What can NFL teams gain from using AI? A play-calling edge and healthier players, some are finding.

MATT MORET: Welcome to Field Study, a sports and science podcast from the Pittsburgh Post-Gazette. I’m Matt Moret.

It should come as no surprise that technology is taking a larger and larger role in how professional sports teams make decisions. Modern baseball, for example, has been transformed by the introduction of advanced analytics.

But other sports have lagged in their embrace of scientific performance analysis. The reality is that humans make mistakes in coaching and training. So what if there’s a better way to decide who plays and who rests, or who spends time in a weight room versus yoga classes?

For this episode, tech reporter Courtney Linder took a look at how the National Football League is turning to artificial intelligence in an attempt to address a host of issues. These systems target human error, and there’s some debate about whether they undermine the sport itself.

We’ll leave that for you to decide, though. With that, here’s Courtney.

* * *

COURTNEY LINDER: Alexa, what is artificial intelligence?

ALEXA: The noun "artificial intelligence" is usually defined as the capacity of a computer to perform operations analogous to learning and decision-making in humans.

LINDER: Alexa, does the NFL use artificial intelligence?

ALEXA: Hmm, I don’t know that.

LINDER: Alexa, how does the NFL use technology?

ALEXA: Sorry, I’m not sure.

LINDER: Alexa, how does the NFL prevent player injuries?

ALEXA: Sorry, I don’t know that one.

LINDER: Alexa, how do NFL coaches make decisions?

ALEXA: Sorry, I’m not sure.

LINDER: Okay, so it may be a bit ironic that Alexa — which, yes, uses AI to give you scores during the game — can’t answer questions about how NFL teams use artificial intelligence.

In her defense, it’s a bit complicated. AI isn’t used widely across the league. And when it is, it can be in completely different areas depending on the football team’s needs. Some teams use artificial intelligence to help prevent player injuries. Others use it to help coaches make better decisions at game time.

And, in many cases, those teams don’t necessarily want to spill the beans and risk their rivals learning their secrets.

I mean, imagine if it came out that the Steelers had an AI system that kept a full record of Antonio Brown’s movements in every game last season and can now use it to optimize their performance against his new employer, the Raiders.

Alexa isn’t going to tell you this, but that’s already reality.

ALEXA: Sorry, I don’t know that one.

LINDER: We’ll let the experts take care of proving that.

* * *

In the heart of Downtown Pittsburgh, one river away from Heinz Field, where the Steelers play, a small startup on Liberty Avenue is working on an AI platform to help NFL teams make better coaching decisions.

That’s a lofty goal, sure, but the company thinks the trick is automating the grunt work — like watching hours upon hours of film — and opening up coaches’ free time to focus on the human side of the operation.

SANJAY CHOPRA: My name is Sanjay Chopra, I am the co-founder and CEO of Cognistx.

LINDER: Sanjay Chopra used to work as a business development head for IBM, the New York City-based company once known for its computer parts. You might recognize the name from your old computer tower, where all of your computer’s data is stored in a hard drive.

IBM has come a long way since then and is now known for Watson. Watson is an artificial intelligence platform that can do quite a bit more than just answer your questions or tell random jokes.

In fact, in 2011, it beat two of the best Jeopardy players in the history of the game.

Businesses use Watson to improve customer service. Consider some of those chatbots you talk to on Facebook.

Despite that background, Chopra says Cognistx moved into working on AI solutions for the NFL only because of some serendipity. He’d been playing golf with some of the Steelers, so he had a unique lens into the team’s issues.

CHOPRA: One of the challenges was, how do the Steelers become better as a team and beat some very other strong teams in the AFC. I can’t name the teams, but how do we get smart about it? Can we apply AI and machine learning and computer vision expertise to get smarter?

LINDER: To figure that out, Chopra and his team did what any NFL coach may do — take a look at some film from prior games. Just one thing: There’s hours upon hours’ worth of data in those videos.

CHOPRA: So it becomes very tedious for human beings — people — to watch this film on an ongoing basis, four or five hours, and you get tired, you miss times. What AI can do very effectively is watch this film for you.

LINDER: This idea sounds like something right out of a football movie, but that’s seriously tedious work.

You don’t have to worry about losing the romanticism of poring over actual film, though, because teams are already able to access every single play from every single game on the computer.

In the 2000s, teams began using software that allowed them to digitally cut up tape and organize it by any type of play they wanted.

Imagine a coach wanted to see how next week’s opponent behaved after converting a third down. All of the film was stored on computer servers, so a video staff simply had to organize the footage.

And if coaches wanted to see how a team responded to a third down with less than 30 seconds left on the clock, they could do that, too. You just had to drill down further into the data — here, the video footage.

Some coaches and even actual players could do this for themselves if they were tech savvy, while other teams were better off doing the backend work for the coaches.

Still, that left an endless pile of data to sort through, an endless reel of clues.

Chopra says that even if you are staring down these specific bits of video, it’s entirely possible that you’re missing something — no matter how seasoned a coach you are.

CHOPRA: We have body posture techniques where we can figure out where the joints are, how far the joints are, where is your hip, how much does the shoulder tilt, the height of the knees, the separation of the knees, the separation of the feet, where the ankles are pointed … the system can now look at multiple frames of information and then begin to analyze and bring insights, which humans might have missed. And now you can use those insights to better train your players.

LINDER: Through this scientific approach, the AI is able to learn about players. If a players’ toes are pointed this way, we know that means he will likely run that direction.

But what does it mean if his shoulders are tilted slightly to the right? That picture may be less clear to the average person.

CHOPRA: So let’s say you’re the defensive coordinator, you can train your linebackers to look for those cues and be ready for the offense.

LINDER: And that’s exactly the type of human problem that AI was built to solve: artificial intelligence is, at its core, all about finding and recognizing patterns.

Ideally, it should use those patterns to come up with predictions.

Think about it: identifying these bits of film or even just watching them can take up a lot of time.

Consider how many more important decisions coaches can make if their free time doesn’t involve so much of this.

Some estimates suggest position coaches spend four hours per day watching film. If roughly half the day is spent with the players, the thinking goes, then the other half is probably spent reviewing footage of their performance to improve it. What if they could have more time with their players?

But here’s where it gets even crazier: AI can even make sense of the coaches’ decisions.

CHOPRA: So based on past games, if it’s a second down situation and long, or if the difference in the score is X, this is what that coach typically does. So we can track all of the data, analyze all of the data, build the right AI models and then give predictions that, hey, it is likely that this particular coach, in this particular situation, will act in a certain way.

LINDER: Don’t panic about seeing robo-coaches, though. That’s not exactly the point.

CHOPRA: Let the data do the talking, and then you obviously need the blocking, and the tackling and your own coach’s sort of mindset on top of that to get the right mix. It’s a combination of man and machine working together for optimal results.

LINDER: Again, this is lofty.

So maybe we should rewind to figure out how it all works.

First, you collect the data.

That’s going to come from the film.

Chopra has a saying about that.

CHOPRA: Typically in AI, we say no data is bad data. The more data you have, the greater the models will be.

LINDER: However, that doesn’t mean all of the information in the videos is smashed into a computer program that spits out valuable information.

You don’t take a bunch of videos and magically get artificial intelligence.

Data must be organized, somehow, to make sense of it.

In artificial intelligence, that usually requires machine learning.

Machine learning is a very specialized type of AI that is able to look at data and make sense of it without being told how to.

In practice, that means a system will consider all of its inputs — which, in this case, are videos — and come up with an output.

For coaches, that output may be some piece of wisdom about how the other team is operating.

Maybe a prediction that the other team’s coach will call a pass play on third down with 20 seconds on the clock.

Things can get messy, though.

Think about all of the film a team has over its history. Do we really care how often the defense rushed ten years ago? Maybe we do, because the team still behaves this way and often wins.

Or, maybe we don’t because none of the defensive players on that version of the team are the same ones on the field today.

Jeff Whitmore, director of business development at Cognistx, says some pieces of data are more valuable than others.

WHITMORE: We don’t want to look back too far from a historical data perspective for training, largely because teams change, teams evolve, make conscious decisions to be different, to be better, to come across more mysterious, perhaps, to their opposing team.

So with that, you know, no data is bad data, but at the same time, if you’re looking at, for example, the Seahawks before Russell Wilson, well that’s going to be not nearly as impactful as looking at Russell Wilson over the course of his career in the last decade or so with the Seahawks.

LINDER: That means the way Tom Brady or Drew Brees played 10 years ago would not be weighed as heavily as how they played last season.

At least, according to the Cognistix system.

That’s great if you’re Mike Tomlin and you want inside the head of Bill Belichick, but is it unfair to use it to your advantage? After all, it’s extremely expensive. Whitmore says a custom AI system, like what Cognistx is building for the Steelers, can cost well into the seven-digit range.

It’s not surprising, then, that Whitmore has faced his fair share of naysayers who think this kind of tech could erase the magic of football as we know it.

WHITMORE: I’ve had conversations with friends and colleagues in general say, "Oh this amount of technology being brought to football, it just, you know it cheapens the sport or weakens it," and i would personally disagree when you have every single halftime report being brought to you by somebody … brought to you by State Farm … the integrity of the sport is not like it was in the 1960s. It’s monetized in every which way and they’re making the most of technology. When people say, "Oh you’re bringing artificial intelligence into professional sports," it’s like, "What’s next? We’re just gonna have robots play the sport?" I don’t buy that. I think that is a little bit ridiculous. But at the same time, if you think about right now, many years ago someone caught on that you can read coaches’ lips. Sometimes when you see coaches speaking, they have their playbook or paper over their mouth so that you can’t actually see what they’re saying or read their lips.

If players or coaches have particular tells that hey, when it’s third-and-long or whatever the case may be when they’re trailing by more than two touchdowns, they have a propensity to behave or take some sort of action. They’ll catch onto that. They’ll say, OK, technology has caught that as a tell, so they’ll adjust accordingly. The game won’t be completely completely different. We’ll see technology further infiltrate into athletics and I think humans will respond and adapt accordingly.

LINDER: And one important thing to mention here, is that the NFL doesn’t allow teams to use this tech in real-time. It’s used for planning purposes to prepare for a game.

After kickoff, though, it’s a level playing field.

The insights you gain from the system, of course, can’t be taken away.

CHOPRA: So you become better at being able to predict what’s happening as a human being because you’ve seen the data. I grew up in India, I knew very little, if anything, about American football. Football for me was soccer. Obviously, living in Pittsburgh for such a long time, I got into the team and knowing the Steelers I got even more into the team, but now having looked at the data and the film and analyzing it, it’s actually making the humans better. Our ability to be able to see things and be able to guess things to see what the model would have said makes you a better person and a better viewer and the same thing is happening from a coaching perspective. And that’s where I think this is definitely going in the future. Coaches, by looking at the data, looking at the models, are also going to become better. And that’s more incentive for teams to be able to do this, because if they don’t, they will be left behind.

* * *

LINDER: On the other side of the country, in sunny Silicon Valley, another startup is using artificial intelligence to help NFL teams with a completely different problem: preventing player injuries.

That’s a competitive space to be in, given that teams lose a significant amount of money to injuries.

According to research from ProFootballLogic.com, a website that tracks and analyzes sports data, players in the NFL have a 4.1 percent chance of being injured during a game.

Injury rates are consistent across positions, for the most part, with the exception of running backs. They face a higher injury rate than others.

Altogether, a player can expect to hit the field 14.2 out of the 16 games each season.

Sparta Science was founded in 2013 to reduce injuries and maximize players’ time on field.

This is CEO Phil Wagner.

WAGNER: I was an athlete in high school, in college, and was injured in quite a few different areas. I was pretty frustrated by the level of guesswork that went on to address some of these injuries, so I really went into coaching and figured, you know what? I’m going to help others avoid this, only to find that other organizations both in the U.S. and where else I worked, Australia and New Zealand, had the same kind of guess-work approach. So I figured, okay, well, I’ve got to learn how people approach death and disease and medicine to really learn the best route to figure out some of these problems.

LINDER: So Wagner attended medical school at the Keck School of Medicine at the University of Southern California.

He quickly realized that one of sports medicine’s faults is what he refers to as the “clipboard approach” to analyzing player health.

Imagine trainers with clipboards standing around players lifting weights and on machines.

A trainer or coach may look at where your hips are compared to your knees when you squat, looking for trouble signs. Then they’ll rate the players’ strengths and weaknesses, quite literally, on a clipboard.

WAGNER: People will write down 1-3 on a clipboard, and if they’re really scientific, they will add up the 1 through 3s on the clipboard to make it even more shiny and "scientific." Then, they might divide it to find some sort of average. So yeah, there’s a lot of confusion on what objectivity means and I think with technology, it’s actually made that confusion harder because, you know, if it’s on an app, then the assumption is that it’s even more scientific, which isn’t the case, it just means it’s digital. And “digital” and “science” are not the same thing.

LINDER: So, just because your Fitbit records your steps and how many flights of stairs you’ve climbed, doesn’t mean there’s any science behind it. It’s still just a pedometer.

Wagner says the clipboard approach leads to subjective screening, meaning that it’s impossible for a trainer to simply look at a player and identify their risk of injury.

WAGNER: People are eyballing it or grading it with a number or saying, “Yeah, you know, your knee collapses when you do this movement.” But what does that mean? Because everybody is so different with the length of their body segments, the attachments of their muscles, their injury history, their ethnicity, their age, the activity they’re doing, that with all those factors, it’s not so easy as eyeballing it and giving it a subjective number.

LINDER: That’s not necessarily a make-or-break for a team, but it does mean that it’s sort of a guessing game when it comes to predicting injuries.

That’s even more difficult when you consider that players, consciously or not, do try to fudge the system. They can overcompensate in practices or while being evaluated by the medical staff to ensure they’re put on the field.

WAGNER: The athletes themselves are always looking for a way to beat the test. And so then our software gets challenged because individuals are going to try to move differently, in a way where they’re gonna, you know, try to compensate in certain directions.

LINDER: In the long run, that’s even more damaging to the athlete. So Sparta Science created a system that could detect these false positives.

Wagner said that in the earliest days of the company, he worked with the military. They, too, tried to trick the system.

WAGNER: The first day we took that in there for the military, they said, “Thank you for this. We’re going to break it.” That was their exact comment to me, meaning that we’re gonna do whatever it takes to beat the software. And it made the technology and the software better.

LINDER: Players interact with the system by stepping onto a force plate, which sort of resembles a bathroom scale. They jump, balance or hold a plank for 60 seconds as data points are collected.

Wagner says players aren’t told much beforehand. That’s intentional — he says the best technology is invisible.

WAGNER: The software uses a lot of machine learning to identify the most consistent parts of their movement and creates a signature saying, "OK, this is how this individual moves. And, relative to others, here’s where he moves better, here’s where he moves worse, here are the injury risks that he has for himself, or herself, and here is the best plan to address those risks."

So the athlete will interact, you know, when they’re done with that jump or that balance, they’ll interact with the data and say, "OK, here’s where I’m at and that makes sense because I’ve been doing this or that, and I need to change my plan accordingly because my body is now different than it was last week."

Because I think one of the challenges is that when people do assessments, sometimes they assume that there’s changes going on, you know, once a year, twice a year, but the reality is, you change everyday. And, you know, how do you constantly hone your routine to make sure you’re always operating at your best?

LINDER: The easiest injuries to prevent are the ones that occur most in a given position, Wagner said. In baseball, it’s about preventing Tommy John injuries in the elbow. With NFL linemen, it’s foot injuries. With wide receivers, it’s about preventing hamstring injuries.

Wagner said the underlying software relies on both artificial intelligence and predictive analytics to make these kinds of assumptions. The two are slightly different.

WAGNER: The difference, I guess from a high level, is that predictive analytics can be done without AI — you export a very large data set and you send it to another individual or have an individual evaluate it in that spreadsheet form to find the relationship and you’re able to be predictive based on that relatively manual process, whereas AI is much more live and dynamic in its insights that it provides, so it’s happening on a daily, or in some cases more frequently, basis. Some of what we’re doing now is still manual, particularly the aspects that we’re relatively unsure of, but the majority of it is automated within the technology.

LINDER: So what is Sparta Science still unsure about?

For one, guesses about far-fetched circumstances like, what happens if you look at different ethnicities?

Wagner wants to be sure his company only puts out information that’s been validated.

Otherwise, the system will identify correlations that actually have nothing to do with athletic performance. And through machine learning, repeat them.

Until Sparta Science can avoid those pitfalls, these outlying situations aren’t automated. Yet.

There’s a ton of other situations to test, though.

The company has already recorded over 900,000 force trials through its partnerships with the military and professional sports teams.

Right now, Sparta Science is working with the Pittsburgh Steelers, the San Francisco 49ers, the Washington Redskins and the Detroit Lions.

Wagner said that altogether, there’s data on at least 25,000 individuals.

Just having a lot of data isn’t enough, though. Similarly to Cognistx, Sparta Science works best when teams remain relatively unchanged. The more player turnover, the harder it is to make predictions.

WAGNER: People don’t recognize that enough, where it’s really difficult to build a database and an evidence-based approach if you have a new coaching staff every two years.

LINDER: Relative to other teams in the AFC North, the Steelers have low turnover in both players and coaches. That’s key to fully benefiting from the Sparta Science system.

Ideally, Wagner said, just about everyone on the team is looking at the data collected. Over time, coaches and trainers will get more familiar with each individual.

WAGNER: Mike Tomlin sees the data, sports med sees the data, scouts see the data, strength conditioning sees the data, with the idea that at the most simplistic level, at a high level, it should be simple enough for everybody to understand — the player, the coach, the doc, the trainer, etc. And there’s layers, right? And certain individuals are going to want to dive in deeper. Tomlin may not be overly concerned about the average rate of force and that really deep layer, he just needs to know how an individual ranks relative to his peers. Yet the medical level is going to want to know the trend of some of those more deeper insights beyond just that initial layer.

LINDER: But if you want to know if Ben Roethlisberger is a better quarterback than, say, Patrick Mahomes of the Kansas City Chiefs, you’re not going to get that from Sparta’s data.

WAGNER: Everyone wants to know who’s the best athlete. And my response is, what’s the best painting? Everybody does it in a different way, in a unique way, and so even if it’s the same position, because there’s running quarterbacks, and then there’s pocket-passers. So how do we basically limit the weaknesses and celebrate the strengths of every individual?

The goal is not to make Ben this amazing running quarterback. It’s to really maximize his rotational abilities as a pocket-passer and make sure that his weaknesses are limited to allow him to do that.

LINDER: But it is possible to measure the financial savings.

Wagner said that college teams using the Sparta Science system have seen their health insurance premiums drop by hundreds of thousands of dollars. Less injuries, lower premiums.

There are still so few teams working in this space, though. Wagner believes there are only one or two other companies using AI to predict and prevent player injury.

WAGNER: The majority of AI that’s being done in sports is based around fan engagement and ticket sales and pricing.

LINDER: Over at Cognistx, Whitmore and Chopra also said that they only have maybe one or two competitors at the moment.

These companies are drastically different, but both rely on a culture shift in order to see success in the sports technology market.

What they really need is trust.

WAGNER: These processes don’t happen by themselves. You’ve got to get a player to actually assess themselves and sign off and agree to the data being collected. And so, the environment is so intense and stressful that it’s almost prohibitive for a lot of companies to gather enough consistent data to have a way to implement any AI and machine learning.

LINDER: He says it’s less about the players, though, and more about the organizations.

WAGNER: It’s the organizations that really have to educate and assure the player that, look, we’re not going to use this data to cut you or make bad decisions. We’re trying to help you extend your career. I think players tend to be fearful. Like if you rush for 1,500 yards, the reality is, if you’re injury free, no one cares about your force profile because you just ran for 1,500 yards in the season.”

LINDER: The tech industry as a whole has had a reckoning with privacy concerns.

The last thing a player wants is private information about their practice performance or their injury risk to leak out.

And with data breaches on the rise, it’s clear that hackers have incentive to try.

Last November, Marriott hotels had up to 500 million accounts breached.

Question-and-answer website Quora reported up to 100 million of its 300 million users may have been impacted by a December data breach.

Facebook came under fire in March 2018 when 87 million accounts were breached.

For his part, Wagner thinks this is an individual concern, whether you’re an NFL player or not.

WAGNER: I think the NFL is pretty renowned for paranoia. There is some concerns about, I think, security, and I think the only really vetted concerns are those that revolve around health care and HIPAA and probably some of the privacy issues our neighbors Facebook have brought to light. So I think there’s a general individual privacy concern, irrespective of necessarily sports or the NFL. I think there’s a legitimate concern there about who owns the data and who has access to it.

LINDER: So all of that being considered, when might we expect to see every NFL team using technology like Sparta Science?

The short answer is we don’t know.

The long answer? When we see a culture shift within teams.

WAGNER: I don’t think people talk about the human element enough in technology and that’s the real limiting factor when we talk about timelines. Because there’s not really a lot of technology in sports medicine in the muscloskeletal space, whenever you introduce new technology in any sector, the people that have never been measured before are not really excited to start being measured … the Steelers staff is always welcoming of it, they see it as a chance to learn and better the craft, but that’s not the norm. So I think that the limiting factor will be the top-down is going to have to explain to sports medicine, strength and conditioning that, look, you’re going to get on the same page as the rest of us. It may not be wins and losses, but you’re getting measured now. Or it’s going to take those individual practitioners to get comfortable that there’s now metrics where their outcomes can be seen.

* * *

MORET: That does it for the second episode of Field Study.

We’d like to thank all of the guests who helped with this episode. I produced this episode and Ryan Winn helped out with scripts.

Check back for the next installment, which is about the power of loud grunting and why athletes outside of tennis may want to embrace it. Trust me, it’ll be a fun one.

Thanks for listening.