The Secret of My eSports Success

Esports has arrived as a major player in the sports world. Games like Dota 2 and League of Legends have hundreds of millions of players, and the best gamers have fan bases and endorsement deals right up there with the stars of “real” sports. As esports grows, so do the analytics surrounding it. But while the nature of esports means that the amount of quantitative data for every game is staggering, the volatile nature of team building and managing in the sport only increases the importance of people analytics and how it leads to success. We explore the role of social science analytics in esports with the head of Shadow and one of the leading voices in esports analytics, Tim Sevenhuysen.

Transcript:

Ben Shields: Hi, it’s Ben. We’d like to ask you a favor. If you enjoy our program, please take two minutes to rate us and, even better, post a review on Apple Podcast or wherever you get Counterpoints. It makes a big difference to our discoverability, and we’d actually like to know what you think. On to the show…

Last summer at The International, the premier world championship of the esport Defense of the Ancients 2, which had a total purse of over $25 million, team OG was nothing more than an afterthought. Not only had the European team disappointed at the tournament the previous two years, but they hardly resembled their once impressive squads. Two of the team’s best players had defected to a rival unit just months prior, and OG’s semiretired coach had to step into one of the spots. The big shake-up meant that OG had to go through a gauntlet of qualifiers just to make it back to The International, but that allowed the team to build chemistry and strategy — so crucial in a game like Dota 2. Huge underdogs coming into the tournament, the new-look OG did what the more vaunted iterations of the team could not. Against all odds, they took home the title.

Paul Michelman: Unless you think that lightning doesn’t strike twice, China’s League of Legends Pro League has recently seen a miracle run of its own. JD Gaming wasn’t rocked by defections like OG, but they were the No. 8 seed in an eight-team playoff bracket where the top squads might as well have been written in Sharpie, but the gamers at JDG knew how to play together and how to play their opponents. They had lost to their first playoff challenger on purpose late in the regular season to gain insights into their strategy, and then they successfully turned the tables when it mattered most. JDG’s status as a non-threat meant they were underestimated by the Goliaths, and during the pre-match draft of characters — always a key strategic battle — the team was able to pick up the pieces they needed to implement their winning formula. The end result: The lowly 8 seed got all the way to the final, and though they didn’t win it all — they, like OG before them — have provided a blueprint of chemistry and strategy that can give any team a shot at the title. I’m Paul Michelman.

Ben Shields: I’m Ben Shields. And this is Counterpoints, the sports analytics podcast from MIT Sloan Management Review. In this episode, we’re exploring the burgeoning world of esports analytics — not just the numbers but how social science might be the biggest difference maker in the game.

Paul Michelman: Esports has arrived as a major player in the sports world, and it’s growing every day. Games like Dota 2 and League of Legends have hundreds of millions of players, and the best gamers now have fan bases and endorsement deals right up there with the stars of “real” sports.

Ben Shields: As esports grows, so do the analytics surrounding it as teams look to find competitive advantages in their combat strategies. But while the nature of esports means that the amount of quantitative data for every game is staggering, the volatile nature of team building and managing in the sport only increases the importance of people analytics and how it leads to success.

Paul Michelman: Could social science be the area in which the underdog esports teams of the world can level the playing field against their more-skilled counterparts? This week, Ben speaks with the head of Shadow and one of the leading voices in esports analytics, Tim Sevenhuysen.

Ben Shields: All right. It is a pleasure to welcome onto Counterpoints Tim Sevenhuysen, head of Shadow. Tim, great to have you on the show.

Tim Sevenhuysen: Yeah, thanks so much for inviting me up.

Ben Shields: Well, we’ve got a thesis here that we’re going to dig into here in a minute about social science holding the secret to success in esports. But before we do that, [I’d] love to set some context. And for our listeners out there that may not be as familiar with esports, can you define it for them?

Tim Sevenhuysen: Sure. So esports is effectively taking video games or computer games [and] playing them competitively at a professional level. So it’s taking a game that you may have heard of, or may not have heard of — something like League of Legends or Counter-Strike or, in some cases, NBA 2K — and just applying that to a more professional structure. So [it’s] having organized teams, organized tournaments, [and] organized competitions that have structure and rules and prize money and trophies, and putting all of the kind of standard sports infrastructure around computer games or video games.

Ben Shields: And what is the rough size of the industry today?

Tim Sevenhuysen: Yeah, the industry has really exploded in the last five to 10 years, especially the last five years or so. Hard to put an actual number on it, because once you get to a certain number, you see these market research companies come out with: Oh, you know, the industry’s worth this many hundreds of millions or this many billions. And once you get into that range… 100 million sometimes sounds bigger than a billion. And you know, you start to just feel like: Well, it’s really, really big.

Ben Shields: Well, maybe that’s an episode for another day — actually quantifying the esports market. But for today, we’re going to focus on esports analytics. And can you give us the 101 of esports analytics and how specifically gamers use analytics to improve their decision-making?

Tim Sevenhuysen: Yeah, absolutely. So when you talk about kind of applying analytics or applying data analysis to esports, there’s a range of things. You can talk about kind of the everyday player at home, and they’re just playing online with kind of a random set of teammates and against a random set of opponents. And the way those people will use data [is] maybe [for] things like: There are different characters you can select to play as in different games. Or different strategies you can use. And understanding what are they most successful on and let’s play more of that, or what are they struggling with and let’s practice more of that. So there are different public tools you can have. The companies that make the games may provide data through their own APIs that they support just to benefit their player base. So there are things at that level that are relatively shallow. Just imagine if you are kind of a recreational baseball player, and you had some website that told you what’s your strikeout rate. You know, everything from that all the way up to the professional level (which I’m more personally involved with), where you have the professional teams with all of their infrastructure of coaches and analysts and people supporting their players. And the way they use analytics is obviously pretty different. They will look at things like scouting their upcoming opponents. They know which teams they’re going to play next weekend. They’ll look at which types of strategies did they play most often and are most successful with so [they] can kind of game-plan against them…. There are certain elements of a team’s practice that may be public when they play on kind of the normal online servers. And you look at those and say, which characters are they using? Which strategies are they using? And try to kind of glean some insight from that for your game planning. They’ll use analytics to scout upcoming talent and see [for instance] here’s a player who hasn’t played professionally before, and in the last six to 12 months they’ve really made a big climb up that competitive ladder. You know, what kind of a player are they? Are they someone we should be recruiting? So it goes all the way from modeling opponent scouting, roster building, identifying the best strategies to use in the game overall at a given time — a whole range of things.

Ben Shields: It’s interesting. You definitely see some of the parallels between esports analytics and traditional sports analytics, but I have to imagine that there are some differences, too. What are some of the unique challenges in esports analytics that may be different from traditional stick and ball sports?

Tim Sevenhuysen: Each individual computer game is going to have its own kind of aspects of how can analytics be applied, what level of data is available to work with — all of those things. So within any given game, the main difference tends to be that a computer game is typically much more complex than a traditional sport, which is not to say that traditional sports are not complex — they’re very complex games at times. But when you look at something like comparing baseball, which is a very kind of start and stop game, very individual discrete actions — you know, you throw a pitch, is it a strike, a ball? Did they swing or not? You know, you can measure and count all of these actions that take place. But you compare that to the average esport. So the game that I work in most closely, personally, is called League of Legends. It’s a game with five players on each team fighting all across a map. And the ultimate objective is to destroy the main base structure inside the enemy space. So to do that, you have to destroy a bunch of other kind of defensive buildings along the way, and you have to fight against the enemy team. And there’s action happening all across the map. It’s continuous. There’s no kind of start and stop, do something for one minute and then take a pause and do something else, like the equivalent to throwing a pitch and now resetting and throwing the next pitch. So it’s constant action. There’s no individual kind of ball or kind of an object of play like there is in most traditional sports where you’ve got the baseball, you’ve got the hockey puck, whatever it is. In traditional sports analytics, most actions that are counted and analyzed statistically relate to how that object of play — how you’re interacting with it. There are a couple of esports that are kind of like modeled after — so there’s a game you may have heard of called Rocket League that is modeled somewhat after soccer. You’ve got a ball, and you’re trying to hit it into a goal. The only difference, very small difference, [is] you’re using rocket-powered cars that fly all around the air.

Ben Shields: Sounds fun.

Tim Sevenhuysen: But other than that where it’s clearly modeled after soccer, you have these games that are fundamentally very different. So the number of actions you can count and measure and analyze is infinitely greater, because you’ve got action happening at so many different points all at the same time.

Ben Shields: So isn’t that an advantage for esports analytics? The fact that everything can be quantified?

Tim Sevenhuysen: It depends on the level of data access. So this is a very interesting structural… In a traditional sport, most things you can see happening with your eyes. And in theory that’s something you could write down or have some technology that can record that and put it into a computer database. In an esport, those actual actions are effectively owned and are the intellectual property of the company that made the game. Major League Baseball does not own the fact that a player swung the bat and missed the ball. But beyond the fact that somebody watching a video of an esports match could in theory write down some of those public things, a lot of the other data you’d want to track, such as this player attacked the other player and did a certain amount of damage — that’s not something that you could see with your eyes, even though the underlying game is recording that and might make it available. But if you’re going to access some API that gives you that action and that measurement, you’re only allowed to do something with that data if the company gives you permission to do so. They also have the structural control over how much of that data is actually even available to be picked up by your systems and made available through an API. So there’s a lot more restriction even though the games themselves are digital and the data might be in theory possible to gather, there [are] basically business elements on top of that, that make it challenging at times.

Ben Shields: I see. So that’s really interesting. So whereas in the NBA where all teams have access to the second spectrum data, the question is what you do with that data in order to gain a competitive advantage. In esports, what we’re saying here is that there’s still a lot of data that’s out there that isn’t necessarily made available to teams and players. And that’s a constraint.

Tim Sevenhuysen: Yeah, for sure. And I think depending on which game you look at, some of them are moving in that direction of trying to provide all the teams with a baseline of data to work with. Some of them don’t really have the professionalism or the infrastructure or the will to kind of make that available in that way. And so it creates a very different landscape from esport to esport of what kind of data is there to work with.

Ben Shields: So that brings us to the thesis that you’ve come on to defend. And it’s interesting because again, coming into this show, many of our listeners may think, wow, you know, it’s such a digitally driven game. There must be data everywhere and that can be easily analyzed in order to improve performance. But you’re sort of coming on and you’re saying, well, first of all, there’s some data access issues. But more to the point, you are arguing that social science and not necessarily hard-core data analytics holds the secret to success in esports.

Tim Sevenhuysen: Right. So this all comes down to the complexity point, I think, at root. Let’s say that you could capture and track and count every action that happens within a game. But even being able to do so, the full kind of quantitative approach to sports analytics really relies on the fact that if you take a similar action at a different time, you can make comparisons between those and say: Was this a successful action? Was this an unsuccessful action? Baseball really is the epitome of this, because like I said before, you can count pretty much every single action and movement and compare them against each other, and you’re going to get thousands or hundreds of thousands of data points across time of a specific type of pitch thrown in a certain way, and [then] see how often does that produce a strike or whatever it is. In most esports, and especially in a game like League of Legends, you have this intense complexity [and] the same action can’t be modeled the same way at different times.

So there’s something I call the snowball effect. Over the course of a game of League of Legends, your characters increase in power, and the more successful you are, the faster you increase in power and gain advantages over the other team. So if you kill an enemy character, you gain a gold reward. You can use that gold to buy an item that makes you stronger, which means that if you get an early kill against them, you are objectively stronger than they are, which means the next time you get a kill, it might be [that] you wouldn’t have gotten that kill if you weren’t already stronger.

So you can compare that to something like in baseball [if] every time you got a hit, your next hit would travel 3% farther than the previous one. Obviously, if that was the case, you couldn’t just model how good of a hitter this person is. You’d have to understand at every single hit what kind of modifier was applied to how far their hits were traveling and… build it into all your modeling. It creates this really intense complexity that reduces the ability to put the game into a statistical model and have something that’s really reliable and really captures all of the subtleties. So when we come to the point of what social science — and what I really mean there is kind of pulling back from the pure quantitative approach to analytics and starting to mix in more qualitative and other methods there. It’s really because the more complex an environment is, the less you’re able to really capture it in a purely quantitative way and the more you need to kind of build up a further pyramid of evidence underneath.

Ben Shields: OK, so let’s dig a little bit deeper here. Is there a research problem that you can cite currently and a qualitative research method that you think would be appropriate to help better understand that research problem?

Tim Sevenhuysen: Yeah, for sure. So to lay a little bit of a framework for this, I tend to look at research and analysis as kind of having three phases to it or three different types. So you’ve got exploratory research, which is something where you go into it knowing [you] don’t really understand anything about this question, and [you] just need to understand even like what are the boundaries of this research question. Usually that would be a really good case for qualitative research. You have explanatory research where you’re starting to get the shape of the question, and now you’re going to try to explain like, “Hey, we know that there are these different kind of types of patterns in place or different behaviors or different actions. Now let’s try to explain how they work.” So this is a good kind of a mixed-methods kind of research problem. You can apply some qualitative, some quantitative, and work them together. And then once you’ve really kind of outlined the problem well, that’s where you can really come in with your quantitative analytics with something more like evaluative research. An example here would be… You know, going into esports, the coach or the analyst is going to have a lot of intuitive knowledge about the right way to play the game, what they think are the correct actions to take. Looking at League of Legends, there’s a position called the jungler. This is someone who kind of roams around the map very freely, that kind of helps out their teammates at different points, and is all about kind of moving in places where the enemy team can’t see them, and then jumping out and taking surprise actions in different places.

If you are going into this with just your preexisting assumptions of how a jungler should play and what they should do, you may kind of miss certain things. And as the game evolves, you might… Everybody can be kind of a slave to their assumptions, right? So applying qualitative research to that, you can step back and say, “Well let’s just drop all the assumptions we have. Let’s observe how do junglers actually play, start categorizing the different types of actions they take, and then just creating that model around it.” I can compare it to, say, a hockey analyst coming in and wanting to understand… You know, first they recognize there’s something… they could call a zone entry. So how do you get the opponent into the offensive zone across the blue line? And categorizing: OK, well there are plays you can carry the puck in. There are plays where you can dump the puck in at forecheck. There are plays where you shoot it at the boards and try to retrieve it, just with better timing, [and] some more advanced plays beyond that. And then you get into the kind of the mixed-methods approach. The explanatory side where you say: “OK, well maybe we dump the puck in if we feel like our players are really good at kind of forechecking and winning puck battles on the wall. Maybe we carry the puck in if we know that our team is faster and has more skilled stick handling than the other team does.”

So in League of Legends, it’s a similar kind of thing. You’d start to understand: When does the jungler — to throw in some more advanced esports terms — when do they power farm? When do they gank? When do they invade aggressively? When do they invade protectively? You know, going into all these things that won’t make any sense to somebody who doesn’t know the game. But the point being of why you have to do this in League of Legends is that the game itself evolves over time as well. The game is incredibly complex to start with, but then also the game changes every few weeks as the company releases balance changes, makes certain characters a little stronger, makes others a little weaker. And suddenly that changes which strategies are most effective at different times. And it’s a constant relearning of the game.

Ben Shields: And theoretically, just to go with this jungler example a little bit farther, are there a core set of skills that a jungler has that are going to be relevant regardless of what the game rules are?

Tim Sevenhuysen: Yeah, absolutely. So that’s where when I talk about this kind of continuum of exploratory, explanatory, and evaluative kind of researcher analysis, the point isn’t that you always need to be reexploring, because at a certain point you’re going to recognize: OK, there’s this maneuver called ganking (a gank is basically a sneak attack) that’s always going to be relevant in certain times and in certain ways. There’s going to be farming, which is just gathering up resources to make yourself more powerful; invading, which is going into enemy territory. These are concepts that are fairly standard and haven’t really changed over the last six to eight years. If you never go back and reexplore those concepts, you won’t understand the subtle ways that they change, the subtle ways that the times you apply them change. And then helping the players to understand: Look, we’ve always done ganking, but look ganking has changed. Here’s how you need to do it a little bit differently now. Let’s re-explain the way that works.

Ben Shields: I think one of my key takeaways already from this conversation is I think I might want to be a jungler. But in addition to that, I see what you’re saying. Especially, void of the rich data set about a jungler in this particular instance, to understand the key success factors, observation of the jungler and exploration over time on how to improve that performance profile is critical. So I’m understanding the social science piece here as well. I want to move from the individual role player to the team context and explore how social science methods could be effective in understanding what makes a strong esports team.

Tim Sevenhuysen: Yeah. Just as there are individual kind of tactics for a specific player to apply and the need to make sure that they know how to enact those tactics, it’s even more important and even more complex at the team level. How do these five players work together to achieve a certain overall goal? So in League of Legends, you pick which characters you’re going to play — there’s a roster of over 130 champions (they’re called). And the mix that you choose is going to very heavily influence which strategy you’re going to use. Some of them are very good early in the game — they’re very strong in the first 10 or 15 minutes before everybody else has kind of powered up. Some of them are very weak early on and are very strong once you hit kind of the 30- to 35-minute range, just because of the way their power kind of scales over time. And the team needs to understand as a whole: Which strengths did we choose with these characters? What does that mean for how we have to act at this moment of the game? What does that mean for what goals we’re trying to achieve? You know, what are the wind conditions? What’s the ideal situation for… our set of champions or our composition of our team… to succeed in this game? So the coach is responsible for making sure that they have a clear way to use these five champions [and] that the team all understands it the same way. And in applying kind of analytics to that, you started to see some of the complexity. It’s kind of like with line changes in hockey. When you put out your really skilled, like your top line, the way they play and the way they understand that they need to play is going to be different than when you put out, say, your checking line — you know, very stereotypically, kind of the third line that’s out there to play defensively and lock down the opponent’s top line. The way they understand and have to play, and the analytics you would apply to whether or not they’re being successful in their role, is going to be quite different.

Ben Shields: And the way that you would do that, the method that you would use to understand the effectiveness, would be more of a social science method. So for instance, would you study that through more of an ethnographic approach? I’m trying to draw this distinction between understanding team effectiveness purely in quantitative terms versus the thesis of using more social science methods. So what type of method would you use in order to understand the effectiveness of the team in that scenario?

Tim Sevenhuysen: Yeah. So I think two of the main frameworks in qualitative research would be ethnography and then grounded theory as well. I think both of them are relevant here. So I’d say ethnographic methods would be applied a lot to teaching and understanding of the individual player in how they act within a certain scenario. So if you can understand the jungler’s role within this team, within this situation, in this game… You know, we have agreed and understood that these are the actions they need to take for the team to be successful. And you understand from the jungler’s perspective, therefore they’re making these certain decisions. This is why they did this thing at this moment. And now we can kind of evaluate it and say whether or not that was a successful decision, whether it had a successful outcome. Whereas if you just tried to apply that statistically, especially if you tried to apply it without building all of the foundations first, it’s going to be very difficult, because you don’t even know what your standards are for whether something was successful or unsuccessful. So you have to really get in the mind of players and the team overall in the same sense and understand: Do we agree on what are the different types of actions that they could be taking? What is leading to the decisions they’re making? And then how are they evaluating the success of those actions?

Applying grounded theory I think in some ways [is] a little more clear if you understand grounded theory to be an approach where you take away all of your existing assumptions and you just start gathering raw data, and you gather such a volume of raw observations and raw data in the sense of: This person did this thing. They made this decision. They took this action. Now let’s start to group these things similarly together, code them, apply labels. Say this situation was the jungler ganking, and now it had the outcome of it forced the enemy to use some resources they didn’t want to use, or it generated a kill or generated some other kind of advantage. And coding all of those actions, helping you to understand — very much at the exploratory stage — what are the actual range of actions, the range of outcomes, applying success or nonsuccess labels to them. And the entire point here is to be systematic and rigorous about it, to intentionally put your assumptions aside.

There’s often an approach in both sports and esports analysis that you just watch a very large volume of games: Hey, I’ve been doing this 20 years, I’ve watched 20 years’ worth of games, and I know what’s good and what’s bad. But [in] the qualitative research approach, you would be taking those preexisting ideas, taking that built-in expertise and putting at least some of it aside and building up a really rigorous approach.

Ben Shields: Yeah, I think that’s a very fair point in what we’ve seen with scouts in traditional sports that certainly have tremendous experience but in the new analytical era are applying even more rigor to their observation. So I think that’s a really nice analog to what we’re seeing in traditional sports as well. All right. I want to bring us full circle here for our final part of our conversation and ask you if, in fact, the games like League of Legends open up their APIs and give away most if not all of the performance data, does that change social science’s role in helping unlock success for esports teams?

Tim Sevenhuysen: I would say that obviously it would have an impact, and it changes certain things. If you’re able to effectively capture and measure deeper data and more granular actions than we currently are, obviously that’s going to affect the types of modeling you can do [and] the types of outcomes you can track. But at the fundamental level, I don’t think it would change the actual value of different types of approaches to analysis.

The reason why a more qualitative approach, a more kind of mixed…. Really, I don’t want to make this seem like it’s all about being qualitative. It’s really about being mixed methods, right? It’s about applying the right scenarios to track things statistically; the right scenarios to kind of set that aside and gather rigorous kind of observational data. That approach and that need to kind of step a little further back on the continuum into the explanatory range compared to something like baseball, which can be very evaluative, very quantitative. That need comes from the complexity of the games. It comes from the constant changing nature of the games, that these are games that have very frequent rule changes compared to most traditional sports. They might have a couple of rule changes each off-season, so once a year. [It’s] incredibly rare to have any kind of rule change mid-season. And those rule changes are pretty minor, right? The games mostly evolve something like the classic example: the corner three in basketball. There were kind of some structural things that led to that, but it’s mostly about evolution within the behavior of the players and the teams themselves. Whereas in most esports, you’re being forced into change because the game itself is changing over time. Because every two weeks to a month, some set of rules changes, some new character comes out that can be played, and it has knock-on effects to everything else. And so you’re constantly in this reactive mode, this adaptation, this reevaluate, this re-explain…. What are the ideal strategies? What are the best ways to play? And that’s why I think kind of the mixed methods or the qualitative approaches are always going to be more valuable in most esports compared to the need for them in most traditional sports.

Ben Shields: Awesome, Tim. That’s a great place to wrap up our conversation. You’ve got a cool job. You’re working in a really fascinating industry, and we appreciate you coming on our show and sharing your insights with us.

Tim Sevenhuysen: Yeah. Again, I really appreciate it. I think it’s always fun to talk through these things and draw the comparisons and identify the differences in what we’re doing, and try to learn from traditional sports, but hopefully also come up with some things that honestly I think could be applied in traditional sports and affect the way they do things as well. So yeah, it’s an exciting space.

Paul Michelman: This has been Counterpoints, the sports analytics podcast from MIT Sloan Management Review.

Ben Shields: You can find us on Apple Podcast, Google Play, Stitcher, Spotify, and wherever fine podcasts are streamed. And if you have an idea for a topic we should cover or a guest we should invite, please drop us a line at counterpoints@mit.edu.

Paul Michelman: Counterpoints is produced by Mary Dooe. Our theme music was composed by Matt Reed. Our coordinating producer is Mackenzie Wise. Our crack researcher is Jake Manashi, and our maven of marketing is Desiree Barry.