Was IEM Rio the most upset-prone Major of all time?

It's fair to say that IEM Rio was a tad short of the competition that fans were hoping to see. Favorites dropped like flies as unlikely candidates rose up the ranks to take places in the playoffs. G2 and Astralis didn't even make it to Rio to compete, discarded in the European Regional Major Ranking competitions. Cloud9 and Vitality, two of the highest-rated teams in the event, were almost eliminated in the Challengers stage. It was only thanks to Imperial blowing a 15-9 and 15-6 lead that would allow the Russians to find their way back into the Major.

Safe to say, it was an absolute mess of a tournament, competition-wise. The playoffs had only two of the top five ranked teams in the world, number four and five, to be exact. It was an upsetting Major, both for the teams who were eliminated and in terms of actual upsets as well. But was it the most upsetting Major of all time?

Throughout the event, community members critiqued the Major as a sort of "PGL Krakow 2.0", a Major that similarly saw favorites drop out early and a Major final between two teams who weren't believed to be deserving of the grand final stage. Well then, how does Rio compare to Krakow, and while we're at it, how does it compare to every other Major?

What makes an upset?

It seems like a simple question with an easier answer: an upset is when a worse team beats a better one. It's hard to qualify which team is "better" than another, though. After all, if team A team beats B, who's to say that team B is the better team?

The best recourse we have to this quandary is the HLTV rankings, which have been continuously updated week after week since October 2015. Of course, there were Majors before October 2015, but since there are no continuous rankings from all the way back then to the present, we can only analyze the Majors from Dreamhack Cluj-Napoca onwards. Still, that gives us a sample size of twelve Majors spanning eight years, more than enough to gather interesting data from.

So then, we can say that an upset is when a lower HLTV-ranked team beats a higher-ranked team. Let's count how many upsets, by that definition, occurred in each Major. In addition, we should divide those by the number of matches in each Major, because earlier Majors had fewer matches than more recent ones.

By this definition of an upset, IEM Rio was in fact the most upset-prone Major of all time (at least, that we can calculate), even more so than PGL Krakow. In 45%, or nearly half of all matches, the lower-ranked team proved victorious. This provides a great deal of explanation as to why the playoffs contained such interesting teams. Highlighted on the graph as well is MLG Columbus, the event with the lowest proportion of upsets—just under 25%. At Columbus, four of the top five teams were in playoffs, paving the way for incredible matches in the final straight of the Major.

It is interesting that even at its lowest, the proportion of upsets is above 25%. That is to say, you could expect the lower-ranked team to win at least 25% of the time. This sure is interesting, but it overlooks a very important facet of upsets that cannot be forgotten: the severity.

When the 17th-ranked team defeats the 16th-ranked team, it can hardly be considered the same situation as if that same team took down the best team in the world. Currently, by this definition of an upset, those two situations are treated the same, when they could hardly be more different. To alleviate this issue, let's go about calculating upsets in a different way.

Rank Differential

Let's introduce a funky new concept: rank differential. To calculate rank differential, you go as follows. Find the ranks of the two teams, and subtract the loser's rank from the winner's. That's it. If the #1 team beats #4, you subtract four from one and obtain a rank differential of -3. If, on the other hand, #4 beats #1, you get a rank differential of 3. So, negative rank differentials represent games where the favorites win, and positive rank differentials represent games where the underdogs win.

By calculating upsets this way, we get a much more accurate representation of the severity of the upset that occurred. Let's take our previous example, for instance. If the 17th-ranked team beat the 16th-ranked team, that would yield a rank differential of 1. If they beat the number one ranked team, that would then be a rank differential of 16, quite a difference.

We can add up all the rank differentials of all the games in each Major, and see their totals, adjusted for the number of matches in each Major.

In this graph, the lower the bar, the more negative the cumulative rank differential. That means that the longer the bar, the fewer upsets occurred, and the shorter the bar, the more upsets occurred.

By this definition of upset, PGL Krakow takes the cake for the most upsetting Major of all time (that we can calculate). While the event was still slightly favored for the higher-ranked team, victories from the 15th-ranked Gambit helped drag PGL Krakow to the top of this list. On the flip side, the FACEIT London Major had the highest cumulative rank differential, where overwhelming favorites Astralis took their second Major win.

Rio, in this graph, sits in second place. By rank differential, then, Rio is almost the most upset-prone Major of all time, but not quite.

Another method

This method of calculating rank differentials is good at tracking the upsets that occur throughout the tournament. However, this leaves out another crucial facet: you can only play who's in front of you. If the fifth-ranked team in the world won the Major, we would certainly consider that an upset, but if they only faced teams ranked worse than them, their games would count towards a negative rank differential in the previous calculation.

To solve this problem, let's calculate another measure, one inspired heavily by NER0cs' HLTV article on home crowd advantage at Majors. We compare the initial rankings of each team to their final placement at the Major, and calculate the sum.

To explain this, let's go back to our trustworthy example of Gambit's run at PGL Krakow. They came into the Major the 15th ranked team in the world, but because some higher ranked teams missed out on the Legends stage, they were the 12th ranked team at the event. As history knows, they finished in first place. So, the absolute difference between their initial ranking, 12, and their final placement, 1, is 11. We calculate this for all 16 teams at the Major and add it all up.

We do this for all the teams at every Major (since 2015), and we are given this graph below.

As observed above, FACEIT London continues to be the least upset-prone Major, while IEM Rio and PGL Krakow are tied for the upset crown. Rio and Krakow had by far the biggest difference in the ranks coming into the tournament and the placements at the end, meaning they had the most surprising and upsetting results.

If we restrict this to just the top eight finishers, so just the teams in the playoffs, we can look at the most surprising playoff brackets of the Majors.

Here, it comes as little surprise that we see Starladder Berlin shoot to the top. With Avangar, the lowest-ranked team coming into the tournament, making the grand finals, and Renegades making the semi-finals, that tournament had playoff teams that nobody envisioned. FACEIT London still takes the cake as the least upsetting Major though, and Krakow and Rio are not far behind Berlin at all.

Conclusion

None of these measures stand alone. We must use all of these graphs in conjunction to create a narrative that will lift the fog on something as abstract as "the most upset-prone Major". So then, is Rio the most upsetting Major of all time? It depends on how you measure it, but if it's not first, it's certainly second. Analysts and critics called IEM Rio "PGL Krakow 2.0" from the eye test, and well, the numbers certainly back that up.

The one and only thing I can say for sure about IEM Rio is that I'm upset that Liquid didn't win it all.