Grace Teng

Speedrunning and the Scientific Method

Dated: Jun 16, 2018

In a previous post, I wrote about speedrunning and I asked myself why I found it so enthralling.

Well, I should clarify that. I’ve watched a couple of Pokémon speedrunners grinding runs, which was supremely uninteresting. I watched pokeguy84 swear and reset repeatedly, and dropped in on an Exarion stream right after his main Pokémon died to a critical hit and the very promising run died with it. The only sound on the stream was the Pokémon battle music playing over and over. A viewer typed in Twitch chat, “He’s not coming back, is he?” Exarion did come back, and he did attempt a few more runs, but none of them were as promising as the dead run had been.

Huh, I thought. So this is speedrunning — long stretches of grinding boredom, punctuated by a few flashes of elusive euphoria.

When I wrote my last speedrunning post, there was no real insight to it. Obviously there was a link to be made between speedrunning and competitive sport, but to my mind, they differ in one key respect: there is a very clear reason to play sport besides the simple desire to be elite at something, or even to simply improve at something. Being able to run farther or lift heavier loads or respond faster to external stimuli has clear advantages outside of the arena of sport. I can’t think of any specific skills in speedrunning, whether mental or motor, that are in any way comparable. (Of course, I don’t speedrun, so perhaps there are real mental advantages that speedrunners develop that I can’t see from here.)

I sound like I think speedrunning is a waste of time, which is not the case. It’s worth repeating that my question is not: “Why do people speedrun?” My question is, “Why do I find good speedruns compelling?”

I was still pondering this question when I decided to watch a video by Summoning Salt, about the most infamous level in Super Mario Bros:

4-2: The History of Super Mario Bros.’ Most Infamous Level, by Summoning Salt

Summoning Salt is a real historian of speedrunning, and a really good storyteller, able to carve out the broad arcs of each game’s story while clearly explaining the little details that motivate each breakthrough. I quickly became a fan. I watched his Super Mario Bros video, then his Pokémon Red/Blue video, then his Sonic 2 video, then his Metroid and Super Metroid videos, then his Portal and Half-Life 2 videos, Donkey Kong (which is something special), Mike Tyson, Legend of Zelda …

And I started to realise, the real comparison here wasn’t speedrunning and sport, it was speedrunning and science.

The Incremental Nature of Science

As a kid, the stories of scientists that I read about tended to focus on the big breakthrough discoveries. I was especially interested in physics, and the history of early 20th century physics feels like it turns on a few major personalities: Max Born, Max Planck, Marie Curie — wait, how about this:

A photo of attendees of the Solvay Conference in 1927, including Schrödinger, Pauli, Heisenberg, Dirac, de Broglie, Born, Bohr, Planck, Curie, Lorentz and Einstein, among others. Attendees of the Solvay Conference in 1927.

This was my impression of what physicists did: they made earth-shattering, groundbreaking discoveries that altered the landscape of human knowledge.

Only in college, when I was taking classes in linguistics and found myself loving the discipline, did I start to understand that this vision of science wasn’t the full picture. The paradigm shifters in any field get all the glory, but science is primarily incremental in nature: for every scientist whose name goes in the history books, there are a thousand others working on the little advances that make the big ones possible.

Speedrunning as a scientific endeavour

The process of speedrunning is, in many ways, similar to the process of science. It isn’t simply about playing a video game as fast as possible. Most of us, when playing video games, are content to live in the virtual world that’s placed before us and take it at face value, but speedrunners — they want to push their understanding of that virtual world to the absolute limit.

In an interview on the speedrunning podcast My Insane Pace, Pokémon speedrunner Shenanagans divided the Pokémon speedrunning community into three groups: the glitch hunters, the routers and the runners. I’m writing here as a total outsider, but I doubt these are strict categories: I imagine most Pokémon speedrunners will straddle two of these categories to some degree. The glitch hunters find ways to break the game that are useful to speedrunners. The routers are the theorycrafters who use their knowledge of the game (including any glitches that are allowed) to plan the fastest route through the game, and the runners are the ones who actually execute the routes.

Shenanagans was talking about Pokémon, but it’s clear from Summoning Salt’s videos that this is true of all speedrunning communities. The first runs in any game are usually superbly-executed playthroughs by highly skilled players, similar to what a casual player might do except highly refined and nearly error-free. At this stage, the runners are the dominant force. They bring the target time down, until someone does a run that is so flawless, so optimal, that the possibilities of the route are exhausted.

Routers: The Engineers

At this point, the routers take over. They examine their assumptions about what the fastest path through the game is, and work to cut out any parts of the game that are not essential to finishing the game. Depending on the nature of the game, this work can take many forms. A game that is relatively linear, like Super Mario Bros or Metroid, might present relatively few routing possibilities. A game that is relatively open, like the main series Pokémon games, presents a multitude of possible routes, but very few of them will allow for a competitive speedrun.

Of the games that are commonly speedrun, there are only a few that I have completed myself and can talk with any degree of confidence about: Pokémon Generations 1-3, and Jet Set Radio Future (JSRF). JSRF is a relatively linear game, where the goals needed to unlock the next game levels are clear, and the fastest path through the game presents itself relatively quickly, even to a novice player. From a routing point of view, there are a few things you can do to save time: not all the characters need to be unlocked, some fights and races can be avoided, and an entire level (the sewers) can be skipped. However, because the game itself gives you a fairly limited set of permutations to progress through the game, the fastest route through JSRF is straightforward to learn.

The main series Pokémon games, on the other hand, are much more complex and present much more elaborate routing possibilities. A regular Pokémon player wants to maximise the total amount of experience points that the six Pokémon in their party have, and to catch as many Pokémon as possible along the way. This entails fighting as many battles and walking through as much tall grass as possible. This takes a great deal of time and makes up the bulk of a Pokémon playthrough.

A Pokémon speedrunner, however, is not trying to maximise total experience points. They are trying to do the opposite, to minimise the total needed to complete the game.

For a certain kind of gamer, this kind of question is an intellectual playground. What is the least amount of battling needed to get through the game? Six strong Pokémon take too long to train. One strong Pokémon is better than six weak Pokémon. What should that one Pokémon be? The biggest advantages you can have in battle are stats, moveset and type, so this one strong Pokémon should have high base stats, a powerful moveset, and preferably have few type weaknesses.

That narrows the 150 (or more) possibilities down to a much smaller pool of candidates, but the work of really optimising the route has only just begun at this point. Which battles are unavoidable? What Pokémon are encountered in these battles? What stats do these Pokémon have? What moves? Working from this information, what is the minimum amount of damage needed at each stage of the game to get through these battles, and the minimum amount of defense, special defense and health needed to survive? In a way, the game becomes an engineering problem, and all routers are trying to do is to find the most time-efficient solution.

Glitch Hunters: The Field Researchers and Lab Scientists

It is impossible to discuss routing in many games without discussing the glitch hunters. Glitch hunters look for ways to break the game, often as a way to try to understand how the game is built. Think about a child playing with wooden blocks who builds a structure and then removes blocks, one by one, until it collapses: this is not too different from what glitch hunters do. They look at a game as a carefully-constructed structure, and try to find which cornerstones and support pillars hold up the game, and how the structure holds up or falls apart when some of these elements are modified or removed.

(There is some discussion about what constitutes a glitch and what doesn’t, but I’m not going to get into that here. For our purposes, I’m going to group glitches, exploits and level skips together, because the differences are not important in this post.)

I could try to give you an example of how glitches help speedrunners to understand the game better, but it’s easier to let the experts do it. If you’ve ever played Pokémon Red/Blue/Yellow, I urge you to watch Shenanagan’s masterclass in catching all 151 Pokémon in Pokémon Blue from Summer Games Done Quick 2015:

Pokémon Blue 151 Pokémon by Shenanagans in 1:58:56 at SGDQ 2015

Many glitches have effects that are undesirable in normal gameplay, sometimes irreversibly breaking the game. However, glitches of all stripes are of interest to speedrunners because they suggest ways that the game can be manipulated, and therefore open up new routing possibilities that non-speedrunners would not even dream of considering. Some speedrunning communities maintain separate glitched and glitchless categories for speedrunners who prefer their games glitchless, but glitches are still useful discoveries in glitchless speedrunning because discoveries about the structure of the game may alter the calculus of routing possibilities in glitchless speedrunning.

Replicability

Here is where, to my mind at least, the parallels to the scientific method really start to take shape. A glitch is useless unless it is replicable.

In this respect, a glitch is no different from any other software bug. If you encounter a bug in a piece of software, the developer needs to know what you were doing right before the bug happened, because that’s the key to figuring out why it happens and how to debug the code. The critical difference between a software bug and a gaming glitch is how it is viewed. Developers understand how their software works, and bugs reveal oversights in the logic of the software. Glitch hunters do not have that privileged view of the software, and glitches reveal insights into the game logic.

Developers are watchmakers who craft a complex piece of machinery, while glitch hunters are more akin to field researchers and lab scientists, rigorously documenting new discoveries and tweaking experimental variables.

If a glitch cannot be replicated, it might as well be an accident caused by the sneezing of the universe. If it can be replicated, however, routers can add it to their arsenal of game knowledge and look at how this glitch or exploit can help gain time on existing routes.

Tool-Assisted Speedrunning: The Theoreticians

In some speedrunning communities, there is an additional group of speedrunners often critical to helping lower records: the tool-assisted speedrunners. A tool-assisted speedrun (TAS) is a run done in an emulator by a player who has meticulously planned out the game inputs (i.e. button presses) frame by frame, so that routes requiring multiple frame-perfect inputs that might be impossible for human reflexes can be run. In other words, the route is run by a computer.

TASes do not qualify for records, but I noticed that they often feature prominently in Summoning Salt’s world record progression videos. If a route is theoretically possible but very difficult to execute, a TAS is often presented as a proof of concept. Sometimes TAS routes are run, then shelved as being too difficult for a human to run, then revisited when the potential of easier routes is exhausted, as in the case of Super Mario Bros Level 4-2 (this link goes to the relevant TAS section, but you may want to start from the beginning for context). Other times, a difficult route is run by TAS to set a mark for human players to work towards, as in the case of Mario Kart’s Choco Mountain’s Weathertenko (this link also goes to the relevant TAS section, but you may want to start from the beginning for context).

The obvious scientific analogue here is the theoretician, the scientists who work with models and predictions, and whose work helps to advance understanding of what is possible but is not immediately applicable.

TASes are not useful for all speedrunning games. They are useful for testing extremely challenging routes in games with almost no randomness in the gameplay, like Super Mario Bros. In a game like Pokémon with a considerable amount of randomness, however, there are too many moving parts for TASes to be useful. In such speedrunning communities, the routers are the ones who take on the job of presenting these proofs of concept.

The Structure of World Record Progressions

Thomas Kuhn is credited with introducing the idea of the paradigm shift in his 1962 book, The Structure of Scientific Revolutions. He argues that “normal science” is punctuated by periods of “revolutionary science”, when existing models of science prove to be inadequate, and a period of tumult follows as new models are constructed to accommodate paradigm-breaking discoveries. These new paradigms demand the re-evaluation of old data, and eventually one will become the basis for the next period of normal science.

Kuhn’s presentation of the idea was controversial, but the concept is nonetheless useful in relation to speedrunning. If we think of a route as a paradigm, then the rest of the analogy falls in place easily.

In the early days of a game’s speedrunning history, skilled players converge on one optimal strategy to speedrun the game, and successive runs refine this route/paradigm until its potential is exhausted. Then, routers start examining assumptions about the best way to run the game, theorycrafting new routes/paradigms and incorporating lesser-known tricks and more difficult manoeuvres. In doing so, they work closely with glitch hunters and tool-assisted speedrunners to figure out how the game is built and what the theoretical limits are. They look at known glitches and explore ways to incorporate them into new and better routes, re-evaluating all their assumptions about established routes along the way.

Then, a breakthrough: either a new glitch is found, or an old glitch is “rediscovered” and successfully incorporated into an improved route/paradigm that might lower the theoretical speed record once more, so that the runners can have another go. The cycle begins again.

Obviously, this is a very abstract and overly simplified picture. The real process of lowering a speedrun record is much more nuanced, and all of the processes I describe in this post — glitch-hunting, tool-assisted speedrunning, routing, and the actual speedrunning — occur simultaneously, not sequentially. If you consider the scientific analogues — engineers, field and experimental researchers, theoreticians — it’s the same thing: progress in each domain happens simultaneously, not sequentially. Theoreticians don’t stop working while they wait for their colleagues in the lab to publish. Nonetheless, I think this division of labour is a useful lens through which to look at speedrunning.

The speedrunners: what about them?

So far, this analogy hasn’t accounted for the runners themselves. The runners are the ones who actually perform the speedruns, so what would their scientific analogues be? Truth be told, I don’t know. The analogy breaks down here, because so much of science is predicated on progressing knowledge through the accumulation of controlled, repeatable experiments, and speedrunning records are, by definition, exceptional. The methods to perform the speedruns may be rigorously documented, but few people on the planet will ever be able to repeat the methods to the required precision, and even then they may not match or beat the speedrun records of any given category. This is antithetical to the idea of replicability.

This is where speedrunning diverges from science and converges with sport. In fact, speedrunning uses the language of speed-based sports: pace, split, personal best, world record.

This is the face of speedrunning, the only dimension that most of us ever interact with twice a year when Games Done Quick rolls around and millions of gamers watch the world’s best speedrunners performing jawdropping feats of gaming. It’s probably why the comparison of speedrunning with sport seemed so natural to me at first.

Every GDQ run includes a commentator’s couch, where other speedrunners explain what the runner is doing and, very very briefly, the underlying mechanics that allow each section of the game to be optimised to such a degree. In a way, the couch commentators represent the real experience of speedrunning — the experimentation, the refinement of each route, and the collective knowledge of the community.

It would be easy to regard speedrunners as the leading edge of the speedrunning community, the ones who push past the limits and establish new frontiers, like the star scientists at the Solvay Conference. Perhaps it is more fitting, though, to think of the star speedrunners at GDQ as the keystone of an arch. Each block, each voussoir, is an incremental step towards the keystone. The keystone is merely the final piece of an architectural puzzle that rests upon all the blocks below it and gives the arch its shape. The speedruns are the raison d’être of the speedrunning community, but the joy of speedrunning lies in the process of discovery and inquiry that makes the speedruns possible.

Additional thoughts