In 2013, competitive Super Mario 64 speedrunner DOTA_Teabag was running through Tick Tock Clock, a level infamous for punishing even the best of players due to a challenging sequence of jumps. While making their way through, Mario suddenly shot directly upwards through the floor, bypassing a full section of the level and saving DOTA_Teabag precious seconds.
TTC Upwarp: Ceiling Warp vs Byte Change (embed video of the flip)
The speedrunning community, which had been playing and dissecting Super Mario 64 for decades, was stunned. A bounty was offered to reproduce the glitch; some speedrunners claimed that DOTA_Teabag had “rocked the cartridge” (a way to sometimes predictably glitch a Nintendo 64 game by slightly modifying the angle of the game cartridge). The mystery came to a close when YouTuber and esteemed video game glitch hunter pannenkoek12 (translated ‘Pancake12’) discovered that a single bit in the Nintendo 64’s memory was the cause. The bit flip happened at `0xC5837800`, the address responsible for Mario’s vertical position. The change in binary was from `1100 0101 to 1100 0100`, and by extraordinary luck, the exact change needed to warp Mario onto a higher platform.
So what caused this in the first place? All signs point to a cosmic ray bit flip, a random error in computer chips caused by a high-energy particle from space that happens to affect a system right at a particular time.
What is a Cosmic Ray Bit Flip?
A cosmic ray bit flip (also called a “single-event upset”, or SEU) occurs when a stray ionizing particle from a cosmic ray in space strikes a single memory cell or transistor, flipping a binary bit from 0 to 1, or 1 to 0. These bit flips can cause a soft error, which leaves no physical damage to the hardware, but they can alter data stored in memory or logic adjusting application behaviours or even disabling security features.
The Super Mario 64 speedrun isn’t the only affected documented case of a cosmic event having on-Earth effects. During the 2003 Belgian election, an unknown candidate received exactly 4,096 extra votes seemingly out of nowhere on the country’s electronic voting machines. The error was only caught because she received more votes than the number of voters, making it mathematically impossible.
Upon investigation, researchers concluded that a cosmic ray had flipped single bit at position 13 in the voting machine’s memory, giving the candidate 4,096 extra votes [1]. In binary, each bit doubles the value, so the bit at that position was computed to be 2^12 - or exactly 4,096.
Rare Doesn’t Mean Impossible, Especially at Scale
These cosmic ray bit flips are incredibly rare on a per-bit basis, but they’re not impossible. In 1996, researchers at IBM were able to estimate that a typical desktop that has 256 megabytes of RAM could potentially experience one cosmic-ray-induced bit flip per month. Fast forward to today, where modern systems commonly run with 16 gigabytes of RAM (62.5x more than the study!), and the risk scales accordingly. Combine that with millions of cosmic rays hitting the Earth every second, and it’s only a matter of time before one hits the wrong bit at the wrong time. Researchers at Vanderbilt University were able to confirm this, stating in their research that an internet service provider’s router farm with 25 gigabytes of memory could experience a bit flip up to every 17 hours [2].
The key insight is that a “one in a million” event will happen daily if you do a million-plus things a day. Here at cside, we scan roughly over 10 million scripts every 24 hours for malicious behavior. Even if a bit flip error has a one-in-100-million chance, scanning at that volume means these impossible odds are, in fact, quite possible.
While enterprise grade memory may not be subjected to cosmic ray bit flips anymore through adoption of error-code correcting memory, which can often detect and correct single-bit errors, consumer hardware isn’t usually protected. It is important to understand that ‘rare’ in a world where millions of actions takes place means it is just a matter of time.
Cosmic Hardening and Embracing Resilience
Given that cosmic bit flips are a game of statistics, how can systems be built that expect the unexpected? The answer lies in engineering for fault tolerance. For example, NASA spacecraft run critical calculations in triplicate - multiple processors run the same operation, and if one disagrees due to a stray bit flip, the other two out-vote the process. Back home on Earth, ECC memory can play a role in safeguarding routers, servers, and cloud infrastructure against single-bit errors (and even sometimes detect two-bit errors!). But like everything, ECC memory has its own limitations. It doesn’t typically protect the CPU register, GPU memory, or network buffers. Even in enterprise environments equipped with ECC memory, multi-bit flips or logic-level faults can go undetected. This is why it is key to have a layered resilience strategy. At cside, we apply the same philosophy to client-side security. Even though a malformed script or edge-case exploit may be rare individually, scanning millions of JavaScript files daily means they’re not rare in aggregate.
Ultimately, cosmic ray bit flips remind us that at scale, we’re dealing with probabilities - not certainties. A one-in-a-million becomes inevitable when your systems process a billion operations at a time. Whether you’re building a database, an election system, or a client-side browser proxy, the lesson remains the same: plan for all possibilities. On the client side, unpredictability scales with volume. That’s why your systems must be built with the assumption that nothing is immune to failure, and designed to withstand even the most improbable, cosmic-level anomalies.
References:
[1] https://web.archive.org/web/20070927185155/http://wiki.ael.be/index.php/ElectronicVotingRandomSpontaneousBitInversionExplained