This is not a game of snake

Almost a hundred years ago, René Magritte painted a very nice pipe and wrote ‘Ceci n’est pas une pipe’ (‘This is not a pipe’) underneath it, and people have been mildly annoyed about it ever since. In fact, there’s been quite a lot of discussion around it, despite the simplicity of the original artwork.

And while there have been many different takes from many different artists, philosophers, poets, historians, and common people, it’s been profoundly difficult for everyone to actually agree on what the image is about, but for the most part, it boils down to this: The representation of a thing, is not that thing. One excellent application of this way of thinking is “the model is not the data”, an aphorism in the sciences that acknowledges models of reality are limited and inaccurate, but can still be useful.

“We confuse the world as it is with the world as it is described.”

Sure, it looks like snake, and it even moves like snake, mostly. But there is no spoon game running in the above pane. In fact, to call what you are seeing a game is almost entirely inaccurate. Just as with Magritte’s pipe, which is not a pipe due to the fact it possesses none of the requisites to be a pipe, this is not a game because it possesses none of the requisites to even be a game. There are no rules, players, goals or end states. This isn’t “an AI playing a game of snake” either, the above is a small language model’s representation of a game of snake.

The idea

The plan for this little project is relatively simple. Build a perfectly normal snake game, and make the underlying game engine write every “frame” or “tick” out as a short “sentence” made up of the game state which is represented in “words” (lots of quotes here but just bear with me, if I explained it all now it would be too boring for you to continue). Then when we have our special game engine, we get something that is actually good at snake to play a lot of games and train a small transformer on nothing but those sentences, so that playing snake turns into predicting the next most probable word to form a “sentence”, which is coincidentally the thing a language model is built to do.

The transformer in the language model never gets any direct information on the game rules. In fact, it doesn’t get direct information from anything, the entire “concept” of finding apples, or what a good move is, or why running into a wall is bad is entirely inferred from the training data.

To be clear, I’m not trying to build a good snake player here. A transformer is a terrible choice for game agents, but that’s exactly why I’m building it. I wanted to see what kinds of mistakes it makes.

Writing snake down

The game itself is as standard as snake gets. It’s more or less the “hello world” of writing games, and I don’t think I have to talk much about how it works or how you play it. The rules and game engine are all just one python file. The only real oddity which sets it apart from a “standard” snake game is that it writes every game state out as a “sentence”, which brings us to the first challenge of this project, and that is inventing a language to play a game of snake.

The first attempt

My first go at this spelled out every item on the board, and was inspired by the g-code commonly used by 3D printers. A typical frame looked like this:

HEAD X9 Y5 BODY X8 Y5 BODY X7 Y5 APPLE X14 Y5

followed by a move like RIGHT to signal an ‘input’, followed by the whole list again except for the small permutations such as the new head position, tail position, and possibly apple position. Three “words” are used for ever “piece” of the snake (segment type, x location, y location) and that works out to 3L + 3 words for a snake of length L. If we assume that, on average, our snake eats 30 apples per game (which is on the low end), we would end up with a final game state of roughly 3(30) + 3 = 93 words in one sentence. The problem with a sentence length like this, is that it’s just one frame of the game, and our tiny little transformer is going to fill up its 256-word memory really fast, and we would be inducing attention collapse in the small attention head of our transformer due to the repetition.

Regardless of the verbosity, if we were to try to use masking to train our small language model, we won’t be able to mask entire frames since there is a good chance we don’t have the space for that on longer games. This means we would mostly be training our transformer to learn (as far as I can tell) that BODY is followed by an X coordinate, which is followed by a Y coordinate. Riveting stuff.

Begin, describe, end

The biggest problem with the above language is the way body segments are unnecessarily repeated. If we know the initial positions of the head and tail, all subsequent positions of the body can be inferred based on how the head moves (and if an apple was eaten). But we also don’t want to entirely disconnect the body of our snake from the head and tail, since the transformer somehow needs to “know” about collisions, otherwise it will just move through itself.

So the revised version treats every frame as its own complete little sentence, with a clear beginning, a description, and a clear end:

snake d 1 left left apple k 14 stop

snake starts the snake, followed by the address of its head (column d, row 1). After that each body word is the direction to the next piece, walking from the head toward the tail, so left left means “one piece to the left of the head, and another one to the left of that”. Then comes apple k 14, and stop closes the sentence. If that frame is the end of the game due to a self or wall collision, it closes with dies instead of stop.

The whole language that can describe any possible game of snake is just 48 words:

Words	What they do
snake piece apple	starting clauses
a … p	the 16 columns
1 … 16	the 16 rows
west east north south	margin ring of cells just outside the board
up down left right	directions (for tail piece and movement)
goes	comes right before a move / direction
stop dies	end a frame, or end a frame and the game
game <pad>	start of a row, and padding

The margin ring is something that was added later due to an unexpected development. During training, sometimes the head of the snake would end up off the board, but its head still has to actually be somewhere due to the way the engine works, and we still want to be able to represent whatever the small language model is outputting. So driving straight up off the top from the start would mean the head is positioned in the “margin” above column d, then the body steps back down onto the board, and the sentence ends in dies.

A frame for a snake of length L is always L + 6 words. Much better than the previous 3L + 3! The general structure of a sentence is now

game <this frame> goes <move> <next frame>

This structure captures a whole frame, and nothing from a few moves ago is needed to know what happens next, so the model never needs to “remember” anything outside the sentence it’s reading. The longest sentence in the longest game of the corpus is 185 words, which was comfortably inside the model’s 256-word window.

Here are a few frames side by side:

One thing I cared about a lot is that this language can spell out things snake can’t do. The renderer built for this reads a sentence and draws whatever it “says”, regardless of how valid it is, or with any relation to the snake game rules. Two apples, no apple, two heads, a loose bit of body, a snake folded back over itself, all of it should work, but for any “illegal” board states we can flag that something’s gone wrong. You can try it yourself below, so type whatever you like, feel free to use the table of the language above for reference.

Training a RL model to play some snake

Before the transformer can read any games “written” by the game engine, somebody has to play a lot of snake first, and it was not going to be me. Snake is small, fully visible, and only has a handful of choices per move, which makes it a natural fit for reinforcement learning, so I tried two different popular RL algorithms, Proximal Policy Optimization and Deep-Q learning

Starting from random weights, both agents ate basically nothing for the first few thousand games, but both soon started improving.

Games	PPO median	DQN median	PPO starved	DQN starved
20,000	4.5 (0–9.1)	0 (0–1)	92	97
50,000	21 (8.9–28)	24 (10–30)	49	16
200,000	48 (27–62.1)	32 (14–46.1)	1	12

Median apples eaten, with the middle 80% of games in brackets, and how many of the 100 games ended by starving.

At 20,000 games PPO has at least begun learning while DQN is still effectively random. By 50,000 DQN has caught up and is slightly ahead, but then by 200,000 it has levelled off while PPO keeps climbing, ending at 48 apples for PPO against 32 for DQN.

For interest’s sake, the statistics include both the performance and the time. PPO reached 200,000 games in 87 minutes, which is pretty respectable considering I didn’t use one of the big ML libraries like pytorch or tensorflow. DQN took 205 minutes, but that’s mostly because its replay-buffer update ran every four environment steps once the warm-up stage ended.

Both of them are “good enough” players in the sense that they eat a lot of apples and then die in an entirely normal, snake-like way, which is all a teacher model needs to do here. Since PPO was performing better at the time (and training faster), I let it cook overnight, and in the morning the median apples eaten for PPO was 57. Not bad!

Turning a lot of games into a little file

The initial plan was to let the PPO agent play 20,000 games for training, plus another 1,000 that the transformer would never see (so I could check its homework and make sure it wasn’t overfitting). Recording all of them took about 10 minutes. The training games average 49.8 apples, with a median of 53 and a longest game of 82. That’s a little under the greedy median of 57 because the moves were sampled from the policy rather than always taking the top choice, which gives the corpus a bit of variety (and the occasional questionable decision).

If all of those games were to be written out as sentences, that would be a lot of data to manage. The typical game above is 1,231 moves long, and as one long document of frames it’s 243,607 bytes. Measured on every 40th game, the 20,000 training games would come to roughly 4.4 GB of text, and that’s before you count each frame a second time for the training rows.

So, what’s the solution? Well, if we want to make the dataset as small as possible, let’s start with the smallest representation of data available to us, a bit, and build from there.

The most commonly used words in our vocabulary are most certainly going to be the directions, so they are the first candidate for encoding. Four directions fit exactly into two bits: 00, 01, 10, and 11. That means four moves fit in half a byte (also called a nibble), and we can encode the four margin positions in another four bits, leaving all “directions” representable in a single 8-bit byte.

Then we need to look at the next most common piece of data, the apple. We need a way to denote in which row and column of the 16x16 board the apple is, and there are 256 possible positions it can spawn in. Conveniently, 2^8 = 256, so eight bits (which, again, is one byte) can point to the exact position of any cell. The high four bits hold the row and the low four hold the column.

The rest of the necessary information can be rolled up in a five-byte header per game: two bytes for the number of moves in total, stored little-endian-ly, then one byte for how many apple cells appear in the entire game, one byte for how the sequence stopped (died, starved, won, or deliberately cut short), and one spare byte as a buffer. After that come the sets of moves and apple cells. Using this representation, our entire vocabulary takes up (at minimum) only 7 bytes in total!

After performing our space-saving encoding, that same 243,607-byte game is 367 bytes in total. The 20,000-game training archive is 6.7 MB, roughly 652 times smaller than writing the same games out as text, and a few megabytes smaller than the size of the blog post you’ve read up until this exact point, which is 15.1 MB of content so far (more or less).

Here are the following three moves of a game snake f 2 up left apple k 14 stop in our new data format:

03 00  01  03  00  1f  da

Or, as the bits that are actually stored:

00000011 00000000   01        00000011  00000000   00011111   11011010
moves = 3           1 apple   stopped   spare      00 01 11   row 13, column 10
                    cell      early                + 1 unused  (d, a)

A game is 5 + ceil(moves / 4) + apple cells bytes large. The typical game has 1,231 moves and 54 apple cells. The list holds the opening apple and one new cell after every eat, so 53 apples eaten means 54 stored cells, the remaining apple being the last on the board when it died.

367 bytes = 5 + ceil(1231 / 4) + 54 = 5 + 308 + 54

The ending byte is related to an end state which I’ve not mentioned earlier. One thing that appeared during training is the tendency for snakes to just spin in circles, because if they don’t collide with anything, they won’t get punished, and without consequences to our actions we are unable to learn. In the game engine, the player “gives up” after too long without eating an apple, which is to say that it “starves”.

The reason this all compresses so well is a bit of luck in the shape of the problem. The biggest contributor is the fact that the engine is deterministic, so given a frame and a move, the next frame is completely determined, except for where a new apple appears after one gets eaten since they are inherently random. In our engine, the only information required to represent a game is the list of moves and the list of apples. This also simplifies how much data we need to feed into our eventual transformer.

Why attention works with this language

The “T” in GPT is for transformer, and the main trick a transformer has is attention. When a position is choosing its next word, it gets to look back over the earlier words in the row and decide how much it cares about each one. Roughly speaking, the current position asks a question (a query), every earlier word holds up a label (a key), and the words whose labels best match the question get to pass their contents (their value) forward. A model has several of these attention “heads” per layer, each free to ask a different question. To make the attention mechanism a bit more transparent, let’s have a look at two frames from a game:

game snake f 2 up left apple k 14 stop --- goes right snake g 2 left up apple k 14 stop

I’ve added the --- between the two frames to make reading a bit easier, but this does not exist in the real data.

Almost everything in the next frame is a copy of the previous, except for a small offset. One useful thing an attention head can learn to ask is: “Which earlier word belongs in the position I’m writing now?”

That is not a sentence written anywhere inside the model, of course. A “query” is just a list of sixteen numbers in one attention head. Every earlier word has its own sixteen-number key, and a dot product measures how well the query and key match. A high match gives that earlier word more “influence” or percieved “relevance” over the current one.

This is why the direction language suits attention unusually well. A bend in the snake remains mostly the same “word” while it moves down the sentence. If we were to use a grid-coordinate representation of our board, that bend would become a different piece of the snake in a different row and column every frame. In a 256-cell board where each cell is an input, almost every input token would be an unchanged empty square and the few useful edits would be buried among countless “unchanged” copies of cells, and as mentioned before, transformers aren’t the best at dealing with repitition.

Borrowing a very small brain

The model is based on Andrej Karpathy’s microgpt, a single file of pure, dependency-free Python that trains a tiny GPT to make up baby names one character at a time. It’s a lovely piece of code, and Karpathy has done a great job of making it easy to digest and simple to follow. It also does every single multiplication as its own little Python object, which means running it on 20,000 games of snake would take much longer than I have the patience for.

So instead of just waiting, I reimplemented it all from scratch in NumPy. To be clear, that wasn’t really necessary, but I wanted an excuse to actually understand it. The GPT built for this uses all the same parts as the original: RMSNorm instead of LayerNorm, no biases, ReLU, learned position embeddings, causal attention and Adam. It just does them as array operations, with every backward pass written out by hand, which is where most of the time spent understanding, and most of the time spent debugging, actually took place.

The resulting parameter matrices look like this:

Piece	Shape	Parameters
Token embedding	48 × 64	3,072
Position embedding	256 × 64	16,384
two RMSNorm scale layers	64 + 64	128
query, key, value, and output layers	4 × (64 × 64)	16,384
widen to 256, then back	64 × 256 + 256 × 64	32,768
Three transformer blocks	3 × 49,280	147,840
Final RMSNorm	64	64
Output head	64 × 48	3,072
Total		170,432

That is a very small model, especially when compared to the 2018 GPT-1 with 117 million parameters. So small in fact, the entire model has been embedded into this page, and you didn’t even notice!

Learning to read

The above transformer was trained for ~136,000 steps, about nine and a half hours on my laptop.

From this data, we can gather that it learned how a snake moves almost perfectly, and it agrees with its PPO teacher about where to go four times out of five. There’s a pretty persistent 6.2% “loss” since it can never guess where the apple will be correctly, but that is exactly what you’d expect. A new apple lands on a uniformly random cell, so each word of its position is a one-in-sixteen guess, and one in sixteen is 6.25%. The model sits right on that floor (6.2% averaged over training), which means it has learned everything there is to learn here: nothing in the frame says where a random apple will appear, so the only honest answer is a shrug.

So now the big question is…

How does it play?

Well, you’ve technically already seen it playing at the start of the blog! That wasn’t a pre-recorded set of moves or anything, apart from the opening snake d 1 left left apple k 14 stop to get the game going, but after that every frame is its own transformer output.

Below is a more fleshed out and interactive version. A pink flash means the frame isn’t legal in a normal game of snake, and “next weird frame” jumps to those illegal frames specifically.

Choosing when to die

One interesting thing about this model is that it somehow never really learned to die. It knows the word, certainly, it’s in the vocabulary and in the training text, but it’s such a rare word, in a sample of 500 training games, 570,403 frames end in stop and 473 end in dies, so it shows up about once per game against well over a thousand stops. The idea behind a transformer, to always choose the single most likely word, is never going to select the statistically improbable death.

I tried to induce dying using the easiest case I could think of. The typical teacher game from our earlier example ends with the head at a 10, against the left wall, and the teacher goes left into the margin, so the frame ends in dies. Given that exact same frame, the transformer goes right to b 10 and writes stop. It just pops right out of the wall!

So, how does a game actually end if the snake can’t die? Eventually, if we let the transformer “play” long enough, it ends up looping through the exact same path, over and over, always passing close to the apple, but never quite reaching it, before starting the statistically most likely path again.

Poking it with things that can’t happen

The game at the top only ever got into trouble on its own. So I induced sentences that no game of snake could reasonably produce, and let each one run for a few frames.

Interesting, doing a little bit of damage to a sentence seems to make it repair itself instead of making it go off the rails. A loose piece of body lasts exactly one frame before it gets welded onto the snake (which jumps from length 3 to 6, and the apple teleports ten rows for good measure). Adding a second head makes one simply get deleted. A body doubled back over itself is straight again two frames later. A head cut off from the body and parked out in the east margin doesn’t die and doesn’t wander around out there either, it gets put back on the board, and then the snake traces the same four-frame square forever.

With some experimentation, I was able to find some “game breaking” sentences after all though!

If we remove the starting apple completely, the first thing it writes is right right right right right apple f 6 stop: some directions, an apple, and no snake. The frame after that is a length-12 snake with its tail hanging off the board, and two frames later it’s legal again.

Two apples sometimes also resulted in something unique. The first continuation deletes both apples and writes two heads instead. Then come frames that aren’t snakes at all, like up up up apple g 15 stop, then a one-cell snake, then a long run of right. On the fourteenth frame it’s legal again, with one snake, length 15, and one apple.

Since the weights are in this page, you can try and break it yourself if you wanted to. The parameters are stored as 16-bit floats to keep the page small, so a long continuation here can drift by a word or so compared to the results above from the real transformer model, but the drift shouldn’t be too significant

What is the model “thinking” about?

I wanted to see what’s actually going on in the latent space of this model, so I’ve made a few ways to try and observe what’s in its “brain” during a game.

Right before it picks a move, the model has boiled the entire sentence down to a list of 64 numbers (64 latent vectors). They are not 64 predetermined “indicators” in the sentence though, and there is no single slot for “head row” or “apple column”. Each number is one coordinate in a learned representation (there’s that word again) of the 64-dimensional world state, and useful pieces of information are spread across combinations of them. Here’s a short form video from 3blue1brown on latent space vectors and how they encode meaning. The output of those 64 numbers combine into a single “which move to make” signal.

Sixty-four dimensions sadly don’t fit on a 2-dimensional screen without a lot of imagination, so the box below shows the same game four different ways that are hopefully digestible to 3-dimensional brains.

Cloud squashes the 64 numbers onto the three directions they vary along the most, which together hold 73% of the variance. Coloured by next move, the cloud splits into rough arms, and guessing the move from whichever arm a frame sits closest to is right 83% of the time (against 25% by chance). Have a look at the end of the game, you’ll see the bright tail stops exploring and traces the same loop over and over.

Decision is the model’s own odds for its four moves. In 87% of frames, just two moves hold more than 95% of the odds, so it’s nearly always choosing between two options, not four. On legal frames the top move gets a median of 89%, on illegal frames 76%. It gets less sure when the sentence stops being snake, but it never stops, and in the closing loop it’s the most certain it ever gets, at 94%.

Mind reading is a simple straight-line readout, fitted on eight teacher games the model never trained on, that tries to recover where the head and the apple are from those 64 numbers. The head comes back well, usually within a cell or two (an R² of about 0.9). The apple’s row comes back roughly (0.71) and its column only partly (0.45). It knows where its own head is far better than where its food is, which explains a few things about the apple-on-snake frames.

Attention shows how much the last word in the row, goes, looks at every earlier word while the model picks the move, in the last of its three layers. Averaged over all four heads the picture is pretty mild, but split apart, we can see the heads look at different things! The second head gives about half its attention to stop, the word that closes the frame, and on about half of the frames that’s the word it looks at hardest. The other three mostly land on a column letter, a row number or a direction: the words that say where the head is, where the apple is, or which way the body bends.

So what is it, if it isn’t snake?

I’ve said before that this is not a game of snake, but what does that make it? A representation of a game of snake is the best I can manage to call it, and something I think we can all more or less agree with. Perhaps an imitation of snake? Or a language of snake? It’s made by a language model after all?

What I find most interesting is how little of this has anything to do with language. Nothing about the transformer knows these are English-ish words. The vocabulary could just as well be 48 random symbols, and it’d learn the exact same thing, because what it’s really learning is the structure of a sequence (what tends to follow what, and what has to be copied from where.) Snake frames, chess moves, a game log, a sensor readout, anything you can write down as a sequence with structure is a “language” as far as a transformer is concerned.

It isn’t learning the rules of the world, though. It’s learning to imitate the behaviour of the thing that wrote the text. It’s learning to imitate the game so well, if I were to have asked a human to point out which game of snake was played by an RL model forced to follow rules and a transformer model that has no concept of them, I doubt I’d be able to tell the difference. And that’s on such a small and simple model where I deliberately chose to induce “noticeable” behaviours and highlight them.

Perhaps, one day, we will have advanced our generative AI models to the point where we too are unable to tell what’s real, and what’s an imitation. What is grounded in the rules of physics, of law, of society, and what is merely a representation of humanity. What flaws will it have, I wonder. What rules will it start breaking first?