The ideas behind the measurements.
Short primers, no background needed. Why running a model costs what it does.
What happens when a model writes a token
A language model is a very large function. You give it a list of numbers, it gives back one prediction. Everything else in these notes follows from what that function is made of, and from the fact that it has to run once for every token it writes.
Read →P2 · PrimerWhy decoding is limited by memory, not arithmetic
A modern chip can do trillions of multiplications per second. A small model needs only a few billion per token. So why does a laptop write tokens at "only" 200 per second? Because the chip spends most of its time waiting for numbers to arrive from memory.
Read →P3 · PrimerWhat quantization is, and what Q4_0 means
Decode is limited by how many bytes it reads (primer 2). The most direct way to read fewer bytes is to store each weight in fewer bits. That is quantization.
Read →P4 · PrimerWhy serving many people at once is cheap
A datacenter GPU can serve a token for a fraction of the energy a laptop needs. Most of that difference is not the chip. It is batching.
Read →P5 · PrimerHow speculative decoding works
A single user cannot batch with anyone else (primer 4). Speculative decoding is a way to get some of batching's benefit anyway, by batching your own future tokens.
Read →P6 · PrimerHow to measure energy without fooling yourself
Most published energy figures for AI disagree with each other, and usually not because anyone lied. They measured different things, or measured the right thing badly. These are the rules we learned, most of them by getting it wrong first.
Read →P7 · PrimerDistillation: teaching a small model to imitate a big one
Speculative decoding (primer 5) is only as good as its draft model's guesses. A draft that agrees with the big model 70% of the time saves energy; one that agrees 30% of the time wastes it ([note 7](../research/speculative-decoding/)). So the obvious question is: can you make a small model guess better? That is what distillation does, and it is how we built our first model family, Trace.
Read →How we measure.
Every result follows one written standard.
Read the methods