Two machines, one token
The study on a Windows laptop with an RTX 5070 Ti, on battery. Q4_0 wins on both machines and a 4-bit token costs the same whole-machine energy on both; one Mac finding does not transfer.
Read note →N3 · 24 October 2026Does it hold? A model five times larger, and a second runtime
The format study at 7B, where every finding replicates and the hardware is used better, and in MLX, where repeat runs do not yet agree, so only claims that survive both passes are made.
Read note →N2 · 24 October 2026Where the joules go: a roofline and an energy model for decode
A decode roofline from measured bandwidth, and a validated energy model whose residuals isolate the cost of unpacking each format. Two wrong first attempts, kept in.
Read note →N1 · 24 October 2026Fewer bits is not fewer joules
Eight quantization formats and batch sizes 1 to 8, energy at the chip and at the board. Chip-only measurement misranks F16 by 2x; batch 2 saves 1% per token and batch 8 saves 72%.
Read note →What we want to know next.
Is it the kernel?
The energy model leaves Q4_0 29% cheaper and Q5_K_M 19% dearer than their bytes predict. Measuring each unpacking kernel alone should account for it.
Does it hold at scale?
Tokens per megawatt-hour for datacenter GPUs, from a roofline and published specs, checked against MLPerf results.
Joules or seconds?
Speculative decoding is sold on latency. On memory-bound hardware it might cost energy instead of saving it.