Note 26 · 26 October 2026

More data made the draft right more often, and no faster

Trace v1 on 35,000 prompts: agreement with the 7B 0.7754, short of the 0.79 bar. Its guesses are accepted more often, but it guesses less at the same threshold, so it is no faster than the stock draft.

Prediction verdict: missed on the bar. Trained on 35,000 prompts, Trace v1 reached 0.7754 held-out agreement with its target, short of 0.79. The eval prediction held: 1.44 times plain decoding's speed with a third less chip energy, within 5% of the stock draft. The prediction that acceptance would rise less than agreement missed on the rate and held on the number that sets speed (section 2).

Spliced research note 26, 26 October 2026. AMD Ryzen AI Max+ 395 (Strix Halo, Radeon 8060S, 128 GB, shared GPU memory 90%), Linux, llama.cpp 620d2225c, Vulkan; training in PyTorch 2.10 on ROCm 7.2. Target Qwen2.5 7B Instruct Q4_0, drafts Qwen2.5 0.5B.

Note 21 trained Trace v1, a 0.5B draft model, on the 7B's own answers to 9,000 prompts and its token probabilities. Agreement with the 7B rose 0.7 points and decoding got no faster. This note asks whether that was a data problem, by repeating the same recipe on 35,000 prompts.

What "agreement" measures

Every number here is measured against the 7B as it is served: Q4_0, through llama.cpp. The training answers and the 180 held-out answers were generated by that model, greedy, with its top 20 probabilities at every token. Held-out agreement is the share of 56,600 answer positions where the draft's top choice is the 7B's. The speculative-decoding eval uses the same model as its target. The bar, 0.79, is 3 points over the stock 0.5B draft.

1. Four times the data, 0.7 points more

One epoch over 35,000 prompts, 4,200 steps, 121 minutes, learning rate 3e-6. Agreement every 200 steps:

step 0 (stock) 200 1,000 2,000 3,000 4,000 4,200
agreement 0.7616 0.7663 0.7683 0.7707 0.7724 0.7753 0.7754

The same recipe on 9,000 prompts reached 0.7685. The curve was still rising at the end, by about half a point over the last 2,000 steps and slowing. A second epoch would land near 0.78 at that slope.

2. Right more often, guessing less

The draft at work: the 7B decoding 40 held-out prompts, 256 tokens each, one request at a time, drafts of up to 4 tokens at a 0.5 confidence threshold, two passes.

draft tok/s vs plain acceptance drafted per pass accepted per generated token chip mJ/token
none 49.6 1.00x 2,096
stock 0.5B 73.3 1.48x 71.7% 8,591 0.601 1,400
Trace v1, 9k 70.1 1.41x 73.6% 8,086 0.581 1,423
Trace v1, 35k 71.7 1.44x 75.3% 8,052 0.592 1,385

The 35k draft's guesses are accepted more often, 75.3% against 71.7%. But it makes fewer guesses: at the same 0.5 confidence threshold it proposes 8,052 tokens per pass against the stock draft's 8,591. Speed depends on accepted tokens per generated token, and that fell from 0.601 to 0.592. Against the stock draft it is 2.3% slower and 1.1% cheaper per token, both inside the stock draft's own 4% gap between passes.

Training on the 7B's top-20 probabilities taught the draft that several tokens are plausible at a position, which is true. A draft that spreads its probability is less often above 0.5, so the threshold that suits the stock draft holds the trained one back.

3. Speculation itself pays on this machine

Any of these drafts makes the 7B about 1.45 times faster at one request and cuts chip energy per token by a third. On the M5 Pro the same kind of draft gives 1.26 times the speed for 3% more energy (note 25). The difference is what the draft does to power. The Strix Halo's package draws about 104 W decoding one request and about 103 W with the stock draft (time-weighted over every row), so finishing 1.45 times sooner saves a third of the energy. On the M5 Pro, power rises from 23 to 30 W with the draft, and that rise eats what the shorter time saves.

What this changes

  • Off-policy distillation has run out at this size. More data and more epochs move agreement by tenths of a point and serving speed not at all.
  • The trained draft's confidence threshold has to be chosen for it, not inherited from the stock draft.
  • One pilot remains, registered before any of these runs: on-policy distillation, where the draft writes and the served 7B's own weights grade every token. If it does not beat the best draft here by at least one point of acceptance at one request, Trace pauses. The 72B Trace stays parked.
  • The stock 0.5B draft remains the one Foundry ships.

Limits

  • One machine, one target, 40 prompts of two kinds (chat and code) for the eval; held-out agreement on 180.
  • One epoch; the curve had not flattened, so this measures the recipe at one epoch, not its ceiling.
  • Chip (package) energy, not the wall.
  • The threshold explanation in section 2 is read from the counts, not tested. A threshold sweep for the trained draft would test it.

Reproduce

On the Strix Halo, with llama.cpp at 620d2225c built for Vulkan and the PyTorch environment in docs/handoff-linux.md:

sh experiments/trace_35k.sh       # harvest the remaining prompts, then train one epoch
sh experiments/trace_35k_eval.sh  # convert, then evaluate.py at k=4 and 8, p 0.5, two passes

Raw: results/drafter/train-log-v1-35k.json, results/drafter/eval-v135k-halo.json, results/drafter/teacher-check-halo.json, and the full write-up results/drafter/trace-v1-35k.md.

Source, scripts and raw data: phase/notes/2026-10-trace-35k.md