Bet 46 — Norm-only replication (5 seeds × 3 eval texts)

Bet 37's norm-only result was a single-seed measurement on a single eval text. Single-seed results in machine learning are notoriously unreliable — the gap between "this works" and "this happened to work that one time on that one corpus" is wide enough that any production decision based on a single data point is taking on more risk than it needs to. Bet 46 ran the same experiment with 5 random seeds and 3 distinct eval texts to see whether the headline survives, and to quantify the variance.

The bet exists because the federation was about to commit to norm-only as the production default per-user adapter format. Before that commitment was acceptable, we needed to verify the result wasn't a fluke. A 15-configuration sweep is the smallest replication that lets us claim the result is structural rather than accidental.

Setup

The replication is a clean cross-product: 5 random seeds × 3 eval texts = 15 training/evaluation runs. Each run repeats the Bet 37 setup exactly:

Model: FractalMoE 30M, programmer-fixture base.
Adapter: norm-only fine-tune (~2,300 RMSNorm gain parameters).
Compared against: full fine-tune (~38M parameters).
Optimiser: AdamW, lr 5e-5.
Steps: 100.
Wall-clock: ~5 minutes per run on M1 Max with MPS backend.
Eval: held-out same-author text, never seen during training.

The 5 seeds vary only the random seed for parameter initialisation. The 3 eval texts are different held-out slices from the same user's corpus, each chosen to span different content regions (function-heavy code, doc-heavy prose, comment-heavy mixed). If norm-only's win in Bet 37 was eval-text-specific, this bet would catch it. If it was seed-specific, this bet would catch that too.

Pre-registered criteria

STRICT: norm-only wins (held-out ppl lower than full FT) on ≥ 12 of 15 configurations.
LENIENT: wins on ≥ 9 of 15.
CATASTROPHIC: wins on < 5 of 15 (would falsify Bet 37's headline).

The STRICT bar of 12/15 was chosen to allow for some configurations where full FT happens to converge well (the seed lottery favouring it on a particular eval text). 12/15 is 80% wins, which is a clear majority.

Result — STRICT 15/15 PASS

Every single seed × eval combination favoured norm-only. The result is unambiguous.

The full table is in experiments/bets/results/46_norm_replication.json; the summary numbers:

| Metric | Norm-only | Full FT | Ratio (norm-only / full FT) | |---|---|---|---| | Median ppl across 15 runs | 187 | 28,400 | 0.0066 | | Min ppl across 15 runs | 1.04 | 76 | 0.014 | | Max ppl across 15 runs | 891 | 412,000 | 0.0022 | | Standard deviation | 312 | 124,000 | — |

The norm-only median ppl is 0.0066 of full FT — equivalently, full FT's median ppl is 150× higher than norm-only's. Across the 15 configurations, full FT regularly explodes to 84,000–400,000 perplexity (essentially noise), while norm-only stays at 80–600 perplexity (within an order of magnitude of useful).

The variance across seeds is informative. Norm-only's standard deviation is 312 — significant in absolute terms, but the result is consistently in the "useful" range. Full FT's standard deviation is 124,000 — the method is not just bad on average; it's wildly inconsistent. Some seeds happen to land near the training-data manifold and produce sub-100 ppl; others overfit catastrophically. There's no way to know in advance which kind of run a particular seed will produce.

What this rules out

The 15/15 result rules out three classes of failure mode that single-seed Bet 37 couldn't:

Single-seed lottery. A single-seed result with a 100,000× margin could be a fluke — perhaps that seed happened to put the optimiser in an unusually bad starting position for full FT and unusually good for norm-only. 5 seeds × 3 eval texts × 1.0 win rate is not a lottery.
Eval-text artefact. A single-eval-text result could be an artefact of that specific text's vocabulary or structure. Three distinct eval texts (each from a different content region of the user's corpus) all favour norm-only. The result is not eval-text-specific.
Implementation bug specific to one configuration. All 15 runs use the same code path; if there were a bug masquerading as the result, it would have to be present in all 15 runs. The bug hypothesis becomes implausible.

These three rule-outs are what makes the production-default decision defensible. The federation can ship norm-only as the per-user adapter format because the result is a structural property of the method, not a single-run accident.

What this does not rule out (and where the framing has been tightened)

This bet was run before Bet 60 (the noise-floor negative control). At the time of Bet 46's writeup, the framing was "norm-only is dramatically better at personalisation than full FT." Bet 60 forced a tightening of that framing.

The current understanding:

Norm-only's win over full FT is dominantly a regularisation effect. Full FT overfits catastrophically on 5-minute corpora; norm-only doesn't. Most of the 100,000× gap is the size of the overfit penalty, not the size of the personalisation benefit.
The personalisation component is real but modest. Bet 61's confusion matrix isolated the personalisation signal at 5%–29% per user.
Bet 46's 15/15 PASS is still valid as a measurement. The result that norm-only beats full FT on 15/15 configurations is true. It just means something different than "norm-only learns the user better." It means "norm-only's structural prior prevents the catastrophic overfit that full FT suffers under fixed-budget training."

The catalogue keeps Bet 46's headline (15/15 STRICT PASS) and updates the interpretation downstream, in Bets 60 and 61. This is the right way to handle interpretation drift — keep the data, update the framing.

Why this is the strongest replication in the catalogue

Most bets in the catalogue run with one or two seeds. Bet 46 ran with five. Most bets evaluate on one held-out text. Bet 46 evaluated on three. The cross-product is 15 configurations, all reporting the same direction of effect. This is the highest-replication result in the catalogue at the time of writing.

The replication standard is something the methodology page explicitly endorses: any production-default decision should be backed by ≥ 15 configurations of evidence. Other bets that touch the production wire format (Bet 49 shootout, Bet 52 quant + adapter composition) follow the same standard. The catalogue's calibration depends on bets like this one having run with enough configurations to rule out single-run artefacts.

What this means for production

Bet 46's STRICT PASS was the trigger for the federation to commit to norm-only as the per-user adapter default. The decision sequence:

Bet 37: norm-only beats full FT on a single seed (interesting, not yet decision-grade).
Bet 46: norm-only beats full FT on 15/15 (replicated, decision-grade).
Bet 49: norm-only Pareto-dominates LoRA-r4 and full FT across 3 distinct users (committed; production default).
Bet 52: norm-only composes with ternary base quantisation (production wire format settled).
Bet 60–61: framing tightened by the negative control + confusion matrix; production default unchanged.

Bet 46 is step 2 in this chain. Without it, the production default would be a single-seed claim, which is not enough to ship.

What stays open

Generalisation to other base models. Bet 46 was on FractalMoE 30M. Whether the result holds on Llama 3.2 1B or Qwen2.5 1.5B or any other architecture is the 1B+ scale open question.
Generalisation to other compute budgets. All 15 runs used 100 steps / 5 minutes. Longer training likely changes the picture; full FT might recover its performance with sufficient data and time. The federation deployment scenario is the 5-minute budget; that's what we tested. Larger budgets are open.
Generalisation to other adapter formats. The 15/15 is for norm-only specifically. LoRA-r4 might also pass a 15/15 replication; it's not yet been replicated at this density. Other formats (steering vectors, soft prompts) would need their own replication studies.

Run command

PYTHONPATH=src python -m experiments.bets.46_norm_replication

Output: experiments/bets/results/46_norm_replication.json contains all 15 cell results, per-cell variance, win counts per seed, win counts per eval text, and the headline 15/15 figure.

Bet 37: the original single-seed result. This bet's replication.
Bet 49: production-default shootout against LoRA-r4 and full FT.
Bet 60: the noise-floor negative control that tightened the framing.
Bet 61: the confusion matrix that isolated the personalisation signal.
Bet 52: norm-only + ternary base composition.

Why it matters

A 15/15 STRICT pass is the strongest evidence the catalogue has produced for any single primitive. It moved norm-only from "interesting result" to "production default for per-user adapters." The wire-format chapter (Bets 48, 52, 53, 54) all assume norm-only as the per-user adapter, which is justified by this replication and by Bet 49's shootout.

The bet is also methodologically important. It's the model the catalogue uses for any future production-default decision: ≥ 5 seeds × ≥ 3 eval texts, all reporting the same direction, with explicit per-cell variance. This standard is what separates a single-shot result from a deployable one. The catalogue holds itself to this standard for any production wire-format decision; Bet 46 is the prototype.