Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
scaling → model-scale
model-scale
investigation
Intent
This investigation asks how a model's size gates its ability to learn
subject-verb number agreement well enough to generalize to unseen
pairings. Size has two axes here, embedding
width (n_embd) and depth (n_layer), and the member
experiments vary each against the full α sweep: first the
width ladder, then depth, then longer-budget continuations of
both to separate what is a property of the architecture from what is a
sign of undertraining at the paper's budget.
Referenced by (1 direct)
Direct references:
Conclusions
The paper's configuration, 256 wide and two layers, sits near the top of the size range this task rewards, and the two ways of making the model bigger fail in different ways.
Width sets which solution the model can represent, following a capacity ladder from agreeing with the nearest noun through memorization to the general rule (the solution ladder), and past the paper's width the wide model splits by seed rather than improving (the 512 split). Depth is not capacity in the same sense: one layer already suffices for the rule (depth_v1), and added layers delay and degrade acquisition (delays and degrades) rather than failing to represent it.
Referenced by (1 direct)
Direct references:
The continuations settle what the paper's budget left ambiguous. The depth penalty at 1200 steps looked like it might be undertraining, and for the onset it partly is: given four times the budget the deeper arms' onset recovers (depth_long), but the recovery splits by seed (the effect splits) and longer training then erodes the peak (peak erosion), leaving a stopping window rather than a clean win. The width split does not recover at all: four times the budget leaves the 512-wide cohort split, so the split is width-intrinsic (msize_long), and it persists across every waypoint measured (the split persists). The 256-wide model, by contrast, has already peaked near the paper's budget and slips with more training (the budget optimum), the same post-peak erosion the training-dynamics work sees at fixed size. Read together the axes dissociate (msize_long): more width changes which basin a seed lands in, more depth changes how long it takes to get there.
What would change this reading. The width ladder tops out at 512 and the depth ladder at four layers, both at the default vocabulary; the vocabulary-scaling result already shows the capacity story bends with vocabulary size, so a wider grid over width × vocabulary is the natural next probe, and the seed-level mechanics of the 512 split belong to the bimodality thread, bimodality.
Referenced by (2 direct, 1 transitive)
Direct references:
Transitive (depth 1):
Experiments
-
20260706_130025_msize_long 2026-07-16
Run on 2026-07-06 between 13:00 and 16:52, about 3 hours 52 minutes, with two children in parallel (
--jobs 2). The run is a warm restart of 300 continuations in all (two widths, fifteen Z values, ten seeds each) resumed from the msize checkpoints at 1200 steps, with the source sweep left untouched. Every anchor waypoint reproduced the msize accuracies exactly, the largest gap being 0.00 points across the 600 checked cells in each child, so each continuation began from the model msize had ended on.Intent: Is the 512-wide seed split undertraining, and does width have a stopping window \@{msizelong-intent}Two questions carry over from msize. The first is whether the 512-wide model's seed split near Z = 1.0, where its replicates divided into a group that learned agreement and a group that did not, is a symptom of too little training rather than a fixed property of that width; this was the open question msize left. The second is whether the window in which more training first helps and then hurts, found along the depth axis in depth_long, also exists along the width axis.
-
20260705_211010_depth_long 2026-07-16
Ran 2026-07-05 21:10 to 2026-07-06 01:32, about four hours and twenty minutes, with
--jobs 2, and was the first production use ofimsyn run continue.It continues the two children of
runs/20260705_133844_depth, the three-layer and four-layer arms, which stay read-only. It takes that run's three-layer and four-layer per-iteration checkpoints and continues each from 1200 to 4800 training iterations, a fresh-AdamW warm restart on each model's own dataset, to see whether the rightward-shifted, seed-bimodal recency-to-agreement transitions recover the two-layer behavior once they are given four times the budget. The sweep covers the two transition bands densely, alpha from 0.9 to 2.1, with tail probes at 2.5 and 3.0. -
20260705_133844_depth 2026-07-16
Run 2026-07-05, 13:38 to 17:34, about 3h56m of wall-clock with
--jobs 4. This was the first production use of the parallelimsyn exprunner and the content-keyed dataset cache (issue #2); run sequentially it would have taken roughly eleven hours. All four children completed with no failures or resumes.How does depth (
n_layer) affect subject-verb agreement generalization? -
20260704_225157_msize 2026-07-16
Run on 2026-07-05 in about 8.1 hours on a single GTX 1660 Ti, training the six widths across the α sweep in sequence: 1,860 models in all (six widths × thirty-one α values × ten seeds), base seed 42, no failures or resumes.
This experiment asks how a model's embedding width,
n_embd, affects whether it learns subject-verb agreement well enough to generalize to noun-verb pairings it never trained on. It moves one axis of model capacity and holds the rest fixed. Every model has two layers and four attention heads, so the only thing that changes from arm to arm is width, the size of the weight-tied embedding and the residual stream it feeds.