Text set in serif type with a dotted bar in the margin was drafted by an LLM and sometimes reviewed by the author. The rest is the author's own. How to read this site

msize

20260704_225157_msize · complete · published 2026-07-16 · seed 42

Intent

Run on 2026-07-05 in about 8.1 hours on a single GTX 1660 Ti, training the six widths across the α sweep in sequence: 1,860 models in all (six widths × thirty-one α values × ten seeds), base seed 42, no failures or resumes.

Intent: How embedding width gates agreement generalization \@{msize-intent} This experiment asks how a model's embedding width, n_embd, affects whether it learns subject-verb agreement well enough to generalize to noun-verb pairings it never trained on. It moves one axis of model capacity and holds the rest fixed. Every model has two layers and four attention heads, so the only thing that changes from arm to arm is width, the size of the weight-tied embedding and the residual stream it feeds.

Background

Background: The compression argument \@{msize-background} When n_embd is below the vocabulary size of 167, the weight-tied embedding cannot give each token its own direction and has to pack them into shared ones, which should force more distributed representations.

Hypothesis

Hypothesis: Peak accuracy falls with width until generalization fails \@{msize-hypothesis} The prediction is that peak unseen_mismatch accuracy falls as width shrinks, and that at some narrow width agreement generalization fails outright. Every arm holds depth at two layers, four heads, the 1200-iteration budget, and the 6e-4 learning rate fixed, so parameter count is the only variable.

Setup

model: GPT (nanoGPT-derived), weight-tied embeddings · block 50 · dropout 0.1 · vocab 167 tokens
training: AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 10 seeds (base 42)
training data: PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code: commit a405b62

arm	n_embd	n_layer	n_head	~params
n-embd_016	16	2	4	10,064
n-embd_032	32	2	4	32,416
n-embd_064	64	2	4	113,984
n-embd_128	128	2	4	424,576
n-embd_256	256	2	4	1,635,584
n-embd_512	512	2	4	6,416,896

data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout a405b62
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS instrument_consistent

Results — unseen_mismatch (mean ± sd over 10 seeds)

Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition

Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)

arm	peak over α (end of training)	flags
n-embd_016	51.1 ±2.0 @ α=2.5
n-embd_032	48.7 ±2.3 @ α=1.8
n-embd_064	71.6 ±13.8 @ α=2.6
n-embd_128	89.2 ±5.9 @ α=1.3
n-embd_256	93.1 ±9.1 @ α=1.2
n-embd_512	76.2 ±14.2 @ α=2.2	⚠ seed_split: range 48.2-90.8 (2 low / 4 high of 10)

Conclusions

Conclusion: The width ladder climbs through distinct solutions and breaks at 512 \@{msize-conclusion} Conclusion: The width ladder is a sequence of distinct solutions \@{msize-solution-ladder} The hypothesis is partly borne out. There is a capacity floor below which the model learns no agreement, and across the lower and middle of the width ladder peak accuracy does fall as width shrinks. What the hypothesis did not anticipate is that the ladder is a sequence of qualitatively different solutions rather than one curve that degrades smoothly, and that the widest model breaks the upward trend instead of extending it. Referenced by (1 direct)Direct references:The two size axes come apart Peak unseen_mismatch, the hardest condition and the one the experiment is read through, summarizes each width. The value is the mean over ten seeds at the α where that mean is highest. n_embd approx params peak unseen_mismatch peak α solution 16 10,064 51.1 2.5 no agreement; chance on all four conditions at every α 32 32,416 48.7 1.8 the nearest-noun shortcut in some seeds, chance in the rest 64 113,984 71.6 2.6 shortcut at spread pairings, agreement at concentrated ones 128 424,576 89.2 1.3 agreement, high and consistent across seeds 256 1,635,584 93.1 1.2 agreement, the highest peak; little gain over 128 512 6,416,896 76.2 2.2 non-monotonic; the seeds split between the two solutions The two narrowest widths sit at or below the shortcut. Width 16, at about ten thousand parameters, stays at chance on all four conditions across the whole α sweep: seen_match 51.5, seen_mismatch 50.4, unseen_match 50.9, and unseen_mismatch 51.1, each within a point or two of the 50 chance line and none tracking α, so its peak α of 2.5 is only where sampling noise happened to land. It has too few parameters to represent either agreement or the shortcut. Width 32 is the first with enough capacity for a number-copying rule, but the rule it finds is recency, agreeing with the nearest noun rather than with the subject. Its match conditions rise to about 80 (seen_match 80.3 and unseen_match 80.5, both at α = 0.8) while its mismatch conditions never clear chance, with peaks of 48.2 and 48.7 and values near 20 at the α where match peaks, because the recency seeds score 0 there. That 80 is a seed split rather than a typical run: at α = 0.8 six of the ten seeds copy the nearest noun, scoring 100 on match and 0 on mismatch, and the other four sit at chance on everything, which the report flags on both match conditions (4 low / 6 high of 10). Width 64 is where a phase transition in α appears. At spread-out pairings, α from 0 up to about 1, the match conditions are perfect and both mismatch conditions are 0: the model copies the nearest noun exactly, so it passes every match probe and fails every mismatch one. As α rises past about 2 the mismatch accuracy climbs, reaching 71.6 unseen_mismatch and 79.0 seen_mismatch at α = 2.6. Width 64 can therefore represent agreement, but only when the training pairings are concentrated enough to push it off the shortcut; on spread pairings it reverts. The flip is incomplete even at its best, with seen_mismatch at α = 2.6 flagged as a seed split (range 49.7 to 98.7, 3 low / 5 high), so some seeds have crossed to agreement while others remain on recency. The next two widths learn agreement outright, and the return to added width narrows. Width 128 reaches 89.2 unseen_mismatch at α = 1.3 with every seed between 79.6 and 97.7 and no split, alongside perfect match conditions and 99.0 seen_mismatch. The transition that width 64 reached only at α = 2.6 now sits at α = 1.3, so more capacity buys the same rule at less concentrated data. Width 256, the paper's default, posts the ladder's highest peak, 93.1 unseen_mismatch at α = 1.2, with 99.6 seen_mismatch and perfect match conditions. The gain over width 128 is about four points, and the spread is a little wider, with one seed at 69.8 lifting the standard deviation to 9.1, so doubling the width from 128 to 256 adds a few points rather than another rung. The widest model does not continue the climb. Result: Width 512 breaks the climb with a seed bimodality \@{msize-width512-bimodal} Width 512 peaks at 76.2 unseen_mismatch at α = 2.2, below both 256 and 128, and the shortfall is a seed bimodality rather than an even decline. At the peak α of 2.2 the ten seeds run from 48.2 to 90.8 with two at chance (flagged 2 low / 4 high). The split is sharper at α = 1.0, where the per-seed values are 48.4, 49.8, 95.3, 45.8, 0.0, 48.3, 0.0, 48.3, 50.3, and 0.0: three seeds have collapsed onto the shortcut, one has reached 95.3, and the rest sit at chance, spanning 0.0 to 95.3 with a standard deviation of 28.8. Referenced by (6 direct, 4 transitive)Direct references:Complementing the msize width ladderMore budget repairs the onset delay but not the peakWidth-axis counterpart to depth_longThe 512-wide seed split is width-intrinsic, not undertrainingA split that is real at a fixed budget, and a budget that is doing much of the workThe two size axes come apartTransitive (depth 1):Width and depth are different axes, and bigger is not betterThe peak is universal, capacity caps it, and the holdout fraction is second orderTransitive (depth 2):The intermediate peak is real but it is a slice of a moving targetA robust peak whose location is set by the budget The seen conditions stay high (seen_match 99.7, seen_mismatch 96.7) while both unseen conditions split (unseen_match 86.0, unseen_mismatch 76.2), so it is specifically generalization to unseen pairings that has become seed-dependent. At the fixed 1200-iteration budget the widest model does not reliably choose agreement over the shortcut; which one a run lands on depends on its seed. Whether that reflects undertraining, the confound that comes with holding the step budget fixed across arms, or a genuine instability of the overparameterized model is the open question the follow-ups address. Result: The capacity floor and the width ladder \@{msize-capacity-ladder} Read against the hypothesis, the capacity floor is real and the predicted fall with shrinking width holds from 256 downward (93.1, then 89.2, 71.6, and chance at 32 and 16). Referenced by (8 direct, 3 transitive)Direct references:The embedding capacity ceilingThe published result reproduces and separates memory from ruleComplementing the msize width ladderCached datasets reproduce fresh resultsmsize_longWidth-axis counterpart to depth_longThe two size axes come apartHow the result scalesTransitive (depth 1):More training does not settle the wide modelThe peak is universal, capacity caps it, and the holdout fraction is second orderTransitive (depth 2):A robust peak whose location is set by the budget The two parts the data does not support are the shape at the top and the location of the boundary. The widest model is worse than the paper's default rather than better or equal, so peak accuracy is not monotone in width. And the compression argument pointed to a boundary where n_embd crosses the vocabulary size of 167, between widths 128 and 256, but the curves show no discontinuity there; the qualitative changes all happen much lower, at 16, 32, and 64. Result: Capacity and concentration trade off in the transition α \@{msize-transition-alpha} What moves smoothly instead is the transition α: narrower models need more concentrated pairings to leave the shortcut (64 at α ≈ 2.6, 128 at 1.3, 256 at 1.2), so within this range capacity and distributional concentration are partly interchangeable. One caution about the figures. The peak-versus-width figure (images/peak_vs_n-embd.png) plots each width's mean, and for width 512 that mean falls in the empty gap between the two seed groups, describing no actual run. Read it together with the per-seed values rather than on its own. Two threads follow from the width-512 result: A waypoint or continued run at width 512, with width 256 as a control, would separate undertraining from instability. If the split closes with more training, the fixed budget was the cause. A per-seed analysis at the two transition regions, width 512 near α = 1.0 and width 64 near α = 2.0 to 2.4, would connect this seed split to the recency-bimodality work, since those are the α where seeds divide between the shortcut and agreement.

`n_embd`	approx params	peak `unseen_mismatch`	peak α	solution
16	10,064	51.1	2.5	no agreement; chance on all four conditions at every α
32	32,416	48.7	1.8	the nearest-noun shortcut in some seeds, chance in the rest
64	113,984	71.6	2.6	shortcut at spread pairings, agreement at concentrated ones
128	424,576	89.2	1.3	agreement, high and consistent across seeds
256	1,635,584	93.1	1.2	agreement, the highest peak; little gain over 128
512	6,416,896	76.2	2.2	non-monotonic; the seeds split between the two solutions

Comparison figures

Children

Child	n-embd	peak unseen_mismatch	Status
n-embd_016	16	51.1	done
n-embd_032	32	48.7	done
n-embd_064	64	71.6	done
n-embd_128	128	89.2	done
n-embd_256	256	93.1	done
n-embd_512	512	76.2	done

experiment.yaml