msize

20260704_225157_msize · complete · published 2026-07-05 · seed 42

Intent

How does embedding width (n_embd) affect subject-verb agreement generalization? One axis of model capacity only — depth (2 layers) and heads (4) are held fixed throughout.

Hypothesis

Below n_embd < vocab (167), the weight-tied embedding must compress tokens into shared directions, forcing distributed representations. Peak unseen_mismatch accuracy should fall as capacity shrinks, with a floor width where agreement generalization fails entirely. All children hold depth (2 layers), heads (4), max_iter (1200), and lr (6e-4) constant — parameter count is the only variable.

Conclusions

Run 2026-07-04 22:51 → 2026-07-05 06:58 (~8.1h wall, sequential, GTX 1660 Ti). All six children completed; 1,860 runs, seed 42, no failures or resumes.

The hypothesis was half right: there IS a capacity floor, but the ladder is a sequence of qualitatively different solutions, not a single degrading curve — and the top of the ladder is non-monotonic.

Per-width unseen_mismatch summary (peak mean over 10 seeds, and its Z):

| n_embd | ~params | peak UMM | at Z | character of solution | |-------:|--------:|---------:|-----:|-----------------------| | 16 | ~10k | 51.1% | — | nothing: chance on all four conditions at every Z | | 32 | ~32k | 48.7% | 1.8 | soft recency: match ~80%, mismatch below chance | | 64 | ~112k | 71.6% | 2.6 | hard recency below Z≈2 (mismatch 0%), sharp flip to agreement above | | 128 | ~421k | 89.2% | 1.3 | same flip, moved left, higher ceiling | | 256 | ~1.63M | 93.1% | 1.2 | baseline; gains saturating vs 128 | | 512 | ~6.4M | 76.2% | 2.2 | NON-MONOTONIC: bimodal across seeds (see below) |

Findings:

1. Capacity floor: d=16 learns no agreement of any kind — flat 49–52% on all conditions across all Z (10 seeds, ±1–2%).
2. Recency rung: d=32 has enough capacity for a number-copying rule but learns the WRONG one — agree with the most recent noun (the PP object). Match conditions rise to ~80% while mismatch falls to 20–48%: the classic attraction error / B_recent solution.
3. Phase transition in Z, shifting with width: d=64 commits fully to recency below Z≈2 (mismatch 0%) then flips to genuine subject agreement (UMM 70%+ at Z≥2.5). Each doubling of width moves the flip left and raises the ceiling: 64→(Z≈2.2, 72%), 128→(Z≈1.2, 89%), 256→(Z≈1.1, 93%). Capacity and distributional concentration appear partially interchangeable: small models need concentrated pairings to escape recency.
4. d=512 breaks the monotone trend — via seed bimodality, not uniform degradation. Mean UMM peaks at only 76.2% (Z=2.2), but per-seed spreads are enormous: at Z=1.0, min 0.0% / max 95.3% (std 28.8). At fixed budget (1200 iters, lr 6e-4), the widest model's solution selection becomes a seed lottery between recency and agreement. Whether this is undertraining (the documented fixed-budget confound) or a genuine overparameterization-instability effect needs the follow-up below.

Follow-ups: - Waypoint run on d=512 (and d=256 as control) to separate undertraining from instability: imsyn run --n-embd 512 --waypoints 300 600 1200 2400 4800 ... - Per-seed bimodality analysis at (d=512, Z≈1.0) and (d=64, Z≈2.0–2.4) — the transition regions — connecting to the B_recent bimodality work. - The d < vocab compression threshold (between 128 and 256) shows no visible discontinuity in these curves; the interesting boundaries are much lower (16→32→64). Hypothesis's compression framing not supported as stated.

Figures: images/comparison_by_condition.png, images/comparison_unseen_mismatch.png, images/peak_vs_n-embd.png (note: mean of per-seed peaks is misleading for d=512 given bimodality — see per-seed follow-up).

Comparison figures

Children

experiment.yaml