Text set in serif type with a dotted bar in the margin was drafted by an LLM and sometimes reviewed by the author. The rest is the author's own. How to read this site
msize_long
20260706_130025_msize_long · complete · published 2026-07-16 · seed None
Intent
Run on 2026-07-06 between 13:00 and 16:52, about 3 hours 52 minutes, with two
children in parallel (--jobs 2). The run is a
warm restart of 300
continuations in all (two widths, fifteen Z values, ten
seeds each) resumed
from the msize checkpoints at 1200
steps, with the source sweep left untouched.
Every anchor waypoint reproduced the msize accuracies exactly, the largest gap
being 0.00 points across the 600 checked cells in each child, so each
continuation began from the model msize had ended on.
Two questions carry over from msize. The first is whether the 512-wide model's seed split near Z = 1.0, where its replicates divided into a group that learned agreement and a group that did not, is a symptom of too little training rather than a fixed property of that width; this was the open question msize left. The second is whether the window in which more training first helps and then hurts, found along the depth axis in depth_long, also exists along the width axis.
Background
This experiment is the width-axis counterpart to depth_long. It continues the two widest models from the msize sweep, the 256-wide and the 512-wide network, from 1200 to 4800 training steps, resuming their checkpoints rather than starting over.
The 256-wide model serves at the same time as the depth study's control, since it is the two-layer architecture used as depth's L = 2 arm.
The 512-wide arm is where msize found its non-monotonic result.
Hypothesis
If width behaves like depth, then more training should resolve the 512-wide model's seed split: the point at which it transitions into agreement should move toward lower Z as the training budget grows, its spread across seeds should shrink, and its accuracy past the transition should then begin to slide once training runs beyond the point of best return. The 256-wide model should show the same rise-then-slide shape, with an earlier and wider useful window. The alternative outcome is that the 512-wide model's seed split is still present at 4800 steps, which would point to an instability that comes from the model having more capacity than the task needs, rather than from the training budget, as the explanation for that non-monotonic result.
Setup
data regime (all arms): single-dataset-recycled (continued) — 1 dataset(s) × 4800 iters each = 16.0 epochs per dataset
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout c626fc6
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yaml⤷ This experiment continues from another run — reproduce the source sweep(s) first:
runs/20260704_225157_msize/n-embd_256, runs/20260704_225157_msize/n-embd_512
Checks
PASS children_complete PASS replicate_coverage PASS instrument_consistent PASS anchor_reproduces_source[n-embd_256] PASS anchor_reproduces_source[n-embd_512]
Results — unseen_mismatch (mean ± sd over 10 seeds)
Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | peak over α (end of training) | per-seed | flags |
|---|---|---|---|
| n-embd_256 | 77.7 ±4.5 @ α=2.1 | ||
| n-embd_512 | 77.1 ±11.0 @ α=1.9 | ⚠ seed_split: range 48.7-93.0 (1 low / 2 high of 10) |
Conclusions
The two widths answer the pre-registered question in opposite ways, and the answer for the wider model is the one this experiment was built to settle. Continuing the 512-wide model to four times its original training budget does not close its seed split, so that split is a property of the width and not a sign of too little training. The 256-wide model behaves the other way, with a clean climb into agreement that has already reached its best point near the paper's budget and then slips back down. Read beside depth_long, the two changes that both count as making the model bigger, more width and more depth, come apart.
Referenced by (2 direct, 3 transitive)
Direct references:
The 256-wide network, which is also the paper's configuration and the depth study's two-layer arm, shows the rise-then-slide shape cleanly. Its peak unseen_mismatch, read at five waypoints between 1200 and 4800 steps, is highest at the shortest budget and falls at every longer one.
| budget (steps) | 256-wide peak | 256-wide onset | 512-wide peak |
|---|---|---|---|
| 1200 | 93.1 at Z = 1.2 | Z = 1.1 | 62.7 at Z = 3.0 |
| 1500 | 88.5 at Z = 1.3 | Z = 1.0 | 69.4 at Z = 1.8 |
| 2400 | 83.3 at Z = 1.5 | Z = 1.1 | 70.6 at Z = 3.0 |
| 3600 | 78.1 at Z = 1.5 | Z = 1.2 | 71.3 at Z = 2.5 |
| 4800 | 77.7 at Z = 2.1 | Z = 1.5 | 77.1 at Z = 1.9 |
The narrow model reaches its best accuracy at 1200 or 1500 steps for thirteen of the fifteen Z settings, so the paper's 1200-step budget sits at the top of its useful window rather than short of it. Training past 1500 only lowers the peak, from 93.1 to 77.7, and moves the left edge of its transition from Z = 1.1 to Z = 1.5 as the low-Z side gives way first.
Referenced by (1 direct, 3 transitive)
Direct references:
The ten replicates stay close together the whole time: the standard deviation across seeds, taken over the transition band from Z = 0.9 to 2.1, rises only from 6.4 at 1200 steps to 8.9 at 4800, and the report raises no seed-split flag on any of the narrow model's conditions.
The 512-wide network never settles, and its spread across seeds is the result. The same band standard deviation that stays between 6.4 and 8.9 for the narrow model runs 28.8 at 1200 steps and 32.5 at 4800 for the wide one, so four times the training leaves the split as wide as it began. The wide model's peak mean does drift upward, from 62.7 to 77.1, and seven of its fifteen Z values are still improving at 4800, so on average it is slowly getting better. Its individual seeds are not converging. At Z = 1.0, the setting where msize first saw the split, four of the ten seeds still score 0.0 at 4800, having settled on agreeing with the nearest noun, while the other six run from 51.8 to 96.8, so the mean of 43.9 there describes no run that happened. At the wide model's best setting, Z = 1.9, the split is milder but still present: one seed at 48.7, two at 86.4 and 93.0, and the other seven between 74.3 and 81.9, which is the dispersion the report flags. By 4800 the two widths reach almost the same peak mean, 77.7 for the narrow model and 77.1 for the wide one, but the narrow model gets there with a standard deviation of 4.5 across seeds and the wide model with 11.0.
Referenced by (1 direct, 3 transitive)
Direct references:
The final-iteration curves for both widths are in
images/comparison_unseen_mismatch.png and their peaks side by side in
images/peak_vs_n-embd.png, both at the 4800-step snapshot.
Set against depth_long, the two size axes separate. Along the depth axis a seed split of the same kind did resolve with training: in that run the four-layer model's spread at Z = 1.5 fell from 36.2 to 5.3 by 2400 steps, and its transition edge moved left from Z = 1.7 to Z = 1.1 over the same span, so depth's delay was a cost the model paid off with more budget. Along the width axis nothing of the sort happens at a comparable budget, since no clean transition edge forms before 4800 and the spread does not fall. Adding depth and adding width therefore mean different things for what more training can repair, even though both enlarge the model.
Referenced by (1 direct, 3 transitive)
Direct references:
A last contrast points the same way. The 256-wide model's high-Z tail is flat: at Z = 3.0 it holds 74 at every budget and shows none of the tail erosion the deeper models showed as they trained on. The falloff at high Z that depth_long found belongs to depth and does not appear at two layers.
The result leaves three questions open, each answerable from checkpoints already on disk:
- Whether the 512-wide split is a fixed lottery, with the same seeds failing from their initialization, or churn, with seeds entering and leaving agreement over training. Every waypoint's checkpoints are saved, so this is analysis only.
- Whether a much longer 512-wide continuation, to around 19,200 steps from this run's 4800 checkpoints, ever converges, which would separate a model that never resolves from one that resolves far more slowly.
- Whether the instability is an artifact of optimization. At a fixed learning rate of 6e-4 the wide model may be mistuned, and a learning rate scaled down with width, the muP 1/width rule, could remove the split, which would confirm the optimization reading.
Comparison figures
Children
| Child | n-embd | peak unseen_mismatch | Status |
|---|---|---|---|
| n-embd_256 | 256 | 93.1 | done |
| n-embd_512 | 512 | 77.1 | done |