Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

paper_long_iter

20260506_205133_paper_long_iter · complete · published 2026-07-16 · seed 42

Intent

Backfilled 2026-07-05 from a pre-manifest run executed 2026-05-06, so the manifest and this report are retroactive. The original invocation used the pre-rebase CLI grammar, and the args recorded in experiment.yaml are its current-grammar equivalent. The sweep is 31 Z values from 0.0 to 3.0 in steps of 0.1 at ten seeds each, 310 trained models in all, from base seed 42.

Intent: How agreement generalization evolves over training across the Z sweep \@{paperlong-intent} How does subject-verb agreement generalization (unseen_mismatch) evolve over the course of training, across the full Z sweep?

Background

Background: The paper's fixed-budget snapshot \@{paperlong-background} The paper reports accuracy at a fixed 1200-iteration budget; this run extends training to 10,000 iterations with checkpoints at six waypoints to see whether the fixed budget under- or over-shoots the agreement solution at each Z.

Hypothesis

Hypothesis: Generalization is not monotone in training time \@{paperlong-hypothesis} Agreement generalization is not monotone in training time. At high Z (concentrated pairings) the model should reach its agreement solution early and then drift away from it with continued training; at low Z it may never reach one. If so, the paper's fixed-budget curve conflates "cannot learn agreement" with "peak missed by the snapshot time".

Setup

model: GPT (nanoGPT-derived), weight-tied embeddings · block 50 · dropout 0.1 · vocab 167 tokens
training: AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 10000 iterations · fresh init per replicate (seeds derived from base seed) · 10 seeds (base 42)
training data: PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code: commit 352694f

data regime (all arms): single-dataset-recycled — 1 dataset(s) × 10000 iters each = 33.33 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 352694f
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS instrument_consistent

Results — unseen_mismatch (mean ± sd over 10 seeds)

Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition

Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)

arm	peak over α (end of training)	per-seed	flags
sweep	75.8 ±4.0 @ α=2.1

Conclusions

Conclusion: The peak is a moving target that longer training erodes \@{paperlong-conclusion} Conclusion: The budget, not the grammar, sets the peak location \@{paperlong-budget-sets-peak} The training budget, not the grammar, fixes where unseen_mismatch peaks in α. The paper reads its central curve at a single 1200-step snapshot and places the best generalization at an intermediate α. This run shows that the location of that peak slides as training continues, so the 1200-step reading captures one slice of a moving target rather than a fixed property of the task. Referenced by (2 direct)Direct references:Concentration does not decide the binding, and above 2.3 it undoes itA robust peak whose location is set by the budget Result: The best-generalizing α slides with the training budget \@{paperlong-moving-peak} Read at each waypoint the α that gives the highest mean unseen_mismatch moves steadily downward as training lengthens. At 300 steps the best setting is α = 2.2 (91.2); at 600 steps it is α = 1.6 (94.8, the highest single cell anywhere in the run); at 1200 steps it is α = 1.2 (93.1). That 1200-step value reproduces the paper's Figure 2 result exactly, which also peaks at 93.1 at Z = 1.2, so the paper's budget happens to land the ridge at α ≈ 1.2. A shorter budget would have placed it higher. Referenced by (4 direct)Direct references:The coarse grid left the peak timing openPeak timing splits into three regimes and arrives earlier with ZThe published result reproduces and separates memory from ruleA robust peak whose location is set by the budget The six per-waypoint panels are in images/waypoint_evolution.png. Seen the other way, concentrated pairings reach their agreement solution sooner. Each setting that learns agreement has a waypoint at which its unseen_mismatch is highest, and that waypoint comes earlier the higher α is: settings at α ≥ 2.1 peak by 300 steps, α from 1.6 to 2.0 by 600, α from 1.1 to 1.5 by 1200, and α from 0.6 to 1.0 only by 2400. This is collocational bootstrapping timing: the more repetitive the pairings, the faster the model builds the verb-to-subject association that agreement needs. Result: Continued training erodes the agreement peak \@{paperlong-post-peak-erosion} Every setting that reaches agreement then loses ground with further training. From its peak to the end of the 10,000-step budget, each α from about 1.0 upward falls by roughly 10 to 25 points and settles near 70. The largest drops are the settings that peaked highest: α = 1.6 falls from 94.8 at 600 steps to 71.8 at 10,000, and α = 1.2 from 93.1 at 1200 to 68.5. Read at the full budget the sweep still has an intermediate peak, but a lower and flatter one. The best over α at the end of training is 75.8 ± 4.0 at α = 2.1, well below the 94.8 the run reached transiently, and its ten seeds are tight (70.1 to 82.6, with no seed split). Referenced by (4 direct, 2 transitive)Direct references:Concentration does not decide the binding, and above 2.3 it undoes itThe two estimators agree, so the comparison is safeWidth and depth are different axes, and bigger is not betterA robust peak whose location is set by the budgetTransitive (depth 1):The intermediate peak is real but it is a slice of a moving targetThe peak is universal, capacity caps it, and the holdout fraction is second order Result: Validation loss keeps improving while agreement generalization erodes \@{paperlong-loss-accuracy-dissociation} The drift is not the language model overfitting its training data. Validation loss keeps falling well after unseen_mismatch has turned over. At α = 1.2 the mean validation loss goes from 2.02 at 300 steps down to 1.84 at 10,000, monotonically, and is still 1.94 at the 1200-step accuracy peak; over that same span from 1200 to 10,000 steps unseen_mismatch drops from 93.1 to 68.5. Across the sweep the validation perplexity reaches its minimum at 4800 or 10,000 steps for every α, long after the accuracy peak at 300 to 1200 steps. The language-model objective and the agreement generalization it induces come apart after the peak: the model keeps predicting the held-out sentences better while its grasp of agreement on unseen pairings erodes. Referenced by (1 direct)Direct references:A robust peak whose location is set by the budget The replay behind this is in images/val_loss_story.png, images/val_loss_vs_unseen_select.png, and images/phase_diagram.png, with the underlying tables in runs/_analysis/20260506_205133_paper_long_iter_val_loss/. At the uniform end the model never learns agreement at all. Every setting at α ≤ 0.6 stays below about 43 at every waypoint, and below the 50 chance line, which means it is agreeing with the nearest noun rather than the subject. Longer training does not rescue these settings; they stay flat or noisy across all six waypoints. This is the other thing the fixed-budget curve was conflating: at low α the model cannot learn the rule, while at high α it learns the rule early and the snapshot simply misses the peak. The seen conditions confirm the models trained normally, so the drift is specific to held-out generalization and not a training failure. At the end of training seen_match is saturated at 100.0 ± 0.0 and seen_mismatch reaches 99.7 ± 0.2 at its concentrated end, so the models handle the attractor trap on the pairings they were trained on. unseen_match ends at 81.5 ± 15.8 at its best α (0.4), but with one seed at 39.7 against the rest up to 99.2, a split worth remembering when reading that condition's mean. A denser follow-up, runs/20260509_153511_peak_umm_iter, brackets each α's true peak iteration more finely than these six waypoints, since for several settings the peak falls between them.

Comparison figures

Children

Child	peak unseen_mismatch	Status
sweep	94.8	done

experiment.yaml