Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
peak_umm_iter
20260509_153511_peak_umm_iter · complete · published 2026-07-16 · seed 42
Intent
These notes were backfilled on 2026-07-05 from a run executed on 9 and 10 May
2026, before the experiment-manifest format existed, so both the manifest and
this write-up are retroactive. The run is Stage 2 of the two-stage
imsyn run peak-iter pipeline. Its Stage 3 analysis, the merged
coarse-and-dense tables merged_metric.csv and
refined_peaks.csv and the two figures copied into
images/, lives in
runs/_analysis/20260509_153511_peak_umm_iter/. The run
directory's measurements.csv holds only each Z's nine-point dense
bracket, so the low-Z peaks that fall at the source sweep's coarse waypoints
appear only in those merged tables, and the per-α peaks reported below are
read from them. The --source-sweep path in experiment.yaml
points at the source run's location after the same backfill.
This run finds, more precisely for every Z, the iteration at which unseen_mismatch peaks. It starts from the coarse argmax per Z in the source sweep, then samples a dense nine-point geometric bracket of iterations around it with ten seeds at each point, so the peak can be read from a finer grid.
Background
paper_long_iter, the paper's companion long-iteration sweep, recorded unseen_mismatch at only six coarse training iterations, which is too few to say when each Zipfian setting reaches its best generalization.
Hypothesis
The six coarse waypoints under-sample the peak in three specific ways. For mid and high Z the true peak iteration should fall inside the coarse bracket rather than at one of its endpoints; the peak iteration should decrease monotonically with Z; and wherever the coarse grid straddled a narrow peak, the refined peak value should exceed the coarse one.
Setup
data regime (all arms): single-dataset-recycled — 1 dataset(s) × 10000 iters each = 33.33 epochs per dataset
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 5423d0c
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yamlChecks
PASS children_complete PASS replicate_coverage PASS instrument_consistent
Results — unseen_mismatch (mean ± sd over 10 seeds)
Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | peak over α (end of training) | per-seed | flags |
|---|---|---|---|
| dense_bracket | 37.7 ±7.3 @ α=0.1 |
Conclusions
Reading each Z at the training iteration where its unseen_mismatch is highest sorts the sweep into three regimes. The dense bracket of waypoints bears out the hypothesis through the middle of the Z range and shows where the idea of a peak stops applying.
Referenced by (1 direct)
Direct references:
At the uniform end, from Z = 0.0 to about 0.7, there is no interior peak. unseen_mismatch is still drifting upward at the last waypoint the run sampled, so the highest value it reaches sits at the right edge of the bracket, between 9123 and 10000 steps, rather than at a maximum the run passed through. The values are low there, from 30.39 at Z = 0.2 to 46.25 at Z = 0.7. Uniform pairings do not reach an agreement peak inside 10000 steps, and what the pipeline records as their peak is the latest iteration it measured.
Referenced by (1 direct)
Direct references:
Through the transition band, Z from about 0.8 to 1.1, interior peaks appear between roughly 1400 and 2000 steps, with wide disagreement across the ten seeds. Z = 0.8 peaks at 1427 steps and 62.99, Z = 0.9 at 1697 and 70.74, Z = 1.0 at 2018 and 74.93, and Z = 1.1 at 2018 and 80.6. The spread is at its largest here and narrows as Z rises: the standard deviation across seeds falls from 31.4 at Z = 0.8 to 13.0 at Z = 1.1, and at the peak waypoint the worst seed climbs from 0.0 at Z = 0.8 to 53.2 at Z = 1.1 while the best stays near 100. This is the band where seeds divide between agreeing with the subject and agreeing with the nearest noun, the seed bimodality the B_recent work studies.
At the concentrated end, from Z = 1.2 upward, the peaks are early and high, and they move earlier as Z rises. The peak iteration falls from about 2000 steps at Z = 1.2 to 245 steps at Z = 3.0, while the peak value holds from the mid 80s to the mid 90s, from 85.77 at Z = 1.3 to the sweep's highest, 95.67 at Z = 1.6, reached at 849 steps.
Referenced by (2 direct)
The seeds also come together: at Z = 1.6 the ten of them run from 90.0 to 100.0, against the 0.0-to-96.4 range at Z = 0.8. For the most concentrated settings the peak arrives before the coarse grid could have seen it. The coarse argmax for every Z from 2.1 to 3.0 sat at 300 steps, the earliest coarse waypoint, and the dense bracket, which reaches down to 100 steps, places the true peak between 245 and 383 steps, at 245 for Z = 2.8 through 3.0.
Across the interior band the peak iteration decreases with Z, which
reproduces the coarse trend from paper_long_iter at finer
resolution (images/peak_iter_vs_alpha.png). The decrease is a
direction rather than a strict law: there are local reversals, for instance
Z = 1.4 peaking at 849 steps and Z = 1.5 at 1200.
Where the coarse grid
straddled a narrow peak, the dense bracket recovered a higher value at an
iteration the coarse grid had skipped (images/peak_iter_grid.png).
At Z = 0.8 the coarse argmax was 2400 steps at 44.86, and the refined peak
is 1427 steps at 62.99, about eighteen points higher; at Z = 1.6 the coarse
argmax was 600 steps at 75.77 and the refined peak is 849 steps at 95.67,
about twenty points higher.
Referenced by (1 direct)
Direct references:
Several interior peaks land on iterations the coarse waypoints never included, such as 2018 steps at Z = 1.1 and 1.2 and 849 steps at Z = 1.4 and 1.6.
One figure in the generated results table needs its context.
RESULTS.md lists a single peak over α of 37.7 ±7.3 at Z = 0.1 for
unseen_mismatch, which the generic sweep report takes at 10000 steps, the
last iteration shared across arms, over only the low-Z arms whose bracket
reached that far. It is the same value as Z = 0.1's own right-edge peak, and
it describes the uniform end at the end of training, not the peak-timing
result this run was built to find. The per-α peaks above, which reach
95.67, are that result.
Three things are settled. The best iteration for unseen_mismatch depends on Z, it decreases with Z through the interior band, and the six coarse waypoints did under-resolve it, both in where the peak falls and in how high it reaches. Two things this design leaves open. The uniform end, Z at 0.7 and below, never peaks within 10000 steps, so its true peak time is outside the window this run observed. And the concentrated end peaks so early that a few settings crest before the coarse grid's first point, which the dense bracket can see only because it reaches below 300 steps.
Comparison figures
Children
| Child | peak unseen_mismatch | Status |
|---|---|---|
| dense_bracket | 95.7 | done |