Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

paper-reproduction

investigation

Intent

Intent: Does the paper's central result reproduce, and how far does it carry \@{paper-reproduction-intent} This investigation asks whether the paper's central result reproduces and how far it carries. The result is Figure 2: on unseen_mismatch, the hardest minimal pair condition, generalization of subject-verb number agreement peaks at an intermediate Zipfian concentration and the oneshot limit generalizes worse. The member experiments reproduce the figure at the paper's own configuration, then probe how the picture moves with the training budget and where in training each α is at its best.

Conclusions

Conclusion: The intermediate peak is real but it is a slice of a moving target \@{paper-reproduction-conclusion} The published result reproduces exactly at the paper's configuration (the intermediate peak), with the oneshot limit memorizing instead of generalizing (oneshot memorizes) and the match conditions separating memory from rule as the paper reads them. Conclusion: A robust peak whose location is set by the budget \@{paper-reproduction-robust-peak} The peak itself is robust: the vocabulary-scale investigation finds it at every vocabulary size and holdout fraction tested (not a vocabulary artifact), though model capacity caps its height at the largest vocabularies (the capacity ceiling). Its location is not a property of the grammar but of the training budget: the best-generalizing α slides downward as training lengthens (the moving peak), the paper's 1200-step snapshot happening to land the ridge at α ≈ 1.2 (budget sets the peak). Per α the story is timing all the way down: more concentrated settings reach their best generalization earlier (peak earlier with Z), the sweep splits into three regimes, and the uniform end never peaks at all (no peak at uniform), while past the peak longer training erodes what was built (post-peak erosion) even as the training loss keeps falling (the dissociation). Together these hand the baseline to the other investigations: a fixed-α run is one point on a budget × concentration surface, which is what makes a schedule worth trying at all (alpha-curriculum) and what the size sweeps hold fixed (model-scale). What would change this reading. The reproduction and its extensions all use the two-preposition PCFG at the paper's sentence lengths; the moving peak has been mapped to 10000 steps at the default vocabulary only, and the deepening seed splits at large vocabularies mean the mean curves get less trustworthy exactly where the capacity questions get interesting. Referenced by (1 direct)Direct references:Summer Summary

Experiments

20260705_105204_paper 2026-07-16

This is the canonical paper reproduction, run fresh on 2026-07-05 through the packaged bare imsyn entry point. An earlier candidate directory, paper_20260703_141230, turned out to be the fast smoke run from rebase validation (Z in {0, 1.4}, one seed) and was archived without publishing.

$Intent: Does Figure 2 reproduce \@{paper-intent} This experiment reproduces the paper's central result, the two panels of Figure 2. It measures how well subject-verb agreement generalizes to noun-verb pairings the model was never trained on as the Zipfian concentration of the training pairings is varied, and it sets that sweep against the oneshot limit, where each verb takes exactly one subject noun. The Zipfian arm sweeps Z from 0.0 to 3.0 in steps of 0.1 with ten seeds at each value; the oneshot arm adds ten more seeds at the deterministic 1:1 limit. Both train the default 256-wide, two-layer, four-head model for 1200 iterations from seed 42.$
20260509_153511_peak_umm_iter 2026-07-16

These notes were backfilled on 2026-07-05 from a run executed on 9 and 10 May 2026, before the experiment-manifest format existed, so both the manifest and this write-up are retroactive. The run is Stage 2 of the two-stage imsyn run peak-iter pipeline. Its Stage 3 analysis, the merged coarse-and-dense tables merged_metric.csv and refined_peaks.csv and the two figures copied into images/, lives in runs/_analysis/20260509_153511_peak_umm_iter/. The run directory's measurements.csv holds only each Z's nine-point dense bracket, so the low-Z peaks that fall at the source sweep's coarse waypoints appear only in those merged tables, and the per-α peaks reported below are read from them. The --source-sweep path in experiment.yaml points at the source run's location after the same backfill.

$Intent: When each Zipfian setting reaches its best generalization \@{peakumm-intent} This run finds, more precisely for every Z, the iteration at which unseen_mismatch peaks. It starts from the coarse argmax per Z in the source sweep, then samples a dense nine-point geometric bracket of iterations around it with ten seeds at each point, so the peak can be read from a finer grid.$
20260506_205133_paper_long_iter 2026-07-16

Backfilled 2026-07-05 from a pre-manifest run executed 2026-05-06, so the manifest and this report are retroactive. The original invocation used the pre-rebase CLI grammar, and the args recorded in experiment.yaml are its current-grammar equivalent. The sweep is 31 Z values from 0.0 to 3.0 in steps of 0.1 at ten seeds each, 310 trained models in all, from base seed 42.

$Intent: How agreement generalization evolves over training across the Z sweep \@{paperlong-intent} How does subject-verb agreement generalization (unseen_mismatch) evolve over the course of training, across the full Z sweep?$

Referenced by

paper-reproduction-conclusion: Summer Summary