Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
bimodality_grid_v2
20260504_211432_bimodality_grid_v2 · complete · published 2026-08-28 · seed 1000
part of investigation bimodality
Intent
Run 2026-05-04 to 2026-05-05 in three batches on one commit, 400 models at 1200 iterations each. Backfilled into the standardized results object on 2026-08-27 from the original run directories; the accuracies were re-read from the per-condition evaluation files and checked against the runner's own log, and no cell disagreed.
This experiment asks whether models that differ only in their seed land in one place or two, and how that depends on α. Each grid point trains fifty replicates at a single α and scores them on the same minimal-pair suite, so the only thing separating one replicate from another is initialization and data order. Eight α values run from 0.8 to 1.2, spaced every 0.05 through the middle of that interval and every 0.025 across 1.10 to 1.20, which is where the cohort spread in earlier runs was widest. The same fifty seeds run at every α, so a cohort at one grid point is comparable to the cohort at its neighbour rather than being a fresh draw.
Background
A first grid and a five-seed pilot had already suggested that cohorts near α = 1 do not concentrate around one accuracy. Both ran before the training pipeline was made deterministic across runners. Evaluation consumed the global random state that model construction and batching also draw on, so two runners given the same α and the same seed did not produce the same weights, and a grid point could not be re-measured exactly or continued from its own checkpoints. That was fixed immediately before this experiment started. This run asks the question again on the corrected pipeline and adds the denser spacing the first pass lacked.
Hypothesis
At a budget of 1200 steps there should be a range of α in which some replicates have found the subject-verb number rule and the rest are still agreeing with the nearest noun, with few cohort members between the two. The prediction is that mean unseen_mismatch rises with α across this grid while the spread around it stays wide, that the wide spread reflects two groups rather than one diffuse one, and that the lower group's accuracy climbs with α until the two groups meet.
Setup
data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout ee28502
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yamlChecks
PASS children_complete PASS replicate_coverage PASS instrument_consistent
Results — unseen_mismatch (mean ± sd over 50 seeds)
Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | peak over α (end of training) | per-seed | flags |
|---|---|---|---|
| alpha_grid | 85.0 ±19.8 @ α=1.175 | ⚠ seed_split: range 4.9-100.0 (1 low / 29 high of 50) |
Conclusions
The cohort does divide, and the division is not confined to a narrow band of α. At every one of the eight grid points the lowest replicate finishes below 20 and the highest above 99, and the spread never contracts the way it would if the grid had passed through an edge. What changes with α is not whether the two groups exist but how many seeds are in each.
Mean unseen_mismatch rises from 35.3 at α = 0.8 to a peak of 85.0 at α = 1.175, easing to 80.9 at α = 1.2, but the standard deviation stays between 17.7 and 37.1 across the whole grid. Counting replicates below 50, the point at which a model is agreeing with the nearest noun more often than with the subject, gives 33 of 50 at α = 0.8, 17 at α = 1.0, 7 at α = 1.1, and 3 at α = 1.15. The rise in the mean is that count draining from one group into the other, not either group moving.
Referenced by (1 direct, 4 transitive)
Direct references:
The denser spacing across 1.10 to 1.20 was placed to catch a boundary and did not find one. The widest single gap between neighbouring sorted replicates is 34.7 points at α = 1.125 and 29.0 at α = 1.15, so the middle of the range stays empty exactly where the spacing was tightest. The hypothesis expected the lower group to climb until it met the upper one; instead the lower group holds its position and loses members.
Every number here is read at 1200 steps, and that qualification carries the weight. Follow-up runs that continued these checkpoints to ten and twenty thousand steps found the two groups converging on a single distribution: the upper group drifts down, the lower group drifts up, and by twenty thousand steps the separation is no longer distinguishable from zero. The split this experiment measures is therefore a statement about where replicates have got to at a fixed budget, and reading it as two attractors in the loss landscape would overstate it. The open question this leaves is not where the boundary sits on the α axis, since there is no boundary on this grid, but what decides which group a given seed joins early and whether that is visible before the two groups separate.
Referenced by (1 direct)
Direct references:
Referenced by (1 direct, 4 transitive)
Direct references:
Comparison figures
Children
| Child | peak unseen_mismatch | Status |
|---|---|---|
| alpha_grid | 85.0 | done |