Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5, claude-opus-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
bimodality
investigation
Intent
This investigation tracks a phenomenon that cuts across the others rather than owning experiments of its own: seed bimodality, where a cohort of replicates differing only in seed splits into two basins, some models learning the subject-verb number rule and the rest agreeing with the nearest noun or memorizing, with almost nothing in between. The question is when the split appears, whether it is a transient of undertraining or a stable property of a configuration, and what it does to any reading based on cohort means.
Conclusions
The split shows up at every edge of the map the other investigations have pushed against. Width past the paper's induces it (the 512 split) and four times the training budget does not close it (width-intrinsic); depth's recovery under longer training happens seed by seed rather than as a cohort (the effect splits); larger vocabularies deepen it (deepening splits); schedules that end at uniform produce it on the way down (the collapse), and short horizons trade seed coverage for it, single seeds reaching the band while the rest never start (the coverage cost). Where a split is present the cohort mean sits between two groups that no individual model occupies, which is why the curriculum work reports per-seed time-to-band and seed coverage rather than means alone (the axes), and why a wide standard deviation next to a middling mean should be read as a split first (mirror_v1's spread claims).
The investigation now has members that measure the split directly rather than meeting it in passing. Two grids of fifty replicates per α put the phenomenon on the α axis, and they agree on its shape: the cohort divides across the middle of the range, and what varies with α is how many seeds sit in each group rather than where the groups are (The low group shrinks with α while both groups stay put). Denser spacing did not find an edge (The cohort splits everywhere on this grid, and the grid never finds its edge), and the spread does close at the top of the axis, where the only coverage the corpus has runs from α = 1.4 to 2.5 (The spread is wide below α = 1.2 and narrow above 1.4).
An earlier reading of this investigation held that the split behaves like basin selection at initialization, stable under more training wherever it had been tested. That is too broad, and the grid members are what narrow it. The splits induced by capacity, vocabulary, horizon and schedule endpoint do persist under a longer budget. The split induced by α at a 1200-step budget does not: continuing those checkpoints to twenty thousand steps brings the two groups together until the separation is no longer distinguishable from zero (What this says is about a 1200-step budget, not about the landscape). Both are worth calling seed bimodality, since in both a cohort mean describes nobody, but only the first is a statement about the loss landscape. The second is a statement about where replicates have got to when the run stops, which makes it a form of undertraining that a cohort mean hides.
What is still missing is the mechanism. No member has followed individual seeds' checkpoints through the early part of training to see when the groups separate, whether the lower group is one solution or several, and whether a seed's destination is readable before it arrives. The replicates in both grids are gone, since only evaluation files were kept, so that study needs fresh training rather than a re-reading of what is on disk.
Referenced by (1 direct)
Direct references:
Referenced by (3 direct, 2 transitive)
Direct references:
Experiments
-
20260504_211432_bimodality_grid_v2 2026-08-28
Run 2026-05-04 to 2026-05-05 in three batches on one commit, 400 models at 1200 iterations each. Backfilled into the standardized results object on 2026-08-27 from the original run directories; the accuracies were re-read from the per-condition evaluation files and checked against the runner's own log, and no cell disagreed.
This experiment asks whether models that differ only in their seed land in one place or two, and how that depends on α. Each grid point trains fifty replicates at a single α and scores them on the same minimal-pair suite, so the only thing separating one replicate from another is initialization and data order. Eight α values run from 0.8 to 1.2, spaced every 0.05 through the middle of that interval and every 0.025 across 1.10 to 1.20, which is where the cohort spread in earlier runs was widest. The same fifty seeds run at every α, so a cohort at one grid point is comparable to the cohort at its neighbour rather than being a fresh draw.
-
20260427_223648_bimodality_grid_v1 2026-08-28
Run 2026-04-27, 400 models at 1200 iterations each. Backfilled into the standardized results object on 2026-08-27 from the original run directory; the accuracies were re-read from the per-condition evaluation files and checked against the runner's own log, and no cell disagreed.
This experiment asks whether models trained at one α and differing only in seed concentrate around a single accuracy or fall into separate groups, and it asks the question across the whole useful range of α rather than near any particular value. Eight grid points run from 0.5 to 2.5, fifty replicates each, with the same fifty seeds at every point. The spacing is deliberately coarse: the purpose is to find out whether a split exists anywhere and roughly where, not to locate its edge.
-
20260427_222400_bimodality_pilot 2026-08-28
Run 2026-04-27, 5 models at 1200 iterations each. Backfilled into the standardized results object on 2026-08-27 from the original run directory; the accuracies were re-read from the per-condition evaluation files and checked against the runner's own log, and no cell disagreed.
This is a pilot, run to size the spread between replicates before committing to a grid of several hundred models. It trains five seeds at α = 1.4, holding everything else at the defaults, and reports how far apart their unseen_mismatch scores land. The point is to learn whether replicate spread at a single grid point is small enough that a cohort mean means anything, which decides how many replicates the full grid needs.