Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

bimodality_pilot

20260427_222400_bimodality_pilot · complete · published 2026-08-28 · seed 1000

part of investigation bimodality

Intent

Run 2026-04-27, 5 models at 1200 iterations each. Backfilled into the standardized results object on 2026-08-27 from the original run directory; the accuracies were re-read from the per-condition evaluation files and checked against the runner's own log, and no cell disagreed.

Intent: How far apart five seeds land at one α \@{bimodpilot-intent} This is a pilot, run to size the spread between replicates before committing to a grid of several hundred models. It trains five seeds at α = 1.4, holding everything else at the defaults, and reports how far apart their unseen_mismatch scores land. The point is to learn whether replicate spread at a single grid point is small enough that a cohort mean means anything, which decides how many replicates the full grid needs.

Hypothesis

Hypothesis: Five seeds at α = 1.4 land close together \@{bimodpilot-hypothesis} At α = 1.4 the pairings are concentrated enough that every replicate should find the subject-verb number rule within the budget. The prediction is that all five land high and within a few points of each other, which would make a cohort mean a fair summary at this α and put any wider spread elsewhere on the axis.

Setup

model: GPT (nanoGPT-derived), weight-tied embeddings — 2 layers × 4 heads × 256 dim (~1,635,584 params) · block 50 · dropout 0.1 · vocab 167 tokens
training: AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 5 seeds (base 1000)
training data: PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code: commit d99f2d5

data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout d99f2d5
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS instrument_consistent

Results — unseen_mismatch (mean ± sd over 5 seeds)

Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition

Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)

arm	end of training	per-seed	flags
alpha_grid	91.6 ±8.1

Conclusions

Conclusion: Five seeds were enough to justify fifty \@{bimodpilot-conclusion} The five replicates landed together, which answered the sizing question and cleared the way for the full grid. Result: The cohort at α = 1.4 spans 21.4 points and sits well above chance \@{bimodpilot-tight-cohort} The five seeds scored between 77.7 and 99.1 on unseen_mismatch, a mean of 91.6 with a standard deviation of 8.1, and none fell near the recency floor. The later fifty-replicate grid reached the same reading at this α, a mean of 89.5 with a standard deviation of 7.4 and no replicate below 50, so nothing in the pilot was an artifact of its size. The pilot's own limitation is worth stating, because it shaped the design that followed. Five replicates can show that a cohort is tight, but they cannot distinguish a genuinely tight cohort from the upper part of a divided one, since a group holding four fifths of the seeds would place all five draws in it more often than not. That is why the grid that followed used fifty rather than five, and at grid points where the cohort does divide, five replicates would have been read as a wide but single group.

Comparison figures

Children

Child	peak unseen_mismatch	Status
alpha_grid	91.6	done

experiment.yaml