Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-4-8) and sometimes reviewed by the author. The rest is the author's own. How to read this site
efficiency_v1
20260712_084956_efficiency_v1 · complete · published 2026-07-16 · seed 42
Intent
This experiment asks whether the anneal-plus-bursts curriculum learns agreement more efficiently than a fixed α of 1.4, measured over 2400 iterations, twice the horizon of the earlier curriculum experiments. Efficiency here has two parts: whether the curriculum reaches its peak sooner, and whether it then stays higher.
Background
The winning prefix from burst_v1, a 3.0-and-1.0 alternating anneal through iteration 900 that held 93 to 97 percent unseen_mismatch across iterations 450 to 900, is continued past 900 under three maintenance diets rather than the deep-uniform tail that undid it in burst_v1.
The build-vs-erosion model is carried over from curriculum_v1 and burst_v1.
Hypothesis
Two models make opposite predictions about stability, and both expect the curriculum to peak sooner. Jason's model expects the anneal-plus-bursts arms to peak sooner than fixed α = 1.4, the efficiency win, but to be unstable: they should degrade more over continued training than the fixed-α model and finish below it at 2400. The build-vs-erosion model predicts the opposite for stability, that under maintenance α at or above 1.0 there is no erosion pressure, so the burst arms should hold their early peak through 2400 and finish at or above the fixed-α arm.
The three maintenance arms, holding at 1.0, holding at 1.4, and continuing to cycle 1.4 and 1.0, separate whether any degradation comes from the maintenance diet or from the peaked start itself.
Curricula
Every arm trains for 2400 steps in sixteen 150-step
segments, twice the horizon of the earlier
curriculum experiments.
Each segment draws its own fresh 12,000-sentence dataset at that segment's
α, sampled without replacement, which is the staged
data regime the curriculum work uses throughout. α is the Zipf exponent
that sets how predictable the noun-verb pairings are: 0 is uniform, higher
values make each verb keep its favorite nouns, and 1.4 is the paper's best
fixed setting. What differs between arms is the sequence of αs, drawn in
images/schedules.png.
The three curriculum arms share one prefix, the burst schedule that won burst_v1: α steps down through teeth of 3.0, 2.14, and 1.29 over the first 900 iterations, each tooth followed by a drop to 1.0. Because they share this prefix and the same seeds, the three arms are the same run through iteration 900 and differ only in what trains the model afterward.
- fixed_14 trains at α = 1.4 for all 2400 iterations. It is the stability benchmark and the gray reference line in the comparison figure.
- ann_burst_hold10 follows the prefix, then holds α = 1.0 for the remaining 1500 iterations.
- ann_burst_hold14 follows the prefix, then holds α = 1.4.
- ann_burst_cycle follows the prefix, then keeps alternating 1.4 and 1.0.
Holding at 1.0, holding at 1.4, and continuing to cycle let the three arms separate degradation caused by the maintenance diet from degradation intrinsic to the peaked start, since all three enter the maintenance phase from an identical model.
Setup
data regime (all arms): fresh-dataset-per-stage — 16 dataset(s) × 150 iters each = 0.5 epochs per dataset
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout eacf604
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yamlChecks
PASS children_complete PASS replicate_coverage PASS schedule_total_matches_final_waypoint PASS instrument_consistent
Claims
| claim | source | expectation | Δ | seeds | verdict |
|---|---|---|---|---|---|
| burst_peaks_sooner translated | both | ann_burst_hold10 < fixed_14 on unseen_mismatch (peak time) @ final | -750.0 | 9/10 | held
referenced by
The curriculum buys speed to the peak
|
| jason_burst_erodes translated | jason | ann_burst_hold10 < fixed_14 on unseen_mismatch @ final | -24.8 | 9/10 | held
referenced by
Maintenance at α ≥ 1.0 does not stop the erosion
|
| bve_burst_holds translated | build-vs-erosion | ann_burst_hold10 >= fixed_14 on unseen_mismatch @ final | -24.8 | 1/10 | rejected
referenced by
Maintenance at α ≥ 1.0 does not stop the erosion
|
Results — unseen_mismatch (mean ± sd over 10 seeds)
Evaluation data: seen_* probed at reference α = 1.4 · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at fixed reference α = 1.4 (not at the schedule's own α, so arms stay comparable)
- @α conditions (e.g. seen_match@0) — the same seen probes regenerated at reference α = 0, 0.7, 2.1, 3
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | end of training (t = 2400) | per-seed | flags |
|---|---|---|---|
| ann_burst_cycle | 64.8 ±14.1 | ||
| ann_burst_hold10 | 61.2 ±14.2 | ⚠ seed_split: range 38.3-78.6 (2 low / 5 high of 10) | |
| ann_burst_hold14 | 78.4 ±9.8 | ||
| fixed_14 | 86.0 ±6.2 |
Conclusions
Jason's model is supported on both counts, and the build-vs-erosion prediction is rejected. The burst arms reach a higher unseen_mismatch sooner than fixed α = 1.4, then lose it over continued training and finish below the fixed baseline, while fixed_14 climbs slowly and largely keeps what it reaches.
The curriculum reaches a higher peak sooner. All three burst arms reach the same peak, a seed-mean unseen_mismatch of 96.8 at 600 steps: they share the first 900 iterations and the same seeds, so the peak, which falls inside that shared prefix, is identical across them. fixed_14 peaks lower and later, at 90.5 by 1800, and eases only to 86.0 by 2400.
The burst arms also reach their peak sooner. Per replicate, the iteration at which unseen_mismatch peaks averages 555 for them and 1305 for fixed_14, and the pre-registered claim burst_peaks_sooner records this time-to-band advantage as held for 9 of 10 seeds. The seed-mean crosses 90 at iteration 450 for the burst arms against 1800 for fixed_14, and at a matched training budget of 600 iterations the two sit at 96.8 and 37.1. The early peak is reached across the cohort rather than propped up by a few seeds: all ten burst-arm seeds reach at least 90 at their peak and nine reach at least 95, so seed coverage of the 95 band is high exactly where the timing is measured, which is what makes the fast timing a real win. This is distinct from the report's replicate-coverage check, which only confirms every arm ran the same ten seeds.
Referenced by (7 direct, 1 transitive)
Direct references:
Transitive (depth 1):
The stability half of Jason's model also holds. Past the peak every burst arm loses ground, and all three finish below fixed_14's 86.0 at 2400: ann_burst_hold10 falls to 61.2, ann_burst_cycle to 64.8, and ann_burst_hold14 to 78.4. The pre-registered claim jason_burst_erodes, that ann_burst_hold10 finishes below fixed_14 on unseen_mismatch, is held for 9 of 10 seeds, a gap of 24.8 points in the means. Its counterpart bve_burst_holds, the build-vs-erosion prediction that maintenance at α at or above 1.0 would keep ann_burst_hold10 at or above fixed_14, is rejected, supported by only 1 of 10 seeds. Maintenance at α = 1.0 does not remove the erosion pressure; the score declines through the whole maintenance phase, so the earlier threshold picture, in which erosion set in only near uniform α, does not survive this longer horizon.
Referenced by (1 direct)
Direct references:
Both the maintenance diet and the peaked start contribute, and the three arms separate them because they enter maintenance from the same model. The endpoint orders by maintenance α: ann_burst_hold14 at α = 1.4 erodes least, to 78.4, while ann_burst_hold10 at α = 1.0 (61.2) and ann_burst_cycle, which spends half its maintenance at 1.0 (64.8), erode more, so the α = 1.0 exposure is the more erosive diet. But the diet is not the only factor. ann_burst_hold14 maintains at exactly α = 1.4, the value fixed_14 trains at throughout, and still finishes 7.6 points below it, 78.4 against 86.0. The only difference between those two arms is the first 900 iterations, so the peaked start itself leaves a lasting deficit in generalization that identical later training does not repair.
What erodes is the agreement rule, not the language model. seen_match stays at essentially 100 everywhere and unseen_match in the mid-90s and above at 2400, so the models still agree correctly when the attractor nouns share the subject's number. The loss is confined to the mismatch conditions, where the model drifts from agreeing with the subject back toward agreeing with the nearest noun, the recency shortcut. The maintenance α decides how far this goes on trained pairings: seen_mismatch at 2400 stays at 96.6 under ann_burst_hold14's α = 1.4 maintenance, matching fixed_14's 96.5, but falls to 79.5 under ann_burst_hold10's α = 1.0. unseen_mismatch, the generalization target, erodes under every burst arm, including the one held at 1.4.
Read across the waypoints, this is interference: the rule is measurably present at the peak, 96.8 at 600, and then lost as training continues, rather than never learned. Unlike curriculum_v1, where the same rise-and-fall came from a deep-uniform tail, here it appears under maintenance held at α at or above 1.0, so a uniform tail is not required to erode the rule. Whether continued training erodes it because of the data the maintenance segments show or because the optimizer keeps updating the weights is not settled here, and it is the first thing a follow-up should separate.
Referenced by (1 direct, 1 transitive)
Direct references:
Transitive (depth 1):
The erosion is uneven across seeds. ann_burst_hold10's endpoint splits, with per-seed unseen_mismatch running from 38.3 to 78.6 and the report flagging two low and five high of ten, so continued α = 1.0 training pushes some seeds much further down than others rather than lowering all of them together. This is the bimodality the curriculum line keeps surfacing, here in the eroded endpoint rather than at the peak, where the cohort was still together.
The reading that fits is that the curriculum buys speed to the peak, not the endpoint. The height the burst arms reach early is not preserved by more training at any diet tried here, so capturing it would mean stopping at the peak or changing how the model is updated after it, not feeding more of any α.
Three follow-ups sharpen this: whether lowering the learning rate after the peak stops the erosion, which would place the cause in the optimizer rather than the data; whether an explicit early-stop-and-consolidate step banks the iteration-600 solution; and the erosion-threshold sweep left open by burst_v1, mapping how maintenance α trades off against how fast the rule is lost.
Comparison figures
Children
| Child | peak unseen_mismatch | Status |
|---|---|---|
| fixed_14 | 90.5 | done |
| ann_burst_hold10 | 96.8 | done |
| ann_burst_hold14 | 96.8 | done |
| ann_burst_cycle | 96.8 | done |