Text set in serif type with a dotted bar in the margin was drafted by an LLM and sometimes reviewed by the author. The rest is the author's own. How to read this site
burst_v1
20260711_222607_burst_v1 · complete · published 2026-07-16 · seed 42
Intent
This experiment takes up GitHub issue #3, α-burst mixing: rather than holding the pairing concentration fixed or annealing it smoothly, it drops short bursts of a different α into an otherwise steady schedule and asks what those bursts do.
Three questions follow. Does brief exposure to a moderate α of 1.0 change a model trained mostly on a fixed α of 1.4? Does brief exposure to deep-uniform data at α = 0.0 damage the rule, or does damage require sustained exposure? And can 1.0-bursts interleaved through a 3-to-0 anneal protect it from the deep-uniform collapse that curriculum_v1 found?
Background
This experiment follows curriculum_v1, where the end of an annealing schedule decided whether a trained model kept the subject-verb agreement rule or lost it.
curriculum_v1 established that sustained
deep-uniform exposure, the final anneal stages
at α ≤ 0.4, collapses unseen_mismatch below chance as the
model flips to agreeing with the nearest noun, while a
moderate α keeps the rule.
The comparison baselines come from curriculum_v1: fixed_14_single at 88.2 ± 9.1 final unseen_mismatch, and anneal_3_0 at 10.4 ± 7.0.
1.0-bursts dropped into a fixed α of 1.4 all stay in the safe range of α.
Hypothesis
This experiment carries three predictions. First, 1.0-bursts dropped into a fixed α of 1.4 are harmless at 25 percent duty at either period, and neutral to mild at 50 percent duty. Second, brief 0.0-bursts of 75 iterations at 25 percent duty do not collapse the rule, because the destruction requires sustained exposure rather than brief visits. Third, alternating 1.0-bursts through the 3-to-0 anneal rescue the collapse when they replace the deep-uniform tail, as in ann_d50_alt which ends at 1.0, but not when the 0.4 and 0.0 tail is kept, as in ann_d375_endzero.
Curricula
Here α is the Zipf exponent that sets how predictable a verb's
noun partners are in the training data: at 0 any noun pairs with any verb,
higher values keep each verb near its favorite partners, and α = 1.4 is the
paper's best fixed setting. Every arm trains for 1200 steps and
draws a fresh dataset at each segment of its schedule,
so what differs between arms is only the sequence of α values, plotted per
arm in images/schedules.png. The comparison baseline, the gray line
in images/curriculum_comparison.png, is
curriculum_v1's
fixed_14_single: one dataset at α = 1.4 for all 1200 steps,
finishing at 88.2 ± 9.1 on unseen_mismatch.
A burst schedule holds the arm at a base α and periodically switches it to a different value for a block of steps before switching back. The labels record two burst settings: d is the duty, the share of training spent in bursts, and p is the period, the number of steps in one base-plus-burst cycle. The six arms fall into two families.
Four arms keep a base of α = 1.4 and vary the bursts:
- f14_d25_p300 spends a quarter of training in bursts to α = 1.0, in four short cycles of 225 steps at 1.4 then 75 steps at 1.0.
- f14_d25_p600 spends the same quarter of training in bursts to α = 1.0, but consolidated into two longer cycles of 450 steps at 1.4 then 150 at 1.0.
- f14_d50_p300 alternates evenly, 150 steps at 1.4 and 150 at 1.0, four times over, so half of training sits in bursts.
- f14_zeroburst_d25 matches
f14_d25_p300step for step, except the four 75-step bursts drop all the way to α = 0.0 rather than 1.0, which tests whether brief deep-uniform exposure is harmful.
Two arms take curriculum_v1's 3-to-0 anneal and replace every second stage with an α = 1.0 burst, so the two share an identical schedule through the first seven segments and part only in the last:
- ann_d50_alt runs 3.0, 1.0, 2.14, 1.0, 1.29, 1.0, 0.43, 1.0, ending at 1.0 in place of the deep-uniform tail.
- ann_d375_endzero runs the same sequence but keeps the α = 0.0 final stage: 3.0, 1.0, 2.14, 1.0, 1.29, 1.0, 0.43, 0.0. The pair isolates what the last 150 steps at deep-uniform α do to a model that reached the same place beforehand.
Setup
| arm | data regime | datasets | iters/dataset | epochs/dataset |
|---|---|---|---|---|
| ann_d375_endzero | fresh-dataset-per-stage | 8 | 150 | 0.5 |
| ann_d50_alt | fresh-dataset-per-stage | 8 | 150 | 0.5 |
| f14_d25_p300 | fresh-dataset-per-stage | 8 | [75, 225] | [0.25, 0.75] |
| f14_d25_p600 | fresh-dataset-per-stage | 4 | [150, 450] | [0.5, 1.5] |
| f14_d50_p300 | fresh-dataset-per-stage | 8 | 150 | 0.5 |
| f14_zeroburst_d25 | fresh-dataset-per-stage | 8 | [75, 225] | [0.25, 0.75] |
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 74aeefb
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yamlChecks
PASS children_complete PASS replicate_coverage PASS schedule_total_matches_final_waypoint PASS instrument_consistent
Claims
| claim | source | expectation | Δ | seeds | verdict |
|---|---|---|---|---|---|
| bursts10_harmless_d25_p300 translated | manifest | f14_d25_p300 ~= baseline:fixed α 1.4 (curriculum_v1 fixed_14_single) (±5) on unseen_mismatch @ final | -13.2 | 3/10 | mixed mean says rejected, seeds 3/10 |
| bursts10_harmless_d25_p600 translated | manifest | f14_d25_p600 ~= baseline:fixed α 1.4 (curriculum_v1 fixed_14_single) (±5) on unseen_mismatch @ final | -0.9 | 3/10 | mixed mean says held, seeds 3/10 |
| bursts10_mild_at_d50 translated | manifest | f14_d50_p300 ~= baseline:fixed α 1.4 (curriculum_v1 fixed_14_single) (±10) on unseen_mismatch @ final | -6.4 | 5/10 | mixed mean says held, seeds 5/10 |
| brief_zerobursts_safe translated | manifest | f14_zeroburst_d25 ~= baseline:fixed α 1.4 (curriculum_v1 fixed_14_single) (±10) on unseen_mismatch @ final | -40.4 | 1/10 | rejected |
| alt_rescues_collapse translated | manifest | ann_d50_alt > chance on unseen_mismatch @ final | +17.1 | 9/10 | held |
| endzero_stays_collapsed translated | manifest | ann_d375_endzero <= chance on unseen_mismatch @ final | -30.3 | 10/10 | held |
| alt_beats_endzero translated | manifest | ann_d50_alt > ann_d375_endzero on unseen_mismatch @ final | +47.4 | 10/10 | held |
Results — unseen_mismatch (mean ± sd over 10 seeds)
Evaluation data: seen_* probed at reference α = 1.4 · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at fixed reference α = 1.4 (not at the schedule's own α, so arms stay comparable)
- @α conditions (e.g. seen_match@0) — the same seen probes regenerated at reference α = 0, 0.7, 2.1, 3
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | end of training (t = 1200) | per-seed | flags |
|---|---|---|---|
| ann_d375_endzero | 19.7 ±11.7 | ⚠ seed_split: range 2.7-45.5 (3 low / 1 high of 10) | |
| ann_d50_alt | 67.1 ±23.6 | ⚠ seed_split: range 10.4-100.0 (1 low / 2 high of 10) | |
| f14_d25_p300 | 75.0 ±24.7 | ⚠ seed_split: range 31.3-98.7 (1 low / 6 high of 10) | |
| f14_d25_p600 | 87.3 ±15.2 | ⚠ seed_split: range 55.5-99.6 (2 low / 7 high of 10) | |
| f14_d50_p300 | 81.8 ±19.7 | ⚠ seed_split: range 43.6-99.7 (2 low / 6 high of 10) | |
| f14_zeroburst_d25 | 47.8 ±33.3 | ⚠ seed_split: range 0.0-100.0 (3 low / 1 high of 10) |
Conclusions
A schedule's outcome tracks a running balance between two
forces: training at a moderate or peaked α builds the
agreement rule, and training near uniform α, roughly α ≤ 0.4,
erodes it, with the late stages weighted most and
the erosion at least partly repairable. The two annealing arms
show this most cleanly. Because ann_d50_alt and
ann_d375_endzero share an identical schedule through
iteration 1050, their per-seed trajectories are identical up to
that point, and they part only in the final 150 steps, where the ending α
alone decides the result. ann_d50_alt returns to α = 1.0 and
recovers to 67.1 ± 23.6 on unseen_mismatch, above the 50 chance line,
while ann_d375_endzero drops to α = 0.0 and collapses to 19.7 ±
11.7, where accuracy below 50 means the model now
agrees with the nearest noun rather than the subject. All three
predictions about the anneal held: the arm ending at 1.0 stays above chance
(9 of 10 seeds), the arm ending at 0.0 stays at or below it (10 of
10), and the first beats the second by 47.4 (10 of 10). The endzero collapse
to 19.7 reproduces curriculum_v1's plain
anneal_3_0 collapse to 10.4, which confirms the deep-uniform
tail as the cause.
Up to that final segment, the two anneal arms reach a higher mid-run generalization than either curriculum_v1 reference point.
Both bring 9 of their 10 replicates to the near-peak band of 95, on average by iteration 467, so the seed coverage of 9 of 10 means their strong mid-run mean is a real cohort effect and not a bimodal average. That cohort mean peaks at 96.8 ± 3.2 at iteration 600, above the fixed α = 1.4 baseline's own peak of 88.3 and above the early lead of about 90.5 that the plain peaked anneal reached in curriculum_v1.
Referenced by (2 direct)
Direct references:
Interleaving moderate 1.0-bursts through the peaked-first anneal builds the rule through collocational bootstrapping at least as strongly as any fixed setting here. The later fall, from 96.8 at iteration 600 down through 94.6 at 900 to 58.1 by 1050 as the shared α = 0.43 stage runs, happens while α is still well above zero, and the waypoints mark it as catastrophic interference: the rule was present and then overwritten by the low-α stage, not never learned.
Brief deep-uniform bursts damage the rule, so the second prediction is rejected. f14_zeroburst_d25 finishes at 47.8 ± 33.3, about 40 points below the baseline, with only 5 of its 10 replicates above chance and a bimodal spread of per-seed finals that runs the full range (sorted: 0.0, 4.3, 6.6, 43.0, 47.1, 56.6, 59.2, 72.5, 88.8, 100.0). Its cohort mean never establishes the rule at all, oscillating between 25 and 57 across training and peaking at only 56.6, and only 2 of 10 seeds ever reach the 95 band. Collapse therefore does not require a single sustained uniform tail; four brief visits to α = 0.0 are enough to keep the rule from consolidating.
Referenced by (2 direct, 1 transitive)
Direct references:
Transitive (depth 1):
What separates this arm from the otherwise identical f14_d25_p300 is the depth of the burst, since dropping briefly to α = 1.0 is largely tolerable while dropping briefly to α = 0.0 is not.
Where a schedule fails, it fails by the same global switch to the
nearest-noun shortcut rather than by a selective loss of generalization.
seen_mismatch tracks unseen_mismatch on every arm, 48.6 beside 47.8 for
f14_zeroburst_d25 and 18.2 beside 19.7 for
ann_d375_endzero, while seen_match and unseen_match stay
near 100 throughout, because on a match probe the nearest
noun already agrees with the subject and the shortcut is not penalized. The
collapse is a change of strategy the mismatch conditions expose, not a
narrower failure to transfer agreement to
unseen pairings.
The 1.0-bursts into fixed α = 1.4 come back mixed rather than cleanly
harmless. On the mean, the consolidated 25-percent-duty arm
f14_d25_p600 matches the baseline (87.3 ± 15.2, a gap of 0.9) and
the 50-percent-duty arm f14_d50_p300 stays within tolerance (81.8
± 19.7, a gap of 6.4), while the finely chopped 25-percent-duty arm
f14_d25_p300 falls short (75.0 ± 24.7, a gap of 13.2). None
matches the baseline seed for seed: f14_d25_p600 and
f14_d25_p300 meet the match on only 3 of 10 seeds each and
f14_d50_p300 on 5 of 10, because each carries a low-seed minority
that the single-dataset baseline, whose worst seed is 71.9, does not.
At matched 25-percent duty the two f14_d25 arms separate on burst
length: the two 150-step bursts of p600 leave the baseline mean intact,
while the four 75-step bursts of p300 lose about 13 points, so chopping the
same burst training budget finer is the harsher choice here, and the
50-percent-duty arm, whose bursts are also 150 steps, stays within
tolerance despite twice the burst time.
Taken with curriculum_v1, the picture is a balance between building and eroding rather than a rule about the direction of the α trajectory. Peaked or moderate α builds the verb-to-subject cue; deep-uniform α erodes it; the final stages weigh most because little training remains to repair them; and the erosion is partly reversible, since a moderate stage after a uniform one pulls accuracy back up. The peaked-to-moderate alternation builds the rule more strongly than any other shape tested here, and a single deep-uniform stage at the end is enough to undo it.
A schedule's outcome tracks a running balance of two forces: moderate or peaked α builds the agreement rule, deep-uniform α (roughly α ≤ 0.4) erodes it, the late segments weigh most because little training remains to repair them, and the erosion is partly reversible. The two annealing arms show the balance directly: sharing one schedule through step 1050, they end 47.4 points apart on unseen_mismatch (67.1 against 19.7) because one returns to α = 1.0 for its final 150 steps and the other drops to 0.
Referenced by (5 direct, 1 transitive)
Direct references:
Transitive (depth 1):
Three follow-ups suggest themselves. The first is the arm this experiment
points at but did not run: the anneal-with-1.0-bursts stopped at iteration
900, or ending on a longer 1.0 stage, where the mid-run mean of 94.6 at
iteration 900 suggests a final score above every baseline. The second is an
erosion-threshold sweep of the final-stage α between 0.43 and 0.86, to
locate where the erosion begins. The third is a recovery dose-response,
continuing training from the ann_d375_endzero
checkpoints at a
moderate α with imsyn run continue, to measure how much of the
collapse a later moderate stage can repair.
Comparison figures
Children
| Child | peak unseen_mismatch | Status |
|---|---|---|
| fixed α 1.4 (curriculum_v1 fixed_14_single) | 88.3 | baseline |
| f14_d25_p300 | 78.7 | done |
| f14_d25_p600 | 87.3 | done |
| f14_d50_p300 | 81.8 | done |
| f14_zeroburst_d25 | 56.6 | done |
| ann_d50_alt | 96.8 | done |
| ann_d375_endzero | 96.8 | done |