Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-4-8) and sometimes reviewed by the author. The rest is the author's own. How to read this site
slopesweep_v1
20260713_095946_slopesweep_v1 · complete · published 2026-07-16 · seed 42
Intent
The question is whether a steeper, model-tuned decline in α does better than the empirical trajectory of child-directed speech. The sweep crosses the starting concentration, from 1 to 5, with the floor the schedule settles to: from each start, α declines linearly to every value between that start and 0. All arms use the continuous, per-batch data setting. The goal is to map where the erosion floor sits as a function of the starting height.
Every arm of this sweep walks the same shape: a staircase of 24 segments that lowers α linearly from a chosen starting concentration to a chosen floor, drawing each segment's training pool fresh at its own α. Arms differ only in their two endpoints.
Referenced by (1 direct)
Direct references:
Background
This experiment is a follow-up to the child-directed speech arm of curriculum_v1, which lowered the pairing concentration α along the trajectory measured in child-directed speech.
The declining slope is the right idea, but the magnitude and range measured in child-directed speech are not expected to carry over to the model. Three results from earlier runs shape the prediction: a bootstrapping cliff between a start of α = 2 and α = 3; deep-uniform tails at α ≤ 0.4 that destroy the learned rule; and erosion that acts as a gradient, nonzero even at a floor of 1.0, with the final α weighted most heavily.
Hypothesis
The prediction is that a steep decline from a high start to a moderate floor at or above about 2, for example the 4-to-2 schedule, both peaks early and stays stable; that any schedule ending at 0 collapses regardless of where it starts; and that higher starts peak earlier.
Setup
data regime (all arms): with-replacement (legacy stream) — fresh batch every step, no dataset reuse
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 49063016e541
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yaml
Checks
PASS children_complete PASS replicate_coverage PASS schedule_total_matches_final_waypoint PASS instrument_consistent
Claims
| claim | source | expectation | Δ | seeds | verdict |
|---|---|---|---|---|---|
| s4e2_stays_high | manifest | s4_e2 > 70 on unseen_mismatch @ final | +10.1 | 8/10 | held
referenced by
The floor sets the height and the start sets the timing
|
| s4e2_peaks_early | manifest | s4_e2 <= 300 on unseen_mismatch (peak time) @ final | +730.0 | 0/10 | rejected
referenced by
Higher starts peak later, not earlier
|
| floor2_beats_floor0 | manifest | s4_e2 > s4_e0 on unseen_mismatch @ final | +54.0 | 10/10 | held
referenced by
The floor sets the height
|
| s5e0_collapses | manifest | s5_e0 < 50 on unseen_mismatch @ final | -17.9 | 7/10 | mixed mean says held, seeds 7/10
referenced by
The floor sets the height and the start sets the timing
|
| s4e0_collapses | manifest | s4_e0 < 50 on unseen_mismatch @ final | -23.9 | 9/10 | held
referenced by
The floor sets the height and the start sets the timing
|
| s3e0_collapses | manifest | s3_e0 < 50 on unseen_mismatch @ final | -31.0 | 10/10 | held
referenced by
The floor sets the height and the start sets the timing
|
| higher_start_peaks_earlier | manifest | s5_e2 < s3_e2 on unseen_mismatch (peak time) @ final | +230.0 | 3/10 | mixed mean says rejected, seeds 3/10
referenced by
Higher starts peak later, not earlier
|
Results — unseen_mismatch (mean ± sd over 10 seeds)
Evaluation data: seen_* probed at reference α = 1.4 · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at fixed reference α = 1.4 (not at the schedule's own α, so arms stay comparable)
- @α conditions (e.g. seen_match@0) — the same seen probes regenerated at reference α = 0, 0.7, 2.1, 3
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | end of training (t = 1200) | per-seed | flags |
|---|---|---|---|
| s1_e0 | 25.5 ±20.5 | ⚠ seed_split: range 0.0-50.3 (4 low / 3 high of 10) | |
| s1_e1 | 86.9 ±25.0 | ⚠ seed_split: range 19.2-100.0 (1 low / 8 high of 10) | |
| s2_e0 | 30.7 ±24.1 | ⚠ seed_split: range 7.4-91.7 (4 low / 1 high of 10) | |
| s2_e1 | 92.4 ±9.8 | ||
| s2_e2 | 86.9 ±4.0 | ||
| s3_e0 | 19.0 ±11.3 | ||
| s3_e1 | 93.2 ±8.1 | ||
| s3_e2 | 86.6 ±4.0 | ||
| s3_e3 | 79.4 ±3.8 | ||
| s4_e0 | 26.1 ±19.8 | ⚠ seed_split: range 3.5-60.3 (5 low / 1 high of 10) | |
| s4_e1 | 94.0 ±5.1 | ||
| s4_e2 | 80.1 ±6.7 | ||
| s4_e3 | 75.9 ±4.3 | ||
| s4_e4 | 71.4 ±5.4 | ||
| s5_e0 | 32.1 ±27.4 | ⚠ seed_split: range 0.4-85.3 (3 low / 1 high of 10) | |
| s5_e1 | 93.0 ±8.9 | ||
| s5_e2 | 82.5 ±4.0 | ||
| s5_e3 | 75.4 ±3.7 | ||
| s5_e4 | 69.1 ±3.7 | ||
| s5_e5 | 66.0 ±3.9 |
Conclusions
The sweep set out to ask whether the declining-α trajectory of child-directed speech helps the model once its magnitude is exaggerated.
Its more useful result was to separate two things the child-directed-speech analogy had treated as one: how high the model's agreement accuracy climbs, and how soon it gets there. The two turn out to be governed by different ends of the schedule, which each arm walks as a staircase of 24 segments from a chosen start down to a chosen floor. Where α ends sets how well the rule is learned; where α begins sets how quickly it appears, among the schedules that end high enough for the timing to matter.
Each arm's α
schedule is plotted in images/schedules.png.
Where the schedule ends decides the height. Two schedules that begin at different concentrations but settle to the same floor reach almost the same final unseen_mismatch, so the endpoint, not the descent that reaches it, carries the outcome. Final accuracy over the ten replicates falls steadily as the floor rises: about 92 to 94 at a floor of α = 1, the 80s at floor 2, the middle-to-high 70s at floor 3, around 70 at floor 4, and 66 at the fixed α = 5. A shallow floor near α = 1 is the best of these. Ending higher leaves the model short of it, because a distribution kept concentrated never forces generalization across the noun class, and at the most concentrated fixed settings even the easier held-out test degrades, with unseen_match at 76.0 for a fixed α = 4 and 69.2 for a fixed α = 5. Ending at uniform is worse than ending short: it removes the rule the model had already built. The pre-registered contrast floor2_beats_floor0 holds on all ten replicates, with the 4-to-2 schedule finishing at 80.1 against 26.1 for 4-to-0.
The collapse at a floor of 0 is worth stating precisely, because the model
does not forget the language. On matching probes its
accuracy stays at the
ceiling, seen_match at 100 and unseen_match near 99, and what
moves is the agreement rule itself. As the tail approaches uniform the model
shifts from preferring the grammatical subject back toward
agreeing with the nearest noun: on the 3-to-0 schedule,
seen_mismatch falls to 15.2 while seen_match holds at 100. The
dense waypoints, plotted per arm in
images/curriculum_comparison.png, place this loss late in training.
The 3-to-0 schedule climbs to a mean of 92.0 by step 750, then
erodes to
47.4 by step 1050 and 19.0 by step 1200 as α descends through the low
concentrations toward 0. These schedules reach their best score on the way
down, as the descent passes through the low-α region a good floor would have
held, and lose it only by continuing to uniform. This is the erosion seen in
the earlier curriculum experiments, now
located in time and identifiable as
catastrophic interference: the rule was measurably present and then
overwritten, rather than never formed.
The erosion does not fall the same way on every seed, which is why the three collapse claims separate. On the 3-to-0 schedule every replicate ends below the chance line of 50, so s3e0_collapses holds on all ten; on 4-to-0 nine of ten do, so s4e0_collapses holds; but on 5-to-0 three replicates hold above 50 while seven fall below, so s5e0_collapses is mixed, its mean of 32.1 sitting in the empty gap of a bimodal split whose per-seed finals run from 0.4 to 85.3. What decides whether a given seed keeps the rule or loses it under a uniform tail is not visible in the schedule alone.
Any schedule that ends at uniform loses the rule it built. Every descending arm with a floor of α = 0 finishes between 19.0 and 32.1 mean unseen_mismatch regardless of where it started, after peaking mid-run (the 3-to-0 arm reaches 92.0 at step 750 before eroding to 19.0 by 1200), which the waypoints identify as catastrophic interference rather than a failure to learn. The erosion splits seeds instead of lowering them uniformly: the 5-to-0 arm ends at 32.1 ±27.4 with per-seed finals from 0.4 to 85.3, a seed bimodality whose cause is not visible in the schedule alone.
Referenced by (4 direct, 4 transitive)
Direct references:
Where the schedule begins decides how soon the rule appears, the speed axis the study tracks separately from height, and here the pre-registration was wrong. A high starting α was expected to bootstrap the rule early, and the steep 4-to-2 schedule was named as one that should peak early. The timing runs the other way.
Measured by the iteration
at which each seed's unseen_mismatch peaks, the 4-to-2 schedule has
a mean peak iteration of 1030, far from the predicted 300 or sooner, so
s4e2_peaks_early is rejected on every seed. The 5-to-2 schedule
peaks later than 3-to-2, 1150 against 920, so
higher_start_peaks_earlier is mixed, its mean running against the
prediction with only 3 of 10 seeds in the predicted direction. Among the
schedules that all end at a floor of 1, the peak instead arrives earliest from
a moderate start near 2, at a mean peak iteration of 785, and raising the
start delays it to 1025, 1170, and 1185 for starts of 3, 4, and 5.
The steep 4-to-2 schedule does stay high, keeping its accuracy above 70 on 8 of 10 seeds so that s4e2_stays_high holds, but it climbs slowly and reaches its maximum only at the final waypoint. It stays high without peaking early.
A start as low as 1 is too diffuse to build the rule reliably, which shows up as seed coverage rather than as timing. The fixed 1-to-1 schedule ends at the same good α = 1 floor yet reaches only 86.9, because it brings the rule to just 8 of its 10 seeds and leaves the other two at 62.4 and 19.2; the 1-to-0 schedule, a diffuse start with a deep tail on top, reaches a near-peak band of 80 and above on only 1 of 10 seeds. This seed coverage is a different quantity from the report's replicate-coverage check, which only confirms that every arm ran the same ten seeds.
Read back onto child-directed speech, the direction of the trajectory transfers but its scale does not. A gentle decline to a low but nonzero floor reaches a higher plateau than any fixed setting while still building the rule early, so falling α is the right shape. The productive band is lower and narrower than a child's input: the model does best starting near α = 2, settling near α = 1, and never reaching uniform. It does not need to begin from a distribution as concentrated as early child-directed speech, and unlike a child it is harmed rather than helped when the predictability of its input decays all the way to flat.
As a working recipe, the schedule from α = 2 to α = 1 is the one to reach for when both early competence and a stable endpoint matter. It has the highest single peak in the sweep, 95.8, and finishes at 92.4, within the band of the other schedules that end at 1. A fixed α = 2 is simpler and reaches its own peak sooner, at a mean peak iteration of 320 against 785, but settles about six points lower, at a peak of 93.8 and a final of 86.9. Falling α buys the extra height at little cost to when competence first appears.
Three questions stay open. The onset optimum sits at the low edge of the integer grid, so a finer sweep of starts between 1 and 3, and of floors between 0 and 1, would place both the speed optimum and the erosion edge more precisely. The stability of the start-2 schedule is established only to 1200 iterations, and the efficiency_v1 result is a reminder that erosion can appear later, so a longer horizon is worth running. And the floor-0 collapse divides seeds sharply, so what decides whether a given seed recovers or is lost remains unknown.
Comparison figures
Children
| Child | peak unseen_mismatch | Status |
|---|---|---|
| s5_e5 | 66.0 | done |
| s5_e4 | 69.1 | done |
| s5_e3 | 75.4 | done |
| s5_e2 | 82.5 | done |
| s5_e1 | 93.0 | done |
| s5_e0 | 93.1 | done |
| s4_e4 | 72.1 | done |
| s4_e3 | 75.9 | done |
| s4_e2 | 80.1 | done |
| s4_e1 | 94.0 | done |
| s4_e0 | 87.8 | done |
| s3_e3 | 79.4 | done |
| s3_e2 | 86.6 | done |
| s3_e1 | 93.2 | done |
| s3_e0 | 92.0 | done |
| s2_e2 | 93.8 | done |
| s2_e1 | 95.8 | done |
| s2_e0 | 94.0 | done |
| s1_e1 | 89.5 | done |
| s1_e0 | 51.2 | done |