imagining syntax

Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-4-8) and sometimes reviewed by the author. The rest is the author's own. How to read this site

slopesweep_v1

20260713_095946_slopesweep_v1 · complete · published 2026-07-16 · seed 42

Intent

The question is whether a steeper, model-tuned decline in α does better than the empirical trajectory of child-directed speech. The sweep crosses the starting concentration, from 1 to 5, with the floor the schedule settles to: from each start, α declines linearly to every value between that start and 0. All arms use the continuous, per-batch data setting. The goal is to map where the erosion floor sits as a function of the starting height.

Result: The 24-segment linear ramp \@{slopesweep-ramp}

Every arm of this sweep walks the same shape: a staircase of 24 segments that lowers α linearly from a chosen starting concentration to a chosen floor, drawing each segment's training pool fresh at its own α. Arms differ only in their two endpoints.

Referenced by (1 direct)

Background

Background: Three prior results shape the prediction \@{slopesweep-background}

This experiment is a follow-up to the child-directed speech arm of curriculum_v1, which lowered the pairing concentration α along the trajectory measured in child-directed speech.

The declining slope is the right idea, but the magnitude and range measured in child-directed speech are not expected to carry over to the model. Three results from earlier runs shape the prediction: a bootstrapping cliff between a start of α = 2 and α = 3; deep-uniform tails at α ≤ 0.4 that destroy the learned rule; and erosion that acts as a gradient, nonzero even at a floor of 1.0, with the final α weighted most heavily.

Hypothesis

The prediction is that a steep decline from a high start to a moderate floor at or above about 2, for example the 4-to-2 schedule, both peaks early and stays stable; that any schedule ending at 0 collapses regardless of where it starts; and that higher starts peak earlier.

Setup

model
GPT (nanoGPT-derived), weight-tied embeddings — 2 layers × 4 heads × 256 dim (~1,635,584 params) · block 50 · dropout 0.1 · vocab 167 tokens
training
AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 10 seeds (base 42)
training data
PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code
commit 49063016e541

data regime (all arms): with-replacement (legacy stream) — fresh batch every step, no dataset reuse

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 49063016e541
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS schedule_total_matches_final_waypoint PASS instrument_consistent

Claims

claimsourceexpectationΔseedsverdict
s4e2_stays_high manifest s4_e2 > 70 on unseen_mismatch @ final +10.1 8/10 held
s4e2_peaks_early manifest s4_e2 <= 300 on unseen_mismatch (peak time) @ final +730.0 0/10 rejected
floor2_beats_floor0 manifest s4_e2 > s4_e0 on unseen_mismatch @ final +54.0 10/10 held
s5e0_collapses manifest s5_e0 < 50 on unseen_mismatch @ final -17.9 7/10 mixed mean says held, seeds 7/10
s4e0_collapses manifest s4_e0 < 50 on unseen_mismatch @ final -23.9 9/10 held
s3e0_collapses manifest s3_e0 < 50 on unseen_mismatch @ final -31.0 10/10 held
higher_start_peaks_earlier manifest s5_e2 < s3_e2 on unseen_mismatch (peak time) @ final +230.0 3/10 mixed mean says rejected, seeds 3/10

Results — unseen_mismatch (mean ± sd over 10 seeds)

Evaluation data: seen_* probed at reference α = 1.4 · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
armend of training (t = 1200)per-seedflags
s1_e0 25.5 ±20.5 ⚠ seed_split: range 0.0-50.3 (4 low / 3 high of 10)
s1_e1 86.9 ±25.0 ⚠ seed_split: range 19.2-100.0 (1 low / 8 high of 10)
s2_e0 30.7 ±24.1 ⚠ seed_split: range 7.4-91.7 (4 low / 1 high of 10)
s2_e1 92.4 ±9.8
s2_e2 86.9 ±4.0
s3_e0 19.0 ±11.3
s3_e1 93.2 ±8.1
s3_e2 86.6 ±4.0
s3_e3 79.4 ±3.8
s4_e0 26.1 ±19.8 ⚠ seed_split: range 3.5-60.3 (5 low / 1 high of 10)
s4_e1 94.0 ±5.1
s4_e2 80.1 ±6.7
s4_e3 75.9 ±4.3
s4_e4 71.4 ±5.4
s5_e0 32.1 ±27.4 ⚠ seed_split: range 0.4-85.3 (3 low / 1 high of 10)
s5_e1 93.0 ±8.9
s5_e2 82.5 ±4.0
s5_e3 75.4 ±3.7
s5_e4 69.1 ±3.7
s5_e5 66.0 ±3.9

Conclusions

Conclusion: The floor sets the height and the start sets the timing \@{slopesweep-conclusion}

The sweep set out to ask whether the declining-α trajectory of child-directed speech helps the model once its magnitude is exaggerated.

Its more useful result was to separate two things the child-directed-speech analogy had treated as one: how high the model's agreement accuracy climbs, and how soon it gets there. The two turn out to be governed by different ends of the schedule, which each arm walks as a staircase of 24 segments from a chosen start down to a chosen floor. Where α ends sets how well the rule is learned; where α begins sets how quickly it appears, among the schedules that end high enough for the timing to matter.

Each arm's α schedule is plotted in images/schedules.png.

Result: The floor sets the height \@{slopesweep-endpoint-height}

Where the schedule ends decides the height. Two schedules that begin at different concentrations but settle to the same floor reach almost the same final unseen_mismatch, so the endpoint, not the descent that reaches it, carries the outcome. Final accuracy over the ten replicates falls steadily as the floor rises: about 92 to 94 at a floor of α = 1, the 80s at floor 2, the middle-to-high 70s at floor 3, around 70 at floor 4, and 66 at the fixed α = 5. A shallow floor near α = 1 is the best of these. Ending higher leaves the model short of it, because a distribution kept concentrated never forces generalization across the noun class, and at the most concentrated fixed settings even the easier held-out test degrades, with unseen_match at 76.0 for a fixed α = 4 and 69.2 for a fixed α = 5. Ending at uniform is worse than ending short: it removes the rule the model had already built. The pre-registered contrast floor2_beats_floor0 holds on all ten replicates, with the 4-to-2 schedule finishing at 80.1 against 26.1 for 4-to-0.

The collapse at a floor of 0 is worth stating precisely, because the model does not forget the language. On matching probes its accuracy stays at the ceiling, seen_match at 100 and unseen_match near 99, and what moves is the agreement rule itself. As the tail approaches uniform the model shifts from preferring the grammatical subject back toward agreeing with the nearest noun: on the 3-to-0 schedule, seen_mismatch falls to 15.2 while seen_match holds at 100. The dense waypoints, plotted per arm in images/curriculum_comparison.png, place this loss late in training. The 3-to-0 schedule climbs to a mean of 92.0 by step 750, then erodes to 47.4 by step 1050 and 19.0 by step 1200 as α descends through the low concentrations toward 0. These schedules reach their best score on the way down, as the descent passes through the low-α region a good floor would have held, and lose it only by continuing to uniform. This is the erosion seen in the earlier curriculum experiments, now located in time and identifiable as catastrophic interference: the rule was measurably present and then overwritten, rather than never formed.

The erosion does not fall the same way on every seed, which is why the three collapse claims separate. On the 3-to-0 schedule every replicate ends below the chance line of 50, so s3e0_collapses holds on all ten; on 4-to-0 nine of ten do, so s4e0_collapses holds; but on 5-to-0 three replicates hold above 50 while seven fall below, so s5e0_collapses is mixed, its mean of 32.1 sitting in the empty gap of a bimodal split whose per-seed finals run from 0.4 to 85.3. What decides whether a given seed keeps the rule or loses it under a uniform tail is not visible in the schedule alone.

Result: The floor-zero collapse \@{slopesweep-collapse}

Any schedule that ends at uniform loses the rule it built. Every descending arm with a floor of α = 0 finishes between 19.0 and 32.1 mean unseen_mismatch regardless of where it started, after peaking mid-run (the 3-to-0 arm reaches 92.0 at step 750 before eroding to 19.0 by 1200), which the waypoints identify as catastrophic interference rather than a failure to learn. The erosion splits seeds instead of lowering them uniformly: the 5-to-0 arm ends at 32.1 ±27.4 with per-seed finals from 0.4 to 85.3, a seed bimodality whose cause is not visible in the schedule alone.

Where the schedule begins decides how soon the rule appears, the speed axis the study tracks separately from height, and here the pre-registration was wrong. A high starting α was expected to bootstrap the rule early, and the steep 4-to-2 schedule was named as one that should peak early. The timing runs the other way.

Result: Higher starts peak later, not earlier \@{slopesweep-start-timing}

Measured by the iteration at which each seed's unseen_mismatch peaks, the 4-to-2 schedule has a mean peak iteration of 1030, far from the predicted 300 or sooner, so s4e2_peaks_early is rejected on every seed. The 5-to-2 schedule peaks later than 3-to-2, 1150 against 920, so higher_start_peaks_earlier is mixed, its mean running against the prediction with only 3 of 10 seeds in the predicted direction. Among the schedules that all end at a floor of 1, the peak instead arrives earliest from a moderate start near 2, at a mean peak iteration of 785, and raising the start delays it to 1025, 1170, and 1185 for starts of 3, 4, and 5.

The steep 4-to-2 schedule does stay high, keeping its accuracy above 70 on 8 of 10 seeds so that s4e2_stays_high holds, but it climbs slowly and reaches its maximum only at the final waypoint. It stays high without peaking early.

A start as low as 1 is too diffuse to build the rule reliably, which shows up as seed coverage rather than as timing. The fixed 1-to-1 schedule ends at the same good α = 1 floor yet reaches only 86.9, because it brings the rule to just 8 of its 10 seeds and leaves the other two at 62.4 and 19.2; the 1-to-0 schedule, a diffuse start with a deep tail on top, reaches a near-peak band of 80 and above on only 1 of 10 seeds. This seed coverage is a different quantity from the report's replicate-coverage check, which only confirms that every arm ran the same ten seeds.

Conclusion: The falling-α shape transfers but its scale does not \@{slopesweep-direction-not-scale}

Read back onto child-directed speech, the direction of the trajectory transfers but its scale does not. A gentle decline to a low but nonzero floor reaches a higher plateau than any fixed setting while still building the rule early, so falling α is the right shape. The productive band is lower and narrower than a child's input: the model does best starting near α = 2, settling near α = 1, and never reaching uniform. It does not need to begin from a distribution as concentrated as early child-directed speech, and unlike a child it is harmed rather than helped when the predictability of its input decays all the way to flat.

Result: The 2-to-1 working recipe \@{slopesweep-two-to-one-recipe}

As a working recipe, the schedule from α = 2 to α = 1 is the one to reach for when both early competence and a stable endpoint matter. It has the highest single peak in the sweep, 95.8, and finishes at 92.4, within the band of the other schedules that end at 1. A fixed α = 2 is simpler and reaches its own peak sooner, at a mean peak iteration of 320 against 785, but settles about six points lower, at a peak of 93.8 and a final of 86.9. Falling α buys the extra height at little cost to when competence first appears.

Three questions stay open. The onset optimum sits at the low edge of the integer grid, so a finer sweep of starts between 1 and 3, and of floors between 0 and 1, would place both the speed optimum and the erosion edge more precisely. The stability of the start-2 schedule is established only to 1200 iterations, and the efficiency_v1 result is a reminder that erosion can appear later, so a longer horizon is worth running. And the floor-0 collapse divides seeds sharply, so what decides whether a given seed recovers or is lost remains unknown.

Comparison figures

curriculum_comparison.png
curriculum_comparison.png
schedules.png
schedules.png

Children

Child peak unseen_mismatchStatus
s5_e5 66.0 done
s5_e4 69.1 done
s5_e3 75.4 done
s5_e2 82.5 done
s5_e1 93.0 done
s5_e0 93.1 done
s4_e4 72.1 done
s4_e3 75.9 done
s4_e2 80.1 done
s4_e1 94.0 done
s4_e0 87.8 done
s3_e3 79.4 done
s3_e2 86.6 done
s3_e1 93.2 done
s3_e0 92.0 done
s2_e2 93.8 done
s2_e1 95.8 done
s2_e0 94.0 done
s1_e1 89.5 done
s1_e0 51.2 done

experiment.yaml