imagining syntax

Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-4-8) and sometimes reviewed by the author. The rest is the author's own. How to read this site

curriculum_v1

20260711_172518_curriculum_v1 · complete · published 2026-07-16 · seed 42

Intent

Intent: Does annealing α within one run beat the best fixed α \@{curriculum-intent}

The paper trains each model at a single fixed pairing distribution.

This experiment asks whether varying that distribution over the course of one run does better. Concretely, does annealing the Zipfian exponent α from peaked toward uniform within a single 1200-iteration run beat the best fixed setting, α = 1.4, at the same training budget?

Background

Background: The child-directed speech motivation \@{curriculum-background}

The peaked-to-uniform shape mirrors child-directed speech, whose measured α falls from about 1.46 to 1.25 over the first 96 months.

The paper's best fixed α is 1.4.

Hypothesis

Hypothesis: An anneal should win via collocational bootstrapping \@{curriculum-hypothesis}

The proposed mechanism is collocational bootstrapping. Early training at high α, where each verb almost always takes its favorite noun, should build strong associations from a verb to its subject, so the model learns to agree with the subject rather than with the nearest noun. The later low-α tail, where pairings spread out, should then force the model to abstract agreement across the whole noun class. If this holds, a curriculum arm (cds or anneal_3_0) should beat both fixed baselines on unseen_mismatch at iteration 1200: fixed_14_single, the paper's best fixed α, and fixed_14_fresh, which draws fresh data each stage at constant α and controls for data diversity. The anti-curriculum arm reverse_0_3 should not beat them, and noisy_14 controls for variability without a trend.

Curricula

Every arm trains for 1200 steps in eight 150-step segments, and each segment draws a fresh 12,000-sentence dataset at its own α. What differs between arms is the sequence of αs. Higher α makes each verb's subject more predictable; α = 1.4 is the paper's best fixed setting, and it is the baseline every arm is measured against (the gray line in images/curriculum_comparison.png; each arm's α over training is in images/schedules.png).

  • fixed_14_single: α = 1.4 throughout, on one dataset for the whole run. This is the paper's original setup and the strongest fixed baseline.
  • fixed_14_fresh: α = 1.4 throughout, but with a fresh dataset every 150 steps. It controls for data freshness, so a curriculum win has to beat varied data at a constant α, not merely varied data.
  • cds: the eight stage αs are the values measured in child-directed speech by child age (1.46, 1.40, 1.44, 1.38, 1.37, 1.28, 1.23, 1.25 for ages 0 to 96 months). A gentle, realistic drift that stays near 1.4.
  • anneal_3_0: a linear annealing from very peaked to fully uniform (3.0, 2.57, 2.14, 1.71, 1.29, 0.86, 0.43, 0.0). The strong form of the curriculum: build the collocations first, then force generalization.
  • reverse_0_3: the same ladder run backwards, from uniform up to very peaked. The anti-curriculum control.
  • noisy_14: α = 1.4 at every stage with Gaussian jitter (σ = 0.3, redrawn per stage and per seed). It controls for variability without a trend.

Setup

model
GPT (nanoGPT-derived), weight-tied embeddings — 2 layers × 4 heads × 256 dim (~1,635,584 params) · block 50 · dropout 0.1 · vocab 167 tokens
training
AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 10 seeds (base 42)
training data
PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code
commit 74aeefb
armdata regimedatasetsiters/datasetepochs/dataset
anneal_3_0fresh-dataset-per-stage 8 150 0.5
cdsfresh-dataset-per-stage 8 150 0.5
fixed_14_freshfresh-dataset-per-stage 8 150 0.5
fixed_14_singlesingle-dataset-recycled 1 1200 4.0
noisy_14fresh-dataset-per-stage 8 150 0.5
reverse_0_3fresh-dataset-per-stage 8 150 0.5

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 74aeefb
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS schedule_total_matches_final_waypoint PASS instrument_consistent

Claims

claimsourceexpectationΔseedsverdict
anneal_beats_single translated manifest anneal_3_0 > fixed_14_single on unseen_mismatch @ final -77.8 0/10 rejected
anneal_beats_fresh translated manifest anneal_3_0 > fixed_14_fresh on unseen_mismatch @ final -78.2 0/10 rejected
cds_beats_single translated manifest cds > fixed_14_single on unseen_mismatch @ final -7.2 6/10 mixed mean says rejected, seeds 6/10
cds_beats_fresh translated manifest cds > fixed_14_fresh on unseen_mismatch @ final -7.6 5/10 mixed mean says rejected, seeds 5/10
reverse_does_not_win translated manifest reverse_0_3 < fixed_14_single on unseen_mismatch @ final -2.8 6/10 mixed mean says held, seeds 6/10
noisy_neutral translated manifest noisy_14 ~= fixed_14_single (±5) on unseen_mismatch @ final -3.2 1/10 mixed mean says held, seeds 1/10

Results — unseen_mismatch (mean ± sd over 10 seeds)

Evaluation data: seen_* probed at reference α = 1.4 · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
armend of training (t = 1200)per-seedflags
anneal_3_0 10.4 ±7.0
cds 81.0 ±25.6 ⚠ seed_split: range 7.6-97.9 (1 low / 5 high of 10)
fixed_14_fresh 88.6 ±12.4 ⚠ seed_split: range 52.1-96.7 (1 low / 9 high of 10)
fixed_14_single 88.2 ±9.1
noisy_14 85.0 ±11.4
reverse_0_3 85.4 ±6.2

Conclusions

Conclusion: Curricula accelerate acquisition but the uniform tail undoes it \@{curriculum-conclusion}

The headline hypothesis is rejected at the 1200-step endpoint, but the mechanism it proposed is half-vindicated earlier in training, and the way it fails is more informative than a clean win would have been.

Result: No curriculum beats fixed α = 1.4 at the endpoint \@{curriculum-endpoint-null}

At the end of training, mean unseen_mismatch over the ten replicates is 88.2 ±9.1 for fixed_14_single, 88.6 ±12.4 for fixed_14_fresh, 81.0 ±25.6 for cds, 10.4 ±7.0 for anneal_3_0, 85.4 ±6.2 for reverse_0_3, and 85.0 ±11.4 for noisy_14. No curriculum arm beats the fixed baseline at 1200 steps, and the strong curriculum, anneal_3_0, finishes far below it.

The mechanism does work early. The waypoints show anneal_3_0 reaching 90.5 mean unseen_mismatch by step 300, while both fixed-α = 1.4 arms are near 45 at that point; by step 750 it leads the baseline on both the mean (88.2 against 70.8) and the worst seed (76.8 against 5.7).

Conclusion: Peaked-first training accelerates acquisition \@{curriculum-bootstrapping-accelerates}

Peaked-first training learns to agree with the subject faster, and far more consistently across seeds, than a fixed α = 1.4 at the same training budget. Collocational bootstrapping accelerates acquisition, as predicted.

Referenced by (1 direct, 1 transitive)

Direct references:

Transitive (depth 1):

What the hypothesis missed is that the uniform tail then undoes the rule. As anneal_3_0 continues below α ≈ 0.9, every seed collapses: final per-seed unseen_mismatch lands between 0.3 and 20.0, below the 50 chance line, which means the models have switched to agreeing with the nearest noun, and seen_mismatch collapses to 7.3 as well. This is interference from the late uniform data, not a failure to learn, because the rule was measurably present first: the same arm's waypoints run 88.2 at step 750, 24.9 at 1050, and 10.4 at 1200 as the schedule passes from α ≈ 0.4 down to 0.0.

Result: Build, then erode \@{curriculum-build-and-erode}

Training that starts at high pairing concentration builds the agreement rule early (anneal_3_0 reaches 90.5 mean unseen_mismatch by step 300 while the fixed α = 1.4 arms sit near 45), and continued training near uniform then erases it, the same arm collapsing to 10.4 by step 1200 as its models switch to agreeing with the nearest noun. The two directions are asymmetric: a peaked tail does not undo what a uniform start built, but a uniform tail undoes what a peaked start built.

Referenced by (13 direct, 4 transitive)

Direct references:

Transitive (depth 1):

Transitive (depth 2):

cds sits at the baseline but with more spread. Nine of its ten seeds finish between 74.8 and 97.9, and one collapses to 7.6. Its αs stay between 1.23 and 1.46 and never enter the deep-uniform range, so the realistic developmental drift neither helps nor hurts at this budget, which is consistent with the damage being specific to sustained near-uniform exposure.

The anti-curriculum is not the failure the strong-curriculum story predicts.

Result: The anti-curriculum's generalization is durable \@{curriculum-reverse-durable}

reverse_0_3 ends at α = 3.0 and holds 85.4 ±6.2 unseen_mismatch, the tightest spread of any arm, along with 92.6 on seen_mismatch. Early uniform exposure appears to confer a durable ability to generalize that survives more than 150 steps of nearly one-to-one data at the end.

Referenced by (1 direct, 1 transitive)

Direct references:

Transitive (depth 1):

Conclusion: The direction asymmetry \@{curriculum-direction-asymmetry}

The two directions are asymmetric: a peaked tail does not erase the abstraction a uniform start built, but a uniform tail does erase the verb-to-subject cue a peaked start built.

The controls behave. fixed_14_fresh matches fixed_14_single, so drawing fresh data each stage is not on its own an advantage and there is no data-diversity confound to explain a curriculum effect. noisy_14 finishes just below the baseline, so jitter without a trend does not help.

Follow-ups this suggests:

  • A truncated anneal that descends from 3.0 to about 1.2 and then holds, which is the schedule the mid-run numbers say should win outright.
  • A sweep of the final-stage α over 0 to 0.9 to locate the threshold where the rule begins to be lost.
  • α-burst mixing (issue #3): whether brief uniform exposure hurts, or only sustained exposure does.
  • Whether anneal_3_0 recovers if training continues at a moderate α, using imsyn run continue on its waypoint checkpoints.

Comparison figures

comparison_by_condition.png
comparison_by_condition.png
comparison_unseen_mismatch.png
comparison_unseen_mismatch.png
curriculum_comparison.png
curriculum_comparison.png
schedules.png
schedules.png

Children

Child peak unseen_mismatchStatus
fixed_14_single 88.3 done
fixed_14_fresh 88.6 done
cds 81.0 done
anneal_3_0 90.5 done
reverse_0_3 86.9 done
noisy_14 88.9 done

experiment.yaml