imagining syntax

Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5, claude-opus-4-8) and sometimes reviewed by the author. The rest is the author's own. How to read this site

schedopt_calib

20260721_150444_schedopt_calib · complete · published 2026-07-23 · seed 2000

part of investigation alpha-curriculumschedule-optschedopt-v1

Intent

Run 2026-07-21 (about 73 minutes on a single GTX 1660 Ti at two concurrent jobs; 240 models, no failures).

Intent: Calibrating the schedule-optimization objective \@{schedopt-calib-intent}

Phase 0 of the schedopt campaign (spec: the design spec): calibrate the objective. Eight probe schedules chart how the probability of reaching a threshold m_star by a step budget k varies across schedule shapes, so m_star, k, and the smoothing/dwell guards can be frozen before any optimization runs. Nothing is optimized here.

Hypothesis

Hypothesis: The 95 band within 250 steps separates the shapes \@{schedopt-calib-hypothesis}

At m_star near 95 and k in 200-250, the probes separate: the incumbent and fast-descending shapes cross early, constants read near zero, and the ascending ramp sits between. If all probes are indistinguishable there, the campaign stops.

Setup

model
GPT (nanoGPT-derived), weight-tied embeddings — 2 layers × 4 heads × 256 dim (~1,635,584 params) · block 50 · dropout 0.1 · vocab 167 tokens
training
AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 900 iterations · fresh init per replicate (seeds derived from base seed) · 30 seeds (base 2000)
training data
PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code
commit c71cfbb772e4
armdata regimedatasetsiters/datasetepochs/dataset
down_floorfresh-dataset-per-stage 16 25 0.02
early_fastfresh-dataset-per-stage 16 25 0.02
fixed_16fresh-dataset-per-stage 16 25 0.02
fixed_30fresh-dataset-per-stage 16 25 0.02
incumbentfresh-dataset-per-stage 48 [18, 19] [0.01, 0.02]
late_fastfresh-dataset-per-stage 16 25 0.02
lin_downfresh-dataset-per-stage 16 25 0.02
lin_upfresh-dataset-per-stage 16 25 0.02

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout c71cfbb772e4
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS schedule_total_matches_final_waypoint PASS instrument_consistent PASS objective_computable

Results — objective: P(unseen_mismatch ≥ 92.5 by t = 400, dwell 2, smoothing 1)

armŝwilson95median crossing [CI]crossed
incumbent1.000[0.886, 1.000]188.1 [184.3, 191.5]30/30
lin_down0.933[0.787, 0.982]311.9 [305.0, 316.4]28/30
lin_up0.567[0.392, 0.726]299.6 [279.2, 321.2]17/30
fixed_160.533[0.361, 0.698]297.2 [272.5, 342.5]16/30
early_fast0.333[0.192, 0.512]225.7 [169.9, 350.0]10/30
fixed_300.033[0.006, 0.167]147.1 [147.1, 147.1]1/30
down_floor0.000[0.000, 0.114]0/30
late_fast0.000[0.000, 0.114]0/30

Trajectory summary — unseen_mismatch

Evaluation data: seen_* probed at reference α = 1.4 · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
armend of training (t = 400)per-seedflags
down_floor 89.1 ±5.1
early_fast 90.7 ±4.2
fixed_16 84.2 ±19.2 ⚠ seed_split: range 49.6-100.0 (7 low / 19 high of 30)
fixed_30 73.2 ±4.4
incumbent 85.2 ±14.0 ⚠ seed_split: range 52.1-100.0 (4 low / 17 high of 30)
late_fast 84.7 ±5.7
lin_down 84.0 ±17.4 ⚠ seed_split: range 43.0-100.0 (2 low / 16 high of 30)
lin_up 88.2 ±5.9

Conclusions

Conclusion: The objective the data would support \@{schedopt-calib-conclusion}

The run returned a usable operating point, but not the one the hypothesis named. Reaching 95 unseen_mismatch within 250 steps is something almost nothing in the smooth three-parameter family under study does: six of the seven in-family probes crossed on none of their 30 seeds, the seventh on 0.10, while the off-family burst schedule carried over from the earlier work crossed on 0.93. An objective that reads zero everywhere the search is about to look offers the optimizer nothing to climb, which is the failure mode this phase exists to catch. Lowering the threshold to 92.5 and extending the budget to 400 recovers a graded response across the same probes, and the campaign is frozen there.

Result: The pre-registered band is out of reach inside the search family \@{schedopt-calib-band-unreachable}

At 95 unseen_mismatch within 250 steps, six of the seven probes drawn from the smooth family crossed on 0.00 of 30 seeds and early_fast on 0.10, against 0.93 for the burst schedule incumbent. At 92.5 within 400 the same probes spread out: lin_down 0.93, lin_up 0.57, fixed_16 0.53, early_fast 0.33, fixed_30 0.03, and down_floor and late_fast 0.00, with the incumbent at 1.00. The frozen objective is the probability that a schedule's run reaches 92.5 within 400 steps.

The pre-registration left open whether the crossing test needs a dwell guard, a requirement that the threshold be met at two consecutive waypoints rather than one. It does, and the guard turns out to do more than suppress noise. Requiring the level to hold for 25 steps removes every one of down_floor's apparent crossings and roughly a third of lin_up's and fixed_16's, while leaving the incumbent untouched. Schedules therefore differ not only in whether they reach the band but in whether they stay there once they do, and a test without the guard would score a graze and a durable crossing alike. Smoothing was rejected on the opposite evidence: a three-waypoint average shifts crossing times and shaves a few points off most arms without separating any pair that the raw curve did not already separate.

Result: A third of apparent crossings do not hold for 25 steps \@{schedopt-calib-grazes}

Moving the dwell requirement from one waypoint to two, at 92.5 within 400 steps, drops down_floor from 0.30 to 0.00, lin_up from 0.83 to 0.57, fixed_16 from 0.73 to 0.53, and early_fast from 0.50 to 0.33, while the incumbent holds at 1.00 and lin_down falls only from 0.97 to 0.93. The guard changes the ordering of the arms, so it is part of the objective rather than a display choice.

The probe ordering itself says something about what the search is likely to find. The two arms that never cross are the two that hold α above the productive region for most of the run: down_floor descends from 5 only as far as 1.3 and reaches that floor at the final waypoint, and late_fast stays near 5 until late. Neither is incapable of the band so much as out of time, since their best unseen_mismatch averages 89.7 and 84.7 across seeds and every late_fast seed and 0.77 of down_floor's are still rising at 400 steps, which reads as undertraining rather than a ceiling. What separates the arms is how much training happens once the schedule has brought α into the region where the rule is learnable, and both a descent through that region and an ascent into it beat holding above it.

Result: Passing through the productive band beats stopping above it \@{schedopt-calib-through-beats-above}

Two arms differ only in where their linear descent ends. lin_down runs from α 5 to 0 and crosses on 0.93 of seeds; down_floor runs from 5 to 1.3 and crosses on 0.00, its best unseen_mismatch averaging 89.7. The ascending lin_up, from 0 to 5, crosses on 0.57. Against the fixed_16 baseline over the shared seed pool, lin_down wins 13 seeds and loses 1 (exact McNemar p = 0.0018).

This does not overturn the finding that a schedule's endpoint governs its fate, because the two measurements answer different questions. Mirror_v1 scored accuracy at the end of a 1200-step run, where a long uniform tail has time to undo the rule through erosion. Here the objective is only whether a run has crossed the band by step 400, and lin_down spends just its final 50 steps below α 0.5, too brief for the tail to do damage before the budget is up. The objective is the probability of crossing by 400, not final accuracy at a longer horizon, so it will favor schedules whose endpoints would be poor choices under a longer run.

Conclusion: The incumbent's calibration edge was a confound, resolved to a tie \@{schedopt-calib-smooth-speed-gap}

On the objective, the probability of crossing the band by step 400, the burst incumbent scored 1.00 in calibration against 0.93 for the best smooth probe. That looked like the discrete teeth, a shape the smooth three-parameter family cannot express, doing something the smooth arms could not, but the comparison is confounded: the incumbent runs 48 segments over 900 steps while every smooth probe runs 16 over 400, so it differs in segment granularity, horizon, and seed set, not in shape alone. A later matched control settled it (matched control): run at the same granularity, horizon, and seeds, the burst and the best smooth schedule cross the objective at an identical rate, so the discrete drop-and-recover structure buys nothing the smooth family does not already reach, and the smooth family is not leaving anything on the table.

Two limits on how far this generalizes are worth stating. The calibration seeds are disjoint from every later phase's pool, so the frozen threshold carries no advantage into the search, but it was chosen to make these eight shapes separate and a different family might have wanted a different band. And no seed in the pool is hopeless: each crossed between two and five of the eight arms, so the achievable probability is not capped by a handful of broken initializations, which seed bimodality in the fixed-α arms might otherwise have suggested.

Comparison figures

curriculum_comparison.png
curriculum_comparison.png
objective_crossing_ecdf.png
objective_crossing_ecdf.png
objective_shat.png
objective_shat.png
objective_survival.png
objective_survival.png
schedules.png
schedules.png

Children

Child peak unseen_mismatchStatus
fixed_16 86.6 done
fixed_30 74.7 done
lin_up 90.8 done
lin_down 97.2 done
down_floor 89.1 done
early_fast 90.7 done
late_fast 84.7 done
incumbent 99.6 done

experiment.yaml