Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5, claude-opus-4-8) and sometimes reviewed by the author. The rest is the author's own. How to read this site
schedopt_calib
20260721_150444_schedopt_calib · complete · published 2026-07-23 · seed 2000
part of investigation alpha-curriculum → schedule-opt → schedopt-v1
Intent
Run 2026-07-21 (about 73 minutes on a single GTX 1660 Ti at two concurrent jobs; 240 models, no failures).
Phase 0 of the schedopt campaign (spec: the design spec): calibrate the objective. Eight probe schedules chart how the probability of reaching a threshold m_star by a step budget k varies across schedule shapes, so m_star, k, and the smoothing/dwell guards can be frozen before any optimization runs. Nothing is optimized here.
Hypothesis
At m_star near 95 and k in 200-250, the probes separate: the incumbent and fast-descending shapes cross early, constants read near zero, and the ascending ramp sits between. If all probes are indistinguishable there, the campaign stops.
Setup
| arm | data regime | datasets | iters/dataset | epochs/dataset |
|---|---|---|---|---|
| down_floor | fresh-dataset-per-stage | 16 | 25 | 0.02 |
| early_fast | fresh-dataset-per-stage | 16 | 25 | 0.02 |
| fixed_16 | fresh-dataset-per-stage | 16 | 25 | 0.02 |
| fixed_30 | fresh-dataset-per-stage | 16 | 25 | 0.02 |
| incumbent | fresh-dataset-per-stage | 48 | [18, 19] | [0.01, 0.02] |
| late_fast | fresh-dataset-per-stage | 16 | 25 | 0.02 |
| lin_down | fresh-dataset-per-stage | 16 | 25 | 0.02 |
| lin_up | fresh-dataset-per-stage | 16 | 25 | 0.02 |
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout c71cfbb772e4
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yaml
Checks
PASS children_complete PASS replicate_coverage PASS schedule_total_matches_final_waypoint PASS instrument_consistent PASS objective_computable
Results — objective: P(unseen_mismatch ≥ 92.5 by t = 400, dwell 2, smoothing 1)
| arm | ŝ | wilson95 | median crossing [CI] | crossed |
|---|---|---|---|---|
| incumbent | 1.000 | [0.886, 1.000] | 188.1 [184.3, 191.5] | 30/30 |
| lin_down | 0.933 | [0.787, 0.982] | 311.9 [305.0, 316.4] | 28/30 |
| lin_up | 0.567 | [0.392, 0.726] | 299.6 [279.2, 321.2] | 17/30 |
| fixed_16 | 0.533 | [0.361, 0.698] | 297.2 [272.5, 342.5] | 16/30 |
| early_fast | 0.333 | [0.192, 0.512] | 225.7 [169.9, 350.0] | 10/30 |
| fixed_30 | 0.033 | [0.006, 0.167] | 147.1 [147.1, 147.1] | 1/30 |
| down_floor | 0.000 | [0.000, 0.114] | — | 0/30 |
| late_fast | 0.000 | [0.000, 0.114] | — | 0/30 |
Trajectory summary — unseen_mismatch
Evaluation data: seen_* probed at reference α = 1.4 · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at fixed reference α = 1.4 (not at the schedule's own α, so arms stay comparable)
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | end of training (t = 400) | per-seed | flags |
|---|---|---|---|
| down_floor | 89.1 ±5.1 | ||
| early_fast | 90.7 ±4.2 | ||
| fixed_16 | 84.2 ±19.2 | ⚠ seed_split: range 49.6-100.0 (7 low / 19 high of 30) | |
| fixed_30 | 73.2 ±4.4 | ||
| incumbent | 85.2 ±14.0 | ⚠ seed_split: range 52.1-100.0 (4 low / 17 high of 30) | |
| late_fast | 84.7 ±5.7 | ||
| lin_down | 84.0 ±17.4 | ⚠ seed_split: range 43.0-100.0 (2 low / 16 high of 30) | |
| lin_up | 88.2 ±5.9 |
Conclusions
The run returned a usable operating point, but not the one the hypothesis named. Reaching 95 unseen_mismatch within 250 steps is something almost nothing in the smooth three-parameter family under study does: six of the seven in-family probes crossed on none of their 30 seeds, the seventh on 0.10, while the off-family burst schedule carried over from the earlier work crossed on 0.93. An objective that reads zero everywhere the search is about to look offers the optimizer nothing to climb, which is the failure mode this phase exists to catch. Lowering the threshold to 92.5 and extending the budget to 400 recovers a graded response across the same probes, and the campaign is frozen there.
At 95 unseen_mismatch within 250 steps, six of the seven probes drawn from the smooth family crossed on 0.00 of 30 seeds and early_fast on 0.10, against 0.93 for the burst schedule incumbent. At 92.5 within 400 the same probes spread out: lin_down 0.93, lin_up 0.57, fixed_16 0.53, early_fast 0.33, fixed_30 0.03, and down_floor and late_fast 0.00, with the incumbent at 1.00. The frozen objective is the probability that a schedule's run reaches 92.5 within 400 steps.
The pre-registration left open whether the crossing test needs a dwell guard, a requirement that the threshold be met at two consecutive waypoints rather than one. It does, and the guard turns out to do more than suppress noise. Requiring the level to hold for 25 steps removes every one of down_floor's apparent crossings and roughly a third of lin_up's and fixed_16's, while leaving the incumbent untouched. Schedules therefore differ not only in whether they reach the band but in whether they stay there once they do, and a test without the guard would score a graze and a durable crossing alike. Smoothing was rejected on the opposite evidence: a three-waypoint average shifts crossing times and shaves a few points off most arms without separating any pair that the raw curve did not already separate.
Moving the dwell requirement from one waypoint to two, at 92.5 within 400 steps, drops down_floor from 0.30 to 0.00, lin_up from 0.83 to 0.57, fixed_16 from 0.73 to 0.53, and early_fast from 0.50 to 0.33, while the incumbent holds at 1.00 and lin_down falls only from 0.97 to 0.93. The guard changes the ordering of the arms, so it is part of the objective rather than a display choice.
The probe ordering itself says something about what the search is likely to find. The two arms that never cross are the two that hold α above the productive region for most of the run: down_floor descends from 5 only as far as 1.3 and reaches that floor at the final waypoint, and late_fast stays near 5 until late. Neither is incapable of the band so much as out of time, since their best unseen_mismatch averages 89.7 and 84.7 across seeds and every late_fast seed and 0.77 of down_floor's are still rising at 400 steps, which reads as undertraining rather than a ceiling. What separates the arms is how much training happens once the schedule has brought α into the region where the rule is learnable, and both a descent through that region and an ascent into it beat holding above it.
Two arms differ only in where their linear descent ends. lin_down runs from α 5 to 0 and crosses on 0.93 of seeds; down_floor runs from 5 to 1.3 and crosses on 0.00, its best unseen_mismatch averaging 89.7. The ascending lin_up, from 0 to 5, crosses on 0.57. Against the fixed_16 baseline over the shared seed pool, lin_down wins 13 seeds and loses 1 (exact McNemar p = 0.0018).
This does not overturn the finding that a schedule's endpoint governs its fate, because the two measurements answer different questions. Mirror_v1 scored accuracy at the end of a 1200-step run, where a long uniform tail has time to undo the rule through erosion. Here the objective is only whether a run has crossed the band by step 400, and lin_down spends just its final 50 steps below α 0.5, too brief for the tail to do damage before the budget is up. The objective is the probability of crossing by 400, not final accuracy at a longer horizon, so it will favor schedules whose endpoints would be poor choices under a longer run.
On the objective, the probability of crossing the band by step 400, the burst incumbent scored 1.00 in calibration against 0.93 for the best smooth probe. That looked like the discrete teeth, a shape the smooth three-parameter family cannot express, doing something the smooth arms could not, but the comparison is confounded: the incumbent runs 48 segments over 900 steps while every smooth probe runs 16 over 400, so it differs in segment granularity, horizon, and seed set, not in shape alone. A later matched control settled it (matched control): run at the same granularity, horizon, and seeds, the burst and the best smooth schedule cross the objective at an identical rate, so the discrete drop-and-recover structure buys nothing the smooth family does not already reach, and the smooth family is not leaving anything on the table.
Referenced by (1 direct, 3 transitive)
Direct references:
Two limits on how far this generalizes are worth stating. The calibration seeds are disjoint from every later phase's pool, so the frozen threshold carries no advantage into the search, but it was chosen to make these eight shapes separate and a different family might have wanted a different band. And no seed in the pool is hopeless: each crossed between two and five of the eight arms, so the achievable probability is not capped by a handful of broken initializations, which seed bimodality in the fixed-α arms might otherwise have suggested.
Comparison figures
Children
| Child | peak unseen_mismatch | Status |
|---|---|---|
| fixed_16 | 86.6 | done |
| fixed_30 | 74.7 | done |
| lin_up | 90.8 | done |
| lin_down | 97.2 | done |
| down_floor | 89.1 | done |
| early_fast | 90.7 | done |
| late_fast | 84.7 | done |
| incumbent | 99.6 | done |