Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-4-8) and sometimes reviewed by the author. The rest is the author's own. How to read this site

schedopt_bo_r4

20260721_222731_schedopt_bo_r4 · complete · published 2026-07-23 · seed 1000

part of investigation alpha-curriculum → schedule-opt → schedopt-v1

Intent

Intent: Round 4 of the Bayesian search \@{schedopt-bo-r4-intent} Phase 2, round 4 of the schedopt campaign: eight candidate schedules proposed by Thompson sampling from a Gaussian process fitted to every prior evaluation, scored on the frozen objective (m_star=92.5, k=400) over the shared search seed pool. Adaptivity lives between rounds only; within this manifest the eight arms are fixed before any of them runs.

Hypothesis

Hypothesis: Each round improves on the incumbent, with shrinking returns \@{schedopt-bo-r4-hypothesis} At least one candidate improves on the current incumbent's crossing probability; the round-over-round gain shrinks as the surface is learned. Search-phase s-hat values are never reported as results.

Setup

model: GPT (nanoGPT-derived), weight-tied embeddings — 2 layers × 4 heads × 256 dim (~1,635,584 params) · block 50 · dropout 0.1 · vocab 167 tokens
training: AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 400 iterations · fresh init per replicate (seeds derived from base seed) · 25 seeds (base 1000)
training data: PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code: commit 9be00c0f7835

data regime (all arms): fresh-dataset-per-stage — 16 dataset(s) × 25 iters each = 0.02 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 9be00c0f7835
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS schedule_total_matches_final_waypoint PASS instrument_consistent PASS objective_computable

Results — objective: P(unseen_mismatch ≥ 92.5 by t = 400, dwell 2, smoothing 1)

arm	ŝ	wilson95	median crossing [CI]	crossed
r4_c2	1.000	[0.867, 1.000]	322.0 [304.8, 327.8]	25/25
r4_c7	1.000	[0.867, 1.000]	305.3 [290.5, 319.6]	25/25
r4_c6	0.960	[0.805, 0.993]	346.9 [340.0, 351.6]	24/25
r4_c1	0.840	[0.653, 0.936]	288.4 [246.6, 315.4]	21/25
r4_c5	0.680	[0.484, 0.828]	296.8 [281.7, 318.1]	17/25
r4_c3	0.520	[0.335, 0.700]	353.8 [324.5, 368.6]	13/25
r4_c0	0.000	[0.000, 0.133]	—	0/25
r4_c4	0.000	[0.000, 0.133]	—	0/25

Trajectory summary — unseen_mismatch

Evaluation data: seen_* probed at reference α = 1.4 · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition

Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at fixed reference α = 1.4 (not at the schedule's own α, so arms stay comparable)
unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)

arm	end of training (t = 400)	flags
r4_c0	96.2 ±8.0	⚠ seed_split: range 57.5-99.9 (1 low / 24 high of 25)
r4_c1	92.2 ±5.0
r4_c2	91.0 ±13.8	⚠ seed_split: range 43.2-100.0 (1 low / 19 high of 25)
r4_c3	87.8 ±15.9	⚠ seed_split: range 46.8-100.0 (3 low / 16 high of 25)
r4_c4	64.1 ±4.7
r4_c5	91.8 ±7.5
r4_c6	98.5 ±2.6
r4_c7	92.0 ±16.1	⚠ seed_split: range 29.6-100.0 (1 low / 19 high of 25)

Conclusions

Conclusion: The breakthrough to a perfect search score, and the stop \@{schedopt-bo-r4-conclusion} Round 4 broke the plateau by raising the starting concentration. Two candidates, a descent from α 3.1 to 0.0 and one from 4.3 to 0.3, each crossed on all 25 seeds, a search score of 1.00. The lesson the optimizer added to the late-descent shape was that the start can go higher, to 3--4, as long as the curvature stays moderate so the run does not sit parked above the productive band; the one round-4 probe that started at 5 and lingered there crossed on none. Reaching 1.00 is the objective's ceiling, so the search stopped here: no round can improve on every seed crossing. Whether these schedules truly reach 1.00 or the pool flattered them is the winner's-curse question the held-out Phase 3 settles. Search-pool figures, not reported results. Result: Two schedules reach a perfect search score \@{schedopt-bo-r4-ceiling} Descents from α 3.1 to 0.0 (curvature 1.96) and 4.3 to 0.3 (curvature 1.07) each crossed on 25 of 25 seeds, a search crossing probability of 1.00, above the 0.96 plateau. The search stopped at this ceiling.

Comparison figures

Children

Child	peak unseen_mismatch	Status
r4_c0	96.2	done
r4_c1	93.9	done
r4_c2	98.1	done
r4_c3	87.8	done
r4_c4	64.1	done
r4_c5	91.8	done
r4_c6	98.5	done
r4_c7	99.1	done

experiment.yaml