Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5, claude-opus-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
alpha-curriculum
investigation
Intent
This investigation asks whether varying the pairing concentration α over the course of a single training run changes what the model learns about subject-verb number agreement, compared with holding α fixed. The paper trains every model at one fixed α; the CHILDES literature shows that child-directed speech moves from concentrated toward uniform as children age, which suggests a schedule might be the natural regime rather than the exception. The member experiments map that question along its axes in turn: whether any schedule beats the best fixed α, whether the gain is speed or height, which direction of travel matters, what shape of schedule is optimal, and what a shorter training budget does to all of it.
Conclusions
Eleven experiments converge on one account and then sharpen it.
Training that starts at high pairing concentration builds the number rule early, through collocational bootstrapping, and sustained training near uniform afterwards erodes it through interference. What a schedule buys is speed and seed coverage, not a higher ceiling; what it risks is where it ends. At a fixed horizon the final α, not the direction of travel, sets the outcome.
The account was assembled in three steps. The first experiment found the asymmetry: a peaked start accelerates acquisition (curriculum_v1) but a uniform tail then undoes the rule it built (build, then erode), while the reverse direction holds what it learns (the anti-curriculum is durable). Burst mixing showed the same forces in miniature, brief deep-uniform bursts doing no harm (burst_v1) while the balance of build against erosion sets the endpoint (the balance). The efficiency comparison made the gain precise: the curriculum reaches a higher peak much sooner than fixed α = 1.4 (the fast peak), and the slide that follows is not specific to a uniform tail (erosion without one), so late decline is erosion at work rather than one bad schedule choice.
The shape of the optimal schedule turned out to be flatter than the search assumed. The starting height saturates (peaksweep_v1, anchor_v1), and the systematic φ-landscape walk found the landscape sloppy in most directions (the deformation map), with the real levers being where the schedule starts its descent (the start lever) and how much of the α range the bursts cover (coverage, not height). Speed and consistency move together rather than trading off (anchor_v1), and no smooth limit of the burst structure recovers its advantage (refine_v1).
Two boundaries close the account. Compressing the winning shape into a shorter horizon buys nothing: the model needs on the order of 196 steps to bootstrap the rule regardless of schedule (the hard floor), and what a short budget costs is seed coverage rather than peak height (the coverage cost). And the mirror test showed direction itself is not the operative variable (mirror_v1): descent wins retention when it lands at a safe floor (retention), but every pair ending at uniform reverses (final α dominates), and on interior pairs the ascending arm reaches the higher peak within the budget (ascent out-peaks), the signature of undertraining in the descending mirror rather than a directional law.
A sub-investigation took the speed half of that account and made it the objective, scoring a schedule by how soon nine seeds in ten reach a fixed generalization band rather than by where the accuracy curve ends (the schedule-opt campaign). Its answer is the account's shape with the descent compressed to a single switch: hold α at 4.76 for 160 steps, then drop once to 0.81 and stay (The fastest schedule the campaign has found). That schedule reaches the band at a held-out 90th percentile of 202.1 steps against 337.9 for the best constant α, which puts a number on the claim that a schedule buys speed and seed coverage rather than a higher ceiling: about 136 steps, more than a third of the horizon. Two searches over free-form curricula, one random and one surrogate-driven, have since found nothing faster, so on this objective the shape appears to be the finding rather than its tuning.
What would change this reading. The interior mirror pairs were read at a fixed horizon with the descending arms' peaks still rising, so a longer horizon could return the peak advantage to descent. And the account is built entirely on schedules that are piecewise-constant in α over a two-preposition grammar; a maintenance recipe that holds the rule indefinitely after the build phase has not been found, only diets that slow the slide.
Referenced by (1 direct)
Direct references:
Sub-investigations
-
schedule-opt 16 experiments
This investigation asks a single question inherited from the schedule line of the alpha-curriculum work: what schedule gets a model to its generalization peak fastest, and how sure can we be of the answer? The objective, fixed before any search ran, is the probability — over random seeds — that the generalization score reaches a frozen threshold within a frozen step budget. The campaign proceeds in gated phases: calibration of the threshold and budget, a space-filling sweep of a three-parameter schedule family, Bayesian optimization within it, confirmation of the winners on held-out seeds never touched during the search, and a nested test of whether a fourth shape parameter buys anything real. Only held-out numbers are reported as results. The design and its statistical reasoning live in the repository spec
2026-07-21-schedule-opt-campaign-design.md.
Experiments
-
20260716_133911_mirror_v1 2026-07-17
This experiment tests directly whether the direction of an α schedule matters in itself. Each mirror pair runs the same α values for the same number of steps with the same seeds, once descending and once ascending; each segment (stage) draws its training pool fresh at its own α, so direction is the only systematic difference.
The grid covers every pair of integer endpoints from 0 to 5 in two shapes: the smooth 24-stage linear ramp of slopesweep_v1, and a jagged ladder with one rung per integer between the endpoints, so an adjacent pair degenerates to a single abrupt switch. Flat arms at each grid value calibrate what no direction looks like. All arms use the with replacement data setting. Design: the mirror_v1 spec.
-
20260713_223526_budget_v1 2026-07-16
Compressing a fixed schedule shape into progressively shorter training horizons T is meant to separate two explanations of the 196-iteration onset. It could be a property of the schedule's timing at T = 900, in which case a compressed schedule should reach the band proportionally sooner; or it could be a hard learning floor, in which case the model cannot bootstrap the agreement rule faster than about 180 to 196 iterations however the schedule is compressed.
Each arm is read as the pair of its per-seed time-to-band and its coverage, because an arm that reaches the band sooner while losing seeds has hit the wall rather than beaten it.
-
20260713_210738_ascent_v1 2026-07-16
This phase, the ascent of the φ-landscape study, combines the uphill directions a prior single-deformation map of the burst ladder identified, a high start, a coverage-safe valley depth at or below 1.0, and a small tooth count of 2 to 4, into candidate shapes and asks whether the combination pushes onset below the roughly 197-iteration single-deformation best while holding full 20 of 20 coverage.
All arms are the continuous burst ladder at a horizon of 900 iterations, and every arm is scored on a pair of numbers, the per-seed time-to-band and the coverage (the fraction of seeds whose peak reaches 95,
peak_per_seed.summary.frac_ge[95]), rather than on the cohort mean. Peak height is saturated near 100 and is not an objective. -
20260713_182951_deform_v1 2026-07-16
This is Phase 3 of the φ-landscape study, whose design and full term definitions live in the design spec. It draws the deformation map, the campaign's headline result.
Holding one shape fixed, a three-tooth descending burst ladder, the experiment perturbs one schedule parameter at a time and measures how each direction moves the two things still open: how soon the model reaches the 95 band (per-seed time-to-band) and how many seeds reach it at all (coverage, the fraction of seeds whose own peak reaches 95). Peak height is saturated, near 100 per seed, so it is not an objective here. The output is a labelled map of which deformations are sloppy, meaning coverage and speed barely move, and which are stiff, meaning they collapse, together with where the cliffs sit.
-
20260713_164535_refine_v1 2026-07-16
This experiment asks whether the discrete burst structure of the winning schedule from Phase 1 of the φ-landscape study is load-bearing for the two things still open, which are how fast the model reaches a near-peak band (time-to-band) and how few seeds fail (consistency), and whether refining the schedule's staircase toward a smooth curve preserves or improves them.
Two shapes, a three-tooth burst ladder like the incumbent and the same descending envelope with the bursts removed, are each realized at increasing staircase fineness K, in the staged setting (fresh finite data per stage) and the continuous setting (every batch sampled fresh).
-
20260713_144937_anchor_v1 2026-07-16
This is Phase 1 of the φ-landscape study, whose design and full term definitions live in the design spec.
Its purpose is to pin down how high the generalization score of the current best training schedule actually peaks, and to reconcile the two ways of delivering that schedule's data, staged and continuous. Both run at a high seed count, forty seeds per arm, so that any run-to-run split in outcomes shows up as itself rather than being averaged away.
-
20260713_095946_slopesweep_v1 2026-07-16
The question is whether a steeper, model-tuned decline in α does better than the empirical trajectory of child-directed speech. The sweep crosses the starting concentration, from 1 to 5, with the floor the schedule settles to: from each start, α declines linearly to every value between that start and 0. All arms use the continuous, per-batch data setting. The goal is to map where the erosion floor sits as a function of the starting height.
Every arm of this sweep walks the same shape: a staircase of 24 segments that lowers α linearly from a chosen starting concentration to a chosen floor, drawing each segment's training pool fresh at its own α. Arms differ only in their two endpoints.
Referenced by (1 direct)
Direct references:
-
20260712_190403_peaksweep_v1 2026-07-16
This experiment asks whether 3.0 is the right height to start the descending α ladder of efficiency_v1's winning schedule from. It holds the winning schedule's structure fixed, keeping the same stage length, the same seeds, and the same 900-iteration prefix, and sweeps only the ladder's starting peak P from 1.0 to 5.0, scaling the three rungs together as P, 0.714P, and 0.429P. The P = 3.0 arm is byte-identical training to efficiency_v1's winner over these 900 iterations, so what is new is the P axis and a finer waypoint spacing, every 50 iterations instead of 150, to resolve when the generalization peak actually arrives.
-
20260712_084956_efficiency_v1 2026-07-16
This experiment asks whether the anneal-plus-bursts curriculum learns agreement more efficiently than a fixed α of 1.4, measured over 2400 iterations, twice the horizon of the earlier curriculum experiments. Efficiency here has two parts: whether the curriculum reaches its peak sooner, and whether it then stays higher.
-
20260711_222607_burst_v1 2026-07-16
This experiment takes up GitHub issue #3, α-burst mixing: rather than holding the pairing concentration fixed or annealing it smoothly, it drops short bursts of a different α into an otherwise steady schedule and asks what those bursts do.
Three questions follow. Does brief exposure to a moderate α of 1.0 change a model trained mostly on a fixed α of 1.4? Does brief exposure to deep-uniform data at α = 0.0 damage the rule, or does damage require sustained exposure? And can 1.0-bursts interleaved through a 3-to-0 anneal protect it from the deep-uniform collapse that curriculum_v1 found?
-
20260711_172518_curriculum_v1 2026-07-16
The paper trains each model at a single fixed pairing distribution.
This experiment asks whether varying that distribution over the course of one run does better. Concretely, does annealing the Zipfian exponent α from peaked toward uniform within a single 1200-iteration run beat the best fixed setting, α = 1.4, at the same training budget?