Text set in serif type with a dotted bar in the margin was drafted by an LLM and sometimes reviewed by the author. The rest is the author's own. How to read this site

Investigations

• anaphor-binding 2 experiments · latest 2026-09-04
  Intent: Does concentration help a dependency it never touched \@{anaphor-binding-intent}

  This investigation asks whether the statistical conditions that help a model learn subject-verb number agreement also help it learn a dependency the Zipfian concentration never touched. The paper's account is collocational bootstrapping: repetitive noun-verb pairings give the model a reason to represent the subject at all, and an intermediate α generalizes best because it is repetitive enough to build that representation and varied enough to stop the model from memorizing the pairs it saw. If that account is right, what the concentration builds is a representation of a position rather than of a verb's partner list, and some other dependency that needs the subject should be able to use it.

  The anaphor task is that test. It holds the sentence space, the α sampling and the held-out pairings fixed, and replaces the number-marked verb with an uninflected past-tense form followed by a clause-final reflexive that has to agree with the subject. The reflexive is never part of the pairing the α distribution is defined over. If the accuracy curve still turns over at an interior α, the concentration is doing positional work that outlives the dependency it was measured on; if the curve is flat, the mechanism is lexical and stops at the verb.

• data-reuse 1 experiment · latest 2026-08-03
  Intent: How does recycling a finite training pool change what the model learns \@{data-reuse-intent}

  This investigation asks how training-sentence Reuse changes subject-verb agreement learning at a fixed optimizer training budget. It separates reuse from with / without replacement: every arm draws unique sentences without replacement, so α controls which noun-verb pairings enter the pool rather than how often a retained sentence is presented.

  Its first experiment starts from the paper's constant-α design. The paper cycles a 9,600-sentence training split four times to fill 38,400 training slots. The matched no-reuse regime instead draws 38,400 unique training sentences and presents each once. Comparing them across the full α grid and the oneshot limit tests whether the published generalization curve depends on seeing the same sentences repeatedly.

• alpha-curriculum 27 experiments · latest 2026-07-28
  Intent: Does varying α within one training run change what the model learns \@{alpha-curriculum-intent}

  This investigation asks whether varying the pairing concentration α over the course of a single training run changes what the model learns about subject-verb number agreement, compared with holding α fixed. The paper trains every model at one fixed α; the CHILDES literature shows that child-directed speech moves from concentrated toward uniform as children age, which suggests a schedule might be the natural regime rather than the exception. The member experiments map that question along its axes in turn: whether any schedule beats the best fixed α, whether the gain is speed or height, which direction of travel matters, what shape of schedule is optimal, and what a shorter training budget does to all of it.

  1 sub-investigation

  • schedule-opt 16 experiments · latest 2026-07-28

    This investigation asks a single question inherited from the schedule line of the alpha-curriculum work: what schedule gets a model to its generalization peak fastest, and how sure can we be of the answer? The objective, fixed before any search ran, is the probability — over random seeds — that the generalization score reaches a frozen threshold within a frozen step budget. The campaign proceeds in gated phases: calibration of the threshold and budget, a space-filling sweep of a three-parameter schedule family, Bayesian optimization within it, confirmation of the winners on held-out seeds never touched during the search, and a nested test of whether a fourth shape parameter buys anything real. Only held-out numbers are reported as results. The design and its statistical reasoning live in the repository spec 2026-07-21-schedule-opt-campaign-design.md.

    5 sub-investigations

    • schedopt-v5 1 experiment · latest 2026-07-28

      The previous round searched a free-form space of block curricula and ranked what it found by median crossing time. The campaign does not report the median. It reports the 90th percentile over seeds, which is the number that says whether a schedule works for nearly every seed or merely for a typical one, and a median-ranked search is blind to it: two candidates on the previous round's board share a median of 187.4 steps and differ by 87 in their 90th percentile.

      This round therefore optimizes the quantity it reports. Each candidate gets its own dispersion parameter, shrunk toward the pooled value by how many of its seeds actually crossed, and two surrogates are fitted over the schedule space, one for location and one for dispersion, so the acquisition can ask how ragged a region is as well as how fast. The search itself is given the half the previous round lacked: trust regions that widen when they find something, shrink when they stop, and are replaced when they collapse, driven unattended across days rather than one round at a time.

      The instrument is unchanged from the previous round, so every row of that board carries forward as training data and the two are directly comparable. A crossing is still confirmed by re-measuring frozen weights against fresh minimal pairs rather than by waiting for a second waypoint. The design is in the repository spec 2026-07-26-schedopt-v5-turbo-driver-design.md, and the campaign log, including what the first session got wrong, is in V5.md.

    • schedopt-v4 1 experiment · latest 2026-07-26

      The first three rounds of this campaign each froze an objective, searched a parameterized family of curricula, and defended a winner. This one drops the family. A candidate is K blocks of α level and duration, with K drawn rather than fixed and the boundaries free integers, so the search space contains constant schedules, monotone descents and oscillations alike as special cases, and contains shapes none of the earlier families could express.

      The reason to spend a round this way is that every family used so far was chosen after the fact to contain the arms that were already winning, which makes a search inside it evidence about the family rather than about the space. The node also carries the instrument change that makes the numbers here incomparable to the earlier rounds: a crossing is confirmed by re-measuring frozen weights against fresh minimal pairs rather than by waiting for a second waypoint.

    • schedopt-v2 6 experiments · latest 2026-07-23

      The v2 arm of the schedule-optimization campaign gathers every experiment scored against the re-frozen objective: the probability, over random seeds, that unseen_mismatch accuracy reaches m_star = 95.5 within k = 350 steps, guarded by the same two-waypoint dwell (experiments/schedopt/config_v2.json). Its members are the Step-D search round that re-searched the three-parameter schedule family at the corrected point, the Phase-4 rounds that added a tooth-amplitude parameter riding the ramp, and the Step-F confirmation on fresh held-out seeds. The operating point was re-frozen only after the Phase-3 confirmation had been seen, and it was chosen from search and calibration data alone so that the held-out comparison stayed honest. Nothing here confirms a v1 result: the re-freeze is always reported as a re-freeze, the two operating points are never merged, and no number recorded under v2 is comparable to a number under v1.

    • schedopt-v1 8 experiments · latest 2026-07-23

      The v1 arm of the schedule-optimization campaign gathers every experiment scored against the original frozen objective: the probability, over random seeds, that unseen_mismatch accuracy reaches m_star = 92.5 within k = 400 steps, guarded by a two-waypoint dwell (experiments/schedopt/config.json). Its members are the Phase-0 calibration that froze that objective, the Sobol design that first mapped the three-parameter schedule family, the four Bayesian-optimization rounds that climbed it, and the Phase-3 confirmation on held-out seeds. The objective reached saturation here: the winning descent crosses on 0.94 of held-out seeds and the top three candidates are statistically inseparable, which is what motivated re-freezing a second operating point. The two operating points are never merged, and no number recorded under v1 is comparable to a number under v2.

    • schedopt-v3 no experiments yet

      The first two operating points of this campaign scored a schedule by a single number: the probability that a seed's generalization accuracy reaches a frozen threshold by a frozen deadline. That objective answered its question and then ran out of resolution — at the second operating point the three leading arms were pairwise inseparable on crossing probability (The re-frozen campaign: reliability and speed are separable virtues) while differing by roughly seventy steps in median crossing time. This investigation replaces the deadline Bernoulli with a frozen objective on the full distribution of the per-seed crossing time: minimize its 90th percentile under right-censoring at a horizon, so that an arm wins only if at least nine in ten seeds cross at all, and among such arms the one whose slowest decile crosses soonest. The change of objective is a fresh freeze; no number here is comparable to the v1 or v2 operating points, and the earlier winners enter only as reference arms with no incumbency. Four sub-investigations carry the phases: the tail diagnostics that decide whether fixed-concentration training stalls heavy-tailed, the restart baselines that classical theory demands if it does, the staged schedule search with its held-out confirmation, and the mechanism studies that turn a descriptive win into a causal account.

      4 sub-investigations

      • schedopt-v3-mechanism no experiments yet

        Phase E of the v3 campaign. The descriptive result — a high-start, late-diving concentration schedule truncates the tail of stalling seeds — leaves the mechanism open. This node measures, per seed and along training, when the agreement rule is built and when it erodes; whether the late dive plays the role a learning-rate cooldown plays in the warmup-stable-decay picture; and whether stalled seeds resemble delayed, grokking-like generalization. Ablations that remove the high start, the hold, or the late dive — and mid-run branch interventions that inject a concentration excursion into stalled fixed-$\alpha$ runs — convert correlation into cause by showing which component reintroduces the tail and which intervention rescues it.

      • schedopt-v3-restarts no experiments yet

        Phase B of the v3 campaign. If fixed-concentration training stalls on a heavy tail of seeds, the classical remedy is not a schedule but random restarts, and any tail-truncation claim for a schedule must beat that remedy at matched compute. This node re-tunes the fixed-$\alpha$ baseline for the v3 objective with no incumbency, then stands up two restart baselines over it: the universal Luby cutoff sequence and an oracle fixed cutoff chosen on calibration data, both simulated from a large bank of independent fixed-$\alpha$ runs and validated by real restart chains executed end to end. The comparison of the descent schedule against these policies is the campaign's honest control: if restarts match the schedule, the headline changes.

      • schedopt-v3-search no experiments yet

        Phases C and D of the v3 campaign. The search space is a staged hierarchy — a four-parameter warmup-stable-decay envelope, an envelope-plus-teeth modulation nesting the v2 winners, and a gated free-form per-waypoint family — searched with a Gaussian-process surrogate whose censored log-time likelihood scores did-not-finish seeds correctly, under Thompson sampling on the posterior of the censored 90th-percentile crossing time. A parametric response surface, population-based training, and a random-schedule control run alongside as cheap cross-checks. The selected arms then face one held-out confirmation on a fresh paired seed cohort, adjudicated by paired restricted-mean survival differences, weighted log-rank tests, almost-stochastic-order dominance, and winner's-curse shrinkage; search-pool numbers are never reported as results.

      • schedopt-v3-tails no experiments yet

        Phase A of the v3 campaign. Before any search spends compute, this node establishes the empirical ground the new objective stands on: extract per-seed crossing times from the existing v1/v2 trajectories, run a reference panel of seven arms to a long horizon so the right tail is actually observed rather than censored at 350 steps, test whether the fixed-concentration baseline's time-to-generalization distribution is heavy-tailed, bimodal, or merely shifted, and calibrate the frozen operating point — the threshold, the horizon, and the instrument — by a pre-registered discrimination criterion computed on calibration seeds only. The phase gate decides the campaign's framing: tail truncation if the tail is real, median speed if it is not.

• scaling 7 experiments · latest 2026-07-16
  Intent: How the result scales \@{scaling-intent}

  This investigation gathers the two axes along which the paper's number-agreement result is scaled: the model itself, and the vocabulary the grammar draws from. Its member investigations ask the same question from opposite sides of a fixed training budget: model-scale varies embedding width and depth against the default vocabulary, while vocabulary-scale holds the paper's architecture still and grows the token inventory. The two meet at capacity: the same 256-dimensional embedding ceiling is reached from one side by shrinking the model (the capacity ladder) and from the other by growing the vocabulary (the embedding ceiling), which is why the axes belong to one investigation rather than two. The open joint probe both members name, a wider grid over width × vocabulary, would land here.

  2 sub-investigations

  • model-scale 4 experiments · latest 2026-07-16
    Intent: How model size gates agreement generalization \@{model-scale-intent}

    This investigation asks how a model's size gates its ability to learn subject-verb number agreement well enough to generalize to unseen pairings. Size has two axes here, embedding width (n_embd) and depth (n_layer), and the member experiments vary each against the full α sweep: first the width ladder, then depth, then longer-budget continuations of both to separate what is a property of the architecture from what is a sign of undertraining at the paper's budget.

    Referenced by (1 direct)

    Direct references:

    • How the result scales
  • vocabulary-scale 3 experiments · latest 2026-07-20
    Intent: How the result scales with vocabulary size and holdout \@{vocabulary-scale-intent}

    This investigation asks how the paper's agreement result scales along the vocabulary axis. The grammar makes vocabulary a free parameter, and two things change with it at once: how many tokens the fixed model must represent, and how much seen pairing evidence training provides. The member experiments hold the model at the paper's architecture and pull those apart: a six-size sweep from vocabulary 20 to 160 at a fixed quarter holdout, and two matched sweeps at a one-third holdout, at vocabulary 60 and 120, that vary the evidence fraction with the vocabulary held still.

    Referenced by (1 direct)

    Direct references:

    • How the result scales
• paper-reproduction 3 experiments · latest 2026-07-16
  Intent: Does the paper's central result reproduce, and how far does it carry \@{paper-reproduction-intent}

  This investigation asks whether the paper's central result reproduces and how far it carries. The result is Figure 2: on unseen_mismatch, the hardest minimal pair condition, generalization of subject-verb number agreement peaks at an intermediate Zipfian concentration and the oneshot limit generalizes worse. The member experiments reproduce the figure at the paper's own configuration, then probe how the picture moves with the training budget and where in training each α is at its best.

• bimodality 3 experiments · latest 2026-08-28
  Intent: When and why seed cohorts split into two basins \@{bimodality-intent}

  This investigation tracks a phenomenon that cuts across the others rather than owning experiments of its own: seed bimodality, where a cohort of replicates differing only in seed splits into two basins, some models learning the subject-verb number rule and the rest agreeing with the nearest noun or memorizing, with almost nothing in between. The question is when the split appears, whether it is a transient of undertraining or a stable property of a configuration, and what it does to any reading based on cohort means.