This page describes what imsyn's controls actually do, so that a flag name or a directory label is never the only source of truth about a run's behavior. It begins with how training data is produced, because that is where the settings interact in ways the names alone do not convey, then how one model's training is structured and how often it shows each sentence, and then what a single training step does to the model's weights. A section on how the trained model is tested for agreement comes next, and a glossary of the recurring terms follows, each definition linking to the other terms it uses.

Between those last two sits a comparison of the measures the schedule-optimization campaign has scored itself by, and of what changed each time the measure did.

Every sentence imsyn trains on comes from one small grammar. A sentence is the <noun> <verb>, optionally wrapped by one or two prepositional phrases of the form by the <object> or near the <object>. The subject noun and the verb agree in number, and any prepositional objects sit between them as attractors for the agreement task. Because the vocabulary is fixed and the grammar has only a few slots, the set of possible training sentences is finite. Four choices fix a sentence: its number, its verb (40 options), its subject noun (one of the verb's 30 seen partners), and how its optional prepositional phrases are filled. That last factor is the number of phrase combinations. A single phrase is one of two prepositions with one of thirty object nouns, so it has 60 forms, and a sentence can carry no phrase (1), a trailing phrase (60), a leading phrase (60), or both (60 × 60). Those cases sum to 1 + 60 + 60 + 60 × 60 = 3,721. Multiplying the four choices, 2 × 40 × 30 × 3,721, gives about 8.9 million sentences.

What varies across the project is not the grammar but the distribution over which noun pairs with which verb. The parameter α, written Z in the code and in directory names, is the exponent of a Zipfian distribution over a verb's possible partners. At α = 0 the subject noun is uniform over a verb's seen partners; at α = 1 it follows classic Zipf; at higher α the mass collapses onto each verb's canonical partner. α sets how repetitive the pairings are, and it is the independent variable in almost every experiment.

Two questions decide how a run's training data behaves, and they are independent of each other. The first is whether α changes during the run. The second is how the training pool is built — how many sentences it holds and whether they are drawn with or without replacement — which together fix how often the model sees each sentence. Reading one question as if it answered the other is the most common way to misjudge a run.

The grammar above describes the default task, subject-verb agreement, where number is marked on the verb. A run may instead select the anaphor task with --task anaphor, which keeps the sentence space, the α distribution and the held-out pairings exactly as they are, and moves the agreement off the verb. Its verbs are regular past-tense forms that never inflect, so the painter painted and the painters painted share one verb token, and every sentence ends in a reflexive: the painter near the trader painted himself. Every noun in a training sentence carries the subject's number, so nothing in the training data tells binding the subject apart from binding the first noun or the nearest one, and the mismatch conditions are what separate them. The two tasks share every other knob, which is the point: a curve measured under one is read against the same axes as a curve measured under the other.

A plain imsyn run holds α fixed for the whole run. A curriculum run varies α according to a schedule, an ordered list of segments that each hold one α for a set number of steps. The schedule 3.0:300,2.0:300,1.0:300 trains at α = 3.0 for 300 steps, then at 2.0, then at 1.0. α is piecewise-constant: it changes only at segment boundaries. No mode varies α smoothly from step to step; a gradual ramp is approximated by using many short segments.

Every run trains on a fixed dataset that it draws once and then cycles. Two independent knobs shape that pool. Its size is set by --dataset-sentence-count, 12,000 by default, split into 9,600 for training and 1,200 each for validation and test. The training loader shuffles the 9,600 training sentences and cycles through them; one full pass is an epoch, which at batch size 32 takes 300 steps. A run consumes T = steps × 32 sentence-slots, so each training sentence is shown about T / 9,600 times — four at the defaults. Raising the count lowers that reuse; setting it near T removes reuse entirely, showing each sentence about once.

How the pool is drawn is the second knob, with or without replacement. By default the draw is without replacement: it keeps sampling until it has N distinct sentences, so the pool is deduplicated and every sentence in it is shown equally often. α is then visible only in which pairings were collected, never in how frequently each is shown — this is the paper's regime. With --with-replacement the draw keeps duplicates: the pool is N independent samples from the α distribution, so common pairings appear in it many times and rare ones seldom. α now weights how often the model sees each pairing, and the expected number of times a given sentence x is trained on is T · p(x) rather than a flat T / 9,600. That is a genuinely different training signal, so with-replacement runs are comparable with each other, not with the without-replacement baseline.

The without-replacement branch above needs one qualification. Deduplication is not a neutral way to collect the same distribution: past a pool size the grammar cannot supply, it lowers the realized α and reshapes how many prepositional phrases a training sentence carries. The page on the paper's own corpus works out where that threshold sits and what it costs, and the page on sampling regimes sets out what can be built instead.

Staged curriculum runs apply both knobs one segment at a time: each segment draws its own N-sentence pool at that segment's α — with or without replacement — and cycles it for the segment's duration before moving on. α still moves in discrete steps; no mode varies it continuously.

The regimes the reports name follow from these two questions:

A single imsyn run invocation trains several models, and each model's training is itself a nested structure. Naming the levels from the outside in makes the reuse question below precise.

run                          one `imsyn run …` invocation
└─ replicate  (×N)           one seeded training of the model   → iter_001 … iter_00N
   │                         seed = base_seed + i
   │  model                  the transformer this replicate trains (fresh seeded init)
   └─ epoch  (×E)            one full pass over the dataset,  E = steps / 300
      └─ step  (300 / epoch) one optimizer update
         └─ batch            32 sentences, one forward-and-backward pass
            └─ sentence ×32  drawn from the dataset

dataset   the fixed pool of 9,600 unique training sentences the epochs draw from

The two entries that are not nesting levels are the model and the dataset. A model is the network being trained; a replicate is one seeded training of it, and a run produces N replicates that differ only in seed. A dataset is the pool a replicate's batches draw from — always a materialized, cycled pool, whether it is drawn with or without replacement.

How often a given sentence is reused within one replicate follows from this structure, and the two data regimes answer it differently.

Without replacement — a plain run and each segment of a staged curriculum — the pool is a fixed set of N distinct sentences (9,600 in the training split by default). Each epoch shuffles the pool and shows every sentence once, so across E = steps / 300 epochs a sentence in the pool is shown exactly E times (⌊E⌋ or ⌈E⌉ when E is not whole) and a sentence outside the pool is never shown. Reuse is certain and even: once training runs two or more epochs, every training sentence recurs, each the same number of times. A 1,200-step run at the default size is four epochs, so it shows each of its 9,600 sentences exactly four times and the remaining eight and a half million possible sentences not at all.

With replacement the pool is instead N independent draws from the α distribution, so it holds duplicates and reuse becomes a matter of probability rather than an even count. A particular sentence occupies about N · p of the pool's slots, where p is its probability under the current α, and each slot is cycled steps × 32 / N_train times, so the sentence is trained on about steps × 32 × p times in expectation. The common, high-probability pairings recur often; the long tail of rare sentences lands in the pool once or, more often, never. At α = 0 a plain sentence with no prepositional phrase has probability near 1/9,600 and is expected about four times — the same as the without-replacement case — while raising α concentrates the mass and lifts the common pairings well above four as the tail empties.

A step turns one batch into one change to the model's weights, and it is the only place the weights change. It has two halves: measuring how wrong the model currently is, which produces the loss, and using that measurement to adjust the weights, which is the optimizer update.

Training treats every sentence as a chain of next-token predictions. A sentence is tokenized, wrapped in a beginning and an end marker, and split into an input and a target that are the same token sequence shifted by one position, so at each position the model sees the tokens so far and is asked for the token that actually comes next.

The loss for the batch is this same per-position average, taken over every real next-token position in all 32 sentences at once, with the padded −1 positions skipped.

The loss never reaches zero, because some of what comes next is genuinely uncertain. Which verb follows a subject is one of many words, and no model can always name it in advance. What is never uncertain is the verb's number, which the subject fixes, so a model that has learned agreement stops spending probability on wrong-number verbs. That avoidable part of the loss is the part training drives down, and it is the signal the whole project studies.

Backpropagation then differentiates the loss. In one backward pass it computes, for every weight, the gradient, the derivative of the loss with respect to that weight. The forward and backward passes only read and differentiate the weights; nothing so far has changed them.

The optimizer makes the change. AdamW keeps, for each weight, a running average of its recent gradients and of their squares, and moves the weight by the first divided by the square root of the second, so directions with small but consistent gradients still make steady progress and noisy ones are damped. The base learning rate is a constant 6e-4, with no warmup and no decay over the run. Two adjustments sit around this update. The gradients are first clipped to a global norm of 1.0, so one surprising batch cannot move the weights too far. And weight decay shrinks each weight slightly toward zero, at a strength set by the coefficient 0.1, which discourages large weights and tends to help the model generalize.

Weight decay is applied only to the 2-D tensors. A model's weights are stored as tensors, arrays of numbers with a fixed number of axes: a weight matrix or an embedding table is two-dimensional, a grid of outputs by inputs, while a bias or a LayerNorm scale is one-dimensional, one number per feature. The two-dimensional tensors are the matrices that mix features and hold the learned transformation, so shrinking them toward zero regularizes what the model has learned; the one-dimensional offsets and scales are left alone, because pulling them toward zero would distort the network without the same benefit. The optimizer tells the two apart by each tensor's number of axes.

Once the optimizer moves the weights the step is done. The model is now slightly different, and the next batch is measured against these new weights.

Training drives the loss down; evaluation asks the sharper question the project is built around, whether the model has learned subject-verb agreement as a rule and not as a habit that happens to fit the pairings it saw. The test isolates that one choice: given a subject, does the model prefer the verb that matches it in number, even when nouns of the other number stand between them.

The probe is a minimal pair. Take a grammatical sentence, "the runner near the skaters challenges", and form its twin by flipping the verb's number, "the runner near the skaters challenge". The two differ in one thing only, whether the verb agrees with its subject, so a model that has learned agreement should give the grammatical member the higher probability. It reads both and scores correct on the pair when it does. A condition's accuracy is the fraction of its pairs, 1,000 by default, that it scores correct, and 50 is chance.

Two independent choices about a pair produce the four conditions the reports name. The first is whether the subject-verb pairing is one training showed. A seen pairing occurred in training; an unseen pairing is one of the held-out partners that training never showed. A seen probe therefore tests agreement on familiar material, and an unseen probe tests whether the rule carries to pairings the model has never met. Unseen probes are always drawn uniformly (α = 0) over the held-out pairings, so their difficulty is the same for every run whatever its training α.

The second choice is whether the prepositional phrases help or interfere. Every probe carries both phrases, and their objects are the attractors. In a match probe the attractors share the subject's number, so following the subject and following the nearest noun call for the same verb. In a mismatch probe the attractors take the opposite number, and the two strategies part ways: "the runner near the skaters ___" wants a singular verb by agreement and a plural one by recency. Mismatch is where they separate, and it is what lets the condition tell them apart.

A model that has genuinely learned agreement does well on all four. A model that has instead settled for agreeing with the most recent noun, the shortcut the attractors exist to catch, passes the match conditions and fails the mismatch ones. unseen_mismatch is the hardest of the four and the outcome most experiments are read through: agreement generalized to a pairing never trained, across two attractors of the wrong number.

Comparing two schedules means collapsing a cohort of seeds into one number, and that collapse is not a formality. The schedule-optimization campaign has scored the same runs four different ways, and the four do not agree about which schedule is best. Each seed either reaches a target accuracy at some step or never reaches it within the budget, so an arm is a distribution over crossing times with some of its mass at "never". A measure is a choice of what to read off that distribution, and every such choice discards something.

The first two are ordinary summaries that turned out to be blind in ways that mattered. The second two are the same quantity estimated two ways, and their disagreement is what decided the fifth round.

Three further quantities are computed and reported but never used to select: restricted mean survival time, which stays defined at any censoring level and so anchors paired comparisons; CVaR, the mean of the slow tail; and the survival curve, which is the crossing probability read at every deadline rather than one. seed coverage travels beside all of them, because a fast time over the seeds that finished is not a result.

The campaign began by asking whether a schedule gets there at all. The first frozen objective was a crossing objective (ŝ): the probability that unseen_mismatch reaches 92.5 within 400 steps and holds. It answered its question, a high-start descent crossing on 0.94 of held-out seeds against 0.67 for the best constant α, and then ran out of resolution, because a probability near 1 cannot rank the arms that all reach it (saturation). Re-freezing at a harder threshold and a shorter deadline bought one round of resolution and then tied again: the top three arms came in at 0.867, 0.847 and 0.793, pairwise inseparable on 150 seeds, while their median crossings differed by about seventy steps. The measure was throwing away exactly the information that separated them.

That is what moved the campaign from a deadline to the distribution. From v3 onward the objective is the 90th percentile of the per-seed crossing time itself, estimated from the Kaplan-Meier curve so that seeds still uncrossed at the horizon shape the answer without being scored as crossings (tail quantile q0.9). The 90th percentile rather than the median is the point: it asks when nearly every seed is finished, so a schedule that strands a tenth of its cohort cannot win by being quick in the rest.

Two rounds of search then ran against that objective, and neither ranked its own candidates on it. The v4 ladder promoted on the median with a censoring guard in front, and the v5 driver promoted on a fitted quantile. Both substitutions were made for the same documented reason, that the reported estimator is unusable at the depths a search actually measures, and the second one is what cost that round its result.

A measure is only defined relative to how a crossing is detected, and that changed once, between v3 and v4, which is why no number crosses that boundary. Under the old rule a crossing was confirmed by waiting: the accuracy had to sit above the threshold at two consecutive waypoints. Waiting cannot tell a lucky probe from a model that has not moved, and it charges training steps to answer either question, so it biased every crossing time late. The confirmation rule now re-measures instead, holding the weights that produced the reading fixed and scoring them against three fresh sets of minimal pairs.

Removing that late bias moved the whole field earlier, by more than most schedule differences: the best constant α went from 366.9 to 300.0 and the previous champion from 282.6 to 225.0. The horizon fell from 600 steps to 400 in the same change, which can only push a measured quantile later or into censoring, so the direction of the shift is the confirmation rule and not the shorter horizon. The waypoint grid also tightened from every 25 steps to every 10 from step 100, and runs now stop as soon as a crossing is confirmed, which buys most of the campaign's throughput and means there is no post-crossing accuracy to read.

The rule for the reader is mechanical. Numbers measured under the waiting rule and numbers measured under re-measurement are different quantities, the leaderboard labels every row with the rule it was scored under, and the two are never merged.

The median crossing time is the cheapest usable summary and the reason the v4 ladder promoted on it. At 20 seeds it has a spread of a few steps, where the reported tail quantile has a standard deviation near 35 and is often not defined at all, so the median can order a board that the 90th percentile leaves mostly blank.

What the median cannot see is a stranded tail. A schedule that abandons a fifth of its seeds looks fast in the seeds that remain: constant α at 2.2 posts the fastest median on the v4 board while censoring 0.20 of its cohort, with nothing in the median to say so. The board answers that with a guard rather than a better statistic, refusing to rank any candidate censoring above 0.10, and the guard runs before the median.

Even among arms that finish, the two orderings come apart. Two of the v4 finalists reached identical medians of 187.4 steps over the same 60 seeds, and their reported 90th percentiles were 198.9 and 286.4. One of them is the fastest schedule the campaign has found and the other is nowhere, and the median cannot tell them apart. A tail objective exists to separate exactly that pair.

The reported tail quantile is an order statistic. With no censoring it is the 135th crossing of 150 seeds, the 54th of 60, the 18th of 20, and the slowest of six; if one of six seeds fails to cross there is no reading at all. That is why it cannot rank a search: across the whole board, 290 of 930 candidates have no reported 90th percentile at their deepest cohort, and 256 of the 640 that never left the first rung have none. The fitted tail quantile exists to fill that gap, extrapolating a tail from the shape of the crossings that did happen.

The two are estimates of the same quantity and they do not agree. Over the 27 candidates measured at the full 150 seeds, where both are defined and both are read on identical data, the fitted value runs a mean 8.8 steps above the measured one, with a spread from −0.3 to 20.7, and the size of the gap tracks the fitted dispersion at a correlation of 0.82. That is the model doing what it was asked to do: a lognormal reads a heavier upper tail as a larger scale parameter and extrapolates further past the data, so the schedules penalized most are the ones whose slowest seeds are furthest out. The two rank those same 27 candidates at a rank correlation of 0.19.

Depth compounds it, in the same direction for both estimators. Of the 26 candidates measured at both 60 and 150 seeds, the median crossing time does not move at all, while the measured 90th percentile rises by a median of 3.0 steps and the fitted one by 9.9; the campaign log records a mean fitted dispersion of 0.209 at six seeds against 0.111 at 150. A shallow cohort is optimistic about the tail, and a search that promotes on shallow readings selects the luckiest shrinkage as readily as the fastest schedule, which is the winner's curse operating on the estimator rather than on the schedule.

A fresh cohort settles which of the two carries information about the future. Across the eleven arms taken to held-out confirmation, the measured quantile reproduced itself to a mean of 2.4 steps while the fitted value sat 6.0 away, and the search ordering predicted the held-out ordering at a rank correlation of 0.86 when both sides were measured against −0.34 when the search side was fitted. The cost was the round: on the fitted ranking 13 of 27 candidates beat the standing incumbent and on the measured one 3 did, the incumbent placed 14th on the quantity the search optimized and 4th on the quantity the campaign publishes, and the candidate with the best measured tail on the entire board placed 19th on the fitted ranking and was never sent to confirmation. The schedule the search did nominate came in 14.1 steps behind the incumbent on held-out seeds.

Rank on the quantity you report. Where that is impossible, and at six seeds it genuinely is, the substitute has to be calibrated against the reported measure at the depths where both are defined, and the ranking has to return to the reported one before anything is selected. A substitution that is never checked is not a shortcut but a different objective.

Two supporting habits do the rest of the work. Every arm in a comparison runs on one shared seed base, so differences are read per seed rather than between two marginal distributions (paired seeds); paired intervals are much tighter than unpaired ones and are what the campaign's claims are adjudicated on. And a time is never reported without the censoring beside it, because the fraction of seeds that never arrived is the part of the distribution every summary statistic is tempted to drop.

Each term links to the others it uses.