Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
anaphor
20260903_042902_anaphor · complete · published 2026-09-03 · seed 42
part of investigation anaphor-binding
Intent
This experiment runs the paper's sweep again with one thing changed:
what has to agree. The anaphor task keeps the sentence space, the
Zipfian sampling over noun-verb pairings, the held-out
unseen pairings and the model exactly as the reproduction had
them, and moves number off the verb. Verbs are regular past-tense forms
that never inflect, and every sentence ends in a reflexive,
himself or themselves, that must agree with the subject.
Each minimal pair flips that reflexive rather than the verb.
Every noun in a training sentence carries the subject's number, so the training data never distinguishes binding the subject from binding the first noun or the nearest one; the mismatch conditions are what separate them. The Zipfian arm sweeps α from 0.0 to 3.0 in steps of 0.1 with ten seeds at each value, and the oneshot arm adds ten more at the deterministic 1:1 limit. Both train the 256-wide, two-layer, four-head model for 1200 iterations from seed 42, and both record waypoints at 300, 600, 900 and 1200 steps, so a dependency that turns out to be easy shows its saturation in the trajectory rather than hiding behind a ceiling at the end.
One difference from the reproduction is worth stating before any number is read. A waypoint run reports the weights at a fixed budget, where the reproduction reported the checkpoint with the lowest validation loss; across that run's 320 replicates those two coincided 275 times, and the other 45 stopped at 900 or 600 steps. The α axis is estimated the same way at every grid point here, so the shape of the curve is internally consistent, but a peak location compared against the reproduction's is compared across two estimators.
Background
The reproduction found the paper's figure intact: generalization to unseen pairings on the hardest condition peaks at an intermediate α rather than climbing as the pairings grow more repetitive (the intermediate peak), and the oneshot limit, whose pairings are perfectly consistent, generalizes worse than the best Zipfian setting (oneshot memorizes). The reading offered for that shape is collocational bootstrapping: repetition gives the model a reason to represent the subject, and variation stops it from solving the task by memorizing pairs.
What the reproduction cannot say is whether the thing repetition builds is positional or lexical. Every measurement in it was taken on the same verb whose partner distribution α controls, so a subject representation and a well-learned partner list predict the same curve. Separating them needs a dependency that requires the subject and never appears in the α distribution.
Hypothesis
The registered expectation is that unseen_mismatch accuracy peaks at an interior α well above 1.4: locating a subject with no agreement task sharpening the representation should need more concentration per verb than the paper's setting supplied, so the optimum moves up rather than landing on 1.4 again. seen_match should sit near ceiling at every α; if it does not, nothing else in the run is interpretable.
Two other outcomes are worth naming in advance because they say different things. A flat unseen_mismatch curve with seen_match at ceiling is a clean negative: concentrating noun-verb pairings does not build a subject representation the reflexive can reach, and the mechanism is lexical. A curve that rises monotonically to α = 3.0 would say the high-concentration failure mode does not bite here, which would in turn mean the seen and unseen pairings axis is carrying no signal at the tested slot and the run has two live conditions rather than four.
Setup
data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout b8c749e03202
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yamlChecks
PASS children_complete PASS replicate_coverage PASS instrument_consistent
Results — unseen_mismatch (mean ± sd over 10 seeds)
Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its reflexive-flipped twin (himself ↔ themselves), the reading that binds an attractor instead of the subject. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between the subject and the clause-final reflexive (the BIND-RECENT trap)
| arm | end of training (t = 1200) | per-seed | flags |
|---|---|---|---|
| oneshot | 62.6 ±13.1 | ⚠ seed_split: range 46.2-90.9 (4 low / 1 high of 10) | |
| zipfian | 85.7 ±16.9 @ α=1.6 | ⚠ seed_split: range 46.4-100.0 (1 low / 5 high of 10) |
Conclusions
The registered expectation was an interior optimum well above 1.4. The sweep does not show one. Between α of 0.1 and 2.3 about half of seeds finish training having learned to bind the reflexive to the subject, and that fraction does not move with α. Above 2.3 it collapses. On this dependency the Zipfian concentration that the paper's account holds responsible for building a subject representation does very little until it starts doing harm.
Counting a replicate as having learned the rule when its unseen_mismatch accuracy exceeds 90, the rate is 118 of 230 across α from 0.1 to 2.3 and 6 of 60 from 2.5 to 3.0 (Fisher exact p ≈ 10⁻⁹, and the same separation holds at thresholds of 80 and 95). Inside the plateau there is no structure the data can resolve: the best single α is 1.2 at 8 of 10, against 110 of 220 for the rest of the plateau, which is p = 0.103. The mean curve's maximum sits elsewhere again, at α = 1.6 with 85.7 ±16.9, and its standard deviations run from 17 to 41 at every α. Ten seeds per grid point cannot tell these apart, so the peak location is not a finding here; the edge of the plateau is.
Referenced by (1 direct)
Direct references:
The cliff is not a failure to learn. At 300 steps the two bands are indistinguishable, 0.50 against 0.50 with p = 1.000, and the gap opens only afterwards, the high-α band running 0.50, 0.28, 0.22, 0.10 across the four waypoints while the plateau holds near 0.55. Models at high α find the binding rule as reliably as any others and then lose it, which is erosion rather than an inability to acquire the rule, and matches what the paper reproduction found on the verb. The uniform end does the same thing later and less sharply, running 0.40, 0.50, 0.60, 0.20.
Of the 310 α-by-seed cells, 129 finish above 90 and 38 finish below 40, leaving 143 in between; the seed either has the rule at the end of training or it does not. The mean at a given α is therefore closer to a mixture weight than to a level of performance, and reading it as an accuracy overstates how much the middle of the range differs from the ends. The estimator-matched agreement arm is not like this: its cells are 40 above 90 and 196 in the middle band, a spread rather than two basins.
Referenced by (1 direct)
Direct references:
Two other things are settled and worth stating. seen_match and unseen_match are at 100.0 at every one of the 31 α values, so nothing here is an artifact of the model collapsing onto a prior over the two reflexives. And the seen and unseen axis carries no signal: at no α do seen_mismatch and unseen_mismatch differ by more than 1.82 points. That is the outcome the pre-registration named for the monotone case, and it arrived without the monotone curve. This experiment has two live conditions, not four, and later work on this task should say so rather than reporting four.
Set beside the estimator-matched control, the two dependencies fail in different places. Below α ≈ 1.0 the agreement task is far below chance, 22.7 at uniform and 11.7 at 0.6, which is a recency solution applied systematically. The anaphor task at the same α is near or above chance and already solved by half its seeds. Concentration is what rescues the verb from recency, and the reflexive never needed rescuing. Above 2.3 the order reverses: the agreement arm holds between 72 and 78 while the anaphor arm decays to between 50 and 55.
The reading this suggests, which the run supports but does not establish, is that the two tasks are limited by different things. Binding a clause-final reflexive to the subject does not require the model to have distinguished the subject from the attractors lexically, and so does not benefit from the repetition that does that work; what it needs is for the representation not to be overwritten later, which is what concentrating the pairings past 2.3 appears to cost. On that reading collocational bootstrapping is doing lexical work rather than positional work, and the oneshot result is consistent with it: at 62.6 ±13.1 the deterministic limit sits below every α from 0.2 to 2.3.
What would change this reading. The plateau is underpowered, and ten seeds cannot separate a genuine interior optimum of the size the hypothesis expected from a flat stretch; a narrower α range at 40 or more seeds would settle whether 1.0 to 1.2 is really elevated. The all-or-nothing structure means the right measurement on this task is probably the fraction of seeds that acquire the rule rather than the mean accuracy, and no claim here was pre-registered in those terms. The erosion account rests on four waypoints; a denser grid through the second half of training would show whether the high-α loss is gradual or sudden. Everything above is measured at a fixed 1200-step budget, and the moving peak on the verb task is a reason to expect the cliff's location to move with the budget rather than to sit at 2.3 for its own sake.
Comparison figures
Children
| Child | peak unseen_mismatch | Status |
|---|---|---|
| zipfian | 97.7 | done |
| oneshot | 68.0 | done |