Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
anaphor-binding
investigation
Intent
This investigation asks whether the statistical conditions that help a model learn subject-verb number agreement also help it learn a dependency the Zipfian concentration never touched. The paper's account is collocational bootstrapping: repetitive noun-verb pairings give the model a reason to represent the subject at all, and an intermediate α generalizes best because it is repetitive enough to build that representation and varied enough to stop the model from memorizing the pairs it saw. If that account is right, what the concentration builds is a representation of a position rather than of a verb's partner list, and some other dependency that needs the subject should be able to use it.
The anaphor task is that test. It holds the sentence space, the α sampling and the held-out pairings fixed, and replaces the number-marked verb with an uninflected past-tense form followed by a clause-final reflexive that has to agree with the subject. The reflexive is never part of the pairing the α distribution is defined over. If the accuracy curve still turns over at an interior α, the concentration is doing positional work that outlives the dependency it was measured on; if the curve is flat, the mechanism is lexical and stops at the verb.
Experiments
-
20260903_122307_anaphor_n60 2026-09-04
This experiment repeats the anaphor sweep with 60 replicates at every α instead of 10, and changes nothing else: the same anaphor task grammar, the same α grid from 0.0 to 3.0 in steps of 0.1, the same 256-wide two-layer four-head model trained for 1200 iterations from seed 42, the same waypoints at 300, 600, 900 and 1200, and a matching oneshot arm at 60 replicates. It is 1,860 models in the Zipfian arm and 60 in the oneshot arm.
The first ten seeds at each α are the ones the earlier run used, because the seed derivation depends only on the α grid and the base seed, both unchanged. That makes the earlier run a strict subset of this one and its ten replicates a free consistency check, and it is also why the pre-registered test below is restricted to the fifty that are new.
The point of the extra replication is one quantity: how often a training run ends up with the binding rule at all. The earlier sweep could not measure that to better than ±0.16, which is wider than any difference worth arguing about. At 60 replicates the standard error is 0.065, and pooling five neighbouring α values brings it to 0.029.
-
20260903_054440_anaphor_control unpublished
-
20260903_042902_anaphor 2026-09-03
This experiment runs the paper's sweep again with one thing changed: what has to agree. The anaphor task keeps the sentence space, the Zipfian sampling over noun-verb pairings, the held-out unseen pairings and the model exactly as the reproduction had them, and moves number off the verb. Verbs are regular past-tense forms that never inflect, and every sentence ends in a reflexive,
himselforthemselves, that must agree with the subject. Each minimal pair flips that reflexive rather than the verb.Every noun in a training sentence carries the subject's number, so the training data never distinguishes binding the subject from binding the first noun or the nearest one; the mismatch conditions are what separate them. The Zipfian arm sweeps α from 0.0 to 3.0 in steps of 0.1 with ten seeds at each value, and the oneshot arm adds ten more at the deterministic 1:1 limit. Both train the 256-wide, two-layer, four-head model for 1200 iterations from seed 42, and both record waypoints at 300, 600, 900 and 1200 steps, so a dependency that turns out to be easy shows its saturation in the trajectory rather than hiding behind a ceiling at the end.
One difference from the reproduction is worth stating before any number is read. A waypoint run reports the weights at a fixed budget, where the reproduction reported the checkpoint with the lowest validation loss; across that run's 320 replicates those two coincided 275 times, and the other 45 stopped at 900 or 600 steps. The α axis is estimated the same way at every grid point here, so the shape of the curve is internally consistent, but a peak location compared against the reproduction's is compared across two estimators.