imagining syntax

Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

paper

20260705_105204_paper · complete · published 2026-07-16 · seed 42

Intent

This is the canonical paper reproduction, run fresh on 2026-07-05 through the packaged bare imsyn entry point. An earlier candidate directory, paper_20260703_141230, turned out to be the fast smoke run from rebase validation (Z in {0, 1.4}, one seed) and was archived without publishing.

Intent: Does Figure 2 reproduce \@{paper-intent}

This experiment reproduces the paper's central result, the two panels of Figure 2. It measures how well subject-verb agreement generalizes to noun-verb pairings the model was never trained on as the Zipfian concentration of the training pairings is varied, and it sets that sweep against the oneshot limit, where each verb takes exactly one subject noun. The Zipfian arm sweeps Z from 0.0 to 3.0 in steps of 0.1 with ten seeds at each value; the oneshot arm adds ten more seeds at the deterministic 1:1 limit. Both train the default 256-wide, two-layer, four-head model for 1200 iterations from seed 42.

Background

Background: The paper's account \@{paper-background}

The paper's reading is that variability in which noun pairs with which verb is what forces the model to learn an abstract agreement rule rather than memorizing the pairings it saw.

Hypothesis

Hypothesis: An intermediate peak and an oneshot deficit \@{paper-hypothesis}

The published finding should reproduce. On the hardest condition, unseen_mismatch, accuracy should peak at an intermediate Z rather than rising all the way as the pairings grow more concentrated, and the oneshot limit should generalize worse than the best Zipfian setting even though its pairings are perfectly consistent.

Setup

model
GPT (nanoGPT-derived), weight-tied embeddings — 2 layers × 4 heads × 256 dim (~1,635,584 params) · block 50 · dropout 0.1 · vocab 167 tokens
training
AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 10 seeds (base 42)
training data
PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code
commit 2898aa9

data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 2898aa9
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS instrument_consistent

Results — unseen_mismatch (mean ± sd over 10 seeds)

Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
armend of trainingper-seedflags
oneshot 62.2 ±4.7
zipfian 93.1 ±9.1 @ α=1.2

Conclusions

Conclusion: The published result reproduces and separates memory from rule \@{paper-conclusion}

The published result reproduces. On the hardest condition, unseen_mismatch, mean accuracy peaks at an intermediate Zipfian concentration and then falls away toward the concentrated end, and the oneshot limit, where each verb takes exactly one subject noun, generalizes worse than that peak. The composed two-panel figure is images/figure_2.png.

Result: The intermediate-Z generalization peak \@{paper-intermediate-peak}

Across the sweep, mean unseen_mismatch traces the intermediate peak the paper describes. At the uniform end it sits near or below the 50 chance line, 22.7 at Z = 0.0 and as low as 10.9 at Z = 0.6. Accuracy below chance there means the model is agreeing with the nearest noun instead of the subject, which the matched conditions confirm: unseen_match and seen_match hold at 100 wherever the attractors and the subject want the same verb. Accuracy then rises steeply through Z = 1.0, 1.1, and 1.2 (64.4, 80.3, 93.1), peaks at 93.1 ±9.1 at Z = 1.2, and drifts back down across the concentrated tail to 74.0 at Z = 3.0. The climb is not monotonic in concentration; the best generalization is in the middle of the range.

Referenced by (9 direct, 10 transitive)

Direct references:

Transitive (depth 1):

Transitive (depth 2):

The peak carries real spread across replicates. Its ten seeds run from 69.8 to 100.0, half of them at 98 or above, so the 93.1 mean sits below the seeds that fully solved the task and is held down by the few that did not.

Result: Oneshot consistency memorizes without transferring \@{paper-oneshot-memorizes}

The oneshot limit shows what perfect consistency buys and what it costs. Both seen conditions are exactly right, seen_match and seen_mismatch at 100.0 ±0.0, so the model handles the attractor trap on the pairings it trained on. On held-out pairings, though, it reaches only 62.2 ±4.7 on unseen_mismatch and 66.9 ±2.7 on unseen_match. That is memory of consistent pairings rather than a rule that transfers, and 62.2 sits below the Zipfian peak of 93.1 and below the mean at every swept Z from 1.0 upward. The most concentrated regime is not where agreement generalizes best.

Referenced by (2 direct)

Direct references:

Conclusion: Memory versus rule \@{paper-memory-vs-rule}

Put side by side, the two panels separate memory from rule. As the pairings concentrate the model memorizes the trained material better: seen_mismatch improves from 22.0 at Z = 0.0 to 99.6 at Z = 3.0 and 100.0 at oneshot, while seen_match stays near 100 throughout. Generalization to unseen pairings runs the other way, weakening through the tail: unseen_match falls from 100 at low Z to about 82.6 at Z = 3.0 and 66.9 at oneshot, and unseen_mismatch turns over at Z = 1.2. The intermediate setting is where the training pairings vary enough to rule out memorization yet stay regular enough to learn from, which is the paper's account of why pairing variability forces an abstract agreement rule.

Referenced by (1 direct)

Direct references:

Two things bear on how firmly to read the peak's location. Its exact position in Z is partly a property of the 1200-step snapshot rather than a fixed feature of the grammar: the companion long-iteration and peak-timing runs (paper_long_iter, peak_umm_iter) show higher-Z settings peaking earlier and then drifting, so Figure 2's shape is best read together with those. The peak itself reproduces independently, appearing at the same 93.1 at Z = 1.2 in the msize d = 256 run on this grid under the same seed policy.

Comparison figures

figure_2.png
figure_2.png

Children

Child peak unseen_mismatchStatus
zipfian 93.1 done
oneshot 62.2 done

experiment.yaml