imagining syntax

Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

vocab_holdout25

20260614_215604_vocab_holdout25 · complete · published 2026-07-20 · seed 42

part of investigation scaling → vocabulary-scale

Intent

These notes and the manifest were backfilled on 2026-07-05 from runs that predate the manifest format, so both are retroactive. The six sweeps ran as chain scripts between 2026-06-14 and 2026-07-04 and are archived under runs/_archive/. Four arms (vocabulary 20, 40, 80, 160) were extended from five to ten seeds on 2026-06-15 and the other two (vocabulary 60, 120) on 2026-07-05, with seeds continuing from base 42 throughout, so every arm now carries ten. The original commands used the pre-rebase CLI, and the arguments in experiment.yaml are their current-grammar equivalents.

Intent: Does the paper's result depend on vocabulary size and holdout \@{vocabholdout-intent}

This experiment asks whether the paper's agreement result depends on the particular vocabulary size and holdout it happened to use. It runs a full Zipfian sweep (Z from 0 to 3.0 in steps of 0.2, ten seeds per point) at each of six vocabulary sizes from 20 to 160 nouns and verbs, and at every size it holds the unseen fraction of noun-verb pairings fixed at a quarter, so the holdout count rises with the vocabulary (5, 10, 15, 20, 30, 40). The model is held at the paper's architecture, 256 wide with two layers and four heads, so the only things that change across arms are the vocabulary and its matching holdout.

Background

Background: More vocabulary means more pairing evidence \@{vocabholdout-background}

Holding the holdout fraction fixed means a larger vocabulary gives the model more seen pairings to induce the agreement rule from.

Hypothesis

Hypothesis: More evidence should raise the peak and shift it left \@{vocabholdout-hypothesis}

If evidence is what limits generalization, then peak unseen_mismatch should rise with vocabulary size, or at least not fall, and the peak should arrive at a lower concentration as vocabulary grows, so its Z should move left. If instead generalization degrades at the large vocabularies, that points to the fixed model capacity, 256 wide with two layers and four heads, as the binding constraint rather than the amount of evidence.

Setup

model
GPT (nanoGPT-derived), weight-tied embeddings — 2 layers × 4 heads × 256 dim (~1,635,584 params) · block 50 · dropout 0.1 · vocab 167 tokens
training
AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 10 seeds (base 42)
training data
PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code
commit cab1539

data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout cab1539
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS instrument_consistent

Results — unseen_mismatch (mean ± sd over 10 seeds)

Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
armpeak over α (end of training)per-seedflags
v020_h05 81.0 ±7.9 @ α=2.4
v040_h10 88.0 ±6.0 @ α=1.4
v060_h15 89.5 ±7.2 @ α=1.6
v080_h20 85.4 ±12.7 @ α=1.8 ⚠ seed_split: range 49.6-95.1 (1 low / 7 high of 10)
v120_h30 85.5 ±15.9 @ α=1.6 ⚠ seed_split: range 50.5-99.2 (2 low / 6 high of 10)
v160_h40 81.5 ±12.5 @ α=1.6 ⚠ seed_split: range 51.9-97.8 (1 low / 3 high of 10)

Conclusions

Conclusion: The peak survives every vocabulary; capacity caps its height \@{vocabholdout-conclusion}

The intermediate peak in unseen_mismatch that the paper reports appears at every vocabulary size and holdout tested here, so it is not an artifact of the default vocabulary of 40 or its holdout of 10. What does not carry over is the hypothesis that a larger vocabulary keeps raising that peak. The peak grows only up to vocabulary 60 and then falls back, and the largest vocabularies degrade in the way the hypothesis attributed to fixed model capacity.

Result: The intermediate peak is not a vocabulary artifact \@{vocabholdout-peak-universal}

Every arm traces the same shape (the per-arm curves are in images/comparison_unseen_mismatch.png). At the uniform end of the sweep, α = 0, mean unseen_mismatch sits below the 50 chance line for all six vocabularies, which is the model agreeing with the nearest noun rather than with the subject. Accuracy then climbs to a peak at an intermediate α and drifts back down across the concentrated tail. The arm that matches the paper's own configuration, vocabulary 40 with a holdout of 10, peaks at 88.0 at α = 1.4, so the shape the paper describes is not specific to that one choice of vocabulary or held-out fraction.

Referenced by (4 direct)

Direct references:

Result: Peak height rises to vocabulary 60 then falls at fixed capacity \@{vocabholdout-capacity-ceiling}

The height of the peak rises and then falls as vocabulary grows. Mean peak unseen_mismatch runs 81.0 at vocabulary 20, 88.0 at 40, and 89.5 at 60, then declines through 85.4 at 80, 85.5 at 120, and 81.5 at 160 (images/peak_vs_vocab-size.png). More seen pairings help up to about vocabulary 60, after which the peak gives the gains back, so the prediction that it should rise or at least hold describes only the first half of the range.

Referenced by (6 direct, 1 transitive)

Direct references:

Transitive (depth 1):

The peak also does not keep moving toward lower concentration. The smallest vocabulary needs the most concentration to reach its peak, at α = 2.4, but every vocabulary from 40 upward peaks in a narrow band between α = 1.4 and 1.8. The leftward shift the hypothesis predicted therefore appears only once, between the smallest vocabulary and the rest, and not as a continuing trend. Past roughly 40 nouns and verbs, extra vocabulary buys the model no further tolerance for flatter pairing distributions.

Conclusion: Capacity binding, not evidence shortage \@{vocabholdout-capacity-signature}

The decline at the large vocabularies has the signature of a capacity limit rather than a shortage of evidence, and the clearest sign is in the conditions that are otherwise easy. From vocabulary 20 through 80, the other three conditions sit at their ceiling: seen_match at 100.0, seen_mismatch just above 99, and unseen_match at 100.0. At vocabulary 120 and 160 even these fall, seen_match to 98.2 then 96.5, seen_mismatch to 97.7 then 96.5, and unseen_match to 89.8 then 85.2 (images/comparison_by_condition.png). When the easiest condition, memory of a trained pairing with attractors that help rather than interfere, can no longer reach 100, the model has run short of room to fit the material it was given, which is what the hypothesis meant by capacity binding.

Referenced by (3 direct, 1 transitive)

Direct references:

Transitive (depth 1):

Result: Seed bimodality deepens as vocabulary grows \@{vocabholdout-bimodality-deepens}

The spread across seeds widens the same way. Through vocabulary 60 the ten replicates of each peak hold together, but from vocabulary 80 upward they split into seeds that learn agreement and seeds that do not. At vocabulary 80 the peak's seeds range from 49.6 to 95.1, with seven of the ten high and one collapsed near chance. At 120 they range from 50.5 to 99.2, with six high and two near chance, at 50.5 and 59.9. At 160 the split is widest: only three seeds are clearly high, one sits at 51.9, and the rest are scattered between. The seeds near 50 have settled on the nearest-noun shortcut, and unseen_match splits the same way at 120 and 160. Because the mean then sits in the gap between the two groups and describes no single run, this is seed bimodality rather than ordinary spread, and it deepens as vocabulary grows, in step with the capacity reading.

Referenced by (6 direct, 2 transitive)

Direct references:

Transitive (depth 1):

Conclusion: The embedding capacity ceiling \@{vocabholdout-embedding-ceiling}

Set against the model-size study, the two experiments meet in the middle from opposite directions. The size study shrank the model at a fixed vocabulary; this one holds the model fixed and grows the vocabulary, so the same 256-dimensional tied embedding has to span an ever larger token inventory. Both locate the ceiling at the embedding's capacity relative to the number of tokens it must represent, not at the amount of pairing evidence available. On this reading the large-vocabulary decline should ease if the model is made wider, which the follow-ups below would test.

Referenced by (2 direct, 1 transitive)

Direct references:

Transitive (depth 1):

Follow-ups this suggests:

  • A width sweep at the largest vocabulary, for example a 512-wide model at vocabulary 160, to see whether the decline reverses when capacity is added.
  • A comparison against the matching runs at a one-third holdout (v60_h20 and v120_h40, currently unpublished at the runs root), to separate the vocabulary axis from the holdout-fraction axis.
Referenced by (2 direct)

Direct references:

Comparison figures

comparison_by_condition.png
comparison_by_condition.png
comparison_unseen_mismatch.png
comparison_unseen_mismatch.png
peak_vs_vocab-size.png
peak_vs_vocab-size.png

Children

Childvocab-size peak unseen_mismatchStatus
v020_h05 20 81.0 done
v040_h10 40 88.0 done
v060_h15 60 89.5 done
v080_h20 80 85.4 done
v120_h30 120 85.5 done
v160_h40 160 81.5 done

experiment.yaml