vocab_holdout25

20260614_215604_vocab_holdout25 · complete · published 2026-07-05 · seed 42

Intent

How does vocabulary size affect the Z-sweep agreement curve when the held-out (unseen) fraction of noun-verb pairs is fixed at 25%? Six Zipf sweeps at vocab sizes 20-160, each holding out a quarter of the pairs.

Hypothesis

With the holdout fraction fixed, larger vocabularies give the model more seen pairs to induce the agreement rule from, so peak unseen_mismatch should rise (or at least not fall) with vocab size, and the peak should need less distributional concentration (peak Z moving left). If instead the curve degrades at large vocab, the fixed model capacity (256/2/4) is the binding constraint, not the evidence.

Conclusions

Peak mean unseen_mismatch per child (10 seeds, n=10 everywhere after the extends):

| vocab | holdout | peak UMM | at Z | |------:|--------:|---------:|-----:| | 20 | 5 | 81.0% | 2.4 | | 40 | 10 | 88.0% | 1.4 | | 60 | 15 | 89.5% | 1.6 | | 80 | 20 | 85.4% | 1.8 | | 120 | 30 | 85.5% | 1.6 | | 160 | 40 | 81.5% | 1.6 |

Findings:

1. The hypothesis holds only up to vocab ≈ 60, then reverses. Peak UMM rises 81.0 → 89.5% from vocab 20 to 60, then declines steadily to 81.5% at 160 (peak_vs_vocab-size.png). More evidence helps until something else binds.
2. The leftward-peak prediction holds at the small end only: vocab 20 needs heavy concentration (peak at Z=2.4) while every vocab ≥ 40 peaks in a flat Z≈1.4–1.8 band — beyond ~40 nouns, extra vocabulary no longer buys tolerance of flatter distributions.
3. The large-vocab decline is consistent with a capacity bind. At vocab 160 the token inventory is 647 against a 256-dim tied embedding; the same architecture that peaks at 93% on the paper's vocab (167 tokens, see msize d=256) manages only 81.5% here. This mirrors the msize result from the other direction: msize shrank the model under fixed vocab, this holds the model and grows the vocab — both point to embedding capacity relative to token count, not pair evidence, as the ceiling.

Follow-up candidates: extend the msize × vocab grid (e.g. d=512 at vocab 160) to test whether the large-vocab decline undoes at higher width; compare against the 33%-holdout pair (v60_h20, v120_h40, unpublished at runs/ root) for a holdout-fraction axis.

Comparison figures

Children

experiment.yaml