Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

zipf_v60_h20

20260614_215604_zipf_v60_h20 · complete · published 2026-07-20 · seed 42

part of investigation scaling → vocabulary-scale

Intent

These notes and the manifest were backfilled on 2026-07-20 from a run that predates the manifest format. The sweep ran on 2026-06-14 as a chain script alongside the vocabulary-scaling series, with seeds from base 42, and is adopted here as the single child sweep; the arguments in experiment.yaml are the current-grammar equivalents of the original command.

Intent: Does the peak survive a one-third holdout at vocabulary 60 \@{zipfv60-intent} This experiment asks whether the paper's agreement result survives when a third of the noun-verb pairings are held out rather than a quarter. It runs one full Zipfian sweep (Z from 0 to 3.0 in steps of 0.2, five seeds per point) at vocabulary 60 with a holdout of 20, so two thirds of the pairings remain as seen evidence, at the paper's model architecture, 256 wide with two layers and four heads, trained for 1200 steps. Referenced by (1 direct)Direct references:Capacity is already binding at vocabulary 120

Background

Background: The scaling series varies vocabulary at a fixed quarter \@{zipfv60-background} The vocabulary-scaling series (vocab_holdout25) varies vocabulary from 20 to 160 while holding the held-out fraction at a quarter, and finds the paper's intermediate peak at every size (peak universal). Holding the fraction fixed ties two axes together: a bigger vocabulary always brings proportionally more seen evidence. This run and its companion at vocabulary 120 (zipf_v120_h40) untie them by raising the held-out fraction to a third at two of the series' vocabularies, the comparison the series' own conclusions name as a follow-up.

Hypothesis

Hypothesis: A thinner seen set should cost height or concentration \@{zipfv60-hypothesis} If the amount of seen pairing evidence is what limits generalization, then holding out a third of the pairings instead of a quarter should lower the peak in unseen_mismatch, or push it toward more concentrated distributions where the surviving pairings repeat often enough to compensate. If the peak is instead a property of the pairing distribution alone, it should appear at the same α with about the same height.

Setup

model: GPT (nanoGPT-derived), weight-tied embeddings · block 50 · dropout 0.1 · vocab 247 tokens
training: AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 5 seeds (base 42)
training data: PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 20
code: commit cab1539

data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout cab1539
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS instrument_consistent

Results — unseen_mismatch (mean ± sd over 5 seeds)

Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition

Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)

arm	peak over α (end of training)	per-seed	flags
sweep	91.5 ±4.7 @ α=1.4

Conclusions

Conclusion: At vocabulary 60 the holdout fraction changes nothing \@{zipfv60-conclusion} The sweep reproduces the quarter-holdout picture at this vocabulary essentially unchanged, so at a size the model fits comfortably, how much of the pairing space is held out is not an operative variable. Result: The peak holds its location and height at a third holdout \@{zipfv60-peak-holds} At α = 0 mean unseen_mismatch sits at 6.0, the model agreeing with the nearest noun rather than the subject; accuracy climbs through the intermediate range to a peak of 91.5 ±4.7 at α = 1.4 and falls back across the concentrated tail to 64.2 at α = 3.0. The peak sits in the same place and at the same height as the quarter-holdout arm at this vocabulary, which reaches 89.5 in the series' 1.4 to 1.8 band (the capacity ceiling), so raising the holdout to a third moves the peak by less than its spread across seeds. Referenced by (1 direct, 1 transitive)Direct references:The peak is universal, capacity caps it, and the holdout fraction is second orderTransitive (depth 1):A robust peak whose location is set by the budget One grid point breaks the otherwise smooth tail: at α = 2.0 the five seeds span 20.0 to 80.0 around a mean of 63.0, a seed bimodality split of the kind the quarter-holdout series only sees from vocabulary 80 upward (deepening splits), which suggests the thinner seen set moves the cohort closer to the split regime even at a vocabulary that otherwise shows no strain. Five seeds per point, against the series' ten, so reading spread here leans on the wider series.

Children

Child	peak unseen_mismatch	Status
sweep	91.5	done

experiment.yaml