imagining syntax

Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

zipf_v120_h40

20260614_215604_zipf_v120_h40 · complete · published 2026-07-20 · seed 42

part of investigation scaling → vocabulary-scale

Intent

These notes and the manifest were backfilled on 2026-07-20 from a run that predates the manifest format. The sweep ran on 2026-06-14 as a chain script alongside the vocabulary-scaling series, with seeds from base 42, and is adopted here as the single child sweep; the arguments in experiment.yaml are the current-grammar equivalents of the original command.

Intent: Does a one-third holdout change the picture at vocabulary 120 \@{zipfv120-intent}

This experiment asks whether the paper's agreement result survives when a third of the noun-verb pairings are held out rather than a quarter, at vocabulary 120. It runs one full Zipfian sweep (Z from 0 to 3.0 in steps of 0.2, five seeds per point) with a holdout of 40, so two thirds of the pairings remain as seen evidence, at the paper's model architecture, 256 wide with two layers and four heads, trained for 1200 steps.

Referenced by (1 direct)

Direct references:

Background

Background: Capacity is already binding at vocabulary 120 \@{zipfv120-background}

The vocabulary-scaling series (vocab_holdout25) finds vocabulary 120 past the point where the fixed 256-wide model stops gaining from more evidence: the peak has fallen back from its vocabulary-60 high (the capacity ceiling), the easy conditions are off their ceilings (capacity binding), and the seeds have begun to split (deepening splits). This run raises the held-out fraction from a quarter to a third at that vocabulary, with its companion at 60 (zipf_v60_h20) providing the unstrained comparison.

Hypothesis

Hypothesis: Thinner evidence should cost more where capacity binds \@{zipfv120-hypothesis}

If the amount of seen pairing evidence is what limits generalization, then holding out a third of the pairings instead of a quarter should lower the peak in unseen_mismatch, or push it toward more concentrated distributions where the surviving pairings repeat often enough to compensate, and the cost should be larger at a vocabulary the model already struggles to fit than at one it fits comfortably.

Setup

model
GPT (nanoGPT-derived), weight-tied embeddings · block 50 · dropout 0.1 · vocab 487 tokens
training
AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 5 seeds (base 42)
training data
PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 40
code
commit cab1539

data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout cab1539
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS instrument_consistent

Results — unseen_mismatch (mean ± sd over 5 seeds)

Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
armpeak over α (end of training)per-seedflags
sweep 82.3 ±3.3 @ α=2.0

Conclusions

Conclusion: Where capacity binds, thinner evidence costs height and needs concentration \@{zipfv120-conclusion}

Unlike at vocabulary 60, where the same change did nothing, raising the holdout to a third at vocabulary 120 measurably worsens generalization, in both of the ways the hypothesis allowed: the peak is lower and it needs a more concentrated distribution to appear.

Result: The peak drops and moves right at a one-third holdout \@{zipfv120-peak-shifts-right}

Peak unseen_mismatch reaches 82.3 ±3.3 at α = 2.0, against 85.5 inside the 1.4 to 1.8 band for the quarter-holdout arm at the same vocabulary (the capacity ceiling). The thinner seen set costs the peak about three points and pushes its location toward more concentrated distributions, where the surviving pairings repeat often enough to compensate.

Referenced by (1 direct, 1 transitive)

Direct references:

Transitive (depth 1):

Result: The approach to the peak is split by seed \@{zipfv120-split-approach}

Across the approach the cohort is bimodal rather than noisy: at α = 1.4 the five seeds span 16.0 to 88.0, at 1.6 they span 47.5 to 95.9, and at 1.8 they span 48.8 to 94.9, with means of 60.3, 74.0, and 72.9 that no individual run sits near. Only at α = 2.0 does the cohort hold together. This is the seed bimodality the scaling series sees deepen with vocabulary (deepening splits), pushed further into the grid by the thinner evidence.

Referenced by (1 direct, 1 transitive)

Direct references:

Transitive (depth 1):

The capacity strain the quarter-holdout series diagnoses at this vocabulary (capacity binding) is visible here too: unseen_match is off its ceiling across much of the grid, falling as low as a 59.4 mean at α = 1.0 against 100.0 at vocabulary 60. Read with the companion run, the holdout fraction only matters where model capacity is already strained. Five seeds per point, against the series' ten, so the spread readings here are coarse.

Children

Child peak unseen_mismatchStatus
sweep 82.3 done

experiment.yaml