Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
scaling → vocabulary-scale
vocabulary-scale
investigation
Intent
This investigation asks how the paper's agreement result scales along the vocabulary axis. The grammar makes vocabulary a free parameter, and two things change with it at once: how many tokens the fixed model must represent, and how much seen pairing evidence training provides. The member experiments hold the model at the paper's architecture and pull those apart: a six-size sweep from vocabulary 20 to 160 at a fixed quarter holdout, and two matched sweeps at a one-third holdout, at vocabulary 60 and 120, that vary the evidence fraction with the vocabulary held still.
Referenced by (1 direct)
Direct references:
Conclusions
The intermediate peak is not an artifact of the paper's vocabulary: it appears at every size from 20 to 160 (peak universal). Its height is set by model capacity rather than by evidence: the peak rises to vocabulary 60 and falls back (the capacity ceiling), the decline carries the signature of capacity binding, with the otherwise easy conditions off their ceilings (capacity binding), and the same 256-dimensional embedding limit the model-scale work reaches by shrinking the model is reached here by growing the token inventory (the embedding ceiling).
The holdout fraction turned out to be a second-order axis that matters only where capacity is already strained. At vocabulary 60, raising the holdout from a quarter to a third changes nothing: the peak keeps its location and height (the peak holds). At vocabulary 120 the same change costs the peak about three points and pushes it toward more concentrated distributions (drops and moves right), and it drives the cohort's seed bimodality deeper into the grid (the split approach), extending the deepening the quarter series already shows (deepening splits); the seed-level mechanics of those splits belong to the bimodality thread (bimodality).
What would change this reading. The one-third-holdout sweeps carry five seeds against the series' ten, so their spread estimates are coarse. Everything here sits at the paper's 1200-step budget and single architecture, and the series' own named follow-up, a wider model at the largest vocabulary, is the direct test of the capacity reading.
Referenced by (1 direct)
Direct references:
Experiments
-
20260614_215604_zipf_v60_h20 2026-07-20
These notes and the manifest were backfilled on 2026-07-20 from a run that predates the manifest format. The sweep ran on 2026-06-14 as a chain script alongside the vocabulary-scaling series, with seeds from base 42, and is adopted here as the single child
sweep; the arguments inexperiment.yamlare the current-grammar equivalents of the original command.This experiment asks whether the paper's agreement result survives when a third of the noun-verb pairings are held out rather than a quarter. It runs one full Zipfian sweep (Z from 0 to 3.0 in steps of 0.2, five seeds per point) at vocabulary 60 with a holdout of 20, so two thirds of the pairings remain as seen evidence, at the paper's model architecture, 256 wide with two layers and four heads, trained for 1200 steps.
Referenced by (1 direct)
Direct references:
-
20260614_215604_zipf_v120_h40 2026-07-20
These notes and the manifest were backfilled on 2026-07-20 from a run that predates the manifest format. The sweep ran on 2026-06-14 as a chain script alongside the vocabulary-scaling series, with seeds from base 42, and is adopted here as the single child
sweep; the arguments inexperiment.yamlare the current-grammar equivalents of the original command.This experiment asks whether the paper's agreement result survives when a third of the noun-verb pairings are held out rather than a quarter, at vocabulary 120. It runs one full Zipfian sweep (Z from 0 to 3.0 in steps of 0.2, five seeds per point) with a holdout of 40, so two thirds of the pairings remain as seen evidence, at the paper's model architecture, 256 wide with two layers and four heads, trained for 1200 steps.
Referenced by (1 direct)
Direct references:
-
20260614_215604_vocab_holdout25 2026-07-20
These notes and the manifest were backfilled on 2026-07-05 from runs that predate the manifest format, so both are retroactive. The six sweeps ran as chain scripts between 2026-06-14 and 2026-07-04 and are archived under
runs/_archive/. Four arms (vocabulary 20, 40, 80, 160) were extended from five to ten seeds on 2026-06-15 and the other two (vocabulary 60, 120) on 2026-07-05, with seeds continuing from base 42 throughout, so every arm now carries ten. The original commands used the pre-rebase CLI, and the arguments inexperiment.yamlare their current-grammar equivalents.This experiment asks whether the paper's agreement result depends on the particular vocabulary size and holdout it happened to use. It runs a full Zipfian sweep (Z from 0 to 3.0 in steps of 0.2, ten seeds per point) at each of six vocabulary sizes from 20 to 160 nouns and verbs, and at every size it holds the unseen fraction of noun-verb pairings fixed at a quarter, so the holdout count rises with the vocabulary (5, 10, 15, 20, 30, 40). The model is held at the paper's architecture, 256 wide with two layers and four heads, so the only things that change across arms are the vocabulary and its matching holdout.