Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

scaling

investigation

Intent

Intent: How the result scales \@{scaling-intent} This investigation gathers the two axes along which the paper's number-agreement result is scaled: the model itself, and the vocabulary the grammar draws from. Its member investigations ask the same question from opposite sides of a fixed training budget: model-scale varies embedding width and depth against the default vocabulary, while vocabulary-scale holds the paper's architecture still and grows the token inventory. The two meet at capacity: the same 256-dimensional embedding ceiling is reached from one side by shrinking the model (the capacity ladder) and from the other by growing the vocabulary (the embedding ceiling), which is why the axes belong to one investigation rather than two. The open joint probe both members name, a wider grid over width × vocabulary, would land here.

Sub-investigations

model-scale 4 experiments
$Intent:$ $How model size gates agreement generalization$ \@{model-scale-intent}

This investigation asks how a model's size gates its ability to learn subject-verb number agreement well enough to generalize to unseen pairings. Size has two axes here, embedding width (n_embd) and depth (n_layer), and the member experiments vary each against the full α sweep: first the width ladder, then depth, then longer-budget continuations of both to separate what is a property of the architecture from what is a sign of undertraining at the paper's budget.
Referenced by (1 direct)
Direct references:
How the result scales
vocabulary-scale 3 experiments
$Intent:$ $How the result scales with vocabulary size and holdout$ \@{vocabulary-scale-intent}

This investigation asks how the paper's agreement result scales along the vocabulary axis. The grammar makes vocabulary a free parameter, and two things change with it at once: how many tokens the fixed model must represent, and how much seen pairing evidence training provides. The member experiments hold the model at the paper's architecture and pull those apart: a six-size sweep from vocabulary 20 to 160 at a fixed quarter holdout, and two matched sweeps at a one-third holdout, at vocabulary 60 and 120, that vary the evidence fraction with the vocabulary held still.
Referenced by (1 direct)
Direct references:
How the result scales

scaling

Intent

Sub-investigations

Direct references:

Direct references: