Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-fable-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site

depth

20260705_133844_depth · complete · published 2026-07-16 · seed 42

Intent

Run 2026-07-05, 13:38 to 17:34, about 3h56m of wall-clock with --jobs 4. This was the first production use of the parallel imsyn exp runner and the content-keyed dataset cache (issue #2); run sequentially it would have taken roughly eleven hours. All four children completed with no failures or resumes.

Intent: How depth affects agreement generalization \@{depth-intent} How does depth (n_layer) affect subject-verb agreement generalization?

Background

Background: Complementing the msize width ladder \@{depth-background} Complements msize (which varied width at 2 layers): depth controls the number of sequential composition steps available, while width (256) and heads (4) are held fixed throughout. The n-layer 2 child is the msize n-embd 256 baseline re-run, giving an in-experiment anchor. Two grounds shaped the expectation. Agreement across an intervening prepositional phrase needs a two-step circuit (locate the subject head noun while suppressing the prepositional object, then move its number feature to the verb slot). And params grow linearly in depth at the fixed 1200-step/lr-6e-4 budget, where the msize d=512 width rung was seed-bimodal.

Hypothesis

Hypothesis: Agreement needs two layers, and more buys little \@{depth-hypothesis} 1 layer should manage at most the recency/attraction heuristic, with the recency-to-agreement phase transition appearing only at 2+ layers. Added depth beyond 2 buys little at this task scale and may destabilize. All children hold width (256), heads (4), max_iter, and lr constant, so depth is the only variable.

Setup

model: GPT (nanoGPT-derived), weight-tied embeddings · block 50 · dropout 0.1 · vocab 167 tokens
training: AdamW (β 0.9/0.95, wd 0.1) · lr 0.0006 constant (no warmup, no decay) · batch 32 · 1200 iterations · fresh init per replicate (seeds derived from base seed) · 10 seeds (base 42)
training data: PCFG, zipfian noun–verb pairing (oneshot where noted) · 12,000 sentences per dataset (9,600 train — 1 epoch = 300 iters) · unseen holdout 10
code: commit a403abb

arm	n_embd	n_layer	n_head	~params
n-layer_1	256	1	4	845,824
n-layer_2	256	2	4	1,635,584
n-layer_3	256	3	4	2,425,344
n-layer_4	256	4	4	3,215,104

data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 4.0 epochs per dataset

Replicate this experiment

Get the code at this exact version:

git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout a403abb
pip install -e ".[dev]"

Download the config (experiment.yaml) and run:

imsyn exp run experiment.yaml

Checks

PASS children_complete PASS replicate_coverage PASS instrument_consistent

Results — unseen_mismatch (mean ± sd over 10 seeds)

Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition

Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)

arm	peak over α (end of training)	per-seed	flags
n-layer_1	90.7 ±3.3 @ α=1.8
n-layer_2	93.1 ±9.1 @ α=1.2
n-layer_3	92.1 ±2.9 @ α=1.7
n-layer_4	86.0 ±5.9 @ α=2.1

Conclusions

Conclusion: One layer suffices and depth returns less than it costs \@{depth-conclusion} The main prediction behind this experiment did not hold. A single transformer layer is enough to learn subject-verb agreement across a prepositional phrase. Adding layers does not raise the ceiling, and it steadily lowers accuracy on the concentrated end of the pairing range, so at this training budget depth returns less than it costs. The single-condition curve is images/comparison_unseen_mismatch.png, the four conditions side by side are in images/comparison_by_condition.png, and the peak-versus-depth summary is images/peak_vs_n-layer.png. Result: One layer suffices for agreement \@{depth-one-layer-suffices} The one-layer model learns the rule, and it learns it well. On the hardest condition, unseen_mismatch, it reaches a peak of 90.7 ±3.3 over ten replicates at α = 1.8, so agreement generalizes to pairings the model never trained on, across two attractor nouns of the wrong number, with one attention pass and its MLP. The hypothesis expected one layer to be stuck at the nearest-noun shortcut and the flip to genuine agreement to appear only at two layers or more. The opposite is true: the one-layer model makes the flip earliest, not latest, so the two-step locate-then-copy circuit is not a precondition for the task at this scale. Referenced by (2 direct)Direct references:The two size axes come apartSummer Summary Result: Depth delays the transition and degrades the tail \@{depth-delays-and-degrades} What depth does instead is push the recency-to-agreement phase transition to higher α. Every arm sits below the 50 chance line at low α, where the model agrees with the nearest noun rather than the subject. The α at which mean unseen_mismatch first rises above chance moves out steadily with depth: 0.8, 1.0, 1.2, and 1.5 for one, two, three, and four layers. The peaks tell the same story from the concentrated end, landing at α = 1.8, 1.2, 1.7, and 2.1 with heights of 90.7, 93.1, 92.1, and 86.0. The height is flat from one to three layers, and only the four-layer arm falls off, so the prediction that depth beyond two buys little holds even as the prediction about one layer fails. Near each crossing the ten seeds do not move together. At the α where their spread is widest they split into a low group still doing recency, near 0, and a high group that has learned agreement, near 90 to 100, with a few seeds between; the across-seed standard deviation there runs from about 29 at one layer (α = 0.3) to 40 at three layers (α = 1.1). This is seed bimodality in the transition band, where the mean describes no actual replicate and the per-seed values carry the information. The band that shows it moves to higher α as depth grows, the same rightward march as the chance-line crossing. Past the transition the depth ordering inverts, and the shallowest model is both the highest and the steadiest. On the concentrated tail at α = 3.0 the four arms hold 81.7, 74.0, 64.7, and 60.0 mean unseen_mismatch, a monotone decline with depth, and the seed spread widens the same way, from 3.0 standard deviation at one layer to 9.3 at four. Three and four layers never climb back to the shallow plateau within the swept range. Even seen_mismatch, the attractor trap on pairings the model did train on, slips a little as depth grows, from 99.8 to 99.6, 99.2, and 98.1 at each arm's best α, so depth costs a small amount even on familiar material. Referenced by (3 direct)Direct references:An undertraining follow-up to the depth ladderMore budget repairs the onset delay but not the peakThe two size axes come apart Result: Cached datasets reproduce fresh results \@{depth-cache-anchor} The two-layer arm reproduces its cross-experiment anchor. It was run as the msize n_embd = 256 baseline, and it lands at 93.1 ±9.1 at α = 1.2, the same peak height and location that the msize sweep and the paper reproduction report. The match holds even though this run drew its datasets from the content-keyed cache on the cache's first production use, so cached datasets reproduce the results of freshly generated ones. Referenced by (2 direct)Direct references:Width-axis counterpart to depth_longMore training does not settle the wide model Conclusion: The depth penalty is entangled with undertraining \@{depth-undertraining-confound} How to read the depth penalty is the one thing this run leaves open. The widening, later transition band and the sagging tail resemble the seed-bimodal behavior of the n_embd = 512 rung in msize, which suggests that adding parameters at the fixed 1200-step, 6e-4 budget slows or unsettles the transition rather than adding capacity the task can use. That reading is entangled with undertraining: this experiment records only the final iteration, with no waypoints, so a shift of the transition to higher α could equally be a shift to later in training that the fixed budget cuts off. The two cannot be separated from these runs alone. Referenced by (1 direct, 3 transitive)Direct references:Width and depth are different axes, and bigger is not betterTransitive (depth 1):The intermediate peak is real but it is a slice of a moving targetThe peak is universal, capacity caps it, and the holdout fraction is second orderTransitive (depth 2):A robust peak whose location is set by the budget Follow-ups this suggests: Rerun the three- and four-layer arms, and the msize n_embd = 512 rung, with waypoints or a longer iteration budget, to test whether the deeper models are merely undertrained: is the rightward shift in α actually a delay in steps? Characterize the per-seed split in each depth's transition band directly, with the protocol from the bimodality work, rather than reading it off the standard deviation. Look at what the one-layer solution does mechanically. An attention-pattern analysis at one layer, comparing α near 1.0 with α near 1.8, would show whether the single head suppresses the prepositional object directly.

Comparison figures

Children

Child	n-layer	peak unseen_mismatch	Status
n-layer_1	1	90.7	done
n-layer_2	2	93.1	done
n-layer_3	3	92.1	done
n-layer_4	4	86.0	done

experiment.yaml