depth

20260705_133844_depth · complete · published 2026-07-06 · seed 42

Intent

How does depth (n_layer) affect subject-verb agreement generalization? Complements msize (which varied width at 2 layers): depth controls the number of sequential composition steps available, while width (256) and heads (4) are held fixed throughout. The n-layer 2 child is the msize n-embd 256 baseline re-run, giving an in-experiment anchor.

Hypothesis

Agreement across an intervening PP needs a two-step circuit (locate the subject head noun while suppressing the prepositional object, then move its number feature to the verb slot), so 1 layer should manage at most the recency/attraction heuristic, with the recency-to-agreement phase transition appearing only at 2+ layers. Added depth beyond 2 buys little at this task scale and may destabilize (cf. the seed-bimodal d=512 width rung) since params grow linearly in depth at the fixed 1200-iter/lr-6e-4 budget. All children hold width (256), heads (4), max_iter, and lr constant — depth is the only variable.

Conclusions

(run 2026-07-05, 13:38–17:34, 3h56m wall with --jobs 4 — first production use of the parallel exp runner + dataset cache from issue #2; ~11h-equivalent sequential)

The hypothesis was wrong in both directions. One layer is sufficient for agreement generalization, and extra depth actively hurts at this training budget.

Per-depth unseen_mismatch (UMM), 10 seeds per α:

| depth | peak UMM | transition onset (first α ≥70%) | UMM @ Z=3.0 | max σ (where) | |---|---|---|---|---| | 1 | 90.7% ± 3.3 @ Z=1.8 | Z=0.9 | 81.7% ± 3.0 | ±29 @ Z=0.3 | | 2 | 93.1% ± 9.1 @ Z=1.2 | Z=1.1 | 74.0% ± 1.4 | ±38 @ Z=0.9 | | 3 | 92.1% ± 2.9 @ Z=1.7 | Z=1.3 | 64.7% ± 6.1 | ±40 @ Z=1.1 | | 4 | 86.0% ± 5.9 @ Z=2.1 | Z=1.7 | 60.0% ± 9.3 | ±38 @ Z=1.6 |

1. A single attention layer learns agreement-across-PP (peak 90.7%), so the two-step locate-then-copy circuit story is not necessary — one attention pass plus the MLP suffices. The recency heuristic is not a depth limitation.
2. Depth delays the recency→agreement transition monotonically: onset Z=0.9 → 1.1 → 1.3 → 1.7 for 1→4 layers. Each depth has a seed-bimodal transition band (σ ≈ 30–40) whose location moves right with depth; this is the same instability signature as msize's d=512 width rung, consistent with "more parameters at the fixed 1200-iter/lr 6e-4 budget destabilize/slow the grokking-like transition" rather than a capacity benefit.
3. The high-Z tail degrades with depth (Z=3.0: 81.7 → 74.0 → 64.7 → 60.0), inverting the ordering: past the transition, the shallowest model is the best and most stable. Depth ≥3 never fully recovers the 2-layer plateau within the sweep range.
4. The n-layer_2 child reproduces the msize n-embd_256 anchor (peak 93.1 @ Z=1.2 vs 93 @ Z=1.2 in msize), validating cross-experiment comparability — and doing so through the new dataset cache, i.e. cached datasets reproduce the fresh-generation science.

Open follow-ups: - Training-budget confound: rerun depth 3–4 (and msize d=512) with waypoints / longer max_iter to test whether deeper models are merely undertrained — is the rightward shift a delay in iterations rather than in Z? - Per-seed bimodality analysis in each depth's transition band (same protocol as the bimodality experiment's B_recent work). - Mechanistic question now sharper: what does the 1-layer solution look like? Attention-pattern analysis at d=256/L=1 around Z=1.0 vs Z=1.8 would show whether the single head suppresses the PP object directly.

Comparison figures

Children

experiment.yaml