Text set in serif type with a dotted bar in the margin was drafted by an LLM and sometimes reviewed by the author. The rest is the author's own. How to read this site
depth_long
20260705_211010_depth_long · complete · published 2026-07-16 · seed None
Intent
Ran 2026-07-05 21:10 to 2026-07-06 01:32, about four hours and twenty
minutes, with --jobs 2, and was the first production use of
imsyn run continue.
It continues the two children of
runs/20260705_133844_depth, the three-layer and four-layer
arms, which stay read-only. It takes that run's
three-layer and four-layer per-iteration
checkpoints and continues each from 1200 to 4800
training iterations, a fresh-AdamW
warm restart on each model's own dataset, to see whether the
rightward-shifted, seed-bimodal
recency-to-agreement transitions
recover the two-layer behavior once they are given four times the
budget. The sweep covers the two transition bands densely,
alpha from 0.9 to 2.1, with tail probes at 2.5 and 3.0.
Background
This is an undertraining follow-up to the
depth experiment
(runs/20260705_133844_depth).
Hypothesis
If the depth effect is only undertraining, then by iteration 4800 three things should follow. The transition onset moves left toward the two-layer onset near Z = 1.1, the seed bimodality in the transition band collapses, and the high-Z tail recovers toward the two-layer plateau. If instead the onset and the bimodality persist at 4800, depth is a real optimization or capacity limit rather than a budget artifact. The anchor waypoint at 1200 must reproduce the source runs, and the waypoint at 1500 doubles as a check on the warm restart transient.
Setup
data regime (all arms): single-dataset-recycled (continued) — 1 dataset(s) × 4800 iters each = 16.0 epochs per dataset
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 63e09ac
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yaml⤷ This experiment continues from another run — reproduce the source sweep(s) first:
runs/20260705_133844_depth/n-layer_3, runs/20260705_133844_depth/n-layer_4
Checks
PASS children_complete PASS replicate_coverage PASS instrument_consistent PASS anchor_reproduces_source[n-layer_3] PASS anchor_reproduces_source[n-layer_4]
Results — unseen_mismatch (mean ± sd over 10 seeds)
Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | peak over α (end of training) | per-seed | flags |
|---|---|---|---|
| n-layer_3 | 78.1 ±7.9 @ α=1.5 | ||
| n-layer_4 | 76.8 ±11.4 @ α=1.2 |
Conclusions
The depth effect splits in two. The delayed transition is undertraining and comes back with more budget, but the peak and tail deficits are not, and continuing past the early peak shows that more training erodes agreement rather than cementing it.
Referenced by (2 direct, 3 transitive)
Direct references:
Both pre-registered checks hold. Each anchor waypoint at 1200 steps reproduces the source depth run to within 0.00 points over 600 cells per arm, and the warm restart leaves no visible transient: 300 steps after the restart, at 1500, the three-layer peak has risen from 92.1 to 93.1 rather than dipped.
The recency-to-agreement onset moves left as the budget grows, which is the signature of undertraining.
Reading the onset as the lowest α whose mean unseen_mismatch reaches 70, the three-layer arm moves from α = 1.3 at 1200 steps to 1.0 by 2400 and settles near 1.0 to 1.2 after, and the four-layer arm, starting further right at 1.7, reaches 1.1 by 2400 and holds there. Both arrive at the two-layer onset near 1.1 that they had missed at the shorter budget. The seeds that sat below chance at low α recover as this happens: the four-layer mean at α = 1.3 rises from 39 to 73, and at α = 1.5 it goes from 63 to 66 while the per-seed spread narrows from ±36 to ±13. By the hypothesis's own rule, the rightward onset shift was a budget artifact.
Referenced by (1 direct, 3 transitive)
Direct references:
The transition-band seed bimodality narrows without fully vanishing. The widest per-seed spread across α falls from about ±40 at 1200 steps to about ±20 at 4800 for both arms, and the transition band stops splitting the seeds across the chance line. At α = 1.2 the three-layer arm has five of its ten seeds below chance and five above at 1200; by 4800 all ten sit between 65 and 81. What spread remains is smaller and sits near the moving peak rather than straddling chance.
The peak and the tail tell the opposite story, and this is what the continuation adds beyond the 1200-step snapshot. Both arms reach their highest unseen_mismatch at any α early and then decline.
For three layers the peak over α climbs from 92.1 at 1200 to 93.1 at 1500 and then falls to 87.3, 82.4, and 78.1 at 2400, 3600, and 4800; for four layers it climbs from 86.0 at 1200 to 88.4 at 2400 and then falls to 80.2 and 76.8 at 3600 and 4800. For nearly every α the best waypoint comes before the last. The end-of-training peaks the report records, 78.1 ±7.9 at α = 1.5 for three layers and 76.8 ±11.4 at α = 1.2 for four layers, sit well below what each arm briefly reached, and below the roughly 93 the two-layer model reached in the paper run. More budget moved the peak the wrong way, so the peak deficit is not a matter of too few steps.
Referenced by (1 direct, 3 transitive)
Direct references:
The decline is confined to generalization. Holding α at 1.5 and following the four conditions across the waypoints, the trained conditions stay high: seen_match holds near 100 for both arms, and seen_mismatch ends near 97 for three layers and near 93 for four layers. The held-out conditions fall: unseen_match drops from about 99 to 89 for three layers and from 100 to 92 for four layers, and unseen_mismatch falls from its early peak, 90.8 at 1200 for three layers and 88.4 at 2400 for four layers, to 78.1 and 65.7 at 4800. The model keeps fitting the pairings it trained on while losing the abstract rule, which is the signature of overfitting.
The high-Z tail does not recover, which marks it as intrinsic to depth. The four-layer arm at α = 3.0 sits at 60 at 1200 steps and never climbs, holding between 51 and 55 at every later budget, while the three-layer tail barely moves, from about 65 at 1200 steps to about 68 at 4800. The depth ladder's tail inversion, where the shallower model does better at the most concentrated pairings, survives the fourfold budget.
Put together, depth acts on training time in two opposite directions. It slows the approach to the agreement solution, so the onset delay and the transition-band bimodality clear with more steps, and it hastens a later drift away from that solution, so the peak erodes as training continues.
Deeper models reach the good solution later and hold it for less time; at these depths agreement generalization has an optimal-stopping window in training time rather than a plateau, and the window arrives later and lasts less for the four-layer arm. The paper's two-layer, 1200-step configuration happens to sit near that stopping optimum.
Referenced by (4 direct, 4 transitive)
Direct references:
Follow-ups this suggests:
- Waypoint the decline more finely between 2400 and 4800, to see whether the slide is gradual forgetting or seeds dropping out of the agreement basin one at a time. The per-seed trajectories are already on disk in this run, so this is analysis only.
- Continue the shallower depth children, one and two layers, under the same protocol, to test whether the stopping window narrows with depth or is depth-independent.
- Examine what the extra layers do at high concentration, α ≥ 2.5, for the four-layer arm, since the tail deficit there is not a budget effect.
- Apply the same continuation to the width ladder's
n_embd= 512 arm, which was the original motivation.
Comparison figures
Children
| Child | n-layer | peak unseen_mismatch | Status |
|---|---|---|---|
| n-layer_3 | 3 | 93.1 | done |
| n-layer_4 | 4 | 88.4 | done |