depth_long

20260705_211010_depth_long · complete · published 2026-07-06 · seed None

Intent

Undertraining follow-up to depth (runs/20260705_133844_depth): continue the existing L=3 and L=4 per-iteration checkpoints from 1200 to 4800 training iterations (fresh-AdamW warm restart on each model's own dataset) and watch whether the rightward-shifted, seed-bimodal recency->agreement transitions recover the 2-layer behavior with a 4x budget. Covers the two transition bands densely (alpha 0.9-2.1) plus tail probes (2.5, 3.0).

Hypothesis

If the depth effect is undertraining, by iter 4800 the transition onset moves left toward the 2-layer onset (Z~1.1), the transition-band seed bimodality collapses, and the high-Z tail recovers toward the 2-layer plateau. If onset and bimodality persist at 4800, depth is a real optimization/capacity pathology, not a budget artifact. The anchor waypoint at 1200 must reproduce the source runs; waypoint 1500 doubles as a check on the warm-restart transient.

Conclusions

(run 2026-07-05 21:10 – 2026-07-06 01:32, 4h22m with --jobs 2; first production use of imsyn run continue. Every anchor waypoint reproduced the source depth run's accuracies exactly — the continuation methodology is validated end-to-end.)

Split verdict: the transition delay IS undertraining; the peak/tail deficit is NOT — and continued training reveals a new degradation phenomenon.

Onset of the recency→agreement transition (first α with UMM ≥ 70%) per cumulative budget:

| budget | L=3 onset | L=4 onset | L=3 peak | L=4 peak | L=3 max σ | L=4 max σ | |---|---|---|---|---|---|---| | 1200 (anchor) | 1.3 | 1.7 | 92 @ 1.7 | 86 @ 2.1 | ±40 | ±38 | | 1500 | 1.2 | 1.5 | 93 @ 1.5 | 88 @ 1.9 | ±38 | ±35 | | 2400 | 1.0 | 1.1 | 87 @ 1.2 | 88 @ 1.5 | ±26 | ±33 | | 3600 | 1.1 | 1.1 | 82 @ 1.5 | 80 @ 1.4 | ±17 | ±25 | | 4800 | 1.2 | 1.1 | 78 @ 1.5 | 77 @ 1.2 | ±19 | ±22 |

1. Undertraining confirmed for the onset. By iter 2400 both depths reach the 2-layer onset (Z=1.1) that they missed at 1200; L=4 needed ~2× budget, consistent with "depth taxes the optimization schedule, not the ceiling". The stuck low-Z models transition dramatically (L4 Z=1.3: 39→79; L4 Z=1.5: 63→88 with σ collapsing ±36→±5). The warm-restart transient is negligible (all 1200→1500 steps move smoothly).
2. Overtraining degradation is real and pervasive. Nearly every trajectory peaks at 1500–2400 and then slides 10–25 points by 4800 (L3 Z=1.7: 92→67; L3 Z=1.0: 80@2400→57@4800). Best-waypoint beats iter-4800 for 13/15 (L=3) and 11/15 (L=4) α values. Agreement generalization at these depths has an optimal-stopping window in training time, not a plateau; the window is later and narrower for deeper models and lower Z.
3. The high-Z tail deficit is depth-intrinsic, not budget. L=4 at Z=3.0 never recovers (60 → 51–55 across all budgets); L=3 stays flat ~65–68. The depth ladder's tail inversion (shallow best at high Z) survives 4× budget.
4. Bimodality narrows but does not vanish (max σ ±40→±17-25), and its location tracks the moving onset — it is a property of the transition edge, wherever the edge currently sits.

Interpretation sketch: depth slows convergence toward the agreement solution (onset = budget effect) but also accelerates a later drift away from it (degradation = capacity/overfit effect), so deeper models see the good solution later and hold it for less time. The 2-layer/1200-iter paper config sits, apparently by luck, near the shallow models' stopping optimum.

Open follow-ups: - Waypoint the DECLINE: is the 2400→4800 slide gradual forgetting or seed-discrete (models dropping out of the agreement basin one by one)? Per-seed trajectories are already on disk in this run — analysis only. - Does L=1/L=2 also degrade by 4800? Continue the remaining depth children (cheap, same command) to test whether the window narrows with depth or is depth-independent. - Tail mechanism at Z≥2.5 for L=4: not budget — look at what the extra layers do to the PP-object suppression at high concentration. - msize d=512 continuation (the original motivation) — same protocol.

Comparison figures

Children

experiment.yaml