paper_long_iter

20260506_205133_paper_long_iter · complete · published 2026-07-05 · seed 42

Intent

How does subject-verb agreement generalization (unseen_mismatch) evolve over the course of training, across the full Z sweep? The paper reports accuracy at a fixed 1200-iteration budget; this run extends training to 10,000 iterations with checkpoints at six waypoints to see whether the fixed budget under- or over-shoots the agreement solution at each Z.

Hypothesis

Agreement generalization is not monotone in training time. At high Z (concentrated pairings) the model should reach its agreement solution early and then drift away from it with continued training; at low Z it may never reach one. If so, the paper's fixed-budget curve conflates "cannot learn agreement" with "peak missed by the snapshot time".

Conclusions

Zipf sweep Z 0→3.0 step 0.1, 10 seeds per Z, max_iter 10,000, seed 42 (31 × 10 = 310 training runs). Waypoint evolution figures per waypoint plus waypoint_evolution.png; val-loss replay analysis lives in runs/_analysis/20260506_205133_paper_long_iter_val_loss/.

Mean unseen_mismatch by waypoint at representative Z:

| Z | 300 | 600 | 1200 | 2400 | 4800 | 10000 | |-----|-----:|-----:|-----:|-----:|-----:|------:| | 0.0 | 34.7 | 25.6 | 22.7 | 15.3 | 13.8 | 36.0 | | 1.0 | 20.5 | 30.4 | 64.4 | 74.3 | 49.6 | 55.3 | | 1.2 | 20.1 | 36.8 | 93.1 | 83.6 | 71.4 | 68.5 | | 1.6 | 39.9 | 94.8 | 85.0 | 77.9 | 71.3 | 71.8 | | 2.0 | 86.2 | 88.9 | 80.2 | 77.0 | 73.6 | 70.8 | | 3.0 | 85.4 | 71.6 | 74.0 | 73.0 | 73.1 | 71.2 |

Findings:

1. The peak moves earlier as Z grows. Z=1.2 peaks at iteration ~1200 (93.1%), Z=1.6 at ~600 (94.8%), Z≥2.0 by the first waypoint (~300). The paper's 1200-iteration budget happens to sit near the Z≈1.2–1.4 peak — which is why that region looks best in Figure 2.
2. Post-peak drift is real. Every Z ≥ 1.2 decays 15–25 points from its peak by iteration 10,000 and settles near ~70%. The high-Z advantage at the paper budget is partly a snapshot-timing artifact.
3. Drift is not val-loss overfitting. Validation loss improves monotonically to 10,000 iterations at every Z shown (e.g. Z=1.2: 1.94 → 1.84) while unseen_mismatch decays — the LM objective and the agreement generalization it induces dissociate after the peak (val_loss_story.png, phase_diagram.png).
4. Low Z never gets there. Z ≤ 0.6 stays below ~43% at every waypoint; longer training does not rescue uniform pairings.

Follow-up executed: the per-α refined peak search (20260509_153511_peak_umm_iter) brackets each Z's true peak iteration more densely than these six waypoints.

Comparison figures

Children

experiment.yaml