Text set in serif type with a dotted bar in the margin was drafted by an LLM (gpt-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
data-reuse
investigation
Intent
This investigation asks how training-sentence Reuse changes subject-verb agreement learning at a fixed optimizer training budget. It separates reuse from with / without replacement: every arm draws unique sentences without replacement, so α controls which noun-verb pairings enter the pool rather than how often a retained sentence is presented.
Its first experiment starts from the paper's constant-α design. The paper cycles a 9,600-sentence training split four times to fill 38,400 training slots. The matched no-reuse regime instead draws 38,400 unique training sentences and presents each once. Comparing them across the full α grid and the oneshot limit tests whether the published generalization curve depends on seeing the same sentences repeatedly.
Experiments
-
20260802_230301_faithful_v1 2026-08-03
Every sweep this project has run drew its training sentences without replacement, and that draw does not deliver the α it was asked for. Requesting a pool of distinct sentences larger than the grammar can comfortably supply pushes the sampler into the tail of the subject offset distribution, so the marginal the model trains on is flatter than the one requested. The paper's headline curve, unseen_mismatch against α with an intermediate peak, is therefore plotted against a nominal α that overstates the realized one by an amount that grows with α.
This experiment holds the paper's training budget, vocabulary, and evaluation fixed and varies only how the pool is drawn. Two arms sweep α from 0.0 to 3.0 at a pool of 48,000 sentences, one without replacement and one with replacement, and two more do the same at the oneshot limit. At that pool size 1,200 steps at batch 32 consume the training split exactly once, so neither regime recycles and the sampler is the only thing that differs.