Text set in serif type with a dotted bar in the margin was drafted by an LLM (claude-opus-5) and sometimes reviewed by the author. The rest is the author's own. How to read this site
faithful_v1
20260802_230301_faithful_v1 · complete · published 2026-08-03 · seed 42
part of investigation data-reuse
Intent
Every sweep this project has run drew its training sentences without replacement, and that draw does not deliver the α it was asked for. Requesting a pool of distinct sentences larger than the grammar can comfortably supply pushes the sampler into the tail of the subject offset distribution, so the marginal the model trains on is flatter than the one requested. The paper's headline curve, unseen_mismatch against α with an intermediate peak, is therefore plotted against a nominal α that overstates the realized one by an amount that grows with α.
This experiment holds the paper's training budget, vocabulary, and evaluation fixed and varies only how the pool is drawn. Two arms sweep α from 0.0 to 3.0 at a pool of 48,000 sentences, one without replacement and one with replacement, and two more do the same at the oneshot limit. At that pool size 1,200 steps at batch 32 consume the training split exactly once, so neither regime recycles and the sampler is the only thing that differs.
Background
Deduplication does not deliver the α it is asked for. The effective α a pool realizes is the lower one, and at the paper's own 12,000-sentence pool the gap is already the distance between a nominal 1.4 and a realized 1.28. It widens with pool size. The arms here are larger still: at vocabulary size 40 a 48,000-sentence pool drawn without replacement realizes 1.18 at nominal 1.4, 1.47 at nominal 1.8, and 2.32 at nominal 3.0. Drawn with replacement the realized and nominal α agree to two decimal places at every level tested, which is why the sampler is the variable here and the pool size is held fixed at 48,000 across all four arms.
Two prior results frame what is at stake. The joint-fidelity capacity bound establishes that uniqueness and fidelity to the grammar exclude each other at any budget this project uses, and the page on the paper's own corpus works out what the paper's pool costs in the form class mix. The schedule-optimization campaign has drawn corpus-faithful pools since its fourth round, so its constant-α reference points and the paper's are not currently on the same axis.
Hypothesis
The unseen_mismatch peak of the with replacement sweep falls at a lower nominal α than the peak of the without replacement sweep, and lands between 1.1 and 1.2. The deduplicated arm peaks near α = 1.4.
The oneshot arms test the same prediction where the effect on corpus composition is largest. The grammar builds only 80 bare sentences at a single pairing, so deduplication reduces them from a quarter of the pool to 0.17 percent of it, against 24.99 percent when the pool is drawn with replacement. No direction is registered for the height of the peak or for the spread across seeds. Those are estimated rather than predicted.
Setup
data regime (all arms): single-dataset-recycled — 1 dataset(s) × 1200 iters each = 1.0 epochs per dataset
Replicate this experiment
Get the code at this exact version:
git clone https://github.com/ClaireHobbs/imagining-syntax
cd imagining-syntax
git checkout 9bb2434dae7c
pip install -e ".[dev]"
Download the config (experiment.yaml) and run:
imsyn exp run experiment.yamlChecks
PASS children_complete PASS replicate_coverage PASS instrument_consistent
Results — unseen_mismatch (mean ± sd over 10 seeds)
Evaluation data: seen_* generated at each grid point's own training α · unseen_* = held-out pairings, uniform (α-independent) · 1000 pairs per condition
- Each probe is a minimal pair: a grammatical sentence and its verb-number-flipped twin. The model scores correct when it assigns the grammatical version higher probability; accuracy = % correct over 1000 pairs per condition.
- seen_match / seen_mismatch — noun–verb pairings that occur in training, sampled at the same α the model was trained on (matched per grid point, so this eval data varies with the training α)
- unseen_match / unseen_mismatch — held-out noun–verb pairings that never occur in training, sampled uniformly (α = 0), so their difficulty is identical across all arms and training αs
- match vs mismatch — whether the prepositional objects agree in number with the subject; mismatch places attractor nouns between subject and verb (the AGREE-RECENT trap)
| arm | end of training | per-seed | flags |
|---|---|---|---|
| dedup_oneshot | 58.5 ±4.0 | ||
| dedup_zipfian | 91.7 ±5.2 @ α=1.6 | ||
| faithful_oneshot | 56.8 ±2.9 | ||
| faithful_zipfian | 96.6 ±2.1 @ α=1.2 |
Conclusions
The intermediate peak the paper reports is a property of the distribution the model trains on, not of the sampler that assembled the dataset. Both sweeps peak, and they peak at nominal α values that are three to four grid points apart, but almost all of that separation disappears once each peak is expressed in the α its pool actually realizes.
The with replacement sweep peaks in unseen_mismatch at nominal α = 1.2, reaching 96.6 ± 2.1, and the without replacement sweep peaks at 1.6, reaching 91.7 ± 5.2, with a flat top spanning 1.5 to 1.6 (90.1 ± 6.0 at 1.5). Measured in realized α, drawn from the subject offset marginal each pool actually contains, those peaks sit at 1.20 and at 1.26 to 1.33. A nominal separation of 0.4 becomes a realized separation of 0.06 to 0.13, at or below the 0.1 grid step this sweep can resolve.
Referenced by (1 direct)
Direct references:
The faithful sweep also peaks higher and far more tightly, 96.6 ± 2.1 against 91.7 ± 5.2, so drawing the pool with replacement both puts the peak where the axis says it is and reduces the spread across seeds at it.
At the oneshot limit the two regimes are 1.7 points apart in unseen_mismatch, 58.5 ± 4.0 without replacement against 56.8 ± 2.9 with it, and the deduplicated arm is the higher of the two. This is where deduplication changes corpus composition most, taking the bare sentences from a quarter of the pool to 0.17 percent of it, and the pre-registered expectation was that the regimes would separate most here. They separate least. Whatever the bare sentences contribute to agreement on held-out pairings, it is not a function of how large a share of the pool they occupy.
The pre-registration predicted a deduplicated peak near α = 1.4, and that figure
does not survive. It came from an earlier measurement that this sweep
reproduces exactly up to α = 1.2 and then departs from, and the departure has a
cause worth recording, because it was a defect in the runner rather than in
either run. Under --resume the parameter grid was filtered to the
values still outstanding and then re-enumerated from zero, so a
seed was derived from a position in the remaining list rather than in
the full grid. In the interrupted run that supplied the 1.4, α = 1.3 trained
under seed 42 and α = 1.4 under 1042, values belonging to α = 0.0 and
α = 0.1; uninterrupted, they are 13042 and 14042. The prediction was therefore
checked against a number no clean run reproduces, which is a reason to hold
the direction of this result and not the stated location it was measured
against.
What remains open is the residual. In realized α the two peaks are 0.06 to 0.13 apart, which this design cannot distinguish from zero, so the experiment establishes that the regime accounts for most of the shift and not that it accounts for all of it. Separating the rest needs a finer grid than 0.1 around the peak, and more than ten seeds, because below the peak the seeds are strongly bimodal and the mean tracks how many of them solved the task at all rather than how well.
Comparison figures
Children
| Child | peak unseen_mismatch | Status |
|---|---|---|
| dedup_zipfian | 91.7 | done |
| faithful_zipfian | 96.6 | done |
| dedup_oneshot | 58.5 | done |
| faithful_oneshot | 56.8 | done |