Summer Summary
Over the summer I played with the imsyn project quite a bit. This is a summary of what I did and learned.
Bimodality
I started off by looking into the bimodality of replicate performance, a replicate being one model trained under a specific α value, after noticing that the spread on performance in the paper was pretty big at some α values. I was curious what caused it: whether it was a property of the initial conditions, the random initial parameter values in the model, or something else.
I didn't really reach a satisfactory conclusion, but one interesting outcome is that the split narrows with more training. Continuing those checkpoints out to twenty thousand steps brings the two groups together until the separation is no longer distinguishable from zero. That turns out to be true of the split α induces at the paper's 1,200-step budget specifically, and not of seed bimodality in general: the splits that capacity, vocabulary, budget and schedule endpoint induce all survive a longer budget (Two different splits, and only one of them survives more training). In some later runs I found another way to reduce bimodality, more on that later.
Every replicate of the paper's own sweep, one dot each. Between α 0.7 and 1.0 the seeds sit near 0 and near 100 with almost nothing in between, and the mean runs through the gap: at α = 0.8 eight seeds land under 40 and two land above 97, and the mean of 40.4 sits at the top edge of the lower group.
Scale
Then I looked at how the result changes when we change the scale involved in the experiment. I scaled the vocabulary up to 120 words and the paper's result was maintained: the shape of the curve as we sweep α changes some, but it still peaks in the same region of α. That holds at every size tested from 20 to 160 (The intermediate peak is not a vocabulary artifact).
I also looked at how changing model size affects the results, changing the number of layers and the size of the embedding space. In both cases additional model capacity hurt more than it helped (The two size axes come apart). Another interesting result here was that reducing the model to one layer didn't hurt performance as much as I expected (One layer suffices for agreement).
I did not scale vocabulary size with the model size, which would be an interesting follow-up. It seems like there should be some way to scale everything together and keep the same kind of performance.
Curricula
Then, inspired by the fact that CHILDES data shows the α value in child-directed speech decreasing with age, I got curious about how changing α over a training run could impact performance. Since we have near-perfect performance on unseen_mismatch by the end of a run, I looked into whether we could learn the correct rule faster, on average, by testing various curricula and asking at which step a given curriculum reaches the target accuracy for 90% of its seeds. The measure is the 90th percentile of time-to-band, censored at a 400-step budget, with the band set at 90.5 unseen_mismatch.
This is an optimization problem on a huge space. I tried a bunch of different techniques: different slopes, smooth against bumpy, mirrored shapes, and optimization over different low-dimensional parameterizations.
A grid of small panels of training α against iteration: descending ramps from 5 to 5, 5 to 4, and on down to 5 to 0, then the same for each lower starting point, then the ascending mirrors of all of them, and a block-staircase version of each.
The best shapes I have found for this metric start α very high, hold it there, and then drop. The fastest known one holds α at 4.76 for 160 steps and then drops once to 0.81 for the remaining 240, reaching the band at a 90th percentile of 202.1 steps, with a median of 190.7 and one seed in 150 not crossing inside the budget (The fastest schedule the campaign has found).
Some of the shapes tried, one panel each, against the dashed line at the best
fixed α. The staircases that hold high and then drop are the fast
ones; flat_1_6 and flat_1_8 are the constants they are
measured against.
Constant α does much worse on this. The better constant, 1.8, reaches the band at 337.9 steps, and the constant at 1.6 has no identifiable 90th percentile at all, because 16% of its seeds never cross inside the budget (Every shaped schedule beats every constant one by more than a hundred steps). Shaping α at all is worth about 136 steps against the best constant, more than a third of the horizon.
This particular metric also punishes bimodality, because we want 90% of our replicates to reach the target and it is not enough that half do really well and half do not, and so this shape reduces bimodality drastically. Maybe sending a strong, peaked set of co-occurrences causes the model to learn the right rule quickly, and then exposing it to a broad set of pairings lets it generalize. Anyhow, that start-high-and-go-low shape matches the CHILDES data too, even if the descent there is smoother. Smooth descents also do well on this test, just not as well as the big step: the smooth v2 descent reaches the band at 285.0 steps against 202.1.
Distributional Discrepancies
When looking into the validity of the curriculum work, I noticed a difference between how the models are trained there and how they are trained in the paper. The paper draws 12,000 distinct sentences, splits them 80/10/10, and shows the resulting 9,600-sentence training split four times over, which is its four epochs. In a curriculum run, each time we shift α we regenerate sentences, and deduplication only occurs within that new pool.
It turns out the deduplication of the paper's pool falls hardest on the most frequent pairings, the ones at the head of the distribution, and especially on sentences with no prepositional phrase. Every subject offset owns the same inventory of distinct strings, so the head is asked for far more sentences than the tail out of an equally sized supply, and it runs out first. The accepted pool is what survives that filter, so the head is shaved and the tail passes intact, which makes the realized distribution flatter than the one it was drawn from. A flatter distribution fits a smaller exponent, and that is why the effective α comes out below the nominal one. The effect grows as α increases, because a higher α makes the nominal distribution peakier and so makes the shaving stronger: at nominal 1.4 the pool realizes about 1.28, and at nominal 3.0 about 2.69.
It also means we don't train on the distribution of sentence shapes the code specifies, because there simply aren't enough no-phrase sentences to go around. The grammar can build 2,400 of them and a faithful 12,000-sentence pool would need 3,000, so the paper's corpus is about one tenth bare sentences against the one quarter its own sampler draws (The paper's corpus is one tenth bare, not one quarter).
I don't think this materially changes the outcome of the experiment. It changes which α value actually fits the data we're training on, shifting it down slightly (see Both regimes peak at the same realized α.). More on all of that at the sentence space page.
Overall
The work gave a lot of data on how the experiment's result changes when we change parts of its setup. I think it helps confirm that we still get an "there is an optimal α" result under a wide variety of changes, although the particular shape of the curve as we sweep α changes with the setup (The intermediate peak is real but it is a slice of a moving target). It also resulted in a bunch of machinery in the imsyn project for changing all of these parameters, running sweeps over spaces of parameters, and collecting and analyzing the results.
The website this is published on was a result of that too. It uses a project called latexblocks, which I developed for my personal mathnotes site. It allows linking between different pieces of information, like definitions, results, conclusions and hypotheses, across LaTeX documents, and renders the result in HTML with tooltips and clickable links. Every link in this posting is one of those.
The LLM-written text feels a bit goofy sometimes, but it's been useful to have it produce summaries. This carries over an approach to development I've used at work, where using LLMs to accelerate coding and experimentation, followed up with human supervision and hand-writing important documents for other humans to consume, can greatly accelerate a workstream. The LLM writing will only get better over time.