The Mix Report
How a mix actually gets chosen
Sellers tend to imagine a person deciding that a corpus is interesting. What actually happens is a search: a lab defines a set of source groups, a target to optimise, and a cheap way to estimate the effect of changing the weights, then runs a procedure that returns a weight vector. Your dataset does not get chosen. It gets assigned a weight — possibly zero — by a procedure with a stated objective. Knowing the objective is worth more than knowing anyone at the lab.
The problem, stated the way a lab states it
Given source groups s₁ … s_k (web, code, maths, papers, forums,
licensed corpus A, licensed corpus B) and a token budget T, choose
sampling weights w₁ … w_k summing to one that minimise loss on a
target distribution. Two facts make this hard and both matter commercially:
- Evaluating one weight vector means training a model. The search space is continuous and high-dimensional; the objective costs a training run. Every method below is a way of buying a cheaper estimate.
- The target is a choice, not a given. "General capability" is not a distribution. Someone picks the validation set, and that pick determines which sources get weight. A corpus that helps on a target nobody selected is worth nothing to that search.
Four families
Proxy-model reweighting (DoReMi, DoGE)
Train one small reference model on a baseline mix, then train a second small model that adjusts group weights during training to concentrate on groups where it is doing worst relative to the reference — a minimax objective over source groups. DoGE varies the criterion: instead of excess loss, it upweights groups whose gradients are most useful to a designated target set. Both output a weight vector that is then reused at full scale, on the assumption that proportions transfer up.
Consequence for a seller: a group gets weight for being hard and useful, not for being large. A corpus that a model already handles well gets downweighted precisely because it is easy — being clean and predictable can score against you here.
Regression on many small runs (RegMix)
Train dozens or hundreds of very small models on randomly sampled mixtures, fit a regression from mixture weights to validation loss, then optimise the fitted surrogate and apply the argmax mix at scale. It is the most straightforwardly empirical of the four: no theory of why a source helps, just a fitted response surface.
Consequence for a seller: your corpus must show an effect at proxy scale. If the benefit only appears in a 70B model after 2T tokens, this search cannot see it, and no amount of narrative fixes that. It is also why our own contribution harness deliberately runs a real fine-tune on a small dense model — the buyer's screening procedure is small-scale, so the seller-side evidence has to be too.
Online and dynamic mixing
Rather than fixing weights up front, adjust them during the run from signals available at training time — per-group loss trajectories, learning speed, gradient conflict. Sources that stop contributing get downweighted mid-flight.
Consequence for a seller: weight is not permanent. A corpus can be included and then effectively switched off partway through, which is one reason "our data was in the mix" is a weaker claim than it sounds and why outcome-linked pricing is hard to negotiate.
The proxy swarm with conditional mixing
The approach used for the most recent fully-open mixes: run many small proxy models across candidate mixtures, but add a step that handles sources changing during development — a corpus is re-filtered, a licence lands, a new tranche arrives. Rather than restarting the search, the procedure conditions on what has already been decided and re-solves for the remainder.
Consequence for a seller: arrival timing matters. A corpus that becomes available while the search is running can be slotted in; one that arrives after the mix is frozen waits for the next model. Labs plan mixes on a cycle, and a deal that closes two weeks after the freeze is a deal for a model a year out.
What all four have in common
| Property of the search | What it means when you are a candidate |
|---|---|
| The unit is a source group, not a record | Your corpus is evaluated as one blob. Mixing your excellent 200k records with 2M mediocre ones drags the whole group's weight down. Segment before you sell. |
| The objective is a named validation set | Ask what the buyer is optimising. "Improves general quality" is unpurchasable; "moves our internal long-horizon tool-use eval" is a specification you can be measured against. |
| Estimates come from small proxies | Evidence that survives at 1B–8B scale is the evidence that gets you weight. |
| The answer is relative | You are not competing against a quality bar, you are competing against the other groups for the same weight. Being good is not enough; being differently good is. |
| Zero is a legitimate output | Sources routinely get weight ≈ 0 and are dropped. A lab saying no is often the search saying no. |
The three questions to ask a buyer
- What is the target set for this mix? Not the public benchmark — the internal validation set the weights are fitted against. Some buyers will not say, and their reluctance still tells you whether they have one.
- At what scale do you screen candidates? This tells you what evidence to bring, and whether an effect you have measured is even visible to their process.
- Which stage is this for? Mixture search over pretraining groups is a different procedure, with a different objective and vastly different token economics, from selecting a post-training set. Most commercial data is being considered for the latter — see midtraining and post-training.
The seller's version of this is short: make your corpus a coherent group, know which target it moves, and be able to show the movement at small scale. That is the entire brief, and it is a technical brief rather than a commercial one.
Read next: Midtraining is where licensed data lands — the stage most commercial datasets are actually competing for.