The Mix Report
Midtraining is where licensed data lands
The most common self-inflicted wound in this market: a data owner compares their 40 million tokens to a six-trillion-token pretraining mix, concludes they are a rounding error, and either walks away or prices accordingly. They are comparing against the wrong stage. Almost no commercial dataset is bought for pretraining. It is bought for the hundred-billion-token stage that comes after, where a corpus that size is not a rounding error at all.
What midtraining is
Between pretraining and post-training sits a stage that goes by several names — midtraining, annealing, the high-quality phase, the decay phase. Mechanically it is the tail of pretraining run on a different diet: the learning rate is decayed toward zero while the data mix shifts hard toward dense, curated, instructionally useful text. Maths, science, code, reading comprehension, instruction-following.
The public reference point is the fully-open Dolma 3 lineage: a ~5.9T-token pretraining mix, then Dolmino at 100B midtraining tokens sampled from a ~2.2T pool, plus Longmino at ~50B for long context. Those are the numbers to hold in your head. Not the 5.9T.
| Stage | Scale | What earns a place | Marginal value per token |
|---|---|---|---|
| Pretraining | Trillions | Being cheap, plentiful and not actively harmful | Near zero — the supply is effectively unbounded |
| Midtraining | ~1011 | Density of useful signal per token | High, and the competition is a pool ~20× the slot |
| Post-training | 106–109 | Demonstrated behaviour, preferences, verifiable tasks | Highest, and priced per item rather than per token |
Why the phase does disproportionate work
Two mechanisms, both well attested in the open literature and both commercially relevant.
Low learning rate, high leverage. Data seen while the learning rate decays has an outsized effect on the final weights relative to the same data seen early. The model is no longer forming general representations; it is settling. What it settles into is disproportionately what it was reading at the end.
Density beats volume once representations exist. By this point the model has seen enough English to model English. What it has not seen much of is worked reasoning, correct derivations, well-structured procedural accounts of doing a real task. Those are rare on the open web at any scale, and they are exactly what specialist data owners hold.
The arithmetic that should change your pitch
Wrong framing
40M tokens / 5,900,000M pretraining tokens = 0.0007%
"we are irrelevant"
Right framing
40M tokens / 100,000M midtraining tokens = 0.04%
...against a candidate pool of ~2,200,000M = a 22:1 competition
...in a category (say, clinical reasoning) where the pool
may hold only a few billion tokens at all
"we are 40M of the maybe 3,000M tokens in existence that do this,
competing for a slot where the buyer needs perhaps 2,000M"The last framing is the one that gets a second meeting, and it is not rhetorical — it is the arithmetic the buyer is doing. Their question is never "is this big?" It is "what fraction of the midtraining budget can I fill with this category, and what is the best available source for that category?"
What makes a corpus midtraining-grade
Five properties, in the order buyers weigh them. None is about size.
- Signal density. High ratio of substantive content to boilerplate, navigation, disclaimers and repetition. A 40M-token corpus after cleaning beats a 400M-token corpus that is 90% template.
- Explicit reasoning. Text that shows the steps rather than only the conclusion. A decision with its justification is worth several multiples of the same decision alone, which is why capture programmes that record operator narration are worth running.
- Verifiability. Content where correctness can be checked — derivations with answers, code with tests, procedures with outcomes. It transfers into the post-training stage as well, which doubles its market.
- Category coherence. The mixture search assigns weight to groups. A corpus that is unambiguously one thing gets a clean weight; a grab-bag gets averaged down.
- Long-form structure. If any part of your corpus consists of genuinely long, coherent documents, say so separately — the long-context slot is its own budget line and the supply of real long documents is much thinner than the supply of short ones.
How to size your asset, concretely
- Name the category your data belongs to as a buyer would name it in a mix table — not "healthcare data" but "clinical decision traces with outcomes".
- Estimate the total public supply in that category. For most specialist categories this is small enough to enumerate, and if you cannot find an equivalent public corpus, that is the finding.
- Estimate your effective token count after deduplication and boilerplate stripping. Use the real number; the buyer will compute it anyway.
- Express the ratio against a 100B budget and against the category pool, not against the pretraining mix.
- Bring evidence of effect at proxy scale, because that is what the selection procedure runs on.
A seller who arrives with those five lines is having a different conversation from one who arrives with a row count — and it is the conversation that reaches a price.
Figures for the open corpora are as published by their authors and are approximate; token counts shift with tokeniser and release revision. The framing holds across an order of magnitude either way.
Read next: Post-training is where the money is — the stage with the highest price per record in the market.