Tools & Code · v0.1

marginal — what would this dataset add?

Apache-2.0Target: under $30 a run

Every mixture tool in the ecosystem answers the same question: given the sources I already hold, what is the optimal blend? Not one of them answers the question that decides a purchase — what would this candidate dataset add to what I have? That gap is why data deals are argued rather than measured. marginal is our attempt to close it, and it is open because whoever ships the standard answer to that question defines how datasets get bought.

One command

marginal run \
  --candidate ./my_dataset.jsonl \
  --control public-domain-matched \
  --base Qwen3-4B \
  --tokens 200M \
  --evals domain.yaml,mmlu,heldout \
  --seeds 3

It trains the base model twice — once on the candidate, once on a matched control — under an identical schedule and token budget, evaluates both plus the untouched base, and reports the difference with an interval. Roughly an hour on a single rented accelerator, at a target cost under $30.

Five design commitments

1 · A matched control, by default

A before-and-after with no control measures "we fine-tuned", not "this data helped". Almost every capability claim in this market is a before-and-after. The default control is public in-domain data at the same token count under the same schedule; if the treated run and the control run land on top of each other, the honest reading is that the buyer can get this effect for free.

2 · A real fine-tune on a small dense model, not LoRA on a large one

LoRA on a 70B model measures whether knowledge already latent in the weights can be surfaced. The buying question is whether the capability can be learned from your data. Those are different experiments and they give different answers on the same dataset — full fine-tune on a 4–8B dense model is the cheaper one that answers the question actually being asked. It is also what the buyer's own screening runs at, per how a mix gets chosen.

3 · Multiple seeds and an interval, always

A 1,000-item benchmark carries roughly 1.5 percentage points of binomial standard error before seed variance, prompt-format variance and decoding noise are added. Any tool reporting a point estimate on a 2-point effect is reporting noise with a decimal place. Three seeds is the floor; --seeds 1 exists and prints a warning on every line of output.

4 · Three diagnostics, not one score

A single number tells a seller they scored 61 and nothing about what to do. These three tell them whether to clean labels, whether to stop claiming uniqueness, or whether to re-scope the listing to the slice that actually moved — and they map onto three dimensions of the rubric.

5 · Cheap enough to give away

The cost target is not a nice-to-have. A $1,000 assay is only paid for by sellers who already know their data is good, which selects for exactly the corpora that least need measuring and produces an adverse-selection spiral. At under $30 the test can sit at the front of a free intake funnel, which is the only way the measurement becomes normal rather than exceptional.

What it prints

$ marginal run --candidate ./triage.jsonl --base Qwen3-4B --seeds 3

candidate   triage.jsonl · 41,200 records · 18.4M tokens
control     public-domain-matched · 18.4M tokens
base        Qwen3-4B · full fine-tune · 3 seeds · identical schedule

                          base      control      candidate     Δ vs control
  domain.yaml            34.1       36.8         48.2          +11.4  [+8.1, +14.6]
  ├ triage-decisions     29.4       31.0         51.7          +20.7  [+16.2, +25.1]
  ├ documentation        38.2       40.1         44.9           +4.8   [+1.3,  +8.2]
  └ escalation           35.8       39.2         41.1           +1.9   [-1.8,  +5.6]  ns
  mmlu                   61.2       61.0         60.7           -0.3   [-1.9,  +1.3]  ns
  heldout-general        52.7       52.9         52.4           -0.5   [-2.2,  +1.1]  ns

diagnostics
  ✓ generalises           held-out gain tracks train loss (no memorisation signal)
  ✓ beats control         +11.4 over matched public data, outside the interval
  ! narrow                78% of the gain sits in one slice (triage-decisions)
  ✓ no regression         general evals unchanged within noise

cost  $23.40 · 3h12m · 3 seeds × 2 arms + base eval

That last diagnostic is the useful one for this seller. The corpus is real and the effect is real, but it is a triage-decisions asset, not a clinical documentation asset — and the listing should say so. Discovering that here costs $23; discovering it in a buyer's pilot costs the deal's credibility.

What it cannot tell you

• Whether the effect holds at frontier scale. Proxy-scale results transfer imperfectly. The tool measures what a small dense model learns, which is the same evidence the buyer's own screening produces — but it is not a guarantee about a 400B model.
• Anything about rights. A dataset can post an excellent contribution and be unlicensable, which is the most expensive way to run this test. Do chain of title first.
• Whether the eval is the right eval. If your domain evaluation is weak, the whole run inherits that weakness. Writing a good domain.yaml is most of the work and the tool cannot do it for you.
• What a buyer will pay. It produces evidence, not a price. Price comes from replacement cost and structure.

Interoperability, on purpose

Training uses standard fine-tuning stacks; evaluation runs through adapters for the common harnesses so that a number produced here is comparable to a number produced anywhere else. We are not interested in a metric only we can compute — a private score is a marketing asset, and this needs to be a measurement.

Every run emits a signed, reproducible manifest: candidate hash, control identity, base model revision, schedule, seeds, eval versions, and raw per-seed results. Hand it to a counterparty and they can rerun it. That is the difference between publishing a claim and publishing a method.

If you run it and the numbers disagree with your own measurements, that is the most useful report we can receive — particularly on thresholds, which are currently calibrated on a small and self-selected intake queue.