Selling to Labs

Ship it training-ready

Delivery is a term of the deal

Sellers treat curation as a cost they hope the buyer will appreciate. Buyers treat it as a property of the asset that they price. Published work on principled curation has found that carefully selected training sets matched or beat their baselines with around 14% fewer samples — which is the whole argument in one number. Fewer, better records produce the same capability. If you deliver the unselected version, you have handed the buyer the selection work and they will pay you the unselected price.

What a delivery contains

northside-triage-v1/
  README.md                  # what this is, in 200 words
  DATASHEET.md               # the long form (below)
  SCHEMA.json                # JSON Schema for one record
  LICENCE.txt                # what the buyer may do
  PROVENANCE.jsonl           # per-record source, instrument, consent basis
  data/
    train-0000.parquet       # ~500MB shards, stable ordering
    train-0001.parquet
    holdout-0000.parquet     # a real held-out split, defined by you
  reports/
    profile.json             # volume, dedup, language, length distributions
    pii-audit.md             # recall figures and method
    assay.json               # deterministic checks, reproducible by the buyer
  CHECKSUMS.sha256

A buyer receiving that can load it the same day. A buyer receiving a zip of CSVs with a column called notes2_final spends a week guessing, and every hour of that week is spent forming an opinion about your organisation.

Format, decided

Use	Format	Why
Text, conversations, demonstrations	JSONL, one record per line, UTF-8, newline-delimited	Streams, shards, greps, and survives partial reads. The default of the field.
Large tabular or heavily-typed corpora	Parquet, ~200–500MB per shard	Columnar reads, compression, and schema carried in the file
Media	Files on disk plus a JSONL manifest referencing them by relative path	Nobody wants base64 inside a JSON field
Anything	Not CSV	Embedded newlines and commas in free text corrupt silently; the failure appears as bad data, not as an error

Schema rules that prevent most support tickets

Every record has a stable unique id that survives revisions.
Fields are present on every record, explicitly null when absent. Optional keys turn into loader branches.
One type per field. A field that is sometimes a string and sometimes a list is the most common defect we see.
Timestamps in ISO 8601 with a timezone. Dates without timezones are a guess.
Nesting no deeper than it needs to be, and never variable-depth.
A version field on the record, so a corrected delivery is distinguishable from the original.

The datasheet

Ten sections. It takes a day to write and it is the document the buyer's reviewer forms their opinion from.

What this is — the corpus in three sentences, including what task it supports.
Collection method — how records came to exist, by whom, with what instrumentation, over what period.
Population and sampling — what universe this is drawn from, and how. Convenience sampling is fine; unstated convenience sampling is not.
Preprocessing — every transformation applied, in order, including what was dropped and how much.
Labels — who produced them, their qualifications, the protocol, and inter-annotator agreement where more than one person was involved.
Privacy — the measurements from your de-identification work, not the assertion.
Rights — the instruments, per source, summarised so the buyer can paste it into a public training-data summary.
Known gaps and biases — see below.
Recommended use and non-use — what this corpus does not support. Saying so raises your credibility on everything else.
Maintenance — cadence, contact, how corrections are issued.

Section 8 is the one that sells

"Coverage of paediatric cases is thin — under 4% of records — because the source clinic does not run a paediatric service. Records from 2020–21 carry a different documentation template, which shows up as shorter narration fields."

That paragraph costs nothing and does two things. It tells a reviewer you have actually read your own corpus, and it pre-empts the discovery that would otherwise happen in week three and cost you leverage. Every buyer assumes gaps exist. The only variable is whether they hear about them from you.

What to clean, and what to leave

Clean	Leave
Exact and near-duplicates (report the rate; do not silently drop)	Natural variation in style, length and quality
Encoding damage, mojibake, truncation	Domain jargon, abbreviations, shorthand
Boilerplate: headers, footers, navigation, disclaimers	Failures, corrections and dead ends — these are signal
Test and staging records, seeded rows, internal QA artefacts	Hard, ambiguous or unusual cases
Contamination against public benchmarks (check, and report)	Records that disagree with each other — annotate, do not resolve

The distinction is simple: remove what carries no information about the task, keep everything that does, and never remove something because it looks untidy. Over-cleaning is as detectable as under-cleaning, and it destroys value that cannot be recovered.

Before you send it

Load your own delivery from scratch, on a clean machine, following only your README.
Run the deterministic checks and ship the output. Numbers the buyer can reproduce beat numbers they have to trust.
Verify checksums after transfer, not before.
Deliver the held-out split as a separate artefact so it can be used for evaluation without contaminating training.

Read next: The five ways a deal dies — where these deals actually end, in the order of frequency.

List a dataset All writing