Tools & Code · v0.1

assay — the rubric, as a command you can run

Apache-2.0No network calls

We publish the rubric we score datasets on. Publishing it as a web page is a claim; publishing it as a command anyone can run against their own file is a method. assay is the deterministic half of our intake — every check that produces the same answer for the same bytes — with the published weights applied on top. It runs locally, it sends nothing anywhere, and it does not need an account.

What it does

$ assay check ./sample.jsonl --modality computer_use

sample.jsonl · 12,481 records · 41.2M tokens · computer_use

  duplicates
    exact                    0.4%
    near (simhash, d≤3)      6.1%   ← 762 records in 214 clusters
    effective volume         38.5M tokens (-6.5%)

  pii
    email                    31 records (0.25%)
    phone                    8  records (0.06%)
    person-name (free text)  1,904 records (15.3%)   ← unreviewed
    detector recall (held-out probe)  0.91

  schema
    fields stable            yes (9/9 across all records)
    null rate, `reasoning`   4.2%
    step-count outliers      17 records >200 steps

  encoding
    mojibake                 3 records
    truncated utf-8          0 records
    control chars            0 records

  markers
    assistant-preamble       0.8%   ← possible undisclosed synthetic
    benchmark overlap        no public-suite n-gram hits

composite  —  needs 4 qualitative scores to compute (see --scores)
deterministic dimensions:  coverage 71 · hygiene 62 · delivery 88

Everything above is mechanical. No model is called, nothing is judged, and two runs over the same file give the same output. That is the point: these are the numbers a buyer can reproduce, so they are the numbers worth fixing before anyone else looks.

The checks, and why each one is in the list

Check	Feeds	Why a buyer cares
Exact and near-duplicate rate (simhash clustering)	Coverage	Duplicate mass inflates a volume claim and shrinks effective volume at the same time. It is the first thing a buyer computes.
PII pattern rates, plus a free-text name sweep	Hygiene, Rights	Structured-field redaction is common; free-text residue is where it survives. Reported with detector recall, because a rate without a recall figure means nothing.
Schema consistency and null rates	Delivery	A field present in 96% of records is a field the buyer's loader will crash on.
Encoding damage	Delivery	Mojibake and truncation are cheap to fix and read as carelessness when found by someone else.
Corpus-marker screening	Hygiene	Assistant preambles, refusal boilerplate and template regularity flag content that may be model-generated and undisclosed.
Public-benchmark n-gram overlap	Hygiene	Contamination against evaluation suites turns a capability claim into an artefact.

What it deliberately does not do

Three of the seven rubric dimensions — capability lift, scarcity, and fidelity — cannot be computed from a file. Lift requires training runs against a matched control. Scarcity requires knowing what else exists. Fidelity requires domain expertise on an audited slice. assay accepts those as inputs and applies the published weights, rather than pretending to derive them:

$ assay check ./sample.jsonl \
    --modality computer_use \
    --scores lift=64,scarcity=80,fidelity=73 \
    --json

{"rubric": "r2", "composite": 72, "dimensions": {
   "rights": null, "lift": 64, "scarcity": 80, "fidelity": 73,
   "coverage": 71, "delivery": 88, "hygiene": 62 },
 "weights": {"rights":24,"lift":20,"scarcity":16,"fidelity":14,
             "coverage":12,"delivery":8,"hygiene":6},
 "sample_sha256": "9c1f…", "incomplete": ["rights"]}

Any dimension left null is reported as null and named in incomplete. The composite is computed over the supplied weights and renormalised, and the tool says so. A scoring tool that silently substitutes a default for a dimension it could not measure is worse than one that refuses.

The open/closed line, stated plainly

Optimising against a published check improves the asset. Optimising against a published prompt improves only the sample.

A seller who reads our duplicate threshold and deduplicates their corpus has made it genuinely more valuable — the buyer gets what the number says. A seller who reads the exact prompt behind our fidelity judge and rewrites 500 records to satisfy it has made the sample score better and the corpus no better at all. So the weights, the deterministic checks, their thresholds and the composite formula are published and versioned; the judge prompts and the audit-slice construction are not. This post is the whole of that policy, and it will not quietly change — a rubric revision gets a version bump, a changelog, and certificates that state which version they were issued under.

Install and use

# requires python 3.11+
pip install s2l-assay

assay check ./sample.jsonl --modality text
assay check ./shard-*.parquet --modality trajectory --json > assay.json
assay explain coverage        # what the dimension contains and how it is computed
assay weights --rubric r2     # the published weight table

Input formats are JSONL, Parquet and CSV. Modality changes which checks run and how they are weighted within a dimension — trajectory data is screened for step-count distribution and recovery presence, computer-use data additionally for screenshot dedup, text for the standard set. Sampling is deterministic given a seed, and the sample hash is printed so that the same sample can be referred to later.

That hash matters more than it looks. It is what makes an assay result checkable after the fact: the certificate we issue names the hash of the sample that was scored, and s2l-verify checks a delivered file against it. A seller cannot be assayed on one sample and deliver another.

Roadmap, and what we want back

Rights annotations as a first-class input, so the heaviest dimension stops being null in the common case. That is the provenance-ops work, and it will land as operators for existing curation pipelines rather than as another standalone tool.
More modalities: audio and time series are next; environments get their own rubric rather than being forced into this one.
Threshold calibration. The current thresholds come from our intake queue, which is small and biased towards the sellers who found us. If you run assay on a corpus and the numbers disagree with your own measurements, that is the most useful bug report we can get.

Reference: the dimensions and weights are set out in full in What your data is actually worth to an AI lab.

Run a full assay All writing