Tools & Code · v0.1

provenance-ops — rights operators for existing pipelines

Apache-2.0Operators, not a product

The data curation stack is mature. GPU-accelerated deduplication, quality classifier ensembles, PII modifiers, decontamination, recipe libraries — the frameworks people actually run cover all of it well. Not one of them touches rights. They will tell you a record is a near-duplicate, is low quality, or contains an email address. None will tell you whether you are allowed to train on it. That is 24% of our rubric and effectively 100% of what ends deals at counsel, and it is missing from every pipeline in production.

Operators, not a platform

The strategic choice here is deliberate: this ships as a set of operators for the frameworks people already use, not as another standalone tool. Rights annotation is worth something only if it runs where the data is already being processed, and every install is distribution. Being the rights layer inside the two most-used curation frameworks is worth more than owning a product nobody adds to their pipeline.

from provenance_ops import (
    ChainOfTitleAnnotator, LicenceCompatibilityFilter,
    RightsReservationDetector, Article53Summariser, ProvenanceManifest,
)

pipeline = Sequential([
    ChainOfTitleAnnotator(sources="sources.yaml", on_missing="fail"),
    RightsReservationDetector(signals=["robots", "tdm", "noai"]),
    LicenceCompatibilityFilter(use="model-training", strictness="counsel"),

    ExactDuplicates(), FuzzyDeduplication(),   # your existing operators,
    QualityClassifier(), PiiModifier(),        # unchanged

    Article53Summariser(out="training-data-summary.md"),
    ProvenanceManifest(sign_with=KEY, out="manifest.jsonl.sig"),
])

Rights operators run first. Filtering a record you had no right to process is wasted compute, and annotating after deduplication loses the source attribution of whichever near-duplicate was dropped.

The five operators

ChainOfTitleAnnotator

Attaches per-record source, licence and consent basis from a declared source map, and — the important part — fails loudly on records that carry none. The default is on_missing="fail" because the alternative silently produces a corpus where some unknown fraction has no rights position, which is precisely the state that kills deals six months later.

# sources.yaml
- match: {bucket: "s3://intake/partner-northside/**"}
  source: "partner:northside-clinic"
  instrument: "DSA-2023-11 §4.2"
  consent_basis: "explicit-research-consent"
  permits: [model-training, evaluation]
  expires: 2029-11-01

- match: {bucket: "s3://intake/crawl-2024/**"}
  source: "web-crawl"
  instrument: "public-web"
  consent_basis: null
  permits: [research]          # note: NOT model-training

LicenceCompatibilityFilter

Drops records whose terms do not permit the declared use. Takes a strictness setting because the honest state of the world is that many licences are ambiguous about model training: permissive keeps anything not explicitly forbidden, counsel keeps only what is explicitly permitted. Both are legitimate; picking one silently is not. Every drop is counted and reported by reason.

RightsReservationDetector

Machine-readable opt-outs: robots directives, TDM reservation declarations, noai-style signals, and per-site policy files. Records the signal and the timestamp it was observed, because the compliance question is what was reserved at collection time, not what is reserved today. Where you have no observation from collection time, it says so rather than checking now and implying you checked then.

Article53Summariser

Emits a training-data summary skeleton from the annotations: sources, kinds of content, approximate proportions, instruments, and the reservation handling. It is a skeleton, not a filing — a human has to finish it — but it turns a document that currently takes weeks of archaeology into an artefact derived from the pipeline that built the corpus. See Article 53 for data owners for why this obligation reaches sellers at all.

ProvenanceManifest

A signed manifest over a processed shard: content hash, record count, per-source counts, the operator chain with versions and parameters, and drop counts by reason. The manifest is what lets a buyer verify that the corpus they received is the corpus that was processed, and lets you demonstrate what was excluded without disclosing the excluded records.

{"shard":"train-0007.parquet","sha256":"1b9e…","records":412880,
 "sources":{"partner:northside-clinic":401220,"web-crawl":11660},
 "dropped":{"licence-incompatible":8140,"reservation-honoured":2317,
            "no-chain-of-title":0},
 "operators":[{"name":"ChainOfTitleAnnotator","version":"0.1.2",
               "params":{"on_missing":"fail"}}, …],
 "signature":"…"}

What it does not claim

It is not legal advice and does not decide anything. It applies a policy you declare and records what it did. The permits field in your source map is a legal judgement made by a person; the operator only enforces it consistently.
It cannot find rights that were never recorded. If your corpus arrived with no source metadata, the annotator will fail on every record, which is the correct behaviour and an unwelcome discovery. There is no automated route out of undocumented provenance.
Detection of reservations is best-effort. Signals are heterogeneous and unevenly adopted; absence of a detected reservation is not evidence that none was expressed.

Why upstream adoption is the real test

If the curation frameworks will not take a rights layer, the ecosystem has decided that rights are not a data problem — and our 24% weighting is a thesis the market has rejected.

That is a falsifiable position and we would rather state it than hedge. The measure of success for this work is not stars or downloads; it is whether the operators end up merged upstream and running inside pipelines that were built before we existed. Everything about the design — operators rather than a platform, permissive licence, no service dependency, no account — follows from that being the goal.

Companion reading: Chain of title, in practice for the documents behind the annotations, and assay for the deterministic checks that run alongside them.

List a dataset All writing