Tools & Code · v0.1 draft rubric
env-assay — grading RL environments
RL environments have become a traded asset with no standards attached. There is no provenance convention, no rights position anyone checks, no assessment of reward quality, and no contamination screening. Buyers are purchasing environments on a README and a demo video. Meanwhile the strongest public evidence that this matters — roughly half of agent patches passing a carefully-built benchmark would not have been merged by maintainers — says plainly that reward quality needs auditing. This is our rubric, one modality over.
Why environments need their own seven
The dataset rubric does not transfer. An environment is executable, which introduces failure modes a corpus does not have: it can be nondeterministic, its reward can be gamed, its tasks can be unsolvable, and it can wrap software you have no right to redistribute. Two dimensions carry over in spirit — rights and coverage — and five are new.
| # | Dimension | Weight | The question |
|---|---|---|---|
| 1 | Rights on the underlying application | 22 | May the wrapped software, its fixtures and its recorded integrations be redistributed inside an image, for this use? |
| 2 | Reward quality | 20 | What is the measured false-pass rate against human judgement? What is programmatic, what is rubric, what is judge — and is the judge capped? |
| 3 | Task provenance | 16 | Generated, authored, or derived from production records? If derived — whose, under what basis? Who wrote the evaluation split? |
| 4 | Determinism and reset fidelity | 14 | Same seed, same trajectory, same reward — verified by replay, not asserted. And do different seeds actually differ? |
| 5 | Coverage of the task space | 12 | Effective task count after clustering, difficulty distribution, and whether hard cases exist at all. |
| 6 | Contamination | 10 | Overlap of task content with public benchmark suites and with likely pretraining corpora. |
| 7 | Failure representation | 6 | Are failure, recovery and genuinely ambiguous cases present — or is every task a clean success path? |
What the tool can check mechanically
$ env-assay ./claims-env-1.2.0 --episodes 200
claims-env 1.2.0 · gymnasium adapter · container image pinned
determinism
replay identical (seed 7, 3 runs) pass
seeds produce distinct states pass (200/200 distinct)
wall-clock independence pass (clock injected)
network egress during episode pass (none observed)
reward
components declared 4 (2 programmatic, 1 rubric, 1 judge)
judge weight 0.15 ok (cap 0.25)
reward decomposition logged pass
constant-agent score 0.31 ← task balance is acceptable
lazy-agent score 0.12
thrash-agent score 0.44 ! within 0.1 of efficient agent
tasks
nominal 48,000
effective (cluster-adjusted) 6,900 ratio 0.14 ! low diversity
solver-verified solvable 47,102 (898 dropped, reported)
metadata leakage classifier 0.51 auc pass (base rate 0.50)
public-benchmark n-gram overlap 0 hits
provenance
application licence declared: internal-only ✗ BLOCKING
fixtures synthetic
eval split authorship declared, uncredentialed !
composite — blocked: dimension 1 fails absolutelyDeterminism, the adversarial baselines, diversity, solvability, leakage and contamination are all mechanical, and they are most of what nobody currently runs. Reward-quality agreement against human graders and task provenance need evidence supplied by the author — the tool records what was supplied and what was not, exactly as assay does for datasets.
Two findings that recur
Diversity ratios are low and unreported. Environments advertise nominal task counts in the tens of thousands and cluster down to a few hundred effective tasks. Nobody is lying; nobody has measured. Publishing the ratio would change how these assets are priced overnight.
The thrash agent is the tell. If an agent that opens everything and clicks everything scores close to an efficient correct agent, the reward is not measuring competence — and this is the cheapest test in the whole rubric. It takes minutes and almost nobody runs it before publishing.
Rights is blocking here too
An environment wrapping software you may not redistribute is not a low-scoring asset. It is an asset that cannot be sold, and the score is withheld rather than computed.
This is the same absolute-failure logic as chain of title for datasets, and it is more common with environments because the wrapping is usually done by engineers solving a technical problem, months before anyone asks a commercial question. Answer it first — part two of the gym series puts the three questions before the Dockerfile for exactly this reason.
Status, honestly
This rubric is a draft and the weights are less well-founded than the dataset ones, because we have far less outcome data behind them. We are publishing it now anyway, for comment, because environments are being traded today and the alternative to an imperfect public standard is the current situation, which is no standard at all.
What would change our minds: evidence that a dimension does not predict buyer outcomes, a failure mode the seven do not cover, or a mechanical check that catches something these miss. All three are useful and all three get published.
Companion reading: the six-part Build an RL Gym series, which is this rubric written as a construction guide rather than as a grading sheet.