Build an RL Gym · Part 5 of 6

Verification

A passing test is not a solved task

The most useful published result in this area is uncomfortable: when maintainers reviewed agent-generated patches that passed the tests on SWE-Bench Verified — a benchmark built specifically to be trustworthy — roughly half would not have been accepted into the codebase. The tests passed. The work was not acceptable. Every environment builder should assume the same gap exists in their own verifier until they have measured it.

Why the gap exists

A checker asserts over observable end state. "Acceptable work" is a much larger predicate that includes things the end state does not record:

The checker sees	The reviewer also sees
The decision field equals the gold outcome	Whether the reasoning that produced it was sound or a lucky guess
The required document was opened	Whether it was actually used
No policy flags were raised	Whether a flag should have been raised and the agent worked around the check
State reached the target	What else changed on the way there
The task terminated in budget	Whether the approach generalises or is specific to this fixture

In our claims environment the concrete version is an agent that discovers it can set the outcome field directly through a form the workflow does not intend, skipping the adjudication path entirely. Final state: correct. Trajectory: a bypass of the control the whole task exists to teach.

Hybrid verification

Four layers, cheapest first, each catching what the previous one cannot:

State assertions. The programmatic checker from part three. Necessary, fast, and insufficient.
Negative assertions — what must not have changed. The most underused layer and the highest yield. Anything outside the task's legitimate footprint must be untouched: other claims, audit records, policy configuration, user permissions. Most exploits show up here first, and this is why the state digest from part two allowlists tables explicitly.
```
def negative_checks(before, after, task):
    footprint = task.allowed_mutations          # declared per task
    diff = state_diff(before, after)
    return {"no_out_of_scope_writes": diff.keys() <= footprint,
            "audit_intact": diff.get("audit_log", []) == expected_audit(task),
            "no_config_change": "policy_config" not in diff}
```
Trajectory assertions. Properties of the path, not the destination: the evidence was opened before the decision; no action was repeated more than n times; the adjudication endpoint was reached through the workflow rather than directly. These encode the process the task is meant to teach.
Narrow judge calls. Only for the residue that genuinely requires reading — is the justification grounded in what was actually seen — and always as bounded questions with the reference supplied, never "grade this trajectory".

Measuring your verifier against people

This is the part almost nobody does, and it is what makes an environment worth buying rather than downloading.

Sample 200 trajectories across the score range, weighted toward passes — the failures are rarely the interesting disagreements.
Have two qualified people independently judge each one as acceptable or not, blind to the verifier's output.
Report inter-rater agreement first. If your humans disagree with each other, your verifier cannot be better than that ceiling and you should fix the specification before the verifier.
Report verifier–human agreement, and both error directions separately.

verifier_agreement.json
{
  "n": 200,
  "human_inter_rater_kappa": 0.81,
  "verifier_vs_human_agreement": 0.88,
  "false_pass_rate": 0.09,   # verifier said pass, humans said no
  "false_fail_rate": 0.03,   # verifier said fail, humans said yes
  "top_false_pass_causes": [
    "justification restates the claim without citing the decisive fact",
    "correct outcome reached without opening the evidence (23% of these)"
  ]
}

False passes are the number that matters. A false fail costs a training signal. A false pass actively teaches the wrong behaviour, and it is the quantity the SWE-Bench result is about. Publish it. An environment shipping a measured 9% false-pass rate is more valuable than one shipping an unmeasured claim of correctness, and any buyer sophisticated enough to matter knows that.

The exploit log

Keep a running record of every reward hack you have found and closed. It is the most persuasive artefact an environment can ship:

exploits.md
  2026-05-02  direct POST to /claims/{id}/outcome bypasses adjudication
              → closed: endpoint requires a workflow token issued at
                the review step; trajectory assertion added
  2026-05-19  agent escalates everything; escalation scored as "safe"
              → closed: escalation on a clear-cut claim now scores 0,
                distribution rebalanced to 15% genuine escalations
  2026-06-11  justification passes rubric by quoting the policy verbatim
              → closed: cites_policy_clause now requires the cited clause
                to be the one the engine used

A buyer reading that log learns three things: the environment has been adversarially probed, the author understands their own reward, and the remaining exploits are the ones nobody has found yet — which is the honest state of every environment.

The standard to hold yourself to is not "the checker passes". It is "a competent practitioner reviewing this trajectory would sign it off". The gap between those two is the quality of your environment, and it is measurable.

Next: Publishing an environment — packaging, licensing, and what a buyer asks about provenance.

List an environment All writing