Build an RL Gym · Part 3 of 6

Reward design

The reward is the specification

An agent optimises what you measure, exactly, including the parts you did not mean. In our claims environment the naive reward — "1.0 if the recorded decision matches the ground-truth decision" — is satisfied perfectly by an agent that approves everything, on a task distribution where 70% of claims are approvable. It will score 0.7 without opening a single document, and every number downstream will be describing that.

Three mechanisms, three failure modes

	Programmatic checker	Rubric grading	LLM judge
How	Assert over final environment state	Fixed criteria, each independently checkable, scored and weighted	A model reads the trajectory and scores it
Costs	Nothing per episode; engineering per task	Some engineering, some judgement	A model call per episode
Strength	Exact, fast, reproducible, cannot be sweet-talked	Handles partial credit and process, still auditable	Handles anything, including free text
Fails when	Success has more than one valid form, or the state does not capture what matters	Criteria are vague, or the weights encode a preference nobody agreed	Always, silently, in a direction correlated with the agent's fluency

Programmatic checkers: prefer them, and know their limit

In a claims workflow a great deal is mechanically checkable: was a decision recorded, does it match the correct outcome, was the required document opened before deciding, was the excess calculated correctly, was an escalation raised when the policy demanded one.

def check(final_state, task) -> dict:
    return {
        "decision_correct":  final_state.claim.outcome == task.gold_outcome,
        "evidence_seen":     task.key_document in final_state.opened_documents,
        "amount_correct":    abs(final_state.claim.payable
                                 - task.gold_payable) <= 0.01,
        "no_policy_breach":  not final_state.audit.flags,
    }

The limit is expressiveness. "Was the justification adequate?" is not assertable, and in this domain the justification is half the job. A checker that ignores it teaches the agent that justifications are decorative.

Rubric grading: the middle that most people skip

A rubric decomposes a judgement into criteria that are each individually checkable — some programmatically, some by a judge with a much narrower question. It is not "ask the model for a score out of ten"; it is a list where every line is nearly binary:

justification_rubric:
  - id: cites_policy_clause
    check: regex(r"clause \d+\.\d+") and clause_exists(cited)
    weight: 0.3
  - id: references_the_document
    check: mentions(final_state.opened_documents)
    weight: 0.2
  - id: states_the_decisive_fact
    check: judge("Does the justification state the fact that decided the
                  outcome, namely: {task.decisive_fact}?")
    weight: 0.4
  - id: no_unsupported_claims
    check: judge("Does the justification assert anything not present in
                  the claim record or the documents opened?")
    weight: 0.1

Two of the four lines are code. The two that need a model ask a narrow, grounded question with the answer supplied in the prompt — which is a very different task from "grade this". Narrow judge calls are dramatically more reliable than broad ones, and the whole rubric is auditable by a buyer.

LLM judges: the failure is not noise, it is bias

An open-ended judge fails in a specific and dangerous direction. It rewards confident, fluent, well-structured output. Since the agent being trained also produces confident, fluent, well-structured output, the judge and the policy are correlated — and the training loop optimises for the correlation.

Known judge pathologies, all of which have to be measured rather than assumed away: position bias in pairwise comparisons, length bias, self-preference for the judge's own family, and sycophancy toward assertive phrasing. If you use a judge, you owe the buyer an agreement study against human graders on a held-out slice, with the number published. Without it, the judge is a reward function nobody has ever validated.

The three-tier reward we use

reward =  0.55 * outcome_correct        # programmatic, binary
        + 0.30 * process_score          # rubric, mostly programmatic
        + 0.15 * justification_score    # rubric with narrow judge calls
        - penalties

penalties:
  invalid_action        0.02 each   # malformed tool calls are not free
  decided_without_evidence  0.25    # the shortcut we most want to kill
  budget_exhausted      0.10

Design notes that matter more than the exact weights:

The judge component is capped at 15%. It cannot rescue a wrong decision and cannot sink a right one. A judge that can move the reward by more than a fraction is a judge you have to trust more than you can.
The shortcut has a named penalty. Deciding correctly without opening the evidence scores worse than deciding correctly with it, because the guess is not the skill we are paying for.
Partial credit is deliberate. Pure binary reward on a long-horizon task gives almost no gradient. Process credit is what makes a sparse task learnable.
Every component is logged separately in info. A scalar reward you cannot decompose is un-debuggable, and a buyer will ask for the decomposition.

Adversarial testing your own reward

Before you generate ten thousand tasks, try to break the reward. Write these four agents and run them:

The constant agent. Always approve. Its score is your reward's floor — if it is high, your task distribution is unbalanced.
The lazy agent. Decide immediately, correct guess or not, no documents. Should be heavily penalised; if it is not, the evidence requirement is not real.
The verbose agent. Correct decision, enormous justification stuffed with policy citations. Reveals length and keyword bias in the rubric.
The thrash agent. Open everything, click everything, then decide correctly. Should not outscore an efficient correct agent.

Publish those four scores with the environment. They are the cheapest possible evidence that the reward means something, and their absence is a signal in itself.

A reward function is a claim about what good work is. Yours will be read by people deciding whether to buy the environment, and by a model that will find every gap in it. The model reads more carefully.

Next: Task generation at scale — and what breaks when tasks are generated rather than authored.

List an environment All writing