Build an RL Gym · Part 1 of 6

What an environment actually is

Six parts, one environment carried end to end

This series builds one environment from a real application and carries it to something a lab would buy. The application is a claims-processing workflow — a queue of submitted insurance claims, a form-driven adjudication interface, a document store, and an escalation path. Deliberately not a terminal, not a browser game, and not another SWE-bench clone: the environments that are scarce are the ones wrapping software that real organisations actually run.

The four-part contract

An environment is not a dataset and it is not a simulator. It is four things an agent can interact with, and nothing else:

The Gymnasium API expresses exactly this and is worth adopting even if you never touch a classic RL algorithm, because every trainer in this ecosystem speaks it:

class ClaimsEnv(gym.Env):
    def reset(self, *, seed=None, options=None) -> tuple[Obs, dict]:
        """Return the environment to a known state and hand back the
        first observation. Same seed → same claim, same queue, same
        document set. Everything in part 2 exists to make this true."""

    def step(self, action: Action) -> tuple[Obs, float, bool, bool, dict]:
        """Apply one action. Returns (observation, reward, terminated,
        truncated, info). `terminated` means the task reached an end
        state; `truncated` means we ran out of budget. Conflating the
        two is the most common bug in home-made environments."""

Four ways LLM agent environments are different

Everything written about classic RL environments assumes properties that do not hold here, and each broken assumption has a practical consequence.

1 · The action space is language, not a fixed set

Classic RL has k discrete actions or an n-dimensional continuous vector. An LLM agent emits text — usually a tool call with arguments, sometimes prose. The space is unbounded and most of it is invalid.

Consequence: parsing and validation are part of the environment, not an afterthought. Decide explicitly what happens on a malformed action: reject with an error observation and burn a step (our choice), or terminate. Silently ignoring it teaches the agent that malformed actions are free.

2 · Episodes are tens of steps, not millions

Classic RL runs billions of cheap steps. An agent episode here is 10–100 steps, each involving a model call costing real money and seconds of latency.

Consequence: sample efficiency is not an optimisation, it is the design constraint. It also means your environment will be run far fewer times than a classic one — so per-episode quality matters more than throughput, and one badly-specified task pollutes a meaningful fraction of training signal.

3 · Observations are for a reader, not a sensor

A pixel buffer is a pixel buffer. An LLM observation is a rendering decision: screenshot, DOM, accessibility tree, structured state, or some combination — and how you serialise it changes measured performance substantially. Two environments over the same application can produce very different scores.

Consequence: the observation format is part of the environment's identity and belongs in its documentation. We emit structured state plus an accessibility tree by default, with screenshots optional, and we say so — a buyer comparing environments needs to know they are not comparing like with like.

4 · Reward is sparse, terminal, and usually judged

There is no dense shaping signal in a claims workflow. Nothing meaningful can be said about step 7 in isolation. The reward arrives at the end and often requires judgement about whether the outcome was right and appropriately reached.

Consequence: reward design is the hard part of the whole exercise and gets its own part. It is also the dimension buyers scrutinise hardest, because a weak reward makes everything else worthless.

The minimum viable environment

Before any Docker, any task generator, any verifier — write this and run it by hand:

from dataclasses import dataclass

@dataclass(frozen=True)
class Obs:
    screen: str          # accessibility-tree rendering of the current view
    claim: dict          # structured state of the open claim
    documents: list[str] # ids the agent may open
    step: int
    budget: int

ACTIONS = {
    "open_document":  {"doc_id": str},
    "set_field":      {"field": str, "value": str},
    "request_info":   {"reason": str},
    "decide":         {"outcome": "approve|deny|escalate", "justification": str},
}

def step(state, action):
    if action["type"] not in ACTIONS:
        return state, 0.0, False, False, {"error": "unknown action"}
    ...

Then drive it yourself for ten episodes, as a person, typing actions. You will discover that two fields you thought were observable are not, that one action you did not include is required to complete the task, and that your step budget is wrong by a factor of three. Every one of those discoveries is cheaper now than after you have generated ten thousand tasks.

What a buyer will ask about this layer

• What exactly is in the observation, and can the agent see anything a human operator could not?
• Is the action space complete — can every task actually be solved with the actions provided?
• What happens on a malformed action, and is that documented?
• Is terminated distinguished from truncated? (This is a five-second check that tells them how carefully the rest was built.)

Next: Wrapping a real application — Docker isolation, state reset, and making reset(seed) mean something.