Build an RL Gym · Part 2 of 6

Wrapping a real application

Determinism is the product

The claims application from part one was written to process claims, not to be an environment. It has a database, background jobs, a clock, outbound integrations and a login. Every one of those is a source of nondeterminism, and nondeterminism is the difference between an environment a lab can train against and a demo. This part is about making reset(seed) a promise you can keep.

The isolation boundary

One container per episode, holding the entire application and its state. Nothing shared between episodes except the image itself.

FROM claims-app:2.14-pinned          # exact tag, never :latest

# everything the app talks to lives inside the image
COPY --from=postgres-seed /seed.sql /seed/
COPY fixtures/ /fixtures/

ENV TZ=UTC \
    APP_CLOCK=fixed \                # the clock is injected, see below
    APP_OUTBOUND=blackhole \         # no real network egress
    APP_LOGIN=bypass-token

# the episode harness, not the app
COPY harness/ /harness/
ENTRYPOINT ["/harness/serve.py"]

The harness exposes the four-part contract over a small HTTP API — POST /reset, POST /step, GET /state — so the trainer never talks to the application directly. That indirection is what lets you change the application without breaking every task.

The five sources of nondeterminism, and what to do about each

Source	Symptom	Fix
The clock	"Submitted 3 days ago" changes; date validations pass on Monday and fail on Friday	Inject a fixed epoch per episode, derived from the seed. Never let the app call the system clock.
Ids and randomness	Claim references, UUIDs and ordering differ per run, so verifiers written against one run fail on the next	Seed the app's RNG; use deterministic id generation; always sort what you serialise.
Background jobs	A queue worker fires between steps and changes state under the agent	Run workers synchronously, or disable them and advance them explicitly as part of `step`.
External calls	A third-party check times out or returns different data	Record-and-replay fixtures. No egress at all in the episode container.
Rendering	Font metrics, animation timing and viewport differences change screenshots byte-for-byte	Pin viewport and fonts, disable animations, and prefer structured observations over pixels where you can.

Reset: snapshot, not teardown

The naive reset reinstalls the application and replays a seed script. It works and it takes 40 seconds, which is unaffordable when you are running thousands of episodes. Three approaches, in ascending order of engineering:

Container per episode. Simple, perfectly isolated, and startup-cost bound. Fine at the pilot stage — and if your image boots in two seconds you may never need more.
Database snapshot restore. Keep the container, restore the data layer from a template. In Postgres, CREATE DATABASE … TEMPLATE is typically a few hundred milliseconds. You must also reset anything living outside the database: caches, uploaded files, session state.
Filesystem snapshots. Copy-on-write at the storage layer restores the whole world, including files and caches, in milliseconds. The most robust and the most infrastructure.

Whichever you pick, the test is the same: run the identical action sequence twice from the same seed and diff the full state, not the screen. If anything differs, you do not have a reset — you have a probability distribution.

# the test that has to be in your CI from day one
def test_reset_is_deterministic():
    a = rollout(seed=7, actions=SCRIPT)
    b = rollout(seed=7, actions=SCRIPT)
    assert a.state_digest == b.state_digest
    assert a.observations == b.observations
    assert a.rewards == b.rewards

def test_seeds_differ():
    assert rollout(seed=7, actions=SCRIPT).state_digest != \
           rollout(seed=8, actions=SCRIPT).state_digest

The second test matters as much as the first. A "deterministic" environment that ignores its seed is deterministic and useless — every episode is the same task, and a model that memorises one claim scores perfectly.

State digests

You need a canonical digest of the environment's state to compare runs, verify outcomes and detect drift. Build it deliberately rather than hashing a database dump:

def state_digest(conn) -> str:
    """Stable across runs; sensitive to everything a task can change."""
    parts = []
    for table in SEMANTIC_TABLES:              # explicit allowlist
        rows = conn.execute(
            f"SELECT {COLUMNS[table]} FROM {table} ORDER BY id")
        parts.append(canonical_json(rows))     # sorted keys, no floats
    return sha256("\n".join(parts))

Exclude what changes for reasons the task does not care about: audit timestamps, sequence counters, session rows. Include everything the task can change. Getting this allowlist right is most of what makes the verifiers in part five possible.

Rights, before you build any of it

An environment wrapping an application inherits that application's licence. Three questions to answer before writing the Dockerfile, because the answers can end the project:

May you redistribute the application? Selling an environment usually means shipping an image containing it. A licence permitting internal use frequently does not permit that.
Whose data is in the fixtures? Seeded records drawn from production carry every obligation the production data carries. Synthetic fixtures avoid the entire problem and are usually better tasks anyway.
Are third-party integrations replayable? Recorded responses from a paid API may not be redistributable, even as fixtures.

If any answer is no, build the environment over an application you control or an open-source equivalent. Discovering this after the environment works is a painful way to learn it — and it is the first thing env-assay checks.

Next: Reward design — the part that decides whether any of this produces a usable training signal.

List an environment All writing