Build an RL Gym · Part 4 of 6

Task generation at scale

Generate the world, derive the task

Hand-authoring tasks produces a few hundred good ones and stops. Procedural generation produces a hundred thousand, of which a large fraction are subtly broken in ways that are invisible until a model exploits them. The rule that resolves this: generate the world, then derive the task and its answer from the world you generated. Never generate a task and then try to build a world that satisfies it.

The generator, in the right direction

def generate_episode(seed: int) -> Task:
    rng = Random(seed)

    # 1. sample a world — a policy, a claimant, a history
    policy   = sample_policy(rng)          # coverage, excess, exclusions
    claimant = sample_claimant(rng, policy)
    history  = sample_claim_history(rng, claimant)

    # 2. sample an incident and let the rules decide the outcome
    incident = sample_incident(rng, policy)
    outcome  = POLICY_ENGINE.adjudicate(policy, claimant, history, incident)

    # 3. render the world into the application's state
    state = render_to_app(policy, claimant, history, incident)

    # 4. the task is what the world already implies
    return Task(
        seed          = seed,
        initial_state = state,
        gold_outcome  = outcome.decision,     # derived, never invented
        gold_payable  = outcome.payable,
        decisive_fact = outcome.decisive_fact,
        key_document  = outcome.evidence_doc,
        difficulty    = estimate_difficulty(outcome),
    )

The ground truth is a by-product of a rules engine that adjudicated a world it fully knows. It is correct by construction. The alternative — writing a scenario and asking a model what the right answer is — produces ground truth that is exactly as reliable as the model, which is to say not reliable enough to train against.

Difficulty has to be a knob, not an accident

Naive sampling produces a distribution dominated by the easy case, because reality is. Control the axes explicitly:

Then sample deliberately across the grid and record the coordinates on every task. A buyer will ask for the difficulty distribution, and "we sampled randomly" means you do not have one.

The four things that break

1 · Unsolvable tasks

The generator produces a world where the decisive fact is not reachable through any sequence of available actions — the document exists but nothing links to it, or the field is on a screen the agent cannot navigate to. The model gets punished for a failure that is yours.

Guard: a solver. Write a scripted agent that has privileged access to the world model and plays the intended solution path. Every generated task must be solved by it before entering the set. Tasks the solver cannot complete are dropped and counted; a rising drop rate means the generator has drifted.

2 · Trivially solvable tasks

The reverse: a spurious shortcut. Approved claims get a longer description field; the correct outcome correlates with a claim-number range; escalations always come from one region because of how you sampled.

Guard: train a small classifier on the task metadata alone — no agent, no environment, just the surface features — to predict the gold outcome. If it does better than the base rate, you have a leak. This takes ten minutes and finds problems that survive months of manual inspection.

3 · Homogeneity

A hundred thousand tasks from one template are one task with a hundred thousand seeds. Models memorise the schema and generalise nothing.

Guard: measure it. Embed each task's rendered initial state, cluster, and report cluster mass — the same effective-volume calculation applied to datasets. Report effective task count alongside nominal count; the ratio is a quality metric buyers understand instantly.

4 · Contamination

Generated worlds built from public sources — real policy documents, scraped forms, public case descriptions — can overlap material that is already in pretraining corpora or in public benchmarks. The model then knows the answer without solving the task.

Guard: n-gram overlap screening of rendered task content against public benchmark suites, plus synthetic-by-construction fixtures wherever possible. Synthetic worlds are also how you avoid the rights problem from part two — the same decision solves both.

The generation pipeline, end to end

seeds ──▶ generate_episode ──▶ solver check ──▶ leak classifier
                                    │                 │
                              drop + count       drop + count
                                    ▼                 ▼
                            diversity cluster ──▶ difficulty balance
                                                        │
                                                        ▼
                                              held-out split (never
                                              shown during training)
                                                        │
                                                        ▼
                                                  task manifest

Every stage counts what it drops, and the counts are published with the environment. An environment whose author cannot tell you how many generated tasks were discarded and why has not run these checks.

Hold out a human-authored set

Generate the training tasks; author the evaluation set by hand, from real cases, with people who do the work. A few hundred is enough. The generated distribution has systematic quirks by construction, and a model evaluated on the same generator that trained it measures how well it fits your generator rather than the job.

This is also the artefact that makes the environment saleable: a human-authored, verified evaluation set is what lets a buyer believe the training set. It is worth more per task than everything else you generate.

Next: Verification — why a passing check is not a solved task, and what to do about it.