data/ holds the raw facts of your world: ids, names, quantities, prices, statuses, dates. One JSON file per collection, named after it.
Facts only, never derived values
If a number can be computed from other fields, do not store it. ERP’s datasets hold quantities and unit prices.lineTotal, subtotal, taxAmount and totalAmount never appear in data/ — they are derived when the world is built. CRM’s stages.json stores a probability per stage; no opportunity stores its own weighted value.
Two reasons, and the second is the one that bites:
- A stored total goes stale. Edit a quantity in a dataset and the total beside it is now wrong, with nothing to catch it.
- A verifier reading a stored total is grading its own copy of the answer. The whole point of deterministic verification is that expected values are derived from the world, not read out of it. Store the answer in the data and the check becomes circular.
opportunities.json
amount × the probability on STG-003. It lives nowhere on disk — it is computed on read, because the agent can move the opportunity to another stage mid-rollout.
Where derived values belong depends on whether they can change during a run. See State for the distinction between deriving once at load and exposing a helper.
Managing datasets
Data is the one part of an environment the CLI fully manages, because it is inert — a JSON file cannot become an agent capability.add and update take the content either inline or from a file:
--json to any of them for machine-readable output.
How data becomes a world
environment.ts imports the datasets, assembles them, and derives whatever is computed at load:
environment.ts
indexById turns a JSON array into a Record<string, T> keyed by id, which is how tools look rows up without scanning.
structuredClone is not optional. Node caches imported JSON modules for the life of the process, so without the clone the second rollout in a --runs 5 sweep starts from whatever the first one left behind.
Seeding realistically
Datasets are the input to every task and every verifier, so their shape decides what your environment can actually test.- Give tasks a unique answer. If two reps tie for “most stalled deals”, the task is ambiguous and the verifier cannot grade it. CRM’s seed data is arranged so each question has exactly one correct answer.
- Include the awkward rows. A deactivated user who still owns open work, an invoice that does not match its purchase order, a lead already converted. Agents fail on edge cases, which is what you want to measure.
- Keep ids stable and readable.
USR-004,OPP-012,VINV-103. They appear in task instructions, verifier code, traces and diffs.
Silo