Skip to main content

​
Every rollout starts clean

createState() is called fresh for each rollout, so one run cannot contaminate the next. That is what makes repeated runs comparable and what lets you attribute a difference in score to the agent rather than to leftover state. It is also why createState() must return a newly cloned object. Return a cached import directly and rollout two inherits rollout one’s mutations — see State.

​
Repeating a task

Agents are not deterministic. One passing run tells you a task is achievable, not that your agent achieves it.
Rollouts run sequentially, each from a fresh world, and each writes its own run directory. The spread is the interesting part: Best 1.00 / Worst 0.20 means an agent that sometimes finds the answer, which is a different problem from one that never does.

​
Comparing agents

--agent is just a path, so the natural way to compare two approaches is to point at each in turn against the same task and seed:
Because the world is identical each time, the difference in pass rate is attributable to the agent.

​
Limits

When either limit trips, the run’s signal aborts, further callTool calls stop, and the run terminates with max_tool_calls or timeout. A long-running loop should check signal.aborted and return what it has rather than spinning. Limits are recorded in run.json, so a run that hit one is legible after the fact rather than looking like an agent that gave up.

​
When a run does not complete

Every run ends with one of four reasons:
agent_error with 0 tool calls and a near-zero duration almost always means the agent never reached its model — usually a model server that is not running. A genuine model failure shows up as tool errors or a timeout, after some work.
A run that fails still produces artifacts. A rollout that dies before its first tool call writes a trace containing the task, the config and the failure, because an empty directory is the least useful thing to find when something broke.

​
Where results go

Run ids are collision-free, so concurrent runs and --runs sweeps never overwrite each other. See Run Artifacts.