Debug a Failed Run

A failure tells you more than a pass, but only if you read the right file. The order below goes from cheapest to most detailed.

1. Which required check failed

result.json names it, and the CLI prints it:

  Result        FAIL
  Reward        0.60

  Checks        3 / 5
  Required      1 / 3

  Failed
  ✗ Every open deal of the departed rep moved to the new owner
  ✗ The departed rep owns no open deals

The detail on each check carries the numbers behind the verdict — 0 of 2 moved to USR-003 tells you the agent did nothing, while 1 of 2 moved tells you it started and stopped.

2. Did the world change at all?

cat .silo/runs/<runId>/state-diff.json

{
  "scalars": [],
  "collections": {}
}

An empty diff on a state-changing task is the single most diagnostic signal in Silo. The agent read but never wrote. It may have described the work convincingly in its output — that is exactly the failure Silo exists to catch. If the diff is non-empty but the task still failed, the agent acted on the wrong thing. Compare the ids it touched against the ones the task was about.

3. How did the run end?

run.json

{ "terminationReason": "agent_error", "toolCallCount": 1, "durationMs": 3 }

Reason	What it means	Where to look
`completed`	The agent returned normally	The verifier checks — it did the wrong work
`max_tool_calls`	Budget exhausted	The trace, for a repeating call
`timeout`	Time exhausted	Usually a slow model, sometimes a loop
`agent_error`	The agent threw	`result.json`’s `error` field

  Result        FAIL
  Tool calls    1
  Ended early: agent_error

{ "error": "model server refused the connection" }

agent_error with 0 tool calls and a near-zero duration almost always means the agent never reached its model — a model server that is not running, or a missing API key. A genuine model failure shows up after some work, as tool errors or a timeout.

4. Read the trace

trace.jsonl is the record of what actually happened, in order.

node -e "require('fs').readFileSync('.silo/runs/<runId>/trace.jsonl','utf8').trim().split('\n').forEach(l=>{const e=JSON.parse(l);console.log(e.type, e.tool ?? '')})"

run_start
tool_call     list_opportunities
tool_result   list_opportunities
tool_call     reassign_opportunity
tool_result   reassign_opportunity
agent_output
run_end
verifier_result

What you see	What it means
No `tool_call` at all	The agent never decided to act, or never reached its model
The same call repeating	A loop with no exit condition
`tool_result` with `isError` repeatedly	The model is not reading the error messages
Calls stop abruptly mid-task	A limit tripped — check `terminationReason`
`agent_output` with an empty diff	Work described but not done

Common failures

A runaway loop

  Result        FAIL
  Tool calls    10
  Ended early: max_tool_calls

The trace shows the same call ten times. The agent is not incorporating results into its next decision. Cap it low while debugging — --max-tool-calls 10 turns a two-minute timeout into an instant, readable failure.

The agent keeps sending bad arguments

{
  "output": {
    "code": "invalid_input",
    "message": "Invalid arguments for \"get_opportunity\": \"opportunityId\" is required; \"opportunityID\" is not a parameter; expected one of: opportunityId."
  },
  "isError": true
}

Silo already names the correct parameter. If the model repeats the mistake, the problem is usually your loop — check that you are feeding result.output back into the conversation rather than discarding errors.

The right answer, graded wrong

Check what the verifier expected:

✗ The reported weighted total is correct
  answered 507500, expected 507499.99

Rounding, currency symbols and thousands separators are the usual culprits. Parse generously and compare with a tolerance.

A task nothing can solve

If no agent can pass, the environment may be at fault rather than the agent. Ask:

Is the information reachable? If the task needs to know which users are inactive and no tool reports it, the task is unsolvable no matter how good the model is. Missing tools look like agent failures.
Is the answer unique? If two reps tie for “most stalled deals”, a correct agent can still fail.
Does the instruction contain what a person would need? Names, ids, and any parameter that is a decision rather than a deduction.

Run the task with a hand-written agent that does exactly the right thing. If that fails, the environment is wrong.

Is it the agent or is it chance?

One failure proves nothing about a non-deterministic agent.

npx @burn0/silo run --env demo --task TASK-002 --agent ./silo.agent.ts --runs 5

  Passed        3 / 5
  Mean reward   0.74
  Best          1.00
  Worst         0.20

Best 1.00 means the task is solvable and your agent can solve it — this is a reliability problem. If every run sits at the same low score, it is a capability or environment problem, and repeating it will not tell you more.

Before blaming the agent

npx @burn0/silo env validate --env demo

An environment that does not compile can still run, because type-only imports are erased before execution. If validation fails, fix that first — the run you are debugging may be meaningless.

Getting Started

Environments

Running Agents

Guides

Reference

Debug a Failed Run

1. Which required check failed

2. Did the world change at all?

3. How did the run end?

4. Read the trace

Common failures

A runaway loop

The agent keeps sending bad arguments

The right answer, graded wrong

A task nothing can solve

Is it the agent or is it chance?

Before blaming the agent

​1. Which required check failed

​2. Did the world change at all?

​3. How did the run end?

​4. Read the trace

​Common failures

​A runaway loop

​The agent keeps sending bad arguments

​The right answer, graded wrong

​A task nothing can solve

​Is it the agent or is it chance?

​Before blaming the agent

1. Which required check failed

2. Did the world change at all?

3. How did the run end?

4. Read the trace

Common failures

A runaway loop

The agent keeps sending bad arguments

The right answer, graded wrong

A task nothing can solve

Is it the agent or is it chance?

Before blaming the agent