Skip to main content
Every rollout produces two numbers that answer different questions. Passed — did the agent achieve the task? A boolean, gated entirely on required checks. Reward — how much of what we hoped for did it get? A fraction between 0 and 1.
The arithmetic lives in Silo rather than in each verifier, so every environment grades the same way.

​
Required and optional checks

A verifier returns a list of checks, each either required or optional.
Required checks are the definition of done. Optional checks are how you tell a near miss from a disaster — did it do the right thing sloppily, or the wrong thing entirely?
A verifier that returns no required checks throws. A task cannot be graded on corroborating evidence alone, so an environment cannot accidentally ship a verifier that can never fail.

​
A failing run can still score well

This surprises people, and it is the point:
Three of five checks passed, so the reward is 0.60. But two required checks failed, so the task failed. The reward says “this agent was partly on track”; the result says “it did not do the job.” Use passed for gating and reward for comparing. An agent that improves from 0.20 to 0.60 while still failing is making real progress, and a pass-rate-only view is blind to it.

​
Reading a result

result.json
failedRequired lists the labels that failed, which is usually all you need to know what went wrong. detail carries the numbers behind each verdict — a well-written verifier makes answered 507499, expected 507500 obvious at a glance.

​
Comparing across rollouts

The spread matters more than the mean. Best 1.00 / Worst 0.20 describes an agent that can solve the task but does not reliably — a different problem from one that consistently scores 0.60 and never passes. The first needs reliability work, the second needs capability work.

​
What not to score

Do not grade which tools were called, or in what order, unless the process itself is what the task asks for. There are usually several reasonable routes to the same outcome, and scoring the route measures conformity rather than competence. toolCalls and toolErrors are recorded because they are useful diagnostics — an agent that took 40 calls to do a 3-call job is worth looking at — but they do not affect the score.