> ## Documentation Index
> Fetch the complete documentation index at: https://docs.burn0.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Verifiers

> Deterministic grading of the world a rollout left behind

A verifier decides whether a task was achieved. It receives the final world, the world the rollout started from, and the agent's output, and returns a list of checks.

No model is involved. Grading is ordinary TypeScript reading ordinary data, so the same rollout always scores the same.

```typescript verifiers/VER-002.ts theme={null}
import { check, defineVerifier, optional } from "@burn0/silo";

import type { State } from "../state.js";

export const ver002 = defineVerifier<State>({
  id: "VER-002",
  taskId: "TASK-002",
  name: "Departed rep's open deals moved to the new owner",
  check(final, initial, context) {
    const shouldMove = Object.values(initial.opportunities).filter(
      (opportunity) => opportunity.ownerId === "USR-004" && opportunity.status === "open",
    );

    const moved = shouldMove.filter(
      (opportunity) => final.opportunities[opportunity.id]?.ownerId === "USR-003",
    );

    return [
      check(
        "Every open deal of the departed rep moved to the new owner",
        shouldMove.length > 0 && moved.length === shouldMove.length,
        `${moved.length} of ${shouldMove.length} moved`,
      ),
      optional(
        "Deals the departed rep already closed kept their owner",
        closedUnchanged,
        "closed history preserved",
      ),
    ];
  },
});
```

## Required and optional checks

|                                   |                                                                       |
| --------------------------------- | --------------------------------------------------------------------- |
| `check(label, passed, detail)`    | **Required.** Failing one fails the task.                             |
| `optional(label, passed, detail)` | Corroborating evidence. Moves the reward, cannot fail the task alone. |

Scoring is fixed so every environment grades the same way:

```
passed = no required check failed
reward = checks passed / total checks     (required and optional together)
```

A task scoring `0.60` while failing is normal — some optional checks passed. Pass is gated purely on required checks.

A verifier that returns **no required checks throws**. A task cannot be graded on corroborating evidence alone.

The `detail` string is what you read when something fails, so make it carry the numbers:

```
✗ Every open deal of the departed rep moved to the new owner
  0 of 2 moved to USR-003
```

## Grade state first, output second

**State-changing task** — check `finalState`, comparing against `initialState` where a delta matters.

**Answer-producing task** — derive the expected answer from `initialState` using your domain helpers, then compare against `context.agentOutput`.

```typescript verifiers/VER-004.ts theme={null}
function expectedWeightedTotal(state: State): number {
  return sumMoney(
    Object.values(state.opportunities)
      .filter((opportunity) => opportunity.status === "open")
      .map((opportunity) => weightedAmount(state, opportunity)),
  ).amount;
}

check(
  "The reported weighted total is correct",
  statesNumber(context.agentOutput, expectedWeightedTotal(initial)),
  `answered ${answered ?? "nothing"}, expected ${expected}`,
),
```

<Warning>
  **Never hardcode an answer that can be derived.** `507500` written into a verifier is correct until someone edits an amount in `data/`, at which point the verifier is confidently wrong. Deriving it through the same helpers the tools use means the data and the grading move together.
</Warning>

Do **not** grade which tools were called, or in what order, unless the process itself is what the task asks for. There are usually several reasonable routes to the same outcome, and scoring the route measures conformity rather than competence.

## Derive from the initial state, not the final one

`initial` is the world as seeded; `final` is what the agent left. Work out what *should* have happened from `initial`:

```typescript theme={null}
const shouldMove = Object.values(initial.opportunities).filter(
  (o) => o.ownerId === DEPARTED_ID && o.status === "open",
);
```

Deriving that set from `final` would be circular — an agent that closed the deals instead of reassigning them would produce an empty set and trivially satisfy the check. Reading `initial` means the expectation is fixed before the agent touches anything, and adding another deal to the seed data strengthens the check automatically.

## Look up by property, not by id, where you can

```typescript theme={null}
const wonStage = Object.values(final.stages).find(
  (stage) => stage.isClosed && stage.probability === 100,
);
```

`VER-003` finds the closed-won stage by what it *is* rather than hardcoding `STG-005`, so renumbering the pipeline in `data/stages.json` does not break grading.

## Reading a free-text answer

Answer-producing tasks need a number or a name out of prose. Keep that parsing in one place rather than reinventing it per verifier:

```typescript verifiers/shared.ts theme={null}
/** Every number in the text, with currency symbols and separators removed. */
export function numbersIn(output: string): number[] {
  const matches = output.replace(/[$,]/g, "").match(/-?\d+(?:\.\d+)?/g);

  return matches ? matches.map(Number) : [];
}

/** True when the output states `value` anywhere, within `tolerance`. */
export function statesNumber(output: string, value: number, tolerance = 0.01): boolean {
  return numbersIn(output).some((candidate) => Math.abs(candidate - value) <= tolerance);
}
```

Be generous about formatting and strict about the value. `$507,500.00` and `507500` are the same answer; `507501` is not.

Checking a wrong answer was *not* given is sometimes as valuable as checking the right one appears — `VER-006` asserts the deactivated rep is not named, because he is the trap.

## Scaffolding

```bash theme={null}
npx @burn0/silo verifier add VER-007 --env demo --task TASK-007 --name "Invoice escalated"
```

```
Created verifiers/VER-007.ts
Registered ver007 in verifiers/index.ts
Next: replace the failing TODO check in verifiers/VER-007.ts.
```

The generated check **fails on purpose**:

```typescript theme={null}
return [
  check(
    "TODO: define success condition",
    false,
    "Verifier \"VER-007\" is not implemented yet.",
  ),
];
```

An unimplemented verifier must never report a passing run. A stub that returned `true` would mark every agent correct and look like a working gate.

Like tools, verifiers are **explicitly registered** in `verifiers/index.ts` — a file on disk is not a grader until it is in the barrel.

## Verifiers are not the whole gate

Deriving expectations protects against stale data: edit an amount in `data/` and grading follows. It does **not** protect against changed logic.

If a verifier derives its expected answer through the same helper the tools use, and that helper is wrong, both sides move together and the check still passes. What catches that is a recorded baseline — an artifact frozen when the behaviour was known good, which cannot follow a formula change.

Derivation and baselines cover different failures. An environment you intend to rely on wants both.