Verifiers

A verifier decides whether a task was achieved. It receives the final world, the world the rollout started from, and the agent’s output, and returns a list of checks. No model is involved. Grading is ordinary TypeScript reading ordinary data, so the same rollout always scores the same.

verifiers/VER-002.ts

import { check, defineVerifier, optional } from "@burn0/silo";

import type { State } from "../state.js";

export const ver002 = defineVerifier<State>({
  id: "VER-002",
  taskId: "TASK-002",
  name: "Departed rep's open deals moved to the new owner",
  check(final, initial, context) {
    const shouldMove = Object.values(initial.opportunities).filter(
      (opportunity) => opportunity.ownerId === "USR-004" && opportunity.status === "open",
    );

    const moved = shouldMove.filter(
      (opportunity) => final.opportunities[opportunity.id]?.ownerId === "USR-003",
    );

    return [
      check(
        "Every open deal of the departed rep moved to the new owner",
        shouldMove.length > 0 && moved.length === shouldMove.length,
        `${moved.length} of ${shouldMove.length} moved`,
      ),
      optional(
        "Deals the departed rep already closed kept their owner",
        closedUnchanged,
        "closed history preserved",
      ),
    ];
  },
});

Required and optional checks


`check(label, passed, detail)`	Required. Failing one fails the task.
`optional(label, passed, detail)`	Corroborating evidence. Moves the reward, cannot fail the task alone.

Scoring is fixed so every environment grades the same way:

passed = no required check failed
reward = checks passed / total checks     (required and optional together)

A task scoring 0.60 while failing is normal — some optional checks passed. Pass is gated purely on required checks. A verifier that returns no required checks throws. A task cannot be graded on corroborating evidence alone. The detail string is what you read when something fails, so make it carry the numbers:

✗ Every open deal of the departed rep moved to the new owner
  0 of 2 moved to USR-003

Grade state first, output second

State-changing task — check finalState, comparing against initialState where a delta matters. Answer-producing task — derive the expected answer from initialState using your domain helpers, then compare against context.agentOutput.

verifiers/VER-004.ts

function expectedWeightedTotal(state: State): number {
  return sumMoney(
    Object.values(state.opportunities)
      .filter((opportunity) => opportunity.status === "open")
      .map((opportunity) => weightedAmount(state, opportunity)),
  ).amount;
}

check(
  "The reported weighted total is correct",
  statesNumber(context.agentOutput, expectedWeightedTotal(initial)),
  `answered ${answered ?? "nothing"}, expected ${expected}`,
),

Never hardcode an answer that can be derived. 507500 written into a verifier is correct until someone edits an amount in data/, at which point the verifier is confidently wrong. Deriving it through the same helpers the tools use means the data and the grading move together.

Do not grade which tools were called, or in what order, unless the process itself is what the task asks for. There are usually several reasonable routes to the same outcome, and scoring the route measures conformity rather than competence.

Derive from the initial state, not the final one

initial is the world as seeded; final is what the agent left. Work out what should have happened from initial:

const shouldMove = Object.values(initial.opportunities).filter(
  (o) => o.ownerId === DEPARTED_ID && o.status === "open",
);

Deriving that set from final would be circular — an agent that closed the deals instead of reassigning them would produce an empty set and trivially satisfy the check. Reading initial means the expectation is fixed before the agent touches anything, and adding another deal to the seed data strengthens the check automatically.

Look up by property, not by id, where you can

const wonStage = Object.values(final.stages).find(
  (stage) => stage.isClosed && stage.probability === 100,
);

VER-003 finds the closed-won stage by what it is rather than hardcoding STG-005, so renumbering the pipeline in data/stages.json does not break grading.

Reading a free-text answer

Answer-producing tasks need a number or a name out of prose. Keep that parsing in one place rather than reinventing it per verifier:

verifiers/shared.ts

/** Every number in the text, with currency symbols and separators removed. */
export function numbersIn(output: string): number[] {
  const matches = output.replace(/[$,]/g, "").match(/-?\d+(?:\.\d+)?/g);

  return matches ? matches.map(Number) : [];
}

/** True when the output states `value` anywhere, within `tolerance`. */
export function statesNumber(output: string, value: number, tolerance = 0.01): boolean {
  return numbersIn(output).some((candidate) => Math.abs(candidate - value) <= tolerance);
}

Be generous about formatting and strict about the value. $507,500.00 and 507500 are the same answer; 507501 is not. Checking a wrong answer was not given is sometimes as valuable as checking the right one appears — VER-006 asserts the deactivated rep is not named, because he is the trap.

Scaffolding

npx @burn0/silo verifier add VER-007 --env demo --task TASK-007 --name "Invoice escalated"

Created verifiers/VER-007.ts
Registered ver007 in verifiers/index.ts
Next: replace the failing TODO check in verifiers/VER-007.ts.

The generated check fails on purpose:

return [
  check(
    "TODO: define success condition",
    false,
    "Verifier \"VER-007\" is not implemented yet.",
  ),
];

An unimplemented verifier must never report a passing run. A stub that returned true would mark every agent correct and look like a working gate. Like tools, verifiers are explicitly registered in verifiers/index.ts — a file on disk is not a grader until it is in the barrel.

Verifiers are not the whole gate

Deriving expectations protects against stale data: edit an amount in data/ and grading follows. It does not protect against changed logic. If a verifier derives its expected answer through the same helper the tools use, and that helper is wrong, both sides move together and the check still passes. What catches that is a recorded baseline — an artifact frozen when the behaviour was known good, which cannot follow a formula change. Derivation and baselines cover different failures. An environment you intend to rely on wants both.

Getting Started

Environments

Running Agents

Guides

Reference

Required and optional checks

Grade state first, output second

Derive from the initial state, not the final one

Look up by property, not by id, where you can

Reading a free-text answer

Scaffolding

Verifiers are not the whole gate

​Required and optional checks

​Grade state first, output second

​Derive from the initial state, not the final one

​Look up by property, not by id, where you can

​Reading a free-text answer

​Scaffolding

​Verifiers are not the whole gate

Required and optional checks

Grade state first, output second

Derive from the initial state, not the final one

Look up by property, not by id, where you can

Reading a free-text answer

Scaffolding

Verifiers are not the whole gate