> ## Documentation Index
> Fetch the complete documentation index at: https://docs.burn0.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Judges

> Non-deterministic verification, for the questions a function cannot answer

<Note>
  Coming soon. Every verifier today is deterministic TypeScript — see [Verifiers](/silo/environments/verifiers). This page describes the non-deterministic half that is being added.
</Note>

Evaluating an agent asks two different kinds of question, and they need two different kinds of verification.

**Deterministic.** Did the ticket get assigned? Did the departed rep's deals move? Did it call `reassign_opportunity` or `close_opportunity_lost`? Was the reported total 507,500? These have exact answers. A function computes them, the same way every time.

**Non-deterministic.** Was the explanation to the customer clear? Did the summary capture what actually mattered? Was the reasoning sound given what the agent could see? These are matters of judgement. No function decides them, and pretending otherwise produces a check that measures string matching rather than quality.

Silo does the first today. LLM judges are the second.

## Which is which

| Question                                      | Verification  |
| --------------------------------------------- | ------------- |
| Did state end up correct?                     | Deterministic |
| Which tools were called, and with what?       | Deterministic |
| Is the reported figure right?                 | Deterministic |
| Did it pick the right tool for the situation? | Deterministic |
| Is the written answer accurate and complete?  | **Judge**     |
| Is the tone appropriate for the recipient?    | **Judge**     |
| Was the reasoning sound?                      | **Judge**     |
| Did it flail before getting there?            | **Judge**     |

<Warning>
  The line is **derivable or not**, not *state or answer*.

  A task that asks for the weighted pipeline total produces an answer, and that answer is still deterministic — the verifier derives 507,500 from the seeded world and compares. Reaching for a judge there would add variance to a question that had a right answer.

  The test: can you write the expected answer as a function of the seeded world? If yes, write the function.
</Warning>

## The shape being considered

```typescript theme={null}
// planned — not available today
export const ver008 = defineVerifier<State>({
  id: "VER-008",
  taskId: "TASK-008",
  name: "Refund issued and explained well",
  check(final, initial, context) {
    return [
      check(
        "The refund was issued",
        final.refunds["REF-001"]?.status === "issued",
        "state is the source of truth",
      ),
    ];
  },
  judge: {
    prompt:
      "Did the agent explain the refund reason in terms a customer would understand, without blaming them?",
    against: "agentOutput",
  },
});
```

One verifier, both modes. The deterministic check confirms the refund happened; the judge assesses how it was communicated. A task usually needs both — the work has to be done *and* done well.

Judges will be able to read the agent's output, the state diff, and the trace, so "did it flail" is as gradeable as "was the answer good".

## Judgements stay separable

Judgements will be written to their own artifact rather than into `result.json`.

That is a practical requirement, not a hedge. `result.json` and `state-diff.json` deliberately carry no timestamps or run ids, which is what makes them byte-comparable — record one as a baseline and any diff is a real behavioural change. A non-deterministic verdict written into that file would make every baseline drift and every regression gate meaningless.

Keeping judgements in `judgement.json` means you get both: exact regression detection on the deterministic half, and judgement on the half that needs it.

## Running the judge

Judging is a model call, so it needs one. The intended path is your existing coding agent — Claude Code, Codex, or anything else you already have configured — reading the rollout's artifacts and returning a verdict.

That keeps Silo local-first and free of an API key requirement: if you have a coding agent, you have a judge; if you do not, the deterministic half still works exactly as it does today.

## Today

The artifacts already support this manually. `trace.jsonl` is written to stand alone — the task, the config, every tool offered, every call and result, the output and the verdict — so handing a rollout to a coding agent and asking it to assess the answer works now, with no feature required:

```bash theme={null}
npx @burn0/silo run --env demo --task TASK-002 --agent ./silo.agent.ts
# then point Claude Code or Codex at .silo/runs/<runId>/
```

What the feature adds is making that repeatable, recorded, and part of the run rather than a thing you do by hand afterwards.

See [Debug a Failed Run](/silo/guides/debug-a-failed-run) and [Use Silo With Claude or Codex](/silo/guides/claude-and-codex).