Skip to main content
Coming soon. Every verifier today is deterministic TypeScript — see Verifiers. This page describes the non-deterministic half that is being added.
Evaluating an agent asks two different kinds of question, and they need two different kinds of verification. Deterministic. Did the ticket get assigned? Did the departed rep’s deals move? Did it call reassign_opportunity or close_opportunity_lost? Was the reported total 507,500? These have exact answers. A function computes them, the same way every time. Non-deterministic. Was the explanation to the customer clear? Did the summary capture what actually mattered? Was the reasoning sound given what the agent could see? These are matters of judgement. No function decides them, and pretending otherwise produces a check that measures string matching rather than quality. Silo does the first today. LLM judges are the second.

​
Which is which

The line is derivable or not, not state or answer.A task that asks for the weighted pipeline total produces an answer, and that answer is still deterministic — the verifier derives 507,500 from the seeded world and compares. Reaching for a judge there would add variance to a question that had a right answer.The test: can you write the expected answer as a function of the seeded world? If yes, write the function.

​
The shape being considered

One verifier, both modes. The deterministic check confirms the refund happened; the judge assesses how it was communicated. A task usually needs both — the work has to be done and done well. Judges will be able to read the agent’s output, the state diff, and the trace, so “did it flail” is as gradeable as “was the answer good”.

​
Judgements stay separable

Judgements will be written to their own artifact rather than into result.json. That is a practical requirement, not a hedge. result.json and state-diff.json deliberately carry no timestamps or run ids, which is what makes them byte-comparable — record one as a baseline and any diff is a real behavioural change. A non-deterministic verdict written into that file would make every baseline drift and every regression gate meaningless. Keeping judgements in judgement.json means you get both: exact regression detection on the deterministic half, and judgement on the half that needs it.

​
Running the judge

Judging is a model call, so it needs one. The intended path is your existing coding agent — Claude Code, Codex, or anything else you already have configured — reading the rollout’s artifacts and returning a verdict. That keeps Silo local-first and free of an API key requirement: if you have a coding agent, you have a judge; if you do not, the deterministic half still works exactly as it does today.

​
Today

The artifacts already support this manually. trace.jsonl is written to stand alone — the task, the config, every tool offered, every call and result, the output and the verdict — so handing a rollout to a coding agent and asking it to assess the answer works now, with no feature required:
What the feature adds is making that repeatable, recorded, and part of the run rather than a thing you do by hand afterwards. See Debug a Failed Run and Use Silo With Claude or Codex.