Deterministic evals
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
A category of evals that rely on logic-based or rule-based checks with fully predictable outputs, as opposed to probabilistic methods like LLM-as-a-judge or human feedback.
Role in the eval landscape
Dat Ngo (Arize) identifies deterministic evals as one of five distinct flavors of eval signal, alongside LLM as a judge, human feedback, golden datasets, and business metrics. He highlights their core advantage directly: "Determinism is super nice, so think about logic or deterministic based evals." (13:35) The appeal is reliability — a deterministic check produces the same result every time, eliminating the variance and cost associated with model-based or human-based evaluation methods.
Cost considerations and selective use
Despite their appeal, Ngo cautions against running deterministic evals (or any evals) exhaustively without discipline. He notes "there's a cost associated with this stuff," framing deterministic evals in contrast to the principle that evals should be run at the minimal sufficient set rather than exhaustively. (19:14) Even low-cost deterministic checks accumulate overhead at scale, so selectivity remains important when designing an eval suite.