LLM as a judge
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
A technique in which a large language model is used to evaluate the outputs, decisions, or behaviors of another AI system, serving as one of several distinct flavors of evals signal alongside human feedback, golden datasets, deterministic checks, and business metrics.
Role in evaluation pipelines
Dat Ngo (Arize) frames LLM as a judge as one of five core signal types for AI observability, noting it is well-known enough that "everyone here in the room has heard of LLM as a judge." → However, he cautions that familiarity should not translate into overuse: "just because you can eval something doesn't mean you always should," arguing that evals should be run at the minimal sufficient set because there is a real cost to running them. →
Philipp Schmid (Google DeepMind) positions LLM as a judge as appropriate for complex, multi-step evaluation — specifically for cases involving whole traces or sequences of agent actions — while arguing it is unnecessary for simpler skill-level checks: "most of the tests or evals for skills can be regex... we don't need to use like LLM as a judge." → This makes regex-based checks the cheaper, preferred default for routine evaluations, with LLM as a judge reserved for cases that genuinely require holistic reasoning over a full trace. →
Rustem Feyzkhanov (Snorkel AI) takes a broader view, treating LLM as a judge as one instance of a general verifier pattern, alongside "harness as a judge" or "agent as a judge," suggesting the role of automated judgment can be filled by various agent-level constructs depending on the task. →
Use in security and policy enforcement
Beyond evaluation, LLM as a judge is applied by Pedro Franceschi and the Crab Trap system as a real-time policy enforcement mechanism at the network layer. In this architecture, an HTTP proxy intercepts agent requests and the LLM determines "is this request something that should be approved or not based on the policy for what that agent should be doing." → This positions LLM as a judge not just as an offline evaluation tool but as a live guardrail enabling "aggressive experimentation while maintaining guardrails" in enterprise AI security.
Points of disagreement
| Position | Speaker | View |
|---|---|---|
| LLM-as-judge is often unnecessary for skill evals | Philipp Schmid | Regex checks are cheaper and sufficient for most cases; LLM as a judge only needed for complex trace-level evaluation |
| LLM-as-judge should be used judiciously | Dat Ngo | Real cost to running evals means it should not be applied exhaustively |
| LLM-as-judge is broadly useful in a verifier pattern | Rustem Feyzkhanov | Advocates for it as a general-purpose judgment mechanism, alongside agent-level and harness-level alternatives |
The core tension is between cost/efficiency (favoring deterministic or regex checks as defaults) and the genuine need for semantic, holistic judgment that only an LLM can provide over complex agent traces or nuanced policy decisions.