Online Evals
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
Online Evals are evals that run continuously against live production traces—rather than in offline test suites—to generate pre-processed signal about agent failures and quality issues in real time.
What They Are and How They Work
As described by Jason Lopatecki of Arize, online evals are layered on top of production traces as agents operate, distinguishing them from batch or pre-deployment evaluation approaches. The core idea is that raw trace data alone is insufficient for self-improvement loops; evals add a structured assessment layer that converts noisy operational telemetry into actionable failure signal. 18:42 30:19
Role in Self-Improving Systems
Lopatecki positions online evals as a critical component of Self-Improving Systems, arguing that evals "add another layer to this"—meaning they sit atop the raw production signal pipeline and pre-process it before it feeds into downstream improvement mechanisms such as fine-tuning, prompt revision, or automated pull requests. 4:57
The practical implication is that online evals bridge the gap between observability (knowing what happened in production) and improvement (knowing what to fix): without this evaluation layer, production traces surface symptoms but not structured diagnoses.
Points of Disagreement
No dissenting views on online evals appear in the current graph. All attributed positions come from Lopatecki in a single talk.