LLM Observability, Evaluation, and Experimentation for AI Agents — Dat Ngo, Arize AI

Dat Ngo, AI Architect at Arize AI, argues that nondeterministic agent systems require a fundamentally different observability and evaluation stack than traditional software—one built on telemetry, not code inspection—and outlines the three-part flywheel (observability → evals → experimentation) that Arize is working to fully automate.

Key Takeaways

Code does not audit agents; telemetry does—traces and spans are the audit record of what an agent actually did 3:29
Evals come in five distinct flavors: LLM-as-a-judge, human feedback, golden datasets, deterministic checks, and business metrics 7:06
Evals should be run at multiple scopes: single-span, multi-span, trajectory eval, and session-level 10:18
Running the minimum necessary evals is a design principle—exhaustive evaluation has real cost 11:54
Arize Phoenix is open-source, single-container, no Kubernetes required; Arize AX is the enterprise tier used by Uber, Booking, and Reddit 15:33
Arize's stated long-term goal is to automate the entire observability-eval-experimentation loop out of the human workflow 14:34

Why Telemetry, Not Code, Audits Agents

Traditional code inspection cannot track nondeterministic execution paths 2:02. Ngo positions OpenTelemetry (Otel) as the correct foundation: a single auto-instrumenter line of code wraps any framework or SDK, emits traces and spans, and produces an audit record of every agent decision 3:06. He warns that without this telemetry layer, ordering bugs—such as calling tool B before tool A when B has a dependency on A—go entirely undetected 6:03.

Beyond individual traces, Ngo identifies additional observability dimensions:

Sessions: captures back-and-forth state across multiple runs, enabling evaluation of multi-turn conversations 3:47
Distributional views: aggregates all instantiations of an agent to reveal what percentage of traffic takes each branch and which branches incur high latency 5:01

Five Flavors of Eval Signal

Ngo categorizes signal sources into five types 7:06:

Flavor	Description
LLM-as-a-judge	Scalable but grows complex; can be tuned against golden datasets
Human feedback	Highest-trust signal from end users or domain experts
Golden datasets	Labeled data from trusted domain experts; used to calibrate LLM judges 7:40
Deterministic checks	Logic-based (e.g., JSON schema validation); cheapest to run 8:13
Business metrics	Make money, save money, or save time 8:36

He advocates matching eval type to persona: engineers own automation and programmatic evals; product managers and subject-matter experts own prompt engineering decisions and defining which evals matter 9:06.

Eval Scope: From Span to Session

Ngo stresses that what to evaluate and at what granularity are separate decisions 11:46:

Span eval: single input/output of one LLM call—most familiar, lowest complexity 10:27
Multi-span eval: requires data from multiple components; example use case—measuring how well agents hand off data to each other 11:02
Trajectory eval: examines the full ordered sequence of spans to verify the agent followed the correct tool-calling order for a business process 11:15
Session eval: evaluates the state machine across a full conversation—e.g., "Was the user ever frustrated? Were all questions answered?" 11:23

Ngo explicitly warns against exhaustive evaluation: "just because you can eval something doesn't mean you always should"—the goal is the minimal set of evals sufficient to confirm the application is working as intended, given the cost of each eval call 11:54.

Experimentation and the Improvement Loop

Experimentation begins by collecting traces where signal is poor into a dataset (or uploading input/output pairs directly), then running controlled changes against that dataset 12:22. Variables include:

Prompt changes
Model swaps
Orchestration changes
Configuration changes 12:55

Experiments can be run via UI or programmatically. Ngo predicts software in this space will compress—it will become easier to build and customize—and frames the manual UI as a stepping stone, not the destination 13:27.

The Alex AI Layer and the Automation Vision

Arize's enterprise product, Arize AX, includes an AI agent called Alex that can be invoked directly or called by an external coding agent (e.g., Claude Code / cloud code, Codex) via CLI primitives and tool definitions 13:46. Alex scans traces, surfaces high-latency spans and errors, and can generate evals automatically 15:06.

Ngo's stated position is that the entire observability-eval-experimentation flywheel is automatable: Arize's ultimate goal is that users should not have to choose their own evals—Alex should infer appropriate evals from the trace context, create them on the fly, and update them when application behavior changes 15:00. He frames this as "automating you out of this process entirely" 14:34.