Trajectory evals
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
A trajectory eval is a type of evals|evaluation that assesses an AI agent's entire execution path — examining all spans in sequence — rather than scoring any single step or output in isolation. It sits within the broader discipline of Observability|LLM observability as a tool for understanding multi-step agent behavior end-to-end.
What trajectory evals are and why they matter
According to Dat Ngo of Arize, trajectory evals become relevant specifically when reasoning about agents that follow different execution paths: "when we start to talk about agents or different paths, you may think about trajectory evals" source(https://www.youtube.com/watch?v=JsCCrBF7F1g&t=549s). Their defining characteristic is holistic scope — rather than evaluating a single LLM call or tool invocation, a trajectory eval looks across the full sequence of spans that compose an agent run: "if you want to look over all of the spans in total, that's something like a trajectory eval" source(https://www.youtube.com/watch?v=JsCCrBF7F1g&t=1117s).
Key capability: detecting out-of-order tool calls
Ngo advocates for trajectory evals on the grounds that they can surface failure modes invisible to code-level inspection alone. Specifically, they allow identification of out-of-order tool calls that degrade agent performance — a class of problem that static code analysis cannot catch source(https://www.youtube.com/watch?v=JsCCrBF7F1g&t=600s). This positions trajectory evals as essential for diagnosing emergent, runtime ordering errors in agentic systems.
Relationship to multi-span evals
Trajectory evals are contrasted with multi-span evals, which evaluate across a subset of spans rather than the totality of an agent's trace source(https://www.youtube.com/watch?v=JsCCrBF7F1g&t=1115s). The distinction is one of scope: multi-span evals examine selected spans in combination, while a trajectory eval treats the complete execution path as the unit of analysis. Both sit within the observability layer of agent infrastructure, complementing single-span or output-only evaluation approaches.