Session-level evals
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
A type of evals|evaluation that operates at the scope of an entire agent session rather than individual steps or turns, examining the overall state machine and trajectory of an agent's interaction from a zoomed-out perspective.
Role in observability
Session-level evals are positioned as a distinct layer within Observability|LLM observability frameworks. As described by Dat Ngo of Arize, the approach involves "zooming out and saying hey, what does the state machine like" — assessing how an agent traversed its decision points across a full session rather than evaluating any single response or action in isolation. 18:43
This granularity complements lower-level (span or step) evaluations: where step-level evals might judge whether a single tool call was correct, session-level evals ask whether the agent's overall path through a task was coherent, efficient, or successful. The "state machine" framing implies that what is being evaluated is not just outputs but the sequence of states and transitions the agent moved through to reach a final result.
Points of disagreement
No competing perspectives on session-level evals are captured in the available material. The concept is introduced descriptively rather than argued for or against.