Observability Traces

Observability traces are the runtime execution logs produced by AI agents in production, capturing the full sequence of steps, tool calls, costs, latencies, and retries that occur during live operation. In the agent engineering context, they serve as the raw material for constructing offline benchmarks and evaluation datasets.

Role in the Agent Ops Loop

Rustem Feyzkhanov (Snorkel AI) positions observability traces as a foundational input to the broader agent development cycle. Rather than treating traces as passive diagnostic artifacts, he argues they should feed directly into a continuously updated benchmark dataset: "It's a constantly populated data set from your production traces." 1:57

This framing makes traces an active component of quality control. Feyzkhanov advocates connecting observability and experimentation as "two sides of the same coin in the agent ops loop," such that "your observability piece and experimentation piece are connected." 15:56

From Traces to Benchmarks

A central use case Feyzkhanov describes is converting production traces into repeatable offline experiments via Agent Simulation: "you take production traces, you construct tasks, and then you can run simulation benchmark." 4:00 This approach enables apples-to-apples comparison of agent versions on metrics beyond simple pass rate, including cost, latency, and retry counts.

The implication is that agent evals should function as a CI pipeline and release gate rather than a static dataset — and that this is only achievable when the benchmark is continuously repopulated from live traces. 1:54

Relationship to Agent Simulation

Agent Simulation is positioned as an alternative mode of use for observability traces rather than a replacement for the traces themselves. Where raw traces reflect what happened in production, simulations constructed from those traces allow controlled re-execution under different agent configurations — preserving the realism of real-world tasks while enabling systematic comparison. 4:00