From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

Rustem Feyzkhanov (AI Platform Lead at Snorkel AI) argues that every company shipping AI agents must build a private agent benchmark derived from production observability traces, because public benchmarks measure pass rate on someone else's tasks while production teams care about cost-per-solved-task, latency, and policy compliance. The core technique is reconstructing the exact database state, tools, files, and environment an agent touched during a real run, then packaging that as a repeatable offline simulation. That simulation becomes simultaneously an evaluation harness, a release gate, and a training-data source — the core of what Feyzkhanov calls the "agent ops loop."

Key Takeaways

• Every company needs its own private benchmark; public benchmarks orient but cannot ship 00:44
• Benchmarks must mimic real tools, API services, policies, and workflows — not toy environments 00:54
• A private benchmark is a living dataset continuously populated from production traces, not a static artifact 01:11
• Metrics that matter in production — cost, latency, retries — are invisible on public leaderboards 03:57
• Benchmarks serve three roles: evaluation, integration/release gating, and training-data generation 06:06
• Observability and experimentation are "two sides of the same coin" and must be connected 16:00

Why Production Traces Are Insufficient Alone

Agent traces (input prompt → action sequence → output) are valuable for spotting failures but cannot support rigorous comparison between agent configurations 01:53. A/B testing in production is non-repeatable: database state, tool versions, and environment differ across runs, so comparisons are never apples-to-apples 02:31. Offline simulation converts traces into repeatable experiments, enabling parallel runs with identical environments under different model choices, prompt variants, thinking levels, or skill/tool configurations 02:41.

Why Public Benchmarks Are Insufficient

Public benchmarks such as SWE-bench (GitHub issue fixing), TerminalBench (terminal agents), and WebArena-style computer-use benchmarks are domain-specific and fixated on pass rate 03:08. Feyzkhanov's position: public benchmarks are "useful to orient and build your prior, but your private benchmark is useful to ship" 03:44. Production releases require metrics — cost per solved task, latency, retry counts — that public leaderboards do not expose 04:11.

Anatomy of a Simulation Task

Each benchmark task consists of two complementary halves 06:34:

Agent-facing components:

• instruction.md — the task prompt 07:52
• Environment: Dockerfile / Docker Compose for databases, API services, MCP tools, and sidecar containers 07:56

Hidden components:

• Oracle solution: a hand-constructed correct solution that validates the task is solvable before any agent runs it 07:08
• Verifiers: scripts that read final environment state, traces, and output artifacts 06:57
• Metadata: difficulty tags (simple/medium/hard) based on how often agents succeed 13:44

Feyzkhanov stresses the Oracle's necessity: if the Oracle cannot pass the verifiers, the task is unsolvable and must not enter the benchmark 07:20. For long-horizon tasks spanning hours, the simulation is broken into multi-step sub-tasks, each with its own prompt and verifier, with early-stopping when the agent clearly goes off track 09:57.

Verifiers, LLM-as-a-Judge, and Reward Hacking

Verifiers analyze three data sources: final environment state, the full trace, and output artifacts 10:55. Feyzkhanov recommends a layered approach 11:03:

1. Deterministic checks — best for final outputs and tool calls where correctness is unambiguous 11:05
2. LLM-as-a-judge — evaluates trace quality, planning correctness, and policy adherence 11:14
3. Subject matter expert review — reserved for cases of disagreement between verifiers, not applied universally 11:38; specifically, when a task appears solved but the judge marks it incorrect, or vice versa 19:43

Edge cases to guard against 12:12:

• Reward hacking: agents detect they are in simulation and exploit the environment 12:14
• Too-broad verifiers: agent always passes even when wrong 12:21
• Broken verifiers: agent always fails even when correct 12:26
• High variance: unstable agent behavior produces inconsistent pass/fail 12:32
• Missing fixtures: incomplete environment setup breaks reproducibility 13:26

Benchmark as CI Pipeline and Agent Ops Loop

Feyzkhanov's central prescription: treat the benchmark as software subject to its own CI pipeline 13:09. That CI pipeline should pin all dependencies, validate base images, run the Oracle solution, and tag tasks by difficulty before approving them into the benchmark 13:20.

Once in production, two loops operate in parallel 15:12:

1. Benchmark expansion loop: observability tool (e.g., Arize) surfaces production failures → failures reconstructed as new simulation tasks → benchmark grows 15:19
2. Release gate loop: simulation runner executes experiments on the expanded benchmark under new agent configs → results recorded → agent released only if performance threshold is met 15:32

Feyzkhanov explicitly warns against the anti-pattern of stuffing fixes into the system prompt ("Never do this / only do that") 14:31. With full-stack simulation, fixes belong in the correct layer: context window management for overload issues, skill definitions for missing procedures, structured output schemas for format problems 14:54. Simulations can also generate training data to fine-tune a smaller model to match a larger model's performance on specific tasks 05:58.