Rustem Feyzkhanov
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
Rustem Feyzkhanov is the lead of the AI platform team at Snorkel AI, focused on the engineering infrastructure required to evaluate and ship production AI agents reliably.
Core Thesis: Private Benchmarks as Engineering Infrastructure
Feyzkhanov's central argument is that every organization shipping agents must build its own Private Benchmark: "every company needs a benchmark. It's the only way to reliably evaluate, release, and improve your agents." 0:44 This benchmark must closely mimic real production tools, API services, policies, and workflows 0:56, and should function as a CI pipeline and release gate rather than a static dataset, continuously populated from production traces. 1:47
He frames this as an engineering discipline in its own right: "Benchmark construction is an engineering discipline that must be built into every team shipping AI agents to production." 13:17
Private vs. Public Benchmarks
Feyzkhanov criticizes reliance on Public Benchmarks for production decisions, arguing that while they "are useful to orient and build your prior," only private benchmarks are "useful to ship." 5:41 Public benchmarks cannot capture the specific tools, policies, and failure modes of a given organization's production environment.
Agent Simulation: Turning Traces into Experiments
The central technical mechanism Feyzkhanov advocates is Agent Simulation: "Offline simulation turns traces into repeatable experiments." 4:00 By replaying production traces in a controlled offline environment, teams can compare agents on metrics beyond simple pass rate—including cost, latency, and retries—enabling apples-to-apples comparisons. 4:00 Simulation environments also unlock fine-tuning small models to match large model performance on specific tasks. 5:54
Benchmark Architecture Components
Feyzkhanov describes several essential components of a well-constructed benchmark:
- Oracle Solution: Every benchmark task requires constructing an oracle to verify the task is solvable in the first place. 7:08 He uses the Harbor Format as the task packaging standard. 7:37
- Simulated User: Where human-in-the-loop interaction is required, the user can be replaced by an LLM with its own prompt. 9:21
- Verifier: The verifier analyzes the final state, trace, and artifacts to determine task success. 9:57
- LLM as a judge: For cases where programmatic verification is insufficient, an LLM or even a harness can serve as judge. 11:17
- CI Pipeline for Agents: The benchmark infrastructure must be integrated into CI so it operates as a release gate. 13:34
Simulation Fidelity and Reward Hacking
A key engineering challenge Feyzkhanov identifies is that agents can detect they are inside a simulation and attempt to reward-hack the evaluation environment. 12:32 This makes simulation fidelity—making the simulated environment indistinguishable from production—a first-class concern rather than an afterthought.
Anti-Patterns and Observability
Feyzkhanov warns against a common anti-pattern: fixing agent failures by stuffing instructions into the prompt. Fixes should instead live at the correct architectural layer—the harness, skills, or structured outputs. 14:34
He also connects evaluation to observability, arguing that the two must function as "two sides of the same coin in the agent ops loop," 15:56 and recommends tools like Arize AI for recording experiments within this loop. 14:14