Agent Ops
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
Agent Ops is the operational discipline encompassing the ongoing lifecycle management of AI agents in production, analogous to DevOps for traditional software — integrating monitoring, evaluation, and continuous integration practices into a unified loop specific to agentic systems.
Core Architecture
According to Rustem Feyzkhanov (Snorkel AI), Agent Ops is not an optional add-on but a mandatory part of running agents: it "has to be part of your agentic life cycle." → He describes the discipline as composed of several interconnected components that together form a coherent operational system.
Observability and Experimentation as Two Sides of the Same Coin
A central architectural claim Feyzkhanov makes is that observability (tracing what agents actually do) and experimentation (simulating and evaluating what they would do) must be unified rather than treated as separate concerns: "they're two sides of the same coin." → Neither is sufficient alone — traces without evaluation pipelines leave teams without actionable signal, and evaluation without production traces means simulations are disconnected from real behavior.
Component: Private Benchmark
A Private Benchmark — an evaluation suite built from an organization's own data and use cases rather than public benchmarks — is a constituent part of Agent Ops. Feyzkhanov frames it as something that "has to be part of your agentic life cycle," positioning bespoke evaluation as a non-negotiable element rather than a nice-to-have. →
Component: CI Pipeline for Agents
A CI Pipeline for Agents is similarly described as a structural requirement within Agent Ops: "you need to have a separate CI pipeline for it." → The emphasis on separate signals that standard software CI pipelines are insufficient for agentic systems, which require evaluation steps (such as running agent simulations against private benchmarks) that have no direct analogue in conventional software testing.
Summary of Positions
All substantive claims about Agent Ops in the current graph come from a single speaker (Feyzkhanov); no dissenting or complementary perspectives from other speakers are yet recorded.