Oracle Solution
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
A reference implementation or idealized agent run used within a Private Benchmark to verify that a benchmark task is actually solvable before it is used to evaluate real agents.
Role in Benchmark Construction
Rustem Feyzkhanov (Snorkel AI) advocates for the Oracle Solution as an essential quality-control step when building agent benchmarks. Rather than assuming a task is well-formed, the benchmark pipeline first substitutes a ground-truth "Oracle" agent in place of the real agent being evaluated and runs it through the full task sequence. This confirms the task has a valid solution path before any real agent is tested against it. As Feyzkhanov explains: "we construct Oracle ourselves to make sure that task is solvable in the first place." (7:08)
Mechanics
The Oracle Solution mirrors the structure of a normal agent run: "When Oracle solution runs, it runs through the whole sequence, but just instead of running the real agent, it runs Oracle." (7:08) This means the same environment, tool calls, and evaluation harness are exercised — the only substitution is the agent itself. The result is a verified upper-bound trace that demonstrates task solvability and can serve as a reference for scoring real agent trajectories.
Significance
Because the Oracle Solution is positioned as a prerequisite component of every benchmark task, it directly shapes the reliability of any evals|evaluation built on top of those tasks. A benchmark that skips this step risks including unsolvable or malformed tasks, which would corrupt downstream performance measurements of real agents.