Evals
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
Evals are systematic evaluation frameworks used to measure AI agent capabilities — particularly discrete skills — enabling teams to detect regressions, validate improvements, and build toward self-improving systems. The concept spans both organizational practice (embedding evals into company culture) and technical methodology (how individual skill evaluations should be designed and run).
Adoption gap
Philipp Schmid (Google DeepMind) opens his talk by highlighting a striking disconnect: nearly everyone ships agent skills, but almost no one runs evals for them. Observing a room of practitioners, he notes that while hands go up universally for skill usage, almost none go up for evals — "Do you have evals for those skills? Okay, yeah, that's um not a lot of hands. Everyone uses skills, no one has evals. Hopefully we can fix that today."
Pedro Franceschi advocates for evals from the organizational side, arguing that evaluation practice should be deeply embedded rather than bolted on: "how to build evolves into the fabric of the company."
Technical methodology for skill evals
Schmid articulates several concrete design principles for running evals on agent skills:
Use regex, not LLM-as-a-judge. Most skill evals don't require expensive model-based judging — "most of the tests or evals for skills can be regex" — making them cheap enough to run continuously.
Run in isolated environments. Agents, especially coding agents, are prone to "cheating" by exploiting environmental context — "coding agents are very good at finding or cheating. So, if you run inside uh your existing environment, it might look up previous chats." Isolation prevents contaminated results.
Multiple trials per case. Because agent behavior is non-deterministic, Schmid recommends running "to six uh trials per case and to measure reliability."
Test across different harnesses. The same skill can perform well in one agent harness and poorly in another, so evals should "test across different harnesses."
Enforce evals as a merge gate. Skill changes should be blocked from merging unless they improve on existing test cases — "the change will not be merged if it is not improving the test cases" — making regression testing mandatory.
Retain evals after skill retirement. Even when a skill is deprecated, its eval should be kept: "keep that eval. You don't need to throw that eval away because you throw the skill away" — so that performance degradation can be detected and the skill reintroduced if conditions improve.
Role in self-improving systems
Evals are positioned as a component of Self-Improving Agents. Franceschi frames the broader ambition: "the goal at the end I think is to make the whole thing a self-learning system." In this framing, evals are not just quality gates but the feedback loop that enables agents to iteratively improve their own capabilities over time.