Philipp Schmid

Philipp Schmid is a researcher and engineer based in Germany on the Google DeepMind team, focused primarily on the Gemini API and agents. His public work centers on best practices for building and evaluating agent skills, with a particular emphasis on the underappreciated role of evals in skill development. 0:12

Core Advocacy: Evals for Agent Skills

Schmid's central thesis is that the near-universal failure to test agent skills with evals is a serious and correctable problem in practice. Observing that almost no hands went up when he asked an audience whether they had evals for their skills—despite widespread skill usage—he opened his talk with: "Everyone uses skills, no one has evals. Hopefully we can fix that today." 0:41 He reinforces this with data from Skills Bench, a benchmark indexing over 50,000 skills, noting that almost none of those skills had accompanying evals and most were AI-written and untested. 0:52

Despite this gap, Schmid reports that skills on average improve agent performance by roughly 15% according to Skills Bench results—making the lack of evals a significant missed opportunity for quality control. 4:18

Skill Quality and Design

Schmid draws a strong distinction between well-crafted and poorly-crafted skills:

Human vs. AI-generated skills: Human-written skills outperform AI-generated ones, and AI-generated skills can actually negatively impact agent performance. A key reason is that AI-generated skills tend to include many no-ops—instructions that do nothing to change the agent's behavior—which he criticizes as wasteful and harmful. 11:14
Skill descriptions: He reports that 50% of skill failures occur because the skill was not triggered correctly due to a weak description. 17:26 His prescription is that skill descriptions should use directives rather than passive information, so the model knows exactly when and how to invoke a skill. 18:45
File size: Skill .md files should be kept below 500 lines, as longer files degrade performance. 12:03
When not to use a skill: If a workflow is fully deterministic and always the same, a script should be used instead of a skill to avoid wasting model tokens. 9:16

Skill Lifecycle: Retirement and Durability

Schmid articulates a principled framework for managing skills over time:

Capability skills are temporary—they compensate for current model weaknesses and should be retired as models improve. Preference skills encode durable behavioral preferences and should be protected with evals indefinitely. 53:12
Evals are the mechanism for detecting when retirement is appropriate: "Evals will tell us when we can retire a skill and when not." 11:43
Critically, evals should be retained even after a skill is retired, so that performance regressions can be detected and the skill reintroduced if a later model regresses. 20:37

Practical Eval Design

Schmid offers concrete operational guidance for running skill evals:

Isolation: Evals must run in isolated environments because coding agents can "cheat" by looking up previous chat histories or executions to infer context without actually invoking the skill. 19:48
Multiple trials: Each test case should be run across multiple trials to account for non-determinism in agent behavior. 20:02
Cross-harness testing: Skills should be tested across different agent harnesses, since the same skill can perform well in one and poorly in another. 20:18
Ablation Testing: Always run evals with and without the skill loaded to measure net impact. 21:13
Regression gates: Skill changes should not be merged unless they improve test cases, enforcing regression testing on every modification. 17:04
Low cost: Most skill evals can use regex-based checks rather than LLM-as-a-judge, making them cheap enough to run frequently. 14:11

Tools and Infrastructure

In practice, Schmid uses Skills Bench as a benchmark and evaluation reference 0:52, and builds new skills targeting the Gemini Interactions API 12:33. He has used the Gemini CLI as the agent harness in demonstrations, noting it passes output through as part of the skill execution pipeline. 13:42