Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind

Philipp Schmid (Staff Engineer, Google DeepMind, working on Gemini API and agents) argues that agent skills are almost universally shipped without evals, creating silent failures in production. Drawing on SkillsBench data and an internal case study building a Gemini Interactions API skill, he presents a complete lifecycle for writing, testing, and retiring skills responsibly.

Key Takeaways

• SkillsBench indexed over 50,000 skills and found almost none had evals; most were AI-written and untested 0:52
• Skills improve average model performance by ~15% according to SkillsBench 1.1 4:30
• Human-written skills outperform AI-generated ones; AI-generated skills can negatively impact performance 5:11
• Skill .md files should stay below 500 lines 5:24
• ~50% of skill failures Schmid observed were caused by the skill description failing to trigger correctly 17:27
• Evals for skills are often just regex checks — cheap, fast, and sufficient for most cases 14:38
• Skills should be retired when models no longer need them; evals tell you when 3:33

Agents You Use vs. Agents You Build

Schmid draws a sharp distinction between tools developers use personally (e.g., Cursor, Claude Code, anti-gravity) and agents built for end customers 1:22. When an engineer uses a coding agent, they notice immediately if a skill doesn't trigger and can reprompt or use slash commands 1:51. End customers have no awareness of skill descriptions and won't write prompts that naturally invoke them 2:06. This means model-invoked skills must be self-triggering based on natural user language, making correct trigger behavior a first-class engineering concern — not a vibe check.

Skill Anatomy and Two Skill Types

A skill is a folder containing a skills.md file plus optional reference assets, structured via progressive disclosure 2:33:

1. Description — always in model context; 100–200 tokens paid on every invocation 8:00
2. Skill body — loaded when the model decides to use the skill 8:14
3. Reference files — deep context the model navigates only when needed (e.g., separate AWS vs. GCP deployment guides) 8:52

Schmid differentiates two types 3:10:

• Capability skills — teach models something they can't do consistently today (e.g., using a newly released API); temporary by nature; evals tell you when to retire them 3:26
• Preference skills — encode company-specific workflows, style, or domain knowledge; durable; must be protected by evals because foundation models won't learn this context 3:37

Eight Tips for Writing Good Skills

1. Write directives, not essays — use imperative instructions ("use the Interactions API if building a chat application"), not passive descriptions 7:28
2. Keep skills lean; layer information — minimize description length to reduce per-call token cost 7:55
3. Set the right level of freedom — if the workflow is fully deterministic, write a script instead; skills should define goals and constraints, not step-by-step paths 9:04
4. Include negative cases — specify when not to use the skill to prevent over-triggering 9:56
5. Test early — write 5 happy-path + 5 negative prompts at creation time; add real production traces when available 10:40
6. Kill no-ops — AI-generated skills are full of instructions that don't change agent behavior (e.g., "write clean code"); these waste tokens without improving performance 11:17 (credit attributed to "Matt," an AI educator)
7. Know when to retire — run ablation tests with and without the skill; retire when performance holds without it 11:49
8. Write the description with care — it is the primary trigger signal for model-invoked skills 6:51

Building a Lightweight Eval Harness: Gemini Interactions API Case Study

Schmid's team needed a skill for the Gemini Interactions API because it was released after Gemini's training cutoff — the model had no knowledge of it 12:42. They built 117 test cases drawn from real user data, synthetic cases, and user feedback (e.g., model defaulting to Gemini 2.0 instead of 3.0) 13:06. Result: performance on generating valid Interactions API code improved to ~90% 13:27.

The eval harness required only two artifacts 13:41:

• A JSON file with fields: prompt, language (TypeScript/Python), should_trigger (boolean), and expected_checks 13:49
• A Python script that runs Gemini CLI, parses output, and evaluates results 14:20

Most checks were regex assertions — correct SDK, correct model ID, correct methods, absence of deprecated patterns 14:38. This avoids the cost of LLM-as-a-judge for the common case; LLM-as-a-judge is reserved for complex traces requiring holistic evaluation 15:15.

At Google DeepMind internally, evals run on every diff to a skill file; a change cannot be merged unless it improves or maintains eval scores 16:44. Evals include: prompt + workspace definition, startup commands to pre-install dependencies, regex/script checks on traces, and LLM-as-a-judge expectations 15:53.

Eval Best Practices Summary

Practice	Rationale
Start with 10–20 samples	Even 5–10 examples surface real failures 18:11
Test outcomes, not paths	Whether the skill loads on turn 1 is irrelevant; task success is what matters 18:22
Use isolated workspaces	Coding agents will find previous chat history and "cheat" without isolation 18:40
Run 3–6 trials per case	Non-determinism means a single pass/fail is unreliable 19:02
Test across multiple harnesses	A skill that works in Gemini may fail in Codex/Cursor 19:16
Keep evals after retiring a skill	Retain as regression guards; reintroduce the skill if degradation reappears 19:43
Ablation: skill on vs. off	The only way to know if a skill is truly helping 21:11