Ablation Testing
person concept tool org talk claim — click a node to jump to its page; hover an arrow for the relation
A technique in AI agent evaluation where a component is removed from a system and performance is measured both with and without it, in order to determine the component's actual contribution.
Advocated Use: Skill Lifecycle Management
Philipp Schmid (Google DeepMind) advocates ablation testing as a mandatory practice when shipping and maintaining agent skills. His position is that developers should "run always evals with your skill loaded and without your skill loaded" — the comparison between these two conditions is the only reliable way to determine whether a skill is still providing value. Critically, Schmid frames this not just as a validation step at deployment time but as an ongoing mechanism for deciding when a skill can be retired: without a baseline comparison, there is no principled basis for knowing if a skill has become redundant or obsolete. Don't Ship Skills Without Evals, 21:13
Key Implication
The practical thrust of this recommendation is that every skill should be paired with an eval suite from the start, enabling the ablation comparison at any point in the skill's lifecycle. A skill that lacks this paired evaluation infrastructure cannot be safely retired — its removal would be guesswork rather than evidence-based.