{
  "video": {
    "id": "0vphxNt4wyk",
    "title": "Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind",
    "duration": 1306,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "0vphxNt4wyk",
    "title": "Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind",
    "one_liner": "Skills measurably help — SkillBench 1.1 puts the average lift at ~15% — but almost nobody writes evals for them, so Philipp Schmid shows the minimal harness (a JSON file of prompts plus a Python script that runs a coding agent and regex-asserts the output) that Google DeepMind now ships alongside every internal skill.",
    "summary": "Schmid opens with a room poll: everyone uses coding agents, most use skills, almost nobody has evals — matching SkillBench's finding that of 50,000+ indexed skills, almost none had evals and most were AI-written and untested. He distinguishes the agents we use (where you notice instantly when a skill doesn't fire and can re-prompt or slash-command it) from the agents we build for customers, who have no idea skills exist and will never say 'use the refund skill' — so model-triggered skills and their descriptions carry all the weight. He gives eight rules for writing skills (directives over essays, layer information, don't over-specify workflows that should be scripts, cover negative cases, kill no-ops, know when to retire) and then walks the Gemini Interactions API skill as a worked example: 117 test cases, a JSON spec, a Python runner over Gemini CLI, cheap regex asserts, and a result of almost 90% valid code. The closing argument is that evals are what let you retire a skill as models improve — and that you keep the eval even after the skill is gone.",
    "key_points": [
      "SkillBench indexed over 50,000 skills and found almost none had evals; most were AI-written and not really tested. Because agents are non-deterministic, you can't tell whether a task failed because the skill is bad or because the task was too hard for the model.",
      "SkillBench 1.1, covering ~100 coding and productivity tasks across open and closed models and multiple harnesses, found skills improve performance by roughly 15% on average. A follow-up analysis found human-written skills are the best you can provide, AI-generated skills can impact performance negatively, and a skill.md should stay below 500 lines.",
      "Two kinds of skill: capability skills teach a model something it cannot do consistently today and are temporary (evals tell you when to retire them); preference skills encode your team's workflow, style or domain knowledge, are durable, and need evals to protect them from regressions when agents update.",
      "The description is the cost you pay on every single model call (~100–200 tokens), so it must be short but carry the why, how and when. 50% of failures were skills not triggering because the user's prompt was too shallow — and customers using an agent you built have no idea your skill descriptions exist.",
      "If your skill spells out step one, step two, step three, it should be a script, not a skill — don't waste model tokens on a fixed workflow. Define goals and constraints and let the model decide the path. Multi-cloud deploy instructions (AWS, Google Cloud, Azure) belong in reference files, not in skill.md.",
      "Worked example: the Gemini Interactions API shipped after Gemini's last training cut-off, so Gemini 3/3.1/3.5 knew nothing about it. 117 test cases — drawn from real user traces, synthetic generation, and feedback like 'the model is using Gemini 2.0 when we're already on 3.0' — took valid Interactions API code generation to almost 90%.",
      "The whole harness was two assets: a JSON file of test cases (prompt, language for TypeScript vs Python, a should_trigger flag, expected checks) and a basic Python script that runs a coding agent (Gemini CLI) and parses the output. Most asserts are regex — right SDK, right model, right methods, no old patterns — which is cheap enough to rerun constantly and easy to update when a new model ships. LLM-as-judge with a rubric is the fallback for skills that need whole-trace evaluation.",
      "At Google DeepMind every skill has evals alongside it: cases with prompts, clean workspaces, startup commands that preinstall libraries, script/regex evals over the traces, and LLM-as-judge expectations. They run on every diff to a skill file, and a change is not merged unless it improves the test cases.",
      "Credit to Matt (AI educator) for the no-ops finding: AI-generated skills are full of instructions that don't change agent behaviour — 'write clear high-quality code', 'make the implementation easy to read' — which the model already does. His 'writing great skills' skill is on GitHub."
    ],
    "takeaways": [
      "Homework, literally: pick your most-used skill (ask your coding agent to look at your trajectories to find it) and write five test prompts. 10–20 samples is enough — you'll be surprised what five to ten turn up.",
      "Write five happy-path prompts and five negative prompts where the skill should NOT fire, so you catch over-triggering; 'use for web development tasks' will load on Angular work when you meant React or Tailwind. Add real production traces if you have them.",
      "Test outcomes, not paths — don't assert the skill loads on turn one; if it loads after five turns that's fine. Run evals in isolated workspaces, because coding agents will happily cheat by pulling context from previous chats instead of using the skill.",
      "Run multiple trials per case (up to six) since models are non-deterministic, and test across harnesses — a skill that's great with Gemini may be bad with Codex, and your users may be on Cursor.",
      "Always run ablations: evals with the skill loaded and without. That's the only way to know a skill still earns its tokens, and when the model catches up, keep the eval and delete the skill — the eval then guards against degradation and tells you when to reintroduce it.",
      "Strip the no-ops from your skills. Even if eval performance doesn't move, tokens that don't change agent behaviour are money you spend for nothing."
    ],
    "topics": [
      "skills",
      "evals",
      "agents",
      "coding-agents",
      "prompt-engineering",
      "regression-testing",
      "context-engineering",
      "llm-as-judge"
    ],
    "tools": [
      "SkillBench",
      "Gemini API",
      "Gemini Interactions API",
      "Gemini 3 / 3.1 / 3.5",
      "Gemini CLI",
      "Google DeepMind",
      "Claude Code",
      "Cursor",
      "Antigravity",
      "Codex",
      "React",
      "Tailwind CSS",
      "AWS",
      "Google Cloud",
      "Azure",
      "GitHub",
      "Python",
      "TypeScript"
    ],
    "quotes": [
      {
        "text": "Everyone uses skills, no one has evals. Hopefully we can fix that today.",
        "at": "00:45",
        "url": "https://www.youtube.com/watch?v=0vphxNt4wyk&t=45s"
      },
      {
        "text": "human-written skills are the best we can provide. Uh AI-generated skills can uh impact performance negatively.",
        "at": "05:11",
        "url": "https://www.youtube.com/watch?v=0vphxNt4wyk&t=311s"
      },
      {
        "text": "you can only change the skill if it improves the eval or add new evals",
        "at": "16:59",
        "url": "https://www.youtube.com/watch?v=0vphxNt4wyk&t=1019s"
      },
      {
        "text": "coding agents are very good at finding or cheating",
        "at": "18:42",
        "url": "https://www.youtube.com/watch?v=0vphxNt4wyk&t=1122s"
      }
    ],
    "words": 4549
  },
  "summary_url": "/#0vphxNt4wyk",
  "transcript": {
    "html": "/transcripts/0vphxNt4wyk.html",
    "txt": "/transcripts/0vphxNt4wyk.txt",
    "vtt": "/transcripts/0vphxNt4wyk.vtt"
  }
}