{
  "video": {
    "id": "q4WC4TowkBo",
    "title": "Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS",
    "duration": 3664,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "q4WC4TowkBo",
    "title": "Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS",
    "one_liner": "Give every agent a deterministic, app-specific verifier CLI (snapshot, reload, storage, console, prove, judge) so it can find out for itself that it isn't done. Don't take its word for it: \"status comes from records, not model prose.\"",
    "summary": "Nick Nisi and Zack Proser of WorkOS argue that when agents write code faster than humans can review it, verification is the foundation for trusting them and for scaling up. An agent saying it is done is not proof. Done should mean receipts: screenshots, videos, test results and deterministic checks. Using a demo 'Midnight Snack Club' waitlist app whose UI says the signup worked but never saves it, they show a custom CLI called `snack` with up, doctor, join, reload, storage, snapshot, screenshot, prove and judge commands. With it an agent can catch its own bug, fix it and re-check without a human. They also present their own verification tools, CTX and Riker, and argue that saved receipts become eval datasets for future self-improving (RSI) code bases.",
    "key_points": [
      "Demo failure mode: an email added in the Midnight Snack Club app shows a success message, but on reload it is gone because it was never saved. A UI-only check like a Playwright form-submit test would pass while the feature is broken. Running `snack join` then `snack storage` shows 'Storage is empty', and that is the moment the agent learns it is not done.",
      "Prompts like 'always verify the change in a browser' are vague and open to interpretation. A script backed by real bash calls is more deterministic and gives the same result across many agents running the same prompt. It feeds exact errors back so the agent can loop: run, check, fix, run again.",
      "The `snack` CLI is a template, not a reusable tool. Its commands: lifecycle (up/down/doctor, which checks Node, Chromium, session, running, built); drive (join, reload, reset); look (snapshot as JSON, network, console, local storage, screenshot); and prove (a deterministic Playwright check) plus judge (a fast 'system one' model that checks intent without needing an exact text match, e.g. 'Waitlisted' vs 'You're in, your seat is confirmed').",
      "Zack's CTX is a vibe-coded web app on Cloudflare. It started as shared memory across Claude Code, ChatGPT and Pi, then handed agents secrets without them seeing the credential. Now it acts as a central verification authority over MCP: an agent asks 'am I done?', CTX runs a known verifier (e.g. it checks through the GitHub API that the PR exists, is healthy and passes its merge checks), and only then marks that step of the workflow satisfied. Otherwise it sends the agent back with what is still failing.",
      "Nick's Riker babysits PRs: it fixes failing tests and review comments while keeping to the goal of working software, keeps the evidence, decides when to stop, and only alerts him when it needs him.",
      "A successful screenshot isn't proof: the agent may not have actually reached that state, may have taken a path the UI doesn't allow, or may have caused side effects. Proof has to run the flow start to finish.",
      "A saboteur/detective game (`pnpm snack up --bug random`) plants a random bug, such as memory-only storage, a wrong email, a lost confirmation, an over-promising message, a console error, or no bug at all. The detective has to name it and prove it using `snack` alone. Each miss becomes a new rule to check for.",
      "Anecdotes: people on their team ship about 900 PRs a month with no incidents. Zack runs 18–30 terminal tabs a day, and Nick usually has 10–15 agents going. Nick trusts a 5/5 'Reptile' review score plus green CI, but notes that 10,000 lines of code that do nothing would still pass, so outcome checks are still needed."
    ],
    "takeaways": [
      "Build a custom verifier CLI for your own app before anything else. If your app doesn't have one, they call it among the highest-leverage things you can add. It should let any agent exercise and check the features you care about: persistence, network, console, storage, screenshots.",
      "Tell agents the outcome and how to check it ('you'll know you're here when this command returns this'), not the step-by-step path. Frontier models do better when they're free to be creative inside clear constraints.",
      "Write explicit, detailed error messages so agents can fix things themselves, and print state that would otherwise be invisible (e.g. ball coordinates on a canvas) so the agent has something to loop on.",
      "Keep receipts of runs (user journeys, failures, recordings) and attach screenshots or walkthrough videos to PRs with the gh CLI. This lets humans review the result before the code, and builds eval datasets for future self-improving loops.",
      "Make verification hard to skip: put 'before you open a PR, run X' in an AGENTS.md (not a CLAUDE.md), use shared Claude Code hooks or CI, and consider an agent that watches production rollouts (metrics, Datadog, feature flags) and reverts bad deploys."
    ],
    "topics": [
      "agents",
      "verification",
      "evals",
      "feedback loops",
      "testing",
      "receipts",
      "llm-as-judge",
      "developer productivity"
    ],
    "tools": [
      "WorkOS",
      "Playwright",
      "pnpm",
      "Chromium",
      "Chrome DevTools",
      "Cloudflare",
      "CTX",
      "Riker",
      "MCP",
      "GitHub API",
      "gh CLI",
      "Claude Code",
      "ChatGPT",
      "Pi",
      "Amp",
      "Bun",
      "Rust",
      "Go",
      "TypeScript",
      "React Testing Library",
      "Reptile",
      "CodeRabbit",
      "Codo",
      "Datadog",
      "AGENTS.md",
      "Fable",
      "Opus 5.5",
      "Jev",
      "Cleft",
      "Laracon",
      "X"
    ],
    "quotes": [
      {
        "text": "if an agent can't verify its own work, nothing else matters.",
        "at": "07:44",
        "url": "https://www.youtube.com/watch?v=q4WC4TowkBo&t=464s"
      },
      {
        "text": "status comes from records, not model pros.",
        "at": "11:48",
        "url": "https://www.youtube.com/watch?v=q4WC4TowkBo&t=708s"
      },
      {
        "text": "a successful screenshot isn't proof.",
        "at": "31:18",
        "url": "https://www.youtube.com/watch?v=q4WC4TowkBo&t=1878s"
      },
      {
        "text": "at this moment your agent realizes I am not done and I'm not going to drop this work on the floor.",
        "at": "36:53",
        "url": "https://www.youtube.com/watch?v=q4WC4TowkBo&t=2213s"
      }
    ],
    "words": 13863
  },
  "summary_url": "/#q4WC4TowkBo",
  "transcript": {
    "html": "/transcripts/q4WC4TowkBo.html",
    "txt": "/transcripts/q4WC4TowkBo.txt",
    "vtt": "/transcripts/q4WC4TowkBo.vtt"
  }
}