{ "video": { "id": "pz74r-4dGfQ", "title": "Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp", "duration": 1160, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "pz74r-4dGfQ", "title": "Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp", "one_liner": "Have LLM judges score every agent conversation against simple markdown rubrics, then let a cron-driven self-improvement agent mine the failures and propose skill and config PRs, because agents silently work around problems and report only that it worked.", "summary": "Ben Holmes of Warp argues that software factories need 'factory engineering': agents that introspect their own past work and improve their own skills and configuration. The loop is to centralize conversation traces (transcript, tool calls, outputs), grade them with LLM-as-judge scorers on rubrics such as redundant tests, verbosity and efficiency, then have a self-improvement agent find commonalities among failures on a cron and suggest changes. He demos the one-shot Skill Doctor skill and describes Warp Factories' continuous version, whose score dashboards let you attribute changes in a metric to a specific merged skill, something he says memory or 'dreaming' systems leave invisible.", "key_points": [ "Persistent agents hide problems: they work around roadblocks and report that it worked. In Warp's test harness, out-of-date skill instructions went unnoticed because the agent pinged port numbers until it figured out its environment; the only visible symptom was a run taking about 2 minutes longer than expected.", "Traces need three things: the transcript, the tool calls (including file system and MCP calls), and the output, such as code diffs or the pull request. Local agents like Claude Code and Codex store these in per-tool folders; some teams zip everyone's .claude folders into a shared S3 bucket, while a shared cloud runner gives you centralization 'for free'.", "Example scorers are redundant test (too many tests, or the wrong kind for your coverage preferences), verbosity (overly wordy messages and code comments, which can be barriers to getting a change reviewed quickly) and efficiency (working towards the result versus spinning its wheels, judged from tool calls).", "A scorer is just markdown, like a specialized skill definition: what is being judged, labels defining success and failure, and a short rubric. In Warp's example the pass/fail bound sits midway through the list, so 0.4 or 0.2 is a failing grade. Holmes advises keeping it succinct and building it up, calling it 'more of an art than a science'.", "In Warp's setup, grading is usually triggered by a PR merge, with one agent run per conversation, either on every conversation or by random sampling to save tokens. Each grade records pass/fail per criterion plus a written explanation of why.", "Every 24 hours a cron job can take a sampling of recent conversations (say 200), filter down to the failures (say 50), and pass the failure notes to a self-improvement agent with access to the skills repository and code base so it can find commonalities and suggest changes.", "Skill Doctor (warp.dev/skilldoctor) does a one-time grading of your local Claude, Codex and other agent conversations, in batches for token efficiency, and produces a skill report card. In the demo it proposed better port-finding instructions in an end-to-end testing skill and more prescriptive language in a changelog skill.", "Warp's own factory has an inbox of self-improvement PRs for team review; one merged PR patched factory config missing from the triage skill, where the agent kept wasting tool calls trying to work out how to spawn sub-agents." ], "takeaways": [ "Get your whole team's agent conversations into one place, and include tool calls, MCP calls and outputs as well as the transcript, so failures that never surface in the chat can be analysed.", "Start with a few scorers for problems your team actually notices, such as redundant tests, verbosity or efficiency, written as a couple of bullet points and a rubric, then refine them as you go.", "Automate the loop rather than relying on someone remembering: grade when a PR merges, then run a daily self-improvement agent over the failures and have it suggest changes to skills and config.", "Chart scores over time and tie inflection points to merged skill changes, so you have proof that a suggested improvement actually did something.", "Try a one-shot pass with Warp's Skill Doctor on your own setup to find low-hanging fruit before building anything continuous." ], "topics": [ "agents", "software factories", "self-improving agents", "evals", "llm as judge", "agent skills", "agent observability" ], "tools": [ "Warp", "Warp Factories", "Skill Doctor", "Claude Code", "Codex", "Pi", "OpenClaw", "SonarQube", "Amazon S3", "Docker", "GPT", "MCP" ], "quotes": [ { "text": "an agent these days are very persistent and will work around whatever roadblock that they run into, and they won't tell you about it.", "at": "08:18", "url": "https://www.youtube.com/watch?v=pz74r-4dGfQ&t=498s" }, { "text": "It's more of an art than a science.", "at": "10:38", "url": "https://www.youtube.com/watch?v=pz74r-4dGfQ&t=638s" }, { "text": "it's very invisible what the agent actually decided or what takeaways that it got from all the conversations.", "at": "16:52", "url": "https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1012s" }, { "text": "you also have some proof points of like oh, I shipped this change and I got this result out of it.", "at": "17:36", "url": "https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1056s" } ], "words": 4516 }, "summary_url": "/#pz74r-4dGfQ", "transcript": { "html": "/transcripts/pz74r-4dGfQ.html", "txt": "/transcripts/pz74r-4dGfQ.txt", "vtt": "/transcripts/pz74r-4dGfQ.vtt" } }