{ "video": { "id": "A3FL_zTQKU0", "title": "Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics", "duration": 1032, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "A3FL_zTQKU0", "title": "Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics", "one_liner": "Benchmarks only shortlist candidates: evaluate the exact model-plus-harness configuration you will run, on tasks drawn from your own workload, using metrics that match your business need, and keep traces so you can find where the pipeline actually broke.", "summary": "Stephanie Jarmak is a former astronomer who built the CodeScaleBench evaluation work at Sourcegraph and is now an AI engineer at Omni Analytics. She argues that agent evals should be run like science experiments and that you should mostly not trust your agents. Her examples include a prompt change that made a run 28% cheaper while the agent still never called the target MCP tool, and a harness swap that reversed a model comparison by more than 30%. She also covers grep pulling in 295 files when only 6 were needed, which shows why the choice of metric matters. Beyond evals, she says flagging a problem must be kept separate from enforcing a gate, human review load has a real cost, agent traces can themselves be untrustworthy, and in the end 'production is truth'.", "key_points": [ "At Sourcegraph she changed the initial prompt to get a coding agent to use a code search MCP tool. The run came out 28% cheaper, but the agent still never used the tool, so the headline metric didn't answer the question she was asking.", "Third-party results on 63 or so tasks from a popular coding benchmark: Opus 5 beat Astra in the OpenHands harness, but swapping only the harness flipped the result to Astra, a difference of 30+%. Evaluate the configuration closest to your real environment.", "An evaluation means taking measurements to make a decision about your product or process. Public benchmarks are for understanding the landscape and building a shortlist. Don't over-index on them, because model training is 'spiky' (a law firm shouldn't choose by Terminal Bench results).", "She built her own coding benchmark because SWE-bench was largely bug fixing. Tasks across the software development life cycle need coverage, since a model may be better at code understanding than at fixing bugs. Design should account for workload coverage via task families, multiple runs (agents are non-deterministic), and interaction effects between model and harness.", "Debug the pipeline step by step: was the tool available in the agent's context, was it called, what did it return, and did the final task outcome change? Even after a better tool gets used, the agent may still not use the information it returns.", "Choose the metric deliberately. She optimized for F1 (precision as well as recall) and found Claude with grep pulled 295 files when 6 were needed. Recall of 1 is only enough if all you care about is pulling in anything that might be relevant.", "In her software factory, an agent noticed things going wrong but had no mechanism to stop the work. Keep the flagging separate from the enforcement, which should be a gate in code, with escalation to a human after that. Human review load counts too: a 1% false alarm rate in an LLM-as-judge system would have meant 10,000 human reviews.", "Researchers seeded security issues and relied on a hash check against tampering; everything passed and nothing was detected. An independent record comparison of traffic history caught all of them. Agent traces can be untrustworthy." ], "takeaways": [ "Before choosing a model, evaluate the full model-plus-harness configuration as close to your real environment as you can, and use public benchmarks only to build a shortlist.", "Build an internal benchmark from production usage, classified into task types (at Omni: easy vs. hard Q&A, dashboard building, app building). Run it several times and improve against it iteratively, hill-climbing toward better configurations.", "Keep all traces, and when debugging walk the pipeline in order: tool available → called → values returned → used → final outcome.", "Have code enforce gates instead of trusting the agent to stop itself. Set alarm rates with your human review capacity in mind, and keep an independent record of what happened rather than relying on the agent's own trace.", "Treat production as the ground truth. Measure after you deploy (in shadow mode where the blast radius is high), and define explicit risk boundaries for when an agent can act autonomously and when it must escalate to a human." ], "topics": [ "evals", "agents", "benchmarks", "coding agents", "observability", "human-in-the-loop", "agent safety" ], "tools": [ "Omni Analytics", "Bloopy", "Sourcegraph", "Amp", "CodeScaleBench", "MCP", "Terminal Bench", "SWE-bench", "OpenHands", "Opus 5", "Astra", "Claude", "grep", "GitHub" ], "quotes": [ { "text": "It says how to trust your agents, but I think largely this talk is mainly about how to not trust your agents.", "at": "01:04", "url": "https://www.youtube.com/watch?v=A3FL_zTQKU0&t=64s" }, { "text": "I want to automate as much as possible with uh for the work to get done, but I also do not trust these systems like at all.", "at": "10:20", "url": "https://www.youtube.com/watch?v=A3FL_zTQKU0&t=620s" }, { "text": "Claude went out and got like 295 files returned when it only really needed six.", "at": "09:16", "url": "https://www.youtube.com/watch?v=A3FL_zTQKU0&t=556s" }, { "text": "ultimately like production is truth.", "at": "14:38", "url": "https://www.youtube.com/watch?v=A3FL_zTQKU0&t=878s" } ], "words": 3389 }, "summary_url": "/#A3FL_zTQKU0", "transcript": { "html": "/transcripts/A3FL_zTQKU0.html", "txt": "/transcripts/A3FL_zTQKU0.txt", "vtt": "/transcripts/A3FL_zTQKU0.vtt" } }