{ "video": { "id": "xyL2Ltkh-SA", "title": "How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads", "duration": 1169, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "xyL2Ltkh-SA", "title": "How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads", "one_liner": "Two YouTube Ads engineers explain how they built production evals for an ad-editing agent: optimize the tools first, deliberately \"vibe\" on a tiny intuition-based eval early, then scale to rated golden sets, rater explanations and agent-trace inspection — because a pass/fail percentage hid the agent stripping legally-required disclaimers.", "summary": "Preetika Bhateja and Daniel Bump, who work on image and video models for YouTube ads, argue that agent reliability is a function of the agent's capabilities, its guardrails and its evals — and that you build them in that order, optimizing a focused set of LLM-friendly tools (plus an independent critique agent with a remediation loop) before investing in a comprehensive eval. Counterintuitively they recommend starting with non-scalable, intuition-based \"vibe\" evals on a few core tasks, because jumping to scaled raters too early means calibrating the eval while the model is still changing radically. As the eval scales they add clear rubrics with worked examples, rater explanations rather than bare pass/fail, LLM judges monitored by human-vs-LLM disagreement sampling, and trace inspection. Their worked example: despite a prompt repeatedly saying disclaimers can never be removed for legal reasons, traces showed the agent finding a disclaimer and deciding to remove it — invisible in the aggregate pass rate.", "key_points": [ "Reliability framing: \"the reliability of your agent is basically a function of the capabilities of the agent, the guard rails and the evals\" — so optimize a focused, strong set of LLM-friendly tools first, before building larger agent evals.", "Add an independent critique agent with a remediation loop as a self-correction mechanism that fills gaps where the base tool set has limitations.", "Early on, non-scalable \"vibing\" beats a comprehensive eval: at that stage failure patterns are easy to spot by eye, prompt tweaks give large performance gains, and radical architecture changes aren't blocked by a half-calibrated eval. Scaling to raters too early produces big ups and downs because you're calibrating the eval while the model is still moving.", "Start small with a few core tasks rather than a massive golden set on day one — and test the negatives: confirming the model *didn't* do something bad is as critical as confirming it did the task.", "Writing the eval is the small part; humans arguing over what the rubric should be is the large part. Raters kept coming back with edge cases, and the team itself disagreed on pass vs fail, so human-human agreement inside the team is the prerequisite.", "Collect rater explanations, not just pass/fail — in their multi-output ad evals (accuracy, brand safety, \"is it what we expected\"), a result is rarely a clean pass/fail; an ad can score well on brand safety and badly on accuracy, and only the explanation tells you where the agent is missing.", "For LLM/auto raters: run a sampling pipeline monitoring agreement/disagreement between human expert ratings and LLM ratings so you can see how it trends, and spot-check the judge's reasoning rather than just its verdict.", "The disclaimer demo: the prompt stated several times that for legal reasons disclaimers can never be removed, yet edge-case traces showed the agent say it found a disclaimer and then remove it (a public-parks ad reading \"America, we can do better\" with \"paid by the community of parks\" bottom right). A categorical x% pass rate could not surface this; the trace could.", "Traditional ML discipline still applies: agents don't generalize beyond their training data, so keep an edge-case/capability dataset and a held-out test set, use the test set sparingly, and refresh it with production data.", "Launch readiness: identify where and why performance degrades to distinguish acceptable trade-offs from critical failures, and fix the gatekeeping rule (e.g. a precision/recall bar) early rather than at launch." ], "takeaways": [ "Don't build the comprehensive eval first. Harden your agent's tools, then run a deliberately small, intuition-based eval on a handful of core tasks while you're still making radical changes — and only bring in scaled/LLM raters once the architecture has stopped moving.", "Instrument reasoning, not just outcomes: read agent traces and require raters (human or LLM) to give explanations, because the failure mode you care about — like an agent removing a legally-required disclaimer it explicitly detected — can be invisible in a pass-rate number.", "Invest in rater templates, rubrics and worked examples, plus actual training sessions for cross-functional and scaled raters, so you stop getting \"I'm not sure how to rate this\" and items marked unknown. Settle human-human agreement within your own team first.", "Fix on patterns, never single runs: put multiple examples of each failure pattern in the golden set and judge by how often the pattern fails, instead of patching the prompt from one non-deterministic bad trace.", "Treat the eval as a living system — online evals, sampling pipelines, golden sets that evolve with use cases, and test sets refreshed from production — and decide your launch criteria/metric up front." ], "topics": [ "evals", "agents", "llm-as-judge", "golden-sets", "agent-traces", "prompting", "human-raters", "launch-readiness" ], "tools": [ "YouTube Ads", "AI Engineer (conference)" ], "quotes": [ { "text": "the reliability of your agent is basically a function of the capabilities of the agent uh the guard rails and the evals", "at": "02:38", "url": "https://www.youtube.com/watch?v=xyL2Ltkh-SA&t=158s" }, { "text": "early on vibing can actually be kind of good for you", "at": "04:03", "url": "https://www.youtube.com/watch?v=xyL2Ltkh-SA&t=243s" }, { "text": "if you if you want to know what it's doing look at it at its thinking", "at": "11:09", "url": "https://www.youtube.com/watch?v=xyL2Ltkh-SA&t=669s" }, { "text": "you should focus on patterns rather than isolated runs so a tempting thing is to hyperfixate on very small examples from the model", "at": "14:50", "url": "https://www.youtube.com/watch?v=xyL2Ltkh-SA&t=890s" } ], "words": 4045 }, "summary_url": "/#xyL2Ltkh-SA", "transcript": { "html": "/transcripts/xyL2Ltkh-SA.html", "txt": "/transcripts/xyL2Ltkh-SA.txt", "vtt": "/transcripts/xyL2Ltkh-SA.vtt" } }