{ "video": { "id": "qZj7sqmidqA", "title": "From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI", "duration": 6687, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "qZj7sqmidqA", "title": "From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI", "one_liner": "Read a dozen or more of your traces before writing any eval. Then stack cheap code checks, LLM judges you have checked against human labels, and human review, and use the judges' written explanations as the feedback a coding agent turns into prompt fixes.", "summary": "Laurie Voss (head of developer relations at Arize, and a co-founder of npm Inc.) argues that most AI features ship on vibes, meaning someone runs a few queries and they look fine, and that evals (testing for AI) and traces (logs for AI) replace that with numbers you can act on. In a Colab notebook he instruments a Claude Agent SDK financial-analyst agent (Haiku, two turns: web research then report writing) with two lines of OpenTelemetry/OpenInference code. He reads its traces and uses open and axial coding to find failures such as silent writes to a missing disk, loops and unverifiable claims. He then builds code evals, built-in LLM evals, and a custom actionability judge, and checks that judge against human annotations. He finishes by putting the failing traces into a dataset, having Claude rewrite the prompts from the judges' explanations, and running an experiment to show the improvement.", "key_points": [ "Agents compound risk at every step, so you have to judge intermediate steps too: tool choice, arguments, knowing when to stop, and handoffs. One example is a cascading failure where a 'Tesla' search returns material about the 18th-century inventor and the agent writes a polished but wrong report. Another is Anthropic's tau-squared-bench, where the agent 'failed' by legitimately upgrading an economy ticket to first class so it could reschedule it.", "Reading the traces showed that 3 of 13 runs tried to write the report as a markdown file to a disk that doesn't exist in Colab. The write failed silently and the output said the report was saved. One Microsoft run looped through endless web searches, and the Rivian report made confident claims nobody could verify. Only reading the traces exposed these.", "A 5-line code eval that checks the requested ticker appears in the output passed 12 of 13. The one failure was an Amazon query about AWS: the report covered only AWS and never mentioned Amazon's ticker.", "The built-in correctness eval, judged by Sonnet 4.6, scored 0 of 13. The agent was citing live 2026 web data, and the judge's training data stops in January. Faithfulness, which gives the judge the same research context the agent had, gave a useful signal: 7 unfaithful and 6 faithful. Which eval works depends on the question you are actually asking.", "His recipe for a custom rubric: define the judge's role and context, list explicit and observable pass/fail criteria drawn from your error analysis, wrap the data in XML tags (Anthropic models like XML), and set the output choices outside the prompt, since AX gives the judge a tool to call with its label. He also adds good and bad examples, which he admits some practitioners dispute because of overfitting. Use binary labels rather than 1–5 scales, and ask the judge to reason before scoring.", "Meta-evaluation treats the judge as a classifier and compares it with human labels collected in AX's annotation UI. He labeled at random on purpose and got 46% agreement (6/13). Advice: split labels 75% dev and 25% held-out test, and usually prioritize recall over precision. Watch for position, length, confidence and self-preference bias. Human inter-rater reliability is often only 0.2–0.3, so occasional disagreement is acceptable; a judge that disagrees all the time is the danger sign.", "The failing traces were filtered (actionability = not actionable), saved as a dataset, and passed to Claude with the requirements and the judge explanations to rewrite the research and writing prompts. An experiment rerun on that dataset took the agent from roughly 50% to 100% actionable. He says real iteration won't be that smooth.", "Sample sizes: 12–20 examples give a directional signal, and 200–400 is a good target for shipping decisions. Returns diminish: you have to quadruple the sample to shrink the margin of error. Order of investment: fix data quality first, then prompting, then model selection. Hyperparameters such as temperature and top-p come last." ], "takeaways": [ "Start small: add the two lines of tracing, spend about 15 minutes reading real outputs, open-code what you see, then axial-code it into categories. Rank fixes by severity times frequency, not frequency alone.", "Write down your success criteria with domain experts such as PMs, QA and support. Then build one focused evaluator per dimension instead of a 'god evaluator'. Grade the output, not the path the agent took, and decide which evals are guardrails (ship-blockers that page someone) and which are north-star metrics.", "Treat eval prompts as code. Version them, test them against human-labeled golden sets with a dev/test split, and if the judge disagrees with humans on around 40% of examples, assume the eval is wrong. Use a stronger judge than your agent, ideally from a different lab.", "Close the loop with experiments on saved failure and regression datasets. When you hand failing traces and explanations to a coding agent, include your requirements so it doesn't cheat by just making the evals pass, and tell it to fix recurring themes rather than individual traces.", "In production, run online evals on a sample of traffic (1–10%) at span or trace scope, feed them into monitors, and turn every caught failure into a new regression test case." ], "topics": [ "evals", "agents", "observability", "llm-as-a-judge", "error-analysis", "meta-evaluation", "experiments", "prompt-engineering" ], "tools": [ "Arize AX", "Arize Phoenix", "Arize skills", "Alex (AX agent)", "Claude Agent SDK", "OpenInference", "OpenTelemetry", "Anthropic", "Claude Haiku", "Claude Sonnet 4.6", "Claude Code", "Cursor", "Codex", "OpenAI", "LangChain", "CrewAI", "Mastra", "Google Colab", "tau-squared-bench", "Descript", "Bolt", "npm" ], "quotes": [ { "text": "Three times is not a test suite.", "at": "09:23", "url": "https://www.youtube.com/watch?v=qZj7sqmidqA&t=563s" }, { "text": "specific criteria produce consistent judgments uh while vague criteria produce coin flips.", "at": "69:16", "url": "https://www.youtube.com/watch?v=qZj7sqmidqA&t=4156s" }, { "text": "If you find yourself writing a rubric that lists six different things that the response should do, then stop. That is six evaluators, not one.", "at": "79:21", "url": "https://www.youtube.com/watch?v=qZj7sqmidqA&t=4761s" }, { "text": "15 minutes of reading real outputs will teach you more about your application than an hour of building dashboards.", "at": "109:27", "url": "https://www.youtube.com/watch?v=qZj7sqmidqA&t=6567s" } ], "words": 22446 }, "summary_url": "/#qZj7sqmidqA", "transcript": { "html": "/transcripts/qZj7sqmidqA.html", "txt": "/transcripts/qZj7sqmidqA.txt", "vtt": "/transcripts/qZj7sqmidqA.vtt" } }