{ "video": { "id": "F0TNSmbo5hE", "title": "From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI", "duration": 2537, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "F0TNSmbo5hE", "title": "From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI", "one_liner": "Laurie Voss argues that observability is moving from humans reading dashboards to agents reading traces. Arize's Signal agent automates the loop of traces, then evals, then signals, then fixes: it scans production traces every six hours and turns recurring failures into GitHub issues, datasets, evaluators or ready-to-review PRs.", "summary": "Agents are non-deterministic, so the code is no longer the source of truth for what an agent does; traces at scale are. The 2025 answer was LLM-judge evals, but in 2026 there are so many evals that humans become the bottleneck again. The fix is to have agents read your traces and evals, find failure patterns (signals) and generate fixes, so that 'now that we are building at agent speed we have to improve at agent speed as well.' In a live demo, Voss used Claude Code with pre-installed Arize skills on a toy-store agent to add tracing, diagnose its traces and add a missing price filter without touching the code himself. He then showed Signal doing the same thing continuously inside Arize AX.", "key_points": [ "Agents are non-deterministic and multi-turn: the same input can give a different output, or the same output by a different path. Traces at scale, not the code, are the source of truth for what an agent does on average.", "Voss lays out three loops. The 2025 loop goes from no visibility to traces, then evals, then manual improvements. The 2026 loop adds signals and automated fixes: no observability, traces, evals, signals, automated fixes, and back again.", "LLM-based evals give an explanation as well as a score of 1 or 0. A coding agent can read that explanation and improve the software, so once you have defined what good looks like, 'your agent can climb' the hill by itself.", "Adding tracing to the Wonder Toys demo needed no hand-written code. The OpenAI Agents SDK is already instrumented with OpenInference, the industry's open tracing standard, so the coding agent only had to point the traces at the Arize endpoint and API key.", "Using the Arize skills and the ax CLI, Claude Code pulled the traces and found that 42% of searches returned zero results. It also found degenerate all-null search calls, exact-age filters that narrowed results too far, and a search_products tool that 'suspiciously returned in one millisecond'. It confirmed there was no price filtering, and then added it.", "Signal is an agent running inside Arize AX that reads all your traces every 6 hours by default (configurable). It found an age constraint the agent had hallucinated, bomb-related toys recommended in reply to a harmful query, unsolicited psychological advice, and the agent obeying an off-topic 'say hello' instruction.", "For each issue Signal can create a GitHub issue naming the trace and a suggested fix, add the trace to a dataset, create an evaluator, or open a PR on a connected repo.", "Signal runs on Claude Code (bring-your-own harness is planned) and Arize currently pays for running it. Signal is evaluated by its own internal eval suite and traced in Arize with LLM judges. It does not create evaluators automatically, because evaluators cost time and money to run. An on-prem version of Arize is available for compliance." ], "takeaways": [ "Treat production traces in aggregate, not the code, as the record of what your agent actually does. Turn on tracing early: frameworks that already emit OpenInference need only an endpoint and a key.", "Use LLM-judge evals whose explanations a coding agent can act on. Define 'good' carefully, because that definition is the hill your system climbs.", "Give your coding agent skills or a CLI for your observability platform (e.g. `npx skills install arize-ai`) so it can pull traces and diagnose quality problems. Look beyond error spans: Voss's demo turned up no error spans, just calls returning 200 with bad results.", "Keep regression evals for behaviour the agent already gets right, so automatically generated fixes don't silently break it.", "Have agents group failing eval results into recurring patterns, rather than reading thousands of eval explanations yourself." ], "topics": [ "agents", "observability", "evals", "tracing", "continuous-improvement", "coding-agents", "llm-as-judge" ], "tools": [ "Arize AI", "Arize AX", "Arize Signal", "Alex (Arize in-product agent)", "ax CLI", "OpenAI Agents SDK", "OpenInference", "OpenAI API", "Claude Code (VS Code plugin)", "npx skills", "GitHub" ], "quotes": [ { "text": "your code isn't the source of truth for what does my agent do. Instead, the traces are your source of truth.", "at": "03:51", "url": "https://www.youtube.com/watch?v=F0TNSmbo5hE&t=231s" }, { "text": "now that we are building at agent speed we have to improve at agent speed as well.", "at": "09:06", "url": "https://www.youtube.com/watch?v=F0TNSmbo5hE&t=546s" }, { "text": "Observability is shifting from humans reading dashboards to agents reading traces.", "at": "09:24", "url": "https://www.youtube.com/watch?v=F0TNSmbo5hE&t=564s" }, { "text": "The software improves itself. The software climbs the hill.", "at": "12:03", "url": "https://www.youtube.com/watch?v=F0TNSmbo5hE&t=723s" } ], "words": 6619 }, "summary_url": "/#F0TNSmbo5hE", "transcript": { "html": "/transcripts/F0TNSmbo5hE.html", "txt": "/transcripts/F0TNSmbo5hE.txt", "vtt": "/transcripts/F0TNSmbo5hE.vtt" } }