{ "video": { "id": "v5tm5EAzsAs", "title": "Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI", "duration": 1165, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "v5tm5EAzsAs", "title": "Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI", "one_liner": "A one-line prompt change can improve one eval dimension by 17 points and degrade another by 17, so gate every agent change with experiments run on real production traces and make every failure a new regression test.", "summary": "Sofia Jakovcevic (Arize AI) argues that agents fail at runtime rather than in the code, so a diff can't show what a change will actually do; the trace can. Using a support/refund agent where a security engineer adds 'confirm before taking any action', she shows action safety rising 17 points while task completion drops 17, for a net-zero average. She lays out a five-step regression loop: curate a production dataset, build evals, run a baseline experiment, rerun with the change, and gate merges in CI. She demos it in Arize's prompt and agent playgrounds. She closes by arguing that observability is moving from humans understanding agents to agents debugging and improving themselves, with coding agents connected to production traces through Arize's agent skills.", "key_points": [ "Worked example: adding a one-line 'confirm before any action' policy gave +17 points on action safety and -17 on task completion, because users leave the session before confirming and tasks go uncompleted. The net average is zero, so improving one dimension doesn't mean the agent as a whole improved.", "Agents fail at runtime, not at the code level. A one-line system-prompt markdown change is easy to review but changes the steps the agent takes, how it hands off to sub-agents and how it calls tools. Those possibly thousands of trajectories exist only in the traces, so traces are becoming the new source of truth.", "The five-step loop: (1) select a dataset from real production traces, (2) build evaluators, (3) run a baseline experiment, (4) add the one change, rerun and compare, (5) push the experiment into CI (e.g. GitHub Actions) so a PR doesn't merge until it meets the regression criteria.", "The dataset should cover four things: representative average workflows, known failures, edge cases, and successful workflows you want to protect. Teams often forget the last one.", "Evaluators turn subjective metrics (hallucination, accuracy, tool trajectory, step efficiency) into numeric scores using an LLM as a judge, e.g. a binary complete/incomplete task-completion eval, or 0–5 scales and multiple labels. Code-based evals suit deterministic checks, LLM judges suit non-deterministic ones, and ML-based evals are also an option.", "In the SDK the experiment is a single run experiment call: pass the dataset, the task and the eval suite. The same rerun-and-compare loop can test prompts, handoffs, tools and models without deploying to production. It also justifies whether a model that is 'five times more expensive' is worth it.", "Arize demo: the prompt playground tests a single agent (a prompt from the prompt hub, version 1, against a 100-row curated dataset) with prompts side by side and pre-built evals such as task completion, action safety, hallucination and user frustration. The agent playground connects a remote agent over HTTP to test multi-agent systems. Experiment views track latency, evals, total cost and error rates over time. The final iteration reached +18 on action safety and +6 on task completion.", "Arize publishes a public repository of agent skills that exposes every UI feature, letting coding agents link the code they produce to what is actually happening in the observability backend." ], "takeaways": [ "Don't ship prompt changes on intuition or 'vibes'. Score every change against a baseline across several eval dimensions and look for trade-offs, not just the metric you meant to improve.", "Build regression datasets from real production traces, and deliberately include successful workflows you want to protect alongside known failures and edge cases.", "Wire experiments into CI (e.g. GitHub Actions) so each PR runs against your team's chosen evals and is blocked until it meets the regression criteria.", "Turn every production failure into a new case in the regression suite so the suite keeps growing.", "Give coding agents access to production traces (e.g. via agent skills) so they aren't 'pushing PRs blind' using only static code." ], "topics": [ "agents", "evals", "observability", "regression-testing", "llm-as-a-judge", "ci-cd", "tracing", "self-improving-agents" ], "tools": [ "Arize AI", "Arize prompt playground", "Arize agent playground", "Arize prompt hub", "Arize agent skills", "GitHub Actions", "Fable (model)" ], "quotes": [ { "text": "the diff tells you what changed, but the trace tells you what actually happened.", "at": "05:43", "url": "https://www.youtube.com/watch?v=v5tm5EAzsAs&t=343s" }, { "text": "just because you improve the dimension of one agent does not mean that you're improving the agent as a whole.", "at": "04:27", "url": "https://www.youtube.com/watch?v=v5tm5EAzsAs&t=267s" }, { "text": "every failure should become a test.", "at": "17:14", "url": "https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1034s" }, { "text": "observability is no longer about humans understanding agents, it's about agents understanding themselves.", "at": "18:15", "url": "https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1095s" } ], "words": 4189 }, "summary_url": "/#v5tm5EAzsAs", "transcript": { "html": "/transcripts/v5tm5EAzsAs.html", "txt": "/transcripts/v5tm5EAzsAs.txt", "vtt": "/transcripts/v5tm5EAzsAs.vtt" } }