Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

AI Engineer · 19 min · 245 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello everyone.
00:13My name is Sophia Yakoubovitch.
00:15I'm a forward deployed engineer at Arize, and for those who aren't familiar, Arize is an observability backend that is designed to make AI work.
00:24Now, what does that mean?
00:25Building AI systems can be inherently intuitive in some sense, especially when a lot of that low-level abstraction is a low-level programming is abstracted into the mind of an agent.
00:37Unfortunately though, that mind can be a bit of a black box.
00:41When that black box is difficult to understand, it's difficult to predict, and therefore difficult to debug.
00:46And that's where observability becomes essential.
00:49Observability helps you understand what your systems are actually doing.
00:53It's essentially shining a light in the black box.
00:56And once you understand what the system is doing step-by-step, you can then evaluate it.
01:01Run evaluation metrics to track its performance, and then use that data to iterate and improve your agents over time.
01:08But I want to take this one step further, because the biggest challenge isn't determining what broke, the biggest challenge is that when you do submit a fix, you want to make sure that you don't break 10 other things along the way.
01:20Now, raise your hand if you've had that exact issue before.
01:23You push Yep, seeing a lot of hands.
01:26You push a simple change, you know, should be a should be easy, no issues, and next thing you know, everything's on fire.
01:34So, today we're going to discuss how you can use your observability as a way to build a robust regression process.
01:40Not only something that your engineers can do, but an automatic process that feeds into your CI/CD.
01:47We're going to break this down to three parts.
01:49First, we'll discuss why agent regression is so different and difficult to detect.
01:53And the first thing we're going to touch on is that agents don't fail at the code level, they fail at runtime.
01:59We're then going to use this premise to build out our regression loop.
02:03It's going to be a five-step process.
02:05We'll touch on it at a high-level overview, and then we'll move on to the Arize platform and show you exactly how you can do it today.
02:11Then, we'll close off the session by closing that regression loop at agentic speed and touching on where we're seeing the next frontier of agent observability move to.
02:21And if you stick with me till the end, I'll introduce a paradigm shift that I hope you'll all find interesting.
02:26So, let's get started.
02:27First, with a concrete example.
02:29So, imagine with me that you have a support agent, and you have a security engineer that has approached you.
02:34And that security engineer has reviewed the policy of your support agent and determined that he wants to add one line, a simple one line to your agent that will ask your agent to confirm before it takes any action.
02:48Sounds simple.
02:49Sounds harmless.
02:50Your security engineer says he has the intuition, and oftentimes we're quick to make this change.
02:56I'm not even going to ask if you've done this before cuz I know everyone has pushed a one-line prompt change and just assumed everything will go well.
03:03And the reason we make that assumption is because we think that because it's written in natural language, that we have this intuition on how these agents are performing.
03:11We're really coding on vibes, and we don't want to be.
03:15We want to be able to code with a numeric evaluation suite backed data.
03:21We want to have scores that prove that these changes are actually making a positive impact and not just crossing our fingers.
03:27But let's say in the scenario you did push that one-line change, and now you are running your evaluations on your production data a week after it's been submitted.
03:35And the first thing you notice is that you actually have a 17-point improvement on a action safety.
03:41So, your security engineer was correct.
03:42It asking the agent to confirm before taking any action does improve its overall safety.
03:48But your observability engineers are continuing to evaluate, and they also check for task completion and they notice a 17-point degradation.
03:58Now, this is occurring because if you have an agent that's confirming before every single action it's taking, now tasks are being uncompleted.
04:06If a user leaves the session, that session is never completed to the end and I have incomplete task and you see this unlikely result.
04:17You may not have caught this initially when you were thinking about the project, but your evaluation suite has caught this.
04:21And ironically enough, the overall average of both is net zero.
04:26And what I want you to take away from here is just because you improve the dimension of one agent does not mean that you're improving the agent as a whole.
04:32The question I have for you is, would you have shipped this change?
04:37Many of us would have before the evaluation test.
04:39And this is why it's so important that we get these numeric results so that we're not shipping on vibes, we're shipping with backed data.
04:48So, to understand where the mistake that we made is is we made an unrealistic assumption.
04:53And that assumption is that the error is going to occur at the code, but the error rarely occurs at the code level.
04:59Instead, these agents are failing at runtime.
05:03What the code reviewer saw is here on the left-hand side.
05:07It was a one-line change in the system prompt markdown file.
05:11Easy change, simple to understand, simple to approve.
05:14But in reality, this code change was local, but the behavioral changes of this code change was in production.
05:21It was in the runtime.
05:22This one simple line has shifted how the agent operates.
05:26It's adding additional steps.
05:27It's changing how the agent works with other sub-agents.
05:30It's changing how the agent calls tools.
05:32All of these different trajectories, which could be thousands of possible trajectories, are not in your static code.
05:37They're only in your observability back end.
05:40They're only in your traces.
05:42And so, what we are saying here is that the diff tells you what changed, but the trace tells you what actually happened.
05:49And there's a shift here, that traces are becoming the new source of truth.
05:53These structured logs of what your agents are actually doing actually is what truly represents what these systems are doing.
06:00In traditional software, logic was written down.
06:03It was very clear to determine what that runtime was before you ever pushed production.
06:08But now the logic only exists at runtime.
06:11The observability becomes essential when we're looking at regression testing.
06:16This gap between what was written and what actually happens is filled with silent errors.
06:20So to capture that, we need to start our regression testing by looking at our traces.
06:26So let's build out this regression loop together.
06:28It's a five-step process.
06:29We're going to start with our production traces.
06:32We're going to build evals, run experiments, and then close it all up to be an automatic gate at CI.
06:39First, I'm going to do a five-step high-level overview.
06:43And then we're going to go into the actual product and show you how you can do it yourself.
06:48So step one is we're going to select a data set.
06:50Seems simple enough.
06:51We want to pull real production traces.
06:54We don't need to synthesize data.
06:55If we already have real user queries, real inputs, real tool calls, we're pulling from that directly.
07:01It's important that our data set is a representative work is representative of the average workflow that our users are going to be pushing into our systems.
07:09But it's also important that we capture known failures.
07:12We filter out com moments where often errors, often fails at eval metrics.
07:18Capture those into your data data sets.
07:21You also want to capture edge cases.
07:23Maybe not as common, but just as important.
07:26And one thing that we often see users forget is capturing successful workflows that we want to protect.
07:32Because again, we're not just trying to make sure that the fix is going to work, but we also want to test that we're not going to break something that was already working.
07:41So it's important that we capture all four of these in our data set.
07:45Once we have this production data set, we can then build out the rest of our regression testing suite.
07:50The second step is we want to have a scoring mechanism.
07:53And this is where we bring in evaluators.
07:54Now, show of hands, how many here are familiar with the concept of evaluating agents?
08:01I'm so happy to see so many hands.
08:03This is a core foundational part of agentic engineering.
08:06Evaluators allow us to take metrics and dimensions that are important to your engineers, your product team, and actually quantify them.
08:14What do I mean by that?
08:15Let's take some metrics that are important for your agents, things like hallucination, accuracy, tool trajectory, step efficiency.
08:24All of these metrics are subjective in nature.
08:28And you may have a human that reviews them and can potentially determine whether or not something was hallucinated, whether or not the most efficient pathway was taken.
08:35But we want to be able to automate this process.
08:37We want to turn a subjective metric into a numeric score.
08:41And we do this by leveraging LLMs.
08:44We use a concept of LM as a judge, where we judge a span, a trace, some log data against an LM.
08:52The LM has a particular guide of how it determines whether or not something was hallucinated or factual, something was efficient or not efficient.
09:01And now we've done essentially a binary classification.
09:04In this example at the top right corner, you could see a task completion eval, where the LM as a judge is has a task of determining whether or not something was complete or incomplete.
09:14Now we've taken a subjective metric and we've essentially turned it into a binary score, 0 1. You can translate the score across 0 to 5. You can have multiple different labels.
09:23But you've now turned a quantitative measurement into a qualitative score that you can measure against.
09:29You're not limited to LM as a judge, you can also use code-based evals or ML-based evals as well, determine based on what that metric is.
09:36A code evaluator is great for your deterministic checks, but an LM as a judge is good for your non-deterministic checks.
09:42So, once we have a list of all of our metrics that are important to us, we build evals for each of them.
09:48Now, we have a testing suite that we can go against.
09:51Step three is we're going to run our baseline experiment.
09:53And what this experiment is is we're going to take our agent and run it against all of our production traces, um all of our production data sets.
10:03We're after we get the output of those production data production data sets, we're then going to run our eval testing suite on top of that.
10:09And that's what's going to be our baseline.
10:11Now, you can do this in the SDK, it's just run experiment.
10:14You'll pass in that data set, pass in that task, pass in your evaluation suite, and you'll get those results.
10:20We're going to touch on how you can do this in the UI, as well as how you can do this with your coding agents.
10:24But, once you have that baseline experiment, you're then going to add your one change.
10:28Change that policy, rerun the evaluation, what are the eval results?
10:33A simple process.
10:34Now that you can quickly iterate on, and not just with one policy change, but test multiple changes within your prompts.
10:41Test multiple models.
10:42If your team comes with you comes to you and says, "We want to switch the new fable model, it's five times more expensive."
10:48You're not going to switch without proving that you're going to get a good ROI.
10:51This is how you can prove that with a numeric set of measurements to determine whether or not it actually is improving your investment to switch to a more expensive model.
11:01And you can adjust this policy, rerun, and compare.
11:04Rerun and compare how changing the prompt, changing handoffs, changing tools, changing models is all going to affect your overall output.
11:11All without having to deploy to production.
11:14And then we can automate this.
11:16We push this experiment into CI to complete this automation.
11:20Now, each time that you have a pull request, let's say you set this up in GitHub actions, for example, each pull request each PR is run through an experiment workflow against the evaluations
11:30that your team has deemed important.
11:32And you don't merge until it meets all of the regression criteria that you have listed.
11:37Now, you're no longer pushing on vibes, you're pushing with confidence that you actually do have a positive impact on your performance.
11:44And if it is negative, you go back, you test, and you reiterate.
11:48So, let's actually show how to do this in the platform.
11:51Let me switch over.
11:53Um here we have Here we have the Arize platform, and I'm showcasing this exact refund agent.
11:59You can see in our baseline experiment what our scores were, and then you can see that plus 17 points on action safety and minus 17 points on task completion.
12:08Each one of these points Each one of these line items here is a different test that I ran with different prompt changes, potentially different model changes, and I'm evaluating each one of them to see which one has the best trade-off
12:20to get to where my end goal is.
12:22And I can see that overall I'm going to have a plus 18 18 points on action safety and plus six on task completion.
12:28Now, how can we build these individual points?
12:30Well, there's a couple ways.
12:31We'll do this first in the UI.
12:34I'm going to jump into the playground here.
12:36I'm going to create a new playground.
12:38We're going to start with the prompt playground to test a single agent.
12:42Let's call this AIE NYC.
12:46Perfect.
12:46Once I'm in the prompt playground, here is where I can test a single agent.
12:51An agent A single agent all is essentially just a single prompt with a set of tools attached to it, a model LLM configurations, and so we're going to replicate that here.
13:00Now, I've already stored my prompt in my prompt hub, so I can pull it right here.
13:03I've got version one, and then I can pull up my data set.
13:07I've already selected a data set that I can use here and that I've gone through and pulled out those edge cases, those known failures, and those successful workflows that I want to protect,
13:18all listed in this 100-row data set.
13:21I can test this right now, or I can test multiple prompts at the same time.
13:25So, I'm actually going to duplicate this, and I can change a line item directly in here, or I can load up a separate version right here and test side by side.
13:35So, I can see just how quick just how both of these prompts are affected.
13:41Now, of course, we have to add our scoring mechanism.
13:43So, that's going to be our evals.
13:44I'm going to click here.
13:46I'm going to add in my task completion, and then I'm also going to add in my action safety.
13:51And you can see here that there's a ton of other pre-built halluc pre-built evaluators you can select from, hallucination, user frustration.
13:59And we've got some code templates, security checks, etc. And you always have the ability to customize your own.
14:04Once I run this, it's going to run on every single record, and then it's going to pull out those eval scores.
14:12And then when I go into my data sets here, go into my experiment run flow, it'll pop up under here as well.
14:25And I can pull up all those metrics that are important for me.
14:27I can look at latency, I can look at my evals, I can look at my total cost and error rates, and track that over time.
14:33So, I can look this for PR changes, for playground runs.
14:37And there's one more other option that I want to show here, too.
14:39And that's if I go into playground, click here, and I do agent playground.
14:46Call this AIE.
14:48Oh.
14:48There we go.
14:49Agent playground allows you to not only evaluate at a single agent, but at multiple agent levels.
14:55If you give us access to connect to your agent remotely, for example, through a HTTP call.
15:01If I go into remote agents here, you can connect your remote agent directly here, so that you can change how different sub sub agents communicate with each other, all directly in your code base, and then we can call that.
15:12So, each record in your data set will call against your agent, will test, will get the output, will run those evals, and then score it in the same mechanism.
15:20So, you have two ways of running play running these experiments within the UI.
15:24You can run them in the prompt playground testing a single agent or run them in the agent playground and test a very complicated multi-agentic system.
15:32Now, this is in the UI, but of course you can do this programmatically with the SDK, rest, or CI or CLI, but my favorite method to do this is actually with coding agents.
15:42How familiar is this group with the concept of agent skills?
15:47Ah, I love to see when people are saying advanced.
15:50Agent skills allows you to connect your coding agents with external databases and advanced contexts.
15:55Arize has a public repository of skills that allows you to connect to every feature available in the Arize UI.
16:01This means that you can close the loop between what your coding agents are producing in your IDE, the code that they're producing, with what is actually happening in your observability back end.
16:11So, let's actually touch more on that closing the loop.
16:15So, how we're going to close this loop at agent speed.
16:17We've got the same loop, but instead of running it by people, we want to run it by agents.
16:21Now, I'm going to highlight two key takeaways.
16:24We only have 2 minutes left here.
16:26And the first key takeaway I want to touch on is what the engineers are doing.
16:30Today, engineers are the ones that are closing the loop.
16:32And to summarize what that loop is, is we have observability first.
16:36That is as your systems are running, we are taking that log data, that tracing data, and we are tracking it for every run in production.
16:43We then take that, evaluate it, get those scores, then we can implement a change at that could be a prompt change, model tool scale policy change.
16:53And then we test that change.
16:54We iterate on it with the experiments workflow.
16:58Finally, we improve it.
16:59If it passes, great, ship it.
17:01But if it fails, we close the loop.
17:04We take those failed cases and we bring it back to the observability layer.
17:08These new production traces that failed then become new cases in our regression testing suite.
17:13And that's the first takeaway here, is that every failure should become a test.
17:18Every failure should make your agents stronger and harder to break.
17:24Once you have these regression tests that are regression test suites that are constantly building on themselves, you have a more robust method of continuously iterating and improving.
17:34That's takeaway number one.
17:35Now, takeaway number two, going back to that loop, I have a question for the group and that is, what if an agent could do the investigation and propose a fix?
17:43Because right now, the manual investigation is that that's the bottleneck.
17:47Humans are the bottleneck here.
17:49So, what we're seeing is a shift from observability to self-improvement.
17:53The manual bottleneck here is that humans are still reviewing dashboards.
17:57They're still triaging and shipping those fixes and reviewing those PRs and that is the ultimate bottleneck.
18:02So, we want to not only close this loop, but we want to close it at agentic speeds.
18:07We don't want to just give humans the ability to debug agents.
18:10We want to give agents the ability to debug and understand themselves.
18:14And that's the shift that we're seeing in observability.
18:17Is observability is no longer about humans understanding agents, it's about agents understanding themselves.
18:23Agents having the tools to improve themselves.
18:25Agents having the tools to be accountable.
18:27Remember what we talked about in the very beginning.
18:29That is that traces is where the real runtime is.
18:33That's where the real logic is stored.
18:35If your coding agents are pushing PRs and I know they are, they're pushing PRs without actually knowing what's happening in production.
18:40That means they're pushing PRs blind.
18:42They're reliant on the static code.
18:44They don't know what's actually happening in production.
18:47By closing that loop, by connecting your coding harnesses, your coding agents with what is actually happening happening in your production database, you are giving your agents the full understanding
18:58of what these changes are actually making.
19:00And that is the second and final takeaway.
19:02That is that the best agents don't just learn from failures.
19:05The best agents fix themselves.
19:08And that's the future of observability that we're seeing is that agents can not only see, but they can evaluate and improve themselves on their own.
19:15Thank you.