WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=v5tm5EAzsAs

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/v5tm5EAzsAs.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.400 --> 00:00:13.400
Hello everyone.

s3
00:00:13.400 --> 00:00:15.240
My name is Sophia Yakoubovitch.

s4
00:00:15.240 --> 00:00:24.160
I'm a forward deployed engineer at Arize, and for those who aren't familiar, Arize is an observability backend that is designed to make AI work.

s5
00:00:24.160 --> 00:00:25.600
Now, what does that mean?

s6
00:00:25.600 --> 00:00:37.280
Building AI systems can be inherently intuitive in some sense, especially when a lot of that low-level abstraction is a low-level programming is abstracted into the mind of an agent.

s7
00:00:37.280 --> 00:00:41.400
Unfortunately though, that mind can be a bit of a black box.

s8
00:00:41.400 --> 00:00:46.880
When that black box is difficult to understand, it's difficult to predict, and therefore difficult to debug.

s9
00:00:46.880 --> 00:00:49.720
And that's where observability becomes essential.

s10
00:00:49.720 --> 00:00:53.760
Observability helps you understand what your systems are actually doing.

s11
00:00:53.760 --> 00:00:56.680
It's essentially shining a light in the black box.

s12
00:00:56.680 --> 00:01:01.520
And once you understand what the system is doing step-by-step, you can then evaluate it.

s13
00:01:01.520 --> 00:01:08.480
Run evaluation metrics to track its performance, and then use that data to iterate and improve your agents over time.

s14
00:01:08.480 --> 00:01:20.760
But I want to take this one step further, because the biggest challenge isn't determining what broke, the biggest challenge is that when you do submit a fix, you want to make sure that you don't break 10 other things along the way.

s15
00:01:20.760 --> 00:01:23.760
Now, raise your hand if you've had that exact issue before.

s16
00:01:23.760 --> 00:01:26.520
You push Yep, seeing a lot of hands.

s17
00:01:26.520 --> 00:01:34.640
You push a simple change, you know, should be a should be easy, no issues, and next thing you know, everything's on fire.

s18
00:01:34.640 --> 00:01:40.960
So, today we're going to discuss how you can use your observability as a way to build a robust regression process.

s19
00:01:40.960 --> 00:01:47.160
Not only something that your engineers can do, but an automatic process that feeds into your CI/CD.

s20
00:01:47.360 --> 00:01:49.360
We're going to break this down to three parts.

s21
00:01:49.360 --> 00:01:53.680
First, we'll discuss why agent regression is so different and difficult to detect.

s22
00:01:53.680 --> 00:01:59.920
And the first thing we're going to touch on is that agents don't fail at the code level, they fail at runtime.

s23
00:01:59.920 --> 00:02:03.440
We're then going to use this premise to build out our regression loop.

s24
00:02:03.440 --> 00:02:05.280
It's going to be a five-step process.

s25
00:02:05.280 --> 00:02:11.520
We'll touch on it at a high-level overview, and then we'll move on to the Arize platform and show you exactly how you can do it today.

s26
00:02:11.520 --> 00:02:21.520
Then, we'll close off the session by closing that regression loop at agentic speed and touching on where we're seeing the next frontier of agent observability move to.

s27
00:02:21.520 --> 00:02:26.000
And if you stick with me till the end, I'll introduce a paradigm shift that I hope you'll all find interesting.

s28
00:02:26.000 --> 00:02:27.400
So, let's get started.

s29
00:02:27.400 --> 00:02:29.600
First, with a concrete example.

s30
00:02:29.600 --> 00:02:34.560
So, imagine with me that you have a support agent, and you have a security engineer that has approached you.

s31
00:02:34.560 --> 00:02:48.120
And that security engineer has reviewed the policy of your support agent and determined that he wants to add one line, a simple one line to your agent that will ask your agent to confirm before it takes any action.

s32
00:02:48.120 --> 00:02:49.200
Sounds simple.

s33
00:02:49.200 --> 00:02:50.520
Sounds harmless.

s34
00:02:50.520 --> 00:02:56.680
Your security engineer says he has the intuition, and oftentimes we're quick to make this change.

s35
00:02:56.680 --> 00:03:03.080
I'm not even going to ask if you've done this before cuz I know everyone has pushed a one-line prompt change and just assumed everything will go well.

s36
00:03:03.080 --> 00:03:11.920
And the reason we make that assumption is because we think that because it's written in natural language, that we have this intuition on how these agents are performing.

s37
00:03:11.920 --> 00:03:15.959
We're really coding on vibes, and we don't want to be.

s38
00:03:15.959 --> 00:03:21.160
We want to be able to code with a numeric evaluation suite backed data.

s39
00:03:21.160 --> 00:03:27.680
We want to have scores that prove that these changes are actually making a positive impact and not just crossing our fingers.

s40
00:03:27.680 --> 00:03:35.560
But let's say in the scenario you did push that one-line change, and now you are running your evaluations on your production data a week after it's been submitted.

s41
00:03:35.560 --> 00:03:41.160
And the first thing you notice is that you actually have a 17-point improvement on a action safety.

s42
00:03:41.160 --> 00:03:42.920
So, your security engineer was correct.

s43
00:03:42.920 --> 00:03:48.680
It asking the agent to confirm before taking any action does improve its overall safety.

s44
00:03:48.680 --> 00:03:58.440
But your observability engineers are continuing to evaluate, and they also check for task completion and they notice a 17-point degradation.

s45
00:03:58.440 --> 00:04:06.760
Now, this is occurring because if you have an agent that's confirming before every single action it's taking, now tasks are being uncompleted.

s46
00:04:06.760 --> 00:04:17.079
If a user leaves the session, that session is never completed to the end and I have incomplete task and you see this unlikely result.

s47
00:04:17.079 --> 00:04:21.840
You may not have caught this initially when you were thinking about the project, but your evaluation suite has caught this.

s48
00:04:21.840 --> 00:04:26.160
And ironically enough, the overall average of both is net zero.

s49
00:04:26.160 --> 00:04:32.919
And what I want you to take away from here is just because you improve the dimension of one agent does not mean that you're improving the agent as a whole.

s50
00:04:32.919 --> 00:04:37.200
The question I have for you is, would you have shipped this change?

s51
00:04:37.200 --> 00:04:39.720
Many of us would have before the evaluation test.

s52
00:04:39.720 --> 00:04:48.080
And this is why it's so important that we get these numeric results so that we're not shipping on vibes, we're shipping with backed data.

s53
00:04:48.080 --> 00:04:53.120
So, to understand where the mistake that we made is is we made an unrealistic assumption.

s54
00:04:53.120 --> 00:04:59.320
And that assumption is that the error is going to occur at the code, but the error rarely occurs at the code level.

s55
00:04:59.320 --> 00:05:03.480
Instead, these agents are failing at runtime.

s56
00:05:03.480 --> 00:05:07.440
What the code reviewer saw is here on the left-hand side.

s57
00:05:07.440 --> 00:05:11.240
It was a one-line change in the system prompt markdown file.

s58
00:05:11.240 --> 00:05:14.960
Easy change, simple to understand, simple to approve.

s59
00:05:14.960 --> 00:05:21.160
But in reality, this code change was local, but the behavioral changes of this code change was in production.

s60
00:05:21.160 --> 00:05:22.760
It was in the runtime.

s61
00:05:22.760 --> 00:05:26.040
This one simple line has shifted how the agent operates.

s62
00:05:26.040 --> 00:05:27.320
It's adding additional steps.

s63
00:05:27.320 --> 00:05:30.040
It's changing how the agent works with other sub-agents.

s64
00:05:30.040 --> 00:05:32.360
It's changing how the agent calls tools.

s65
00:05:32.360 --> 00:05:37.680
All of these different trajectories, which could be thousands of possible trajectories, are not in your static code.

s66
00:05:37.680 --> 00:05:40.080
They're only in your observability back end.

s67
00:05:40.080 --> 00:05:42.000
They're only in your traces.

s68
00:05:42.000 --> 00:05:49.320
And so, what we are saying here is that the diff tells you what changed, but the trace tells you what actually happened.

s69
00:05:49.320 --> 00:05:53.160
And there's a shift here, that traces are becoming the new source of truth.

s70
00:05:53.160 --> 00:06:00.600
These structured logs of what your agents are actually doing actually is what truly represents what these systems are doing.

s71
00:06:00.600 --> 00:06:03.000
In traditional software, logic was written down.

s72
00:06:03.000 --> 00:06:08.400
It was very clear to determine what that runtime was before you ever pushed production.

s73
00:06:08.400 --> 00:06:11.800
But now the logic only exists at runtime.

s74
00:06:11.800 --> 00:06:16.800
The observability becomes essential when we're looking at regression testing.

s75
00:06:16.800 --> 00:06:20.760
This gap between what was written and what actually happens is filled with silent errors.

s76
00:06:20.760 --> 00:06:26.040
So to capture that, we need to start our regression testing by looking at our traces.

s77
00:06:26.040 --> 00:06:28.800
So let's build out this regression loop together.

s78
00:06:28.800 --> 00:06:29.919
It's a five-step process.

s79
00:06:29.919 --> 00:06:32.040
We're going to start with our production traces.

s80
00:06:32.040 --> 00:06:39.320
We're going to build evals, run experiments, and then close it all up to be an automatic gate at CI.

s81
00:06:39.320 --> 00:06:43.120
First, I'm going to do a five-step high-level overview.

s82
00:06:43.120 --> 00:06:48.080
And then we're going to go into the actual product and show you how you can do it yourself.

s83
00:06:48.080 --> 00:06:50.880
So step one is we're going to select a data set.

s84
00:06:50.880 --> 00:06:51.840
Seems simple enough.

s85
00:06:51.840 --> 00:06:54.280
We want to pull real production traces.

s86
00:06:54.280 --> 00:06:55.440
We don't need to synthesize data.

s87
00:06:55.440 --> 00:07:01.240
If we already have real user queries, real inputs, real tool calls, we're pulling from that directly.

s88
00:07:01.240 --> 00:07:09.840
It's important that our data set is a representative work is representative of the average workflow that our users are going to be pushing into our systems.

s89
00:07:09.840 --> 00:07:12.760
But it's also important that we capture known failures.

s90
00:07:12.760 --> 00:07:18.120
We filter out com moments where often errors, often fails at eval metrics.

s91
00:07:18.120 --> 00:07:21.000
Capture those into your data data sets.

s92
00:07:21.000 --> 00:07:23.200
You also want to capture edge cases.

s93
00:07:23.200 --> 00:07:26.120
Maybe not as common, but just as important.

s94
00:07:26.120 --> 00:07:32.560
And one thing that we often see users forget is capturing successful workflows that we want to protect.

s95
00:07:32.560 --> 00:07:41.320
Because again, we're not just trying to make sure that the fix is going to work, but we also want to test that we're not going to break something that was already working.

s96
00:07:41.320 --> 00:07:45.280
So it's important that we capture all four of these in our data set.

s97
00:07:45.280 --> 00:07:50.200
Once we have this production data set, we can then build out the rest of our regression testing suite.

s98
00:07:50.200 --> 00:07:53.160
The second step is we want to have a scoring mechanism.

s99
00:07:53.160 --> 00:07:54.760
And this is where we bring in evaluators.

s100
00:07:54.760 --> 00:08:00.920
Now, show of hands, how many here are familiar with the concept of evaluating agents?

s101
00:08:01.400 --> 00:08:03.120
I'm so happy to see so many hands.

s102
00:08:03.120 --> 00:08:06.320
This is a core foundational part of agentic engineering.

s103
00:08:06.320 --> 00:08:14.360
Evaluators allow us to take metrics and dimensions that are important to your engineers, your product team, and actually quantify them.

s104
00:08:14.360 --> 00:08:15.600
What do I mean by that?

s105
00:08:15.600 --> 00:08:24.400
Let's take some metrics that are important for your agents, things like hallucination, accuracy, tool trajectory, step efficiency.

s106
00:08:24.400 --> 00:08:28.120
All of these metrics are subjective in nature.

s107
00:08:28.120 --> 00:08:35.000
And you may have a human that reviews them and can potentially determine whether or not something was hallucinated, whether or not the most efficient pathway was taken.

s108
00:08:35.000 --> 00:08:37.800
But we want to be able to automate this process.

s109
00:08:37.800 --> 00:08:41.760
We want to turn a subjective metric into a numeric score.

s110
00:08:41.760 --> 00:08:44.120
And we do this by leveraging LLMs.

s111
00:08:44.120 --> 00:08:52.240
We use a concept of LM as a judge, where we judge a span, a trace, some log data against an LM.

s112
00:08:52.240 --> 00:09:01.360
The LM has a particular guide of how it determines whether or not something was hallucinated or factual, something was efficient or not efficient.

s113
00:09:01.360 --> 00:09:04.760
And now we've done essentially a binary classification.

s114
00:09:04.760 --> 00:09:14.640
In this example at the top right corner, you could see a task completion eval, where the LM as a judge is has a task of determining whether or not something was complete or incomplete.

s115
00:09:14.640 --> 00:09:23.800
Now we've taken a subjective metric and we've essentially turned it into a binary score, 0 1. You can translate the score across 0 to 5. You can have multiple different labels.

s116
00:09:23.800 --> 00:09:29.080
But you've now turned a quantitative measurement into a qualitative score that you can measure against.

s117
00:09:29.080 --> 00:09:36.839
You're not limited to LM as a judge, you can also use code-based evals or ML-based evals as well, determine based on what that metric is.

s118
00:09:36.839 --> 00:09:42.720
A code evaluator is great for your deterministic checks, but an LM as a judge is good for your non-deterministic checks.

s119
00:09:42.720 --> 00:09:48.080
So, once we have a list of all of our metrics that are important to us, we build evals for each of them.

s120
00:09:48.080 --> 00:09:51.000
Now, we have a testing suite that we can go against.

s121
00:09:51.000 --> 00:09:53.480
Step three is we're going to run our baseline experiment.

s122
00:09:53.480 --> 00:10:03.000
And what this experiment is is we're going to take our agent and run it against all of our production traces, um all of our production data sets.

s123
00:10:03.000 --> 00:10:09.720
We're after we get the output of those production data production data sets, we're then going to run our eval testing suite on top of that.

s124
00:10:09.720 --> 00:10:11.520
And that's what's going to be our baseline.

s125
00:10:11.520 --> 00:10:14.160
Now, you can do this in the SDK, it's just run experiment.

s126
00:10:14.160 --> 00:10:20.240
You'll pass in that data set, pass in that task, pass in your evaluation suite, and you'll get those results.

s127
00:10:20.240 --> 00:10:24.400
We're going to touch on how you can do this in the UI, as well as how you can do this with your coding agents.

s128
00:10:24.400 --> 00:10:28.360
But, once you have that baseline experiment, you're then going to add your one change.

s129
00:10:28.360 --> 00:10:33.120
Change that policy, rerun the evaluation, what are the eval results?

s130
00:10:33.120 --> 00:10:34.400
A simple process.

s131
00:10:34.400 --> 00:10:41.040
Now that you can quickly iterate on, and not just with one policy change, but test multiple changes within your prompts.

s132
00:10:41.040 --> 00:10:42.880
Test multiple models.

s133
00:10:42.880 --> 00:10:48.640
If your team comes with you comes to you and says, "We want to switch the new fable model, it's five times more expensive."

s134
00:10:48.640 --> 00:10:51.880
You're not going to switch without proving that you're going to get a good ROI.

s135
00:10:51.880 --> 00:11:01.040
This is how you can prove that with a numeric set of measurements to determine whether or not it actually is improving your investment to switch to a more expensive model.

s136
00:11:01.040 --> 00:11:04.120
And you can adjust this policy, rerun, and compare.

s137
00:11:04.120 --> 00:11:11.680
Rerun and compare how changing the prompt, changing handoffs, changing tools, changing models is all going to affect your overall output.

s138
00:11:11.680 --> 00:11:14.920
All without having to deploy to production.

s139
00:11:14.920 --> 00:11:16.440
And then we can automate this.

s140
00:11:16.440 --> 00:11:20.640
We push this experiment into CI to complete this automation.

s141
00:11:20.640 --> 00:11:30.520
Now, each time that you have a pull request, let's say you set this up in GitHub actions, for example, each pull request each PR is run through an experiment workflow against the evaluations

s142
00:11:30.520 --> 00:11:32.880
that your team has deemed important.

s143
00:11:32.880 --> 00:11:37.880
And you don't merge until it meets all of the regression criteria that you have listed.

s144
00:11:37.880 --> 00:11:44.800
Now, you're no longer pushing on vibes, you're pushing with confidence that you actually do have a positive impact on your performance.

s145
00:11:44.800 --> 00:11:48.600
And if it is negative, you go back, you test, and you reiterate.

s146
00:11:48.600 --> 00:11:51.920
So, let's actually show how to do this in the platform.

s147
00:11:51.920 --> 00:11:53.080
Let me switch over.

s148
00:11:53.080 --> 00:11:59.120
Um here we have Here we have the Arize platform, and I'm showcasing this exact refund agent.

s149
00:11:59.120 --> 00:12:08.400
You can see in our baseline experiment what our scores were, and then you can see that plus 17 points on action safety and minus 17 points on task completion.

s150
00:12:08.400 --> 00:12:20.240
Each one of these points Each one of these line items here is a different test that I ran with different prompt changes, potentially different model changes, and I'm evaluating each one of them to see which one has the best trade-off

s151
00:12:20.240 --> 00:12:22.040
to get to where my end goal is.

s152
00:12:22.040 --> 00:12:28.160
And I can see that overall I'm going to have a plus 18 18 points on action safety and plus six on task completion.

s153
00:12:28.160 --> 00:12:30.520
Now, how can we build these individual points?

s154
00:12:30.520 --> 00:12:31.920
Well, there's a couple ways.

s155
00:12:31.920 --> 00:12:34.000
We'll do this first in the UI.

s156
00:12:34.000 --> 00:12:36.760
I'm going to jump into the playground here.

s157
00:12:36.760 --> 00:12:38.360
I'm going to create a new playground.

s158
00:12:38.360 --> 00:12:42.000
We're going to start with the prompt playground to test a single agent.

s159
00:12:42.000 --> 00:12:45.920
Let's call this AIE NYC.

s160
00:12:46.400 --> 00:12:46.920
Perfect.

s161
00:12:46.920 --> 00:12:51.040
Once I'm in the prompt playground, here is where I can test a single agent.

s162
00:12:51.040 --> 00:13:00.080
An agent A single agent all is essentially just a single prompt with a set of tools attached to it, a model LLM configurations, and so we're going to replicate that here.

s163
00:13:00.080 --> 00:13:03.880
Now, I've already stored my prompt in my prompt hub, so I can pull it right here.

s164
00:13:03.880 --> 00:13:07.480
I've got version one, and then I can pull up my data set.

s165
00:13:07.480 --> 00:13:18.160
I've already selected a data set that I can use here and that I've gone through and pulled out those edge cases, those known failures, and those successful workflows that I want to protect,

s166
00:13:18.160 --> 00:13:21.160
all listed in this 100-row data set.

s167
00:13:21.160 --> 00:13:25.560
I can test this right now, or I can test multiple prompts at the same time.

s168
00:13:25.560 --> 00:13:35.840
So, I'm actually going to duplicate this, and I can change a line item directly in here, or I can load up a separate version right here and test side by side.

s169
00:13:35.840 --> 00:13:41.320
So, I can see just how quick just how both of these prompts are affected.

s170
00:13:41.320 --> 00:13:43.640
Now, of course, we have to add our scoring mechanism.

s171
00:13:43.640 --> 00:13:44.720
So, that's going to be our evals.

s172
00:13:44.720 --> 00:13:46.000
I'm going to click here.

s173
00:13:46.000 --> 00:13:51.560
I'm going to add in my task completion, and then I'm also going to add in my action safety.

s174
00:13:51.560 --> 00:13:59.040
And you can see here that there's a ton of other pre-built halluc pre-built evaluators you can select from, hallucination, user frustration.

s175
00:13:59.040 --> 00:14:04.920
And we've got some code templates, security checks, etc. And you always have the ability to customize your own.

s176
00:14:04.920 --> 00:14:12.560
Once I run this, it's going to run on every single record, and then it's going to pull out those eval scores.

s177
00:14:12.560 --> 00:14:25.400
And then when I go into my data sets here, go into my experiment run flow, it'll pop up under here as well.

s178
00:14:25.400 --> 00:14:27.840
And I can pull up all those metrics that are important for me.

s179
00:14:27.840 --> 00:14:33.720
I can look at latency, I can look at my evals, I can look at my total cost and error rates, and track that over time.

s180
00:14:33.720 --> 00:14:37.320
So, I can look this for PR changes, for playground runs.

s181
00:14:37.320 --> 00:14:39.960
And there's one more other option that I want to show here, too.

s182
00:14:39.960 --> 00:14:45.839
And that's if I go into playground, click here, and I do agent playground.

s183
00:14:46.480 --> 00:14:48.040
Call this AIE.

s184
00:14:48.040 --> 00:14:48.760
Oh.

s185
00:14:48.760 --> 00:14:49.720
There we go.

s186
00:14:49.720 --> 00:14:55.320
Agent playground allows you to not only evaluate at a single agent, but at multiple agent levels.

s187
00:14:55.320 --> 00:15:01.160
If you give us access to connect to your agent remotely, for example, through a HTTP call.

s188
00:15:01.160 --> 00:15:12.480
If I go into remote agents here, you can connect your remote agent directly here, so that you can change how different sub sub agents communicate with each other, all directly in your code base, and then we can call that.

s189
00:15:12.480 --> 00:15:20.600
So, each record in your data set will call against your agent, will test, will get the output, will run those evals, and then score it in the same mechanism.

s190
00:15:20.600 --> 00:15:24.280
So, you have two ways of running play running these experiments within the UI.

s191
00:15:24.280 --> 00:15:32.600
You can run them in the prompt playground testing a single agent or run them in the agent playground and test a very complicated multi-agentic system.

s192
00:15:32.600 --> 00:15:42.280
Now, this is in the UI, but of course you can do this programmatically with the SDK, rest, or CI or CLI, but my favorite method to do this is actually with coding agents.

s193
00:15:42.280 --> 00:15:46.840
How familiar is this group with the concept of agent skills?

s194
00:15:47.160 --> 00:15:50.160
Ah, I love to see when people are saying advanced.

s195
00:15:50.160 --> 00:15:55.600
Agent skills allows you to connect your coding agents with external databases and advanced contexts.

s196
00:15:55.600 --> 00:16:01.320
Arize has a public repository of skills that allows you to connect to every feature available in the Arize UI.

s197
00:16:01.320 --> 00:16:11.360
This means that you can close the loop between what your coding agents are producing in your IDE, the code that they're producing, with what is actually happening in your observability back end.

s198
00:16:11.360 --> 00:16:15.200
So, let's actually touch more on that closing the loop.

s199
00:16:15.560 --> 00:16:17.880
So, how we're going to close this loop at agent speed.

s200
00:16:17.880 --> 00:16:21.800
We've got the same loop, but instead of running it by people, we want to run it by agents.

s201
00:16:21.800 --> 00:16:24.080
Now, I'm going to highlight two key takeaways.

s202
00:16:24.080 --> 00:16:26.040
We only have 2 minutes left here.

s203
00:16:26.040 --> 00:16:30.320
And the first key takeaway I want to touch on is what the engineers are doing.

s204
00:16:30.320 --> 00:16:32.920
Today, engineers are the ones that are closing the loop.

s205
00:16:32.920 --> 00:16:36.520
And to summarize what that loop is, is we have observability first.

s206
00:16:36.520 --> 00:16:43.320
That is as your systems are running, we are taking that log data, that tracing data, and we are tracking it for every run in production.

s207
00:16:43.320 --> 00:16:53.400
We then take that, evaluate it, get those scores, then we can implement a change at that could be a prompt change, model tool scale policy change.

s208
00:16:53.400 --> 00:16:54.839
And then we test that change.

s209
00:16:54.839 --> 00:16:58.080
We iterate on it with the experiments workflow.

s210
00:16:58.080 --> 00:16:59.400
Finally, we improve it.

s211
00:16:59.400 --> 00:17:01.040
If it passes, great, ship it.

s212
00:17:01.040 --> 00:17:04.199
But if it fails, we close the loop.

s213
00:17:04.199 --> 00:17:08.400
We take those failed cases and we bring it back to the observability layer.

s214
00:17:08.400 --> 00:17:13.760
These new production traces that failed then become new cases in our regression testing suite.

s215
00:17:13.760 --> 00:17:18.199
And that's the first takeaway here, is that every failure should become a test.

s216
00:17:18.199 --> 00:17:24.600
Every failure should make your agents stronger and harder to break.

s217
00:17:24.600 --> 00:17:34.080
Once you have these regression tests that are regression test suites that are constantly building on themselves, you have a more robust method of continuously iterating and improving.

s218
00:17:34.080 --> 00:17:35.600
That's takeaway number one.

s219
00:17:35.600 --> 00:17:43.720
Now, takeaway number two, going back to that loop, I have a question for the group and that is, what if an agent could do the investigation and propose a fix?

s220
00:17:43.720 --> 00:17:47.760
Because right now, the manual investigation is that that's the bottleneck.

s221
00:17:47.760 --> 00:17:49.960
Humans are the bottleneck here.

s222
00:17:49.960 --> 00:17:53.960
So, what we're seeing is a shift from observability to self-improvement.

s223
00:17:53.960 --> 00:17:57.320
The manual bottleneck here is that humans are still reviewing dashboards.

s224
00:17:57.320 --> 00:18:02.000
They're still triaging and shipping those fixes and reviewing those PRs and that is the ultimate bottleneck.

s225
00:18:02.000 --> 00:18:07.160
So, we want to not only close this loop, but we want to close it at agentic speeds.

s226
00:18:07.160 --> 00:18:10.120
We don't want to just give humans the ability to debug agents.

s227
00:18:10.120 --> 00:18:14.440
We want to give agents the ability to debug and understand themselves.

s228
00:18:14.440 --> 00:18:17.000
And that's the shift that we're seeing in observability.

s229
00:18:17.000 --> 00:18:23.160
Is observability is no longer about humans understanding agents, it's about agents understanding themselves.

s230
00:18:23.160 --> 00:18:25.920
Agents having the tools to improve themselves.

s231
00:18:25.920 --> 00:18:27.640
Agents having the tools to be accountable.

s232
00:18:27.640 --> 00:18:29.760
Remember what we talked about in the very beginning.

s233
00:18:29.760 --> 00:18:33.360
That is that traces is where the real runtime is.

s234
00:18:33.360 --> 00:18:35.160
That's where the real logic is stored.

s235
00:18:35.160 --> 00:18:40.920
If your coding agents are pushing PRs and I know they are, they're pushing PRs without actually knowing what's happening in production.

s236
00:18:40.920 --> 00:18:42.800
That means they're pushing PRs blind.

s237
00:18:42.800 --> 00:18:44.760
They're reliant on the static code.

s238
00:18:44.760 --> 00:18:47.120
They don't know what's actually happening in production.

s239
00:18:47.120 --> 00:18:58.000
By closing that loop, by connecting your coding harnesses, your coding agents with what is actually happening happening in your production database, you are giving your agents the full understanding

s240
00:18:58.000 --> 00:19:00.240
of what these changes are actually making.

s241
00:19:00.240 --> 00:19:02.240
And that is the second and final takeaway.

s242
00:19:02.240 --> 00:19:05.880
That is that the best agents don't just learn from failures.

s243
00:19:05.880 --> 00:19:08.320
The best agents fix themselves.

s244
00:19:08.320 --> 00:19:15.800
And that's the future of observability that we're seeing is that agents can not only see, but they can evaluate and improve themselves on their own.

s245
00:19:15.800 --> 00:19:18.040
Thank you.