WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=nxokqOq1imY

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/nxokqOq1imY.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.960 --> 00:00:13.599
Hello everyone.

s3
00:00:13.599 --> 00:00:18.480
My name is Amaya Bhavadkar and I am the field CTO at Brain Trust.

s4
00:00:18.480 --> 00:00:27.439
Uh Brain Trust is a eval observability platform that helps AI teams build and improve their AI with confidence.

s5
00:00:28.000 --> 00:00:41.120
So, I'm sure all of you, if not, you know, I I'm sure everyone here has built some application over the last couple of years that has a model at the center of it,

s6
00:00:41.120 --> 00:00:41.520
right?

s7
00:00:41.520 --> 00:00:51.840
Some sort of a chatbot or a AI agent or some system that's doing batch processing using AI at the heart of it.

s8
00:00:52.000 --> 00:00:59.280
And I'm sure all of you over that time span have done significant uh changes to that application.

s9
00:00:59.280 --> 00:01:11.360
You have either rewritten that application entirely or you have like done some pretty complex surgery on your application and the way it looks now compared to how it looked when you started

s10
00:01:11.360 --> 00:01:14.080
is likely very very different.

s11
00:01:14.080 --> 00:01:26.720
And I think everyone's probably uh experienced the same pattern which is like how building a demo with AI is really easy but making it production quality is really hard.

s12
00:01:26.720 --> 00:01:34.960
The same way when you're evolving your AI application and making significant changes to it, it can be very very challenging.

s13
00:01:34.960 --> 00:01:36.240
Right?

s14
00:01:36.240 --> 00:01:40.560
And uh the challenge is not because you built it the wrong way.

s15
00:01:40.560 --> 00:01:46.159
The challenge is because the system around you is evolving and changing so dynamically, so rapidly.

s16
00:01:46.159 --> 00:01:53.840
You know, the models are changing, the way your users use your application changes, the data that your application works with changes.

s17
00:01:53.840 --> 00:02:00.799
And all of those things require you to continually make changes to your applications.

s18
00:02:01.520 --> 00:02:09.599
And so if you look at, you know, the rate at which the models have evolved over the last couple of years, it's truly astonishing.

s19
00:02:09.599 --> 00:02:24.160
like every few months there's a new release and that locks unlocks a you know a ton of new capabilities a ton of new features that were not present in the previous generation of the models right we have started seeing like models

s20
00:02:24.160 --> 00:02:38.800
that got really good at working with tools models getting really good at handling very long context uh we started seeing models generate code that can be reliably and safely executed in

s21
00:02:38.800 --> 00:02:40.239
uh sandboxes.

s22
00:02:40.239 --> 00:02:45.519
We've seen memory systems becoming very sophisticated and practical.

s23
00:02:45.519 --> 00:02:48.319
And so each of those was not a minor upgrade.

s24
00:02:48.319 --> 00:02:51.040
It was not an incremental change to the previous state-of-the-art.

s25
00:02:51.040 --> 00:02:53.440
It was a step function change, right?

s26
00:02:53.440 --> 00:03:06.640
And so now we are moving from this era of like re uh sort of iterating on improving our applications to replplatforming our applications because everything is changing so dramatically.

s27
00:03:07.920 --> 00:03:14.000
So why can't you just drop in a new model and ask expect your system to work?

s28
00:03:14.000 --> 00:03:28.879
Um well the models the previous system that you built was built with some assumptions around the existing limitations and the constraints that the models had.

s29
00:03:28.879 --> 00:03:29.200
Right?

s30
00:03:29.200 --> 00:03:36.000
Your previous systems were built to account for the fact that your models weren't really as good at tool calling.

s31
00:03:36.000 --> 00:03:41.760
for example, and so your system implemented a bunch of logic to make it work with those limitations.

s32
00:03:41.760 --> 00:03:54.480
And so when you drop in a new model, right, uh you are not able to tap into the new capabilities, the new state-of-the-art without really restructuring your systems in a pretty dramatic way,

s33
00:03:54.480 --> 00:03:55.200
right?

s34
00:03:55.200 --> 00:04:03.040
And so in order to capture that kind of capability, the new unlock, you have to rearchitect.

s35
00:04:04.640 --> 00:04:11.120
And so as you rearchitect right um what happens is u models evolve.

s36
00:04:11.120 --> 00:04:17.840
So you've got to go in and change your application architecture do a lot of work on on getting it to work with the new models.

s37
00:04:17.840 --> 00:04:22.000
But that means that you also now have to update your evals.

s38
00:04:22.000 --> 00:04:26.880
The way you ensure that your system is going to operate reliably, right?

s39
00:04:26.880 --> 00:04:36.320
Because every new um uh unlock is potentially also giving you new surface area where things can go wrong.

s40
00:04:36.320 --> 00:04:41.680
And so your evals now have to adapt and evolve to your new architecture.

s41
00:04:41.680 --> 00:04:48.320
And so you know architecture follows model updates and your evals have to follow your architecture.

s42
00:04:48.639 --> 00:05:01.840
So as I talked through the various generations of the AI systems architectures and how you do uh you know what that architecture is and how the eval to evolve with those architectural changes.

s43
00:05:01.840 --> 00:05:10.720
I want to ground it in a real example and so what I want to talk about is um on the subsequent slides I'll share a bunch of notional evals

s44
00:05:10.720 --> 00:05:13.600
but I want them to be grounded in a real agent.

s45
00:05:13.600 --> 00:05:16.400
In this case we are going to look at this S sur agent.

s46
00:05:16.400 --> 00:05:26.400
the SR agent is able to not only read information but it's able to take actions on and update systems so it can you know roll back a deployment or

s47
00:05:26.400 --> 00:05:40.400
uh escalate it to a human page someone uh so it has access to read tools and write tools so let's see how this system would have evolved through the various generations of AI architectures

s48
00:05:40.400 --> 00:05:46.800
so let's start with the simplest case right this is how a lot of AI applications started about 3 years ago.

s49
00:05:46.800 --> 00:05:50.800
This is a single prompt, a single model call.

s50
00:05:50.800 --> 00:05:54.639
You have one input, one model call, one output.

s51
00:05:54.639 --> 00:06:00.880
And so the focus of evaluations was on the final answer quality, right?

s52
00:06:00.880 --> 00:06:06.400
Did you get the correct answer in terms of uh accuracy and factuality?

s53
00:06:06.400 --> 00:06:10.080
Um or did the uh model hallucinate something?

s54
00:06:10.080 --> 00:06:11.280
Did it make up stuff?

s55
00:06:11.280 --> 00:06:22.319
or did it reference uh old non u the the previous knowledge that it had been trained on and not the latest uh information related to that subject.

s56
00:06:22.319 --> 00:06:31.120
Um so in this case um you were really focusing primarily on the final answer that was your unit of evaluation.

s57
00:06:31.120 --> 00:06:34.800
And so the approach was you would put together a golden data set.

s58
00:06:34.800 --> 00:06:45.120
You will create a bunch of various scores that were looking at um encoding your definition of what good looks like that then you could evaluate the answers against.

s59
00:06:45.120 --> 00:06:45.919
And this was great.

s60
00:06:45.919 --> 00:06:47.280
This was a good way to get started.

s61
00:06:47.280 --> 00:06:49.280
It was narrow because there's no tool calling.

s62
00:06:49.280 --> 00:06:50.319
There's no orchestration.

s63
00:06:50.319 --> 00:06:53.600
there's no uh retrieval, no other steps.

s64
00:06:53.600 --> 00:06:56.560
It's just a simple call to the model.

s65
00:06:56.560 --> 00:06:59.280
Uh but the next iteration of this was the chain.

s66
00:06:59.280 --> 00:07:04.720
This is where you started doing a set of steps before you actually made the model call, right?

s67
00:07:04.720 --> 00:07:09.599
Uh the typical rag application looked like it took the user input.

s68
00:07:09.599 --> 00:07:13.199
It parsed some information from the user output input.

s69
00:07:13.199 --> 00:07:19.759
It then used that to retrieve information, then generate the context and then hand it over to the model.

s70
00:07:19.759 --> 00:07:27.919
And then the model synthesizes reasons on that information, synthesizes an answer and you evaluate the answer.

s71
00:07:27.919 --> 00:07:31.520
But there's a number of other places where things could go wrong.

s72
00:07:31.520 --> 00:07:34.800
Yeah, your parser could extract the wrong information.

s73
00:07:34.800 --> 00:07:37.199
It could retrieve the wrong context.

s74
00:07:37.199 --> 00:07:39.280
The model could struggle with the context.

s75
00:07:39.280 --> 00:07:48.639
Like in the early days, even though the model windows were the context window sizes were increasing, the models struggled to um reason over large context.

s76
00:07:48.639 --> 00:07:51.599
So context stuffing could be an issue for the model performance.

s77
00:07:51.599 --> 00:07:56.000
And so now you had multiple uh areas of failure.

s78
00:07:56.000 --> 00:07:59.039
And so you needed to eval.

s79
00:08:05.280 --> 00:08:14.080
But this was kind of very um what I would call very limited like it did things a very specific way all the time, right?

s80
00:08:14.080 --> 00:08:22.160
And so in late mid late 2023 early 24 the React paper became really popular.

s81
00:08:22.160 --> 00:08:34.399
And so folks were looking at building um model um in a loop running a model in a loop where it could uh reason and act uh in a step-wise way.

s82
00:08:34.399 --> 00:08:37.200
So the model could make tool calls.

s83
00:08:37.200 --> 00:08:54.560
It could then understand what the tool calls returned uh reason on that data and then figure out what the next step was so that it could then continue to run this in a loop till the user intent was finally satisfied or the model ran out of the iteration budget.

s84
00:08:54.560 --> 00:08:55.519
Right?

s85
00:08:55.519 --> 00:09:02.560
And so this was great because it now gives you gives the model the AI system a lot more flexibility.

s86
00:09:02.560 --> 00:09:06.560
It's not pinned down to operating in a very specific workflow.

s87
00:09:06.560 --> 00:09:14.800
It now is able to reason on the various intents and it's able to self-organize, self-chestrate and complete the user tasks.

s88
00:09:14.800 --> 00:09:19.600
Unfortunately, the models of that era were not as robust as they needed to be.

s89
00:09:19.600 --> 00:09:22.560
So, you know, models struggled with tool callings.

s90
00:09:22.560 --> 00:09:24.080
They got the arguments wrong.

s91
00:09:24.080 --> 00:09:26.399
The models struggled with orchestration.

s92
00:09:26.399 --> 00:09:28.160
So, they called the wrong tools.

s93
00:09:28.160 --> 00:09:31.440
The models still had challenges with reasoning.

s94
00:09:31.440 --> 00:09:37.200
they weren't necessarily doing a great job of, you know, dealing with long context.

s95
00:09:37.200 --> 00:09:39.120
So you had things like context collapse.

s96
00:09:39.120 --> 00:09:46.800
And so while the idea was like really really exciting, um, it fell short of delivering on the actual promise.

s97
00:09:46.800 --> 00:09:51.120
And so what does what do you do when your model can't be controlled, right?

s98
00:09:51.120 --> 00:09:56.320
You take the control and you bake that control into the system that you're building around the model.

s99
00:09:56.320 --> 00:10:02.000
And so teams started moving towards these kind of workflow graphs, right?

s100
00:10:02.000 --> 00:10:15.519
Um they started building the orchestration and the execution and planning logic into the the system itself either as a graph or as a state machine.

s101
00:10:15.519 --> 00:10:23.200
And so you took control of the orchestration while you allowed the models to operate at the node level.

s102
00:10:23.200 --> 00:10:35.279
And that way you got a lot more uh reliability and predictability in how your AI was going to operate across those various intents.

s103
00:10:35.360 --> 00:10:44.320
But then the problem is you are now building a system that is designed to work for a specific set of intents for a specific types of use cases.

s104
00:10:44.320 --> 00:10:55.519
And as you start hand, you know, the system starts interacting with with instances that are outside that distribution, the system starts struggling with that, right?

s105
00:10:55.519 --> 00:11:04.959
You expect um you know a certain set of applications or u user interactions to work well because they can be fulfilled by the orchestration that you have designed.

s106
00:11:04.959 --> 00:11:17.680
But when your the user intent needs to be requires other things to happen beyond what's specified in the orchestration, the system can start um you know breaking at the seams.

s107
00:11:17.680 --> 00:11:23.440
And uh in order to do that, folks were now building a lot more complexity into their orchestration logic.

s108
00:11:23.440 --> 00:11:33.519
And so you're building these special uh branches and way you handle special intents in the complex graph that described your system.

s109
00:11:33.519 --> 00:11:39.360
And so what that means is like you had now a ton of different surfaces for failure.

s110
00:11:39.360 --> 00:11:47.360
So you now had to deal with uh you know uh dealing with uh branch consistency and branching logic failures.

s111
00:11:47.360 --> 00:11:53.200
You had to deal with things like the contracts between the nodes not working out well.

s112
00:11:53.200 --> 00:12:03.360
Uh you had to deal with the limitations of uh the nodes that were you know built for a specific set of use cases.

s113
00:12:03.360 --> 00:12:17.040
So you know there were classifier nodes for example and they could make mistakes and so you could now have a significant amount of um you know areas where you could uh where the system could fail.

s114
00:12:17.040 --> 00:12:29.040
And so your evals now have to not only look at u you know the overall orchestration but they now have to you have to have node level evals.

s115
00:12:29.040 --> 00:12:41.200
you have to uh make sure that you have evalu uh you know how you do retry loops.

s116
00:12:41.200 --> 00:12:52.079
There's a lot of complex behaviors of the system that now need to be evaluated in addition to all the other things that you were evaluating before.

s117
00:12:54.720 --> 00:13:08.639
So the graphs were kind of popular like in in late 24 early 25 and so a lot of systems were now implemented using certain frameworks and they were now in production.

s118
00:13:08.639 --> 00:13:22.240
Uh but then um Anthropic and OpenAI launched some amazing new model capabilities mid late 25 and what that was like tool calling became extremely reliable.

s119
00:13:22.240 --> 00:13:26.880
We started uh seeing uh much better orchestration control.

s120
00:13:26.880 --> 00:13:32.240
Uh the models were able to plan a lot more effectively accurately.

s121
00:13:32.240 --> 00:13:34.959
They were able to manage long horizon tasks.

s122
00:13:34.959 --> 00:13:39.680
they were able to do a much better job of introspecting and course correcting.

s123
00:13:39.680 --> 00:13:48.480
And so like as things went a little off track, the models were able to, you know, understand that and bring the execution back on track.

s124
00:13:48.480 --> 00:13:59.519
And so what that meant was a lot of these u u graph-based systems were not able to take advantage of these new capabilities.

s125
00:13:59.519 --> 00:14:16.959
they were still running into some of those like brittleleness issues that the new model state-of-art had unlocked and so um we started looking at building out um the react loop again that's that started working and so now you had this new

s126
00:14:16.959 --> 00:14:32.880
AI systems that could effectively reliably work in a loop they could make those tool calls they could figure out the next step and then they could u essentially go in and um fulfill the user intent.

s127
00:14:32.880 --> 00:14:36.320
But the way they worked was very it had a high degree of variance.

s128
00:14:36.320 --> 00:14:44.959
So every trajectory for the same input if you ran it a couple of times you would see you know dramatically different trajectories while yielding the right answer.

s129
00:14:44.959 --> 00:14:47.920
And so now there's a lot of variance that you have to deal with.

s130
00:14:47.920 --> 00:15:08.160
So now instead of just focusing on a specific eval the unit of eval was no longer just one eval now you're looking at doing an analysis of the distribution of the evals you're taking the same eval you're running it multiple times you're running it k times and you're ensuring that uh you get a statistically relevant signal from

s131
00:15:08.160 --> 00:15:21.040
that eval so now new metrics like uh pass at k and pass raise to k or pass wedge k these were the new metrics that certainly started to make a lot of sense.

s132
00:15:21.040 --> 00:15:31.120
pass at K is like if you take the same that eval and you run it K times does it succeed at least once and that is a measure of its capability

s133
00:15:31.120 --> 00:15:41.199
and pass wedge K is like if you run that eval multiple times how many times of those K instances does it run successfully that's a measure of its uh reliability

s134
00:15:41.199 --> 00:15:55.680
and so now you can understand whether your system with a high pass at K uh you know is reliable by seeing seeing how it you know by measuring the pass wedge K metric for example.

s135
00:15:55.680 --> 00:15:56.310
So

s136
00:15:56.310 --> 00:15:56.720
[snorts]

s137
00:15:56.720 --> 00:16:07.279
this gives you a lot more um you know u understanding of like how your system is working what the failure sources are and how you work on those right

s138
00:16:07.279 --> 00:16:19.120
and then more recently what we've seen is um there's a big shift from it's your system is not just a model running in a loop right it becomes a product system it's that

s139
00:16:19.120 --> 00:16:29.920
there's a model in the loop that's augmented by a lot of peripheral components you know you have a memory system that is able to provide robust memory storage and memory

s140
00:16:29.920 --> 00:16:34.320
um retrieval capabilities uh within a session cross sessions.

s141
00:16:34.320 --> 00:16:43.279
Uh models can tap into this memory to you know improve upon their runs in subsequent instances by learning from previous runs for example.

s142
00:16:43.279 --> 00:16:55.920
You've got robust code execution uh sandboxes now and so you can run model generated code reliably robustly on uh uh during uh execution.

s143
00:16:55.920 --> 00:17:04.959
You've got um MCP and skill uh directories that the model can now tap into and you can you know weave in extensibility.

s144
00:17:04.959 --> 00:17:17.280
You now have things like a skills repository or a skill systems that can be used to continually augment the the the capabilities of models through you know symbolic instructions.

s145
00:17:17.280 --> 00:17:32.880
And so uh now you know like uh these systems are getting pretty complex and as a result uh you know if you are continuing to to use the eval from the previous generation

s146
00:17:32.880 --> 00:17:36.240
you're going to get sort of a partial coverage of your system.

s147
00:17:36.240 --> 00:17:49.760
you're not going to see uh how your system is fragile in ways because of the unlock because of the new surface that you have uh you know uh unlocked in your new system.

s148
00:17:49.760 --> 00:18:07.039
So what that means is um just reflecting back on the pattern is like you know all of these model innovations resulted in in you know corresponding shift in the architectures and so so you've seen these waves of architecture

s149
00:18:07.039 --> 00:18:21.120
and then what's needed is like your evals to be congru congruent with that architecture right uh because ultimately it's the eval that are sort of your durable asset that describe how your system is supposed to work.

s150
00:18:21.120 --> 00:18:35.520
And as you go through these generational shifts, that's a good way to ensure that you know your system your user users experience your system in a way that things that were working are not broken, but it's unlocked a bunch of new capability.

s151
00:18:35.840 --> 00:18:39.840
And so everyone's seen this, you know, diagram of this flywheel.

s152
00:18:39.840 --> 00:18:42.799
Everyone's sort of like bought into it conceptually, right?

s153
00:18:42.799 --> 00:18:50.080
the idea of harvesting data from production to inform your eval so that your evals are reflective of the real world.

s154
00:18:50.080 --> 00:18:52.000
I think that all makes sense, right?

s155
00:18:52.000 --> 00:19:03.600
And and this is the way that you know teams that are doing a great job at building and shipping and improving their AI systems, they they they follow this workflow pretty religiously.

s156
00:19:03.600 --> 00:19:15.200
Um so I've talked to a lot of teams and I think while there is a general acceptance that yeah you need to run that workflow um in practice a lot of teams don't do that their eval are somewhat static

s157
00:19:15.200 --> 00:19:32.160
and even if you're not changing your AI agent architecture you're you know by not really being disciplined about running that that workflow that flywheel you are now getting stagnant evals that are not being as effective in helping you

s158
00:19:32.160 --> 00:19:34.799
measure and improve the quality of your AI.

s159
00:19:34.799 --> 00:19:45.919
And especially as you go through this generational shift, it's really important that you need a mechanism to not only harvest data from production in a way that shows you failures

s160
00:19:45.919 --> 00:19:53.440
that you are looking out for because you defined what good looks like as part of your evals.

s161
00:19:53.440 --> 00:19:59.039
But you also want something to shine a light on the new failure types, right?

s162
00:19:59.039 --> 00:20:08.880
the system is going to fail in new and novel ways in ways that you might not have anticipated and you now need to start harvesting that data in a meaningful way.

s163
00:20:08.880 --> 00:20:12.320
And you want to do this again as as part of the flywheel.

s164
00:20:12.320 --> 00:20:26.720
And so this is where you need systems to come in and u shine a light on things that are broken in ways that you had anticipated, but also broken in a way in ways that you had not anticipated.

s165
00:20:26.720 --> 00:20:29.200
And this is really important.

s166
00:20:29.200 --> 00:20:33.360
So I'm going to quickly talk a little bit about like how we do this in brain trust.

s167
00:20:33.360 --> 00:20:37.600
So brain trust provides all the components that you need to run this flywheel.

s168
00:20:37.600 --> 00:20:40.400
We've got evals, we've got observability.

s169
00:20:40.400 --> 00:20:53.840
We have ways in which you can get insights from your production data to harvest u new eval cases that you can then pass off to the to the team that they can then use to hill climb and

s170
00:20:53.840 --> 00:20:56.240
improve your AI system.

s171
00:20:56.240 --> 00:20:58.400
But topics is a really cool feature.

s172
00:20:58.400 --> 00:21:04.400
What topics does, it does a cluster analysis on all of your production data.

s173
00:21:04.400 --> 00:21:12.960
And so the idea over here is now you are able to find new categories of failures that you had not anticipated.

s174
00:21:12.960 --> 00:21:23.440
So your system is now able to look at all what's going on in production and it's able to now start surfacing these new failure modes that tell you here's a new new

s175
00:21:23.440 --> 00:21:31.360
failure uh you know um uh situation that you hadn't thought about and you didn't have any guardrails in place.

s176
00:21:31.360 --> 00:21:42.720
didn't have any eval in place and so now it's really easy for teams to expand the set of their evals to now cover those kind of new failures.

s177
00:21:42.720 --> 00:21:56.720
And so this is this is a pretty exciting uh capability in brain trust that enables these teams to continually not only get new failure examples for known failure modes

s178
00:21:56.720 --> 00:22:10.000
but more importantly as they make these systemic architectural changes they're able to also understand the new ways in which your system is going to fail and build out effective data sets from production data.

s179
00:22:11.200 --> 00:22:19.200
So I think the takeaway for today's talk is that the models will keep on changing.

s180
00:22:19.200 --> 00:22:21.520
Uh I I don't think we're going to see any slowdown.

s181
00:22:21.520 --> 00:22:23.039
I don't think we have hit a plateau yet.

s182
00:22:23.039 --> 00:22:28.240
I think there are lots of unlocks that are coming down um this road.

s183
00:22:28.240 --> 00:22:35.840
Um and as a result you will be making significant changes to your AI systems.

s184
00:22:35.840 --> 00:22:42.720
you know, you'll be doing a lot of surgery on your AI agents in the coming months, years.

s185
00:22:42.720 --> 00:22:55.520
And so it's really important that you have a robust workflow system in place to ensure that as you make those changes, as you incorporate these new models into your systems,

s186
00:22:55.520 --> 00:23:06.240
that your systems continue to get better at doing new things, but also continue to work well for the things that they were doing before.

s187
00:23:06.240 --> 00:23:19.440
And so building out like a robust eval discipline uh with the right tools and the right automation and the right systems becomes paramount to manage these generational changes.

s188
00:23:19.440 --> 00:23:34.320
And so ultimately what you want is um to really uh index on that flywheel and make it part of your workflow so that uh you know the ability to

s189
00:23:34.320 --> 00:23:47.039
improve incrementally when the changes in the system are incremental and the ability to improve your system in a in sort of a step function way are both supported by your

s190
00:23:47.039 --> 00:23:48.640
evals.

s191
00:23:48.640 --> 00:23:54.240
So with that, I want to say thank you.

s192
00:23:54.466 --> 00:23:56.466
[applause]