Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust

AI Engineer · 24 min · 192 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello everyone.
00:13My name is Amaya Bhavadkar and I am the field CTO at Brain Trust.
00:18Uh Brain Trust is a eval observability platform that helps AI teams build and improve their AI with confidence.
00:28So, I'm sure all of you, if not, you know, I I'm sure everyone here has built some application over the last couple of years that has a model at the center of it,
00:41right?
00:41Some sort of a chatbot or a AI agent or some system that's doing batch processing using AI at the heart of it.
00:52And I'm sure all of you over that time span have done significant uh changes to that application.
00:59You have either rewritten that application entirely or you have like done some pretty complex surgery on your application and the way it looks now compared to how it looked when you started
01:11is likely very very different.
01:14And I think everyone's probably uh experienced the same pattern which is like how building a demo with AI is really easy but making it production quality is really hard.
01:26The same way when you're evolving your AI application and making significant changes to it, it can be very very challenging.
01:34Right?
01:36And uh the challenge is not because you built it the wrong way.
01:40The challenge is because the system around you is evolving and changing so dynamically, so rapidly.
01:46You know, the models are changing, the way your users use your application changes, the data that your application works with changes.
01:53And all of those things require you to continually make changes to your applications.
02:01And so if you look at, you know, the rate at which the models have evolved over the last couple of years, it's truly astonishing.
02:09like every few months there's a new release and that locks unlocks a you know a ton of new capabilities a ton of new features that were not present in the previous generation of the models right we have started seeing like models
02:24that got really good at working with tools models getting really good at handling very long context uh we started seeing models generate code that can be reliably and safely executed in
02:38uh sandboxes.
02:40We've seen memory systems becoming very sophisticated and practical.
02:45And so each of those was not a minor upgrade.
02:48It was not an incremental change to the previous state-of-the-art.
02:51It was a step function change, right?
02:53And so now we are moving from this era of like re uh sort of iterating on improving our applications to replplatforming our applications because everything is changing so dramatically.
03:07So why can't you just drop in a new model and ask expect your system to work?
03:14Um well the models the previous system that you built was built with some assumptions around the existing limitations and the constraints that the models had.
03:28Right?
03:29Your previous systems were built to account for the fact that your models weren't really as good at tool calling.
03:36for example, and so your system implemented a bunch of logic to make it work with those limitations.
03:41And so when you drop in a new model, right, uh you are not able to tap into the new capabilities, the new state-of-the-art without really restructuring your systems in a pretty dramatic way,
03:54right?
03:55And so in order to capture that kind of capability, the new unlock, you have to rearchitect.
04:04And so as you rearchitect right um what happens is u models evolve.
04:11So you've got to go in and change your application architecture do a lot of work on on getting it to work with the new models.
04:17But that means that you also now have to update your evals.
04:22The way you ensure that your system is going to operate reliably, right?
04:26Because every new um uh unlock is potentially also giving you new surface area where things can go wrong.
04:36And so your evals now have to adapt and evolve to your new architecture.
04:41And so you know architecture follows model updates and your evals have to follow your architecture.
04:48So as I talked through the various generations of the AI systems architectures and how you do uh you know what that architecture is and how the eval to evolve with those architectural changes.
05:01I want to ground it in a real example and so what I want to talk about is um on the subsequent slides I'll share a bunch of notional evals
05:10but I want them to be grounded in a real agent.
05:13In this case we are going to look at this S sur agent.
05:16the SR agent is able to not only read information but it's able to take actions on and update systems so it can you know roll back a deployment or
05:26uh escalate it to a human page someone uh so it has access to read tools and write tools so let's see how this system would have evolved through the various generations of AI architectures
05:40so let's start with the simplest case right this is how a lot of AI applications started about 3 years ago.
05:46This is a single prompt, a single model call.
05:50You have one input, one model call, one output.
05:54And so the focus of evaluations was on the final answer quality, right?
06:00Did you get the correct answer in terms of uh accuracy and factuality?
06:06Um or did the uh model hallucinate something?
06:10Did it make up stuff?
06:11or did it reference uh old non u the the previous knowledge that it had been trained on and not the latest uh information related to that subject.
06:22Um so in this case um you were really focusing primarily on the final answer that was your unit of evaluation.
06:31And so the approach was you would put together a golden data set.
06:34You will create a bunch of various scores that were looking at um encoding your definition of what good looks like that then you could evaluate the answers against.
06:45And this was great.
06:45This was a good way to get started.
06:47It was narrow because there's no tool calling.
06:49There's no orchestration.
06:50there's no uh retrieval, no other steps.
06:53It's just a simple call to the model.
06:56Uh but the next iteration of this was the chain.
06:59This is where you started doing a set of steps before you actually made the model call, right?
07:04Uh the typical rag application looked like it took the user input.
07:09It parsed some information from the user output input.
07:13It then used that to retrieve information, then generate the context and then hand it over to the model.
07:19And then the model synthesizes reasons on that information, synthesizes an answer and you evaluate the answer.
07:27But there's a number of other places where things could go wrong.
07:31Yeah, your parser could extract the wrong information.
07:34It could retrieve the wrong context.
07:37The model could struggle with the context.
07:39Like in the early days, even though the model windows were the context window sizes were increasing, the models struggled to um reason over large context.
07:48So context stuffing could be an issue for the model performance.
07:51And so now you had multiple uh areas of failure.
07:56And so you needed to eval.
08:05But this was kind of very um what I would call very limited like it did things a very specific way all the time, right?
08:14And so in late mid late 2023 early 24 the React paper became really popular.
08:22And so folks were looking at building um model um in a loop running a model in a loop where it could uh reason and act uh in a step-wise way.
08:34So the model could make tool calls.
08:37It could then understand what the tool calls returned uh reason on that data and then figure out what the next step was so that it could then continue to run this in a loop till the user intent was finally satisfied or the model ran out of the iteration budget.
08:54Right?
08:55And so this was great because it now gives you gives the model the AI system a lot more flexibility.
09:02It's not pinned down to operating in a very specific workflow.
09:06It now is able to reason on the various intents and it's able to self-organize, self-chestrate and complete the user tasks.
09:14Unfortunately, the models of that era were not as robust as they needed to be.
09:19So, you know, models struggled with tool callings.
09:22They got the arguments wrong.
09:24The models struggled with orchestration.
09:26So, they called the wrong tools.
09:28The models still had challenges with reasoning.
09:31they weren't necessarily doing a great job of, you know, dealing with long context.
09:37So you had things like context collapse.
09:39And so while the idea was like really really exciting, um, it fell short of delivering on the actual promise.
09:46And so what does what do you do when your model can't be controlled, right?
09:51You take the control and you bake that control into the system that you're building around the model.
09:56And so teams started moving towards these kind of workflow graphs, right?
10:02Um they started building the orchestration and the execution and planning logic into the the system itself either as a graph or as a state machine.
10:15And so you took control of the orchestration while you allowed the models to operate at the node level.
10:23And that way you got a lot more uh reliability and predictability in how your AI was going to operate across those various intents.
10:35But then the problem is you are now building a system that is designed to work for a specific set of intents for a specific types of use cases.
10:44And as you start hand, you know, the system starts interacting with with instances that are outside that distribution, the system starts struggling with that, right?
10:55You expect um you know a certain set of applications or u user interactions to work well because they can be fulfilled by the orchestration that you have designed.
11:04But when your the user intent needs to be requires other things to happen beyond what's specified in the orchestration, the system can start um you know breaking at the seams.
11:17And uh in order to do that, folks were now building a lot more complexity into their orchestration logic.
11:23And so you're building these special uh branches and way you handle special intents in the complex graph that described your system.
11:33And so what that means is like you had now a ton of different surfaces for failure.
11:39So you now had to deal with uh you know uh dealing with uh branch consistency and branching logic failures.
11:47You had to deal with things like the contracts between the nodes not working out well.
11:53Uh you had to deal with the limitations of uh the nodes that were you know built for a specific set of use cases.
12:03So you know there were classifier nodes for example and they could make mistakes and so you could now have a significant amount of um you know areas where you could uh where the system could fail.
12:17And so your evals now have to not only look at u you know the overall orchestration but they now have to you have to have node level evals.
12:29you have to uh make sure that you have evalu uh you know how you do retry loops.
12:41There's a lot of complex behaviors of the system that now need to be evaluated in addition to all the other things that you were evaluating before.
12:54So the graphs were kind of popular like in in late 24 early 25 and so a lot of systems were now implemented using certain frameworks and they were now in production.
13:08Uh but then um Anthropic and OpenAI launched some amazing new model capabilities mid late 25 and what that was like tool calling became extremely reliable.
13:22We started uh seeing uh much better orchestration control.
13:26Uh the models were able to plan a lot more effectively accurately.
13:32They were able to manage long horizon tasks.
13:34they were able to do a much better job of introspecting and course correcting.
13:39And so like as things went a little off track, the models were able to, you know, understand that and bring the execution back on track.
13:48And so what that meant was a lot of these u u graph-based systems were not able to take advantage of these new capabilities.
13:59they were still running into some of those like brittleleness issues that the new model state-of-art had unlocked and so um we started looking at building out um the react loop again that's that started working and so now you had this new
14:16AI systems that could effectively reliably work in a loop they could make those tool calls they could figure out the next step and then they could u essentially go in and um fulfill the user intent.
14:32But the way they worked was very it had a high degree of variance.
14:36So every trajectory for the same input if you ran it a couple of times you would see you know dramatically different trajectories while yielding the right answer.
14:44And so now there's a lot of variance that you have to deal with.
14:47So now instead of just focusing on a specific eval the unit of eval was no longer just one eval now you're looking at doing an analysis of the distribution of the evals you're taking the same eval you're running it multiple times you're running it k times and you're ensuring that uh you get a statistically relevant signal from
15:08that eval so now new metrics like uh pass at k and pass raise to k or pass wedge k these were the new metrics that certainly started to make a lot of sense.
15:21pass at K is like if you take the same that eval and you run it K times does it succeed at least once and that is a measure of its capability
15:31and pass wedge K is like if you run that eval multiple times how many times of those K instances does it run successfully that's a measure of its uh reliability
15:41and so now you can understand whether your system with a high pass at K uh you know is reliable by seeing seeing how it you know by measuring the pass wedge K metric for example.
15:55So
15:56[snorts]
15:56this gives you a lot more um you know u understanding of like how your system is working what the failure sources are and how you work on those right
16:07and then more recently what we've seen is um there's a big shift from it's your system is not just a model running in a loop right it becomes a product system it's that
16:19there's a model in the loop that's augmented by a lot of peripheral components you know you have a memory system that is able to provide robust memory storage and memory
16:29um retrieval capabilities uh within a session cross sessions.
16:34Uh models can tap into this memory to you know improve upon their runs in subsequent instances by learning from previous runs for example.
16:43You've got robust code execution uh sandboxes now and so you can run model generated code reliably robustly on uh uh during uh execution.
16:55You've got um MCP and skill uh directories that the model can now tap into and you can you know weave in extensibility.
17:04You now have things like a skills repository or a skill systems that can be used to continually augment the the the capabilities of models through you know symbolic instructions.
17:17And so uh now you know like uh these systems are getting pretty complex and as a result uh you know if you are continuing to to use the eval from the previous generation
17:32you're going to get sort of a partial coverage of your system.
17:36you're not going to see uh how your system is fragile in ways because of the unlock because of the new surface that you have uh you know uh unlocked in your new system.
17:49So what that means is um just reflecting back on the pattern is like you know all of these model innovations resulted in in you know corresponding shift in the architectures and so so you've seen these waves of architecture
18:07and then what's needed is like your evals to be congru congruent with that architecture right uh because ultimately it's the eval that are sort of your durable asset that describe how your system is supposed to work.
18:21And as you go through these generational shifts, that's a good way to ensure that you know your system your user users experience your system in a way that things that were working are not broken, but it's unlocked a bunch of new capability.
18:35And so everyone's seen this, you know, diagram of this flywheel.
18:39Everyone's sort of like bought into it conceptually, right?
18:42the idea of harvesting data from production to inform your eval so that your evals are reflective of the real world.
18:50I think that all makes sense, right?
18:52And and this is the way that you know teams that are doing a great job at building and shipping and improving their AI systems, they they they follow this workflow pretty religiously.
19:03Um so I've talked to a lot of teams and I think while there is a general acceptance that yeah you need to run that workflow um in practice a lot of teams don't do that their eval are somewhat static
19:15and even if you're not changing your AI agent architecture you're you know by not really being disciplined about running that that workflow that flywheel you are now getting stagnant evals that are not being as effective in helping you
19:32measure and improve the quality of your AI.
19:34And especially as you go through this generational shift, it's really important that you need a mechanism to not only harvest data from production in a way that shows you failures
19:45that you are looking out for because you defined what good looks like as part of your evals.
19:53But you also want something to shine a light on the new failure types, right?
19:59the system is going to fail in new and novel ways in ways that you might not have anticipated and you now need to start harvesting that data in a meaningful way.
20:08And you want to do this again as as part of the flywheel.
20:12And so this is where you need systems to come in and u shine a light on things that are broken in ways that you had anticipated, but also broken in a way in ways that you had not anticipated.
20:26And this is really important.
20:29So I'm going to quickly talk a little bit about like how we do this in brain trust.
20:33So brain trust provides all the components that you need to run this flywheel.
20:37We've got evals, we've got observability.
20:40We have ways in which you can get insights from your production data to harvest u new eval cases that you can then pass off to the to the team that they can then use to hill climb and
20:53improve your AI system.
20:56But topics is a really cool feature.
20:58What topics does, it does a cluster analysis on all of your production data.
21:04And so the idea over here is now you are able to find new categories of failures that you had not anticipated.
21:12So your system is now able to look at all what's going on in production and it's able to now start surfacing these new failure modes that tell you here's a new new
21:23failure uh you know um uh situation that you hadn't thought about and you didn't have any guardrails in place.
21:31didn't have any eval in place and so now it's really easy for teams to expand the set of their evals to now cover those kind of new failures.
21:42And so this is this is a pretty exciting uh capability in brain trust that enables these teams to continually not only get new failure examples for known failure modes
21:56but more importantly as they make these systemic architectural changes they're able to also understand the new ways in which your system is going to fail and build out effective data sets from production data.
22:11So I think the takeaway for today's talk is that the models will keep on changing.
22:19Uh I I don't think we're going to see any slowdown.
22:21I don't think we have hit a plateau yet.
22:23I think there are lots of unlocks that are coming down um this road.
22:28Um and as a result you will be making significant changes to your AI systems.
22:35you know, you'll be doing a lot of surgery on your AI agents in the coming months, years.
22:42And so it's really important that you have a robust workflow system in place to ensure that as you make those changes, as you incorporate these new models into your systems,
22:55that your systems continue to get better at doing new things, but also continue to work well for the things that they were doing before.
23:06And so building out like a robust eval discipline uh with the right tools and the right automation and the right systems becomes paramount to manage these generational changes.
23:19And so ultimately what you want is um to really uh index on that flywheel and make it part of your workflow so that uh you know the ability to
23:34improve incrementally when the changes in the system are incremental and the ability to improve your system in a in sort of a step function way are both supported by your
23:47evals.
23:48So with that, I want to say thank you.
23:54[applause]