How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs

AI Engineer · 13 min · 156 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hey, so I'm Milind, MTS at Brainbase.
00:14We help enterprises deploy internal agents at scale and manage them.
00:18Today I'm going to be talking about hill climbing back office agents and how we did that successfully with a customer.
00:26So yeah, our customer in this case I'm going going to be talking about throughout and like the famous local video we use repeatedly is is is Domino's.
00:32As you all know, they have 20 plus thousand stores, 90 plus markets.
00:36We started off with a single region with their CMO office.
00:40We wanted to answer this question that they kept asking.
00:43What does a pizza cost at the competitor down the street?
00:46What what does it cost in their delivery app?
00:48What does it cost on their actual website with promos, without promos, things like that.
00:52They essentially wanted a spreadsheet at the end with the lowest price possible for a bunch of different items.
00:58Before we built the agent for them, they had an analyst manually doing these steps.
01:04They all look easy in isolation.
01:05You open a competitor's website, pick the right store, compare prices with delivery, takeaway, find the right promos, find the cheapest price and write it all down into a spreadsheet.
01:15It it all looks easy enough, but if you get one of these steps wrong, you look like you you you get a cell at the end that looks finished, but it's just wrong.
01:23We had to build an agent that does the same.
01:25Reads a bunch of items, finds the website, interprets prices, fills the spreadsheets, hands it off to an agent, and flags anything that it thinks it's not sure of.
01:35When we created version one of this agent for them, we had 41% accuracy.
01:40By the end at version 11, we had 97% accuracy.
01:44Our cost per source per source means per item per per competitor per website went from $2 per source to $1 per source.
01:54And the amount of minutes it took per run went from 129 to 71.
01:57All of this went just with just just instructions and and skills and playbooks changing.
02:03Here is like the six steps that we use, not just with Domino's, but with other customers as well.
02:08We start off with kickoff, followed by anthology, benchmarks, first agent, iterating, and handing off.
02:14I'll talk about what these steps mean in detail and what we did for Domino's in each of these and hopefully it's a repeatable framework that y'all can use internally as well.
02:23So, step one is kickoff.
02:24You should ideally talk to your customer or end user and agree on what the win looks like.
02:30As of today, what what is the job?
02:32What did the human sign off on at the end of it?
02:34How long does it take?
02:35How much does it cost?
02:37How accurate does your agent need to be?
02:39Who judges the output?
02:40And what must the agent never do alone?
02:42What assumptions should it never make?
02:45For Domino's, we ended up creating this brief that worked really well for us.
02:48Our outcome at the end had to be a faster, repeatable way to collect local competitor pricing without losing the judgment edge cases need.
02:56So, we asked them uh all of these questions.
03:00As of today, the analysts create a spreadsheet and the pricing team uh find finally analyzes it.
03:05An analyst spent 2 hours daily manually actively working on creating the spreadsheet.
03:11Every cell was then checked and and just confirmed and the agent at the end just had to ask for the analyst for sign off and flag anything else it had to.
03:20Our win in numbers looked like just having the highest accuracy, the lowest cost of sources possible, run time to be as low as possible, and before we even started, they weren't even sure if we can do this discount search at all.
03:33[snorts]
03:34Step two is anthology.
03:35What does that exactly mean?
03:36We we try to write down all the terms that that the agent needs to know to never make assumptions, which means we need to tell the agent exactly what it's supposed to collect,
03:47where each answer comes from, where does it go, what does each term mean to the analysts that actually do the do the work manually, and what should the agent
03:56do when it can't figure something out.
03:58If if there's one jewel here, it is that every term should ideally have a definition.
04:02If you don't write something down, the agent's going to guess and probably get something wrong.
04:07For Domino's, here's what it looked like.
04:09We at Brainbase have a memory primitive of our own that has memory that looks like a Postgres database that the agent can do whatever it wants with.
04:16We gave it a target stable where we gave it up all the items.
04:20In this case, let's look at ID number 14.
04:22ID 14 is brand B. It's a large pizza.
04:26Uh takeaway method is Sorry, it's delivery and on app B. The agent hypothetically does browser use, finds the price on the website, finds that it's 24 and 19.
04:3624 is the striked out price, 19 is the sale price.
04:40We also now have a rules file where we actually tell the agent to specifically mark down the price that it that a user pays at checkout, not 24.
04:49This came from like a valid failure.
04:51What it saves at the end, we created a table again called price observations where it with the corresponding ID stores what price it saw, what the full price was,
05:00and the actual source.
05:02And at the end, we give it a spreadsheet that has all placeholder item IDs, and all it has to do is look up the ID and replace the price.
05:10When a cell is wrong, it points to the exact page and the rule behind why it chose the selected price.
05:17Other rules that we mentioned that came through actual life traces were definition of what a large pizza looks like.
05:23It's anything that's actually the item itself or a bundle.
05:28Delivery is not equal to takeaway.
05:30It It got confused with that.
05:32It's supposed to do promotion math.
05:33If it's 30% off the price, it should do that before entering it in the spreadsheet.
05:37And if it can't verify it, write it down as a gap and do not guess.
05:43Step three is finding out benchmarks and emails.
05:45Now Now, these steps can be interchangeable slightly, but essentially you're trying to figure out what you can run offline emails on and what you can run online emails on.
05:54For offline emails, you should ideally start off by asking them if they have past cases that they already have answers for.
06:00In this case, we had analysts from Domino's who had spreadsheets that they had worked on.
06:04Can we actually rerun those um input cases to get the same final output?
06:09And then at the end, how do you figure out how to score each thing that the agent is doing?
06:14For evals, you try to figure out what you can check deterministically, what you can check why an LM as a judge, and what you can check as an agent as a judge.
06:22That's a concept that we think is slightly more unique, where if there's an output artifact, you probably want an agent to take multiple turns, multiple tool calls to figure out how correct that that artifact is.
06:34And at the end, what happens when something goes wrong, and what happens when it's flagged?
06:37You should try to score as many things as you can with these three things.
06:42So, yeah, at Domino's, our benchmark essentially started off with all the spreadsheets that they had.
06:47We tried to create the input environment by going to archive.org and having snapshots of competitor websites at those given dates.
06:54What we saw that it was that it didn't work with delivery apps because they didn't serve data the same way.
06:59So, we ended up storing browser screenshots for everything the agent was doing starting from version one of the agent, and that's how we create up our environment for for the internal benchmark.
07:08For evals, we use code for things like checking if every ID cell was replaced with the price, if the discount map was computed correctly, if the price was 20%
07:19plus or minus compared to the previous run, was it flagged or not.
07:23LM as a judge just checks the trace to see if the right item was selected, and if the right promo was applied.
07:29Agent as a judge actually goes to every flagged price and actually confirms if the flag was valid or not.
07:34And the analyst at the end will go look at the final flags and try to figure out what the reason for it was if something went wrong.
07:42Step four is actually shipping some version of a first agent.
07:45You should ideally ship a narrow first agent in your stack that you can score from from day one.
07:51You should probably figure out what the smallest slice worth scoring is.
07:54You should figure out how accurate things should be, how long and and you should use this agent to figure out how long does it run take, what what does it cost,
08:03what does it get wrong, and things like that.
08:05Version one's job is to be measured, it's not to be good.
08:08You should widen the scope uh widen the scope after you get some baseline score they're trying to meet.
08:14For Domino's, as like I mentioned earlier, version one for us was just 41% correct.
08:19It took more than 2 hours.
08:21It was $2 per source.
08:23And and it just wasn't working well.
08:25We found a bunch of changes, made fixes, and then improved and increased the scope.
08:31So, we started off with just one app, one item in one city.
08:35Moved on to adding burgers and takeaway options.
08:38Moved on to adding another app, and finally other websites as well.
08:42So, starting off small narrow scope worked out really well for us to figure out how we can fix things.
08:49Step five is to just iterate.
08:51You should score, fix, and ship.
08:52Here is where you'll do steps two, three, and four.
08:55Keep iterating, improving your anthology, improving your emails, your benchmarks to get things that work.
09:01You should figure out which failures are most expensive, most frequent, fix them, what changes should be used to fix them.
09:08If that change results in increased run time or cost, if yes, are you okay with that increased run time and cost?
09:13And how can you use those failures to create and add to your benchmark cases or eval criteria?
09:18Ideally, change one thing at a time and fix the instructions before you want to change the model entirely.
09:26At Domino's, we highlighted two examples.
09:28Before I get to that, here is a graph of how our progression went.
09:33We one started off at 41%.
09:35Um more than $2 a source.
09:37We 11 after a bunch of changes, it's $1 a source, and and 97% accurate.
09:44So, yeah.
09:44Here's an example of a very simple rule change that we made.
09:48We gave the agent a link of of a large pizza to find for brand B. Via delivery on its own website.
09:55It went to the link which had the section already highlighted.
09:59It clicked on it, found the large pizza, found the price, and ended there.
10:04The analyst got that somebody manually caught at the end that there was actually a deal below that website below on the same page, that was just $21 instead, and
10:15and it missed that, right?
10:16So, all we had to do there was add a simple rule to not stop at the section that it gets linked to, but actually look at the entire page.
10:22And a simple change like that helped us just like catch this and then make sure it doesn't happen again.
10:28Another more intricate change that we had to make was actually browser use behavior of set agent.
10:35So, in one of the runs, we saw that for brand D this is a this is a brand called brand D promotions page, where all the promotion prices were inside images, so not actually captured in text.
10:45What the agent did is it looked at all 26 promo images to find the item it was looking for.
10:51It looked them looked at them again by zooming in 4x, 5x, 6x.
10:56It found the right card at minute 13, but it still kept zooming to double-check and verify, and it it was just in a loop, and we had to manually stop it at minute 15.
11:06We realized after manually looking at the trace and the website, that the HTML DOM is actually super important and could have saved the agent a lot of work here.
11:14So, we we tell the agent in in our latest rules to go look at the HTML DOM first, look at things like image titles, all text before looking at the actual image rates,
11:24and just stopping past a certain zoom level.
11:27What that resulted in is is 1/3 the number of image rates, reducing cost by over 50%, and just us getting an actual answer and a finished report.
11:36So, while we have automated checks and eval criteria to help us catch a lot of things, some things require our engineers to actually look at traces and and browser on use tool calls and and just see what went wrong.
11:49So, yeah, we made a bunch of these changes, got to V11, got to 97% accuracy, and now it's just figuring out, hey, like like what next?
11:56So, so our our way of this is trying to figure out how best to hand off to the customer at the end.
12:01We want to figure out who's running these agents, who's reviewing the outputs, who's trying to correct who's trying to take the corrections into and converting into rules and benchmark cases,
12:11and who decides when new version is ready to ship.
12:13All these questions are things you should probably have written down.
12:16Um you're trying to hand off an entire loop and not just the agent.
12:19Ideally the the user should be empowered enough to make these changes if they're small enough themselves.
12:24So for for for Domino's, that's exactly what we did.
12:27The agent now runs on their schedule either multi um daily or even by email trigger manually.
12:33Analysts review only the exceptions at this point.
12:36Each correction that they catch, they're automatically able to add as a benchmark case.
12:41If they can't, then they obviously they're always available for them.
12:44And we are now at a place where we are expanding with them.
12:48Um trying to get it into multiple markets and multiple other um areas.
12:53So yeah, this is basically how we we used um our framework to work with Domino's to go from an agent that barely works to an agent that works almost perfectly
13:04while reducing the cost.
13:06We found this framework to be very effective for various other use cases, especially where you have messy sources of data where you're trying to figure out the right answer.
13:15And um thanks.