WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=4VmugsE-iP0 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/4VmugsE-iP0.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.680 --> 00:00:14.920 Hey, so I'm Milind, MTS at Brainbase. s3 00:00:14.920 --> 00:00:18.880 We help enterprises deploy internal agents at scale and manage them. s4 00:00:18.880 --> 00:00:25.240 Today I'm going to be talking about hill climbing back office agents and how we did that successfully with a customer. s5 00:00:26.280 --> 00:00:32.800 So yeah, our customer in this case I'm going going to be talking about throughout and like the famous local video we use repeatedly is is is Domino's. s6 00:00:32.800 --> 00:00:36.560 As you all know, they have 20 plus thousand stores, 90 plus markets. s7 00:00:36.560 --> 00:00:40.520 We started off with a single region with their CMO office. s8 00:00:40.520 --> 00:00:43.560 We wanted to answer this question that they kept asking. s9 00:00:43.560 --> 00:00:46.520 What does a pizza cost at the competitor down the street? s10 00:00:46.520 --> 00:00:48.360 What what does it cost in their delivery app? s11 00:00:48.360 --> 00:00:52.440 What does it cost on their actual website with promos, without promos, things like that. s12 00:00:52.440 --> 00:00:58.800 They essentially wanted a spreadsheet at the end with the lowest price possible for a bunch of different items. s13 00:00:58.800 --> 00:01:04.080 Before we built the agent for them, they had an analyst manually doing these steps. s14 00:01:04.080 --> 00:01:05.800 They all look easy in isolation. s15 00:01:05.800 --> 00:01:15.840 You open a competitor's website, pick the right store, compare prices with delivery, takeaway, find the right promos, find the cheapest price and write it all down into a spreadsheet. s16 00:01:15.840 --> 00:01:23.560 It it all looks easy enough, but if you get one of these steps wrong, you look like you you you get a cell at the end that looks finished, but it's just wrong. s17 00:01:23.560 --> 00:01:25.720 We had to build an agent that does the same. s18 00:01:25.720 --> 00:01:35.960 Reads a bunch of items, finds the website, interprets prices, fills the spreadsheets, hands it off to an agent, and flags anything that it thinks it's not sure of. s19 00:01:35.960 --> 00:01:40.960 When we created version one of this agent for them, we had 41% accuracy. s20 00:01:40.960 --> 00:01:44.240 By the end at version 11, we had 97% accuracy. s21 00:01:44.240 --> 00:01:54.200 Our cost per source per source means per item per per competitor per website went from $2 per source to $1 per source. s22 00:01:54.200 --> 00:01:57.560 And the amount of minutes it took per run went from 129 to 71. s23 00:01:57.560 --> 00:02:03.932 All of this went just with just just instructions and and skills and playbooks changing. s24 00:02:03.932 --> 00:02:08.572 Here is like the six steps that we use, not just with Domino's, but with other customers as well. s25 00:02:08.572 --> 00:02:14.932 We start off with kickoff, followed by anthology, benchmarks, first agent, iterating, and handing off. s26 00:02:14.932 --> 00:02:23.292 I'll talk about what these steps mean in detail and what we did for Domino's in each of these and hopefully it's a repeatable framework that y'all can use internally as well. s27 00:02:23.292 --> 00:02:24.492 So, step one is kickoff. s28 00:02:24.492 --> 00:02:30.252 You should ideally talk to your customer or end user and agree on what the win looks like. s29 00:02:30.252 --> 00:02:32.692 As of today, what what is the job? s30 00:02:32.692 --> 00:02:34.892 What did the human sign off on at the end of it? s31 00:02:34.892 --> 00:02:35.972 How long does it take? s32 00:02:35.972 --> 00:02:37.372 How much does it cost? s33 00:02:37.372 --> 00:02:39.292 How accurate does your agent need to be? s34 00:02:39.292 --> 00:02:40.492 Who judges the output? s35 00:02:40.492 --> 00:02:42.252 And what must the agent never do alone? s36 00:02:42.252 --> 00:02:45.292 What assumptions should it never make? s37 00:02:45.292 --> 00:02:48.932 For Domino's, we ended up creating this brief that worked really well for us. s38 00:02:48.932 --> 00:02:56.892 Our outcome at the end had to be a faster, repeatable way to collect local competitor pricing without losing the judgment edge cases need. s39 00:02:56.892 --> 00:03:00.092 So, we asked them uh all of these questions. s40 00:03:00.092 --> 00:03:05.772 As of today, the analysts create a spreadsheet and the pricing team uh find finally analyzes it. s41 00:03:05.772 --> 00:03:11.252 An analyst spent 2 hours daily manually actively working on creating the spreadsheet. s42 00:03:11.252 --> 00:03:20.092 Every cell was then checked and and just confirmed and the agent at the end just had to ask for the analyst for sign off and flag anything else it had to. s43 00:03:20.092 --> 00:03:33.532 Our win in numbers looked like just having the highest accuracy, the lowest cost of sources possible, run time to be as low as possible, and before we even started, they weren't even sure if we can do this discount search at all. s44 00:03:33.532 --> 00:03:34.012 [snorts] s45 00:03:34.012 --> 00:03:35.412 Step two is anthology. s46 00:03:35.412 --> 00:03:36.452 What does that exactly mean? s47 00:03:36.452 --> 00:03:47.292 We we try to write down all the terms that that the agent needs to know to never make assumptions, which means we need to tell the agent exactly what it's supposed to collect, s48 00:03:47.292 --> 00:03:56.212 where each answer comes from, where does it go, what does each term mean to the analysts that actually do the do the work manually, and what should the agent s49 00:03:56.212 --> 00:03:58.412 do when it can't figure something out. s50 00:03:58.412 --> 00:04:02.412 If if there's one jewel here, it is that every term should ideally have a definition. s51 00:04:02.412 --> 00:04:07.374 If you don't write something down, the agent's going to guess and probably get something wrong. s52 00:04:07.374 --> 00:04:09.094 For Domino's, here's what it looked like. s53 00:04:09.094 --> 00:04:16.974 We at Brainbase have a memory primitive of our own that has memory that looks like a Postgres database that the agent can do whatever it wants with. s54 00:04:16.974 --> 00:04:20.814 We gave it a target stable where we gave it up all the items. s55 00:04:20.814 --> 00:04:22.854 In this case, let's look at ID number 14. s56 00:04:22.854 --> 00:04:26.254 ID 14 is brand B. It's a large pizza. s57 00:04:26.254 --> 00:04:36.894 Uh takeaway method is Sorry, it's delivery and on app B. The agent hypothetically does browser use, finds the price on the website, finds that it's 24 and 19. s58 00:04:36.894 --> 00:04:40.094 24 is the striked out price, 19 is the sale price. s59 00:04:40.094 --> 00:04:49.054 We also now have a rules file where we actually tell the agent to specifically mark down the price that it that a user pays at checkout, not 24. s60 00:04:49.054 --> 00:04:51.334 This came from like a valid failure. s61 00:04:51.334 --> 00:05:00.574 What it saves at the end, we created a table again called price observations where it with the corresponding ID stores what price it saw, what the full price was, s62 00:05:00.574 --> 00:05:02.054 and the actual source. s63 00:05:02.054 --> 00:05:10.574 And at the end, we give it a spreadsheet that has all placeholder item IDs, and all it has to do is look up the ID and replace the price. s64 00:05:10.574 --> 00:05:17.734 When a cell is wrong, it points to the exact page and the rule behind why it chose the selected price. s65 00:05:17.734 --> 00:05:23.574 Other rules that we mentioned that came through actual life traces were definition of what a large pizza looks like. s66 00:05:23.574 --> 00:05:28.534 It's anything that's actually the item itself or a bundle. s67 00:05:28.534 --> 00:05:30.014 Delivery is not equal to takeaway. s68 00:05:30.014 --> 00:05:32.094 It It got confused with that. s69 00:05:32.094 --> 00:05:33.854 It's supposed to do promotion math. s70 00:05:33.854 --> 00:05:37.734 If it's 30% off the price, it should do that before entering it in the spreadsheet. s71 00:05:37.734 --> 00:05:43.334 And if it can't verify it, write it down as a gap and do not guess. s72 00:05:43.334 --> 00:05:45.534 Step three is finding out benchmarks and emails. s73 00:05:45.534 --> 00:05:54.734 Now Now, these steps can be interchangeable slightly, but essentially you're trying to figure out what you can run offline emails on and what you can run online emails on. s74 00:05:54.734 --> 00:06:00.934 For offline emails, you should ideally start off by asking them if they have past cases that they already have answers for. s75 00:06:00.934 --> 00:06:04.895 In this case, we had analysts from Domino's who had spreadsheets that they had worked on. s76 00:06:04.895 --> 00:06:09.975 Can we actually rerun those um input cases to get the same final output? s77 00:06:09.975 --> 00:06:14.735 And then at the end, how do you figure out how to score each thing that the agent is doing? s78 00:06:14.735 --> 00:06:22.735 For evals, you try to figure out what you can check deterministically, what you can check why an LM as a judge, and what you can check as an agent as a judge. s79 00:06:22.735 --> 00:06:34.055 That's a concept that we think is slightly more unique, where if there's an output artifact, you probably want an agent to take multiple turns, multiple tool calls to figure out how correct that that artifact is. s80 00:06:34.055 --> 00:06:37.855 And at the end, what happens when something goes wrong, and what happens when it's flagged? s81 00:06:37.855 --> 00:06:42.135 You should try to score as many things as you can with these three things. s82 00:06:42.135 --> 00:06:47.215 So, yeah, at Domino's, our benchmark essentially started off with all the spreadsheets that they had. s83 00:06:47.215 --> 00:06:54.375 We tried to create the input environment by going to archive.org and having snapshots of competitor websites at those given dates. s84 00:06:54.375 --> 00:06:59.535 What we saw that it was that it didn't work with delivery apps because they didn't serve data the same way. s85 00:06:59.535 --> 00:07:08.695 So, we ended up storing browser screenshots for everything the agent was doing starting from version one of the agent, and that's how we create up our environment for for the internal benchmark. s86 00:07:08.695 --> 00:07:19.855 For evals, we use code for things like checking if every ID cell was replaced with the price, if the discount map was computed correctly, if the price was 20% s87 00:07:19.855 --> 00:07:23.735 plus or minus compared to the previous run, was it flagged or not. s88 00:07:23.735 --> 00:07:29.055 LM as a judge just checks the trace to see if the right item was selected, and if the right promo was applied. s89 00:07:29.055 --> 00:07:34.975 Agent as a judge actually goes to every flagged price and actually confirms if the flag was valid or not. s90 00:07:34.975 --> 00:07:42.375 And the analyst at the end will go look at the final flags and try to figure out what the reason for it was if something went wrong. s91 00:07:42.375 --> 00:07:45.415 Step four is actually shipping some version of a first agent. s92 00:07:45.415 --> 00:07:51.495 You should ideally ship a narrow first agent in your stack that you can score from from day one. s93 00:07:51.495 --> 00:07:54.895 You should probably figure out what the smallest slice worth scoring is. s94 00:07:54.895 --> 00:08:03.135 You should figure out how accurate things should be, how long and and you should use this agent to figure out how long does it run take, what what does it cost, s95 00:08:03.135 --> 00:08:05.295 what does it get wrong, and things like that. s96 00:08:05.295 --> 00:08:08.537 Version one's job is to be measured, it's not to be good. s97 00:08:08.537 --> 00:08:14.977 You should widen the scope uh widen the scope after you get some baseline score they're trying to meet. s98 00:08:14.977 --> 00:08:19.777 For Domino's, as like I mentioned earlier, version one for us was just 41% correct. s99 00:08:19.777 --> 00:08:21.537 It took more than 2 hours. s100 00:08:21.537 --> 00:08:23.457 It was $2 per source. s101 00:08:23.457 --> 00:08:25.657 And and it just wasn't working well. s102 00:08:25.657 --> 00:08:31.297 We found a bunch of changes, made fixes, and then improved and increased the scope. s103 00:08:31.297 --> 00:08:35.417 So, we started off with just one app, one item in one city. s104 00:08:35.417 --> 00:08:38.337 Moved on to adding burgers and takeaway options. s105 00:08:38.337 --> 00:08:42.297 Moved on to adding another app, and finally other websites as well. s106 00:08:42.297 --> 00:08:49.297 So, starting off small narrow scope worked out really well for us to figure out how we can fix things. s107 00:08:49.297 --> 00:08:51.297 Step five is to just iterate. s108 00:08:51.297 --> 00:08:52.737 You should score, fix, and ship. s109 00:08:52.737 --> 00:08:55.457 Here is where you'll do steps two, three, and four. s110 00:08:55.457 --> 00:09:01.177 Keep iterating, improving your anthology, improving your emails, your benchmarks to get things that work. s111 00:09:01.177 --> 00:09:08.017 You should figure out which failures are most expensive, most frequent, fix them, what changes should be used to fix them. s112 00:09:08.017 --> 00:09:13.737 If that change results in increased run time or cost, if yes, are you okay with that increased run time and cost? s113 00:09:13.737 --> 00:09:18.977 And how can you use those failures to create and add to your benchmark cases or eval criteria? s114 00:09:18.977 --> 00:09:26.217 Ideally, change one thing at a time and fix the instructions before you want to change the model entirely. s115 00:09:26.217 --> 00:09:28.777 At Domino's, we highlighted two examples. s116 00:09:28.777 --> 00:09:33.297 Before I get to that, here is a graph of how our progression went. s117 00:09:33.297 --> 00:09:35.657 We one started off at 41%. s118 00:09:35.657 --> 00:09:37.697 Um more than $2 a source. s119 00:09:37.697 --> 00:09:44.377 We 11 after a bunch of changes, it's $1 a source, and and 97% accurate. s120 00:09:44.737 --> 00:09:44.977 So, yeah. s121 00:09:44.977 --> 00:09:48.297 Here's an example of a very simple rule change that we made. s122 00:09:48.297 --> 00:09:55.857 We gave the agent a link of of a large pizza to find for brand B. Via delivery on its own website. s123 00:09:55.857 --> 00:09:59.137 It went to the link which had the section already highlighted. s124 00:09:59.137 --> 00:10:04.017 It clicked on it, found the large pizza, found the price, and ended there. s125 00:10:04.017 --> 00:10:15.298 The analyst got that somebody manually caught at the end that there was actually a deal below that website below on the same page, that was just $21 instead, and s126 00:10:15.298 --> 00:10:16.338 and it missed that, right? s127 00:10:16.338 --> 00:10:22.698 So, all we had to do there was add a simple rule to not stop at the section that it gets linked to, but actually look at the entire page. s128 00:10:22.698 --> 00:10:28.738 And a simple change like that helped us just like catch this and then make sure it doesn't happen again. s129 00:10:28.738 --> 00:10:35.098 Another more intricate change that we had to make was actually browser use behavior of set agent. s130 00:10:35.098 --> 00:10:45.818 So, in one of the runs, we saw that for brand D this is a this is a brand called brand D promotions page, where all the promotion prices were inside images, so not actually captured in text. s131 00:10:45.818 --> 00:10:51.778 What the agent did is it looked at all 26 promo images to find the item it was looking for. s132 00:10:51.778 --> 00:10:56.738 It looked them looked at them again by zooming in 4x, 5x, 6x. s133 00:10:56.738 --> 00:11:06.378 It found the right card at minute 13, but it still kept zooming to double-check and verify, and it it was just in a loop, and we had to manually stop it at minute 15. s134 00:11:06.378 --> 00:11:14.378 We realized after manually looking at the trace and the website, that the HTML DOM is actually super important and could have saved the agent a lot of work here. s135 00:11:14.378 --> 00:11:24.418 So, we we tell the agent in in our latest rules to go look at the HTML DOM first, look at things like image titles, all text before looking at the actual image rates, s136 00:11:24.418 --> 00:11:27.578 and just stopping past a certain zoom level. s137 00:11:27.578 --> 00:11:36.018 What that resulted in is is 1/3 the number of image rates, reducing cost by over 50%, and just us getting an actual answer and a finished report. s138 00:11:36.018 --> 00:11:49.258 So, while we have automated checks and eval criteria to help us catch a lot of things, some things require our engineers to actually look at traces and and browser on use tool calls and and just see what went wrong. s139 00:11:49.258 --> 00:11:56.378 So, yeah, we made a bunch of these changes, got to V11, got to 97% accuracy, and now it's just figuring out, hey, like like what next? s140 00:11:56.378 --> 00:12:01.178 So, so our our way of this is trying to figure out how best to hand off to the customer at the end. s141 00:12:01.178 --> 00:12:11.058 We want to figure out who's running these agents, who's reviewing the outputs, who's trying to correct who's trying to take the corrections into and converting into rules and benchmark cases, s142 00:12:11.058 --> 00:12:13.139 and who decides when new version is ready to ship. s143 00:12:13.139 --> 00:12:16.019 All these questions are things you should probably have written down. s144 00:12:16.019 --> 00:12:19.539 Um you're trying to hand off an entire loop and not just the agent. s145 00:12:19.539 --> 00:12:24.899 Ideally the the user should be empowered enough to make these changes if they're small enough themselves. s146 00:12:24.899 --> 00:12:27.259 So for for for Domino's, that's exactly what we did. s147 00:12:27.259 --> 00:12:33.379 The agent now runs on their schedule either multi um daily or even by email trigger manually. s148 00:12:33.379 --> 00:12:36.219 Analysts review only the exceptions at this point. s149 00:12:36.219 --> 00:12:41.099 Each correction that they catch, they're automatically able to add as a benchmark case. s150 00:12:41.099 --> 00:12:44.059 If they can't, then they obviously they're always available for them. s151 00:12:44.059 --> 00:12:48.099 And we are now at a place where we are expanding with them. s152 00:12:48.099 --> 00:12:53.619 Um trying to get it into multiple markets and multiple other um areas. s153 00:12:53.979 --> 00:13:04.939 So yeah, this is basically how we we used um our framework to work with Domino's to go from an agent that barely works to an agent that works almost perfectly s154 00:13:04.939 --> 00:13:06.579 while reducing the cost. s155 00:13:06.579 --> 00:13:15.219 We found this framework to be very effective for various other use cases, especially where you have messy sources of data where you're trying to figure out the right answer. s156 00:13:15.219 --> 00:13:18.459 And um thanks.