WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=PXj0p_mW9nI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/PXj0p_mW9nI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:19.760 --> 00:00:20.720 Good morning. s2 00:00:20.720 --> 00:00:24.800 We're super excited to be here at AI Engineer with all of you. s3 00:00:24.800 --> 00:00:28.080 I'm Caitlyn and I lead platform engineering at Anthropic. s4 00:00:28.080 --> 00:00:28.880 And I'm Angela. s5 00:00:28.880 --> 00:00:30.880 I lead platform product at Anthropic. s6 00:00:30.880 --> 00:00:41.520 And today we want to talk to you about a concept that we've been spending a lot of time thinking about and working on with our team, which is this idea that we think that tokens should have jobs. s7 00:00:41.760 --> 00:00:59.120 So if you're building an agentic system and you're trying to accomplish some specific outcome, you're trying to get something done with agents, there's one lever that everybody pulls in order to get a better outcome, and that's usually increasing your budget, which means you spend more tokens or you spend more expensive tokens. s8 00:00:59.760 --> 00:01:11.200 But we've been wondering is that all there is underlying this assumption of uh using the budget is this kind of implicit perspective that every single token is basically fungeable. s9 00:01:11.200 --> 00:01:13.520 And we've been wondering is that actually true? s10 00:01:13.520 --> 00:01:15.840 Are all these tokens actually fungeible? s11 00:01:15.840 --> 00:01:20.240 And to test that, we've been thinking, what if we gave tokens jobs? s12 00:01:20.640 --> 00:01:29.759 So, if you think about the way that you would normally set up an agent to go accomplish a task, you give it that task, you give it this token budget, and then all the tokens that are being spent are basically indiscriminate s13 00:01:29.759 --> 00:01:31.680 in the sense that they're all doing one job. s14 00:01:31.680 --> 00:01:34.079 They're just executing. s15 00:01:34.240 --> 00:01:40.159 But what if you take some of those tokens and they're not just executing, they're doing some other job. s16 00:01:40.159 --> 00:01:46.000 So, for example, maybe you take some of your tokens and they're advising the tokens that are executing. s17 00:01:46.000 --> 00:01:56.799 Or maybe the tokens that are executing try to get something done well and you take some other tokens and you actually grade how well the executor is doing so that it can iterate and try again. s18 00:01:56.799 --> 00:01:58.960 Or maybe you have tokens that are dreaming. s19 00:01:58.960 --> 00:02:06.799 They're reflecting back on the job that other executors have done and writing learnings to memory so that they can do it again. s20 00:02:06.799 --> 00:02:12.959 And what we call each of these if you take some tokens that are executing and some tokens that are doing some other job. s21 00:02:12.959 --> 00:02:15.840 Let's call this a strategy. s22 00:02:16.160 --> 00:02:19.520 So let's go take a look at the first strategy, the advising strategy. s23 00:02:19.520 --> 00:02:22.560 Here we're splitting up an executor and an adviser. s24 00:02:22.560 --> 00:02:26.959 The executor obviously executes, but crucially they can call out to an adviser for advice. s25 00:02:26.959 --> 00:02:31.440 And then they can take this advice and figure out if they're doing the next step correctly. s26 00:02:31.760 --> 00:02:33.280 This is really helpful in use cases. s27 00:02:33.280 --> 00:02:41.840 For example, if you're building a sales agent, in an ideal world, you'd have that sales agent be able to actually help the sales rep flag when a follow-up is overdue or deal is stalling. s28 00:02:41.840 --> 00:02:49.360 In this construct, having an adviser to be able to kind of make sure that all the different pieces are actually working is really helpful. s29 00:02:49.360 --> 00:02:51.360 So another example is grading. s30 00:02:51.360 --> 00:02:56.480 Let's say you're executing and you kind of know exactly what good really does look like. s31 00:02:56.480 --> 00:03:08.480 You can define this in a rubric and then each time an executor tries to accomplish that outcome, you can have a grader provisioned that grades how well the executor did while looking at that rubric. s32 00:03:08.480 --> 00:03:11.760 And if the executor did a good job, then great, it can be done. s33 00:03:11.760 --> 00:03:17.360 But if it didn't do such a great job, you can iterate again until you get that good outcome. s34 00:03:17.360 --> 00:03:27.519 So an example in practice of when you might want to use this is let's say you have a customer service agent and you're running a store and your customers are writing in and they're saying, "H, I should get a refund for this thing." s35 00:03:27.519 --> 00:03:30.480 And your customer service agent needs to be able to respond. s36 00:03:30.480 --> 00:03:36.000 You probably have some like pretty specific criteria on when you would give somebody a refund. s37 00:03:36.000 --> 00:03:39.920 And so what you can do is define a rubric that uses that criteria. s38 00:03:39.920 --> 00:03:49.040 You can have a grader that goes and looks at the work that the customer service agent is doing and decide is it getting it right and is it coming to the right outcome. s39 00:03:49.040 --> 00:03:51.440 And the last strategy we have is dreaming. s40 00:03:51.440 --> 00:03:55.680 So in dreaming there's an executor who naturally executes and then there's a dreamer. s41 00:03:55.680 --> 00:04:03.599 The dreamer is actually able to inspect the work and the transcripts of the executor and then it takes any of the findings that it has and it writes them to memory. s42 00:04:03.599 --> 00:04:06.560 This memory is repicked up by the executor for the next round. s43 00:04:06.560 --> 00:04:09.280 So ideally would have improved. s44 00:04:09.280 --> 00:04:12.080 A great use case for this is if you're building a recruiting agent. s45 00:04:12.080 --> 00:04:19.040 Now recruiting requires a lot of interaction with feedback on whether or not a candidate does or doesn't make sense and if it's a good fit between both parties. s46 00:04:19.040 --> 00:04:28.800 And so by taking all this type of data, if you build a dreaming type of strategy on this agent, it's actually able to kind of sharpen the next round so that it's more and more increasingly useful. s47 00:04:28.800 --> 00:04:31.919 So let's make this concrete with some experiments. s48 00:04:31.919 --> 00:04:37.520 So what we did was we created a bench of a bunch of tasks related to financial an analysis. s49 00:04:37.520 --> 00:04:43.840 And what we were doing with each of these tasks is trying to replicate in the real world a expert human financial analyst. s50 00:04:43.840 --> 00:04:46.560 How well would they do on each of these various tasks? s51 00:04:46.560 --> 00:04:50.000 And so what we did was we start with a control that's just executing. s52 00:04:50.000 --> 00:04:53.919 Let's try each of these tasks and we'll eval them when we're literally just executing. s53 00:04:53.919 --> 00:04:59.280 But then we can experiment with each of our strategies and see how well we perform. s54 00:05:00.400 --> 00:05:02.400 So, we start with a super basic experiment. s55 00:05:02.400 --> 00:05:03.759 Let's just oneshot it. s56 00:05:03.759 --> 00:05:10.240 Let's take each of our strategies and we'll go and just make an attempt to accomplish these tasks and we'll see how accurate we are. s57 00:05:10.240 --> 00:05:13.759 And so, you can see here with executing um it didn't do so well. s58 00:05:13.759 --> 00:05:21.120 15% accuracy, but because it was just a oneshot, the strategy got to choose how many tokens it would actually spend on its own. s59 00:05:21.120 --> 00:05:26.639 And so, you can actually see that execute decided not to spend that many tokens, only 39,000. s60 00:05:26.639 --> 00:05:33.280 And as we go into our larger strategies, our more complex strategies, we did choose to spend more tokens, but we did a better job. s61 00:05:33.280 --> 00:05:40.000 So, this isn't really telling us much because sure, Drain did really, really well, but it used a whopping 600,000 tokens to get there. s62 00:05:40.000 --> 00:05:41.120 That's right. s63 00:05:41.120 --> 00:05:47.840 So, in order to actually figure out if varying the jobs produces any alpha, what we need to do is hold the budget constant. s64 00:05:47.840 --> 00:05:53.440 And to do this, we're going to take Dreaming's budget, that 600,000 or so, as the maximum budget that is fixed across the board. s65 00:05:53.440 --> 00:05:58.479 And we give every single strategy this budget in order to analyze how well it's performing. s66 00:05:58.479 --> 00:06:04.240 And as expected again, if you give a lot of strategies more budget, you are going to see performance increase across the board. s67 00:06:04.240 --> 00:06:07.440 So execute went from 0.15 to 76. s68 00:06:07.440 --> 00:06:10.960 Advise and grade went from the 60s to closer to the 90s. s69 00:06:10.960 --> 00:06:17.199 And that's again expected given the fact that if you give things more test time compute, they should generally perform better. s70 00:06:17.199 --> 00:06:26.639 But if that was the only thing that mattered, we should actually expect to see execute, advise, grade, dream actually all be at the exact same level given the exact same token budget. s71 00:06:26.639 --> 00:06:32.400 But what we're actually seeing is that there is an alpha or there is a difference and therefore an alpha for us to exploit. s72 00:06:32.400 --> 00:06:38.720 If you look at execute at this exact same budget level, it gets to 76 but advise is at 89. s73 00:06:38.720 --> 00:06:44.080 So while a minimal, it does exist and so there is alpha for us to take a look at. s74 00:06:44.479 --> 00:06:48.400 Now, we decided to take a look at this analysis from a completely different lens. s75 00:06:48.400 --> 00:06:54.560 And as Kayla mentioned, you know, we're doing this bench for a very complex set of financial tasks in the real world. s76 00:06:54.560 --> 00:06:59.919 And we wanted to analyze the usage of agents with actual experts. s77 00:06:59.919 --> 00:07:11.360 So, if we look at a financial analyst expert, right, the kind of task that they need to do with an agent is that they're giving it something very concrete like let's say make a P&L and then they're getting the result back. s78 00:07:11.360 --> 00:07:28.639 Now if that result is 80% accurate on a bench that sounds great but in reality what that means for that expert is they have to go back and recomputee that P&L themselves or alter or alternatively send it through another run and that's because in this kind of domain for this kind of task if you're not 100% accurate s79 00:07:28.639 --> 00:07:31.199 it's actually not useful. s80 00:07:31.199 --> 00:07:36.400 You cannot make up an income number or you can't make up a cost number right you have to make sure that it's 100% accurate. s81 00:07:36.400 --> 00:07:43.280 So with this lens of the real world consequence associated with this domain, we needed to recomputee our experiments and score them a bit differently. s82 00:07:43.280 --> 00:07:50.080 Crucially, we needed to make sure that our experiment had this kind of construct where if it was scored perfectly, we'd actually give it a pass. s83 00:07:50.080 --> 00:07:56.479 And if it scored anything less than 100% on that kind of task, we would actually mark it as a failure. s84 00:07:56.560 --> 00:08:04.319 So let's look at a different cut of our data from our experiments with this lens where we're looking for this perfect run 100% accuracy pass. s85 00:08:04.319 --> 00:08:09.840 And let's look at what percent of the time each of these strategies was able to achieve a pass. s86 00:08:09.840 --> 00:08:18.319 Um so we we've got executes um down at 42% and we've got our more complex strategies doing a bit better up to 75% accuracy. s87 00:08:18.319 --> 00:08:27.440 Um and again this doesn't necessarily tell us a ton because um you know each of these strategies might um choose to use different budgets over time, right? s88 00:08:27.440 --> 00:08:34.880 So what we did here was we fixed the budget and we said within a fixed budget, how well do each of these strategies perform? s89 00:08:35.599 --> 00:08:46.640 And so what really matters to us actually is if you're trying to get this perfect answer and you're in the real world, you're running a business, what matters to you is the cost to you to get to that perfect answer. s90 00:08:46.640 --> 00:08:50.640 And so one way we can think about this is we had our execute strategy for example. s91 00:08:50.640 --> 00:08:54.959 The execute strategy around 40% of the time will give you that perfect answer. s92 00:08:54.959 --> 00:09:01.760 So on average, you can expect to have to run it three times and you should hopefully sometime in those three runs get a perfect answer. s93 00:09:01.760 --> 00:09:07.760 And as we talked about earlier, we fixed our budget to that highest token budget strategy, which was 600,000 tokens. s94 00:09:07.760 --> 00:09:13.200 So if you spend 600,000 tokens in each individual run, you have to run approximately three times. s95 00:09:13.200 --> 00:09:21.279 You can expect on average to have to spend 1.8 million tokens with the execution strategy to get to your perfect answer. s96 00:09:21.279 --> 00:09:31.360 And so if we take this analysis and run it across the board against all these strategies, this is actually the true cost it took in this domain for that agent to be useful for that strategy. s97 00:09:31.360 --> 00:09:38.959 So as Caitlyn mentioned for execute, which is our baseline, this is going to be 1.8 million true total token cost for you. s98 00:09:38.959 --> 00:09:42.000 But advise, grade, and dream are showing us a bit of difference. s99 00:09:42.000 --> 00:09:50.720 Crucially, advise and grade are actually quite token efficient when you think about the actual usage of the end output of each of these agents. s100 00:09:51.200 --> 00:09:53.279 So what does this mean for you as a business? s101 00:09:53.279 --> 00:09:59.120 Well, it actually really depends on what kind of thing you want to optimize for and it's going to vary, right? s102 00:09:59.120 --> 00:10:04.080 There's going to be businesses who say, "Actually, for me, the most important thing is to be really token efficient. s103 00:10:04.080 --> 00:10:10.399 In that case, you should probably pick the advised type of strategy in order to solve for that particular domain in which you want to optimize that." s104 00:10:10.399 --> 00:10:22.399 There's going to be other areas or other businesses where you're going to say, I'm not going to care so much about token efficiency because what I really care about is reliability of that answer and so I need to maximize the percentage of runs in which I get that perfect answer. s105 00:10:22.399 --> 00:10:24.800 In which case, you would actually pick completely different strategies. s106 00:10:24.800 --> 00:10:28.399 You probably lean towards grade or dream. s107 00:10:28.880 --> 00:10:34.160 So if you take away one thing, the thing we want everyone to think about is this idea that tokens are not fungeible. s108 00:10:34.160 --> 00:10:35.920 You can use your tokens to execute. s109 00:10:35.920 --> 00:10:39.440 You can brute force your way through your task and you can throw more budget at it. s110 00:10:39.440 --> 00:10:52.079 But if you get really smart about having your tokens do these different jobs and try these different strategies, you're very very likely to be able to get a better outcome for the task at hand within a fixed budget. s111 00:10:52.079 --> 00:10:56.079 And so let's talk a little bit about how we actually build strategies and how we bring this to life. s112 00:10:56.079 --> 00:11:00.800 Um so we've done a lot of work to create a really excellent harness for individual agents. s113 00:11:00.800 --> 00:11:12.240 Um and if you see this uh picture at the bottom here, this is actually the architecture that we've used for cloud managed agents um which is our Aentic solution that we give to you within the cloud platform. s114 00:11:12.240 --> 00:11:24.640 And what we do on top of this is we start to get into the meta harness level like the multi- aent orchestration and execution level where this strategy can go and be um coordinated between our executor and our adviser s115 00:11:24.640 --> 00:11:27.519 or the other agents within our strategy. s116 00:11:27.519 --> 00:11:34.399 And some of these um like dreaming and outcomes we actually give to you out of the box within cloud manage agents. s117 00:11:34.399 --> 00:11:45.839 So with those set of primitives it's actually relatively trivial for us to construct this kind of you know architecture where we're able to combine these different types of strategies and figure out how to they should work together. s118 00:11:45.839 --> 00:11:55.360 So for example it's relatively trivial for us to say okay now with this I can take a task and I should be able to execute it but also allow it to advise and fable is back online. s119 00:11:55.360 --> 00:11:59.519 So we could actually say Fable is the one that's actually advising uh the executor. s120 00:11:59.519 --> 00:12:05.760 And then I can take all these results and say send them to a greater so that I can make sure that this is verifying in a loop that makes sense. s121 00:12:05.760 --> 00:12:07.360 And if it passes, that's awesome. s122 00:12:07.360 --> 00:12:12.959 I want to send all of that stuff to Dreaming and make sure that my next run is better than ever. s123 00:12:12.959 --> 00:12:14.720 And of course, you don't have to stop there, right? s124 00:12:14.720 --> 00:12:24.240 If the right primitives are there and the right coordination is there, then you can actually construct really complex setups that fit for all the different types of dynamic problems that you have. s125 00:12:24.240 --> 00:12:28.639 You can invent these kinds of large-scale architectures, again, very triv. s126 00:12:28.639 --> 00:12:36.160 And you could also invent completely new jobs, not just the ones of the pieces that Caitlyn and I have presented in this conversation. s127 00:12:36.240 --> 00:12:45.839 So, a big goal that we have over time is to get our models better and better and our platform better and better at dynamically constructing these strategies for you as you're doing work. s128 00:12:45.839 --> 00:12:59.920 But in the meantime, as we're working our way there, we would love for you to continue to think about this idea that you should give your tokens jobs and you should use different novel strategies by combining these primitives in order to get the outcomes that you want for your tasks. s129 00:12:59.920 --> 00:13:02.560 Thanks for joining us.