WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=maitMe9HCvw NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/maitMe9HCvw.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.720 --> 00:00:01.640 Hi everyone. s2 00:00:01.640 --> 00:00:10.960 My name is Nuha, I'm a senior researcher at Cohere and associate industry member at Mila and an adjunct PI at the Allen Institute to begin. s3 00:00:10.960 --> 00:00:18.320 So excited to tell you today about how to optimize your agent team and get frontier coding at 70% lower the cost. s4 00:00:18.320 --> 00:00:21.920 This is a work that I did at Cohere. s5 00:00:21.920 --> 00:00:26.800 All right, so traditionally there are two ways to make an agent stronger at inference time. s6 00:00:26.800 --> 00:00:32.880 First, you can let it think longer, meaning you scale compute sequentially with longer chain of reasoning. s7 00:00:32.880 --> 00:00:34.840 And second, you can sample more. s8 00:00:34.840 --> 00:00:39.320 You run many independent attempts in parallel and then you aggregate them. s9 00:00:39.320 --> 00:00:45.850 But as agents take on longer and longer task, they can navigate large repo, they can edit the code, they s10 00:00:45.850 --> 00:00:45.880 [snorts] s11 00:00:45.880 --> 00:00:48.240 can debug their own work over hours. s12 00:00:48.240 --> 00:00:52.520 So what can happen here is that the context window may blow up. s13 00:00:52.520 --> 00:00:59.360 And so we can see here that, you know, the context window eventually fills up and the longer it runs, the longer you wait. s14 00:00:59.360 --> 00:01:05.519 And sampling more doesn't either fix it because each attempt here still has to do the whole task on its own. s15 00:01:05.519 --> 00:01:05.920 Right? s16 00:01:05.920 --> 00:01:09.080 So what if we split the work? s17 00:01:09.080 --> 00:01:20.040 So here in this case the key the lead keeps the task and hands out subtask and each worker starts with a fresh context so it can read files, it can run tests, it can dig through logs, s18 00:01:20.040 --> 00:01:23.600 etc. etc. and then it can send back only a compact result. s19 00:01:23.600 --> 00:01:33.960 So here the lead keeps the full picture and it can finish faster if the task is decomposable and agents can work on independent pieces. s20 00:01:33.960 --> 00:01:38.400 And so here we're talking about a new axe for scaling inference time compute. s21 00:01:38.400 --> 00:01:39.920 You work as a team. s22 00:01:39.920 --> 00:01:45.360 You give a task to a team of coding agent and watch them do the task for you. s23 00:01:45.360 --> 00:01:48.680 But then scaling the team gets expensive very very quickly. s24 00:01:48.680 --> 00:01:53.080 And so as you add more agents, your token budget can blow up. s25 00:01:53.080 --> 00:01:57.680 And it can actually and it raises a harder question actually. s26 00:01:57.680 --> 00:02:00.572 So first, how does coordination actually happen? s27 00:02:00.572 --> 00:02:08.732 How should we organize the team so that agent can work together etc. And so, you know, this is the question that we set out to answer. s28 00:02:08.732 --> 00:02:12.572 How should a team be orchestrated to be both effective and efficient? s29 00:02:12.572 --> 00:02:18.212 By effective, I mean it actually solves more of the task so we get higher reward. s30 00:02:18.212 --> 00:02:24.092 And by efficient, I mean it doesn't burn through your budget getting there. s31 00:02:24.252 --> 00:02:24.652 All right. s32 00:02:24.652 --> 00:02:36.252 So, since these models now are extremely capable, I was wondering what if we throw an extremely challenging coding task as a society of agent and see what they can do. s33 00:02:36.252 --> 00:02:48.092 And so, we would expect them to automatically form clean groups, split up the work neatly, and build it together, communicate nicely, and coordinate, etc. etc. But that's not what not quite what happened actually. s34 00:02:48.092 --> 00:02:52.252 In reality, what happened the agent preferred to solve the task on their own. s35 00:02:52.252 --> 00:02:55.132 So, each went off and build their own version. s36 00:02:55.132 --> 00:03:03.372 Even when we asked them to communicate, they mostly discussed things, sometimes that didn't matter, and sometimes they kept also stepping on each other's toes. s37 00:03:03.372 --> 00:03:16.972 They also preferred to communicate back to the lead, etc. etc. So, here we were wondering, okay, how should we design a team so that, you know, we can actually get the best out of it in terms of reward and in terms of cost. s38 00:03:16.972 --> 00:03:18.612 And hopefully in terms of latency. s39 00:03:18.612 --> 00:03:23.492 And so, you know, we set out to figure out what a multi-agent harness should actually look like. s40 00:03:23.492 --> 00:03:25.492 And like we were wondering about the team design. s41 00:03:25.492 --> 00:03:32.572 So, there are many design choices and each one changes how well the team does and how much it cost. s42 00:03:32.572 --> 00:03:34.492 So, first the organization, right? s43 00:03:34.492 --> 00:03:36.252 How do you structure the team? s44 00:03:36.252 --> 00:03:37.052 Who leads? s45 00:03:37.052 --> 00:03:43.132 And at which stage of the work, planning, implementation, or verification do actually workers come in? s46 00:03:43.132 --> 00:03:45.732 Um second thing uh is communication. s47 00:03:45.732 --> 00:03:49.732 Should agent talk to each other directly or only report to the lead? s48 00:03:49.732 --> 00:03:51.892 Third is the team size. s49 00:03:51.892 --> 00:03:53.652 How many workers is worth it? s50 00:03:53.652 --> 00:03:59.132 Like should it be 1 2 16 100 etc. And fourth, the models, right? s51 00:03:59.132 --> 00:04:00.092 Which model should play? s52 00:04:00.092 --> 00:04:06.894 Which role do we need, you know, a strong model everywhere, or, you know, can cheaper ones do the same work, right? s53 00:04:06.894 --> 00:04:11.734 And finally here that something that we cared about is the state of the software engineering, right? s54 00:04:11.734 --> 00:04:24.494 Should should, you know, we should allocate more workers towards planning, towards implementation, towards verification, etc. And so, we actually framed team design as a compute allocation problem. s55 00:04:24.494 --> 00:04:29.734 So, what we cared about is reward per added dollar over the solo agent, right? s56 00:04:29.734 --> 00:04:35.494 So, starting from a solo agent, we ask where does the next dollar of inference buy the most reward? s57 00:04:35.494 --> 00:04:36.614 This is what we cared about. s58 00:04:36.614 --> 00:04:42.494 And so, here we actually define a metric called efficiency, and it's actually a function four things. s59 00:04:42.494 --> 00:04:54.774 First, the stage, it's the the stage at which the worker is at, whether it's implementation, planning, or verification, or all of them together, we call it We also define that as a function of the worker model. s60 00:04:54.774 --> 00:04:56.374 Is it like a very capable model? s61 00:04:56.374 --> 00:05:00.414 Either is it a very, you know, you know, less capable model? s62 00:05:00.414 --> 00:05:05.294 The number of workers, the lead model, etc. Um, and all right. s63 00:05:05.294 --> 00:05:12.654 So, basically, if we To answer this question, we actually design our multi-agent harness as a top-down hierarchy. s64 00:05:12.654 --> 00:05:17.854 We build deliberately a minimal harness, a mini suite agent, and it's a hierarchical, as I mentioned. s65 00:05:17.854 --> 00:05:24.334 Here, we have a lead, and based on the task, it decides dynamically how big that team should be and um, you know, who should do what. s66 00:05:24.334 --> 00:05:28.054 And so, here we made sure that every part of the harness is exposed. s67 00:05:28.054 --> 00:05:30.934 So, we could change one choice at a time. s68 00:05:30.934 --> 00:05:36.294 We keep everything else fixed and see how each one moves reward and cost. s69 00:05:36.294 --> 00:05:36.894 Right? s70 00:05:36.894 --> 00:05:40.574 And so, here more closely, how does a team of coding agent do work? s71 00:05:40.574 --> 00:05:45.294 You can see here every agent works through a bash tool in one shared workspace. s72 00:05:45.294 --> 00:05:51.254 So, basically, the lead receives the task and hands out work with a tool we call delegate agent. s73 00:05:51.254 --> 00:05:54.814 And then the worker sees only its subtask and reports back to the lead. s74 00:05:54.814 --> 00:05:58.694 And it can also communicate directly to the other worker. s75 00:05:58.694 --> 00:06:13.095 And so, here basically, the lead can decide based on the task, oh, actually, I need to just, you know, power an implementation worker or the verification worker, etc. And so, as I mentioned also, the lead decides how many workers to sprout, how to combine, also consolidate the results. s76 00:06:13.095 --> 00:06:22.524 It also has the capability to resume a worker, to kill a worker, and to tap in and, for example, supervise and tell the worker that you have made s77 00:06:22.524 --> 00:06:22.655 [snorts] s78 00:06:22.655 --> 00:06:31.255 some errors here, fix it, etc. And so, yeah, and you may be wondering, "Okay, what is the lead doing while the other workers are working?" s79 00:06:31.255 --> 00:06:33.615 We actually have two variations. s80 00:06:33.615 --> 00:06:37.255 One is what we call the blocking lead, the other one is asynchronous lead. s81 00:06:37.255 --> 00:06:42.495 So, the blocking lead actually hands out subtask and waits until every worker reports back. s82 00:06:42.495 --> 00:06:45.055 It doesn't do anything, you know, in between. s83 00:06:45.055 --> 00:06:50.935 Uh whereas an async lead, it keeps actually working while, you know, the results keep coming from the other workers. s84 00:06:50.935 --> 00:06:56.935 It can supervise what other workers are doing, it can send them feedback, etc., etc. And so, how did we test this? s85 00:06:56.935 --> 00:07:03.335 We test this on a benchmark called program bench, which is basically a reverse engineering task. s86 00:07:03.335 --> 00:07:11.455 So, basically, the agent starts from an empty workspace with only the original program executable, which it can run but not read. s87 00:07:11.455 --> 00:07:13.735 And it also receives some documentation. s88 00:07:13.735 --> 00:07:23.695 So, here the goal is to basically probe the program to discover its interface, to discover the edge cases, the behavior, and then write a new implementation that behaves in the same way. s89 00:07:23.695 --> 00:07:26.375 And then, the task here were extremely challenging. s90 00:07:26.375 --> 00:07:40.135 They range from SQLite, sorry, uh for example, the PHP interpreter, and for example, you know, on the leaderboard on the official website, on the task FFmpeg, the best model only passes 10% of the test. s91 00:07:40.135 --> 00:07:43.855 So, this is also done by, you know, frontier model. s92 00:07:43.855 --> 00:07:49.495 So, it was quite challenging, and it was a good test data for for our, you know, investigation. s93 00:07:49.495 --> 00:07:53.735 And so, the question here is that does a team beat a solo agent, yes or no? s94 00:07:53.735 --> 00:07:54.855 The answer is yes. s95 00:07:54.855 --> 00:07:59.815 It actually can beat a single agent, but at four to seven times the cost. s96 00:07:59.815 --> 00:08:14.137 And we can actually see here in the figure that you know, with a minimal prompt, uh a team with a blocking lead, actually, beats the solo agent by almost 10 points, but it costs about uh costs about 4 to 7 s97 00:08:14.137 --> 00:08:15.057 times as much. s98 00:08:15.057 --> 00:08:23.857 Now, you can see that the blocking actually performed better than the asynchronous, but costs higher, and it actually takes more time to complete. s99 00:08:23.857 --> 00:08:30.417 So, what we noticed here is that we noticed that the async lead doesn't know when to wait, when its workers are still exploring. s100 00:08:30.417 --> 00:08:39.417 It starts building on its own guesses before the workers finding come back, and then when the reports finally arrive, it has already moved on. s101 00:08:39.417 --> 00:08:44.297 And so, either it ignores them, or it has to go back and redo the work. s102 00:08:44.297 --> 00:08:48.737 And sometimes it just decides the task is done while the workers are still actually running. s103 00:08:48.737 --> 00:08:50.457 So, basically, it decides, "Okay, done. s104 00:08:50.457 --> 00:08:51.377 It's time to submit the work." s105 00:08:51.377 --> 00:08:54.297 And so, the work of these workers is simply lost. s106 00:08:54.297 --> 00:09:05.257 But then, you know, this problem never occurs with a blocking lead because it will wait until the workers report back, and then it actually consolidates the work and do the submission. s107 00:09:05.417 --> 00:09:05.737 All right. s108 00:09:05.737 --> 00:09:08.737 So, the next thing I want to emphasize on is the prompt minimal. s109 00:09:08.737 --> 00:09:15.457 So, I I mentioned, okay, in the earlier results, I we used the minimal prompt, which is an earlier prompt, and I can't emphasize on this enough. s110 00:09:15.457 --> 00:09:17.697 This is how our agent operates. s111 00:09:17.697 --> 00:09:22.457 We give them prompt and instructions, and essentially, they do what we ask them. s112 00:09:22.457 --> 00:09:25.937 And so, the first one, we tried to go very, very minimal. s113 00:09:25.937 --> 00:09:37.897 We just told the lead about the task, you know, that it should create you know, sub workers and agents, and it should inform them about the task, and should tell them about the tools. s114 00:09:37.897 --> 00:09:39.737 We gave also the lead tools. s115 00:09:39.737 --> 00:09:45.377 We wanted to see why we started minimal because we wanted to see how the agent behave on their own. s116 00:09:45.377 --> 00:09:50.057 Do they need a lot of guidance, or is it is a minimal prompt enough, et cetera. s117 00:09:50.057 --> 00:09:51.377 And so, this is very minimal. s118 00:09:51.377 --> 00:09:58.977 We also were wondering, okay, what happens if we give it a very heavy prompt, meaning like, you know, what the model should exactly do. s119 00:09:58.977 --> 00:10:01.297 We tell it exactly about different tracks. s120 00:10:01.297 --> 00:10:11.778 Say, follow exactly this plan, or try to, you know, delegate agents, and you know, create agent throughout the whole process of the implementation from exploring and planning to implementation, s121 00:10:11.778 --> 00:10:18.698 create also adversarial workers, verify the task, etc. etc. And actually that helped a lot. s122 00:10:18.698 --> 00:10:32.298 We can see here, you know, compared to the minimal prompt, you know, we can see that the solo agent alone jumps almost six points just from better instructions and telling them, "Okay, don't step on each other's toes. s123 00:10:32.298 --> 00:10:42.538 I don't, you know, there is this folder, write in this folder, etc." So, lots lots of guidance and lots of prompt tuning, which is not ideal cuz we spent too much time there. s124 00:10:42.538 --> 00:10:46.258 And now these agents are supposed to be smart enough to figure out how to do it. s125 00:10:46.258 --> 00:10:55.778 And so, we can actually see that the teams, both, you know, async and blocking, both go up and we can see that the async lead now catches up with the blocking one, right? s126 00:10:55.778 --> 00:10:58.978 So, once we tell the lead how to coordinate, it knows when to wait. s127 00:10:58.978 --> 00:11:02.898 So, this actually really helped address the problem that I talked about earlier. s128 00:11:02.898 --> 00:11:08.018 But again, you know, we have a team that works better than the solo, but it's really really expensive. s129 00:11:08.018 --> 00:11:10.538 It costs six to seven times as much as the solo agent. s130 00:11:10.538 --> 00:11:16.018 So, we were wondering, now that a team beats a solo agent, but it costs more. s131 00:11:16.018 --> 00:11:23.218 So, if a team is a new way to spend inference time compute, where should the next dollar go, basically? s132 00:11:23.218 --> 00:11:32.618 Can we actually optimize our agent setup so that actually we get very good reward but at a very, you know, much cheaper cost. s133 00:11:32.618 --> 00:11:39.818 So, to answer this question, actually we took the same lead and gave it workers at only one stage of the software engineering process. s134 00:11:39.818 --> 00:11:47.698 So, the first setup that we considered here, the workers are only responsible for the planning and exploration, right? s135 00:11:47.698 --> 00:12:02.618 So, basically here the lead will give them the task and then each of the workers will go and try to prompt the binary to discover, okay, what kind of plan I need to do, and then it will report the plan to the lead and the lead will take care of the implementation verification. s136 00:12:02.618 --> 00:12:12.899 The second one is, you know, workers are only responsible for implementation and so the lead is the one responsible for coming up with a plan, distributing that, splitting the work between workers. s137 00:12:12.899 --> 00:12:18.819 they will do the implementation, report back with the code actually to the to the lead. s138 00:12:18.819 --> 00:12:23.219 And so, basically and then the lead does the verification on its own. s139 00:12:23.219 --> 00:12:30.019 Now, the third actually set up that we had, the lead is responsible for both the planning and the implementation. s140 00:12:30.019 --> 00:12:41.059 And then here the workers are responsible to do the verification, which is the final stage and then they will report back to the lead and then the lead will decide whether to submit or not. s141 00:12:41.059 --> 00:12:50.619 And so, Okay, we wanted to see where exactly extra agent will pay off the most and here are the conclusion of our results like extra agent pay off most s142 00:12:50.619 --> 00:12:52.379 in program bench in planning. s143 00:12:52.379 --> 00:12:57.339 So, here you can see okay, going from solo with minimal prompt to solo with PIV. s144 00:12:57.339 --> 00:12:59.019 PIV is the heavy prompt. s145 00:12:59.019 --> 00:13:06.619 Actually, we get like a jump already and then we go with a team of heavy prompting, we get extra, you know, increase and gain in performance. s146 00:13:06.619 --> 00:13:10.699 But then now when we allocate more agent towards the planning, right? s147 00:13:10.699 --> 00:13:15.979 We can see that we get almost the same performance as the PIV but at half the cost. s148 00:13:15.979 --> 00:13:27.859 So, Actually, the conclusion here is that, you know, the extra agent pay off the most in planning and um at at at the task that bottlenecks actually uh the the s149 00:13:27.859 --> 00:13:46.019 the task, which is planning in this case because you need to figure out how exactly what what's the binary behavior and come up with an detailed plan on what should you implement first, etc. And so, here yeah, I Actually, we computed even, you know, the efficiency average that I discussed earlier and we can see that, you know, s150 00:13:46.019 --> 00:13:48.379 it dominates the PIV actually on the frontier. s151 00:13:48.379 --> 00:13:56.979 So, each planning dollar buys almost 1.6 more reward than any other stage that we have here, whether it's implementation or verification. s152 00:13:56.979 --> 00:14:07.739 And actually, we we we noticed that, you know, you know, doubling down on the number of agents in the implementation stage in the implementation or verification stage doesn't actually really help. s153 00:14:07.739 --> 00:14:13.739 We can see that it just increased the cost but we are actually getting the same performance as the solo, so it's not worth it. s154 00:14:13.739 --> 00:14:20.381 And so, again, like program Bench only exposes a black box executable, so discovery here is the bottleneck. s155 00:14:20.381 --> 00:14:25.901 So, planners paralyze this probing and it brings and they bring lots of benefits, right? s156 00:14:25.901 --> 00:14:26.941 And so, yes. s157 00:14:26.941 --> 00:14:33.661 So, this is to explain why planning workers pay off and we were wondering why not, for example, verification or implementation. s158 00:14:33.661 --> 00:14:37.861 And, you know, as I mentioned, Program Bench will give you this binary and its documentation. s159 00:14:37.861 --> 00:14:42.101 So, before you write anything, you have to work out what the program actually does. s160 00:14:42.101 --> 00:14:44.981 So, discovery is the real bottleneck here. s161 00:14:44.981 --> 00:14:49.021 And you can also think about this in terms of context, right? s162 00:14:49.021 --> 00:14:57.181 So, we have one agent that's exploring, you know, alone files uh in its own context with probing and also still misses edge cases, right? s163 00:14:57.181 --> 00:15:06.181 And on the other hand, you know, we have several workers each explore different part of the program in a fresh context and each one hands back a compact report to the lead. s164 00:15:06.181 --> 00:15:10.741 So, basically here the lead, you know, learns more than it could do alone. s165 00:15:10.741 --> 00:15:17.261 And so, now, you know, this is also to explain better this phenomena and you can and we actually looked at our data. s166 00:15:17.261 --> 00:15:25.261 And we came up with these nice figure based on what we saw and we specifically looked at uh when the lead delegates, right? s167 00:15:25.261 --> 00:15:30.621 So, here we have planning, we have implementation and verification and we have all the stages as well. s168 00:15:30.621 --> 00:15:37.221 And so, every row that we can see here in the figure is one run uh and actually time goes left to right, right? s169 00:15:37.221 --> 00:15:46.181 So, the yellow is planning, the blue is implementation, the green is verification and every dot here that you see in the figure is the lead sending work to a worker. s170 00:15:46.181 --> 00:15:55.621 And so, with planning guidance, all the delegation happens in one burst at the very, very first of, you know, the the work before any code is written. s171 00:15:55.621 --> 00:16:01.021 So, basically here the the workers go explore and then they report back and then the lead builds. s172 00:16:01.021 --> 00:16:16.541 With a full um PIV guidance, where, you know, um there is no um specific guidance on where exactly the the the number of workers should be allocated, we get the same early burst, but then, if you You here, s173 00:16:16.541 --> 00:16:18.942 actually the lead keeps delegating all the way through. s174 00:16:18.942 --> 00:16:22.262 Another 830 calls and they buy nothing actually. s175 00:16:22.262 --> 00:16:24.822 They just increase the cost as we saw. s176 00:16:24.822 --> 00:16:29.662 It was around $500, but it didn't actually improve in performance. s177 00:16:29.662 --> 00:16:38.622 So, it's the same reward for planning we can see, but it's 2.4 times more reward per call that we we actually measured. s178 00:16:38.622 --> 00:16:46.662 And so, what pays off here is is one early burst of discovery and not delegating all the way through. s179 00:16:46.662 --> 00:16:46.942 Right? s180 00:16:46.942 --> 00:16:49.342 So, why implement workers fall short? s181 00:16:49.342 --> 00:16:52.822 Here, first of all, as I mentioned, the plan was the bottleneck. s182 00:16:52.822 --> 00:16:55.622 And so, if you don't do it right, it's not going to work out. s183 00:16:55.622 --> 00:16:57.662 And so, here we have only the lead that explored. s184 00:16:57.662 --> 00:17:03.222 It had limited capacity to discover all, you know, the the edge cases and the full binary. s185 00:17:03.222 --> 00:17:08.542 And so, we had a weak foundation and then basically each worker got like a narrow slice. s186 00:17:08.542 --> 00:17:18.182 It doesn't have visibility over the entire task and basically they were already doing errors by implementing each feature on their own. s187 00:17:18.182 --> 00:17:19.422 That's one problem. s188 00:17:19.422 --> 00:17:22.222 And on top of that, they were building on a weak foundation. s189 00:17:22.222 --> 00:17:27.742 And also, we noticed that so, here you might ask me that, you know, why not give workers more context? s190 00:17:27.742 --> 00:17:28.102 We did. s191 00:17:28.102 --> 00:17:35.102 But, we noticed that when you they have more context, actually they didn't respect the boundaries of their specific task. s192 00:17:35.102 --> 00:17:39.862 They get excited and then they want to build extra features that nobody asked them for it. s193 00:17:39.862 --> 00:17:51.862 So, they just step on each other's toes, etc. The other issue that we noticed is that, you know, they are supposed to communicate together to figure out who's doing what to avoid conflicts and duplicated work, but they didn't do that. s194 00:17:51.862 --> 00:17:58.302 They actually they worked alone and 84 of messages go to the lead directly and nobody sees who's doing what. s195 00:17:58.302 --> 00:18:09.702 And sometimes here what happens is the lead gets overwhelmed and it doesn't know doesn't communicate well where the problem is to the other workers, etc. So, it can be actually quite challenging. s196 00:18:09.702 --> 00:18:16.262 And so, what happens here is that when, you know, implementation is not fully correct, you know, the the the lead pays for the gap. s197 00:18:16.262 --> 00:18:22.263 It actually tries to, you know, make up for that duplicated work and does lots of stitching. s198 00:18:22.263 --> 00:18:31.623 And so, here what happens is that the latency goes up, and also the reward goes down, and you know, it's also expensive. s199 00:18:31.623 --> 00:18:39.423 Uh Okay, so basically so far I've been talking about program bench, but we also run experiments also on commit zero. s200 00:18:39.423 --> 00:18:49.103 And what I want to say here is that the result like are are different, and the hierarchy and the the allocation of the team is different according to the task. s201 00:18:49.103 --> 00:18:52.223 Because the bottleneck for program bench was planning. s202 00:18:52.223 --> 00:18:58.183 And so, allocating more agents there really really helped, but for commit zero it was not the case. s203 00:18:58.183 --> 00:19:02.943 The commit zero the bottleneck was not planning, because basically you're given the skeleton. s204 00:19:02.943 --> 00:19:07.183 So, the docs are known and the unit tests are actually known to the workers. s205 00:19:07.183 --> 00:19:09.063 So, the plan is basically given to you. s206 00:19:09.063 --> 00:19:15.343 And the the bottleneck was for the you know, for the reward to go up was the implementation. s207 00:19:15.343 --> 00:19:17.023 You need to get the implementation right. s208 00:19:17.023 --> 00:19:24.903 And so, here when we doubled the number of agent on the implementation stage, this is where we saw the biggest improvement. s209 00:19:24.903 --> 00:19:26.703 I'm not going to present results on this. s210 00:19:26.703 --> 00:19:29.823 Please check out the paper for for for these results. s211 00:19:29.823 --> 00:19:31.143 You'll find them in the appendix. s212 00:19:31.143 --> 00:19:39.623 So, the conclusion here is that you need to allocate more agents at the software engineering stage that bottleneck your task. s213 00:19:39.623 --> 00:19:43.743 All right, so planning already gets us the same reward at half the cost. s214 00:19:43.743 --> 00:19:47.183 So, we were wondering, can we go even cheaper than this? s215 00:19:47.183 --> 00:19:49.383 Can we cut the cost even further? s216 00:19:49.383 --> 00:20:03.343 And so, here we did some experiments, and we focused for example on three GPT-5.6 models, Luna, Terra, and Sol in a this increasing order of capability and cost. s217 00:20:03.343 --> 00:20:12.743 And so, like the question here is that what would happen if we keep Sol as the lead, like the strongest model, and then use the cheap models as the workers? s218 00:20:12.743 --> 00:20:13.103 Right? s219 00:20:13.103 --> 00:20:15.423 So, remember here we used to have this. s220 00:20:15.423 --> 00:20:20.023 This is all the P here is the all Sol, you know, model. s221 00:20:20.023 --> 00:20:29.185 By the way, here the the the Sol the basic This is the version like of P where the lead dynamically decides on the size of the team and it ended up s222 00:20:29.185 --> 00:20:32.505 on average creating and spawning five agents, right? s223 00:20:32.505 --> 00:20:35.545 And so, basically here, we have this. s224 00:20:35.545 --> 00:20:49.225 We go from PIV, we actually allocate more agent towards P. And basically, when we replace the sole, you know, sub workers with cheap ones like Luna, we actually get almost, s225 00:20:49.225 --> 00:20:52.345 you know, same performance but 60% less the cost. s226 00:20:52.345 --> 00:21:02.905 Like just we lose a little bit on performance, but it's still good because we're actually maximizing reward compared to the solo and then we're we're getting less cost. s227 00:21:02.905 --> 00:21:05.225 And so, you know, now the other way around. s228 00:21:05.225 --> 00:21:07.425 We were wondering like can the lead be cheap, right? s229 00:21:07.425 --> 00:21:09.505 We just even get a cheaper lead. s230 00:21:09.505 --> 00:21:20.545 And so, already here, we can see that okay, you know, going from like a cheap lead like solo, Luna, Terra, sole, we keep increasing in performance and also in cost. s231 00:21:20.545 --> 00:21:25.185 And so, can we have like this homogeneous team like Luna plus Luna, Terra plus Terra. s232 00:21:25.185 --> 00:21:31.025 Already we can see the scaling law the same as the model gets more capable, we get better results. s233 00:21:31.025 --> 00:21:42.065 But then, actually, you know, we noticed that when we have a strong lead sole plus Luna, we can get the same performance as sole plus sole, but you know, at much lower the cost. s234 00:21:42.065 --> 00:21:45.625 But the other way around like cheap workers can't make up for a weak a weak lead. s235 00:21:45.625 --> 00:21:48.865 Basically, the lead sets the ceiling and the workers set the price. s236 00:21:48.865 --> 00:21:55.785 Here, Terra with Luna workers still feels sole working alone 51% versus 57%. s237 00:21:55.785 --> 00:22:02.505 So, essentially here, the lead sets, you know, basically this also matches Meta's Meta reasoning work recently. s238 00:22:02.505 --> 00:22:07.945 Like they just mentioned that a better controller raised what the same workers achieved. s239 00:22:07.945 --> 00:22:08.265 All right. s240 00:22:08.265 --> 00:22:14.145 So, so far the lead decided how many workers to spawn as I mentioned and it did on average five workers. s241 00:22:14.145 --> 00:22:16.745 Now, we were wondering what would happen. s242 00:22:16.745 --> 00:22:20.065 We want to scale ourselves the number of agent at each stage. s243 00:22:20.065 --> 00:22:22.865 So, we will impose the team size. s244 00:22:22.865 --> 00:22:37.666 So, here, uh going to, you know, work with Sol as a lead, Luna as planners because this was the best um reward cost um um uh relation and we go from two to 16 planners, right? s245 00:22:37.666 --> 00:22:41.266 Uh and so, here I can actually look at the cost. s246 00:22:41.266 --> 00:22:54.205 You know, the 16 planner team is 38% cheaper than all Sol planning team and actually 70% um cheaper than workers at every stage here for the PID, which is great s247 00:22:54.205 --> 00:22:54.226 [snorts] s248 00:22:54.226 --> 00:23:03.706 cuz here we and it seems we we you do some budget constraint, we couldn't like scale up the the number of agent further, but it seemed like going up and up and up. s249 00:23:03.706 --> 00:23:09.626 Obviously, there is a ceiling to it, but it's really helping and the good news is that it's not that expensive. s250 00:23:09.626 --> 00:23:17.066 It's not that it's not more expensive than the five Sol team uh all allocated towards planning. s251 00:23:17.066 --> 00:23:28.226 Now, we also uh try to see whether uh scaling up the number the size of the team for the other variants like implementation and verification and it didn't help at all. s252 00:23:28.226 --> 00:23:39.426 Uh we already saw that uh you know, allocating teams um agents toward in these stages is not helping because of, you know, the foundation was weak and other considerations. s253 00:23:39.426 --> 00:23:43.746 And so, the last thing we were actually curious about, how do agents communicate, right? s254 00:23:43.746 --> 00:23:58.106 And so, we gave every agent uh a send message tool to reach any other agent actually in the sandbox and we were curious whether communication uh happens directly or, you know, it happens through the lead and whether communication s255 00:23:58.106 --> 00:24:00.746 will lead to better performance or worse. s256 00:24:00.746 --> 00:24:04.786 And so, basically, the first thing is that are they communicating together? s257 00:24:04.786 --> 00:24:05.506 Not really. s258 00:24:05.506 --> 00:24:08.466 They are actually essentially communi- communicating through the lead. s259 00:24:08.466 --> 00:24:12.506 Like 84% of all messages went from a worker to the lead. s260 00:24:12.506 --> 00:24:16.946 Worker to worker, you can see that actually very little, less than one message per task. s261 00:24:16.946 --> 00:24:21.306 So, even when they can talk to each other, they actually behave like a hierarchy. s262 00:24:21.306 --> 00:24:24.386 Like, do your piece, you report up. s263 00:24:24.386 --> 00:24:31.107 Expect the lead to communicate back to the right sub-agent, etc. And you can also see this pattern consistent across all models. s264 00:24:31.107 --> 00:24:43.187 The lead takes much more talks Sorry, talks much more to workers than they talk to to each other, and this is also true for Cloud Opus 4.7. Also, we did the experiments on Gemini, etc. And so, s265 00:24:43.187 --> 00:24:46.587 also, as I mentioned, we were curious about the reward. s266 00:24:46.587 --> 00:24:51.028 Does it, you know, does extra communication between workers help or not? s267 00:24:51.028 --> 00:24:54.908 In our experiments, we didn't find it helps, and I will explain why. s268 00:24:54.908 --> 00:25:14.867 Um so, first, already you can see, you know, on the left you can see that about half of Luna messages never even get through, like, because they're malformed and the harness reject them, etc. Um and so, we notice also in the middle figure that Sol writes about three to four times as much message text as as Luna. s269 00:25:14.867 --> 00:25:23.867 And under the PIV guidance, stronger teams, actually, we can see that they tend to send more messages overall, with leads steering their workers more, which is good. s270 00:25:23.867 --> 00:25:30.028 So, we can see that, you know, as the you know, capability of the model so, they message rather more. s271 00:25:30.028 --> 00:25:33.827 But then, on the right figure, you can see that more talking doesn't mean more reward. s272 00:25:33.827 --> 00:25:41.187 For example, for the Sol and Tora Terra, adding messaging changes nothing, basically, beyond noise. s273 00:25:41.187 --> 00:25:44.227 And Luna teams actually lose four to seven points. s274 00:25:44.227 --> 00:25:46.187 So, it just seems distracting to them. s275 00:25:46.187 --> 00:25:53.507 And we noticed that communication mostly carries status report to a lead, you know, who already receives each worker results, anyway. s276 00:25:53.507 --> 00:25:56.227 So, it it was just like an overhead. s277 00:25:56.227 --> 00:25:58.587 There was nothing important to be discussed. s278 00:25:58.587 --> 00:26:00.307 And I want to be clear here. s279 00:26:00.307 --> 00:26:05.947 I don't want to say that peer messaging is not necessary, and we should not do it. s280 00:26:05.947 --> 00:26:17.467 I do think it's necessary, and I do think as we scale the number of agents, they should be capable to communicate efficiently together, because it doesn't make sense to all the time communicate through your manager for you to work with your team members. s281 00:26:17.467 --> 00:26:18.827 No, it doesn't work like that. s282 00:26:18.827 --> 00:26:33.989 I think that it's also a product of our of the harness that we built, and also of the, you know, the task itself, because basically the lead was splitting the work and making sure that each work is independent of the others and so s283 00:26:33.989 --> 00:26:50.869 they give them that the lead gives them directly the task and so they are not also aware of other you know, agent in the same sandbox and so we gave them the tool only to communicate together but they had didn't have visibility who exists there and so they have to do extra efforts and maybe a message word s284 00:26:50.869 --> 00:27:04.309 would have solved the problem and so would incentivize them more to sort of communicate and to figure out oh, this actually agent is doing something similar to what I'm doing let me communicate or I need more help let me reach out to someone else. s285 00:27:04.309 --> 00:27:15.589 Anyways, we leave this to future work and so the other thing that we also were curious about is like you know, whether other you know, models frontier models know how to lead. s286 00:27:15.589 --> 00:27:27.629 So we also run five other models family with the same harness with identical tools and so here I'm talking about office 4.8.4.7.5 also Gemini family. s287 00:27:27.629 --> 00:27:34.109 We noticed that 5.6 teams delegates often and it gained up to 7 points in reward. s288 00:27:34.109 --> 00:27:44.789 Cloud on the other hand actually behave much you know, the same gaining almost 6 points but then office 4.7 rarely delegated and actually gained very little. s289 00:27:44.789 --> 00:27:54.989 High crew and Gemini you know, delegated as really lead as 5.6 but then gained nothing and so what I want to say here the conclusion is that you know, s290 00:27:54.989 --> 00:27:57.309 two things have basically to to go right. s291 00:27:57.309 --> 00:27:58.909 The lead has to choose to delegate. s292 00:27:58.909 --> 00:28:08.749 This is important because it's actually deciding on who's going to do the work and it has to actually make use of what comes back and at which stage of the task do also delegate. s293 00:28:08.749 --> 00:28:13.669 All right, so here the takeaways three things to remember. s294 00:28:13.669 --> 00:28:21.909 The first thing is that teams are now a real new axe for instance and compute but the catch here is that they cost several times more. s295 00:28:21.909 --> 00:28:27.549 We got up to 10 points which is you know, great but still it's very very expensive. s296 00:28:27.549 --> 00:28:35.590 Now you know, we tried our best to optimize our teams of agent and we were successful to some extent. s297 00:28:35.590 --> 00:28:46.070 Still some work to be done, but essentially here the conclusion is that, you know, where you put agent matters more than how many. s298 00:28:46.070 --> 00:28:49.030 So, you need to find the bottleneck and scale the team there. s299 00:28:49.030 --> 00:28:50.470 This is what matters the most. s300 00:28:50.470 --> 00:28:53.110 And finally, the team does not have to all be strong. s301 00:28:53.110 --> 00:29:00.830 Like, a strong lead with cheap workers can get you near frontier results at actually a fraction of the cost. s302 00:29:00.870 --> 00:29:02.950 And so, what's next here? s303 00:29:02.950 --> 00:29:06.510 These are open questions that we are so curious and excited about. s304 00:29:06.510 --> 00:29:10.590 First, here we're excited about this hierarchy versus self-organization. s305 00:29:10.590 --> 00:29:21.790 We, in our work, we tested up to 16 workers in a hierarchical lead-worker structure, but then how should teams of thousands of agent be organized, communicate, and spend compute? s306 00:29:21.790 --> 00:29:25.750 Can agents design the team themselves given task, budget, and deadline? s307 00:29:25.750 --> 00:29:27.510 We think so. s308 00:29:27.510 --> 00:29:31.750 And so, but that requires, you know, careful investigation of the problem. s309 00:29:31.750 --> 00:29:36.230 Now, the second thing is that we think that there must be alternative forms of communication. s310 00:29:36.230 --> 00:29:43.430 As I mentioned, we observed 84% of messages going from workers to the lead and only 4% between workers. s311 00:29:43.430 --> 00:29:53.190 And so, as I think project outgrow, like coding project, you know, they can outgrow grow one lead, will effective communication here emerge, right? s312 00:29:53.190 --> 00:29:57.190 Will scale or like, should it be trained, right? s313 00:29:57.190 --> 00:29:59.670 How should the lead adapt to the task? s314 00:29:59.670 --> 00:30:12.670 It's not scalable to just for each task that we have, we spend, you know, a huge amount of time optimizing the prompt and giving very detailed instruction to the lead. s315 00:30:12.670 --> 00:30:13.830 That should not be the case. s316 00:30:13.830 --> 00:30:20.750 The lead should figure out how to actually, you know, organize the team according to the task that it has. s317 00:30:20.750 --> 00:30:24.390 Um now, finally, uh multi-agent safety and security. s318 00:30:24.390 --> 00:30:32.870 Um you know, uh this is I can't emphasize on this enough, especially given what happens with the Penny Lane incident. s319 00:30:32.870 --> 00:30:41.432 You know, right now we have teams of agents that can run autonomously 24/7 with no supervision. s320 00:30:41.432 --> 00:30:48.351 They're quite capable and we're not talking anymore about the single agent, but we talking about teams of agent and even swarm of agent. s321 00:30:48.351 --> 00:30:56.312 And so, here the the key question is that how do we secure our systems and our infra? s322 00:30:56.312 --> 00:31:03.072 And we need to urgently basically figure out ways on how to align these models. s323 00:31:03.072 --> 00:31:07.351 All right, with that I actually would like to thank you for making it to the end. s324 00:31:07.351 --> 00:31:13.072 I want to give a special shout out to the first author here, Kenan and Daniel, for their hard work on this. s325 00:31:13.072 --> 00:31:18.312 And for more information, please check out the paper and the blog using these QR codes. s326 00:31:18.312 --> 00:31:21.872 And thank you so much for listening.