Why One AI Agent Isn’t Enough (And How to Run 1,000) - CJ Avilla & Harrison Stall, Anthropic

AI Engineer · 101 min · 1079 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12In 1969, just about 30 miles from here, Ken Thompson uh was sitting at Bell Labs and his wife had just taken their 1-year-old uh son out to California to visit his parents
3. 00:25for a month.
4. 00:26And so, he sat down and he said, "I have a month and I have this project that I want to do.
5. 00:30I'm going to carve it up and in 1 week, I'm going to build a kernel.
6. 00:35In 1 week, I'm going to build an editor.
7. 00:37In 1 week, I'm going to build an assembler.
8. 00:40And then, uh you know, 1 week, I'm going to build uh uh one other thing."
9. 00:47And he had at the end of that month, he had uh Unix.
10. 00:51Okay?
11. 00:51So, what what's beautiful about this process that Ken Thompson went through is that he had what's called conceptual integrity.
12. 00:58And that means that one person can hold the entire project in their head and execute the entire project end to end.
13. 01:05There's another operating system uh that uh was uh you know, a huge effort and that happened also in the '60s and the early '70s at IBM when they were trying to build System/360
14. 01:16and OS/360.
15. 01:18And this was an undertaking where previously, IBM computers, if you wanted to upgrade your computer, every time you upgraded the actual hardware, you had to rewrite all of the software
16. 01:28that came along with it.
17. 01:30And so, OS/360 was this massive effort where they really bet the company and they hired tons and tons of engineers and they they sat down just some back of the map back back of napkin math and said, "Okay, this is going to take about 5,000
18. 01:42sort of man-years.
19. 01:43So, let's hire and throw as many as 1,000 engineers at this project and see if we can get it done in uh in a a short window."
20. 01:52And there was a lot of learnings that came out of that project.
21. 01:55Fred Brooks was one of the the managers of the project and he came away and said like, "Okay, we learned a whole bunch of stuff.
22. 02:00Let's write it down."
23. 02:01So, he wrote it in in this book called The Mythical Man-Month.
24. 02:04So, it's like a very popular book.
25. 02:06And this is a bunch of organizational learnings and management learnings from how to build and ship this really massive project in OS/360.
26. 02:15One of the big takeaways was that a really effective way to organize a team is kind of like a surgical team where you have one sort of leader or one chief,
27. 02:24and they're surrounded by 10 experts.
28. 02:26And so, you might have you know, the surgeon, and then they have you know, a language lawyer, and they have a sidekick, and they have a co-pilot, and then all of those people have their specialties,
29. 02:35but the surgeon ultimately has everything in their head, and they can maintain that conceptual integrity.
30. 02:40And that's great, and you can perform miracles.
31. 02:42Surgeons do it all the time with this small team.
32. 02:45But we also want to take that same sort of structure and scale it up much further.
33. 02:50We fast-forward to just a a month ago, a couple months ago, we saw Linux 7.2 was released.
34. 02:57So, this is another operating system project.
35. 02:59This took nine weeks, and we had contributions from 2,600 something different developers across 300 different companies.
36. 03:07And it didn't start out that way.
37. 03:09Like, the actual contribution to Linux didn't start out that way, but we sort of evolved the way that we can contribute to Linux into this structure.
38. 03:17And so, if you look back at the kernel mailing list from 1998, you'll see that Linus was actually like kind of having a bit of a breakdown.
39. 03:26He was receiving patches from people they were kind of like emailing him, and then you know, saying like, "I've emailed you three times with my patch.
40. 03:33Why haven't you submitted this to the you know, to the kernel or whatever?"
41. 03:36And he's like, "That's it.
42. 03:37I'm going on vacation.
43. 03:38I don't want to hear it.
44. 03:39Get out of my inbox."
45. 03:41And sort of you know, he he he hit his breaking point.
46. 03:44And I think a lot of us who are orchestrating and managing lots of agents now might start to feel this feeling of the being the bottleneck and being the breaking point as we have lots of agents contributing
47. 03:56tons and tons of PRs and we're sitting there, you know, in our in our GitHub trying to like approve PRs and make sure that everything works as expected.
48. 04:03And so uh it wasn't until a couple years later that we actually changed.
49. 04:08But actually the day after Linus posts this like kind of uh you know, I'm done, I'm going on vacation, uh another person posts and they Larry McVoy and he says,
50. 04:18"The the problem is that Linus doesn't scale."
51. 04:22And uh Larry also did a little bit of self-promotion and he's like, "I have an idea.
52. 04:27How about we use version control?"
53. 04:29And was kind of like pushing this commercial product.
54. 04:31Uh and so it actually wasn't until maybe four years later around 2002 that Linus was like, "Okay, you know what?
55. 04:40It's true.
56. 04:40I don't scale.
57. 04:42And instead of uh operating as a star structure where everyone is sort of contributing directly to me, we need to work more as a network where we have these layers of trust and people can submit contributions
58. 04:53to trusted maintainers who are going to review that code.
59. 04:56And then only those exceptional cases will be surfaced up.
60. 05:00Uh but Linus can still kind of like ultimately be the person that is uh committing to main."
61. 05:06On the left you'll see this star structure has like 36 different actors that are all kind of contributing in.
62. 05:11On the right we end up in this like nice beautiful structure that only has like maybe at most four people at each level sort of reporting those changes up.
63. 05:22Uh in San Francisco, uh if you were at AI Engineer you heard from Tarek and Gary and the message was, "We have Fable.
64. 05:31It's really exciting and it's time to be unreasonable.
65. 05:34It's time to be more ambitious and build bigger things."
66. 05:38And Gary told you it's time to boil the ocean.
67. 05:41And so we have these very very capable models and we're excited to talk a little bit today about how we can orchestrate these models into uh systems that can collaborate with each other to build really, really massive
68. 05:56uh projects and and solve even bigger problems.
69. 06:00So, I'm CJ Avilla.
70. 06:01I am a member of technical staff at Anthropic and this is
71. 06:06Harrison, and I'm also a member of technical staff at Anthropic.
72. 06:10And good morning, everybody.
73. 06:12How we doing?
74. 06:13Excited?
75. 06:15Nice.
76. 06:15Lot of energy.
77. 06:16Lot of energy.
78. 06:17WOO.
79. 06:20UH ALL RIGHT.
80. 06:21WELL, AS WE GET GOING HERE, um just want to give you guys a quick lay of the land of how this session's going to run.
81. 06:25Um it's going to be some part like telling you what the heck even is Claude managed agents and what are we about to talk about and like this whole multi-agent
82. 06:33uh orchestration engine that we're uh going to spend a little while explaining.
83. 06:36Um it's also going to be part hands-on workshop, so um laptops out.
84. 06:41We'll have uh developer credits uh on on slide later on, so you can have actually get your hands dirty.
85. 06:46Uh try now some of these features that we're super excited to ship.
86. 06:48Um but before that, we uh we uh we'll give you a a little bit of over of an overview on what the heck is going on here.
87. 06:56So, on Friday, um we are so excited to announce within a product surface called Claude managed agents that Anthropic ships or shipped back in um April that we support a new feature called flows.
88. 07:10Flows are essentially like the next version of multi-agent orchestration, where you can let Claude fully orchestrate the entirety of its own multi-agent topology.
89. 07:19So, like CJ was just talking about these really complicated interconnected webs uh that map to organizations in the real world, we envision that there's a future where Claude is able to basically
90. 07:30orchestrate itself, right?
91. 07:32And uh flows are the way that we do that on our API surfaces.
92. 07:37So, before we get too deep into uh the API surface and the experience specifically, um a lot of you are probably familiar with dynamic workflows in Claude code.
93. 07:45Um, you know, you type in, "Hey Claude, like go and do X, Y, and Z. Use ultra code."
94. 07:50It spins up this like pretty complicated web of agents.
95. 07:53It's fully orchestrated itself.
96. 07:54It uses a really interesting JavaScript engine under the hood to basically support like map reduced ask like activities in deep within Claude code.
97. 08:03Um, and we know that this has become a really, really popular product uh for like typing into a terminal and like using it live.
98. 08:09Um, so there's a bunch of examples on slide, but um, the idea is that instead of having to like literally draw out this classifying act or fan out and synthesize adversarial verification
99. 08:18style um, like topology where you say like, "This is agent one and it does this and this is agent two and like wait until agent one takes some action and yada yada."
100. 08:28Um, you can literally just tell Claude to use ultra code to uh achieve its goal, right?
101. 08:34And Claude will just know based on uh model training that we've done internally and generally just uh its own capabilities, how to effectively orchestrate itself in order to accomplish harder and harder work.
102. 08:46Um, so we we think this is like we're getting a taste of how we're going to solve some of what we think are humanity's hardest problems using like fully agentic automation.
103. 08:55And so this is like, I don't know, I I definitely nerd out about this stuff, but I'm like so, so excited about the direction that this is even headed.
104. 09:01Um, what's really cool about dynamic workflows and just generally like this you using multi uh multi-agent topologies rather than just a single agent is that it a single agent keels over much, much earlier with much less ambitious tasks.
105. 09:15Um, so uh you know, hand waved numbers here, right?
106. 09:19Uh a single agent runs for an hour.
107. 09:21A coordinator and roster, which is like our first pass at multi-agent where we had like kind of this like single coordinator that like has just one level of agents underneath it that can go off and accomplish very specific tasks um, can run a little bit longer,
108. 09:33but then dynamic workflows, we think this is like the key to solving actually hard problems, which enables arbitrarily complex nested sets of uh of agents operating all in parallel and interacting with each other.
109. 09:49So, we spent a lot of time talking to our users.
110. 09:52Um and I'm also like super excited to talk to all of you guys today because I imagine some of y'all have some hot takes on the latest in the industry as well.
111. 09:59And we we've we've been hearing a lot of common common themes throughout.
112. 10:04Um so, first of all, you guys are doing some crazy stuff building out your own multi-agent harnesses, right?
113. 10:12You've got incredibly complicated systems that you're building in order to basically solve for some of those problems that we were talking about just a second ago with the single agent.
114. 10:20Um so, you're you're hand rolling multi-agent orchestration.
115. 10:23You're having to handle container restarts, right?
116. 10:25Like what if one of your agents out of a hundred dies uh you know, like two hours into the task.
117. 10:31How do you actually pick back up where that agent left off?
118. 10:33Make sure you understand generally the state of the whole topology, etc. Um and it's often like very non-trivial to do so.
119. 10:40Um additionally, we uh this this bottom left point in the slide is basically like you're having a single agent have to keep track of like 60 as an as an example
120. 10:51agents underneath it.
121. 10:52And like CD was talking about Linus doesn't scale like that, you know, he's an incredible coordinator, but unfortunately, there's just like some cap to how much the single agent um is able to keep in their head.
122. 11:04And then last but not least, you have to figure out how do you even inspect the graph of agents that you've you've built out.
123. 11:10Do you have to go and like poke at each of the nodes in the graph and like ask where it's at in the in the process?
124. 11:16Do you run everything through the coordinator?
125. 11:18Generally speaking, like how how do you understand the state of the graph at any point in the process?
126. 11:23Um so, we've already heard this just from people using Cloud Code, right?
127. 11:28Um and our first renditions of our own APIs as well.
128. 11:32Don't let me remember there there was this meme that when I think it was probably like post Opus 4 5 when when the models really took off and the the capability just made autonomous coding so so so
129. 11:44powerful where people were like walking around with their laptops slightly open so that their cloud code wouldn't die and like making sure that their agents were still running.
130. 11:51It's like people in the park like pressing two like approve approve approve approve approve just to keep the the thing running.
131. 11:58And we think that that's like hugely limiting, right?
132. 12:01Like that is if we want to solve truly hard like society level problems, that's just not going to scale for our ambitions.
133. 12:09So fast forward to Claude managed agents and how we've embedded this kind of complex multi-agent topology within our own ecosystem.
134. 12:17So I recognize that a lot of you in this room probably actually let me ask how many of you have heard of Claude managed agents?
135. 12:24[snorts]
136. 12:24Oh wow, that's actually pretty solid.
137. 12:27It's about 50% of the room.
138. 12:28For those of you who are not in fact did you should check it out.
139. 12:31It's also like all I think about every day of my life basically for the past year or so.
140. 12:37But it's essentially an API surface that lets you have Anthropic manage the the agent life cycle.
141. 12:44So you define an agent it's basically like a list of skills, tools, MCP servers, etc. You instantiate that agent within a session which is like a request essentially to to get real work done.
142. 12:54And then that agent just goes off and goes off and cooks based on whatever it is that it's trying to accomplish.
143. 12:59So we have a whole API surface that makes it possible to store these like stateful API resources and then monitor everything in real time via durable event log.
144. 13:08So yeah, check it out Claude managed agents and what we're here of course to to talk you talk to you all about it.
145. 13:14Um but back in I think it was May we shipped what we called like multi-agent V1, right?
146. 13:21And this is where you have to define each of the sub agents that your primary agent is able to spin up.
147. 13:27And then you have to listen to basically all of what we call the threads that each of those agents are operating on, right?
148. 13:32So, um, it's a lot like, you know, an operating system, right?
149. 13:36Where you have the the overall process has different threads, each of those threads have their own context.
150. 13:41Um, there is some uh state that's shared between them, but typically in order to share state, you have to like manage message message sends uh back and forth between all the threads.
151. 13:50Um, my point in saying all this is that um it it wasn't the most uh trivial thing to set up even though our API surface supports it.
152. 13:57Um, but fast forward to last Friday, um, where we shipped this new thing called dynamic workflows within cloud managed agents, and we think that this is like way easier and way more possible to get power out of the these increasingly capable models
153. 14:12in a very trivial way.
154. 14:14So, the way that it works is that you have a lead agent that is essentially responsible for creating a plan for how it wants to accomplish a task.
155. 14:20So, you know, migrate my million line mono repo from Python to Rust as as an example.
156. 14:26Um, that agent is able to then spin up Once that agent spins up it its plan, it's able to automatically fan out to its own set of agents that it spins up and down based on uh its own
157. 14:38uh uh I I guess rationality or um capability, and across multiple different phases of the multi-agent orchestration, it's able to accomplish like discrete, verifiable units of work without you having to like manage any of the interconnections between these things.
158. 14:55Um, so, very basically, it might be like phase one, like read the files, and it spins up a bunch of explorer agents to go off and do so.
159. 15:02Phase two, maybe I I don't know, identify something about those files.
160. 15:07Obviously, the the agents are way smarter than me uh these days and would would come up with a better plan than than I could on the spot.
161. 15:13Um, but then at the end, um, you we think that this is essentially like the path to getting really, really, really hard work done on platform via an API surface.
162. 15:24So, if you want to build like this into your products, this is a really trivial way for you to do so without you having to somehow like have Cloud Code open again and I don't know, like connect that to your to your own systems.
163. 15:36So, while these dynamic workflows are running, it's very trivial to first of all watch what's happening in real time.
164. 15:41There's a single event stream that basically gives you everything that the agents are doing in real time.
165. 15:46You can check agent by agent what the events what events are happening in real time.
166. 15:50On top of that, we know that token hunger is like a real problem and you can set budgets on your sessions before kicking them off so you don't actually only burn
167. 15:59substantial amounts of tokens.
168. 16:00And then if you let's say that you have like some system that's listening to generally what the agents are doing in real time.
169. 16:06You feel not so excited about the direction that they might be going or you you want to steer them in real time, then you can do so by just like stopping them by interrupting
170. 16:15via our API surface.
171. 16:18So, in comparison to Cloud Code, I I recognize that like this can be a little bit confusing because the they're both workflows, right?
172. 16:27But we're talking today about these workflows within Cloud Managed Agents.
173. 16:31So, we're an API surface that you should be able to integrate into your own application surfaces or your own internal processes or your own, you know, whatever it is that you might want to
174. 16:42wherever you want to like embed Cloud within your systems.
175. 16:45You don't have to be like typing on a terminal or interacting with the thing in real time.
176. 16:50Maybe you want to hook it up to CI, right?
177. 16:52Like this is like the the trivial way to do so.
178. 16:54Um So, in our comparison chart here, I think the thing that I would highlight is that we we see a lot of users using like running these things on schedules,
179. 17:05listening to webhooks.
180. 17:06GitHub example is a pretty pretty classic one of like maybe you want to build out a bug hunter workflow, right?
181. 17:11We'll actually talk about and and show show off a little bit later.
182. 17:15Um and yeah, I think that that's probably about what I want to share here.
183. 17:20Um, so with that said, let's go ahead and start one of these things.
184. 17:24Um, enough of me yapping.
185. 17:26Let's uh, let's check it out.
186. 17:30Is that Yeah.
187. 17:33Yeah, seamless transition.
188. 17:35Let's see.
189. 17:36Can you see that?
190. 17:37Yeah, there
191. 17:38All right, sweet.
192. 17:40All right, we all good?
193. 17:42Okay, I'm going to make it a little bit bigger.
194. 17:44Um, so I also recognize that some of y'all might not have ever seen the Cloud Console before, so just I'll spend a minute here talking a little bit about it.
195. 17:55So so so so hang with me here.
196. 17:57The Cloud Console is where you as an API integrator can manage essentially all of your resources within Anthropic's API service.
197. 18:04So in the console, I'm in the default workspace and I'll see this nice drop-down.
198. 18:09I'll zoom out a second just so you can see on the left side this not nice drop-down called manage agents.
199. 18:14This is where all of the API resources live for the manage agents product.
200. 18:18It's also where we're going to be spending our time today.
201. 18:20Um, if you click into the quick start, you can see that there's a whole bunch of basically one click to get you going kinds of uh, workflows that you can run.
202. 18:31But if you scroll to the bottom, you'll see this nice little tab here called dynamic workflow.
203. 18:37And that's what I've spent the last few minutes talking about.
204. 18:40So because I recognize there is a a bunch of finance oriented folks in the room, we're going to run the stock watch list scanner dynamic workflow.
205. 18:52Um, so you [clears throat] can see what this config is doing is essentially creating a agent that's using Claude Opus 5.5 passing in this type called multi-agent 2026-10-01 and then providing a system prompt that is basically like what what the agent itself is is
206. 19:10uh, what the persona of the agent is.
207. 19:12Um, I'm not going to belabor this too much, but um, basically the what for all of your edification, this is creating the agent resource under the hood when we click use this template.
208. 19:22So, let's go ahead and get it rolling and see see how this goes.
209. 19:25Um, so the first thing that your agent is going to need in order to accomplish anything in the real world is an environment, right?
210. 19:31This is a a computer essentially for for Claude to operate on.
211. 19:35Um, for the last, oh my gosh, like maybe something like 10 or 12 months at this point, we we've really been about in that giving Claude access to a computer makes it way way way more capable just across the board
212. 19:45and configuring your environment is like the the way uh, that you give uh, Claude access to this computer.
213. 19:50I've already created an environment.
214. 19:52Um, so I'm not going to recreate it, but if I was creating a new one from scratch, I would just hit this four button.
215. 19:57But the um, stock watchlist scanner env is the environment that I used when testing things out this morning and you can see that I can just click this button that says start a session.
216. 20:09Um, and so I am interested in Apple, Nvidia, Amazon, JPM, XOM, COST.
217. 20:14I don't know.
218. 20:15Anything else I should add in here?
219. 20:17Um, Mhm.
220. 20:21I don't know.
221. 20:21I'll leave it because I'm not feeling very creative right now.
222. 20:24Um, so let's go ahead and hit enter.
223. 20:27And what this is doing is kicking off a session.
224. 20:29That's where the actual multi-agent orchestration is going to happen.
225. 20:33Um, I'll jump into that in just a second.
226. 20:34We'll like actually take a look at what's going on, but broadly speaking, this is just using the existing primitives that our API supports today in managed agents and going off and getting real work done.
227. 20:44So, let's jump into the sessions pane.
228. 20:46This is where you can see my other stuff I've been doing on the platform.
229. 20:51Um, and sorry, the zoom is a little funky here.
230. 20:55Um, but I'm going to do a whirlwind tour of this because again, I'm not sure how many folks have seen this live in the real world.
231. 21:01Um, but in this case, we've got our agent that's our stock watchlist scanner agent.
232. 21:06This is like the uh, I guess in the analogy um, I forget the the the name of the final person who's like the overseeing the web of everything, but this is like the person at the top of the pyramid, the the CEO if you will who's
233. 21:18managing everybody else underneath them.
234. 21:20Um then we've got the environment that's the again the computer that I was talking about where Claude is operating and then we've got a bunch of information about what's happening in real time.
235. 21:30Um so you can see the agent is off and doing something.
236. 21:33Um so here's here's the actual run happening.
237. 21:37Um and we'll explain more in detail about like what some of these abstractions are in just a second, but basically like Claude is going step-by-step based on its own self-orchestration
238. 21:46using its own self-defined essentially set of agents to go off and step-by-step achieve each of these things.
239. 21:52So you can see like the the session thread thing I was talking about earlier which is like the the single context window, the single instantiation of an agent um is is being spun up in parallel.
240. 22:02Um so yeah, pretty cool.
241. 22:03I I I don't I don't know.
242. 22:04I think it's pretty pretty sweet that all this just happens basically by by prompting Claude and uh we we can jump into uh what what uh what's happening in our prior session
243. 22:14because it's a little bit farther along than this one.
244. 22:17Um so yeah, the this is basically just showing the the same thing I was just showing, but these are all done now and I don't know.
245. 22:28Uh I our output file that's I'm not going to go into too much detail on, but basically the the the main point here is that and we've we've had I think in this case it was only like six
246. 22:42concurrent agents or so at each step.
247. 22:44Um but Claude can can spin up and down the the number of agents based on the complexity of the task.
248. 22:50Um yeah, I'm trying to think if there's anything else I want to show there.
249. 22:53Um I think that that mostly covers just like the the what this surface is.
250. 23:00Um but we can of course uh take questions in a little bit to, uh, uh go a little bit deeper.
251. 23:05Do you mind jumping back?
252. 23:06Yeah, no, let's do it.
253. 23:08Um.
254. 23:09Yeah.
255. 23:16Another seamless transition.
256. 23:17Yeah.
257. 23:18[laughter]
258. 23:19Yeah, so in the in the previous demo, thank you, Harrison.
259. 23:23Yeah, good walk-through.
260. 23:24Woo.
261. 23:25Yeah.
262. 23:26Um.
263. 23:28So, that was all like, uh, an example of how you might prototype in the console and going through and setting up your your your, uh, your flow, testing it out in the console.
264. 23:39This is a really common way to go through and experiment and see if the flow is behaving as you expected.
265. 23:46And then once you're happy with the way that your flow works, for whatever your use case is, typically you'll go and then integrate this into some part of your
266. 23:55your own application or your own workflow.
267. 23:57So, whether that's like, oh, I actually want to trigger this uh, watch list scanner on a schedule so that it runs every day or every weekday, you know, before market open so I can like schedule this to run 5 minutes before or something so that I get my report
268. 24:10right before market open.
269. 24:11Uh, or you might want to do something where you want to kick off these flows or trigger these tasks from Slack or some some other like interface.
270. 24:18And so this is, uh, the code for how you would, uh, create the agent.
271. 24:23So, it's just like passing in this multi-agent config to enable multi-agent.
272. 24:29And then when you create your first event and send that, uh, that first event to the lead agent, you're going to just say, run a workflow.
273. 24:36And that's all you need to do to get flows up and running and working.
274. 24:40It's just to like tell tell Claude, yes, use multi-agent and yes, run a workflow.
275. 24:46Now, by default, this is kind of like the easiest way to get started, but, um, in this example, Claude is going to like on the fly create whatever sub-agents are required to
276. 24:58uh to perform that work.
277. 25:00And so um it's it's also possible in the uh in the agent config to pass in preconfigured uh subagents.
278. 25:08So, if you have like a set of subagents that know how to like look at uh whatever Yahoo Finance or certain news sources or, you know, the the SEC's Edgar website or whatever.
279. 25:18Like if if you have subagents that specialize in certain activities that are important to your research process, you can also configure those uh as part of setting up your agent.
280. 25:28So, you can set up uh individual agents that specialize in something.
281. 25:32Another thing that's kind of like um really uh I think is is nice to know at this point is that you can also give these agents access to custom skills,
282. 25:41MCP servers.
283. 25:43You can give them access to uh run your your own um uh yeah, like custom tools that run inside of the environment.
284. 25:50So, uh they're really really powerful and then all that works in concert with flows.
285. 25:56Um and we talked a little bit about the environment and you can also run that environment on your own um like you can bring your own sandboxes.
286. 26:05So, you can run this self-hosted environment in your own infra if you want to.
287. 26:08So, um just running through again kind of like what that watchlist scanner is doing.
288. 26:14So, uh typically, if you were to create a watchlist scanner with a single agent, you might go into Cloud Code and say, "Here's the tickers.
289. 26:21Please go research them and come back and tell me."
290. 26:24And um Cloud or any any like single agent would try to do its best to look at each of those tickers, but there's a couple different things that agents are going to struggle with when it's just a single agent.
291. 26:37And we'll talk about those struggles later and how uh flows addresses those.
292. 26:40Um but what uh what with flows, what we're going to do is like split up that list of tickers and for each ticker, we're going to pass it to a scan agent.
293. 26:49So, we're going going about like the scan phase.
294. 26:51So, we take one single ticker and pass it to a scan agent, and that is an isolated context where all it cares about is its one ticker, and it can operate without being like influenced by any of the other tickers or any of the other search results from the other contexts.
295. 27:08Once that scan is done, it can pass it off to the next phase, and now like the review step is also its own isolated context, which may or may not need to like pass back to the scanner or surface those report details that need to be written out.
296. 27:21Um and we went through and we ran kind of like this watch list scanner as a single agent and also as flows.
297. 27:28And um in in most cases, when you think about using flows, you want to think about like, "Okay, I want to make sure that I have a really, really comprehensive
298. 27:40look at this problem that I'm trying to solve, and I want to go really deep and make sure that I'm leaving no stone unturned, and I'm willing to spend the money
299. 27:49to do so."
300. 27:49And so, what you'll see on the bottom right here is that like it costs uh much more to run a flow, but it also in the middle pane, you'll see that it like is much more
301. 27:59effective at finding all of the details.
302. 28:02And uh if you've used uh you know, dynamic workflows, you've probably experienced something similar with Claude Code.
303. 28:07Um so, yeah, it's this is uh also an opportunity to talk about like your um when you're prompting Claude to build a flow, you can ask it to throw different resources at different phases.
304. 28:20So, you might want to say, "Okay, I actually want to allocate more test time compute to the scan phase."
305. 28:26So, you can prompt Claude and say, "Please use uh um you know, or actually let's say for the scan phase, we can use Haiku, a really fast, efficient model that's all it's going to do is just go out and scan the internet, right?"
306. 28:38And so, we can say, "You know, spin up uh maybe two Haiku agents per ticker to go and like do a really comprehensive scan."
307. 28:44And that might be like a very affordable phase.
308. 28:47And then you say, "Okay, actually once I get all the web pages back and I'm going through the details, sometimes I'm missing those details.
309. 28:54So, actually I want to like invest more in the review phase.
310. 28:57So, I want to have Fable review all the details that came back in depth.
311. 29:01So, I'm going to say, 'Claude, please use a Fable agent at the review phase and maybe like do two or three agents per ticker at the review phase.'"
312. 29:08And so, you can kind of like imagine each of these flows being something that you can kind of like expand and contract based on how much you want to invest in that point in the process.
313. 29:19Um and what's Yeah, so this is like a a really effective way to kind of like go through, play around with your investment at each of these phases in order to get the exact result that you want.
314. 29:29And so, um definitely recommend going through like a like a comprehensive eval set to try and make sure that the flows are doing what you want when it's like a very
315. 29:38uh you know, big important task.
316. 29:43The watch list scanner that we showed you from the console is uh one of these templates that you can start.
317. 29:48You can kind of just go, click through.
318. 29:49Um there's also some of the templates are built directly and bundled with Claude code and the Claude API skill.
319. 29:55So, if you have Claude code um and you installed it starting on Sunday night, uh uh then you should be able to run Claude API managed agents on board with watch list scanner and that can spin up a template
320. 30:08um and then allow you to kind of like manage your agents directly from Claude code.
321. 30:12I think this is actually like a really effective way to drive Claude managed agents and generally to use the Claude API is from inside of Claude code.
322. 30:21You can direct Claude code to say, you know, spin up this resource, spin up that resource and it will use another command line interface called Ant CLI to drive and and interact with the Claude API.
323. 30:31And so, um often times instead of using the console, I'll use Claude code and say, you know, spin up a managed agent that does X. Um so, uh today when you're going through and experimenting,
324. 30:42uh this is one of the ways that I would recommend kind of jumping in if you don't want to use the console.
325. 30:46Um, and then there's we we experimented quite a bit with like what kind of initial system prompt or, you know, initial message is best suited for flows.
326. 30:57And we found that like the the more ambiguous and kind of like free form you let, uh, you are about letting Claude do its own thing in the beginning,
327. 31:05the more effective it can be.
328. 31:06So, just tell Claude, "Hey, I want you to, um, use a workflow one ticker for agent and, uh, you know, go out and do this scan.
329. 31:13This is what I'm kind of looking for."
330. 31:15And let Claude take control over like building out the initial flows and then you can kind of like, uh, fine-tune that afterwards.
331. 31:23Um, so let's, uh, let's play around.
332. 31:26So, on screen you'll see a QR code.
333. 31:27This is for, um, uh, some API credits so you can get up and running.
334. 31:32If you don't already have a Claude platform account, another thing we just announced this week is that, uh, with Max and team subscriptions, you, uh, you get some like API credits that are part of your subscription.
335. 31:45So, um, you can also like go link your your subs to make sure that you have those API credits.
336. 31:50So, if you have Max, I think it's like, uh, Max 5X, you'll get like $100 a month.
337. 31:55Um, and then Teams is up to 500.
338. 31:57And then this is like another 50 bucks to go and, uh, experiment with, uh, this Claude managed agents flows.
339. 32:04And, uh, yeah, I think at this point we can pause.
340. 32:06We have like a bit more content that we want to go through, but what let's pause.
341. 32:10Please try it out and then, um, yeah, we can we can open up the floor for questions, uh, if anybody has any questions off the bat.
342. 32:17Okay, so, uh, good question.
343. 32:18So, the question was, when you're in Claude Code and you run the {slash} Claude API skill in Claude Code, is that running on my machine or is that running as part of Claude managed agents, right?
344. 32:28So, um, the Claude API skill and Claude Code are going to run it on your machine or in the cloud depending on how you have Claude Code set up.
345. 32:36And that is like a completely separate system from Claude managed agents.
346. 32:40So, you have you'll have like your Claude Code is going to be running either locally or in the cloud depending on how you have Claude Code set up.
347. 32:47And then, when you kick that off, that's going to make API calls from the Claude Code environment to create the agent, create the session, create all of these different resources in
348. 32:57the API.
349. 32:58Uh is there a roadmap for how it merges with plugins?
350. 33:02So, plugins right now encompass a bunch of different like types of resources, right?
351. 33:06One of those is is skills, right?
352. 33:08Um and so, when you take a look at the API surface, you'll notice that there's there's a field called resources associated with it, and then there's also like some other API fields that are related to things like skills that show up in plugins.
353. 33:20And so, the the short answer is that like the individual components of a plugin should just drop into the API surface already today.
354. 33:27As far as like merging it into Sorry?
355. 33:30Oh, no worries.
356. 33:30So, the other direction being like your workflow generates a scalar something, and you can pull it back into your Claude Code?
357. 33:38Or Do you mind just saying a little bit more about that?
358. 33:40I see.
359. 33:40Okay, so the question is basically given a dynamic workflow that you've created for, in this case, um research on potential acquisitions, can you distribute effectively the workflow uh uh as a plugin?
360. 33:55Um The answer is not today, and I don't have a good answer for if that will be like fully supported out of the box at some point in the future,
361. 34:02but you could take the prompt basically that's generated as part of your workflow runs, and just drop that into like Claude Code or wherever, um check it in, it's a text file, it'll uh
362. 34:13probably be pretty pretty close.
363. 34:15Yeah, yeah.
364. 34:15So, the question is, what about running like some agents in like a Mac environment or other environments that aren't just like presumably the hosted surface where uh Claude managed agents are running?
365. 34:24Um we have this abstraction called custom tools that's basically like a way for the model to kick out to you to ask you to take some action.
366. 34:31So, the way that this would work is you would define a custom tool uh for your agent as part of the workflow run.
367. 34:36And then anytime you want to run something like on, let's say, a local Mac OS machine or like some, you know, arbitrary other computer on the internet or something,
368. 34:45you would basically just like take that custom tool call, do whatever you want to do on that machine, and then return the result back to the model.
369. 34:53Did that answer your question?
370. 34:55Okay, sweet.
371. 34:56Yeah, totally.
372. 34:56So, the question is we all care about token spend at some point and we need to be able to optimize our flows over time to not basically spend tons of tokens.
373. 35:08So, there are a bunch of different ways to do this.
374. 35:10This is kind of a generalizable problem that's like hill climbing on some prompt based on whatever criteria it is that you internally care about, whether it's token, whether it's
375. 35:18capability, etc. And so, um there is I forget if we published the hill climb skill
376. 35:25Yeah.
377. 35:26in Cloud Code.
378. 35:26So, so there's a hill climb skill within Cloud Code that can probably help you hill climb at least the prompt itself.
379. 35:33As far as the tuning of like the the flow operations, um this is kind of a cop-out answer, but you effect you'll have the full transcripts of everything that Cloud has done as part of the
380. 35:48the flow runs.
381. 35:49And so, you can just pull those transcripts, give them to Cloud, ask Cloud to identify any like places where the flow is maybe doing too much or not doing enough,
382. 35:58and it's able to essentially optimize itself.
383. 36:00Um and that way you can ask Cloud to optimize based on whatever criteria you you you want.
384. 36:04But, I will say like we shipped this on Friday and I suspect that this type of like native integration is something that is going to be top of mind for a lot of folks.
385. 36:15Yeah, one thing I would add to that, too, is there's another product called Dreams and Dreaming where you can within Cloud managed agents give a set of sessions to
386. 36:28go through this dreaming process where it can kind of like crystallize and like basically like improve the memory store that is related to those sessions.
387. 36:38So you can attach a memory store, run your flows, and then after you've done it, you know, 10 times or whatever, you can have those sessions run through dreams
388. 36:46and have those dreams kind of like crystallize and improve the memory from from like session to session.
389. 36:51So I don't know if you have uh more dreams thoughts or
390. 36:55ideas, but
391. 36:55Yeah, I think I basically described hand rolling your own dreaming.
392. 36:59But I forgot we have our own product that does this for you.
393. 37:01Yeah.
394. 37:03Right.
395. 37:03So you can you you can either use the product dreaming or you can use flows to implement dreaming.
396. 37:08So
397. 37:10Yeah, totally.
398. 37:11So you can you can visualize it directly in the console if you just want like sorry, stepping back.
399. 37:15The question is can you can you visualize the the multi-agent topology as the flow is running?
400. 37:21Within the console you you can and we'll we'll give you guys a chance to check it out in just a second.
401. 37:27Um but if you're if you want to build like a custom UI around this thing as well, you totally can because each of these flows spins up sub-agents that have their own session threads.
402. 37:36That's like the context in which it's operating and all of that information on a per thread basis is accessible via API surfaces.
403. 37:44So you can spin up arbitrarily interesting UIs around it for for whatever it is that you want.
404. 37:50Yeah, I see Jay's probably about to
405. 37:52Yes, you can like
406. 37:54Yeah.
407. 37:54Look at all the threads.
408. 37:56You can drill into individual sub-agents.
409. 37:58You can see what they were doing.
410. 38:00You can also look at if you're you're accessing this programmatically, there is a huge like really comprehensive set of events that are emitted like as the session is streaming and so you'll see every single like tool call and result
411. 38:14and like when it's handing off between from from the sub-agent back to the main uh, know, the lead agent, etc. So, like, um, yeah, within the flow right now,
412. 38:25the UI in the dashboard is definitely something that we'll continue to improve and iterate on.
413. 38:30Um, but, yeah, you can you can absolutely dig in and see like lots and lots of detail here.
414. 38:35Um, so, you can see like, okay, yeah, this is the thread message received event.
415. 38:40And so, you can tell like, okay, what did the lead agent actually pass to the sub agent?
416. 38:44And you can look at that payload here to deeply understand kind of like, what is actually happening for like every transition in this entire graph.
417. 38:53And I'll also say this is built on the public API, like the dashboard itself.
418. 38:56So, anything that you see here, you could totally hand roll yourself.
419. 38:59The question is, is it self-aware of its own budget?
420. 39:02So, right now, the model does not know its budget like as it's as it's running.
421. 39:07The budget is primarily like guardrails to make sure that it doesn't go off and uh, run away token spend.
422. 39:12Um, we found that at least in today's latest models, um, if you tell it the budget, um, Claude can get a little bit anxious and we don't uh, we we don't we don't want that to happen.
423. 39:24Um, but I would highly suspect that over time that that will improve.
424. 39:29So, the way that it works with flows is if you hit $5, you get an event basically being like, budget reached.
425. 39:34And then if the if you want to like take a look at what happened and potentially up the budget, you can as the session is running.
426. 39:42And if you want to add more money, then you can and the flow will pick off like where it left off.
427. 39:46So, there's this really nice pausing component where any work that you've already accomplished is not thrown away if you up the budget.
428. 39:52Um, but that's totally up to you based on your own Yeah.
429. 39:57So, the question is, what is the plan for integrating with Claude Tag?
430. 40:00Um, so, Claude Tag is essentially like a Slack bot that connects all of your information, uh, uh, together within the Slack surface.
431. 40:08And so, it's a slightly orthogonal product surface, whereas this is an API surface where if you want to like build an integration with Claude directly, you would use something like this.
432. 40:18Um, I doubt at least for now that there would be any direct connection other You know what?
433. 40:25Yeah, yeah.
434. 40:25If you wanted to like build something like tag yourself, you could use flows.
435. 40:31Yes, if you're using
436. 40:32[snorts]
437. 40:35Exactly, exactly.
438. 40:36So like, yeah.
439. 40:37Yeah.
440. 40:38Yeah.
441. 40:38I I would say the other thing too is that like um this is this is something like we want to improve on a lot is like making all of these product surfaces work better together.
442. 40:49And so I would say expect, you know, over the next you know, several months or whatever to continue to see these seams start to like disappear.
443. 40:58You may have noticed that like, you know, co-work and chat kind of like we merged some of those.
444. 41:02You can see now you can use Cloud Code from lots and lots of different surfaces.
445. 41:05So over time we'll kind of like uh close the gap there.
446. 41:08And yeah, um Cloud Tag is definitely like a really effective way to to use a lot of these tools.
447. 41:14And in like we use that a surprising amount.
448. 41:17Cloud Tag is like a huge daily driver for us for for a lot of implementations.
449. 41:20So
450. 41:21So the the question is is there a way for the agents to communicate with each other um like the sub-agents within the flow?
451. 41:27Um there's not like a direct like uh uh way for you to like via the API say sub-agent A like message sub-agent B and like send payload X. Um but
452. 41:40the the sub-agents all share a file system.
453. 41:43And so if there is a need uh within your workflow for them to communicate with each other, they can effectively just use the file system as a way to like send messages back and forth.
454. 41:53Um but there's not like a direct abstraction that uh other than these agent message sent and received where you can like see the message sends back and forth.
455. 42:01There's not a direct way for you to like force one of the sub-agents to message another one.
456. 42:05Yes, so the question is is there a way for the flows to essentially modify themselves and hill climb their own prompts based on issues that they might find.
457. 42:14Um so the best way to do this is probably similar to like the hill climb scale that we were talking about just a second ago, as well as potentially the the dreaming product surface, so that the the agent can see like
458. 42:24what it did in the past and learn from its mistakes.
459. 42:27There's not like such a thing as um so actually here I'll I'll just show really quickly.
460. 42:32We we have this um concept of deployment, and basically deployments are um a way to run these sessions on a schedule.
461. 42:41And you there's not a way for you to like on a schedule like self-improve the thing, but I could totally imagine a world where you like you could build that using these components, and that'd be pretty cool.
462. 42:53Also like for the the hill climb scale that we keep mentioning, this is another sub-command of the Claude API skill.
463. 42:59Um so if you're in Claude code, you can do like {slash} Claude API hill climb, and then tell it what thing you want to like iteratively improve on.
464. 43:07This is kind of like a uh taking um Andrej Karpathy's auto research and sort of baking it into a skill that everybody can use to kind of like iteratively improve on their own prompts.
465. 43:17And so if you start with a prompt that you know, the one off the shelf to do the watch list scan, then you can uh take that prompting and go into Claude code and say, you know, Claude API hill climb this prompt, and then it will work with you to iteratively like figure out, okay, what are we
466. 43:32optimizing for, and then it'll just kind of like grind until it gets like a really nice prompt for you to then like use that uh to to yeah, meet your needs.
467. 43:42Some So depending on uh what happens during the hill climb, sometimes it will give you it really depends on exactly the project or like the problem you're trying to solve.
468. 43:51So I uh I when Haiku was announced a couple weeks ago, or everything happened so fast, I think it might have been last week, I don't know.
469. 43:58Uh
470. 43:58[laughter]
471. 43:59when Haiku was announced, I built this tool for playing charades with my kids.
472. 44:02So it just has like a webcam, and it'll give us a card, we try to act it out, and then Haiku will guess like you know what we're doing.
473. 44:08And so I use this the hill climb um command and Claude generated an artifact for me that let me record sessions of myself doing like these hilarious things rowing a boat and like playing tennis and all this so that it could like improve and iterate on the prompt and say oh we need this many frames per second
474. 44:28and we need to like update the prompt to do this.
475. 44:30And so depending on what the problem is like when you run hill climb Claude may or may not give you an a human interface to kind of like iterate on that and you can also like request Claude to do that for you.
476. 44:41So it's actually like quite powerful.
477. 44:44Yeah so the question is about Navier-Stokes and like these scientific discoveries that are all of a sudden happening kind of all at once and if something like this is like
478. 44:53what we would use to to to to do to do that.
479. 44:56I think maybe in our our demos our our stock stock ticker is not ambitious enough because you pretty you pretty much hit the nail on the head right that like the way that you pull forward substantial amounts of scientific progress
480. 45:09is basically shorting the shortening the timeline that it takes to do this kind of work.
481. 45:13Um and flows are basically a way excuse me to throw a test time compute at hard problems and get more capability faster out of the models.
482. 45:22And so um I can't speak to like specifics of like how internally we like we what our harnessing looks looks like for running these types of research workloads but like we think especially as an API surface
483. 45:35that unlocking the power of this is like the first step for us making these types of discoveries or um scientific progress like broadly doable by the general public.
484. 45:48So um yeah I'm pretty jazzed about that.
485. 45:50Yeah one other thing that I'll add to that too is that like in our previous multi-agent setup for Claude managed agents if you where you have like a coordinator and then a roster of agents
486. 45:59we had some limits around like how many sub agents you can spin up.
487. 46:03With the flow with flows right now, I think it's 1,000 agents.
488. 46:06And so like the other the other like interesting piece of this is that your lead agent could spin up multiple flows at a time or like after one flow finishes, start another flow.
489. 46:19And so you might have like a really massive project that you want to like give to give to a set of agents.
490. 46:25I actually like was really hoping it would be done by now.
491. 46:28It is not.
492. 46:29But I I on Friday kicked off a flow with 1,000 agents to build an operating system to show to you.
493. 46:37And it has already been working for like 85 hours.
494. 46:40And so like it's not done yet and it's like three or four flows in and has been working with like tons and tons of agents on this like bigger project.
495. 46:48And so these are things you can absolutely throw at it.
496. 46:51And obviously because they take a little bit longer, this is something that you I kind of like just started on Friday and then walked away and was hoping that maybe by 9:00 a.m. on Monday it would be ready, but
497. 47:02definitely like the type of project that you could um you could throw at yeah, cloud management agent flows.
498. 47:08Yeah.
499. 47:09Let's save that for an after, but yeah, it was like it was actually like very lightweight.
500. 47:12I was like, "Hey, Ken Thompson made Unix in a week.
501. 47:14Can you try and make an operating system that's Unix compatible with a Python interpreter?
502. 47:19Use flows, you know, feel free to be brave and use up to 1,000 agents."
503. 47:23And that was basically it.
504. 47:25And then like let it run and I also gave it a GitHub repo so that it could work in that GitHub repo.
505. 47:30The last time I checked there was like 1,000 commits in the GitHub repo.
506. 47:33So it's yeah, the agents are still grinding on it.
507. 47:36Yeah.
508. 47:39[laughter]
509. 47:41Uh the question is if we've noticed diminishing returns from like spinning up more sub-agents in the same way that like an organization like on an incremental basis for every new employee you add at a certain size,
510. 47:51there's like coordination costs and such that that increase over time.
511. 47:55That's actually a really good question that I don't have a perfect number or answer for you on.
512. 48:01Um I think that like this whole research area is pretty new and I would be very unsurprised if like this these types of exact like uh numerical assignments to
513. 48:12like or basically like the it's almost like the capability curve per subagent or like the incremental you know, the slope of the line as you're adding a new an extra agent
514. 48:22um is more calculable in the future.
515. 48:25Um but I I don't have a good number right now.
516. 48:26Um yeah, that's a good question.
517. 48:28Yeah, one thing I will say is like with flows, you get the benefit of like throwing them at a problem in an organized way versus an unorganized way.
518. 48:37So like if you take the same 100 agents and throw them at the problem like it's going to do better with flows because there's some organization and isolation between those contexts
519. 48:46with gating and these different phases.
520. 48:48So.
521. 48:49So the question is is there a way to prevent infinite loops or cascading errors?
522. 48:54And the answer is yes, sort of.
523. 48:57[laughter]
524. 48:58Um the there's mechanisms to make sure that like you don't end up with runaway sessions.
525. 49:02So for that I would throw on a budget, right?
526. 49:04Based on what whatever it is that you uh your budget might be.
527. 49:08But then otherwise Claude is actually pretty good at detecting subagents that are misbehaving essentially and managing them accordingly.
528. 49:15Um and so we found that at least in our own evaluations like Claude is Claude is pretty good at like detecting these types of errors and and cutting them out um
529. 49:23uh automatically.
530. 49:24The other thing is we were talking about like dreaming and hill climbing and stuff and like learning from past runs and we think that that's a really powerful way to if you like very specifically
531. 49:33want to optimize parts of your flow or parts of what Claude is trying to accomplish, um that's like a really really nice way to do so.
532. 49:39Let me state your question back to you and tell me if it's if it's correct.
533. 49:42Um so basically you're you're trying to understand if there's a way to like based on some criteria like a smaller set may maybe be a signal like a shot change,
534. 49:52you should only execute potentially like some parts of a flow or some some parts of a like like basically you don't want to burn tokens because like nothing happened.
535. 50:02Right.
536. 50:03There's not really anything like that on platform today other than just like telling Claude to conditionally do stuff with a single flow.
537. 50:11Like if I had to be totally honest with you, I would probably like layer something else on top of flows yourself within your application layer to make that detection.
538. 50:20Um so you could like size basically the uh the the the work that you're trying to do.
539. 50:25I don't think that flows would be a good fit for something that's like, you know, single agent can accomplish in 30 seconds anyway.
540. 50:31Um it's more for like these ambitious, complex tasks.
541. 50:36So, yeah.
542. 50:38Yeah, for sure.
543. 50:38So, the question is uh if I can talk a little bit about the context guardrails between the agents and effectively like the world view that each of the agents have.
544. 50:47Um so each of the agents is spawned into its own session thread that's like totally like physically in the database like separate context windows.
545. 50:55And so there's there's no leakage of those context windows via like file systems or anything like that um by default.
546. 51:03Um beyond that, each of the agents has their own system prompts.
547. 51:07So, whether you want to bring your own system prompts or just have Claude spin up the system prompts, um you can do so and that is like a really good way to enforce the totally separate personas.
548. 51:16Um on top of that, um the I mean the thread message sends and thread message receives are a good way to see the inter inter-agent communication if you want to make sure that like,
549. 51:26you know, middle management agent isn't like doing something with uh its underlings that that you don't want it to do.
550. 51:33Um but generally speaking, those are like that's how it works.
551. 51:35Did that answer your question?
552. 51:37I see.
553. 51:37So, the question is about the shared file system that I alluded to as like a methodology for sub-agents interacting with each other and if there's a way for for to like physically separate those out.
554. 51:47Um the way to do that today would basically just be prompting um, and telling like if you're if you want your agents to only focus on like a very specific set uh, like a subset of directory paths or something for
555. 51:57where it does its work, um, you would just prompt that.
556. 52:00There's no like permissioning layer for the directory paths, um, but honestly like that seems pretty useful.
557. 52:06Um, maybe we'll uh, maybe we'll take that back and come back come back with something better.
558. 52:10So, so the the the follow-up was what about using something like hooks to uh, control access to different parts of the file system?
559. 52:17Um, that's the the way that this would work in managed agents is not quite like hooks, but you're actually kind of answering the question a bit for me because
560. 52:26um, one way that you could do this is we have an abstraction called tool permissions that basically like anytime an agent wants to read a file, it depending on the configuration that you provide,
561. 52:36might need to ask you for permission or you being like your system for for permission.
562. 52:41And in this case, if you wanted to like lock down, you know, {slash}mount{slash}fubar, um, then you could just read the tool basically the request that Claude is making to read
563. 52:49that file, check the agent that's actually like running that uh, read request and just deny it if if you don't want the agent to have access.
564. 52:57Wow, thank you.
565. 52:58Appreciate that.
566. 52:59Oh man, you're you're teeing me up.
567. 53:01Um, we're going to So, the question is about uh, credential management.
568. 53:04Um, so like the the the punchline here, right, is like, "Oh my gosh, like a thousand agents, that's crazy.
569. 53:09Like that doesn't seem like safe to just like let let run wild on the internet."
570. 53:14Um, and we're going to spend like basically the entire second half of this presentation talking about agentic safety.
571. 53:18The question is if there's a way to allow human feedback as part of a flow.
572. 53:22The way that you would implement this today is using something called custom tools.
573. 53:25Basically like give Claude a tool that's like request human feedback and then Claude would request that human feedback from you whenever it needs it and you could provide whatever response you wanted and
574. 53:36from that tool call.
575. 53:37So, the question is um, in the example of building an operating system, uh, what if like what if the thing dies or something and like like how do you see the intermediate states?
576. 53:47How do you interact with it as it's going?
577. 53:49Do you want to take that since
578. 53:49Yeah, sure.
579. 53:50I so um part of the prompt I asked it to have like a a team of docs writers and I was like use Sonnet 5.5. I love the like the documentation that comes out of that and
580. 54:02uh at the end of every single flow a team of docs writers goes back to write documentation about what that phase implemented.
581. 54:10And so I can just go to the GitHub repo and like look at the documentation as it's going.
582. 54:14That's one one way to do it.
583. 54:16Another way uh so you kind of asked about like is there a way to like pause it or stop it?
584. 54:20Um and so there is uh another thing that we'll talk about later is um interrupting a session so you can interrupt the flow and say like actually like you're going the wrong direction.
585. 54:29Please stop right here.
586. 54:30Um you can also prompt Claude to give you reports or like to update a certain report as you go or update it you know like a typically it'll like um give you these marked out files that you can open.
587. 54:42Yeah, and one thing that's pretty cool about the um the the product surface is that you like no matter what type whether you're using flows or not um if if you have Claude write to the out its outputs directory you have like a set of downloadable files that are like totally
588. 54:56uh durable with that another API surface we have called V1 files.
589. 55:00And so or or maybe you're like writing commits to GitHub like um CJ's has been and so there's just like a lot of like outputs um as part of these runs that uh you can inspect yourself.
590. 55:11Yeah, so the question is like halfway through some crazy ambitious problem you find some information like externally or something that can potentially change the trajectory of the the overall run.
591. 55:21Uh do you just want to burn two days of tokens and like start from scratch?
592. 55:24Um so the way this would work today is you would interrupt the session and like provide that more information, you know, lemma X is actually probably probably useful for you Claude.
593. 55:32Take a look at that.
594. 55:34Um and Claude like the the primary agent that you're interacting with has this like full state of where the flow is in its process and it can inspect where where it basically left off in order to
595. 55:46uh kick off essentially a new flow that takes into account that information as well as the past history.
596. 55:51Um what's not supported today is like you can't like take the first flow, like the first plan, and like inject a single step like halfway through um and like rerun later steps or anything like that.
597. 56:02Um but yeah, that's like how you would ingest more information.
598. 56:06[laughter]
599. 56:07Uh so the question is like as a game developer, let's say you want to put in like in-game telemetry um via something like flows.
600. 56:15Um I I I want to make sure that I I'm Tell me if I I don't answer your question because I might I might be going in the wrong direction.
601. 56:21Um so when I think of telemetry in this case, I think of like information about what's happening as the agent's operating, right?
602. 56:29So you could you could drop an agent into whatever product experience you want, whether it's a game, whether it's you know, your website, whatever.
603. 56:35Um but the the telemetry aspect is more like what's happening right now.
604. 56:39Do I need to take actions based on like what the the the session is doing?
605. 56:42Am I going in the right direction here?
606. 56:44Okay, sweet.
607. 56:45Um so we have as part of our API surface, we have this extremely rich, durable event stream where you can stream in all the events as they're happening in real time or you can like list them via
608. 56:54a static API um endpoint and then we um Yeah, I'm trying to think.
609. 57:02I don't know if we've
610. 57:04Is it multi-modal?
611. 57:05So like if you want to throw in like basically in Claude managed agents, if you want to take like a game screenshot and like throw it in and like take some action,
612. 57:12um I think today that would be a pretty bad user experience because it would just take too long depending on what it is you're trying to build in your game.
613. 57:19But I will say that Claude managed agents does support image inputs and Claude is like really good or getting really really good at inspecting images.
614. 57:27And so you could there's totally a world where you like just drop in the screenshot and like Claude can uh take some action uh depending on whatever it is that you want to do.
615. 57:35It's not going to feel like real time right now, but yeah.
616. 57:41Oh, yeah.
617. 57:42Yeah, yeah, totally.
618. 57:43Yeah, that's actually a a great point, which is like maybe telemetry like debugging a crash or something.
619. 57:47You like take a screenshot of what's happening in the game and like throw it into an managed agent.
620. 57:51Totally totally feasible.
621. 57:53So, the question is about sharing these things, right?
622. 57:55Like you want to share these resources.
623. 57:57Maybe you're like an organization that like you have like a set of templated agents, right?
624. 58:01They're useful for you and you want to make sure it's not just you who has access to them.
625. 58:04The way that this would work today, number one, we we have a Terraform provider that's like you can use Terraform to like create agents for you.
626. 58:14But also we have a bunch of like export style functionality in in the console that lets you export agents and drop them into other places, but there's not like a one-click share
627. 58:26share button where you like share an agent with other people.
628. 58:30Basically, we've just tried to break down the barriers to make like taking the body of an agent or you know, whatever API resource.
629. 58:40We've tried to make it as easy as possible to take that and just like upload it somewhere else.
630. 58:43But there's not like a first-class like sharing mechanism within the API right now.
631. 58:47I would say like the most common way that you might give this tool to a customer is to build a product interface for it.
632. 58:55So, like you would build your own like website or something or app and they come to that and interact with it through something that you've built that is just like a layer on top of your managed agent.
633. 59:04And so, like a lot of a lot of Claude managed agent interaction is just through the API directly.
634. 59:09And many many customers are offering that to their end customers as like built like just built into the product.
635. 59:17So, like if you're you're in very popular project management tool that has a Kanban board and you drag one card from like a column to to next column, that might trigger
636. 59:27a Claude managed agent session to go like, you know, using the API, spin up a flow or spin up some session that does something um in the background.
637. 59:35And so, instead of having it like be a like front-end component that is directly available that you can kind of like share out across Claude services, it's something that you would build into your own product
638. 59:46um that's kind of more behind the scenes.
639. 59:49Yeah.
640. 59:51By the what which is being given to them.
641. 59:52So, the question is like is there a way to tweak the behavior of agents rather than like in the system prompt of the agent instead to be keyed based on the type of work that you give the agent?
642. 60:05Um a little bit of a cop-out answer, but I would probably just system prompt that based on the like like like if you want the agent to think about
643. 60:14let's let's say that you know rough sizes of work, like the agent should like take different types of action based on that size of work.
644. 60:20There's not like a knob Okay, well, actually I I take that back.
645. 60:24One way that you might be able to do this um depending on what exactly you're trying to accomplish um is that we have effort knobs on the um uh on a per session basis.
646. 60:34And we also have the ability to like override some subset of fields on a per session basis.
647. 60:39And so, if you like given a task and an agent, you can like set those effort knobs or like change different parts of the agent based on what it is you're trying to accomplish.
648. 60:49That's probably how I would do that.
649. 60:51Yeah, so the question is how does Claude even know, right?
650. 60:53Like how does Claude know what scale uh to to operate?
651. 60:56Um my I know it's probably not the most like satisfying open answer, um but the answer really is like evals and lower defaults where you like if you're if you're asking like the stock ticker thing to run, like we shouldn't be spinning up a thousand agents that are all burning a ton of tokens, right?
652. 61:14But if you're asking about Nav your Stokes, right?
653. 61:16Like maybe you would have to be pushing a little bit harder on the flow and like asking it specifically to do that uh like like spin up that uh level of agent
654. 61:26yeah that level of parallelization so yeah.
655. 61:29All right, I think we probably should keep rolling.
656. 61:33We got one more.
657. 61:35One last question and then yeah we can also take more questions at the end too.
658. 61:39Got you so the question is is using basically like markdown files to both like see what the agent is doing but also presumably like define the agent etc. The right way to do it.
659. 61:52I I think there's a lot of It depends
660. 61:58[laughter]
661. 61:59for lack of a better answer.
662. 62:01The the reality is that our API surfaces have like a ton ton ton of functionality depending on what it is that you're trying to do.
663. 62:07Like we'll talk about credential vaults in an example like if you throw in your API key to a markdown file that you just like give to Claude that's probably bad
664. 62:17and sorry I should say that's definitely bad.
665. 62:19Do not do that.
666. 62:21And so it really just depends on like the complexity of what what you're trying to express with the API surface.
667. 62:26I will say that if you're just trying to like literally have like a basic markdown with like a basic set of like like Claude basically take that and like do something with it
668. 62:35on kind of a one-off basis then Claude manager agent probably isn't isn't the right abstraction for that but like a lot of what how Claude works under the hood with you know all this stuff is basically like text right so it's it's not too far off.
669. 62:48Okay let's okay.
670. 62:50Yeah I'll just add one more thing and then we can keep going.
671. 62:52So one like thing to consider is like how relevant certain context is to each sub agent and so one technique is to gradually reveal that context by having like a markdown file that links to other markdown files so like
672. 63:08if you are part of the engineering team that's working on the kernel maybe you have like the main document will say okay here's the separate docs like an an index of like kernel is here and assembler is here and editor is here.
673. 63:20And so like it can then read the details that are relevant to that sub agent and I can kind of like save some tokens.
674. 63:26So.
675. 63:26We good to roll?
676. 63:27Let's do it.
677. 63:27Yeah.
678. 63:28All right.
679. 63:28I'm sorry.
680. 63:29We've got to keep moving.
681. 63:30We'll we'll have questions at the end.
682. 63:31So so save your questions and we'll uh we'll also probably hang out for a little bit after if I I had to guess.
683. 63:36We didn't even come around and like look at people's consoles.
684. 63:38I I I hope it worked.
685. 63:39Uh if it didn't, we're happy to help you uh run your first flow uh later on.
686. 63:44Um okay.
687. 63:45So earlier we talked about or uh Harrison mentioned Bug Hunter and another common theme that fell out of the last AIE was around loops.
688. 63:55How can we close these loops to make all of these systems kind of like work more on their own and run more autonomously?
689. 64:02And Bug Hunter is a very very effective flow and a pattern that we have been using um and I I want to talk a little bit about it.
690. 64:10It's also available as a template and also in cloud code.
691. 64:14So the idea behind Bug Hunter is you can put up a PR and then you can have a Bug Hunter flow review that PR and uh surface problems that might be uh performance issues or uh you know, like actual like uh breakages
692. 64:29um and uh allow you to more quickly review PRs as a human.
693. 64:35And so uh instead of just having a single agent review a PR, Bug Hunter is a is is a much more effective way to go about reviewing PRs because it uses
694. 64:44a flow where it takes your code, splits it up into these slices, each of those slices is handed to an individual context.
695. 64:52You have this like uh an agent that has uh the responsibility of reviewing only that slice of code and starting to like write down this might be a problem, this might be a problem, this might be a problem.
696. 65:03Then handing that to a second phase where that reviewer can say like, "Okay, given this slice of code, I agree or disagree with these different problems, and then having that all be surfaced to one
697. 65:16phase that can like pick candidates that might actually be real problems that we need to go out and verify.
698. 65:23So, like if you were a human reviewer of a PR and you are looking through some code, you often will have context about different parts of the code base that aren't even mentioned as part of this PR.
699. 65:34And so, that's often why we trust humans to go and continue to review the code that's entering our code bases.
700. 65:40Uh but this becomes much more effective because the agents can start to surface those problems to you.
701. 65:47Um one of the things that you might do as a human is say like, "I think this is going to be a problem because of this this other like corner of the code base that I know about.
702. 65:55Let me pull down this PR and verify that that's actually like going to be an issue and how big of an issue it might be."
703. 66:01And so, you can run through this prover phase where you could download have an agent download the repo, try to reproduce the bug, or like verify that the bug is actually like a legit problem.
704. 66:12And then have these surfaced in a report.
705. 66:14In this case, we just say like, you know, it's it's a report, but in practice, these are added as comments to your your actual PR where Bug Hunter would raise an issue like inline and say, "Okay,
706. 66:26you're making this change to this file and then way off on like this this distant corner of the code base, that's going to cause like this unexpected problem for you."
707. 66:34Um and it can it can give you priorities, you know, this is like a very severe or like less severe thing.
708. 66:39Um So, again, this is a one of the instances where you can like throw test time compute at different parts of the problem depending on your code base and the composition of like, you know, is it a mono repo or you have like lots of different repos, etc.
709. 66:50Um Bug Hunter, really really effective tool for uh helping close the loop on um your PR process.
710. 66:58This is how you would spin it up with Cloud Code.
711. 67:00Cloud API scale manage agents on board Bug Hunter.
712. 67:03Um and uh yeah, this is kind of like how how that works.
713. 67:09Another really fun flow.
714. 67:11Um how many people follow us on Twitter, Cloud Devs?
715. 67:14Do you follow follow Cloud Devs on Twitter?
716. 67:16Okay.
717. 67:16All right.
718. 67:16Oh, not enough of you.
719. 67:17Not enough of you.
720. 67:18Yeah, okay.
721. 67:20So, uh if you follow Cloud Devs on Twitter, you'll see that a lot of times when we announce new products or new features, they come with like an animated video that explains how that product or feature works.
722. 67:30So, we actually use a flow to generate many of those videos.
723. 67:33And so, uh you can I can just say, you know, kick I can kick off a a flow and say, "Explain how dynamic workflows works."
724. 67:41And then it can run through each of these phases or even uh as sub flows or like I'm sorry, not sub flows, just individual separate flows if we wanted.
725. 67:49Uh so, it's going to have like a research phase.
726. 67:51It's going to go out and find what are all the actual details about how this works.
727. 67:55And plan and scout and fact-check.
728. 67:57And then it will build a brief.
729. 67:58And then it can take that design brief and hand it off to production that will go and make individual scenes and animate those scenes.
730. 68:04And then uh in practice, we actually have like another set of phases that will do like sound effects and voiceovers and scripting and like all this other stuff.
731. 68:12And at the end, you get a video.
732. 68:14Um so, Explainer Studio, this is like actually like a quite quite a fun flow to use.
733. 68:19Um and uh as someone who helps a little bit with our Twitter profile, I think it like saves me a ton of time.
734. 68:26I think like, you know, 4 years ago, making the videos that we publish uh on Twitter would have taken like, I don't know, 20 hours or something to animate and like make beautiful.
735. 68:35And now it's just, you know, one very small prompt off to uh off to a flow.
736. 68:40Um you too can do this in Cloud Code uh with the Cloud API skill managed agent on board Explainer Video Maker.
737. 68:46Um and yeah, just ask how What is DNS?
738. 68:50How does DNS even work?
739. 68:51I don't know.
740. 68:52Let's see.
741. 68:53Uh and it'll generate a flow.
742. 68:55So, we've got Yeah, we've got a couple uh of videos that it made.
743. 69:00So, let's let's see.
744. 69:02All right.
745. 69:02So, what if the task has too many pieces for one agent?
746. 69:06All right.
747. 69:07So, we have 300 contracts.
748. 69:08We want it to go through all those contracts and find something out.
749. 69:11So, with dynamic workflows, the agent is going to write the workflow.
750. 69:16And then find these change of control clauses.
751. 69:20So, the server in the background is going to run that workflow.
752. 69:26This is well, this one is just my voice.
753. 69:28Yeah, cuz we
754. 69:29[laughter]
755. 69:29were uh yeah.
756. 69:31Sounds so good, right?
757. 69:32No. The results come back.
758. 69:34They're all combined.
759. 69:35So, these are like the different events that we talked about.
760. 69:37So, you can kind of like see it's actually like pretty well animated explaining, you know, how session threads work, how to enable it through the API by passing this multi-agent
761. 69:47uh argument, and then telling you, "Oh, yeah, it's probably important to set up a budget.
762. 69:52And if the budget changes, you can or like if if you if you if you uh end early without spending it, you can extend it, etc." Um yeah.
763. 69:59How do I deploy to YouTube or
764. 70:02It does not deploy to YouTube, no. Uh and
765. 70:06[laughter]
766. 70:07Um that's a good question.
767. 70:08It's a um We should, though.
768. 70:11We should.
769. 70:11Yeah.
770. 70:13Right.
771. 70:13So, if you if you uh a good follow on YouTube is the Claude AI channel on YouTube where we have done some like really fun things where like it'll stream like music and uh Claude FM did like a very kind of like nice autumnal
772. 70:26kind of like vibe music thing uh live streamed recently.
773. 70:30Who wants to see how a bicycle is made?
774. 70:32Okay.
775. 70:33How many people want to see how a toilet is made or how a toilet works?
776. 70:36Yeah, not not how they're made, how they work.
777. 70:38So, okay.
778. 70:38All right, let's see.
779. 70:39Yeah, toilet I think was the winner just like not by a ton, but
780. 70:44[laughter]
781. 70:45Yeah, I think [clears throat] we know how bicycles work, so Yeah.
782. 70:50Also love that it's like Claude branded the videos.
783. 70:52[laughter]
784. 70:53Like we ask them, it'll give us like a lockup that has, you know, sort of the the the same colors and everything Uh for the backgrounds and you can kind of like give it your own brand style.
785. 71:04Another like plug would be to use Claude design to give it your brand style and go through and create your own like design system.
786. 71:11So, you can go in like Claude AI {slash} design or use design directly from the the Claude uh desktop app and then given your Claude design style, you can
787. 71:22uh give that to your flow and say like, you know, follow my brand uh my my brand settings and it even like the font at the bottom is even like the Anthropic fonts and everything, which is pretty cool.
788. 71:32So, really really powerful tool, especially if you are responsible for animating videos to post on the internet uh and yeah.
789. 71:40Uh another really fun example of flows, but there's lots and lots of different use cases, whether you're trying to solve some um you know, really important millennium problem or you're trying to generate video or you're trying to like improve your stock picks or
790. 71:53uh you know, your dev flow.
791. 71:53[clears throat] There's lots and lots of different things that you can build with flows.
792. 71:58Um so, one thing that we talked about a little bit earlier was a a handful of things that single agents struggle with.
793. 72:07And this is a big takeaway that we learned as part of dynamic workflows and I wanted to just talk a little bit about them and how they apply to uh sort of from from Claude Code and how they apply to Claude managed agents.
794. 72:18So, one is agentic laziness, another is self-preferential bias, and the third is goal drift.
795. 72:24So, um it turns out that like if our prompt is something like, "We're having an outage right now.
796. 72:31Please go look at these 50 different services and tell me if uh you know, tell me the root cause of uh of the outage, but don't touch production.
797. 72:42Like don't actually like change anything in production.
798. 72:44Go research is there an outage?
799. 72:46Don't touch production."
800. 72:48A single agent might have this problem where after it sees 35 services are healthy, then it starts to like convince itself that oh, it's like everything looks good.
801. 72:57It's probably fine.
802. 72:58Like I don't really actually need to look at the rest of them the services.
803. 73:01So, um we have seen that agentic laziness can be like one of the things that a single agent will run into and after it's seen so many that are good, it can kind of just be like, "Okay, I think we're good.
804. 73:11Let me just not look at the rest and let's save some tokens and tell tell our human that like we're good to go."
805. 73:18Coordinator runs into a sort of a similar problem where the you might have three sub-agents reporting back to the reporter or to to the coordinator and the coordinator still has to like keep track of like, "Okay, how many have we seen?"
806. 73:30and are all of the services healthy?
807. 73:32And those sub-agents might also themselves get lazy at a certain point.
808. 73:35And so, um you don't really like improve too much over having the single agent.
809. 73:40When we have flows, we have this sort of it's it's not necessarily deterministic, but we have the ability to split up and dedicate a an individual agent in context to each of those services.
810. 73:51So, they're only responsible for their small scope of work and we can get a much better answer back because we can tell from the flow you know, fan out across all 50 services and dedicate an agent to each of those services to go and see if that service is healthy.
811. 74:06And by the end, we know for sure that we went and checked every single service.
812. 74:10So, this is like the fan out and loop intel done kind of like combined.
813. 74:14Self-preferential bias.
814. 74:16Um if you ask an agent to write a poem and then you ask the agent, "Is that good?"
815. 74:21It's going to say, "Yeah."
816. 74:22Um this is like a pretty common thing.
817. 74:24If you ask an agent to write some code and then you say like, "Is the code good?"
818. 74:27It's going to say, "Yeah."
819. 74:28And it's because it like generally was the one who did the work and like is happy with the work that it did.
820. 74:34And so, if you can uh split that up, you get better results.
821. 74:38We know this from the advisor pattern where you might have like one agent that is the generator and one agent that is like judging the results of that.
822. 74:45Like kind of uh taking advantage of that generator discriminator gap.
823. 74:50But, when you have like a really massive problem, you're still going to like run into this pro- this you'll you'll run into a situation where like you your roster is not going to like necessarily
824. 75:01scale up to a point that you need without having like these separate and distinct kind of like checkpoints.
825. 75:08And so, with a flow, we can spread out across lots of these different agents and have like adversarial review that has its own isolated context that isn't being like
826. 75:18influenced at all by previous things that it checked.
827. 75:22The last one is goal drift.
828. 75:23So, if you remember like the the initial prompt was something like, go out and scan across all these services and tell me if we're healthy, but the last thing that I said was like, don't touch prod.
829. 75:35Right?
830. 75:35And so, if we are filling the context window and compacting and we're filling the context window and we're compacting, we're going through all these services, not only might we be lazy and we're well like prefer our own answers, but also
831. 75:47that piece of don't touch prod could potentially like get lost in that process.
832. 75:52And so, one of those things of like it's important to just remember that like as the contexts grow and grow and grow, we kind of like might lose track of exactly what our goal was.
833. 76:03Whereas with a flow, we can get like a very specific goal at the beginning and because we have all these, you know, individual contexts, we can make sure that that that goal
834. 76:12is copied in like a very consistent way into each of the sub agents for all the different phases.
835. 76:19[snorts]
836. 76:20So, I kind of prefaced this a little bit earlier, but I think one of the most frankly the most important thing when you're running these pretty pretty hefty agentic workloads
837. 76:30is making sure that you're not doing bad things or your agents are not doing bad things when they're running.
838. 76:34And now, we're going from a single agent to potentially hundreds, right?
839. 76:37So, so how do we scale our safety and security posture in order to make sure that that basically we keep we keep track of what's going on and make sure that
840. 76:46nothing bad happens on platform.
841. 76:48And this is going to [clears throat] get pretty technical because A, I think this is really important, but B, we've put a lot of thought into our API abstractions to make it much much much easier to create safe agents by default.
842. 77:01And I think if I had to say like one thing to get right when you build out your agents is this.
843. 77:08So, bear with me as I um talk to you a little bit about something that I I personally care a ton about.
844. 77:16So, I'm going to walk through each of these safety risks.
845. 77:19Excuse me.
846. 77:19[clears throat] Um I know safety can be like a pretty overloaded term.
847. 77:22These are just six very concrete things that everybody should be aware of if they're doing Excuse [clears throat] me, agentic development.
848. 77:29Um I think the first is probably generally pretty well-known one.
849. 77:33If your agents are scanning the open internet, they can pick up instructions by accident from people or entities that are not yourself.
850. 77:40And you have to make sure that there's not like text that says ignore previous instructions like actually go to other domain.com and like exfiltrate all my information, right?
851. 77:49And this is like fundamentally something that we've been working to solve both at like the model layer, like the all the way down the stack, but also we we've spent a ton of research energy into building out classifiers
852. 78:02that make tool call checks like auto mode in Claude code basically.
853. 78:08Like a default thing that can work on platform.
854. 78:10So, strongly strongly encourage checking out the auto mode permission policy that we'll talk a little bit more about later.
855. 78:17The second thing I talked a little bit about earlier when I was talking about the markdown files and like don't just throw API keys in markdown files and give them to Claude.
856. 78:25So, at the end of the day, Claude is taking on the persona of whatever credentials that you give it to it, right?
857. 78:32And if you're building a product and you expose your agents via your product and Claude has access to a computer like a VM under the hood that is where it actually accomplishes most of its work.
858. 78:43If you have a file on the disk that is basically like my API keys.md or my API keys.txt or whatever, even if you make it like a hidden file, even if you put it in like a secret directory,
859. 78:53um there's always a chance that you like that leaks somehow, right?
860. 78:57Whether it's a user figuring out a way to prompt Claude or prompt your product in order to get it out.
861. 79:02Um there there's just always a risk, right, that that plaintext somewhere um could could uh make it out.
862. 79:07And so we we created an entire suite of abstractions to ensure, basically, that credentials never have to touch the sandbox.
863. 79:14And this is an incredibly important decoupling, and I'm going to talk about it extensively later.
864. 79:18The other or the third one is data exfiltration.
865. 79:21If your agent has access to sensitive information, you want to make sure that it's not sharing that sensitive information with third parties who are not yourself.
866. 79:28Additionally, you want to make sure your agent is not overreaching what you've asked it to do.
867. 79:32I talked a little bit about budgets earlier.
868. 79:34There's other limits that you can set on the platform.
869. 79:36Um on top of that, you want to make sure your agents are not operating outside of like their uh their wheelhouse by establishing permission policies on the agent so that only tools that you give the agent access to can actually run.
870. 79:49And then last but not least, like it's pretty table stakes to have some visibility into what the agent is doing.
871. 79:56And so we have a durable event stream that has essentially the full trajectory of everything uh your agent is doing across the entirety of its life cycle.
872. 80:04So, your job as a developer is three different things.
873. 80:08We like this nice like very catchy declare, contain, detect.
874. 80:12Um but that's that's the reality of it, right?
875. 80:13You have to declare what your agent is, the permissions it has, what it can do, what you what persona it should take on, skills, tools, etc. Um and then
876. 80:22like the contain piece is literally like the the physical limitations of the agent.
877. 80:27So, how do you contain the agent so that even in the absolute worst of case, the amount of harm that it could do on the open internet is is minimal.
878. 80:35And so this is where we have very explicit configurations in your environment policy where you can say you can only access these domains or you can only access a web search but not web fetch or vice versa depending on on on your needs.
879. 80:48And last but not least, that durable event stream that I keep talking about is like a very crucial way for you to to detect things that are happening in real time and if if the agent is doing something that you aren't are not happy with depending on whatever like your definition of happy is, and you can always
880. 81:03listen to these event stream in real time and immediately interrupt any sessions that are going in a different direction than you'd like.
881. 81:10Um, let's see.
882. 81:13Cool.
883. 81:13So when we shipped Claude managed agents back in April, there was almost like this meme that Anthropic like settled the debate, right?
884. 81:20The the brain and hands should should be separate when you're building out an agentic infrastructure.
885. 81:26And like jokes aside, we think this is a very very critical component of our infrastructure that makes safety kind of just default possible.
886. 81:35The idea here is that the brain, which is like the loop that's actually running inference in the just general agentic harness, as well as the session state, so like what's happened across
887. 81:46all of the life cycle of every agent and a session is fully separate from what we call the hands of Claude, which is the place that the agent is actually able to take action in the real world.
888. 81:56And you can think of hands as like running a grep on a computer that Claude has access to.
889. 82:02You can think of it like a custom tool that or rather a custom tool that might run on a you know, your Mac OS machine that we we talked about earlier.
890. 82:12Broadly speaking, there should be like no way for you to given a hand access and totally steer the brains in any way that could be considered harmful.
891. 82:22And this is like the key thing of like how we separate all of the secret information that can that needs to be accessible in the brain and the harness
892. 82:30from the hands themselves.
893. 82:32And so, the way um there's this really interesting way that we have made sure that uh credential injection works.
894. 82:40And I think I talk about it on a little bit more detail later if I remember correctly, but um the idea is that every container that has access to any secret,
895. 82:49you just totally fake the secrets, right?
896. 82:51Like those values don't actually work if you try to call like my API.com with that secret.
897. 82:56And only on the way out um at this egress proxy layer here on the right side, do those credentials actually get substituted for what like the actual credential is.
898. 83:05So, there's no way like physically to prompt Claude or to um like give an access to a sandbox, get that credential off of the sandbox or off of off of Claude Claude's brain.
899. 83:17Um so, this is is like totally decouples the the agentic stuff from the um the actual like uh credentials themselves.
900. 83:26So, I yeah, I kind of preface this already, but basically like you by default uh like, you know, you should not be able to foot gun yourself um in using Cloud Manager agents or building any agentic workloads
901. 83:40uh yourselves um into leaking your credentials.
902. 83:43So, like definitely definitely definitely take this seriously.
903. 83:46I think this is very very crucial.
904. 83:48Um So, I'm going to hand it over to CJ to talk about
905. 83:51Yeah, so the uh we'll go through each of these different things and how you can control it from uh from uh the API.
906. 83:57So, again, one of the important ways that you can make sure to contain uh Cloud Manager agents, and we've talked about this a few times, is setting a budget.
907. 84:06And so, we can say, "Okay, run this flow.
908. 84:09You have a $1.25."
909. 84:10Uh and then if it hits $1.25 and the flow is not finished, then you can decide whether or not you want to up the budget and let it keep running.
910. 84:19And it won't have have redo any of the work that it had already done, um or you can just say like, "Okay, that that's what we had budgeted for that specific session."
911. 84:27Um so, every time that you kick off any session, whether it's flows or not, you can use this budget.
912. 84:33And this pairs really nicely with um this other part of Claude managed agents called Outcomes.
913. 84:39And so, the combination of these two things is really beautiful because now you can say, "I would like this result.
914. 84:45Here is the outcome that I want and the rubric that I expect, and here is a budget."
915. 84:49So, go accomplish this outcome given this budget.
916. 84:52Um and the budget, again, is this way to just uh make sure that we're containing and not letting these agents just like uh continue to spend tokens um unexpectedly.
917. 85:03The second uh code example there below is how you can interrupt a flow in the middle of the process from the API.
918. 85:09So, you can just pass this event, user.interrupt, and this will stop the flow, uh and you can like then reprompt Claude or um do whatever you need to do to continue on.
919. 85:19So, these are two different things.
920. 85:20So, one is the budget, the other was you can interrupt in the middle of uh of a session that's running.
921. 85:25And so, one thing to kind of like test this out is I would take, you know, take the flow that you ran today, and then set a budget that's lower than what you expect it to go over, maybe like, you know, uh a dollar or something, and then run the flow,
922. 85:38increase the budget, watch it continue, so you can kind of like experiment with that a little bit.
923. 85:42Um the second one we talked about the um this ability to have a reviewer agent go back through all of your sessions and understand what happened as part of that session and getting visibility
924. 85:54into what were the tools that were called and did anything unexpected happen in any of these sessions?
925. 86:01This is another use case where you could spin up a Claude managed agent with a prompt that says, "Here's the things that I want you to look for that happened."
926. 86:08And because we have that durable log and we have every single event that happened, you can pull that transcript with the API and pass it to a managed agent and say like, "Review this entire transcript
927. 86:19and all of these past transcripts over the last week or whatever and tell me if my agents did anything that was unexpected or accessed anything that was they weren't supposed to or called any tools that were kind of like odd or unexpected."
928. 86:31And so, um this is another um kind of like a handy thing that you could do is build a Claude managed agent and hand it the sessions from today and just ask it like, "Was there anything odd that happened in these sessions?"
929. 86:42The last and I think this is actually like the easiest lowest-hanging fruit uh if you have any Claude managed agents integrations today and that is uh to take your permission policy from either always ask or like uh and turn on auto mode.
930. 86:58Turn on auto mode.
931. 86:59This is like super super important and also very very low level or like low-hanging fruit.
932. 87:03Um we did uh a bunch of research and there's some like public research about how uh you know, when your laptop was open and cracked a little bit and you're at the park and Claude keeps asking you like, "Do you approve?"
933. 87:14And everyone's like, "Two two two two two.
934. 87:16Yes, yes, yes, yes, yes.
935. 87:17Approve, approve, approve."
936. 87:18You know, like 97% of the time we're going to click approve as humans and so, um we are yeah, we easily get like lulled into just being like, "Yes, yes, yes."
937. 87:27But Claude is like much more patient and using auto mode, it can like determine whether or not a single tool call is safe or it's going to take some destructive action
938. 87:37and whether or not like it should actually like do that thing.
939. 87:40So, auto mode all you have to do is switch from always ask to auto mode or whatever your permission policy is today to auto mode and it's like the easiest lowest-hanging
940. 87:47uh fruit thing to do.
941. 87:48So.
942. 87:50Sweet.
943. 87:50And so, I want to go into detail a little bit about the vaults thing that I alluded to earlier um and why we think credential management is such a crucial part of any platform that
944. 87:59uh you run your agents on top of.
945. 88:01So, in this example, we are using the Twilio API via an environment variable.
946. 88:06So, in the naive situation here, we'd like take our container or our rather our uh session, maybe like upload a file that has this like .env in it that includes this Twilio API key so that our agents could read it, and then actually call the Twilio API from that container if it was trying to take some some
947. 88:24action in let's say a Python script.
948. 88:26Um but the problem with that again is like all the plain text stuff I've I've been talking about for for a minute now.
949. 88:31Um and so instead of like that default just like getting thrown out of the container, um you can use credentials and credential vaults in order to manage these secrets.
950. 88:40Um so in this case we're using a vault credential in order to create a uh in order to store our Twilio API key as an environment variable, and this injection location thing is basically for all network egress on the way out of the container,
951. 88:55where do I expect the secret to show up?
952. 88:57Do I expect it to be in like a a bearer off uh token on the way out?
953. 89:01Or do I expect it to be like literally in the request body?
954. 89:04Um uh you can configure um based on this field.
955. 89:08Then additionally, the networking hash here with this allowed hosting is basically where given a call to domain X, uh like what what where or sorry, given our credential X, what domain should we substitute on the way out?
956. 89:22Because if we just naively were able to like guess these secrets um and or or these like placeholder secrets that exist on the container, we don't we don't want like our Twilio API key to ever go out to
957. 89:33I don't know, like mydomain.com.
958. 89:35Um so really really important to um uh handle this stuff safely.
959. 89:40And then once you've created your credentials uh inside of your vault, then you just pass in the vault and when you create your session and the the platform will be smart enough to know like based on what's in the vault and what secrets exist and at the egress layer what
960. 89:56um like placeholder secrets exist, it can like figure out what uh how to inject in the correct values.
961. 90:02We also talked about giving agents a the smallest tool set possible.
962. 90:06Um so one of the like most um, I think interesting tools that we have in our agent toolset is web fetch and web search.
963. 90:13And maybe you want to disable those by default because you don't want your agent to have any access to the internet um, without you very explicitly allowing it.
964. 90:21In that case, you can set the enabled field to false for any of the the the types of tools that you want to disable.
965. 90:27Um, so, here's just some basic examples for agent tools.
966. 90:32MCP tools are are a really interesting one because um, you don't know the MCP tools that are going to come back from an MCP server up front.
967. 90:38So, like let's say that you're calling out to the GitHub MCP.
968. 90:42Like, yeah, we we trust GitHub.
969. 90:43Um, but like what if something happened and they were compromised and an attacker added like, you know, my exfiltration tool or something to it.
970. 90:52Um, you probably don't want that to be picked up by your agent.
971. 90:54And so, you you on a tool-by-tool basis can can enable um, uh, per per MCP toolset.
972. 91:00So, um, additionally, you're able to limit the network access that your agent has access to um, like physically using the environment configuration.
973. 91:09Um, so, you can think of environments again as like the computer that Claude is operating on.
974. 91:14And as such, if I only wanted api.twilio.com to be accessible, then I can like literally literally say that in the environment configuration.
975. 91:23And so, um, now I have like my API key on the like like hidden away from um, any potential like theft.
976. 91:31And I can call api.twilio.com physically.
977. 91:34If I tried to call, you know, some other website.com, it would like physically fail at the uh, firewall layer um, on as uh, like like when egress tries to occur.
978. 91:46All right.
979. 91:46So, just kind of bringing it back to the beginning, uh, we think that we're in many cases trying to like reinvent the wheel of these organizational learnings that we had,
980. 91:59you know, in the the and '70s about how to build really massive teams of humans to go out and achieve these like engineering feats.
981. 92:07And so, we're really excited to give you access to flows, and we're excited to see what you'll build with flows.
982. 92:13So, again, hopefully you can take these and go achieve some really really ambitious projects.
983. 92:22So, again, start from a template, shape it shape it to be your own.
984. 92:26Tell us what you find, and yeah, we would love to take some more questions.
985. 92:30I think we're getting close to the end here, but we can also we'll also hang out outside in the hallway track.
986. 92:35Absolutely happy to answer any questions.
987. 92:37So, I see one over here, yeah.
988. 92:39Yeah, so the question is like you try to you know, give an agent a task, and the thing that it gives you back is like not very verifiable.
989. 92:48For example, with the video, that's like very much, you know, your human taste.
990. 92:52So, then you give it some feedback, and how do you sort of like iterate on that within flows?
991. 92:57I would say one place to start is to start with Claude code and iterating on some process first.
992. 93:04So, you can either use dynamic workflows in Claude code or directly try to work through the process of generating your first video, and then have Claude encode like what you did into a skill that you can then use
993. 93:15inside of Claude managed agents as part of a flow.
994. 93:18So, I think like usually there is like quite a bit of iteration in the beginning to like figure out, okay, what are all the different phases, and how does this process actually work?
995. 93:27And maybe you need to do fact finding across the organization to understand how an existing process is built today.
996. 93:33And then trying to like do your best to encode that, and then use the hill climb skill to like iterate on the the actual prompt and improve it.
997. 93:41And that hill climb thing can be sort of human driven.
998. 93:44So, you can say, you know, let's hill climb on making these videos it like really great.
999. 93:48And so, it'll generate one, you can say or it'll it might generate two, and you can say which one you prefer.
1000. 93:53And then it'll generate two more, and you can say which one you prefer, and it kind of like will uh iterate on its own prompt that way.
1001. 93:58Yeah.
1002. 93:59So, I think the question was going back, tell me if I'm wrong on this, but going back to the operating system example, when would you use Ultra Code versus when would you use Claude managed agents, like flows, right?
1003. 94:08Cool, cool.
1004. 94:09Yeah, so so Ultra Code is going to be the situation where you're like sitting at your laptop, like typing into the terminal, and using Claude Code directly and interactively.
1005. 94:17The Claude managed agents flows tool would be if you have like a repeatable thing that you want to happen on some regular basis, and want to integrate an API against an API
1006. 94:27that can hook into like whatever product surface uh you want, rather than like directly just using Claude Code.
1007. 94:32Yeah, I I think that that way that you described it at the end there is is is the correct way to think about it, that Ultra Code is like the the single instance of one of these things essentially.
1008. 94:41And we actually took the Ultra Code like uh abstraction or methodology, and generalized it into what flows is on the Claude managed agents API.
1009. 94:50Um and so, you can think of Ultra Code as like the my one-off, like me interacting with the thing, and then Claude managed agents is like the, "Okay, I want to like level this up uh a layer, and
1010. 94:59reuse them, and um run them on whatever trigger schedule I want."
1011. 95:03And that's a great question.
1012. 95:04So, the question is, how do you think about liability if like, well, something malicious happens on platform?
1013. 95:10Um unfortunately, I am not a lawyer.
1014. 95:11Um and I will not be able to like like explicitly weigh in too much, other than um I would think of it a lot like, you know, running any other
1015. 95:20like platform-y kind of business, where if you integrate against an API, and like build a product on top of that, um at the end of the day, like the the the platform itself is going to have the relationship with with the user directly.
1016. 95:33And so, it's going to be kind of like a defense-in-depth kind of thing of like making like each layer of the stack should be doing their best to make sure nothing bad happens on platform.
1017. 95:40Yeah, so the question is about the durable event stream and use cases um that the folks have have built out internally.
1018. 95:47Um so, it I'm trying to think of a a fun one.
1019. 95:51Um somebody internally built out like a uh a competitive Catan uh like I guess simulation where it uses like all the events generally just to explain like why each of the players is making the choices that they're making.
1020. 96:06It uses like custom tools, which is one of the it's like the way that you would expose your Mac OS machine to uh to cloud manage agents in order to
1021. 96:14uh like take action like outside of just the hosted environment.
1022. 96:19Um, I don't know, you you might have some fun fun ones here.
1023. 96:22I think like the easiest way to think about it is if you want to build your own UI that has like shows all of the different tool calls and everything.
1024. 96:28So, I I think a lot of people are reinventing this the wheel here and there are some like front endy tools to help you expose your agents event streams and stream back generally like the messages that are coming back from agents, but being able to like build a chat interface,
1025. 96:43you know, it's kind of like um pretty table stakes these days, but a chat interface where you're interacting with a you know, a conversation with an agent and you can see the tool calls and then you can open those tool calls
1026. 96:53and maybe you're you're having a a tool call that has like some generative UI and it'll show you like the chart and then for especially for like the finance use case, you might say like, "Okay,
1027. 97:03every time I talk my my stock ticker watch list thing encounters something interesting, also render like the charts so I can do like my or like add technical analysis things to my chart and make it so that it's like really interactive and you could do a lot of those with like the event stream that's coming back."
1028. 97:18Um, but yeah, hopefully that answers the question.
1029. 97:20I think there's a because there's like a lot of events, it's a very generic set of tools that you can go and use to build basically any UI on top of.
1030. 97:29So, the question is if there's any compatibility difference between like native Anthropic and self-hosted.
1031. 97:33Um, there are some features that aren't fully supported on self-hosted sandboxes, but flows are like totally supported.
1032. 97:39Like will will work out of the gate um with self-hosted sandboxes.
1033. 97:42Um, yeah, I'm trying to think what which ones aren't supported, but generally speaking we try and support like features like across all environments, whether it's self-hosted or not.
1034. 97:51Um just the environment that you pass in when you create your session will change um types.
1035. 97:56So, so the question is in the bug hunter example, where there's like a multi-repo dependency graph, how do you actually or how do we actually structure like what like how does Cloud actually know, right, what what uh what to do?
1036. 98:07Um and the the truth is that it's kind of up to Cloud.
1037. 98:10Um if you give Cloud access to the GitHub repositories, those will all end up mounted inside of the container where Cloud is operating.
1038. 98:16And so then Cloud can make the decision like if you have a really nice like Basil build and like have a really really crisp clean dependency tree, Cloud can use whatever tools it wants on the container in order to inspect
1039. 98:27uh those dependencies.
1040. 98:28Whereas if you have, you know, plain text language ABC that doesn't have like a very explicit uh dependency mapping or it's like non-trivial to uh build out that graph, then Cloud would probably have to like literally search files for symbols or something like that.
1041. 98:43For every PR review that you have, you would have to remount the container remount the uh repository on the container, yes.
1042. 98:50It's a lot like um if how you think about like uh just a CI job generally, where that you have to like run some checkout on the Git repository
1043. 98:57um and uh then you can take action on it.
1044. 99:01So, the question is if you are able to build out a dependency graph ahead of time, will that help you optimize the flow?
1045. 99:06And the answer is totally.
1046. 99:07If you if you have some representation of the dependency graph that you know ahead of time, you think would take a long time to generate, you can upload it as a file into the session.
1047. 99:16Um like we support just general file uploads as part of the platform.
1048. 99:19And then you can be like, "Hey Cloud, like by the way, your full dependency graph you care about is at like this file path."
1049. 99:24So, the question is about how many limits of nesting are there in these multi-agent topologies?
1050. 99:29Um the the real answer today is it's just one layer of nesting.
1051. 99:32We don't support like arbitrarily complex like mega fan outs of uh topologies.
1052. 99:37Um we we probably at some point will want to, but we need to make sure like everything is really uh like the safety and stuff is really locked in.
1053. 99:46Um so the first one I'm going to I'm going to try and restate it and and tell me if I'm understanding it correctly.
1054. 99:51Basically like there's some some form of like identity or data access or something that you don't have access to on your machine because it sits behind a VPN and is that
1055. 100:02data accessible via any like internal MCP servers or only on your machine.
1056. 100:10Okay, got you.
1057. 100:12Because the the way that you would abstract this would be using something we have called MCP tunnels, which is basically a way for you to give private access like private MCP access to the agents on platform
1058. 100:25using Cloudflare tunnels today.
1059. 100:27And basically it's like a way for you to give access such that you don't expose anything over the open internet.
1060. 100:33And I think that's like probably the closest to what you would be trying to do or that you would need to do in order to give these things access to there would be some
1061. 100:42Yeah, yeah, yeah.
1062. 100:43It's yeah.
1063. 100:43It's It's like the connection is directly from wherever your private MCP server is running to to the place that the MCP calls happened.
1064. 100:53So yeah, I would look into that.
1065. 100:54Number the second question was on just like how how stuff works at Anthropic.
1066. 101:00We're very like Claude tag pilled basically where Claude tag does a bunch of work for us autonomously and like the these workloads where like the thing is looking at Jira and all the activity in the past 24 hours or whatever and even like proposing proactive
1067. 101:15changes to tickets and such.
1068. 101:17We we do a lot of that work through something Claude tag shaped.
1069. 101:20So um Yeah, yeah.
1070. 101:23Claude tag supports
1071. 101:24Yeah, Claude tag Claude tag can do that.
1072. 101:25I think if you just say at Claude like every every day at 6:00 a.m. like do this.
1073. 101:29I I think it would work out of the box.
1074. 101:31Yeah.
1075. 101:32Thank you all so much for your attention.
1076. 101:33Appreciate it.
1077. 101:34Appreciate you guys.
1078. 101:35All right.
1079. 101:36[applause]