Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic

AI Engineer · 18 min · 252 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:13All right, we have a question here.
3. 00:21Hey.
4. 00:23How do you see ICD the agent convex?
5. 00:26Um
6. 00:27Ah, that's a great question.
7. 00:28Uh so, we have something called Ancho CLI, which is a CLI that basically wraps our API to make calls um and allows you to do sort of Terraform-like things where you pass in YAML.
8. 00:39So, you can put your uh long-term uh resources in YAML in your code base and then build um CICD to basically update the Ancho with Ancho CLI using those YAMLs
9. 00:54uh every time you make a change to your to your resource.
10. 00:57Does that make sense?
11. 00:58Got it.
12. 00:58Uh so, I assume you guys use the same internally too.
13. 01:02Yeah, that's how if if we need to build durable long-term resources, we just we can check them in.
14. 01:07All right, thanks.
15. 01:08Thanks.
16. 01:09Um so, I think you mentioned you can send messages whenever you want.
17. 01:14Um can you still send messages if the model's waiting for a a custom tool result?
18. 01:19Uh not when it's waiting for a custom tool result.
19. 01:21We basically pause execution uh at that time.
20. 01:24Um but if the model is doing other work, uh then we do allow you to queue up messages.
21. 01:30Um but when it's doing a custom tool result, we're expecting you to send that um so, because of ordering purposes, we we block the calls.
22. 01:38Well, what happens if you do send the message?
23. 01:40You'll get a 400.
24. 01:41Okay, that makes sense.
25. 01:43Uh there was an ask to see the QR code slide one more time.
26. 01:47The uh
27. 01:48The uh the credits one, yeah.
28. 01:54Any any other questions?
29. 01:56If you're if you're shy, we
30. 01:57Yeah, we'll hang out after for any any questions you don't feel
31. 02:07Yeah, I'm looking through a repo.
32. 02:08There's a lot of interesting workshop talks.
33. 02:10Is there anything that's missing from that like slides that would help us go through other
34. 02:16Yeah, yeah.
35. 02:16sessions like that you can post or is it pretty much self-explanatory through the the readme files you just like have you know cuz you had like a skill that did workshop.
36. 02:25Yeah, yeah.
37. 02:25The other workshops came from our Code with Cloud conference.
38. 02:29Most of which you can find online for the version we did in London about a month ago.
39. 02:34Yeah, all all the all those talks are on YouTube if you're interested in the other repos that are in there, but this one should give you like a pretty good overview and it's pretty similar to what we've done at other conferences.
40. 02:45Yeah.
41. 02:46Yeah, so just going back to evals, can you talk a little bit about you know how you guys are maintaining quality, best practices around the the eval harness?
42. 02:55Uh yeah, we run a lot of evals especially as new models come up as new capabilities come up.
43. 03:00We have access to the models before you all do obviously.
44. 03:04So one of the things that we as a managed offering want to make sure we do is that we update our assumptions about the harness as early as possible so that we can remove things that are no longer needed
45. 03:15and we do all of that in a eval base.
46. 03:17So we have internal tooling that basically looks like different data sets that we can then run our different sessions.
47. 03:25It's because it's all in the cloud it's easy to say like spin up 10,000 sessions to do this all in parallel and then we evaluate those results and tune prompts and tune assumptions,
48. 03:36remove things, add things to make sure that for every model release we're doing the best we can and we're evaluating all sorts of things for generic purposes not just coding.
49. 03:48So so how does that extend into the agents that we would build?
50. 03:51Is there a recommended path for for evals or is there kind of just bring your own solution?
51. 03:56Uh I think the the best advice we can give you is to get high quality data to evaluate over.
52. 04:02I think Claude Claude code is pretty good at building an eval for you, but garbage in, garbage out.
53. 04:08We've seen that some of our initial ideas for how to evaluate things were sort of ill-defined and you get 100%, but what does that mean in performance?
54. 04:18So, iterating on your data and making sure that you're setting up the problem correctly.
55. 04:22Um I know it's like not out of the box solution, but it generally the advice that we've seen works.
56. 04:27And you can probably look at like your historical sessions and like what results you got and then sort of try to score it and see how the model's doing.
57. 04:36Um that would be a good way of like building your data set looking at your historical sessions.
58. 04:42I have a question over here.
59. 04:43Hey, thanks for sharing.
60. 04:45Uh I have a question about how to share those agent and all the resources across all the the entire organization.
61. 04:52The reason I'm asking is because we use Claude as code quite a lot.
62. 04:55We're building shared skills.
63. 04:57We're building shared connectors.
64. 04:58But for the everything you shared today, for example, agent, memory store, or MCB channel, can we actually share them or can we partially share them across the organization to different departments?
65. 05:11So, what is boundary?
66. 05:12What is the privilege about those resources?
67. 05:15Are those resources bonded to my account or how can I control this?
68. 05:19So, what's the what's the way to do that?
69. 05:21They're bound to your workspace.
70. 05:22So, any API key within your workspace can have access to the resources that
71. 05:27So, can my can my teammates edit or modify what I'm building or is Yes, completely controlled, right?
72. 05:34So, I think your Claude platform admin, so you'll have an org for your organization.
73. 05:40And then your admin will be the one setting up the workspaces and deciding who gets access to which workspace.
74. 05:47So, I think
75. 05:47you would basically design your workspaces depending on your access control.
76. 05:51Correct.
77. 05:51You'd probably want to do like wh- whatever group of people you want access to resources X, put them in one.
78. 05:58So, you can share whatever you want, even partially of the, you know, you build an agent that can do something, but I only want to share part of that.
79. 06:04For example, I share the memory to someone that share the MCP channel to the others.
80. 06:08Is that possible?
81. 06:08Or I need to share the entire tech stack of what I'm building?
82. 06:12We don't have fine-grained controls of like this to share this or this not, but you as a as a builder and integration would be able to control that at pretty granularly so that your end users
83. 06:22can have access to partial sharing.
84. 06:25Cool.
85. 06:25Sweet.
86. 06:26Thanks very much.
87. 06:27Thank you.
88. 06:27Yeah.
89. 06:29Yeah, so basically we can generate a lot of code with a lot of agents, and then we're getting to the bottleneck of uh reviewing that code, making sure you can actually merge it.
90. 06:39Um how do you guys think about that in terms of are you deploying like I see you deploy every 30 days, which uh might be a different definition of deployment, but
91. 06:49is it deploying something you can actually test to make sure it works well, or how are you reviewing uh this many lines of code from all the agents?
92. 06:57So, the the original workshop that we wanted to do was actually answering that question.
93. 07:03How do you How do you build agents to review code?
94. 07:05Um what we found is you can build an integration with CMA so that on PR, it kicks off one, two, three, many sessions with many sub-agents to evaluate what has changed and find bugs.
95. 07:19Internally, we we have uh these sorts of agents do a lot of the uplift for us, and we've seen that that enables us to uh reduce the amount of time we need to spend on a per PR basis and reduce the number of bugs and keep the quality
96. 07:33uh high.
97. 07:34Um but this is partially an open question of now we've removed the boundary of uh typing and exploded the number of PRs we can make per day, hour, second.
98. 07:46Um how do we keep up?
99. 07:47And we're still exploring that, but
100. 07:49Yeah.
101. 07:50CMA is part of the answer, I think.
102. 07:52Yeah, and you can iterate in a loop, you know, to like review, add comments, review until it's done, but then at some point you still need to run the thing to make sure it works well locally.
103. 08:02So, I think that's where the deployments can come in, where you can um you know, auto deploy something that you can test with a custom deploy agent, like you said.
104. 08:11Um it just depends, like if you have different environments for each branch, but
105. 08:15You you can run your application in the sandbox as well and have Claude self-validate the changes that it made.
106. 08:22Um and make sure that like it acts as a QA agent as well.
107. 08:28Are these like long-running sessions that you can go in and actually play around on the UI?
108. 08:33We don't have uh kind of tunneling today,
109. 08:36So, you would have Claude go in there, play around with the browser.
110. 08:40That's right.
111. 08:40And then
112. 08:41and validate it.
113. 08:41And then maybe even give you back a video, maybe.
114. 08:43Yep, exactly.
115. 08:45Cool.
116. 08:46Thanks.
117. 08:48Any other questions?
118. 08:49Hello.
119. 08:50Hi.
120. 08:50Uh for an enterprise company, if at all we want to maintain true tenant isolation that we want to support, I heard that you are not supporting yet the enterprise
121. 09:00uh company uh use cases, probably.
122. 09:03We're not supporting on-prem.
123. 09:05On-prem.
124. 09:05Yeah.
125. 09:05But what if I want to have uh let's say I have 100 customers that as an enterprise company, can I have separate uh Claude-managed agent instances
126. 09:16Yeah.
127. 09:16per company of ours that we want to support?
128. 09:20Is that a viable option that you support?
129. 09:22Is it only one instance that we get, or can we get one instance per company that we support as an
130. 09:28It's however you choose to define it.
131. 09:30You can define an agent per customer, and then that way the data will be segregated, or you can use a workspace per customer if you want like true isolation you could do
132. 09:41an org per customer.
133. 09:43So it depends on how you want to set up.
134. 09:45Your agents and you would probably want to maintain that mapping of like which customer maps to which agent.
135. 09:51Okay.
136. 09:54I think I have a follow up question.
137. 09:56So just to given the scenario that we have two customers and we want to build the agent once but the agent going to be shared by the two customers.
138. 10:08How to achieve that?
139. 10:11There's many ways to do it.
140. 10:13You could do it through automation.
141. 10:14So you could have your agent definition live in Yamo in your code and then build tooling to update it to both of your workspaces and basically synchronize them or you can model everything in a single organization
142. 10:25and then do the the modeling that way.
143. 10:28Uh
144. 10:28You could create like a single agent across all your customers and then you manage the mapping of like which customer is using which session.
145. 10:37Yeah, then in in which workspace I can accurate this agent?
146. 10:42Yeah, if if you do it per workspace then you need we give you tooling to synchronize a single definition in your code to multiple places so that whenever you update it in your code through CI you can update all of the definitions at once.
147. 10:56So you can either make a copy of your agent which is essentially a template of system prompt model name tools and you can like make a copy of across every workspace.
148. 11:06Let's say you go down like the one workspace per customer or you can just put all your customers in a single workspace and then you maintain the mapping of
149. 11:14which either you do again have like copies of your agent or you use a single agent and then have the mapping of like which session maps to which customer.
150. 11:23We have like a metadata field on the session where you can put in like let's say customer ID so you know which session what map to which customer.
151. 11:31Okay, that you know like a putting all the customer data in one workspace, how do we satisfy the data segre- segregation
152. 11:41Yeah.
153. 11:41requirement, you know.
154. 11:42Yeah, if you if you care about data segregation, then you need to put them in individual workspaces, and the resources are allocated per workspace.
155. 11:48But, there's basically tooling to enable you to do each of each of those things.
156. 11:54And when you say data segregation, like is is that like a legal requirement, or is that something just you want to
157. 11:59Yes, so
158. 11:59something legal, like customer doesn't want to share data with other customers, you know, they want to dedicate
159. 12:06Yeah, you would do it per workspace.
160. 12:08Okay.
161. 12:09Okay.
162. 12:10Thank you.
163. 12:10I saw a question here.
164. 12:12You have to speak up, or
165. 12:14Um Is there a way to modify the compute environment, meaning uh in some cases, I may need to have higher memory for my agent to do the work, or I need a GPU for my agent to actually do the work.
166. 12:31Is it supported in this, or
167. 12:33To to the Today, if you want to do custom resource allocations for your sandbox, you would use self-hosted sandboxes.
168. 12:40You would use something like Modal that has GPU offerings.
169. 12:43And we have cookbooks and kind of first-party integration, so that you can hook up your Modal instances to run your agent loop inside of Modal instead of our provided sandboxes, which today are all the same.
170. 12:57So, if you care about different scale, you would use a different provider for that.
171. 13:00And do you have like a list of specifications of what the environment these agents have access to?
172. 13:07Yeah, you you can find that in our docs, and you can find cookbooks to integrate with third-party providers as well in our docs.
173. 13:13Yeah, we have a bunch of cookbooks on like how to work with Modal or EB2, Daytona, whatever provider that you want to use for sandboxes.
174. 13:20Yeah.
175. 13:21Yeah.
176. 13:22Hi.
177. 13:22Hi.
178. 13:23Thanks again.
179. 13:24So, out of curiosity, how to build a sim- version of Cloud Tag using CMA in a different chat channel, especially for the multiplayer mode.
180. 13:36Uh this definitely something you could do today where you would on a per thread level map it to a session and then anytime a user messages that session, you inject that as a user.message
181. 13:51letting the agent know, hey, this is coming from Gabe or if it's coming from Priyanka.
182. 13:55And so everybody can message that agent independently.
183. 13:59You can have it share memory across sessions so that it learns from each other.
184. 14:03You can give it tools to reply back using like the Slack post tool or something like that so that the agent can decide when it wants to post.
185. 14:12You don't necessarily want to just mirror whatever the agent is responding.
186. 14:16You want it to be explicit.
187. 14:18Uh but you could definitely model this using CMA and
188. 14:21And if you want it to be like Cloud Tag, you'd probably do like one agent per Slack channel and then every agent has its separate memory and only has access to
189. 14:30messages within that Slack channel.
190. 14:34Yes.
191. 14:34Hey, thank you for the presentation.
192. 14:36Very useful.
193. 14:38Quick question in terms of um looping and doing long runs um in this Cloud Console, how do you see that?
194. 14:47Um is this something you have like some specific module that I I'm not finding or um are you you're planning to develop that um more in the future?
195. 14:58Um thank you.
196. 15:00Do you mean how do we provide the observability for really long sessions or
197. 15:06No, no, no. Um so more focus on um using this platform to build those uh long-running agents like hours and hours like last 6 hours for example.
198. 15:17Um is this something that is already embedded in what you have built in the agents or I'm more curious to know if you are developing something specifically for that or how you see that trend.
199. 15:29The the outcome system that I demoed is the best way to do really long sessions because you define an exit criteria and Claude will just keep going until it meets that criteria.
200. 15:40And it could be a really long task like you know, rebuild blank and it will just mercilessly iterate until it matches the the rubric.
201. 15:51There there there's nothing special you would need to do to get the session running longer than six hours.
202. 15:57It should it should just keep keep going.
203. 15:59One quick follow-up.
204. 16:01So, if I run at the agent for hours and hours and I want to understand like checkpoints status, the best way to that will be through the session tab.
205. 16:10Uh yeah.
206. 16:11be the session event stream.
207. 16:12Yeah, there's the events API on under the session.
208. 16:15Okay.
209. 16:15You could have a custom tool called like update status for human or something like that and Claude would on intervals whenever it feels like it's made progress call that tool and give you a status update.
210. 16:28Thank you.
211. 16:29Thank you very much.
212. 16:30All right, any other final questions?
213. 16:32Yes.
214. 16:33Um following up on that previous question about using this for Claude tag or just similar functionality.
215. 16:39I've been uh analyzing a whole bunch of uh agent frameworks.
216. 16:45So, if you get a message while the um agent is waiting for a custom tool result for example, human in the loop, um you said it would 400.
217. 16:54Do you have uh any suggestions on how you would deal with that?
218. 16:58Um
219. 16:59Can you do custom tools on a sub agent?
220. 17:03I forgot.
221. 17:03You you could do custom tools on a sub
222. 17:04Cuz then cuz you could basically have like a sub agent that's handling your custom tool and then that way your main kind of coordinator agent's always free for receiving.
223. 17:13In the API, can you directly return a custom tool result for the sub agents tool.
224. 17:20Well, the sub agent would emit the custom tool use and then you would populate it back to that.
225. 17:25you you define like your main coordinator agent and then you can say agent A can talk to agent B and C and then agent B is the one let's say processing your custom tools and your agent B definition is where you'll define your custom tool and then that's the session thread that you would listen to get the
226. 17:42Is that a different session thread or the same?
227. 17:44No, it's it's part of like the same core session.
228. 17:46a different thread technically, but you only interact with the main thread and we correctly delegate it cuz we know where it needs to go based on IDs.
229. 17:54Um
230. 17:54Okay.
231. 17:55And then the main agent will pass it along to the sub agent.
232. 17:59So the session is still idle even though the
233. 18:02If if if a sub agent is doing work, it's not idle.
234. 18:05It's
235. 18:06But like if the main coordinator agent is idle the sub agents could still be
236. 18:11still doing work.
237. 18:12Yeah, you you can see that here.
238. 18:13Okay, that's pretty impressive.
239. 18:14This uh gap here, the main session is is idle.
240. 18:18The main the main thread is idle while the sub agents were doing work.
241. 18:21And they could be waiting for tool results.
242. 18:22Exactly.
243. 18:23Yeah.
244. 18:23So if you're trying to do something where you want like the main agent to be always chatty, then I think the coordinator pattern works pretty well for that.
245. 18:30That's awesome.
246. 18:30Yeah.
247. 18:31I think we're we're at time, but we'll we'll be just outside if you want to come talk to us.
248. 18:36Uh thank you for having us.
249. 18:37All right.
250. 18:38Th- Thank you so much.
251. 18:40[applause]
252. 18:54[music]