Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic
https://www.youtube.com/watch?v=QSeaZ6qHyXs
AI Engineer · 18 min · 252 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:13] All right, we have a question here.
[00:21] Hey.
[00:23] How do you see ICD the agent convex?
[00:26] Um
[00:27] Ah, that's a great question.
[00:28] Uh so, we have something called Ancho CLI, which is a CLI that basically wraps our API to make calls um and allows you to do sort of Terraform-like things where you pass in YAML.
[00:39] So, you can put your uh long-term uh resources in YAML in your code base and then build um CICD to basically update the Ancho with Ancho CLI using those YAMLs
[00:54] uh every time you make a change to your to your resource.
[00:57] Does that make sense?
[00:58] Got it.
[00:58] Uh so, I assume you guys use the same internally too.
[01:02] Yeah, that's how if if we need to build durable long-term resources, we just we can check them in.
[01:07] All right, thanks.
[01:08] Thanks.
[01:09] Um so, I think you mentioned you can send messages whenever you want.
[01:14] Um can you still send messages if the model's waiting for a a custom tool result?
[01:19] Uh not when it's waiting for a custom tool result.
[01:21] We basically pause execution uh at that time.
[01:24] Um but if the model is doing other work, uh then we do allow you to queue up messages.
[01:30] Um but when it's doing a custom tool result, we're expecting you to send that um so, because of ordering purposes, we we block the calls.
[01:38] Well, what happens if you do send the message?
[01:40] You'll get a 400.
[01:41] Okay, that makes sense.
[01:43] Uh there was an ask to see the QR code slide one more time.
[01:47] The uh
[01:48] The uh the credits one, yeah.
[01:54] Any any other questions?
[01:56] If you're if you're shy, we
[01:57] Yeah, we'll hang out after for any any questions you don't feel
[02:07] Yeah, I'm looking through a repo.
[02:08] There's a lot of interesting workshop talks.
[02:10] Is there anything that's missing from that like slides that would help us go through other
[02:16] Yeah, yeah.
[02:16] sessions like that you can post or is it pretty much self-explanatory through the the readme files you just like have you know cuz you had like a skill that did workshop.
[02:25] Yeah, yeah.
[02:25] The other workshops came from our Code with Cloud conference.
[02:29] Most of which you can find online for the version we did in London about a month ago.
[02:34] Yeah, all all the all those talks are on YouTube if you're interested in the other repos that are in there, but this one should give you like a pretty good overview and it's pretty similar to what we've done at other conferences.
[02:45] Yeah.
[02:46] Yeah, so just going back to evals, can you talk a little bit about you know how you guys are maintaining quality, best practices around the the eval harness?
[02:55] Uh yeah, we run a lot of evals especially as new models come up as new capabilities come up.
[03:00] We have access to the models before you all do obviously.
[03:04] So one of the things that we as a managed offering want to make sure we do is that we update our assumptions about the harness as early as possible so that we can remove things that are no longer needed
[03:15] and we do all of that in a eval base.
[03:17] So we have internal tooling that basically looks like different data sets that we can then run our different sessions.
[03:25] It's because it's all in the cloud it's easy to say like spin up 10,000 sessions to do this all in parallel and then we evaluate those results and tune prompts and tune assumptions,
[03:36] remove things, add things to make sure that for every model release we're doing the best we can and we're evaluating all sorts of things for generic purposes not just coding.
[03:48] So so how does that extend into the agents that we would build?
[03:51] Is there a recommended path for for evals or is there kind of just bring your own solution?
[03:56] Uh I think the the best advice we can give you is to get high quality data to evaluate over.
[04:02] I think Claude Claude code is pretty good at building an eval for you, but garbage in, garbage out.
[04:08] We've seen that some of our initial ideas for how to evaluate things were sort of ill-defined and you get 100%, but what does that mean in performance?
[04:18] So, iterating on your data and making sure that you're setting up the problem correctly.
[04:22] Um I know it's like not out of the box solution, but it generally the advice that we've seen works.
[04:27] And you can probably look at like your historical sessions and like what results you got and then sort of try to score it and see how the model's doing.
[04:36] Um that would be a good way of like building your data set looking at your historical sessions.
[04:42] I have a question over here.
[04:43] Hey, thanks for sharing.
[04:45] Uh I have a question about how to share those agent and all the resources across all the the entire organization.
[04:52] The reason I'm asking is because we use Claude as code quite a lot.
[04:55] We're building shared skills.
[04:57] We're building shared connectors.
[04:58] But for the everything you shared today, for example, agent, memory store, or MCB channel, can we actually share them or can we partially share them across the organization to different departments?
[05:11] So, what is boundary?
[05:12] What is the privilege about those resources?
[05:15] Are those resources bonded to my account or how can I control this?
[05:19] So, what's the what's the way to do that?
[05:21] They're bound to your workspace.
[05:22] So, any API key within your workspace can have access to the resources that
[05:27] So, can my can my teammates edit or modify what I'm building or is Yes, completely controlled, right?
[05:34] So, I think your Claude platform admin, so you'll have an org for your organization.
[05:40] And then your admin will be the one setting up the workspaces and deciding who gets access to which workspace.
[05:47] So, I think
[05:47] you would basically design your workspaces depending on your access control.
[05:51] Correct.
[05:51] You'd probably want to do like wh- whatever group of people you want access to resources X, put them in one.
[05:58] So, you can share whatever you want, even partially of the, you know, you build an agent that can do something, but I only want to share part of that.
[06:04] For example, I share the memory to someone that share the MCP channel to the others.
[06:08] Is that possible?
[06:08] Or I need to share the entire tech stack of what I'm building?
[06:12] We don't have fine-grained controls of like this to share this or this not, but you as a as a builder and integration would be able to control that at pretty granularly so that your end users
[06:22] can have access to partial sharing.
[06:25] Cool.
[06:25] Sweet.
[06:26] Thanks very much.
[06:27] Thank you.
[06:27] Yeah.
[06:29] Yeah, so basically we can generate a lot of code with a lot of agents, and then we're getting to the bottleneck of uh reviewing that code, making sure you can actually merge it.
[06:39] Um how do you guys think about that in terms of are you deploying like I see you deploy every 30 days, which uh might be a different definition of deployment, but
[06:49] is it deploying something you can actually test to make sure it works well, or how are you reviewing uh this many lines of code from all the agents?
[06:57] So, the the original workshop that we wanted to do was actually answering that question.
[07:03] How do you How do you build agents to review code?
[07:05] Um what we found is you can build an integration with CMA so that on PR, it kicks off one, two, three, many sessions with many sub-agents to evaluate what has changed and find bugs.
[07:19] Internally, we we have uh these sorts of agents do a lot of the uplift for us, and we've seen that that enables us to uh reduce the amount of time we need to spend on a per PR basis and reduce the number of bugs and keep the quality
[07:33] uh high.
[07:34] Um but this is partially an open question of now we've removed the boundary of uh typing and exploded the number of PRs we can make per day, hour, second.
[07:46] Um how do we keep up?
[07:47] And we're still exploring that, but
[07:49] Yeah.
[07:50] CMA is part of the answer, I think.
[07:52] Yeah, and you can iterate in a loop, you know, to like review, add comments, review until it's done, but then at some point you still need to run the thing to make sure it works well locally.
[08:02] So, I think that's where the deployments can come in, where you can um you know, auto deploy something that you can test with a custom deploy agent, like you said.
[08:11] Um it just depends, like if you have different environments for each branch, but
[08:15] You you can run your application in the sandbox as well and have Claude self-validate the changes that it made.
[08:22] Um and make sure that like it acts as a QA agent as well.
[08:28] Are these like long-running sessions that you can go in and actually play around on the UI?
[08:33] We don't have uh kind of tunneling today,
[08:36] So, you would have Claude go in there, play around with the browser.
[08:40] That's right.
[08:40] And then
[08:41] and validate it.
[08:41] And then maybe even give you back a video, maybe.
[08:43] Yep, exactly.
[08:45] Cool.
[08:46] Thanks.
[08:48] Any other questions?
[08:49] Hello.
[08:50] Hi.
[08:50] Uh for an enterprise company, if at all we want to maintain true tenant isolation that we want to support, I heard that you are not supporting yet the enterprise
[09:00] uh company uh use cases, probably.
[09:03] We're not supporting on-prem.
[09:05] On-prem.
[09:05] Yeah.
[09:05] But what if I want to have uh let's say I have 100 customers that as an enterprise company, can I have separate uh Claude-managed agent instances
[09:16] Yeah.
[09:16] per company of ours that we want to support?
[09:20] Is that a viable option that you support?
[09:22] Is it only one instance that we get, or can we get one instance per company that we support as an
[09:28] It's however you choose to define it.
[09:30] You can define an agent per customer, and then that way the data will be segregated, or you can use a workspace per customer if you want like true isolation you could do
[09:41] an org per customer.
[09:43] So it depends on how you want to set up.
[09:45] Your agents and you would probably want to maintain that mapping of like which customer maps to which agent.
[09:51] Okay.
[09:54] I think I have a follow up question.
[09:56] So just to given the scenario that we have two customers and we want to build the agent once but the agent going to be shared by the two customers.
[10:08] How to achieve that?
[10:11] There's many ways to do it.
[10:13] You could do it through automation.
[10:14] So you could have your agent definition live in Yamo in your code and then build tooling to update it to both of your workspaces and basically synchronize them or you can model everything in a single organization
[10:25] and then do the the modeling that way.
[10:28] Uh
[10:28] You could create like a single agent across all your customers and then you manage the mapping of like which customer is using which session.
[10:37] Yeah, then in in which workspace I can accurate this agent?
[10:42] Yeah, if if you do it per workspace then you need we give you tooling to synchronize a single definition in your code to multiple places so that whenever you update it in your code through CI you can update all of the definitions at once.
[10:56] So you can either make a copy of your agent which is essentially a template of system prompt model name tools and you can like make a copy of across every workspace.
[11:06] Let's say you go down like the one workspace per customer or you can just put all your customers in a single workspace and then you maintain the mapping of
[11:14] which either you do again have like copies of your agent or you use a single agent and then have the mapping of like which session maps to which customer.
[11:23] We have like a metadata field on the session where you can put in like let's say customer ID so you know which session what map to which customer.
[11:31] Okay, that you know like a putting all the customer data in one workspace, how do we satisfy the data segre- segregation
[11:41] Yeah.
[11:41] requirement, you know.
[11:42] Yeah, if you if you care about data segregation, then you need to put them in individual workspaces, and the resources are allocated per workspace.
[11:48] But, there's basically tooling to enable you to do each of each of those things.
[11:54] And when you say data segregation, like is is that like a legal requirement, or is that something just you want to
[11:59] Yes, so
[11:59] something legal, like customer doesn't want to share data with other customers, you know, they want to dedicate
[12:06] Yeah, you would do it per workspace.
[12:08] Okay.
[12:09] Okay.
[12:10] Thank you.
[12:10] I saw a question here.
[12:12] You have to speak up, or
[12:14] Um Is there a way to modify the compute environment, meaning uh in some cases, I may need to have higher memory for my agent to do the work, or I need a GPU for my agent to actually do the work.
[12:31] Is it supported in this, or
[12:33] To to the Today, if you want to do custom resource allocations for your sandbox, you would use self-hosted sandboxes.
[12:40] You would use something like Modal that has GPU offerings.
[12:43] And we have cookbooks and kind of first-party integration, so that you can hook up your Modal instances to run your agent loop inside of Modal instead of our provided sandboxes, which today are all the same.
[12:57] So, if you care about different scale, you would use a different provider for that.
[13:00] And do you have like a list of specifications of what the environment these agents have access to?
[13:07] Yeah, you you can find that in our docs, and you can find cookbooks to integrate with third-party providers as well in our docs.
[13:13] Yeah, we have a bunch of cookbooks on like how to work with Modal or EB2, Daytona, whatever provider that you want to use for sandboxes.
[13:20] Yeah.
[13:21] Yeah.
[13:22] Hi.
[13:22] Hi.
[13:23] Thanks again.
[13:24] So, out of curiosity, how to build a sim- version of Cloud Tag using CMA in a different chat channel, especially for the multiplayer mode.
[13:36] Uh this definitely something you could do today where you would on a per thread level map it to a session and then anytime a user messages that session, you inject that as a user.message
[13:51] letting the agent know, hey, this is coming from Gabe or if it's coming from Priyanka.
[13:55] And so everybody can message that agent independently.
[13:59] You can have it share memory across sessions so that it learns from each other.
[14:03] You can give it tools to reply back using like the Slack post tool or something like that so that the agent can decide when it wants to post.
[14:12] You don't necessarily want to just mirror whatever the agent is responding.
[14:16] You want it to be explicit.
[14:18] Uh but you could definitely model this using CMA and
[14:21] And if you want it to be like Cloud Tag, you'd probably do like one agent per Slack channel and then every agent has its separate memory and only has access to
[14:30] messages within that Slack channel.
[14:34] Yes.
[14:34] Hey, thank you for the presentation.
[14:36] Very useful.
[14:38] Quick question in terms of um looping and doing long runs um in this Cloud Console, how do you see that?
[14:47] Um is this something you have like some specific module that I I'm not finding or um are you you're planning to develop that um more in the future?
[14:58] Um thank you.
[15:00] Do you mean how do we provide the observability for really long sessions or
[15:06] No, no, no. Um so more focus on um using this platform to build those uh long-running agents like hours and hours like last 6 hours for example.
[15:17] Um is this something that is already embedded in what you have built in the agents or I'm more curious to know if you are developing something specifically for that or how you see that trend.
[15:29] The the outcome system that I demoed is the best way to do really long sessions because you define an exit criteria and Claude will just keep going until it meets that criteria.
[15:40] And it could be a really long task like you know, rebuild blank and it will just mercilessly iterate until it matches the the rubric.
[15:51] There there there's nothing special you would need to do to get the session running longer than six hours.
[15:57] It should it should just keep keep going.
[15:59] One quick follow-up.
[16:01] So, if I run at the agent for hours and hours and I want to understand like checkpoints status, the best way to that will be through the session tab.
[16:10] Uh yeah.
[16:11] be the session event stream.
[16:12] Yeah, there's the events API on under the session.
[16:15] Okay.
[16:15] You could have a custom tool called like update status for human or something like that and Claude would on intervals whenever it feels like it's made progress call that tool and give you a status update.
[16:28] Thank you.
[16:29] Thank you very much.
[16:30] All right, any other final questions?
[16:32] Yes.
[16:33] Um following up on that previous question about using this for Claude tag or just similar functionality.
[16:39] I've been uh analyzing a whole bunch of uh agent frameworks.
[16:45] So, if you get a message while the um agent is waiting for a custom tool result for example, human in the loop, um you said it would 400.
[16:54] Do you have uh any suggestions on how you would deal with that?
[16:58] Um
[16:59] Can you do custom tools on a sub agent?
[17:03] I forgot.
[17:03] You you could do custom tools on a sub
[17:04] Cuz then cuz you could basically have like a sub agent that's handling your custom tool and then that way your main kind of coordinator agent's always free for receiving.
[17:13] In the API, can you directly return a custom tool result for the sub agents tool.
[17:20] Well, the sub agent would emit the custom tool use and then you would populate it back to that.
[17:25] you you define like your main coordinator agent and then you can say agent A can talk to agent B and C and then agent B is the one let's say processing your custom tools and your agent B definition is where you'll define your custom tool and then that's the session thread that you would listen to get the
[17:42] Is that a different session thread or the same?
[17:44] No, it's it's part of like the same core session.
[17:46] a different thread technically, but you only interact with the main thread and we correctly delegate it cuz we know where it needs to go based on IDs.
[17:54] Um
[17:54] Okay.
[17:55] And then the main agent will pass it along to the sub agent.
[17:59] So the session is still idle even though the
[18:02] If if if a sub agent is doing work, it's not idle.
[18:05] It's
[18:06] But like if the main coordinator agent is idle the sub agents could still be
[18:11] still doing work.
[18:12] Yeah, you you can see that here.
[18:13] Okay, that's pretty impressive.
[18:14] This uh gap here, the main session is is idle.
[18:18] The main the main thread is idle while the sub agents were doing work.
[18:21] And they could be waiting for tool results.
[18:22] Exactly.
[18:23] Yeah.
[18:23] So if you're trying to do something where you want like the main agent to be always chatty, then I think the coordinator pattern works pretty well for that.
[18:30] That's awesome.
[18:30] Yeah.
[18:31] I think we're we're at time, but we'll we'll be just outside if you want to come talk to us.
[18:36] Uh thank you for having us.
[18:37] All right.
[18:38] Th- Thank you so much.
[18:40] [applause]
[18:54] [music]