Scheduled Agents, Dreaming and Q&A: Claude Managed Agents (Part 3 of 4) — Anthropic

AI Engineer · 26 min · 344 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Okay.
00:13So, yeah, let's let's see what this looks like.
00:15Uh, analyze uh Google and Microsoft.
00:20Um, so ideally what would happen and I hope this works live for y'all um is that uh this main agent is going to um evaluate if it has information and then it's going to give us a custom tool where we're going to then spin up individual sessions
00:36for each of those financial artifacts.
00:38So you'll see this dispatch analysis thing.
00:41Um it's going to call it for two tickers and then it's going to kick runs for each of those.
00:46Um so this is how we can orchestrate our multiseession.
00:50um harness and each of those sessions has multi- aent uh that are doing collaborative work and each of those uh also have outcome definition so that there's self-supervision happening
01:00using memory stores to learn from one another um etc etc. Um so let's move on to the finale which is um what if you want to do recurring work?
01:11What if you want to do uh you want to have a list of companies you care about and you want every day at 9:00 am or first of the month to do the analysis over all of these companies
01:21uh on a schedule.
01:22So that's what deployments are.
01:24Deployments is everything is API based.
01:26So we you can obviously interact with our platform via the console.
01:29Um but their main way that we intend to you for you to interact with it is via the API and the SDK.
01:35So you can build uh custom UIs and custom workflows.
01:39Um all of these are just building blocks uh that you can compose together to build your application into something bigger.
01:45Um so you can see here what cloud did is it created a uh deployment um and that is right here.
01:53A deployment is takes in a agent definition.
01:58Uh in this case it's this research uh desk weekly memo using this specific agent.
02:04um it takes in a schedule uh of when it's going to run and it gives it an initial set of messages that it's going to run with.
02:10You can also pass in uh resources like uh uh memory stores or GitHub repos or files that it's going to use on each of those.
02:19Uh and so what this means is that every I think this is first of the month at 1 p.m. or something like that.
02:25It's going to run um this uh specific session.
02:30I could run one now.
02:31So I could trigger it via the console.
02:33it's going to do that.
02:34Um, but I could also build a UI for it and that's what we've done here.
02:39Um, so I can, you know, I have all the crowd.
02:42I can list and and search uh for my deployments and then I can also kick one off um at any given point.
02:49And I don't know why they're failing, but I'll ask Cloud about it later.
02:53Um, and this has come up a few times, which is what about dreaming?
02:56What about self-improvement?
02:58Um, so before I get to your questions, which I'm sure there are many, uh, I'm going to show you a preview that we've, uh, have out for for dreaming.
03:07So what dreaming is, it lets you select, um, so, okay, before I show you a previous one, it lets you select a memory store.
03:15In this case, it we'll we'll use the memory store that we've been using.
03:19You can select a model.
03:20Um, you can give it custom instructions if you want to, but we'll use a default prompt.
03:25Um, and then we're going to let it uh run uh the the last 30 days of sessions through this streaming analysis.
03:32So when I click start streaming, what it's going to do is it's gonna spin up a session using manage agents actually um spin up many sub agents to analyze the transcripts of each of the other sessions that we told it to evaluate at all of them which use the memory store.
03:47Um and it's going to analyze, hey, did the memory store encode all the information that was required?
03:52did a session uh run into any issue that uh that caused problems?
03:56Can we learn anything from it?
03:58Did this happen multiple times?
03:59Can we update the instructions in any way?
04:02You can see here what it actually did is that it found that the skill that we created actually had a few gaps um on the way that it was written and it updated the memory store to tell it even though there is this skill, you should extend this knowledge with the fact that we've learned from it.
04:17Uh and you can imagine that we will continue to extend this to not only improve memory stores but improve skills directly uh agent prompts, sub agent prompts, etc., etc.
04:28Um that's that's it.
04:30That's the demo.
04:30I sorry about the Wi-Fi.
04:32I hope you follow uh later on when when you get home.
04:35Um and let's uh keep going with some questions.
04:38Yeah, we can all take questions.
04:40We have a question over here.
04:41Yep.
04:42Go ahead.
04:43Uh thank you for the demo.
04:44Uh we didn't see any uh telemetry uh instrumentation in the agent.
04:49Is that a default native feature?
04:51What do you mean by telemetry?
04:52Because you have the full observability of all of the events uh that happened.
04:56Right?
04:56If you go into the console, you can see it live, but you can also obviously get it from the API where this transcript view is like a nice view, but if you go to debug,
05:05it's every event that occurred in that session.
05:08So you can see exactly, for example, when did we start calling the model?
05:12uh when did we finish calling the model?
05:14So you can get latency metrics in that.
05:15Is there something else that you mean by telemetry?
05:18Yeah.
05:18Uh the thing is that uh usually in software development cycle you kind of define what you want to observe.
05:24Uh how did we kind of pick what did we want to observe.
05:28Uh right now you get all the events and you just filter them on your end.
05:32If there are things that you think are missing, uh hit us up and we'll talk about adding them.
05:37Okay.
05:38And can I bring my own observability stack in here?
05:41Uh you mean like export to OTEL?
05:43Yeah.
05:44Uh you you could because it's an API you could build that connector fairly easily.
05:49Uh we just everybody has different requirements and we don't want to be opinionated but everything is just API calls.
05:56Mhm.
05:56Thank you.
05:58We have a question.
05:58Uh I I had a question.
06:00So how long can these agents run?
06:03I I I know you mentioned this dreaming thing but can you run them indefinitely or do you see that persistent store keep increasing in size and at some point you need to recompact it offline or something?
06:15We we are running compaction for you so that you can run these for days.
06:19We've seen 10-hour sessions in production uh from some of our users.
06:23So you could run these agents like for a year or indefinitely in theory.
06:27Yeah.
06:28Are you talking about running the agent for a year or you talking about like one specific session that keeps going for
06:33No, basically the agent if I had to if I had a task that I need to do every day, can I just keep it running for 6 months?
06:39Will it not blow its uh persistence?
06:42That's right.
06:42Yeah.
06:43And is there any orchestration piece like where I could have three agents, agent A does something and delegates to B and C. That's some capability that you have or
06:54for for now you would spin you would do sub agents and you would tell claude what you want.
06:58We find that uh there's a desire as humans to constrain what claude does and the way it does it.
07:05As cloud gets smarter we have continuously remove those guard rails and let claude just let it cook uh let it do what it thinks it's best.
07:13So uh right now you think maybe three sub aents is the best topology for it.
07:18But if we just let Claude figure that out, it actually ends up outperforming.
07:23But any topological or like graph based workflow thing in the UI here like where I could have
07:30Not yet.
07:30Not yet.
07:31Not yet.
07:31But but maybe soon.
07:34Uh
07:36you can probably just like speak up.
07:38Yeah.
07:39Sorry.
07:39Sorry.
07:40We'll come back to you.
07:41Sorry about that.
07:42Hey over here on your right.
07:45On my right.
07:46Okay.
07:46Yeah.
07:46So question.
07:47Um, looks great.
07:48A lot of integration points.
07:50What I'm curious about, uh, for, you know, an organ, a a complex organization that potentially manages multiple agents, um, you know, I I I think they want some sort of separation,
08:03arbback, whatever else.
08:04Um, how is that what is what is I guess the recommended approach for uh, a complex organization that has those different agents in here to effectively manage?
08:15We we have workspaces that's our primitive today where you can put different customers in different workspaces and then uh segment them that way and give access to specific users
08:27into those workspaces.
08:28Uh but we know that there's still a gap in what we offer and um that's that's something
08:36our back is on our road map.
08:37So we we do want to provide more like granular controls but right now workspace is like a clean way of like separating your agents where you have like a group of people access to workspace A and they can access all agents within that.
08:48You can have workspace B with like different set of users that can have different levels of access.
08:54So hi excellent presentation really excited.
08:57Uh two questions.
08:58One is deployment.
08:59My understanding was that a deployment is like at a snapshot a moment in time of all the agents.
09:06No
09:07deployment is essentially a way for you to run sessions for a specific agent on a schedule.
09:12So you can say like oh research desk agent go research NVIDIA once every quarter and it'll just create a session every quarter to go research and produce you that report.
09:22And then memory which is a state is persisted across all sessions.
09:27That's right.
09:27You attach your memory store to the session.
09:29So you decide which which session you want to attach which memory store too.
09:34So the memory store is like a independent entity that you just attach whenever you need to
09:38right and the final question is harnesses right one what is harnesses for you because everyone have their own definition
09:47and second is the confident right.
09:49So when I change something here, how do I know that it's not
09:54it's working as expected?
09:55Right?
09:56My own confident as a developer.
09:58Um for you're right, everybody has a different definition of a harness.
10:02Uh we when we talk about a harness in CMA, we mainly mean the tool loop execution.
10:09So managing the context and managing the session state and making sure that the agent can do the work that it wants to do.
10:16When most builders talk about harnesses, they they really mean, you know, they really mean um the ability to coordinate multiple agents and sessions together as a unit of work to achieve some sort of bigger task in a topology of sorts.
10:30Um we're giving you the abstraction of a single session as a unit of work for you to go out and build your harness uh with it as the session being one of the primitives.
10:41Um
10:41and then your second question about how do how do I know how my agent is performing?
10:45Our best recommendation would be to run evals on your agent with like different types of tasks that you want the agent to perform and see how the agent does on those tasks.
10:54Um outcomes is a really good way where we steer the agent for you to get you the specific outcome where you can define an exact rubric for like hey this is the exact outcome I want and you can define max iterations and the agent will keep going into it until it gets you that outcome.
11:10Yeah, the the the verifier agent that that Gabe talked about.
11:14Hi So
11:16after.
11:17Cool.
11:18Hey, thanks for the demo.
11:19I enjoyed it.
11:20Um my my question at the core is a bit about how to choose between skills and agents and I wanted to ask how would you all say what's like the fundamental difference?
11:30Is there an advantage to the harnessing that surrounds an agent that makes it produce better results than a scale?
11:36Um, and one of the reasons I'm asking is because I'm afraid of always investing in new kind of like I guess harnessing and tooling around models and it's it can be kind of tiring always trying to do a new thing.
11:49Um, so yeah, I just want to hear what you think is like is there a big difference and is it going to change any time?
11:54I yes, no. Um, the qu thick hardness, thin harness, thick skills um is is a question that's constantly evolving.
12:05I think what we find is that as Claude gets smarter, the boundaries and and and help that we want to give Claude start to melt away and disappear.
12:14And so what you need today might not be what you need tomorrow.
12:18The boring answer is just run evals and and evaluate these things.
12:22That sounds exhausting obviously like yeah
12:24every time a new model comes out I need to evaluate all these things.
12:27Um what what we find is skills is a good abstraction today.
12:31uh it gives the model enough
12:34information to do what it needs to do.
12:35But I presume that it will start to shift and melt and become thinner and thinner where the model will just be able to do things on its own without any
12:43special requests and instructions uh down the line.
12:47So uh multi-agent seems to be more useful uh in my opinion, but your your mileage may vary on your task.
12:57Hi.
12:57Hey.
12:57Hi.
12:58So my question ties up to last two questions and especially on top of evals.
13:03One thing I'm excited about is dreaming but how will dreaming actually affect the final outcome
13:09something that's where eval comes into the picture but you can have multiple iterations of dreaming sel that can go in different directions.
13:17So is there a plan or do you already support some automated kind of dreaming with run with evals and see that hey how to different direction go into which one should we pick based upon what's going on.
13:30Yeah.
13:30So so dreaming what it does is it looks over you you can choose the duration for which you want the agent to dream over.
13:36So you can say I want to dream over this agent session over the last 30 days and then look at all the transcripts and analyze what task you're trying to achieve and then come up with like a new memory store that's more geared towards like what you were trying to do and we typically will run evals
13:52on our end to make sure that dreaming is providing you with better outcomes and we've seen customers were able to get be get a significant improvement on their evals by running Dreaming and then using the new memory server but again everything kind of ties back to eval.
14:07So like that that's how you would know whether something is performing
14:10as you expect it.
14:20Yeah, I think it'd be good to have your own set of evals.
14:22Um
14:23the other thing is dreaming is in research preview.
14:26So these are early early.
14:28It's like a big experimental thing.
14:30We've seen uplift uh on it generally.
14:32This is we we hear you right like uh how do I gain the confidence that this is
14:36yeah it's like eval connected to dreaming and then we can find out multiple pathways that that has been taken and decide oh this makes more sense because even though
14:47nine sessions may have been saying that hey this is wrong
14:50one says that hey this is good so if you just rely on that session dreaming thing I feel like we may not be grabbing all insights without evalu
14:59if I tell you just to trust me is that no
15:01[laughter]
15:03Thank you.
15:06Yeah.
15:07Hi.
15:07Uh thank you very much.
15:08Um this is very impressive.
15:10Uh those of us who have been working with cloud code for a year or more um have been doing things that are similar to this but this was nice to see this in a really mature form.
15:20Um, one question I have is how much of this was just built on top of cloud code?
15:27That is if somebody was, you know, ambitious, could they could they build something similar to this?
15:33And um, what I wanted to ask you was uh, encourage you to keep things open.
15:38uh some of us are already harvesting what's in the JSON session files uh and starting to experiment with things like uh hepo and that kind of thing for for managing the um
15:50the memories and um and creating a memory pool.
15:53So uh congratulations.
15:55Thank you.
15:56Please keep it open and um can we is there are you doing anything here that's not just built on site on top of cloud code?
16:04Thank you.
16:06We we took the cloud code harness as an inspiration.
16:08We found that it was very well attuned for coding tasks.
16:12Um and different it's different levels of flexibility for other tasks.
16:17We're building an API where we want you to bring your context, your information, your capabilities and build on top of it where cloud code is trying to give you a a product out of the box.
16:27So we have slightly different uh you know value uh props um and so it's not just clocked out in a box uh though you know ideally it performs as well if not better and all sorts of tasks.
16:44Um so not not exactly but it's definitely inspired and motivated by by some of the uplifts that we have seen from cloud code
16:51and and cloud code is a terminal tool and we obviously want to be an API product that and infrastructure uh that is going to scale that's going to be reliable.
17:01So there there there were like challenges that I talked to you about um in the beginning of the presentation that we really worked in to make sure we're providing a really solid
17:10um product for you all to use.
17:14Yeah.
17:14Um, so for a bedrock, is there a console like clawed console like you have here?
17:20That's quite straightforward question, but yeah, I just want to manage and see everything.
17:24This is not available on bedrock, but it is available via cloud platform for AWS.
17:29So there is a AWS version of console that you could go use this with.
17:36Yep.
17:37Yeah.
17:38Uh first uh uh is there any uh how you say is there any restriction on the model to be used like uh non-anthropic models
17:47uh
17:47just like cloud code I can run on nonanthropic models
17:51for for now we we you
17:52yeah it's cloud models only for now
17:54okay so so the second question is what about end user use uh do they have to have a console accounts in order to use it or just a free
18:03uh I can create a customized UI for this and they can use it
18:07so cloud Cloud Console helps you with creating your API key.
18:11So that's how you get access to our APIs.
18:13So you cloud you're not paying separately for cloud console.
18:16It's free to sign up.
18:17You create an API key and then you're paying based on like token usage.
18:20So for for for the case like uh SAS like uh maybe we have multiple tenants right.
18:27So in order to provide this uh agent what kind of a uh how you say model you have already uh depict on so that we can follow for example what kind of a each company are supposed tenant are supposed to have a uh a different space or
18:42or even there is a a sub module for this uh each tenant.
18:48So are are we going to how do we actually going to organize that?
18:51In terms of this
18:52we we we have seen that the best model for multi-tenency is not bring your own key because you lose observability because it runs on your end users API key rather than than yours.
19:03So we encourage you to model things with workspaces as a tenency primitive.
19:08Um and we're improving that so that it's easier and easier to build uh on top of this for multi-tenant applications like
19:15yeah so I think currently you could either have put all your customers data into like a single workspace and then you maintain a mapping on your end on like which agent maps to which of your customers.
19:26Alternatively like Gabe said you could create a workspace per customer of yours but that can get more complicated because then you have to like recreate your agent.
19:34So, so is any you see the examples that uh people did the additional security layer or RBAC layer over the this workspace because to me it may be a little bit you know
19:45yeah that that would probably be like a good good way to do it right now because we don't have any arvback
19:52uh controls uh on our level other than like workspace because everything within your workspace is accessible to everyone in the workspace but we are working on making our back better and then making the experience better if you're working with cloud manage agents but building on behalf of your customers and how do we make that easy something
20:09so so is there any way you can show how to scale this uh if there's a multiple uh how you say tenant working on that
20:16yes
20:17any scaling parameters like uh what is the how you say the the minimal uh pod or whatever these parameters that's was yeah
20:26uh I mean you're you're defining on a you can create as many sessions as you want yeah like there's no limitation I mean we have some rate limits obviously but um but there's no scaling parameter for you to say now I need to run 10,000 pods
20:40so so currently no no such thing as like uh like HPA in the Kubernetes like you can do these kind of setting scaling uh parameters right like timeout or whatever
20:50you you don't need that
20:51you don't need to because we're managing it all for you you're just making API calls that's all
20:55okay okay sure uh thanks for your uh answer the last uh by the way I the offering link does not work for me I'm not sure if it's the only one.
21:05Yeah.
21:05Thank you for the sharing.
21:06Thank you.
21:06Thank you.
21:08Yeah.
21:08Go ahead.
21:09Hi.
21:10First, thank you very much.
21:11I've been having a ball with everything that you've been releasing and um I'm really interested in how you're using UI.
21:18Um you're spinning up a server, you're distributing this, but how might we deploy that either internally, but I'm interested in to sort of a customer base.
21:29If I want to make for example an astrologer
21:33um that person has to enter sensitive data.
21:36So it's actually really nice for me to be able to like have that in a
21:39have a layer.
21:40Yeah.
21:40So there are some procedural things you get like a star chart example.
21:44I'm not building this but it's a good example like I can have a sub agent do an interpretation that could last for like 90 seconds.
21:52It could create some assets etc. When the outcome is met it could shut down.
21:57So this gives me a way to actually just make a really clean implementation where all I need is off
22:03and then everything can spin up from there.
22:05Yeah, exactly.
22:06So there are a couple of things like MCP channels that kind of point to this with like fake chat for example, a way for you to kind of just use claude
22:16and have a channel to communicate with it through UI.
22:19But I'm seeing a lot where you're just spinning up a server inside a project that's easy to distribute internally.
22:25But how can you deploy that at scale?
22:28Um I mean we've this is uh being deployed today at scale in production.
22:33For example, notion external agents are built on top of this.
22:36So this is something that exists today that you can go and use and uh basically leverage cloud managed agents through notion in Atlassian for Jira integrations for example.
22:47Uh there's definitely ways to do it.
22:48I think if you have specific questions we could talk uh
22:51yeah we we can talk after this but typically you would probably build your product your UI that you're distributing to your customers and then calling into us for like your actual agent
23:01the phrase
23:02management everything is through the API
23:04everything that's right
23:06so you will manage all the API calls and build your product and UI around that yeah
23:12hi
23:13hi hi first thanks for a great workshop and demo and uh my question is Not really to like the agent itself.
23:20It's just like I saw the Android just released the sunet five.
23:24Yes.
23:24So I just was wondering if any anything you guys could share with us.
23:30Yeah.
23:31It's a great model.
23:32Yeah.
23:32[laughter]
23:34Try it.
23:37It is supported in cloud manage agents.
23:40Go give it a try.
23:41You can use it today and see.
23:43Hey there.
23:43Um hi if we've got uh business partners that are using like cloud co-work today some of them more techsavvy cloud code to build some like spot solutions for specific business cases.
23:53What do you see as sort of the level of sophistication up level to start using man like managed agents instead because this seems like more powerful
24:02but the you know cloud platform could be intimidating to folks that aren't engineers let's say.
24:06Yeah, I think the console quick start is probably like the part of my demo where you're literally just like telling Claude the agent that you're trying to build should make that pretty easy for
24:17even non-technical users to come in and build their first agent.
24:20But obviously this is an API product and is naturally going to be a little bit more intimidating.
24:25But if they're comfortable with cloud code, we do have a manage agent skill with cloud code that basically just like in natural language will walk you through building your first agent.
24:33And that would probably be my advice uh to try it out.
24:37I think the biggest difference is co-work building building personal agent.
24:40With this one, you're trying to build probably an agent that's like available to like your team, your internal organization, or you're building a product that you're shipping to your end customers.
24:49But if you're building personal agents, co-work is still a really great tool to continue doing that.
24:54Are there any skills to kind of like take something that exists in co-work?
25:01there's a skill to interact with our API.
25:04So that cloud is probably good enough.
25:06So if you say here's what I did in co and here's the skill to call the API make the translation I feel pretty confident.
25:14Yeah five we'll we'll do it for you.
25:17Um my question is about enterprise.
25:20uh there are c lot of enterprises who'd like to run these things inhouse because they have sensitive data sets that they won't want going out also they may have already set up their own secure proxies to anthropics models
25:33um GIS and also the other question was about scaling so within the enterprise now you can actually allocate compute to scale it our own way right uh so an on-prem kind of solution
25:45have you all thought about it
25:47uh definitely been a customer ask from a bunch of for enterprise customers and and we're seriously thinking considering it at the moment, but we we don't already have something that we can we can share with you, but we're we're we're seriously considering it.
26:01Excuse me.
26:02Could you show the last slide that show the QR code of the workshop?
26:08Uh are you looking for the repo or the credits?
26:11The repo.
26:12Okay.
26:12Yeah.