WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=K0X9QDRkIdg NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/K0X9QDRkIdg.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.120 --> 00:00:14.160 All right. s3 00:00:14.160 --> 00:00:15.360 Hi everyone. s4 00:00:15.360 --> 00:00:17.119 Thank you for joining us today. s5 00:00:17.119 --> 00:00:18.240 I'm Gagen s6 00:00:18.240 --> 00:00:19.359 and I'm Isabella. s7 00:00:19.359 --> 00:00:23.680 We're both members of technical staff here at Enthropic at the applied AI team. s8 00:00:23.680 --> 00:00:28.240 Our team sits at the intersection of product research and go to market. s9 00:00:28.240 --> 00:00:35.760 And we spend a lot of our daytime building agents, evaluating claude, and finding ways to make it better in different use cases. s10 00:00:35.760 --> 00:00:40.800 We're here today to talk about how the surfaces for building agents has evolved in the last three years. s11 00:00:40.800 --> 00:00:46.879 What our teams have learned building agents both internally at Enthropic and externally with our enterprise customers along the way. s12 00:00:46.879 --> 00:00:49.280 I'll hand it off to Gogen to kick us off. s13 00:00:49.280 --> 00:00:51.039 So here's the plan. s14 00:00:51.039 --> 00:00:59.520 We'll first start off by talking about how we've seen agentic surfaces evolve all the way from the messages API to cloud managed agents. s15 00:00:59.520 --> 00:01:05.920 Isabelle will then cover the engineering principles behind cloud managed agents and how it works under the hood. s16 00:01:05.920 --> 00:01:10.880 We'll share a demo that kind of shows what it feels like to build with this latest surface. s17 00:01:10.880 --> 00:01:15.840 And then we'll talk about lessons that we've learned from the field by taking this to customers. s18 00:01:15.840 --> 00:01:22.799 And finally, we'll close with what it feels like to build at the frontier as model capabilities evolve dramatically. s19 00:01:22.799 --> 00:01:26.960 Okay, with that, let's talk about how agentic surfaces have evolved. s20 00:01:26.960 --> 00:01:38.960 To set the stage, it's important to realize that AI progress is accelerating from back in the days when the transformer architecture was first defined to the scaling laws discovered by our founders s21 00:01:38.960 --> 00:01:41.600 to the models that have been released today. s22 00:01:41.600 --> 00:01:44.560 Every model release in incre and we all feel it. s23 00:01:44.560 --> 00:01:54.399 Every model release improves capabilities that the previous model did not have and we see this translate over to the tasks that we give models as well. s24 00:01:54.399 --> 00:01:57.600 The complexity of these have grown dramatically. s25 00:01:57.600 --> 00:02:01.439 Initially we used to only give them questions simple Q&A. s26 00:02:01.439 --> 00:02:08.319 We then started delegating tasks to them and now we let agents own entire outcomes. s27 00:02:08.319 --> 00:02:17.200 What this means is that as task complexity and model capability has improved, the agentic surfaces has to evolve with them. s28 00:02:17.200 --> 00:02:20.560 And we're going to talk about that in this section. s29 00:02:20.560 --> 00:02:29.840 So go back in time to when we first launched claw 3. We launched it alongside the first agentic surface called the messages API. s30 00:02:29.840 --> 00:02:33.760 This messages API was simply tokens in and tokens out. s31 00:02:33.760 --> 00:02:37.920 You gave information in and you got information and text completions out. s32 00:02:37.920 --> 00:02:48.400 As I mentioned earlier, task complexity grew over time and there was a need for the model to fetch information and manages context as it runs for longer and longer. s33 00:02:48.400 --> 00:02:52.080 And so we saw the invention of the agentic loop. s34 00:02:52.080 --> 00:03:02.480 This loop is something that every customer started building manually from scratch that calls claude, you know, runs the tools and manages its context and it was painstaking. s35 00:03:02.480 --> 00:03:09.440 And on top of that lived their product where it had the AI feature and it would call the agentic loop to accomplish some sort of task. s36 00:03:09.440 --> 00:03:11.120 But this was not all. s37 00:03:11.120 --> 00:03:26.480 In order to take this to production, there was a whole slew of production infrastructure challenges like session management, observability, credentials, hosting infrastructure, sandboxing, and more. s38 00:03:26.480 --> 00:03:34.239 These challenges were tedious to deal with and it didn't allow teams to focus on building what mattered most, their product. s39 00:03:34.239 --> 00:03:37.519 One of the core challenges here was the agentic loop. s40 00:03:37.519 --> 00:03:48.799 It was surprisingly complex to maintain which brings me to the second part, the second evolution of the agentic surface was the claude agent SDK. s41 00:03:48.799 --> 00:04:00.640 This agent SDK essentially packaged the harness that we know and love which is cloud code and it shipped with a built-in agentic loop along with file system access tools s42 00:04:00.640 --> 00:04:02.959 and a system for doing sandboxing. s43 00:04:02.959 --> 00:04:16.639 Your product would then embed the SDK and there are some primitives already provided for session management observability but you still had to handroll things like credentials and hosting infrastructure s44 00:04:16.639 --> 00:04:17.759 and much more. s45 00:04:17.759 --> 00:04:22.639 You had to figure out how to put this in a box and scale it for your customers. s46 00:04:22.639 --> 00:04:26.320 And I want to double tap on these production infrastructure challenges. s47 00:04:26.320 --> 00:04:31.040 And I'm going to enumerate some of these as questions to thought provoke. s48 00:04:31.040 --> 00:04:33.280 First, hosting and scaling. s49 00:04:33.280 --> 00:04:37.600 Where does the agent run and how long does the process live for? s50 00:04:37.600 --> 00:04:39.919 What scales under load and what doesn't? s51 00:04:39.919 --> 00:04:41.280 Session management. s52 00:04:41.280 --> 00:04:43.919 Where does the history and progress of sessions live? s53 00:04:43.919 --> 00:04:47.759 How can you have multiple concurrent agents running at scale? s54 00:04:47.759 --> 00:04:48.960 File system. s55 00:04:48.960 --> 00:04:54.639 How does Claude actually have access to create files and edit files? s56 00:04:54.639 --> 00:04:56.960 Fourth, execution isolation. s57 00:04:56.960 --> 00:05:02.080 Where does Claude actually run the code that it writes and how do we keep it secure? s58 00:05:02.080 --> 00:05:04.960 Fifth, credential credentials. s59 00:05:04.960 --> 00:05:12.240 How does Claude reach into your sensitive systems without actually getting exposed to the security tokens that you want to protect? s60 00:05:12.240 --> 00:05:14.400 And finally, observability. s61 00:05:14.400 --> 00:05:21.360 With all the comp complex agent orchestration going on, how do you figure out what's actually happening under the hood? s62 00:05:21.360 --> 00:05:31.919 These were production infrastructure challenges that most teams spend significant portions of their time doing instead of being able to focus on just their product, their task, and their context. s63 00:05:31.919 --> 00:05:34.800 Which is why we built cloud managed agents. s64 00:05:34.800 --> 00:05:37.360 The idea of cloud managed agents is simple. s65 00:05:37.360 --> 00:05:47.360 You own the product, you own the task, you own your context and you call cloud managed agents to get production infrastructure for your agents. s66 00:05:47.360 --> 00:05:59.520 So your product would call cloud managed agents and get a brain which is the agentic loop and claude itself and all the custom harnesses and evolutions built inside of it along with the hands s67 00:05:59.520 --> 00:06:05.759 which is a sandbox that spins up just in time for things like file system access and code execution. s68 00:06:05.759 --> 00:06:18.880 And of course all the bells and whistles that were difficult to maintain before with production infrastructure like credentials, session management, observability, hosting infrastructure, all of this is run by anthropic. s69 00:06:18.880 --> 00:06:25.120 And what's yours to build and run is your task, your context, and your domain knowledge. s70 00:06:25.120 --> 00:06:30.960 So if we step back and look at how this has evolved, we can kind of see the evolution over time. s71 00:06:30.960 --> 00:06:38.560 The messages API introduced the ability for the model the for you to interact with the model with tokens in and tokens out. s72 00:06:38.560 --> 00:06:50.800 The cloud agent SDK brought in a built-in agent harness for tasks and cloud managed agents covers everything that you need from your product and everything under that stack. s73 00:06:50.800 --> 00:06:57.680 So now I will hand it over to Isabella to talk a bit more about the engineering principles that drove how we built cloud managed agents. s74 00:06:57.680 --> 00:06:58.720 Over to you. s75 00:06:58.720 --> 00:06:59.520 Perfect. s76 00:06:59.520 --> 00:07:00.400 Thank you. s77 00:07:00.400 --> 00:07:06.400 And as GoGan walked through, a takeaway from that entire last section is that models evolve quickly. s78 00:07:06.400 --> 00:07:15.840 And so when our team set out to build manage agents, we drew inspiration from the lessons that our teams learned building effective agents and harnesses for these models to capture that evolution s79 00:07:15.840 --> 00:07:18.479 as it rapidly advances alongside us. s80 00:07:18.479 --> 00:07:31.120 So for the next few minutes here, I want to talk you through some of the fundamentals that underpin the design of claude manage agents and the lessons that our team learning learned along the way as we went about engineering this harness built for model evolution. s81 00:07:31.680 --> 00:07:42.319 Let's start with one of the very core principles that inspired our team to build manage agents and that is that harnesses encode assumptions about what claude cannot do on its own. s82 00:07:42.319 --> 00:07:48.880 This is things like resetting context and managing compaction and some of the other core primitives that you see here on the screen. s83 00:07:48.880 --> 00:07:55.680 The thing about these assumptions is that they have to be questioned frequently because they go stale as models improve. s84 00:07:56.240 --> 00:07:58.160 Let's dive into one concrete example. s85 00:07:58.160 --> 00:08:04.240 Back when sonnet 4.5 came out, it exhibited an interesting behavior that came to be known as context anxiety. s86 00:08:04.240 --> 00:08:08.960 What this is is that the agent literally got anxious at his approach to its context window limit. s87 00:08:08.960 --> 00:08:10.639 It started to wrap up tasks early. s88 00:08:10.639 --> 00:08:15.440 is to terminate work even when it actually had room left to spare in its context window. s89 00:08:15.440 --> 00:08:27.360 In order to accommodate for this behavior, what our team did is built in fixes into the harness itself, adding in context resets so that Sonnet 4.5 would be able to reset its context and continue working. s90 00:08:27.520 --> 00:08:33.360 But when Opus 4.5 came out, the interesting thing here was that this behavior went away entirely. s91 00:08:33.360 --> 00:08:40.560 Opus 4.5 no longer exhibited context anxiety which means that the fixes that we had added into the harness itself became dead weight. s92 00:08:40.560 --> 00:08:48.160 In fact, it became pure overhead, adding things like latency and causing issues with the cache being discarded incorrectly at times. s93 00:08:48.160 --> 00:09:03.120 The takeaway here is that we saw that the harness fixes were no longer needed and we're actually detracting from model performance with Opus 4.5. So when the model moves and the harness doesn't, it degrades the agent. s94 00:09:03.120 --> 00:09:11.600 What we've seen across these last couple of examples is that there's significant maintenance burden that comes with maintaining a harness that can keep up with Claude's rapid evolution. s95 00:09:11.600 --> 00:09:19.120 As we work with a range of enterprise teams that are building on top of Claude, we also see a range in the harnesses that are ready for this level of adaptation. s96 00:09:19.120 --> 00:09:26.480 Some of the harnesses that we see from customers are more agile and others are more rigid because they were built around older cloud models. s97 00:09:26.480 --> 00:09:37.200 What you don't want to do is have a stale harness that takes weeks or even months to migrate to a new model, especially with how model release cycles have been coming out shorter and shorter. s98 00:09:37.200 --> 00:09:48.320 Now, to build an effective harness, what this means is that you have to be designing for the model capabilities of tomorrow, anticipating what the future cloud models will be able to accomplish and building your harnesses for that capability. s99 00:09:48.320 --> 00:09:56.720 It also means that your harnesses have to be agile, making it easy to iterate for the model capabilities to capture them quickly as soon as they're ready. s100 00:09:57.040 --> 00:10:06.160 That brings me to claw manage agents which is a harness designed around a small set of primitives with individual components that you see here on the screen that are independent s101 00:10:06.160 --> 00:10:13.600 making it easily to swap them out and iterate upon them as individual pieces while keeping the overall architecture stable. s102 00:10:13.600 --> 00:10:18.079 Another key thing that cloud manage agents is designed around is longunning agents. s103 00:10:18.079 --> 00:10:29.920 Now when I look to internal products that are exciting and enthropic like claude code and claude tag which our team is really excited about and as I work with other enterprise teams we're also building exciting truly agentic products. s104 00:10:29.920 --> 00:10:38.959 A common pattern that I see is that these agents are becoming increasingly asynchronous and are tackling tasks that are increasingly complex and challenging. s105 00:10:38.959 --> 00:10:44.560 In order to design a harness that's actually able to capture those level of work, it needs to have a couple of things. s106 00:10:44.560 --> 00:10:50.320 It needs to be good at context engineering as the context will accumulate over those long horizon and bodies of work. s107 00:10:50.320 --> 00:10:55.600 It needs to be good at giving the agent a sandbox that's secure so the agent can actually take action within an environment. s108 00:10:55.600 --> 00:10:59.600 It has to be reliable so the agent can run on hours or even days at a time. s109 00:10:59.600 --> 00:11:07.279 And it also has to be able to do things like parallelize workflows so the agent can tackle multiple parts of a complex problem at once and many many more. s110 00:11:07.279 --> 00:11:15.519 So now what I want to do is dive into some of the engineering fundamentals that go into manage agents to make it possible to tackle some of those challenges. s111 00:11:16.399 --> 00:11:27.360 One of the core architectural decisions that went into manage agents that sets the foundation for the rest of the slides that we're going to walk through is the decision to decouple the brain from the hands of the agent. s112 00:11:27.360 --> 00:11:34.880 When our team first set out to build manage agents, we started by putting the agent loop and the tool execution in the same box in the same environment. s113 00:11:34.880 --> 00:11:42.160 What this meant was that the agent loop would be able to easily able to call tools and read in tool results because it had it right there in the same container. s114 00:11:42.160 --> 00:11:44.880 But then we ran into a series of limitations. s115 00:11:44.880 --> 00:11:48.959 This being that the container was blocking the agent being able to start its model reasoning. s116 00:11:48.959 --> 00:11:52.640 So the agent would be able to kick off until the container was fully set up. s117 00:11:52.640 --> 00:12:00.240 It also meant challenges for reliability because if one part of this component went down, the entire box of the agent would go down. s118 00:12:00.240 --> 00:12:03.120 Our solution to this was to decouple the two elements. s119 00:12:03.120 --> 00:12:08.959 Separating the brain or the agent loop from the hands or the tool execution environment of the agent. s120 00:12:08.959 --> 00:12:10.160 This meant several things. s121 00:12:10.160 --> 00:12:17.920 It meant improved reliability and it also meant that the brain could only spin up sessions when it actually needed it on demand. s122 00:12:19.839 --> 00:12:25.760 And now let's dive into some how some of these replaceable components meant keeping long running agents safe. s123 00:12:25.760 --> 00:12:36.720 First of all, if the sandbox or the hands of the agent died, because the brain was in a separate component, the brain could just spin up a new sandbox and retry and then continue as it left off. s124 00:12:36.720 --> 00:12:51.519 If the brain of the agent dies, we're actually going to walk through something in just a moment here about how everything that the agent does is logged into a durable session resource in a session log, which means that the brain of the agent can actually just read from that session log, go back into context, and resume exactly s125 00:12:51.519 --> 00:12:54.480 where it left off as well. s126 00:12:55.200 --> 00:13:05.760 This means that manage agents is designed around three core primitives and that is the agent and that is what your agent is especially defining what your agent does for your use case. s127 00:13:05.760 --> 00:13:13.120 This is a thing like the model that goes into your agent, the prompts, the tools, the skills, everything that makes your agent work for your particular use case. s128 00:13:13.120 --> 00:13:18.320 Next up is the environment and this is the container that the agent actually runs in. s129 00:13:18.320 --> 00:13:22.240 You can actually have multiple sessions run on the same environment definition. s130 00:13:22.240 --> 00:13:28.880 And you can even attach multiple sessions to run on the same environment at once, but each with its own isolated container instance. s131 00:13:28.880 --> 00:13:32.800 When you combine an agent with an environment, you get a session. s132 00:13:32.800 --> 00:13:42.880 What a session is is it's a durable resource persisted in the cloud of every single interaction that you have with the agent, which unlocks several things like observability, a longunning instance, reliability, s133 00:13:42.880 --> 00:13:46.480 all through this core architectural decisions. s134 00:13:47.600 --> 00:13:56.959 Now, when I work with many teams that are bringing an agent from a prototyping phase all the way to a production phase, one of the main challenges that we see is that reliability s135 00:13:56.959 --> 00:13:58.639 is a core concern. s136 00:13:58.639 --> 00:14:09.760 It's a different story to build an agent that runs on your laptop and serves you as a single user compared to when you actually want to deploy it in production and run it at scale for hundreds of thousands or even millions of users. s137 00:14:09.760 --> 00:14:18.240 You need to make sure especially if your agent is going to long run for long hours at a time that it's going to be reliable and can actually recover from tool failures. s138 00:14:18.240 --> 00:14:30.000 Manage agents because of the way it's designed around those three primitives that we just walked over is able to have four distinct session states and that is idle when your agent is waiting on user input, running when it's actually executing, s139 00:14:30.000 --> 00:14:35.839 rescheduling when it encounters an error and is going to retry or terminated if it's unreoverable. s140 00:14:35.839 --> 00:14:40.480 This means that the agent can always go back to an existing session and resume where it left off. s141 00:14:40.480 --> 00:14:45.440 And it also has a mechanism for it to recover from those failures in production. s142 00:14:46.720 --> 00:14:54.240 Another common theme that we see with designing effective agents, especially as model capabilities evolve is context engineering. s143 00:14:54.240 --> 00:15:04.000 Now, context engineering is something that our team has done a ton of research into because it is one of the things that separates an effective agent from an agent that gets lost in context rot. s144 00:15:04.000 --> 00:15:21.519 Context engineering is also difficult and with many traditional harness implementations the context window and the session are one and the same which means that claude if wants to come in and discard portions of the context that it's in current session run it doesn't have a mechanism to be able to recover pieces of that context back into its s145 00:15:21.519 --> 00:15:25.519 window if it loses it at one point in its current session run. s146 00:15:25.519 --> 00:15:38.160 However, because everything in manage agents is logged to a durable persisted session log resource, what this unlocks is that the harness can actually just read in slices of that context from the session log into its current window. s147 00:15:38.160 --> 00:15:49.120 If it then has Claude coming in and editing or discarding portions of that run, it can simply recover it by just rereading it from the session log because everything is persisted in that log resource. s148 00:15:50.800 --> 00:15:58.959 What this means is that what we see is increasingly developers are able to rely on portions of the manage agents harness that come with the harness itself. s149 00:15:58.959 --> 00:16:05.360 This is things like the agent loop, memory, observability, all that comes alongside building with cloud manage agents. s150 00:16:05.360 --> 00:16:13.279 It also exposes key areas for the developer to be able to customize and this is context managements and domain expertise. s151 00:16:13.279 --> 00:16:21.440 This is what separates a coding agent from a legal agent or go to market agent and again is what makes your agent truly ready for your users. s152 00:16:21.440 --> 00:16:29.199 For instance, with cloud code, cla code uses a set of tools like bash and gp on your laptop just like how developers do when they open up their terminal. s153 00:16:29.199 --> 00:16:33.519 But a go to market agent or a legal agent would need a vastly different set of tools. s154 00:16:33.519 --> 00:16:47.279 So by having this part of the manage agent harness managed by enthropic, what that means is that developer can developer can focus on their time designing the right system prompts, the right skills, and the right tools to make their agent truly work for their users. s155 00:16:47.279 --> 00:16:59.680 Now what I want to do is hand it over back to Gogan to walk you through one example in a live demo where you can see a customized agent for a production use case and how simple it is to build a production ready agent with manage agents. s156 00:16:59.680 --> 00:17:02.000 All right, thank you Isabella. s157 00:17:02.000 --> 00:17:04.720 Let's switch over. s158 00:17:05.199 --> 00:17:07.199 All right. s159 00:17:08.480 --> 00:17:15.839 So, now that you heard from Isabella how managed agents works under the hood, let's look at what it feels like to actually build with it. s160 00:17:15.839 --> 00:17:20.000 What does it feel like to actually create your own production grade agent from scratch? s161 00:17:20.000 --> 00:17:22.720 This is a semi-interactive uh demo. s162 00:17:22.720 --> 00:17:25.600 So, please bear with me here and follow along. s163 00:17:25.600 --> 00:17:32.799 So imagine that you're an engineer and you own a dashboard that basically contains all the key metrics for the services that you own. s164 00:17:32.799 --> 00:17:34.320 It's called Atlas. s165 00:17:34.320 --> 00:17:41.679 And one day you're just enjoying life and you start to see that the P99 latency suddenly starts spiking. s166 00:17:41.679 --> 00:17:43.520 It's 10x over baseline. s167 00:17:43.520 --> 00:17:45.440 So you have an incident on your hands. s168 00:17:45.440 --> 00:17:51.840 You see your logs, you see a bunch of text, and you have to figure out like what exactly is going wrong. s169 00:17:51.840 --> 00:18:03.120 Wouldn't it be nice if there was a site reliability engineering agent that could investigate all these tedious data points and come back to you with the root cause before even you open your dashboard? s170 00:18:03.120 --> 00:18:04.720 Well, that's what we're going to implement today. s171 00:18:04.720 --> 00:18:12.559 We're going to build this from scratch using cloud managed agents and I'll walk you through the steps using the primitives that Isabella shared earlier. s172 00:18:12.559 --> 00:18:14.559 Okay, so let's build it. s173 00:18:14.559 --> 00:18:19.039 The first primitive as mentioned earlier is the agent definition. s174 00:18:19.039 --> 00:18:25.679 You define what the agent does and everything that it needs to accomplish that task. s175 00:18:25.679 --> 00:18:29.360 So in this case I define the name as the S sur investigator. s176 00:18:29.360 --> 00:18:31.840 I give a model as cloud opus 48. s177 00:18:31.840 --> 00:18:39.360 I have a system prompt that kind of defines the t the the instructions for the agent on how to behave and tools. s178 00:18:39.360 --> 00:18:52.400 The agent tool set gives it a standard set of tools like bash gp and blob etc. And there's an MCP tool set that connects to my dashboard and allows it to pull specific things like deploys and the metrics that I showed earlier. s179 00:18:52.400 --> 00:18:55.679 That's step one, agent definition. s180 00:18:55.679 --> 00:19:00.080 Step number two, let's now define where does it run. s181 00:19:00.080 --> 00:19:02.880 This is the environment that we were mentioning before. s182 00:19:02.880 --> 00:19:15.679 Here we create an environment that is S sandbox and I configure it to run on the entropic cloud with the networking limited and allow us hosts only being the MCP server that I wanted to communicate to. s183 00:19:15.679 --> 00:19:19.600 This environment effectively stops claude from doing things that you didn't intend. s184 00:19:19.600 --> 00:19:22.720 So you can control boundaries here. s185 00:19:22.720 --> 00:19:26.640 Next, let's give it the logistics and the details that it needs to solve this task. s186 00:19:26.640 --> 00:19:29.520 And one of the key things is the application logs I showed earlier. s187 00:19:29.520 --> 00:19:32.400 You can upload files and skills just like this. s188 00:19:32.400 --> 00:19:35.280 And you can set it up so that it reads from those. s189 00:19:35.280 --> 00:19:37.919 So now we have the Asian definition. s190 00:19:37.919 --> 00:19:38.960 So we what it is. s191 00:19:38.960 --> 00:19:41.760 We have the environment which is where it runs. s192 00:19:41.760 --> 00:19:43.600 We have relevant evidence. s193 00:19:43.600 --> 00:19:47.760 And so now we set it up so we can kickstart a session. s194 00:19:47.760 --> 00:19:58.640 The session combines these durable resources into a new session and it specified the specifies the log point as a resource and allows the agent to kick off. s195 00:19:58.640 --> 00:19:59.600 That's it. s196 00:19:59.600 --> 00:20:06.000 what you have defined here is now living in the entropic cloud and this can be dynamic as needed in your application. s197 00:20:06.000 --> 00:20:13.360 So once we do this, we kick kick started the session and let's go back to the investigator agent earlier and say, "Hey, I have an incident. s198 00:20:13.360 --> 00:20:15.520 My checkout is like super high. s199 00:20:15.520 --> 00:20:17.200 Can you please investigate?" s200 00:20:17.200 --> 00:20:18.720 Claude immediately spins up. s201 00:20:18.720 --> 00:20:21.600 The brain spins up in cloud managed agents in the cloud. s202 00:20:21.600 --> 00:20:27.200 It uses the hands which is a sandbox to GP and finds specific details in the application logs. s203 00:20:27.200 --> 00:20:35.919 It uses relevant metrics using MCP tools and finds the recent deploys and isolates where the incident started. s204 00:20:35.919 --> 00:20:44.799 It does further investigation, finds the code diff and synthesizes this information to figure out a final root cause. s205 00:20:44.799 --> 00:20:47.679 Just like that, cloud manage agents was able to run in the cloud. s206 00:20:47.679 --> 00:20:52.640 We were able to define it in code and have running have everything running end to end. s207 00:20:52.640 --> 00:20:56.320 And to clarify, this is just one session. s208 00:20:56.320 --> 00:20:58.640 All of this is production infrastructure. s209 00:20:58.640 --> 00:21:05.840 So you can imagine multiple sessions kicked off by all of your users ready to go immediately. s210 00:21:05.840 --> 00:21:11.600 And we have a beautiful observability dashboard that allows you to see all of these sessions on demand. s211 00:21:11.600 --> 00:21:26.720 If I click into one of the observability dashboards on the cloud console, you'll find the different the exact event trace, the session logs for that event, including the tools that it used, the results of it, and any other agent messages that came into picture. s212 00:21:26.720 --> 00:21:36.159 With that, we were able to walk through what it feels like to build a cloud managed agent from scratch with just a piece with just a few lines of code. s213 00:21:36.320 --> 00:21:40.080 All right, perfect. s214 00:21:40.080 --> 00:21:44.240 So, with that, let's go into the next section. s215 00:21:44.240 --> 00:21:48.080 We've learned now what is how cloud manage agents works under the hood. s216 00:21:48.080 --> 00:21:52.799 We've learned what it feels like to build a production grade agent at scale using it. s217 00:21:52.799 --> 00:21:55.440 Now, let's talk about what we've learned from taking this to the field. s218 00:21:55.440 --> 00:22:00.640 what we've heard from customers and what we've learned generally building production scale agents. s219 00:22:00.640 --> 00:22:03.360 There's four lessons here and I'll start off with the first one. s220 00:22:03.360 --> 00:22:07.840 The first lesson is to keep the credentials away from your agent. s221 00:22:07.840 --> 00:22:16.159 A lot of customers ask me how do I make sure my agent doesn't read or see the environment file that contains all my security tokens. s222 00:22:16.159 --> 00:22:18.240 This is a very important aspect. s223 00:22:18.240 --> 00:22:24.880 We already get some of this because we as Isabel mentioned earlier, we separated the brain from the hands. s224 00:22:24.880 --> 00:22:30.320 So where the agentic loop runs is separate from where the tool execution happens. s225 00:22:30.320 --> 00:22:41.840 We took it a step further by introducing the concept of vaults where you can store security credentials in a secure way and they're decrypted only when needed at tool execution runtime. s226 00:22:41.840 --> 00:22:49.120 This way you can effectively keep credentials away from your agent and the model never sees your security tokens. s227 00:22:50.559 --> 00:22:51.039 Perfect. s228 00:22:51.039 --> 00:22:53.039 Now time for lesson two. s229 00:22:53.039 --> 00:22:59.919 And I'm sure if any of you in this room have built a productionready agent, latency has been one of the things that's top of mind for you. s230 00:22:59.919 --> 00:23:09.600 What we realized when we decoupled the brain from the hands of the agent as we talked about is that this actually unlocked a key benefit that really mattered for a lot of our customers building on manage agents out in the wild s231 00:23:09.600 --> 00:23:17.440 and that is that it improved latency significantly because the agent was no longer blocked on reasoning based on container setup. s232 00:23:17.440 --> 00:23:32.080 So if we go back to the first version with the coupled design where we had the harness in the same container and in one single box essentially the model wouldn't be able to start reasoning or outputting its first token until that container setup was fully complete which meant that it had delays in the latency especially for time s233 00:23:32.080 --> 00:23:36.799 to first token when we then decoupled the brain from the hands of the agent. s234 00:23:36.799 --> 00:23:38.000 This is what we get. s235 00:23:38.000 --> 00:23:43.360 Now we can have model reasons start immediately and we can run container setup in parallel. s236 00:23:43.360 --> 00:23:59.039 What this means is that the model can then run container setup so that the brain from the brain of the agent actually has the hands when it needs it or we can actually skip the container setup entirely if for this particular task we actually don't need the container setup in the first place. s237 00:23:59.120 --> 00:24:12.559 What we then saw when we tested this is that we saw 60% faster time to first token for P50 use cases or median use cases and over 90% improvements in latency for time to first token in P95 s238 00:24:12.559 --> 00:24:14.799 use cases. s239 00:24:17.760 --> 00:24:20.480 Lesson three is about session logs. s240 00:24:20.480 --> 00:24:30.480 A lot of customers asked us the question, how can I figure out what's actually going on in my agent under the hood and how can I make my agent better over time? s241 00:24:30.480 --> 00:24:36.400 Turns out the answer to both of these question lies in something that we called the session log or traces. s242 00:24:36.400 --> 00:24:41.919 The session log essentially contains events of everything that happened during an agent execution. s243 00:24:41.919 --> 00:24:49.120 So the user message, the model response, the tool executions, the results, everything is written playbyplay. s244 00:24:49.120 --> 00:24:56.000 Now if you surface a session log in a UI that the that users can see it provides observability. s245 00:24:56.000 --> 00:25:03.200 It turns out that the same session log also improves memory and self and and provides self-improvement for the agent. s246 00:25:03.200 --> 00:25:11.919 Memory essentially allows the agent to remember things about the user and session logs gives a history of past executions. s247 00:25:11.919 --> 00:25:19.039 And if you combine that with something that we call dreaming, it allows memory to be updated and improved over time. s248 00:25:19.039 --> 00:25:21.600 So the next time your agent runs, it gets better. s249 00:25:21.600 --> 00:25:24.880 We'll talk a bit more about this later. s250 00:25:25.840 --> 00:25:31.120 And now for the last lesson that we have for you today, that is security for tool execution. s251 00:25:31.120 --> 00:25:40.799 And this is something that we heard from a lot of enterprise teams that were wanting to build on manage agents is that it really mattered to them how they were able to control the environment where they ran tool execution. s252 00:25:40.799 --> 00:25:46.960 For a lot of teams that were very security conscious, they wanted to be able to have everything controlled in their own virtual private cloud. s253 00:25:46.960 --> 00:25:56.400 And because of the key decision that we made, decoupling the brain from the hands of the agent, what we actually get is that the hands can run anywhere, including in your virtual private cloud. s254 00:25:56.400 --> 00:25:59.279 So the feature that we released called self-hosted sandboxes. s255 00:25:59.279 --> 00:26:12.720 We built this from an engineering perspective because of the feedback that we heard and essentially made it available to have customers control their sandbox control plane exactly for their own execution environments and to have tools run exactly under their own policies. s256 00:26:12.720 --> 00:26:15.600 Another feature that we unblocked is MCP tunnels. s257 00:26:15.600 --> 00:26:23.840 And this is from teams that were saying that they wanted to expose MCP servers to their agent but didn't want to have their MCP servers running over the public internet. s258 00:26:23.840 --> 00:26:24.559 for those teams. s259 00:26:24.559 --> 00:26:33.679 Essentially with MCP tunnels, they can have their MCP servers run only within their private network and only making outbound calls to the clawed agent loop. s260 00:26:34.720 --> 00:26:45.440 And so with everything that we've talked through and those four lessons that Goan and I just walked over, we talked about how manage agents is helping you build for this iterative capability and helping you follow along with model evolution. s261 00:26:45.440 --> 00:26:58.480 What I now also want to talk you through is how those fundamentals are built and expanded upon with some of our most exciting frontier features to show you how cloud manage agents will continue to evolve and capture the model capabilities of tomorrow. s262 00:27:01.360 --> 00:27:02.559 cloud manage agents. s263 00:27:02.559 --> 00:27:14.320 We touched on just the tip of the iceberg of the features that are available today and these cover the fundamentals, but we're also excited to see how the harness evolves as model capabilities evolve. s264 00:27:14.320 --> 00:27:27.600 And I'm excited for some of the new features that we've been experimenting with like scheduled deployments, self-hosted sandboxes, multi- aent orchestration, dreaming outcomes, memory, and more. s265 00:27:27.600 --> 00:27:29.679 We won't have time to cover all of them. s266 00:27:29.679 --> 00:27:34.799 So, I'm going to talk about two of our favorites, which is dreaming and outcomes. s267 00:27:34.799 --> 00:27:36.400 Let's start with dreaming. s268 00:27:36.400 --> 00:27:42.159 As I mentioned before, we have access to the transcripts or the session logs from the agent's daily sessions. s269 00:27:42.159 --> 00:27:45.919 And the agent also has a current memory state. s270 00:27:45.919 --> 00:27:57.600 What we found is that as models have evolved and become more capable, if you feed the transcripts and the memory state as a periodic batch process with what we call dreaming, s271 00:27:57.600 --> 00:28:11.039 it allows us to extract new insights and new organized structures that essentially feed back and edit the memory as needed to make the next day's agent sessions automatically much more intelligent. s272 00:28:11.039 --> 00:28:17.840 This is how we're seeing self-improving agents as they execute more and more over time. s273 00:28:18.240 --> 00:28:25.279 Dreaming and memory we feel are just two cornerstones of a new frontier unified memory system. s274 00:28:25.279 --> 00:28:32.080 Memory gives the agent the ability to remember things across the user that's specific to uh its use case. s275 00:28:32.080 --> 00:28:34.799 Dreaming allows agents to self-improve. s276 00:28:34.799 --> 00:28:44.399 But we see a new form of memory emerge that is organizational scale and that kind of illustrates and stores the team's runbooks and details. s277 00:28:44.399 --> 00:28:54.320 And we're and we believe that this is just the initial areas in which we can see harnesses evolve towards as models become more capable. s278 00:28:55.440 --> 00:28:56.480 I also love dreaming. s279 00:28:56.480 --> 00:29:00.320 But one of my other favorite features is something called outcomes. s280 00:29:00.320 --> 00:29:06.000 What you get with an outcome is that what we allow users to essentially define is success criterias for their agents. s281 00:29:06.000 --> 00:29:07.200 You define a rubric. s282 00:29:07.200 --> 00:29:10.799 You say what means that the agent is actually able to complete the task successfully. s283 00:29:10.799 --> 00:29:23.919 You define failure cases and then outcomes essentially starts a separate greater agent that runs alongside your agent loop and looks at whether that agent was actually able to accomplish the task based on your defined success criteria. s284 00:29:23.919 --> 00:29:27.360 What this then does is that the agent will execute the task at hand. s285 00:29:27.360 --> 00:29:30.720 It will then look at that greater and check across the rubric that you defined. s286 00:29:30.720 --> 00:29:39.200 If the greater determines that the agent was not able to complete the task, it'll keep trying until it reaches that success criteria that you have defined for your agent. s287 00:29:39.200 --> 00:29:52.399 What really excites me about outcomes is that we're moving more towards a world where we can have an agent understand what success actually means for a task and have a mechanism to keep iterating it, which gives us more reliability that the agent can actually complete the outcomes. s288 00:29:52.399 --> 00:30:06.159 and we can start to unlock a new set of tasks that were not possible just a couple of months ago as models continue to evolve and can accomplish outcomes or tasks that are increasingly complex especially as claude achieves new levels of intelligence. s289 00:30:07.600 --> 00:30:18.640 So across everything that Goagan and I have talked about today, what manage agents is trying to do is to close the gap between what products offer today on many surfaces with static harnesses s290 00:30:18.640 --> 00:30:22.399 and what models can actually do. s291 00:30:22.480 --> 00:30:32.559 What we see as cloud models and other models essentially evolve alongside this exponential trajectory is that harnesses have become the limiting factor to what models can achieve. s292 00:30:32.559 --> 00:30:47.440 And so with manage agents, with the core architectural foundation that we offer to developers, as long as with all of these new exciting features that we're continuing to build, what we're trying to do is to close that gap so that products can get closer to what models can actually achieve today. s293 00:30:47.440 --> 00:31:03.520 And so with that, I hope all of you walk out of this room learning something new about how our team went about building manage agents and how cloud manage agents is structured to capture frontier intelligence as models continue to evolve and be a harness that's production ready for real workloads. s294 00:31:03.600 --> 00:31:08.240 Thank you all so much today for being here and for listening to our talk.