WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=cI7zfqusmFU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/cI7zfqusmFU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.480 --> 00:00:13.440 Hi everyone. s2 00:00:13.440 --> 00:00:17.600 This talk will be about how to run agents reliably in production. s3 00:00:17.600 --> 00:00:25.600 It will not be about the eile part, but it will be about all the other things you need to get going in order to run agents resiliently. s4 00:00:25.600 --> 00:00:28.240 So the infrastructure layer basically. s5 00:00:28.240 --> 00:00:33.120 I want to set the scene with this uh quote of Andre Apathy of last week. s6 00:00:33.120 --> 00:00:39.360 It describes that the way we interact with agents and LLMs has been evolving in three waves. s7 00:00:39.360 --> 00:00:49.039 The first wave was an LLM being something like a website where we go to we ask it a question, it thinks for a few seconds and then gives us a response. s8 00:00:49.039 --> 00:00:52.399 The second wave was going towards agents. s9 00:00:52.399 --> 00:00:54.719 It was an app that we download to our computer. s10 00:00:54.719 --> 00:01:00.559 It has some tools at its disposal and it can do some work with our interaction. s11 00:01:00.559 --> 00:01:06.080 Now the third wave will be going more and more towards persistent and asynchronous entities. s12 00:01:06.080 --> 00:01:16.720 So agents being longunning processes in our infrastructure with access to tools and other agents around the organization and context. s13 00:01:17.040 --> 00:01:29.600 And so as our use cases are evolving more and more from single agents to agentic platforms that connect parts around uh the organization our infrastructure layer should also evolve with that. s14 00:01:29.600 --> 00:01:39.680 So when we look at the types of tools that are currently out there to implement agents, a lot of innovation has been done on sites such as agent SDKs and memory. s15 00:01:39.680 --> 00:01:51.439 And agent SDKs are really cool to implement PC's and get started quickly, but they don't necessarily help with like connecting the distributed bits around an organization. s16 00:01:51.439 --> 00:01:57.439 And if you want to implement more complex agentic systems, you actually need all of those things. s17 00:01:57.439 --> 00:02:03.600 So that is the layer that you see below here where um you have to deploy extra infrastructure. s18 00:02:03.600 --> 00:02:17.520 Uh you need to write things like retry logic, recovery logic and all of that is actually pretty complex to get right but completely necessary to run longunning stateful and distributed processes in production. s19 00:02:17.520 --> 00:02:21.599 So today I want to talk about an open-source framework called restate. s20 00:02:21.599 --> 00:02:27.280 And you can see it a bit as a flexible durable foundation that lets you build any backend. s21 00:02:27.280 --> 00:02:36.000 So it's not specific for agents but a as agents are also just a type of a backend uh it also works well for them. s22 00:02:36.000 --> 00:02:49.120 The ideas behind restate come from Apache Flink which is a popular distributed stream processing engine and also from some of the exarchitects behind Meta Score event infra. s23 00:02:49.120 --> 00:02:51.519 So what are the ingredients in restate? s24 00:02:51.519 --> 00:02:54.000 Basically four parts. s25 00:02:54.000 --> 00:02:58.480 First of all, it makes sure that a single run of an agent is resilient. s26 00:02:58.480 --> 00:03:01.440 This is called durable execution in the industry. s27 00:03:01.440 --> 00:03:05.840 Think about things like when an agent runs for a week and then crashes. s28 00:03:05.840 --> 00:03:11.200 We want to be able to bring it back and let it continue exactly at the point where it failed. s29 00:03:11.200 --> 00:03:14.720 We don't want it to start over from the beginning. s30 00:03:14.720 --> 00:03:20.159 Another um area here is running many concurrent sessions in parallel. s31 00:03:20.159 --> 00:03:31.440 Imagine running thousands of concurrent agent sessions at the same time and needing needing to make sure that state is always consistent and that different agents don't interfere with each other. s32 00:03:31.440 --> 00:03:38.000 And then going more towards things like communication between agents, between agents and MCP servers and other tools. s33 00:03:38.000 --> 00:03:50.799 And finally also control, making sure that when an agent for example uh is doing um something you don't want it to continue or when it's stuck being able to actually cancel or kill the execution. s34 00:03:51.280 --> 00:03:54.400 So the way that you can think of it is as follows. s35 00:03:54.400 --> 00:03:58.480 Restate is basically a server which runs in front of your agent service. s36 00:03:58.480 --> 00:04:09.680 So as a separate component it sits there a bit like a like a message broker or a proxy and when there's a request for your agent restate proxies the request to the service s37 00:04:09.680 --> 00:04:21.199 and pushes it to the service basically and from that moment there's a connection open connection between restate and the agent and that connection will basically be a bit like a lifeline for the agent. s38 00:04:21.199 --> 00:04:33.040 So as the agent is doing stuff, it sends events over to restate and restate will use that journal of events to recover the process after a failure. s39 00:04:33.040 --> 00:04:49.040 So from a slightly higher level um explanation, you could say that it's turning a normal function in your application into something that is long running, durable, and stateful without having to do um a lot of the complex things you otherwise need to do for this. s40 00:04:49.040 --> 00:04:52.160 So my talk today will be mainly a demo. s41 00:04:52.160 --> 00:04:56.479 So I'll be showing you um a research agent that is connected to Slack. s42 00:04:56.479 --> 00:05:03.840 Imagine we are like working at some company and we want to make an Slack agent available to all of our employees. s43 00:05:03.840 --> 00:05:12.639 So if I go here into Slack then can I can here in this channel for example ask what is new in AI. s44 00:05:12.800 --> 00:05:15.520 Now let's have a look at what it's doing under the hood. s45 00:05:15.520 --> 00:05:19.600 So if I go back here, I have here the restate uh UI. s46 00:05:19.600 --> 00:05:22.800 This is a bit like a cockpit for your agents. s47 00:05:22.800 --> 00:05:31.280 So you can see a registry of all the agents that are currently registered and you can also see for example which execution is currently happening. s48 00:05:31.280 --> 00:05:36.000 So here is the deep research agent that I spinned up a few seconds ago. s49 00:05:36.000 --> 00:05:38.080 We can see what it's currently doing. s50 00:05:38.080 --> 00:05:43.199 Now it called first an LLM and then it sent me an answer via Slack. s51 00:05:43.199 --> 00:05:46.560 This first LLM call was a planner agent. s52 00:05:46.560 --> 00:05:53.840 So what it did is it planned the research and sent me um a list of subtopics that it wants to research. s53 00:05:53.840 --> 00:06:01.919 Now if I press here approve then this will unblock the workflow and will spin up a set of parallel research agents. s54 00:06:01.919 --> 00:06:05.520 So this is basically like the classical deep research workflow, right? s55 00:06:05.520 --> 00:06:22.400 You have a planner then a set of subress research agents and then finally someone uh who writes a report on this like a writer agent and so this journal you see here on the left that is basically the events that get sent from the agent to the restate server s56 00:06:22.400 --> 00:06:30.720 and if this now crashes at some point this journal is what will be used to uh recover the execution to the point where it failed. s57 00:06:30.720 --> 00:06:33.120 I don't know if uh there were some errors. s58 00:06:33.120 --> 00:06:36.319 I injected a bit of like tool errors in here. s59 00:06:36.319 --> 00:06:53.280 Yeah, here you can for example see that um the sub agent first did an LLM call then started doing some web searches and eventually uh one of the web searches didn't go through because the API was down and then you see here on the right how it got retrieded and eventually completed successfully. s60 00:06:53.280 --> 00:06:58.720 So instead of starting over, it uses the journal to recover the progress. s61 00:06:58.720 --> 00:07:02.720 Let's now have a look at what this looks like in code. s62 00:07:02.800 --> 00:07:13.599 So the basic unit of how you implement applications in restate is by writing HTTP handlers and those handlers become durable by using the restate SDK. s63 00:07:13.599 --> 00:07:24.319 So here in this case we have here our deep research handler and here as a first argument we have a restate object context and the way you can imagine that is basically as that s64 00:07:24.319 --> 00:07:26.880 uh connection to that restate server. s65 00:07:26.880 --> 00:07:34.000 whenever I do an action on this uh restate object, it will lead to an event being sent to restate. s66 00:07:34.000 --> 00:07:42.560 So for example, when I did that planner LLM call, what actually happened under the hood was it executed here this Python function. s67 00:07:42.560 --> 00:07:54.080 This is just a simple light um light lm like LLM call and the way I made it durable is by wrapping it in restate.run. s68 00:07:54.879 --> 00:08:05.199 So what happens is by doing these durable steps if this fails somewhere here two hours or two months later it will recover to exactly that point. s69 00:08:05.199 --> 00:08:07.280 So that's the idea of durable execution. s70 00:08:07.280 --> 00:08:10.400 You're always able to recover a process to where it was. s71 00:08:10.400 --> 00:08:15.120 You can also use that for other things not necessarily for failure recovery. s72 00:08:15.120 --> 00:08:22.240 For example, imagine we want to ask a human to approve something and this approval might take weeks or a month. s73 00:08:22.240 --> 00:08:34.479 this process needs to be able to um to survive restarts and redeploys uh over those kind of long periods of time and so with durable execution you can actually also s74 00:08:34.479 --> 00:08:40.320 uh suspend a function and let bring it back when it's able to make progress. s75 00:08:40.320 --> 00:08:49.920 So in the case of a human approval what we do here is basically we we create a durable promise which lives in that journal a bit like a suspension point. s76 00:08:49.920 --> 00:08:58.800 Then we ask uh a human to click that button in select as I showed in the beginning and while we are waiting this process actually suspends. s77 00:08:58.800 --> 00:09:06.720 So if it's running on serverless this is not using uh execution uh time on our functions. s78 00:09:06.720 --> 00:09:11.920 Once the response comes in this then gets unblocked and can continue where it left off. s79 00:09:11.920 --> 00:09:14.480 So what we see here is a bit like a workflow. s80 00:09:14.480 --> 00:09:18.080 It's a set of steps that get executed durably. s81 00:09:18.080 --> 00:09:29.120 But when we think about agents and also the way that Karpathy described it in the tweet, it's more like a persistent stateful entity that lives for a longer period of time that has some memory. s82 00:09:29.120 --> 00:09:34.320 Um, so a workflow is not the nicest way to model this kind of thing. s83 00:09:34.320 --> 00:09:40.320 So the way that we can model this in restate is by using something called a virtual object. s84 00:09:40.320 --> 00:09:44.320 So imagine in the use case that I'm showing this slack research agent. s85 00:09:44.320 --> 00:09:53.360 Imagine that I don't want to wait for 10 minutes to give it some follow-up context or maybe I think about something else that I should have told it. s86 00:09:53.360 --> 00:10:00.959 Um I want to actually be able to interact with it, not wait till that research is finished before I can send a follow-up. s87 00:10:00.959 --> 00:10:04.399 And so this is basically what a virtual object in restate is. s88 00:10:04.399 --> 00:10:06.320 It's a bit like a stateful actor. s89 00:10:06.320 --> 00:10:09.360 It has a unique ID, for example, a session ID. s90 00:10:09.360 --> 00:10:16.560 It has uh some key value states that is isolated for that specific session that you can write to. s91 00:10:16.560 --> 00:10:27.839 Uh imagine for example your history of messages and it also has like a set of handlers that can execute durable functions uh for this session. s92 00:10:27.839 --> 00:10:36.240 So here the way I implemented this use case that I mentioned of interacting with a running process is as follows. s93 00:10:36.240 --> 00:10:39.839 This is a um a bit a session controller. s94 00:10:39.839 --> 00:10:46.880 Again, it has like this restate object context at its disposal to do things in a recoverable way. s95 00:10:46.880 --> 00:10:49.440 Uh it can write to this session store. s96 00:10:49.440 --> 00:10:52.720 Here it I'm retrieving the chat history. s97 00:10:52.720 --> 00:11:02.320 And one thing that's interesting there is that in order to run these kind of sessions in very high uh paralyzed ways, so thousands of sessions at the same time, s98 00:11:02.320 --> 00:11:06.640 we need to make sure that agents do not interfere with each other. s99 00:11:06.640 --> 00:11:13.680 Imagine I'm sending two messages on Slack and now two agents are actually overwriting each other each other's session state. s100 00:11:13.680 --> 00:11:19.120 To prevent that, this will guarantee that only one execution is running at a time. s101 00:11:19.120 --> 00:11:24.959 So a second execution will be cued behind the current one. s102 00:11:26.959 --> 00:11:31.760 Then let's have a look at how we implement this like interacting with another execution. s103 00:11:31.760 --> 00:11:40.000 So an execution in reset has a unique identifier and you can use that identifier to connect to it from other processes. s104 00:11:40.000 --> 00:11:51.440 for example, to retrieve uh the output, but also to cancel it or maybe to signal it being injecting a bit of state into an already running agent loop. s105 00:11:51.440 --> 00:12:02.880 And so this is like a very flexible type of um uh capabilities that you can do to implement things like for example signaling an already ongoing agent loop. s106 00:12:02.880 --> 00:12:13.200 So what we do here is if there is a current execution ongoing then we will ask an LLM is this like something that is relevant for the current agent loop. s107 00:12:13.200 --> 00:12:26.079 If that is the case inject this via a signal if it's not really relevant for what we're currently doing then cancel what you're currently doing and start over again with this new information. s108 00:12:26.160 --> 00:12:29.040 And so this goes a little bit further than workflows. s109 00:12:29.040 --> 00:12:38.000 it goes a bit more towards like writing persistent stateful entities that can interact with each other and have memory at uh their disposal. s110 00:12:38.000 --> 00:12:40.639 So let me show you uh how this works. s111 00:12:40.639 --> 00:12:49.680 So here if I now ask again what is new in AI and I wait a few seconds then it should respond again with a plan. s112 00:12:49.680 --> 00:12:58.079 Um and then I can say for example some extra info focus on frontier models let's say. s113 00:12:59.120 --> 00:13:05.360 So once I have the plan I will inject that bit of extra state. s114 00:13:06.720 --> 00:13:09.920 Now let's look at the UI of what this is now doing. s115 00:13:09.920 --> 00:13:13.120 So here I have that controller which I just showed. s116 00:13:13.120 --> 00:13:19.200 It started calling an LLM to classify uh this new input. s117 00:13:19.680 --> 00:13:26.560 Once this comes back, it will probably decide that it should signal it because it's it's still relevant to the research it's currently doing. s118 00:13:26.560 --> 00:13:31.200 So this inject that new message into the ongoing agent loop. s119 00:13:31.200 --> 00:13:34.560 So let me show you in the deep research agent again. s120 00:13:34.560 --> 00:13:46.079 Um so first it called an LLM then asked us then we injected this uh new message of focus on frontier models and then it uh took that into account and started s121 00:13:46.079 --> 00:13:47.440 over again. s122 00:13:47.440 --> 00:14:00.160 Here I can now for example also say something like uh forget about that research AI policy. s123 00:14:00.160 --> 00:14:11.600 And if I send this then the coord coordinator will um decide to cancel the ongoing run and start a new one that will research this new topic. s124 00:14:11.600 --> 00:14:20.240 And so this cancellation is basically like a signal that gets um sent down the stack of or the call chain. s125 00:14:20.240 --> 00:14:30.320 So if my agent was already spinning up sub agents first those sub aents would be cancelled then uh the controller itself and like that it would basically rewind the stack s126 00:14:30.320 --> 00:14:34.480 and give agents also the ability to roll back. s127 00:14:34.480 --> 00:14:34.639 Okay. s128 00:14:34.639 --> 00:14:42.160 Okay, so this went a bit more into the direction of like stateful persistent entities that we can interact with over longer periods of time. s129 00:14:42.160 --> 00:14:50.480 Now the last part of the demo that I want to show is um going more towards like being able to write highly customized applications. s130 00:14:50.480 --> 00:15:06.880 Imagine that we deploy this in production but then a few months later a new model provider brings out a new model for example fabulous and even though the model is very good it's also very expensive and we notice that this research agent is actually starting to cost a lot. s131 00:15:06.880 --> 00:15:17.600 These kind of uh things that pop up halfway through a project require you to then deploy a a lot of new extra infra or like find a good way to solve this. s132 00:15:17.600 --> 00:15:21.279 This is the kind of things that Restate really excels at. s133 00:15:21.279 --> 00:15:25.680 It doesn't really peg you into a specific way of how you should write your application. s134 00:15:25.680 --> 00:15:35.360 It basically gives you like a durable programming model that lets you implement an application in the way that fits for you and also extend it if necessary. s135 00:15:35.360 --> 00:15:42.079 So first I showed this um LLM call in the first example as an inline step. s136 00:15:42.079 --> 00:15:45.920 It was just a Python function that got persisted. s137 00:15:45.920 --> 00:15:51.199 But imagine this use case that we want to actually have a bit more control over those LLM calls. s138 00:15:51.199 --> 00:15:56.160 For example, what you can do is then pull this out into its own handler. s139 00:15:56.160 --> 00:16:03.440 And this handler can now do things like for example a policy check and then uh do the LLM call. s140 00:16:03.440 --> 00:16:17.279 And the other agents instead of doing this LLM call inline can now use restates like distributed communication primitives to actually just call this LLM gateway instead of doing it as an inline step. s141 00:16:17.279 --> 00:16:24.959 And this service fabric that lets you communicate between agents also gives you some um things like flow control. s142 00:16:24.959 --> 00:16:33.600 So we can for example say one department is only allowed to run 300 calls to this LLM gateway at the same time. s143 00:16:33.600 --> 00:16:37.040 So the reason why I showed this was just to show you a bit like that. s144 00:16:37.040 --> 00:16:39.839 Uh it's basically just a a resilient foundation. s145 00:16:39.839 --> 00:16:50.000 It makes sure that your process can uh recover from even a more advanced types of infrastructure failures, things like network partitions and zombie failures. s146 00:16:50.000 --> 00:16:57.680 And um it gives you like tooling to extend and customize as your use case grows. s147 00:16:57.839 --> 00:17:08.160 Let's go back to the slides to have a little more of an idea of how this thing is actually implemented on the inside because it's actually a pretty interesting s148 00:17:08.160 --> 00:17:11.439 um design or architecture. s149 00:17:11.439 --> 00:17:18.959 So the way it's implemented is basically by having a a event-driven distributed log implementation. s150 00:17:19.039 --> 00:17:34.080 So inside the box you basically on one side have the clients on the other side the services and inside the box is a log which persists all those journal events and an event loop and that event loop basically gets the events from the service s151 00:17:34.080 --> 00:17:35.440 based on what the event is. s152 00:17:35.440 --> 00:17:42.799 It either persists some state in the embedded state store or it sets a timer or it sends a request to another agent. s153 00:17:42.799 --> 00:17:50.799 And by doing that you basically have a durable um foundation for whatever an application is doing. s154 00:17:50.799 --> 00:17:59.360 The design of this distributed log is heavily inspired by the way that the core event infra layer at meta works. s155 00:17:59.360 --> 00:18:02.799 Uh it's basically like an iteration on top of that. s156 00:18:02.799 --> 00:18:11.919 Um and some of those architects are now have designed that for restate as an more generic solution that is available in open source. s157 00:18:11.919 --> 00:18:16.799 There are two important things related to this architecture that make it interesting. s158 00:18:16.799 --> 00:18:19.520 The first one is that it works as a push model. s159 00:18:19.520 --> 00:18:34.799 So whereas most workflow orchestrators actually pull for new tasks um for pull from the workflow server, restate actually pushes the invocations and the benefit you get from that is that it has a much lower latency. s160 00:18:34.799 --> 00:18:48.400 So you can use these kind of workflow guarantees in functions around your application and uh have like a latencies of for example 45 milliseconds p99 for like a 10-step workflow. s161 00:18:48.400 --> 00:18:58.799 Pushing invocations also works very well for serverless because they require you to basically uh send the request and wake up the function. s162 00:18:58.799 --> 00:19:03.120 So this design that I show here includes everything you need. s163 00:19:03.120 --> 00:19:08.480 It includes uh as well that state store where we were embedding the state as the UI. s164 00:19:08.480 --> 00:19:14.880 It's a single binary so it's pretty easy to operate as well to run it in like a highly available way. s165 00:19:14.880 --> 00:19:21.120 You just spin it up multiple times and let it snapshot to object storage. s166 00:19:21.440 --> 00:19:24.400 So restate has six different SDKs. s167 00:19:24.400 --> 00:19:29.919 We also have integrations for most of the popular agent frameworks out there. s168 00:19:29.919 --> 00:19:42.240 And of course, because it's just like a flexible layer, you can also just use any LLM SDK and implement custom agents by just wrapping some steps into uh these SDK constructs. s169 00:19:42.240 --> 00:19:43.280 So, it's open source. s170 00:19:43.280 --> 00:19:44.320 You can self-host it. s171 00:19:44.320 --> 00:19:54.559 We also have a BYOC offering where we deploy restate in your cloud account and uh that gives you the benefit that data doesn't leave your cloud account. s172 00:19:54.559 --> 00:19:58.720 Otherwise, there's also a managed cloud offering. s173 00:19:58.799 --> 00:20:01.039 This was mainly what I wanted to show. s174 00:20:01.039 --> 00:20:05.440 If you want to explore the code a bit further, there is here this the GitHub repo. s175 00:20:05.440 --> 00:20:07.039 It's publicly available. s176 00:20:07.039 --> 00:20:11.679 If you like the project, then have a look at the restate repo itself. s177 00:20:11.679 --> 00:20:18.799 We are hiring across the board for all sorts of roles going from engineering to marketing, especially also here in the Bay Area. s178 00:20:18.799 --> 00:20:23.360 So, if you're interested in that, uh, then definitely check out our careers page. s179 00:20:23.360 --> 00:20:30.640 and I will be outside in front of the conference hall here if you want to ask any questions or learn more about restate. s180 00:20:30.640 --> 00:20:33.280 Thank you very much. s181 00:20:47.789 --> 00:20:49.789 [music]