The Human Is an Async API — Melanie Warrick, Temporal

AI Engineer · 19 min · 234 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:13All right.
00:13Well, I'm here to talk to you today about this demo and I've already stepped outside the box they told me not to step outside of.
00:19Anyways, I'm here to talk to you today about an ice cream delivery demo that I've got for you and ultimately I'm here to talk to you about Temporal.
00:26So, you're going to hear me pitching about the company that I work for and specifically in a way that will be about this ice cream delivery that's happening as you can see online right now.
00:37So, what's happening?
00:38Well, this is Ziggy.
00:39Ziggy's Ziggy's is our mascot.
00:41I love Ziggy.
00:42And Ziggy has an ice cream store, not really, but let's say he does down in the Ferry Building.
00:47Currently working hard delivering all kinds of ice cream.
00:50Now, what's happening on this screen is that um we have multiple agents that are currently coordinating the delivery that's happening for deliver for Ziggy's ice cream service.
01:01We have a fleet agent and a customer agent and a dispatch agent all working in coordination to make sure that the ice cream goes where it needs to go.
01:10In this specific demo, what I've got up here is showing you we have 80K integrated with Temporal.
01:17So, 80K is a um solution that Google's providing to help you orchestrate agents or multiple agents um and in this case it's specifically helping to coordinate the activities in relation to these agents.
01:30And Temporal is baked into it.
01:32We are integrated into it so that we can help track the state of what's happening with this kind of delivery.
01:38So, Temporal actually helps you track your state and manage that state and we provide a UI that allows you to see what's going on on that screen.
01:48So, this is what you're seeing over here.
01:50And in that UI it's showing you the event history.
01:53There's this this is the parent workflow for all the activities that are happening for Ziggy right now in its delivery suite.
01:59And in this world, sometimes what we'll have in multiple agents is we need to have human in the loop.
02:06So, let's say a customer made an order and the customer said, "Hey, I think I changed my mind about that order.
02:12So, I'm going to submit a change."
02:14And of course, when you do submit that change, it will not it if you depending on how you build it out, it could actually break your system.
02:24But, heaven forbid you have it break your system when somebody has a change to submit.
02:29In this case, we have an order and this order is currently pausing driver A. And so, driver A is waiting for that update to come through.
02:40The system is running while this one specific change is only pausing one part of your system.
02:48That one change coming from a human.
02:51And another human needs to make a call and whether or not they want to accept that.
02:56So, they say approve, and you'll watch that as I mentioned, the rest of the system kept running, orders were coming in, they're being filled, but in this specific question that came up about changing that order,
03:09the update is getting processed and should go through in a second.
03:16I suspect internet is part of it.
03:18You can see it does get processed, the update is read, and that order will now move on to its next location, which is to the Oracle Park.
03:25And there goes little A delivering the order for you.
03:28So, there you go.
03:29That is durable execution with Temporal and 80K integrated.
03:34All right.
03:35I'm here to talk to you about the human is an async API.
03:41Let's go into First off, we've been hearing about agentic loops, right?
03:46You can reason, act, and observe.
03:49That is our whole thing that everybody's been talking about when they're talking about building out these agents and these agents running as autonomously as we possibly can.
03:58Um there's this kind of loop that we're looking at, right?
04:01We've heard other words for this where we're thinking, we're acting, we're perceiving.
04:05Um and that's that's the essence of it.
04:09It can be a for loop, it can be a while loop.
04:12Most of this year we've been hearing people tell us about the harness and what the harness is supposed to be to help us get those agents into production.
04:21So, the harness I'm not going to sit here and tell you I I am the expert on the harness.
04:26Um I'm going to tell you from everything I've researched, I'm trying to give you at least a map to a degree of what I understand so far with the understanding that there's a lot of debate about this.
04:35I've heard that the harness is possibly the model is is part of all of this or it's not.
04:40I think there's there's debates around whether the model's in it, whether tools are in it, memory's in it, uh guardrails, security, but ultimately it's all about getting these agents
04:50with structure so that they can work reasonably well when they're put out into production.
04:55And part of that picture should be durability.
05:00The system that I showed you, the demo that I just showed you, this is what it is.
05:04It's got these loops.
05:05We've got a loop for the fleet agent, we got a loop for the customer agent, we got a loop for the dispatch agent.
05:11The fleet and the customer are doing research.
05:13One's doing it on the actual drivers using thing tools like Google Maps.
05:18Um the other one is doing research on the orders like Google with tools like Google Search.
05:23While the dispatch agent is taking that input and making decisions with it.
05:28All right.
05:29So, that's great that these activities are happening with these three loops and we've got the human potentially providing input in part of this structure.
05:38But what's important is that we need to have some durability in that system to make sure that it stays up and running and doesn't fall over.
05:45And that's where Temporal comes in.
05:47So, Temporal is helping you to standardize your fai- failure handling and your state management in your system.
05:55Any system, any distributed system, especially any systems with agents, we are going to help you standardize that.
06:00We're software and we're also a service.
06:03The software itself is creating a lot of primitives that you can just use out of the box and basically apply to your code base.
06:10We come in all kinds of flavors of code, uh so you can use this in Python or TypeScript or Rust, um but ultimately we're giving you some standard primitives to apply to your code to be able to make sure your your state your
06:24your system is stateful.
06:26So, the three main primitives we tend to talk about are the worker, the workflow, and the activity.
06:31The worker is really running that code.
06:34The workflow is where you're going to be able to uh keep track of the steps in your system.
06:40And so, this is where like an 80K framework would be running inside a workflow.
06:44Uh your agent could work run inside of a workflow cuz it's a loop, right?
06:50Whereas the activity is all the stuff that's integrating or interacting with stuff externally.
06:55So, if you've got a loop, uh any kind of model call or any kind of tool call, those would be set up as activities.
07:01So, these are the primitives.
07:02So, you structure your code with our primitives to be able to track the state of your system.
07:10And we're free and open source at that GitHub URL.
07:13Um you pay us if you you have us manage your state with our service.
07:17Otherwise, you can host it self-host it on your own system.
07:22Let's get back to human in the loop.
07:24Why uh what I wanted to really get across to you about this whole talk.
07:29So, the human, granted, you know, we're not tools, but we are a tool to the agent.
07:35Um and you might think, "Okay, well, we're going to call the human or we're going to have the human come in and give us uh some kind of information to pass into this loop.
07:44If you write it just straight out as a function to say call the human or let the human have a an input, it can become a blocking call.
07:54And it can get lost if your service went down.
07:57It might lose track of what was going on with that human's engagement.
08:04What you want is these two core primitives to be applied to your your code and your agents.
08:09You want a wait condition and a signal.
08:13You're allowing yourself to isolate an event, apply a wait condition, store that in that workflow, in that durability layer, so that if anything fell over in the system, when that
08:27system came back up, it would know where it left off.
08:31And you could send a signal at any point, but the whole system could keep running without you necessarily being blocked on that one call to the human.
08:42The code pretty much is is is fairly simple in terms of the primitives.
08:46And granted, I know this is Python, but it's the same situation.
08:49You're going to apply a wait condition and you're going to apply a signal.
08:54You basically you're not holding a thread, you're not it's going to survive any kind of crashes.
08:59It's got all kinds of built-in primitives like timeouts, for example.
09:04If you're waiting on a human to respond and you want to make sure that it only waits for so long, you can apply timeouts and other types of primitives.
09:12Um and this can scale.
09:14You can have millions of workflows out there parked, and it can keep running.
09:19So, what I want to show you actually I'm going to show you some more code, but I'm going to kick off the next demo so that it's up and running.
09:25And I'll share with all of you, if you haven't already figured this out, uh my eyes are injured.
09:31[laughter]
09:31I am squinting at times and also realizing that I need additional in support here that I did not prepare for, but that's okay cuz I can adapt.
09:41All right, so let's get this other example up and running and then I'm going to tell you a little bit more about the code that's up here.
09:48Let's make sure that's running.
09:51I'm resetting my example and of course it's not going to be as easy peasy as I'd like.
09:58Also, is everybody hearing me okay?
10:01Think so.
10:02Great.
10:04It's an interesting condition to be working in this space with the noises around us.
10:09All right, I think it's up and running.
10:12All right, so let's talk a little bit more about the code.
10:14Now, I already told you about our primitives of activity workflow and worker.
10:20And so with the activity, you are basically for ADK in particular, we have we have the temporal model class that you will wrap around the model and then you'll pass it in for ADK to be able to set up that actual agent.
10:35It's pretty simple.
10:36You apply this temporal model, you apply our activity tool to wrap around your tools and you you run your agent.
10:47You run your agent like you would with any kind of ADK setup that you're going to do.
10:50And you also set up a worker and you pass in the Google ADK plugin.
10:56So that's setting up your worker workflow and activity.
10:59And then similarly, you have your wait condition and the wait condition is using this async io when it uses a wait to say like for that specific workflow, I want you to pause it, don't run any other code, but you can keep running the co-routines
11:13on that workflow or any other workflows in the system.
11:16Meanwhile, that workflow.signal, it will be able to take any kind of input and inject it into a running workflow.
11:25And the workflow will be taken off of your memory or taken off of your system and put in in parked if it's not running anything at that point.
11:36So, I showed you human to agent.
11:38What I want to show you is what does it look like when the agent comes in and needs the human to be involved?
11:44It's pretty similar.
11:45You're using I'm just going to go ahead and tell you right now, you're going to use a workflow you're going to use a wait condition and a signal.
11:52I'm going to use LangChain in this example and also 80K because the reality is with Temporal we can work with multiple frameworks.
12:00We are framework agnostic.
12:02We work across a variety of different tools.
12:05Um so, in terms of LangChain they build out graphs and to build out the multiple agent setup.
12:11And in this integration the way we work, you're passing us into each node for your graph.
12:17So, you're setting up with the metadata looks like.
12:19And if you're making any kind of model call like an invoke the model, that would be set up as an activity.
12:25If you're making any type of other tool call, it would also be set up as an activity.
12:30But if it's a step, a step in that process, then that's set up as a workflow.
12:35This is determinism and non-determinism in terms of workflows are deterministic, activities are non-deterministic.
12:43And then of course, you pass in for the worker the plugin for the LangChain.
12:50In terms of LangChain in particular, we use the interrupt to call out from their node.
12:55So, this is part of LangChain and also to be fair, LangChain is more than a framework, but in this case I'm using it as a framework.
13:01But you use interrupt to call out and then what you're going to do next is use that wait condition in the loop tracking for that interrupt signal to pause that workflow.
13:13And then the signal will capture whatever the human human responds with to the question that's been asked.
13:20Okay, let's see what that looks like.
13:22I'm going to bring up my demo and show you as they're running.
13:26This is the design that I've got.
13:28I've got the customer and the fleet agents running on 80K.
13:31I've got the dispatch agent running on LangGraph.
13:35And Temporal is interwoven throughout.
13:40So, that's how that's working.
13:42Now, let's see if I can show you.
13:47In this case, I'm going to drop a high-value order.
13:51This is an order that will most likely want a human involved.
13:55The workflow for the whole system is showing that all these agents are doing their job.
14:01They're continuing to keep assessing orders.
14:03And the way I've got this set up is I have workflows uh child workflows that are split off for each order per framework.
14:12So, that special order actually had its own assessment done.
14:17And here we see that the customer and the fleet agents did their work.
14:21They did their assessment.
14:22They came back with a result.
14:24And they sent that off to the dispatch agent.
14:28The dispatch agent said, "Hey, okay, I need Oh, well, there you go.
14:33[laughter]
14:34I need to get uh a human involved in this.
14:38So, I'm going to go into a pause state while I'm waiting for the human to respond."
14:42And we know that humans are not going to respond in 200 milliseconds.
14:46If you are, that's great.
14:47Maybe there's a there's probably a couple people out there who are.
14:50But humans are most likely going to take several minutes, maybe days, maybe weeks, maybe longer.
14:54And you can you can because this is just pausing that one specific event.
15:01And as you can see, the little cars are continuing to keep going.
15:04So, now what I'm going to try to do is show you what happens if I was to take this service offline.
15:11So, let's make it bigger for all of us.
15:15And kill worker.
15:20Cool.
15:21So, that takes it offline.
15:25You can see it's disconnected.
15:27Don't mind me as I do this in a very interesting way.
15:29All right.
15:30If I say approve while it's offline, so what's happened is as I mentioned, this is stuff that's happening off of your service.
15:39It's over on its own data store.
15:42So, we can still keep track of that event and queue it.
15:46And then when I bring it back up, yeah, I know.
15:50I have this really well organized here here for y'all.
15:52But, I'm going to I'm going to bring it back up and then we're going to see if it works.
15:55Let's see.
15:57Make it.
15:59And it's back online.
16:00And you'll notice that the dispatch agent will What's happening is that your event log is getting replayed.
16:05It's not redone.
16:06It's just replayed up until the point at which it last left off, so it knows, here's my current state.
16:13And this is now the next step in that state.
16:15And it's been queued up to then have that step know that it that the person approved it, and that order got assigned, and it's already been delivered, actually.
16:24So, there you go.
16:25That is durability in your system.
16:29Your system can come down, and it can recover.
16:33And it can have all these other primitives that will help you manage a complex state because these kinds of distributed systems are complex.
16:41All right.
16:41Last couple things I want to mention.
16:44When should the human be in the loop?
16:46This is the real question.
16:48It's a hard one.
16:49I will say this.
16:50The big thing you want to take into consideration is the cost of being wrong is high.
16:54Okay, it's a case-by-case basis, and we know we're trying to get to autonomy as much as we possibly can.
16:59We also know these models are probabilistic, and they are hard to get to a place to be reliable.
17:04That's why we got all this stuff happening around harnesses that are interesting.
17:07I could go into a whole 'nother conversation about that if y'all want it at a later date.
17:10But, ultimately, you want to assess the cost of being wrong.
17:14What is that worth to you?
17:15And especially in a security standpoint.
17:18And we know like on the other side of this picture, alert fatigue is real.
17:21Like a lot of us are seeing, you know, you just start saying yes, yes, yes, yes, yes every time it asks you questions.
17:26So, you're trying to balance between those two.
17:29And we're trying to get the models better so that you can trust them, so that you can address that alert fatigue, but they're still probabilistic models.
17:37So, when should they be in the When should the human be in the loop?
17:40That's the thing you've got to evaluate for the problems that you're solving.
17:45At the end of the day though, what you saw there, what I gave you as an example, whether it's the human initiating it or the computer initiating it, um you were seeing that we are using two main primitives
17:57in terms of you're applying a weight condition to pause that workflow, and a signal from the human to be able to continue the work and say, "Here's the result.
18:07Now keep moving forward."
18:09And And whether you've actually paused it and taken it offline or you brought it back online.
18:15And I've shown you that we can use multiple frameworks.
18:20All right, if you have any questions, I'm going to stick around for a few more minutes.
18:24Um if you I've got up here a QR code that will take you to some resources about Temporal, as well as some of the demos that I've done, this demo included.
18:32This is up on GitHub.
18:34Uh the slides itself will be up later cuz of course in true form I've been tweaking them up until now.
18:40So, I'll be posting those up and linking them to the GitHub repo.
18:44But yeah, if you have questions, please reach out.
18:47Um and you know, have fun building your agent clips.
18:49I'm also curious to hear what your what problems you're solving with agents and how you're building them autonomously currently.
18:55So, feel free to find me.
18:56And thank you.
19:14Woo!