WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=mJ7q8CuVmbw

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/mJ7q8CuVmbw.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:01.309
All

s2
00:00:01.309 --> 00:00:03.309
[music]

s3
00:00:12.400 --> 00:00:12.719
right.

s4
00:00:12.719 --> 00:00:13.280
Hi everyone.

s5
00:00:13.280 --> 00:00:13.920
I'm Shai.

s6
00:00:13.920 --> 00:00:16.240
I am a developer advocate at Temporal.

s7
00:00:16.240 --> 00:00:18.720
Uh I cannot see anyone in the room.

s8
00:00:18.720 --> 00:00:23.519
So if you boo me or like want to throw tomatoes or something, I I will have no idea.

s9
00:00:23.519 --> 00:00:27.039
Um all right.

s10
00:00:27.039 --> 00:00:27.680
Oop.

s11
00:00:27.680 --> 00:00:29.439
Clickers are difficult.

s12
00:00:29.439 --> 00:00:31.920
There we go.

s13
00:00:33.440 --> 00:00:35.040
I guess I'm not using a clicker today.

s14
00:00:35.040 --> 00:00:37.360
So, uh, agents, they are everywhere.

s15
00:00:37.360 --> 00:00:42.079
I'm guessing if you are in this room, you are probably building an agent or shipping an agent.

s16
00:00:42.079 --> 00:00:43.520
Does that sound about right?

s17
00:00:43.520 --> 00:00:44.399
Yeah.

s18
00:00:44.399 --> 00:00:55.920
Um, so last year, KPMC uh conducted a survey and they figured out that uh we have seen $3 trillion in terms of corporate productivity gains in 2025.

s19
00:00:55.920 --> 00:00:58.559
And that's from their uh Agentic AI report.

s20
00:00:58.559 --> 00:01:04.799
Uh and from that 3 trillion, it's really fin uh financial services that are leading the way here.

s21
00:01:04.799 --> 00:01:10.880
So 94% of fintech are saying that AI is core to how their businesses operate.

s22
00:01:10.880 --> 00:01:15.680
And they've decided that AI isn't just a side project, but but it's like it's part of the business now.

s23
00:01:15.680 --> 00:01:17.439
Like this is how we operate.

s24
00:01:17.439 --> 00:01:24.400
Uh and 62% of those teams lose measurable time uh or revenue to reliability issues.

s25
00:01:24.400 --> 00:01:31.360
So they haven't decid, you know, deciding that AI is uh that something we're doing isn't necessarily the same as shipping it.

s26
00:01:31.360 --> 00:01:34.720
There's definitely still a gap between those two things there.

s27
00:01:34.720 --> 00:01:38.240
Now to break it down, uh part of this is the shape of the work.

s28
00:01:38.240 --> 00:01:43.520
Finance runs on longunning workflows, not not on individual models.

s29
00:01:43.520 --> 00:01:51.680
Um you know, for some examples of their process, you can think of loan origination, KYC and AML, fraud review, uh paying people.

s30
00:01:51.680 --> 00:01:58.399
Each of these processes tan spans multiple days, multiple systems, and usually multiple people.

s31
00:01:58.399 --> 00:02:00.320
Uh so there's many many steps.

s32
00:02:00.320 --> 00:02:03.520
There's many many systems and there's real money involved.

s33
00:02:03.520 --> 00:02:05.759
So they can't mess anything up.

s34
00:02:05.759 --> 00:02:12.000
And now imagine that we're going to go ahead and put an agent into the middle of that process.

s35
00:02:12.000 --> 00:02:15.760
So uh at Temporal, we wanted to understand how things are actually going.

s36
00:02:15.760 --> 00:02:17.440
So we conducted a survey.

s37
00:02:17.440 --> 00:02:20.720
This is our second year running this survey, the state of development survey.

s38
00:02:20.720 --> 00:02:26.800
uh we surveyed 554 engineers and engineering leaders who are actively using AI agents right now.

s39
00:02:26.800 --> 00:02:35.040
Every number on this slide deck is either from that report uh or it'll be qualified in the bottom left from the report that I'm pulling it from.

s40
00:02:35.040 --> 00:02:38.879
So to start things off, we can talk take a look at the demographic of that survey.

s41
00:02:38.879 --> 00:02:43.280
48.4% have 11 plus years of experience working in industry.

s42
00:02:43.280 --> 00:02:45.200
So we're not talking about hobbyists.

s43
00:02:45.200 --> 00:02:49.360
And then the rema segment also has 6 to 10 years.

s44
00:02:49.360 --> 00:02:51.040
So none of these people are hobbyists.

s45
00:02:51.040 --> 00:02:54.480
They're all professionals working um every day.

s46
00:02:54.480 --> 00:02:59.599
Um it's a mix of engineers, AI engineers, data engineers, platform teams, IT leaders.

s47
00:02:59.599 --> 00:03:04.159
Twothirds of them were from the US, a third of them were from the UK in EMA.

s48
00:03:04.159 --> 00:03:06.800
Uh and what we found uh is four gaps.

s49
00:03:06.800 --> 00:03:10.080
I promised you all that this talk was about gaps in the title.

s50
00:03:10.080 --> 00:03:16.560
Uh and the data shows that there are four major gaps and I wanted to walk through each of them and then close with kind of one big idea.

s51
00:03:16.560 --> 00:03:21.760
So, let's go into gap one, and that's the gap between uh usage versus production.

s52
00:03:21.760 --> 00:03:31.440
So, daily agent use from our 2025 version of this survey had us at about 47.3% of the folks that we talked to were using agents daily.

s53
00:03:31.440 --> 00:03:40.080
Uh today, uh nearly a year later, we've doubled that to 80.8% uh of users are using agents in their day-to-day.

s54
00:03:40.080 --> 00:03:46.159
Um, now using agents isn't necessarily the same thing as shipping agents.

s55
00:03:46.159 --> 00:03:55.840
Uh, and so if we take a look at those teams again and ask them what they're doing, about half of them are running those agents in production or they say that those agents are core to how they ship.

s56
00:03:55.840 --> 00:04:02.159
And then the other half of that is still experimenting or still only really using assistance.

s57
00:04:02.159 --> 00:04:13.120
So we've got 49.1% of the folks that we talk to, half of the folks that we talk to are running agents in production are saying that those agents are core to how how things ship.

s58
00:04:13.120 --> 00:04:14.959
Um, so that's a huge gap, right?

s59
00:04:14.959 --> 00:04:24.400
Like that's a 30% gap between 80% of the people are using agents every day and 50% 49.1% are actually shipping them and using them in production.

s60
00:04:24.400 --> 00:04:28.160
So far fewer people in production than actually using them daily.

s61
00:04:28.160 --> 00:04:31.759
A and something seems to keep breaking on the way to production.

s62
00:04:31.759 --> 00:04:33.199
And let's talk about what that is.

s63
00:04:33.199 --> 00:04:37.440
And that brings us to gap number two, which is trust versus reality.

s64
00:04:37.440 --> 00:04:42.800
Now, one of the things we asked folks in this survey was how much do you actually trust your agent output?

s65
00:04:42.800 --> 00:04:44.320
And trust is really high.

s66
00:04:44.320 --> 00:04:46.639
Actually, I'm really impressed by this number.

s67
00:04:46.639 --> 00:04:51.360
Uh, a quarter of the folks that we we surveyed trusted the agent output almost completely.

s68
00:04:51.360 --> 00:04:55.360
Most people trusted it somewhat and almost no one disrupted distrusted it.

s69
00:04:55.360 --> 00:05:01.840
I did not get asked to to participate in this survey because I'm in that complete somewhat distrust category myself.

s70
00:05:01.840 --> 00:05:05.120
So, I'm actually really surprised by this number from that survey.

s71
00:05:05.120 --> 00:05:10.400
Um, yeah, 85.5% trust agent output at least somewhat.

s72
00:05:10.400 --> 00:05:12.320
Uh, and that's going to be an important number.

s73
00:05:12.320 --> 00:05:15.199
Let's come back to that 85.5 in a little bit.

s74
00:05:15.199 --> 00:05:19.600
So, we also asked, how often do your agents have issues?

s75
00:05:19.600 --> 00:05:22.160
Um, the pink bars are daily or worse.

s76
00:05:22.160 --> 00:05:27.120
And that's a lot of pink for a number that uh 85% of people trust.

s77
00:05:27.120 --> 00:05:31.600
That's a lot more pink than I would expect for an 85% trust rating.

s78
00:05:31.600 --> 00:05:36.639
Uh, 41.1% of those folks hit those agent issues daily or more.

s79
00:05:36.639 --> 00:05:40.320
So that's four in 10 of the people that we talked to see issues every day.

s80
00:05:40.320 --> 00:05:42.800
And then 16.4 hit them every hour.

s81
00:05:42.800 --> 00:05:47.199
So one in six of the people that we surveyed were getting an agent issue every hour.

s82
00:05:47.199 --> 00:05:55.360
And then that brings us back to that trust, that 85.5% gap versus the 41.1% uh system breaking.

s83
00:05:55.360 --> 00:05:59.600
Uh so so we trust the output and the system, but it still breaks every day.

s84
00:05:59.600 --> 00:06:02.880
Uh both of those things are true at the same time.

s85
00:06:02.880 --> 00:06:06.960
And getting good at agents doesn't seem to to make this go away.

s86
00:06:06.960 --> 00:06:10.080
You can't skill your way out of this.

s87
00:06:10.080 --> 00:06:18.080
um we were seeing teams that call themselves successful were actually 1.7 more times likely to hit issues continuously.

s88
00:06:18.080 --> 00:06:20.720
So be in that hourly or daily category.

s89
00:06:20.720 --> 00:06:23.199
Uh and it's not because they run more agents, right?

s90
00:06:23.199 --> 00:06:28.319
The report found that agent count and issue frequency aren't actually correlated in any way.

s91
00:06:28.319 --> 00:06:35.919
I'm guessing and the team at Temporal are guessing that maybe those folks are just looking harder for issues and so they're finding those issues some more.

s92
00:06:35.919 --> 00:06:40.639
That's definitely something we need to dig into a little more when we do next year's version of this survey.

s93
00:06:40.639 --> 00:06:46.080
But the important, I guess, takeaway is that even with how much you trust your agents, failure isn't going away.

s94
00:06:46.080 --> 00:06:47.520
Uh you have to account for it.

s95
00:06:47.520 --> 00:06:48.800
You have to plan for it.

s96
00:06:48.800 --> 00:06:51.039
So the goal isn't zero failures, right?

s97
00:06:51.039 --> 00:06:56.720
When we're using agents in production, it's just how do we make it so these failures don't cost us anything.

s98
00:06:56.720 --> 00:07:05.039
Now, a really interesting stat is saying that 40% uh of the teams that we talk to says security is truly the thing that is blocking them from autonomous agents.

s99
00:07:05.039 --> 00:07:06.160
And that's its own talk.

s100
00:07:06.160 --> 00:07:12.319
Mason uh from the temporal team and Kim from the key card team actually gave a talk about this yesterday on the workshop day.

s101
00:07:12.319 --> 00:07:18.720
So I'm going to defer to the recording of that talk uh rather than going to it myself um as well.

s102
00:07:18.720 --> 00:07:26.000
And let's jump into that third gap the prototype uh that we were probably all seeing and working on last year versus the enterprise deployment.

s103
00:07:26.000 --> 00:07:30.400
You know customers are actually touching it that prototype to production ready code system.

s104
00:07:30.400 --> 00:07:35.599
And most teams going from an AI prototype want to go to production ready code uh really fast.

s105
00:07:35.599 --> 00:07:38.560
We're seeing it in like really low numbers.

s106
00:07:38.560 --> 00:07:46.720
Uh 51.3% uh are able to get an AI prototype to production ready in hours or faster.

s107
00:07:46.720 --> 00:07:48.400
Uh that's like startup speed, right?

s108
00:07:48.400 --> 00:07:50.000
That's traditional startup speed.

s109
00:07:50.000 --> 00:07:52.960
Uh this falls apart though if you work somewhere big.

s110
00:07:52.960 --> 00:07:57.520
Uh so if you work somewhere uh 5,000 plus employees, it tends to take months.

s111
00:07:57.520 --> 00:07:59.840
Uh and the data breaks down at that point.

s112
00:07:59.840 --> 00:08:02.960
The biggest companies about one in five says it takes months.

s113
00:08:02.960 --> 00:08:09.440
Everywhere else it's about one in 20 saying that it takes months to get an AI agent deployed into production.

s114
00:08:09.440 --> 00:08:13.039
So big companies uh are shipping at very different speeds.

s115
00:08:13.039 --> 00:08:15.680
We're using the same tools uh and very different speeds.

s116
00:08:15.680 --> 00:08:21.360
Half are shipping in hours and then at these large companies were doing it one in five taking five months.

s117
00:08:21.360 --> 00:08:23.680
Um big companies, regulated work.

s118
00:08:23.680 --> 00:08:27.680
Uh we're talking about banks, we're talking about insurers, we're talking about payment companies.

s119
00:08:27.680 --> 00:08:30.720
I'm guessing there's a lot of those in the room right now.

s120
00:08:30.720 --> 00:08:34.560
And those companies all need something really need a bunch of things that are really important, right?

s121
00:08:34.560 --> 00:08:35.519
They need approvals.

s122
00:08:35.519 --> 00:08:36.479
They need audit trails.

s123
00:08:36.479 --> 00:08:39.120
They need systems that will not lose a payment.

s124
00:08:39.120 --> 00:08:44.640
My read is that's where the five months are going into the auditability and the trust of of all of those things.

s125
00:08:44.640 --> 00:08:46.399
And the survey doesn't break that down, right?

s126
00:08:46.399 --> 00:08:49.760
But it's not bureaucracy for its own sake, or at least I hope it's not.

s127
00:08:49.760 --> 00:08:55.279
I hope it's it's bureaucracy for something productive and like getting to that reliability bar.

s128
00:08:55.279 --> 00:09:00.800
Which brings us to that last reli reliability gap uh seeing versus recovering.

s129
00:09:00.800 --> 00:09:03.120
And this is one of the ones that was in the title, right?

s130
00:09:03.120 --> 00:09:06.320
So we asked what limits your team's productivity.

s131
00:09:06.320 --> 00:09:09.760
Number one is tracking state and number two is debugging.

s132
00:09:09.760 --> 00:09:13.200
The models aren't in the top two uh at all.

s133
00:09:13.200 --> 00:09:24.000
Um 35.7% were saying uh state uh tracking state across service is their top locker and then 30.9% says that it's isolating and debug debugging issues.

s134
00:09:24.000 --> 00:09:30.480
22.2 2% or uh of that survey are losing progress and redoing work because of crashes and restarts.

s135
00:09:30.480 --> 00:09:32.399
So that's one in five losing work.

s136
00:09:32.399 --> 00:09:38.959
And then last year in a different survey um 27% called their state management fragile.

s137
00:09:38.959 --> 00:09:40.720
This year it's the number one blocker.

s138
00:09:40.720 --> 00:09:43.040
Uh different surveys, different questions.

s139
00:09:43.040 --> 00:09:46.000
Um so it's not a trend line, but the story is consistent.

s140
00:09:46.000 --> 00:09:50.560
State got more important as agents got more real.

s141
00:09:50.560 --> 00:09:55.839
And one of the wildest responses that we got was when when an agent breaks, where do you go for help?

s142
00:09:55.839 --> 00:09:59.839
And the number one answer to that being YouTube is kind of wild to me.

s143
00:09:59.839 --> 00:10:01.680
I don't really code with YouTube.

s144
00:10:01.680 --> 00:10:03.920
And so this is very different from me.

s145
00:10:03.920 --> 00:10:06.080
Uh, and the way that I build things.

s146
00:10:06.080 --> 00:10:14.397
Um, but that's what debugging looks like when your system can't tell you what happened is you end up going to YouTube and trying to find someone that can.

s147
00:10:14.397 --> 00:10:14.720
[gasps]

s148
00:10:14.720 --> 00:10:17.519
Observability right now just tells you your agent failed.

s149
00:10:17.519 --> 00:10:18.640
Most teams have that, right?

s150
00:10:18.640 --> 00:10:22.079
traces, logs, LLM observability tools.

s151
00:10:22.079 --> 00:10:27.440
But reliability means that failure doesn't cost you anything.

s152
00:10:28.160 --> 00:10:35.200
Uh only 13% of the people that we talked to were very confident that they can observe and debug AI workflows at scale.

s153
00:10:35.200 --> 00:10:39.680
And then 22% of the people that we talked to lose progress because of crashes.

s154
00:10:39.680 --> 00:10:47.360
And so the gap between seeing uh and recovering is a person reading a trace at 2 a.m. in the morning and then fixing their state by hand.

s155
00:10:47.360 --> 00:10:48.800
And that's not realistic, right?

s156
00:10:48.800 --> 00:10:53.360
Like that's not a realistic thing to expect, uh, is someone fixing things at 2 a.m. in the morning.

s157
00:10:53.360 --> 00:10:56.800
So, I want to show a quick little demo, right, of what this looks like.

s158
00:10:56.800 --> 00:11:00.079
And I'm going to move 25,000 $2,500.

s159
00:11:00.079 --> 00:11:03.200
This is a synthetic customer, but it's a real temporal workflow.

s160
00:11:03.200 --> 00:11:04.959
It's a real LLM running on this laptop.

s161
00:11:04.959 --> 00:11:09.920
I have Quen 36B uh 9B running on this laptop right now.

s162
00:11:09.920 --> 00:11:14.800
Uh, and I'll show you a demo of what we're about to go through, but basically, we're running a payout workflow.

s163
00:11:14.800 --> 00:11:16.880
So, this is going to be a nine-stage workflow.

s164
00:11:16.880 --> 00:11:21.279
uh which is going to be much shorter than I think it would be in like in production.

s165
00:11:21.279 --> 00:11:22.640
Uh money is going to move.

s166
00:11:22.640 --> 00:11:25.040
Uh the money is going to look kind of weird.

s167
00:11:25.040 --> 00:11:28.800
Uh and so an agent is going to jump in and is going to do a fraud investigation.

s168
00:11:28.800 --> 00:11:36.079
The agent is going to just take a look at the transaction and decide do we need to escalate this to a human being or is this good to just wave it through.

s169
00:11:36.079 --> 00:11:45.279
Uh and then we're going to have a human in the loop that actually makes the decision and decides what to uh whether or not to make the final call of this transaction getting approved or not.

s170
00:11:45.279 --> 00:11:50.160
We're going to validate the request, reserve the funds, check in with the FX quote, and those funds will be reserved.

s171
00:11:50.160 --> 00:12:01.680
So the real state will exist in another system, and then I'm going to go ahead and kill uh sorry, and we're going to have that agent do the real work, do those tool calls, uh and then I'm going to kill the process.

s172
00:12:01.680 --> 00:12:03.920
Um so I'm going to just tell the worker to die.

s173
00:12:03.920 --> 00:12:05.120
I'm going to stop the worker.

s174
00:12:05.120 --> 00:12:06.720
Uh and so our process will die.

s175
00:12:06.720 --> 00:12:08.639
Uh and that could be for any reason, right?

s176
00:12:08.639 --> 00:12:12.399
Like it could be a timeout, it could be a deploy error, we could run out of memory.

s177
00:12:12.399 --> 00:12:16.880
I'm going to just intentionally kill it to simulate a worker crash and we'll see what we lose.

s178
00:12:16.880 --> 00:12:22.639
Um, what we would lose traditionally is that the funds would be reserved with no payouts.

s179
00:12:22.639 --> 00:12:26.639
Uh, and a brief would get produced that never actually makes its way to the approver.

s180
00:12:26.639 --> 00:12:33.760
So, we're going to have to respend our LLM tokens traditionally to regenerate that AI uh, draft or summary for the check.

s181
00:12:33.760 --> 00:12:38.720
Um, in that tickle in like a traditional stack, all of that is going to live in process memory.

s182
00:12:38.720 --> 00:12:41.279
And when that worker crashes, it's all gone.

s183
00:12:41.279 --> 00:12:43.839
And so the reserve funds are still reserved.

s184
00:12:43.839 --> 00:12:44.959
It's just not in our system.

s185
00:12:44.959 --> 00:12:46.079
It's in the bank system.

s186
00:12:46.079 --> 00:12:48.320
So traditionally, we'd have to write recovery code.

s187
00:12:48.320 --> 00:12:49.760
We'd have to build state tables.

s188
00:12:49.760 --> 00:12:51.279
We'd have to build retry logic.

s189
00:12:51.279 --> 00:12:53.440
We'd have to create reconciliation jobs.

s190
00:12:53.440 --> 00:12:58.480
And that's all code that's unrelated to the core business logic that I'm building to just do a payment transaction.

s191
00:12:58.480 --> 00:13:06.399
And that's something that 30% of our respondents complained about is that they're building all of this code unrelated to B business logic.

s192
00:13:06.399 --> 00:13:09.040
Or you could use a platform that remembers you.

s193
00:13:09.040 --> 00:13:11.600
And uh that's where durable execution comes in.

s194
00:13:11.600 --> 00:13:17.519
So durable execution is a temporal concept that code essentially should run to completion even when the process doesn't.

s195
00:13:17.519 --> 00:13:19.440
We know that crashes happen.

s196
00:13:19.440 --> 00:13:22.959
But with temporal state survives, progress survives.

s197
00:13:22.959 --> 00:13:27.200
Uh and every step is recorded in an event history that looks like this.

s198
00:13:27.200 --> 00:13:30.000
And so we've also recorded the AI in this state.

s199
00:13:30.000 --> 00:13:32.800
So there's a timeline that'll we'll see in a second live.

s200
00:13:32.800 --> 00:13:35.920
That's a run from the payout demo that we're going to run in a second.

s201
00:13:35.920 --> 00:13:52.160
uh each of those steps uh each of those AI steps choosing the investigation the tool calls that the agentic loop is going to make uh for the lookup of the customer and then drafting the brief each of those is its own individual activity with a recorded result and the yellow bar is going to be our our essentially

s202
00:13:52.160 --> 00:13:54.880
activity waiting for a human approval.

s203
00:13:54.880 --> 00:13:59.120
Um the workflow code is deciding what's next and it's deterministic.

s204
00:13:59.120 --> 00:14:03.760
Anything non-deterministic or sideffecting like the LLM is going to be run in an activity.

s205
00:14:03.760 --> 00:14:11.199
So a tool call or a bank API call is going to be an activity and then the person when they approve or reject the transaction is going to be an event too.

s206
00:14:11.199 --> 00:14:20.959
It's a signal is what we call them in temporals which is like recorded in history uh like everything else is recorded in the temporal history even though the worker is not running that signal is still going to get captured

s207
00:14:20.959 --> 00:14:22.079
uh and get processed.

s208
00:14:22.079 --> 00:14:27.760
So when the worker dies, another worker is able to resume from the last previous success.

s209
00:14:27.760 --> 00:14:30.720
Uh yeah, and the history lives in the temporal service.

s210
00:14:30.720 --> 00:14:33.839
Uh and we're not running our LLM tool calls multiple times.

s211
00:14:33.839 --> 00:14:36.800
So we're not wasting tokens uh on multiple tool calls.

s212
00:14:36.800 --> 00:14:40.160
We're going to retain the agents work uh and not redo it.

s213
00:14:40.160 --> 00:14:47.040
Uh if the call was mid-flight while the worker died, we would retry that activity, but everything before that is uh is kept.

s214
00:14:47.040 --> 00:14:48.399
Um and that matters.

s215
00:14:48.399 --> 00:14:51.760
Eight and 10 people say that cost limits how they use their agent.

s216
00:14:51.760 --> 00:14:55.680
So being able to hold on to the agents memory uh is really helpful.

s217
00:14:55.680 --> 00:14:59.600
Rerunning an agent after every cache is a lot of money that you're burning right there.

s218
00:14:59.600 --> 00:15:01.600
Uh so let's go into the quick demo.

s219
00:15:01.600 --> 00:15:08.160
And yeah, so this is my uh payment transfer.

s220
00:15:08.160 --> 00:15:10.720
So uh I'm going to go ahead and set it to $2,500.

s221
00:15:10.720 --> 00:15:12.240
I'm going to say it's a new recipient.

s222
00:15:12.240 --> 00:15:15.360
I'm going to say it's an unusual amount for this customer.

s223
00:15:15.360 --> 00:15:23.360
And just to make sure that uh our AI flags it as something that needs an investigation, we're going to go ahead and set the customer record to say that there's an unresolved account alert,

s224
00:15:23.360 --> 00:15:26.480
I'm going to go ahead and start this job, which is going to trigger this new workflow.

s225
00:15:26.480 --> 00:15:30.959
And you can see down here in the timeline, all of those nine steps that we talked about are happening.

s226
00:15:30.959 --> 00:15:33.440
Um so we're going to reserve our funds, validate.

s227
00:15:33.440 --> 00:15:42.720
Uh the AI is doing an agentic loop and deciding, okay, we're going to look at the customer profile, we're going to look at the payouts, and then we're going to draft a breach, uh a brief for a human being to review.

s228
00:15:42.720 --> 00:15:46.240
So, I've got an approve or reject button that I can click right here.

s229
00:15:46.240 --> 00:15:50.720
But before I click them, uh like I said, I am going to intentionally kill my workers.

s230
00:15:50.720 --> 00:15:52.320
I'm just going to hit stop.

s231
00:15:52.320 --> 00:15:54.880
And you can see that there are no workers running at this point.

s232
00:15:54.880 --> 00:15:58.320
So, I'm going to go ahead and approve my transaction.

s233
00:15:58.320 --> 00:16:01.199
And we can see the approval gets logged as a signal.

s234
00:16:01.199 --> 00:16:04.720
Now, traditionally, that would be something that just gets lost.

s235
00:16:04.720 --> 00:16:11.120
Um we'll get a timeout in a second, but that's not really a big deal uh because we're going to have a retry policy that'll handle this automatically.

s236
00:16:11.120 --> 00:16:18.560
So, as soon as I return on my worker, it's just going to pick up that signal and resume the behavior and continue on from that last successful event.

s237
00:16:18.560 --> 00:16:19.839
So, we'll start a worker.

s238
00:16:19.839 --> 00:16:22.560
This will take a second.

s239
00:16:24.399 --> 00:16:27.600
Of course, it takes a little longer when I'm demoing it live.

s240
00:16:27.600 --> 00:16:29.519
Usually, it's instantaneous.

s241
00:16:29.519 --> 00:16:34.320
Um, let me just go ahead and give it a refresh.

s242
00:16:34.800 --> 00:16:35.279
You know what?

s243
00:16:35.279 --> 00:16:40.720
I have a video for just this scenario, so we can go ahead and just skip straight forward in the video and see what it's like.

s244
00:16:40.720 --> 00:16:44.880
Uh, if the video plays.

s245
00:16:48.000 --> 00:16:48.880
Yeah.

s246
00:16:48.880 --> 00:16:52.480
Let me see if I can skip to the middle of this.

s247
00:16:54.800 --> 00:16:56.399
Oh, it's in the other tab.

s248
00:16:56.399 --> 00:16:57.881
What am I doing?

s249
00:16:57.881 --> 00:16:59.881
[sighs]

s250
00:17:02.079 --> 00:17:04.400
So, I'm going to approve while the worker is dead.

s251
00:17:04.400 --> 00:17:07.439
Uh, and then I'm going to go ahead and resume the worker right below.

s252
00:17:07.439 --> 00:17:12.880
Uh, and it should go ahead and just uh pick things up and like resume uh resume.

s253
00:17:12.880 --> 00:17:23.120
So, the agent is going to investigate, write the brief, uh, and then I give the approval and then, you know, I can go ahead and crash the worker again while we're waiting for the bank to approve the transaction or mark that it was completed.

s254
00:17:23.120 --> 00:17:28.799
Again, the signal logs and then whenever I turn the worker back on, it's going to pick it up and rerun again.

s255
00:17:28.799 --> 00:17:32.480
So, so this is what happened according to the event history.

s256
00:17:32.480 --> 00:17:35.760
And let me full screen that again for y'all.

s257
00:17:37.760 --> 00:17:38.880
There we go.

s258
00:17:38.880 --> 00:17:46.320
Um, so reading from top to bottom, each of our AI activities ran once, they attempted once, they worked once, and they didn't run again.

s259
00:17:46.320 --> 00:17:47.679
We didn't rerun that workflow.

s260
00:17:47.679 --> 00:17:49.120
We didn't respend those tokens.

s261
00:17:49.120 --> 00:17:51.840
We just ran it once, and so we got that onetime cost.

s262
00:17:51.840 --> 00:17:55.919
Uh, our approval signal came through while there were no s while there were no workers running.

s263
00:17:55.919 --> 00:18:00.080
Uh, and then our bank submission also, you know, came through while no workers were running.

s264
00:18:00.080 --> 00:18:02.720
And both of those things happened on new workers.

s265
00:18:02.720 --> 00:18:04.160
We stopped the workers each time.

s266
00:18:04.160 --> 00:18:06.240
Uh and then we got new workers that just picked things up.

s267
00:18:06.240 --> 00:18:07.440
So zero workers.

s268
00:18:07.440 --> 00:18:08.880
Uh the payout waited.

s269
00:18:08.880 --> 00:18:12.799
Uh the approval landed and we didn't have to rerun anything from the beginning.

s270
00:18:12.799 --> 00:18:19.280
So what are the full list uh of what those engineering leaders are saying uh slow them down.

s271
00:18:19.280 --> 00:18:25.360
Um our demo just handled seven of the top use cases right there that they gave us for problems.

s272
00:18:25.360 --> 00:18:30.400
Um so state debugging, cost, retries, approval, longunning work, and crash recovery.

s273
00:18:30.400 --> 00:18:32.640
Um and we can take a look at the code for each of them.

s274
00:18:32.640 --> 00:18:40.400
So state management in temporal is the number one blocker or sorry was the number one blocker in that survey in a workflow state is just fields.

s275
00:18:40.400 --> 00:18:41.440
It's plain variables.

s276
00:18:41.440 --> 00:18:42.720
There's no database that I built.

s277
00:18:42.720 --> 00:18:43.679
There's no cache.

s278
00:18:43.679 --> 00:18:47.120
After a crash the replay just rebuilt them exactly.

s279
00:18:47.120 --> 00:18:52.320
Uh our LLM calls uh would cost us normally for multiple runs.

s280
00:18:52.320 --> 00:18:56.160
In this run the agent we maintained the state so we only ran it once.

s281
00:18:56.160 --> 00:18:58.400
The tool run was held was sustained.

s282
00:18:58.400 --> 00:19:00.799
So we didn't need to spend any new tokens.

s283
00:19:00.799 --> 00:19:03.840
Retries themselves were a policy, not a loop.

s284
00:19:03.840 --> 00:19:11.280
We declared the retry policy uh in our uh configuration once and then it applied to every single call in that transaction.

s285
00:19:11.280 --> 00:19:13.679
There's no loop anywhere in that code.

s286
00:19:13.679 --> 00:19:19.120
Um the uh this is probably the most important slide for the room.

s287
00:19:19.120 --> 00:19:21.679
So the agent itself gave us advice.

s288
00:19:21.679 --> 00:19:29.120
The workflow itself owned the money and the state and the AI had a really bounded uh attempt to to jump in.

s289
00:19:29.120 --> 00:19:32.080
And if the model had failed, the payout would have continued without the brief.

s290
00:19:32.080 --> 00:19:34.400
A model outage wouldn't have degraded anything.

s291
00:19:34.400 --> 00:19:36.400
Um, we had our human approval.

s292
00:19:36.400 --> 00:19:39.039
Uh, and we could have set that timer for as long as we wanted.

s293
00:19:39.039 --> 00:19:40.559
Uh, and it would have been the same code.

s294
00:19:40.559 --> 00:19:45.120
Um, we could have waited for a year and resumed that worker and it would have been fine.

s295
00:19:45.120 --> 00:19:47.600
And then I didn't write any crash recovery code.

s296
00:19:47.600 --> 00:19:50.000
That was all built into temporal.

s297
00:19:50.000 --> 00:19:58.480
Um, so debugging uh, in most stacks observability and recovery are two separate records that drift apart and then a human brings them together.

s298
00:19:58.480 --> 00:20:09.840
Here we were able to have the history that we debugged in the UI be the same history that our worker replays which closed gap 4. So we had seven blockers and we solved them with one workflow.

s299
00:20:09.919 --> 00:20:11.440
And this isn't just a demo.

s300
00:20:11.440 --> 00:20:14.240
So there are people using this stuff in production for real right now.

s301
00:20:14.240 --> 00:20:16.720
OpenAI's codeex runs on temporal.

s302
00:20:16.720 --> 00:20:20.240
Uh the coding agent that you use a lot of the day is running on temporal.

s303
00:20:20.240 --> 00:20:22.559
Uh Replet runs their replet agents on temporal.

s304
00:20:22.559 --> 00:20:28.720
Their engineering team says it's a bad user experience when the agent gets super far into something and then hits a catastrophic error.

s305
00:20:28.720 --> 00:20:32.640
And that's just what we ha watched happen or watch not happen.

s306
00:20:32.640 --> 00:20:37.039
Um, and I'm going to skip through this because I know I'm running out of time.

s307
00:20:37.039 --> 00:20:44.080
Uh, so from the AI engineer side, you write the prompt, you run it, and the if the output's bad, you throw it away, and then temporal retries it for you.

s308
00:20:44.080 --> 00:20:46.559
Um, bad model output is just an activity failure.

s309
00:20:46.559 --> 00:20:50.000
So same similar retry policy uh as well.

s310
00:20:50.000 --> 00:20:53.840
Um so back to the four gaps, one record uh closed them all.

s311
00:20:53.840 --> 00:20:57.614
Um and just to skip through what those four gaps were uh again

s312
00:20:57.614 --> 00:20:58.559
[laughter]

s313
00:20:58.559 --> 00:21:03.840
uh and resummarize that uh that yeah reliability and observability uh were just one record.

s314
00:21:03.840 --> 00:21:12.640
And so every number of this talk that I I um mentioned is in that temporal IO report state of development and I'll leave that up screen if anyone wants to take a photo and come check it out.

s315
00:21:12.640 --> 00:21:18.240
Uh and there's a bunch of us that'll be at the temporal booth running this demo live if anyone wants to come see it or say hi.

s316
00:21:18.240 --> 00:21:21.360
So, um, yeah, that's that's my talk.

s317
00:21:21.360 --> 00:21:23.600
Let's hope

s318
00:21:24.469 --> 00:21:26.469
[applause]