WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=APh1Vx0oLmQ

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/APh1Vx0oLmQ.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:03.840 --> 00:00:04.880
Hey everyone.

s2
00:00:04.880 --> 00:00:06.280
My name is Nishant Gupta.

s3
00:00:06.280 --> 00:00:12.520
I'm a software engineering tech lead at Meta, working on building the training and inference infrastructure.

s4
00:00:12.520 --> 00:00:19.000
And today, we're going to be talking about building deterministic infrastructure for non-deterministic AI agents.

s5
00:00:19.000 --> 00:00:23.200
So, most of the conversations around AI over the last few years has been focused on models.

s6
00:00:23.200 --> 00:00:26.720
Bigger models, more parameters, better reasoning.

s7
00:00:26.720 --> 00:00:31.200
But as organizations move from chatbots to autonomous agents, a different problem emerges.

s8
00:00:31.200 --> 00:00:33.840
The challenge is no longer in intelligence.

s9
00:00:33.840 --> 00:00:36.480
The challenge is is reliability.

s10
00:00:36.480 --> 00:00:46.680
At Meta and across the industry, we are seeing agents move beyond answering questions and beginning to plan, call tool calls, coordinate workflows, and make decisions that affect production systems.

s11
00:00:46.680 --> 00:00:50.040
These systems are fundamentally probabilistic.

s12
00:00:50.040 --> 00:00:52.240
Infrastructure is not allowed to be.

s13
00:00:52.240 --> 00:00:56.440
Today, I want to discuss this topic in more detail.

s14
00:00:58.200 --> 00:01:01.520
The modern cloud infrastructure evolved around a set of assumptions.

s15
00:01:01.520 --> 00:01:03.760
A request Most of the requests are short-lived.

s16
00:01:03.760 --> 00:01:06.520
Services are deterministic, more or less.

s17
00:01:06.520 --> 00:01:08.160
Execution paths are known.

s18
00:01:08.160 --> 00:01:09.960
Failures are bounded.

s19
00:01:09.960 --> 00:01:14.000
However, autonomous AI agents violate nearly every one of those assumptions.

s20
00:01:14.000 --> 00:01:14.840
They're stateful.

s21
00:01:14.840 --> 00:01:15.800
They're long-running.

s22
00:01:15.800 --> 00:01:17.320
They make decisions dynamically.

s23
00:01:17.320 --> 00:01:21.000
They may may execute different workflows for same inputs.

s24
00:01:21.000 --> 00:01:23.760
This is what I call the great mismatch.

s25
00:01:23.760 --> 00:01:29.560
We're trying to run autonomous systems on infrastructure that was designed for deterministic workflows.

s26
00:01:30.560 --> 00:01:32.960
This is probably the most important mindshift.

s27
00:01:32.960 --> 00:01:36.320
Most AI demos showcase capability.

s28
00:01:36.400 --> 00:01:37.600
But can it solve a problem?

s29
00:01:37.600 --> 00:01:38.600
Can it use a tool?

s30
00:01:38.600 --> 00:01:40.720
Can it compete complete a workflow?

s31
00:01:40.720 --> 00:01:43.840
Production systems be have a different objective.

s32
00:01:43.840 --> 00:01:45.320
Can it do it reliably?

s33
00:01:45.320 --> 00:01:49.160
Can it do it 10,000 times, 100,000 times, million times?

s34
00:01:49.160 --> 00:01:50.280
Can it recover from failures?

s35
00:01:50.280 --> 00:01:51.840
Can it operate safely?

s36
00:01:51.840 --> 00:01:55.680
Can it do it in an at an acceptable cost with an acceptable latency?

s37
00:01:55.680 --> 00:01:57.560
With an acceptable outcome?

s38
00:01:57.560 --> 00:02:05.560
The majority of the engineering effort moves below the model layer into orchestration, monitoring, safety evaluation, and recovery systems.

s39
00:02:06.280 --> 00:02:09.399
When people hear AI failures, they immediately think hallucinations.

s40
00:02:09.399 --> 00:02:14.080
In reality In reality, hallucinations are often the least interesting failure mode.

s41
00:02:14.080 --> 00:02:24.959
What we see instead are infrastructure failures, recursive reasoning loops, workflow dead locks, retry amplification, context corruption, memory poisoning, cost explosions.

s42
00:02:24.959 --> 00:02:29.280
The model makes a mistake, but however, the infrastructure turns that mistake into an outage.

s43
00:02:29.280 --> 00:02:31.840
That's the real challenge.

s44
00:02:33.160 --> 00:02:42.000
So, as this slide shows a pattern that distributed system engineers will probably recognize immediately, an agent calls a tool incorrectly, the tool returns an error.

s45
00:02:42.000 --> 00:02:47.959
Instead of recovering, the agent generates a slightly different, but still invalid request.

s46
00:02:47.959 --> 00:02:49.200
The cycle repeats.

s47
00:02:49.200 --> 00:02:51.760
Each retry consumes more compute.

s48
00:02:51.760 --> 00:02:57.880
Reasoning depth increases, GPU consumption rises, eventually you get exponential resource growth.

s49
00:02:57.880 --> 00:03:01.640
What started as a minor API error became a compute incident.

s50
00:03:01.640 --> 00:03:07.400
This is why unco- uncontrolled retries are one of the biggest risk in agentic systems.

s51
00:03:08.200 --> 00:03:10.840
This is the architecture principle I recommend most strongly.

s52
00:03:10.840 --> 00:03:13.760
Never let the model directly control production systems.

s53
00:03:13.760 --> 00:03:20.720
The model should generate proposals, infrastructure validates them, policy engine approves them, execution gateway enforces them.

s54
00:03:20.720 --> 00:03:23.480
The model just suggests, the platform decides.

s55
00:03:23.480 --> 00:03:29.160
This separation allows us to build reliable systems even when the underlying model remains probabilistic.

s56
00:03:30.640 --> 00:03:35.600
As we know, containers gave rise to Kubernetes, microservices created service meshes.

s57
00:03:35.600 --> 00:03:38.840
AI agents are creating something new, an agentic control plane.

s58
00:03:38.840 --> 00:03:48.959
This layer becomes responsible for scheduling, memory coordination, policy enforcement, evaluation, monitoring, workload routing, which is very important.

s59
00:03:48.959 --> 00:03:51.680
And think of it as an operating system for autonomous AI.

s60
00:03:51.680 --> 00:03:56.720
The organizations that build this layer will have significantly more competitive advantages.

s61
00:03:57.600 --> 00:03:59.440
So, traditional logs tell us what happened.

s62
00:03:59.440 --> 00:04:01.920
Agentic systems require understanding why it happened.

s63
00:04:01.920 --> 00:04:08.760
We need traces to capture planning decisions, tool calls, memory lookups, state transitions.

s64
00:04:08.760 --> 00:04:15.400
When debugging an autonomous workflow, understanding the chain of decisions and reasoning is often more important than the final output.

s65
00:04:15.400 --> 00:04:18.799
Observability becomes multi-dimensional.

s66
00:04:19.200 --> 00:04:23.200
Without it, production debugging becomes nearly impossible.

s67
00:04:23.600 --> 00:04:27.960
So, as you can see, memory is one of the most underestimated challenges in agentic architectures.

s68
00:04:27.960 --> 00:04:32.040
Once multiple agents share state, familiar distributed system issues appear.

s69
00:04:32.040 --> 00:04:37.280
Stale reads, conflicting updates, context drifts, inconsistent views.

s70
00:04:37.280 --> 00:04:42.360
The challenge becomes even harder when memory itself may be probabilistic and retrieval-based.

s71
00:04:42.360 --> 00:04:48.680
Many multi-agent failures are actually consistency failures masquerading as reasoning failures.

s72
00:04:50.720 --> 00:04:52.480
So, safety cannot be a single component.

s73
00:04:52.480 --> 00:04:53.400
It must be layered.

s74
00:04:53.400 --> 00:04:58.800
Prompt level controls, tool permissions, policy validations, human approvals, audit systems.

s75
00:04:58.800 --> 00:05:01.840
Each of these layers catches a different class of failures.

s76
00:05:01.840 --> 00:05:05.120
Defense in depth is a well-understood security principle.

s77
00:05:05.120 --> 00:05:09.120
It applies equally well to autonomous AI systems.

s78
00:05:09.600 --> 00:05:13.960
Many people frame human involvement as temporarily temporary necessity.

s79
00:05:13.960 --> 00:05:15.080
I don't think that's correct.

s80
00:05:15.080 --> 00:05:18.760
The most successful systems are likely to remain human supervised.

s81
00:05:18.760 --> 00:05:20.919
Humans became become exception handlers.

s82
00:05:20.919 --> 00:05:22.960
They review ambiguous situations.

s83
00:05:22.960 --> 00:05:24.840
They handle normal scenarios.

s84
00:05:24.840 --> 00:05:27.000
They provide calibration signals.

s85
00:05:27.000 --> 00:05:28.919
The goal is not to remove humans.

s86
00:05:28.919 --> 00:05:34.320
The goal is allocating human attention where it provides the maximum value.

s87
00:05:36.280 --> 00:05:41.880
So, one of the biggest infrastructure shifts is that AI workloads increasingly resemble cluster scheduling problems.

s88
00:05:41.880 --> 00:05:43.400
Demand is bursting.

s89
00:05:43.400 --> 00:05:48.840
Reasoning depth is and Workflows may run for minutes instead of milliseconds.

s90
00:05:48.840 --> 00:05:51.800
Resource requirements vary dramatically.

s91
00:05:51.800 --> 00:05:57.920
As a result, GPU efficiency, workload placement, elastic capacity management, and scheduling becomes critical.

s92
00:05:57.920 --> 00:05:59.640
Inference is no longer just a model problem.

s93
00:05:59.640 --> 00:06:03.160
It becomes a resource orchestration problem.

s94
00:06:03.520 --> 00:06:07.160
The good news is that many of these problems are not entirely new.

s95
00:06:07.160 --> 00:06:10.680
Distributed systems have solved something similar for decades.

s96
00:06:10.680 --> 00:06:13.040
Circuit breakers become tool isolation.

s97
00:06:13.040 --> 00:06:14.400
Rate limits become agent limits.

s98
00:06:14.400 --> 00:06:15.920
Retries become control recovery.

s99
00:06:15.920 --> 00:06:18.560
Resource quotas become cost governance.

s100
00:06:18.560 --> 00:06:21.160
Observability becomes agent tracing.

s101
00:06:21.160 --> 00:06:27.560
Instead of inventing entirely new infrastructure, we can adapt to reliability patterns for autonomous systems.

s102
00:06:27.600 --> 00:06:29.760
The industry has gone through several phases.

s103
00:06:29.760 --> 00:06:31.880
The initially prompts were the differentiator.

s104
00:06:31.880 --> 00:06:33.560
Then the models became the differentiator.

s105
00:06:33.560 --> 00:06:36.840
And both are rapid rapidly commoditizing.

s106
00:06:36.840 --> 00:06:38.400
The next frontier is infrastructure.

s107
00:06:38.400 --> 00:06:42.240
The organization that won't that wins is not necessarily have the best prompts.

s108
00:06:42.240 --> 00:06:44.280
They'll have the most reliable systems.

s109
00:06:44.280 --> 00:06:48.320
The competitive advantage is moving up the stack.

s110
00:06:48.840 --> 00:06:53.040
If there's one thing I want you to remember, it's this.

s111
00:06:53.040 --> 00:06:55.520
AI agents should be treated as distributed systems.

s112
00:06:55.520 --> 00:06:56.640
Models are stochastic.

s113
00:06:56.640 --> 00:06:59.240
Infrastructures must be deterministic.

s114
00:06:59.240 --> 00:07:01.520
Reliability is increasingly an infrastructure problem.

s115
00:07:01.520 --> 00:07:03.040
Observability is mandatory.

s116
00:07:03.040 --> 00:07:05.240
Control planes are emerging as a foundation layer.

s117
00:07:05.240 --> 00:07:09.000
And ultimately, the future of the AI AI won't be won by better prompts.

s118
00:07:09.000 --> 00:07:12.000
It will be won by better systems.