Deterministic Infra for Non-Deterministic AI Agents - Nishant Gupta, Meta Superintelligence Labs
https://www.youtube.com/watch?v=APh1Vx0oLmQ
AI Engineer · 7 min · 118 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:03] Hey everyone.
[00:04] My name is Nishant Gupta.
[00:06] I'm a software engineering tech lead at Meta, working on building the training and inference infrastructure.
[00:12] And today, we're going to be talking about building deterministic infrastructure for non-deterministic AI agents.
[00:19] So, most of the conversations around AI over the last few years has been focused on models.
[00:23] Bigger models, more parameters, better reasoning.
[00:26] But as organizations move from chatbots to autonomous agents, a different problem emerges.
[00:31] The challenge is no longer in intelligence.
[00:33] The challenge is is reliability.
[00:36] At Meta and across the industry, we are seeing agents move beyond answering questions and beginning to plan, call tool calls, coordinate workflows, and make decisions that affect production systems.
[00:46] These systems are fundamentally probabilistic.
[00:50] Infrastructure is not allowed to be.
[00:52] Today, I want to discuss this topic in more detail.
[00:58] The modern cloud infrastructure evolved around a set of assumptions.
[01:01] A request Most of the requests are short-lived.
[01:03] Services are deterministic, more or less.
[01:06] Execution paths are known.
[01:08] Failures are bounded.
[01:09] However, autonomous AI agents violate nearly every one of those assumptions.
[01:14] They're stateful.
[01:14] They're long-running.
[01:15] They make decisions dynamically.
[01:17] They may may execute different workflows for same inputs.
[01:21] This is what I call the great mismatch.
[01:23] We're trying to run autonomous systems on infrastructure that was designed for deterministic workflows.
[01:30] This is probably the most important mindshift.
[01:32] Most AI demos showcase capability.
[01:36] But can it solve a problem?
[01:37] Can it use a tool?
[01:38] Can it compete complete a workflow?
[01:40] Production systems be have a different objective.
[01:43] Can it do it reliably?
[01:45] Can it do it 10,000 times, 100,000 times, million times?
[01:49] Can it recover from failures?
[01:50] Can it operate safely?
[01:51] Can it do it in an at an acceptable cost with an acceptable latency?
[01:55] With an acceptable outcome?
[01:57] The majority of the engineering effort moves below the model layer into orchestration, monitoring, safety evaluation, and recovery systems.
[02:06] When people hear AI failures, they immediately think hallucinations.
[02:09] In reality In reality, hallucinations are often the least interesting failure mode.
[02:14] What we see instead are infrastructure failures, recursive reasoning loops, workflow dead locks, retry amplification, context corruption, memory poisoning, cost explosions.
[02:24] The model makes a mistake, but however, the infrastructure turns that mistake into an outage.
[02:29] That's the real challenge.
[02:33] So, as this slide shows a pattern that distributed system engineers will probably recognize immediately, an agent calls a tool incorrectly, the tool returns an error.
[02:42] Instead of recovering, the agent generates a slightly different, but still invalid request.
[02:47] The cycle repeats.
[02:49] Each retry consumes more compute.
[02:51] Reasoning depth increases, GPU consumption rises, eventually you get exponential resource growth.
[02:57] What started as a minor API error became a compute incident.
[03:01] This is why unco- uncontrolled retries are one of the biggest risk in agentic systems.
[03:08] This is the architecture principle I recommend most strongly.
[03:10] Never let the model directly control production systems.
[03:13] The model should generate proposals, infrastructure validates them, policy engine approves them, execution gateway enforces them.
[03:20] The model just suggests, the platform decides.
[03:23] This separation allows us to build reliable systems even when the underlying model remains probabilistic.
[03:30] As we know, containers gave rise to Kubernetes, microservices created service meshes.
[03:35] AI agents are creating something new, an agentic control plane.
[03:38] This layer becomes responsible for scheduling, memory coordination, policy enforcement, evaluation, monitoring, workload routing, which is very important.
[03:48] And think of it as an operating system for autonomous AI.
[03:51] The organizations that build this layer will have significantly more competitive advantages.
[03:57] So, traditional logs tell us what happened.
[03:59] Agentic systems require understanding why it happened.
[04:01] We need traces to capture planning decisions, tool calls, memory lookups, state transitions.
[04:08] When debugging an autonomous workflow, understanding the chain of decisions and reasoning is often more important than the final output.
[04:15] Observability becomes multi-dimensional.
[04:19] Without it, production debugging becomes nearly impossible.
[04:23] So, as you can see, memory is one of the most underestimated challenges in agentic architectures.
[04:27] Once multiple agents share state, familiar distributed system issues appear.
[04:32] Stale reads, conflicting updates, context drifts, inconsistent views.
[04:37] The challenge becomes even harder when memory itself may be probabilistic and retrieval-based.
[04:42] Many multi-agent failures are actually consistency failures masquerading as reasoning failures.
[04:50] So, safety cannot be a single component.
[04:52] It must be layered.
[04:53] Prompt level controls, tool permissions, policy validations, human approvals, audit systems.
[04:58] Each of these layers catches a different class of failures.
[05:01] Defense in depth is a well-understood security principle.
[05:05] It applies equally well to autonomous AI systems.
[05:09] Many people frame human involvement as temporarily temporary necessity.
[05:13] I don't think that's correct.
[05:15] The most successful systems are likely to remain human supervised.
[05:18] Humans became become exception handlers.
[05:20] They review ambiguous situations.
[05:22] They handle normal scenarios.
[05:24] They provide calibration signals.
[05:27] The goal is not to remove humans.
[05:28] The goal is allocating human attention where it provides the maximum value.
[05:36] So, one of the biggest infrastructure shifts is that AI workloads increasingly resemble cluster scheduling problems.
[05:41] Demand is bursting.
[05:43] Reasoning depth is and Workflows may run for minutes instead of milliseconds.
[05:48] Resource requirements vary dramatically.
[05:51] As a result, GPU efficiency, workload placement, elastic capacity management, and scheduling becomes critical.
[05:57] Inference is no longer just a model problem.
[05:59] It becomes a resource orchestration problem.
[06:03] The good news is that many of these problems are not entirely new.
[06:07] Distributed systems have solved something similar for decades.
[06:10] Circuit breakers become tool isolation.
[06:13] Rate limits become agent limits.
[06:14] Retries become control recovery.
[06:15] Resource quotas become cost governance.
[06:18] Observability becomes agent tracing.
[06:21] Instead of inventing entirely new infrastructure, we can adapt to reliability patterns for autonomous systems.
[06:27] The industry has gone through several phases.
[06:29] The initially prompts were the differentiator.
[06:31] Then the models became the differentiator.
[06:33] And both are rapid rapidly commoditizing.
[06:36] The next frontier is infrastructure.
[06:38] The organization that won't that wins is not necessarily have the best prompts.
[06:42] They'll have the most reliable systems.
[06:44] The competitive advantage is moving up the stack.
[06:48] If there's one thing I want you to remember, it's this.
[06:53] AI agents should be treated as distributed systems.
[06:55] Models are stochastic.
[06:56] Infrastructures must be deterministic.
[06:59] Reliability is increasingly an infrastructure problem.
[07:01] Observability is mandatory.
[07:03] Control planes are emerging as a foundation layer.
[07:05] And ultimately, the future of the AI AI won't be won by better prompts.
[07:09] It will be won by better systems.