Deterministic Infra for Non-Deterministic AI Agents - Nishant Gupta, Meta Superintelligence Labs

AI Engineer · 7 min · 118 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:03Hey everyone.
2. 00:04My name is Nishant Gupta.
3. 00:06I'm a software engineering tech lead at Meta, working on building the training and inference infrastructure.
4. 00:12And today, we're going to be talking about building deterministic infrastructure for non-deterministic AI agents.
5. 00:19So, most of the conversations around AI over the last few years has been focused on models.
6. 00:23Bigger models, more parameters, better reasoning.
7. 00:26But as organizations move from chatbots to autonomous agents, a different problem emerges.
8. 00:31The challenge is no longer in intelligence.
9. 00:33The challenge is is reliability.
10. 00:36At Meta and across the industry, we are seeing agents move beyond answering questions and beginning to plan, call tool calls, coordinate workflows, and make decisions that affect production systems.
11. 00:46These systems are fundamentally probabilistic.
12. 00:50Infrastructure is not allowed to be.
13. 00:52Today, I want to discuss this topic in more detail.
14. 00:58The modern cloud infrastructure evolved around a set of assumptions.
15. 01:01A request Most of the requests are short-lived.
16. 01:03Services are deterministic, more or less.
17. 01:06Execution paths are known.
18. 01:08Failures are bounded.
19. 01:09However, autonomous AI agents violate nearly every one of those assumptions.
20. 01:14They're stateful.
21. 01:14They're long-running.
22. 01:15They make decisions dynamically.
23. 01:17They may may execute different workflows for same inputs.
24. 01:21This is what I call the great mismatch.
25. 01:23We're trying to run autonomous systems on infrastructure that was designed for deterministic workflows.
26. 01:30This is probably the most important mindshift.
27. 01:32Most AI demos showcase capability.
28. 01:36But can it solve a problem?
29. 01:37Can it use a tool?
30. 01:38Can it compete complete a workflow?
31. 01:40Production systems be have a different objective.
32. 01:43Can it do it reliably?
33. 01:45Can it do it 10,000 times, 100,000 times, million times?
34. 01:49Can it recover from failures?
35. 01:50Can it operate safely?
36. 01:51Can it do it in an at an acceptable cost with an acceptable latency?
37. 01:55With an acceptable outcome?
38. 01:57The majority of the engineering effort moves below the model layer into orchestration, monitoring, safety evaluation, and recovery systems.
39. 02:06When people hear AI failures, they immediately think hallucinations.
40. 02:09In reality In reality, hallucinations are often the least interesting failure mode.
41. 02:14What we see instead are infrastructure failures, recursive reasoning loops, workflow dead locks, retry amplification, context corruption, memory poisoning, cost explosions.
42. 02:24The model makes a mistake, but however, the infrastructure turns that mistake into an outage.
43. 02:29That's the real challenge.
44. 02:33So, as this slide shows a pattern that distributed system engineers will probably recognize immediately, an agent calls a tool incorrectly, the tool returns an error.
45. 02:42Instead of recovering, the agent generates a slightly different, but still invalid request.
46. 02:47The cycle repeats.
47. 02:49Each retry consumes more compute.
48. 02:51Reasoning depth increases, GPU consumption rises, eventually you get exponential resource growth.
49. 02:57What started as a minor API error became a compute incident.
50. 03:01This is why unco- uncontrolled retries are one of the biggest risk in agentic systems.
51. 03:08This is the architecture principle I recommend most strongly.
52. 03:10Never let the model directly control production systems.
53. 03:13The model should generate proposals, infrastructure validates them, policy engine approves them, execution gateway enforces them.
54. 03:20The model just suggests, the platform decides.
55. 03:23This separation allows us to build reliable systems even when the underlying model remains probabilistic.
56. 03:30As we know, containers gave rise to Kubernetes, microservices created service meshes.
57. 03:35AI agents are creating something new, an agentic control plane.
58. 03:38This layer becomes responsible for scheduling, memory coordination, policy enforcement, evaluation, monitoring, workload routing, which is very important.
59. 03:48And think of it as an operating system for autonomous AI.
60. 03:51The organizations that build this layer will have significantly more competitive advantages.
61. 03:57So, traditional logs tell us what happened.
62. 03:59Agentic systems require understanding why it happened.
63. 04:01We need traces to capture planning decisions, tool calls, memory lookups, state transitions.
64. 04:08When debugging an autonomous workflow, understanding the chain of decisions and reasoning is often more important than the final output.
65. 04:15Observability becomes multi-dimensional.
66. 04:19Without it, production debugging becomes nearly impossible.
67. 04:23So, as you can see, memory is one of the most underestimated challenges in agentic architectures.
68. 04:27Once multiple agents share state, familiar distributed system issues appear.
69. 04:32Stale reads, conflicting updates, context drifts, inconsistent views.
70. 04:37The challenge becomes even harder when memory itself may be probabilistic and retrieval-based.
71. 04:42Many multi-agent failures are actually consistency failures masquerading as reasoning failures.
72. 04:50So, safety cannot be a single component.
73. 04:52It must be layered.
74. 04:53Prompt level controls, tool permissions, policy validations, human approvals, audit systems.
75. 04:58Each of these layers catches a different class of failures.
76. 05:01Defense in depth is a well-understood security principle.
77. 05:05It applies equally well to autonomous AI systems.
78. 05:09Many people frame human involvement as temporarily temporary necessity.
79. 05:13I don't think that's correct.
80. 05:15The most successful systems are likely to remain human supervised.
81. 05:18Humans became become exception handlers.
82. 05:20They review ambiguous situations.
83. 05:22They handle normal scenarios.
84. 05:24They provide calibration signals.
85. 05:27The goal is not to remove humans.
86. 05:28The goal is allocating human attention where it provides the maximum value.
87. 05:36So, one of the biggest infrastructure shifts is that AI workloads increasingly resemble cluster scheduling problems.
88. 05:41Demand is bursting.
89. 05:43Reasoning depth is and Workflows may run for minutes instead of milliseconds.
90. 05:48Resource requirements vary dramatically.
91. 05:51As a result, GPU efficiency, workload placement, elastic capacity management, and scheduling becomes critical.
92. 05:57Inference is no longer just a model problem.
93. 05:59It becomes a resource orchestration problem.
94. 06:03The good news is that many of these problems are not entirely new.
95. 06:07Distributed systems have solved something similar for decades.
96. 06:10Circuit breakers become tool isolation.
97. 06:13Rate limits become agent limits.
98. 06:14Retries become control recovery.
99. 06:15Resource quotas become cost governance.
100. 06:18Observability becomes agent tracing.
101. 06:21Instead of inventing entirely new infrastructure, we can adapt to reliability patterns for autonomous systems.
102. 06:27The industry has gone through several phases.
103. 06:29The initially prompts were the differentiator.
104. 06:31Then the models became the differentiator.
105. 06:33And both are rapid rapidly commoditizing.
106. 06:36The next frontier is infrastructure.
107. 06:38The organization that won't that wins is not necessarily have the best prompts.
108. 06:42They'll have the most reliable systems.
109. 06:44The competitive advantage is moving up the stack.
110. 06:48If there's one thing I want you to remember, it's this.
111. 06:53AI agents should be treated as distributed systems.
112. 06:55Models are stochastic.
113. 06:56Infrastructures must be deterministic.
114. 06:59Reliability is increasingly an infrastructure problem.
115. 07:01Observability is mandatory.
116. 07:03Control planes are emerging as a foundation layer.
117. 07:05And ultimately, the future of the AI AI won't be won by better prompts.
118. 07:09It will be won by better systems.