Production Evals For Agentic AI Systems - Nishant Gupta, Meta Superintelligence Labs

AI Engineer · 8 min · 121 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:03Hey everyone.
2. 00:04My name is Nishant Gupta and I'm a software engineering tech lead at Meta working on building the training and inference infrastructure for the Meta Super Dangerous Lab and the infrastructure organization.
3. 00:17Today we're going to be talking about production of ads for agentic systems.
4. 00:21When most people hear the word evaluation, they think about benchmarks.
5. 00:25A model scores 90% on a benchmark.
6. 00:28A new version scores 92%.
7. 00:30The team celebrates.
8. 00:31But agentic systems have fundamentally changed what the evaluation means.
9. 00:35Today the systems don't simply generate answers.
10. 00:38They plan.
11. 00:39They call tools.
12. 00:40They retrieve information.
13. 00:41They execute workflows.
14. 00:42They interact with the production infrastructure.
15. 00:45The question is no longer did the model generate the right answer?
16. 00:49The question is did the system behave correctly?
17. 00:51Today I would like to discuss how evaluation is evolving from model benchmarking into production infrastructure.
18. 01:02This is the problem almost every AI organization is encountering today.
19. 01:06Offline benchmarks continue improving.
20. 01:09Yet production reliability often remains unpredictable.
21. 01:12Why is that?
22. 01:13Because benchmarks measure model capability.
23. 01:15Production measures system behavior.
24. 01:17A benchmark doesn't capture tool failure, API outage, context changes, user variability, long running workflows.
25. 01:25And as systems become more autonomous, the gap between the benchmark performance and production performance grows.
26. 01:30The result is what many teams experience today.
27. 01:34High benchmark scores, as you can see, but unreliable production behavior.
28. 01:41Traditional LLM evaluation focus on outputs.
29. 01:44But we should ask the question did the model produce the correct answer?
30. 01:48Agentic systems force us to ask a different question.
31. 01:51Did the system behave correctly?
32. 01:52Behavior includes planning quality, tool usage, execution, workflow execution, recovery from failures, decision-making.
33. 02:01In other words, we are moving from evaluating answers to evaluating workflows.
34. 02:06And that requires fundamentally different evaluation architectures.
35. 02:10Many teams still think hallucinations are the primary AI failure modes.
36. 02:15In production, they are often just one category.
37. 02:17Agentic systems introduce an entire hierarchy of failure modes.
38. 02:21At the very foundation, the memory failures, retrieval failures, safety failures.
39. 02:26As you go up, you have to think about reasoning mistakes, poor planning, incorrect tool execution.
40. 02:31At the highest layer, you have to think about multi-agent coordination failures.
41. 02:35And this is why evaluating only model output misses the most production risk we observe.
42. 02:43One of the most useful mindset shifts is to stop thinking like researchers and start thinking like a SRE or a production engineer.
43. 02:50SREs don't measure success using accuracy.
44. 02:53They measure reliability, availability, latency, cost, recovery.
45. 02:57And agentic systems require the same approach.
46. 02:59The goal is not maximizing the benchmark scores.
47. 03:02The goal is to maximize dependable outcomes.
48. 03:05Reliability becomes the North Star metric.
49. 03:07Accuracy becomes the only input.
50. 03:14In this pyramid is how I think personally think about modern AI evaluation systems.
51. 03:18At the bottom, you can see they are benchmarks.
52. 03:21They are useful.
53. 03:21They are scalable.
54. 03:22They are repeatable.
55. 03:23But their operational value is limited.
56. 03:25In the middle, there are scenario-based evaluations.
57. 03:28These simulate realistic workflows.
58. 03:30And at the very top, you see production telemetry.
59. 03:33This is where the highest value evaluation signals come from.
60. 03:36The surprising insight is that the most evaluation data often comes from real users interacting with real systems.
61. 03:45Now, let's talk about AI offline evaluations.
62. 03:46So, offline evaluation still matters, but the methodology changes.
63. 03:50Instead of evaluating prompts, we evaluate scenarios.
64. 03:52For example, a customer support workflow, a code generation workflow, a research workflow.
65. 03:57The agent operates inside that simulated environment.
66. 03:59We measure the task completion rate, tool correctness, planning quality, resource usage, which is which becomes exponentially high at high scale.
67. 04:08The key takeaway, agent evaluation should be scenario driven, not prompt driven.
68. 04:14Once a system reaches production, every interaction becomes a signal.
69. 04:18This is one of the biggest shifts in evaluation thinking.
70. 04:21Production traffic is no longer just traffic.
71. 04:23It becomes evaluation data.
72. 04:25We collect execution traces, user outcomes, escalations, failures, feedback signals.
73. 04:31Production is the largest and the most representative evaluation data any organization will ever have.
74. 04:39Many organizations view humans as fallback systems.
75. 04:42I think that's a wrong framing.
76. 04:44Humans are the evaluators.
77. 04:46They provide signals that automated systems cannot.
78. 04:48They assess correctness, trust, usefulness, safety.
79. 04:52These signals become really critical for calibrating evaluation pipelines and identifying blind spots in automated metrics.
80. 04:59The most successful systems combine automated evaluation with targeted human review.
81. 05:06Now, agent systems drift constantly.
82. 05:08Model changes.
83. 05:09You have a new version every couple of weeks or months.
84. 05:13Prompts can change, tools can change, user behavior can change.
85. 05:17The challenge is that no longer single change appear catastrophic.
86. 05:20Reliability slowly degrades.
87. 05:22Success rate declines, escalation increases, tool failure rises.
88. 05:26Without continuous evaluation, teams often don't discover drift until users complain.
89. 05:31Continuous monitoring becomes essential.
90. 05:36Observability and evaluation are inseparable inseparable.
91. 05:40To evaluate an agent, we need visibility into the reasoning paths, the tool calls, the memory access, execution timelines, the state transitions, as you can see here in this chart,
92. 05:49traditional logs are not sufficient.
93. 05:51We need detailed traces, just like with any deep nested microservice architecture for any application or service you're talking about.
94. 06:00Agent traces become the equivalent of distributed tracing for autonomous workloads.
95. 06:03Without observability, evaluation becomes the guesswork.
96. 06:10Let's talk about the continuous evaluation loop because evaluation is an always-running service, not a testing phase.
97. 06:17Historically, evaluation always happened before deployment, but now evaluation continues after deployment till you may identify issues, as you can see in A. Human reviews the edge cases, feedback improves the data sets, offline scenarios validate updates,
98. 06:31the loop never stops.
99. 06:32Evaluation is no longer just a phase, it's an operational capability.
100. 06:38This is probably the most important slide in this presentation.
101. 06:42Every metric shown here maps directly to a business outcome.
102. 06:45Task completion measures value delivered.
103. 06:47Tool success measures operational reliability.
104. 06:50Escalation rate measures human burden.
105. 06:52Safety violations measure risk exposure.
106. 06:55Latency affects user experience.
107. 06:57Cost determines scalability.
108. 06:58Recovery rate reflects resilience.
109. 07:02And notice but notice that accuracy is missing.
110. 07:05It's not because accuracy doesn't matter, but because business success depends on much more than just accuracy.
111. 07:12Now, this is the architecture where the industry is heading more or less.
112. 07:16Evaluation becomes part of the control plane, not a separate tool, not an offline process.
113. 07:20The control plane continuously, which observes the systems, collects telemetry, runs simulations, coordinates human review, and the execution plane performs the work.
114. 07:30The control plane measures and governs the behavior, and this separation is becoming a foundational pattern for production AI systems.
115. 07:39Now, let's summarize the key lessons.
116. 07:40First, benchmarks remains necessary, but they are insufficient.
117. 07:45Second, agent systems must be valued as workflows, not individual outputs.
118. 07:50Third, production telemetry is a most important evaluation signal.
119. 07:55Fourth, reliability ultimately matters more than raw model accuracy.
120. 07:58And finally, evaluation is becoming the infrastructure, not testing, not QA infrastructure.
121. 08:06This is the shift every organization building agent AI will eventually need to make.