Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft

AI Engineer · 14 min · 181 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:00Imagine something your agent didn't prod was wrong.
2. 00:04Called the wrong tool, it wrote the wrong thing, and now suddenly your team is on on correlation to figure out what actually went wrong.
3. 00:14Pretty common, right?
4. 00:15Now, as for the standard engineering response, your gut will tell you to pull the raw prompt from the telemetry logs, pass it to the same model using the same prompt, and run it locally to isolate the bug.
5. 00:30Which we'll all do.
6. 00:31And surprisingly, it will work as well.
7. 00:33Run it again, it will work again.
8. 00:35You'll run it 10 more times, it will be just perfect every time.
9. 00:40But now let's talk about that one run which cost you, and that will be gone.
10. 00:46You can't reproduce it.
11. 00:48And if you can't reproduce it, you can't debug it.
12. 00:50And if you can't debug it, you can't promise it won't happen to your next customer or user, right?
13. 00:57Now, I am Tisha.
14. 00:58I have Sachin with me as my co-presenter.
15. 01:01We both run agents against real production backends.
16. 01:05You know, the kind of place where a bad write isn't, "Oh well, done it again."
17. 01:09It's you on a call with a customer explaining where the data actually went.
18. 01:15This whole talk is going to be about that one thing you lose the second an agent goes haywire in production, which is being able to reproduce it.
19. 01:24That will be our not start for the next 10 minutes to follow.
20. 01:29Now let's look at how this actually blows up.
21. 01:33You've got an agent hooked to a broker API, which is the scenario I'm taking.
22. 01:37The user says, "Hey, sell $1,000 of stock."
23. 01:42Now comes the interesting part.
24. 01:44Instead of doing the math, the agent sells the raw number 1,000 and dumps it straight into the quantity field.
25. 01:53Guess what?
26. 01:53It sells 1,000 shares instead.
27. 01:56Now, at of 190 bucks a share, a $1,000 Intel will become how much?
28. 02:02A $190,000 disaster, right?
29. 02:07And the terrifying part is that the API or my inferring returned a clean 200 okay in 30 milliseconds.
30. 02:14We got zero exceptions, zero alerts.
31. 02:17If you see the trade is completely wrong.
32. 02:20But, your dashboards are sitting there perfectly green, perfectly flawless.
33. 02:26Then, such a scenario as we last discussed comes up.
34. 02:30What's the first thing which you would do to try and fix this?
35. 02:35The reflex here is to, you know, just turn the model temperature down to absolute zero, assuming greedy decoding will make everything deterministic, right?
36. 02:45But, that's a complete misconception.
37. 02:48Setting the temperature to zero doesn't fix a broken reasoning path.
38. 02:53It just means the model is going to make the exact same logical error, the exact same way, at the exact same time, and honestly, even worse than that.
39. 03:05To back up the scenario we just discussed, look at the engineering threads on Reddit and Hacker News.
40. 03:11The hard data shows that temperature zero isn't even truly deterministic on a hardware level.
41. 03:17Running the same prompt a thousand times can still return dozens of completely different responses just due to the underlying GPU non-determinism and the MOE architectures which are there.
42. 03:32So, to understand why this actually happens, we'll have to look at it from first principles.
43. 03:38It comes down to four simple things.
44. 03:41One, sampling determinism isn't system determinism.
45. 03:46Temperature zero just means always take the argmax, but it doesn't guarantee that the underlying scores stay identical run to run.
46. 03:55Two, floating point math isn't associative.
47. 03:59The order you add your decimal matters, right?
48. 04:03But a tiny shift in matrix operation alters the final logits and which in turn will flip the winning token.
49. 04:12Three, it's not a concurrency issue.
50. 04:15Run the same matrix multiplication alone on a GPU a thousand times and I'll guarantee you'll get this exact same bits.
51. 04:23So the real culprit is batch invariance here because a request gets grouped with whatever else hits the server that millisecond.
52. 04:32Four, mixture of experts routing has the exact same bottleneck.
53. 04:36Experts have strict capacity limits.
54. 04:39If a batch overflows a specific subnetwork, tokens get rerouted.
55. 04:44Whether the token makes the cut depends entirely on the traffic you got batched with.
56. 04:51So the ultimate takeaway here is that chasing text outputs is a losing battle, completely.
57. 04:57We don't need the model to return the exact same token back every time.
58. 05:02We just need our system to execute the exact same state transition.
59. 05:09Which means we've been asking the wrong question all along, right?
60. 05:12The wrong question is how do I make the model deterministic?
61. 05:17And I've seen teams burning weeks on that and walk away deciding the system is just unknowable.
62. 05:23The right question is how do I debug and retest a run I can't reproduce because determinism was never the North Star.
63. 05:31Debugging was.
64. 05:33Two words we keep mixing up, which I'll talk about now, is bitwise determinism and replayability.
65. 05:40Bitwise determinism is same input, same output.
66. 05:44That's controllability.
67. 05:45You're not getting it from a hosted API, and you don't actually want it because the randomness is what makes the model good.
68. 05:53Like once the model explores more, you'll get more creative answers.
69. 05:56The other one is replayability, which is re-validates a run that already happened well enough to debug it.
70. 06:03That's observability.
71. 06:04You don't need the model deterministic.
72. 06:07You need the run recorded, and you don't freeze the model.
73. 06:10You capture what it did.
74. 06:14Now, the question which we are all thinking about is where do you record?
75. 06:20For sure, not at the network layer because half your agent will never touch the network, the local retrieval, the in-process tools, the memory, and the parts that do not shred under streaming and async.
76. 06:33Record at the boundary instead because you need to capture what enters each node and what leaves it.
77. 06:41The meaning of each step and not the packets.
78. 06:45What replay adds here is a deterministic CI where you stop the model, you rerun the exact failure of line with zero model calls.
79. 06:57Let's talk about the loop end to end now.
80. 06:59It starts with annotation, recording, visualization, understanding, fixing, then the part we're working on, which is deep playing, and finally verifying.
81. 07:08All right.
82. 07:09Let's now see how to bring the workflow we just discussed into action.
83. 07:15So, we've established that replayability is a core tenant of productionizing any AI agent.
84. 07:20But, how do we build this in code?
85. 07:22As a proof of concept for this, what we have done is we have built something called Chronicle.
86. 07:26At the heart of Chronicle lies the concept of a boundary.
87. 07:30Think of a boundary as a bounding box around any node in your agentic workflow.
88. 07:35A node can be a tool call.
89. 07:36It can be a call to an LLM or a retrieval from a rag.
90. 07:40It doesn't matter.
91. 07:41As long as it's a method, it can be annotated with the boundary annotation.
92. 07:46Now, what does this annotation do?
93. 07:47It ensures that anything that goes into the method and comes out of the method gets recorded.
94. 07:51So, any input and output pair will get recorded.
95. 07:54On top of that, you can define parameters like your model version or the version of the code that is running so that the entire state during which the agent run happened gets frozen and saved as a trace.
96. 08:05Now, let's see this in action.
97. 08:07We've been talking about this stock selling agent which went haywire in production.
98. 08:11This is a representation of the same.
99. 08:13You have your initial planning step which takes into account the user input.
100. 08:17It can use the place order tool to do the actual selling and buying of the stocks, and then finally it delegates to the finalize agent which generates a succinct response for the end user.
101. 08:26We have annotated all three of these methods with the boundary annotation.
102. 08:31The first one is a tool, and the second and the third one is an LLM.
103. 08:34Let's run this.
104. 08:40So, here's what happened.
105. 08:41You gave a user request to sell about $1,000 worth of Acme stocks.
106. 08:45You have the three nodes.
107. 08:46You have the input and the output recorded for every one of them.
108. 08:49You can see that the LLM mistook the thousand as the quantity and generated a tool call of place order with the symbol Acme and quantity thousand.
109. 08:59And the place order tool obviously executed this input and sold $1,000 or 1,000 units of Acme stock at $190 per piece.
110. 09:08This is where the problem started.
111. 09:10Now, you have a trace for it, but is this all you can see?
112. 09:13No. We record much more details than this.
113. 09:17So, you can go and see the hyper detailed JSON for each of the nodes that the call went into or went out of.
114. 09:24You can see the metadata like the model version, the sampling versions that was there in the LLM call.
115. 09:28You can see the input and the output.
116. 09:30For example, this is for the place order tool.
117. 09:32You can see the input went in as a sell of thousand quantity on Acme, and the output was obviously that it sold that quantity.
118. 09:39And if you go one step back, you can see the agent one node creating that tool call that caused this entire problem.
119. 09:46It created a tool call to place order with the symbol Acme and quantity 1,000.
120. 09:51Now, this is all good.
121. 09:52You have a recording, you have a trace, you figured out what the problem is.
122. 09:56Now, what do you do with it?
123. 09:58You figured out that the problem is the LLM created a wrong tool call.
124. 10:02You cannot control the LLM.
125. 10:03That's what this entire discussion is about.
126. 10:06You cannot enforce bitwise determinism.
127. 10:08What you can do is you can put guardrails on your tools to enforce some level of uh credibility on your production agent.
128. 10:17But, how do you test this?
129. 10:18Once you have built the guardrails, how do you test it?
130. 10:21That is another tenant that Boundary offers.
131. 10:24Since the Boundary annotation is already providing a bounding box around your methods, it can be used to stub your methods in during testing.
132. 10:32So, think of it like this.
133. 10:33You have a run which is recorded.
134. 10:35That run recorded every input and output for every node.
135. 10:38Now, you have fixed your code at let's say the tool level, but you want the rest of the nodes to be stubbed so that the entire exact stack trace remains the same.
136. 10:47How do you do it?
137. 10:48You run a test suite with the same trace that was recorded earlier.
138. 10:53You stub every node other than the node that you changed, and you let Boundary handle the rest, right?
139. 11:00So, let's see this one in action.
140. 11:02This is your test case.
141. 11:03You have loaded the trace, and you have enabled the replay mode on Boundary, which allows Boundary to stub every node that you want.
142. 11:12So, for example, in this case, you want to stub the first agent call which generated the tool output.
143. 11:17But, you want the tool to be run live so that you can test out your code changes.
144. 11:21Once you do this, you run your agent.
145. 11:23Now, another good thing about Boundary is that since it's already capturing the output of your tool as well, you can use it to run assertions.
146. 11:30So, you can take the output and assert that this time around the tool call got blocked.
147. 11:35So, let's run this once.
148. 11:41Perfect.
149. 11:41So, you can see that agent one, which was the LLM, was stubbed.
150. 11:45You can see the same input output as the recorded trace, but the tool and the LLM ran live.
151. 11:52And you can see that for the tool, this time the output is blocked.
152. 11:56And your assertion on the tool has passed because the order has blocked.
153. 11:59So, this is the power of merging your replayability traces with auto-generated testing and stubbing and assertions.
154. 12:08tool and the LLM.
155. 12:09So, we just saw how Chronicle not only records your agentic sessions, but it also uses those recordings as test cases.
156. 12:16Now, when we talk about testing in AI agents, I want to draw a very clear distinction here.
157. 12:20There are two ways of testing AI agents, and both of them are equally important.
158. 12:24There's the deterministic testing, and then the behavioral testing.
159. 12:27The deterministic testing applies to, obviously, the deterministic nodes of your agent graph.
160. 12:32This could be your guardrails or your tool calls.
161. 12:35Now, this is exactly where Chronicle shines because, as we just saw, Chronicle freezes the entire agent run as a context.
162. 12:42So, you can use the LLM nodes context to stub the LLM outputs.
163. 12:47This essentially kicks the probability out of the window, and your entire agent run can become a test case.
164. 12:52This is rerun-able, and since it never calls the model, it is free.
165. 12:56On the behavioral side of things, you measure things like the tone of the agent or whether the trajectory it took was right.
166. 13:03This is more subjective, and this is where techniques like LLM as a judge are better off.
167. 13:08Now, at this point, I want to talk about the key takeaways.
168. 13:11First, stop chasing bitwise determinism through the API.
169. 13:14The fundamental principles on which the APIs are built today do not make this possible.
170. 13:18Second, know what are variables for your session, for example, your LLM version or your build ID or your rag chunks, and make sure that you are logging these.
171. 13:28Third, capture the full envelope.
172. 13:30Don't focus on just the prompt.
173. 13:32There are There are a lot more ingredients that go into that final response.
174. 13:36Fourth, use the replace to debug.
175. 13:38Make sure that you find the issues, fix the failures, and then finally, use the same trace as a test case.
176. 13:44Fifth and final, keep the generation time variation alive.
177. 13:47Don't try to pin the temperature to zero.
178. 13:49After all, that is what brings the agency into your agent.
179. 13:52So, the QR on your screen, you can scan that to get access to the code for Chronicle and a bunch of nicely written articles.
180. 13:59And finally, thanks again for your time, and we hope that the traces that you put today in your agent make sure that your on-call cycle tomorrow is much better.
181. 14:08Thank you.