WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Lc8zRh9muoY

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Lc8zRh9muoY.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.360 --> 00:00:04.240
Imagine something your agent didn't prod was wrong.

s2
00:00:04.240 --> 00:00:14.240
Called the wrong tool, it wrote the wrong thing, and now suddenly your team is on on correlation to figure out what actually went wrong.

s3
00:00:14.240 --> 00:00:15.760
Pretty common, right?

s4
00:00:15.760 --> 00:00:30.080
Now, as for the standard engineering response, your gut will tell you to pull the raw prompt from the telemetry logs, pass it to the same model using the same prompt, and run it locally to isolate the bug.

s5
00:00:30.080 --> 00:00:31.360
Which we'll all do.

s6
00:00:31.360 --> 00:00:33.880
And surprisingly, it will work as well.

s7
00:00:33.880 --> 00:00:35.960
Run it again, it will work again.

s8
00:00:35.960 --> 00:00:40.520
You'll run it 10 more times, it will be just perfect every time.

s9
00:00:40.520 --> 00:00:46.480
But now let's talk about that one run which cost you, and that will be gone.

s10
00:00:46.480 --> 00:00:48.040
You can't reproduce it.

s11
00:00:48.040 --> 00:00:50.800
And if you can't reproduce it, you can't debug it.

s12
00:00:50.800 --> 00:00:57.240
And if you can't debug it, you can't promise it won't happen to your next customer or user, right?

s13
00:00:57.240 --> 00:00:58.400
Now, I am Tisha.

s14
00:00:58.400 --> 00:01:01.480
I have Sachin with me as my co-presenter.

s15
00:01:01.480 --> 00:01:05.360
We both run agents against real production backends.

s16
00:01:05.360 --> 00:01:09.480
You know, the kind of place where a bad write isn't, "Oh well, done it again."

s17
00:01:09.480 --> 00:01:15.480
It's you on a call with a customer explaining where the data actually went.

s18
00:01:15.480 --> 00:01:24.640
This whole talk is going to be about that one thing you lose the second an agent goes haywire in production, which is being able to reproduce it.

s19
00:01:24.640 --> 00:01:29.200
That will be our not start for the next 10 minutes to follow.

s20
00:01:29.200 --> 00:01:33.160
Now let's look at how this actually blows up.

s21
00:01:33.160 --> 00:01:37.640
You've got an agent hooked to a broker API, which is the scenario I'm taking.

s22
00:01:37.640 --> 00:01:42.360
The user says, "Hey, sell $1,000 of stock."

s23
00:01:42.360 --> 00:01:44.320
Now comes the interesting part.

s24
00:01:44.320 --> 00:01:53.280
Instead of doing the math, the agent sells the raw number 1,000 and dumps it straight into the quantity field.

s25
00:01:53.280 --> 00:01:53.840
Guess what?

s26
00:01:53.840 --> 00:01:56.320
It sells 1,000 shares instead.

s27
00:01:56.320 --> 00:02:02.760
Now, at of 190 bucks a share, a $1,000 Intel will become how much?

s28
00:02:02.760 --> 00:02:07.160
A $190,000 disaster, right?

s29
00:02:07.160 --> 00:02:14.280
And the terrifying part is that the API or my inferring returned a clean 200 okay in 30 milliseconds.

s30
00:02:14.280 --> 00:02:17.280
We got zero exceptions, zero alerts.

s31
00:02:17.280 --> 00:02:20.600
If you see the trade is completely wrong.

s32
00:02:20.600 --> 00:02:26.200
But, your dashboards are sitting there perfectly green, perfectly flawless.

s33
00:02:26.280 --> 00:02:30.520
Then, such a scenario as we last discussed comes up.

s34
00:02:30.520 --> 00:02:35.000
What's the first thing which you would do to try and fix this?

s35
00:02:35.000 --> 00:02:45.000
The reflex here is to, you know, just turn the model temperature down to absolute zero, assuming greedy decoding will make everything deterministic, right?

s36
00:02:45.000 --> 00:02:48.120
But, that's a complete misconception.

s37
00:02:48.120 --> 00:02:53.160
Setting the temperature to zero doesn't fix a broken reasoning path.

s38
00:02:53.160 --> 00:03:05.600
It just means the model is going to make the exact same logical error, the exact same way, at the exact same time, and honestly, even worse than that.

s39
00:03:05.600 --> 00:03:11.440
To back up the scenario we just discussed, look at the engineering threads on Reddit and Hacker News.

s40
00:03:11.440 --> 00:03:17.720
The hard data shows that temperature zero isn't even truly deterministic on a hardware level.

s41
00:03:17.720 --> 00:03:32.600
Running the same prompt a thousand times can still return dozens of completely different responses just due to the underlying GPU non-determinism and the MOE architectures which are there.

s42
00:03:32.600 --> 00:03:38.200
So, to understand why this actually happens, we'll have to look at it from first principles.

s43
00:03:38.200 --> 00:03:41.040
It comes down to four simple things.

s44
00:03:41.040 --> 00:03:46.519
One, sampling determinism isn't system determinism.

s45
00:03:46.519 --> 00:03:55.800
Temperature zero just means always take the argmax, but it doesn't guarantee that the underlying scores stay identical run to run.

s46
00:03:55.800 --> 00:03:59.680
Two, floating point math isn't associative.

s47
00:03:59.680 --> 00:04:03.400
The order you add your decimal matters, right?

s48
00:04:03.400 --> 00:04:12.120
But a tiny shift in matrix operation alters the final logits and which in turn will flip the winning token.

s49
00:04:12.120 --> 00:04:15.560
Three, it's not a concurrency issue.

s50
00:04:15.560 --> 00:04:23.000
Run the same matrix multiplication alone on a GPU a thousand times and I'll guarantee you'll get this exact same bits.

s51
00:04:23.000 --> 00:04:32.160
So the real culprit is batch invariance here because a request gets grouped with whatever else hits the server that millisecond.

s52
00:04:32.160 --> 00:04:36.640
Four, mixture of experts routing has the exact same bottleneck.

s53
00:04:36.640 --> 00:04:39.280
Experts have strict capacity limits.

s54
00:04:39.280 --> 00:04:44.640
If a batch overflows a specific subnetwork, tokens get rerouted.

s55
00:04:44.640 --> 00:04:51.040
Whether the token makes the cut depends entirely on the traffic you got batched with.

s56
00:04:51.040 --> 00:04:57.960
So the ultimate takeaway here is that chasing text outputs is a losing battle, completely.

s57
00:04:57.960 --> 00:05:02.680
We don't need the model to return the exact same token back every time.

s58
00:05:02.680 --> 00:05:09.320
We just need our system to execute the exact same state transition.

s59
00:05:09.320 --> 00:05:12.919
Which means we've been asking the wrong question all along, right?

s60
00:05:12.919 --> 00:05:17.440
The wrong question is how do I make the model deterministic?

s61
00:05:17.440 --> 00:05:23.160
And I've seen teams burning weeks on that and walk away deciding the system is just unknowable.

s62
00:05:23.160 --> 00:05:31.840
The right question is how do I debug and retest a run I can't reproduce because determinism was never the North Star.

s63
00:05:31.840 --> 00:05:33.560
Debugging was.

s64
00:05:33.560 --> 00:05:40.680
Two words we keep mixing up, which I'll talk about now, is bitwise determinism and replayability.

s65
00:05:40.680 --> 00:05:44.000
Bitwise determinism is same input, same output.

s66
00:05:44.000 --> 00:05:45.560
That's controllability.

s67
00:05:45.560 --> 00:05:53.120
You're not getting it from a hosted API, and you don't actually want it because the randomness is what makes the model good.

s68
00:05:53.120 --> 00:05:56.840
Like once the model explores more, you'll get more creative answers.

s69
00:05:56.840 --> 00:06:03.520
The other one is replayability, which is re-validates a run that already happened well enough to debug it.

s70
00:06:03.520 --> 00:06:04.960
That's observability.

s71
00:06:04.960 --> 00:06:07.080
You don't need the model deterministic.

s72
00:06:07.080 --> 00:06:10.880
You need the run recorded, and you don't freeze the model.

s73
00:06:10.880 --> 00:06:14.360
You capture what it did.

s74
00:06:14.360 --> 00:06:20.120
Now, the question which we are all thinking about is where do you record?

s75
00:06:20.120 --> 00:06:33.760
For sure, not at the network layer because half your agent will never touch the network, the local retrieval, the in-process tools, the memory, and the parts that do not shred under streaming and async.

s76
00:06:33.760 --> 00:06:41.880
Record at the boundary instead because you need to capture what enters each node and what leaves it.

s77
00:06:41.880 --> 00:06:45.280
The meaning of each step and not the packets.

s78
00:06:45.280 --> 00:06:57.320
What replay adds here is a deterministic CI where you stop the model, you rerun the exact failure of line with zero model calls.

s79
00:06:57.320 --> 00:06:59.160
Let's talk about the loop end to end now.

s80
00:06:59.160 --> 00:07:08.760
It starts with annotation, recording, visualization, understanding, fixing, then the part we're working on, which is deep playing, and finally verifying.

s81
00:07:08.760 --> 00:07:09.640
All right.

s82
00:07:09.640 --> 00:07:15.160
Let's now see how to bring the workflow we just discussed into action.

s83
00:07:15.160 --> 00:07:20.320
So, we've established that replayability is a core tenant of productionizing any AI agent.

s84
00:07:20.320 --> 00:07:22.640
But, how do we build this in code?

s85
00:07:22.640 --> 00:07:26.960
As a proof of concept for this, what we have done is we have built something called Chronicle.

s86
00:07:26.960 --> 00:07:30.560
At the heart of Chronicle lies the concept of a boundary.

s87
00:07:30.560 --> 00:07:35.440
Think of a boundary as a bounding box around any node in your agentic workflow.

s88
00:07:35.440 --> 00:07:36.840
A node can be a tool call.

s89
00:07:36.840 --> 00:07:40.560
It can be a call to an LLM or a retrieval from a rag.

s90
00:07:40.560 --> 00:07:41.720
It doesn't matter.

s91
00:07:41.720 --> 00:07:46.240
As long as it's a method, it can be annotated with the boundary annotation.

s92
00:07:46.240 --> 00:07:47.640
Now, what does this annotation do?

s93
00:07:47.640 --> 00:07:51.760
It ensures that anything that goes into the method and comes out of the method gets recorded.

s94
00:07:51.760 --> 00:07:54.240
So, any input and output pair will get recorded.

s95
00:07:54.240 --> 00:08:05.280
On top of that, you can define parameters like your model version or the version of the code that is running so that the entire state during which the agent run happened gets frozen and saved as a trace.

s96
00:08:05.280 --> 00:08:07.280
Now, let's see this in action.

s97
00:08:07.280 --> 00:08:11.520
We've been talking about this stock selling agent which went haywire in production.

s98
00:08:11.520 --> 00:08:13.240
This is a representation of the same.

s99
00:08:13.240 --> 00:08:17.240
You have your initial planning step which takes into account the user input.

s100
00:08:17.240 --> 00:08:26.919
It can use the place order tool to do the actual selling and buying of the stocks, and then finally it delegates to the finalize agent which generates a succinct response for the end user.

s101
00:08:26.919 --> 00:08:31.200
We have annotated all three of these methods with the boundary annotation.

s102
00:08:31.200 --> 00:08:34.599
The first one is a tool, and the second and the third one is an LLM.

s103
00:08:34.599 --> 00:08:36.880
Let's run this.

s104
00:08:40.000 --> 00:08:41.560
So, here's what happened.

s105
00:08:41.560 --> 00:08:45.839
You gave a user request to sell about $1,000 worth of Acme stocks.

s106
00:08:45.839 --> 00:08:46.920
You have the three nodes.

s107
00:08:46.920 --> 00:08:49.839
You have the input and the output recorded for every one of them.

s108
00:08:49.839 --> 00:08:59.600
You can see that the LLM mistook the thousand as the quantity and generated a tool call of place order with the symbol Acme and quantity thousand.

s109
00:08:59.600 --> 00:09:08.880
And the place order tool obviously executed this input and sold $1,000 or 1,000 units of Acme stock at $190 per piece.

s110
00:09:08.880 --> 00:09:10.880
This is where the problem started.

s111
00:09:10.880 --> 00:09:13.720
Now, you have a trace for it, but is this all you can see?

s112
00:09:13.720 --> 00:09:17.000
No. We record much more details than this.

s113
00:09:17.000 --> 00:09:24.240
So, you can go and see the hyper detailed JSON for each of the nodes that the call went into or went out of.

s114
00:09:24.240 --> 00:09:28.920
You can see the metadata like the model version, the sampling versions that was there in the LLM call.

s115
00:09:28.920 --> 00:09:30.400
You can see the input and the output.

s116
00:09:30.400 --> 00:09:32.560
For example, this is for the place order tool.

s117
00:09:32.560 --> 00:09:39.760
You can see the input went in as a sell of thousand quantity on Acme, and the output was obviously that it sold that quantity.

s118
00:09:39.760 --> 00:09:46.800
And if you go one step back, you can see the agent one node creating that tool call that caused this entire problem.

s119
00:09:46.800 --> 00:09:51.040
It created a tool call to place order with the symbol Acme and quantity 1,000.

s120
00:09:51.040 --> 00:09:52.960
Now, this is all good.

s121
00:09:52.960 --> 00:09:56.720
You have a recording, you have a trace, you figured out what the problem is.

s122
00:09:56.720 --> 00:09:58.120
Now, what do you do with it?

s123
00:09:58.120 --> 00:10:02.560
You figured out that the problem is the LLM created a wrong tool call.

s124
00:10:02.560 --> 00:10:03.760
You cannot control the LLM.

s125
00:10:03.760 --> 00:10:06.240
That's what this entire discussion is about.

s126
00:10:06.240 --> 00:10:08.920
You cannot enforce bitwise determinism.

s127
00:10:08.920 --> 00:10:17.200
What you can do is you can put guardrails on your tools to enforce some level of uh credibility on your production agent.

s128
00:10:17.200 --> 00:10:18.800
But, how do you test this?

s129
00:10:18.800 --> 00:10:21.240
Once you have built the guardrails, how do you test it?

s130
00:10:21.240 --> 00:10:24.000
That is another tenant that Boundary offers.

s131
00:10:24.000 --> 00:10:32.160
Since the Boundary annotation is already providing a bounding box around your methods, it can be used to stub your methods in during testing.

s132
00:10:32.160 --> 00:10:33.760
So, think of it like this.

s133
00:10:33.760 --> 00:10:35.880
You have a run which is recorded.

s134
00:10:35.880 --> 00:10:38.880
That run recorded every input and output for every node.

s135
00:10:38.880 --> 00:10:47.920
Now, you have fixed your code at let's say the tool level, but you want the rest of the nodes to be stubbed so that the entire exact stack trace remains the same.

s136
00:10:47.920 --> 00:10:48.760
How do you do it?

s137
00:10:48.760 --> 00:10:53.320
You run a test suite with the same trace that was recorded earlier.

s138
00:10:53.320 --> 00:11:00.000
You stub every node other than the node that you changed, and you let Boundary handle the rest, right?

s139
00:11:00.000 --> 00:11:02.400
So, let's see this one in action.

s140
00:11:02.400 --> 00:11:03.520
This is your test case.

s141
00:11:03.520 --> 00:11:12.160
You have loaded the trace, and you have enabled the replay mode on Boundary, which allows Boundary to stub every node that you want.

s142
00:11:12.160 --> 00:11:17.120
So, for example, in this case, you want to stub the first agent call which generated the tool output.

s143
00:11:17.120 --> 00:11:21.680
But, you want the tool to be run live so that you can test out your code changes.

s144
00:11:21.680 --> 00:11:23.720
Once you do this, you run your agent.

s145
00:11:23.720 --> 00:11:30.680
Now, another good thing about Boundary is that since it's already capturing the output of your tool as well, you can use it to run assertions.

s146
00:11:30.680 --> 00:11:35.720
So, you can take the output and assert that this time around the tool call got blocked.

s147
00:11:35.720 --> 00:11:38.760
So, let's run this once.

s148
00:11:41.120 --> 00:11:41.720
Perfect.

s149
00:11:41.720 --> 00:11:45.560
So, you can see that agent one, which was the LLM, was stubbed.

s150
00:11:45.560 --> 00:11:52.280
You can see the same input output as the recorded trace, but the tool and the LLM ran live.

s151
00:11:52.280 --> 00:11:56.040
And you can see that for the tool, this time the output is blocked.

s152
00:11:56.040 --> 00:11:59.600
And your assertion on the tool has passed because the order has blocked.

s153
00:11:59.600 --> 00:12:08.720
So, this is the power of merging your replayability traces with auto-generated testing and stubbing and assertions.

s154
00:12:08.720 --> 00:12:09.800
tool and the LLM.

s155
00:12:09.800 --> 00:12:16.440
So, we just saw how Chronicle not only records your agentic sessions, but it also uses those recordings as test cases.

s156
00:12:16.440 --> 00:12:20.880
Now, when we talk about testing in AI agents, I want to draw a very clear distinction here.

s157
00:12:20.880 --> 00:12:24.840
There are two ways of testing AI agents, and both of them are equally important.

s158
00:12:24.840 --> 00:12:27.760
There's the deterministic testing, and then the behavioral testing.

s159
00:12:27.760 --> 00:12:32.560
The deterministic testing applies to, obviously, the deterministic nodes of your agent graph.

s160
00:12:32.560 --> 00:12:35.000
This could be your guardrails or your tool calls.

s161
00:12:35.000 --> 00:12:42.720
Now, this is exactly where Chronicle shines because, as we just saw, Chronicle freezes the entire agent run as a context.

s162
00:12:42.720 --> 00:12:47.280
So, you can use the LLM nodes context to stub the LLM outputs.

s163
00:12:47.280 --> 00:12:52.880
This essentially kicks the probability out of the window, and your entire agent run can become a test case.

s164
00:12:52.880 --> 00:12:56.560
This is rerun-able, and since it never calls the model, it is free.

s165
00:12:56.560 --> 00:13:03.640
On the behavioral side of things, you measure things like the tone of the agent or whether the trajectory it took was right.

s166
00:13:03.640 --> 00:13:08.200
This is more subjective, and this is where techniques like LLM as a judge are better off.

s167
00:13:08.200 --> 00:13:11.040
Now, at this point, I want to talk about the key takeaways.

s168
00:13:11.040 --> 00:13:14.360
First, stop chasing bitwise determinism through the API.

s169
00:13:14.360 --> 00:13:18.800
The fundamental principles on which the APIs are built today do not make this possible.

s170
00:13:18.800 --> 00:13:28.080
Second, know what are variables for your session, for example, your LLM version or your build ID or your rag chunks, and make sure that you are logging these.

s171
00:13:28.080 --> 00:13:30.240
Third, capture the full envelope.

s172
00:13:30.240 --> 00:13:32.000
Don't focus on just the prompt.

s173
00:13:32.000 --> 00:13:36.400
There are There are a lot more ingredients that go into that final response.

s174
00:13:36.400 --> 00:13:38.760
Fourth, use the replace to debug.

s175
00:13:38.760 --> 00:13:44.640
Make sure that you find the issues, fix the failures, and then finally, use the same trace as a test case.

s176
00:13:44.640 --> 00:13:47.360
Fifth and final, keep the generation time variation alive.

s177
00:13:47.360 --> 00:13:49.120
Don't try to pin the temperature to zero.

s178
00:13:49.120 --> 00:13:52.680
After all, that is what brings the agency into your agent.

s179
00:13:52.680 --> 00:13:59.680
So, the QR on your screen, you can scan that to get access to the code for Chronicle and a bunch of nicely written articles.

s180
00:13:59.680 --> 00:14:08.200
And finally, thanks again for your time, and we hope that the traces that you put today in your agent make sure that your on-call cycle tomorrow is much better.

s181
00:14:08.200 --> 00:14:10.400
Thank you.