Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog

AI Engineer · 25 min · 199 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:02Hello.
2. 00:04This session is about why do your AI agents disagree with itself and what to do about it.
3. 00:13And this is a Dyan Huang Lin, and you can call me Dyan.
4. 00:19A little bit about my background.
5. 00:22My journey with AGI start from my PhD at Imperial College London, where I was working on continual learning.
6. 00:31Later, I worked with Professor Josh Tenenbaum at MIT on one-shot learning, meaning learning from one single examples.
7. 00:42Afterwards, I was lucky that Alexa was launching.
8. 00:46I was the first three appliance scientists on the question answering team at the Alexa.
9. 00:53Later, attracted by a startup working on AGI at the Silicon Valley called Vicarious.
10. 01:01So, today Vicarious is part of the Google DeepMind.
11. 01:05And there I got to work on the exciting frontier research on zero-shot transfer learning.
12. 01:13After working on AGI for a few years, I decided to pivot to the applied world at Zscaler, which is cybersecurity company.
13. 01:25There I got to apply a variety of different machine learning model to adjust the cybersecurity challenge.
14. 01:34There for 5 years, uh I decided to co-found my own company because of pinpoint I have seen at the Zscaler.
15. 01:44So, company is building AI agent to auto-triage your security alerts and which make your SOC more efficient.
16. 01:55Uh we are super proud that Chrome innate has been acquired by Data Dog earlier this year.
17. 02:02So now I'm part of the Data Dog and leading the development of self-evolved agent.
18. 02:14So today I'm going to share with you the problem you're probably seeing your day-to-day building AI agent.
19. 02:21The inconsistency.
20. 02:24Then we are discussed about where this inconsistency come from in order to figure out the solutions.
21. 02:32And we will discuss a few trade-off among the different solutions and then eventually we'll show you the experimental results to show how feasible they are.
22. 02:46You'll probably see this.
23. 02:48Same model, same input.
24. 02:51But different output.
25. 02:52Here I don't mean the wording different.
26. 02:55I mean semantically different output.
27. 03:01You might be thinking that's the stochastic nature of LM.
28. 03:07Hold on to that thought.
29. 03:09Let's look at a few concrete examples.
30. 03:13So here imagine you're giving a task for sentiment analysis where we're supposed to label each of the review of the hotel either positive or negative.
31. 03:26This is typical NLP task.
32. 03:29And today's LM is really good at this.
33. 03:33However, there's one problem.
34. 03:36When you pipe this input to the same LM model, occasionally you will see different verdict coming out of it when you run multiple times.
35. 03:49That means a single evaluation run didn't tell you the whole story.
36. 03:55You need to repeat your evaluation multiple times in order to get a holistic picture by the average over the different runs results.
37. 04:10And this sounds like a technical inconvenience, but the reality is is way more problematic than just inconvenience.
38. 04:21Let me show you an example in a different domain in cybersecurity.
39. 04:26Imagine you are a security analyst where your day-to-day job is triage the security alerts and decide whether it's malicious that you need to take certain actions to stop the attack
40. 04:40or it's false alarm, meaning it's actually benign, which you could ignore.
41. 04:47So, here example alert you got is there's a failed logging attempt to Gmail account detected from suspicious IP.
42. 04:58And if we give such similar task to this one to AI agent and run it different times, you will see sometimes it stay consistently benign, sometimes it stay consistently suspicious
43. 05:13across different runs.
44. 05:15However, there are cases where it flip-flop between benign and suspicious.
45. 05:23Now, your customer will have a difficulty using such product because they will wonder, "Which one should I trust?"
46. 05:33So, the inconsistency is causing a trust issue in your product.
47. 05:41Imagine if you were in a POC bake-off, one vendor give the consistent wording all the time while the other vendor have this flip-flop wording.
48. 05:59You can imagine which one would win the deal.
49. 06:03Now, I'm pretty sure you want to solve this problem.
50. 06:11First, we have we have to figure out where the inconsistency come from.
51. 06:21The good news is the data points tend to flip-flop actually concentrate around the decision boundary, the so-called gray zone.
52. 06:34Let me illustrate with the earlier example.
53. 06:37Here in the sentiment analysis, we have two reviews.
54. 06:42The first one actually no matter how many times you run through the LM model, it will always give you the positive label.
55. 06:53In this case, it's pretty obvious is the review is super positive.
56. 06:59However, the second case is the one tend to flip.
57. 07:05If we look closer, it's kind of on the boundary.
58. 07:09Sometimes there's certain word is kind of a relative positive, but sometimes there's a little bit negative.
59. 07:20In this case, you'll find that even human experts will have a disagreement on these cases.
60. 07:29In fact, there's no right or wrong answer.
61. 07:32It really depends on particular company's policy and their preference.
62. 07:37For hotel thinking that the such review is not something I can do to improve the memorial kind of experience, memorable experience.
63. 07:49Then, that mean not to label as naughty because there's nothing I can do.
64. 07:55But some hotel might care about it and want to label them as natives so they can improve.
65. 08:01So, that comes down your preference.
66. 08:06Let's look the other cybersecurity example.
67. 08:10Here is another case on the boundary.
68. 08:14And why it's in the boundary is here there's a few attempts.
69. 08:21Indeed, there's attacker knocking on the door.
70. 08:26But attacker might not getting yet.
71. 08:32So, for enterprise customers who always have attacker knocking on the door, well, if they haven't getting in the environment, they don't want to worry about it.
72. 08:42Otherwise, they will be flooded by this type of alerts.
73. 08:45There's nothing they need to do because they are blocked outside already.
74. 08:51However, if the attacker do manage to get the correct password and pass MFA and do manage getting the environment, then it become very seriously and you need to take certain actions.
75. 09:04So, almost similarly at the beginning behavior, but very different outcome and different response needed.
76. 09:14Depends on whether it's truly attack or attacker is still outside.
77. 09:20So, here whether we label them malicious or benign depends on company preference whether they wanted to be notified about such situation.
78. 09:33So, again, this is the example where it depends on your preference.
79. 09:41And additional information to disambiguate between the two different scenarios.
80. 09:50So from these two very different domains, you will see that actually the data points tend to flip-flop.
81. 10:00Are the one close to the decision boundary where it need your clarity to disambiguate which side it should really belongs to.
82. 10:14So these data points are where human experts tend to make a mistake.
83. 10:19And the traditional machine learning classifier will also struggle with.
84. 10:25So in other words it's not your AI agent's fault.
85. 10:30Your AI agent simply point out the ambiguity that already exists.
86. 10:38So now we know where the root cause of the inconsistent is.
87. 10:47The next step is identify where exactly they are.
88. 10:53Who are they?
89. 10:54And fix them.
90. 10:59Surprisingly and not surprised the solution to identify the gray zone actually exists already in machine learning.
91. 11:09And it's called active learning.
92. 11:13So the idea of active learning is you have a lot of data point to be labeled.
93. 11:21Imagine you'd have your initial model and you put it in production, maybe in the monitor mode.
94. 11:28And you wanted to check the quality.
95. 11:32But you don't have bandwidth to check all of them.
96. 11:35Otherwise, it lose a point of having agent to do the work.
97. 11:39But you still want to identify where the model didn't do well.
98. 11:46And then active learning is to find a way to select the data points where tend to make mistake.
99. 11:56And that's where it tells you where to pull your attention.
100. 12:03So let's go over the traditional um machine learning models active learning pipeline and see how is different in the case of the new LM or AI agent error.
101. 12:16So first, you train your initial model and then you make predictions on the unlabeled data set.
102. 12:24It could be the production online data.
103. 12:29And then from there, you see there's certain cases where the model wasn't sure because the probability is close to 2.5. And those data points are the high informational one
104. 12:42that your model is going to learn the most.
105. 12:46The uncertainty is a type of the signal to tell you where the model tend to make a mistake.
106. 12:53Another popular type of way to figure out the problematic one is query by committee, meaning that it by a few different models or even same model multiple times and then you identify
107. 13:08the disagreement.
108. 13:10And that's where the model tend to make a mistake as well and that's where you need human clarification on label so the model can learn it from.
109. 13:23The after you surface these problematic data points and for human to take a second look to make sure the label is actually correct or do we doesn't need additional information to dis
110. 13:37ambiguate those cases meaning adding new features.
111. 13:41And then you can add those label data set or potentially additional features to the next round of training.
112. 13:48Then you retrain the model and then proceed uh continue this cycle.
113. 13:54So then you can see in this particular active learning cycle you spend very little uh amount or at least in other way it's very efficient way to identify the problematic
114. 14:08data points and spend your attention on those point where your model tend to learn the most.
115. 14:18So that's the traditional one.
116. 14:20The good news is the LM way is not very different.
117. 14:27There's a major two part which we need some refinement.
118. 14:30One is on this selection strategy.
119. 14:34The other part is about retraining.
120. 14:37On the selection strategy one um here we suggest you use the one about identify disagreement.
121. 14:47At least from what we have tried so far the uncertainty score from LM is not very reliable.
122. 14:54It's kind of a LM model doesn't know what it doesn't know.
123. 14:58So when it's very confident it doesn't mean that it was the it would is correct.
124. 15:04So that However, the the disagreement from different runs or from different models actually give you a more reliable signal where the model actually not sure about its verdict and it need human guidance.
125. 15:25And then after you identify this group or the uh discriminant and need attention, you label them and provide a feedback like what you did before.
126. 15:37Then afterwards, you might be thinking, "Oh, time to retrain the model."
127. 15:42Yes or no. Yes, one option is retrain the model, but fine-tuning model is expensive.
128. 15:50Here, we are proposing a more lighter weight solution, easier to iterate.
129. 16:01It's about augment your agent with semantic and episodic memory.
130. 16:21Let's use the previous example to illustrate what I mean here.
131. 16:25Um as we talked about, in this case, the attempt to flip, it's mainly missing the information about you a particular company policy or preference.
132. 16:39Here, we can add it to the um the knowledge base.
133. 16:45If this customer complain about something outside hotel's control, then classify as positive because you don't want to pay attention to this.
134. 17:00And then in the cybersecurity example, something similar, you can put in in that the password spray, meaning that the attacker trying to guess your password and try it with different times, even fail, and continuously.
135. 17:16So, if without successful logging, then that should be benign.
136. 17:19But, in contrast, if it succeed with final logging, then that should be malicious.
137. 17:26So, here you actually identify the additional information to help disambiguate the two and then label them separately.
138. 17:35And that's a clarity that help your model to learn.
139. 17:41Also, help your human experts to label more consistently.
140. 17:48So, this type of domain knowledge sharpen your decision boundary.
141. 17:53And also, help the human beings.
142. 17:55And this type of knowledge is like factual knowledge, is part of actually belong to your semantic memory.
143. 18:02And this is in contrast with the episodic memory.
144. 18:05Episodic memory is in the case where you haven't get around to distill the reasoning behind the earlier case why it should be on malicious or the benign side.
145. 18:21Instead, you said that I have seen these similar cases before and it was labeled as benign.
146. 18:27And let me leverage those past similar cases and they made a decision accordingly.
147. 18:33And the advantage of the this episodic memory way is it's relatively automatically.
148. 18:38It has less human intervention.
149. 18:40I mean, the human intervention was done in the past already when during when you're doing a job online, um you can reference them, at least your AI agent can reference them automatically.
150. 18:54They can making decisions without waiting for you to distill the reasoning and putting in the semantic knowledge.
151. 19:02So, the idea is that the identify a similar case to the one under examination and then leverage their past decision as a reference.
152. 19:13And then imagine there's a new case which is coming in the middle, doesn't belong to any of past group, and that's the one you still need a human attention.
153. 19:23And this is how also um where episodic memory resolve recurring case automatically.
154. 19:33And this particular useful in the case of cyber security alerts because a lot of noise especially those false positive noise are the one keep recurring.
155. 19:43And that's the one are the low hanging fruit you can automate away.
156. 19:48And with that human review which is a precious attention and bandwidth you have, now you can concentrate on those where episodic memory failed to address.
157. 20:00And this is also how we chose between semantic memory and episodic memory.
158. 20:07The two are actually not contradicting.
159. 20:10In fact, they are complementary.
160. 20:12Basically, you use episodic memory to address those recurring situations automatically.
161. 20:18And then for what remains still reflecting after the reference from the past or no reference to be able to use, then you pass on to human review.
162. 20:28And then human review distill the domain knowledge and put it in the semantic memory to be used next time.
163. 20:37So here we talk about a solution actually brings three benefits.
164. 20:43First, we identify these data point near the decision boundary and use the domain knowledge in semantic memory and this past similar case in the episodic memory, we're able to improve the the consistency
165. 21:02significantly.
166. 21:04So, now you have a more trustworthy AI agent for your user to use.
167. 21:11And two other byproduct of this solution is you have a very efficient quality control.
168. 21:19You don't need to check every agent output, but instead a subset of them, which your algorithm uh active learning think is potentially problematic, and you check them, either provide additional
169. 21:36information or clarify the label, and your model will learn from there.
170. 21:44So, most importantly, your labeling effort is very small, but high return.
171. 21:56Last but not least, now you are gathering your customer feedback along this way.
172. 22:03You not only have a trustworthy consistent model and high efficacy one, you also have one adapt to your customer environment.
173. 22:11I'm pretty sure your customer will love your product if you your AI agent listen to them and adapt to them.
174. 22:21So, here are some experiment results from uh real data points.
175. 22:26So, here we collect uh 93 alerts, cybersecurity alerts, and then we run them three times.
176. 22:35If without the solution we are proposing here, a quarter of them will flip-flop the verdict.
177. 22:43In contrast, after applying our solution or using episodic memory, about 15% of them become consistent.
178. 22:55However, there's still 10% of them remaining inconsistent.
179. 22:59They still flip.
180. 23:02Sometimes because there's no reference, similar cases in past paper used.
181. 23:08Sometimes even there's a reference to be used, they still disagree after the second thought.
182. 23:15But not a problem.
183. 23:16The episodic memory do the automatic lock down all the inconsistency for 15%.
184. 23:23The remaining 10% human review and provide the additional knowledge to disambiguate those tricky cases.
185. 23:34After that, you also adapt to the customer environment that you want to learn.
186. 23:46So, the three takeaways that I hope you have after this talk is number one, inconsistency isn't usually your model problem.
187. 24:00Stop blaming your model, but instead focus your energy on label issue and potentially insufficient information and help your AI agent with this additional clarity.
188. 24:15Second, the model disagreement isn't a bug, but feature.
189. 24:22Treat each disagreement as an opportunity for your model to learn.
190. 24:29Third, fine-tuning isn't your only option.
191. 24:34Augment your agent with semantic and episodic memory.
192. 24:39I hope you enjoy seeing your model keep improving and adapting to your customer environment.
193. 24:50Last but not least, I want to thanks to my colleagues and wonderful friends at Data Dog.
194. 24:58Special thanks to Ditun and Harsh, Anusha, and Stefan, Saeed, and Mati to give me a lot of enjoyable discussions and helped me to clarify some of the um the concept.
195. 25:14And also, thank you for everyone in the Beats Secured Analyst team to make this journey enjoyable.
196. 25:22And thank you for listening.
197. 25:24Hope you will enjoy training your model and having it improving.
198. 25:31Thank you.
199. 25:32Bye.