Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

AI Engineer · 36 min · 451 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Good afternoon everyone.
2. 00:14My name is Encore.
3. 00:15Uh I'm going to be teaching you how to make your agent cook.
4. 00:17Again, not food related.
5. 00:20I don't know.
6. 00:20Uh but yeah, we're going to get into it.
7. 00:22Um so what I mean by that is actually before we get into that, let's go ahead and start with a quick question and a poll with everyone.
8. 00:29Um, I'm sure a lot of you are building agents or have even shipped agents.
9. 00:32I'm just curious how many of you currently are shipping based off vibes.
10. 00:36And when I say vibes, it's not a bad thing.
11. 00:38I mean like, hey, you're checking your answers.
12. 00:40You're looking at it.
13. 00:41You're making sure that it's potentially doing the right thing just based on the input and output.
14. 00:45You're making sure that it's correct or not.
15. 00:47How many of you all are doing that?
16. 00:50Okay, pretty good reasonable amount of you.
17. 00:53And then how many of you guys actually have like set eval set up like you know like oh you have LM as a judge or codebased evals or some sort of eval setup.
18. 01:03Okay, a little less.
19. 01:04Not too much less though.
20. 01:05That's pretty good actually.
21. 01:07And then how many of you that actually have your eval setup trust it enough that you know you're good enough?
22. 01:12You have a full coverage of your evals and you're ready to you're continuously ready to keep going and moving forward.
23. 01:18Wow, that's that's pretty impressive.
24. 01:20That was a solid hand up, which is kind of what I expected.
25. 01:23I mean, it's very hard to really trust your emails because they're hard to build.
26. 01:28You don't really know what you're looking for.
27. 01:30It's not a deterministic use case.
28. 01:31And it's very hard to determine what we need to actually evaluate to make sure our agents are working properly.
29. 01:36And that's kind of one of the biggest issues is a lot of people are going to start off in a very good spot, which is based off vibes.
30. 01:42It gets you going.
31. 01:43It makes sure that your agent's doing what you expect it to.
32. 01:45But realistically, you can't really ship based on this feeling of, "Yeah, it's correct."
33. 01:49Because when you're in production, you're not going to know where things are broken, what's going wrong, unless you have actual eval set up to find out where things are breaking and how you can actually directly fix it.
34. 02:02So before we get into eval, what I want to kind of take is a whole step back and talk about how you can like what your eval would be on top of.
35. 02:13And that would be on top of traces.
36. 02:15And why we need traces is if you've ever used traditional software engineering, you have something that you do call traces.
37. 02:23It's for APM monitoring.
38. 02:24It's making sure your systems are reliable and are working.
39. 02:27But realistically, what you you can actually go ahead and debug those and it's very easy to do because you have a very deterministic flow, right?
40. 02:34You can even see right here, this is very easy to follow and this is more or less what like code looks like.
41. 02:39You have a handle submit method.
42. 02:41You check if it's valid.
43. 02:42If it doesn't, it errors out.
44. 02:44If it is, then it goes down the next list and continues down that process.
45. 02:47So, it's very easy to figure out where things are b are breaking based on a just based on that sort of uh path history.
46. 02:54But in an agent, unfortunately, with your code, really all you have is that one line of code, which is just going to be agent.run and then the query.
47. 03:02And ultimately, what it's going to do is kind of up to it.
48. 03:05Will it call a tool?
49. 03:06Will it go ahead and do some analysis?
50. 03:07Will it do some internal thinking?
51. 03:09What is its next best step to do?
52. 03:12and its actual best course of action.
53. 03:13We don't know until we actually see the traces.
54. 03:16You won't know what your agent's doing.
55. 03:18So that trace kind of helps you define that.
56. 03:22And so as a kind of a decent example as to what a trace would look like would be this.
57. 03:28You have your agent call.
58. 03:30You go ahead and see the LM calls.
59. 03:33You have your documents you're retrieving, your tools that you're retrieving.
60. 03:36You basically have every step along your journey that you would see of what your agent's doing.
61. 03:42And here you can even go ahead and see, hey, we have for example this tool.get order status.
62. 03:49It's the wrong tool occult.
63. 03:50The user asked us, hey, I was supposed to get a refund here.
64. 03:54Where's my refund?
65. 03:55If we respond back, hey, your refund was issued and they know that that was their status based on their order status.
66. 04:01That's not what they were looking for.
67. 04:02They were looking to understand why hasn't their refund arrived?
68. 04:05So we need to potentially call different tools, do different actions, and there's so many different varieties even just knowing that there's an issue here.
69. 04:12But we'll dive into that in a little bit.
70. 04:15So when you actually go ahead and understand, okay, we have a trace.
71. 04:20Now I want to go ahead and debug what's going wrong with my agent, we're going to start with where first you're going to identify those issues, right?
72. 04:27Those production failures like we just did in that last slide.
73. 04:29We found out that tool was wrong.
74. 04:31But what's your next step?
75. 04:33You want to go ahead and find out, okay, not only where did it go wrong, can I get a nice sample size of that sort of information that I can then use for experimentation
76. 04:42and iteration to improve the agent when I actually want to go back and test against those cases.
77. 04:47Then what actually was the issue, right?
78. 04:50Sure, the tool call was wrong, but should the LM prompt change?
79. 04:53Should we add in guardrails?
80. 04:55Should we change up the tool?
81. 04:57Was the tool just doing the wrong things?
82. 04:58Right?
83. 04:59There's so many different areas that we need to test and go against.
84. 05:02And so we can go and test against those build out evaluations to continuously monitor those use cases.
85. 05:08So for example, if you find out, hey, we need to make sure that we set up a guardrail here.
86. 05:14How are we going to do that?
87. 05:16We want to and then we want to make sure that guardrail is doing its job.
88. 05:18So we set up a evaluator on top of that guardrail making sure is that guardrail doing its job by prohibiting certain information from leaking.
89. 05:27Finally, you actually make those changes.
90. 05:30run that experiment that I was talking about in the beginning and finally ship it to production and do that whole loop over and over again continuously.
91. 05:38Now, this is your standard go-to process for actually improving your agent.
92. 05:43It's pretty long, pretty manual, and pretty tedious.
93. 05:47It does it does work, but we want to kind of iterate and improve that.
94. 05:51How can we do that?
95. 05:52We want to make sure that we're not spending hours, days finding the issue, finding the trace, finding all these different issue points, coming up with the potential solution, going through that full eight-step process manually.
96. 06:04We want to pull all of that together and bring it into minutes.
97. 06:08The best way to do that would be right here.
98. 06:11We're going to pull in all the data from all our different sources into one specific area.
99. 06:16We're going to start with, you know, pulling our ap our arise traces so you can get all the information of what your agent did.
100. 06:24You're going to pull in your evaluations to see where things actually went wrong.
101. 06:29You're going to bring in your annotations from any humans that could theoretically tell you a little bit more information as to, you know, where it could go right, how they can improve it.
102. 06:39You bring in your actual APM traces which could be hey is the system up or down.
103. 06:46you bring in your codebase so you can understand all right based on this specific area where it failed the tool call what changes can I make in my codebase
104. 06:54so that we're c we're we're going down the right pathways we're calling the right tools we're fixing our tools that are broken and so you'll bring in all of these into one final space which is which is our coding agent whether you use cloud code cursor codeex it doesn't matter
105. 07:09all of those you can connect all that data into and it can automatically do all of that work for for you.
106. 07:17So, I know this may sound really complicated and I promise this also probably doesn't help to make it look simpler, but it is very simple to get it set up.
107. 07:25It's going to be just a couple steps of just, you know, setting up your CLI and skills and you're ready to go.
108. 07:32You get access to a bunch of information, a bunch of different tools that your agent can then do, which could be for example here, which are like, hey, you can get your code instrumented.
109. 07:42What that means is you can get all the traces from your agent into whatever platform you choose like Arise.
110. 07:48You could go ahead and pull down those traces, build out data sets, run experiments, run evaluations.
111. 07:55You have the full gamut of the experience all from within your coding agent.
112. 08:01So, what I'm going to do here is we're going to go ahead and look at a specific issue.
113. 08:06And I want a little bit of kind of uh voices from this side of the room uh where we can get some idea of like hey what do you guys think could be the actual issue.
114. 08:17So I'll give you a little bit of background.
115. 08:19This agent right here is a simple financial agent.
116. 08:22I've asked it should I invest in this company?
117. 08:25I think I chose Tata Motors.
118. 08:27Um anyways and ultimately it gave me a response which said yes you should.
119. 08:33I don't think it's directly in the screenshot, but I promise you it told me that.
120. 08:37Um, and ultimately that's not really what we want to do because I'm not registered to give any financial advice, nor is this agent, and I don't want to get in legal trouble.
121. 08:47If you guys can help me figure out maybe, hey, what do you guys think could be the issue in this agent?
122. 08:52So, what she said was, "Hey, maybe the agent needs to ask a lot more questions back to get a little bit more information before I can give a solid, concrete answer."
123. 09:00That's a very good guess.
124. 09:01Anyone else have any ideas?
125. 09:03Okay.
126. 09:04And how do you suggest we potentially improve that?
127. 09:06By the way, he said it should ask, again, pretty similar.
128. 09:08It should ask more questions before making that recommendation, but how do you suggest maybe we can go ahead and do that?
129. 09:14More context, more asking questions, more details.
130. 09:17Okay, so maybe some prompt changes to add in some context to get more context.
131. 09:22One more.
132. 09:24Sorry.
133. 09:25Where's validation?
134. 09:28Yeah, that's true.
135. 09:29We don't have any sort of ground truth to compare against.
136. 09:31We don't know if any of this is correct.
137. 09:33So there's no validation that any of the answers are even correct.
138. 09:36You're right.
139. 09:37There could be so many different issues.
140. 09:38And truthfully, oh yeah, go ahead.
141. 09:40Question.
142. 09:41Does the agent know the user's financial background?
143. 09:46It's a great question.
144. 09:46In this case, no, it does not.
145. 09:48And yeah, it should definitely get a lot more information into that.
146. 09:51Uh he asked, by the way, should the user does the does the agent know any information about the user's financial background?
147. 09:58Um and yeah, it definitely does not.
148. 10:00And there's a lot of areas that this agent could be fixed.
149. 10:02And there's genuinely genuinely not one correct answer, right?
150. 10:07But and I genuinely don't even know what the right answer is to this because this agent is still messed up right now.
151. 10:12And we're going to go ahead and try to fix it today in this demo.
152. 10:15Um, and so what we're going to do is we're going to go ahead and set up the CLI and skills, get that set up so we can actually get our coding agent
153. 10:24to go ahead and understand what's going wrong with our to go ahead and pull down the traces, understand what's going wrong with our agent, look at the evaluations we already have run,
154. 10:33look at the annotations that exist, maybe even do some of its own analysis, and potentially come back with some future suggestions just like many of you have done today
155. 10:43for how to actually improve this agent.
156. 10:45So that hey, maybe we could have it ask more questions.
157. 10:48Maybe we could add a guard rail so it only responds to certain questions and doesn't give financial advice where it's not warranted.
158. 10:56So there's lots of different ways that it can come to do those fixes.
159. 11:01To get it started, what I'm going to do is I'm going to go ahead and pull open let's see, do I have it here?
160. 11:07Yes.
161. 11:08Let's go ahead and pull open a new session.
162. 11:11Um, okay, cool.
163. 11:13I'm in the right space.
164. 11:15All right.
165. 11:16So, generally what you'd want to do whenever you want to set up the CLI and skills, super easy.
166. 11:21It would just be a couple steps to get that set up.
167. 11:23Um I apologize.
168. 11:25Oh, here we go.
169. 11:26I know what I'm doing, guys.
170. 11:28Uh here we go.
171. 11:30So, we're going to start by setting up the CLI.
172. 11:32You would just go ahead and copy this one line of code to get that set up.
173. 11:35So, I'd go into back into my terminal and install that.
174. 11:38Of course, I already had that installed right before this, but just goes just very simple and quick to set up.
175. 11:44Then what you do is you would create your actual profile, which would help you connect it to wherever your data stored.
176. 11:50In this case, it would be arise.
177. 11:51And you create that profile so it can go ahead and pull in those traces and visual and get that sort of information from those evals and annotations.
178. 12:01Once you have that profile created, you'd go ahead and set up your skills.
179. 12:05You could do this in a multitude of ways depending on your coding agent.
180. 12:08For today, we're using Cloud Code, so I use the marketplace to set it up, but it's pretty easy to set up however you want.
181. 12:17From here, let's go ahead and we're just going to ask it.
182. 12:21And again, uh just as a warning, I am using dangerously skip permissions.
183. 12:25Please don't do this.
184. 12:26This is not actually good practice.
185. 12:28This is just for a demo, so I don't have to keep pressing yes.
186. 12:31Um, but we're going to go ahead and talk to Claude and have it go ahead and take a look at these traces.
187. 12:37So, I'm going to go ahead and see if I can find the prompt I asked earlier, which was just use this skill and pull down the traces from that specific agent.
188. 12:48Find out what issues exist that we can determine from any, like I said earlier, right?
189. 12:53What any issues that exist from any of the evals, any of the annotations that we've done.
190. 12:57do your own analysis and tell me what eval maybe I can build out in the future to make sure that I don't run into any of these issues again.
191. 13:05And so this is going to take a couple minutes.
192. 13:07Um, in the meantime, I'm going to go ahead and first go and show you kind of how those traces look and how you can actually start triaging it if you want to do it manually a little bit.
193. 13:19But while this is running in the background, let's go back into the actual uh run into the actual agent that has those sort of queries that we've asked.
194. 13:30So, we'll go ahead and ask it a pretty similar question.
195. 13:32Um, you know, tell me if I should invest into, let's say, Nvidia this time.
196. 13:38It's a little bit more of a well-known stock.
197. 13:41And we're going to get that and it's going to try to tell us some sort of information.
198. 13:45And as that data comes in, looks like I've run this earlier.
199. 13:48So, and you can see it's coming in.
200. 13:50Um, but from an earlier run, we can kind of see what it's done already where you can see that query we asked, you can see the output that it'll finally generate, which we'll see how similar it is in a minute from this previous run.
201. 14:02Um, and you can see all the steps that actually happened in your agent, right?
202. 14:05So, you can understand, hey, we had a supervisor agent, a financial data agent, you can see the LLM calls, you can see all your different tool calls.
203. 14:14you get a lot of information right away that you can start using just like we done in the initial slide to see where those issues could lie.
204. 14:22Now, let's see.
205. 14:23Did Claude finish up?
206. 14:25Looks like it's still running.
207. 14:26It's doing its thing.
208. 14:27We'll let it cook.
209. 14:27You know, it's kind of the point of this.
210. 14:30Um, while it's cooking, anyone have any questions as to like, you know, how you can set up any sort form evals, any sort of traces, anything along those lines?
211. 14:41Anyone wanted to learn a little bit more about Yeah.
212. 14:57Great question.
213. 14:58Yeah.
214. 14:58So, you would go ahead.
215. 14:59So, what it does, it actually goes ahead and downloads a lot of those traces um into the into your local system.
216. 15:06So, what it'll do is actually let me go ahead and show you what it how it actually goes ahead and accesses the system itself.
217. 15:13So when you give it access to these skills, what you're actually giving it access to is the CLI underneath.
218. 15:18And what I mean by that is you're going to have all these different CLIs that it can integrate with.
219. 15:24So you can go ahead and set up and find the data sets.
220. 15:26You can pull down data sets.
221. 15:28You can have it interact with the essentially interact with the UI where it can set up evaluations on the platform on your behalf.
222. 15:35So you can monitor continuously monitor when your agents are failing, build out data sets, run experiments.
223. 15:41It can basically do anything that you that you would be able to do in the platform.
224. 15:46Sometimes it'll download, sometimes it'll make API requests, but that's generally the uh idea here.
225. 15:52Great question.
226. 15:55Any other questions in regards to CLI tracing?
227. 16:02Okay, let's go ahead and see what Claude's done.
228. 16:04Oh, still crunching away.
229. 16:06Um, so as we can see here, let's let's walk through what Claude's done so far.
230. 16:10So, we can see that we've gone ahead and gotten the spans.
231. 16:15It's digging into the evaluations to figure out where those issues are.
232. 16:19Um, it looks like it's not quite there's definitely found one very big issue with some numeric groundedness.
233. 16:25I know someone mentioned that earlier as a potential issue or something related to that.
234. 16:30Um, we're also seeing some tool calls directly fail.
235. 16:34Um, and let's see.
236. 16:37We also see, yeah, so we're getting a lot of different errors.
237. 16:41Oh, looks like it just finished up.
238. 16:43And so you can see here we get a full list as to like, hey, what's going wrong with our agent and what we can do to kind of fix it.
239. 16:50So while it's finishing up that, you can see here one of the biggest alarming things right out of the gate is about 44% of our agent interactions are resulting in a bad summary.
240. 17:03That's really bad.
241. 17:04I did not know that.
242. 17:05So, we definitely need to fix that one.
243. 17:07Another big issue is Oh, did not mean to scroll there.
244. 17:12Another big issue is, hey, we're also just getting a lot of fabricated answers, which is just as bad as this the summarization being wrong.
245. 17:22So, it sounds like it may be related, it may not be.
246. 17:25We can kind of debug that.
247. 17:27We're also seeing a relatively high rate of us just redundantly calling tools.
248. 17:33It's a waste of our time, our resources, and our money.
249. 17:35So clearly, we're doing a lot of bad things that we need to go ahead and fix.
250. 17:39And that's just from the eval from Claude's own analysis.
251. 17:45It's able to even go ahead and figure out, hey, there's other issues that are coming along.
252. 17:49We're getting silent tool failures.
253. 17:51It's telling us it succeeded, but ultimately it's not giving us any actual responses.
254. 17:57We're getting the agent hallucinates when tool fails instead of disclosing it.
255. 18:02So again related but hey that means maybe we need to not just do uh you know some eval built on top of this to understand when the tool is failing but we also need to make some prompt based changes to make sure that the the agent doesn't hallucinate here
256. 18:16and you have a multitude of like different things that it'll go ahead and tell you as well as the evals it can build out.
257. 18:22So what we could do is we can actually go ahead and tell it, hey, let's go ahead and build number one, which was the tool success silent failure eval,
258. 18:32and fix numeric groundedness so that it'll make sure that whenever it does respond, it's always grounded to some sort of actual reference point and it's not making up these numbers.
259. 18:43But we're also going to head and make sure you test this against a data set uh against the spans that caused the failure initially using and then we're going to go ahead and use another Arise skill
260. 19:00which would be here the Arise uh data set skill, the Arise Oh, there we go.
261. 19:07Arise um evaluator skill as well as the Arise experiment skill.
262. 19:15So here it's going to be able to let me just go ahead and run let this run and build out evals.
263. 19:22All right.
264. 19:23So again I'm being pretty vague here but essentially I'm just telling it go ahead and make the changes that you recommended as well as go ahead and run these experiments for us.
265. 19:31So now we can actually compare against where it failed to make sure it passes against that right and so in this scenario we can see okay it's going to go through and take its time.
266. 19:42Um I'm going to go ahead and show you a previous run because again this will take some time and while we want to let it cook I don't want to have you guys staring at a screen where Claude's just discombobulating
267. 19:51or whatever it does in its free time.
268. 19:54Um, so let's go ahead and look at where I've actually gone ahead and asked Alex or sorry Cloud Code to go ahead and answer some of our previous issue that I ran into.
269. 20:05And one of those was related to hey, I've had some times where I'm trying to get some answers about uh stocks that aren't necessarily from the United States.
270. 20:17And so we're having errors where we're talking about like South Korea or looks like they're all South Korea based, but let's assume there's other answers here.
271. 20:26Maybe Japan or China and whatever, right?
272. 20:28And you want to see and you want to be able to understand, hey, I want to be able to compare stocks across different markets.
273. 20:35And clearly people are asking that, but we don't support it.
274. 20:39And so what Claude did, it went ahead found those areas where that issue existed.
275. 20:44Like I said, it looks like it was all South Korea here, but it found those areas.
276. 20:49It built out a data set for me, which I can then go ahead and test against to make sure it works.
277. 20:54And then it went ahead and ran experiments.
278. 20:56And it made a couple changes.
279. 20:57So, it started with the initial one, which was the baseline, which was buggy, which as we expected would result in 0% uh passing rate, which is what happened when we initially got the actual productionbased
280. 21:09traces.
281. 21:11It also made a quick fix where it normalized it by adding in that uh country name.
282. 21:18It didn't work.
283. 21:19Why?
284. 21:20Because the tool I was calling was also outdated and I needed to update to the stable version.
285. 21:25So then it made a third iteration.
286. 21:26And you can see that on this chart here as well where we were able to see, hey, the first iteration didn't work.
287. 21:33It went back, made a fix, and finally got us to the final iteration where we're getting about a 90% success rate.
288. 21:39Clearly still not 100%.
289. 21:41We still need to make some changes, but hey, we're doing a lot better than we were earlier.
290. 21:47So, this is like a great way to get you going so you can actually start from, hey, I know there's an issue to here's a fix.
291. 21:56Now, I know we have quite a bit more time.
292. 21:58I want to check in.
293. 22:00How is everyone doing?
294. 22:01How do you guys feel?
295. 22:01Do you guys have any questions about how you can go ahead and get this started?
296. 22:05Yeah.
297. 22:23Great question.
298. 22:24So his question was how where do the experiments actually run?
299. 22:27Where do the evaluations run?
300. 22:29Where does everything run?
301. 22:31Um so you do have a little bit of customization there.
302. 22:34In this scenario, uh what I did is I had the experiment run on my local machine, which is I think what will be the case most of the time.
303. 22:43And then what you what I did afterwards was I went or you can host it on a cloud or wherever you'd like to experiment on.
304. 22:49And then you can go ahead and actually run the evaluations on Arise itself.
305. 22:53So in this case, I built out LM as a judge, which would then go ahead and let's see if I can see the evaluator here itself.
306. 23:00So you can see we built out two different evaluators, a codebased and an LLM based.
307. 23:04I actually had Claude go ahead and do this.
308. 23:07It built it out for me in the platform.
309. 23:09And this was the final LLM as a judge because it actually kind of failed in getting the codebased data to set up.
310. 23:15So that's why we have two.
311. 23:16And so you can see the LM as a judge.
312. 23:18It went ahead and built it out for us with this prompt and got it going so that we can actually compare.
313. 23:24Hey, did the agent work properly?
314. 23:28Any other questions?
315. 23:28That was a really good question by the way.
316. 23:31Yeah.
317. 23:40Very very good question.
318. 23:42Um so eval can be set up however you like.
319. 23:46We have of course people can people can definitely use whatever sort of models they want.
320. 23:52No one wants to be limited to just claude or just open AAI especially in today's modern age where you can see AI models literally ship every other day and then you want to be like okay I want to use that one now right and also some people have a preference for slightly cheaper models some people have a
321. 24:07preference for making sure their evals are topofthe line because ultimately they know their agent's only going to be as good as their evals so they use the highest end models for their evals even if they're not necessarily using those in production so it kind of varies you have full customizable
322. 24:21control over cut.
323. 24:22And so what you can see here for example, you have a full list of whatever sort of model provider you want to use and you can connect those even if you want to use any sort of local
324. 24:31um open source models as well.
325. 24:35Yeah, good question.
326. 24:38So her question was if we have if I spin up a smaller agent to go ahead and make some of these bug fixes, how does this compare to that?
327. 24:48Realistically, you don't have to use cloud codeex.
328. 24:51You can use whatever sort of agent you want.
329. 24:53The benefit of what we've provided is a skill that you're able to go ahead and pull down the information from your traces to make those fixes.
330. 25:01So, you're not making those fixes just based on, hey, I know this is wrong and the output's incorrect.
331. 25:07You're making those fixes based on, hey, I know this is wrong.
332. 25:10Here's where it failed.
333. 25:11Here's why it failed.
334. 25:12Here's how we can go ahead and fix it.
335. 25:14because you have not just the information of your codebase, you have the information of any of your other APM tickets, whatever or APM logs, any of your other tickets that exist in your Jira systems, etc. as well as your actual arise traces, which would be like here's the pathway the agent took, here's where it messed up, your
336. 25:33eval, your annotations, etc. Any other questions?
337. 25:38Yes.
338. 25:38Are these features available in Phoenix?
339. 25:41Great question.
340. 25:42Yes.
341. 25:42Uh we do have so he asked is are these features available in Phoenix.
342. 25:47Phoenix is Aris's open- source version.
343. 25:50We do have I believe skills and MCP as well in Phoenix which you can go ahead and hook up to your agent and it can go ahead and pull down the same sort of information.
344. 26:00Of course there are going to be some nice to haves in the enterprise edition where you have access to these online evals in a certain way that are much easier to format and set up.
345. 26:08You have a little bit more dashboarding and a few other features.
346. 26:11But yes, the the skills in CLI you can connect to your coding agent.
347. 26:16Yes.
348. 26:20Agent said that summary.
349. 26:22So how would you classify something as like a bad agent know that's a good summary or a bad?
350. 26:30Very good question.
351. 26:31So her question was in what I showed earlier there was a eval or there was a response from Claude that said hey there was a bad summary.
352. 26:39How do we know what is a bad summary or a good summary?
353. 26:43In that case, what I actually had set up, it was actually based on one of the evals that I had previously set up.
354. 26:48And so, we can actually go ahead and look at that evaluator here.
355. 26:52We can go ahead and look at let's see some I actually don't remember which one, but I'm going to go ahead and pull open this one and hope this is the right one.
356. 27:03Um, so as you can see here, this is basically the summary eval that I'd created.
357. 27:10It's basically taking a look at the information we've gathered throughout our entire agent, all the sort of context we've given to that summary agent and making sure that it's referencing the right points, it's answering the question accordingly,
358. 27:23etc. Um, and so that's how we've gotten to determine whether or not it was a good summary or not.
359. 27:33Uh, what if we set up a skill in the first place to see if it can do a good summary?
360. 27:38give it given us a good summary all the time just like we set up a skill instead instead of running.
361. 27:48Yeah, great question.
362. 27:49So, she's asking why can't we just set up skills that you know ensure that it gives a good summary and yes absolutely that is what you want to do.
363. 27:58You want to make sure that you update your prompts, update your skills, update your coding workflows so that your agents will give the best summaries.
364. 28:05The emails are only there to help you determine where there are issues, monitor those issues, make sure they don't repeat, right?
365. 28:11And so you set up that skill, great, it works for the next two months, two months later because there's a model regression because people start asking questions in a different way.
366. 28:22You end up getting different results and you don't want your e you don't want your performance to degrade because of that.
367. 28:28So you want to be able to monitor and observe that.
368. 28:30That's the point of the evals.
369. 28:31Did that help?
370. 28:33Beautiful.
371. 28:34Um I don't know for how much time we have left but go ahead
372. 28:39later agents are assessed and soc.
373. 28:56So your question was if I've run into issues with linting with the agents
374. 29:00agents seems like more
375. 29:11good question.
376. 29:11Yeah.
377. 29:12So he's basically saying hey um have you seen the issue where hey you've given access claude to all these skills and all these sort of traces and all this sort of information
378. 29:23but ultimately most of the time I've seen with cloud code or any of these other sort of coding agents they end up focusing on linting rather than any other sort of issue am I correct
379. 29:33yeah that's a very good understanding and that's kind of why we built out these skills which kind of focus on ensuring that they hyperfixate on the actual issues that have been given to them, right?
380. 29:43So they'll get all these trace information so they'll understand, hey, it's not going to be a linting based issue because you can see that it's calling the wrong tool
381. 29:51and that's not going to happen because of linting and it does have some reasoning capability to understand that.
382. 29:56Um, and so genuinely I haven't seen that so much when using the skills.
383. 30:03Awesome.
384. 30:04So as it Oh yeah.
385. 30:21Does does the system make it easy to understand?
386. 30:23Sorry.
387. 30:27Great question.
388. 30:28Yeah.
389. 30:28So, how so his question was, does it make it easy to understand how much it costs to run those evals?
390. 30:34Absolutely.
391. 30:35So, one of the best parts of running evals is in the platform is you're going to be able to get full trace history of what your emails did, which also comes with the added benefit
392. 30:45of, you can see right here, a nice little cost.
393. 30:48And so, you'll be able to understand how much your emails are actually costing you because yes, of course, anything anytime you're running another LLM, it's going to be additional cost to you.
394. 30:58So, you want to make sure it's not.
395. 30:59And of course, what's really nice about building out evals, whether it's in the UI or via the coding agent, you can have those set up basic requirements like, hey, we want to sample specific amounts so we don't run it across 100% of our traces
396. 31:14and 100% of our calls.
397. 31:16Um, you can make it focus on specific areas.
398. 31:19And of course, you can always dictate that in the coding agent or on the platform itself.
399. 31:25Um, so as a kind of refresher as to everything we kind of covered today, again, it that's the wrong one.
400. 31:32Oh, I think I have it over here.
401. 31:35Um, so essentially what I really want to give you as a takeaway, these AI agents, they're great.
402. 31:46They will do basically everything you ask them to, but again, they're not magic.
403. 31:50Like he said, they can definitely focus on linting.
404. 31:53They can definitely get caught up in small little areas, but the best part of them is they will automate your full workflow from beginning to end.
405. 32:02So whether it's that starting from observing where the issues lie, finding those issues, being able to actually pull it down, understand where the issues lie, create those data sets,
406. 32:15build out exact fixes, test them to make sure that they're working.
407. 32:21And finally, you can then go ahead and be the manual human to make sure that hey, it didn't just go back and fix the linting changes, right?
408. 32:29And we can make sure that it actually made changes that actually help improve the agent and not go for degradation.
409. 32:39That's kind of the general gist of what I wanted to show today.
410. 32:41Um, any other questions on this before, you know, I can maybe show one last thing of seeing.
411. 32:46Oh, still finishing up what I was trying to do earlier.
412. 32:48I was hoping it might be done after this.
413. 32:50took about 14 minutes.
414. 32:52But um while that's finishing up, any other questions on how we can use our agents to automate our workflows and improving your agents?
415. 33:04Yeah.
416. 33:19Great question.
417. 33:20So his question was, are we primarily relying on Claude or any of these coding agents to actually go ahead and give us the evaluation if things are working well or not?
418. 33:30Um, yes, you can definitely rely on Claude to build out everything.
419. 33:34You can also manually do it.
420. 33:36You can also ask Claude to do the specific evaluations you want.
421. 33:40So in this case, when you saw that nice clean table, that was actually defined by the evaluations me and my team had set up prior to this.
422. 33:47Um, we'd gone ahead and looked at, hey, these are specific areas that we know are potential limitations in most agents that work in this workflow and we set up those as generic ones, which is why we got a bad summary.
423. 33:58We had some, hey, is the agent healthy?
424. 34:01Is it giving the right are we getting tool calls correct?
425. 34:03Right?
426. 34:04We know those are general areas, but what Claude was able to do was look at our actual traces, understand based on those traces, what are the actual issues we're falling into today.
427. 34:15Right?
428. 34:15we have our live agent in production.
429. 34:17I'm not going to be able to go through all of those manually and find those, but Claude can or Claude cursor codeex, whatever, right?
430. 34:24And that's where like one of the biggest superpowers that you're able to give these coding agents to be able to do that for you.
431. 34:32I saw another question here.
432. 34:34Good question.
433. 34:35So his question was is there any sort of benchmark data set that we can see or any sort of like visualization that can tell us hey this is a higher quality LLM to use for evaluations
434. 34:45more expensive less expensive um realistically no
435. 34:49[laughter]
436. 34:50that would be sort of something you'd have to do for your own analysis you'd have to figure it out because ultimately one of the things that I like to say is if you're going to use out of the box eval to judge your agent
437. 35:00you're going to get out of the box results and that's not what you want when you're building customtailored AI agents which are built to be personal for you.
438. 35:08Um, and the same goes for benchmarks, right?
439. 35:11Benchmarks are great.
440. 35:12They'll show you, hey, this this is a good agent.
441. 35:15This is a good sorry, this is a good LLM.
442. 35:17But realistically, what you want is, is it a good LLM for me?
443. 35:21And that's going to be tailored to your use case.
444. 35:23So, if you're building for finance, maybe you'll use a specific one.
445. 35:26If you're building for, you know, law in finance, it might be even this slightly different model, right?
446. 35:32So, it really depends on what you're building it for.
447. 35:37Cool.
448. 35:37I think that's more or less it more or less it from me.
449. 35:40Um, if anyone has any other questions, I'm happy to answer them.
450. 35:42But yeah, thank you guys.
451. 35:46[applause]