The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

AI Engineer · 51 min · 758 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:13All right.
2. 00:14Uh since it's already like 11:12, I want to start talking about the workshop.
3. 00:20Um so, I'm going to do this as a guided workshop, so I'll go through all the examples up here on screen, um but you can also uh try and follow along.
4. 00:30So, the repo is there, the environment variables are there.
5. 00:34Uh so, the environment variables give you access to a bunch of Foundry models that I've deployed on uh Microsoft Foundry project.
6. 00:42Um and they have a certain amount of token capacity.
7. 00:46It is possible that we are going to go over the token capacity um in you know, in this session.
8. 00:52Um I only got found out very recently that this workshop was accepted, so I tried to get as much resources as I could in the small period of time.
9. 01:00Um so, in that EMB file, there is actually uh multiple different possible environment variables that you can comment in or comment out.
10. 01:09If we do start getting 429s, um you know, uh rate limit exceeded errors.
11. 01:14Uh so, yeah.
12. 01:15So, I'm hoping that there's enough capacity for all of you to be able to try out those models, and uh we will see.
13. 01:22Uh I also have my colleague Cedric here.
14. 01:24Uh so, Cedric is going to be wandering around while I'm talking, so he might be able to help uh with with issues as well.
15. 01:32Uh if you In terms of setting up the code, if you already have Python and UV on your machine, then you can do a get clone of the repo and run UV sync, and that should have everything you need to run the examples
16. 01:44uh once you have that .env.
17. 01:46Uh otherwise, you can make a GitHub code space.
18. 01:49That's uh basically a VS code in the browser that has everything set up for you.
19. 01:55Uh who here uses UV?
20. 01:57Cedric wants to know.
21. 01:59Okay.
22. 02:01So, it's not number one yet.
23. 02:03But, the other question here, who uses Python?
24. 02:06All right, then more people.
25. 02:07Okay.
26. 02:08Pretty much everyone uses Python here.
27. 02:11Yeah.
28. 02:11Oh, did you count the overlap?
29. 02:12All right.
30. 02:13Okay.
31. 02:14All right, so if you didn't take a picture of that screen yet, just take a picture of it now.
32. 02:18Otherwise, Cedric also has a picture of it on his phone, so he can go around if you need those links.
33. 02:23Uh, so the goal of this workshop is to be able to get a feel for the differences between the different, you know, uh, top-of-the-line models and see how they differ and, you know, what you can improve about them.
34. 02:35So, I've set up a bunch of different examples and also evaluations and optimizations that we can do to improve our experience across models.
35. 02:42Uh, and I think this is a really important thing for all of us to just be to be doing as AI engineers because, you know, as you've seen, like, there's so many new models that come out all the time.
36. 02:51And, you know, the US government at any point could cut off access to a model.
37. 02:55So, we need to like, we need to have options, right?
38. 02:58So, we need to be able to, you know, point our AI applications at a different model and and have it still work or know how to improve it.
39. 03:06Uh, so that's the goal here is to get more experience with more models.
40. 03:09So, uh, these are the models I've deployed inside Microsoft Foundry.
41. 03:13Uh, so I've got uh, GPT-5 5, that's a standard one I use all the time.
42. 03:18But, then I've got Kimmy K2.6, that's a uh, more recent one that's quite good.
43. 03:22I've got Deep Seek V4 Flash.
44. 03:25And then I have Mistral Large 3. So, these are the four models that we'll be comparing in this workshop.
45. 03:32Um, so as I was saying, you can in order to set up the workshop, here's the repo that that you're all into.
46. 03:40So, if you don't have Python on your machine, you can just click on the code button and there'll be a big button here that says create code space.
47. 03:47So, you click on that, it'll make a code space.
48. 03:50So, then here's the uh, the code space and then this has um, I've already put the dot env, but then you go and you grab that other um URL that I put up there to get the env.
49. 04:02That goes to this file here.
50. 04:05You download it.
51. 04:06You open that.
52. 04:08And then you create a dot env with the contents, okay?
53. 04:13So, this is pointing at a that foundry project that has those four models deployed.
54. 04:19As you've seen, that does have you know, it has token limits.
55. 04:24So, if we start getting rate limits, you can start you can try these other models that also have the four project deploys.
56. 04:30And then I have a separate one specifically for Anthropic just because it has different terms for that project.
57. 04:36All right.
58. 04:37So, so yeah, and then once you've got that set up, then you v sync will sync all of the packages that you need for Python.
59. 04:47All right, so that's the setup.
60. 04:50Now, I'm going to actually go through it and do it do it up here so that you can you know, hopefully follow along if you want, or you can just watch
61. 04:59watch what I do.
62. 05:00So, as I was saying, we do have the four models, and you can compare models just doing benchmarks.
63. 05:05You're like, oh, you know, this one has good quality benchmarks and good safety benchmarks.
64. 05:11You can also look at you know, the cost and thoroughput and stuff like that.
65. 05:14And you can get this approximate notion for how good these models are, but you don't really know how good a model is until you use it for your particular scenario because there's so many different ways of using models.
66. 05:26And a particular benchmark isn't really going to tell you how it's going to work for your scenario.
67. 05:31So, that's what we have in this repo here is a ton of different scenarios, and we can try out these models for each of them.
68. 05:38So, we're going to start really really basic, all right?
69. 05:40So, just a single LLM call that asks the LLM to do something.
70. 05:45So, what you see for each of these files is that they all start off with the model variable and they default to GPT-3.5.
71. 05:51That's kind of our baseline.
72. 05:52It's obviously it's a very good model.
73. 05:55But then if we want to try a different model, we can just uncomment that line and that will be the model that gets used.
74. 06:00So, this file here is connecting using the OpenAI Python SDK to the Foundry URL and passing in the key from the environment.
75. 06:11And the prompt we're doing here is how many times does the letter E appear in the following sentence?
76. 06:17The elderly gentleman eagerly entered the elevator.
77. 06:20Count carefully.
78. 06:21All right.
79. 06:21So, then we send that to the OpenAI responses API and say, "Here, use this model with this prompt and then let's see the raw the output, right?"
80. 06:30So, we're doing the classic thing of asking LLMs to count.
81. 06:34And so, we'll see, you know, how well they're able to do.
82. 06:36So, I'm going to say UB run examples with the name of this this file name here.
83. 06:42And this is all in, you know, in the read me.
84. 06:45So, we're just going through one by one.
85. 06:48So, you can just copy-paste from there.
86. 06:50All right.
87. 06:50So, GPT-3.5 is actually very good at math, surprisingly good at math.
88. 06:54My colleagues have been running a lot of finance benchmarks on it lately and they're like, "Oh my god, like it does so much math in its head."
89. 07:00So, it does actually get this correct that there are 13 E's in this sentence, right?
90. 07:04So, now let's try Now we just change it, right?
91. 07:07So, okay, let's say we want to try Kimmy.
92. 07:08So, we go and we change to Kimmy and now Kimmy will be the model that's used.
93. 07:14And we see, "Okay, can Kimmy count as well as GPT-3.5?"
94. 07:18And it does.
95. 07:19It gets 13.
96. 07:20All right.
97. 07:20Now you might be thinking, "Oh, is this just too easy?
98. 07:23Let's try DeepSeek V4 Flash.
99. 07:26And this time it gets six.
100. 07:29Okay.
101. 07:30So, it missed out on seven of the E's, right?
102. 07:33So, we can immediately see that there's definitely difference between you know, DeepSeek V4 and those first two models.
103. 07:40And then finally, we'll try Mistral.
104. 07:42And this one gets eight.
105. 07:43Uh so, what we see there's a difference here.
106. 07:45All right.
107. 07:46So, when you see this and you're like, "Well, I want to be able to use all these models."
108. 07:48Then you start doing some prompt engineering.
109. 07:50You're like, "Okay, what if I say, uh you know, break the sentence into words, count per word, uh sum, show your reasoning, right?"
110. 08:03Let's try Let's try that, right?
111. 08:06And you know, we can fix our typos, not that it matters.
112. 08:09All right.
113. 08:09So, let's try it with Mistral and see with a little bit of prompt engineering, can we get a better better response here?
114. 08:18All right.
115. 08:19And it got it, right?
116. 08:20So, you know, by breaking it up, doing a little chain of thought here, it was able to come up with the correct answer, right?
117. 08:28So, immediately, we can see that there's obviously differences across models and if we change prompts, you know, we can get uh different outputs.
118. 08:36All right.
119. 08:36So, now let's get more interesting and go to more real-world scenarios.
120. 08:40Uh so, let's look at RAG.
121. 08:42So, RAG is still one of the big uses.
122. 08:44Arguably, really everything is RAG.
123. 08:46Agents are RAG, too.
124. 08:47Um So, we have this So, RAG is when you ask an LLM to answer a question based off some context, right?
125. 08:54And the thing that's important for RAG is that you want it to answer the question and stay grounded in the context.
126. 08:59You don't want it to make up anything that is just from its weights, right?
127. 09:03You're trying to force it stick to the context, right?
128. 09:07And that's what I find really varies across the models is that the smaller models, the less powerful models are actually much more likely to go to their weights and pretend it was part of the context.
129. 09:16Um so, what we have here is um we have a data set about carpenter bees and we ask it a question, "What are the key differences differences between how honey bees and carpenter bees build their nests?"
130. 09:30But this data set only has carpenter bee information.
131. 09:34Uh so, it should only actually be able to answer about carpenter bees.
132. 09:38So a good response in this case would tell me the carpenter bee information and then it say, "Sorry, I can't tell you about honey bees.
133. 09:44I don't know."
134. 09:45Um, so we tell it like, "Hey, your job is to answer based on the sources, cite the sources.
135. 09:50Um, here's your here's your question, here's your sources."
136. 09:52All right, so let's go ahead and try that out and see what we get.
137. 10:02All right, and it's Yeah, so when it's doing the search first, very basic search.
138. 10:05Okay, so it's sending did the search.
139. 10:08All right, and what we see and it says the data set does not provide details on honey bee nest construction.
140. 10:16So this is what we actually want to see.
141. 10:17This is from GB55.
142. 10:19GB55 is very good at knowing what it doesn't know.
143. 10:22So now, let's try switching Um, Kimmy's also good as well.
144. 10:26So I'm not going to show every model every time now just for for sake of time.
145. 10:32Um, so let's try Deep Seek, right?
146. 10:34Cuz we've already seen that, you know, there's a distinction here.
147. 10:37Now Deep Seek, once again, says provided sources do not contain information.
148. 10:40All right, so Deep Seek did a good job here.
149. 10:42And now let's go to Mistral.
150. 10:44Uh, Mistral, sorry.
151. 10:45Cedric, how do we say Mistral?
152. 10:46It's French, right?
153. 10:47French company?
154. 10:48Mistral.
155. 10:48Mistral.
156. 10:49Cedric's going to be my French translator.
157. 10:51Every time I need to say it, you're just going to say it.
158. 10:55Now, Mistral.
159. 10:56Mhm?
160. 10:58Yeah, okay.
161. 10:59Mistral.
162. 11:00Um, so Mis- Mis- With Mistral
163. 11:03[laughter]
164. 11:04Um, we do see that Mistral actually answered uh, from its weights uh, even though that information was not in the sources.
165. 11:13So this is an example where a model went into its weights even though we didn't want it to do, right?
166. 11:18Now we can try to force it.
167. 11:20Once again, we can try to do prompt engineering.
168. 11:23Uh, we can say uh, wait, actually so it would say uh, if the if the answer for a bit of info is not in sources.
169. 11:37Say no source provided.
170. 11:41I think this is what I got it to work.
171. 11:43Uh oh, I will try that and do not make up an answer.
172. 11:46All right, let's try this.
173. 11:48[snorts]
174. 11:52So, we're trying to force it to realize when it's giving us information that's not in the weights.
175. 11:57And sometimes if you force it to say, "Hey, for every bit of information that you're giving back in your answer, you have to prove that you got that from
176. 12:06from the sources, right?"
177. 12:08Um so, here I'm trying to have it associate each of the things in that in that answer with a source and um and maybe that will force it to realize, "Oh, wait a second, that wasn't actually,
178. 12:21you know, in my in my answers here."
179. 12:24All right, it's still going out.
180. 12:26It's going to think it's thinking really hard about it here.
181. 12:30Okay.
182. 12:31And there we go.
183. 12:32Honeybee details are no source provided beyond interactions with carpenter bees.
184. 12:37[laughter and gasps]
185. 12:38Um so, it definitely knew when things were not source provided.
186. 12:42It still gave the information.
187. 12:44So, we got a little closer.
188. 12:45But as you can see, this is definitely uh a difference that I see with rag and rag is what I work with the most.
189. 12:51Uh so, it's it's something you want to evaluate when you're looking at models to use with rag is are the models willing to admit what they don't know, how likely are they to make stuff up?
190. 13:03All right, so now let's move to image input.
191. 13:06A lot of these models are multimodal, so they will accept images as input.
192. 13:10And you know, there's lots of really cool um situations you can do with image input.
193. 13:14So, here we've got some files locally and we're just going to send them to the LLMs and ask them some questions about the files, right?
194. 13:24Um I'll show you the files.
195. 13:27Examples, image input.
196. 13:28Okay.
197. 13:30All right.
198. 13:31So, the files we have here are this picture.
199. 13:34This is an aurochs, not a unicorn.
200. 13:36This picture.
201. 13:37This is of some crocodiles that I saw in Costa Rica that I was very afraid of.
202. 13:43And then also we have this picture which is a list of plants.
203. 13:49Basically like a screenshot of a table with some really small fonts in it.
204. 13:53So, we're asking each of the LLMs to interpret each of the images, right?
205. 13:58So, first we say, "Oh, is this a unicorn?"
206. 14:01Right?
207. 14:01That should be easy to say it's not a unicorn.
208. 14:03Great.
209. 14:03Then we say, "Hey, are these alligators or crocodiles?"
210. 14:05That's a bit tricky.
211. 14:07They are actually crocodiles because of their snout.
212. 14:10I spent a lot of time reading the Wikipedia page about crocodiles and you got to look at their snout and then you know.
213. 14:15Luckily, GPT-5.5 has also been on a Wikipedia binge and so it knows that when it sees that snout, then that is crocodiles.
214. 14:23And then the cheapest plant, it correctly found the cheapest plant in that like fairly small text screenshot of a table.
215. 14:31So, that's pretty impressive that GPT-5.5 could do that.
216. 14:34So, now let's try it with Now, for these models, we can't use DeepSeek cuz DeepSeek only accepts text.
217. 14:41It's not a multimodal model.
218. 14:43So, we're only going to try Kimmy and Mistral.
219. 14:47So, you know, it figured out the aurochs.
220. 14:50Okay, good.
221. 14:50Just making sure it works.
222. 14:52All right.
223. 14:52And then for crocodiles, it does say crocodiles correctly.
224. 14:57And then cheapest plant, it got the cheapest plant.
225. 14:59So, Kimmy is doing a great job as well.
226. 15:02And then finally, let's go to Mistral.
227. 15:06Here.
228. 15:08Let's see.
229. 15:09All right.
230. 15:10So, you know, it knows it knows it's not a unicorn.
231. 15:13It didn't know it was an aurochs.
232. 15:14This one, it wasn't sure.
233. 15:17It thinks they're alligators.
234. 15:19So, it you know, it's having a harder time looking at the it thinks that they're alligators.
235. 15:23And then for the the plants, it pulled out a plant that's $2.30, which is way more expensive than the cheapest plant.
236. 15:30It was supposed to find one for 58 cents.
237. 15:32So, the Mistral isn't able to answer these kind of images.
238. 15:36Now, here's the thing, when it comes to multimodal understanding, you can't really do much prompt engineering cuz either like the like the the model either understands an image or it doesn't.
239. 15:46Like you might be able to do some prompt engineering if you know you're in a very particular domain and there's some hints that you can give to it about how to read a particular kind of graph.
240. 15:54Um but really with image input, it's about figuring out, you know, for the kind of images that you're going to send to the LLM, does the LLM understand those images?
241. 16:03Is it going to, you know, is it going to work for that scenario, right?
242. 16:08Um because it may be that, you know, maybe Mistral is really good at particular kinds of image tasks, but just not this, right?
243. 16:13So, it's all figuring out what is the task you want to do and is the model going to do a good job at it?
244. 16:19Now, what's interesting with the multimodal, too, is that a lot of these also depend on the language, right?
245. 16:24Here I'm using English.
246. 16:25Remember, all these models are super English-centric, but if you switch to a different language, like if I ask in Spanish if they're caimanes or cocodrilos, it actually thinks uh it says it gives the wrong answer because it's getting so confused in its head between these different like words and how they map across English.
247. 16:44Like it just becomes too hard of a task both understand the difference between alligators and crocodiles and also remember that caiman means alligator in Spanish and is not the same thing as the caiman that we have in like Florida.
248. 16:54Anyway, it's like it's fascinating to see that once we switch to Spanish, it fails at the task.
249. 17:00Um so, that's another thing to keep in mind, especially we're in California, a lot of times when we develop applications here, they are bilingual cuz a lot of, you know, 50% Spanish-speaking here.
250. 17:08Um so, you know, your your mileage may vary as you change languages.
251. 17:13Okay, so now let's go on to function calling, also known as tool calling.
252. 17:17So, tool calling is in fact the basis of like all agents.
253. 17:22All agents are based off of tool calling.
254. 17:24And this is a a capability that the LLMs are trained with specifically during the training process.
255. 17:31So, they are trained to be able to look at a definition of a function or multiple function definitions and look at the user input and say, "Okay, based off that user input, I want to call this function with these arguments
256. 17:41and and then, you know, just return that."
257. 17:45So, in this example here, I'm defining a function using JSON schema.
258. 17:50This is what the LLMs understand.
259. 17:52So, I say, "Okay, here's my function.
260. 17:53It has the name create calendar event.
261. 17:56It has this description and these are the parameters it takes.
262. 17:59It takes a title with this that's a string.
263. 18:01Here's a description, right?"
264. 18:03And all of this is all going to get sent to the LLM.
265. 18:07So, we send this full JSON schema to the LLM and say, "Hey, here's the function.
266. 18:11You got to decide whether you're going to call this and how you're going to call it, right?"
267. 18:16So, this is in fact, even though this this is like an object, this is prompt engineering here.
268. 18:20Everything here is actually like me deciding what is going to be the best for the LLM, right?
269. 18:24What type should I use?
270. 18:26What description should I add here?
271. 18:27What should be the names of these parameters?
272. 18:30Cuz the LLM's going to decide based on all that information how it's going to call this function.
273. 18:36Uh so, then we have a user message.
274. 18:37So, it's like So, the user's like, "Oh, can you throw something on my calendar?
275. 18:40It's a platform team sync.
276. 18:42Me, Sarah from Ange, Marcus, the new PM Priya.
277. 18:44Tomorrow at 1:30 p.m. It's virtual on Microsoft Teams, okay?"
278. 18:48So, then our then we tell the LLM like, "Hey, you're a calendar assistant.
279. 18:52Today is Monday, June 29th.
280. 18:53Use the available tools to process the user's request."
281. 18:56And we send it the tools.
282. 18:57And so, now the LLM is looking at the user request, looking at the tools.
283. 19:01And in this case, it just has that create calendar event.
284. 19:04And it's going to decide, "Hey, is there a tool here I can use to answer the user?"
285. 19:08Uh and then we're just going to go through and see what tool calls it decides to make.
286. 19:13Uh so there we go.
287. 19:14So in and this is like really low-level tool calling here.
288. 19:18Uh once we build an agent, we're going to abstract on top of this, but I wanted to start with the most low-level tool calling we can do.
289. 19:25And the LM just responds back with here's the name of the function and here's the arguments that I want you to send.
290. 19:32The LM itself doesn't call the function.
291. 19:34That's up to us to do.
292. 19:35All right, so what we can see is that uh I put my I put the expected on the left just so you can see and we can see the actual.
293. 19:41So GPT 55 did add me as an attendee, which I didn't really want.
294. 19:46Um but it did pretty good otherwise.
295. 19:47So let's try uh Deep Seek and see how Deep Seek does with this one.
296. 19:54Cuz what we want to see like when we're doing tool calling, we want to see is it calling the right tool?
297. 19:58So it is in this case.
298. 19:58It is calling the create calendar event tool.
299. 20:01And is it passing in the arguments that we expect, right?
300. 20:04We have certain arguments we expect.
301. 20:06In this case, Deep Seek put you as an attendee.
302. 20:09And then for the location, it said virtual {parentheses} Microsoft Teams, which is not what I asked for, right?
303. 20:14So it's not adhering to my argument um constraints as much.
304. 20:19Let's try Mistral as well.
305. 20:22And see how it does.
306. 20:23So once I see this, I can start to realize like, "Okay, there's some ambiguity, right?"
307. 20:28And you know, if it if it called it with the wrong arguments, that means there's ambiguity.
308. 20:31I need to go back into my descriptions and say, you know, uh you know, don't you know, don't include me or you, just named attendees.
309. 20:41Um you know, you could say first and last if provided, right?
310. 20:46So you start to see, you know, what what you haven't been specific enough about and start to clarify that ambiguity and then, you know, evaluate all the different um inputs and see how it's doing.
311. 20:58All right.
312. 20:58So that was a basic tool calling.
313. 21:00The next thing we want to do, we're building up to agents here.
314. 21:03The next thing is to do tool calling in a loop, right?
315. 21:06So when we have an agent, it's calling tools in a loop until it reaches a goal.
316. 21:12Um so, we're going to demonstrate using this just a function calculate.
317. 21:15So, this calculate takes a mathematical expression and returns back the output, right?
318. 21:21And we have, you know, basic support for uh evaluating these uh mathematical expressions.
319. 21:26So, we're giving it basically a word problem where it's like, "Oh, this store is having a sale.
320. 21:30If I buy three shirts and pay, you know, this much tax, what's my total?"
321. 21:34And this can be done several different ways.
322. 21:36It could be done with multiple tool calls to that calculator call with like one little expression at a time, like, "Do this, and then do this, and then do this, and then this."
323. 21:44Or, if the LM is very clever, it could batch all that into one long expression and do a single tool call.
324. 21:51So, here we're testing efficiency, right?
325. 21:53Where, you know, there's more like a lot of times once you have tool calling in a loop, the differences between the models is their efficiency.
326. 22:01Are they able to get to the goal faster?
327. 22:04Can they accomplish the same goal with less tool calls, right?
328. 22:08So, that's what we're going to say.
329. 22:11Uh the only system prompt is do not do math in your head.
330. 22:14Okay.
331. 22:15[laughter]
332. 22:16[gasps]
333. 22:16So, let's see.
334. 22:18Uh that one's really for GB55 cuz it's so proud of its math abilities that it's always trying to do it in its head.
335. 22:23Like, "Come on."
336. 22:24All right.
337. 22:25So, GB55 indeed does the entire calculation in one expression.
338. 22:30So, it only required one tool call to get to the final result.
339. 22:34So, that's good.
340. 22:35That's good.
341. 22:35That's, you know, fast.
342. 22:36It saves us time.
343. 22:37So, now let's try with um DeepSeek.
344. 22:42The only reason I skip Kimmy all the time is cuz it's actually a very good model, and so it does pretty much uh on par with uh GB55 uh for much of these.
345. 22:50Okay.
346. 22:50So, now we can see with DeepSeek, it is going through and it's doing and and and and it's doing like so many how many it did like four.
347. 23:00It did four tool calls, right?
348. 23:02So, it still got to the correct answer, but it took four tool calls.
349. 23:06And every one of those tool calls is another network request, right?
350. 23:11And so, you know, when we're evaluating two two models, we want to both look at the final answer, but also how how long did it take to get to that final answer, right?
351. 23:22Um actually, Kimmy did do something really interesting the other day.
352. 23:24Let's see if it does it again with this one.
353. 23:27Cuz Kimmy most of the time works quite well, but Kimmy does sometimes get into weird loops um when it's kind of like double-checking itself.
354. 23:36Uh okay, this time it did well.
355. 23:37Yeah, the other day when I did Kimmy, it like got the right answer on the first try and then decided to check its work with the like four other calls.
356. 23:43I was like, "No, you're good.
357. 23:44You're good."
358. 23:45[laughter]
359. 23:45[gasps]
360. 23:46Um so that's what that's what we're looking at here, okay?
361. 23:48Is how many how many calls does it actually need to get to the end.
362. 23:54All right, so next um we can go to uh agent loops.
363. 24:00Okay, I'm actually going quite fast through this now, actually.
364. 24:03Uh okay, good.
365. 24:04[laughter]
366. 24:05Um all right.
367. 24:08So now we've got the basis, right?
368. 24:09We've seen tool calling, tool calling in a loop.
369. 24:11So now we can actually build like a proper agent.
370. 24:14And for this, we'd usually use an agent framework.
371. 24:16Uh I've gone ahead and put three different agent frameworks in here uh so that you can pick your favorite.
372. 24:22So we have LangChain, Pydantic AI, and Agent Framework.
373. 24:26Uh I'm sure there's other agent frameworks as well that I missed, but hopefully you can find one that you like there.
374. 24:32Uh so these are all, you know, the same idea.
375. 24:34All right, which one do we want to show?
376. 24:37Someone vote.
377. 24:39LangChain.
378. 24:39All right, LangChain is still very popular and uh hopefully there's some LangChain folks here today at the conference.
379. 24:45All right, so with LangChain, we are using LangChain V1, um which is a a nice uh newest TK version of theirs if you haven't tried it in the last year.
380. 24:54And so we connect to the Foundry models from LangChain.
381. 24:57and what we have here is a travel planning agent, which is a classic example.
382. 25:02So, it has tools.
383. 25:03So, we have a tool for search flights, search hotels, and check budget to check if a certain flight and hotel fits within the budget, uh and then search activities within a city.
384. 25:13Uh then I've got some middleware, which is just for me to see what tools are happening.
385. 25:17And then we have the actual agent call.
386. 25:19So, the agent has a system prompt that says you're going to help users find flights and hotels.
387. 25:24Uh you're you know, it has some rules here like search for both flights and hotels, prefer direct flights, yada yada, always check check budget, and then ask for activities
388. 25:33at the end based off the remaining budget.
389. 25:36Then we have the user message.
390. 25:37I need to fly from SF to New York City next month for 3 nights.
391. 25:40My total budget is 6 600, and I want direct flights and an activity.
392. 25:45All right.
393. 25:46So, we have all that, and then we are running the agent.
394. 25:50There's some additional code here that's just logging out the tool calls so that we can see it.
395. 25:54So, let's go ahead and try that out.
396. 26:00Okay.
397. 26:03And so, this is going to log out the tool calls that happened.
398. 26:06We're starting off with GPT 55 here.
399. 26:08Now, what's interesting here, you'll see turn one.
400. 26:10Let me make this bigger.
401. 26:12Turn one, it says is parallel.
402. 26:15So, one thing is you can do with LLMs is uh is like many LLMs support parallel tool calling where if you give them a bunch of tool definitions, they can suggest a multiple tools that can be called at the same time in the same turn.
403. 26:29And this is really good from a performance perspective because on your server, you can actually file like you can send off requests in parallel to like two different databases, right?
404. 26:39Uh so, as long as they're independent of each other, this is a great thing that LLMs can do is say, "Hey, these two these two things can be done in parallel.
405. 26:46Why don't you do them?"
406. 26:47So, uh GPT 55 on the first turn and "Hey, search for hotels in New York City and search for flights in uh from SF to New York City."
407. 26:56cuz it's decided that there wasn't a dependency between them.
408. 26:59And then, once that came back, once it had all the options, it did a check budget um based off of the flight cost and the hotel cost that it got back from that call.
409. 27:09And then, it saw what the remaining budget was and then it did a call for search activities with that remaining budget.
410. 27:15So, this is probably the most efficient agent that we can have for this scenario.
411. 27:20So, let's see if we get similar um results when we change models here.
412. 27:28Do do do.
413. 27:32And of course, here you can even change and compare across both models and frameworks, right?
414. 27:37Because every framework has a slightly different tool calling loop.
415. 27:40We also call it the harness, right?
416. 27:41So, a lot of times that you're you want to compare both models and harnesses and and you'll see there's interesting differences across them.
417. 27:49All right.
418. 27:49So, this time we're with Kimmy and Kimmy is not paralyzing uh as well.
419. 27:54I don't know if that's because Kimmy doesn't support parallel tool calling or it's just deciding not to in this case.
420. 27:59So, this time it did the same Actually, it did even more calls.
421. 28:03So, it did search flight and search hotels.
422. 28:05It did over two turns, so that's already less efficient cuz it didn't paralyze.
423. 28:09Then, it decided to do two calls to check budget cuz I think it was checking like one more option.
424. 28:14And then, it did the final thing.
425. 28:15So, this one took two extra tool calls and it took longer because it didn't paralyze that first uh set of turns.
426. 28:23All right, let's see how we do with DeepSeek.
427. 28:31And you can start to see why it's hard cuz everybody asked me like, "How do you evaluate agents?"
428. 28:36Agents are the hardest to evaluate because there's so much going on.
429. 28:39There's so many different things that are happening along the way.
430. 28:42It's like, are is it calling the right tools?
431. 28:44Is it calling them efficiently?
432. 28:45Is it calling things in parallel?
433. 28:47Um in this case, you know, this one uh didn't do any parallelizing, um but at least it only did four tool calls.
434. 28:54So, you know, reasonably efficient.
435. 28:58And then Mistral Mistral often will see Mistral often forgets the very last step.
436. 29:04And so, we'll see.
437. 29:05Um you know, that's the other problem with the agents.
438. 29:07Sometimes agents, when they get to the like once their context window fills up, they might forget some earlier instructions.
439. 29:14They may not be as thorough um with, you know, paying attention to what the original steps were.
440. 29:19And sometimes you have to like kind of prompt them along the way to to remember like, "Oh, wait, you're supposed to you know, finish up with this final step here, right?"
441. 29:29Okay, so it did it didn't do parallel.
442. 29:31So, what we're going to see is GPT-5.5 is very good at parallel tool calling.
443. 29:35Um the other ones not not aren't doing it.
444. 29:38Okay, so Mistral decided to only search flights and search hotels and then not do anything else.
445. 29:44Um so,
446. 29:44[laughter]
447. 29:45it did it did it kind of gave up there.
448. 29:47Um so, that's an example where, okay, we'd have to do some additional steering there to be like, "Hey, are you sure that you have everything, right?"
449. 29:54Um you could use something like the route loop.
450. 29:57Uh Jeff Huntley is here today.
451. 29:58He made the route loop.
452. 29:59And you know, you can you can force uh you know, you can force agents to be more aggressive uh instead of just using the default loop.
453. 30:09All right.
454. 30:09Okay, so that uh that was an agent example of saying we also have agent examples in the other frameworks.
455. 30:16Um but what you see is the code is really similar across them.
456. 30:19Uh I tend to think most agent frameworks are, you know, pre- more similar than they are different at this point.
457. 30:25Um but, you know, you're welcome to check them out.
458. 30:29All right.
459. 30:30So, now we can talk about evaluations, right?
460. 30:33So, we've been doing everything just, you know, manually running them and observing the results.
461. 30:38That's nice to get a feel for the kind of differences that you can see, but when you're actually going to be, you know, doing something for production, you do want to actually have bulk automated evaluations
462. 30:50that you can reliably run and compare results, right?
463. 30:54So, the first step that we can do is just uh doing evals for um without, you know, with just using string comparison, right?
464. 31:03Um so, the example I have here is with that create calendar event tool call, cuz what we can do is like we're actually getting back the arguments, we can actually just do a straight string comparison say, "Hey, did it give back virtual, right?
465. 31:16We wanted virtual for this, did it give back virtual?"
466. 31:19So, when you're testing a single tool call, you can just check to see, did it check did it call call the right tool and did it use the right arguments?
467. 31:26So, that's what we're going to do in this one.
468. 31:30Um and we set up a bunch of test cases, right?
469. 31:32Cuz when you're running evals, you want to have lots and lots of test cases.
470. 31:35You also want it to be easy to add more test cases, cuz once your app is in production, you're going to see how users really use it and you're like, "Oh, okay.
471. 31:43Thank you.
472. 31:44Everything's broken."
473. 31:45No, and then you're going to go and add those new cases of user input to uh your, you know, your evaluations to make sure that you don't drift, right?
474. 31:53This is called like model drift or your your your drift away from it.
475. 31:58So, you don't want your evals to drift away from your user inputs.
476. 32:01So, once they're in production, you want a good system for getting new user inputs into your eval cases.
477. 32:06So, here what I do is I've got like lots of different um you know, uh cases here.
478. 32:11Like, here's a user message, here's the checks, and these checks are just doing really string comparison, uh some list comparison, and we check different things, right?
479. 32:20We check the date math.
480. 32:22Uh I always check LLMs on date math, cuz usually they're kind of bad at it.
481. 32:25So, another tip, if you're doing dates, give them just give them a tool to do date math, okay?
482. 32:29Cuz it's too hard to do date math in your head.
483. 32:32Uh especially on leap years.
484. 32:33I was evaluating
485. 32:34[laughter]
486. 32:35And LLM on a leap year, that was hard.
487. 32:37Uh then we had um okay, ambiguous durations.
488. 32:40I did one in Spanish, right?
489. 32:42It's nice to test across different languages, whatever languages you're going to support.
490. 32:46[snorts]
491. 32:46Uh this one I guess is written written real slangy.
492. 32:49Yo, can you book the big conference room?
493. 32:51Okay.
494. 32:51All right, so we have various edge cases there, right?
495. 32:55And then we're let me to run these evals.
496. 33:02Okay, and these evals in this case I'm going to run it across each of the models and the use cases.
497. 33:09A lot of times in reality when I'm running evals, I'm just running it on a single model and then I'm comparing to like a previous run where maybe I'd done a different model.
498. 33:17Uh but you know, for today uh since we are actively comparing across models, I'm going to run it across all the models so we can see um it's for each of them it's doing the tool call, checking the args that come back,
499. 33:30and deciding if it's a pass or a fail.
500. 33:33Uh now here's the thing about evals is that you don't want to just have these like just be part of like your pie test, like your test suite because these are slow, these are expensive.
501. 33:42Yeah, question?
502. 33:42Um I see you just roll, you know, checking strings against Uh do you know if any frameworks do this a bit more elegantly
503. 33:52You don't like my string checks.
504. 33:54Um yeah, so framework-wise for evals a lot of so first of all, most observability providers have evals built in, right?
505. 34:00So if you're doing LangFuse, LangSmith, LangChain, um Arize, like they all have their own framework built in.
506. 34:06For Foundry, we have um we have two different frameworks uh for it and I actually I do have the examples in the repo.
507. 34:13Um um those ones there's Azure AI evaluation SDK, Open uh AI.evals um from the OpenAI package.
508. 34:20So there's a bunch of frameworks, there's also DeepEval and Prompt Prompt foo in JavaScript if you're a JavaScript person.
509. 34:29And which one?
510. 34:30Vali.
511. 34:30Vali?
512. 34:31Something like that.
513. 34:32Mark Vali, okay.
514. 34:33V A L L Y
515. 34:34V A L L Y, okay.
516. 34:36I mean you use that one.
517. 34:37Um so there's a number of eval frameworks um I'll I'm going to use a fancier one soon.
518. 34:41But for string check, but here's the thing like like for string check, you don't really have to do a fancy one um because you're like
519. 34:49If you're looking for it to do a specific
520. 34:51Yeah.
521. 34:52Do the parallelization you mentioned for
522. 34:54Yeah, yeah.
523. 34:55So for tool calls, yeah, that's where you're going to get a bit fancier, yeah.
524. 34:57Yeah, so I wanted to start with the basic, but yeah, most of the time we can't get away with just the basic, right?
525. 35:03Um oh, they did actually really well this time.
526. 35:05Wow, nice.
527. 35:06Um so usually we're going to have to move to fancier evaluation.
528. 35:11Um so that's when we start to bring in LLM as a judge.
529. 35:15Uh so this is cuz a lot of times, you know, the the output is a lot fuzzier and we need to use an LLM to help us reason about that fuzziness.
530. 35:25And there's many frameworks that do wrap this um For this example, I just decided to write it entirely in the in the code so you could see.
531. 35:34So the idea when you use an LLM as a judge is that you need to give it, you know, put the LLM in judge mode and say, "Okay, you are an evaluator.
532. 35:44This is what you're evaluating and you need to either say does it pass or does it fail?"
533. 35:49Right?
534. 35:49We're trying to force the LLM to say does this pass or fail?
535. 35:52Um and we also want it to give it reason, right?
536. 35:54It's really really helpful to have it give it reason.
537. 35:56So we want pass, fail, and a reason.
538. 35:59And then we give it the query, we give it the response.
539. 36:02And for grounding this, we also want to give it the context cuz that helps it actually reason about it.
540. 36:06So sometimes in order to really evaluate, you do need access to more information.
541. 36:09So like tool calls, if you're trying to evaluate whether it was an efficient user of tool calls, you can't you have to give it everything.
542. 36:16You have to give it the tool call definitions.
543. 36:17You have to give it the traces of the tool calls it made, right?
544. 36:20So to build really good evaluators, a lot of times you're you're not just giving it the input output, you're giving it every everything that happened along the way, everything that was shown.
545. 36:30Uh so here we, you know, we tell the LLM, "Hey, we're going to use the judge model of GPT-5.
546. 36:36It's a pretty good judge model."
547. 36:38And we're going to ask for it to output some JSON that we can look at.
548. 36:42So let me go ahead and run uh this LLM as a judge.
549. 36:48Now, here's the thing, um this is going to be even slower because now each time we're running an eval, we're both running our own code, which is using a model,
550. 36:57and we're then using another LLM to judge that code, right?
551. 37:01So every single eval case is actually two LLM calls.
552. 37:05And then many times you actually run multiple LLMs as a judge.
553. 37:08Like you'll have like five different evaluators that you're running.
554. 37:11Each of those is going to be a separate LLM call, right?
555. 37:14Um so that that's what is tricky about evals is that you can't just run them all the time.
556. 37:20You can't necessarily run them on every single CI uh and every single commit because they are fairly slow to run.
557. 37:27Cuz normally in production, like for rag, when I'm evaluating rag, you know, you usually want like 200 test cases at least, maybe 800.
558. 37:35And that's that's like a lot of calls to an LLM, right?
559. 37:38So that that takes some amount of time.
560. 37:41Um so you do want to think about how often you're going to be, you know, running evals.
561. 37:44So you can see that it's going through its text it it's got a bunch of different test cases, different questions and answers.
562. 37:51And it's it's looking at the results.
563. 37:53Now, in this case, what I did was that I like pre-ran the um the questions and answers.
564. 37:58This is a common way of doing it, too.
565. 37:59Like get, you know, get all your get your context for the evaluator beforehand um because then you can change your evaluator independently from your app, right?
566. 38:11Um and that's more efficient.
567. 38:12Cuz a lot of times what you realize is that your evaluator is incorrect, right?
568. 38:15Like that you haven't written good evaluator.
569. 38:18Uh so that's what you have to figure out when you're doing LLM as judge, you have to figure out like wait, is the problem my app or is the problem my judge,
570. 38:26right?
571. 38:26And uh you also want to calibrate your judges with human input as well.
572. 38:29So, it definitely helps me working on a domain you know, or work with a domain expert that can help you label your data and say and you know, do they agree with your LLM as a judge?
573. 38:39Was it actually a pass or a fail?
574. 38:42Uh so here we get the results back and we can see you know, the model um and their scores um most of them got like two out of three.
575. 38:52Mr. Owl got one out of three.
576. 38:54You know, so they're they're they have some work to do on being more grounded here.
577. 39:00So that is the generator of using LLM as a judge.
578. 39:03As you see, there's quite a few frameworks that uh make it, you know, that have built-in LLM as a judge that make it easier to do it.
579. 39:10Um but I want to show that you you know, you can't you can't also do it yourself.
580. 39:13Oh, also I should recommend if you're interested in evaluation, Hamel.dev.
581. 39:18I don't know if he's at the conference this week, but he has the most amazing resources and he teaches courses all the time about evals.
582. 39:25And also he's very good memes.
583. 39:27I actually used these last night.
584. 39:28So, if you're ever chatting with someone about evals and you need an eval meme, uh what was it what I was doing last night?
585. 39:35Yeah, building an LLM as a judge it's it's hard.
586. 39:38[laughter]
587. 39:40Uh so very important resource here.
588. 39:43How can we trust the LLM judge?
589. 39:44You have to calibrate your LLM judge.
590. 39:47Okay.
591. 39:49All right.
592. 39:49So, that was LLM as a judge.
593. 39:51So now um but then they as you see, it is hard to build judge evaluators, right?
594. 39:58Like you have to like follow Hamel on Twitter and take his course and learn all these best practices for how you write uh a good eval.
595. 40:06Um uh so we have this new tool from Microsoft Research.
596. 40:10It's very new, like the last week.
597. 40:12Um it's called Assert.
598. 40:14And Assert helps you write the judges based off of the spec.
599. 40:19So, you write a spec.
600. 40:20You say, "Hey, this is the agent I'm building and this is, you know, roughly what I want it to be doing, my idea about how it should work, and um what kind of users it should support."
601. 40:31And then it will generate lots of test cases.
602. 40:34It'll like flesh out your your spec into um into actual judging rubrics, and figure all of that out for you based off of, you know, all the the it's very based in in research, right?
603. 40:45So, we've done that for the travel planner already in here.
604. 40:51Um so, here's the um so, it's under this assert eval.
605. 40:54Okay.
606. 40:55So, what we can see is that we write a YAML file cuz we all wanted more YAML in our life.
607. 41:00What I realized about YAML is that I might hate YAML, but it's also better than everything else for what I need it for.
608. 41:05So, there we go.
609. 41:07Um so, in this YAML, you know, we describe uh the context, and we describe, "Okay, we're going to use this judge model.
610. 41:16Um these are the kind of dimensions we care about is budget, you know, budget pressure."
611. 41:20So, we can we can give it more information if we know, like, "Oh, we care about ample budget, tight budget, invisible."
612. 41:26Um you know, so, you can you can stratify it.
613. 41:28This is called stratification.
614. 41:30You can you can pre-stratify if you want, uh or you can just have it come up with the stratification.
615. 41:36Uh and then we have more dimensions here.
616. 41:39And there's this there's also a a skill an agent skill that helps you write this YAML file so that you don't have to write it yourself cuz no one wants to write it, right?
617. 41:46Um and then we have the actual thing that's evaluating.
618. 41:49This is the LangChain travel planner that we saw before.
619. 41:51Uh so, this just makes sure that it's tracing everything cuz it's saying, "When you are doing evaluations, you do want to see every single tool call so that you can really properly do these evaluations."
620. 42:02All right.
621. 42:02So, then it it runs it all, and then we can see the results.
622. 42:06It does take a, you know, a decent amount of time.
623. 42:08So, we've already got the um evals in here.
624. 42:12Uh so, I've got them running.
625. 42:15Uh there's like a little like front end that you can use to view them.
626. 42:18Okay.
627. 42:20All right.
628. 42:21So, here is the result of this um a search evaluation.
629. 42:27Um so, basically it you know, it came up with the different behavior categories, right?
630. 42:33So, making like budget in infeasibility, direct flight preference rule violation, required tool use, right?
631. 42:40So, uh it comes up with what behaviors are you know, allowed but not allowed.
632. 42:46And then it comes up with the test set.
633. 42:47The test set is really is pretty fun.
634. 42:50Um and here we go.
635. 42:52Scenario 65.
636. 42:53Red eye before the installer walked through the coach with six parent text one weekend to see Alma.
637. 42:58I really like that these are obviously LLM generated titles, right?
638. 43:01So, it came up with all of these coach with a clipboard.
639. 43:05Um it can it comes up with all these scenarios based off that that YAML to try and simulate users, right?
640. 43:12Cuz the whole thing it's really hard to bootstrap your evaluation before you go into production, but you want to have evals before you go into production, right?
641. 43:19So, how do you figure out, you know, what your test set is going to be?
642. 43:23And so, this assert framework helps a lot with making that test set and bootstrapping your evals.
643. 43:30Uh so, it has prompts and scenarios.
644. 43:32And then we have the results.
645. 43:33Okay.
646. 43:33So, for the results, it ran it um across Deep Sea, Kimmy, and GBD 5.5 and uh came up with the you know, with the numbers here.
647. 43:42Um and in this case the the lower numbers are better.
648. 43:46And so, we can see that uh GBD 5.5 did way better than the other ones.
649. 43:51I was actually surprised cuz Kimmy, we've seen that Kimmy has generally been quite good today um and I was thinking it was kind of on par, but at least for this scenario,
650. 44:00uh GBD 5.5 had the lowest number of violations, and so it is it's scoring the best here.
651. 44:08Um so So yeah.
652. 44:10So this is assert.
653. 44:12I will say I'm I'm brand new to assert since it only came out like a week ago.
654. 44:16Um so this is all I've I've used it so far, but the idea here is that it's helping you to build those evaluators to build that test set and to follow the best practices and bootstrap your evals so that you can start
655. 44:29start off um you know, with with some really good evals.
656. 44:33Cuz what we're seeing is that we keep telling people like you got to run evals, you got to run evals, and then no one's running evals or they're running like the default evals like coherence
657. 44:41and fluency.
658. 44:42Never run those.
659. 44:43Like LLMs are great at being coherent.
660. 44:46You never have to check if an LLM is coherent.
661. 44:48If some if they're not, like something really bad has happened.
662. 44:51Um I do sometimes use the groundness and relevance evaluators for my rag things, but most of the times the built-in evaluators are not what you want.
663. 44:59You want to build custom evaluators.
664. 45:01And that's what this is helping you with is like okay, you need custom evaluators to really see if your agent is doing a good job.
665. 45:09Uh but you know, you need some help bootstrapping those custom evaluators.
666. 45:14All right.
667. 45:14Oh, we have 3 minutes left.
668. 45:15I've been blabbing.
669. 45:16Okay, finally, um you know, earlier we were talking about prompt engineering like okay, how can we tweak this prompt in order to work better for these models?
670. 45:25That's pretty painful, right?
671. 45:26If you have to tweak the prompt every time you switch models and like rerun evals, that's like a lot of work.
672. 45:35Oh, there's a little timer.
673. 45:36Thank you.
674. 45:36Okay.
675. 45:37Uh so if you Yeah, if you have to tweak the model every time, uh that's a lot of work and you don't know which way to tweak tweak the model, right?
676. 45:43You don't know the details of every single model, what's going to really work well for it.
677. 45:48So uh what I've added is this as a final example is using Dspy.
678. 45:52Has anyone used Dspy here?
679. 45:55Cool, a couple people.
680. 45:56So Dspy, the idea of Dspy is that you declare your program like you say, "Hey, I'm making this program that's expecting to take user input and then turning it into a structured counter bit."
681. 46:08So, you declare the constraints of your program and then Dspy is going to figure out the prompt that adheres to those constraints.
682. 46:15So, Dspy is going to write your prompt.
683. 46:17So, you you're basically checking your spec in and then having Dspy by write the prompts based off of doing evaluations and seeing, "Hey, what prompt is going to satisfy those constraints?"
684. 46:27And it actually uses these this cool algorithm called Jeppa, genetic credo algorithms.
685. 46:34It can use these genetic algorithms in order to come up with the next best prompt to try, which is really really cool to watch it do that.
686. 46:41Um so, that's the idea of Dspy.
687. 46:43And so, basically we have to set So, it's very different, right?
688. 46:45You can't just take your standard code and just use it.
689. 46:48You kind of have to rewrite your whole code cuz now, instead of checking your pro- like your prompt isn't in your code, your constraints are in your code.
690. 46:55Your spec is your code.
691. 46:56The prompt is an output from this Dspy process, right?
692. 47:00So, here I'm doing I'm asking it to generate a numbered list where each line has exact word count and the first letter of the line spell out a target word.
693. 47:06So, this is the kind of input and then we have a bunch of constraints that are checking to see is the result correct, right?
694. 47:12So, you need to set up your spec and then set up like checks, tests to see, "Hey, does the output satisfy this constraint?"
695. 47:19And that way Dspy can keep iterating, hill climbing, right?
696. 47:22Basically hill climbing until it finds a prompt that satisfies those constraints, right?
697. 47:27So, you give it the student model, the reflection model, uh then it runs this Jeppa uh optimizer, which you know, tries to find the best um prompt, and then we get back the result.
698. 47:37So, the result I have it stored here so we can see it.
699. 47:41It does take a fair amount of time.
700. 47:43It took like 6 minutes on this task.
701. 47:45Um but here, this result is the actual instructions.
702. 47:49So, let me add word wrap into there.
703. 47:53Right.
704. 47:53So, this is the system prompt that it came up with based off of those constraints and those you know those checks, right?
705. 48:01And so this is nice because this is a system prompt that works with Mistral.
706. 48:05Cuz I I picked Mistral cuz that was the one we were having the hardest time with, right?
707. 48:09So especially with using smaller models, this means we can use the cheapest model and still get the results we need, right?
708. 48:15So there's been some great talks by like Dropbox, Shopify about how they've used DSPy plus Jeppa so that they can move off the larger models and use the smaller models and maybe even deploy it on their infrastructure
709. 48:27and get the same results.
710. 48:29But the key was trying to you know using this order to figure out what that optimal prompt is.
711. 48:34So this is a definitely a different way of writing your apps and it can feel a little different to you know to have the prompt be the output of this process,
712. 48:43but it's also potentially really powerful way of writing it and can give you a lot more model independence and more flexibility to be able to use more of these models.
713. 48:53So I wanted to yeah, make sure that people knew about that as an option as well.
714. 48:59Okay.
715. 49:00All right.
716. 49:00So So yeah, so that was all the things that I wanted to show.
717. 49:06As you can see there's lots of differences between models, different ways you can improve the model output and then evaluate it and then improve it.
718. 49:15I do encourage you like yeah, just just tinker with the models.
719. 49:19It's I always learn so much whenever I you know try out different models um and see how different they are.
720. 49:26Oh, I never showed Anthropic.
721. 49:27I meant to show Anthropic.
722. 49:28Okay.
723. 49:29I do have some Anthropic's deployed as well.
724. 49:31With Anthropic you do have to use a different API.
725. 49:34That's why I wasn't using Anthropic most of the time.
726. 49:37But of course you might be interested in Anthropic.
727. 49:39They They use the messages API instead, so you do have to switch your code around.
728. 49:43But I actually specifically like it for rag because here's interesting is that with Anthropic they actually have this notion of citations built into the models.
729. 49:56So, you can actually you provide the sources in this structured format.
730. 50:00So, you say, "Hey, this is my source.
731. 50:02It's a document.
732. 50:02It's text.
733. 50:03Here's the text of it."
734. 50:04And then you enable citations.
735. 50:06And then the Anthropic models are actually trained to return back blocks of text with citation.
736. 50:12So, you can extract those in a programmatic way, which is really nice when you're like building front ends.
737. 50:17You can just link your citations.
738. 50:19So, that's really cool.
739. 50:20So, that's even another thing to look at is you know, a whole different ways of calling models, too.
740. 50:27Okay.
741. 50:28All right.
742. 50:29And yeah, we're kind of at time here.
743. 50:32The the repo will stay up.
744. 50:35So, you're welcome to keep working on the repo.
745. 50:38I do need to rotate the keys that I put in that .env because I mean, I trust you, but I don't know how much I trust you.
746. 50:45And Cedric always already said I was bad for doing that.
747. 50:49So, I I won't be able to keep those keys up for much longer.
748. 50:53But, hopefully with the repo you can try it out.
749. 50:55You can also try it out with a llama.
750. 50:56I added a llama support yesterday.
751. 50:58So, if you want you know, do you want to try a llama, too?
752. 51:00If you're doing local models, you basically just change the examples to point at the local llama server.
753. 51:07And you just have to change the model names as well, right?
754. 51:11But, generally they should all work with a llama as well because it's same API.
755. 51:16All right.
756. 51:17Thank you, everyone.
757. 51:18Oh, and there's stickers.
758. 51:20[applause]