WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=UGjk3Uk0MxM NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/UGjk3Uk0MxM.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:13.440 --> 00:00:14.040 All right. s2 00:00:14.040 --> 00:00:20.440 Uh since it's already like 11:12, I want to start talking about the workshop. s3 00:00:20.440 --> 00:00:30.400 Um so, I'm going to do this as a guided workshop, so I'll go through all the examples up here on screen, um but you can also uh try and follow along. s4 00:00:30.400 --> 00:00:34.040 So, the repo is there, the environment variables are there. s5 00:00:34.040 --> 00:00:42.280 Uh so, the environment variables give you access to a bunch of Foundry models that I've deployed on uh Microsoft Foundry project. s6 00:00:42.280 --> 00:00:46.040 Um and they have a certain amount of token capacity. s7 00:00:46.040 --> 00:00:52.200 It is possible that we are going to go over the token capacity um in you know, in this session. s8 00:00:52.200 --> 00:01:00.720 Um I only got found out very recently that this workshop was accepted, so I tried to get as much resources as I could in the small period of time. s9 00:01:00.720 --> 00:01:09.600 Um so, in that EMB file, there is actually uh multiple different possible environment variables that you can comment in or comment out. s10 00:01:09.600 --> 00:01:14.680 If we do start getting 429s, um you know, uh rate limit exceeded errors. s11 00:01:14.680 --> 00:01:15.360 Uh so, yeah. s12 00:01:15.360 --> 00:01:22.320 So, I'm hoping that there's enough capacity for all of you to be able to try out those models, and uh we will see. s13 00:01:22.320 --> 00:01:24.880 Uh I also have my colleague Cedric here. s14 00:01:24.880 --> 00:01:32.400 Uh so, Cedric is going to be wandering around while I'm talking, so he might be able to help uh with with issues as well. s15 00:01:32.400 --> 00:01:44.520 Uh if you In terms of setting up the code, if you already have Python and UV on your machine, then you can do a get clone of the repo and run UV sync, and that should have everything you need to run the examples s16 00:01:44.520 --> 00:01:46.600 uh once you have that .env. s17 00:01:46.600 --> 00:01:49.560 Uh otherwise, you can make a GitHub code space. s18 00:01:49.560 --> 00:01:55.400 That's uh basically a VS code in the browser that has everything set up for you. s19 00:01:55.840 --> 00:01:57.760 Uh who here uses UV? s20 00:01:57.760 --> 00:01:59.880 Cedric wants to know. s21 00:01:59.880 --> 00:02:01.320 Okay. s22 00:02:01.320 --> 00:02:03.440 So, it's not number one yet. s23 00:02:03.440 --> 00:02:06.720 But, the other question here, who uses Python? s24 00:02:06.720 --> 00:02:07.680 All right, then more people. s25 00:02:07.680 --> 00:02:08.679 Okay. s26 00:02:08.679 --> 00:02:11.240 Pretty much everyone uses Python here. s27 00:02:11.240 --> 00:02:11.680 Yeah. s28 00:02:11.680 --> 00:02:12.840 Oh, did you count the overlap? s29 00:02:12.840 --> 00:02:13.600 All right. s30 00:02:13.600 --> 00:02:14.280 Okay. s31 00:02:14.280 --> 00:02:18.000 All right, so if you didn't take a picture of that screen yet, just take a picture of it now. s32 00:02:18.000 --> 00:02:23.440 Otherwise, Cedric also has a picture of it on his phone, so he can go around if you need those links. s33 00:02:23.440 --> 00:02:35.200 Uh, so the goal of this workshop is to be able to get a feel for the differences between the different, you know, uh, top-of-the-line models and see how they differ and, you know, what you can improve about them. s34 00:02:35.200 --> 00:02:42.680 So, I've set up a bunch of different examples and also evaluations and optimizations that we can do to improve our experience across models. s35 00:02:42.680 --> 00:02:51.760 Uh, and I think this is a really important thing for all of us to just be to be doing as AI engineers because, you know, as you've seen, like, there's so many new models that come out all the time. s36 00:02:51.760 --> 00:02:55.760 And, you know, the US government at any point could cut off access to a model. s37 00:02:55.760 --> 00:02:58.040 So, we need to like, we need to have options, right? s38 00:02:58.040 --> 00:03:06.440 So, we need to be able to, you know, point our AI applications at a different model and and have it still work or know how to improve it. s39 00:03:06.440 --> 00:03:09.600 Uh, so that's the goal here is to get more experience with more models. s40 00:03:09.600 --> 00:03:13.760 So, uh, these are the models I've deployed inside Microsoft Foundry. s41 00:03:13.760 --> 00:03:18.200 Uh, so I've got uh, GPT-5 5, that's a standard one I use all the time. s42 00:03:18.200 --> 00:03:22.760 But, then I've got Kimmy K2.6, that's a uh, more recent one that's quite good. s43 00:03:22.760 --> 00:03:25.160 I've got Deep Seek V4 Flash. s44 00:03:25.160 --> 00:03:32.760 And then I have Mistral Large 3. So, these are the four models that we'll be comparing in this workshop. s45 00:03:32.760 --> 00:03:40.480 Um, so as I was saying, you can in order to set up the workshop, here's the repo that that you're all into. s46 00:03:40.480 --> 00:03:47.200 So, if you don't have Python on your machine, you can just click on the code button and there'll be a big button here that says create code space. s47 00:03:47.200 --> 00:03:50.160 So, you click on that, it'll make a code space. s48 00:03:50.160 --> 00:04:02.560 So, then here's the uh, the code space and then this has um, I've already put the dot env, but then you go and you grab that other um URL that I put up there to get the env. s49 00:04:02.560 --> 00:04:05.280 That goes to this file here. s50 00:04:05.280 --> 00:04:06.720 You download it. s51 00:04:06.720 --> 00:04:08.880 You open that. s52 00:04:08.880 --> 00:04:13.000 And then you create a dot env with the contents, okay? s53 00:04:13.000 --> 00:04:19.680 So, this is pointing at a that foundry project that has those four models deployed. s54 00:04:19.680 --> 00:04:24.000 As you've seen, that does have you know, it has token limits. s55 00:04:24.000 --> 00:04:30.160 So, if we start getting rate limits, you can start you can try these other models that also have the four project deploys. s56 00:04:30.160 --> 00:04:36.120 And then I have a separate one specifically for Anthropic just because it has different terms for that project. s57 00:04:36.120 --> 00:04:37.760 All right. s58 00:04:37.760 --> 00:04:47.200 So, so yeah, and then once you've got that set up, then you v sync will sync all of the packages that you need for Python. s59 00:04:47.200 --> 00:04:50.000 All right, so that's the setup. s60 00:04:50.000 --> 00:04:59.560 Now, I'm going to actually go through it and do it do it up here so that you can you know, hopefully follow along if you want, or you can just watch s61 00:04:59.560 --> 00:05:00.960 watch what I do. s62 00:05:00.960 --> 00:05:05.440 So, as I was saying, we do have the four models, and you can compare models just doing benchmarks. s63 00:05:05.440 --> 00:05:11.000 You're like, oh, you know, this one has good quality benchmarks and good safety benchmarks. s64 00:05:11.000 --> 00:05:14.720 You can also look at you know, the cost and thoroughput and stuff like that. s65 00:05:14.720 --> 00:05:26.760 And you can get this approximate notion for how good these models are, but you don't really know how good a model is until you use it for your particular scenario because there's so many different ways of using models. s66 00:05:26.760 --> 00:05:31.040 And a particular benchmark isn't really going to tell you how it's going to work for your scenario. s67 00:05:31.040 --> 00:05:38.240 So, that's what we have in this repo here is a ton of different scenarios, and we can try out these models for each of them. s68 00:05:38.240 --> 00:05:40.800 So, we're going to start really really basic, all right? s69 00:05:40.800 --> 00:05:45.280 So, just a single LLM call that asks the LLM to do something. s70 00:05:45.280 --> 00:05:51.800 So, what you see for each of these files is that they all start off with the model variable and they default to GPT-3.5. s71 00:05:51.800 --> 00:05:52.840 That's kind of our baseline. s72 00:05:52.840 --> 00:05:55.040 It's obviously it's a very good model. s73 00:05:55.040 --> 00:06:00.600 But then if we want to try a different model, we can just uncomment that line and that will be the model that gets used. s74 00:06:00.600 --> 00:06:11.520 So, this file here is connecting using the OpenAI Python SDK to the Foundry URL and passing in the key from the environment. s75 00:06:11.520 --> 00:06:17.040 And the prompt we're doing here is how many times does the letter E appear in the following sentence? s76 00:06:17.040 --> 00:06:20.120 The elderly gentleman eagerly entered the elevator. s77 00:06:20.120 --> 00:06:21.520 Count carefully. s78 00:06:21.520 --> 00:06:21.960 All right. s79 00:06:21.960 --> 00:06:30.240 So, then we send that to the OpenAI responses API and say, "Here, use this model with this prompt and then let's see the raw the output, right?" s80 00:06:30.240 --> 00:06:34.120 So, we're doing the classic thing of asking LLMs to count. s81 00:06:34.120 --> 00:06:36.560 And so, we'll see, you know, how well they're able to do. s82 00:06:36.560 --> 00:06:42.400 So, I'm going to say UB run examples with the name of this this file name here. s83 00:06:42.400 --> 00:06:45.120 And this is all in, you know, in the read me. s84 00:06:45.120 --> 00:06:48.520 So, we're just going through one by one. s85 00:06:48.520 --> 00:06:50.000 So, you can just copy-paste from there. s86 00:06:50.000 --> 00:06:50.440 All right. s87 00:06:50.440 --> 00:06:54.320 So, GPT-3.5 is actually very good at math, surprisingly good at math. s88 00:06:54.320 --> 00:07:00.720 My colleagues have been running a lot of finance benchmarks on it lately and they're like, "Oh my god, like it does so much math in its head." s89 00:07:00.720 --> 00:07:04.840 So, it does actually get this correct that there are 13 E's in this sentence, right? s90 00:07:04.840 --> 00:07:07.320 So, now let's try Now we just change it, right? s91 00:07:07.320 --> 00:07:08.919 So, okay, let's say we want to try Kimmy. s92 00:07:08.919 --> 00:07:14.240 So, we go and we change to Kimmy and now Kimmy will be the model that's used. s93 00:07:14.240 --> 00:07:18.480 And we see, "Okay, can Kimmy count as well as GPT-3.5?" s94 00:07:18.480 --> 00:07:19.280 And it does. s95 00:07:19.280 --> 00:07:20.560 It gets 13. s96 00:07:20.560 --> 00:07:20.919 All right. s97 00:07:20.919 --> 00:07:23.280 Now you might be thinking, "Oh, is this just too easy? s98 00:07:23.280 --> 00:07:26.640 Let's try DeepSeek V4 Flash. s99 00:07:26.640 --> 00:07:29.040 And this time it gets six. s100 00:07:29.040 --> 00:07:30.080 Okay. s101 00:07:30.080 --> 00:07:33.440 So, it missed out on seven of the E's, right? s102 00:07:33.440 --> 00:07:40.480 So, we can immediately see that there's definitely difference between you know, DeepSeek V4 and those first two models. s103 00:07:40.480 --> 00:07:42.120 And then finally, we'll try Mistral. s104 00:07:42.120 --> 00:07:43.880 And this one gets eight. s105 00:07:43.880 --> 00:07:45.760 Uh so, what we see there's a difference here. s106 00:07:45.760 --> 00:07:46.120 All right. s107 00:07:46.120 --> 00:07:48.960 So, when you see this and you're like, "Well, I want to be able to use all these models." s108 00:07:48.960 --> 00:07:50.640 Then you start doing some prompt engineering. s109 00:07:50.640 --> 00:08:03.960 You're like, "Okay, what if I say, uh you know, break the sentence into words, count per word, uh sum, show your reasoning, right?" s110 00:08:03.960 --> 00:08:06.560 Let's try Let's try that, right? s111 00:08:06.560 --> 00:08:09.640 And you know, we can fix our typos, not that it matters. s112 00:08:09.640 --> 00:08:09.920 All right. s113 00:08:09.920 --> 00:08:18.600 So, let's try it with Mistral and see with a little bit of prompt engineering, can we get a better better response here? s114 00:08:18.720 --> 00:08:19.320 All right. s115 00:08:19.320 --> 00:08:20.560 And it got it, right? s116 00:08:20.560 --> 00:08:28.000 So, you know, by breaking it up, doing a little chain of thought here, it was able to come up with the correct answer, right? s117 00:08:28.000 --> 00:08:36.159 So, immediately, we can see that there's obviously differences across models and if we change prompts, you know, we can get uh different outputs. s118 00:08:36.159 --> 00:08:36.560 All right. s119 00:08:36.560 --> 00:08:40.919 So, now let's get more interesting and go to more real-world scenarios. s120 00:08:40.919 --> 00:08:42.479 Uh so, let's look at RAG. s121 00:08:42.479 --> 00:08:44.640 So, RAG is still one of the big uses. s122 00:08:44.640 --> 00:08:46.320 Arguably, really everything is RAG. s123 00:08:46.320 --> 00:08:47.839 Agents are RAG, too. s124 00:08:47.839 --> 00:08:54.680 Um So, we have this So, RAG is when you ask an LLM to answer a question based off some context, right? s125 00:08:54.680 --> 00:08:59.680 And the thing that's important for RAG is that you want it to answer the question and stay grounded in the context. s126 00:08:59.680 --> 00:09:03.920 You don't want it to make up anything that is just from its weights, right? s127 00:09:03.920 --> 00:09:07.040 You're trying to force it stick to the context, right? s128 00:09:07.040 --> 00:09:16.480 And that's what I find really varies across the models is that the smaller models, the less powerful models are actually much more likely to go to their weights and pretend it was part of the context. s129 00:09:16.480 --> 00:09:30.160 Um so, what we have here is um we have a data set about carpenter bees and we ask it a question, "What are the key differences differences between how honey bees and carpenter bees build their nests?" s130 00:09:30.160 --> 00:09:34.960 But this data set only has carpenter bee information. s131 00:09:34.960 --> 00:09:38.280 Uh so, it should only actually be able to answer about carpenter bees. s132 00:09:38.280 --> 00:09:44.320 So a good response in this case would tell me the carpenter bee information and then it say, "Sorry, I can't tell you about honey bees. s133 00:09:44.320 --> 00:09:45.640 I don't know." s134 00:09:45.640 --> 00:09:50.280 Um, so we tell it like, "Hey, your job is to answer based on the sources, cite the sources. s135 00:09:50.280 --> 00:09:52.800 Um, here's your here's your question, here's your sources." s136 00:09:52.800 --> 00:09:57.680 All right, so let's go ahead and try that out and see what we get. s137 00:10:02.120 --> 00:10:05.880 All right, and it's Yeah, so when it's doing the search first, very basic search. s138 00:10:05.880 --> 00:10:08.320 Okay, so it's sending did the search. s139 00:10:08.320 --> 00:10:16.240 All right, and what we see and it says the data set does not provide details on honey bee nest construction. s140 00:10:16.240 --> 00:10:17.840 So this is what we actually want to see. s141 00:10:17.840 --> 00:10:19.160 This is from GB55. s142 00:10:19.160 --> 00:10:22.800 GB55 is very good at knowing what it doesn't know. s143 00:10:22.800 --> 00:10:26.240 So now, let's try switching Um, Kimmy's also good as well. s144 00:10:26.240 --> 00:10:32.680 So I'm not going to show every model every time now just for for sake of time. s145 00:10:32.680 --> 00:10:34.200 Um, so let's try Deep Seek, right? s146 00:10:34.200 --> 00:10:37.480 Cuz we've already seen that, you know, there's a distinction here. s147 00:10:37.480 --> 00:10:40.640 Now Deep Seek, once again, says provided sources do not contain information. s148 00:10:40.640 --> 00:10:42.760 All right, so Deep Seek did a good job here. s149 00:10:42.760 --> 00:10:44.360 And now let's go to Mistral. s150 00:10:44.360 --> 00:10:45.400 Uh, Mistral, sorry. s151 00:10:45.400 --> 00:10:46.520 Cedric, how do we say Mistral? s152 00:10:46.520 --> 00:10:47.400 It's French, right? s153 00:10:47.400 --> 00:10:48.320 French company? s154 00:10:48.320 --> 00:10:48.839 Mistral. s155 00:10:48.839 --> 00:10:49.880 Mistral. s156 00:10:49.880 --> 00:10:51.920 Cedric's going to be my French translator. s157 00:10:51.920 --> 00:10:55.040 Every time I need to say it, you're just going to say it. s158 00:10:55.040 --> 00:10:56.640 Now, Mistral. s159 00:10:56.640 --> 00:10:58.120 Mhm? s160 00:10:58.120 --> 00:10:59.320 Yeah, okay. s161 00:10:59.320 --> 00:11:00.520 Mistral. s162 00:11:00.520 --> 00:11:03.522 Um, so Mis- Mis- With Mistral s163 00:11:03.522 --> 00:11:04.760 [laughter] s164 00:11:04.760 --> 00:11:13.120 Um, we do see that Mistral actually answered uh, from its weights uh, even though that information was not in the sources. s165 00:11:13.120 --> 00:11:18.320 So this is an example where a model went into its weights even though we didn't want it to do, right? s166 00:11:18.320 --> 00:11:20.760 Now we can try to force it. s167 00:11:20.760 --> 00:11:23.080 Once again, we can try to do prompt engineering. s168 00:11:23.080 --> 00:11:37.280 Uh, we can say uh, wait, actually so it would say uh, if the if the answer for a bit of info is not in sources. s169 00:11:37.360 --> 00:11:41.200 Say no source provided. s170 00:11:41.200 --> 00:11:43.680 I think this is what I got it to work. s171 00:11:43.680 --> 00:11:46.560 Uh oh, I will try that and do not make up an answer. s172 00:11:46.560 --> 00:11:48.063 All right, let's try this. s173 00:11:48.063 --> 00:11:50.063 [snorts] s174 00:11:52.200 --> 00:11:57.160 So, we're trying to force it to realize when it's giving us information that's not in the weights. s175 00:11:57.160 --> 00:12:06.640 And sometimes if you force it to say, "Hey, for every bit of information that you're giving back in your answer, you have to prove that you got that from s176 00:12:06.640 --> 00:12:08.000 from the sources, right?" s177 00:12:08.000 --> 00:12:21.240 Um so, here I'm trying to have it associate each of the things in that in that answer with a source and um and maybe that will force it to realize, "Oh, wait a second, that wasn't actually, s178 00:12:21.240 --> 00:12:24.280 you know, in my in my answers here." s179 00:12:24.280 --> 00:12:26.040 All right, it's still going out. s180 00:12:26.040 --> 00:12:30.720 It's going to think it's thinking really hard about it here. s181 00:12:30.720 --> 00:12:31.680 Okay. s182 00:12:31.680 --> 00:12:32.360 And there we go. s183 00:12:32.360 --> 00:12:37.843 Honeybee details are no source provided beyond interactions with carpenter bees. s184 00:12:37.843 --> 00:12:38.160 [laughter and gasps] s185 00:12:38.160 --> 00:12:42.640 Um so, it definitely knew when things were not source provided. s186 00:12:42.640 --> 00:12:44.000 It still gave the information. s187 00:12:44.000 --> 00:12:45.440 So, we got a little closer. s188 00:12:45.440 --> 00:12:51.320 But as you can see, this is definitely uh a difference that I see with rag and rag is what I work with the most. s189 00:12:51.320 --> 00:13:03.720 Uh so, it's it's something you want to evaluate when you're looking at models to use with rag is are the models willing to admit what they don't know, how likely are they to make stuff up? s190 00:13:03.720 --> 00:13:06.040 All right, so now let's move to image input. s191 00:13:06.040 --> 00:13:10.760 A lot of these models are multimodal, so they will accept images as input. s192 00:13:10.760 --> 00:13:14.839 And you know, there's lots of really cool um situations you can do with image input. s193 00:13:14.839 --> 00:13:24.440 So, here we've got some files locally and we're just going to send them to the LLMs and ask them some questions about the files, right? s194 00:13:24.440 --> 00:13:27.360 Um I'll show you the files. s195 00:13:27.600 --> 00:13:28.920 Examples, image input. s196 00:13:28.920 --> 00:13:30.920 Okay. s197 00:13:30.920 --> 00:13:31.280 All right. s198 00:13:31.280 --> 00:13:34.560 So, the files we have here are this picture. s199 00:13:34.560 --> 00:13:36.920 This is an aurochs, not a unicorn. s200 00:13:36.920 --> 00:13:37.480 This picture. s201 00:13:37.480 --> 00:13:43.080 This is of some crocodiles that I saw in Costa Rica that I was very afraid of. s202 00:13:43.080 --> 00:13:49.240 And then also we have this picture which is a list of plants. s203 00:13:49.240 --> 00:13:53.760 Basically like a screenshot of a table with some really small fonts in it. s204 00:13:53.760 --> 00:13:58.760 So, we're asking each of the LLMs to interpret each of the images, right? s205 00:13:58.760 --> 00:14:01.000 So, first we say, "Oh, is this a unicorn?" s206 00:14:01.000 --> 00:14:01.240 Right? s207 00:14:01.240 --> 00:14:03.000 That should be easy to say it's not a unicorn. s208 00:14:03.000 --> 00:14:03.720 Great. s209 00:14:03.720 --> 00:14:05.920 Then we say, "Hey, are these alligators or crocodiles?" s210 00:14:05.920 --> 00:14:07.480 That's a bit tricky. s211 00:14:07.480 --> 00:14:10.200 They are actually crocodiles because of their snout. s212 00:14:10.200 --> 00:14:15.240 I spent a lot of time reading the Wikipedia page about crocodiles and you got to look at their snout and then you know. s213 00:14:15.240 --> 00:14:23.280 Luckily, GPT-5.5 has also been on a Wikipedia binge and so it knows that when it sees that snout, then that is crocodiles. s214 00:14:23.280 --> 00:14:31.240 And then the cheapest plant, it correctly found the cheapest plant in that like fairly small text screenshot of a table. s215 00:14:31.240 --> 00:14:34.760 So, that's pretty impressive that GPT-5.5 could do that. s216 00:14:34.760 --> 00:14:41.800 So, now let's try it with Now, for these models, we can't use DeepSeek cuz DeepSeek only accepts text. s217 00:14:41.800 --> 00:14:43.280 It's not a multimodal model. s218 00:14:43.280 --> 00:14:47.320 So, we're only going to try Kimmy and Mistral. s219 00:14:47.440 --> 00:14:50.080 So, you know, it figured out the aurochs. s220 00:14:50.080 --> 00:14:50.839 Okay, good. s221 00:14:50.839 --> 00:14:52.520 Just making sure it works. s222 00:14:52.520 --> 00:14:52.920 All right. s223 00:14:52.920 --> 00:14:57.120 And then for crocodiles, it does say crocodiles correctly. s224 00:14:57.120 --> 00:14:59.200 And then cheapest plant, it got the cheapest plant. s225 00:14:59.200 --> 00:15:02.240 So, Kimmy is doing a great job as well. s226 00:15:02.240 --> 00:15:06.080 And then finally, let's go to Mistral. s227 00:15:06.080 --> 00:15:08.080 Here. s228 00:15:08.640 --> 00:15:09.720 Let's see. s229 00:15:09.720 --> 00:15:10.120 All right. s230 00:15:10.120 --> 00:15:13.080 So, you know, it knows it knows it's not a unicorn. s231 00:15:13.080 --> 00:15:14.720 It didn't know it was an aurochs. s232 00:15:14.720 --> 00:15:17.640 This one, it wasn't sure. s233 00:15:17.640 --> 00:15:19.640 It thinks they're alligators. s234 00:15:19.640 --> 00:15:23.840 So, it you know, it's having a harder time looking at the it thinks that they're alligators. s235 00:15:23.840 --> 00:15:30.560 And then for the the plants, it pulled out a plant that's $2.30, which is way more expensive than the cheapest plant. s236 00:15:30.560 --> 00:15:32.640 It was supposed to find one for 58 cents. s237 00:15:32.640 --> 00:15:36.400 So, the Mistral isn't able to answer these kind of images. s238 00:15:36.400 --> 00:15:46.400 Now, here's the thing, when it comes to multimodal understanding, you can't really do much prompt engineering cuz either like the like the the model either understands an image or it doesn't. s239 00:15:46.400 --> 00:15:54.880 Like you might be able to do some prompt engineering if you know you're in a very particular domain and there's some hints that you can give to it about how to read a particular kind of graph. s240 00:15:54.880 --> 00:16:03.720 Um but really with image input, it's about figuring out, you know, for the kind of images that you're going to send to the LLM, does the LLM understand those images? s241 00:16:03.720 --> 00:16:08.280 Is it going to, you know, is it going to work for that scenario, right? s242 00:16:08.280 --> 00:16:13.960 Um because it may be that, you know, maybe Mistral is really good at particular kinds of image tasks, but just not this, right? s243 00:16:13.960 --> 00:16:19.200 So, it's all figuring out what is the task you want to do and is the model going to do a good job at it? s244 00:16:19.200 --> 00:16:24.240 Now, what's interesting with the multimodal, too, is that a lot of these also depend on the language, right? s245 00:16:24.240 --> 00:16:25.560 Here I'm using English. s246 00:16:25.560 --> 00:16:44.080 Remember, all these models are super English-centric, but if you switch to a different language, like if I ask in Spanish if they're caimanes or cocodrilos, it actually thinks uh it says it gives the wrong answer because it's getting so confused in its head between these different like words and how they map across English. s247 00:16:44.080 --> 00:16:54.800 Like it just becomes too hard of a task both understand the difference between alligators and crocodiles and also remember that caiman means alligator in Spanish and is not the same thing as the caiman that we have in like Florida. s248 00:16:54.800 --> 00:17:00.240 Anyway, it's like it's fascinating to see that once we switch to Spanish, it fails at the task. s249 00:17:00.240 --> 00:17:08.160 Um so, that's another thing to keep in mind, especially we're in California, a lot of times when we develop applications here, they are bilingual cuz a lot of, you know, 50% Spanish-speaking here. s250 00:17:08.160 --> 00:17:13.000 Um so, you know, your your mileage may vary as you change languages. s251 00:17:13.000 --> 00:17:17.959 Okay, so now let's go on to function calling, also known as tool calling. s252 00:17:17.959 --> 00:17:22.240 So, tool calling is in fact the basis of like all agents. s253 00:17:22.240 --> 00:17:24.800 All agents are based off of tool calling. s254 00:17:24.800 --> 00:17:31.000 And this is a a capability that the LLMs are trained with specifically during the training process. s255 00:17:31.000 --> 00:17:41.960 So, they are trained to be able to look at a definition of a function or multiple function definitions and look at the user input and say, "Okay, based off that user input, I want to call this function with these arguments s256 00:17:41.960 --> 00:17:45.640 and and then, you know, just return that." s257 00:17:45.640 --> 00:17:50.360 So, in this example here, I'm defining a function using JSON schema. s258 00:17:50.360 --> 00:17:52.120 This is what the LLMs understand. s259 00:17:52.120 --> 00:17:53.800 So, I say, "Okay, here's my function. s260 00:17:53.800 --> 00:17:56.160 It has the name create calendar event. s261 00:17:56.160 --> 00:17:59.000 It has this description and these are the parameters it takes. s262 00:17:59.000 --> 00:18:01.320 It takes a title with this that's a string. s263 00:18:01.320 --> 00:18:03.400 Here's a description, right?" s264 00:18:03.400 --> 00:18:07.040 And all of this is all going to get sent to the LLM. s265 00:18:07.040 --> 00:18:11.680 So, we send this full JSON schema to the LLM and say, "Hey, here's the function. s266 00:18:11.680 --> 00:18:16.040 You got to decide whether you're going to call this and how you're going to call it, right?" s267 00:18:16.040 --> 00:18:20.080 So, this is in fact, even though this this is like an object, this is prompt engineering here. s268 00:18:20.080 --> 00:18:24.960 Everything here is actually like me deciding what is going to be the best for the LLM, right? s269 00:18:24.960 --> 00:18:26.080 What type should I use? s270 00:18:26.080 --> 00:18:27.800 What description should I add here? s271 00:18:27.800 --> 00:18:30.480 What should be the names of these parameters? s272 00:18:30.480 --> 00:18:36.440 Cuz the LLM's going to decide based on all that information how it's going to call this function. s273 00:18:36.440 --> 00:18:37.920 Uh so, then we have a user message. s274 00:18:37.920 --> 00:18:40.560 So, it's like So, the user's like, "Oh, can you throw something on my calendar? s275 00:18:40.560 --> 00:18:42.000 It's a platform team sync. s276 00:18:42.000 --> 00:18:44.960 Me, Sarah from Ange, Marcus, the new PM Priya. s277 00:18:44.960 --> 00:18:48.280 Tomorrow at 1:30 p.m. It's virtual on Microsoft Teams, okay?" s278 00:18:48.280 --> 00:18:52.160 So, then our then we tell the LLM like, "Hey, you're a calendar assistant. s279 00:18:52.160 --> 00:18:53.680 Today is Monday, June 29th. s280 00:18:53.680 --> 00:18:56.400 Use the available tools to process the user's request." s281 00:18:56.400 --> 00:18:57.840 And we send it the tools. s282 00:18:57.840 --> 00:19:01.720 And so, now the LLM is looking at the user request, looking at the tools. s283 00:19:01.720 --> 00:19:04.000 And in this case, it just has that create calendar event. s284 00:19:04.000 --> 00:19:08.760 And it's going to decide, "Hey, is there a tool here I can use to answer the user?" s285 00:19:08.760 --> 00:19:13.200 Uh and then we're just going to go through and see what tool calls it decides to make. s286 00:19:13.200 --> 00:19:14.360 Uh so there we go. s287 00:19:14.360 --> 00:19:18.680 So in and this is like really low-level tool calling here. s288 00:19:18.680 --> 00:19:25.840 Uh once we build an agent, we're going to abstract on top of this, but I wanted to start with the most low-level tool calling we can do. s289 00:19:25.840 --> 00:19:32.280 And the LM just responds back with here's the name of the function and here's the arguments that I want you to send. s290 00:19:32.280 --> 00:19:34.080 The LM itself doesn't call the function. s291 00:19:34.080 --> 00:19:35.360 That's up to us to do. s292 00:19:35.360 --> 00:19:41.640 All right, so what we can see is that uh I put my I put the expected on the left just so you can see and we can see the actual. s293 00:19:41.640 --> 00:19:46.480 So GPT 55 did add me as an attendee, which I didn't really want. s294 00:19:46.480 --> 00:19:47.880 Um but it did pretty good otherwise. s295 00:19:47.880 --> 00:19:54.040 So let's try uh Deep Seek and see how Deep Seek does with this one. s296 00:19:54.040 --> 00:19:58.040 Cuz what we want to see like when we're doing tool calling, we want to see is it calling the right tool? s297 00:19:58.040 --> 00:19:58.920 So it is in this case. s298 00:19:58.920 --> 00:20:01.040 It is calling the create calendar event tool. s299 00:20:01.040 --> 00:20:04.040 And is it passing in the arguments that we expect, right? s300 00:20:04.040 --> 00:20:06.000 We have certain arguments we expect. s301 00:20:06.000 --> 00:20:09.480 In this case, Deep Seek put you as an attendee. s302 00:20:09.480 --> 00:20:14.440 And then for the location, it said virtual {parentheses} Microsoft Teams, which is not what I asked for, right? s303 00:20:14.440 --> 00:20:19.040 So it's not adhering to my argument um constraints as much. s304 00:20:19.040 --> 00:20:22.400 Let's try Mistral as well. s305 00:20:22.520 --> 00:20:23.600 And see how it does. s306 00:20:23.600 --> 00:20:28.400 So once I see this, I can start to realize like, "Okay, there's some ambiguity, right?" s307 00:20:28.400 --> 00:20:31.640 And you know, if it if it called it with the wrong arguments, that means there's ambiguity. s308 00:20:31.640 --> 00:20:41.600 I need to go back into my descriptions and say, you know, uh you know, don't you know, don't include me or you, just named attendees. s309 00:20:41.600 --> 00:20:46.080 Um you know, you could say first and last if provided, right? s310 00:20:46.080 --> 00:20:58.080 So you start to see, you know, what what you haven't been specific enough about and start to clarify that ambiguity and then, you know, evaluate all the different um inputs and see how it's doing. s311 00:20:58.080 --> 00:20:58.720 All right. s312 00:20:58.720 --> 00:21:00.920 So that was a basic tool calling. s313 00:21:00.920 --> 00:21:03.400 The next thing we want to do, we're building up to agents here. s314 00:21:03.400 --> 00:21:06.640 The next thing is to do tool calling in a loop, right? s315 00:21:06.640 --> 00:21:12.000 So when we have an agent, it's calling tools in a loop until it reaches a goal. s316 00:21:12.000 --> 00:21:15.920 Um so, we're going to demonstrate using this just a function calculate. s317 00:21:15.920 --> 00:21:21.040 So, this calculate takes a mathematical expression and returns back the output, right? s318 00:21:21.040 --> 00:21:26.960 And we have, you know, basic support for uh evaluating these uh mathematical expressions. s319 00:21:26.960 --> 00:21:30.480 So, we're giving it basically a word problem where it's like, "Oh, this store is having a sale. s320 00:21:30.480 --> 00:21:34.560 If I buy three shirts and pay, you know, this much tax, what's my total?" s321 00:21:34.560 --> 00:21:36.520 And this can be done several different ways. s322 00:21:36.520 --> 00:21:44.320 It could be done with multiple tool calls to that calculator call with like one little expression at a time, like, "Do this, and then do this, and then do this, and then this." s323 00:21:44.320 --> 00:21:51.280 Or, if the LM is very clever, it could batch all that into one long expression and do a single tool call. s324 00:21:51.280 --> 00:21:53.800 So, here we're testing efficiency, right? s325 00:21:53.800 --> 00:22:01.560 Where, you know, there's more like a lot of times once you have tool calling in a loop, the differences between the models is their efficiency. s326 00:22:01.560 --> 00:22:04.000 Are they able to get to the goal faster? s327 00:22:04.000 --> 00:22:08.920 Can they accomplish the same goal with less tool calls, right? s328 00:22:08.920 --> 00:22:11.400 So, that's what we're going to say. s329 00:22:11.400 --> 00:22:14.760 Uh the only system prompt is do not do math in your head. s330 00:22:14.760 --> 00:22:15.560 Okay. s331 00:22:15.560 --> 00:22:16.215 [laughter] s332 00:22:16.215 --> 00:22:16.640 [gasps] s333 00:22:16.640 --> 00:22:18.280 So, let's see. s334 00:22:18.280 --> 00:22:23.680 Uh that one's really for GB55 cuz it's so proud of its math abilities that it's always trying to do it in its head. s335 00:22:23.680 --> 00:22:24.800 Like, "Come on." s336 00:22:24.800 --> 00:22:25.080 All right. s337 00:22:25.080 --> 00:22:30.800 So, GB55 indeed does the entire calculation in one expression. s338 00:22:30.800 --> 00:22:34.760 So, it only required one tool call to get to the final result. s339 00:22:34.760 --> 00:22:35.360 So, that's good. s340 00:22:35.360 --> 00:22:35.840 That's good. s341 00:22:35.840 --> 00:22:36.920 That's, you know, fast. s342 00:22:36.920 --> 00:22:37.920 It saves us time. s343 00:22:37.920 --> 00:22:42.720 So, now let's try with um DeepSeek. s344 00:22:42.720 --> 00:22:50.240 The only reason I skip Kimmy all the time is cuz it's actually a very good model, and so it does pretty much uh on par with uh GB55 uh for much of these. s345 00:22:50.240 --> 00:22:50.800 Okay. s346 00:22:50.800 --> 00:23:00.440 So, now we can see with DeepSeek, it is going through and it's doing and and and and it's doing like so many how many it did like four. s347 00:23:00.440 --> 00:23:02.480 It did four tool calls, right? s348 00:23:02.480 --> 00:23:06.640 So, it still got to the correct answer, but it took four tool calls. s349 00:23:06.640 --> 00:23:11.080 And every one of those tool calls is another network request, right? s350 00:23:11.080 --> 00:23:22.160 And so, you know, when we're evaluating two two models, we want to both look at the final answer, but also how how long did it take to get to that final answer, right? s351 00:23:22.160 --> 00:23:24.560 Um actually, Kimmy did do something really interesting the other day. s352 00:23:24.560 --> 00:23:27.280 Let's see if it does it again with this one. s353 00:23:27.280 --> 00:23:36.360 Cuz Kimmy most of the time works quite well, but Kimmy does sometimes get into weird loops um when it's kind of like double-checking itself. s354 00:23:36.360 --> 00:23:37.840 Uh okay, this time it did well. s355 00:23:37.840 --> 00:23:43.840 Yeah, the other day when I did Kimmy, it like got the right answer on the first try and then decided to check its work with the like four other calls. s356 00:23:43.840 --> 00:23:44.760 I was like, "No, you're good. s357 00:23:44.760 --> 00:23:45.296 You're good." s358 00:23:45.296 --> 00:23:45.951 [laughter] s359 00:23:45.951 --> 00:23:46.200 [gasps] s360 00:23:46.200 --> 00:23:48.720 Um so that's what that's what we're looking at here, okay? s361 00:23:48.720 --> 00:23:54.040 Is how many how many calls does it actually need to get to the end. s362 00:23:54.640 --> 00:24:00.720 All right, so next um we can go to uh agent loops. s363 00:24:00.720 --> 00:24:03.360 Okay, I'm actually going quite fast through this now, actually. s364 00:24:03.360 --> 00:24:04.291 Uh okay, good. s365 00:24:04.291 --> 00:24:05.640 [laughter] s366 00:24:05.640 --> 00:24:08.080 Um all right. s367 00:24:08.080 --> 00:24:09.800 So now we've got the basis, right? s368 00:24:09.800 --> 00:24:11.880 We've seen tool calling, tool calling in a loop. s369 00:24:11.880 --> 00:24:14.200 So now we can actually build like a proper agent. s370 00:24:14.200 --> 00:24:16.960 And for this, we'd usually use an agent framework. s371 00:24:16.960 --> 00:24:22.040 Uh I've gone ahead and put three different agent frameworks in here uh so that you can pick your favorite. s372 00:24:22.040 --> 00:24:26.520 So we have LangChain, Pydantic AI, and Agent Framework. s373 00:24:26.520 --> 00:24:32.840 Uh I'm sure there's other agent frameworks as well that I missed, but hopefully you can find one that you like there. s374 00:24:32.840 --> 00:24:34.960 Uh so these are all, you know, the same idea. s375 00:24:34.960 --> 00:24:37.200 All right, which one do we want to show? s376 00:24:37.200 --> 00:24:39.000 Someone vote. s377 00:24:39.000 --> 00:24:39.600 LangChain. s378 00:24:39.600 --> 00:24:45.920 All right, LangChain is still very popular and uh hopefully there's some LangChain folks here today at the conference. s379 00:24:45.920 --> 00:24:54.720 All right, so with LangChain, we are using LangChain V1, um which is a a nice uh newest TK version of theirs if you haven't tried it in the last year. s380 00:24:54.720 --> 00:24:57.600 And so we connect to the Foundry models from LangChain. s381 00:24:57.600 --> 00:25:02.600 and what we have here is a travel planning agent, which is a classic example. s382 00:25:02.600 --> 00:25:03.800 So, it has tools. s383 00:25:03.800 --> 00:25:13.320 So, we have a tool for search flights, search hotels, and check budget to check if a certain flight and hotel fits within the budget, uh and then search activities within a city. s384 00:25:13.320 --> 00:25:17.720 Uh then I've got some middleware, which is just for me to see what tools are happening. s385 00:25:17.720 --> 00:25:19.560 And then we have the actual agent call. s386 00:25:19.560 --> 00:25:24.880 So, the agent has a system prompt that says you're going to help users find flights and hotels. s387 00:25:24.880 --> 00:25:33.640 Uh you're you know, it has some rules here like search for both flights and hotels, prefer direct flights, yada yada, always check check budget, and then ask for activities s388 00:25:33.640 --> 00:25:36.280 at the end based off the remaining budget. s389 00:25:36.280 --> 00:25:37.480 Then we have the user message. s390 00:25:37.480 --> 00:25:40.240 I need to fly from SF to New York City next month for 3 nights. s391 00:25:40.240 --> 00:25:45.800 My total budget is 6 600, and I want direct flights and an activity. s392 00:25:45.800 --> 00:25:46.560 All right. s393 00:25:46.560 --> 00:25:50.240 So, we have all that, and then we are running the agent. s394 00:25:50.240 --> 00:25:54.600 There's some additional code here that's just logging out the tool calls so that we can see it. s395 00:25:54.600 --> 00:25:58.720 So, let's go ahead and try that out. s396 00:26:00.880 --> 00:26:02.880 Okay. s397 00:26:03.000 --> 00:26:06.120 And so, this is going to log out the tool calls that happened. s398 00:26:06.120 --> 00:26:08.600 We're starting off with GPT 55 here. s399 00:26:08.600 --> 00:26:10.920 Now, what's interesting here, you'll see turn one. s400 00:26:10.920 --> 00:26:12.560 Let me make this bigger. s401 00:26:12.560 --> 00:26:15.320 Turn one, it says is parallel. s402 00:26:15.320 --> 00:26:29.680 So, one thing is you can do with LLMs is uh is like many LLMs support parallel tool calling where if you give them a bunch of tool definitions, they can suggest a multiple tools that can be called at the same time in the same turn. s403 00:26:29.680 --> 00:26:39.040 And this is really good from a performance perspective because on your server, you can actually file like you can send off requests in parallel to like two different databases, right? s404 00:26:39.040 --> 00:26:46.720 Uh so, as long as they're independent of each other, this is a great thing that LLMs can do is say, "Hey, these two these two things can be done in parallel. s405 00:26:46.720 --> 00:26:47.840 Why don't you do them?" s406 00:26:47.840 --> 00:26:56.240 So, uh GPT 55 on the first turn and "Hey, search for hotels in New York City and search for flights in uh from SF to New York City." s407 00:26:56.240 --> 00:26:59.280 cuz it's decided that there wasn't a dependency between them. s408 00:26:59.280 --> 00:27:09.240 And then, once that came back, once it had all the options, it did a check budget um based off of the flight cost and the hotel cost that it got back from that call. s409 00:27:09.240 --> 00:27:15.560 And then, it saw what the remaining budget was and then it did a call for search activities with that remaining budget. s410 00:27:15.560 --> 00:27:20.640 So, this is probably the most efficient agent that we can have for this scenario. s411 00:27:20.640 --> 00:27:28.240 So, let's see if we get similar um results when we change models here. s412 00:27:28.240 --> 00:27:30.640 Do do do. s413 00:27:32.200 --> 00:27:37.440 And of course, here you can even change and compare across both models and frameworks, right? s414 00:27:37.440 --> 00:27:40.400 Because every framework has a slightly different tool calling loop. s415 00:27:40.400 --> 00:27:41.800 We also call it the harness, right? s416 00:27:41.800 --> 00:27:49.680 So, a lot of times that you're you want to compare both models and harnesses and and you'll see there's interesting differences across them. s417 00:27:49.680 --> 00:27:49.880 All right. s418 00:27:49.880 --> 00:27:54.680 So, this time we're with Kimmy and Kimmy is not paralyzing uh as well. s419 00:27:54.680 --> 00:27:59.240 I don't know if that's because Kimmy doesn't support parallel tool calling or it's just deciding not to in this case. s420 00:27:59.240 --> 00:28:03.000 So, this time it did the same Actually, it did even more calls. s421 00:28:03.000 --> 00:28:05.640 So, it did search flight and search hotels. s422 00:28:05.640 --> 00:28:09.440 It did over two turns, so that's already less efficient cuz it didn't paralyze. s423 00:28:09.440 --> 00:28:14.200 Then, it decided to do two calls to check budget cuz I think it was checking like one more option. s424 00:28:14.200 --> 00:28:15.760 And then, it did the final thing. s425 00:28:15.760 --> 00:28:23.200 So, this one took two extra tool calls and it took longer because it didn't paralyze that first uh set of turns. s426 00:28:23.200 --> 00:28:27.560 All right, let's see how we do with DeepSeek. s427 00:28:31.840 --> 00:28:36.320 And you can start to see why it's hard cuz everybody asked me like, "How do you evaluate agents?" s428 00:28:36.320 --> 00:28:39.520 Agents are the hardest to evaluate because there's so much going on. s429 00:28:39.520 --> 00:28:42.040 There's so many different things that are happening along the way. s430 00:28:42.040 --> 00:28:44.560 It's like, are is it calling the right tools? s431 00:28:44.560 --> 00:28:45.800 Is it calling them efficiently? s432 00:28:45.800 --> 00:28:47.960 Is it calling things in parallel? s433 00:28:47.960 --> 00:28:54.920 Um in this case, you know, this one uh didn't do any parallelizing, um but at least it only did four tool calls. s434 00:28:54.920 --> 00:28:58.000 So, you know, reasonably efficient. s435 00:28:58.000 --> 00:29:04.480 And then Mistral Mistral often will see Mistral often forgets the very last step. s436 00:29:04.480 --> 00:29:05.680 And so, we'll see. s437 00:29:05.680 --> 00:29:07.920 Um you know, that's the other problem with the agents. s438 00:29:07.920 --> 00:29:14.080 Sometimes agents, when they get to the like once their context window fills up, they might forget some earlier instructions. s439 00:29:14.080 --> 00:29:19.440 They may not be as thorough um with, you know, paying attention to what the original steps were. s440 00:29:19.440 --> 00:29:29.560 And sometimes you have to like kind of prompt them along the way to to remember like, "Oh, wait, you're supposed to you know, finish up with this final step here, right?" s441 00:29:29.800 --> 00:29:31.880 Okay, so it did it didn't do parallel. s442 00:29:31.880 --> 00:29:35.560 So, what we're going to see is GPT-5.5 is very good at parallel tool calling. s443 00:29:35.560 --> 00:29:38.040 Um the other ones not not aren't doing it. s444 00:29:38.040 --> 00:29:44.520 Okay, so Mistral decided to only search flights and search hotels and then not do anything else. s445 00:29:44.520 --> 00:29:44.895 Um so, s446 00:29:44.895 --> 00:29:45.040 [laughter] s447 00:29:45.040 --> 00:29:47.200 it did it did it kind of gave up there. s448 00:29:47.200 --> 00:29:54.800 Um so, that's an example where, okay, we'd have to do some additional steering there to be like, "Hey, are you sure that you have everything, right?" s449 00:29:54.800 --> 00:29:57.040 Um you could use something like the route loop. s450 00:29:57.040 --> 00:29:58.280 Uh Jeff Huntley is here today. s451 00:29:58.280 --> 00:29:59.520 He made the route loop. s452 00:29:59.520 --> 00:30:09.000 And you know, you can you can force uh you know, you can force agents to be more aggressive uh instead of just using the default loop. s453 00:30:09.200 --> 00:30:09.960 All right. s454 00:30:09.960 --> 00:30:16.360 Okay, so that uh that was an agent example of saying we also have agent examples in the other frameworks. s455 00:30:16.360 --> 00:30:19.800 Um but what you see is the code is really similar across them. s456 00:30:19.800 --> 00:30:25.680 Uh I tend to think most agent frameworks are, you know, pre- more similar than they are different at this point. s457 00:30:25.680 --> 00:30:29.560 Um but, you know, you're welcome to check them out. s458 00:30:29.560 --> 00:30:30.520 All right. s459 00:30:30.520 --> 00:30:33.960 So, now we can talk about evaluations, right? s460 00:30:33.960 --> 00:30:38.480 So, we've been doing everything just, you know, manually running them and observing the results. s461 00:30:38.480 --> 00:30:50.720 That's nice to get a feel for the kind of differences that you can see, but when you're actually going to be, you know, doing something for production, you do want to actually have bulk automated evaluations s462 00:30:50.720 --> 00:30:54.880 that you can reliably run and compare results, right? s463 00:30:54.880 --> 00:31:03.880 So, the first step that we can do is just uh doing evals for um without, you know, with just using string comparison, right? s464 00:31:03.880 --> 00:31:16.680 Um so, the example I have here is with that create calendar event tool call, cuz what we can do is like we're actually getting back the arguments, we can actually just do a straight string comparison say, "Hey, did it give back virtual, right? s465 00:31:16.680 --> 00:31:19.080 We wanted virtual for this, did it give back virtual?" s466 00:31:19.080 --> 00:31:26.960 So, when you're testing a single tool call, you can just check to see, did it check did it call call the right tool and did it use the right arguments? s467 00:31:26.960 --> 00:31:30.040 So, that's what we're going to do in this one. s468 00:31:30.040 --> 00:31:32.080 Um and we set up a bunch of test cases, right? s469 00:31:32.080 --> 00:31:35.720 Cuz when you're running evals, you want to have lots and lots of test cases. s470 00:31:35.720 --> 00:31:43.760 You also want it to be easy to add more test cases, cuz once your app is in production, you're going to see how users really use it and you're like, "Oh, okay. s471 00:31:43.760 --> 00:31:44.720 Thank you. s472 00:31:44.720 --> 00:31:45.440 Everything's broken." s473 00:31:45.440 --> 00:31:53.920 No, and then you're going to go and add those new cases of user input to uh your, you know, your evaluations to make sure that you don't drift, right? s474 00:31:53.920 --> 00:31:58.480 This is called like model drift or your your your drift away from it. s475 00:31:58.480 --> 00:32:01.880 So, you don't want your evals to drift away from your user inputs. s476 00:32:01.880 --> 00:32:06.720 So, once they're in production, you want a good system for getting new user inputs into your eval cases. s477 00:32:06.720 --> 00:32:11.720 So, here what I do is I've got like lots of different um you know, uh cases here. s478 00:32:11.720 --> 00:32:20.400 Like, here's a user message, here's the checks, and these checks are just doing really string comparison, uh some list comparison, and we check different things, right? s479 00:32:20.400 --> 00:32:22.240 We check the date math. s480 00:32:22.240 --> 00:32:25.560 Uh I always check LLMs on date math, cuz usually they're kind of bad at it. s481 00:32:25.560 --> 00:32:29.800 So, another tip, if you're doing dates, give them just give them a tool to do date math, okay? s482 00:32:29.800 --> 00:32:32.040 Cuz it's too hard to do date math in your head. s483 00:32:32.040 --> 00:32:33.280 Uh especially on leap years. s484 00:32:33.280 --> 00:32:34.542 I was evaluating s485 00:32:34.542 --> 00:32:35.000 [laughter] s486 00:32:35.000 --> 00:32:37.480 And LLM on a leap year, that was hard. s487 00:32:37.480 --> 00:32:40.800 Uh then we had um okay, ambiguous durations. s488 00:32:40.800 --> 00:32:42.360 I did one in Spanish, right? s489 00:32:42.360 --> 00:32:46.332 It's nice to test across different languages, whatever languages you're going to support. s490 00:32:46.332 --> 00:32:46.440 [snorts] s491 00:32:46.440 --> 00:32:49.400 Uh this one I guess is written written real slangy. s492 00:32:49.400 --> 00:32:51.120 Yo, can you book the big conference room? s493 00:32:51.120 --> 00:32:51.840 Okay. s494 00:32:51.840 --> 00:32:55.040 All right, so we have various edge cases there, right? s495 00:32:55.040 --> 00:32:58.240 And then we're let me to run these evals. s496 00:33:02.640 --> 00:33:09.680 Okay, and these evals in this case I'm going to run it across each of the models and the use cases. s497 00:33:09.680 --> 00:33:17.680 A lot of times in reality when I'm running evals, I'm just running it on a single model and then I'm comparing to like a previous run where maybe I'd done a different model. s498 00:33:17.680 --> 00:33:30.560 Uh but you know, for today uh since we are actively comparing across models, I'm going to run it across all the models so we can see um it's for each of them it's doing the tool call, checking the args that come back, s499 00:33:30.560 --> 00:33:33.640 and deciding if it's a pass or a fail. s500 00:33:33.640 --> 00:33:42.280 Uh now here's the thing about evals is that you don't want to just have these like just be part of like your pie test, like your test suite because these are slow, these are expensive. s501 00:33:42.280 --> 00:33:42.960 Yeah, question? s502 00:33:42.960 --> 00:33:51.000 Um I see you just roll, you know, checking strings against Uh do you know if any frameworks do this a bit more elegantly s503 00:33:52.120 --> 00:33:54.000 You don't like my string checks. s504 00:33:54.000 --> 00:34:00.680 Um yeah, so framework-wise for evals a lot of so first of all, most observability providers have evals built in, right? s505 00:34:00.680 --> 00:34:06.800 So if you're doing LangFuse, LangSmith, LangChain, um Arize, like they all have their own framework built in. s506 00:34:06.800 --> 00:34:13.080 For Foundry, we have um we have two different frameworks uh for it and I actually I do have the examples in the repo. s507 00:34:13.080 --> 00:34:20.560 Um um those ones there's Azure AI evaluation SDK, Open uh AI.evals um from the OpenAI package. s508 00:34:20.560 --> 00:34:29.080 So there's a bunch of frameworks, there's also DeepEval and Prompt Prompt foo in JavaScript if you're a JavaScript person. s509 00:34:29.080 --> 00:34:30.080 And which one? s510 00:34:30.080 --> 00:34:30.760 Vali. s511 00:34:30.760 --> 00:34:31.399 Vali? s512 00:34:31.399 --> 00:34:32.120 Something like that. s513 00:34:32.120 --> 00:34:33.600 Mark Vali, okay. s514 00:34:33.600 --> 00:34:34.800 V A L L Y s515 00:34:34.800 --> 00:34:36.040 V A L L Y, okay. s516 00:34:36.040 --> 00:34:37.280 I mean you use that one. s517 00:34:37.280 --> 00:34:41.440 Um so there's a number of eval frameworks um I'll I'm going to use a fancier one soon. s518 00:34:41.440 --> 00:34:49.560 But for string check, but here's the thing like like for string check, you don't really have to do a fancy one um because you're like s519 00:34:49.560 --> 00:34:51.560 If you're looking for it to do a specific s520 00:34:51.560 --> 00:34:52.480 Yeah. s521 00:34:52.480 --> 00:34:54.600 Do the parallelization you mentioned for s522 00:34:54.600 --> 00:34:55.120 Yeah, yeah. s523 00:34:55.120 --> 00:34:57.920 So for tool calls, yeah, that's where you're going to get a bit fancier, yeah. s524 00:34:57.920 --> 00:35:03.320 Yeah, so I wanted to start with the basic, but yeah, most of the time we can't get away with just the basic, right? s525 00:35:03.320 --> 00:35:05.320 Um oh, they did actually really well this time. s526 00:35:05.320 --> 00:35:06.840 Wow, nice. s527 00:35:06.840 --> 00:35:11.560 Um so usually we're going to have to move to fancier evaluation. s528 00:35:11.560 --> 00:35:15.520 Um so that's when we start to bring in LLM as a judge. s529 00:35:15.520 --> 00:35:25.480 Uh so this is cuz a lot of times, you know, the the output is a lot fuzzier and we need to use an LLM to help us reason about that fuzziness. s530 00:35:25.480 --> 00:35:34.920 And there's many frameworks that do wrap this um For this example, I just decided to write it entirely in the in the code so you could see. s531 00:35:34.920 --> 00:35:44.440 So the idea when you use an LLM as a judge is that you need to give it, you know, put the LLM in judge mode and say, "Okay, you are an evaluator. s532 00:35:44.440 --> 00:35:49.280 This is what you're evaluating and you need to either say does it pass or does it fail?" s533 00:35:49.280 --> 00:35:49.480 Right? s534 00:35:49.480 --> 00:35:52.680 We're trying to force the LLM to say does this pass or fail? s535 00:35:52.680 --> 00:35:54.720 Um and we also want it to give it reason, right? s536 00:35:54.720 --> 00:35:56.480 It's really really helpful to have it give it reason. s537 00:35:56.480 --> 00:35:59.160 So we want pass, fail, and a reason. s538 00:35:59.160 --> 00:36:02.200 And then we give it the query, we give it the response. s539 00:36:02.200 --> 00:36:06.200 And for grounding this, we also want to give it the context cuz that helps it actually reason about it. s540 00:36:06.200 --> 00:36:09.520 So sometimes in order to really evaluate, you do need access to more information. s541 00:36:09.520 --> 00:36:16.080 So like tool calls, if you're trying to evaluate whether it was an efficient user of tool calls, you can't you have to give it everything. s542 00:36:16.080 --> 00:36:17.880 You have to give it the tool call definitions. s543 00:36:17.880 --> 00:36:20.560 You have to give it the traces of the tool calls it made, right? s544 00:36:20.560 --> 00:36:30.520 So to build really good evaluators, a lot of times you're you're not just giving it the input output, you're giving it every everything that happened along the way, everything that was shown. s545 00:36:30.520 --> 00:36:36.320 Uh so here we, you know, we tell the LLM, "Hey, we're going to use the judge model of GPT-5. s546 00:36:36.320 --> 00:36:38.200 It's a pretty good judge model." s547 00:36:38.200 --> 00:36:42.600 And we're going to ask for it to output some JSON that we can look at. s548 00:36:42.600 --> 00:36:48.000 So let me go ahead and run uh this LLM as a judge. s549 00:36:48.000 --> 00:36:57.800 Now, here's the thing, um this is going to be even slower because now each time we're running an eval, we're both running our own code, which is using a model, s550 00:36:57.800 --> 00:37:01.560 and we're then using another LLM to judge that code, right? s551 00:37:01.560 --> 00:37:05.360 So every single eval case is actually two LLM calls. s552 00:37:05.360 --> 00:37:08.600 And then many times you actually run multiple LLMs as a judge. s553 00:37:08.600 --> 00:37:11.640 Like you'll have like five different evaluators that you're running. s554 00:37:11.640 --> 00:37:14.880 Each of those is going to be a separate LLM call, right? s555 00:37:14.880 --> 00:37:20.360 Um so that that's what is tricky about evals is that you can't just run them all the time. s556 00:37:20.360 --> 00:37:27.120 You can't necessarily run them on every single CI uh and every single commit because they are fairly slow to run. s557 00:37:27.120 --> 00:37:35.400 Cuz normally in production, like for rag, when I'm evaluating rag, you know, you usually want like 200 test cases at least, maybe 800. s558 00:37:35.400 --> 00:37:38.160 And that's that's like a lot of calls to an LLM, right? s559 00:37:38.160 --> 00:37:41.160 So that that takes some amount of time. s560 00:37:41.160 --> 00:37:44.680 Um so you do want to think about how often you're going to be, you know, running evals. s561 00:37:44.680 --> 00:37:51.360 So you can see that it's going through its text it it's got a bunch of different test cases, different questions and answers. s562 00:37:51.360 --> 00:37:53.280 And it's it's looking at the results. s563 00:37:53.280 --> 00:37:58.560 Now, in this case, what I did was that I like pre-ran the um the questions and answers. s564 00:37:58.560 --> 00:37:59.760 This is a common way of doing it, too. s565 00:37:59.760 --> 00:38:11.040 Like get, you know, get all your get your context for the evaluator beforehand um because then you can change your evaluator independently from your app, right? s566 00:38:11.040 --> 00:38:12.080 Um and that's more efficient. s567 00:38:12.080 --> 00:38:15.800 Cuz a lot of times what you realize is that your evaluator is incorrect, right? s568 00:38:15.800 --> 00:38:18.800 Like that you haven't written good evaluator. s569 00:38:18.800 --> 00:38:26.040 Uh so that's what you have to figure out when you're doing LLM as judge, you have to figure out like wait, is the problem my app or is the problem my judge, s570 00:38:26.040 --> 00:38:26.360 right? s571 00:38:26.360 --> 00:38:29.600 And uh you also want to calibrate your judges with human input as well. s572 00:38:29.600 --> 00:38:39.800 So, it definitely helps me working on a domain you know, or work with a domain expert that can help you label your data and say and you know, do they agree with your LLM as a judge? s573 00:38:39.800 --> 00:38:42.640 Was it actually a pass or a fail? s574 00:38:42.640 --> 00:38:52.840 Uh so here we get the results back and we can see you know, the model um and their scores um most of them got like two out of three. s575 00:38:52.840 --> 00:38:54.600 Mr. Owl got one out of three. s576 00:38:54.600 --> 00:39:00.400 You know, so they're they're they have some work to do on being more grounded here. s577 00:39:00.880 --> 00:39:03.840 So that is the generator of using LLM as a judge. s578 00:39:03.840 --> 00:39:10.600 As you see, there's quite a few frameworks that uh make it, you know, that have built-in LLM as a judge that make it easier to do it. s579 00:39:10.600 --> 00:39:13.880 Um but I want to show that you you know, you can't you can't also do it yourself. s580 00:39:13.880 --> 00:39:18.160 Oh, also I should recommend if you're interested in evaluation, Hamel.dev. s581 00:39:18.160 --> 00:39:25.280 I don't know if he's at the conference this week, but he has the most amazing resources and he teaches courses all the time about evals. s582 00:39:25.280 --> 00:39:27.600 And also he's very good memes. s583 00:39:27.600 --> 00:39:28.920 I actually used these last night. s584 00:39:28.920 --> 00:39:35.720 So, if you're ever chatting with someone about evals and you need an eval meme, uh what was it what I was doing last night? s585 00:39:35.720 --> 00:39:38.332 Yeah, building an LLM as a judge it's it's hard. s586 00:39:38.332 --> 00:39:40.332 [laughter] s587 00:39:40.400 --> 00:39:43.520 Uh so very important resource here. s588 00:39:43.520 --> 00:39:44.680 How can we trust the LLM judge? s589 00:39:44.680 --> 00:39:47.600 You have to calibrate your LLM judge. s590 00:39:47.600 --> 00:39:49.400 Okay. s591 00:39:49.400 --> 00:39:49.800 All right. s592 00:39:49.800 --> 00:39:51.720 So, that was LLM as a judge. s593 00:39:51.720 --> 00:39:58.680 So now um but then they as you see, it is hard to build judge evaluators, right? s594 00:39:58.680 --> 00:40:06.920 Like you have to like follow Hamel on Twitter and take his course and learn all these best practices for how you write uh a good eval. s595 00:40:06.920 --> 00:40:10.840 Um uh so we have this new tool from Microsoft Research. s596 00:40:10.840 --> 00:40:12.920 It's very new, like the last week. s597 00:40:12.920 --> 00:40:14.520 Um it's called Assert. s598 00:40:14.520 --> 00:40:19.200 And Assert helps you write the judges based off of the spec. s599 00:40:19.200 --> 00:40:20.280 So, you write a spec. s600 00:40:20.280 --> 00:40:31.400 You say, "Hey, this is the agent I'm building and this is, you know, roughly what I want it to be doing, my idea about how it should work, and um what kind of users it should support." s601 00:40:31.400 --> 00:40:34.240 And then it will generate lots of test cases. s602 00:40:34.240 --> 00:40:45.840 It'll like flesh out your your spec into um into actual judging rubrics, and figure all of that out for you based off of, you know, all the the it's very based in in research, right? s603 00:40:45.840 --> 00:40:51.080 So, we've done that for the travel planner already in here. s604 00:40:51.080 --> 00:40:54.400 Um so, here's the um so, it's under this assert eval. s605 00:40:54.400 --> 00:40:55.560 Okay. s606 00:40:55.560 --> 00:41:00.040 So, what we can see is that we write a YAML file cuz we all wanted more YAML in our life. s607 00:41:00.040 --> 00:41:05.600 What I realized about YAML is that I might hate YAML, but it's also better than everything else for what I need it for. s608 00:41:05.600 --> 00:41:07.320 So, there we go. s609 00:41:07.320 --> 00:41:16.200 Um so, in this YAML, you know, we describe uh the context, and we describe, "Okay, we're going to use this judge model. s610 00:41:16.200 --> 00:41:20.080 Um these are the kind of dimensions we care about is budget, you know, budget pressure." s611 00:41:20.080 --> 00:41:26.320 So, we can we can give it more information if we know, like, "Oh, we care about ample budget, tight budget, invisible." s612 00:41:26.320 --> 00:41:28.760 Um you know, so, you can you can stratify it. s613 00:41:28.760 --> 00:41:30.200 This is called stratification. s614 00:41:30.200 --> 00:41:36.880 You can you can pre-stratify if you want, uh or you can just have it come up with the stratification. s615 00:41:36.880 --> 00:41:39.120 Uh and then we have more dimensions here. s616 00:41:39.120 --> 00:41:46.800 And there's this there's also a a skill an agent skill that helps you write this YAML file so that you don't have to write it yourself cuz no one wants to write it, right? s617 00:41:46.800 --> 00:41:49.120 Um and then we have the actual thing that's evaluating. s618 00:41:49.120 --> 00:41:51.840 This is the LangChain travel planner that we saw before. s619 00:41:51.840 --> 00:42:02.360 Uh so, this just makes sure that it's tracing everything cuz it's saying, "When you are doing evaluations, you do want to see every single tool call so that you can really properly do these evaluations." s620 00:42:02.400 --> 00:42:02.760 All right. s621 00:42:02.760 --> 00:42:06.080 So, then it it runs it all, and then we can see the results. s622 00:42:06.080 --> 00:42:08.160 It does take a, you know, a decent amount of time. s623 00:42:08.160 --> 00:42:12.120 So, we've already got the um evals in here. s624 00:42:12.120 --> 00:42:15.360 Uh so, I've got them running. s625 00:42:15.360 --> 00:42:18.840 Uh there's like a little like front end that you can use to view them. s626 00:42:18.840 --> 00:42:20.600 Okay. s627 00:42:20.600 --> 00:42:21.040 All right. s628 00:42:21.040 --> 00:42:27.680 So, here is the result of this um a search evaluation. s629 00:42:27.680 --> 00:42:33.120 Um so, basically it you know, it came up with the different behavior categories, right? s630 00:42:33.120 --> 00:42:40.560 So, making like budget in infeasibility, direct flight preference rule violation, required tool use, right? s631 00:42:40.560 --> 00:42:46.000 So, uh it comes up with what behaviors are you know, allowed but not allowed. s632 00:42:46.000 --> 00:42:47.640 And then it comes up with the test set. s633 00:42:47.640 --> 00:42:50.240 The test set is really is pretty fun. s634 00:42:50.240 --> 00:42:52.240 Um and here we go. s635 00:42:52.240 --> 00:42:53.720 Scenario 65. s636 00:42:53.720 --> 00:42:58.800 Red eye before the installer walked through the coach with six parent text one weekend to see Alma. s637 00:42:58.800 --> 00:43:01.880 I really like that these are obviously LLM generated titles, right? s638 00:43:01.880 --> 00:43:05.520 So, it came up with all of these coach with a clipboard. s639 00:43:05.520 --> 00:43:12.640 Um it can it comes up with all these scenarios based off that that YAML to try and simulate users, right? s640 00:43:12.640 --> 00:43:19.760 Cuz the whole thing it's really hard to bootstrap your evaluation before you go into production, but you want to have evals before you go into production, right? s641 00:43:19.760 --> 00:43:23.400 So, how do you figure out, you know, what your test set is going to be? s642 00:43:23.400 --> 00:43:30.160 And so, this assert framework helps a lot with making that test set and bootstrapping your evals. s643 00:43:30.160 --> 00:43:32.200 Uh so, it has prompts and scenarios. s644 00:43:32.200 --> 00:43:33.280 And then we have the results. s645 00:43:33.280 --> 00:43:33.800 Okay. s646 00:43:33.800 --> 00:43:42.560 So, for the results, it ran it um across Deep Sea, Kimmy, and GBD 5.5 and uh came up with the you know, with the numbers here. s647 00:43:42.560 --> 00:43:46.320 Um and in this case the the lower numbers are better. s648 00:43:46.320 --> 00:43:51.200 And so, we can see that uh GBD 5.5 did way better than the other ones. s649 00:43:51.200 --> 00:44:00.440 I was actually surprised cuz Kimmy, we've seen that Kimmy has generally been quite good today um and I was thinking it was kind of on par, but at least for this scenario, s650 00:44:00.440 --> 00:44:08.400 uh GBD 5.5 had the lowest number of violations, and so it is it's scoring the best here. s651 00:44:08.400 --> 00:44:10.680 Um so So yeah. s652 00:44:10.680 --> 00:44:12.440 So this is assert. s653 00:44:12.440 --> 00:44:16.920 I will say I'm I'm brand new to assert since it only came out like a week ago. s654 00:44:16.920 --> 00:44:29.480 Um so this is all I've I've used it so far, but the idea here is that it's helping you to build those evaluators to build that test set and to follow the best practices and bootstrap your evals so that you can start s655 00:44:29.480 --> 00:44:33.000 start off um you know, with with some really good evals. s656 00:44:33.000 --> 00:44:41.320 Cuz what we're seeing is that we keep telling people like you got to run evals, you got to run evals, and then no one's running evals or they're running like the default evals like coherence s657 00:44:41.320 --> 00:44:42.320 and fluency. s658 00:44:42.320 --> 00:44:43.400 Never run those. s659 00:44:43.400 --> 00:44:46.200 Like LLMs are great at being coherent. s660 00:44:46.200 --> 00:44:48.440 You never have to check if an LLM is coherent. s661 00:44:48.440 --> 00:44:51.320 If some if they're not, like something really bad has happened. s662 00:44:51.320 --> 00:44:59.800 Um I do sometimes use the groundness and relevance evaluators for my rag things, but most of the times the built-in evaluators are not what you want. s663 00:44:59.800 --> 00:45:01.960 You want to build custom evaluators. s664 00:45:01.960 --> 00:45:09.080 And that's what this is helping you with is like okay, you need custom evaluators to really see if your agent is doing a good job. s665 00:45:09.080 --> 00:45:14.120 Uh but you know, you need some help bootstrapping those custom evaluators. s666 00:45:14.120 --> 00:45:14.800 All right. s667 00:45:14.800 --> 00:45:15.760 Oh, we have 3 minutes left. s668 00:45:15.760 --> 00:45:16.640 I've been blabbing. s669 00:45:16.640 --> 00:45:25.520 Okay, finally, um you know, earlier we were talking about prompt engineering like okay, how can we tweak this prompt in order to work better for these models? s670 00:45:25.520 --> 00:45:26.760 That's pretty painful, right? s671 00:45:26.760 --> 00:45:35.200 If you have to tweak the prompt every time you switch models and like rerun evals, that's like a lot of work. s672 00:45:35.200 --> 00:45:36.080 Oh, there's a little timer. s673 00:45:36.080 --> 00:45:36.800 Thank you. s674 00:45:36.800 --> 00:45:37.960 Okay. s675 00:45:37.960 --> 00:45:43.800 Uh so if you Yeah, if you have to tweak the model every time, uh that's a lot of work and you don't know which way to tweak tweak the model, right? s676 00:45:43.800 --> 00:45:48.080 You don't know the details of every single model, what's going to really work well for it. s677 00:45:48.080 --> 00:45:52.520 So uh what I've added is this as a final example is using Dspy. s678 00:45:52.520 --> 00:45:55.400 Has anyone used Dspy here? s679 00:45:55.400 --> 00:45:56.320 Cool, a couple people. s680 00:45:56.320 --> 00:46:08.200 So Dspy, the idea of Dspy is that you declare your program like you say, "Hey, I'm making this program that's expecting to take user input and then turning it into a structured counter bit." s681 00:46:08.200 --> 00:46:15.440 So, you declare the constraints of your program and then Dspy is going to figure out the prompt that adheres to those constraints. s682 00:46:15.440 --> 00:46:17.240 So, Dspy is going to write your prompt. s683 00:46:17.240 --> 00:46:27.800 So, you you're basically checking your spec in and then having Dspy by write the prompts based off of doing evaluations and seeing, "Hey, what prompt is going to satisfy those constraints?" s684 00:46:27.800 --> 00:46:34.200 And it actually uses these this cool algorithm called Jeppa, genetic credo algorithms. s685 00:46:34.200 --> 00:46:41.000 It can use these genetic algorithms in order to come up with the next best prompt to try, which is really really cool to watch it do that. s686 00:46:41.000 --> 00:46:43.280 Um so, that's the idea of Dspy. s687 00:46:43.280 --> 00:46:45.720 And so, basically we have to set So, it's very different, right? s688 00:46:45.720 --> 00:46:48.640 You can't just take your standard code and just use it. s689 00:46:48.640 --> 00:46:55.400 You kind of have to rewrite your whole code cuz now, instead of checking your pro- like your prompt isn't in your code, your constraints are in your code. s690 00:46:55.400 --> 00:46:56.600 Your spec is your code. s691 00:46:56.600 --> 00:47:00.080 The prompt is an output from this Dspy process, right? s692 00:47:00.080 --> 00:47:06.840 So, here I'm doing I'm asking it to generate a numbered list where each line has exact word count and the first letter of the line spell out a target word. s693 00:47:06.840 --> 00:47:12.880 So, this is the kind of input and then we have a bunch of constraints that are checking to see is the result correct, right? s694 00:47:12.880 --> 00:47:19.440 So, you need to set up your spec and then set up like checks, tests to see, "Hey, does the output satisfy this constraint?" s695 00:47:19.440 --> 00:47:22.320 And that way Dspy can keep iterating, hill climbing, right? s696 00:47:22.320 --> 00:47:27.200 Basically hill climbing until it finds a prompt that satisfies those constraints, right? s697 00:47:27.200 --> 00:47:37.760 So, you give it the student model, the reflection model, uh then it runs this Jeppa uh optimizer, which you know, tries to find the best um prompt, and then we get back the result. s698 00:47:37.760 --> 00:47:41.880 So, the result I have it stored here so we can see it. s699 00:47:41.880 --> 00:47:43.200 It does take a fair amount of time. s700 00:47:43.200 --> 00:47:45.280 It took like 6 minutes on this task. s701 00:47:45.280 --> 00:47:49.280 Um but here, this result is the actual instructions. s702 00:47:49.280 --> 00:47:53.040 So, let me add word wrap into there. s703 00:47:53.040 --> 00:47:53.720 Right. s704 00:47:53.720 --> 00:48:01.960 So, this is the system prompt that it came up with based off of those constraints and those you know those checks, right? s705 00:48:01.960 --> 00:48:05.600 And so this is nice because this is a system prompt that works with Mistral. s706 00:48:05.600 --> 00:48:09.160 Cuz I I picked Mistral cuz that was the one we were having the hardest time with, right? s707 00:48:09.160 --> 00:48:15.680 So especially with using smaller models, this means we can use the cheapest model and still get the results we need, right? s708 00:48:15.680 --> 00:48:27.720 So there's been some great talks by like Dropbox, Shopify about how they've used DSPy plus Jeppa so that they can move off the larger models and use the smaller models and maybe even deploy it on their infrastructure s709 00:48:27.720 --> 00:48:29.040 and get the same results. s710 00:48:29.040 --> 00:48:34.440 But the key was trying to you know using this order to figure out what that optimal prompt is. s711 00:48:34.440 --> 00:48:43.080 So this is a definitely a different way of writing your apps and it can feel a little different to you know to have the prompt be the output of this process, s712 00:48:43.080 --> 00:48:53.160 but it's also potentially really powerful way of writing it and can give you a lot more model independence and more flexibility to be able to use more of these models. s713 00:48:53.160 --> 00:48:59.000 So I wanted to yeah, make sure that people knew about that as an option as well. s714 00:48:59.000 --> 00:49:00.280 Okay. s715 00:49:00.280 --> 00:49:00.960 All right. s716 00:49:00.960 --> 00:49:06.560 So So yeah, so that was all the things that I wanted to show. s717 00:49:06.560 --> 00:49:15.480 As you can see there's lots of differences between models, different ways you can improve the model output and then evaluate it and then improve it. s718 00:49:15.480 --> 00:49:19.560 I do encourage you like yeah, just just tinker with the models. s719 00:49:19.560 --> 00:49:26.440 It's I always learn so much whenever I you know try out different models um and see how different they are. s720 00:49:26.440 --> 00:49:27.920 Oh, I never showed Anthropic. s721 00:49:27.920 --> 00:49:28.920 I meant to show Anthropic. s722 00:49:28.920 --> 00:49:29.720 Okay. s723 00:49:29.720 --> 00:49:31.840 I do have some Anthropic's deployed as well. s724 00:49:31.840 --> 00:49:34.080 With Anthropic you do have to use a different API. s725 00:49:34.080 --> 00:49:37.000 That's why I wasn't using Anthropic most of the time. s726 00:49:37.000 --> 00:49:39.040 But of course you might be interested in Anthropic. s727 00:49:39.040 --> 00:49:43.760 They They use the messages API instead, so you do have to switch your code around. s728 00:49:43.760 --> 00:49:56.040 But I actually specifically like it for rag because here's interesting is that with Anthropic they actually have this notion of citations built into the models. s729 00:49:56.040 --> 00:50:00.640 So, you can actually you provide the sources in this structured format. s730 00:50:00.640 --> 00:50:02.080 So, you say, "Hey, this is my source. s731 00:50:02.080 --> 00:50:02.800 It's a document. s732 00:50:02.800 --> 00:50:03.640 It's text. s733 00:50:03.640 --> 00:50:04.600 Here's the text of it." s734 00:50:04.600 --> 00:50:06.440 And then you enable citations. s735 00:50:06.440 --> 00:50:12.360 And then the Anthropic models are actually trained to return back blocks of text with citation. s736 00:50:12.360 --> 00:50:17.120 So, you can extract those in a programmatic way, which is really nice when you're like building front ends. s737 00:50:17.120 --> 00:50:19.320 You can just link your citations. s738 00:50:19.320 --> 00:50:20.080 So, that's really cool. s739 00:50:20.080 --> 00:50:27.800 So, that's even another thing to look at is you know, a whole different ways of calling models, too. s740 00:50:27.800 --> 00:50:28.480 Okay. s741 00:50:28.480 --> 00:50:29.560 All right. s742 00:50:29.560 --> 00:50:32.480 And yeah, we're kind of at time here. s743 00:50:32.480 --> 00:50:35.080 The the repo will stay up. s744 00:50:35.080 --> 00:50:38.560 So, you're welcome to keep working on the repo. s745 00:50:38.560 --> 00:50:45.880 I do need to rotate the keys that I put in that .env because I mean, I trust you, but I don't know how much I trust you. s746 00:50:45.880 --> 00:50:49.120 And Cedric always already said I was bad for doing that. s747 00:50:49.120 --> 00:50:53.600 So, I I won't be able to keep those keys up for much longer. s748 00:50:53.600 --> 00:50:55.520 But, hopefully with the repo you can try it out. s749 00:50:55.520 --> 00:50:56.840 You can also try it out with a llama. s750 00:50:56.840 --> 00:50:58.280 I added a llama support yesterday. s751 00:50:58.280 --> 00:51:00.560 So, if you want you know, do you want to try a llama, too? s752 00:51:00.560 --> 00:51:07.400 If you're doing local models, you basically just change the examples to point at the local llama server. s753 00:51:07.400 --> 00:51:11.040 And you just have to change the model names as well, right? s754 00:51:11.040 --> 00:51:16.360 But, generally they should all work with a llama as well because it's same API. s755 00:51:16.360 --> 00:51:17.520 All right. s756 00:51:17.520 --> 00:51:18.680 Thank you, everyone. s757 00:51:18.680 --> 00:51:20.500 Oh, and there's stickers. s758 00:51:20.500 --> 00:51:22.500 [applause]