WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=GgLQ02aO-hs NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/GgLQ02aO-hs.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.920 --> 00:00:18.340 Please welcome to the stage Maxime Rivest and Isaac Miller. s3 00:00:18.340 --> 00:00:19.440 [applause] s4 00:00:19.440 --> 00:00:20.480 Wow. s5 00:00:20.480 --> 00:00:37.520 Isaac, myself, all of the DS9 community are so grateful to be here today to get to talk to you about AI programming, DS9, and the unreasonable effectiveness of separating the task from the model, its harness, and all of the implementation details. s6 00:00:38.880 --> 00:00:46.880 When you think about it, in programming, if we want to repeat a task often, we make it a function. s7 00:00:46.880 --> 00:00:51.000 We believe the same should be true for AI programs. s8 00:00:51.320 --> 00:00:52.680 Functions are awesome. s9 00:00:52.680 --> 00:00:58.360 Functions are reusable, composable, testable, and optimizable. s10 00:00:58.360 --> 00:01:08.000 To make a function, you give it a name, you define some inputs, some outputs, and then you have some implementation logic inside of it. s11 00:01:08.720 --> 00:01:11.560 You get to reuse your functions dozens of times. s12 00:01:11.560 --> 00:01:16.720 You can optimize it, but you can also compose it into bigger programs. s13 00:01:16.920 --> 00:01:24.640 One of the really nice things about functions is that you can also package it and distribute it, and someone else can use it, and they just need to know about s14 00:01:24.640 --> 00:01:29.760 the contract on top of it to use it, and they can treat it as a black box. s15 00:01:30.560 --> 00:01:34.720 DS9 brings all of these properties to AI programs. s16 00:01:34.720 --> 00:01:46.960 And so, DS9 is an open-source software in Python that lets you, like I said, bring these properties to your AI workflows and AI programs. s17 00:01:46.960 --> 00:01:51.080 And it gives you all of the toolings you need to do that. s18 00:01:52.440 --> 00:01:54.520 Why do you want that? s19 00:01:54.520 --> 00:01:59.200 Well, we have been inventing a lot of terms in our field in the last 3 years. s20 00:01:59.200 --> 00:02:00.840 It's growing fast. s21 00:02:00.840 --> 00:02:04.360 We have new models coming every other week. s22 00:02:04.360 --> 00:02:06.880 We have new techniques, new strategies. s23 00:02:06.880 --> 00:02:09.960 And if you're like me, you want to try all of them. s24 00:02:09.960 --> 00:02:19.040 But will any of these new specific techniques coming out at a different time really help on your task, on your job? s25 00:02:19.040 --> 00:02:23.000 Well, these are all just implementation tactics. s26 00:02:23.280 --> 00:02:27.040 And you want to put them inside of clear contract. s27 00:02:27.040 --> 00:02:34.959 If for your repeated AI task, you define an input interface and an output interface, you get to play in the internals. s28 00:02:34.959 --> 00:02:37.280 You get a lot of agility. s29 00:02:37.280 --> 00:02:40.280 Let's make it concrete for AI. s30 00:02:40.280 --> 00:02:49.280 So, my first AI program I made when I discovered the S bar was that I had some invoices from my farm, and I wanted to extract them to do my taxes. s31 00:02:49.280 --> 00:02:52.840 I wanted to extract the tax values from there. s32 00:02:52.840 --> 00:03:06.040 Then, another AI program I did is that on my keyboard in my computer, I have a little command that reads my keyboard shortcuts, reads my clipboard, and will correct the grammar for me. s33 00:03:06.040 --> 00:03:09.239 Sometimes, I actually wanted it to also rewrite for clarity. s34 00:03:09.239 --> 00:03:14.360 So, I have another program that takes text and just rewrites it for clarity, put it back in my keyboard keyboard. s35 00:03:14.360 --> 00:03:19.239 And that's a command, and then I can like have a lot of agility and bring it different places. s36 00:03:19.239 --> 00:03:22.000 Inside of that, I can change it however I want. s37 00:03:22.000 --> 00:03:23.880 A new model comes out, and I can change that. s38 00:03:23.880 --> 00:03:27.760 It's super easy cuz my interface is fixed like that. s39 00:03:27.760 --> 00:03:29.800 I'll skip that one. s40 00:03:29.800 --> 00:03:35.720 But, they're not restrained to very easy things and small input outputs. s41 00:03:35.720 --> 00:03:38.160 You can be very ambitious with AI programs. s42 00:03:38.160 --> 00:03:46.800 So, in this examples, you could have your entire inbox and a new email coming in, and you want to compose a new drafted reply. s43 00:03:46.800 --> 00:03:50.000 We can do that in the S bar with RLM, recursive language models. s44 00:03:50.000 --> 00:03:53.720 This is an idea that came from around our community. s45 00:03:53.720 --> 00:03:58.880 Or more like things we probably all do, agentic engineering or vibe coding. s46 00:03:58.880 --> 00:04:02.800 You can give it a spec, a repository, and you get a PR. s47 00:04:02.800 --> 00:04:05.600 Those are repeatable tasks. s48 00:04:06.000 --> 00:04:17.120 And so, as I have been telling you, when you fix that boundary, you can focus on the how on the top, and then inside of it, you can I have a little chat with just a simple prompt. s49 00:04:17.120 --> 00:04:18.799 You can iterate on that prompt. s50 00:04:18.799 --> 00:04:20.880 Agents come out, you change it to be an agent. s51 00:04:20.880 --> 00:04:23.360 Tools gets invented, you add tools. s52 00:04:23.360 --> 00:04:28.040 And then we get into loop engineering, you put that inside of it too. s53 00:04:28.040 --> 00:04:31.520 Anything on the outside of it doesn't change. s54 00:04:31.520 --> 00:04:35.360 Your integration and and anything else doesn't change. s55 00:04:35.360 --> 00:04:42.360 And when you have such a hard boundary, you can also start to automatically optimize. s56 00:04:42.960 --> 00:04:47.240 But how can you automatically optimize with just that simple signature? s57 00:04:47.240 --> 00:04:48.480 This is not enough. s58 00:04:48.480 --> 00:04:51.160 This is not enough to specify your task. s59 00:04:51.160 --> 00:05:01.480 And even before ChatGPT came out, the creator of DSPy had started to land on this idea that you need three things to specify your task. s60 00:05:01.480 --> 00:05:13.240 And if you have this language and this ability to express your task in a programming language, you can start to automatically optimize and delegate away the implementation details. s61 00:05:13.240 --> 00:05:16.200 So, the first one is what should happen. s62 00:05:16.200 --> 00:05:18.000 This is instructions. s63 00:05:18.000 --> 00:05:20.960 The signatures that I've been showing you are part of that. s64 00:05:20.960 --> 00:05:25.000 Here on the screen, you see the beginning of a real script in DSPy. s65 00:05:25.000 --> 00:05:36.640 You set your model at the top, you configure that, and it's fully independent of the signatures here, where you have natural language instruction to extract all taxes, and um s66 00:05:36.640 --> 00:05:39.200 and if it's eligible to output zero. s67 00:05:39.200 --> 00:05:42.000 Then you say it, "I'm going to give you an input, it's going to be a string. s68 00:05:42.000 --> 00:05:45.680 I want you to give me an output, and it goes it's going to be a string and a float." s69 00:05:45.680 --> 00:05:48.760 This is natural language expressing my needs. s70 00:05:48.760 --> 00:05:50.960 This is very powerful and efficient. s71 00:05:50.960 --> 00:05:58.520 If you think about it, if you have a friend over coming to play a board game with you, and you give them the instructions and they're ready to play. s72 00:05:58.520 --> 00:06:07.360 But if you want to do like AlphaGo or AlphaZero, and you tell them you're just going to learn from example, you're going to have a long night. s73 00:06:07.360 --> 00:06:10.360 And then, the second one is what must happen. s74 00:06:10.360 --> 00:06:15.000 There's some constraints you have that they have to be listened to. s75 00:06:15.000 --> 00:06:16.680 They have to be enforced. s76 00:06:16.680 --> 00:06:20.120 The best way to do that is with code. s77 00:06:20.120 --> 00:06:24.440 So, I want you to go to the third line or fourth line. s78 00:06:24.440 --> 00:06:28.840 You have self.extract and self.recheck. s79 00:06:28.840 --> 00:06:36.840 You can see we're doing a predict on the extract taxes, and we're doing a chain of thought on the extract taxes. s80 00:06:36.840 --> 00:06:38.760 The first one is a vanilla program. s81 00:06:38.760 --> 00:06:41.800 The second one makes it do some reasoning. s82 00:06:41.800 --> 00:06:45.880 Now, I'm taking them inside in the forward. s83 00:06:45.880 --> 00:06:56.320 And you can see in the if not pred tax, this is a requirement I have that if my first simple vanilla program doesn't extract my taxes, I want you to rerun with more reasoning. s84 00:06:56.320 --> 00:06:58.960 I mean, I got to get my taxes right. s85 00:06:58.960 --> 00:07:03.680 And then, another requirement I have is if the value is below zero, throw. s86 00:07:03.680 --> 00:07:05.040 I want to show that to a human. s87 00:07:05.040 --> 00:07:06.720 I don't want to let you go. s88 00:07:06.720 --> 00:07:08.080 This will not change. s89 00:07:08.080 --> 00:07:12.360 Like, even if I have AGI, I would hope it doesn't make mistake. s90 00:07:12.360 --> 00:07:19.400 But whatever is in the predictor, if they make these mistakes, I still want these things to be true. s91 00:07:19.400 --> 00:07:23.400 So, the last one is what good look like. s92 00:07:23.400 --> 00:07:31.600 And when I was young, was on the farm with my dad, and I asked him, "How do you know that this tree is a maple?" s93 00:07:31.600 --> 00:07:32.800 And he couldn't tell me. s94 00:07:32.800 --> 00:07:39.800 He couldn't give me the instruction on how to know this tree is a maple, and he certainly couldn't give me code on how to know this tree is a maple. s95 00:07:39.800 --> 00:07:45.640 And so, through time with example, I learned how to know that the tree is a maple. s96 00:07:45.640 --> 00:07:50.200 But, this is not limited to things like classifying plants. s97 00:07:50.200 --> 00:07:56.360 It's also for all of the long tails in your specifications that are things that are more latent. s98 00:07:56.360 --> 00:08:01.080 These are sometimes the reason why you would do internship, and you would have a mentor and a mentee. s99 00:08:01.080 --> 00:08:07.800 You're looking at a lot of of examples, and there are long tails of successful behaviors that you have to see and learn. s100 00:08:07.800 --> 00:08:14.320 Now that you have all of these, you have express fully, you have all these three languages you can put together. s101 00:08:14.320 --> 00:08:17.080 You have the specs, the code, and the evals. s102 00:08:17.080 --> 00:08:20.400 And now, your goal is fully specified. s103 00:08:20.400 --> 00:08:22.200 And so, you can start optimizing. s104 00:08:22.200 --> 00:08:27.800 You can use things like Chip A on your metrics and on your program, and you can start optimizing. s105 00:08:27.800 --> 00:08:31.480 At the beginning of the Aspire, the ChatGPT didn't exist. s106 00:08:31.480 --> 00:08:41.840 The models were not good enough to optimize, and so we were using code to find few-shots examples to make the base models uh act in the proper way. s107 00:08:41.840 --> 00:08:45.600 Then, models got better, and so we could automatically optimize instruction. s108 00:08:45.600 --> 00:08:54.400 And in the future, we are starting to be able to be liberated more and more from the implementation details, and delegate that away. s109 00:08:54.400 --> 00:09:03.480 And at the end, our hope in the Aspire is that you can stick to all of that, and then just the news and the implementation details will be automated for you. s110 00:09:03.480 --> 00:09:11.320 Isaac will talk to you a lot more about what has been released in the last year, what we're releasing now, and all of the future plans we have. s111 00:09:11.320 --> 00:09:13.480 Thank you. s112 00:09:15.446 --> 00:09:16.080 [applause] s113 00:09:16.080 --> 00:09:17.839 Thanks, Max. s114 00:09:17.839 --> 00:09:23.080 So, we've given you a pretty big abstract overview of specs, code, and evals. s115 00:09:23.080 --> 00:09:27.520 But, these aren't things that are just res- restricted to the academic sphere. s116 00:09:27.520 --> 00:09:32.800 These are used in production by some of the biggest enterprises for massive gains. s117 00:09:32.800 --> 00:09:36.960 And we see two main benefits when you use the Aspire in the enterprise. s118 00:09:36.960 --> 00:09:40.920 First, is that your implementation becomes cheaper. s119 00:09:40.920 --> 00:09:51.880 When you're flexible to what the implementation is, you can use the bitter lesson to search over different solutions, find something that solves your problem cheaply. s120 00:09:51.880 --> 00:09:58.000 And you can use this to scale to data sizes that weren't possible with a more expensive implementation. s121 00:09:58.000 --> 00:10:01.920 Shopify, 550 times cheaper. s122 00:10:01.920 --> 00:10:10.760 They're able to do that because they went from an expensive model to a cheap model, but they could keep the same eval's, keep iterating on their business logic inside, s123 00:10:10.760 --> 00:10:13.040 and try new things. s124 00:10:13.040 --> 00:10:16.920 There's three awesome case studies here, and you should check them out after the talk. s125 00:10:16.920 --> 00:10:21.320 They give you a lot of details on how you can do this in your own enterprise. s126 00:10:22.720 --> 00:10:32.600 Now, part of the reason why you want to build in the DSPy ecosystem is that we're constantly adding new techniques for you to try. s127 00:10:32.600 --> 00:10:38.760 And it's important to know, none of these techniques we add will definitely solve your problem, because that's your job. s128 00:10:38.760 --> 00:10:44.760 What we can do is we can solve sub problems for you that make your implementation easier. s129 00:10:44.760 --> 00:10:53.680 For instance, Alex Zeng, a PhD student at MIT, came out with this paper called recursive language models. s130 00:10:53.680 --> 00:10:57.920 Recursive language models are a way to solve some kinds of long context programs. s131 00:10:57.920 --> 00:10:58.960 And guess what? s132 00:10:58.960 --> 00:11:02.080 We can bring this into DSPy for you to try. s133 00:11:02.080 --> 00:11:04.400 See if it helps your long context tasks. s134 00:11:04.400 --> 00:11:06.440 Maybe it will, maybe it won't. s135 00:11:06.440 --> 00:11:10.400 But the thing is, it's one line, and your signature stays the same. s136 00:11:10.400 --> 00:11:11.640 That's what's important here. s137 00:11:11.640 --> 00:11:17.240 Everything gets to stay constant, and you get to see if this solves your problem or not. s138 00:11:17.720 --> 00:11:24.560 And we've had a number of examples of this just in the last year from people building in and around the DSPy community. s139 00:11:24.560 --> 00:11:26.000 We've had RLLMs. s140 00:11:26.000 --> 00:11:29.600 We've had Jeppa, which is an incredible prompt optimizer out of Berkeley. s141 00:11:29.600 --> 00:11:32.640 Better together, multi-module GRPO. s142 00:11:32.640 --> 00:11:40.720 All these are incredible research innovations that you get to try in your implementation just by being in the DSPy ecosystem. s143 00:11:41.360 --> 00:11:46.400 And we have more coming in DSPy 4. I'm excited to talk to you about two of those today. s144 00:11:46.400 --> 00:11:50.080 DSPy Flex and qualitative learning. s145 00:11:51.040 --> 00:11:54.640 DSPy.flex is a new kind of module. s146 00:11:54.640 --> 00:11:59.360 In DSPy, when we let you optimize things, it started with few-shot examples. s147 00:11:59.360 --> 00:12:00.800 Then it became prompts. s148 00:12:00.800 --> 00:12:03.880 And now that's becoming code. s149 00:12:03.880 --> 00:12:12.080 For any function that you want to implement, you can actually learn a harness over time to solve that function. s150 00:12:12.080 --> 00:12:18.600 And this is completely custom, and you don't care about the implementation as long as it solves your business problem. s151 00:12:18.600 --> 00:12:24.960 Which you've created ways to measure it because you've defined the three core parts of specs, code, and evals. s152 00:12:26.000 --> 00:12:31.440 The second thing I'm excited to talk about is qualitative learning. s153 00:12:31.800 --> 00:12:36.640 One of the hard hard problems in AI engineering is building evals. s154 00:12:36.640 --> 00:12:39.440 And there's a few reasons why this is hard. s155 00:12:39.440 --> 00:12:45.920 One is that defining what good looks like is really challenging for any real-world problem. s156 00:12:46.000 --> 00:12:51.280 The second is that when you define good, often times you have to lose detail. s157 00:12:51.280 --> 00:12:58.880 If an email is good or bad, contains a lot less information than if you know what could change in that email in order to improve. s158 00:12:59.320 --> 00:13:06.400 And the third is that whenever you create a hill in a dataset, you're really trying to create a proxy for reality. s159 00:13:06.400 --> 00:13:12.920 What if instead we could use reality to inform our evals automatically? s160 00:13:13.880 --> 00:13:17.560 What qualitative learning asks is how do we decrease this question? s161 00:13:17.560 --> 00:13:19.080 How do we decrease assistance? s162 00:13:19.080 --> 00:13:21.600 And it's a research question right now. s163 00:13:21.600 --> 00:13:33.720 But what we believe is that models are now good enough to interpret whatever textual feedback is present in the environment and convert that into evals and a hill that the model can climb. s164 00:13:33.720 --> 00:13:44.320 And so, as you get more feedback from production, it's traces, it's user actions, it's product analytics, it's asking you it's the model asking you questions about how data should be represented. s165 00:13:44.320 --> 00:13:53.440 As you do this, the model can iteratively refine the hill over time and continue climbing it to solve your actual business problem. s166 00:13:55.280 --> 00:13:59.720 And DSPy focuses on these kinds of last mile problems. s167 00:13:59.720 --> 00:14:04.160 We have a really strong research ecosystem and we collaborate really closely with them. s168 00:14:04.160 --> 00:14:14.120 And that's part of the beauty is that we can see the problems that happen in applied AI engineering, solve define them, build a benchmark, and then solve them with techniques. s169 00:14:14.120 --> 00:14:20.880 And then we get to democratize the results of that to everyone because it's open source, open research. s170 00:14:21.560 --> 00:14:26.840 Now, one common question is what happens when we have AGI. s171 00:14:26.840 --> 00:14:32.480 Well, even when we have an incredibly smart model, the model won't know how to solve your problems. s172 00:14:32.480 --> 00:14:36.000 It won't know how to do your tasks or have your context. s173 00:14:36.000 --> 00:14:42.839 And so, this genre of last mile learning is trying to ask, how do we efficiently do this learning? s174 00:14:42.839 --> 00:14:47.200 Intelligence is very different from being all-knowing. s175 00:14:47.360 --> 00:14:54.839 If you were to ask Albert Einstein to help you with your emails, he'd probably ask what's an email. s176 00:14:54.839 --> 00:14:59.600 But, if you AGI will know how to do your emails. s177 00:14:59.600 --> 00:15:05.320 Nevertheless, it won't know how to actually solve your problem and interact with the people you need to interact with. s178 00:15:05.320 --> 00:15:10.440 It won't understand your relationships without learning this context over time. s179 00:15:12.200 --> 00:15:22.680 Since 2022, DSPy has been focused on these three core ideas of specs, code, and evals, all defined as a a interface. s180 00:15:24.160 --> 00:15:28.760 We've certainly evolved over time, and new techniques are incredible. s181 00:15:28.760 --> 00:15:35.600 We've gone from evolving few shots to prompts to now harnesses, and now evolving your emails over time, too. s182 00:15:35.680 --> 00:15:45.160 But what you need to ask for any of these new techniques is how do they help you solve harder problems or solve your own problems better? s183 00:15:46.360 --> 00:15:50.360 And you should ask this question in a data-driven manner. s184 00:15:50.360 --> 00:15:55.440 You should look at this new technique, say how can I apply this to the business problem that I have? s185 00:15:55.440 --> 00:16:03.480 You should define your problem, and you should hold your prompts, models, and code accountable to the problem that you need them to solve. s186 00:16:04.520 --> 00:16:17.200 And what's awesome about when you build in this way where you have flexible implementations, what you unlock is you unlock the ecosystem of all the techniques that anyone in this room is constantly inventing. s187 00:16:17.200 --> 00:16:23.600 You unlock access to the collective intelligence of everyone here, all sharing techniques together. s188 00:16:26.120 --> 00:16:32.920 So, if you want to build reliable AI software, I encourage you to come check out DSPy. s189 00:16:32.920 --> 00:16:35.960 We're completely open source, open research. s190 00:16:35.960 --> 00:16:41.280 And we're here to help you solve your problems by building reliable software. s191 00:16:41.280 --> 00:16:43.600 We have a Discord that you should come join. s192 00:16:43.600 --> 00:16:52.040 And when you come up with the next technique, you should come contribute it to DSPy, and we can help you distribute it and make this awesome technique available for everyone. s193 00:16:52.040 --> 00:16:54.240 Thank you.