The Unreasonable Effectiveness of Separating the Task from the Model — Maxime Rivest & Isaac Miller

AI Engineer · 17 min · 193 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Please welcome to the stage Maxime Rivest and Isaac Miller.
00:18[applause]
00:19Wow.
00:20Isaac, myself, all of the DS9 community are so grateful to be here today to get to talk to you about AI programming, DS9, and the unreasonable effectiveness of separating the task from the model, its harness, and all of the implementation details.
00:38When you think about it, in programming, if we want to repeat a task often, we make it a function.
00:46We believe the same should be true for AI programs.
00:51Functions are awesome.
00:52Functions are reusable, composable, testable, and optimizable.
00:58To make a function, you give it a name, you define some inputs, some outputs, and then you have some implementation logic inside of it.
01:08You get to reuse your functions dozens of times.
01:11You can optimize it, but you can also compose it into bigger programs.
01:16One of the really nice things about functions is that you can also package it and distribute it, and someone else can use it, and they just need to know about
01:24the contract on top of it to use it, and they can treat it as a black box.
01:30DS9 brings all of these properties to AI programs.
01:34And so, DS9 is an open-source software in Python that lets you, like I said, bring these properties to your AI workflows and AI programs.
01:46And it gives you all of the toolings you need to do that.
01:52Why do you want that?
01:54Well, we have been inventing a lot of terms in our field in the last 3 years.
01:59It's growing fast.
02:00We have new models coming every other week.
02:04We have new techniques, new strategies.
02:06And if you're like me, you want to try all of them.
02:09But will any of these new specific techniques coming out at a different time really help on your task, on your job?
02:19Well, these are all just implementation tactics.
02:23And you want to put them inside of clear contract.
02:27If for your repeated AI task, you define an input interface and an output interface, you get to play in the internals.
02:34You get a lot of agility.
02:37Let's make it concrete for AI.
02:40So, my first AI program I made when I discovered the S bar was that I had some invoices from my farm, and I wanted to extract them to do my taxes.
02:49I wanted to extract the tax values from there.
02:52Then, another AI program I did is that on my keyboard in my computer, I have a little command that reads my keyboard shortcuts, reads my clipboard, and will correct the grammar for me.
03:06Sometimes, I actually wanted it to also rewrite for clarity.
03:09So, I have another program that takes text and just rewrites it for clarity, put it back in my keyboard keyboard.
03:14And that's a command, and then I can like have a lot of agility and bring it different places.
03:19Inside of that, I can change it however I want.
03:22A new model comes out, and I can change that.
03:23It's super easy cuz my interface is fixed like that.
03:27I'll skip that one.
03:29But, they're not restrained to very easy things and small input outputs.
03:35You can be very ambitious with AI programs.
03:38So, in this examples, you could have your entire inbox and a new email coming in, and you want to compose a new drafted reply.
03:46We can do that in the S bar with RLM, recursive language models.
03:50This is an idea that came from around our community.
03:53Or more like things we probably all do, agentic engineering or vibe coding.
03:58You can give it a spec, a repository, and you get a PR.
04:02Those are repeatable tasks.
04:06And so, as I have been telling you, when you fix that boundary, you can focus on the how on the top, and then inside of it, you can I have a little chat with just a simple prompt.
04:17You can iterate on that prompt.
04:18Agents come out, you change it to be an agent.
04:20Tools gets invented, you add tools.
04:23And then we get into loop engineering, you put that inside of it too.
04:28Anything on the outside of it doesn't change.
04:31Your integration and and anything else doesn't change.
04:35And when you have such a hard boundary, you can also start to automatically optimize.
04:42But how can you automatically optimize with just that simple signature?
04:47This is not enough.
04:48This is not enough to specify your task.
04:51And even before ChatGPT came out, the creator of DSPy had started to land on this idea that you need three things to specify your task.
05:01And if you have this language and this ability to express your task in a programming language, you can start to automatically optimize and delegate away the implementation details.
05:13So, the first one is what should happen.
05:16This is instructions.
05:18The signatures that I've been showing you are part of that.
05:20Here on the screen, you see the beginning of a real script in DSPy.
05:25You set your model at the top, you configure that, and it's fully independent of the signatures here, where you have natural language instruction to extract all taxes, and um
05:36and if it's eligible to output zero.
05:39Then you say it, "I'm going to give you an input, it's going to be a string.
05:42I want you to give me an output, and it goes it's going to be a string and a float."
05:45This is natural language expressing my needs.
05:48This is very powerful and efficient.
05:50If you think about it, if you have a friend over coming to play a board game with you, and you give them the instructions and they're ready to play.
05:58But if you want to do like AlphaGo or AlphaZero, and you tell them you're just going to learn from example, you're going to have a long night.
06:07And then, the second one is what must happen.
06:10There's some constraints you have that they have to be listened to.
06:15They have to be enforced.
06:16The best way to do that is with code.
06:20So, I want you to go to the third line or fourth line.
06:24You have self.extract and self.recheck.
06:28You can see we're doing a predict on the extract taxes, and we're doing a chain of thought on the extract taxes.
06:36The first one is a vanilla program.
06:38The second one makes it do some reasoning.
06:41Now, I'm taking them inside in the forward.
06:45And you can see in the if not pred tax, this is a requirement I have that if my first simple vanilla program doesn't extract my taxes, I want you to rerun with more reasoning.
06:56I mean, I got to get my taxes right.
06:58And then, another requirement I have is if the value is below zero, throw.
07:03I want to show that to a human.
07:05I don't want to let you go.
07:06This will not change.
07:08Like, even if I have AGI, I would hope it doesn't make mistake.
07:12But whatever is in the predictor, if they make these mistakes, I still want these things to be true.
07:19So, the last one is what good look like.
07:23And when I was young, was on the farm with my dad, and I asked him, "How do you know that this tree is a maple?"
07:31And he couldn't tell me.
07:32He couldn't give me the instruction on how to know this tree is a maple, and he certainly couldn't give me code on how to know this tree is a maple.
07:39And so, through time with example, I learned how to know that the tree is a maple.
07:45But, this is not limited to things like classifying plants.
07:50It's also for all of the long tails in your specifications that are things that are more latent.
07:56These are sometimes the reason why you would do internship, and you would have a mentor and a mentee.
08:01You're looking at a lot of of examples, and there are long tails of successful behaviors that you have to see and learn.
08:07Now that you have all of these, you have express fully, you have all these three languages you can put together.
08:14You have the specs, the code, and the evals.
08:17And now, your goal is fully specified.
08:20And so, you can start optimizing.
08:22You can use things like Chip A on your metrics and on your program, and you can start optimizing.
08:27At the beginning of the Aspire, the ChatGPT didn't exist.
08:31The models were not good enough to optimize, and so we were using code to find few-shots examples to make the base models uh act in the proper way.
08:41Then, models got better, and so we could automatically optimize instruction.
08:45And in the future, we are starting to be able to be liberated more and more from the implementation details, and delegate that away.
08:54And at the end, our hope in the Aspire is that you can stick to all of that, and then just the news and the implementation details will be automated for you.
09:03Isaac will talk to you a lot more about what has been released in the last year, what we're releasing now, and all of the future plans we have.
09:11Thank you.
09:15[applause]
09:16Thanks, Max.
09:17So, we've given you a pretty big abstract overview of specs, code, and evals.
09:23But, these aren't things that are just res- restricted to the academic sphere.
09:27These are used in production by some of the biggest enterprises for massive gains.
09:32And we see two main benefits when you use the Aspire in the enterprise.
09:36First, is that your implementation becomes cheaper.
09:40When you're flexible to what the implementation is, you can use the bitter lesson to search over different solutions, find something that solves your problem cheaply.
09:51And you can use this to scale to data sizes that weren't possible with a more expensive implementation.
09:58Shopify, 550 times cheaper.
10:01They're able to do that because they went from an expensive model to a cheap model, but they could keep the same eval's, keep iterating on their business logic inside,
10:10and try new things.
10:13There's three awesome case studies here, and you should check them out after the talk.
10:16They give you a lot of details on how you can do this in your own enterprise.
10:22Now, part of the reason why you want to build in the DSPy ecosystem is that we're constantly adding new techniques for you to try.
10:32And it's important to know, none of these techniques we add will definitely solve your problem, because that's your job.
10:38What we can do is we can solve sub problems for you that make your implementation easier.
10:44For instance, Alex Zeng, a PhD student at MIT, came out with this paper called recursive language models.
10:53Recursive language models are a way to solve some kinds of long context programs.
10:57And guess what?
10:58We can bring this into DSPy for you to try.
11:02See if it helps your long context tasks.
11:04Maybe it will, maybe it won't.
11:06But the thing is, it's one line, and your signature stays the same.
11:10That's what's important here.
11:11Everything gets to stay constant, and you get to see if this solves your problem or not.
11:17And we've had a number of examples of this just in the last year from people building in and around the DSPy community.
11:24We've had RLLMs.
11:26We've had Jeppa, which is an incredible prompt optimizer out of Berkeley.
11:29Better together, multi-module GRPO.
11:32All these are incredible research innovations that you get to try in your implementation just by being in the DSPy ecosystem.
11:41And we have more coming in DSPy 4. I'm excited to talk to you about two of those today.
11:46DSPy Flex and qualitative learning.
11:51DSPy.flex is a new kind of module.
11:54In DSPy, when we let you optimize things, it started with few-shot examples.
11:59Then it became prompts.
12:00And now that's becoming code.
12:03For any function that you want to implement, you can actually learn a harness over time to solve that function.
12:12And this is completely custom, and you don't care about the implementation as long as it solves your business problem.
12:18Which you've created ways to measure it because you've defined the three core parts of specs, code, and evals.
12:26The second thing I'm excited to talk about is qualitative learning.
12:31One of the hard hard problems in AI engineering is building evals.
12:36And there's a few reasons why this is hard.
12:39One is that defining what good looks like is really challenging for any real-world problem.
12:46The second is that when you define good, often times you have to lose detail.
12:51If an email is good or bad, contains a lot less information than if you know what could change in that email in order to improve.
12:59And the third is that whenever you create a hill in a dataset, you're really trying to create a proxy for reality.
13:06What if instead we could use reality to inform our evals automatically?
13:13What qualitative learning asks is how do we decrease this question?
13:17How do we decrease assistance?
13:19And it's a research question right now.
13:21But what we believe is that models are now good enough to interpret whatever textual feedback is present in the environment and convert that into evals and a hill that the model can climb.
13:33And so, as you get more feedback from production, it's traces, it's user actions, it's product analytics, it's asking you it's the model asking you questions about how data should be represented.
13:44As you do this, the model can iteratively refine the hill over time and continue climbing it to solve your actual business problem.
13:55And DSPy focuses on these kinds of last mile problems.
13:59We have a really strong research ecosystem and we collaborate really closely with them.
14:04And that's part of the beauty is that we can see the problems that happen in applied AI engineering, solve define them, build a benchmark, and then solve them with techniques.
14:14And then we get to democratize the results of that to everyone because it's open source, open research.
14:21Now, one common question is what happens when we have AGI.
14:26Well, even when we have an incredibly smart model, the model won't know how to solve your problems.
14:32It won't know how to do your tasks or have your context.
14:36And so, this genre of last mile learning is trying to ask, how do we efficiently do this learning?
14:42Intelligence is very different from being all-knowing.
14:47If you were to ask Albert Einstein to help you with your emails, he'd probably ask what's an email.
14:54But, if you AGI will know how to do your emails.
14:59Nevertheless, it won't know how to actually solve your problem and interact with the people you need to interact with.
15:05It won't understand your relationships without learning this context over time.
15:12Since 2022, DSPy has been focused on these three core ideas of specs, code, and evals, all defined as a a interface.
15:24We've certainly evolved over time, and new techniques are incredible.
15:28We've gone from evolving few shots to prompts to now harnesses, and now evolving your emails over time, too.
15:35But what you need to ask for any of these new techniques is how do they help you solve harder problems or solve your own problems better?
15:46And you should ask this question in a data-driven manner.
15:50You should look at this new technique, say how can I apply this to the business problem that I have?
15:55You should define your problem, and you should hold your prompts, models, and code accountable to the problem that you need them to solve.
16:04And what's awesome about when you build in this way where you have flexible implementations, what you unlock is you unlock the ecosystem of all the techniques that anyone in this room is constantly inventing.
16:17You unlock access to the collective intelligence of everyone here, all sharing techniques together.
16:26So, if you want to build reliable AI software, I encourage you to come check out DSPy.
16:32We're completely open source, open research.
16:35And we're here to help you solve your problems by building reliable software.
16:41We have a Discord that you should come join.
16:43And when you come up with the next technique, you should come contribute it to DSPy, and we can help you distribute it and make this awesome technique available for everyone.
16:52Thank you.