Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean

AI Engineer · 15 min · 203 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello everyone.
00:14So preferences over benchmarks.
00:17The talk today is about model routing and specifically why the way most people think about picking a model which usually is chasing you know to the top of a benchmark
00:28is actually the wrong instinct.
00:32I'm Archa VP of engineering for inference engine and AI infrastructure at Digital Ocean and I'll be joined by Tyler who built parts of the router and will actually do a live demo for us today.
00:44We both work on the managed agent orchestration and inference engine products at digital ocean.
00:52So you may know digital ocean as droplets, databases and app platform.
00:57All of that is true.
00:58We are also the AI native cloud.
01:01This is five integrated layers starting from infrastructure all the way up to the managed agents with the inference engine right in the middle.
01:10And that's why we are here talking about inference router.
01:14Routing lives in the inference engine.
01:15And if you want to know more about our stack and the full story, please come find us at the booth.
01:24So everybody is reaching out for the model routing and let's look at three reasons why the three reasons that are breaking the one model habit for most users.
01:33The first one I want to talk about is cost.
01:36Spend is exploding and even companies like Walmart, Uber, Microsoft, they're actively capping usage to control the inference bills.
01:46The second one is fit.
01:48One model for every task is likely an overkill.
01:51We're essentially paying frontier rates for a work that a much smaller model will be able to handle really well.
01:58And the third one, which for me is the most important one, is the risk.
02:02The risk associated with one single model.
02:05models can go down and if you bet your entire product and production on one model, you have no failover when something degrades and model orchestration is actually the new phas.
02:20As you all know, cloud cost optimization took us about 15 years for it to actually become a real good discipline and for companies to get it right.
02:29This one actually is arriving in months and not years.
02:32And here's the premise that I think everybody gets wrong about this.
02:37We all think of like what is the best model for a job.
02:41Here's the thing.
02:42There is no single best model.
02:45The right one depends on the actual request.
02:48For example, if you're doing classification and labeling, a small open model may very well work really well for you and will give you really good cost optimizations.
02:59However, if you're running code completion in line, you will likely need really fast routing and that is where a faster larger routing model comes into picture.
03:09Think about code generation and bug fixing.
03:12You're likely good with an mid openw weight model.
03:15Uh and again, it'll bring you like really good cost optimizations over using a frontier for something that is likely an overkill in this situation.
03:23But then you're looking at like really accuracy critical tasks like code review and security, you're likely going to lean towards a frontier model.
03:34So essentially what makes a model right for a request?
03:38It's a mix that no public leaderboard can actually encode for you because it's the task itself.
03:44What are you actually trying to achieve?
03:45What is your model trying to achieve?
03:47The system prompts and tools around it.
03:49that is the methodology by which you're getting something done using a model.
03:54The cost you're willing to spend.
03:55This is a very very important aspect and latency the use case needs.
04:00Not all use cases need the same amount of latency.
04:04So depending on what you're trying to do, this can vary widely.
04:06And finally, the end user preference.
04:08All of this is driven by what the end user really wants out of your application.
04:13So, I'm going to welcome Tyler on to stage so that he can actually show you the inference light router live in action and show you how it can really help with all of these key aspects that I'm calling out here.
04:31Testing.
04:32All right.
04:32Thank you, Archa.
04:35Okay.
04:36So many builders have tried auto routing before, but the problem was that it feels like a black box.
04:44The router makes a choice and if that choice results in poor performance, you really have no way of improving it.
04:49We built ours differently at the architecture level, which is what you can see on the screen.
04:54A request runs through our open proxy plan and our purpose-built routing model.
04:59Both open source.
05:01There is no vendor lockin, which is a key digital ocean value.
05:06You describe what matters for your workload, costs, latency, quality, preferred models or hard rules.
05:15Then the router uses that context to pick the right model per request.
05:20Because the routing model is specialized for this job, it's super fast, under 200 milliseconds, and it costs customers nothing extra.
05:27In our evaluations, it actually has beating frontier models like the GBT 5 series models at routing task itself with a fraction of the latency.
05:37So the difference is simple.
05:39This is routing you can customize, evaluate and improve without vendor lock in.
05:48So you bring your preferences and we honor them.
05:51You describe a task in natural language and set what matters, cost, latency, and task description.
05:57You bring your rules and we execute them intelligently.
06:01Layer decision tree rules on top.
06:03Start from presets, change anything you want in a single line of code.
06:08And you validate with your own evaluations, not someone else's leaderboard.
06:11Route, evaluate, adjust, then feed that back in.
06:16That loop is key.
06:20Okay, we're going to switch gears here.
06:21We're going to do a live demo.
06:25Bear with me here.
06:29All right, I'm going to show you a couple things.
06:31First, I'll show you router configuration in the UI, how to use it, and then how you can use evaluations to measure and improve your router's performance.
06:39And then I'll show you a real router that I created inside a coding agent workflow.
06:45So I'm here in the cloud console, the digital ocean cloud console and you can see my routers.
06:50We have several presets.
06:51You can see software engineering, general writing, knowledge bases and document intelligence.
06:56In this case, I've actually created my own.
06:58So I I customized our preset software engineering.
07:01Uh if we click into this, we can see that I have sever several different tasks here.
07:06I have bug fixing, code generation, test writing, and a few others.
07:11This also shows that you can specify more than one model per task in the bug fixing case and code generation case.
07:18Um, in the code generation, I have GLM 5.2 and GPT 5.2. And because I really want to always route to GLM 5.2 unless it's down, I use this manual ranking option.
07:28So, it'll always go to GLM 5.2. If GLM fails, it'll fail over to GPD 5.2. In the bug fixing one, you can see a little bit of a different one.
07:37In this case, I have selection policy fastest.
07:41So out of this model pool, if it matches to bug fixing, it'll pick whichever one's been fastest in about the last 30 minutes.
07:48Okay, let's do this in action a little bit.
07:50Here's our playground where I'll show a couple of examples side by side.
07:54First, I'll start with just a simple prompt.
07:56Write a basic Fibonacci function.
08:01And as this runs, we can see on the left we're writing to Opus.
08:05On the right, we're using our software engineering router that I just showed you.
08:08And you're going to see that it picks different models on the right.
08:11So in this case, it matched to the code snippets task and just used the long before Maverick model that I had configured for that one.
08:17And if we scroll down, I mean this is this is obvious, right?
08:20But this model is extremely fast and extremely cheap compared to Opus.
08:27Now let's say optimize my function and we'll see the same thing happen.
08:32In this case, it matched the code performance optimization task using GBT 5.2. And again, it's obviously significantly faster.
08:39If we scroll down here, we can also see that it's significantly cheaper.
08:43We'll do one more.
08:44Write some unit tests.
08:50Okay.
08:50And in this case, it matched to Cloud 5 Summit on the test writing and code verification.
08:56And again, we're going to see faster and cheaper.
08:58So, it's a pattern.
08:59It matches my, you know, vibe check, right?
09:02It's still vibes, though.
09:03How you actually prove it is working it through evaluations.
09:07So, I have an evaluation that I ran here, comparing Opus on the left or actually on the right hand side to my router on the left hand side.
09:17You can see that the scores 90% for my router, 95% correctness for Opus are very very close.
09:23In fact, that's pretty much within a judge uh margin of error.
09:29But what what's really interesting is if we scroll down here, we can see that the router knew significantly less tokens and was significantly faster than Opus.
09:41Okay, let's jump into a real workflow here.
09:44This is where the inference router really becomes impactful.
09:47Here I have two terminals running open code.
09:50On the left I have a single model approach using quad opus.
09:53So I have opus set up or open code set up with opus.
09:56On the right, I've configured open code to send requests to our software engineering router that I just showed you configured.
10:03Um, below I kind of have this customuilt open code where you'll be able to see live uh, observability essentially.
10:09So, let's go ahead and get these started.
10:11It's just a simple feature request pre-loaded into here.
10:14Build me a spinning wheel app.
10:16I'll run the same prompt in both.
10:18And as this runs, we can focus on the bottom panel.
10:20So, it'll start to show up here.
10:22Hopefully, we can see that on the screen.
10:24uh you'll be able to see token usage in real time, which models are being selected, what tasks those map to, and the cost accumulating live.
10:32So on the right, we can already see that we're starting to route to GLM 5.2 because our requests are starting to match the code generation.
10:40And on the left, of course, we're just routing to quadopus.
10:42I think open code sometimes routes to to haiku by itself.
10:45So that's what you see there.
10:47And we'll notice latency, too.
10:49How quickly things start to come back.
10:51In this case, it wants me to create a temporary directory.
10:58So the key difference here is that on the left we'll see every single request that I write goes to the same premium model.
11:04Cost and latency is going to stay high for pretty much every single task.
11:07On the right the router is selecting models based on the task.
11:13[snorts]
11:13So we're optimizing both cost and speed.
11:16And we can see that our software engineering router already finished.
11:19And if we look here, it actually matched to two models throughout.
11:22So, let's go and open this up and see how it looks.
11:25Okay, this actually looks really solid to me in Opus 4.7 finish at a similar time.
11:30Let's take a look at that.
11:32We can compare them.
11:33I mean, it's this is a vibe check, right?
11:34But honestly, I would say the software engineering router did better because this is an interesting approach that you I'm not even sure it works too well.
11:41So, in this case, the router did a little bit better.
11:45So now that step is done, you know, we get similar outputs, but if we look here, the software engineering router has only spent 8 cents on the session while Opus directly has spent 25 cents.
11:56So we have a about a 3x in cost and very very similar quality so far.
12:03Let's try another another prompt here.
12:05What what comes next in a software engineering life cycle?
12:07Probably write some unit test, right?
12:12So we'll write this in both start up this first.
12:17[clears throat] On the right we have the router again.
12:20And we can see that it got matched to the test writing and code verification which picked the claude 5 sonnet model because that's what I configured earlier.
12:29And we'll see the same pattern.
12:30It's going to be significantly cheaper overall across the entire session than going straight to opus.
12:37So we'll let this finish here.
12:42Okay.
12:43And that finished.
12:44Let's just queue up one more, write some documentation in a readme, and then we'll compare the total session cost.
13:03Okay.
13:03And as this runs, we'll wait and see what it does.
13:09Okay, it created the read me and if we look here, we can see that the total session cost for the router was 14 while the total session cost for Opus was 44.
13:20So at this point, we can see the cost is significantly lower.
13:24Latency is optimized per step and the quality remains pretty similar across.
13:29So you can see as you scale this, the cost performance really add up.
13:35Okay, Archa, back to you.
13:41[applause]
13:44Thank you so much, Tyler.
13:46And that that was actually a live demo that we ran here.
13:49So, thanks to Tyler for setting it up and taking us through that.
13:53So, now that you've seen it work, let's look at some quick facts.
13:58Routing decision and under 200 milliseconds per request.
14:03It runs on a custom mixture of experts model purpose-built for routing.
14:08Zero application code changes needed from you to get it to adopt and it's free and included so you do not have to roll out your own router.
14:16And we open source the whole routing model via Plano.
14:19So you can actually check how that looks as well.
14:24The last thing I wanted to talk about was a bit about um routing is the foundation layer.
14:29It's not really the destination.
14:31And there are three things that we usually build on top of it.
14:34The first one is eval to prove that the right model works with your use case and your test well.
14:41Caching so that you can stop paying twice or more for the same answer each time.
14:47And personalization so that the router learns what works for your team over time.
14:53This is a continuous improvement loop maturing over time.
14:56That means that the more you route and evaluate, the better the router does for your workload.
15:04So to summarize, where does this leave you?
15:07There is no single best model.
15:09There's only the right model for the request and benchmarks will only tell you part of the story.
15:15Your preferences will tell you the rest.
15:17And we built the router to honor your preferences and stay open so that you're never locked into a single stack.
15:24And that's how teams actually built.
15:27We are digital ocean and AI native cloud.
15:30Come find us at the booth and route your next workload with us.
15:33Thank you so much for being here.
15:36[applause]
15:50[music]