WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=FvxY8oPoI8o

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/FvxY8oPoI8o.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.719 --> 00:00:14.559
Hello everyone.

s3
00:00:14.559 --> 00:00:17.520
So preferences over benchmarks.

s4
00:00:17.520 --> 00:00:28.320
The talk today is about model routing and specifically why the way most people think about picking a model which usually is chasing you know to the top of a benchmark

s5
00:00:28.320 --> 00:00:31.279
is actually the wrong instinct.

s6
00:00:32.079 --> 00:00:44.000
I'm Archa VP of engineering for inference engine and AI infrastructure at Digital Ocean and I'll be joined by Tyler who built parts of the router and will actually do a live demo for us today.

s7
00:00:44.000 --> 00:00:50.640
We both work on the managed agent orchestration and inference engine products at digital ocean.

s8
00:00:52.160 --> 00:00:57.520
So you may know digital ocean as droplets, databases and app platform.

s9
00:00:57.520 --> 00:00:58.640
All of that is true.

s10
00:00:58.640 --> 00:01:01.760
We are also the AI native cloud.

s11
00:01:01.760 --> 00:01:10.560
This is five integrated layers starting from infrastructure all the way up to the managed agents with the inference engine right in the middle.

s12
00:01:10.560 --> 00:01:14.000
And that's why we are here talking about inference router.

s13
00:01:14.000 --> 00:01:15.840
Routing lives in the inference engine.

s14
00:01:15.840 --> 00:01:22.000
And if you want to know more about our stack and the full story, please come find us at the booth.

s15
00:01:24.000 --> 00:01:33.840
So everybody is reaching out for the model routing and let's look at three reasons why the three reasons that are breaking the one model habit for most users.

s16
00:01:33.840 --> 00:01:36.640
The first one I want to talk about is cost.

s17
00:01:36.640 --> 00:01:46.079
Spend is exploding and even companies like Walmart, Uber, Microsoft, they're actively capping usage to control the inference bills.

s18
00:01:46.079 --> 00:01:48.159
The second one is fit.

s19
00:01:48.159 --> 00:01:51.600
One model for every task is likely an overkill.

s20
00:01:51.600 --> 00:01:58.240
We're essentially paying frontier rates for a work that a much smaller model will be able to handle really well.

s21
00:01:58.240 --> 00:02:02.479
And the third one, which for me is the most important one, is the risk.

s22
00:02:02.479 --> 00:02:05.680
The risk associated with one single model.

s23
00:02:05.680 --> 00:02:20.800
models can go down and if you bet your entire product and production on one model, you have no failover when something degrades and model orchestration is actually the new phas.

s24
00:02:20.800 --> 00:02:29.840
As you all know, cloud cost optimization took us about 15 years for it to actually become a real good discipline and for companies to get it right.

s25
00:02:29.840 --> 00:02:32.800
This one actually is arriving in months and not years.

s26
00:02:32.800 --> 00:02:37.599
And here's the premise that I think everybody gets wrong about this.

s27
00:02:37.920 --> 00:02:41.280
We all think of like what is the best model for a job.

s28
00:02:41.280 --> 00:02:42.480
Here's the thing.

s29
00:02:42.480 --> 00:02:45.280
There is no single best model.

s30
00:02:45.280 --> 00:02:48.560
The right one depends on the actual request.

s31
00:02:48.560 --> 00:02:59.040
For example, if you're doing classification and labeling, a small open model may very well work really well for you and will give you really good cost optimizations.

s32
00:02:59.040 --> 00:03:09.680
However, if you're running code completion in line, you will likely need really fast routing and that is where a faster larger routing model comes into picture.

s33
00:03:09.680 --> 00:03:12.080
Think about code generation and bug fixing.

s34
00:03:12.080 --> 00:03:15.360
You're likely good with an mid openw weight model.

s35
00:03:15.360 --> 00:03:23.280
Uh and again, it'll bring you like really good cost optimizations over using a frontier for something that is likely an overkill in this situation.

s36
00:03:23.280 --> 00:03:33.040
But then you're looking at like really accuracy critical tasks like code review and security, you're likely going to lean towards a frontier model.

s37
00:03:34.080 --> 00:03:38.080
So essentially what makes a model right for a request?

s38
00:03:38.080 --> 00:03:44.000
It's a mix that no public leaderboard can actually encode for you because it's the task itself.

s39
00:03:44.000 --> 00:03:45.519
What are you actually trying to achieve?

s40
00:03:45.519 --> 00:03:47.599
What is your model trying to achieve?

s41
00:03:47.599 --> 00:03:49.599
The system prompts and tools around it.

s42
00:03:49.599 --> 00:03:54.239
that is the methodology by which you're getting something done using a model.

s43
00:03:54.239 --> 00:03:55.599
The cost you're willing to spend.

s44
00:03:55.599 --> 00:04:00.879
This is a very very important aspect and latency the use case needs.

s45
00:04:00.879 --> 00:04:04.000
Not all use cases need the same amount of latency.

s46
00:04:04.000 --> 00:04:06.879
So depending on what you're trying to do, this can vary widely.

s47
00:04:06.879 --> 00:04:08.480
And finally, the end user preference.

s48
00:04:08.480 --> 00:04:13.760
All of this is driven by what the end user really wants out of your application.

s49
00:04:13.760 --> 00:04:26.720
So, I'm going to welcome Tyler on to stage so that he can actually show you the inference light router live in action and show you how it can really help with all of these key aspects that I'm calling out here.

s50
00:04:31.360 --> 00:04:32.080
Testing.

s51
00:04:32.080 --> 00:04:32.639
All right.

s52
00:04:32.639 --> 00:04:35.040
Thank you, Archa.

s53
00:04:35.840 --> 00:04:36.720
Okay.

s54
00:04:36.720 --> 00:04:44.080
So many builders have tried auto routing before, but the problem was that it feels like a black box.

s55
00:04:44.080 --> 00:04:49.759
The router makes a choice and if that choice results in poor performance, you really have no way of improving it.

s56
00:04:49.759 --> 00:04:54.560
We built ours differently at the architecture level, which is what you can see on the screen.

s57
00:04:54.560 --> 00:04:59.759
A request runs through our open proxy plan and our purpose-built routing model.

s58
00:04:59.759 --> 00:05:01.600
Both open source.

s59
00:05:01.600 --> 00:05:06.639
There is no vendor lockin, which is a key digital ocean value.

s60
00:05:06.639 --> 00:05:15.919
You describe what matters for your workload, costs, latency, quality, preferred models or hard rules.

s61
00:05:15.919 --> 00:05:20.720
Then the router uses that context to pick the right model per request.

s62
00:05:20.720 --> 00:05:27.840
Because the routing model is specialized for this job, it's super fast, under 200 milliseconds, and it costs customers nothing extra.

s63
00:05:27.840 --> 00:05:37.440
In our evaluations, it actually has beating frontier models like the GBT 5 series models at routing task itself with a fraction of the latency.

s64
00:05:37.440 --> 00:05:39.039
So the difference is simple.

s65
00:05:39.039 --> 00:05:45.919
This is routing you can customize, evaluate and improve without vendor lock in.

s66
00:05:48.240 --> 00:05:51.039
So you bring your preferences and we honor them.

s67
00:05:51.039 --> 00:05:57.759
You describe a task in natural language and set what matters, cost, latency, and task description.

s68
00:05:57.759 --> 00:06:01.440
You bring your rules and we execute them intelligently.

s69
00:06:01.440 --> 00:06:03.600
Layer decision tree rules on top.

s70
00:06:03.600 --> 00:06:08.080
Start from presets, change anything you want in a single line of code.

s71
00:06:08.080 --> 00:06:11.919
And you validate with your own evaluations, not someone else's leaderboard.

s72
00:06:11.919 --> 00:06:16.560
Route, evaluate, adjust, then feed that back in.

s73
00:06:16.560 --> 00:06:19.520
That loop is key.

s74
00:06:20.000 --> 00:06:21.520
Okay, we're going to switch gears here.

s75
00:06:21.520 --> 00:06:24.240
We're going to do a live demo.

s76
00:06:25.919 --> 00:06:28.560
Bear with me here.

s77
00:06:29.440 --> 00:06:31.280
All right, I'm going to show you a couple things.

s78
00:06:31.280 --> 00:06:39.840
First, I'll show you router configuration in the UI, how to use it, and then how you can use evaluations to measure and improve your router's performance.

s79
00:06:39.840 --> 00:06:45.759
And then I'll show you a real router that I created inside a coding agent workflow.

s80
00:06:45.759 --> 00:06:50.479
So I'm here in the cloud console, the digital ocean cloud console and you can see my routers.

s81
00:06:50.479 --> 00:06:51.600
We have several presets.

s82
00:06:51.600 --> 00:06:56.479
You can see software engineering, general writing, knowledge bases and document intelligence.

s83
00:06:56.479 --> 00:06:58.639
In this case, I've actually created my own.

s84
00:06:58.639 --> 00:07:01.759
So I I customized our preset software engineering.

s85
00:07:01.759 --> 00:07:06.880
Uh if we click into this, we can see that I have sever several different tasks here.

s86
00:07:06.880 --> 00:07:11.280
I have bug fixing, code generation, test writing, and a few others.

s87
00:07:11.280 --> 00:07:18.400
This also shows that you can specify more than one model per task in the bug fixing case and code generation case.

s88
00:07:18.400 --> 00:07:28.720
Um, in the code generation, I have GLM 5.2 and GPT 5.2. And because I really want to always route to GLM 5.2 unless it's down, I use this manual ranking option.

s89
00:07:28.720 --> 00:07:37.759
So, it'll always go to GLM 5.2. If GLM fails, it'll fail over to GPD 5.2. In the bug fixing one, you can see a little bit of a different one.

s90
00:07:37.759 --> 00:07:41.039
In this case, I have selection policy fastest.

s91
00:07:41.039 --> 00:07:48.720
So out of this model pool, if it matches to bug fixing, it'll pick whichever one's been fastest in about the last 30 minutes.

s92
00:07:48.720 --> 00:07:50.639
Okay, let's do this in action a little bit.

s93
00:07:50.639 --> 00:07:54.720
Here's our playground where I'll show a couple of examples side by side.

s94
00:07:54.720 --> 00:07:56.879
First, I'll start with just a simple prompt.

s95
00:07:56.879 --> 00:08:00.080
Write a basic Fibonacci function.

s96
00:08:01.039 --> 00:08:05.199
And as this runs, we can see on the left we're writing to Opus.

s97
00:08:05.199 --> 00:08:08.800
On the right, we're using our software engineering router that I just showed you.

s98
00:08:08.800 --> 00:08:11.199
And you're going to see that it picks different models on the right.

s99
00:08:11.199 --> 00:08:17.840
So in this case, it matched to the code snippets task and just used the long before Maverick model that I had configured for that one.

s100
00:08:17.840 --> 00:08:20.479
And if we scroll down, I mean this is this is obvious, right?

s101
00:08:20.479 --> 00:08:25.919
But this model is extremely fast and extremely cheap compared to Opus.

s102
00:08:27.680 --> 00:08:32.240
Now let's say optimize my function and we'll see the same thing happen.

s103
00:08:32.240 --> 00:08:39.760
In this case, it matched the code performance optimization task using GBT 5.2. And again, it's obviously significantly faster.

s104
00:08:39.760 --> 00:08:43.919
If we scroll down here, we can also see that it's significantly cheaper.

s105
00:08:43.919 --> 00:08:44.880
We'll do one more.

s106
00:08:44.880 --> 00:08:47.519
Write some unit tests.

s107
00:08:50.640 --> 00:08:50.880
Okay.

s108
00:08:50.880 --> 00:08:56.000
And in this case, it matched to Cloud 5 Summit on the test writing and code verification.

s109
00:08:56.000 --> 00:08:58.800
And again, we're going to see faster and cheaper.

s110
00:08:58.800 --> 00:08:59.600
So, it's a pattern.

s111
00:08:59.600 --> 00:09:02.320
It matches my, you know, vibe check, right?

s112
00:09:02.320 --> 00:09:03.519
It's still vibes, though.

s113
00:09:03.519 --> 00:09:07.680
How you actually prove it is working it through evaluations.

s114
00:09:07.680 --> 00:09:17.200
So, I have an evaluation that I ran here, comparing Opus on the left or actually on the right hand side to my router on the left hand side.

s115
00:09:17.200 --> 00:09:23.360
You can see that the scores 90% for my router, 95% correctness for Opus are very very close.

s116
00:09:23.360 --> 00:09:29.120
In fact, that's pretty much within a judge uh margin of error.

s117
00:09:29.120 --> 00:09:40.880
But what what's really interesting is if we scroll down here, we can see that the router knew significantly less tokens and was significantly faster than Opus.

s118
00:09:41.839 --> 00:09:44.399
Okay, let's jump into a real workflow here.

s119
00:09:44.399 --> 00:09:47.519
This is where the inference router really becomes impactful.

s120
00:09:47.519 --> 00:09:50.080
Here I have two terminals running open code.

s121
00:09:50.080 --> 00:09:53.360
On the left I have a single model approach using quad opus.

s122
00:09:53.360 --> 00:09:56.640
So I have opus set up or open code set up with opus.

s123
00:09:56.640 --> 00:10:03.279
On the right, I've configured open code to send requests to our software engineering router that I just showed you configured.

s124
00:10:03.279 --> 00:10:09.600
Um, below I kind of have this customuilt open code where you'll be able to see live uh, observability essentially.

s125
00:10:09.600 --> 00:10:11.200
So, let's go ahead and get these started.

s126
00:10:11.200 --> 00:10:14.320
It's just a simple feature request pre-loaded into here.

s127
00:10:14.320 --> 00:10:16.079
Build me a spinning wheel app.

s128
00:10:16.079 --> 00:10:18.880
I'll run the same prompt in both.

s129
00:10:18.880 --> 00:10:20.959
And as this runs, we can focus on the bottom panel.

s130
00:10:20.959 --> 00:10:22.480
So, it'll start to show up here.

s131
00:10:22.480 --> 00:10:24.480
Hopefully, we can see that on the screen.

s132
00:10:24.480 --> 00:10:32.560
uh you'll be able to see token usage in real time, which models are being selected, what tasks those map to, and the cost accumulating live.

s133
00:10:32.560 --> 00:10:40.240
So on the right, we can already see that we're starting to route to GLM 5.2 because our requests are starting to match the code generation.

s134
00:10:40.240 --> 00:10:42.399
And on the left, of course, we're just routing to quadopus.

s135
00:10:42.399 --> 00:10:45.680
I think open code sometimes routes to to haiku by itself.

s136
00:10:45.680 --> 00:10:47.920
So that's what you see there.

s137
00:10:47.920 --> 00:10:49.279
And we'll notice latency, too.

s138
00:10:49.279 --> 00:10:51.040
How quickly things start to come back.

s139
00:10:51.040 --> 00:10:57.680
In this case, it wants me to create a temporary directory.

s140
00:10:58.480 --> 00:11:04.160
So the key difference here is that on the left we'll see every single request that I write goes to the same premium model.

s141
00:11:04.160 --> 00:11:07.760
Cost and latency is going to stay high for pretty much every single task.

s142
00:11:07.760 --> 00:11:13.347
On the right the router is selecting models based on the task.

s143
00:11:13.347 --> 00:11:13.360
[snorts]

s144
00:11:13.360 --> 00:11:16.320
So we're optimizing both cost and speed.

s145
00:11:16.320 --> 00:11:19.600
And we can see that our software engineering router already finished.

s146
00:11:19.600 --> 00:11:22.399
And if we look here, it actually matched to two models throughout.

s147
00:11:22.399 --> 00:11:25.040
So, let's go and open this up and see how it looks.

s148
00:11:25.040 --> 00:11:30.320
Okay, this actually looks really solid to me in Opus 4.7 finish at a similar time.

s149
00:11:30.320 --> 00:11:32.399
Let's take a look at that.

s150
00:11:32.399 --> 00:11:33.200
We can compare them.

s151
00:11:33.200 --> 00:11:34.720
I mean, it's this is a vibe check, right?

s152
00:11:34.720 --> 00:11:41.200
But honestly, I would say the software engineering router did better because this is an interesting approach that you I'm not even sure it works too well.

s153
00:11:41.200 --> 00:11:45.200
So, in this case, the router did a little bit better.

s154
00:11:45.200 --> 00:11:56.880
So now that step is done, you know, we get similar outputs, but if we look here, the software engineering router has only spent 8 cents on the session while Opus directly has spent 25 cents.

s155
00:11:56.880 --> 00:12:02.880
So we have a about a 3x in cost and very very similar quality so far.

s156
00:12:03.680 --> 00:12:05.600
Let's try another another prompt here.

s157
00:12:05.600 --> 00:12:07.760
What what comes next in a software engineering life cycle?

s158
00:12:07.760 --> 00:12:10.880
Probably write some unit test, right?

s159
00:12:12.160 --> 00:12:17.538
So we'll write this in both start up this first.

s160
00:12:17.538 --> 00:12:20.959
[clears throat] On the right we have the router again.

s161
00:12:20.959 --> 00:12:29.360
And we can see that it got matched to the test writing and code verification which picked the claude 5 sonnet model because that's what I configured earlier.

s162
00:12:29.360 --> 00:12:30.560
And we'll see the same pattern.

s163
00:12:30.560 --> 00:12:37.440
It's going to be significantly cheaper overall across the entire session than going straight to opus.

s164
00:12:37.440 --> 00:12:40.399
So we'll let this finish here.

s165
00:12:42.959 --> 00:12:43.279
Okay.

s166
00:12:43.279 --> 00:12:44.480
And that finished.

s167
00:12:44.480 --> 00:12:59.040
Let's just queue up one more, write some documentation in a readme, and then we'll compare the total session cost.

s168
00:13:03.519 --> 00:13:03.920
Okay.

s169
00:13:03.920 --> 00:13:08.320
And as this runs, we'll wait and see what it does.

s170
00:13:09.600 --> 00:13:20.959
Okay, it created the read me and if we look here, we can see that the total session cost for the router was 14 while the total session cost for Opus was 44.

s171
00:13:20.959 --> 00:13:24.480
So at this point, we can see the cost is significantly lower.

s172
00:13:24.480 --> 00:13:29.440
Latency is optimized per step and the quality remains pretty similar across.

s173
00:13:29.440 --> 00:13:35.200
So you can see as you scale this, the cost performance really add up.

s174
00:13:35.200 --> 00:13:38.560
Okay, Archa, back to you.

s175
00:13:41.379 --> 00:13:43.379
[applause]

s176
00:13:44.800 --> 00:13:46.320
Thank you so much, Tyler.

s177
00:13:46.320 --> 00:13:49.760
And that that was actually a live demo that we ran here.

s178
00:13:49.760 --> 00:13:53.600
So, thanks to Tyler for setting it up and taking us through that.

s179
00:13:53.600 --> 00:13:58.560
So, now that you've seen it work, let's look at some quick facts.

s180
00:13:58.560 --> 00:14:03.279
Routing decision and under 200 milliseconds per request.

s181
00:14:03.279 --> 00:14:08.320
It runs on a custom mixture of experts model purpose-built for routing.

s182
00:14:08.320 --> 00:14:16.720
Zero application code changes needed from you to get it to adopt and it's free and included so you do not have to roll out your own router.

s183
00:14:16.720 --> 00:14:19.440
And we open source the whole routing model via Plano.

s184
00:14:19.440 --> 00:14:23.440
So you can actually check how that looks as well.

s185
00:14:24.399 --> 00:14:29.360
The last thing I wanted to talk about was a bit about um routing is the foundation layer.

s186
00:14:29.360 --> 00:14:31.440
It's not really the destination.

s187
00:14:31.440 --> 00:14:34.959
And there are three things that we usually build on top of it.

s188
00:14:34.959 --> 00:14:41.600
The first one is eval to prove that the right model works with your use case and your test well.

s189
00:14:41.600 --> 00:14:47.760
Caching so that you can stop paying twice or more for the same answer each time.

s190
00:14:47.760 --> 00:14:53.279
And personalization so that the router learns what works for your team over time.

s191
00:14:53.279 --> 00:14:56.959
This is a continuous improvement loop maturing over time.

s192
00:14:56.959 --> 00:15:03.360
That means that the more you route and evaluate, the better the router does for your workload.

s193
00:15:04.240 --> 00:15:07.279
So to summarize, where does this leave you?

s194
00:15:07.279 --> 00:15:09.199
There is no single best model.

s195
00:15:09.199 --> 00:15:15.199
There's only the right model for the request and benchmarks will only tell you part of the story.

s196
00:15:15.199 --> 00:15:17.600
Your preferences will tell you the rest.

s197
00:15:17.600 --> 00:15:24.560
And we built the router to honor your preferences and stay open so that you're never locked into a single stack.

s198
00:15:24.560 --> 00:15:27.040
And that's how teams actually built.

s199
00:15:27.040 --> 00:15:30.160
We are digital ocean and AI native cloud.

s200
00:15:30.160 --> 00:15:33.440
Come find us at the booth and route your next workload with us.

s201
00:15:33.440 --> 00:15:36.005
Thank you so much for being here.

s202
00:15:36.005 --> 00:15:38.005
[applause]

s203
00:15:50.415 --> 00:15:52.415
[music]