WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=QHBjufYK8TA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/QHBjufYK8TA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:13.640
have been really exciting.

s3
00:00:13.640 --> 00:00:21.840
We've tried to get a bunch of the industry leaders together to talk about some of the problems that are that we're facing as we try to run more on local.

s4
00:00:21.840 --> 00:00:26.120
If you guys were here for the first panel, one of the things that we talked about was model routing.

s5
00:00:26.120 --> 00:00:29.640
We firmly believe that we're in a multi-model world.

s6
00:00:29.640 --> 00:00:32.320
I think you heard this from many of the panelists.

s7
00:00:32.320 --> 00:00:37.600
Anyone who is deploying AI in production and who is doing so locally is seeing that multi-model world.

s8
00:00:37.600 --> 00:00:40.160
That's why we released these NeMo Triton models at NVIDIA.

s9
00:00:40.160 --> 00:00:44.920
Everything is released from the data sets to the weights with recipes so that you can customize them.

s10
00:00:44.920 --> 00:00:49.440
We do that because we know that people customizing models is going to be huge.

s11
00:00:49.440 --> 00:00:54.080
And so, this panel is really exciting cuz we're going to talk specifically about model routing.

s12
00:00:54.080 --> 00:01:00.880
So, as you are picking which model to use, how does that how essentially how does that tooling itself look?

s13
00:01:00.880 --> 00:01:03.000
Um, do you guys want to introduce yourselves?

s14
00:01:03.000 --> 00:01:03.720
Yeah, sure.

s15
00:01:03.720 --> 00:01:04.519
Um, I'm Walden.

s16
00:01:04.519 --> 00:01:05.960
I'm the co-founder of Cognition.

s17
00:01:05.960 --> 00:01:08.600
We build Devin, AI software engineer.

s18
00:01:08.600 --> 00:01:15.360
Uh, in addition to the product, we spend a lot of time partnering with our customers to figure out how they should deploy these models uh and these agents.

s19
00:01:15.360 --> 00:01:29.240
And one of the things they're constantly asking us nowadays is basically how do I know the ROI of our models and how do I know which tasks I can actually let our engineers spend the most expensive models on versus, you know, letting them use a more cost-efficient model.

s20
00:01:29.240 --> 00:01:32.520
And so, that's why we're we're also thinking a lot more about multi-model model routing nowadays.

s21
00:01:32.520 --> 00:01:33.440
Totally.

s22
00:01:33.440 --> 00:01:33.840
Yeah.

s23
00:01:33.840 --> 00:01:34.360
I'm Carter.

s24
00:01:34.360 --> 00:01:39.400
You guys heard from me a little bit earlier, but if you weren't here, I'm a developer tech engineer at NVIDIA.

s25
00:01:39.400 --> 00:01:46.320
And uh ultimately, I spend a lot of time thinking about how to get intelligence into as many developers' hands as possible.

s26
00:01:46.320 --> 00:01:56.800
And uh something that is continually becoming a not an issue, but something that is top of mind for a lot of developers is uh as you use more intelligence and the frontier models get more expensive,

s27
00:01:56.800 --> 00:02:04.440
uh it becomes somewhat cost prohibitive to to use the best uh tools, what feels like the best tools, as much as you would like to use them.

s28
00:02:04.440 --> 00:02:21.120
And so, this has become a recent uh you know, focus is, how can we, you know, still get the same desired outputs, but actually uh both as an individual developer, but also imagine startups and small companies, how can you leverage this incredible tool without totally breaking the bank?

s29
00:02:21.320 --> 00:02:22.480
Uh I'm Donne.

s30
00:02:22.480 --> 00:02:31.400
I have worked on model evaluations uh both in terms of its accuracies and efficiency and cost understanding of the model.

s31
00:02:31.400 --> 00:02:37.840
Uh and then I try and understand those implement those learnings and uh help build a router.

s32
00:02:37.840 --> 00:02:52.560
So, it's basically my job is to understand the behavior of of the model on an intimate level and then use those learnings to both improve the model and try and design a system of model that can work together with each other.

s33
00:02:52.560 --> 00:02:53.440
Totally.

s34
00:02:53.440 --> 00:02:58.040
Um yeah, I love a lot of the research that you're doing at Nvidia as we kind of see the space through.

s35
00:02:58.040 --> 00:03:01.800
I think what's really interesting is model routing itself is pretty new still.

s36
00:03:01.800 --> 00:03:05.680
And so, what you'll notice is there isn't a very clear solution here.

s37
00:03:05.680 --> 00:03:15.400
That was something that came up on the first panel is that there is a lot of space for startups and for companies in the ecosystem to fill in a solution here cuz we're still figuring out how to best do these patterns.

s38
00:03:15.400 --> 00:03:17.320
And I think, Walden, I want to kind of ask you.

s39
00:03:17.320 --> 00:03:21.320
So, um Cognition just released Fusion, your guys's model router.

s40
00:03:21.320 --> 00:03:28.440
And when you guys released it, you in your blog said that you're actually getting better performance than Fable, um than these frontier models.

s41
00:03:28.440 --> 00:03:40.720
And I feel like that was a very surprising statement to hear because we were thinking that you're getting as good or close enough usually when we're running on edge, when we're running local in these compute strained, smaller footprint models.

s42
00:03:40.720 --> 00:03:41.760
But you guys are getting better.

s43
00:03:41.760 --> 00:03:43.880
Can you explain how?

s44
00:03:43.880 --> 00:03:44.760
Yeah, absolutely.

s45
00:03:44.760 --> 00:03:54.840
Um so, I also want to be be clear about something here is like, you know, we're not saying that that we gap above Fable level performance in in the same way that maybe Fable level performance gaps above other models.

s46
00:03:54.840 --> 00:04:03.320
I think actually there's this really unintuitive dynamic where smarter models actually get better and better at delegating work.

s47
00:04:03.320 --> 00:04:13.760
And so one of the philosophies we had with building a model router is we don't want to route people to a dumber model and then suddenly you're stuck with a model that doesn't know how to do your task.

s48
00:04:13.760 --> 00:04:18.640
Next thing you know you're you're switching yourself back to smarter model anyways and now taking that expensive cost.

s49
00:04:18.640 --> 00:04:24.480
And in general we think a lot of the existing model routing systems out there are probably the same ones people have been using like a year ago.

s50
00:04:24.480 --> 00:04:33.880
And so we really wanted to put out a new framework that actually lets people still feel like and and still have a frontier model in their system while getting all these like cost benefits.

s51
00:04:33.880 --> 00:04:37.520
So yeah we we're reducing the cost of Fable level intelligence by 40%.

s52
00:04:37.520 --> 00:04:46.919
The way we do that is we allow Fable to still do like the planning and the the hard decision making but delegate a lot of the work to an implementation model.

s53
00:04:46.919 --> 00:04:51.560
And the implementation model can be you know one of these open source models be it a cheaper mini model.

s54
00:04:51.560 --> 00:05:00.480
Um the unintuitive thing is even though it's cheaper because you're delegating the work to another model you can let that model go at the task with much more depth and intensity

s55
00:05:00.480 --> 00:05:01.320
than you might otherwise.

s56
00:05:01.320 --> 00:05:10.360
Like you can spin off like you know three sub agents to go and like explore the code base and maybe that's actually more comprehensive than if you had just let Fable explore the code base itself.

s57
00:05:10.360 --> 00:05:15.760
So you're actually getting this like nice trade-off where it's both more cost efficient and it's also more comprehensive overall.

s58
00:05:15.760 --> 00:05:16.080
Interesting.

s59
00:05:16.080 --> 00:05:16.280
I see.

s60
00:05:16.280 --> 00:05:26.840
So you're saying by using a bunch of smaller models you're essentially like for one example scouring the code base you can you you can explore it potentially better than if you were to just have one model

s61
00:05:26.840 --> 00:05:31.520
I don't know figure out with what it's limited with its limited context with whatever path it's on.

s62
00:05:31.520 --> 00:05:32.400
Yeah, totally.

s63
00:05:32.400 --> 00:05:45.360
But also if you think about the budget of if you were to say the frontier model costs this amount per token and the smaller model is this amount per per token and it's significantly cheaper then you can use a lot more tokens from the smaller model

s64
00:05:45.360 --> 00:05:45.600
with

s65
00:05:45.600 --> 00:05:48.560
still within the budget that it would have been from the Frontier model.

s66
00:05:48.560 --> 00:05:54.240
I would also like to encourage everyone to think there are jagged capabilities in most models, right?

s67
00:05:54.240 --> 00:05:56.440
So, coding is not one domain.

s68
00:05:56.440 --> 00:06:02.920
Within, let's say data visualization, there'll be scikit-learn, there'll be matplotlib, there'll be something else.

s69
00:06:02.920 --> 00:06:10.400
It largely comes down comes down to the training corpora that went into each of the models, so right?

s70
00:06:10.400 --> 00:06:19.960
So, one model, while you're trying to do X type of work, let's say data visualization, and the other type is Y, that means let's say model building.

s71
00:06:19.960 --> 00:06:29.440
Let's say you're trying to have a data science work stream, where you're trying to optimize for some kind of prediction, and then visualizing your results.

s72
00:06:29.440 --> 00:06:33.360
Within that task, different models will have different strengths.

s73
00:06:33.360 --> 00:06:42.480
So, not it's not necessary that model A if scores higher on a coding benchmark, is just plain better at every task that is.

s74
00:06:42.480 --> 00:06:54.520
So, routing is a task of intimately intimately understanding of behavior of and strengths and weaknesses of different models, and then applying them thusly, right?

s75
00:06:54.520 --> 00:07:01.960
I would I would encourage everyone to think, "Hey, models are strong at different things, rather than like there's one model to rule them all."

s76
00:07:01.960 --> 00:07:02.480
I see.

s77
00:07:02.480 --> 00:07:04.480
And by the way, real quick, thank you Alex for joining.

s78
00:07:04.480 --> 00:07:05.080
Yeah.

s79
00:07:05.080 --> 00:07:05.800
Sorry I'm late.

s80
00:07:05.800 --> 00:07:07.919
No. Oh, is that so?

s81
00:07:07.919 --> 00:07:09.280
I might need yours in.

s82
00:07:09.280 --> 00:07:10.339
Sorry I'm late.

s83
00:07:10.339 --> 00:07:11.120
[laughter]

s84
00:07:11.120 --> 00:07:13.200
I'm Alex from OpenRouter.

s85
00:07:13.200 --> 00:07:14.040
Um thanks for having me now.

s86
00:07:14.040 --> 00:07:14.640
Yeah, of course.

s87
00:07:14.640 --> 00:07:15.160
Thank you so much.

s88
00:07:15.160 --> 00:07:17.200
You came right from the airport, so this is perfect.

s89
00:07:17.200 --> 00:07:19.080
I think um tonight that's that's super interesting.

s90
00:07:19.080 --> 00:07:34.440
So, um the way that you're thinking through through model routing, it's not even just uh delegating to necessarily a smaller model, but like and maybe this is kind of what you're saying is can you can you put essentially a swarm of agents to accomplish the same task, and suddenly routing the task between them is is much is

s91
00:07:34.440 --> 00:07:36.680
a is a problem to solve in and of itself.

s92
00:07:36.680 --> 00:07:37.160
Yeah.

s93
00:07:37.160 --> 00:07:39.720
So, if you look at like let's say let's take an easy example.

s94
00:07:39.720 --> 00:07:44.200
Let's take a science or like scientific discovery as an example, right?

s95
00:07:44.200 --> 00:07:46.120
Usually these are one-shot problems.

s96
00:07:46.120 --> 00:07:47.840
It's incredibly hard.

s97
00:07:47.840 --> 00:07:51.440
You have models think through this process, right?

s98
00:07:51.440 --> 00:07:55.400
So, in that you have tons of sub-domains.

s99
00:07:55.400 --> 00:07:57.120
Like tons and tons and tons.

s100
00:07:57.120 --> 00:08:06.120
So, in that aspect, if you think about post-training like the post-training process of a model, they'd be tuned with different teachers.

s101
00:08:06.120 --> 00:08:09.000
They'd be tuned on different sub-tasks.

s102
00:08:09.000 --> 00:08:20.720
So, those those um those overlapping strengths will be readily apparent when you're trying to understand failures of each models on different different sub-tasks.

s103
00:08:20.720 --> 00:08:27.919
Once you understand that, you can orchestrate your system to leverage that arbitrage essentially, and that essentially becomes free.

s104
00:08:27.919 --> 00:08:31.520
So, I think this is on LM router bench.

s105
00:08:31.520 --> 00:08:34.200
There was there are tons of benchmarks out there.

s106
00:08:34.200 --> 00:08:39.360
But, if you use these techniques, you can get like up to 10% higher accuracy even, right?

s107
00:08:39.360 --> 00:08:40.760
It depends on the model pool.

s108
00:08:40.760 --> 00:08:42.440
Depends on the task at hand.

s109
00:08:42.440 --> 00:08:48.280
But, I would encourage to think about the complementary nature of models.

s110
00:08:48.280 --> 00:08:48.800
I see.

s111
00:08:48.800 --> 00:08:57.480
Do you see so in kind of the way that you were describing the way that the task is broken up, do you see that the some of the smaller models because the token cost is cheaper,

s112
00:08:57.480 --> 00:08:58.800
are they using more tokens?

s113
00:08:58.800 --> 00:09:03.880
Like is it Are Are you specifically routing so that they do or that they so that they are chattier?

s114
00:09:03.880 --> 00:09:07.400
Uh oh yeah, they absolutely do use more tokens.

s115
00:09:07.400 --> 00:09:19.240
I I actually I want to kind of like riff on something that Tuhin was saying, which is like you know, a lot of times when you look at these different benchmarks, you'll see that the small models will perform better than like even the frontier models in certain cases.

s116
00:09:19.240 --> 00:09:28.160
I think a lot of people they look at this and they immediately jump to, "Oh, how can we just route like, you know, the task where the small models do better just straight to the smaller models?"

s117
00:09:28.160 --> 00:09:40.560
I I think that one of the things we really want to emphasize with our recent blog post and recent Devin Fusion was that this kind of like naive like initial routing to based on the task type is extremely fragile,

s118
00:09:40.560 --> 00:09:43.000
especially the more agentic the task you you work on is.

s119
00:09:43.000 --> 00:09:48.320
So, for example, like a real developer, you might ask your agent first, "Oh, how does this code base work?"

s120
00:09:48.320 --> 00:09:51.640
And then you you go deeper and you're like, "Okay, actually, can you implement some features for me?"

s121
00:09:51.640 --> 00:09:56.280
And then you go deeper and you're like, "Oh, can you like now go to like a live test of this feature and debug deep cases?"

s122
00:09:56.280 --> 00:10:04.880
The complexity changes and the type of task changes over time, and you don't want to be left with some subpar model for the the task that you're now on.

s123
00:10:04.880 --> 00:10:15.640
I think this is why people like frontier models so much is they're they're just like generally intelligent and they're capable of shifting between various different domains, even if you can eke out better performance in very specific tasks.

s124
00:10:15.640 --> 00:10:26.320
Um, and and what So, the challenge is, how do you get a small model to know that it's out of its depth and you need to now like go switch to another model or go like, you know, go to a smart smarter model.

s125
00:10:26.320 --> 00:10:33.240
And and our solution to this is you kind of just always have this like main frontier agent that's watching, even if it's not the one doing the work.

s126
00:10:33.240 --> 00:10:38.000
It should at least be keeping tabs to figure out, "Okay, we like the the agent I delegated to now is like out of its depth.

s127
00:10:38.000 --> 00:10:40.440
I need to kind of like move it to to something else."

s128
00:10:40.440 --> 00:10:48.640
Um, and and overall, just the guarantee of always having frontier intelligence present, I think reduces the the fragility of of these systems quite a lot.

s129
00:10:48.640 --> 00:10:59.360
How does the sharing of context between one of those smaller agents who has basically completed up to some level of a task and decides, "Actually, I don't think I'm the right person for this.

s130
00:10:59.360 --> 00:11:01.720
I need to hand it back to the the foundational model."

s131
00:11:01.720 --> 00:11:05.160
Um, of course, you don't want to have the entire trace of that

s132
00:11:05.160 --> 00:11:05.440
Yeah.

s133
00:11:05.440 --> 00:11:07.880
smaller agent be passed back to the larger model.

s134
00:11:07.880 --> 00:11:13.920
But so, how do you get that level of specificity while basically providing the information it needs, but not more?

s135
00:11:13.920 --> 00:11:14.480
Yeah, absolutely.

s136
00:11:14.480 --> 00:11:25.200
So, I I I think like the the context here is it's actually very easy to actually create a system that's more expensive as soon as you're running like, you know, multiple models together cuz oh, no, like, you know, this one file reading.

s137
00:11:25.200 --> 00:11:27.680
Now, every every one of these models is now reading this one file reading.

s138
00:11:27.680 --> 00:11:30.200
So, now you're you're you're being charged like three times as much.

s139
00:11:30.200 --> 00:11:37.560
Um, the the the the trick that we we spent a lot of our time on is um, most of the context by default will only be going to like one model.

s140
00:11:37.560 --> 00:11:39.840
So, like most of the context let's say will be going to the small model.

s141
00:11:39.840 --> 00:11:47.880
But, the thing you need to then tune very well is okay, like maybe you still show like what files it's reading, maybe you show like the high-level thinking of what it's doing back to the main model.

s142
00:11:47.880 --> 00:11:53.440
Maybe you have the small model um, you tune its ability to present the context back to the main model.

s143
00:11:53.440 --> 00:12:03.440
Um, and actually a lot of these problems already have been well studied in many domains already like context compaction is something you already have to solve if you want to do like really long-running agents.

s144
00:12:03.440 --> 00:12:13.320
And so, this problem of taking long context compacting it in a way that is now understandable is the one that you can also apply to this domain and just kind of give the compacted context back to back to the main agent.

s145
00:12:13.320 --> 00:12:27.000
Context compaction is something that you know, I'm familiar with but I hadn't really thought about as you're doing model routing and as you're trying to share context across now potentially many models, you're you're expanding the amount of what could be seen as wasteful tokens or redundant

s146
00:12:27.000 --> 00:12:30.360
tokens just just because you have to process that across the many models.

s147
00:12:30.360 --> 00:12:31.240
Yeah.

s148
00:12:31.240 --> 00:12:31.840
Yeah.

s149
00:12:31.840 --> 00:12:37.880
Um, I I I think there's the way I describe it is I think we are early in in the in the model routing domain.

s150
00:12:37.880 --> 00:12:49.560
Um, I I I hope that a year from now that even the techniques we kind of use for Devin Fusion you people look back on that and are like, "Oh, these are some like really legacy ideas and and now we have like much better methods at routing between models."

s151
00:12:49.560 --> 00:12:55.760
Um, and when people actually start co-designing their models with this in mind we're going to be be in a much better world.

s152
00:12:55.760 --> 00:13:06.960
Yeah, I I echo what you said that um, I think routing will evolve as the task evolves when when you start task, right?

s153
00:13:06.960 --> 00:13:21.600
So, um, it's more useful to see things in terms of subtasks and sessions uh, than individual problems that you're trying to solve because more than likely you're when you're working through a problem you're asking a lot of questions, you're you're exploring different things.

s154
00:13:21.600 --> 00:13:38.200
Um, and it is imperative that you try like people who design routers is imperative that they try and understand these phases of different complexities and then try and apply some logic

s155
00:13:38.200 --> 00:13:48.920
for essentially side kicking tasks or leveraging expertise from other models that's that's pretty on point.

s156
00:13:48.920 --> 00:13:51.859
Yeah, I'd love to hear from the the router guy.

s157
00:13:51.859 --> 00:13:53.320
[laughter]

s158
00:13:53.320 --> 00:14:04.720
Yeah, um I think those are these are like important points and one of the biggest debates I think we have internally is whether that outer model that's doing the orchestration

s159
00:14:04.720 --> 00:14:08.000
should be the big model or the small model.

s160
00:14:08.000 --> 00:14:20.520
You get like very different results depending on your choice and in fact and it's not even clear what the pricing impact would be because if your outer model that's doing the orchestration is the big model it can leverage its caching

s161
00:14:20.520 --> 00:14:29.760
to like make more of its decisions and it's caching is going to be like a dramatic price savings compared to the small models caching a lot of the time

s162
00:14:29.760 --> 00:14:41.160
especially for like perform for you know issues that are on the bright line like zooming out a little bit I think the what you want from all the models out there when you do model fusion

s163
00:14:41.160 --> 00:14:56.800
is to benefit from all the data that is being trained on across all the labs and and not just the data from one lab right or one source and a model is just like a combination of like the data

s164
00:14:56.800 --> 00:15:12.280
and its understanding of the data both its compute and the quality of its RL so long term I think you want you want models where they know that oh this is like in distribution

s165
00:15:12.280 --> 00:15:23.720
like this is in my data you can use small models pretty easily and get a cost savings but if it's out of distribution small models may actually increase your cost because of how often

s166
00:15:23.720 --> 00:15:26.360
they'll like call tools and how crazy their loops will be.

s167
00:15:26.360 --> 00:15:40.840
Like if you run terminal bench on Opus and Haiku, like Opus will do about three times better at 1/10 the cost of Haiku, even though Haiku's significantly cheaper per token.

s168
00:15:40.840 --> 00:15:48.960
So it really becomes a huge problem if you use a too small of a model, particularly on tasks that are out of domain for the train data.

s169
00:15:48.960 --> 00:15:54.920
When you're doing something like classifying text, like hey, is this like a person's name or is this an organization's name?

s170
00:15:54.920 --> 00:15:56.800
That's super in domain.

s171
00:15:56.800 --> 00:16:00.200
So you would you don't want that kind of task to go to a large model.

s172
00:16:00.200 --> 00:16:01.960
You want it to go to a a small model.

s173
00:16:01.960 --> 00:16:03.920
Everyone has that in their domain.

s174
00:16:03.920 --> 00:16:12.400
Um so deep like being able to understand in domain, out of domain is a lot of work that we're doing for for um open router fusion.

s175
00:16:12.400 --> 00:16:18.880
And uh and then also figuring out like what like how to orchestrate the outer and inner models um for different types of task.

s176
00:16:18.880 --> 00:16:22.600
And like um yeah, it's it's an early industry.

s177
00:16:22.600 --> 00:16:24.839
It's like an early field of research.

s178
00:16:24.839 --> 00:16:32.240
Uh most research on model fusion has not been very detailed, not been very like, you know, optimistic sometimes.

s179
00:16:32.240 --> 00:16:36.839
Um it's only like just, you know, recently getting uh more optimistic.

s180
00:16:36.839 --> 00:16:40.160
And I think um I'm personally very optimistic about it.

s181
00:16:40.160 --> 00:16:46.240
And uh you know, like we we're very like ecosystem-driven, collaborative company.

s182
00:16:46.240 --> 00:16:57.280
And a lot of our like we work with a lot of partners to try like help improve their orchestration pipelines with good primitives, like the, you know, sub-agent and like the advisor tool,

s183
00:16:57.280 --> 00:16:59.280
um which is kind of similar to what you were talking about.

s184
00:16:59.280 --> 00:16:59.920
I'm curious.

s185
00:16:59.920 --> 00:17:07.680
So uh help me understand when uh if a it makes total sense that um a small model, if it's in domain, would be cheaper.

s186
00:17:07.680 --> 00:17:12.400
But if it's not, then it's going to thrash around uh as it tries to get an answer.

s187
00:17:12.400 --> 00:17:22.520
When you're describing whether the uh con- like the main agent should be the the the local model or the cloud model?

s188
00:17:22.520 --> 00:17:29.440
Does that is that a decision that's then dependent on whether the task is something that it that's in domain or not?

s189
00:17:29.440 --> 00:17:30.520
Does my question kind of make sense?

s190
00:17:30.520 --> 00:17:32.040
I don't know.

s191
00:17:32.040 --> 00:17:36.240
Um, I basically I don't it's kind of early to

s192
00:17:36.240 --> 00:17:38.440
um to say.

s193
00:17:38.440 --> 00:17:47.400
I I think the I wait the results that we published, which are a couple weeks ago, which were focused on deep research, not coding.

s194
00:17:47.400 --> 00:17:47.720
Mhm.

s195
00:17:47.720 --> 00:17:56.560
Um we had the smart model be the wrapper model, be the outer model, and we got the best results from doing that.

s196
00:17:56.560 --> 00:18:01.400
Um, but it for deep research it works the best.

s197
00:18:01.400 --> 00:18:05.280
Um, for other tasks, it's like kind of unclear.

s198
00:18:05.280 --> 00:18:09.360
Like we there it's fusion is not super well optimized for coding.

s199
00:18:09.360 --> 00:18:21.480
And uh it might be that like a smaller model ends up being like a higher um efficiency per you know, fewer dollars um per like completed successfully completed task,

s200
00:18:21.480 --> 00:18:22.600
but it's kind of early to say.

s201
00:18:22.600 --> 00:18:23.032
Yeah.

s202
00:18:23.032 --> 00:18:23.280
[clears throat]

s203
00:18:23.280 --> 00:18:27.440
One thing you said earlier is like, oh, you get the caching benefit from like the main line agent.

s204
00:18:27.440 --> 00:18:30.280
Um, you actually can get the caching benefit from the side agent.

s205
00:18:30.280 --> 00:18:40.760
And um, this is actually one of the key things we talked about with our Devin fusion launch is that um, you kind of are leaving a lot on the table if you do a main agent and sub agents type system.

s206
00:18:40.760 --> 00:18:42.840
Um, so we don't use sub agents.

s207
00:18:42.840 --> 00:18:49.120
We use what we call a sidekick, which is um, one sub agent that continually has a running context.

s208
00:18:49.120 --> 00:18:52.680
So the main agent doesn't need to re-provide uh, context from earlier.

s209
00:18:52.680 --> 00:18:54.720
Um, it's all still in the KB cache, right?

s210
00:18:54.720 --> 00:18:57.640
Like it's 10 times cheaper on on all those cache tokens.

s211
00:18:57.640 --> 00:19:08.320
Um, and then if you want to like switch the smart model to be like the one on the side or the one in charge, um it's actually totally fine, and you can kind of like do do the swapping back and forth.

s212
00:19:08.320 --> 00:19:15.280
We're also spending a lot of time right now thinking about how do you train models to actually work collaboratively with other models?

s213
00:19:15.280 --> 00:19:19.960
I actually I think there's a lot of literature out there on how you RL one model to do a task end-to-end.

s214
00:19:19.960 --> 00:19:22.880
How can you RL model to also be good at collaboration?

s215
00:19:22.880 --> 00:19:32.680
And when we think about it, we actually try both of these setups where let's RL the model being the orchestrator and the one deciding what gets delegated to other models, see how well that performs.

s216
00:19:32.680 --> 00:19:42.120
Um and we also orchestrate it in a way where the model we're training is actually the executor, the sidekick, and see how well is it at executing other models' instructions.

s217
00:19:42.120 --> 00:19:53.400
Um and we expect that to to be a probably a big lift in this next step of like multi-model orchestration is don't just like take models as they are and orchestrate them, but like can you actually co-design your models with the orchestration system?

s218
00:19:53.400 --> 00:19:54.000
Yeah, that makes sense.

s219
00:19:54.000 --> 00:20:00.360
I mean, with Nemo Tron and you know, with all the foundational models where um we're essentially post-training them for the harnesses that they're getting used.

s220
00:20:00.360 --> 00:20:05.400
If the harness is going to include a lot of routing, then that makes sense that makes its way in to the post-training.

s221
00:20:05.400 --> 00:20:05.880
Yeah.

s222
00:20:05.880 --> 00:20:12.240
Um are you guys thinking a lot about the kind of like the model training at at Nvidia for this kind of purposes?

s223
00:20:12.240 --> 00:20:15.440
Yeah, so we have a technology called

s224
00:20:16.960 --> 00:20:18.000
The mic's go?

s225
00:20:18.000 --> 00:20:18.960
Right.

s226
00:20:18.960 --> 00:20:20.040
Hello.

s227
00:20:20.040 --> 00:20:20.600
Testing.

s228
00:20:20.600 --> 00:20:22.244
Oh, this one works.

s229
00:20:22.244 --> 00:20:23.520
[laughter]

s230
00:20:23.520 --> 00:20:24.040
Okay.

s231
00:20:24.040 --> 00:20:25.800
Uh so we have a technology called Flex Run.

s232
00:20:25.800 --> 00:20:36.080
So you have uh we we have a setup where uh there's a the uh there's a main model, then we distill it into smaller uh footprints.

s233
00:20:36.080 --> 00:20:43.200
And then based on the based on the task at hand, you can switch which model does the decoding.

s234
00:20:43.200 --> 00:20:43.480
Right?

s235
00:20:43.480 --> 00:20:49.040
So there's a lot of fancy stuff you can do uh within a model artifact, too.

s236
00:20:49.040 --> 00:20:59.360
Uh to essentially only activate a class of model or a section of weights, depending on the task at hand or the complexity at hand.

s237
00:20:59.360 --> 00:21:11.240
Most In most cases, you can essentially understand the novelty of a question to a model if you have access to the recipe with which it was trained.

s238
00:21:11.240 --> 00:21:13.360
So, this works very well for open models, right?

s239
00:21:13.360 --> 00:21:16.280
Like or any model you have access to its data for, right?

s240
00:21:16.280 --> 00:21:21.280
Because you can literally decide if it's in like see if it's in distribution or not.

s241
00:21:21.280 --> 00:21:34.120
Uh again, if you have studies from when it was trained, you can also see how much essentially how much was your distillation gap across teachers and the artifact that you trained, right?

s242
00:21:34.120 --> 00:21:44.440
Because sure, you have domain data from all all the different domains you're tuning, but it's not guaranteed that it uh absorbed all the that data evenly across the models, right?

s243
00:21:44.440 --> 00:21:56.760
So, um it becomes it becomes very interesting uh to start thinking about these flexible weights and flexible model sizes essentially.

s244
00:21:56.760 --> 00:21:59.640
There's also I wanted to add with the context piece, right?

s245
00:21:59.640 --> 00:22:06.600
So, how do you think about ASTs and and context compression representations?

s246
00:22:06.600 --> 00:22:10.680
Uh compaction in its very nature is lossy, right?

s247
00:22:10.680 --> 00:22:14.960
So, just like headroom is there, RTK is there, right?

s248
00:22:14.960 --> 00:22:28.080
Uh these these spaces code bases are usually designed to have representations that we carry forward through life and you essentially give models the capability to further expand on them.

s249
00:22:28.080 --> 00:22:35.920
It's it's more more like loss less-ish compression, which can retain states of a models or states of agents.

s250
00:22:35.920 --> 00:22:37.520
What do you think about that?

s251
00:22:37.520 --> 00:22:43.000
Yeah, I think this gets to like kind of like a fundamental philosophy of how agents and context should work.

s252
00:22:43.000 --> 00:22:48.920
One exercise I like to do is like you know, as a human, like how many numbers can you like if I just start spitting out numbers now, right?

s253
00:22:48.920 --> 00:22:53.200
Like how many can you remember before, you know, like you start losing track of them.

s254
00:22:53.200 --> 00:22:54.240
I think it's actually very few, right?

s255
00:22:54.240 --> 00:22:58.400
So, in some ways you could argue that your context window is actually shorter than these language models.

s256
00:22:58.400 --> 00:23:01.320
And yet, you can actually be very effective at that, right?

s257
00:23:01.320 --> 00:23:03.480
Your context is is very lossy.

s258
00:23:03.480 --> 00:23:10.280
Um, I think one of the nice things that people are starting to realize with agents is like you have a lot of non-lossy systems that you can fall back to.

s259
00:23:10.280 --> 00:23:11.600
So, you have a file system.

s260
00:23:11.600 --> 00:23:16.720
Like uh if in your memory all your memory is that you read some file earlier, you don't need to remember the whole file.

s261
00:23:16.720 --> 00:23:21.560
You maybe remember the important parts, but you can still have the full version of the file on your system.

s262
00:23:21.560 --> 00:23:35.600
And that's kind of my goal when I'm thinking about how do we build a good context engineered harness is the harness should have everything in needs to find what it needs to have even if it doesn't have everything immediately available.

s263
00:23:35.600 --> 00:23:45.080
In that case, uh do you think that the context sharing problem will become cheaper and less of a problem in future?

s264
00:23:45.080 --> 00:23:47.080
Yeah, it's definitely possible as well.

s265
00:23:47.080 --> 00:23:51.200
I've seen cases where um you know, the psychic agent does a bunch of work.

s266
00:23:51.200 --> 00:23:54.800
It tells the main model, "Oh yeah, like here's here's all the things I found."

s267
00:23:54.800 --> 00:23:57.840
Instead of dumping the full thing, it just references them by file.

s268
00:23:57.840 --> 00:24:05.360
And then the main model is actually generally you find these larger, smarter models, they're actually more token efficient with how they use tools and how they read.

s269
00:24:05.360 --> 00:24:09.400
And so, they actually read the files in a way where they only see the important parts, right?

s270
00:24:09.400 --> 00:24:12.080
Or they decide that, "Oh, actually I only need to look at a subset of this."

s271
00:24:12.080 --> 00:24:16.320
Or, "Oh, I can run a single command and just know if everything is done properly."

s272
00:24:16.320 --> 00:24:27.680
Um, it's actually quite amazing um the fact that you know, these multi-model systems they actually seem to scale and get better with intelligence, which is um not something we should just take for granted, right?

s273
00:24:27.680 --> 00:24:33.080
It's not obvious that actually more expensive models are actually creating an overall cheaper system.

s274
00:24:33.080 --> 00:24:40.240
Yeah, like the scaling laws uh if you have a larger model, it's going to be more efficient with its tokens.

s275
00:24:40.240 --> 00:24:43.720
Uh smaller models less efficient with its tokens.

s276
00:24:43.720 --> 00:24:45.880
I guess I had a question for you, Alex.

s277
00:24:45.880 --> 00:24:52.560
Uh do you guys uh like weigh more importance on uh the actual infrastructure side of routing?

s278
00:24:52.560 --> 00:25:05.920
So, for instance, KV cache aware routing or uh like this where most of the businesses right now Uh or the are you seeing strong pull off like people actually deploying routers in production?

s279
00:25:07.280 --> 00:25:13.480
So open router is a like a marketplace for language models.

s280
00:25:13.480 --> 00:25:22.840
We we exist at like we can't see into the KV caches of models unless we're running them ourselves, which is pretty rare.

s281
00:25:22.840 --> 00:25:32.040
Um we we we do spend a lot of time optimizing for cash hits and we we like pass through cash hits directly to users.

s282
00:25:32.040 --> 00:25:38.400
Um but like in terms of KV cash optimizations we we we can't do any of like specific work there.

s283
00:25:38.400 --> 00:25:51.440
What we do for um model routing is we we try to like find the best model or best combination of models for the prompt and then when we see a cash hit,

s284
00:25:51.440 --> 00:26:00.000
we will will like use up the duration of the cash and send the downstream customer like the the full savings of the cash hit.

s285
00:26:00.000 --> 00:26:11.240
Um there's more work that we can do here where we could say, okay, this looks like something where there's significant benefit to switching the model right now, but you haven't used up the full cash.

s286
00:26:11.240 --> 00:26:13.520
You still have like 2 minutes left.

s287
00:26:13.520 --> 00:26:25.120
Um and it we think it's probably worth switching the model and losing the rest of your cash um and letting people kind of like tweak their tolerance for that behavior.

s288
00:26:25.120 --> 00:26:29.703
Um we've we've been doing a little bit of that, but we haven't like exposed it to customers yet.

s289
00:26:29.703 --> 00:26:30.080
[clears throat]

s290
00:26:30.080 --> 00:26:30.520
Awesome.

s291
00:26:30.520 --> 00:26:33.200
What what is next for you guys in terms of your your your model routing?

s292
00:26:33.200 --> 00:26:39.360
Cuz as you mentioned, you know, it is kind of like a different direction from like the the marketplace business that exists today.

s293
00:26:39.360 --> 00:26:41.480
So I'd love to hear.

s294
00:26:41.480 --> 00:26:47.400
So I mean we've been doing like we we've had a an auto router for like 2 years almost.

s295
00:26:47.400 --> 00:26:51.320
Um but when we launched it, there was like no adoption of it.

s296
00:26:51.320 --> 00:26:54.880
It was people really wanted to use specific models.

s297
00:26:54.880 --> 00:26:58.880
Um and the the auto router just had like no real usage.

s298
00:26:58.880 --> 00:27:02.080
It was it we mostly saw it as like a discovery play point.

s299
00:27:02.080 --> 00:27:05.640
Like, hey, this is how you discover which model might be good for your prompt.

s300
00:27:05.640 --> 00:27:14.160
And then um at around like January this year with open claw, it exploded.

s301
00:27:14.160 --> 00:27:31.929
And the reason it exploded is because there's this fundamental um idiosyncrasy in open claw where it sends heartbeats every like 10 minutes to your model of choice just to see like if the you know, if the the the client was still active.

s302
00:27:31.929 --> 00:27:32.520
[laughter]

s303
00:27:32.520 --> 00:27:39.520
And that means that if you set Opus to be your default model, it would be like using a lot of tokens on this like heartbeat process.

s304
00:27:39.520 --> 00:27:49.480
And so um this was the very beginning of like a very popular app with two completely different intelligence needs, completely different.

s305
00:27:49.480 --> 00:27:57.880
And the models the open source models have improved to a point where like it makes sense to segment the market and at at least those two areas.

s306
00:27:57.880 --> 00:28:00.080
And so that's that was how it got started.

s307
00:28:00.080 --> 00:28:03.240
And then we saw a lot more segmentation blossom afterwards.

s308
00:28:03.240 --> 00:28:10.080
And so um and now a whole bunch of agents and apps on open router use the the different routers that we have.

s309
00:28:10.080 --> 00:28:11.400
We have a couple of them.

s310
00:28:11.400 --> 00:28:20.760
We have Pareto code which gives you the Pareto Pareto optimal model for coding tasks given a certain like threshold that you can tune.

s311
00:28:20.760 --> 00:28:25.960
Um we have fusion which orchestrates multiple models and gives you a fused result.

s312
00:28:25.960 --> 00:28:29.720
And um and then we'll have other experiments in the future.

s313
00:28:29.720 --> 00:28:39.400
What we want to do is basically like create good primitives that developers can use to like get really advanced with how they use model orchestration.

s314
00:28:39.400 --> 00:28:43.120
Um kind of like sidecar sort of like that.

s315
00:28:43.120 --> 00:28:52.240
Um but also give people like a really easy thing that they can just like set a slug to that works with all harnesses and just like gets the job done.

s316
00:28:52.240 --> 00:29:06.080
I feel like it's super interesting how much of a perfect storm there is for model routing right now because on one hand, you know, ignore agents, ignore open claw for a second, like just to squeeze better performance, it seems like we should be smarter about how we tackle problems.

s317
00:29:06.080 --> 00:29:07.200
That's obvious, right?

s318
00:29:07.200 --> 00:29:11.680
If you make a plan, if you make a strategy, that's a that's a better way to to go about your day.

s319
00:29:11.680 --> 00:29:19.200
So, I'm not surprised that you're going to see, you know, better code get written or more performing code get written and less buggy code get written if you break the problem down.

s320
00:29:19.200 --> 00:29:23.400
And so, routing specifically for that uh use case makes a ton of sense.

s321
00:29:23.400 --> 00:29:27.680
But then hearing this, yeah, like the profile of workloads changed with agents, right?

s322
00:29:27.680 --> 00:29:37.720
They're very they it went from like I ask questions, I get a response, then it went to reasoning where I ask questions, it reasons, and then it comes back, and then it went to uh yeah, this heartbeat, right?

s323
00:29:37.720 --> 00:29:43.600
Like if um if my agents are running optimally, uh there's a token being generated every second.

s324
00:29:43.600 --> 00:29:52.440
Um and suddenly that is its own need for model routing, and it feels like um hearing the different solutions uh to kind of tackle each of those is very interesting.

s325
00:29:52.440 --> 00:29:53.680
Even

s326
00:29:53.680 --> 00:29:59.600
Well, I was just going to say I think that yeah, the the use cases for model routing are are there are many of them.

s327
00:29:59.600 --> 00:30:05.680
And so, one could be getting a better answer, one could be, you know, saving money and trying to get the same answer.

s328
00:30:05.680 --> 00:30:13.040
One that we haven't even talked about yet, which is probably the most relevant maybe even to this crowd, is uh when do you want to actually run a model

s329
00:30:13.040 --> 00:30:18.800
locally versus when do you actually need something like a frontier model to to perform that task.

s330
00:30:18.800 --> 00:30:23.640
And that might be something to the effect of like uh for privacy protecting information.

s331
00:30:23.640 --> 00:30:28.320
You know, when I'm running some local, it can you detect that my prompt has sensitive information?

s332
00:30:28.320 --> 00:30:37.680
And if so, do that on on device, um but then maybe even anonymize some of that information to go do the more advanced workloads up on top of that information

s333
00:30:37.680 --> 00:30:38.800
in the cloud.

s334
00:30:38.800 --> 00:30:47.160
Um another example would be again for the cost savings, but it's like, "Hey, I bought this DGX Spark and I'm not I know I'm not at 100% utilization.

s335
00:30:47.160 --> 00:31:00.040
How can I make sure that as part of my workloads, whether it's the heartbeat and open claw or what have you, that I'm leveraging that compute um to the fullest of its ability because I'm only paying for the electrons that are coming in for my power bill, um but I'm paying full price for the tokens in the

s336
00:31:00.040 --> 00:31:00.760
cloud?"

s337
00:31:00.760 --> 00:31:10.400
And I think that's a whole other area of model routing that I know that we're doing some work with at NVIDIA um that I think will be will be really cool as it starts to uh the hybrid

s338
00:31:10.400 --> 00:31:14.120
of local and cloud starts to to really emerge as its own sector.

s339
00:31:14.120 --> 00:31:18.040
Yeah, I'd be curious to know what you guys take uh take is on.

s340
00:31:18.040 --> 00:31:21.920
Like if you self-host a model, the cost dynamics change, right?

s341
00:31:21.920 --> 00:31:29.200
You have a considerably higher cost at a higher context length because your throughput slows down as the context gets deeper.

s342
00:31:29.200 --> 00:31:38.720
So, rather than switching to a cheaper model, even if you have self-hosted models in data center, you can use compaction uh to bring your throughput back up.

s343
00:31:38.720 --> 00:31:41.360
Uh have you guys uh thought about this part of it?

s344
00:31:41.360 --> 00:31:47.720
Like compaction versus uh just routing because one is you have fewer tokens to work with.

s345
00:31:47.720 --> 00:31:50.800
One is we have cheaper tokens.

s346
00:31:50.800 --> 00:32:04.640
Yeah, I I think in practice um compacting alone doesn't solve the the cost or or throughput problems because a lot of times it's just like the uh the differential in like model intelligence and cost is just so big where also by the way, when you compact, you're taking a cache miss.

s347
00:32:04.640 --> 00:32:09.480
So, you're actually then now like paying 10 times as much for the for those input tokens if you didn't compact.

s348
00:32:09.480 --> 00:32:11.960
Um the main reason we compact is actually intelligence.

s349
00:32:11.960 --> 00:32:16.600
Um all all these like multipliers they advertise some like insane context window, like a million tokens.

s350
00:32:16.600 --> 00:32:22.440
Uh I would like never recommend using like these models past like 200K tokens, under 100K if you can.

s351
00:32:22.440 --> 00:32:26.760
Um the the the intelligence uh just kind of like falls off a cliff at some point.

s352
00:32:26.760 --> 00:32:28.680
Um sorry, Anthropic if you're if you're watching.

s353
00:32:28.680 --> 00:32:43.000
But I I think that uh you know, compaction is like a very useful tool if you are going to have to take a cache miss anyway um one way or another, like when you're routing to another model and you want to just like minimize the the window there.

s354
00:32:43.160 --> 00:32:46.760
Do you find that, um, in the sidecar,

s355
00:32:46.760 --> 00:32:47.120
Yeah.

s356
00:32:47.120 --> 00:32:58.520
um, when small models are generating lots of tokens, um, is that like an is that like one of the best reasons to switch it to a larger model?

s357
00:32:58.520 --> 00:33:08.080
Like, basically, when when small models generate lots of tokens, I wonder if that's like a box of root the root cause of of intelligence problems down the road.

s358
00:33:08.080 --> 00:33:13.280
You want your big model to generate the big token chunks, the small models to generate smaller token chunks, right?

s359
00:33:13.280 --> 00:33:19.600
that question, you mentioned like a small model essentially needing to flag that it needs help from the larger model.

s360
00:33:19.600 --> 00:33:21.160
What is that mechanism?

s361
00:33:21.160 --> 00:33:25.080
Cuz that seems like the what's the indicator and then what's the mechanism for it to do so?

s362
00:33:25.080 --> 00:33:25.600
Yeah, totally.

s363
00:33:25.600 --> 00:33:32.920
Uh, so there are a lot of mechanisms we we talk about in our blog post about how uh we just detect that we need to like change the model up.

s364
00:33:32.920 --> 00:33:37.560
Um, to to uh I guess to answer your question first, how does the the small model detect?

s365
00:33:37.560 --> 00:33:43.560
Um, actually, the the thing that we spend a lot of time on is, um, how do we make sure the small model is like good at detecting it.

s366
00:33:43.560 --> 00:33:47.000
Unfortunately, there's a lot of cases where you do need the big model to detect it.

s367
00:33:47.000 --> 00:33:56.360
Um, one thing that we could don't go into the blog post is, um, you have some kind of cadence on which you're refreshing the cache anyways, cuz by default there's some like 5-minute lifetime on these caches.

s368
00:33:56.360 --> 00:34:04.600
If you're going to go refresh a cache anyways, you basically can get a free like big like frontier model call uh if you kind of like ask the right question.

s369
00:34:04.600 --> 00:34:08.159
So, it's at that point where you might say, "Hey, just take a look at what the small model is doing.

s370
00:34:08.159 --> 00:34:12.080
Does it feel like it's kind of like going into some rabbit hole and and need some help now?"

s371
00:34:12.080 --> 00:34:12.600
Um,

s372
00:34:12.600 --> 00:34:14.720
What's the need for the 5-minute refresh?

s373
00:34:14.720 --> 00:34:21.639
Uh, it's just like a practical like you you you have to pay some kind of like cost to like keep these like KV cache caches warm.

s374
00:34:21.639 --> 00:34:24.679
Um, and so most caches just get evicted on some kind of cadence.

s375
00:34:24.679 --> 00:34:31.159
works is like at inference time you only have so many so many cache you can keep kept loaded in the GPU.

s376
00:34:31.159 --> 00:34:35.280
So, once if a cache is not being used again and again, it's offloaded.

s377
00:34:35.280 --> 00:34:35.560
I see.

s378
00:34:35.560 --> 00:34:37.399
So, it's lost essentially.

s379
00:34:37.399 --> 00:34:40.639
So, that's why the inference provider asks you for money.

s380
00:34:40.639 --> 00:34:44.000
Uh but if you self-hosted, you can get around this problem.

s381
00:34:44.000 --> 00:34:48.200
You can you can make it as long as you want based on your big business logic.

s382
00:34:48.200 --> 00:34:57.200
But do do you see a world where we'll have like much more dynamic cache durations rather than just the 5-minute, 1-hour?

s383
00:34:57.200 --> 00:35:02.840
It depends on who's do who's deploying the model where, right?

s384
00:35:02.840 --> 00:35:16.920
So, if you have a GPU which has like a lot of memory uh which like the ratio of let's say SCMs to memory is memory more heavily skewed or if you're working with unified memory

s385
00:35:16.920 --> 00:35:22.960
uh and you have systems like where I Rubin, uh you you have a lot of tricks to play here, right?

s386
00:35:22.960 --> 00:35:37.760
Uh the 5-minute uh window is what a lot of providers right now put, but that's uh that's that's more an operational operational operational determination rather than a like a science-based

s387
00:35:37.760 --> 00:35:40.600
or like a core physics law determination.

s388
00:35:40.600 --> 00:35:53.000
So, you can technically see over time maybe some uh some APIs are priced differently, uh but uh if you do self-deploy again, you can you can get past a lot of this.

s389
00:35:53.000 --> 00:36:07.840
The the cost economics really change when you move from self-hosted models to uh API providers uh because you have a lot more control and uh you don't have to guess the shape of your workload.

s390
00:36:07.840 --> 00:36:23.640
So, let's say if your workload is 32K uh like on average 32K cache, 1K input, 1K output, uh and someone else's like let's say 64K uh 1K 1K, uh if you use

s391
00:36:23.640 --> 00:36:30.040
some provider, they are amortizing everyone's use uh use case and then giving you a price, right?

s392
00:36:30.040 --> 00:36:33.280
And they have optimized, quote and quote, for general use.

s393
00:36:33.280 --> 00:36:39.160
If you self-host, you can optimize specifically for your use, and you'll likely pay much less.

s394
00:36:39.160 --> 00:36:39.840
Yeah.

s395
00:36:39.840 --> 00:36:45.920
This is like kind of like the level of like, you know, hardware-software frontier that we kind of like think about.

s396
00:36:45.920 --> 00:36:52.160
When When we started Cognition, we were working on the first agents, I think one reason why no one else worked on agents is they were just extremely expensive.

s397
00:36:52.160 --> 00:36:56.600
This was before cash tokens was a thing that API providers paid for.

s398
00:36:56.600 --> 00:37:04.520
Like if you were sending 100,000 tokens and the same 100,000 tokens, you were paying full price for those tokens back in 2024 when we started.

s399
00:37:04.520 --> 00:37:16.840
One of the key things that let us build Devin and build these first agents was we actually bought direct compute capacity from these providers, and instead of paying on a per-token basis, we just paid for the underlying compute, knowing

s400
00:37:16.840 --> 00:37:23.160
that the economics of the compute was that we were actually paying far less for for the cash tokens that we'd send over.

s401
00:37:23.160 --> 00:37:34.840
Um And And then nowadays, you know, that uh you know, there's there's like similar, you know, things I would like about like, you know, having a version of the cache that maybe like you can just back out to like storage in S3 or something and just like hold for much longer.

s402
00:37:34.840 --> 00:37:35.080
Yeah.

s403
00:37:35.080 --> 00:37:44.440
Now, this is not extremely relevant to a DGX organization like setup, but if anyone's looking to do what you guys have wanted to try out Dynamo.

s404
00:37:44.440 --> 00:37:48.360
We have a lot of pre-fixed cache optimizations in there.

s405
00:37:48.360 --> 00:37:49.040
Yeah.

s406
00:37:49.040 --> 00:37:58.560
Um And then going back to your question, Alex, I think you said like, "Oh, like you When a small model is going off and generating a ton of tokens, is that like a interesting time to back off?"

s407
00:37:58.560 --> 00:38:00.400
Um to be honest, we haven't explored that right yet.

s408
00:38:00.400 --> 00:38:03.880
So, that that might actually be a very interesting thing to to take a look at.

s409
00:38:03.880 --> 00:38:05.080
Um It It is weird.

s410
00:38:05.080 --> 00:38:15.040
I think like some small models do tend to kind of be less like token efficient than others, but um they also seem to be trained on like their own traces, so maybe in a way it ends up like balancing out.

s411
00:38:15.040 --> 00:38:20.840
A lot of these things I I feel like we have to be like very empirical about to actually know.

s412
00:38:20.840 --> 00:38:28.720
So, uh just to add on that um um you have a lot of like these days there are a lot of hallucination probes.

s413
00:38:28.720 --> 00:38:35.280
So, probes that work on either the internal state like internal state of the models directly.

s414
00:38:35.280 --> 00:38:51.960
So, you can have some form of either magnitude analysis done or linear probes or just uh the end types of probes that you can see and you can essentially rate like how how much you think is is tending towards hallucination.

s415
00:38:51.960 --> 00:38:56.480
Uh so, that kind of gives you a proxy for how lost it is.

s416
00:38:56.480 --> 00:38:59.400
Uh like how lost a model is in its thinking.

s417
00:38:59.400 --> 00:39:05.720
Uh so, you can use like different kinds of probes to understand like the perplexity within a model.

s418
00:39:05.720 --> 00:39:06.360
Well, that's interesting.

s419
00:39:06.360 --> 00:39:14.080
So, yeah, instead of using the quantity of tokens that are as indicative of a of a model being lost, it's the it's hallucinating more and you're

s420
00:39:14.080 --> 00:39:16.520
Yeah, so uh so, essentially what is cache, right?

s421
00:39:16.520 --> 00:39:20.080
It's It's the prefill It's the prefill stage, right?

s422
00:39:20.080 --> 00:39:21.680
So, you What is a prefill stage?

s423
00:39:21.680 --> 00:39:23.440
It's just a vector at the end of the day.

s424
00:39:23.440 --> 00:39:32.480
So, you can do tune all kinds of classifiers to understand uh different aspects of those collections of vectors.

s425
00:39:32.480 --> 00:39:41.360
So, with those kind of probes, you can guesstimate a lot of uh states of a model.

s426
00:39:41.360 --> 00:39:41.920
I see.

s427
00:39:41.920 --> 00:39:50.920
One One question I have is um you know, different models behave uh differently and um the kind of means that these prompts aren't portable.

s428
00:39:50.920 --> 00:40:00.040
So, as you're doing model routing, how do you handle essentially if you're if you're going to a different model architecture, um what do you have to do to the prompts and how much is that a factor

s429
00:40:00.040 --> 00:40:03.480
into either of you guys' model routing solution?

s430
00:40:03.640 --> 00:40:07.840
Like what like how how is the prompt itself a factor in the in the routing?

s431
00:40:07.840 --> 00:40:08.440
Yeah.

s432
00:40:08.440 --> 00:40:27.440
Yeah, um well, I think with building agents, there are all kinds of paper cuts and edge cases that are domain-specific and like the value of an agent company like the value of Devin is all these like doom loops that you've discovered

s433
00:40:27.440 --> 00:40:38.800
that are across all industries and the best ways to recover from them and like man here I mean it manifests big time in what the prompts are going to be both for like

s434
00:40:38.800 --> 00:40:46.453
you know how the the advisor model gets called um uh you know the smart friend

s435
00:40:46.453 --> 00:40:47.240
[laughter]

s436
00:40:47.240 --> 00:41:01.960
the how the like subtask agents get called and and the best thing is that like anyone can like like any engineer or any like agent can inspect the traces and like adjust the prompt and then see the like live accuracy

s437
00:41:01.960 --> 00:41:02.320
long time.

s438
00:41:02.320 --> 00:41:16.880
So I mean basically I just think that that's part the prompt is part of the the startup building process and is also really easy to observe and like and have and have like multiple people and agents collaborating on them.

s439
00:41:16.880 --> 00:41:17.440
Yeah.

s440
00:41:17.440 --> 00:41:24.680
One thing I'd love to do with our fusion product and we don't have this yet and so this is kind of maybe a preview of some some things we work on is

s441
00:41:24.680 --> 00:41:34.800
you know you can tune it against a data set but the the real thing you want when you're building a real agent someone uses is just like tune it against what actual people use it for and and what actual models they get routed to.

s442
00:41:34.800 --> 00:41:48.720
And so there's a lot of signals for this like if if someone sends a prompt and then Devin is working and then you see that the user decides themselves like upgrade to a different model or they decide to downgrade or the system detects that we originally sent to the wrong one we now got to replace.

s443
00:41:48.720 --> 00:41:56.040
Like that's actually a really useful stream of signals and we're actually getting into this world of like auto research where like maybe we can just have like a constant stream

s444
00:41:56.040 --> 00:42:08.640
of prompts what it should have been what it was instead and build a system internally that's just capturing all of this and then reiterating on our routing system until it eventually kind of like fits the real production data.

s445
00:42:08.640 --> 00:42:15.040
Um that's kind of like now that we it's it's it's public and and people are using it, this is now something that we're we're thinking about.

s446
00:42:15.040 --> 00:42:20.880
Have you guys looked into prompt tuning and do you find it useful like say Japa?

s447
00:42:20.880 --> 00:42:29.840
Yeah, so there are like these prompt tuning frameworks from like a few years ago that tried to do some kind of like gradient descent type thing.

s448
00:42:29.840 --> 00:42:43.320
I'm like I'm actually personally less bullish on these kind of like low level mechanical prompt tuning harnesses versus just telling like a smart model like here is the decision that was made

s449
00:42:43.320 --> 00:42:46.920
and the context figure out why it went wrong.

s450
00:42:46.920 --> 00:42:58.760
Sometimes you can do something as dumb as asking the model why did you do this instead of this and cite the prompts and then just have your agent your dev and just go and update the prompts, rerun the the test as a regression, make sure make sure it changes.

s451
00:42:58.760 --> 00:43:03.920
Like it's a lot heavier weight of a system, but I kind of trust the intelligence of a system like that a lot more.

s452
00:43:03.920 --> 00:43:11.920
So we're we're running out of time, so we're going to wrap up real quick, but I think what's really interesting is just from talking to you guys, we can kind of see how

s453
00:43:11.920 --> 00:43:13.160
new this space is, right?

s454
00:43:13.160 --> 00:43:15.640
How much of this is actually just research.

s455
00:43:15.640 --> 00:43:17.520
We're starting to see new products come in.

s456
00:43:17.520 --> 00:43:20.520
I'm really excited about your guys' solutions as you guys enter the space.

s457
00:43:20.520 --> 00:43:30.320
The the ways and the needs that you need routing for, you know, even on a DGX Spark when you're doing local inference, you have more compute and if the memory is filled, one

s458
00:43:30.320 --> 00:43:35.120
or if it's if the memory utilization is high, one thing you need to do is increase the compute utilization.

s459
00:43:35.120 --> 00:43:39.280
And so one way you can do that is by spawning multiple agents that are working collaboratively.

s460
00:43:39.280 --> 00:43:49.120
So that collaborative piece is something that not only is optimal for all of these cloud workloads that you guys are doing, but but specifically that is how you extract more performance out of this edge hardware.

s461
00:43:49.120 --> 00:44:00.480
And I think um you know, a question here is and maybe to to end on is a router going to be something that we see as a product or is that going to be seen as part of the plumbing here?

s462
00:44:00.480 --> 00:44:11.080
Are models going to get good at routing to other models because they know they need to be collaborative or or harnesses going to know that they are working across multiple models.

s463
00:44:11.200 --> 00:44:12.400
I think we already see this.

s464
00:44:12.400 --> 00:44:17.040
Like you know, at Cognition we're training our models to be able to be good collaborators.

s465
00:44:17.040 --> 00:44:26.080
I think it's very clear that new frontier models like the Fable models and GPT-5.5 and 5.6 models are like themselves like naturally collaborative and and better at delegation.

s466
00:44:26.080 --> 00:44:27.440
So I think we're ready there at that point.

s467
00:44:27.440 --> 00:44:28.560
Interesting.

s468
00:44:28.560 --> 00:44:42.440
Yeah, I I think that the systems are are kind of becoming not muddied in some sense, but I think that ultimately we're understanding that as we step up the the abstraction ladder and build more more things to create this smarter

s469
00:44:42.440 --> 00:44:55.240
blob, which obviously we should hopefully and we do understand how we're building it and why we're building it, that it's going to it's going to become a system that you look at kind of both the the different components of the system, but

s470
00:44:55.240 --> 00:44:56.840
it's not just going to be just models.

s471
00:44:56.840 --> 00:45:03.000
There's not going to There's not going to be a thing as like a really great harness that is in absence of a really great model and vice versa.

s472
00:45:03.000 --> 00:45:03.920
Yeah.

s473
00:45:03.920 --> 00:45:05.440
Makes sense.

s474
00:45:05.440 --> 00:45:15.640
I think applications, especially as built on non-deterministic systems like models, operate in a very low-trust environment.

s475
00:45:15.640 --> 00:45:30.720
So yes, most of the improvements will likely be distributed across both models and the harnesses, but I think overall it's it's it's mostly it's There There will There will have to be some form of controller

s476
00:45:30.720 --> 00:45:37.440
trying to have some form of arbitration because even from the model perspective you aren't in a perfectly visible world.

s477
00:45:37.440 --> 00:45:45.760
You don't know the behavior of every model, so it's it's going to be at the orchestration level where you have these kind of things.

s478
00:45:45.760 --> 00:45:52.400
And this has traditionally been shown by other industries like when web when web launched, you know, you had traffic-based routing.

s479
00:45:52.400 --> 00:45:53.840
So it's different.

s480
00:45:53.840 --> 00:45:59.560
But all the sort of routing controls have been centralized over time.

s481
00:45:59.560 --> 00:46:01.280
Makes sense.

s482
00:46:01.280 --> 00:46:05.240
I I it's most likely going to be good news in the future.

s483
00:46:05.240 --> 00:46:10.040
Um and and I I think like caching is a big reason for that.

s484
00:46:10.040 --> 00:46:22.880
Even if you I think like a to take the flip side of of this argument, um the you know, it might be that in the future we have like one big model that's like, "I know I am the like most efficient at everything

s485
00:46:22.880 --> 00:46:25.080
and I'm like way more efficient than Haiku.

s486
00:46:25.080 --> 00:46:29.280
I'll solve every task better than Haiku can at like a lower price.

s487
00:46:29.280 --> 00:46:31.920
Um why should I ever delegate to Haiku?"

s488
00:46:31.920 --> 00:46:36.400
There's something like that actually could could be a model that we have in the future.

s489
00:46:36.400 --> 00:46:41.280
Um but you're always going to have these like, you know, for example, caching.

s490
00:46:41.280 --> 00:46:52.320
It could be that like you tell the model that this other model like does have the right context in cash and uh you know, the the orchestrator model just always has more context

s491
00:46:52.320 --> 00:46:54.200
and the models have to be aligned.

s492
00:46:54.200 --> 00:47:04.040
So, I I think like it's I I don't really see a world where like we wouldn't be able to get models to collaborate really well and and I think they're going to get better over time.

s493
00:47:04.040 --> 00:47:08.920
Um in part because they're you know, they just have limited memory.

s494
00:47:08.920 --> 00:47:25.880
So, I I think that's kind of one one deciding factor and another is that um there will be like uh there will continue to be like if you just look at like the the the rankings on OpenRouter, if you look at our our public data and you look at like

s495
00:47:25.880 --> 00:47:33.480
the top model being used by dollar spent on classification tasks, well, guess what it is.

s496
00:47:33.480 --> 00:47:35.177
It's Opus.

s497
00:47:35.177 --> 00:47:35.640
[laughter]

s498
00:47:35.640 --> 00:47:51.680
I think there are there are there are big opportunities for like using small models for in distribution easy tasks and the and like as time goes on, that's going to be a larger and larger percentage of tasks relative to like the most valuable tasks that

s499
00:47:51.680 --> 00:47:54.160
um very smart models spend most of their time on.

s500
00:47:54.160 --> 00:47:54.880
Totally.

s501
00:47:54.880 --> 00:47:56.760
Well, I want to thank you guys so much.

s502
00:47:56.760 --> 00:47:58.640
Can we all give everyone a round of applause?

s503
00:47:58.640 --> 00:47:59.413
Thank you.

s504
00:47:59.413 --> 00:48:01.413
[applause]

s505
00:48:04.653 --> 00:48:06.653
[music]