The State of Model Routing — NVIDIA, Cognition, OpenRouter

AI Engineer · 48 min · 505 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12have been really exciting.
00:13We've tried to get a bunch of the industry leaders together to talk about some of the problems that are that we're facing as we try to run more on local.
00:21If you guys were here for the first panel, one of the things that we talked about was model routing.
00:26We firmly believe that we're in a multi-model world.
00:29I think you heard this from many of the panelists.
00:32Anyone who is deploying AI in production and who is doing so locally is seeing that multi-model world.
00:37That's why we released these NeMo Triton models at NVIDIA.
00:40Everything is released from the data sets to the weights with recipes so that you can customize them.
00:44We do that because we know that people customizing models is going to be huge.
00:49And so, this panel is really exciting cuz we're going to talk specifically about model routing.
00:54So, as you are picking which model to use, how does that how essentially how does that tooling itself look?
01:00Um, do you guys want to introduce yourselves?
01:03Yeah, sure.
01:03Um, I'm Walden.
01:04I'm the co-founder of Cognition.
01:05We build Devin, AI software engineer.
01:08Uh, in addition to the product, we spend a lot of time partnering with our customers to figure out how they should deploy these models uh and these agents.
01:15And one of the things they're constantly asking us nowadays is basically how do I know the ROI of our models and how do I know which tasks I can actually let our engineers spend the most expensive models on versus, you know, letting them use a more cost-efficient model.
01:29And so, that's why we're we're also thinking a lot more about multi-model model routing nowadays.
01:32Totally.
01:33Yeah.
01:33I'm Carter.
01:34You guys heard from me a little bit earlier, but if you weren't here, I'm a developer tech engineer at NVIDIA.
01:39And uh ultimately, I spend a lot of time thinking about how to get intelligence into as many developers' hands as possible.
01:46And uh something that is continually becoming a not an issue, but something that is top of mind for a lot of developers is uh as you use more intelligence and the frontier models get more expensive,
01:56uh it becomes somewhat cost prohibitive to to use the best uh tools, what feels like the best tools, as much as you would like to use them.
02:04And so, this has become a recent uh you know, focus is, how can we, you know, still get the same desired outputs, but actually uh both as an individual developer, but also imagine startups and small companies, how can you leverage this incredible tool without totally breaking the bank?
02:21Uh I'm Donne.
02:22I have worked on model evaluations uh both in terms of its accuracies and efficiency and cost understanding of the model.
02:31Uh and then I try and understand those implement those learnings and uh help build a router.
02:37So, it's basically my job is to understand the behavior of of the model on an intimate level and then use those learnings to both improve the model and try and design a system of model that can work together with each other.
02:52Totally.
02:53Um yeah, I love a lot of the research that you're doing at Nvidia as we kind of see the space through.
02:58I think what's really interesting is model routing itself is pretty new still.
03:01And so, what you'll notice is there isn't a very clear solution here.
03:05That was something that came up on the first panel is that there is a lot of space for startups and for companies in the ecosystem to fill in a solution here cuz we're still figuring out how to best do these patterns.
03:15And I think, Walden, I want to kind of ask you.
03:17So, um Cognition just released Fusion, your guys's model router.
03:21And when you guys released it, you in your blog said that you're actually getting better performance than Fable, um than these frontier models.
03:28And I feel like that was a very surprising statement to hear because we were thinking that you're getting as good or close enough usually when we're running on edge, when we're running local in these compute strained, smaller footprint models.
03:40But you guys are getting better.
03:41Can you explain how?
03:43Yeah, absolutely.
03:44Um so, I also want to be be clear about something here is like, you know, we're not saying that that we gap above Fable level performance in in the same way that maybe Fable level performance gaps above other models.
03:54I think actually there's this really unintuitive dynamic where smarter models actually get better and better at delegating work.
04:03And so one of the philosophies we had with building a model router is we don't want to route people to a dumber model and then suddenly you're stuck with a model that doesn't know how to do your task.
04:13Next thing you know you're you're switching yourself back to smarter model anyways and now taking that expensive cost.
04:18And in general we think a lot of the existing model routing systems out there are probably the same ones people have been using like a year ago.
04:24And so we really wanted to put out a new framework that actually lets people still feel like and and still have a frontier model in their system while getting all these like cost benefits.
04:33So yeah we we're reducing the cost of Fable level intelligence by 40%.
04:37The way we do that is we allow Fable to still do like the planning and the the hard decision making but delegate a lot of the work to an implementation model.
04:46And the implementation model can be you know one of these open source models be it a cheaper mini model.
04:51Um the unintuitive thing is even though it's cheaper because you're delegating the work to another model you can let that model go at the task with much more depth and intensity
05:00than you might otherwise.
05:01Like you can spin off like you know three sub agents to go and like explore the code base and maybe that's actually more comprehensive than if you had just let Fable explore the code base itself.
05:10So you're actually getting this like nice trade-off where it's both more cost efficient and it's also more comprehensive overall.
05:15Interesting.
05:16I see.
05:16So you're saying by using a bunch of smaller models you're essentially like for one example scouring the code base you can you you can explore it potentially better than if you were to just have one model
05:26I don't know figure out with what it's limited with its limited context with whatever path it's on.
05:31Yeah, totally.
05:32But also if you think about the budget of if you were to say the frontier model costs this amount per token and the smaller model is this amount per per token and it's significantly cheaper then you can use a lot more tokens from the smaller model
05:45with
05:45still within the budget that it would have been from the Frontier model.
05:48I would also like to encourage everyone to think there are jagged capabilities in most models, right?
05:54So, coding is not one domain.
05:56Within, let's say data visualization, there'll be scikit-learn, there'll be matplotlib, there'll be something else.
06:02It largely comes down comes down to the training corpora that went into each of the models, so right?
06:10So, one model, while you're trying to do X type of work, let's say data visualization, and the other type is Y, that means let's say model building.
06:19Let's say you're trying to have a data science work stream, where you're trying to optimize for some kind of prediction, and then visualizing your results.
06:29Within that task, different models will have different strengths.
06:33So, not it's not necessary that model A if scores higher on a coding benchmark, is just plain better at every task that is.
06:42So, routing is a task of intimately intimately understanding of behavior of and strengths and weaknesses of different models, and then applying them thusly, right?
06:54I would I would encourage everyone to think, "Hey, models are strong at different things, rather than like there's one model to rule them all."
07:01I see.
07:02And by the way, real quick, thank you Alex for joining.
07:04Yeah.
07:05Sorry I'm late.
07:05No. Oh, is that so?
07:07I might need yours in.
07:09Sorry I'm late.
07:10[laughter]
07:11I'm Alex from OpenRouter.
07:13Um thanks for having me now.
07:14Yeah, of course.
07:14Thank you so much.
07:15You came right from the airport, so this is perfect.
07:17I think um tonight that's that's super interesting.
07:19So, um the way that you're thinking through through model routing, it's not even just uh delegating to necessarily a smaller model, but like and maybe this is kind of what you're saying is can you can you put essentially a swarm of agents to accomplish the same task, and suddenly routing the task between them is is much is
07:34a is a problem to solve in and of itself.
07:36Yeah.
07:37So, if you look at like let's say let's take an easy example.
07:39Let's take a science or like scientific discovery as an example, right?
07:44Usually these are one-shot problems.
07:46It's incredibly hard.
07:47You have models think through this process, right?
07:51So, in that you have tons of sub-domains.
07:55Like tons and tons and tons.
07:57So, in that aspect, if you think about post-training like the post-training process of a model, they'd be tuned with different teachers.
08:06They'd be tuned on different sub-tasks.
08:09So, those those um those overlapping strengths will be readily apparent when you're trying to understand failures of each models on different different sub-tasks.
08:20Once you understand that, you can orchestrate your system to leverage that arbitrage essentially, and that essentially becomes free.
08:27So, I think this is on LM router bench.
08:31There was there are tons of benchmarks out there.
08:34But, if you use these techniques, you can get like up to 10% higher accuracy even, right?
08:39It depends on the model pool.
08:40Depends on the task at hand.
08:42But, I would encourage to think about the complementary nature of models.
08:48I see.
08:48Do you see so in kind of the way that you were describing the way that the task is broken up, do you see that the some of the smaller models because the token cost is cheaper,
08:57are they using more tokens?
08:58Like is it Are Are you specifically routing so that they do or that they so that they are chattier?
09:03Uh oh yeah, they absolutely do use more tokens.
09:07I I actually I want to kind of like riff on something that Tuhin was saying, which is like you know, a lot of times when you look at these different benchmarks, you'll see that the small models will perform better than like even the frontier models in certain cases.
09:19I think a lot of people they look at this and they immediately jump to, "Oh, how can we just route like, you know, the task where the small models do better just straight to the smaller models?"
09:28I I think that one of the things we really want to emphasize with our recent blog post and recent Devin Fusion was that this kind of like naive like initial routing to based on the task type is extremely fragile,
09:40especially the more agentic the task you you work on is.
09:43So, for example, like a real developer, you might ask your agent first, "Oh, how does this code base work?"
09:48And then you you go deeper and you're like, "Okay, actually, can you implement some features for me?"
09:51And then you go deeper and you're like, "Oh, can you like now go to like a live test of this feature and debug deep cases?"
09:56The complexity changes and the type of task changes over time, and you don't want to be left with some subpar model for the the task that you're now on.
10:04I think this is why people like frontier models so much is they're they're just like generally intelligent and they're capable of shifting between various different domains, even if you can eke out better performance in very specific tasks.
10:15Um, and and what So, the challenge is, how do you get a small model to know that it's out of its depth and you need to now like go switch to another model or go like, you know, go to a smart smarter model.
10:26And and our solution to this is you kind of just always have this like main frontier agent that's watching, even if it's not the one doing the work.
10:33It should at least be keeping tabs to figure out, "Okay, we like the the agent I delegated to now is like out of its depth.
10:38I need to kind of like move it to to something else."
10:40Um, and and overall, just the guarantee of always having frontier intelligence present, I think reduces the the fragility of of these systems quite a lot.
10:48How does the sharing of context between one of those smaller agents who has basically completed up to some level of a task and decides, "Actually, I don't think I'm the right person for this.
10:59I need to hand it back to the the foundational model."
11:01Um, of course, you don't want to have the entire trace of that
11:05Yeah.
11:05smaller agent be passed back to the larger model.
11:07But so, how do you get that level of specificity while basically providing the information it needs, but not more?
11:13Yeah, absolutely.
11:14So, I I I think like the the context here is it's actually very easy to actually create a system that's more expensive as soon as you're running like, you know, multiple models together cuz oh, no, like, you know, this one file reading.
11:25Now, every every one of these models is now reading this one file reading.
11:27So, now you're you're you're being charged like three times as much.
11:30Um, the the the the trick that we we spent a lot of our time on is um, most of the context by default will only be going to like one model.
11:37So, like most of the context let's say will be going to the small model.
11:39But, the thing you need to then tune very well is okay, like maybe you still show like what files it's reading, maybe you show like the high-level thinking of what it's doing back to the main model.
11:47Maybe you have the small model um, you tune its ability to present the context back to the main model.
11:53Um, and actually a lot of these problems already have been well studied in many domains already like context compaction is something you already have to solve if you want to do like really long-running agents.
12:03And so, this problem of taking long context compacting it in a way that is now understandable is the one that you can also apply to this domain and just kind of give the compacted context back to back to the main agent.
12:13Context compaction is something that you know, I'm familiar with but I hadn't really thought about as you're doing model routing and as you're trying to share context across now potentially many models, you're you're expanding the amount of what could be seen as wasteful tokens or redundant
12:27tokens just just because you have to process that across the many models.
12:30Yeah.
12:31Yeah.
12:31Um, I I I think there's the way I describe it is I think we are early in in the in the model routing domain.
12:37Um, I I I hope that a year from now that even the techniques we kind of use for Devin Fusion you people look back on that and are like, "Oh, these are some like really legacy ideas and and now we have like much better methods at routing between models."
12:49Um, and when people actually start co-designing their models with this in mind we're going to be be in a much better world.
12:55Yeah, I I echo what you said that um, I think routing will evolve as the task evolves when when you start task, right?
13:06So, um, it's more useful to see things in terms of subtasks and sessions uh, than individual problems that you're trying to solve because more than likely you're when you're working through a problem you're asking a lot of questions, you're you're exploring different things.
13:21Um, and it is imperative that you try like people who design routers is imperative that they try and understand these phases of different complexities and then try and apply some logic
13:38for essentially side kicking tasks or leveraging expertise from other models that's that's pretty on point.
13:48Yeah, I'd love to hear from the the router guy.
13:51[laughter]
13:53Yeah, um I think those are these are like important points and one of the biggest debates I think we have internally is whether that outer model that's doing the orchestration
14:04should be the big model or the small model.
14:08You get like very different results depending on your choice and in fact and it's not even clear what the pricing impact would be because if your outer model that's doing the orchestration is the big model it can leverage its caching
14:20to like make more of its decisions and it's caching is going to be like a dramatic price savings compared to the small models caching a lot of the time
14:29especially for like perform for you know issues that are on the bright line like zooming out a little bit I think the what you want from all the models out there when you do model fusion
14:41is to benefit from all the data that is being trained on across all the labs and and not just the data from one lab right or one source and a model is just like a combination of like the data
14:56and its understanding of the data both its compute and the quality of its RL so long term I think you want you want models where they know that oh this is like in distribution
15:12like this is in my data you can use small models pretty easily and get a cost savings but if it's out of distribution small models may actually increase your cost because of how often
15:23they'll like call tools and how crazy their loops will be.
15:26Like if you run terminal bench on Opus and Haiku, like Opus will do about three times better at 1/10 the cost of Haiku, even though Haiku's significantly cheaper per token.
15:40So it really becomes a huge problem if you use a too small of a model, particularly on tasks that are out of domain for the train data.
15:48When you're doing something like classifying text, like hey, is this like a person's name or is this an organization's name?
15:54That's super in domain.
15:56So you would you don't want that kind of task to go to a large model.
16:00You want it to go to a a small model.
16:01Everyone has that in their domain.
16:03Um so deep like being able to understand in domain, out of domain is a lot of work that we're doing for for um open router fusion.
16:12And uh and then also figuring out like what like how to orchestrate the outer and inner models um for different types of task.
16:18And like um yeah, it's it's an early industry.
16:22It's like an early field of research.
16:24Uh most research on model fusion has not been very detailed, not been very like, you know, optimistic sometimes.
16:32Um it's only like just, you know, recently getting uh more optimistic.
16:36And I think um I'm personally very optimistic about it.
16:40And uh you know, like we we're very like ecosystem-driven, collaborative company.
16:46And a lot of our like we work with a lot of partners to try like help improve their orchestration pipelines with good primitives, like the, you know, sub-agent and like the advisor tool,
16:57um which is kind of similar to what you were talking about.
16:59I'm curious.
16:59So uh help me understand when uh if a it makes total sense that um a small model, if it's in domain, would be cheaper.
17:07But if it's not, then it's going to thrash around uh as it tries to get an answer.
17:12When you're describing whether the uh con- like the main agent should be the the the local model or the cloud model?
17:22Does that is that a decision that's then dependent on whether the task is something that it that's in domain or not?
17:29Does my question kind of make sense?
17:30I don't know.
17:32Um, I basically I don't it's kind of early to
17:36um to say.
17:38I I think the I wait the results that we published, which are a couple weeks ago, which were focused on deep research, not coding.
17:47Mhm.
17:47Um we had the smart model be the wrapper model, be the outer model, and we got the best results from doing that.
17:56Um, but it for deep research it works the best.
18:01Um, for other tasks, it's like kind of unclear.
18:05Like we there it's fusion is not super well optimized for coding.
18:09And uh it might be that like a smaller model ends up being like a higher um efficiency per you know, fewer dollars um per like completed successfully completed task,
18:21but it's kind of early to say.
18:22Yeah.
18:23[clears throat]
18:23One thing you said earlier is like, oh, you get the caching benefit from like the main line agent.
18:27Um, you actually can get the caching benefit from the side agent.
18:30And um, this is actually one of the key things we talked about with our Devin fusion launch is that um, you kind of are leaving a lot on the table if you do a main agent and sub agents type system.
18:40Um, so we don't use sub agents.
18:42We use what we call a sidekick, which is um, one sub agent that continually has a running context.
18:49So the main agent doesn't need to re-provide uh, context from earlier.
18:52Um, it's all still in the KB cache, right?
18:54Like it's 10 times cheaper on on all those cache tokens.
18:57Um, and then if you want to like switch the smart model to be like the one on the side or the one in charge, um it's actually totally fine, and you can kind of like do do the swapping back and forth.
19:08We're also spending a lot of time right now thinking about how do you train models to actually work collaboratively with other models?
19:15I actually I think there's a lot of literature out there on how you RL one model to do a task end-to-end.
19:19How can you RL model to also be good at collaboration?
19:22And when we think about it, we actually try both of these setups where let's RL the model being the orchestrator and the one deciding what gets delegated to other models, see how well that performs.
19:32Um and we also orchestrate it in a way where the model we're training is actually the executor, the sidekick, and see how well is it at executing other models' instructions.
19:42Um and we expect that to to be a probably a big lift in this next step of like multi-model orchestration is don't just like take models as they are and orchestrate them, but like can you actually co-design your models with the orchestration system?
19:53Yeah, that makes sense.
19:54I mean, with Nemo Tron and you know, with all the foundational models where um we're essentially post-training them for the harnesses that they're getting used.
20:00If the harness is going to include a lot of routing, then that makes sense that makes its way in to the post-training.
20:05Yeah.
20:05Um are you guys thinking a lot about the kind of like the model training at at Nvidia for this kind of purposes?
20:12Yeah, so we have a technology called
20:16The mic's go?
20:18Right.
20:18Hello.
20:20Testing.
20:20Oh, this one works.
20:22[laughter]
20:23Okay.
20:24Uh so we have a technology called Flex Run.
20:25So you have uh we we have a setup where uh there's a the uh there's a main model, then we distill it into smaller uh footprints.
20:36And then based on the based on the task at hand, you can switch which model does the decoding.
20:43Right?
20:43So there's a lot of fancy stuff you can do uh within a model artifact, too.
20:49Uh to essentially only activate a class of model or a section of weights, depending on the task at hand or the complexity at hand.
20:59Most In most cases, you can essentially understand the novelty of a question to a model if you have access to the recipe with which it was trained.
21:11So, this works very well for open models, right?
21:13Like or any model you have access to its data for, right?
21:16Because you can literally decide if it's in like see if it's in distribution or not.
21:21Uh again, if you have studies from when it was trained, you can also see how much essentially how much was your distillation gap across teachers and the artifact that you trained, right?
21:34Because sure, you have domain data from all all the different domains you're tuning, but it's not guaranteed that it uh absorbed all the that data evenly across the models, right?
21:44So, um it becomes it becomes very interesting uh to start thinking about these flexible weights and flexible model sizes essentially.
21:56There's also I wanted to add with the context piece, right?
21:59So, how do you think about ASTs and and context compression representations?
22:06Uh compaction in its very nature is lossy, right?
22:10So, just like headroom is there, RTK is there, right?
22:14Uh these these spaces code bases are usually designed to have representations that we carry forward through life and you essentially give models the capability to further expand on them.
22:28It's it's more more like loss less-ish compression, which can retain states of a models or states of agents.
22:35What do you think about that?
22:37Yeah, I think this gets to like kind of like a fundamental philosophy of how agents and context should work.
22:43One exercise I like to do is like you know, as a human, like how many numbers can you like if I just start spitting out numbers now, right?
22:48Like how many can you remember before, you know, like you start losing track of them.
22:53I think it's actually very few, right?
22:54So, in some ways you could argue that your context window is actually shorter than these language models.
22:58And yet, you can actually be very effective at that, right?
23:01Your context is is very lossy.
23:03Um, I think one of the nice things that people are starting to realize with agents is like you have a lot of non-lossy systems that you can fall back to.
23:10So, you have a file system.
23:11Like uh if in your memory all your memory is that you read some file earlier, you don't need to remember the whole file.
23:16You maybe remember the important parts, but you can still have the full version of the file on your system.
23:21And that's kind of my goal when I'm thinking about how do we build a good context engineered harness is the harness should have everything in needs to find what it needs to have even if it doesn't have everything immediately available.
23:35In that case, uh do you think that the context sharing problem will become cheaper and less of a problem in future?
23:45Yeah, it's definitely possible as well.
23:47I've seen cases where um you know, the psychic agent does a bunch of work.
23:51It tells the main model, "Oh yeah, like here's here's all the things I found."
23:54Instead of dumping the full thing, it just references them by file.
23:57And then the main model is actually generally you find these larger, smarter models, they're actually more token efficient with how they use tools and how they read.
24:05And so, they actually read the files in a way where they only see the important parts, right?
24:09Or they decide that, "Oh, actually I only need to look at a subset of this."
24:12Or, "Oh, I can run a single command and just know if everything is done properly."
24:16Um, it's actually quite amazing um the fact that you know, these multi-model systems they actually seem to scale and get better with intelligence, which is um not something we should just take for granted, right?
24:27It's not obvious that actually more expensive models are actually creating an overall cheaper system.
24:33Yeah, like the scaling laws uh if you have a larger model, it's going to be more efficient with its tokens.
24:40Uh smaller models less efficient with its tokens.
24:43I guess I had a question for you, Alex.
24:45Uh do you guys uh like weigh more importance on uh the actual infrastructure side of routing?
24:52So, for instance, KV cache aware routing or uh like this where most of the businesses right now Uh or the are you seeing strong pull off like people actually deploying routers in production?
25:07So open router is a like a marketplace for language models.
25:13We we exist at like we can't see into the KV caches of models unless we're running them ourselves, which is pretty rare.
25:22Um we we we do spend a lot of time optimizing for cash hits and we we like pass through cash hits directly to users.
25:32Um but like in terms of KV cash optimizations we we we can't do any of like specific work there.
25:38What we do for um model routing is we we try to like find the best model or best combination of models for the prompt and then when we see a cash hit,
25:51we will will like use up the duration of the cash and send the downstream customer like the the full savings of the cash hit.
26:00Um there's more work that we can do here where we could say, okay, this looks like something where there's significant benefit to switching the model right now, but you haven't used up the full cash.
26:11You still have like 2 minutes left.
26:13Um and it we think it's probably worth switching the model and losing the rest of your cash um and letting people kind of like tweak their tolerance for that behavior.
26:25Um we've we've been doing a little bit of that, but we haven't like exposed it to customers yet.
26:29[clears throat]
26:30Awesome.
26:30What what is next for you guys in terms of your your your model routing?
26:33Cuz as you mentioned, you know, it is kind of like a different direction from like the the marketplace business that exists today.
26:39So I'd love to hear.
26:41So I mean we've been doing like we we've had a an auto router for like 2 years almost.
26:47Um but when we launched it, there was like no adoption of it.
26:51It was people really wanted to use specific models.
26:54Um and the the auto router just had like no real usage.
26:58It was it we mostly saw it as like a discovery play point.
27:02Like, hey, this is how you discover which model might be good for your prompt.
27:05And then um at around like January this year with open claw, it exploded.
27:14And the reason it exploded is because there's this fundamental um idiosyncrasy in open claw where it sends heartbeats every like 10 minutes to your model of choice just to see like if the you know, if the the the client was still active.
27:31[laughter]
27:32And that means that if you set Opus to be your default model, it would be like using a lot of tokens on this like heartbeat process.
27:39And so um this was the very beginning of like a very popular app with two completely different intelligence needs, completely different.
27:49And the models the open source models have improved to a point where like it makes sense to segment the market and at at least those two areas.
27:57And so that's that was how it got started.
28:00And then we saw a lot more segmentation blossom afterwards.
28:03And so um and now a whole bunch of agents and apps on open router use the the different routers that we have.
28:10We have a couple of them.
28:11We have Pareto code which gives you the Pareto Pareto optimal model for coding tasks given a certain like threshold that you can tune.
28:20Um we have fusion which orchestrates multiple models and gives you a fused result.
28:25And um and then we'll have other experiments in the future.
28:29What we want to do is basically like create good primitives that developers can use to like get really advanced with how they use model orchestration.
28:39Um kind of like sidecar sort of like that.
28:43Um but also give people like a really easy thing that they can just like set a slug to that works with all harnesses and just like gets the job done.
28:52I feel like it's super interesting how much of a perfect storm there is for model routing right now because on one hand, you know, ignore agents, ignore open claw for a second, like just to squeeze better performance, it seems like we should be smarter about how we tackle problems.
29:06That's obvious, right?
29:07If you make a plan, if you make a strategy, that's a that's a better way to to go about your day.
29:11So, I'm not surprised that you're going to see, you know, better code get written or more performing code get written and less buggy code get written if you break the problem down.
29:19And so, routing specifically for that uh use case makes a ton of sense.
29:23But then hearing this, yeah, like the profile of workloads changed with agents, right?
29:27They're very they it went from like I ask questions, I get a response, then it went to reasoning where I ask questions, it reasons, and then it comes back, and then it went to uh yeah, this heartbeat, right?
29:37Like if um if my agents are running optimally, uh there's a token being generated every second.
29:43Um and suddenly that is its own need for model routing, and it feels like um hearing the different solutions uh to kind of tackle each of those is very interesting.
29:52Even
29:53Well, I was just going to say I think that yeah, the the use cases for model routing are are there are many of them.
29:59And so, one could be getting a better answer, one could be, you know, saving money and trying to get the same answer.
30:05One that we haven't even talked about yet, which is probably the most relevant maybe even to this crowd, is uh when do you want to actually run a model
30:13locally versus when do you actually need something like a frontier model to to perform that task.
30:18And that might be something to the effect of like uh for privacy protecting information.
30:23You know, when I'm running some local, it can you detect that my prompt has sensitive information?
30:28And if so, do that on on device, um but then maybe even anonymize some of that information to go do the more advanced workloads up on top of that information
30:37in the cloud.
30:38Um another example would be again for the cost savings, but it's like, "Hey, I bought this DGX Spark and I'm not I know I'm not at 100% utilization.
30:47How can I make sure that as part of my workloads, whether it's the heartbeat and open claw or what have you, that I'm leveraging that compute um to the fullest of its ability because I'm only paying for the electrons that are coming in for my power bill, um but I'm paying full price for the tokens in the
31:00cloud?"
31:00And I think that's a whole other area of model routing that I know that we're doing some work with at NVIDIA um that I think will be will be really cool as it starts to uh the hybrid
31:10of local and cloud starts to to really emerge as its own sector.
31:14Yeah, I'd be curious to know what you guys take uh take is on.
31:18Like if you self-host a model, the cost dynamics change, right?
31:21You have a considerably higher cost at a higher context length because your throughput slows down as the context gets deeper.
31:29So, rather than switching to a cheaper model, even if you have self-hosted models in data center, you can use compaction uh to bring your throughput back up.
31:38Uh have you guys uh thought about this part of it?
31:41Like compaction versus uh just routing because one is you have fewer tokens to work with.
31:47One is we have cheaper tokens.
31:50Yeah, I I think in practice um compacting alone doesn't solve the the cost or or throughput problems because a lot of times it's just like the uh the differential in like model intelligence and cost is just so big where also by the way, when you compact, you're taking a cache miss.
32:04So, you're actually then now like paying 10 times as much for the for those input tokens if you didn't compact.
32:09Um the main reason we compact is actually intelligence.
32:11Um all all these like multipliers they advertise some like insane context window, like a million tokens.
32:16Uh I would like never recommend using like these models past like 200K tokens, under 100K if you can.
32:22Um the the the intelligence uh just kind of like falls off a cliff at some point.
32:26Um sorry, Anthropic if you're if you're watching.
32:28But I I think that uh you know, compaction is like a very useful tool if you are going to have to take a cache miss anyway um one way or another, like when you're routing to another model and you want to just like minimize the the window there.
32:43Do you find that, um, in the sidecar,
32:46Yeah.
32:47um, when small models are generating lots of tokens, um, is that like an is that like one of the best reasons to switch it to a larger model?
32:58Like, basically, when when small models generate lots of tokens, I wonder if that's like a box of root the root cause of of intelligence problems down the road.
33:08You want your big model to generate the big token chunks, the small models to generate smaller token chunks, right?
33:13that question, you mentioned like a small model essentially needing to flag that it needs help from the larger model.
33:19What is that mechanism?
33:21Cuz that seems like the what's the indicator and then what's the mechanism for it to do so?
33:25Yeah, totally.
33:25Uh, so there are a lot of mechanisms we we talk about in our blog post about how uh we just detect that we need to like change the model up.
33:32Um, to to uh I guess to answer your question first, how does the the small model detect?
33:37Um, actually, the the thing that we spend a lot of time on is, um, how do we make sure the small model is like good at detecting it.
33:43Unfortunately, there's a lot of cases where you do need the big model to detect it.
33:47Um, one thing that we could don't go into the blog post is, um, you have some kind of cadence on which you're refreshing the cache anyways, cuz by default there's some like 5-minute lifetime on these caches.
33:56If you're going to go refresh a cache anyways, you basically can get a free like big like frontier model call uh if you kind of like ask the right question.
34:04So, it's at that point where you might say, "Hey, just take a look at what the small model is doing.
34:08Does it feel like it's kind of like going into some rabbit hole and and need some help now?"
34:12Um,
34:12What's the need for the 5-minute refresh?
34:14Uh, it's just like a practical like you you you have to pay some kind of like cost to like keep these like KV cache caches warm.
34:21Um, and so most caches just get evicted on some kind of cadence.
34:24works is like at inference time you only have so many so many cache you can keep kept loaded in the GPU.
34:31So, once if a cache is not being used again and again, it's offloaded.
34:35I see.
34:35So, it's lost essentially.
34:37So, that's why the inference provider asks you for money.
34:40Uh but if you self-hosted, you can get around this problem.
34:44You can you can make it as long as you want based on your big business logic.
34:48But do do you see a world where we'll have like much more dynamic cache durations rather than just the 5-minute, 1-hour?
34:57It depends on who's do who's deploying the model where, right?
35:02So, if you have a GPU which has like a lot of memory uh which like the ratio of let's say SCMs to memory is memory more heavily skewed or if you're working with unified memory
35:16uh and you have systems like where I Rubin, uh you you have a lot of tricks to play here, right?
35:22Uh the 5-minute uh window is what a lot of providers right now put, but that's uh that's that's more an operational operational operational determination rather than a like a science-based
35:37or like a core physics law determination.
35:40So, you can technically see over time maybe some uh some APIs are priced differently, uh but uh if you do self-deploy again, you can you can get past a lot of this.
35:53The the cost economics really change when you move from self-hosted models to uh API providers uh because you have a lot more control and uh you don't have to guess the shape of your workload.
36:07So, let's say if your workload is 32K uh like on average 32K cache, 1K input, 1K output, uh and someone else's like let's say 64K uh 1K 1K, uh if you use
36:23some provider, they are amortizing everyone's use uh use case and then giving you a price, right?
36:30And they have optimized, quote and quote, for general use.
36:33If you self-host, you can optimize specifically for your use, and you'll likely pay much less.
36:39Yeah.
36:39This is like kind of like the level of like, you know, hardware-software frontier that we kind of like think about.
36:45When When we started Cognition, we were working on the first agents, I think one reason why no one else worked on agents is they were just extremely expensive.
36:52This was before cash tokens was a thing that API providers paid for.
36:56Like if you were sending 100,000 tokens and the same 100,000 tokens, you were paying full price for those tokens back in 2024 when we started.
37:04One of the key things that let us build Devin and build these first agents was we actually bought direct compute capacity from these providers, and instead of paying on a per-token basis, we just paid for the underlying compute, knowing
37:16that the economics of the compute was that we were actually paying far less for for the cash tokens that we'd send over.
37:23Um And And then nowadays, you know, that uh you know, there's there's like similar, you know, things I would like about like, you know, having a version of the cache that maybe like you can just back out to like storage in S3 or something and just like hold for much longer.
37:34Yeah.
37:35Now, this is not extremely relevant to a DGX organization like setup, but if anyone's looking to do what you guys have wanted to try out Dynamo.
37:44We have a lot of pre-fixed cache optimizations in there.
37:48Yeah.
37:49Um And then going back to your question, Alex, I think you said like, "Oh, like you When a small model is going off and generating a ton of tokens, is that like a interesting time to back off?"
37:58Um to be honest, we haven't explored that right yet.
38:00So, that that might actually be a very interesting thing to to take a look at.
38:03Um It It is weird.
38:05I think like some small models do tend to kind of be less like token efficient than others, but um they also seem to be trained on like their own traces, so maybe in a way it ends up like balancing out.
38:15A lot of these things I I feel like we have to be like very empirical about to actually know.
38:20So, uh just to add on that um um you have a lot of like these days there are a lot of hallucination probes.
38:28So, probes that work on either the internal state like internal state of the models directly.
38:35So, you can have some form of either magnitude analysis done or linear probes or just uh the end types of probes that you can see and you can essentially rate like how how much you think is is tending towards hallucination.
38:51Uh so, that kind of gives you a proxy for how lost it is.
38:56Uh like how lost a model is in its thinking.
38:59Uh so, you can use like different kinds of probes to understand like the perplexity within a model.
39:05Well, that's interesting.
39:06So, yeah, instead of using the quantity of tokens that are as indicative of a of a model being lost, it's the it's hallucinating more and you're
39:14Yeah, so uh so, essentially what is cache, right?
39:16It's It's the prefill It's the prefill stage, right?
39:20So, you What is a prefill stage?
39:21It's just a vector at the end of the day.
39:23So, you can do tune all kinds of classifiers to understand uh different aspects of those collections of vectors.
39:32So, with those kind of probes, you can guesstimate a lot of uh states of a model.
39:41I see.
39:41One One question I have is um you know, different models behave uh differently and um the kind of means that these prompts aren't portable.
39:50So, as you're doing model routing, how do you handle essentially if you're if you're going to a different model architecture, um what do you have to do to the prompts and how much is that a factor
40:00into either of you guys' model routing solution?
40:03Like what like how how is the prompt itself a factor in the in the routing?
40:07Yeah.
40:08Yeah, um well, I think with building agents, there are all kinds of paper cuts and edge cases that are domain-specific and like the value of an agent company like the value of Devin is all these like doom loops that you've discovered
40:27that are across all industries and the best ways to recover from them and like man here I mean it manifests big time in what the prompts are going to be both for like
40:38you know how the the advisor model gets called um uh you know the smart friend
40:46[laughter]
40:47the how the like subtask agents get called and and the best thing is that like anyone can like like any engineer or any like agent can inspect the traces and like adjust the prompt and then see the like live accuracy
41:01long time.
41:02So I mean basically I just think that that's part the prompt is part of the the startup building process and is also really easy to observe and like and have and have like multiple people and agents collaborating on them.
41:16Yeah.
41:17One thing I'd love to do with our fusion product and we don't have this yet and so this is kind of maybe a preview of some some things we work on is
41:24you know you can tune it against a data set but the the real thing you want when you're building a real agent someone uses is just like tune it against what actual people use it for and and what actual models they get routed to.
41:34And so there's a lot of signals for this like if if someone sends a prompt and then Devin is working and then you see that the user decides themselves like upgrade to a different model or they decide to downgrade or the system detects that we originally sent to the wrong one we now got to replace.
41:48Like that's actually a really useful stream of signals and we're actually getting into this world of like auto research where like maybe we can just have like a constant stream
41:56of prompts what it should have been what it was instead and build a system internally that's just capturing all of this and then reiterating on our routing system until it eventually kind of like fits the real production data.
42:08Um that's kind of like now that we it's it's it's public and and people are using it, this is now something that we're we're thinking about.
42:15Have you guys looked into prompt tuning and do you find it useful like say Japa?
42:20Yeah, so there are like these prompt tuning frameworks from like a few years ago that tried to do some kind of like gradient descent type thing.
42:29I'm like I'm actually personally less bullish on these kind of like low level mechanical prompt tuning harnesses versus just telling like a smart model like here is the decision that was made
42:43and the context figure out why it went wrong.
42:46Sometimes you can do something as dumb as asking the model why did you do this instead of this and cite the prompts and then just have your agent your dev and just go and update the prompts, rerun the the test as a regression, make sure make sure it changes.
42:58Like it's a lot heavier weight of a system, but I kind of trust the intelligence of a system like that a lot more.
43:03So we're we're running out of time, so we're going to wrap up real quick, but I think what's really interesting is just from talking to you guys, we can kind of see how
43:11new this space is, right?
43:13How much of this is actually just research.
43:15We're starting to see new products come in.
43:17I'm really excited about your guys' solutions as you guys enter the space.
43:20The the ways and the needs that you need routing for, you know, even on a DGX Spark when you're doing local inference, you have more compute and if the memory is filled, one
43:30or if it's if the memory utilization is high, one thing you need to do is increase the compute utilization.
43:35And so one way you can do that is by spawning multiple agents that are working collaboratively.
43:39So that collaborative piece is something that not only is optimal for all of these cloud workloads that you guys are doing, but but specifically that is how you extract more performance out of this edge hardware.
43:49And I think um you know, a question here is and maybe to to end on is a router going to be something that we see as a product or is that going to be seen as part of the plumbing here?
44:00Are models going to get good at routing to other models because they know they need to be collaborative or or harnesses going to know that they are working across multiple models.
44:11I think we already see this.
44:12Like you know, at Cognition we're training our models to be able to be good collaborators.
44:17I think it's very clear that new frontier models like the Fable models and GPT-5.5 and 5.6 models are like themselves like naturally collaborative and and better at delegation.
44:26So I think we're ready there at that point.
44:27Interesting.
44:28Yeah, I I think that the systems are are kind of becoming not muddied in some sense, but I think that ultimately we're understanding that as we step up the the abstraction ladder and build more more things to create this smarter
44:42blob, which obviously we should hopefully and we do understand how we're building it and why we're building it, that it's going to it's going to become a system that you look at kind of both the the different components of the system, but
44:55it's not just going to be just models.
44:56There's not going to There's not going to be a thing as like a really great harness that is in absence of a really great model and vice versa.
45:03Yeah.
45:03Makes sense.
45:05I think applications, especially as built on non-deterministic systems like models, operate in a very low-trust environment.
45:15So yes, most of the improvements will likely be distributed across both models and the harnesses, but I think overall it's it's it's mostly it's There There will There will have to be some form of controller
45:30trying to have some form of arbitration because even from the model perspective you aren't in a perfectly visible world.
45:37You don't know the behavior of every model, so it's it's going to be at the orchestration level where you have these kind of things.
45:45And this has traditionally been shown by other industries like when web when web launched, you know, you had traffic-based routing.
45:52So it's different.
45:53But all the sort of routing controls have been centralized over time.
45:59Makes sense.
46:01I I it's most likely going to be good news in the future.
46:05Um and and I I think like caching is a big reason for that.
46:10Even if you I think like a to take the flip side of of this argument, um the you know, it might be that in the future we have like one big model that's like, "I know I am the like most efficient at everything
46:22and I'm like way more efficient than Haiku.
46:25I'll solve every task better than Haiku can at like a lower price.
46:29Um why should I ever delegate to Haiku?"
46:31There's something like that actually could could be a model that we have in the future.
46:36Um but you're always going to have these like, you know, for example, caching.
46:41It could be that like you tell the model that this other model like does have the right context in cash and uh you know, the the orchestrator model just always has more context
46:52and the models have to be aligned.
46:54So, I I think like it's I I don't really see a world where like we wouldn't be able to get models to collaborate really well and and I think they're going to get better over time.
47:04Um in part because they're you know, they just have limited memory.
47:08So, I I think that's kind of one one deciding factor and another is that um there will be like uh there will continue to be like if you just look at like the the the rankings on OpenRouter, if you look at our our public data and you look at like
47:25the top model being used by dollar spent on classification tasks, well, guess what it is.
47:33It's Opus.
47:35[laughter]
47:35I think there are there are there are big opportunities for like using small models for in distribution easy tasks and the and like as time goes on, that's going to be a larger and larger percentage of tasks relative to like the most valuable tasks that
47:51um very smart models spend most of their time on.
47:54Totally.
47:54Well, I want to thank you guys so much.
47:56Can we all give everyone a round of applause?
47:58Thank you.
47:59[applause]
48:04[music]