{ "video": { "id": "QHBjufYK8TA", "title": "The State of Model Routing — NVIDIA, Cognition, OpenRouter", "duration": 2897, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "QHBjufYK8TA", "title": "The State of Model Routing — NVIDIA, Cognition, OpenRouter", "one_liner": "A panel from Cognition, OpenRouter and NVIDIA argues that model routing is not about downgrading to a cheaper model but about keeping a frontier model in charge while it delegates work — Cognition claims 40% cheaper Fable-level intelligence, and OpenRouter warns a too-small model on out-of-distribution work can cost 10x more than Opus.", "summary": "The panel's shared thesis is that naive task-type routing — 'this benchmark says the small model is better at X, so send X to the small model' — is extremely fragile once tasks are agentic, because complexity shifts mid-session as a developer moves from 'explain this code base' to 'implement this' to 'debug this live'. Cognition's Devin Fusion instead keeps a frontier model always present to plan and supervise while a 'sidekick' with a continuously running context does the implementation, cutting the cost of Fable-level intelligence by 40% while exploring more comprehensively because cheap tokens buy more depth. OpenRouter's Alex adds the counter-case: small models out of distribution thrash on tool calls, and on terminal bench Opus scores ~3x better at 1/10 the cost of Haiku despite Haiku being far cheaper per token. All three agree the field is early, that caching economics (5-minute KV cache windows, 10x cache discounts, compaction cache misses) drive most real design decisions, and that the next step is co-designing models with the orchestration system rather than routing between models as they are.", "key_points": [ "Cognition's Devin Fusion reduces the cost of Fable-level intelligence by 40% by letting Fable do planning and hard decisions while delegating implementation to a cheaper/open-source model; because delegated tokens are cheap you can spin off three sub-agents to explore a code base more comprehensively than Fable would alone.", "Cognition deliberately rejects a main-agent-plus-sub-agents design in favour of a 'sidekick' — one sub-agent with a continuously running context, so the main agent never re-provides context and everything stays in the KV cache at ~10x cheaper cached-token rates; the smart model can sit on either side and be swapped.", "Alex (OpenRouter): on terminal bench, Opus scores about three times better at 1/10 the cost of Haiku even though Haiku is much cheaper per token — out-of-distribution small models blow up cost through excessive tool calls and crazy loops, so in-domain/out-of-domain detection is the core routing work.", "The NVIDIA researcher frames routing as exploiting 'jagged capabilities': a model's strengths come from its training corpora (scikit-learn vs matplotlib within data visualisation), post-training teachers and distillation gaps, so a higher coding-benchmark score doesn't mean better at every sub-task — on LM Router Bench these complementarity techniques can yield up to 10% higher accuracy 'essentially for free'.", "OpenRouter's auto router existed for ~2 years with almost no adoption until it exploded around January this year, driven by open claw's 10-minute heartbeat pings — one app with two completely different intelligence needs, which made market segmentation make sense; OpenRouter now ships Pareto Code (Pareto-optimal coding model at a tunable threshold) and Fusion (multi-model orchestration), with published results so far on deep research, not coding, where the smart model as the outer/wrapper model worked best.", "Compaction is mainly an intelligence tool, not a cost tool: compacting takes a cache miss so you pay ~10x for those input tokens, and Walden would 'never recommend' using million-token-context models past ~200K, preferring under 100K — compaction pays off when you're taking a cache miss anyway, e.g. when switching models.", "Detecting that a small model is out of its depth: Cognition exploits the ~5-minute KV cache refresh cadence to get an effectively free frontier-model check-in on whether the sidekick is going down a rabbit hole; NVIDIA suggests hallucination/linear probes over internal state and perplexity as a proxy for 'how lost a model is'.", "NVIDIA's Flex Run distills a main model into smaller footprints and switches which model does the decoding by task, activating only a section of weights; open models let you check in-distribution-ness directly against the training recipe, and self-hosting lets you optimise cache duration and pricing for your actual workload shape instead of a provider's amortised average.", "Both labs are moving toward co-designing models for collaboration — Cognition is RL-ing models both as orchestrator and as sidekick executor, and sees new frontier models (Fable, GPT-5.5/5.6) as already naturally collaborative and better at delegation." ], "takeaways": [ "Don't route by task type to a dumber model and walk away — keep a frontier model watching even when it isn't doing the work, because the complexity and domain of an agentic session change mid-flight and a stranded small model sends users back to the expensive model anyway.", "Classify in-domain vs out-of-domain before routing down. Straightforwardly in-distribution work (e.g. is this string a person's name or an organisation's?) belongs on a small model; novel or out-of-domain work on a small model can cost more than Opus.", "Design routing around the cache, not just per-token price: use one long-lived sidekick context instead of many sub-agents, pass results back by file reference rather than dumping full context, and treat a scheduled cache refresh as a free opportunity for a frontier-model check-in.", "Use compaction for intelligence and for model switches — the cache miss makes it a poor pure cost optimisation — and lean on non-lossy external stores (the file system, ASTs) so the harness can re-find detail it didn't keep in context.", "Prefer having a smart model diagnose and rewrite prompts (ask it why it made the decision, cite the prompt, rerun as a regression) over low-level mechanical prompt-tuning frameworks; and tune your router against real production signals like users manually upgrading or downgrading the model." ], "topics": [ "model-routing", "agents", "cost-optimization", "kv-caching", "context-engineering", "orchestration", "local-inference", "evals" ], "tools": [ "Cognition", "Devin", "Devin Fusion", "OpenRouter", "OpenRouter Fusion", "Pareto Code", "NVIDIA", "Nemotron / NeMo Triton models", "Flex Run", "Dynamo", "DGX Spark", "Fable", "Opus", "Haiku", "GPT-5.5 / 5.6", "Anthropic", "terminal bench", "LM Router Bench", "open claw", "scikit-learn", "matplotlib", "S3", "GEPA" ], "quotes": [ { "text": "we're reducing the cost of Fable level intelligence by 40%. The way we do that is we allow Fable to still do like the planning and the the hard decision making but delegate a lot of the work to an implementation model.", "at": "04:34", "url": "https://www.youtube.com/watch?v=QHBjufYK8TA&t=274s" }, { "text": "Like if you run terminal bench on Opus and Haiku, like Opus will do about three times better at 1/10 the cost of Haiku, even though Haiku's significantly cheaper per token.", "at": "15:26", "url": "https://www.youtube.com/watch?v=QHBjufYK8TA&t=926s" }, { "text": "I would I would encourage everyone to think, \"Hey, models are strong at different things, rather than like there's one model to rule them all.\"", "at": "06:54", "url": "https://www.youtube.com/watch?v=QHBjufYK8TA&t=414s" }, { "text": "It's not obvious that actually more expensive models are actually creating an overall cheaper system.", "at": "24:27", "url": "https://www.youtube.com/watch?v=QHBjufYK8TA&t=1467s" } ], "words": 10894 }, "summary_url": "/#QHBjufYK8TA", "transcript": { "html": "/transcripts/QHBjufYK8TA.html", "txt": "/transcripts/QHBjufYK8TA.txt", "vtt": "/transcripts/QHBjufYK8TA.vtt" } }