{ "video": { "id": "maitMe9HCvw", "title": "How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere", "duration": 1882, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "maitMe9HCvw", "title": "How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere", "one_liner": "In a lead-plus-workers coding agent team, where you put the extra agents matters more than how many: scale workers at your task's bottleneck stage (for ProgramBench, that was planning) and pair a strong lead with cheap workers. This gets near-frontier results at up to 70% lower cost.", "summary": "Dziri presents Cohere research on agent teams as a new way to scale inference-time compute. A lead hands out subtasks to workers that each start with a fresh context, which avoids the context blow-up of one long-running agent. Using a deliberately minimal hierarchical harness tested on ProgramBench, the team frames team design as a compute allocation problem: reward per added dollar over a solo agent. Teams beat solo agents by up to ~10 points but cost 4-7x as much. Putting cheap workers only at the bottleneck stage, under a strong lead, recovers most of the reward at a fraction of the cost.", "key_points": [ "The harness is a top-down hierarchy. A lead delegates through a 'delegate agent' tool, all agents share one workspace through a bash tool, and the lead can resume, kill or supervise workers. A blocking lead waits for every report; an async lead keeps working meanwhile.", "ProgramBench is a reverse-engineering task: rebuild a program (e.g. SQLite, the PHP interpreter, FFmpeg) from an executable you can run but not read, plus its docs. On FFmpeg the best model on the leaderboard passes only 10% of the tests.", "With a minimal prompt, a blocking-lead team beats the solo agent by almost 10 points at 4-7x the cost. The async lead does worse: it builds on its own guesses before workers report back, ignores or redoes their work, and sometimes submits while workers are still running.", "A heavy 'PIV' prompt (explicit plan, folders, adversarial verifier workers) lifts the solo agent by ~6 points and lets the async lead catch up with the blocking one. But the team then costs 6-7x as much as solo, and the prompt tuning took a lot of time.", "Putting workers only at planning matches full PIV reward at half the cost. Each planning dollar buys ~1.6x more reward than other stages, and 2.4x more reward per call. Full PIV kept delegating through the whole run: 830 extra calls, around $500, no reward gain. Extra implementation or verification workers just add cost.", "Implementation workers fall short because they build on a weak plan and each sees only a narrow slice. Given more context, they overstep their boundaries and build features nobody asked for. 84% of messages go worker-to-lead and only ~4% worker-to-worker (less than one message per task).", "On Commit0 the bottleneck is implementation, not planning, because the skeleton, docs and unit tests are given. There, doubling implementation agents gave the biggest gain.", "Using GPT-5.6 Luna/Terra/Sol (cheapest to strongest): a Sol lead with Luna workers gets almost the same reward as all-Sol at 60% less cost. A weak lead can't be rescued, though: Terra plus Luna workers scores 51% vs 57% for Sol alone. Sol lead plus 16 Luna planners is 38% cheaper than an all-Sol planning team and 70% cheaper than workers at every stage.", "Peer messaging didn't help. About half of Luna's messages were malformed and rejected by the harness, Sol wrote 3-4x as much message text, and Luna teams lost 4-7 points with messaging on. Across model families, 5.6 teams delegated often and gained up to 7 points, Claude gained ~6, Opus 4.7 rarely delegated and gained little, and Gemini delegated a lot but gained nothing." ], "takeaways": [ "Find the software-engineering stage that bottlenecks your task (planning/discovery vs implementation vs verification) and add workers there, not everywhere.", "Use one early burst of parallel exploration workers that send compact reports back, rather than letting the lead keep delegating all the way through.", "Put your strongest model in the lead role and use cheap models as workers. The lead sets the ceiling and the workers set the price.", "Prefer a blocking lead, or explicitly tell an async lead when to wait, so worker results aren't lost or redone.", "Don't assume peer-to-peer messaging improves results. Measure it, especially with weaker models, where it can distract or produce malformed messages." ], "topics": [ "multi-agent", "coding agents", "inference-time compute", "cost optimization", "orchestration", "agent communication", "benchmarks" ], "tools": [ "Cohere", "Mila", "Allen Institute", "ProgramBench", "Commit0", "GPT-5.6 Luna", "GPT-5.6 Terra", "GPT-5.6 Sol", "Claude Opus 4.7", "Gemini", "SQLite", "PHP interpreter", "FFmpeg", "Meta" ], "quotes": [ { "text": "the lead sets the ceiling and the workers set the price", "at": "21:47", "url": "https://www.youtube.com/watch?v=maitMe9HCvw&t=1307s" }, { "text": "where you put agent matters more than how many", "at": "28:40", "url": "https://www.youtube.com/watch?v=maitMe9HCvw&t=1720s" }, { "text": "Another 830 calls and they buy nothing actually.", "at": "16:20", "url": "https://www.youtube.com/watch?v=maitMe9HCvw&t=980s" }, { "text": "They get excited and then they want to build extra features that nobody asked them for it.", "at": "17:36", "url": "https://www.youtube.com/watch?v=maitMe9HCvw&t=1056s" } ], "words": 6550 }, "summary_url": "/#maitMe9HCvw", "transcript": { "html": "/transcripts/maitMe9HCvw.html", "txt": "/transcripts/maitMe9HCvw.txt", "vtt": "/transcripts/maitMe9HCvw.vtt" } }