{ "video": { "id": "7vn4WpqNpck", "title": "Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs", "duration": 1088, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "7vn4WpqNpck", "title": "Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs", "one_liner": "Wisedocs' Denys Linkov re-ran his team's 6-month, 10-repo-to-monorepo AI-pipeline refactor against today's models to answer whether they should have just waited — the same task that took 3 hours of back-and-forth with o3 and 10 major mistakes was near one-shot by Opus 4.8, but GPT 5.5 still only scaffolded the full refactor, so the refactor was worth doing when they did it.", "summary": "Linkov frames a real business decision — Wisedocs was too slow for customer demand, its AI pipeline was too complicated to update, and nobody wanted to touch 10+ legacy repos — and asks whether the 6-month refactor into a monorepo was right, or whether they should have waited for better models. He replays the actual refactor tasks on newer models to measure how much cheaper the work has become, and shows the productivity curve after the rebuild reached parity. His answer is yes: even though agentic coding is getting exponentially better, models still can't self-validate a full refactor, AI-native code starts to look like legacy code, and the rebuilt monorepo cut pipeline time and cost while letting almost every developer in the company contribute.", "key_points": [ "The use case: Wisedocs processes complex medical claims as PDFs over 10,000 pages — 'some of these files are bigger than video files' — so scaling the ML pipeline is non-trivial.", "Orchestrator selection took ~2 months: five open-source projects, 17 evaluation criteria in a Confluence doc, proof-of-concepts built by a team of three. He says he's 'pretty confident we could do this 90% faster now' with deep research plus sub-agents per criterion — but warns of 'AI psychosis', where a 20-page deep research report describes product features that don't actually exist.", "Benchmark replay of one Temporal activities/workflow refactor: with o3 in Cursor it took 3 hours of back-and-forth and the model made 10 major mistakes; Sonnet 4.6 solved it with one additional iteration; Opus 4.8 basically one-shot it. Rebuilding the same task today would take ~1/5 the time.", "The interaction pattern changed, not just the score: o3 made few tool calls in certain categories, while modern harnesses bring sub-agents, plan calls, shell commands and verification steps — more expensive per run, far less manual.", "On METR's time-horizon graph: the usual 50% success-rate version overstates things; look at 80%, and ideally 90% or 99%, because that's where the plan-then-hand-off mental model works. On METR's Metis preview eval (~a month before the talk), success declines sharply around the 4-hour mark, and some tasks fail even below 15 minutes.", "Zero-shot attempt: GPT 5.5 extra high was given the repo names, models and components and 'completed' the refactor goal in 10 minutes 22 seconds — but wrote only 2,000 lines, having implemented scaffolding and skipped the models (its own note: it had not added a Ray Serve deployment or bootstrap command). He predicts substantial refactors will be consistent in ~6 months.", "Post-refactor productivity: the previous repos were 6+ years old with slow progress; the rebuild hit parity in six months with a steep commit curve that kept climbing as new features were added, and almost every developer in the company now commits to the monorepo even outside their area of expertise.", "Business outcome: shorter pipeline runtime, lower costs, support for larger files, and features that used to take multiple months now ship in under a week — plus developers actually want to work in the codebase, and its patterns have spread to other repos.", "Counter-argument he takes seriously: shipping velocity is up industry-wide but product quality and reliability aren't — he showed two unnamed leading companies below three-nine or four-nine uptime.", "Q&A specifics: they hit 15 of 17 requirements in the refactor; plan mode was barely arriving in Claude Code and absent from Cursor when they started, and they adopted it; PR reviews were all human, with local skill-driven code review, because PRs built shared context across the few developers doing the refactor." ], "takeaways": [ "Judge model capability at the 80/90/99% success rate, not the headline 50% — kicking off an hour-long agent run with a coin-flip success rate usually means you wasted the hour and your attention.", "Treat deep research output as a hypothesis, not a finding: verify that the features it attributes to a library or product actually exist (and aren't in beta) before building on them.", "Put the guardrails and primitives in place — plan/spec discipline, sandboxes, verification, monitoring — whether or not you do a full refactor; AI-native code accumulates the same legacy problems (lots of code, low quality, nobody understands it) faster.", "If you're weighing multiple repos vs a monorepo: models navigate multiple repos fine from a higher-level folder, but end-to-end testing, verification, deployment and sandbox setup are all materially harder and slower to clone across repos.", "Don't defer a refactor purely because models will be better next year — evaluate the business ROI of the cleanup now, since velocity, cost and developer willingness to touch the code compound while you wait." ], "topics": [ "coding-agents", "refactoring", "tech-debt", "monorepo", "benchmarking", "agentic-workflows", "developer-productivity", "evals" ], "tools": [ "Wisedocs", "Anthropic", "Spotify", "Stripe", "OpenAI", "Google", "deep research", "Confluence", "Temporal", "o3", "Cursor", "Claude Code", "Sonnet 4.6", "Opus 4.8", "GPT 5.5", "METR", "Metis", "Ray Serve" ], "quotes": [ { "text": "This refactor took uh 3 hours of back and forth chatting with Incursr, but it made 10 major mistakes.", "at": "05:32", "url": "https://www.youtube.com/watch?v=7vn4WpqNpck&t=332s" }, { "text": "it's very easy to undergo AI psychosis, where you look at a deep research report that's 20 pages long and you say, \"Wow, this looks good.\" And then those features don't actually exist in the product, and you've set yourself back.", "at": "04:50", "url": "https://www.youtube.com/watch?v=7vn4WpqNpck&t=290s" }, { "text": "if you're kicking off a process that is going to take an hour and it has a 50% chance of completing, there's a very high chance you just wasted that hour and you could have been doing something different.", "at": "08:48", "url": "https://www.youtube.com/watch?v=7vn4WpqNpck&t=528s" }, { "text": "So, models will continue to get better, uh but sometimes it's good to pause, build a monorepo, and forge ahead.", "at": "14:51", "url": "https://www.youtube.com/watch?v=7vn4WpqNpck&t=891s" } ], "words": 4025 }, "summary_url": "/#7vn4WpqNpck", "transcript": { "html": "/transcripts/7vn4WpqNpck.html", "txt": "/transcripts/7vn4WpqNpck.txt", "vtt": "/transcripts/7vn4WpqNpck.vtt" } }