WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Rx8f05JI_WA NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Rx8f05JI_WA.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.160 --> 00:00:02.080 Hi everyone. s2 00:00:02.080 --> 00:00:04.040 My name is Rishi Desai. s3 00:00:04.040 --> 00:00:11.920 I'm an ML engineer at Abundant AI, where we build reinforcement learning environments for Frontier Labs. s4 00:00:11.920 --> 00:00:18.880 Today, I'm going to talk about SWE Marathon, a benchmark that answers a question that is starting to matter a lot more. s5 00:00:18.880 --> 00:00:23.600 Can coding agents stay coherent over a billion token budget? s6 00:00:23.600 --> 00:00:26.560 Can they build Slack from scratch? s7 00:00:26.560 --> 00:00:30.800 Can they rewrite an entire JAX code base in PyTorch? s8 00:00:30.800 --> 00:00:34.160 Can they build a C compiler in Rust? s9 00:00:34.440 --> 00:00:36.680 This is what SWE Marathon is trying to measure. s10 00:00:36.680 --> 00:00:43.920 What happens when coding agents move from fixing bugs to owning entire projects end to end? s11 00:00:47.640 --> 00:00:52.560 There's been a tremendous amount of interest in autonomous agent systems. s12 00:00:52.560 --> 00:00:57.840 Anthropic has explored teams of agents building a C compiler. s13 00:00:57.840 --> 00:01:04.800 Cloudflare rebuilt the entire Next.js on Vite, completely hands-off with agents. s14 00:01:04.800 --> 00:01:11.440 And Cursor has experimented with their days-long running autonomous agent harness. s15 00:01:11.440 --> 00:01:20.120 The pattern is that coding agents are being pointed at whole projects, not just GitHub issues or linear tickets. s16 00:01:20.560 --> 00:01:28.760 My question is, can we turn some of these Frontier Labs style case studies into reproducible eval tasks? s17 00:01:32.720 --> 00:01:36.520 Let's talk about the SWE benchmark lineage. s18 00:01:36.680 --> 00:01:42.440 Human Eval asked whether models could write individual Python functions. s19 00:01:43.040 --> 00:01:54.720 SWE-bench was a big jump to real GitHub issues, where agents had to inspect a repository, make a patch, and patch some unit tests. s20 00:01:54.840 --> 00:02:01.160 Terminal-bench pushed this even further by making each task a full environment with a verifier. s21 00:02:01.160 --> 00:02:10.479 So, agents could use a terminal, run bash commands, inspect files, and leave behind a final container state. s22 00:02:10.800 --> 00:02:18.959 SWE Marathon takes that environment plus verifier framing and stretches the horizon to project scale work. s23 00:02:18.959 --> 00:02:23.800 Multi-hour trajectories and coordinated changes across many, many components. s24 00:02:23.800 --> 00:02:31.040 These are literally hundreds of hours of human work compressed into a single agent rollout. s25 00:02:34.000 --> 00:02:39.520 But, once you make tasks this long, a big problem shows up. s26 00:02:39.520 --> 00:02:41.520 Verification. s27 00:02:42.640 --> 00:02:48.120 In a short benchmark, a weak test could just be considered as noise. s28 00:02:48.120 --> 00:02:55.000 But, in a multi-hour environment, a weak verifier becomes an attack surface. s29 00:02:55.000 --> 00:03:02.360 The agent has hours, a file system, unrestricted network access potentially, and a reward signal. s30 00:03:02.360 --> 00:03:09.680 So, it could spend hours probing the verifier instead of actually doing the intended engineering work. s31 00:03:09.680 --> 00:03:15.519 That's a big reason why SWE Marathon uses multiple independent checks. s32 00:03:15.959 --> 00:03:26.080 We have hidden tests, reference parity checks, computer use agent checks for the product clone tasks, and anti-cheating tests. s33 00:03:26.360 --> 00:03:32.080 We wanted independent verified channels that fail in different ways. s34 00:03:32.080 --> 00:03:46.120 I'll first show you the computer use agent verification example, and then later the failure case where an agent tries to solve the C compiler task by secretly calling GCC. s35 00:03:49.880 --> 00:03:57.680 You You have noticed that there are basically no full stack product clone tasks in any long horizon three benchmark out there. s36 00:03:57.680 --> 00:04:00.600 And the reason is verification. s37 00:04:01.600 --> 00:04:10.200 Unit test can pass, but the product is probably still unusable and the front end looks terrible. s38 00:04:10.680 --> 00:04:18.959 Sweet marathon is the first benchmark to use a computer use agent or CUA verifier for these full stack tasks. s39 00:04:18.959 --> 00:04:26.440 For the clone slack task, we have deterministic unit tests to check the API and the back end functionality. s40 00:04:26.840 --> 00:04:31.400 But then a computer use agent uses the browser like a human. s41 00:04:31.400 --> 00:04:33.680 That's what you're seeing in this GIF. s42 00:04:33.680 --> 00:04:38.600 The verifier isn't reading code or calling an API directly. s43 00:04:38.600 --> 00:04:43.560 It's driving the submitted slack clone through the UI. s44 00:04:43.560 --> 00:04:53.560 So it's logging in, creating channels, posting messages, reacting with emotes, and checking that the app actually works with the rubric. s45 00:04:54.120 --> 00:05:00.320 The big takeaway is that full stack eval's are hard because correctness is not just an API contract. s46 00:05:00.320 --> 00:05:05.919 It's whether the user can actually complete the product's intended workflow. s47 00:05:09.680 --> 00:05:15.360 Sweet marathon has 20 project scale tasks across four families. s48 00:05:15.360 --> 00:05:23.320 There are library clones, full stack product clones, ML engineering, and algorithmic tasks. s49 00:05:23.320 --> 00:05:25.720 And some of these tasks even use external APIs. s50 00:05:25.720 --> 00:05:33.600 For example, we have a post train task where the agent must post train a language model using the tinker API. s51 00:05:34.480 --> 00:05:40.400 Expert contributors from the eval's community propose the tasks and reference solutions. s52 00:05:40.400 --> 00:05:47.960 And then we work together to standardize them into executable environments with the multi-layer verifier suites. s53 00:05:47.960 --> 00:05:51.880 Tasks all follow the harbor format. s54 00:05:52.440 --> 00:05:56.080 A lot of my work was spent on the QA and the hardening layer. s55 00:05:56.080 --> 00:06:10.320 So, running the agent trials, inspecting the failure modes, patching the shortcuts, patching the verifier, and then rerunning until the tasks were both solvable, but also hard to game. s56 00:06:15.360 --> 00:06:17.919 This is the main leaderboard result. s57 00:06:17.919 --> 00:06:26.840 The best configuration here is Claude Opus 4.8 with Claude Code, and it only achieves a 26% resolution rate. s58 00:06:26.840 --> 00:06:34.600 So, even with the strongest agent setup we evaluated, it's only solving like one in four tasks. s59 00:06:35.880 --> 00:06:38.680 The important thing is that these aren't shallow failures. s60 00:06:38.680 --> 00:06:46.360 The average trial used 31 million tokens, and the longest rollout consumed 877 million tokens. s61 00:06:46.360 --> 00:06:54.280 So, the agents are exploring, editing, testing, getting stuck, recovering, running for hours. s62 00:06:55.480 --> 00:07:06.400 So, the takeaway is that current agents are very impressive, but end-to-end project ownership ownership is still very far from being solved. s63 00:07:10.240 --> 00:07:15.680 This plot puts cost on the x-axis and resolution rate on the y-axis. s64 00:07:15.680 --> 00:07:21.080 So, higher success rate for less money is always better. s65 00:07:21.600 --> 00:07:24.440 Claude Opus 4.8 is the top point. s66 00:07:24.440 --> 00:07:30.520 It gets 26%, but it's also the most expensive configurations, or one of them. s67 00:07:30.520 --> 00:07:37.240 Whereas GPT 4.5 with Codex is far cheaper and only gets 12%. s68 00:07:38.880 --> 00:07:41.280 So, the model isn't just the full picture. s69 00:07:41.280 --> 00:07:51.240 The agent scaffold makes a huge difference uh how the how it plans, uses tools, um summarizes context, and decides on when to test. s70 00:07:51.240 --> 00:07:56.800 I won't get too deep into the cost analysis here, but the paper has the full details. s71 00:08:00.560 --> 00:08:06.560 I wanted to show you what a full marathon rollout actually looks like. s72 00:08:06.560 --> 00:08:13.800 This is one I picked with GLM 5.2 on the Next.js fight rewrite task. s73 00:08:13.800 --> 00:08:23.840 So, there's over, you know, 356 million tokens uh over 9 hours and over 800 trajectory steps and and tool actions. s74 00:08:23.840 --> 00:08:36.080 So, for the top half, you can see the the agent starts by exploring the repo and the fixtures, um gets its first full test suite at zero of zero out of 325 s75 00:08:36.080 --> 00:08:46.440 test passing, and then spends the next few hours pushing through routing, hydration, uh server actions, middleware, and cache behavior. s76 00:08:47.680 --> 00:08:51.400 The bottom part of the chart shows the work pattern over time. s77 00:08:51.400 --> 00:08:59.480 So, you can see like lots of reading and searching early, then huge waves of like editing, building, testing, and debugging. s78 00:08:59.480 --> 00:09:03.520 The key intuition is that these are like long engineering loops. s79 00:09:03.520 --> 00:09:06.480 They're not simple coding tasks. s80 00:09:10.560 --> 00:09:16.120 Reward hacking is an arms race between coding agents and our environment. s81 00:09:16.280 --> 00:09:23.120 This is why strong verifiers are are central to Sweep Marathon's task design and not an afterthought. s82 00:09:23.280 --> 00:09:26.360 This chart has two levels of behavior. s83 00:09:26.360 --> 00:09:32.120 The lighter bars are the suspicious suspicious uh shortcut behavior. s84 00:09:32.120 --> 00:09:37.880 So, things like looking for solution files, messing with data, messing with the configs. s85 00:09:37.880 --> 00:09:44.520 Whereas the darker bar is like a clear exploit that has actually gotten shipped in the final submission. s86 00:09:44.800 --> 00:09:56.000 And across the 1,400 rollouts, um we found 12.8% had suspicious shortcut behavior, and 9% had the clear verifier bypass. s87 00:09:56.000 --> 00:10:05.600 So, if these verifiers were weak, these wouldn't just be amusing failure cases, they would actually delegitimize the benchmark. s88 00:10:05.600 --> 00:10:08.360 And the important number is the zero. s89 00:10:08.360 --> 00:10:13.720 Zero rollouts earned reward through an exploit, because our defenses caught them. s90 00:10:13.720 --> 00:10:18.040 That should be the bar for long-horizon evals. s91 00:10:20.200 --> 00:10:25.040 This is my favorite concrete reward hacking example. s92 00:10:25.560 --> 00:10:28.600 The task is to build a C compiler in Rust from scratch. s93 00:10:28.600 --> 00:10:34.760 The lexer, the parser, semantic analysis, codegen, the whole thing. s94 00:10:35.640 --> 00:10:45.200 But Gemini found a much shorter implementation strategy, which is call GCC from inside the Rust program. s95 00:10:45.200 --> 00:10:53.800 So, under a weak verifier, this task would look almost solved, because the compiler outputs match the reference behavior behavior. s96 00:10:53.800 --> 00:10:57.440 But obviously it's not a real compiler in Rust. s97 00:10:57.640 --> 00:11:07.520 The anti-cheat layers ca- uh catch this by using S trace to find the forbidden subprocesses um called like GCC. s98 00:11:07.960 --> 00:11:14.120 So, even though the partial scores look high, the final reward is zero. s99 00:11:14.600 --> 00:11:21.200 I have the full failure mode taxonomy in the paper, um which I hope you guys will check out. s100 00:11:25.600 --> 00:11:32.680 If you remember one thing from this video, it's that the future of SWE evals is not just harder unit tests. s101 00:11:32.680 --> 00:11:47.640 Once agents run for hours, each task becomes a complex environment, and agents not only trying to write code, it's also navigating tools, tests, your hidden assumptions, and the verifier itself. s102 00:11:48.000 --> 00:11:56.400 So, the two big takeaways are first, long horizons we is still unsolved. s103 00:11:56.600 --> 00:11:58.480 The best agents only at 26%. s104 00:11:58.480 --> 00:12:01.080 There's plenty of headroom left. s105 00:12:01.080 --> 00:12:06.160 Second, the big bottleneck is robust verification. s106 00:12:06.160 --> 00:12:15.280 At hour and day scale length tasks, we need the multi-channel checks, anti-cheat hardening, product style validation. s107 00:12:20.600 --> 00:12:26.120 The tasks, the code, the paper, the logs, and the trajectories are all public. s108 00:12:26.120 --> 00:12:36.880 I've released 320 GB of trajectories that are especially important because they make SWE-bench fully inspectable and transparent. s109 00:12:36.880 --> 00:12:42.800 I also want to thank all of my collaborators on this project, all of whom are listed here. s110 00:12:42.800 --> 00:12:50.520 SWE-bench was very much a community-driven effort across task contributors, advisors, and paper writing. s111 00:12:50.520 --> 00:12:53.920 You can find everything at swe-bench.org. s112 00:12:54.960 --> 00:12:57.160 Thank you.