WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Yk87oUPVaxU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Yk87oUPVaxU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.480 --> 00:00:14.080 Hey everyone, can you guys hear me? s3 00:00:14.080 --> 00:00:15.440 Okay, this is good. s4 00:00:15.440 --> 00:00:17.359 Um, yeah, my name is James. s5 00:00:17.359 --> 00:00:19.920 I'm one of the founding engineers at Data Curve. s6 00:00:19.920 --> 00:00:23.920 Unfortunately, Serena's been out with a fever for the past couple of days. s7 00:00:23.920 --> 00:00:26.400 Um, she was supposed to be here giving this talk. s8 00:00:26.400 --> 00:00:33.520 Um, so I'm just filling in uh in her place, but I've been at uh data curve working on the research and engineering side of things. s9 00:00:33.520 --> 00:00:40.559 Um, as well as Deep Suite, which is our uh frontier long horizon coding benchmark, which you guys may be familiar. s10 00:00:40.559 --> 00:00:45.520 Um, I'll just be going over, you know, some of the most important findings about Deep Suite. s11 00:00:45.520 --> 00:00:49.840 Um, a brief overview of what it is for those of you who don't who who may not know. s12 00:00:49.840 --> 00:00:56.719 um and then going deeper into our methodology and exactly how we came about this uh uh frontier coding benchmark. s13 00:00:56.719 --> 00:01:07.119 So um deepu is a long horizon software engineering software engineering benchmark comprised of 113 original software engineering tasks. s14 00:01:07.119 --> 00:01:12.960 So this means unlike something like sweet bench pro we didn't scrape this from um existing PRs that have been closed. s15 00:01:12.960 --> 00:01:15.600 Um there's a variety of benefits for this. s16 00:01:15.600 --> 00:01:22.880 Uh namely one of them is to resist against contamination and agents being able to cheat uh through the course of their rollouts. s17 00:01:22.880 --> 00:01:29.520 Um Swebench has or Swebench Pro uh pulls thousands of tasks from only 40 repositories. s18 00:01:29.520 --> 00:01:33.280 The median task per repository for us is one. s19 00:01:33.280 --> 00:01:38.000 So you can see across over a 100 tasks we pull from nearly 100 repositories. s20 00:01:38.000 --> 00:01:42.799 And the language spans across uh Typescript, JavaScript, Python, Rust, and Go. s21 00:01:42.799 --> 00:01:46.799 And we have plans to add uh more languages later on. s22 00:01:46.799 --> 00:01:51.840 Um since its uh release, we've received uh very positive reception. s23 00:01:51.840 --> 00:02:05.040 It's replaced SweetBench Pro in the artificial analysis uh coding um agent index as well as being cited by uh numerous Frontier model labs and us helping with them in uh tracking their models on our benchmark as well. s24 00:02:05.040 --> 00:02:08.560 So, we've been really really appreciative of that. s25 00:02:08.560 --> 00:02:10.239 Uh bit of context about us. s26 00:02:10.239 --> 00:02:17.920 Data curve works on building uh training data for high ceiling domains including coding as well as coding adjacent fields. s27 00:02:17.920 --> 00:02:32.480 Uh we also are trying to answer the very uh elusive question of what exactly makes uh good data, what is data quality and how can we demonstrate that our training data in fact moves the needle. s28 00:02:32.480 --> 00:02:40.000 So Deep Suite is one in a long line of initiatives that we have towards answering this question. s29 00:02:40.239 --> 00:02:42.000 So why did we create Deep Suite? s30 00:02:42.000 --> 00:02:46.319 Well, it was very clear that the existing benchmarks are not hitting the mark. s31 00:02:46.319 --> 00:02:51.280 Um with benches like Sweetbench Pro, uh top models are clustering at the top. s32 00:02:51.280 --> 00:02:58.480 It's very hard to differentiate between uh which one is good because they all have overlapping confidence intervals. s33 00:02:58.480 --> 00:03:03.680 uh contamination is also rampant because again all of these tasks are mined from public PRs. s34 00:03:03.680 --> 00:03:10.560 So all the solution tests even the discussion around the PRs those are all available out in the wild for these agents to access. s35 00:03:10.560 --> 00:03:19.120 The verifiers are also very very brittle because we're anchoring them to a specific implementation often derived from the PR that was merged in. s36 00:03:19.120 --> 00:03:30.560 And oftentimes you also have tests that check for private helpers and functions uh created by the task author which is uh very opinionated right and is not something that models should have to adhere to. s37 00:03:30.560 --> 00:03:32.640 Um and finally uh leakage. s38 00:03:32.640 --> 00:03:47.200 So one thing about Sweetbench Pro is uh for very uh insightful models such as Claude, they're able to directly run git log and then go through the commit hashes and cherrypick the ones out that contain the golden patches which again very very serious issue. s39 00:03:47.200 --> 00:03:48.640 So this is deep suite. s40 00:03:48.640 --> 00:03:51.599 This is the updated leaderboard as of July 1st. s41 00:03:51.599 --> 00:03:59.680 You can see um I was mentioning before uh the problem of differentiating uh but you can see on deep suite here there is a very clear difference. s42 00:03:59.680 --> 00:04:07.040 there's a very clear performance gap between the top performing models versus you know at 10th place you have Gemini 3.1 Pro. s43 00:04:07.040 --> 00:04:17.840 Um also within the cloud and the GPT models as well we're able to see the some uh deviants and um yeah if you go on deepsweet.datacurve.ai you'll also be able to see s44 00:04:17.840 --> 00:04:26.720 um the token efficiency uh cost um token usage context window peak context all of that stuff on the deep site as well. s45 00:04:26.720 --> 00:04:33.280 But yeah, as of July 1st, uh, Fable 5 is retaining the top spot on our leaderboard. s46 00:04:33.919 --> 00:04:38.000 So, um, the ranking information is available online. s47 00:04:38.000 --> 00:04:46.000 I wanted to talk about some of the qualitative insights into how these different models are performing, which I think is the most interesting part. s48 00:04:46.000 --> 00:04:52.880 Um, starting with the first one is we find claude is generally a very very um, thorough and exhaustive model. s49 00:04:52.880 --> 00:04:57.840 it will um try to explore everything including uh go through all of the git logs. s50 00:04:57.840 --> 00:05:04.720 Uh so one interesting insight was seeing that it becomes quite forgetful when it comes to multi-art prompts. s51 00:05:04.720 --> 00:05:18.080 So when you tell it uh within the scope of a task, let's say to in support both synchronous and async versions of calling a hook, it will go ahead and implement the synchronous part, but it may drop the asynchronous s52 00:05:18.080 --> 00:05:19.039 part. s53 00:05:19.039 --> 00:05:25.520 We observed this in roughly two out of three cloud rollouts across all of the uh trials, all of the rollouts that we ran. s54 00:05:25.520 --> 00:05:37.520 So this was definitely quite interesting because from my experiences and developers I've talked to as well, Claude is generally very very thorough thorough and able to get at the developer intent quite well. s55 00:05:37.919 --> 00:05:43.039 Um another thing about claude is it pays very close attention to its environment. s56 00:05:43.039 --> 00:05:56.320 So it will often run uh this is taken from the trials uh we ran ourselves independently and also from examining sweetbench pro it'll attempt to run git log and recover the golden patch from the git history. s57 00:05:56.320 --> 00:06:12.800 We found that for opus 4.6 6 and 4.7 it did this 25% and 18% of the time respectively compared to all the Gemini models uh averaging at roughly 1% of the time and we found zero instances of this for the GPT models. s58 00:06:12.800 --> 00:06:21.199 So um thankfully within deep suite 1.1 we safeguarded further against models being able to cheat by pulling from the git history. s59 00:06:21.199 --> 00:06:29.039 Um but this was something yeah we observed quite frequently for claude within um the sweepbench pro rollouts. s60 00:06:29.039 --> 00:06:38.240 Um third finding is that GBT is very good at implementing exactly what it is asked across our failure mode analysis. s61 00:06:38.240 --> 00:06:42.639 We found that it was the least likely model to miss requirements. s62 00:06:42.639 --> 00:06:56.800 Um GPT 5.4 was the second best model at this ranking only behind GPT 5.5. it always uh learns to read the prompts and theos rep repository contract very literally and producing a patch that honors the existing conventions s63 00:06:56.800 --> 00:07:03.520 um signatures within the repository which is very helpful and we found that these traits converge across all rollouts. s64 00:07:03.520 --> 00:07:13.120 So these were not just uh lucky attempts but on average like this was the uh favorable um behavior exhibited by GPT. s65 00:07:13.280 --> 00:07:21.919 Um and finally we found that on average stronger me stronger models have a great tendency to want to test their own work. s66 00:07:21.919 --> 00:07:34.240 Um but with a caveat in SweetBench Pro's template they explicitly tell the model that the tests are handled and therefore they do not need to uh write uh any new tests of their own. s67 00:07:34.240 --> 00:07:45.680 With that single line in the prompt it will uh prevent the models from even uh 5.5 and uh Opus 4.8 from attempting to verify its own work through the course of the rollout. s68 00:07:45.680 --> 00:07:58.400 Um in deep suite we do not have anything that says to write or to not to write tests and so we observe this uh divergence between uh the percentage of the time where these models are actually engaging and writing tests. s69 00:07:58.400 --> 00:08:08.720 So this is quite an important behavior as it can uh provide the models are trying to obtain their own ways to verify and validate their work through the course of a rollout. s70 00:08:08.720 --> 00:08:20.479 uh we find on average that stronger models like 5.4 4.7 exhibit this the majority of the time whereas uh models like three flash and 3.1 pro are far less frequently s71 00:08:20.479 --> 00:08:22.879 um willing to test their own work. s72 00:08:22.879 --> 00:08:34.320 Um yeah so uh takeaways from the findings I think is is very interesting how um yeah stronger models on average exhibit um or converge on these behaviors. s73 00:08:34.320 --> 00:08:46.480 Um so moving on to the tasks right the the methodology behind Deep Suite um we made a decision to want to have every task authored uh from scratch uh rather than being mined. s74 00:08:46.480 --> 00:08:51.839 Um aside from the uh issues with contamination that we mentioned previously. s75 00:08:51.839 --> 00:09:05.680 Um this also plays into uh one of our core strengths which is that we offer a bespoke platform where we have um software engineers, machine learning enthusiasts come on and create these challenges and compete against one another. s76 00:09:05.680 --> 00:09:14.240 Um this platform is like shipped and or sorry this platform is called shipped and we have a version of this platform for every single domain that we're interested in. s77 00:09:14.240 --> 00:09:19.760 For example, for software engineering, it takes a lot after code forces or GitHub. s78 00:09:19.760 --> 00:09:22.560 Um, and we're really looking for enthusiasts. s79 00:09:22.560 --> 00:09:31.600 So, these are oftent times open-source engineers who are core contributors or maintainers of the projects that they're actively making tasks for. s80 00:09:31.600 --> 00:09:40.640 So, by creating these tasks from scratch, we know that the outputs are intrinsically aligned with our objective of providing a fair and comprehensive test to models. s81 00:09:40.640 --> 00:09:47.360 We also know that these people have have very thorough understandings of the repositories philosophy and the existing conventions. s82 00:09:47.360 --> 00:09:59.279 So they can make tasks that are both realistic in terms of the prompt but also realistic in the sense that uh this is an actual PR that you might see getting merged into the repositories. s83 00:10:00.720 --> 00:10:08.240 Um another uh very important design decision is we try as much as possible to make our prompts read like real tasks. s84 00:10:08.240 --> 00:10:17.760 On average, the average prompt uh characters within SweetBench Pro is over 4,500 characters, whereas for us, it's uh roughly half of that. s85 00:10:17.760 --> 00:10:35.680 And it's this is important because when you're prompting say a junior engineer or you're prompting a model to solve a very uh high ceiling ambiguous task, you're not going to be coming in there with a to-do list uh telling it to oh first do this and then do this and then write this function signature s86 00:10:35.680 --> 00:10:39.200 in exactly uh this way that I've prescribed on to you. s87 00:10:39.200 --> 00:10:48.880 oftentimes you're going to give it the high level objective, get it to explore and get it to reason about to uh the list of to-dos and ultimately to the solution on its own. s88 00:10:48.880 --> 00:10:51.519 Um so this was not the case in SweetBench Pro. s89 00:10:51.519 --> 00:10:57.279 It's very overly verbose and trying to prescribe a certain solution method onto agents. s90 00:10:57.279 --> 00:11:03.920 Um, as much as we could, we try and make Deep Suite prompts as tur and as uh high level as possible. s91 00:11:03.920 --> 00:11:13.760 Um, mirroring what you might see in the real world if you were to prompt, say, another another engineer or one of your agents to go and solve an engineering task. s92 00:11:14.160 --> 00:11:20.320 Um, so even though our prompts are short, we still are able to maintain the long horizon nature of these tasks. s93 00:11:20.320 --> 00:11:31.360 Even with our prompts again being roughly half the size of Sweetbench Pros, we find that the average size of our solution is five times the lines of code um compared to Sweepbench Pros. s94 00:11:31.360 --> 00:11:38.000 Um we also verified that uh there are on average seven files being touched in the agents solution. s95 00:11:38.000 --> 00:11:43.839 Um and across the course of a roll out we have two times more output tokens being emitted. s96 00:11:43.839 --> 00:11:55.279 Um and um finally we have uh we verifier design is of course one of the most um important and tricky parts of building good environments. s97 00:11:55.279 --> 00:12:00.800 In Sweepbench Pro we have these verifiers that are testing again for specific implementations. s98 00:12:00.800 --> 00:12:14.800 It will fail the model if it uh produces a function that may address the objective but is not named or is not defined within a specific module or if there is the absence of specific helpers or other private functions. s99 00:12:14.800 --> 00:12:20.240 Um because again these are derived from the solutions that were merged in the actual PR. s100 00:12:20.240 --> 00:12:26.240 Um so for us we want to uh emphasize on the observable behavior as much as possible. s101 00:12:26.240 --> 00:12:36.000 We want to ensure that any correct implementation uh anything that correctly solves the problem is rewarded and this will prevent against uh false negatives. s102 00:12:36.000 --> 00:12:43.680 We also make sure that um there's the absence of these PR derived tests that rely on naming relying on specific implementations. s103 00:12:43.680 --> 00:12:49.839 Um and so this will prevent again again uh again uh towards false negatives as well. s104 00:12:49.839 --> 00:13:00.720 And we observed through a combination of these um you know considerations we're able to drastically reduce the false negative as well as the false positive rates when we analyzed s105 00:13:00.720 --> 00:13:13.839 um our rollouts compared to SweetBench pros uh using both human experts as well as LLM as judge and yes the coverage for us spans across these 91 repositories. s106 00:13:13.839 --> 00:13:20.399 Um our criteria for these repositories was uh ones that had more than 500 stars on GitHub. s107 00:13:20.399 --> 00:13:35.279 Uh they are actively being contributed towards and for for our pool of uh uh subject matter experts to validate that these are you know in fact repositories that are actively used in in in the in the real world and are representative s108 00:13:35.279 --> 00:13:40.720 and can field uh real world uh and realistic software engineering tasks. s109 00:13:40.959 --> 00:13:47.200 Um but with all that said um there's still a lot of work to be done for deep suite and for benchmarks in general. s110 00:13:47.200 --> 00:13:55.680 One of the things that we outlined in our blog is our uh choice to use mini suite agent which is an agent agnostic harness. s111 00:13:55.680 --> 00:14:05.680 The reason why here is we really want to be focusing on the model's base performance and so we use mini suite agent also uh ran rollouts to test that the s112 00:14:05.680 --> 00:14:11.600 uh performance is comparable both using mini suite and against each model's native harness. s113 00:14:11.600 --> 00:14:23.839 But I think there's a lot of work to be done in the future for benchmarks that focus solely or more so on harnesses and comparing the effects that these harnesses whether it's native or third party ones like mini suite s114 00:14:23.839 --> 00:14:28.240 um towards the efficiency and the output of these models. s115 00:14:28.240 --> 00:14:31.199 Another thing we want to improve on is task mix. s116 00:14:31.199 --> 00:14:49.440 So given that we are targeting long horizon tasks naturally this meant that there's less um less uh emphasis on bug localization and refactoring these are obviously very representative of real work that software engineers are doing uh under reppresented in our current taxonomy s117 00:14:49.440 --> 00:14:51.040 uh for deep suite. s118 00:14:51.040 --> 00:14:53.199 Um and finally, repository pool. s119 00:14:53.199 --> 00:15:01.360 Um we put an emphasis on trying to field as many diverse repositories as possible, keeping the median tasks per repository to a very low count. s120 00:15:01.360 --> 00:15:11.440 But uh further work here just to pull in more or more repos, more tasks that software engineers find interesting and find them to be good and maybe also um s121 00:15:11.440 --> 00:15:18.160 more niche t tests of models performance would also be a great addition here. s122 00:15:18.240 --> 00:15:32.880 So we've already uh released deep suite v 1.1. So in here we've taken some additional measures to guard against cheating uh reward hacking uh by ensuring you know the verifier runtime is fully separate now from the agent runtime. s123 00:15:32.880 --> 00:15:43.600 Um also making sure the test reports are in a more standardized format and also making sure that we've trimmed all of the git refs and the commits besides the s124 00:15:43.600 --> 00:15:46.240 uh base commit that our agents are working on. s125 00:15:46.240 --> 00:15:51.600 So all of this in service of just making the environments more robust and more cheating proof. s126 00:15:51.600 --> 00:15:58.880 But uh as I mentioned looking ahead we want to support an even greater diversity of like t uh tasks corpus. s127 00:15:58.880 --> 00:16:11.199 We also want to look into hybrid verification because if we're able to use LLM as judge or other um methodologies, it's possible for us to make our prompts even more tur and even more s128 00:16:11.199 --> 00:16:18.639 um even more uh high level and focus on the objective rather than prescribing anything onto the agent. s129 00:16:18.639 --> 00:16:32.720 Um there is of course like a certain degree that we have to in our current prompts like um hint the agents steering them towards a current methodology just because otherwise they they may not be well positioned at all to make meaningful progress towards the task. s130 00:16:32.720 --> 00:16:43.600 Um but something like LLM as a judges and hybrid verifiers would potentially help us towards that and beyond deep suite we're also working on new benchmarks that are in the works. s131 00:16:43.600 --> 00:16:54.720 These are again focused on the highv value domains that uh data curve we prioritize uh as being the domains where we want to be uh most meaningfully advancing model capabilities. s132 00:16:54.959 --> 00:17:05.520 Um but with that said we're actively hiring uh both researchers, engineers helping us with these new benchmarks, new trading data pipelines in service of advancing these capabilities. s133 00:17:05.520 --> 00:17:09.439 So um definitely reach out at datacurve.ai/careers. s134 00:17:09.439 --> 00:17:15.039 Um, and yeah, if you're interested about any of this research benchmark or any of our works, uh, come find me after. s135 00:17:15.039 --> 00:17:16.876 Thank you very much. s136 00:17:16.876 --> 00:17:18.876 [applause] s137 00:17:31.942 --> 00:17:33.942 [music]