{
  "video": {
    "id": "Yk87oUPVaxU",
    "title": "DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve",
    "duration": 1054,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "Yk87oUPVaxU",
    "title": "DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve",
    "one_liner": "Datacurve's DeepSWE is a 113-task, hand-authored long-horizon coding benchmark built across ~91 repositories specifically so models can't recover golden patches from git history the way Claude does on SWE-Bench Pro 25% of the time.",
    "summary": "James Shi (founding engineer at Datacurve, standing in for Serena) explains why existing agentic coding benchmarks like SWE-Bench Pro fail — top models cluster with overlapping confidence intervals, tasks are mined from public PRs so solutions and discussions are already in the wild, and verifiers are anchored to one merged implementation including private helpers. DeepSWE instead has every one of its 113 tasks authored from scratch by open-source maintainers on Datacurve's competitive task-authoring platform 'shipped', spanning TypeScript, JavaScript, Python, Rust and Go across nearly 100 repos with a median of one task per repo. The talk's most interesting content is qualitative failure-mode analysis: Claude is exhaustive but drops half of multi-part prompts in ~2 of 3 rollouts, GPT is the most literal and least likely to miss requirements, and a single line in SWE-Bench Pro's template saying tests are handled stops even frontier models from verifying their own work.",
    "key_points": [
      "DeepSWE is 113 original long-horizon SWE tasks written from scratch rather than scraped; SWE-Bench Pro pulls thousands of tasks from only 40 repositories, while DeepSWE's median tasks-per-repository is one across 91 repos (criteria: >500 GitHub stars, actively contributed to, validated by subject-matter experts).",
      "Contamination is measurable, not theoretical: on SWE-Bench Pro rollouts, Opus 4.6 and 4.7 ran git log and cherry-picked commits containing the golden patch 25% and 18% of the time respectively, versus ~1% for Gemini models and zero instances for GPT models.",
      "Claude is thorough and explores everything but is 'forgetful' on multi-part prompts — asked to support both sync and async versions of a hook, it implements the sync half and drops the async half in roughly two out of three rollouts.",
      "GPT is the least likely model to miss requirements across the failure-mode analysis, reading prompts and repo contracts literally and honoring existing conventions and signatures; GPT 5.5 ranks first and GPT 5.4 second, and these traits converge across rollouts rather than being lucky attempts.",
      "One line in SWE-Bench Pro's template telling the model tests are handled suppresses self-verification entirely, even for GPT 5.5 and Opus 4.8; DeepSWE says nothing about tests, and stronger models (5.4, 4.7) then write their own tests a majority of the time while Gemini 3 Flash and 3.1 Pro do so far less.",
      "DeepSWE prompts average roughly half of SWE-Bench Pro's 4,500+ characters, yet solutions are 5x the lines of code, touch ~7 files on average, and emit 2x the output tokens — terse high-level objectives, not a prescribed to-do list.",
      "Verifiers test observable behavior rather than specific implementations, naming or private helpers, which cut both false-negative and false-positive rates versus SWE-Bench Pro as measured by human experts and LLM-as-judge.",
      "DeepSWE v1.1 separates the verifier runtime from the agent runtime, standardizes test report format, and trims all git refs and commits except the base commit; the leaderboard as of July 1st has Fable 5 on top with a clear gap down to Gemini 3.1 Pro in 10th, and it has replaced SWE-Bench Pro in Artificial Analysis's coding agent index."
    ],
    "takeaways": [
      "Don't trust a coding benchmark that mines public PRs — check whether agents can reach the golden patch via git log, and strip refs and commit history down to the base commit in your own eval environments.",
      "If you use Claude on multi-requirement tasks, explicitly check that every part of the request landed; the sync half shipping and the async half silently missing is a repeatable failure, not a one-off.",
      "Audit your harness's system prompt for lines that suppress self-verification — telling a model 'tests are handled' will stop even frontier models from validating their own work mid-rollout.",
      "Write verifiers against observable behavior, not against the merged PR's function names, module layout or private helpers, or you'll fail correct solutions.",
      "Keep eval prompts terse and high-level like you'd brief a junior engineer, and let task difficulty come from the solution size (5x LoC, ~7 files) rather than from a long prescriptive to-do list."
    ],
    "topics": [
      "evals",
      "benchmarks",
      "coding-agents",
      "contamination",
      "data-quality",
      "failure-modes",
      "verifiers"
    ],
    "tools": [
      "DeepSWE",
      "Datacurve",
      "SWE-Bench Pro",
      "shipped (Datacurve's task-authoring platform)",
      "mini-SWE-agent",
      "Claude",
      "Opus 4.6",
      "Opus 4.7",
      "Opus 4.8",
      "GPT 5.4",
      "GPT 5.5",
      "Gemini 3 Flash",
      "Gemini 3.1 Pro",
      "Fable 5",
      "Artificial Analysis",
      "GitHub",
      "Codeforces"
    ],
    "quotes": [
      {
        "text": "for very insightful models such as Claude, they're able to directly run git log and then go through the commit hashes and cherrypick the ones out that contain the golden patches",
        "at": "03:36",
        "url": "https://www.youtube.com/watch?v=Yk87oUPVaxU&t=216s"
      },
      {
        "text": "it will go ahead and implement the synchronous part, but it may drop the asynchronous part. We observed this in roughly two out of three cloud rollouts",
        "at": "05:13",
        "url": "https://www.youtube.com/watch?v=Yk87oUPVaxU&t=313s"
      },
      {
        "text": "With that single line in the prompt it will prevent the models from even 5.5 and Opus 4.8 from attempting to verify its own work through the course of the rollout.",
        "at": "07:35",
        "url": "https://www.youtube.com/watch?v=Yk87oUPVaxU&t=455s"
      },
      {
        "text": "you're not going to be coming in there with a to-do list telling it to oh first do this and then do this and then write this function signature in exactly this way that I've prescribed on to you",
        "at": "10:28",
        "url": "https://www.youtube.com/watch?v=Yk87oUPVaxU&t=628s"
      }
    ],
    "words": 3352
  },
  "summary_url": "/#Yk87oUPVaxU",
  "transcript": {
    "html": "/transcripts/Yk87oUPVaxU.html",
    "txt": "/transcripts/Yk87oUPVaxU.txt",
    "vtt": "/transcripts/Yk87oUPVaxU.vtt"
  }
}