{
  "video": {
    "id": "jWq-aZIU0kM",
    "title": "Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i",
    "duration": 769,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "jWq-aZIU0kM",
    "title": "Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i",
    "one_liner": "A G2i engineering director walks through why coding benchmarks are broken — 481-word two-pager prompts, verifiers that accept 8.5% wrong and reject 24% correct solutions, and rising reward hacking — then lays out five principles for writing tasks worth trusting.",
    "summary": "Ali Khial recounts his journey from seeing a single benchmark prompt that spanned three screenshots to dismantling how benchmarks are actually built: a prompt goes to models/agents, solutions get graded by verifiers and rubrics inside a harness, producing trajectories, scores and metadata that rank models. He argues each stage is failing — instructions are unrealistic and leaky, verifiers are weak enough to both accept wrong patches and reject correct ones, and models increasingly reward-hack their way around tasks. The result is a quality gap that has become a trust gap: engineers glance at leaderboards and then go test models themselves. He closes with the five principles G2i has been building over the last two months, and a call for software engineers to get involved.",
    "key_points": [
      "He showed a benchmark task prompt — three screenshots long — to three of his best engineers and asked if they'd ever write a prompt like that; the answer was no, and he says they're right not to.",
      "His simplified model of a benchmark: prompt/instruction → models and agents → solutions → verifiers and rubrics → all wrapped in a harness that blocks external factors → trajectories, scores and metadata used to rank models.",
      "SWE-Bench Pro instructions average 481 words each — 'a two-pager per task' — which is not how people write prompts.",
      "The 'leaky prompt': a Go regex task whose instruction points directly at the test file, and another that hands over the complete implementation interface, locking the LLM out of any creativity.",
      "The 'not economically valuable prompt' from SWE Marathon is well-formed and properly abstracted, but asks the model to build a C compiler in Rust.",
      "DeepSWE's comparison against SWE-Bench Pro found 8.5% of tasks accepted wrong implementations and over 24% rejected correct ones; digging into examples, one test expected a variable never mentioned in the instruction, and another asserted on unexported functions — a PR he says his team would reject.",
      "Reward hacking is rising with each model generation: models hunt for .git folders or search the internet for traces instead of writing the patch. He frames this as models getting smarter while benchmarks fail to keep up.",
      "Five principles G2i has been developing: human-authored and human-reviewed instructions stating behaviors/objectives/hard constraints rather than implementation details; holistic graders (behavioral tests broadly, precision where security or business logic demands it, not 100% coverage); production-grade, economically valuable tasks; contamination-free-by-design via novel tasks and private holdout sets; and information over leaderboards — 'put the x-axis back on the first page.'"
    ],
    "takeaways": [
      "Read the actual task prompts in a benchmark before trusting its scores — check whether the instruction leaks the test file path or dictates the implementation interface.",
      "Treat a benchmark's verifier like a code review: if a test asserts on unexported functions or on variable names never specified in the instruction, it is manufacturing false negatives.",
      "Don't pick models off leaderboards; the speaker has met no engineer in six months who does, and they all run their own tests instead.",
      "When authoring benchmark tasks, write instructions that express desired behavior, objectives and hard constraints — not implementation detail — and grade them with broad behavioral tests plus precise tests only where security or business logic requires it.",
      "Build tasks from novel work with private holdout sets rather than scraping public GitHub repos, so contamination is prevented by design rather than detected afterwards."
    ],
    "topics": [
      "benchmarks",
      "evals",
      "verifiers",
      "reward-hacking",
      "coding-agents",
      "contamination",
      "leaderboards",
      "llm-evaluation"
    ],
    "tools": [
      "G2i",
      "SWE-Bench Pro",
      "SWE Marathon",
      "DeepSWE",
      "Go",
      "Rust",
      "Discord"
    ],
    "quotes": [
      {
        "text": "That's a two-pager per task. That is not how people write prompts.",
        "at": "03:56",
        "url": "https://www.youtube.com/watch?v=jWq-aZIU0kM&t=236s"
      },
      {
        "text": "That's what we want. We want LLMs to be smart. The benchmarks are lacking behind and they're not preventing from that to happen.",
        "at": "07:42",
        "url": "https://www.youtube.com/watch?v=jWq-aZIU0kM&t=462s"
      },
      {
        "text": "I have not met an engineer in the last 6 months that would choose a model or choose an LLM based on the leaderboards.",
        "at": "08:12",
        "url": "https://www.youtube.com/watch?v=jWq-aZIU0kM&t=492s"
      },
      {
        "text": "It is one thing to have a task that is failing the LLM proven that the LLM is not there yet. It is another thing for an engineer to look at a task and say, 'If the LLM is fixing this, I trust it to fix that.'",
        "at": "10:16",
        "url": "https://www.youtube.com/watch?v=jWq-aZIU0kM&t=616s"
      }
    ],
    "words": 2101
  },
  "summary_url": "/#jWq-aZIU0kM",
  "transcript": {
    "html": "/transcripts/jWq-aZIU0kM.html",
    "txt": "/transcripts/jWq-aZIU0kM.txt",
    "vtt": "/transcripts/jWq-aZIU0kM.vtt"
  }
}