{
  "video": {
    "id": "YIxzNZDDy1A",
    "title": "From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases",
    "duration": 3150,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "YIxzNZDDy1A",
    "title": "From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases",
    "one_liner": "When you benchmark, you are testing a whole configuration, not just a model: the dataset version, harness, sandbox, agent, inference endpoint and every setting all count. Run each configuration several times (Ravenwolf uses 5), report the solid base, best, worst and ceiling as well as the average, and publish the traces so others can check why a model scored as it did.",
    "summary": "Wolfram Ravenwolf argues that one leaderboard score is not enough, because performance is a distribution, not a point. Using WolfBench, his open-source (Apache 2) toolkit built on Terminal Bench 2.0 and Harbor, he walks through six swappable layers: benchmark, evaluation harness, sandbox, agent harness, model/service and settings. He shows that changing any one of them makes it a different experiment. He runs a live Harbor eval of GLM 5.2 with the Terminus 2 agent in CoreWeave serverless sandboxes. He then uses the WolfBench dashboard and leaderboard to show what the average hides: token burn, cost, variance, refusals and overthinking. Every bar links to Weave traces so it can be checked.",
    "key_points": [
      "WolfBench charts four numbers per bar. The 'solid base' is the share of tasks solved in all 5 runs, which he calls consistency. Then come the average (the usual single score), best and worst runs as up/down arrows, and a 'ceiling' star for tasks solved at least once. One example ranged from 79% to 60%, a sign of high randomness. Two configs averaging 73% and 74% had solid bases 6 points apart.",
      "Gemini 3.5 Flash came second on his board, close to GPT 5.5. But it used 1.65 billion tokens against 136 million for GPT 5.5, so it cost more: $620 vs $497. A 'cheap, fast' model can end up slower and more expensive.",
      "His top configuration is GPT 5.5 with Codex, at almost $1,000. Cursor with the same model cost $282, which may be the better price-performance choice. Testing Fable cost $11,000, and it scored poorly because it refused 13 tasks.",
      "Raising thinking to max on Opus 4.6 lowered the score, both with Terminus and with Claude Code. The traces showed overthinking: with a 1-hour limit, spending minutes thinking about each step leaves fewer turns than fast trial and error. GPT 5.5 improved as thinking went up, yet Opus with thinking off already beat GPT 5.5 on medium, solid base included. Fable sometimes ignored the task spec ('I know better') and failed the verifier.",
      "He stays on Terminal Bench 2.0 even though 2.1 fixes 28 of the 89 tasks: switching would make it a new benchmark and force him to retest everything. He pins the Harbor version for the same reason ('at least testing all the models with the same bug'). Serious evaluators should fork the harness and backport only what they need.",
      "Settings that quietly change results include: context window (CoreWeave serves GLM 5.2 at 256K, not 1M), thinking effort, system prompt (models lost points asking a user who wasn't there), tool config (a model with web access looked up the public Terminal Bench solution), sandbox CPU/memory, task timeout, concurrency against rate limits, retry policy, temperature, top-p, max new tokens and seed. He has seen a driver update change results even at temperature 0.",
      "He cut his per-task timeout from 2 hours to 1. He had chosen 2 hours so open-weights models on slower endpoints weren't handicapped, but models that can't solve a task just loop, burning time and money.",
      "After each run he checks the errors for infrastructure problems. Repeated characters, or prompt-template tokens leaking into responses, point to tokenizer or inference-server issues. These are common with newly released models; the same eval run a week later scored better once the inference engine had improved."
    ],
    "takeaways": [
      "Run every configuration several times (he uses 5) and report the solid base, best/worst range and ceiling, not just the average. When two configurations are close, prefer the one with the higher solid base, and do 10+ runs if you need certainty.",
      "Pin every layer: benchmark version, harness version (fork it if you're serious), agent version and provider. On OpenRouter, set the provider explicitly, or one 89-task run may be split across different backends. Publish the exact configuration with your results.",
      "Track tokens and dollar cost alongside the score, because a cheaper model can cost more overall.",
      "Run each task in its own isolated sandbox, and run the harness on a VM rather than your laptop. Retry infrastructure failures (sandbox crashes, rate limits), but count crashes the model caused as real fails.",
      "When a result surprises you, read the traces, or have an agent read them and link to the evidence. Publish the traces so others can check your leaderboard."
    ],
    "topics": [
      "evals",
      "agents",
      "benchmarking",
      "leaderboards",
      "reproducibility",
      "observability",
      "cost"
    ],
    "tools": [
      "WolfBench (wolfbench.ai)",
      "Weights & Biases",
      "W&B Weave",
      "CoreWeave serverless sandboxes",
      "CoreWeave serverless inference",
      "Terminal Bench 2.0 / 2.1",
      "Harbor",
      "Terminus 2",
      "Inspect (UK AI Security Institute)",
      "LM Evaluation Harness",
      "OpenAI Evals",
      "Daytona",
      "Docker",
      "OpenRouter",
      "uv",
      "Claude Code",
      "Codex",
      "Cursor",
      "Cline",
      "OpenClaw",
      "Hermes Agent",
      "GPT 5.5",
      "Gemini 3.5 Flash",
      "Claude Opus 4.6",
      "Claude Fable",
      "GLM 5.2",
      "DeepSeek",
      "MMLU",
      "MMMU Pro",
      "Hetzner",
      "1Password",
      "ThursdAI podcast"
    ],
    "quotes": [
      {
        "text": "And always do multiple runs. A single run is benchmark lottery.",
        "at": "06:49",
        "url": "https://www.youtube.com/watch?v=YIxzNZDDy1A&t=409s"
      },
      {
        "text": "Even if it has a bug, you are at least testing all the models with the same bug.",
        "at": "13:52",
        "url": "https://www.youtube.com/watch?v=YIxzNZDDy1A&t=832s"
      },
      {
        "text": "The bar is a claim, the trace is evidence.",
        "at": "47:38",
        "url": "https://www.youtube.com/watch?v=YIxzNZDDy1A&t=2858s"
      },
      {
        "text": "So the model was too smart for its own good in some cases.",
        "at": "47:32",
        "url": "https://www.youtube.com/watch?v=YIxzNZDDy1A&t=2852s"
      }
    ],
    "words": 11115
  },
  "summary_url": "/#YIxzNZDDy1A",
  "transcript": {
    "html": "/transcripts/YIxzNZDDy1A.html",
    "txt": "/transcripts/YIxzNZDDy1A.txt",
    "vtt": "/transcripts/YIxzNZDDy1A.vtt"
  }
}