{
  "video": {
    "id": "7WUzHIF32vA",
    "title": "If Your AI Scores 100%, Your Eval Is Wrong - Ramakrishnan Lokanathan, eslams.com",
    "duration": 1125,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "7WUzHIF32vA",
    "title": "If Your AI Scores 100%, Your Eval Is Wrong - Ramakrishnan Lokanathan, eslams.com",
    "one_liner": "If your eval scores 100/100 (or even 85–90), it isn't ambitious enough: test every step of the job rather than only the final answer, keep regression cases for what used to work, and turn real user failures into new tests.",
    "summary": "Lokanathan describes evals as exams for AI systems. Each exam answers a bounded question and gives one signal of readiness, never a measure of complete ability. Its job is to check that production performance matches the promise made to the user. He sets out a framework for high-stakes applications: define the most ambitious success first, test the whole workflow step by step, build a holistic system (CI checks, human review, adversarial tests and other paradigms), and evolve the tests using real-world failures. He also promotes his open-source E-Slams package, which has models and agents play games such as chess, Go and poker as one more evaluation paradigm.",
    "key_points": [
      "No single eval is a full measure of ability. Like the SAT in US college admissions, each benchmark (SWE-bench, Humanity's Last Exam) is one signal, so don't over-index on any of them.",
      "He predicts evals won't be relevant in a decade because model capability keeps improving, but production can't wait. He cites the MIT GenAI Divide report: 95% of organisations get no returns because they don't deliver in production what they promised with the LLM.",
      "Define the most ambitious success before designing any test (\"assume AGI\"). Ask what must happen, what must never happen and when the system should stop. A 100/100 score means the eval is wrong; frontier model evals top out around 85–86.",
      "Test the whole workflow, not just the output. In his example, an image edit asked only to change the background produced a cream background but gave the object two handles. With 100 steps, each step's input, output and expected outcome needs evaluating, and each test should fit what that step does (e.g. tool-call accuracy for a tool-calling step).",
      "A holistic eval system has three parts: automated eval checks in the CI pipeline on every production deployment, continuous human review (don't rely on agents to review your eval), and adversarial tests that try to break the workflow.",
      "E-Slams (\"electronic slams\") is an open-source Python package on PyPI that makes AI play games: about 50 games and about 90 providers. Chess and Go test planning; poker tests decisions on hidden information; there are cooperation games and arcade trials. In a demo on eslams.com, Fable 5.1 vs GPT-6 Astra at tic-tac-toe ended in a tie. A 'Grand Slams of AI' Olympics-style competition is planned for February.",
      "When Google Assistant moved to Gemini, it stopped setting reminders and said it was a language model that couldn't. He uses this as an example of missing regression evals: a much better model can still be unable to do something the old one did well.",
      "Real-world failures should feed back into tests: observability tools and thumbs up/down feedback, then classic QA (reproduce the issue, find where it failed, such as a tool call, and which part of the eval suite missed it), review, and add it to the eval pipeline. This catches the last 5% of edge cases."
    ],
    "takeaways": [
      "Write the most ambitious definition of success before writing tests, and make the eval harder (fresh challenge cases) once models reach about 85–90 on it.",
      "Evaluate each step of an agent workflow with a test designed for what that step does, not only the start and end points.",
      "Run evals in CI on every deployment, keep humans reviewing results, and add adversarial and non-traditional paradigms (e.g. game play, search-index evals) for a holistic picture.",
      "Keep a regression suite of known tasks and past failures, especially when swapping in a new model.",
      "Build a pipeline that turns user-reported production failures into new eval cases."
    ],
    "topics": [
      "evals",
      "agents",
      "benchmarks",
      "regression testing",
      "adversarial testing",
      "human review",
      "game-based evaluation",
      "ci"
    ],
    "tools": [
      "SWE-bench",
      "Humanity's Last Exam",
      "SAT",
      "GMAT",
      "GPT-3",
      "GPT-3.5",
      "ChatGPT",
      "OpenAI",
      "Nemotron 3",
      "Lawrence Berkeley Lab",
      "London Business School",
      "MIT GenAI Divide report",
      "E-Slams (eslams.com)",
      "PyPI",
      "AlphaGo",
      "Fable 5.1",
      "GPT-6 Astra",
      "Google Assistant",
      "Gemini",
      "Parallel search index"
    ],
    "quotes": [
      {
        "text": "If you have an eval that is scoring a 100 of a 100, your eval is wrong, right? And you're not ambitious enough and you're not testing enough",
        "at": "06:18",
        "url": "https://www.youtube.com/watch?v=7WUzHIF32vA&t=378s"
      },
      {
        "text": "a model that is extremely better than the old one could very well be incapable of something the old one did very well, for example, setting a reminder",
        "at": "12:55",
        "url": "https://www.youtube.com/watch?v=7WUzHIF32vA&t=775s"
      },
      {
        "text": "test the job and not just the answer",
        "at": "15:10",
        "url": "https://www.youtube.com/watch?v=7WUzHIF32vA&t=910s"
      },
      {
        "text": "Reality is going to be your best teacher",
        "at": "17:54",
        "url": "https://www.youtube.com/watch?v=7WUzHIF32vA&t=1074s"
      }
    ],
    "words": 4148
  },
  "summary_url": "/#7WUzHIF32vA",
  "transcript": {
    "html": "/transcripts/7WUzHIF32vA.html",
    "txt": "/transcripts/7WUzHIF32vA.txt",
    "vtt": "/transcripts/7WUzHIF32vA.vtt"
  }
}