{
  "video": {
    "id": "c_SPW__jF9w",
    "title": "How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada",
    "duration": 1190,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "c_SPW__jF9w",
    "title": "How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada",
    "one_liner": "When no benchmark exists, flip the problem around: start from a known answer (a source page, a rule you deliberately break, a clause you remove, a filled-in template), generate the input from it, and grade whether the agent finds it again with code rather than an LLM judge.",
    "summary": "Nathanael Weill says you cannot improve an agent you cannot grade. That matters most at Air Canada, where tasks are too specific for public benchmarks and domain experts are busy and hard to reach. He first learned this in 2024 on a RAG Q&A assistant for telecom engineers. There, having an LLM write questions from known pages took his team from one expert-graded trial a week to dozens of automated evaluations a day, and their solution worked where competing teams' did not. He then applies the same answer-first idea to a 2026 Air Canada agent that reviews supplier contracts. He walks through five ways to build test sets for it, and argues for exact-match and code-based grading over LLM-as-a-judge, which he found too noisy.",
    "key_points": [
      "Telecom RAG assistant (2024): about 10,000 documents plus vendor docs. Weekly meetings put five chatbots on a giant screen for experts to compare, and the experts often disagreed. That capped the team at one trial a week.",
      "The reverse-RAG trick: an LLM writes a question from a page, the agent searches for it, and code checks whether the agent retrieved the source page. This took them from one evaluation a week to dozens a day. It also showed that removing complexity helped, and that the real gains came from the embedding model, the chunking and the BM25 alpha, not from techniques in papers.",
      "Adding complexity kept breaking things that used to work. He thinks the other teams had no test set and relied only on the slow loop with experts.",
      "Air Canada contract agent, four steps: spot clauses (classify sections into domains and find defects), retrieve supporting documents, write the amendment, then classify whether to keep it.",
      "Five test-generation techniques: (1) diff the original and signed contracts against past decisions. This failed because the team keeps only the final version. (2) Take a known rule, such as a cap above $1 million, write a clause that breaks it, and generate a contract around it; this produced 275 tests with known answers. (3) Remove a clause from a signed contract and mix these with clean contracts that should raise no flags. He calls this the most powerful approach so far; it showed the agent missed most of the removed clauses. (4) Invariance checks: change the supplier name or address, reorder clauses, or rerun the same contract, and expect the same flags. (5) Fill the experts' own contract templates with arrays of values, some of which break rules, to generate many labelled contracts.",
      "Feedback that took about 4 weeks now comes back from a script in 5 minutes.",
      "He avoids LLM-as-a-judge wherever possible. Getting its prompt and harness right was almost as hard as the original problem, and they never found one that worked for this project.",
      "Other answer-first examples: trajectory tests (known endpoints in a known order, an LLM-generated story, then a check of the trace) and text-to-SQL. Sometimes they redesigned the solution so that it could be evaluated."
    ],
    "takeaways": [
      "Generate the answer first, then build the input around it (plant a defect or pick a source page), so the agent's output can be checked by code instead of by expert opinion.",
      "Use invariance tests (irrelevant field changes, reordering, repeated runs) as cheap code-checked consistency checks.",
      "Reuse what experts already have, such as templates, rule documents and signed contracts, to produce many labelled cases without much of their time.",
      "Feed expert feedback into better tests (better prompts, better ways to remove clauses), not into direct fixes to the agent.",
      "Treat generated test sets as a tool for developers, not as the go/no-go decision for shipping. Be willing to redesign an agent you cannot otherwise evaluate."
    ],
    "topics": [
      "evals",
      "agents",
      "rag",
      "synthetic-data",
      "llm-as-a-judge",
      "contract-analysis",
      "enterprise-ai"
    ],
    "tools": [
      "Air Canada",
      "BM25",
      "Rise (as transcribed)",
      "LinkedIn"
    ],
    "quotes": [
      {
        "text": "if you cannot grade the agent, uh you can't make it better.",
        "at": "07:33",
        "url": "https://www.youtube.com/watch?v=c_SPW__jF9w&t=453s"
      },
      {
        "text": "So, complexity doesn't always help.",
        "at": "04:44",
        "url": "https://www.youtube.com/watch?v=c_SPW__jF9w&t=284s"
      },
      {
        "text": "we got so much noise from the LLM as a judge compared to the noise with the with the agent that we building.",
        "at": "16:57",
        "url": "https://www.youtube.com/watch?v=c_SPW__jF9w&t=1017s"
      },
      {
        "text": "we don't use this feedback to fix the agent. We use this feedback to generate better test.",
        "at": "18:40",
        "url": "https://www.youtube.com/watch?v=c_SPW__jF9w&t=1120s"
      }
    ],
    "words": 3801
  },
  "summary_url": "/#c_SPW__jF9w",
  "transcript": {
    "html": "/transcripts/c_SPW__jF9w.html",
    "txt": "/transcripts/c_SPW__jF9w.txt",
    "vtt": "/transcripts/c_SPW__jF9w.vtt"
  }
}