{
  "video": {
    "id": "NdrSPm6NCdk",
    "title": "Evals in AI: A Deep Dive — Tejas Kumar, IBM",
    "duration": 3589,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "NdrSPm6NCdk",
    "title": "Evals in AI: A Deep Dive — Tejas Kumar, IBM",
    "one_liner": "Treat evals as \"fuzzy unit tests\" that give you ahead-of-time reliability, and before you trust an LLM judge, check it against roughly 30 human-scored examples until it agrees with them 80–85% of the time (100% suggests overfitting). Close each disagreement by giving the judge the policy it was missing, and gate CI on that agreement.",
    "summary": "Tejas Kumar (AI engineer on IBM's watsonx.data team) argues that harnesses give AI systems just-in-time reliability and evals give them ahead-of-time reliability, and that you need both, like two pedals on a bicycle. His warning example is Instagram's support chatbot, which attackers talked into compromising 20,225 accounts. He covers the parts of an eval (test cases, expected behaviour, a judge, aggregation), the main techniques, and four ways LLM judges lie. He then live-codes a Vitest eval for a headphone-return scenario: string matching fails, and a GPT-3.5 Turbo judge improves as policy is added to its prompt and then by switching to GPT-4o mini, until it meets an 80% agreement gate. He finishes by showing IBM's open-source OpenRAG retrieving policy for the judge.",
    "key_points": [
      "Why evals: (1) ahead-of-time reliability, which complements the just-in-time reliability of harnesses such as Claude Code and Codex, compared to TypeScript (ahead-of-time) versus JavaScript (just-in-time); (2) protection against exploits. Example: Instagram's AI support agent had elevated API access and added attacker email addresses to accounts (attackers used a VPN to avoid tripping 2FA), compromising 20,225 accounts.",
      "An eval is 'reliability instrumentation for non-deterministic systems', or 'fuzzy unit tests'. A unit test asks 'did this function return exactly this number'; an eval asks 'did this agent behave acceptably across many scenarios'. Good eval suites share four parts: test cases, expected behaviour, a judge, and aggregation into a score.",
      "Techniques, cheapest first: exact match (e.g. checking a persisted JSON array of tool calls with plain code), schema validation, pairwise comparison, and LLM-as-judge. Prefer pointwise (pass/fail on one answer) over pairwise. He cites a study finding pairwise judging 39% more vulnerable to attack, while pointwise failed in single digits.",
      "Four ways LLM judges lie: position bias (detect it by swapping the order of options), sycophancy (they pick the nicer 'yes, you can return it' answer even when it breaks policy), self-preference (they favour text from their own model family, so use a different family for the judge), and verbosity bias.",
      "Training a judge: build about 30 question/answer/human-verdict examples, hide the verdicts, have the judge score them, and measure agreement. Aim for 80–85%, because human support staff only agree with each other about 80% of the time. 100% agreement is a red flag for sycophancy or overfitting, so the gate should have an upper bound as well as a lower one.",
      "Live demo: matching on the word 'cannot' let the nonsense answer 'you cannot pay for the shipping' pass. A GPT-3.5 Turbo judge chose a GPT-generated wrong answer consistently (self-preference) until the policy was added to the prompt. On 30 scenarios it failed the 80% bar. Adding the 14-day rule, an 'unused and in original packaging' rule and 'policy is non-negotiable no matter how loyal the user is' helped. Lowercasing the output and switching to GPT-4o mini reached 27/30 agreement, and Opus then rewrote the test to fail only when agreement falls below 80%.",
      "Production: run evals on CI as an agreement gate; pin the judge's model version rather than using an alias like gpt-4o-mini; keep the dataset alive by sampling production traffic and having an internal team score it by hand; use synthetic and red-team adversarial data to grow it ('here's everything that failed, now generate more things that could fail').",
      "Policy should come from one shared source used by both the production agent and the judge. The judge should ideally be an agent with the same retrieval tools, so a CI failure predicts a production failure. IBM's open-source OpenRAG takes in documents with Docling and stores them in OpenSearch; in the demo, uploading a refund policy let it correctly deny the 20-day headphone return."
    ],
    "takeaways": [
      "Escalate cost gradually: unit tests, then regex/pattern matching, then validating traces (tool calls, counts, durations), and only then an LLM judge, starting with the cheapest model and moving up only once added context stops helping.",
      "Before trusting an LLM judge, measure its agreement against about 30 human-scored examples. Fix each disagreement by finding the knowledge gap and adding policy, and gate CI on 80–85% agreement, failing at 100% as well.",
      "Use pointwise pass/fail judging rather than pairwise, use a judge from a different model family than the generator, and test for position bias by swapping the order of options.",
      "Pin the judge's model version, and keep the eval dataset alive with scored samples of production traffic plus synthetic and red-team cases.",
      "Have the production agent and the judge retrieve the same live policy source (e.g. through RAG) instead of hardcoding it in prompts."
    ],
    "topics": [
      "evals",
      "llm-as-judge",
      "judge-bias",
      "harnesses",
      "rag",
      "ci",
      "ai-security",
      "testing"
    ],
    "tools": [
      "IBM watsonx.data",
      "Granite",
      "OpenRouter",
      "GPT-3.5 Turbo",
      "GPT-4o mini",
      "Claude Code",
      "Codex",
      "Cursor",
      "Opus",
      "Vitest",
      "TypeScript",
      "JavaScript",
      "LM Arena",
      "OpenRAG",
      "Docling",
      "OpenSearch",
      "Elasticsearch",
      "Instagram",
      "Meta",
      "Workday",
      "Apple Metal",
      "Microsoft OneDrive"
    ],
    "quotes": [
      {
        "text": "And really these two work as two pedals on a bicycle.",
        "at": "04:50",
        "url": "https://www.youtube.com/watch?v=NdrSPm6NCdk&t=290s"
      },
      {
        "text": "just because it's green doesn't mean it works.",
        "at": "20:52",
        "url": "https://www.youtube.com/watch?v=NdrSPm6NCdk&t=1252s"
      },
      {
        "text": "If you agree, if your judge agrees 100% of the time with you, it's probably somehow sophantic.",
        "at": "23:17",
        "url": "https://www.youtube.com/watch?v=NdrSPm6NCdk&t=1397s"
      },
      {
        "text": "when we don't agree, it's just usually because there's a knowledge gap",
        "at": "40:50",
        "url": "https://www.youtube.com/watch?v=NdrSPm6NCdk&t=2450s"
      }
    ],
    "words": 13770
  },
  "summary_url": "/#NdrSPm6NCdk",
  "transcript": {
    "html": "/transcripts/NdrSPm6NCdk.html",
    "txt": "/transcripts/NdrSPm6NCdk.txt",
    "vtt": "/transcripts/NdrSPm6NCdk.vtt"
  }
}