{
  "video": {
    "id": "SHMCGrNJJGc",
    "title": "How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI",
    "duration": 6973,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "SHMCGrNJJGc",
    "title": "How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI",
    "one_liner": "Read your traces before writing any evals, then build several small evals (code checks, then built-in LLM judges, then your own rubric). Check the judge against human labels, prove each prompt or model change with a controlled experiment, and gate CI on the evals that pass reliably.",
    "summary": "Laurie Voss (Arize AI) runs a hands-on TypeScript workshop on a deliberately weak two-turn financial-analyst agent built with the Claude Agent SDK and traced into Phoenix. It covers the whole loop: trace the agent, do error analysis (open coding, then axial coding), write code evals, use built-in LLM-judge evals, write a custom rubric, check the judge against human labels, run experiments, and gate CI with Vitest. The main argument is that 'looks good to me' is not a test suite. Evals turn vague disappointment into numbers you can track. Picking the right eval matters more than tuning it, and the explanation an LLM judge gives is where most of the value lies.",
    "key_points": [
      "Mental model: traces are basically logs, evals are basically tests, and evals run tests on traces. A trace is made of nested spans, each recording input, output, latency, tokens and sometimes cost. In TypeScript, tracing takes about five lines with Phoenix's register on top of OpenTelemetry/OpenInference. One gotcha: the Claude Agent SDK is an ES module with read-only exports, so you must import the copy that manuallyInstrument returns. Otherwise nothing gets logged.",
      "Three kinds of eval stack up like the Swiss cheese model. Code evals are fast, free and brittle. LLM judges understand meaning but cost money and can be confidently wrong. Humans are the gold standard but slow and prone to boredom. A simple code check (is the requested ticker in the report?) failed 2 of 17 runs, including an Amazon report that covered only AWS.",
      "Built-in correctness eval scored 0/16, because the judge measures correctness against its own knowledge cutoff and treated current stock data as hallucination. Switching to faithfulness (is the report grounded in the search results and research notes it was given?) flagged 14/16 as unfaithful, with useful explanations such as fabricated 2027 targets and conflated data points.",
      "Traces exposed failures the output alone hid. With the 'dontAsk' permission mode and an ambiguous 'write a report' prompt, 13 of 16 writing turns tried to save the report to a file (which the agent named 'harness leakage'). A Tesla report cited analyst ratings adding up to 116% after merging two unrelated searches, a reasoning error rather than a retrieval failure.",
      "A custom actionability judge (must say buy/sell/hold) has five parts: role and context, explicit criteria taken from error analysis, data wrapped in XML tags, labelled examples, and a constrained output via tool-call choices. Leaving the XML tags off the examples flipped results. Removing the examples slightly improved agreement with human labels (16/16 twice vs 15/16).",
      "Checking the judge: human labels in Phoenix become a golden dataset with a dev/test split (every fourth example by span ID goes to test). The judge matched the human on 12/12 dev reports. Adding 'must meet all four criteria' dropped agreement to 11/12 and broke Apple on the test split, showing that a stricter rubric can make the eval worse.",
      "Experiment results: an evidence-driven prompt fix took actionability from 5/16 to 48/48 (3 repetitions) with no regressions, but faithfulness only went from 2/16 to 3/48. Moving from Haiku 4.5 to Haiku 5.5 made faithfulness more than 3x better, but it is still poor and costs about 4x as much. Calls became more consistent mainly because Haiku 5.5 said 'hold' 41 of 45 times.",
      "CI: agent.eval.ts is an ordinary Vitest file over a regression set (Coca-Cola, Apple vs Microsoft, Rivian, ZZZQ). The ticker check is a hard guardrail, and the actionability average must stay at or above 0.8 because a single judgment is noisy. Phoenix runs as a service container in GitHub Actions."
    ],
    "takeaways": [
      "Read a dozen or more traces end to end before automating. Write free-form notes first and group them into categories second (open coding, then axial coding) rather than inventing categories up front. Rank failures by frequency × severity.",
      "Build several narrow evals instead of one 'god evaluator'. Grade the outcome, not the path the agent took. Pick evals whose question the judge can actually answer, such as faithfulness to the retrieved context rather than correctness against the judge's own knowledge.",
      "Treat the judge as a classifier. Label a golden dataset against the same criteria the judge uses, split it into dev and test, don't tune on the test set, and track precision and recall (usually favouring recall). Use a stronger judge than the agent, ideally from a different provider, to avoid self-preference bias.",
      "Treat eval prompts as code: version and test them. Wrap data and examples in XML tags. Test the rubric with and without examples. Make each prompt change answer a specific eval finding.",
      "Run each experiment several times and decide whether you need pass@k or pass^k (a financial analyst needs pass^k). Once a capability eval passes reliably, promote it to a regression eval in CI. In production, evaluate a sample of real traffic and alert on trends, not single failures."
    ],
    "topics": [
      "evals",
      "agents",
      "llm-as-judge",
      "observability",
      "tracing",
      "error-analysis",
      "ci",
      "experiments"
    ],
    "tools": [
      "Arize AI",
      "Phoenix",
      "Claude Agent SDK",
      "Claude Code",
      "Claude Haiku 4.5",
      "Claude Haiku 5.5",
      "Claude Sonnet 5.5",
      "Claude Opus 4.5",
      "Anthropic",
      "OpenAI",
      "OpenTelemetry",
      "OpenInference",
      "Vercel AI SDK",
      "Mastra",
      "LangChain",
      "OpenAI Agents SDK",
      "Vitest",
      "GitHub Actions",
      "uvx",
      "Node.js",
      "TypeScript",
      "Cursor",
      "Codex",
      "τ²-bench",
      "CORE-Bench",
      "SWE-bench",
      "npm"
    ],
    "quotes": [
      {
        "text": "Looks good to me is not a test suite.",
        "at": "06:15",
        "url": "https://www.youtube.com/watch?v=SHMCGrNJJGc&t=375s"
      },
      {
        "text": "Confidently wrong is worse than obviously wrong.",
        "at": "11:58",
        "url": "https://www.youtube.com/watch?v=SHMCGrNJJGc&t=718s"
      },
      {
        "text": "the lesson here is that choosing the right eval beats tuning it.",
        "at": "70:17",
        "url": "https://www.youtube.com/watch?v=SHMCGrNJJGc&t=4217s"
      },
      {
        "text": "A prompt change that you haven't measured is just a vibe.",
        "at": "99:32",
        "url": "https://www.youtube.com/watch?v=SHMCGrNJJGc&t=5972s"
      }
    ],
    "words": 23090
  },
  "summary_url": "/#SHMCGrNJJGc",
  "transcript": {
    "html": "/transcripts/SHMCGrNJJGc.html",
    "txt": "/transcripts/SHMCGrNJJGc.txt",
    "vtt": "/transcripts/SHMCGrNJJGc.vtt"
  }
}