{
  "video": {
    "id": "gv4w23xYj-0",
    "title": "How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI",
    "duration": 2120,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "gv4w23xYj-0",
    "title": "How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI",
    "one_liner": "Treat traces as the logs and evals as the tests: start with one specific, concrete criterion per LLM-as-a-judge eval on about 20 traces, read the explanations, iterate, then add more evals as production data shows what else matters.",
    "summary": "SallyAnn DeLucia, head of product at Arize, argues that vibe checks don't scale. Without evals you can't catch regressions when a prompt changes, objectively compare prompts or models, or put quality gates into CI. She explains why an AI eval is not a unit test: there are many valid outputs, and the agent's logic only shows up afterwards in traces. The workshop then walks through a travel-planning agent demo in Arize AX: reading traces and spans, writing concrete pass/fail criteria, saving filtered spans to a dataset, and building and iterating on an LLM-as-a-judge template in the prompt playground.",
    "key_points": [
      "Without evals you can't detect regressions when you change a prompt, objectively compare prompts or model versions, or put official quality gates into CI. The 'vibes problem' is that a fix shipped Monday is broken a week later and nobody can prove either way.",
      "Evals differ from unit tests: a test has one input, one correct output and logic you can read, while an agent has many valid outputs and its logic only appears afterwards in traces. 'Traces are the logs for AI, evals are actual tests', so you need observability to do evals.",
      "Eval types covered: LLM-as-a-judge (a label plus an explanation), decision models (cheap and fast, returning a probability and a label but no explanation; OpenAI had just released a decision model API), code evals (deterministic checks such as JSON validity or whether a word appears, plus ROUGE/BLEU), annotations/golden datasets, agent-as-a-judge (newer, for complex success criteria), and user feedback and business metrics for production. The first four can be used across development and production.",
      "Use concrete criteria rather than vague ones. Instead of 'helpful, accurate, good response', write 'answers the user's follow-ups, matches the requested number of days, keeps totals within the stated budget'.",
      "Judge one quality per eval: research shows LLM-as-a-judge only does well with one criterion at a time. Decision models and agent-as-a-judge can handle several.",
      "Arize's own agent Alex started with a few evals (QA correctness, user frustration, jailbreak/toxicity safety evals) and now runs 32.",
      "Criteria template: task, success, failure, what's being judged, and three bullets each for pass and fail. These become the judge prompt, written as if you were describing success to a coworker.",
      "Workflow in Arize: filter spans (e.g. parent ID = null for root spans, or by tool name), save them to a dataset, open the prompt playground, write the template with variable mapping so evals stay portable across datasets and traces, set labels (correct/incorrect, optionally partially correct or not applicable), choose the optimization direction and explanations, run it (she used GPT-4.1 mini), iterate, then save the evaluator."
    ],
    "takeaways": [
      "Start small: about 20 traces, one dimension, one judge. Read the explanations, iterate on the template, then expand.",
      "Write success and failure as specific, checkable bullets tied to your product's requirements, and give each LLM judge only one criterion.",
      "Generic eval-hub templates (QA correctness, hallucination, session coherence) are fine to start with, since any eval beats none, but make them specific to your use case over time.",
      "Next steps: align judges against human annotations on your dataset, run them as online evals on every incoming trace for monitoring, and try converting an LLM judge into a decision judge to compare performance.",
      "Get every stakeholder (product, engineering, support, domain experts, security) to define success. Use metrics to see that something is broken and evals to see what the failure mode is."
    ],
    "topics": [
      "evals",
      "llm-as-a-judge",
      "agents",
      "observability",
      "tracing",
      "decision models",
      "eval criteria"
    ],
    "tools": [
      "Arize AX",
      "Arize Alex",
      "Arize Signal",
      "OpenInference",
      "OpenAI",
      "OpenAI decision model API",
      "Anthropic",
      "Gemini",
      "GPT-4.1 mini",
      "LangGraph",
      "Notion",
      "ROUGE",
      "BLEU"
    ],
    "quotes": [
      {
        "text": "traces are the logs for AI Um, evals are actual tests.",
        "at": "03:58",
        "url": "https://www.youtube.com/watch?v=gv4w23xYj-0&t=238s"
      },
      {
        "text": "an eval is better than no eval at all.",
        "at": "21:26",
        "url": "https://www.youtube.com/watch?v=gv4w23xYj-0&t=1286s"
      },
      {
        "text": "a metric tells you that something might be broken, but evals tell you what the actual failure mode is.",
        "at": "34:03",
        "url": "https://www.youtube.com/watch?v=gv4w23xYj-0&t=2043s"
      },
      {
        "text": "start small, 20 traces maybe, one dimension, one judge",
        "at": "34:30",
        "url": "https://www.youtube.com/watch?v=gv4w23xYj-0&t=2070s"
      }
    ],
    "words": 8047
  },
  "summary_url": "/#gv4w23xYj-0",
  "transcript": {
    "html": "/transcripts/gv4w23xYj-0.html",
    "txt": "/transcripts/gv4w23xYj-0.txt",
    "vtt": "/transcripts/gv4w23xYj-0.vtt"
  }
}