{
  "video": {
    "id": "q2JrUKBMf0w",
    "title": "The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI",
    "duration": 366,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "q2JrUKBMf0w",
    "title": "The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI",
    "one_liner": "Arize's Aparna Dhinakaran argues that fixed-rubric LLM-as-a-judge evals can't catch the failure modes of modern looping, sub-agent, long-horizon systems, and introduces Signal — a long-running 'agent as a judge' that reads production traces, finds patterns of failure, and opens PRs with fixes.",
    "summary": "Drawing on Arize's view of over 100 million evals run per month across customer teams, Dhinakaran says the thing being evaluated changed underneath first-gen evals: 2023 was answering a prompt, 2024 added tool calls, reasoning and deep research, and now teams run loops on real-world data with sub-agents on long-horizon tasks. Because Arize's own in-product agent Alex started forgetting context, failing to know when it was done, and getting stuck in loops — and because it generates a new dynamic UI on every user interaction, so no two trajectories match — classical LLM-as-a-judge evals with fixed rubrics and fixed scores couldn't catch the failures. Her revelation: the best way to evaluate an agent is with an agent. Arize has released Signal, a long-running agent that reads incoming traces, discovers issue patterns like repeated tool calls and inefficient trajectories, and can open a PR with a fix.",
    "key_points": [
      "Arize runs over 100 million evals a month; the average team runs about 12 different eval jobs, and top teams run over 3,800 different evaluators.",
      "The talk focuses specifically on evals run on live production agent traces — that's what tells teams what's working, catches failures, and fuels continual learning loops.",
      "The evaluation target escalated in complexity year by year: 2023 = answering a prompt; 2024 = frontier models adding tool calls, reasoning and deep research; now = loops on real-world data with sub-agents on long-horizon tasks. Each jump wasn't just harder, it made it a fundamentally different type of problem.",
      "As Arize added each new frontier-lab capability to its own agent Alex (longer memory, dynamic UI generation, search across enormous volumes of traces), new failure modes appeared: forgetting context, not knowing when something was done, getting stuck in loops.",
      "Classical LLM-as-a-judge gives a fixed rubric with fixed scores — fine for a deterministic flow, but Alex creates a new UI on every user interaction, meaning a fundamentally different trajectory each time.",
      "Agent as a judge is framed as adaptive, dynamic analysis, not a replacement: deterministic evals and LLM-as-a-judge still matter — it's a different tool for a different problem.",
      "Her take: most teams today do the first two (deterministic evals, LLM as a judge); the future of evals is having all three.",
      "Arize released Signal, a long-running agent that reads traces sent in, discovers patterns of issues, surfaces subtle failures (the same tool called repeatedly in a loop, an inefficient trajectory), and — because it already holds the analysis — can put up a PR with a fix."
    ],
    "takeaways": [
      "Run evals against live production traces, not just offline datasets, if you want the data that feeds continual learning loops.",
      "Stop expecting a fixed-rubric LLM judge to catch agent failures; when every run produces a different trajectory, a fixed score can't express what went wrong.",
      "Add agent-as-a-judge as a third layer alongside deterministic evals and LLM-as-a-judge rather than replacing them — match the eval tool to the type of problem.",
      "Look explicitly for trajectory-level failures your current evals are blind to: repeated identical tool calls, loops that never terminate, lost context, no completion signal, inefficient paths.",
      "Consider closing the loop past detection — an eval agent holding the full analysis can propose the fix as a PR, not just a score."
    ],
    "topics": [
      "evals",
      "agents",
      "llm-as-a-judge",
      "agent-as-a-judge",
      "observability",
      "tracing",
      "production-monitoring"
    ],
    "tools": [
      "Arize",
      "Alex (Arize's in-product agent)",
      "Signal (Arize's agent-as-a-judge release)",
      "Anthropic",
      "OpenAI",
      "Terminal Bench",
      "Uber",
      "Snorkel",
      "Luma"
    ],
    "quotes": [
      {
        "text": "When we were building all of these first-gen evals, the thing that we were actually evaluating has changed underneath us.",
        "at": "02:00",
        "url": "https://www.youtube.com/watch?v=q2JrUKBMf0w&t=120s"
      },
      {
        "text": "So, this led to our really big revelation. What if the best way to an evaluate an agent was actually with an agent.",
        "at": "03:45",
        "url": "https://www.youtube.com/watch?v=q2JrUKBMf0w&t=225s"
      },
      {
        "text": "LLM as a judge just gives you a fixed rubric with these fixed scores. It's what everyone's doing, but when your agent's doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval.",
        "at": "04:12",
        "url": "https://www.youtube.com/watch?v=q2JrUKBMf0w&t=252s"
      },
      {
        "text": "My take is that most teams today are doing the first two, but the future of evals is actually having all three.",
        "at": "04:27",
        "url": "https://www.youtube.com/watch?v=q2JrUKBMf0w&t=267s"
      }
    ],
    "words": 1135
  },
  "summary_url": "/#q2JrUKBMf0w",
  "transcript": {
    "html": "/transcripts/q2JrUKBMf0w.html",
    "txt": "/transcripts/q2JrUKBMf0w.txt",
    "vtt": "/transcripts/q2JrUKBMf0w.vtt"
  }
}