The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

AI Engineer · 6 min · 66 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:12Awesome.
  3. 00:13Well, hey everyone.
  4. 00:14My name is Aparna, one of the founders of Arize.
  5. 00:16We work with some amazing teams to help them build evals.
  6. 00:21Um, and we have an incredible lineup of talks for you all today at the evals track.
  7. 00:26Um, it's happening in room 2005 and there's going to be amazing speakers from Term Bench and Uber and Snorkel kind of all happening after this.
  8. 00:35Um, but today I'm here to talk to you about the future of evals.
  9. 00:38Evals have gone from the new skill that every PM and every AI engineer has to learn to the thing that every serious AI team is betting on.
  10. 00:50We've been really fortunate to get to work with some of the best AI teams in the world.
  11. 00:54So, we get a front row seat into not just what's happening when they're building their actual agents and before they actually ship, but actually the evals that teams are running on their live
  12. 01:06production agent via their traces.
  13. 01:09Little bit of some stats for you guys.
  14. 01:10We run over 100 million evals every month.
  15. 01:13The average team runs about 12 different eval jobs with the top teams running over 3,800 different evaluators.
  16. 01:22And offline evals, online evals, they each have their own place, but today what I'm actually going to talk to you about is the teams that are running evals on their traces.
  17. 01:32This is actually what's helping teams figure out what's working, catch their failures, and that's the type of data you need to fuel your continual learning loops.
  18. 01:43And the industry kind of agrees.
  19. 01:45I mean, all the CPOs of Anthropic, OpenAI, all you know, GDB, you have Garry Tan saying, "Evals are everything you need."
  20. 01:53And the whole industry kind of agrees.
  21. 01:55So, we added evals, they catch all the failures, right?
  22. 02:00Here's the problem.
  23. 02:01When we were building all of these first-gen evals, the thing that we were actually evaluating has changed underneath us.
  24. 02:08In 2023, it was about just answering a prompt.
  25. 02:12In 2024, we started to see all the frontier models.
  26. 02:16They've added tool calls, they've added reasoning, they've added deep research.
  27. 02:21Now, what we have is teams running loops on real-world data with sub-agents kicked off on long-horizon tasks.
  28. 02:30Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem.
  29. 02:41What that meant is that as these systems got more complex, so did the way that they actually fail.
  30. 02:46We're really lucky cuz we have our own agent that we've built, Alex, that lives in our UI, and we get our kind of get to feel this pain ourselves.
  31. 02:53Every time the frontier labs added new functionality, we added it to our agent.
  32. 02:58And now Alex can has much longer memory.
  33. 03:02It has the ability to create dynamic UIs.
  34. 03:05It can go search across an enormous volume of traces.
  35. 03:09But, we also realized that it would forget context.
  36. 03:12It wouldn't know when something was done.
  37. 03:14Um sometimes it would just get stuck in these loops.
  38. 03:17And the key thing here is that the classical LLM as a judge evals, that probably many of you have written in this room, just weren't for us to be able to catch
  39. 03:28all the types of failures that we were experiencing.
  40. 03:32I mean, it's just fundamentally different, right?
  41. 03:34You have a deterministic flow, and now what we have is literally every time a user interacted with Alex, it would create a new UI.
  42. 03:41That's a fundamentally different trajectory.
  43. 03:45So, this led to our really big revelation.
  44. 03:48What if the best way to an evaluate an agent was actually with an agent.
  45. 03:54Doesn't mean that all of the ways that we did evals, with deterministic evals, with LLM as a judge, classic evals, doesn't matter anymore, but it just means that we have a different type of tool to solve a different type of problem.
  46. 04:07Agent as a judge is about adaptive dynamic analysis.
  47. 04:12LLM as a judge just gives you a fixed rubric with these fixed scores.
  48. 04:15It's what everyone's doing, but when your agent's doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval.
  49. 04:27My take is that most teams today are doing the first two, but the future of evals is actually having all three.
  50. 04:35And today I'm actually excited to share we've released agent as a judge to help our teams on their eval journey.
  51. 04:42We've released signal.
  52. 04:43Signal's actually a long-running agent that can read traces sent in, discover patterns of issues.
  53. 04:51Um, it can figure out types of problems that a classical LLM as a judge eval just would never be able to do with these deterministic rubrics.
  54. 05:00It's helped us figure out very subtle failures that you wouldn't even think of doing, such as something going on in a loop for multiple times, it was calling the same tool
  55. 05:10for repeatedly long time, the trajectory was inefficient.
  56. 05:13And actually what this does is because it has all that analysis, it can go put up a PR and put up a fix.
  57. 05:19So, if you want to learn more, come to our come to our booth.
  58. 05:23We're right by the OpenAI booth.
  59. 05:25We'll give you a demo, we'll show you a bit more about it.
  60. 05:28We're also, like I said, taking over the evals track, so come to room 2005.
  61. 05:33We're going to be talking a lot about the future of evals and what they look like.
  62. 05:37And if you just want to hang out with our team, we're throwing a viewing party for the USA World Cup game tonight, so check out the Luma and register to come join us.
  63. 05:47Awesome.
  64. 05:47Thank you all so much.
  65. 05:56[music]
  66. 06:02[music]