WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=q2JrUKBMf0w

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/q2JrUKBMf0w.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.720 --> 00:00:13.400
Awesome.

s3
00:00:13.400 --> 00:00:14.160
Well, hey everyone.

s4
00:00:14.160 --> 00:00:16.520
My name is Aparna, one of the founders of Arize.

s5
00:00:16.520 --> 00:00:21.520
We work with some amazing teams to help them build evals.

s6
00:00:21.520 --> 00:00:26.000
Um, and we have an incredible lineup of talks for you all today at the evals track.

s7
00:00:26.000 --> 00:00:35.200
Um, it's happening in room 2005 and there's going to be amazing speakers from Term Bench and Uber and Snorkel kind of all happening after this.

s8
00:00:35.200 --> 00:00:38.880
Um, but today I'm here to talk to you about the future of evals.

s9
00:00:38.880 --> 00:00:50.280
Evals have gone from the new skill that every PM and every AI engineer has to learn to the thing that every serious AI team is betting on.

s10
00:00:50.280 --> 00:00:54.480
We've been really fortunate to get to work with some of the best AI teams in the world.

s11
00:00:54.480 --> 00:01:06.280
So, we get a front row seat into not just what's happening when they're building their actual agents and before they actually ship, but actually the evals that teams are running on their live

s12
00:01:06.280 --> 00:01:09.400
production agent via their traces.

s13
00:01:09.400 --> 00:01:10.840
Little bit of some stats for you guys.

s14
00:01:10.840 --> 00:01:13.560
We run over 100 million evals every month.

s15
00:01:13.560 --> 00:01:22.240
The average team runs about 12 different eval jobs with the top teams running over 3,800 different evaluators.

s16
00:01:22.240 --> 00:01:32.120
And offline evals, online evals, they each have their own place, but today what I'm actually going to talk to you about is the teams that are running evals on their traces.

s17
00:01:32.120 --> 00:01:43.160
This is actually what's helping teams figure out what's working, catch their failures, and that's the type of data you need to fuel your continual learning loops.

s18
00:01:43.240 --> 00:01:45.080
And the industry kind of agrees.

s19
00:01:45.080 --> 00:01:53.320
I mean, all the CPOs of Anthropic, OpenAI, all you know, GDB, you have Garry Tan saying, "Evals are everything you need."

s20
00:01:53.320 --> 00:01:55.880
And the whole industry kind of agrees.

s21
00:01:55.880 --> 00:02:00.680
So, we added evals, they catch all the failures, right?

s22
00:02:00.680 --> 00:02:01.840
Here's the problem.

s23
00:02:01.840 --> 00:02:08.720
When we were building all of these first-gen evals, the thing that we were actually evaluating has changed underneath us.

s24
00:02:08.720 --> 00:02:12.640
In 2023, it was about just answering a prompt.

s25
00:02:12.640 --> 00:02:16.400
In 2024, we started to see all the frontier models.

s26
00:02:16.400 --> 00:02:21.560
They've added tool calls, they've added reasoning, they've added deep research.

s27
00:02:21.560 --> 00:02:30.240
Now, what we have is teams running loops on real-world data with sub-agents kicked off on long-horizon tasks.

s28
00:02:30.240 --> 00:02:41.000
Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem.

s29
00:02:41.000 --> 00:02:46.920
What that meant is that as these systems got more complex, so did the way that they actually fail.

s30
00:02:46.920 --> 00:02:53.920
We're really lucky cuz we have our own agent that we've built, Alex, that lives in our UI, and we get our kind of get to feel this pain ourselves.

s31
00:02:53.920 --> 00:02:58.520
Every time the frontier labs added new functionality, we added it to our agent.

s32
00:02:58.520 --> 00:03:02.959
And now Alex can has much longer memory.

s33
00:03:02.959 --> 00:03:05.160
It has the ability to create dynamic UIs.

s34
00:03:05.160 --> 00:03:09.400
It can go search across an enormous volume of traces.

s35
00:03:09.400 --> 00:03:12.040
But, we also realized that it would forget context.

s36
00:03:12.040 --> 00:03:14.160
It wouldn't know when something was done.

s37
00:03:14.160 --> 00:03:17.120
Um sometimes it would just get stuck in these loops.

s38
00:03:17.120 --> 00:03:28.760
And the key thing here is that the classical LLM as a judge evals, that probably many of you have written in this room, just weren't for us to be able to catch

s39
00:03:28.760 --> 00:03:32.760
all the types of failures that we were experiencing.

s40
00:03:32.760 --> 00:03:34.480
I mean, it's just fundamentally different, right?

s41
00:03:34.480 --> 00:03:41.920
You have a deterministic flow, and now what we have is literally every time a user interacted with Alex, it would create a new UI.

s42
00:03:41.920 --> 00:03:45.040
That's a fundamentally different trajectory.

s43
00:03:45.040 --> 00:03:48.120
So, this led to our really big revelation.

s44
00:03:48.120 --> 00:03:54.440
What if the best way to an evaluate an agent was actually with an agent.

s45
00:03:54.480 --> 00:04:07.280
Doesn't mean that all of the ways that we did evals, with deterministic evals, with LLM as a judge, classic evals, doesn't matter anymore, but it just means that we have a different type of tool to solve a different type of problem.

s46
00:04:07.280 --> 00:04:12.040
Agent as a judge is about adaptive dynamic analysis.

s47
00:04:12.040 --> 00:04:15.920
LLM as a judge just gives you a fixed rubric with these fixed scores.

s48
00:04:15.920 --> 00:04:27.680
It's what everyone's doing, but when your agent's doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval.

s49
00:04:27.680 --> 00:04:35.400
My take is that most teams today are doing the first two, but the future of evals is actually having all three.

s50
00:04:35.400 --> 00:04:42.200
And today I'm actually excited to share we've released agent as a judge to help our teams on their eval journey.

s51
00:04:42.200 --> 00:04:43.640
We've released signal.

s52
00:04:43.640 --> 00:04:51.400
Signal's actually a long-running agent that can read traces sent in, discover patterns of issues.

s53
00:04:51.400 --> 00:05:00.440
Um, it can figure out types of problems that a classical LLM as a judge eval just would never be able to do with these deterministic rubrics.

s54
00:05:00.440 --> 00:05:10.480
It's helped us figure out very subtle failures that you wouldn't even think of doing, such as something going on in a loop for multiple times, it was calling the same tool

s55
00:05:10.480 --> 00:05:13.760
for repeatedly long time, the trajectory was inefficient.

s56
00:05:13.760 --> 00:05:19.840
And actually what this does is because it has all that analysis, it can go put up a PR and put up a fix.

s57
00:05:19.840 --> 00:05:23.640
So, if you want to learn more, come to our come to our booth.

s58
00:05:23.640 --> 00:05:25.000
We're right by the OpenAI booth.

s59
00:05:25.000 --> 00:05:28.280
We'll give you a demo, we'll show you a bit more about it.

s60
00:05:28.280 --> 00:05:33.360
We're also, like I said, taking over the evals track, so come to room 2005.

s61
00:05:33.360 --> 00:05:37.919
We're going to be talking a lot about the future of evals and what they look like.

s62
00:05:37.919 --> 00:05:47.320
And if you just want to hang out with our team, we're throwing a viewing party for the USA World Cup game tonight, so check out the Luma and register to come join us.

s63
00:05:47.320 --> 00:05:47.600
Awesome.

s64
00:05:47.600 --> 00:05:50.160
Thank you all so much.

s65
00:05:56.324 --> 00:05:58.324
[music]

s66
00:06:02.874 --> 00:06:04.874
[music]