WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=aVbRSIQO2dE

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/aVbRSIQO2dE.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.520 --> 00:00:13.680
Good afternoon.

s3
00:00:13.680 --> 00:00:16.600
Super excited to be talking about Evals today.

s4
00:00:16.600 --> 00:00:23.640
So today we'll be covering the difficulty of creating Evals for real world workflows in production.

s5
00:00:23.640 --> 00:00:32.400
How Google DeepMind and Kaggle are collaborating to solve this problem and a case study that we did with our partners at Link Alpha.

s6
00:00:32.400 --> 00:00:43.560
So they'll present next about a specific benchmark called Mandate Bench which basically looks at how how well models can perform as a buy-side equity analyst.

s7
00:00:43.800 --> 00:00:46.640
So to start with some quick intros.

s8
00:00:46.640 --> 00:00:47.360
I'm Jenny.

s9
00:00:47.360 --> 00:00:49.960
I'm a product manager at Google DeepMind.

s10
00:00:49.960 --> 00:00:53.120
I lead our Kaggle benchmarks team.

s11
00:00:53.120 --> 00:00:56.800
Ivan is our amazing forward deployed engineer at GDM.

s12
00:00:56.800 --> 00:00:59.600
He can't make it today unfortunately.

s13
00:00:59.600 --> 00:01:04.440
Jacob is in the crowd and from our Link Alpha partners.

s14
00:01:04.440 --> 00:01:08.840
So he'll be presenting with me and Jack also couldn't make it unfortunately.

s15
00:01:10.000 --> 00:01:14.800
So just starting with a quick intro about Kaggle and how we got here.

s16
00:01:14.800 --> 00:01:21.600
So Kaggle you guys probably know Kaggle as the home for data science and ML competitions.

s17
00:01:21.600 --> 00:01:27.880
I'm sure many of you have have interacted with Kaggle or competed in a Kaggle competition in in the past.

s18
00:01:27.880 --> 00:01:30.480
And so we've been doing this for for 15 years.

s19
00:01:30.480 --> 00:01:35.520
Basically evaluating humans on different capabilities in in the real world.

s20
00:01:35.520 --> 00:01:41.160
And we've built up a community of 31 million AI and ML enthusiasts across the platform.

s21
00:01:41.160 --> 00:01:47.480
Thousands of competitions over the last 15 years and one leaderboard that everyone really trusts.

s22
00:01:47.480 --> 00:01:53.240
And what has really made Kaggle succeed in the last 15 years is is our rigor.

s23
00:01:53.240 --> 00:01:55.840
So we have sealed private test data sets.

s24
00:01:55.840 --> 00:02:10.119
We um leak detection, anti-cheating infrastructure, and really make sure that that every single competition that that is put on Kaggle uh is designed really well and good at evaluating humans on on the capability that it's supposed to be.

s25
00:02:10.119 --> 00:02:11.720
Um and so that brings us to today.

s26
00:02:11.720 --> 00:02:23.560
So, in February 2026, uh Kaggle actually moved into Google DeepMind um and we're moving our subject from evaluating humans to evaluating frontier models.

s27
00:02:23.560 --> 00:02:28.480
Uh and so Kaggle is basically become GDM's public platform for for benchmarks.

s28
00:02:28.480 --> 00:02:43.160
And our goal is to make an open, neutral proving ground for every single frontier model uh that uses the same infrastructure, the same tasks, and the same grading um so that you every score is representative.

s29
00:02:44.040 --> 00:02:51.680
Um and our goal is really to become the industry standard for benchmarks the way that Kaggle uh already is for competitions.

s30
00:02:52.600 --> 00:02:54.200
So, why does this actually matter?

s31
00:02:54.200 --> 00:02:56.240
Why are we Why are we getting into this problem?

s32
00:02:56.240 --> 00:02:58.760
Why is Kaggle trying to tackle this problem?

s33
00:02:58.760 --> 00:03:01.239
Um and it's because benchmarks are really important.

s34
00:03:01.239 --> 00:03:05.560
Benchmarks are how we measure progress in the AI industry.

s35
00:03:06.080 --> 00:03:10.239
And right now, the state of benchmarks is fundamentally broken.

s36
00:03:10.239 --> 00:03:13.519
So, you think about all the different ways that benchmarks are used today.

s37
00:03:13.519 --> 00:03:17.480
Uh frontier labs are using them to train and hill climb models.

s38
00:03:17.480 --> 00:03:22.440
Startups are using them to decide what models that they want to build their product around.

s39
00:03:22.440 --> 00:03:25.800
Investors are using it to judge technical amounts.

s40
00:03:25.800 --> 00:03:34.640
Um and how you measure like what whether the benchmark is correctly measuring uh what you want it to measure is exactly what the models end up hill climbing.

s41
00:03:34.640 --> 00:03:41.480
Um and when you measure the wrong thing, the models actually get worse at at what they're supposed to be doing.

s42
00:03:41.720 --> 00:03:45.280
So, there's a few different ways that benchmarks fail today.

s43
00:03:45.280 --> 00:03:47.800
Um the first one is is relevance.

s44
00:03:47.800 --> 00:03:54.440
So, most public benchmarks don't actually resemble the real work that that your agents are doing in production today.

s45
00:03:54.440 --> 00:04:06.680
So, most of them test synthetic puzzles, academic trivia, random math problems, um and that bears little little resemblance to what your agent actually does in in production.

s46
00:04:07.200 --> 00:04:17.720
The second problem with benchmarks today is it's very very expensive in terms of a compute perspective, uh which is the like most scarce re- resource in the AI industry today.

s47
00:04:17.720 --> 00:04:28.600
So, take a benchmark, let's say it has 500 multi-turn agent tasks, and you want to run it across 16 frontier models, um that costs tens of thousands of dollars per sweep,

s48
00:04:28.600 --> 00:04:37.440
um and every time that a new model launches, which is basically every week or two at this point, um you have to do the sweep again in order to keep your benchmark current,

s49
00:04:37.440 --> 00:04:41.280
otherwise it it becomes stale.

s50
00:04:41.600 --> 00:04:48.160
And the third problem that we see, uh which is probably the most underappreciated, is verification.

s51
00:04:48.160 --> 00:05:02.320
So, everyone like all the benchmarks in the industry, everyone runs their own harnesses, they report their own scores, they they build their own like self-reported benchmark implementation, and the numbers are wildly different across different implementations.

s52
00:05:02.320 --> 00:05:07.280
And this is mostly because of infrastructure, not because of actual differences in model quality.

s53
00:05:07.280 --> 00:05:16.520
So, if you have different scaffolding, if you have different ways that you're calling tools, if you input if you just serialize the input into the model in different ways,

s54
00:05:16.520 --> 00:05:22.880
um you end up with different benchmark scores, even if it's the exact same benchmark on the same model.

s55
00:05:24.240 --> 00:05:31.200
Um so, that last point, uh around verification, I think, uh is probably the most abstract.

s56
00:05:31.200 --> 00:05:34.720
Uh so, wanted to to deep dive into that a little bit.

s57
00:05:34.720 --> 00:05:45.560
Um so, we, uh in order to really understand the the state of the benchmark industry and all of the different problems, uh we at Kaggle spent the last 2 months reproducing

s58
00:05:45.560 --> 00:06:00.880
65 different industry benchmarks, um and recreated all of them, ran all of them, and read every single trajectory trace by trace to to really understand what the problems were so that we could figure out like what the infrastructure we wanted to build was to fix it.

s59
00:06:00.880 --> 00:06:05.600
And we saw tons of issues across all of these different benchmarks.

s60
00:06:05.600 --> 00:06:10.120
So I'm listing a few of them here just to give you an idea of all of the different problems.

s61
00:06:10.120 --> 00:06:27.320
So for one benchmark we found the model that was being evaluated actually found the API keys in the eval container for a different model and then called different model and then use that response and submitted it as as their response to the eval.

s62
00:06:27.320 --> 00:06:37.000
Second, most people when they when they build their benchmark, they they don't actually separate the agent being evaluated from the agent being graded.

s63
00:06:37.000 --> 00:06:41.760
They share the same the same container and models are really good at at hacking their container now.

s64
00:06:41.760 --> 00:06:54.240
So we found models basically modified the grading script in order to give them whatever score that they wanted which was 100% and and let themselves pass.

s65
00:06:54.600 --> 00:06:57.200
The third is just pure like statistical variance.

s66
00:06:57.200 --> 00:07:09.360
So we found for all 65 benchmarks we ran all of them multiple times and we found that running the exact same benchmark even on the same infrastructure multiple times produced

s67
00:07:09.360 --> 00:07:13.040
variance of up to plus or minus 20% in the score.

s68
00:07:13.040 --> 00:07:17.520
And again, this is the same model on the same exact harness on the same exact infra.

s69
00:07:17.520 --> 00:07:24.760
So this is just reflective of instability in in the the harness or the benchmark design itself.

s70
00:07:25.680 --> 00:07:29.680
I think everyone's familiar with the hugging face incident.

s71
00:07:29.680 --> 00:07:39.919
So we found instances of models basically trying to look up the answers online in different forums like hugging face or Reddit or Stack Overflow.

s72
00:07:40.040 --> 00:07:43.040
And then this one we were very surprised by.

s73
00:07:43.040 --> 00:07:48.520
So, each harness uh actually ships with its own default reasoning effort.

s74
00:07:48.520 --> 00:07:56.160
Um, so if you use the default uh without setting it, um then you actually end up comparing models uh at different reasoning efforts.

s75
00:07:56.160 --> 00:08:05.320
So, you might have one model that's running on high and one that's running on low if you uh if you don't actually configure it uh in order to make sure that it's a head-to-head comparison.

s76
00:08:05.320 --> 00:08:14.760
Um, and a lot of the times when we were recreating it, we found that people did not do this uh because our scores actually matched parity with the original benchmark paper,

s77
00:08:14.760 --> 00:08:19.160
um but we found issues with with the score itself.

s78
00:08:20.000 --> 00:08:27.520
Um, and so given all of these problems in in the industry, uh Google DeepMind and Kaggle are are partnering to solve this.

s79
00:08:27.520 --> 00:08:29.440
So, we're doing a few things here.

s80
00:08:29.440 --> 00:08:41.520
Uh the first is we are partnering with companies and researchers uh to build to build benchmarks around real workflows uh economically viable real real-world workflows.

s81
00:08:41.520 --> 00:08:49.560
Um, and to make this happen, we are offering free compute and frontier model inference uh to anyone who wants to build a benchmark.

s82
00:08:49.560 --> 00:09:03.200
We're giving out $5 million in grants in in Q4 to fund any benchmark creation or research into fixing this space of of benchmarks given how early a space it is.

s83
00:09:03.200 --> 00:09:15.200
Um, and third, we will uh we will provide you a forward deployed engineer from from GDM to help you co-design your eval uh to make sure the methodology is actually rigorous.

s84
00:09:16.120 --> 00:09:24.720
Second, uh we we talked about this a lot on the last slide, but a lot of these problems are just issues all up and down the infrastructure stack.

s85
00:09:24.720 --> 00:09:33.400
Um, and so what we're building at Kaggle is standardized infrastructure so that every score is auditable, uh high quality, and and repeatable.

s86
00:09:33.400 --> 00:09:43.560
So, this includes um sandboxing every run, having logged proxy calls and open traces for for every model call, and containerized grading.

s87
00:09:44.280 --> 00:09:50.920
And then lastly, uh, the reason why we specifically are doing this, uh, is is for our expertise.

s88
00:09:50.920 --> 00:10:01.080
Um, so Kaggle has 15 years of designing rigorous tests and preventing cheating by humans, which we are now applying to to the model evaluation space.

s89
00:10:01.080 --> 00:10:07.240
Um, we're bringing GDM researcher expertise to to help us design all of these these real-world benchmarks.

s90
00:10:07.240 --> 00:10:20.200
Um, and lastly, we're actually partnering with independent research orgs, uh, like Epoch to audit all of the benchmarks on our platform, so you don't just have to take our word word for it.

s91
00:10:20.440 --> 00:10:25.680
So, at this point, I hope I've convinced you, uh, how important benchmarks are.

s92
00:10:25.680 --> 00:10:34.600
Um, but you're probably wondering, why should I build a benchmark as me as like a company with other priorities and other things to do?

s93
00:10:34.600 --> 00:10:37.640
Um, and there's a two two big reasons.

s94
00:10:37.640 --> 00:10:42.360
Um, the first is it actually allows you to build faster internally.

s95
00:10:42.360 --> 00:10:55.880
So, having a benchmark and having a rigorous evaluation suite, um, allows you to easily tell when when you model modify your your prompt, when you modify your hardness, whether you've broken something or how you've improved your product.

s96
00:10:55.880 --> 00:11:07.000
Um, and then the second is if you then want to publish your benchmark externally, uh, then labs will be encouraged to actually hill climb on your use case.

s97
00:11:07.000 --> 00:11:10.920
Um, like most of the labs are are looking for real-world benchmarks.

s98
00:11:10.920 --> 00:11:25.400
Um, and once the labs want to hill climb on your benchmark, then they'll pour trillions of dollars in of compute into improving the base models, uh, for your specific use case, so that your product gets better for your users over time.

s99
00:11:26.360 --> 00:11:37.760
Um, and with that, I will pass it over to Link Alpha, um, to to talk through a case study of Mandate Bench, which is one of the first benchmarks that we built on Kaggle.

s100
00:11:39.548 --> 00:11:40.858
[clears throat]

s101
00:11:40.858 --> 00:11:42.858
[applause]

s102
00:11:44.040 --> 00:11:45.200
Great.

s103
00:11:45.200 --> 00:11:45.800
Hi everyone.

s104
00:11:45.800 --> 00:11:47.080
Um this is Suyo.

s105
00:11:47.080 --> 00:11:52.520
I'm head of AI at Link Analytics, and we create we created a benchmark called Mandate Bench.

s106
00:11:52.520 --> 00:11:57.520
It's built co-developed with Google DeepMind and Kaggle.

s107
00:11:57.520 --> 00:12:00.360
So before start, let me share briefly about our company.

s108
00:12:00.360 --> 00:12:09.680
So Link Analytics builds AI research agent for institutional investors, institutions with more than $5 trillion in assets under management using Link Analytics.

s109
00:12:09.680 --> 00:12:13.000
And more than 200 investors use it actively.

s110
00:12:13.000 --> 00:12:16.960
There are some of the firms that trust us I listed up here.

s111
00:12:16.960 --> 00:12:19.160
And we're we're also a research team.

s112
00:12:19.160 --> 00:12:33.913
We have published 13 papers at at the venue like ICLR or ICML with co-authored with the BlackRock, Blackstone, JP Morgan, etc. So um

s113
00:12:33.913 --> 00:12:34.400
[clears throat]

s114
00:12:34.400 --> 00:12:39.360
I'd like to first go through the existing financial benchmarks.

s115
00:12:39.920 --> 00:12:48.000
So most of these benchmarks prepare a a golden answers and compare it with the agent answer.

s116
00:12:48.000 --> 00:12:53.280
But in real world task two analysts can reach two different answers and both could be right.

s117
00:12:53.280 --> 00:12:56.760
So existing benchmark use relatively short and simple task.

s118
00:12:56.760 --> 00:13:03.440
They don't um um they don't task a full unit of analyst work.

s119
00:13:03.440 --> 00:13:05.680
So they only get the fragments of it.

s120
00:13:05.680 --> 00:13:07.960
So Mandate Bench aims to do two things.

s121
00:13:07.960 --> 00:13:13.040
First, its rubric explicit grades the reasoning path.

s122
00:13:13.040 --> 00:13:17.280
Second, we take the real world real unit of work seriously.

s123
00:13:17.280 --> 00:13:30.120
For our analyst, the unit goes from the funds mandate or and from the mandate all the way to an investment memo.

s124
00:13:31.080 --> 00:13:35.800
So, let's first look at what a task in the benchmark looks like.

s125
00:13:35.800 --> 00:13:38.520
Let me start with a simple question.

s126
00:13:38.520 --> 00:13:43.560
Um how does an analyst work at a um hedge fund?

s127
00:13:43.560 --> 00:13:46.440
Nobody gives an analyst a simple task.

s128
00:13:46.440 --> 00:13:51.480
The PM gave analyst the funds mandate and a company.

s129
00:13:51.480 --> 00:14:01.120
For example, the mandate looks like we are an event-driven fund and we trade on mergers and acquisitions with a holding period 3 to 12 months.

s130
00:14:01.120 --> 00:14:09.520
And the PM asked, "Okay, here's a company A and we know company A will be bought by company B. Should we go long, short, or just pass on this company?

s131
00:14:09.520 --> 00:14:16.320
And please bring me a investment memo with detailed evidence and the reasoning behind it."

s132
00:14:16.480 --> 00:14:21.040
And this can take an analyst hours, 20 to 22 hundred hours easily.

s133
00:14:21.040 --> 00:14:25.040
And this is the one real unit of analyst work.

s134
00:14:26.760 --> 00:14:31.000
So, let's go deeper into an example task.

s135
00:14:31.000 --> 00:14:34.800
Here's a company called Sonoco and it's a US packaging company.

s136
00:14:34.800 --> 00:14:42.200
Recently, Sonoco bought about a company called Eviosys and it's also sold one of its business called TFP.

s137
00:14:42.200 --> 00:14:46.560
To value Sonoco, you first need one basic number, which is a sales.

s138
00:14:46.560 --> 00:14:52.240
But five different sources give the different um sales numbers from 5 5 billion to 8 billion dollars.

s139
00:14:52.240 --> 00:14:52.680
Why?

s140
00:14:52.680 --> 00:14:56.920
Because each source counts a different different Sonoco.

s141
00:14:56.920 --> 00:15:03.080
So, first, the agent has to understand the numbers doesn't match.

s142
00:15:03.080 --> 00:15:07.120
Second, it hasn't it has to find out why.

s143
00:15:07.120 --> 00:15:20.360
And and um and third, and also second, it has to find out why and third, it has to rebuild Sonoco as it was in May 2025 with Eviosys for the

s144
00:15:20.360 --> 00:15:21.760
full year and without TFP.

s145
00:15:21.760 --> 00:15:26.360
This it will be about 7.5 billion dollars of sales.

s146
00:15:26.360 --> 00:15:32.960
And that is just a small part of the entire reasoning to skill analyst to work through.

s147
00:15:32.960 --> 00:15:35.640
So, then how we grade the agent work?

s148
00:15:35.640 --> 00:15:45.200
So, agent submits an investment memo long together with a reasoning graph in the graph and then every fact should point to exact two call it came from.

s149
00:15:45.200 --> 00:15:48.450
Then our former buy-side analyst

s150
00:15:48.450 --> 00:15:49.600
[clears throat]

s151
00:15:49.600 --> 00:15:53.520
write down the key reasoning steps that any good memo would include.

s152
00:15:53.520 --> 00:15:56.080
Each step is a node in the graph.

s153
00:15:56.080 --> 00:16:00.680
First, the no-code agent must rebuild the company it was in May 2025.

s154
00:16:00.680 --> 00:16:09.440
And what the new new new business really earns and net out what was sold and rebuild the debt.

s155
00:16:09.640 --> 00:16:15.200
If first that we ask HR, we also attacked the memo with the what if test.

s156
00:16:15.200 --> 00:16:20.160
For example, management pro- promised cost savings from the EVOC still.

s157
00:16:20.160 --> 00:16:23.600
But, what if the sales savings were zero?

s158
00:16:23.600 --> 00:16:25.400
Would the call change?

s159
00:16:25.400 --> 00:16:31.240
And does the memo explain this with clear reasoning about the moving parts?

s160
00:16:31.240 --> 00:16:41.400
So, the So, the So, the memos can reach different calls and the both can score well as long as the work behind the each calls are there.

s161
00:16:41.400 --> 00:16:47.120
So, this is the result and you can see the cloud of This is a preliminary result.

s162
00:16:47.120 --> 00:16:53.040
Only runs over 10 tasks of 50 entire task universe.

s163
00:16:53.040 --> 00:16:56.560
And the cloud with 5.5 scores best.

s164
00:16:56.560 --> 00:16:59.640
In the matter of time, let's skip to the next page.

s165
00:16:59.640 --> 00:17:01.160
And every model behaves differently.

s166
00:17:01.160 --> 00:17:11.000
For example, Astra 6 really good at making a financial modeling, while the Opus 5.5 is a balanced very well over the cross different um test.

s167
00:17:11.000 --> 00:17:14.000
Okay, let me get back to Shane.

s168
00:17:17.680 --> 00:17:18.600
Um awesome.

s169
00:17:18.600 --> 00:17:22.520
So, that was the first initial 10 tasks um of Mandate Bench.

s170
00:17:22.520 --> 00:17:28.920
The the full 20 50 task benchmark will be launching uh next month in in November 2026.

s171
00:17:28.920 --> 00:17:37.480
Um and uh this is part of many many other benchmarks that are launching in on Kaggle and and DeepMind over the next few weeks.

s172
00:17:37.480 --> 00:17:43.080
Uh we've partnered with a bunch of leading AI startups uh to build benchmarks in different domains.

s173
00:17:43.080 --> 00:17:47.640
Uh we launched HeyGen's code code to video bench uh last week.

s174
00:17:47.640 --> 00:18:00.560
Um we're working on a couple other benchmarks with Octava, Quadrillion, and Rehearsals uh on on meta uh clinician workflows, research workflows, uh and user research workflows.

s175
00:18:00.560 --> 00:18:11.080
Um so, if anyone in the crowd is interested in in building a real-world benchmark for for your company, uh we're we're very interested in collaborating.

s176
00:18:11.080 --> 00:18:19.200
Um as I mentioned before, we're we're providing free model compute, uh FTE support from from JDM, and $5 million in grants.

s177
00:18:19.200 --> 00:18:20.800
Um so, please reach out to us.

s178
00:18:20.800 --> 00:18:22.760
We we'd love to work together.

s179
00:18:22.760 --> 00:18:25.560
Um thank you so much.