WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=rWtbOgl5FTU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/rWtbOgl5FTU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.520 --> 00:00:19.320
I'm going to talk a little bit about uh going from beyond the model card, how evaluations can be used to measure this jagged frontier.

s3
00:00:19.320 --> 00:00:23.240
Uh special focus on finance for this audience as well.

s4
00:00:23.240 --> 00:00:32.320
Um yeah, so first off talking about model cards themselves, we've built uh some of the most important benchmarks over the last couple of years that the major foundation model labs have included in their model releases.

s5
00:00:32.320 --> 00:00:35.320
So people I think are familiar with artifacts like this.

s6
00:00:35.320 --> 00:00:44.240
Uh last week we have Sunder talking about our Val's index on stage and then uh even just yesterday Muse models uh reciting our our Finagint benchmark.

s7
00:00:44.240 --> 00:00:48.320
You know, it keeps our lives interesting to have to keep re-updating these slides as new models come out.

s8
00:00:48.320 --> 00:00:57.320
I had to double-check new models didn't come out this morning, but um these are say these these kind of model cards reflect a general understanding of what's newly capable with the models.

s9
00:00:57.320 --> 00:01:02.880
Um and so uh they they can reveal from a single number how intelligent is a model.

s10
00:01:02.880 --> 00:01:12.600
In particular, our Val's index is compiling results to try and measure the economic impact of models uh in domains like finance, legal, but also coding, health care, and things like that.

s11
00:01:12.600 --> 00:01:18.640
So from these high-level numbers, you can get a sense of how capable a model is for uh valuable work.

s12
00:01:18.640 --> 00:01:20.480
Um but that doesn't give the whole picture.

s13
00:01:20.480 --> 00:01:27.760
And so when we dial in a little bit more closely and look at the constituent benchmarks that comprise the Val's index, we can see a more nuanced picture.

s14
00:01:27.760 --> 00:01:30.360
And this may be something you all are familiar with, the jagged frontier.

s15
00:01:30.360 --> 00:01:37.480
And the idea that within a particular domain or set of tasks, you may see different performance uh from when you aggregate results to produce that single number.

s16
00:01:37.480 --> 00:01:43.280
And so you can see here across the different models, uh we actually see an inconsistent performance for who's number one.

s17
00:01:43.280 --> 00:01:47.880
Uh but that's made even more so when you look at particular organizations or companies.

s18
00:01:47.880 --> 00:01:52.680
So in a lot of ways, coding is indicative of what's happening across every major vertical.

s19
00:01:52.680 --> 00:02:01.052
Um when we look at coding and actually create custom benchmarks for particular repositories, we see that that jaggedness just increases even more.

s20
00:02:01.052 --> 00:02:04.892
And so on on the x-axis you've got different programming languages and libraries.

s21
00:02:04.892 --> 00:02:09.932
And on the y-axis you see the relative performance of models and see that it's even more fragmented.

s22
00:02:09.932 --> 00:02:20.012
But this isn't even to take into account that there are different harnesses in which these models can can be used or the idea that you might be optimizing for cost and latency as well as performance.

s23
00:02:20.012 --> 00:02:33.252
And so this all feeds into the idea that the the granularity of of the performance you see in a general model card benchmark is not necessarily what's most applicable for people who are using these models or or products in practice.

s24
00:02:33.252 --> 00:02:36.332
In parallel, AI adoption has gone up considerably.

s25
00:02:36.332 --> 00:02:40.812
So year over year there's roughly a 2x increase in enterprise AI spend.

s26
00:02:40.812 --> 00:02:47.412
But with all that spend it's still really obtuse to understand where the ROI is coming from.

s27
00:02:47.412 --> 00:02:55.412
You can see that people regularly using AI is now 90% but if you look at the smallest bar now we have established ROI just at 7%.

s28
00:02:55.412 --> 00:03:02.412
So for all this token spend and increased adoption it's still very unclear where AI is actually driving value for a lot of companies.

s29
00:03:02.412 --> 00:03:11.772
And I think two interesting stats that jump out at me, the groups that are able to actually track AI cost and spend are much much more likely to actually understand where ROI is going to come from.

s30
00:03:11.772 --> 00:03:15.372
And this is also a key blocker for adoption itself.

s31
00:03:15.372 --> 00:03:23.012
The groups that are unable to actually quantify the cost to the return on their investment often see this as a big bottleneck to actually widely roll out AI.

s32
00:03:23.012 --> 00:03:24.772
And so I think evals is the key story here.

s33
00:03:24.772 --> 00:03:36.772
If people are able to construct key evaluations that measure what our model's capable of and what will be the most efficient utilization of those dollars, they'll be able to adopt AI in their firm more broadly.

s34
00:03:40.692 --> 00:03:49.372
I think one particular interesting finding that that I want to share with you all is an indication of why the intuition that are intuition of model performance as presented by labs

s35
00:03:49.372 --> 00:03:51.092
doesn't often align with experience.

s36
00:03:51.092 --> 00:03:56.452
So if you look at Opus 5.5 that's the kind of second tier model from Anthropic.

s37
00:03:56.452 --> 00:03:57.812
And on paper it's actually cheaper.

s38
00:03:57.812 --> 00:04:02.492
$4 and $20 out compared to Fable 5.1's $10 and $50 out.

s39
00:04:02.492 --> 00:04:08.334
But in practice, and particularly for this tax benchmark, you see that it's actually more expensive, 14 more expensive.

s40
00:04:08.334 --> 00:04:13.214
And that's because the less intelligent model is trying to expend more tokens to try and get its work done.

s41
00:04:13.214 --> 00:04:16.774
But this actually bears out as a 3% lower score.

s42
00:04:16.774 --> 00:04:20.454
Um and so you are actually would have been better off using the frontier intelligence.

s43
00:04:20.454 --> 00:04:23.014
You would have spent less and got a better performance.

s44
00:04:23.014 --> 00:04:27.734
Um but that kind of intuition isn't isn't directly available for people who are using the models in practice.

s45
00:04:27.734 --> 00:04:30.494
And so they may try and say, "Actually, I should use a Sonnet or an Opus."

s46
00:04:30.494 --> 00:04:34.014
But end up overpaying for uh inferior performance.

s47
00:04:34.014 --> 00:04:39.174
Uh and so I think this, you know, reveals that we should actually take a more diligent effort to figuring out where return is coming from.

s48
00:04:39.174 --> 00:04:45.334
And and that will inform a more realistic picture as opposed to the intuition that labs are feeding directly.

s49
00:04:45.614 --> 00:04:54.014
If we look at look at finance in particular, so that this kind of jagged frontier concept, uh I've highlighted the Vals Index, but we also have a finance agent benchmark

s50
00:04:54.014 --> 00:04:58.614
looking at research tasks for finance, and then Excel modeling, uh and then tax agent.

s51
00:04:58.614 --> 00:05:06.014
Uh and even across these benchmarks again, it's not immediately obvious which is number one or will be consistently two or three.

s52
00:05:06.254 --> 00:05:09.334
This picture is made even more nuanced when you look at the Pareto curve.

s53
00:05:09.334 --> 00:05:11.454
So on the x-axis, you've got cost.

s54
00:05:11.454 --> 00:05:15.054
In particular, it's cost per task, not just token in token out cost.

s55
00:05:15.054 --> 00:05:16.934
And on the y-axis, you have performance.

s56
00:05:16.934 --> 00:05:19.774
And and here what we're looking at is that Excel modeling benchmark.

s57
00:05:19.774 --> 00:05:27.254
So seeing whether these AI models can create and work with Excel models that are core to work that's happening in finance.

s58
00:05:27.254 --> 00:05:31.294
Uh in the green, you have the overall performance across all of the questions.

s59
00:05:31.294 --> 00:05:35.254
And then in red, what we've done is is segmented out the LBO tasks.

s60
00:05:35.254 --> 00:05:42.374
So this is a particular subset of Excel model tasks that you might you might be working with in the in uh um using any of these models.

s61
00:05:42.374 --> 00:05:44.134
Um and I want to call out a couple things.

s62
00:05:44.134 --> 00:05:50.254
In particular, first what you see is that the entire frontier for LBO is lower than the overall task score.

s63
00:05:50.254 --> 00:05:57.534
So the intuition you would have from just the the high-level EMB results would not match what you see for the LBO categories of tasks.

s64
00:05:57.534 --> 00:06:05.415
And so if your firm is especially doing, you know, one type of Excel models over another, it's important to have a clearer picture of what's going to be actually relevant to you.

s65
00:06:05.415 --> 00:06:11.495
But the other is that the frontier is also changing when you go between the overall scores to just the LBO subcategory.

s66
00:06:11.495 --> 00:06:18.975
So, whereas you might have thought you would have get the best possible performance from using Feable 5.1, when you go to the LBO it actually drops off the Pareto frontier.

s67
00:06:18.975 --> 00:06:24.095
You're going to be massively overpaying for what is no longer the state of the art model for that category.

s68
00:06:24.095 --> 00:06:35.895
Uh and so again, this kind of reiterates this idea that it's it's less important what you see at the top, but actually the subset of tasks that are most relevant for you and the cost performance optimization that you want to see within your company.

s69
00:06:36.495 --> 00:06:39.455
So, I want to share uh some some lessons from actually building the benchmarks.

s70
00:06:39.455 --> 00:06:43.815
We talked a little bit about the results and and why the results may be intuitive unintuitive.

s71
00:06:43.815 --> 00:06:50.055
Um when we actually build benchmarks, one of the most important things that we need to look at is the metric we use to report accuracy.

s72
00:06:50.055 --> 00:06:53.375
Uh and so you you you have a different takeaway if you pick a different metric.

s73
00:06:53.375 --> 00:06:55.455
A lot of benchmarks deal with partial credit.

s74
00:06:55.455 --> 00:07:07.215
And so what this is answering is in expectation, if I were to give a question or task to a model, what percentage of the points would I expect to uh see satisfied by the the the work the work output of that model,

s75
00:07:07.215 --> 00:07:09.015
especially compared to humans?

s76
00:07:09.015 --> 00:07:16.655
And then the fully corrected answers metric is more looking at of the percentage of questions that I give to models, what percentage of them will come back perfect?

s77
00:07:16.655 --> 00:07:18.375
And that's that's a very different metric.

s78
00:07:18.375 --> 00:07:25.615
It may mean uh that a model is very good at producing uh an incremental response that you can work with to build a better response.

s79
00:07:25.615 --> 00:07:32.535
Um but if you were to pass this directly to a client, it would be a coin toss of whether it was meeting all the criteria that you have.

s80
00:07:32.535 --> 00:07:40.495
Uh and so picking the right metric is very important to accurately representing the performance of a model and actually forming a good intuition of what it can do.

s81
00:07:40.615 --> 00:07:43.055
There's another three components that we think a lot about.

s82
00:07:43.055 --> 00:07:46.655
So, validity, are you actually evaluating what's going on in the real world?

s83
00:07:46.655 --> 00:07:50.935
That means are the questions or tasks reflective of actual work going on or are they contrived?

s84
00:07:50.935 --> 00:07:55.975
Um and then are the rubrics actually describing what we're looking for or the standard of excellence we want to see?

s85
00:07:55.975 --> 00:08:00.215
Reliability, this is usually a function of the number of tasks or the sample count.

s86
00:08:00.215 --> 00:08:06.695
Um a lot of companies, they have maybe five or six samples and they're kind of vibe checking new models or products as they get access.

s87
00:08:06.695 --> 00:08:14.777
Um but it's actually really important to have a rigorous approach and have enough tasks and samples to make sure your repeated runs actually show consistent results.

s88
00:08:14.777 --> 00:08:15.857
And then lastly, discriminability.

s89
00:08:15.857 --> 00:08:21.857
So, we we don't think it's that valuable to have benchmarks where all models score 100% or all models score 0%.

s90
00:08:21.857 --> 00:08:26.297
Ideally, we can we can actually discriminate between models and see what the frontier looks like.

s91
00:08:26.297 --> 00:08:29.817
In particular, we also want to make sure we're building benchmarks forward-looking.

s92
00:08:29.817 --> 00:08:40.817
So, what are the new set of tasks that are even more ambitious than what models can do today uh to make sure that there's room at the frontier for us to uh you could uh to continue to see model progress as they get better.

s93
00:08:41.497 --> 00:08:41.737
Okay.

s94
00:08:41.737 --> 00:08:48.257
So, and then some of the high-level problems that we've seen and I think others uh you know, can and and do run into.

s95
00:08:48.257 --> 00:08:50.577
The first is that agents solve the task, not the problem.

s96
00:08:50.577 --> 00:08:53.257
So, this is talked a lot about as reward hacking.

s97
00:08:53.257 --> 00:08:59.137
Um, and we've seen a lot of instances where we'll have a sandbox and models will be run with our model proxy.

s98
00:08:59.137 --> 00:09:06.177
And sometimes they may try and actually call other models uh within that proxy to escape the sandbox or make use of the internet.

s99
00:09:06.177 --> 00:09:11.057
Uh, so it's really important to to build the the containers in a way that you're actually testing for what matters.

s100
00:09:11.057 --> 00:09:16.937
Can they can they do the task at hand um or are they actually leveraging other tools or maybe even other models to get there?

s101
00:09:16.937 --> 00:09:25.777
Um, we've also seen this issue in finance in particular where a lot of the times uh evals are looking at a set of rubrics or or items that we want to see included in the response.

s102
00:09:25.777 --> 00:09:34.937
And so, models can hedge and give three possible answers uh which has a higher chance of satisfying all the criteria um but in practice, that's way way too much.

s103
00:09:34.937 --> 00:09:37.017
A person does not want to consider three different answers.

s104
00:09:37.017 --> 00:09:40.377
They just want to get the the first answer and and have it be right.

s105
00:09:40.377 --> 00:09:47.417
Um, and so, the goal here is to try and build evals that are reflective of the actual work people are doing and the work output they want to see

s106
00:09:47.417 --> 00:09:51.537
um not to build these contrived evals that that the models can actually gain.

s107
00:09:51.537 --> 00:09:59.897
The second is I think a lot of people are starting to see that the cost of software is coming down and it's it's actually very very easy to produce code.

s108
00:09:59.897 --> 00:10:06.817
But unfortunately, we still see in practice that running evals is incredibly challenging, especially from an infrastructure side of things.

s109
00:10:06.817 --> 00:10:15.698
Um, and in particular, as we get to longer-running agents, it means that it's really important to have infrastructure that can stay consistent over a long period of time and then if interrupted,

s110
00:10:15.698 --> 00:10:17.938
can be resumed and respawned back up.

s111
00:10:17.938 --> 00:10:22.578
Um, we have I think just recently crossed 6 trillion tokens just in evaluation.

s112
00:10:22.578 --> 00:10:29.778
Um, so we run a considerable asset of evals and we run everything in as highest degree parallel as possible.

s113
00:10:29.778 --> 00:10:35.258
Um, so I think this has really been load tested, but I think other people are going to run into this even on a much smaller scale.

s114
00:10:35.258 --> 00:10:43.298
Uh, and the last is that I think a lot of the existing benchmarks, they deal with a single prompt, single response, but in practice everything is multi-turn.

s115
00:10:43.298 --> 00:10:52.818
Uh, you have an initial spec which you think covers everything, but then you get questions back or you have additional feedback once something's built and then it takes more of this iteration to actually drive to the end output.

s116
00:10:52.818 --> 00:10:57.058
And so I think benchmarks and evals need to actually cover this multi-turn component.

s117
00:10:57.058 --> 00:10:59.698
That's not going to go away.

s118
00:10:59.978 --> 00:11:08.258
So to look at a case study in particular, when we looked at our team, we did a little token maxing experiment uh, where we we got free access to some tools and just gave it to all the engineers.

s119
00:11:08.258 --> 00:11:11.658
Um, and they they really made use of these unlimited tokens.

s120
00:11:11.658 --> 00:11:17.098
Um, and and if if we actually look back for that month, we spent 1.5 million dollars worth of tokens.

s121
00:11:17.098 --> 00:11:18.978
This is like a team of I think 10 or 12.

s122
00:11:18.978 --> 00:11:23.858
So it's roughly 10 times the amount we were spending on salary for tokens in that month.

s123
00:11:23.858 --> 00:11:26.178
Uh, and so that was pretty insane, but we saw great output.

s124
00:11:26.178 --> 00:11:33.178
And so our goal was to try and maintain the same level of output across all these engineers, but massively cut cut down on cost.

s125
00:11:33.178 --> 00:11:36.418
And and really our solution to do that was of course to build an eval.

s126
00:11:36.418 --> 00:11:41.538
Um, we built a bunch of tools internally and then have now exposed this as a product called Val Smith.

s127
00:11:41.538 --> 00:11:47.498
Uh, and so this is you know, what we're doing in coding and I think it's again a sign of what's going to what's going to come in other parts of knowledge work.

s128
00:11:47.498 --> 00:11:55.938
Um, so what we were able to do is look at the GitHub repo um, as a essentialized way of seeing what work has already been done and what work is continuing to happen.

s129
00:11:55.938 --> 00:12:01.498
Um, and in particular GitHub repos are comprised of PRs or pull requests which have a GitHub issue and a resolution.

s130
00:12:01.498 --> 00:12:08.578
And so what we can do is unwind the clock and give those same tasks to agents to see if they could have equivalently done the work of humans.

s131
00:12:08.578 --> 00:12:18.419
Um, and in doing that we found a a a much clearer understanding of which models were overkill, were way too expensive, and did not need to be used, and then which ones were actually capable of getting the task done.

s132
00:12:18.419 --> 00:12:23.979
Uh and I think again, this is another place where the intuitions that a lot of people have may not be what's actually born out in practice.

s133
00:12:23.979 --> 00:12:27.139
So, I think you may see, "Oh, we're using a lot of close-source models.

s134
00:12:27.139 --> 00:12:30.619
We should try and cut costs and use our own self-hosted open-source model."

s135
00:12:30.619 --> 00:12:37.619
Uh you can see the comparison here we have between GPT-5.6 Soul and Kimiko 3, and we see that 5.6 Soul is better.

s136
00:12:37.619 --> 00:12:40.019
It's It's going 2% better on this internal eval.

s137
00:12:40.019 --> 00:12:45.579
It's also considerably cheaper than than hosting it through a very, you know, inference-optimized uh stack.

s138
00:12:45.579 --> 00:12:49.379
So, it's $2.50 instead of $11, and it's faster, half the time.

s139
00:12:49.379 --> 00:12:56.099
And so, there's no real reason why um this kind of intuitive sense of using open source would be a legitimate way to drive down cost.

s140
00:12:56.099 --> 00:12:58.299
Um and so, this has resulted in two main differences.

s141
00:12:58.299 --> 00:13:08.379
One, we can actually figure out which tools we're going to procure, what the usage limits of those things should be across the organization, but then also, we can give recommendations for engineers

s142
00:13:08.379 --> 00:13:15.099
and say, "We think that for this GitHub issue, you should actually use this particular coding tool with this model."

s143
00:13:15.099 --> 00:13:17.619
Uh and this is able to drive the cost down considerably.

s144
00:13:17.619 --> 00:13:22.699
I think we're spending 90% less, but still able to get the same output.

s145
00:13:23.139 --> 00:13:31.739
I think the question I would have for you all We're We're continuing to expand this approach and and take this model of coding to other domains, knowledge work, legal, finance, things like that.

s146
00:13:31.739 --> 00:13:37.299
But the key question is, where are you tracking the history of work that has already been done by that firm?

s147
00:13:37.299 --> 00:13:40.259
Some of these law firms have been around for like 100 years.

s148
00:13:40.259 --> 00:13:43.699
And so, where Where is that I think quality of the organization stored?

s149
00:13:43.699 --> 00:13:45.979
Where are you tracking the standard of excellence?

s150
00:13:45.979 --> 00:13:55.099
And so, that's today may exist in places like email from the history of responses that people have had with their clients, may exist in SharePoint, may exist in Databricks dumps,

s151
00:13:55.099 --> 00:14:05.979
but I think this is the key treasure trove that a lot of companies have and don't realize will be the major unlock for evals, and then the unlock for AI adoption more systematically.

s152
00:14:10.899 --> 00:14:18.541
Once these evals are constructed, they can enable adoption to happen, but we've also seen that there's more advantageous ways that they can be used.

s153
00:14:18.541 --> 00:14:20.901
So, first is right-sizing.

s154
00:14:20.901 --> 00:14:26.821
So, rather than just token maximizing, how do you figure out what the right usage limit should be across the organization?

s155
00:14:26.821 --> 00:14:27.701
Negotiation.

s156
00:14:27.701 --> 00:14:33.861
Once you realize uh you have actual leverage, you can take that to the conversation you have in bulk pricing.

s157
00:14:33.861 --> 00:14:34.581
Then routing.

s158
00:14:34.581 --> 00:14:41.461
This is becoming a core primitive in uh in a lot of AI products, um but can also be done by users directly.

s159
00:14:41.461 --> 00:14:46.101
And so, they can build intelligent routers that actually figure out the right model to use.

s160
00:14:46.101 --> 00:14:48.181
And then fine-tuning or post-training.

s161
00:14:48.181 --> 00:14:56.861
Uh this this can be the most reliable way for you to actually mint the tasks and then have an external eval set to make sure that this approach is working.

s162
00:14:57.581 --> 00:15:06.661
Last thought I'd leave you with is I think for a lot of companies, they have a competitive advantage that has existed in the heads of employees through things like institutional knowledge.

s163
00:15:06.661 --> 00:15:13.581
And what evals allow you to do is actually codify the competitive advantage and put that down in words or in scripts in language.

s164
00:15:13.581 --> 00:15:19.421
Uh and that will actually be the the biggest unlock for uh making sure that a company succeeds through the agentic era.

s165
00:15:19.421 --> 00:15:20.101
Okay.

s166
00:15:20.101 --> 00:15:20.381
That's all.

s167
00:15:20.381 --> 00:15:22.250
Thanks, everyone.

s168
00:15:22.250 --> 00:15:24.250
[applause]