When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI

AI Engineer · 17 min · 173 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Let's get started.
00:14When will the benchmaxing plague end?
00:18In the tech industry, we love a hype cycle.
00:21And in AI, we really love a hype cycle.
00:24And the way we do that is when a model comes out, there's a big announcement, there's a lot of benchmark cited.
00:32Sometimes to keep things interesting, we do a little chart crime.
00:36And then people actually go and use it.
00:38And if the expectations aren't met by the reality, then we have allegations of benchmaxing.
00:45Benchmaxing, of course, being when labs are training too hard on benchmarks in a way that deviates from what people actually care about.
00:55So the existence of that term indicates that we have a sense that benchmarks don't always equal reality.
01:00And so in this talk we're going to figure out why does benchmaxing happen?
01:04Why are traditional benchmarks not always accurate reflections of real world value?
01:10Is this intrinsic to all benchmarks?
01:12And will we ever know which models are best?
01:14And the answers are incentives, poor methodologies, no and yes.
01:20All right, that was my talk.
01:21Thank you so much for coming.
01:22Um actually it looks like I have a few extra minutes so let's let's move on.
01:27I have a few extra slides we'll we'll go through.
01:31So we have a sense that benchmarks don't equal reality but the industry is dominated by a lot of popular but very bad benchmarks.
01:39So there's millions of dollars on prediction markets being wagered on Elm Marina outcomes even as we have industry leaders openly bragging about gaming Elm Marina and you have thought leaders like Wor saying it can be easily gamed.
01:56It's past time for the Elm Marina people to sit down and think about whether they're doing more harm than good.
02:02Andre Karpathy had a similar observation when he noticed that the models that he thought were best were not lining up with what Elmarina was ranking.
02:11He said unfortunately the teams are not getting better models overall but better Elm Marina models whatever that is possibly something with a lot of nested list bullet points and emojis.
02:21So why does this happen that sort of industry insiders are telling us that this benchmark is not useful but it still gets a lot of play.
02:30The problem is that AI is aimed at everyone in the world is is something everyone in the world can use.
02:37And so everyone needs some tool to figure out which models are best.
02:41And benchmarks are what we have for that.
02:43But if you can't if you don't have the ability to assess if a benchmark is good, what you do have is the ability to assess what's popular.
02:50And this creates this avalanche, this feedback effect where the conversation is very much driven by incumbency and marketing and less by real world value.
02:59and even myself, right?
03:00Like unless I actually look at a benchmark in a fair amount of detail, I don't have an opinion on it.
03:05So, it's a very challenging problem.
03:08So, what are the things that benchmarks do that lead to these problems?
03:14There are a handful of key antiatterns that we're going to go through.
03:19The first is price.
03:20Let's say you want to make an agentic coding benchmark, which these days is a very popular thing to want to do, and you want a thousand tasks in your benchmark.
03:28Each task takes 60 hours to make.
03:30Each software engineer in your workforce costs half a million a year.
03:34That's $15 million to make your benchmark.
03:37And if you think that over time about a third of those tasks are going to get washed away every year due to models getting better, that's $5 million to replace them.
03:48So that puts you out of budget for most projects.
03:52So then people turn to a variety of workarounds that have their own problems.
03:56One of which is trying to use a lot of AI assistance which ultimately does not really work.
04:02Like you can't push the frontier forward from within the frontier.
04:05You need to inject that external human expertise and it needs to be good expertise.
04:12If you try to use cheap labor, you're going to get what you pay for and the whole result is not going to be that useful.
04:19At Surge, one of our differentiators has long been that we are not trying to minimize cost.
04:24We are trying to maximize quality and part of that means paying a lot of money for good workers.
04:31We've always believed that but especially in 2026 models are just beyond the point where you can make do with anything less than the best workers.
04:41Contamination is often thought of as when labs are explicitly training on the test set and that does happen sometimes but really contamination is the default outcome unless you are very very good.
04:53So labs put a lot of effort into holding back this flood of data that's going to contaminate their models.
05:00But inevitably if you have public questions and answers on the internet that's going to get memorized to some extent.
05:08So SweetBench verified here's an example prompt.
05:11You can give opus the first part of the prompt and it will verbatim spit out the rest.
05:16It does that with the answers as well.
05:20And we actually did an investigation where we compared looking at the repos that Sweepbench verified was built out of.
05:27How much has Opus memorized the Sweepbench verified contents versus the rest of the repo?
05:33And we found very clear evidence that Opus had memorized a lot of Sweetbench.
05:38In the most recent model card, Opus 4.8 talks about its SWE score.
05:41It does not disclose this contamination.
05:44We as an industry aren't really in the habit of doing those disclosures.
05:48And so what that means is that as benchmarking consumers, we're just missing that information.
05:54Reward hacking is also a big problem.
05:57Reward hacking is basically when a model finds a lazy and creative way to meet the letter of the law, but not the spirit.
06:04You need to think about designing your rewards as a adversarial process against this maximally lazy agent.
06:12Gradient descent is basically like water flowing downhill looking for the path of least resistance.
06:18And so your verifiers need to be robust to that.
06:23Another key challenge is simply just not having the ambition to make a sophisticated enough benchmark.
06:29Automation bench tests that agents are able to make tool calls in an enterprise environment.
06:35The problem is that a lot of the verifiers are these hard-coded string matches.
06:40And so you'll see it for things like phone numbers where there are many different acceptable phone number formats.
06:46But this verifier just picks one and the prompt doesn't tell you which one it is.
06:51So the result of this is that Haiku and Fable both score 20% on this task.
06:55Haiku scores 20% because it makes a bunch of mistakes and Fable scores 20% because it gets it right 80% of the time but then just happens to pick different formats.
07:05So if the benchmark task is not differentiating between Haiku and Fable, it's not a useful task.
07:11And more broadly, in 2026, many of us in this room are looking towards AI that's about to remake entire industries.
07:18And benchmarks are ideally our lighthouse on the horizon to let us know when that's coming.
07:24And a simple hard-coded string match is just not going to do it to measure that sort of impact.
07:32Another important aspect of a good benchmark is taste.
07:35Perhaps it used to be the case that benchmarks were these dry academic, you know, questions.
07:40and answer sets.
07:41But nowadays, a benchmark is an artifact expressing what it's an aspirational artifact.
07:48It's an expression of values of what you want your AI to do and how you want it to behave.
07:52And so you need to have some product sense in this process, some sort of a sense of what you want the AI to do.
07:58And that sense is unfortunately missing from ifal if has been cited on many model cards.
08:04And the way it was constructed was taking a bunch of arbitrary prompts that no user has ever asked in earnest and mashing them up with a bunch of other prompts to create a prompt set.
08:16The problem is that because no user actually has asked do not use any commas in your response or use the letter T at most once.
08:23You have to believe for this to be useful, you have to believe that there's a generalization from this to actual things that users are going to ask.
08:33If eval just happens also to have a bunch of prompts that are fully unsolvable due to having contradictory instructions.
08:40So this one starts by saying repeat this response verbatim and it ends by saying translate this into Hindi.
08:45Obviously you can't do both of those at once.
08:49Here's one that says write a riddle that includes exactly one bullet point.
08:53Make sure to include a few bullet points.
08:56Again this is just fully impossible.
08:59It uses a sentence splitter that does not align with how humans would actually split the sentences.
09:06And a lot of the prompts are not fully verified.
09:10So this one says write a story.
09:11There's nothing in the verifier that checks that a story was written.
09:15It just checks that the asky character I is not used more than once, which means that all of these responses get a full score, including response D. The way it gets a full score is by reward hacking and using the cerrillic eye character instead of the asy eye character.
09:30If is totally fine with that another challenge is operational ability.
09:38Making a big benchmark requires a lot of QC work and plenty of organizations just don't make that investment.
09:46Apex is a rag benchmark where the agent is given files and then asked questions about them.
09:52And in some instances, what's in the file and then what's expected in the rubric don't line up.
09:58So an agent that does the thing that it's seeing in the ground truth is going to get a negative score.
10:05And a lot of the data in Apex is seemingly synthetically generated because it's full of obvious placeholder values or dates or places that don't exist.
10:17And so as a result, the model is more likely to develop eval awareness where it realizes that it's being tested which undermines the entire exercise.
10:26It also just takes you out of distribution from actual real world data to something that is obviously fake.
10:34So that's an overview of some of the key antiatterns that happen during benchmark creation.
10:40But benchmaxing is a two-way process and there are all sorts of fun things that labs can do to benchmax and that's what we're going to talk about next.
10:51So the the core value that we're all trying to get towards as human eval right AI exists to serve humans and so just having humans look at the responses and make ratings like that's what we care about.
11:03The problem is that human eval is very expensive.
11:05And so a lot of what benchmarks are doing is trying to get around that and you are trying to distill human preference into something more scalable and you're hoping you do that distillation in a way that's still sufficiently faithful to what human eval wants.
11:18But what this means is that inevitably there is a point where you can keep hill climbing on a benchmark and the human eval stays flat.
11:27And you can actually take it even further if you want where you keep hill climbing on a benchmark even as the human eval goes down.
11:35But if for whatever reason you think this is necessary for marketing or we have sort of organizational politics or incentives that are demanding this that's how it can end up happening.
11:44In this instance the prompt is what time is it?
11:47And the response is absolutely deranged.
11:50No human eval is ever going to choose this but El Marina puts it at the top of the leaderboard.
11:55So again, you have this divergence and if you're trying to benchmax, you just cannot care about that.
12:02Another thing you can do that I've heard stories of is you can actually hire a crowdsource army to vote for you in Elmarina since Elmarina basically does no filtering of their workforce.
12:11And you might say, well, we anonym, you know, Elmarina anonymizes.
12:15So how are they going to know who to vote for?
12:17That's actually quite simple.
12:18You have your model include a watermark that tells the crowd who to vote for.
12:25There's also all sorts of things you can do with running your evals in conditions that are like not fully representative of the applesto apples comparison you're trying to make
12:34and then not always being super transparent about those conditions in such a way that undermines the validity that the community is trying to interpret because they don't have that contextualizing
12:45information.
12:47This was a paper um again about Elmarina and talking about how some of the dynamics of how it's run lead to models overfitting on Elmarina.
12:55Um in this instance, the specific chart we're seeing is that Meta tested 27 models without disclosing that it was doing so.
13:02Um which you know distorts the results.
13:09So how are we going to end benchmaxing?
13:12We need to hold the benchmark industry and the labs to a higher standard.
13:18The first thing we need to do when making a good benchmark is start with great human experts.
13:24And those experts inform everything that is downstream from what types of tasks are we going to have the agent do?
13:32How is success measured?
13:34What are the input files that agents are given?
13:36What are the tools that they're given?
13:38But we also do need that product sense.
13:39So imagine you're making a medical benchmark.
13:42It's not enough to have doctors who can answer specific medical questions because if you're trying to test how ready are we for agents to be deployed into hospitals.
13:50You also need someone with the business sense to know what's the regulatory environment, what's the legal requirements because that is going to impact what types of tasks you're trying to have the AI solve.
14:01You need high fidelity input data which is best done by going out and getting it from the real world, having actual people create this data.
14:10Synthetic approaches are possible, but it is very very hard to do it reliably.
14:14The tools need to actually work.
14:16A lot of benchmarks have tools that are buggy in various ways.
14:19And unless you're intentionally making a benchmark about buggy tools, this just introduces noise.
14:26You need verifiers that are fully aligned with the prompts.
14:28And this is a two-way alignment.
14:30So the verifiers need to be verifying everything the prompt asks for.
14:35And everything the prompt asks for needs to be covered by the verifiers.
14:39And if you get either side of those two misaligned, then it's going to be unfair to models and you're introducing random noise.
14:46You need to thoroughly QC everything and you need to have a private hold out set so you don't get contaminated.
14:54And if you do all this right, then you'll avoid what often happens with benchmarks, which is when labs get to like 80% and say, "Okay, this is saturated."
15:03And I used to think that saturation was just them saying again we don't think training on this further is going to increase real world value.
15:10And it often does mean that but it can mean that because the lab is saying we realize 20% of these tasks are broken.
15:18But the problem is that as you're hill climbing you don't know what 20% are broken until you solve all the others.
15:25And so as a result you have a lot of noise.
15:28And if that 20% of broken tasks is randomly but in a biased way assigning the rewards, it's going to really distort the model relative ranking you're trying to get.
15:41So at Serge, we created a benchmark called Hemingway bench to measure writing.
15:45There have been a number of writing benchmarks that use various mechanical means to try to assess writing quality, but we believe that writing is just too rich and deep and nuanced and frankly human of an activity to measure
15:59with mechanical benchmarks and LM as a judge doesn't really work either because LLMs don't have good taste in writing.
16:05Again, this is sort of the you can't expand the frontier from within the frontier situation.
16:10So what we've done is we've just created a workforce of thousands of professional writers in various domains, technical writers, poets, journalists, editors, and we just have them do blind model comparisons and then we create this leaderboard
16:26and it is quite expensive, right?
16:28Human eval is very expensive.
16:29Getting the time of these professionals is quite expensive.
16:32But again, our goal is to maximize quality, not to minimize costs.
16:37So in conclusion, benchmaxing is the exploitation of benchmark misalignments between human preference, but we can do better and we can hold the industry to a higher standard.
16:49Both the people making the benchmarks like myself and the people who are reporting on the benchmarks.
16:55And if you'd like to be a part of that, of course, obligatory pitch at Serge, we're hiring for basically all aspects of that.
17:00Uh and if you'd like more spicy takes from me, uh please follow my substack.
17:04Thank you very much.