WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=jWq-aZIU0kM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/jWq-aZIU0kM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.480 --> 00:00:13.640
Hello everyone.

s2
00:00:13.640 --> 00:00:16.000
Um this is the last talk of this session.

s3
00:00:16.000 --> 00:00:17.640
So hopefully it's going to be short.

s4
00:00:17.640 --> 00:00:20.040
I know that you guys had to go through a long day.

s5
00:00:20.040 --> 00:00:23.760
So try to keep it short and light for you all.

s6
00:00:23.760 --> 00:00:25.440
Um I'm going to present myself.

s7
00:00:25.440 --> 00:00:26.560
Um I'm Ali.

s8
00:00:26.560 --> 00:00:29.360
I'm the director of AI and ML at G2I.

s9
00:00:29.360 --> 00:00:31.600
Um I have zero experience in ML.

s10
00:00:31.600 --> 00:00:34.160
So I don't know why they put the ML in my title.

s11
00:00:34.160 --> 00:00:36.640
I'm a software engineer uh at heart.

s12
00:00:36.640 --> 00:00:40.400
And to prove that I have more than 50 abandoned side projects in my machine.

s13
00:00:40.400 --> 00:00:42.880
So uh you can know.

s14
00:00:42.880 --> 00:00:44.840
So uh I'm going to make a disclaimer.

s15
00:00:44.840 --> 00:00:49.240
The the title of the the presentation is a little bit misleading.

s16
00:00:49.240 --> 00:00:58.520
Uh as I was working on it, I realized that it would be better if I presented my journey uh into benchmarks and what I learned instead of trying to find

s17
00:00:58.520 --> 00:01:02.040
a dichotomy of the the bad, the ugly, and and the good.

s18
00:01:02.040 --> 00:01:06.960
So um let's start with um I want to grab your attention.

s19
00:01:06.960 --> 00:01:10.040
And I invite you to look at this.

s20
00:01:10.160 --> 00:01:16.600
These beautiful three screenshots are a single prompt on one of the benchmark tasks.

s21
00:01:16.600 --> 00:01:20.760
As I was looking at it, I was like how can an engineer write a task like this?

s22
00:01:20.760 --> 00:01:23.520
So I said, "Nah, it's impossible.

s23
00:01:23.520 --> 00:01:26.000
No one writes prompts like these ever."

s24
00:01:26.000 --> 00:01:28.240
But I wanted to double-check with my engineers.

s25
00:01:28.240 --> 00:01:30.560
So I took three of our best engineers.

s26
00:01:30.560 --> 00:01:34.240
I showed them the prompt and I said, "Would you ever write a prompt like this?"

s27
00:01:34.240 --> 00:01:40.360
And the answer was no. And they're right.

s28
00:01:40.360 --> 00:01:41.880
They shouldn't.

s29
00:01:41.880 --> 00:01:48.720
And so at that point I'm I was like what is a what are benchmarks anyway?

s30
00:01:48.720 --> 00:01:50.000
Uh I needed to take a step back.

s31
00:01:50.000 --> 00:01:50.920
I needed to look more.

s32
00:01:50.920 --> 00:01:52.280
I needed to understand.

s33
00:01:52.280 --> 00:01:57.800
And so as I was researching, I faced a wall of keywords.

s34
00:01:57.800 --> 00:02:04.400
Um graders, long horizon, verifiers, bench benchmarks, and and a lot of jargon.

s35
00:02:04.400 --> 00:02:15.840
So, I was like, either this is too complicated or um there's a lot of jargon and a lot of um words to to to work through here.

s36
00:02:15.840 --> 00:02:20.040
So, um I worked through it, worked with my team.

s37
00:02:20.040 --> 00:02:27.240
I have a lot of good researchers in the team, and we uh kind of like nailed like simplified it to the most basics.

s38
00:02:27.240 --> 00:02:34.000
Um and so, the way I see it is that it starts as a prompt or an instruction.

s39
00:02:34.000 --> 00:02:39.120
That prompt is fed to models and agents.

s40
00:02:40.080 --> 00:02:42.920
Agents provide solutions.

s41
00:02:42.920 --> 00:02:48.320
Those solutions are verified uh and graded through verifiers and rubrics.

s42
00:02:48.320 --> 00:02:55.920
All of that is wrapped in a harness that's that's preventing it from um from the external factors.

s43
00:02:55.920 --> 00:03:13.120
And if it all goes good, uh we have um trajectories, scores, and um metadata that we can use um to to to verif- to basically uh rank um models.

s44
00:03:13.120 --> 00:03:15.680
And so, the equation is simple.

s45
00:03:15.680 --> 00:03:28.239
If prompts and instructions are great and verifiers and rubrics are doing their job while the harness is preventing um or creating an environment that is good for a benchmark,

s46
00:03:28.239 --> 00:03:31.400
we should have amazing results.

s47
00:03:31.400 --> 00:03:33.200
Um but, that's not the reality.

s48
00:03:33.200 --> 00:03:36.600
So, what what what went wrong?

s49
00:03:37.120 --> 00:03:45.000
So, the first thing is when looking deeper in benchmarks, uh most of the instructions are unrealistic.

s50
00:03:45.000 --> 00:03:55.760
Um I did a quick research on SweetBench Pro, and um there's 481 words per instruction in average.

s51
00:03:55.760 --> 00:03:58.840
That's a two-pager per task.

s52
00:03:58.840 --> 00:04:01.800
That is not how people write prompts.

s53
00:04:01.800 --> 00:04:07.280
And to illustrate more of that, um I took a couple examples here.

s54
00:04:07.280 --> 00:04:09.920
The first one I looked at I I call the leaky prompt.

s55
00:04:09.920 --> 00:04:18.640
It's a go um task that's basically um that's trying to match in some rejects and doing test it's on on some rejects.

s56
00:04:18.640 --> 00:04:31.680
So, in the first screenshot here, um the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that.

s57
00:04:31.680 --> 00:04:34.800
The second one is is even worse.

s58
00:04:34.800 --> 00:04:39.640
Um it's basically providing a complete interface of the implementation.

s59
00:04:39.640 --> 00:04:46.040
Basically locking the LLM from any kind of uh creativity and it's forcing it to do it that way.

s60
00:04:46.040 --> 00:04:49.040
So, that's the leaky prompt.

s61
00:04:49.040 --> 00:04:54.240
The second example, it's the the not economically valuable prompt.

s62
00:04:54.240 --> 00:04:56.840
Uh this is from Sweet Marathon.

s63
00:04:56.840 --> 00:04:59.800
And this prompt is well-formed.

s64
00:04:59.800 --> 00:05:08.600
It's It's It's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compiler in Rust.

s65
00:05:08.600 --> 00:05:12.640
So, I don't know if any of you ever tried to do that, but I don't think it's a good idea.

s66
00:05:12.640 --> 00:05:15.240
We should not do that.

s67
00:05:16.680 --> 00:05:18.240
All right, moving on.

s68
00:05:18.240 --> 00:05:21.320
The second problem, weak verifiers.

s69
00:05:21.320 --> 00:05:32.480
Um so, the screenshot here is is a uh is the work that Deep Sweet um did uh to compare their uh their bench against Sweet Bench Pro.

s70
00:05:32.480 --> 00:05:37.960
And um let me just fix here so I can see the numbers.

s71
00:05:39.800 --> 00:05:55.960
In Sweet Bench Pro, 8.5 of 8.5% of all the tasks uh accepted wrong implementation in one hand and more than 20 24% of the tasks uh rejected um correct implementations.

s72
00:05:55.960 --> 00:06:02.960
And so, I kind of went again, dug a little bit, and I extracted one of the tasks, and I started looking at it.

s73
00:06:02.960 --> 00:06:13.480
Um and and here's here's what's happening in the example of uh re- rejecting um possibly rejecting good good answers.

s74
00:06:13.480 --> 00:06:19.960
So, in this example, the test is is basically expecting a variable to exist.

s75
00:06:19.960 --> 00:06:28.640
But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the variable name this way?

s76
00:06:28.640 --> 00:06:36.760
So, this test is cornering the LLM and basically uh causing uh those false negatives.

s77
00:06:36.760 --> 00:06:44.600
In the other example, it's base the test is basically checking functions that are unexported.

s78
00:06:44.600 --> 00:06:51.560
So, if that was a PR in any of our projects, and exposed these type of tests, we would not accept it.

s79
00:06:51.560 --> 00:06:57.240
So, this is what a weak verifier looks like.

s80
00:06:58.200 --> 00:07:00.480
All right, moving on.

s81
00:07:00.480 --> 00:07:02.000
Re- reward hacking.

s82
00:07:02.000 --> 00:07:12.200
So, what's happening is models are becoming increasingly increasingly able to optimize and figure out solutions to hard problems by going around the problem.

s83
00:07:12.200 --> 00:07:26.680
So, instead of actually trying to fix the to to apply a patch to a task, they try to go and find dot git folders, or they look up the internet for any kind of traces that would allow them to

s84
00:07:26.680 --> 00:07:29.640
um to do the task.

s85
00:07:29.640 --> 00:07:43.320
And this first graph here shows like shows that as models evolve, they are now more smarter and smarter in being able to do reward hacking, but that's what we want.

s86
00:07:43.320 --> 00:07:45.120
We want LLMs to be smart.

s87
00:07:45.120 --> 00:07:50.240
The benchmarks are lacking behind and they're not preventing from from that to happen.

s88
00:07:50.240 --> 00:08:04.600
Um More in detail, as you can see here, the more you go in time and the more you have new versions, the delta of um of um reward hacking is increasing.

s89
00:08:06.040 --> 00:08:12.480
So, the conclusion here is there's a quality gap and it's causing a trust gap.

s90
00:08:12.480 --> 00:08:21.680
I have not met an engineer in the last 6 months that would choose a model or choose um an LLM based on the leaderboards.

s91
00:08:21.680 --> 00:08:23.720
Um they look at them.

s92
00:08:23.720 --> 00:08:30.320
There's a lot of hype, but then they move on and they test things by themselves and they apply that.

s93
00:08:31.919 --> 00:08:34.080
So, how do we close the gap?

s94
00:08:34.080 --> 00:08:46.400
Um in the last 2 months, we've been working with our team at G2i to basically try to define a framework, uh a set of principles that would allow us to

s95
00:08:46.400 --> 00:08:53.160
build tasks for benchmarks that are um better than what we have today.

s96
00:08:54.120 --> 00:08:57.880
The first one, human instructions.

s97
00:08:57.880 --> 00:09:00.720
Authored by humans, reviewed by humans.

s98
00:09:00.720 --> 00:09:03.640
This is basically the entry point for any great tasks.

s99
00:09:03.640 --> 00:09:20.640
The instructions given to an agent or an LLM should lean towards expressing desired behaviors, objectives, and hard constraints, not implement details or try to guarantee self-containment when the task itself is is expressing too much uh

s100
00:09:20.640 --> 00:09:23.120
too much details.

s101
00:09:24.120 --> 00:09:27.600
The second principle is holistic graders.

s102
00:09:27.600 --> 00:09:31.400
Behavioral tests in one hand and then precision what were needed.

s103
00:09:31.400 --> 00:09:37.040
This is very similar to how we approach um tests in engineering.

s104
00:09:37.240 --> 00:09:44.840
We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed.

s105
00:09:44.840 --> 00:09:53.520
So, for security issues or business logic, we want to have the whole stack units unit test integration tests and then end-to-end tests.

s106
00:09:53.520 --> 00:10:03.480
But, for the rest of the the rest of the the software, we don't want to have 100% coverage because that's um not efficient.

s107
00:10:05.120 --> 00:10:08.000
The third principle, production grade.

s108
00:10:08.000 --> 00:10:15.960
The tasks have to be tasks have to have value um and they have to be economically valuable.

s109
00:10:15.960 --> 00:10:24.520
Um it is one thing to have a test a task that is failing the LLM proven that the LLM is not there yet.

s110
00:10:24.520 --> 00:10:33.840
It is another for it's another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that."

s111
00:10:33.840 --> 00:10:34.880
Currently, we don't have that.

s112
00:10:34.880 --> 00:10:38.160
So, production grade.

s113
00:10:39.720 --> 00:10:44.320
The fourth principle, contamination free by design.

s114
00:10:44.560 --> 00:10:51.440
We want to do novel tasks only and we want to make sure that we keep private holdout sets.

s115
00:10:51.440 --> 00:11:06.560
This is a principle that is very important as currently the tasks that are existing in benchmarks are all put from GitHub repos or from um from from public repos.

s116
00:11:06.560 --> 00:11:10.000
So, our approach here is that it should always be novel.

s117
00:11:10.000 --> 00:11:13.600
This way, it's contamination free by design.

s118
00:11:13.600 --> 00:11:19.160
And the fifth and last principle here is information about leaderboards.

s119
00:11:19.160 --> 00:11:23.400
The benchmark needs to tell a story and needs to help people make decisions.

s120
00:11:23.400 --> 00:11:26.880
Leaderboards are what we see in benchmarks today.

s121
00:11:26.880 --> 00:11:31.320
They tell you who wins, but they don't to you why.

s122
00:11:31.320 --> 00:11:36.960
And so, we want to basically put the x-axis back on um on on the first page.

s123
00:11:36.960 --> 00:11:50.120
Uh the idea here is that there's um there's a lot of um data that we can extract from those these runs, and unfortunately, they're not being put in the forefront.

s124
00:11:50.120 --> 00:11:56.680
And people have to dig uh a lot and do their own experiments to get to those data points.

s125
00:11:58.400 --> 00:12:08.800
And so, finally, uh initially, I wanted to have a kind of a a lofty like ending to this, but I think I I I pivoted to something more interesting.

s126
00:12:08.800 --> 00:12:11.800
Uh this is a call to action to software engineers.

s127
00:12:11.800 --> 00:12:15.120
Um benchmarks are not hard.

s128
00:12:15.120 --> 00:12:17.200
We need to look under the hood.

s129
00:12:17.200 --> 00:12:23.240
And we need to understand them and join the Discord because engineers' input is valuable.

s130
00:12:24.480 --> 00:12:27.040
And thank you.

s131
00:12:28.673 --> 00:12:30.673
[applause]

s132
00:12:45.703 --> 00:12:47.703
[music]