Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

AI Engineer · 12 min · 132 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12Hello everyone.
2. 00:13Um this is the last talk of this session.
3. 00:16So hopefully it's going to be short.
4. 00:17I know that you guys had to go through a long day.
5. 00:20So try to keep it short and light for you all.
6. 00:23Um I'm going to present myself.
7. 00:25Um I'm Ali.
8. 00:26I'm the director of AI and ML at G2I.
9. 00:29Um I have zero experience in ML.
10. 00:31So I don't know why they put the ML in my title.
11. 00:34I'm a software engineer uh at heart.
12. 00:36And to prove that I have more than 50 abandoned side projects in my machine.
13. 00:40So uh you can know.
14. 00:42So uh I'm going to make a disclaimer.
15. 00:44The the title of the the presentation is a little bit misleading.
16. 00:49Uh as I was working on it, I realized that it would be better if I presented my journey uh into benchmarks and what I learned instead of trying to find
17. 00:58a dichotomy of the the bad, the ugly, and and the good.
18. 01:02So um let's start with um I want to grab your attention.
19. 01:06And I invite you to look at this.
20. 01:10These beautiful three screenshots are a single prompt on one of the benchmark tasks.
21. 01:16As I was looking at it, I was like how can an engineer write a task like this?
22. 01:20So I said, "Nah, it's impossible.
23. 01:23No one writes prompts like these ever."
24. 01:26But I wanted to double-check with my engineers.
25. 01:28So I took three of our best engineers.
26. 01:30I showed them the prompt and I said, "Would you ever write a prompt like this?"
27. 01:34And the answer was no. And they're right.
28. 01:40They shouldn't.
29. 01:41And so at that point I'm I was like what is a what are benchmarks anyway?
30. 01:48Uh I needed to take a step back.
31. 01:50I needed to look more.
32. 01:50I needed to understand.
33. 01:52And so as I was researching, I faced a wall of keywords.
34. 01:57Um graders, long horizon, verifiers, bench benchmarks, and and a lot of jargon.
35. 02:04So, I was like, either this is too complicated or um there's a lot of jargon and a lot of um words to to to work through here.
36. 02:15So, um I worked through it, worked with my team.
37. 02:20I have a lot of good researchers in the team, and we uh kind of like nailed like simplified it to the most basics.
38. 02:27Um and so, the way I see it is that it starts as a prompt or an instruction.
39. 02:34That prompt is fed to models and agents.
40. 02:40Agents provide solutions.
41. 02:42Those solutions are verified uh and graded through verifiers and rubrics.
42. 02:48All of that is wrapped in a harness that's that's preventing it from um from the external factors.
43. 02:55And if it all goes good, uh we have um trajectories, scores, and um metadata that we can use um to to to verif- to basically uh rank um models.
44. 03:13And so, the equation is simple.
45. 03:15If prompts and instructions are great and verifiers and rubrics are doing their job while the harness is preventing um or creating an environment that is good for a benchmark,
46. 03:28we should have amazing results.
47. 03:31Um but, that's not the reality.
48. 03:33So, what what what went wrong?
49. 03:37So, the first thing is when looking deeper in benchmarks, uh most of the instructions are unrealistic.
50. 03:45Um I did a quick research on SweetBench Pro, and um there's 481 words per instruction in average.
51. 03:55That's a two-pager per task.
52. 03:58That is not how people write prompts.
53. 04:01And to illustrate more of that, um I took a couple examples here.
54. 04:07The first one I looked at I I call the leaky prompt.
55. 04:09It's a go um task that's basically um that's trying to match in some rejects and doing test it's on on some rejects.
56. 04:18So, in the first screenshot here, um the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that.
57. 04:31The second one is is even worse.
58. 04:34Um it's basically providing a complete interface of the implementation.
59. 04:39Basically locking the LLM from any kind of uh creativity and it's forcing it to do it that way.
60. 04:46So, that's the leaky prompt.
61. 04:49The second example, it's the the not economically valuable prompt.
62. 04:54Uh this is from Sweet Marathon.
63. 04:56And this prompt is well-formed.
64. 04:59It's It's It's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compiler in Rust.
65. 05:08So, I don't know if any of you ever tried to do that, but I don't think it's a good idea.
66. 05:12We should not do that.
67. 05:16All right, moving on.
68. 05:18The second problem, weak verifiers.
69. 05:21Um so, the screenshot here is is a uh is the work that Deep Sweet um did uh to compare their uh their bench against Sweet Bench Pro.
70. 05:32And um let me just fix here so I can see the numbers.
71. 05:39In Sweet Bench Pro, 8.5 of 8.5% of all the tasks uh accepted wrong implementation in one hand and more than 20 24% of the tasks uh rejected um correct implementations.
72. 05:55And so, I kind of went again, dug a little bit, and I extracted one of the tasks, and I started looking at it.
73. 06:02Um and and here's here's what's happening in the example of uh re- rejecting um possibly rejecting good good answers.
74. 06:13So, in this example, the test is is basically expecting a variable to exist.
75. 06:19But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the variable name this way?
76. 06:28So, this test is cornering the LLM and basically uh causing uh those false negatives.
77. 06:36In the other example, it's base the test is basically checking functions that are unexported.
78. 06:44So, if that was a PR in any of our projects, and exposed these type of tests, we would not accept it.
79. 06:51So, this is what a weak verifier looks like.
80. 06:58All right, moving on.
81. 07:00Re- reward hacking.
82. 07:02So, what's happening is models are becoming increasingly increasingly able to optimize and figure out solutions to hard problems by going around the problem.
83. 07:12So, instead of actually trying to fix the to to apply a patch to a task, they try to go and find dot git folders, or they look up the internet for any kind of traces that would allow them to
84. 07:26um to do the task.
85. 07:29And this first graph here shows like shows that as models evolve, they are now more smarter and smarter in being able to do reward hacking, but that's what we want.
86. 07:43We want LLMs to be smart.
87. 07:45The benchmarks are lacking behind and they're not preventing from from that to happen.
88. 07:50Um More in detail, as you can see here, the more you go in time and the more you have new versions, the delta of um of um reward hacking is increasing.
89. 08:06So, the conclusion here is there's a quality gap and it's causing a trust gap.
90. 08:12I have not met an engineer in the last 6 months that would choose a model or choose um an LLM based on the leaderboards.
91. 08:21Um they look at them.
92. 08:23There's a lot of hype, but then they move on and they test things by themselves and they apply that.
93. 08:31So, how do we close the gap?
94. 08:34Um in the last 2 months, we've been working with our team at G2i to basically try to define a framework, uh a set of principles that would allow us to
95. 08:46build tasks for benchmarks that are um better than what we have today.
96. 08:54The first one, human instructions.
97. 08:57Authored by humans, reviewed by humans.
98. 09:00This is basically the entry point for any great tasks.
99. 09:03The instructions given to an agent or an LLM should lean towards expressing desired behaviors, objectives, and hard constraints, not implement details or try to guarantee self-containment when the task itself is is expressing too much uh
100. 09:20too much details.
101. 09:24The second principle is holistic graders.
102. 09:27Behavioral tests in one hand and then precision what were needed.
103. 09:31This is very similar to how we approach um tests in engineering.
104. 09:37We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed.
105. 09:44So, for security issues or business logic, we want to have the whole stack units unit test integration tests and then end-to-end tests.
106. 09:53But, for the rest of the the rest of the the software, we don't want to have 100% coverage because that's um not efficient.
107. 10:05The third principle, production grade.
108. 10:08The tasks have to be tasks have to have value um and they have to be economically valuable.
109. 10:15Um it is one thing to have a test a task that is failing the LLM proven that the LLM is not there yet.
110. 10:24It is another for it's another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that."
111. 10:33Currently, we don't have that.
112. 10:34So, production grade.
113. 10:39The fourth principle, contamination free by design.
114. 10:44We want to do novel tasks only and we want to make sure that we keep private holdout sets.
115. 10:51This is a principle that is very important as currently the tasks that are existing in benchmarks are all put from GitHub repos or from um from from public repos.
116. 11:06So, our approach here is that it should always be novel.
117. 11:10This way, it's contamination free by design.
118. 11:13And the fifth and last principle here is information about leaderboards.
119. 11:19The benchmark needs to tell a story and needs to help people make decisions.
120. 11:23Leaderboards are what we see in benchmarks today.
121. 11:26They tell you who wins, but they don't to you why.
122. 11:31And so, we want to basically put the x-axis back on um on on the first page.
123. 11:36Uh the idea here is that there's um there's a lot of um data that we can extract from those these runs, and unfortunately, they're not being put in the forefront.
124. 11:50And people have to dig uh a lot and do their own experiments to get to those data points.
125. 11:58And so, finally, uh initially, I wanted to have a kind of a a lofty like ending to this, but I think I I I pivoted to something more interesting.
126. 12:08Uh this is a call to action to software engineers.
127. 12:11Um benchmarks are not hard.
128. 12:15We need to look under the hood.
129. 12:17And we need to understand them and join the Discord because engineers' input is valuable.
130. 12:24And thank you.
131. 12:28[applause]
132. 12:45[music]