From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases

AI Engineer · 52 min · 707 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Before we get started, if you want to follow along, though this is a practical workshop, so I see a lot of notebooks out there, great.
00:19Um if you if you want to follow along exactly as I'm going, you should have a wand B account or weights and biases account.
00:26If you don't have one, you can register and get a free one.
00:30And we also give you inference credits, so you can do an evaluation that way, and sandbox access is also included.
00:37And um if you want to use something else like like open router, you can do that as well.
00:42Or even a different provider, whatever you want to use.
00:45I will just use this as the example in the workshop.
00:48And if you have an AI agent, you can just tell it change something and make it work with my setup.
00:55So, let's go.
00:58Quick about me.
00:59I'm Wolfram Reimann Oh.
01:00And my slogan is I have evaluate models for breakfast, build agents at night, and preach AI all day long.
01:07I do that as an AI evangelist at Corvus and weights and biases.
01:10And um I'm also a co-host on Thursday AI, the podcast.
01:15Alex is also here.
01:16He's a the main guy.
01:18And if you have technical issues during the workshop, you can contact him.
01:21He will help you out so we don't get interrupted.
01:24But he's your man.
01:26Thanks, Alex, for the help.
01:28And yeah, you can follow me on X or LinkedIn, and Wolf Benches wolfbench.ai.
01:33Easy to remember.
01:35Um but this workshop isn't just about Wolf Bench.
01:37It's why and how I built it, so you can stand up your own model and agent evil from zero to leaderboard.
01:44Um yeah, my backstory basically I started as a little child loved AI from the beginning.
01:50Then when ChatGPT happened, I decided I want to focus my career on this.
01:54Um if anyone knows the old good old times in local llama when the block was quantizing models, I was testing the models or I started that way.
02:03Was asked a lot about helping with inference and stuff.
02:07So, I became a consultant.
02:08And yeah, last year I went full in in a German AI startup and this year I joined Qorvo.
02:17And I've been on the podcast for a long time, right Alex?
02:22So, but this is about you.
02:24So, the workshop, who you are and why you are here.
02:27I think there are different kinds of people and I'm interested in which category you think you belong to.
02:33So, for instance, the agent builders.
02:35Are people building agents here that want to evaluate their agents?
02:40Someone here, someone here, a couple.
02:43And evil newcomers.
02:44Any people who want to do meaningful evaluations but have no idea what they are doing yet and want to learn this?
02:51Okay, those are more.
02:53And model evaluators.
02:54Who is interested in which model and agent should I trust?
02:57Which is the one I want to use?
02:58Anyone interested in that specifically?
03:00Over there and over there.
03:02Great.
03:03Platform and evil leads.
03:04The people who want to do comparable and reproducible evaluations.
03:08Someone here.
03:10A couple here.
03:10Great.
03:11Trace debuggers.
03:12Those are the people who are looking at the actual traces and trying to find out what went wrong and how to improve this.
03:18Over here.
03:19Great.
03:20And skeptics.
03:21Who does not trust the leaderboards?
03:24I mean No, every hand should go up, right?
03:26[laughter]
03:28But why is that and how can we change that?
03:30And that is part of what we are talking about and how you can build your own.
03:34But first, my my guiding principles are mindset is greater than skill set greater than tool set.
03:41So, the tool set is what we are using and that changes in AI changes almost every week, every month.
03:46Something new comes out and it changes all the time.
03:50So, skill set is how we are using tools that we have.
03:53That applies to different tools.
03:55So, if you have a great skill, uh, you know how to use the tools, you can switch to other tools.
04:00But, the mindset is the most important thing because that is the hardest to change or to improve.
04:05And for evaluation specifically, that is why we benchmark and what to watch out for.
04:09And skill set is just how to use the tools.
04:12And tool sets, I will show you a couple of tool sets and you can build your own that way like I did with Wolf Bench.
04:20And the premise is one score is not enough because performance is a distribution, not a point.
04:25The thing I disliked about the leaderboards and a lot of evaluations I have seen is you see some nice bars and you see some numbers and it's an average, an aggregate,
04:35and it doesn't really tell you very much.
04:38And that's why people always say or often say, "I don't trust this and I have a different experience."
04:44And there are things hidden behind the numbers that we can find out and look at.
04:49Because is it a really a good model?
04:52Did it have a lucky run?
04:53What are the hidden trade-offs?
04:54Stuff like that.
04:56And that's why I built Wolf Bench because I wanted to know which models can I use with which agents.
05:01The agent changed over time.
05:03In January, it was Open Claw.
05:04In February, it was Hermes Agent.
05:06And uh, I'm using a lot of Codex now.
05:08It was Cloud Code before.
05:09And I still want to know which models work the best with which settings.
05:13And yeah, like I said, most evaluation results collapse into just one score.
05:18And most leaderboards rank the models.
05:20You don't even see the agentic systems behind it.
05:24Or yeah, different versions, settings, even if you even see what uh, settings they have used for this.
05:31And they vary a lot, so you see some scores and somebody else reports the same benchmark, but they use different settings.
05:36So, can you really compare them?
05:39And is there a way to inspect the results to those people got?
05:42So, Wolf Bench is a framework.
05:44It has charts, dashboards, leaderboard.
05:46It's open source, by the way, and there's also methodology, and that is what I want to talk about most.
05:51So, you can use that methodology with your own tools.
05:55And for WolfBench, one thing that is that is applicable to all the benchmarks is you have to decide which metrics are interested to you.
06:03Do you want to look at an average?
06:05Do you want to look at the worst and best scores?
06:07Do you want to look at how many percent of the benchmarks it can solve at all?
06:12And how much does it consistently and also cost.
06:16That is also something that not everybody takes into account.
06:18How many tokens has it burned?
06:20How much did it cost?
06:22And it can happen that a cheaper and faster model takes longer and costs more than one of the bigger models because it just used so many tokens.
06:30Also very important are uniform conditions.
06:32So, you have the same setup every time because if you change just the variable, one of the many that are there, and we will look at a bunch of those, if you change it, it is basically a new benchmark now.
06:43And also multi-agent, if you want to do different agents, which ones do you look at?
06:48And always do multiple runs.
06:49A single run is benchmark lottery.
06:52It is a random result, and it is just a ballpark number.
06:55So, you have to do multiple runs.
06:57I do five with WolfBench, but you could do more if you have the money, of course.
07:02It's always a trade-off.
07:03Uh transparency is also super important to me because I publish all the results, and people can look at that.
07:09So, I don't just show a score and say this is why this is my number one model, number two model.
07:14I upload everything and you can inspect it and check it out, and have your agent check it out, which makes a lot of sense with all the data we've collected.
07:22So, basically get the thinking right, and then the tools become interchangeable.
07:27And the most important thing I I think that people need to realize is you're not ranking a model.
07:32If you're benchmarking a model, it's this model, but you are actually benchmarking a specific configuration.
07:38So, you use a data set, and if you change the data set, it's a new benchmark.
07:41You have the model, and behind the model you have the inference engine and whatever is happening on the endpoint that you often can't control.
07:48But if they do an update for the inference software, it can change the results again and you you have no way to actually see that.
07:54Unless it's your own inference on your own system.
07:56Agent orange.
07:57Also the agent orange, if I test blood code or code X, it is a different version if I test it next week and I do the update.
08:04So that is also important to keep in mind.
08:07And same thing with the sandbox.
08:10Even if you are using a sandbox provider and they change something in their back end, you may not even notice and it could have an effect and all the settings you choose yourself.
08:18So change any part and it's a different experiment.
08:21You have to keep that in mind.
08:23Um okay, here's a glossary.
08:25You will get the presentation, so I'm not going through all of this.
08:28Most of you are very familiar with this and I will explain whatever is needed at the moment.
08:34So we are looking at a specific tool set because it's a practical workshop and this is one way to build it, not the only way and will show you the different ways and you can pick your own stuff.
08:43So to follow along, you should have your laptop and your terminal, Python, UV and Git are recommended.
08:51We have files to download, so if you are not following, you can download the results and import them and use the tool set anyway.
08:57So you don't have to follow along everything, but if you want to, you can do it and use your own results that way.
09:02Um yeah, you should have the Wi-Fi and an API key is optional.
09:06I showed it in the beginning.
09:07I'll show it in a minute again.
09:09So you can get this.
09:11So here again, take a screenshot.
09:14Scan the QR code.
09:15This is where you can get you get to a form where you can get free inference credits for the duration of the AI engineer conference and to do that you need a free One B account.
09:25There you can sign up if you don't have already.
09:28If you don't already have one.
09:29Who has an account already?
09:30Does anyone here use One B?
09:34Bunch of people.
09:34Great.
09:36Okay, and also the sandboxes, I'm using the core weave serverless sandboxes in your offering where you have an isolated box for each task, which is super important because otherwise you would change your own system with every evaluation run and then it would be even more random.
09:53Okay, we are looking at six swappable layers and we'll choose each one and you can do the same.
09:57So this is the skill set how to do it and the mindset in the background and now we choose the different tools for this toolkit that we are building.
10:06So we need a benchmark, which is a data set.
10:08We need an evaluation harness to run the data set and the evaluation and yeah, to orchestrate all of this and to run it in sandboxes so it doesn't run on your system.
10:19And also we need an agent harness.
10:21What which agent do we want to use?
10:23And the model and service.
10:26That is of course usually the thing that people are most interested in.
10:29How does this model do?
10:30And the thing is you are not just using a model, you're always using an endpoint.
10:35It's always on a provider if it's open router, they are routing to other providers.
10:40They are a lot and if it's open AI or Google or XAI or whatever it is, they have their own setup and you are also evaluating that in a way.
10:49And all the settings you can choose.
10:52So the first thing is which benchmark do we use?
10:54You can start with the benchmark.
10:56I did it that way because I wanted to test agentic use with different harnesses so I decided I want a benchmark that does that.
11:02So I looked at all the benchmarks that have been used by the frontier labs and I saw terminal bench was on pretty much everyone.
11:09And it's an agentic benchmark.
11:11It's not a coding benchmark, not just it's a agentic benchmark where there are tasks like like fix the git repository, crack a password, repair a database, stuff you would ask your agent in normal use
11:25as an example.
11:26And that's why I picked terminal bench.
11:28But you could if you are interested in multilingual stuff, MMLU is one thing.
11:32or if you are interested in multimodal stuff, MMU Pro is something.
11:36If you are used interested in GUI agent use, then you would book pick a different benchmark.
11:42If you are interested in mathematics, coding, whatever it is, there's a benchmark for it and the thing is find the benchmark that is yeah, most interesting to you and that is comparable.
11:52You could also make your own, of course.
11:53You can make your own data set.
11:55And regarding Terminal Bench, that is a gentic, so that's why I chose it to benchmark agents.
12:01There are versions 2.0 and 2.1 and 3.0 is in the making right now.
12:06I chose 2.0 because 2.1 wasn't available when I started.
12:10If you are starting now, choose the latest version 2.1, but I'm using 2.0 because choosing a new version would be a new benchmark and I could not compare the scores anymore.
12:20It actually fixes only 28 of the 89 tests, but it would be a different benchmark for all the models.
12:26I would have to retest everything I already did.
12:29And I did a lot and never enough, but I did a lot, so I don't want to make a new version of Wolf Bench right now.
12:36I will do that at a later point, maybe with 3.0 or if a big new jump in capabilities happens and we need a different benchmark.
12:44Evaluation harness.
12:46This is you have chosen which benchmark you want to use, which data set to use.
12:50Now you need something to actually run the evaluation and Harbor is a default framework that is for Terminal Bench.
12:57There are others.
12:57Inspector is a great framework by the UK AI Security Institute, which has a lot of different benchmarks you can use.
13:05It's also very popular.
13:06The LM Evaluation harness, I used that years ago, so it's been around forever and OpenAI has their own eval stuff.
13:12You can build your own runner or one of the others.
13:15You pick what you want.
13:17Some people can also Some people start with picking their runner and then they pick one of the benchmarks it supports.
13:22That is also a valid way.
13:25And now for the first practical step, let's install Harbor.
13:28In this case, I'm using the current version.
13:31It was current yesterday.
13:32I'm not sure if it's current right now.
13:33It changes a lot, but I pin it to the to the version, and I highly recommend people stick to one version because anytime you change one of the variables,
13:43if it has an effect on the evaluation, and many have, then, yeah, you are changing already a variable, and the comparison comparability is suffering.
13:52Even if it has a bug, you are at least testing all the models with the same bug.
13:56But, it's a visual thinking, of course, because sometimes you have to update if a new version of the harness of the evaluation harness supports a different agent or provider, then you may be forced to update.
14:08What I recommend for the people who are really serious about this, fork the repository, maintain your own fork, and just backport whatever you need.
14:17And if you install it, this is There's a line break missing.
14:21It's UV tool install Harbor.
14:23So, it's installing in a UV environment, so it doesn't affect your system, and you can cleanly uninstall it again.
14:28Then, you should have this version.
14:31The next step, agents need a sandbox because if you just run it locally on your own system, then second agent running, it would find a system that has already been changed,
14:41and it would interfere with your system, so I would not recommend that.
14:44You can use Docker.
14:45That is very common, but it's very hard to parallelize that depending on what system you are using.
14:51So, you need a lot of resources.
14:52So, that's why there are cloud providers, and I have to give a big shout out to Daytona.
14:57They supported me in the beginning before Qualif had their own sandboxes, and I wanted to start with this.
15:02So, they really supported me a lot, and I have done a lot of the evaluations I have with Daytona.
15:08Now, we have our own offering, so I'm dogfooding it, and um that's why there are two options here that I recommend.
15:15The Qualif sandboxes are for those using the Kubernetes service, and the the golden one here is the Qualif serverless sandboxes.
15:22You just need your Vent B account.
15:24What about the Ah, okay.
15:29We also have a Um, I Can you uh I I turned off Slack, so it's not shared here.
15:37If anyone wants to get the PDF right away, just go to Alex.
15:41This is probably the easiest way right now.
15:44Um, okay.
15:46So, after you've installed this, uh Harbor is installed.
15:50Now, we need a sandbox client that is part of Harbor.
15:52And in this case, there's a part of a couple of things.
15:57Here's the QR code for the presentation.
16:07Yeah.
16:09Good.
16:09Um, UV tool install Harbor in square brackets won't be installs the provider to use the sandboxes, and you don't have to do anything.
16:17Once it's installed, you just have to have your API key.
16:21And you configure this in a YAML file, and here's just the next step.
16:26You don't have to copy this.
16:28There is a link and an easy URL you can just enter with the full configuration a couple of slides ahead.
16:35So, just so you This is just so you see it.
16:37But, if you are following along, you should just type the UV tool install command to install the Harbor won't be provider if you are using this.
16:46Let's continue.
16:48Next step, agent.
16:50This is the most fun part for me because there are so many agents, and I want to test how does this agent do with this model?
16:56And here's a list of some of the agents.
16:59These are all the agents, and there are even more that are supported by the Harbor framework.
17:04So, we have Cursor, we have Claude, we have Klein, we have Codex, Open Claude Hermes agent, um, and Terminus 2 is the default agent for the terminal bench and Harbor framework.
17:16And this is not a real agent.
17:18You can't use it for your coding stuff, or you shouldn't.
17:21It's more like simulating CLI environment to the actual model, and it sends keystrokes, and the agent executes them.
17:28It's one way to do this.
17:30So, in simple agent, which is a default if you are seeing Terminal Bench scores somewhere, and nobody said it's tested with a different agent, that is the agent they used.
17:41And then, which model do you want to test?
17:43You have to pick a model, and you have to pick a provider.
17:46If you are just using GPT 5.5, you will be using OpenAI.
17:49If you are using Claude, Fable, lucky if you are to use it, then you will be using Anthropic.
17:57Also, Google also provides some of the Anthropic models.
18:00And these are the just some of those.
18:03OpenRouter provides almost all of them, probably all of them, even with DDR enabled, so you are not sending any data to the model vendors.
18:13And we also have our own inference core, with serverless inference, where you also get selection of open-source models.
18:19And in this case, what I wanted to test is GLM 5.2, which was the latest model I tested, and also the best open-source model, open-weights model.
18:30So, I just picked this for the demonstration.
18:33You could choose any other, of course.
18:36And now the thing is the settings.
18:38There are a lot of settings.
18:39Some are very familiar to people, some maybe not as much.
18:42So, the context window size is also very important, because you always see this model has this context window size, but every provider has the Yeah, some providers choose to deliver a different size, a smaller size, for performance
18:55performance reasons.
18:57Like our own inference also doesn't offer the full 1 million of G GLM 5.2, only 256K, but that way we can serve it faster, and the quality of the model
19:12de- degrades after a certain amount of tokens.
19:14So, basically, half context is already very um very much noticeable.
19:20And the thing is it costs a lot if you always go to the million and then you compact.
19:24So, I personally think 256K, that's why I recommended it to our inference team to use this in that situation.
19:31Um I also use the same with Codex.
19:34Um basically, right now that gives you the best quality at the best price performance and speed.
19:40Thinking and reasoning efforts.
19:41This is a big factor and I have done different evaluations with different thinking modes and they really have a big effect on the model and sometimes a very surprising effect where if you raise the thinking,
19:53it goes down, but I will tell about this when we get to the slide.
19:56So, system prompt.
19:58You don't always have the ability to change the system prompt or even inspect it, but it is a very important factor.
20:05And if you think about an agent like Open Law Hermes agent, you have the sole MD where and put a lot of instructions.
20:11And now imagine you are using this with the benchmark.
20:14And if you give the instruction you are helpful assistant, it will be very different if you give it an instruction it is an autonomous agent doing stuff.
20:24I have seen results where the model or the benchmark delivered a bad result because the model tried to ask the user, but there was no user and it didn't know.
20:33So, if I change the system prompt and tell it you can't ask the user, it would behave differently and it would get a better score, I'm pretty sure.
20:40And that would be completely changing what I'm testing.
20:44So, there are so many variables here.
20:46Also, tool config.
20:47If you give the agent web access, I've seen where a model went out and looked up the solution for the benchmark because Terminal Bench is a public benchmark.
20:56And you have to look in the traces to see what happened here.
20:59It It's not really It's not even a bad thing if you want an agent, it should be smart.
21:04And if there's a solution for the problem, why should it try to think of it if it there is a published solution?
21:09So, it depends what you really want to measure and if you want to give it access or not.
21:14And not all of those do that.
21:16Or not every time.
21:18Sandbox resources, all the thing.
21:19If you give it less CPU or memory, it may run out or it be maybe slower, so it takes longer to do something or even crash and then you get a bad result.
21:28And it's not the model's fault, it's because of the sandbox configuration.
21:32Task timeout, every task gets a timeout.
21:34By default, they range from I think 5 minutes 15 minutes to over 3 hours.
21:40One is 3 hours something and another one is 2 hours or so.
21:44And that means you have to decide how much time you want to invest.
21:48When I started with Wolf Bench, I gave every task a fixed 2 hours because I didn't want to basically handicap open weights models which are often running on less powerful inference
22:00endpoints than the big SOTA frontier lab models.
22:05So, if you are running DeepSeek on a local system, it's usually not as fast as GPT on OpenAI systems.
22:11And that would skew the score if you don't give it unlimited time.
22:15But I noticed when I was going with 2 hours that if the model can't do the task, it sometimes just goes in circles and tries and tries and tries and fails and fails and fails.
22:23So, it was very expensive and took a long time because if I do 2 hours per task and there are 89 tasks and I parallelize them depending on how much concurrency I'm using, which is the next point,
22:34it takes so long.
22:35It takes so long and it costs so much and you don't get a result because it just can't do the task.
22:41So, I noticed 1 hour is a good mark.
22:44If you use the default terminal bench the test duration will rise.
22:49Concurrency, that is what I meant.
22:51You don't want to wait 98 89 hours or even twice the amount of time.
22:56You want to do concurrency, so you have multiple trials happening at the same time.
23:01But then you have to decide how high can I go?
23:04If If I go concurrency 89, I can do all the 89 tasks of terminal bench at once.
23:10But, if my inference endpoint is not fast enough and I get rate limited, then I will get bad scores or even no scores at the the end.
23:18The same with the sandboxes.
23:19If I have sandbox provider, I get resources and if I use 8 GB per sandbox, yeah, it sums up quickly and maybe you get over the limit you are allowed.
23:29Retry policy, how often to retry something.
23:31That is also something if I don't retry a task and it fails because a sandbox crashed, I get a bad score for the model.
23:38But, it wasn't the model's fault.
23:39So, if there are errors that you should retry and there are errors you should not retry.
23:43Like if the model causes a crash on its own by doing something stupid, that is actually a fail you want to measure.
23:51Temperature, the usual stuff.
23:53Um that is a difficult thing for a benchmark because which temperature do you use?
23:58Some model providers they give you the recommended temperature to use with their model.
24:02Some don't.
24:03And some agents have a default.
24:05Some agents decide based on the model.
24:08And there are so many combinations that you have to decide, do you want to look for every model what is the recommended one?
24:15If there is none, which do you use?
24:17How do you make sure that people who are doing the same benchmark are using the same setting you chose?
24:21It's difficult.
24:23And top P as well.
24:24Do you use 0.95 or do you use one or do you any value?
24:30It's just one setting and there are so many others.
24:32And max new tokens, how many tokens is the model allowed to generate?
24:36You have to decide on this because there are recommendations by the providers, there are limitations by the by the the hosters.
24:44You have to choose something and usually you have to configure it.
24:47And if you can, you can also set a seed so the randomness is reduced.
24:51We I when I started benchmark, I used temperature zero so there was less randomness.
24:55There was never no randomness.
24:57And I've seen things as small as a traffic driver update to change the actual results I got.
25:02Even if everything was uh deterministic or looked like that.
25:05Because we have floating point rounding errors, we have performance optimization, so many effects that change the result even if you use a temperature zero.
25:14And nowadays, when the model is recommending temperature one, then you should not set zero, of course.
25:20Otherwise, you would be going against the recommendation.
25:22In that case, you could set a seed, but even if you choose a seed like 42, it's very popular.
25:27You know why.
25:28And the thing is, some model may be more have more luck with that seed than others have less luck with that seed.
25:35It's all choices you have to make.
25:38Now, for this choice, which inference we want to use, I have on the left side the normal inference core with serverless inference.
25:46If you want to follow along and you want to use open router because you don't didn't go through the getting your own key here, you could use open router as well.
25:55In either case, you just use an end file and put in the variable at the bottom here.
26:00You just put the variable, the key in the end file, and then we will use that with the agent.
26:07Um if you use open router, you have to set some special settings to use our provider, the One API provider.
26:15This is very important if you are using open router and benchmarking a model.
26:18There are so many providers, and by default, you get one of these.
26:21They decide which is the best at the moment.
26:23So, if you are running a single benchmark run 89 tasks, you may get half the task with this provider and half the task with the other provider.
26:31So, you definitely have to set which one you want to use.
26:36And after all this, you have chosen one part for each of the six layers we went through.
26:42We have chosen Terminal Bench for our evaluation, we have chosen Harbor and a specific version of that, we have chosen our sandboxes, we have chosen Terminus 2 as the agent.
26:52GLM 5.2 and we have specific settings.
26:56If you are doing evaluation, you will do the same.
26:59Whatever framework you use, whatever model you use, you have to make choices and you should be very clear what you are using.
27:05This is important when you are publishing your results, so others know what you have actually used and measured.
27:12So, here is another QR code.
27:14This gives you the full configuration, so you don't have to copy stuff from the slides and here's a formatting that has been messed up a little.
27:21Excuse this.
27:22Um If you download this configuration, I will show you in the next slide how to run it and it will take about 5 minutes to actually get the results in this case.
27:32And I will also do the same.
27:34So, if you want to download the config, here's the QR code.
27:39And I will go to the next one.
27:41So, let me just not just show it, but actually run it.
27:44So, take a look at this.
27:45What we are doing, we are sourcing the end file to get the API key we find two slides back and then we are running it with the UV run harbor run config command
27:55and giving it the configuration you downloaded or created and then we can actually get the results.
28:02I will do this right now in the shell.
28:06So, the first thing, installing the harbor framework.
28:10Executing the command.
28:11I will zoom in a bit.
28:17So, if my Wi-Fi is working the way it should, it's full.
28:24Okay.
28:24We are letting this run.
28:26I will go back in a minute.
28:29So, continuing with the presentation.
28:33So, we are running the thing and it will do just one test.
28:38I didn't want to use 89 tests for this, otherwise we would be sitting an hour overtime.
28:43And that is only the time I give every test, so afterwards the verifier has to run and check if it's correct.
28:49That takes also some time, so it takes even longer.
28:52But, um Harbor pulls the tasks, the Terminal Bench 2.0 data set, it uses that from the configuration.
28:59So, this is the part where it has Terminal Bench version 2.0 in the config.
29:08Then, it starts the sandboxes using a provider we are using.
29:11So, it creates a sandbox per task, depending on the concurrency, how many it uses.
29:15Then, it runs the agent.
29:16It actually goes in the sandbox, installs the agent, and executes the task.
29:21Give it Give it a system user prompt, basically, saying here, "That is what you have to do."
29:26The agent does it.
29:27And afterwards, it collects the results from the sandbox before it is destroyed.
29:34And it gets the trajectories, really, what is the agent doing?
29:37You get a lot of information that way.
29:39And this small task, it's one of the smallest, it's just a Git repository where something has been lost, and it should be find it in the Git tree and restore it.
29:49This takes about 3 to 5 minutes.
29:51Let's see how far we are here.
29:53It's installed now.
29:55Then, we install our One B client.
29:59And the next step, in this case, I'm just getting my API key from the One Password.
30:11And then, I'm writing the configuration here.
30:14This is the one you can download.
30:15I'm just catting it into a file.
30:21And loading the end file.
30:23Now, the API key is in my environment variables, and I will just This is the run command.
30:29Run the evaluation.
30:30So, now it's starting to go out, starts the sandboxes, install the agent, run it.
30:37And this is all happening here.
30:41And while it's doing that, I will go back to the presentation.
30:48Next part.
30:50Run as evidence, not only a score.
30:52The formatting is totally messed up here.
30:53I will fix the presentation and we will upload it.
30:56This is just the directory of what is in the in the evaluation folder.
31:01We can create this live as soon as it's done.
31:04Of course, it will take about 5 minutes should it take.
31:07Right now, it's starting the environment, which is creating the sandboxes.
31:11So, just letting it run.
31:13Oh.
31:14This is too early.
31:18And when we have that, we get one task and it is scored.
31:23It either It gets a score of one or zero.
31:25Did it succeed or did it fail?
31:27If you do that for the 89 tasks, you get an amount how many tasks it did and then you do that over five runs, for instance, and then you get a percentage and you know how many percent of Terminal Bench did the model get.
31:39And then this is usually the score that the labs report.
31:42Most people just report the average across five runs.
31:47And this is the installation of the Wolf Bench Toolkit.
31:50What we have done before, it was all the Harbor framework and Terminal Bench and this is what I programmed, what I created.
31:56The Wolf Bench Toolkit is a way to work with this data because otherwise you now have a couple of files and you have some score, but how do you analyze it?
32:06So, this is a collector, what Wolf Bench does and I will It's best if I show it when I open the program.
32:13So, let me go back here.
32:16While this is happening, I will just start it in a new shell.
32:23And we just clone it and go in there.
32:26So, it's open source, Apache 2 license.
32:28You can use it for your own projects.
32:30Of course, you have to adapt it if you are using something else than Terminal Bench, but it's easy to adapt nowadays.
32:36Just tell your agent.
32:38Quick question.
32:39Who's using agents for stuff like that?
32:43Not everybody?
32:45Oh, I'm surprised.
32:46I still I still see a lot of notebooks, by the way.
32:49Because some people now just pull out their phone, take a screenshot, and say, "Okay, agent, go.
32:54Do whatever he is showing on the stage."
32:58Next year.
33:00Okay, so it's downloading and I'm also downloading all the results I did with all the runs I did so far.
33:07So, quick spoiler, I did uh 430 runs right now across um where is it?
33:16Uh it was 430 runs.
33:22But anyway, here are the agents I tested so far.
33:26Cloud code, Hermes agent, open clock cursor, Codex, Klein is not published yet.
33:30That is on my list and some other agents.
33:33If you are making an agent, contact me if you want to appear as well.
33:36And some models I tested, many on those W&B, that was on our own inference.
33:42So, the the final cost was unfortunately not as high as it is displayed, almost 40K here.
33:48So, this is some of this is on our own inference, but I just use the prices that have been published.
33:54So, let's see where we are here.
33:56It's downloaded.
33:58So, the 13 megabytes, those are the results.
34:01If you want to see all the trajectories, those have been uploaded to weave and you can just inspect them.
34:05You uh um don't have to um download gigabytes of data.
34:10Now, if you want to use the Wolf Bench, you can set some variables, so it is also an uploader, but I will show you if I just start it.
34:19And this uses a Remeo notebook to um provide a dashboard, an easy way to handle all the evaluation data.
34:28So, any moment now.
34:30There it is.
34:31So, this is a dashboard.
34:32You can enter your information where you want to upload the traces if you have the We stuff.
34:37Then you can discover VMs if you are using XE dev or Hetzner, it will find all your VMs on there and you can filter which VM names are evaluation VMs.
34:47Because I don't run the Harbor Framework on my own laptop usually.
34:50I run it in a VM so I can turn off my laptop or my Wi-Fi doesn't go off and the connection to the sandboxes fails because that could also affect something.
35:00Um in this case, everything is local as soon as I download the stuff which I did.
35:06So, I can just scan my local directory and it will find all the run data I already did.
35:10So, here are the 443 unique runs.
35:14And it shows you here a filter so I every run that has a score is is measured.
35:21It has to have all the tests.
35:22If some tests have not been run, it's not a valid res a run.
35:25It was just a test run.
35:27It shouldn't have too many errors or timeouts.
35:29And then I get a list of all of this stuff and I can choose which models I want to create um uh nice chart for.
35:37In this case, I will just use all of them.
35:39Then you can also just change the names of the models and sometimes you also have to change the thinking because the parameter you give can be changed by the agent.
35:48Like if you are using a thinking extra high or if you want to use a thinking max mode and your agent doesn't support it, maybe it is called a different way.
36:00And then you have to change it so it's really reflecting what has been used actually.
36:05Now, after all of this is done, I can just create my results file.
36:11You can upload to We view if you have the key.
36:13I do not have set the key so I right now don't have the button.
36:16Now, I create a save file and this is one JSON file that contains all the results.
36:21You can share that with other people.
36:23And in the end, I generate a chart which is a artifact I generate.
36:28So, now from all the results I did, I get a new chart.
36:32And this is the same that I then in the next step, I have an uploader as well.
36:36After the chart is generated, I just upload it to wolfbenchmark.ai and it's updated.
36:41If I activate this option to replace the index.html and it is online immediately.
36:49So, going back to the present Oh, let's just check how far the benchmark run.
36:54Oh, it's done.
36:55See, it's done.
36:55Trials one, mean score 1.0. So, it had one task, it solved the task.
37:01So, it worked.
37:04It finished and it fixed the Git repository.
37:07And this task is now in my directory and I could import it, but it's not a full run, so it is not displayed by default.
37:16And once we have done that and uploaded this, we can look at what Wolf benchmark provides.
37:21This is also the mindset thing again, what do we want to look at?
37:27And in this case, this is what we just did.
37:31I showed you the dashboard.
37:33And now we have to take a look at the leaderboard and what it means, because the leaderboard itself, whatever it generates, it is not just an answer, it helps you to find a better question.
37:45Like, what are you looking at?
37:47Do you care about the capability of a model or the reliability of a model?
37:51And I will show you what that means by going to wolfbenchmark.ai.
38:00And the thing is, I will just make it a little smaller.
38:05So, they are sorted from left to right and I have all of the information here.
38:08Most benchmarks you look at, they just show the terminal's two score and they only show the average.
38:13So, this is an interactive page, which is used not only to display results, but to create charts that you can then copy and paste and put somewhere else.
38:21There's a screenshot symbol so I can show stuff on X for instance.
38:26So if I want to show how did this model do, I can just mark it.
38:29I can remove some of the bars or reorder the bars.
38:32So it's interactive.
38:34So I can create my report that way.
38:37And what it also has it in the URL.
38:39So everybody can if you give the URL to this, they see the same thing you had on your screen.
38:47So I'm switching over here.
38:51And it looks a bit overwhelming sometimes because there are so many information so so many information on the same bar.
39:00Like I said, you can filter to different information to make it a simple bar, but I like to see this because the darker part at the bottom, that percentage
39:08is how many percent of the whole benchmark has a model solved in every run of the five runs runs I did.
39:14So that is what I call consistency.
39:16This model and this agent, they have solved the task 60% of the whole benchmark every time.
39:23And this means if that is a very high score, the model is very reliable because if it solves the task, you can expect it to solve the task all the time.
39:32While the yellow score is the average, the one score that usually is reported.
39:37And the up and down arrows around it, those are the best and the worst scores in one of the five runs.
39:43So if those are very close to each other, then there's not a high variance.
39:47So you can expect the same score.
39:49If they are very far apart like we see here from 79% to 60%, that means there's a lot of variance.
39:56And that is a sign that the model there's a lot of randomness involved here.
40:01You can't really trust it to execute the the task.
40:04Even if it has a high score, it's a bit random if it does it.
40:07And the highest mark is the star, that is the ceiling.
40:10How many percent of the benchmark has it solved at all, at least once.
40:15The amount of task that tells you how Yeah, in an ideal case, that would be the score the model should get all the time if it's perfect and always doing it.
40:24Although, that would also be an indicator that it is probably benchmarked for this.
40:28Um but it shows you how far it can go.
40:31So, what you want is a very very high ceiling, a very stable average, and a very high solid base.
40:39And if a model has the same score and they are almost the same, you can still see that the solid base is different.
40:46And then in that case, I would pick one that has a higher solid base.
40:50Like here, it's 73% to 74%.
40:53It's almost the same.
40:54If you just look at the average score, it's almost exactly the same thing.
40:58Could be a variance, just variance.
41:01But this is a bigger difference.
41:03It's 6% points, and that is a noticeable gap.
41:06So, you see the extra high thinking is better than the medium thinking, as we expected.
41:12But if we are looking at um This is also very interesting.
41:16I will show it in the presentation.
41:17Let me switch to the presentation.
41:20So, these were the different ways to imagine the model.
41:23Ah, risk and cost tolerance.
41:24I have to show you this because this is something If you look at the second best model on my evaluation, it was Gemini 3.5 Flash.
41:33So, this model, a flash model, is almost as good as GPT 5.5. So, what gives?
41:39If I just look at terminus, so we don't have so many bars, and I can also say I just want to compare these two.
41:45And yeah, it's a great model, Gemini 3.5 Flash.
41:47But if I look at how did it achieve that, I have a 3D view, which shows you how many tokens it used to achieve that.
41:54And it has been billions of tokens to get that score, while GPT 5.5 used much less.
42:02Let me zoom in.
42:06So, 136 million tokens for GPT 5.5, compared to 1.65 billion tokens for Gemini 3.5 Flash.
42:14And the thing is, if we also look at the price, which is another 3D view, this is the cost.
42:21And I can even show both at the same time.
42:24The cost is now a shadow.
42:25You see that the cost is over here.
42:27It's overlap because it used so many tokens, but this shows that Gemini 3.5 Flash is very close to GBD 5.5, but it used so many tokens that it cost
42:37even more than GBD 5.5 of course.
42:40So, it costs in this case $620 compared to $497.
42:45So, 500 to over 600.
42:48So, it's not that you Yeah, you have to think about this.
42:51Do you want to use a fast and cheap model?
42:54But you have to compare is it really faster and cheaper because all those tokens also take time.
42:59That is one thing.
43:02Going back to the presentation.
43:05And it's a decision surface.
43:07It's not just a chart.
43:08So, you load your snapshot and you can add your own if you did it and you execute the code, you would see them.
43:15And then you can filter like I showed you different metrics and models.
43:18You can reorder the bars and export your own.
43:21And then you can show this and make a social post about this or write an article, whatever you do.
43:28Um you can do this.
43:29In this case, I was looking at which is the best model on this benchmark.
43:33So, my number one right now is GBD 5.5 with Codex.
43:38It is even better than Fable because if you are asking what happened to Fable, that is also very interesting to take a look at.
43:45If I put Fable in here, it's had 13 refusals.
43:49So, it refused.
43:50And if you take a look at the cost, it goes so far.
43:54It was my most expensive benchmark.
43:56Don't tell anyone because it cost 11,000 just to test Fable and it failed because it refused so many tests.
44:04Whoops.
44:05And that is also something of course you have to decide.
44:09That is when people say oh I don't trust this benchmark this model is in the first place.
44:13Look at what happened here and the benchmark has to be transparent so you can look at the traces and see what has happened then it makes sense suddenly.
44:23Okay.
44:24Going back and yeah like I said the best model was Codex with GPT 5.5 but as you can see it was pretty very expensive.
44:33Let's zoom in.
44:34It cost almost a thousand bucks.
44:36While Cursor with the same model cost only 282.
44:42So that was very cheap compared to the other one and it used a little less tokens here.
44:48But yeah that's you could say okay this is the best combination but maybe the other one is from a price and performance perspective the better one.
44:59And Terminus also had the highest ceiling but if the scores are very close to each other that could be random.
45:05You have would have to do 10 runs or even more to be very sure but you have to make a choice sometime and pick one and go with it.
45:16This was also the thing that I just showed Gemini 3.5 Flash compared to GPT 5.5 lower score bigger bill so in that case it would make more sense to use the other model.
45:28And this was also super interesting and surprised me a lot when I tested Opus 4.6 at the time I raised the thinking to max and the score decreased.
45:39It really got worse and I did the same with Claude Code because I used Titanus as well and there I saw the same effect.
45:46When I do that with GPT if I increase from thinking off to extra high or from medium to extra high if I raise the thinking level the model gets smarter as you would expect.
45:54So why did it get worse with Opus?
45:57That was surprising to me and the difference if If look at the traces is that you see the model it thinks too much.
46:04It's overthinking.
46:05And that means I give it 1 hour.
46:07And if the model thinks for a minute or two and then decides I want to execute this command and the command doesn't exist on the machine yet, then it thinks another two or three minutes.
46:17And how do I install this?
46:18While an agent that is not even thinking, it's just doing stuff.
46:21So the first impulse is I run this command, get an error message.
46:25I run this command, get an error message.
46:26I install it, get an error message, fix it.
46:29And it gets more turns.
46:31So it can do more.
46:32That is the difference between thinking for an hour and not doing anything or just trying, trying, trying, trial and error.
46:39And the interesting stuff is that the GPT models, they are by default they appeared not as smart as Opus.
46:45Opus is very capable out of the box.
46:47Even with thinking off, it already got a great score.
46:50If you compare that here, it is already with off better than medium here on GPT 5.5. And even the solid base was higher.
47:00So it did more tasks in every run.
47:04Yeah.
47:05So that is stuff you have to look at.
47:07And if it's surprising, take a look at the traces to find out what is happening here.
47:11Model overthinking.
47:13I also saw the same with Fable, by the way, where the model was very smart and it seemed to know it was so smart.
47:19So what it did is it decided to go this direction.
47:22And even if the benchmark said it should be done that way, said, "No, I know better.
47:26I will do it my way."
47:27And then the verifier at the end, it ran, tested, and failed.
47:31So the model was too smart for its own good in some cases.
47:36And this is also an important feature I have to show you.
47:39The bar is a claim, the trace is evidence.
47:41I'm talking about traces all the time.
47:43So we have the bar, which is what we see on Wolf Bench.
47:47You have the nice bars.
47:48And the thing is you can click any of the bars, any of these.
47:53You just click it.
47:54And you get taken to our One page, where we have weave, which allows you to inspect all the traces.
48:01In this case, I just clicked one bar, and I'm seeing all the five runs that composed that score.
48:07And I can now pick any of the runs, go in here, and uh this I should remove.
48:16I'll show this later.
48:18Uh go away.
48:20So, we have 89 chords.
48:21That are the 89 tasks that I have been doing, and you can see all of the traces here.
48:27It's a lot, so it's loading a bit.
48:29And then you can go into any of these, and you see the task name, and you see the reward, did it succeed or not, and you see all the steps.
48:37You see the initial request, the user message that says, "You are an assistant.
48:42You are tasked to do with this."
48:43Then you see what the LLM did as a first command, what analysis it did, and what it wants to do, and then the command it sends, and so on.
48:53And you can really inspect all of this what has happened, and look at it.
48:57Or use your agent, and have this done.
48:59And we have a new feature, which is also an agent built into the platform.
49:03So, you have an agent inside of weave, and you can just tell it to do this.
49:07There's a talk about this by my colleague, Tim Sweeney is doing it on July 1. There you can check it out.
49:14I use agents a lot, because reading all the traces takes a long time.
49:18And if the agent gives you a link to weave, and you see that it's true what it says, that is also very good.
49:24So, to recap, there are so many variables relevant to benchmarks.
49:28You have the benchmark version, the OneBench 2.0, 2.1, the agent version, the model service and route, even updates to the infer- inference endpoint can change the results and the settings.
49:39Sandbox resources, like I said, if the if the agent runs out of resources, it will fail, and then you get a bad score for the model, although it wasn't the model's fault.
49:49Sampling, temperature, all things we discussed.
49:52Thinking defaults, like maximum not being higher if it thinks too long and a smaller, faster agent reaction could work better.
50:00Time out policy, retry policy.
50:02If you don't retry and there has been infrastructure error, you would get a bad score.
50:05So, after every run I go through all of the errors and mistakes and check if there have been inference issues.
50:11You can find that out if there are repeated characters or if prompt template um tokens leak into the actual response, then you know there's a tokenizer issue, an inference server issue.
50:23And especially with new models, that is a big thing.
50:25If you are benchmarking models as they come out, they are so new that the inference engine maybe has not the best support yet.
50:32So, I have done evaluations where the new model wasn't that great.
50:36A week later I did the same, everything was the same, and it was better because the inference engine has been improved.
50:42That is also part of this.
50:44So, use the message, bring your own everything.
50:47You have the building blocks, check out the presentation, think about what you want to do, which evaluation to use, which agent to use, which model to test.
50:54You have the mindset now to know what to look out for and apply this to make uh leaderboards and charts and evaluations that are more reliable than if you just run something and
51:05publish a score.
51:07So, run your own reproducible benchmark configurations.
51:10Share the results.
51:12If you want to work with Wolf Bench, contact me.
51:13If you want to run some models, that would be great.
51:16We could collaborate and publish the results so I don't have to run everything.
51:20And uh if you want to make an off-spin, that's also great.
51:25Use another framework, whatever you want.
51:28And if you are building models or agents, feel free to contact me so I can benchmark them.
51:32I work with some of the agent uh creators already and will publish results soon.
51:38Yeah, that's basically the end of the presentation.
51:41I hope you could get something out of it.
51:43And if you have some questions, now is the time.
51:45Otherwise, you can catch me at the booth and uh big shout out to Alex for the help.
51:50The podcast Thursday I is every Thursday and in the morning the time zone I'm from another time zone.
51:588:30 um Pacific time.
52:03Every week because it's so hard to keep track of what's happening and it's hard for us as well so we are forced to do it so we can report it to you.
52:10Okay, thank you.
52:27[music]