WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=YIxzNZDDy1A

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/YIxzNZDDy1A.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.920 --> 00:00:19.440
Before we get started, if you want to follow along, though this is a practical workshop, so I see a lot of notebooks out there, great.

s2
00:00:19.440 --> 00:00:26.960
Um if you if you want to follow along exactly as I'm going, you should have a wand B account or weights and biases account.

s3
00:00:26.960 --> 00:00:30.480
If you don't have one, you can register and get a free one.

s4
00:00:30.480 --> 00:00:37.800
And we also give you inference credits, so you can do an evaluation that way, and sandbox access is also included.

s5
00:00:37.800 --> 00:00:42.440
And um if you want to use something else like like open router, you can do that as well.

s6
00:00:42.440 --> 00:00:45.440
Or even a different provider, whatever you want to use.

s7
00:00:45.440 --> 00:00:48.680
I will just use this as the example in the workshop.

s8
00:00:48.680 --> 00:00:55.000
And if you have an AI agent, you can just tell it change something and make it work with my setup.

s9
00:00:55.000 --> 00:00:58.080
So, let's go.

s10
00:00:58.080 --> 00:00:59.040
Quick about me.

s11
00:00:59.040 --> 00:01:00.560
I'm Wolfram Reimann Oh.

s12
00:01:00.560 --> 00:01:07.040
And my slogan is I have evaluate models for breakfast, build agents at night, and preach AI all day long.

s13
00:01:07.040 --> 00:01:10.920
I do that as an AI evangelist at Corvus and weights and biases.

s14
00:01:10.920 --> 00:01:15.080
And um I'm also a co-host on Thursday AI, the podcast.

s15
00:01:15.080 --> 00:01:16.000
Alex is also here.

s16
00:01:16.000 --> 00:01:18.200
He's a the main guy.

s17
00:01:18.200 --> 00:01:21.440
And if you have technical issues during the workshop, you can contact him.

s18
00:01:21.440 --> 00:01:24.880
He will help you out so we don't get interrupted.

s19
00:01:24.880 --> 00:01:26.800
But he's your man.

s20
00:01:26.800 --> 00:01:28.440
Thanks, Alex, for the help.

s21
00:01:28.440 --> 00:01:33.680
And yeah, you can follow me on X or LinkedIn, and Wolf Benches wolfbench.ai.

s22
00:01:33.680 --> 00:01:35.160
Easy to remember.

s23
00:01:35.160 --> 00:01:37.640
Um but this workshop isn't just about Wolf Bench.

s24
00:01:37.640 --> 00:01:44.960
It's why and how I built it, so you can stand up your own model and agent evil from zero to leaderboard.

s25
00:01:44.960 --> 00:01:50.440
Um yeah, my backstory basically I started as a little child loved AI from the beginning.

s26
00:01:50.440 --> 00:01:54.640
Then when ChatGPT happened, I decided I want to focus my career on this.

s27
00:01:54.640 --> 00:02:03.880
Um if anyone knows the old good old times in local llama when the block was quantizing models, I was testing the models or I started that way.

s28
00:02:03.880 --> 00:02:07.160
Was asked a lot about helping with inference and stuff.

s29
00:02:07.160 --> 00:02:08.920
So, I became a consultant.

s30
00:02:08.920 --> 00:02:17.400
And yeah, last year I went full in in a German AI startup and this year I joined Qorvo.

s31
00:02:17.400 --> 00:02:22.040
And I've been on the podcast for a long time, right Alex?

s32
00:02:22.080 --> 00:02:24.240
So, but this is about you.

s33
00:02:24.240 --> 00:02:27.200
So, the workshop, who you are and why you are here.

s34
00:02:27.200 --> 00:02:33.640
I think there are different kinds of people and I'm interested in which category you think you belong to.

s35
00:02:33.640 --> 00:02:35.280
So, for instance, the agent builders.

s36
00:02:35.280 --> 00:02:40.080
Are people building agents here that want to evaluate their agents?

s37
00:02:40.080 --> 00:02:43.120
Someone here, someone here, a couple.

s38
00:02:43.120 --> 00:02:44.959
And evil newcomers.

s39
00:02:44.959 --> 00:02:51.640
Any people who want to do meaningful evaluations but have no idea what they are doing yet and want to learn this?

s40
00:02:51.640 --> 00:02:53.000
Okay, those are more.

s41
00:02:53.000 --> 00:02:54.360
And model evaluators.

s42
00:02:54.360 --> 00:02:57.040
Who is interested in which model and agent should I trust?

s43
00:02:57.040 --> 00:02:58.280
Which is the one I want to use?

s44
00:02:58.280 --> 00:03:00.800
Anyone interested in that specifically?

s45
00:03:00.800 --> 00:03:02.200
Over there and over there.

s46
00:03:02.200 --> 00:03:03.080
Great.

s47
00:03:03.080 --> 00:03:04.480
Platform and evil leads.

s48
00:03:04.480 --> 00:03:08.840
The people who want to do comparable and reproducible evaluations.

s49
00:03:08.840 --> 00:03:10.040
Someone here.

s50
00:03:10.040 --> 00:03:10.800
A couple here.

s51
00:03:10.800 --> 00:03:11.600
Great.

s52
00:03:11.600 --> 00:03:12.519
Trace debuggers.

s53
00:03:12.519 --> 00:03:18.880
Those are the people who are looking at the actual traces and trying to find out what went wrong and how to improve this.

s54
00:03:18.880 --> 00:03:19.519
Over here.

s55
00:03:19.519 --> 00:03:20.360
Great.

s56
00:03:20.360 --> 00:03:21.920
And skeptics.

s57
00:03:21.920 --> 00:03:24.600
Who does not trust the leaderboards?

s58
00:03:24.600 --> 00:03:26.982
I mean No, every hand should go up, right?

s59
00:03:26.982 --> 00:03:28.000
[laughter]

s60
00:03:28.000 --> 00:03:30.400
But why is that and how can we change that?

s61
00:03:30.400 --> 00:03:34.760
And that is part of what we are talking about and how you can build your own.

s62
00:03:34.760 --> 00:03:41.080
But first, my my guiding principles are mindset is greater than skill set greater than tool set.

s63
00:03:41.080 --> 00:03:46.519
So, the tool set is what we are using and that changes in AI changes almost every week, every month.

s64
00:03:46.519 --> 00:03:50.440
Something new comes out and it changes all the time.

s65
00:03:50.440 --> 00:03:53.360
So, skill set is how we are using tools that we have.

s66
00:03:53.360 --> 00:03:55.320
That applies to different tools.

s67
00:03:55.320 --> 00:04:00.240
So, if you have a great skill, uh, you know how to use the tools, you can switch to other tools.

s68
00:04:00.240 --> 00:04:05.000
But, the mindset is the most important thing because that is the hardest to change or to improve.

s69
00:04:05.000 --> 00:04:09.840
And for evaluation specifically, that is why we benchmark and what to watch out for.

s70
00:04:09.840 --> 00:04:12.400
And skill set is just how to use the tools.

s71
00:04:12.400 --> 00:04:20.519
And tool sets, I will show you a couple of tool sets and you can build your own that way like I did with Wolf Bench.

s72
00:04:20.600 --> 00:04:25.760
And the premise is one score is not enough because performance is a distribution, not a point.

s73
00:04:25.760 --> 00:04:35.440
The thing I disliked about the leaderboards and a lot of evaluations I have seen is you see some nice bars and you see some numbers and it's an average, an aggregate,

s74
00:04:35.440 --> 00:04:38.080
and it doesn't really tell you very much.

s75
00:04:38.080 --> 00:04:44.120
And that's why people always say or often say, "I don't trust this and I have a different experience."

s76
00:04:44.120 --> 00:04:49.480
And there are things hidden behind the numbers that we can find out and look at.

s77
00:04:49.480 --> 00:04:52.120
Because is it a really a good model?

s78
00:04:52.120 --> 00:04:53.200
Did it have a lucky run?

s79
00:04:53.200 --> 00:04:54.600
What are the hidden trade-offs?

s80
00:04:54.600 --> 00:04:56.320
Stuff like that.

s81
00:04:56.320 --> 00:05:01.640
And that's why I built Wolf Bench because I wanted to know which models can I use with which agents.

s82
00:05:01.640 --> 00:05:03.080
The agent changed over time.

s83
00:05:03.080 --> 00:05:04.440
In January, it was Open Claw.

s84
00:05:04.440 --> 00:05:06.240
In February, it was Hermes Agent.

s85
00:05:06.240 --> 00:05:08.320
And uh, I'm using a lot of Codex now.

s86
00:05:08.320 --> 00:05:09.680
It was Cloud Code before.

s87
00:05:09.680 --> 00:05:13.600
And I still want to know which models work the best with which settings.

s88
00:05:13.600 --> 00:05:18.919
And yeah, like I said, most evaluation results collapse into just one score.

s89
00:05:18.919 --> 00:05:20.960
And most leaderboards rank the models.

s90
00:05:20.960 --> 00:05:24.240
You don't even see the agentic systems behind it.

s91
00:05:24.240 --> 00:05:31.280
Or yeah, different versions, settings, even if you even see what uh, settings they have used for this.

s92
00:05:31.280 --> 00:05:36.680
And they vary a lot, so you see some scores and somebody else reports the same benchmark, but they use different settings.

s93
00:05:36.680 --> 00:05:39.120
So, can you really compare them?

s94
00:05:39.120 --> 00:05:42.880
And is there a way to inspect the results to those people got?

s95
00:05:42.880 --> 00:05:44.040
So, Wolf Bench is a framework.

s96
00:05:44.040 --> 00:05:46.080
It has charts, dashboards, leaderboard.

s97
00:05:46.080 --> 00:05:51.840
It's open source, by the way, and there's also methodology, and that is what I want to talk about most.

s98
00:05:51.840 --> 00:05:55.160
So, you can use that methodology with your own tools.

s99
00:05:55.160 --> 00:06:03.400
And for WolfBench, one thing that is that is applicable to all the benchmarks is you have to decide which metrics are interested to you.

s100
00:06:03.400 --> 00:06:05.080
Do you want to look at an average?

s101
00:06:05.080 --> 00:06:07.640
Do you want to look at the worst and best scores?

s102
00:06:07.640 --> 00:06:12.760
Do you want to look at how many percent of the benchmarks it can solve at all?

s103
00:06:12.760 --> 00:06:16.400
And how much does it consistently and also cost.

s104
00:06:16.400 --> 00:06:18.960
That is also something that not everybody takes into account.

s105
00:06:18.960 --> 00:06:20.520
How many tokens has it burned?

s106
00:06:20.520 --> 00:06:22.040
How much did it cost?

s107
00:06:22.040 --> 00:06:30.680
And it can happen that a cheaper and faster model takes longer and costs more than one of the bigger models because it just used so many tokens.

s108
00:06:30.680 --> 00:06:32.560
Also very important are uniform conditions.

s109
00:06:32.560 --> 00:06:43.480
So, you have the same setup every time because if you change just the variable, one of the many that are there, and we will look at a bunch of those, if you change it, it is basically a new benchmark now.

s110
00:06:43.480 --> 00:06:48.400
And also multi-agent, if you want to do different agents, which ones do you look at?

s111
00:06:48.400 --> 00:06:49.840
And always do multiple runs.

s112
00:06:49.840 --> 00:06:52.200
A single run is benchmark lottery.

s113
00:06:52.200 --> 00:06:55.720
It is a random result, and it is just a ballpark number.

s114
00:06:55.720 --> 00:06:57.480
So, you have to do multiple runs.

s115
00:06:57.480 --> 00:07:02.160
I do five with WolfBench, but you could do more if you have the money, of course.

s116
00:07:02.160 --> 00:07:03.640
It's always a trade-off.

s117
00:07:03.640 --> 00:07:09.360
Uh transparency is also super important to me because I publish all the results, and people can look at that.

s118
00:07:09.360 --> 00:07:14.000
So, I don't just show a score and say this is why this is my number one model, number two model.

s119
00:07:14.000 --> 00:07:22.640
I upload everything and you can inspect it and check it out, and have your agent check it out, which makes a lot of sense with all the data we've collected.

s120
00:07:22.640 --> 00:07:27.320
So, basically get the thinking right, and then the tools become interchangeable.

s121
00:07:27.320 --> 00:07:32.520
And the most important thing I I think that people need to realize is you're not ranking a model.

s122
00:07:32.520 --> 00:07:38.200
If you're benchmarking a model, it's this model, but you are actually benchmarking a specific configuration.

s123
00:07:38.200 --> 00:07:41.800
So, you use a data set, and if you change the data set, it's a new benchmark.

s124
00:07:41.800 --> 00:07:48.280
You have the model, and behind the model you have the inference engine and whatever is happening on the endpoint that you often can't control.

s125
00:07:48.280 --> 00:07:54.200
But if they do an update for the inference software, it can change the results again and you you have no way to actually see that.

s126
00:07:54.200 --> 00:07:56.400
Unless it's your own inference on your own system.

s127
00:07:56.400 --> 00:07:57.440
Agent orange.

s128
00:07:57.440 --> 00:08:04.640
Also the agent orange, if I test blood code or code X, it is a different version if I test it next week and I do the update.

s129
00:08:04.640 --> 00:08:07.960
So that is also important to keep in mind.

s130
00:08:07.960 --> 00:08:10.040
And same thing with the sandbox.

s131
00:08:10.040 --> 00:08:18.760
Even if you are using a sandbox provider and they change something in their back end, you may not even notice and it could have an effect and all the settings you choose yourself.

s132
00:08:18.760 --> 00:08:21.000
So change any part and it's a different experiment.

s133
00:08:21.000 --> 00:08:23.400
You have to keep that in mind.

s134
00:08:23.400 --> 00:08:25.040
Um okay, here's a glossary.

s135
00:08:25.040 --> 00:08:28.240
You will get the presentation, so I'm not going through all of this.

s136
00:08:28.240 --> 00:08:34.200
Most of you are very familiar with this and I will explain whatever is needed at the moment.

s137
00:08:34.200 --> 00:08:43.200
So we are looking at a specific tool set because it's a practical workshop and this is one way to build it, not the only way and will show you the different ways and you can pick your own stuff.

s138
00:08:43.200 --> 00:08:51.480
So to follow along, you should have your laptop and your terminal, Python, UV and Git are recommended.

s139
00:08:51.480 --> 00:08:57.080
We have files to download, so if you are not following, you can download the results and import them and use the tool set anyway.

s140
00:08:57.080 --> 00:09:02.560
So you don't have to follow along everything, but if you want to, you can do it and use your own results that way.

s141
00:09:02.560 --> 00:09:06.280
Um yeah, you should have the Wi-Fi and an API key is optional.

s142
00:09:06.280 --> 00:09:07.160
I showed it in the beginning.

s143
00:09:07.160 --> 00:09:09.680
I'll show it in a minute again.

s144
00:09:09.680 --> 00:09:11.880
So you can get this.

s145
00:09:11.880 --> 00:09:14.040
So here again, take a screenshot.

s146
00:09:14.040 --> 00:09:15.400
Scan the QR code.

s147
00:09:15.400 --> 00:09:25.920
This is where you can get you get to a form where you can get free inference credits for the duration of the AI engineer conference and to do that you need a free One B account.

s148
00:09:25.920 --> 00:09:28.200
There you can sign up if you don't have already.

s149
00:09:28.200 --> 00:09:29.520
If you don't already have one.

s150
00:09:29.520 --> 00:09:30.960
Who has an account already?

s151
00:09:30.960 --> 00:09:34.040
Does anyone here use One B?

s152
00:09:34.040 --> 00:09:34.680
Bunch of people.

s153
00:09:34.680 --> 00:09:36.280
Great.

s154
00:09:36.280 --> 00:09:53.280
Okay, and also the sandboxes, I'm using the core weave serverless sandboxes in your offering where you have an isolated box for each task, which is super important because otherwise you would change your own system with every evaluation run and then it would be even more random.

s155
00:09:53.280 --> 00:09:57.560
Okay, we are looking at six swappable layers and we'll choose each one and you can do the same.

s156
00:09:57.560 --> 00:10:06.040
So this is the skill set how to do it and the mindset in the background and now we choose the different tools for this toolkit that we are building.

s157
00:10:06.040 --> 00:10:08.800
So we need a benchmark, which is a data set.

s158
00:10:08.800 --> 00:10:19.160
We need an evaluation harness to run the data set and the evaluation and yeah, to orchestrate all of this and to run it in sandboxes so it doesn't run on your system.

s159
00:10:19.160 --> 00:10:21.520
And also we need an agent harness.

s160
00:10:21.520 --> 00:10:23.760
What which agent do we want to use?

s161
00:10:23.760 --> 00:10:26.360
And the model and service.

s162
00:10:26.360 --> 00:10:29.320
That is of course usually the thing that people are most interested in.

s163
00:10:29.320 --> 00:10:30.920
How does this model do?

s164
00:10:30.920 --> 00:10:35.560
And the thing is you are not just using a model, you're always using an endpoint.

s165
00:10:35.560 --> 00:10:40.240
It's always on a provider if it's open router, they are routing to other providers.

s166
00:10:40.240 --> 00:10:49.640
They are a lot and if it's open AI or Google or XAI or whatever it is, they have their own setup and you are also evaluating that in a way.

s167
00:10:49.640 --> 00:10:52.120
And all the settings you can choose.

s168
00:10:52.120 --> 00:10:54.680
So the first thing is which benchmark do we use?

s169
00:10:54.680 --> 00:10:56.000
You can start with the benchmark.

s170
00:10:56.000 --> 00:11:02.280
I did it that way because I wanted to test agentic use with different harnesses so I decided I want a benchmark that does that.

s171
00:11:02.280 --> 00:11:09.880
So I looked at all the benchmarks that have been used by the frontier labs and I saw terminal bench was on pretty much everyone.

s172
00:11:09.880 --> 00:11:11.040
And it's an agentic benchmark.

s173
00:11:11.040 --> 00:11:25.080
It's not a coding benchmark, not just it's a agentic benchmark where there are tasks like like fix the git repository, crack a password, repair a database, stuff you would ask your agent in normal use

s174
00:11:25.080 --> 00:11:26.400
as an example.

s175
00:11:26.400 --> 00:11:28.600
And that's why I picked terminal bench.

s176
00:11:28.600 --> 00:11:32.680
But you could if you are interested in multilingual stuff, MMLU is one thing.

s177
00:11:32.680 --> 00:11:36.600
or if you are interested in multimodal stuff, MMU Pro is something.

s178
00:11:36.600 --> 00:11:42.000
If you are used interested in GUI agent use, then you would book pick a different benchmark.

s179
00:11:42.000 --> 00:11:52.080
If you are interested in mathematics, coding, whatever it is, there's a benchmark for it and the thing is find the benchmark that is yeah, most interesting to you and that is comparable.

s180
00:11:52.080 --> 00:11:53.640
You could also make your own, of course.

s181
00:11:53.640 --> 00:11:55.640
You can make your own data set.

s182
00:11:55.640 --> 00:12:01.320
And regarding Terminal Bench, that is a gentic, so that's why I chose it to benchmark agents.

s183
00:12:01.320 --> 00:12:06.160
There are versions 2.0 and 2.1 and 3.0 is in the making right now.

s184
00:12:06.160 --> 00:12:10.280
I chose 2.0 because 2.1 wasn't available when I started.

s185
00:12:10.280 --> 00:12:20.480
If you are starting now, choose the latest version 2.1, but I'm using 2.0 because choosing a new version would be a new benchmark and I could not compare the scores anymore.

s186
00:12:20.480 --> 00:12:26.880
It actually fixes only 28 of the 89 tests, but it would be a different benchmark for all the models.

s187
00:12:26.880 --> 00:12:29.440
I would have to retest everything I already did.

s188
00:12:29.440 --> 00:12:36.440
And I did a lot and never enough, but I did a lot, so I don't want to make a new version of Wolf Bench right now.

s189
00:12:36.440 --> 00:12:44.680
I will do that at a later point, maybe with 3.0 or if a big new jump in capabilities happens and we need a different benchmark.

s190
00:12:44.680 --> 00:12:46.240
Evaluation harness.

s191
00:12:46.240 --> 00:12:50.960
This is you have chosen which benchmark you want to use, which data set to use.

s192
00:12:50.960 --> 00:12:57.240
Now you need something to actually run the evaluation and Harbor is a default framework that is for Terminal Bench.

s193
00:12:57.240 --> 00:12:57.800
There are others.

s194
00:12:57.800 --> 00:13:05.040
Inspector is a great framework by the UK AI Security Institute, which has a lot of different benchmarks you can use.

s195
00:13:05.040 --> 00:13:06.640
It's also very popular.

s196
00:13:06.640 --> 00:13:12.480
The LM Evaluation harness, I used that years ago, so it's been around forever and OpenAI has their own eval stuff.

s197
00:13:12.480 --> 00:13:15.760
You can build your own runner or one of the others.

s198
00:13:15.760 --> 00:13:17.000
You pick what you want.

s199
00:13:17.000 --> 00:13:22.680
Some people can also Some people start with picking their runner and then they pick one of the benchmarks it supports.

s200
00:13:22.680 --> 00:13:25.680
That is also a valid way.

s201
00:13:25.680 --> 00:13:28.840
And now for the first practical step, let's install Harbor.

s202
00:13:28.840 --> 00:13:31.400
In this case, I'm using the current version.

s203
00:13:31.400 --> 00:13:32.400
It was current yesterday.

s204
00:13:32.400 --> 00:13:33.760
I'm not sure if it's current right now.

s205
00:13:33.760 --> 00:13:43.760
It changes a lot, but I pin it to the to the version, and I highly recommend people stick to one version because anytime you change one of the variables,

s206
00:13:43.760 --> 00:13:52.440
if it has an effect on the evaluation, and many have, then, yeah, you are changing already a variable, and the comparison comparability is suffering.

s207
00:13:52.440 --> 00:13:56.720
Even if it has a bug, you are at least testing all the models with the same bug.

s208
00:13:56.720 --> 00:14:08.080
But, it's a visual thinking, of course, because sometimes you have to update if a new version of the harness of the evaluation harness supports a different agent or provider, then you may be forced to update.

s209
00:14:08.080 --> 00:14:17.360
What I recommend for the people who are really serious about this, fork the repository, maintain your own fork, and just backport whatever you need.

s210
00:14:17.360 --> 00:14:21.760
And if you install it, this is There's a line break missing.

s211
00:14:21.760 --> 00:14:23.160
It's UV tool install Harbor.

s212
00:14:23.160 --> 00:14:28.280
So, it's installing in a UV environment, so it doesn't affect your system, and you can cleanly uninstall it again.

s213
00:14:28.280 --> 00:14:31.040
Then, you should have this version.

s214
00:14:31.040 --> 00:14:41.600
The next step, agents need a sandbox because if you just run it locally on your own system, then second agent running, it would find a system that has already been changed,

s215
00:14:41.600 --> 00:14:44.320
and it would interfere with your system, so I would not recommend that.

s216
00:14:44.320 --> 00:14:45.400
You can use Docker.

s217
00:14:45.400 --> 00:14:51.480
That is very common, but it's very hard to parallelize that depending on what system you are using.

s218
00:14:51.480 --> 00:14:52.800
So, you need a lot of resources.

s219
00:14:52.800 --> 00:14:57.760
So, that's why there are cloud providers, and I have to give a big shout out to Daytona.

s220
00:14:57.760 --> 00:15:02.560
They supported me in the beginning before Qualif had their own sandboxes, and I wanted to start with this.

s221
00:15:02.560 --> 00:15:08.960
So, they really supported me a lot, and I have done a lot of the evaluations I have with Daytona.

s222
00:15:08.960 --> 00:15:15.800
Now, we have our own offering, so I'm dogfooding it, and um that's why there are two options here that I recommend.

s223
00:15:15.800 --> 00:15:22.640
The Qualif sandboxes are for those using the Kubernetes service, and the the golden one here is the Qualif serverless sandboxes.

s224
00:15:22.640 --> 00:15:24.480
You just need your Vent B account.

s225
00:15:24.480 --> 00:15:29.640
What about the Ah, okay.

s226
00:15:29.640 --> 00:15:37.840
We also have a Um, I Can you uh I I turned off Slack, so it's not shared here.

s227
00:15:37.840 --> 00:15:41.720
If anyone wants to get the PDF right away, just go to Alex.

s228
00:15:41.720 --> 00:15:44.800
This is probably the easiest way right now.

s229
00:15:44.800 --> 00:15:46.520
Um, okay.

s230
00:15:46.520 --> 00:15:50.040
So, after you've installed this, uh Harbor is installed.

s231
00:15:50.040 --> 00:15:52.800
Now, we need a sandbox client that is part of Harbor.

s232
00:15:52.800 --> 00:15:57.360
And in this case, there's a part of a couple of things.

s233
00:15:57.360 --> 00:16:01.280
Here's the QR code for the presentation.

s234
00:16:07.960 --> 00:16:09.000
Yeah.

s235
00:16:09.000 --> 00:16:09.440
Good.

s236
00:16:09.440 --> 00:16:17.960
Um, UV tool install Harbor in square brackets won't be installs the provider to use the sandboxes, and you don't have to do anything.

s237
00:16:17.960 --> 00:16:21.360
Once it's installed, you just have to have your API key.

s238
00:16:21.360 --> 00:16:26.680
And you configure this in a YAML file, and here's just the next step.

s239
00:16:26.680 --> 00:16:28.200
You don't have to copy this.

s240
00:16:28.200 --> 00:16:35.360
There is a link and an easy URL you can just enter with the full configuration a couple of slides ahead.

s241
00:16:35.360 --> 00:16:37.920
So, just so you This is just so you see it.

s242
00:16:37.920 --> 00:16:46.760
But, if you are following along, you should just type the UV tool install command to install the Harbor won't be provider if you are using this.

s243
00:16:46.960 --> 00:16:48.760
Let's continue.

s244
00:16:48.760 --> 00:16:50.160
Next step, agent.

s245
00:16:50.160 --> 00:16:56.560
This is the most fun part for me because there are so many agents, and I want to test how does this agent do with this model?

s246
00:16:56.560 --> 00:16:59.839
And here's a list of some of the agents.

s247
00:16:59.839 --> 00:17:04.040
These are all the agents, and there are even more that are supported by the Harbor framework.

s248
00:17:04.040 --> 00:17:16.839
So, we have Cursor, we have Claude, we have Klein, we have Codex, Open Claude Hermes agent, um, and Terminus 2 is the default agent for the terminal bench and Harbor framework.

s249
00:17:16.839 --> 00:17:18.680
And this is not a real agent.

s250
00:17:18.680 --> 00:17:21.319
You can't use it for your coding stuff, or you shouldn't.

s251
00:17:21.319 --> 00:17:28.680
It's more like simulating CLI environment to the actual model, and it sends keystrokes, and the agent executes them.

s252
00:17:28.680 --> 00:17:30.320
It's one way to do this.

s253
00:17:30.320 --> 00:17:41.200
So, in simple agent, which is a default if you are seeing Terminal Bench scores somewhere, and nobody said it's tested with a different agent, that is the agent they used.

s254
00:17:41.200 --> 00:17:43.320
And then, which model do you want to test?

s255
00:17:43.320 --> 00:17:46.120
You have to pick a model, and you have to pick a provider.

s256
00:17:46.120 --> 00:17:49.920
If you are just using GPT 5.5, you will be using OpenAI.

s257
00:17:49.920 --> 00:17:57.480
If you are using Claude, Fable, lucky if you are to use it, then you will be using Anthropic.

s258
00:17:57.480 --> 00:18:00.760
Also, Google also provides some of the Anthropic models.

s259
00:18:00.760 --> 00:18:03.520
And these are the just some of those.

s260
00:18:03.520 --> 00:18:13.880
OpenRouter provides almost all of them, probably all of them, even with DDR enabled, so you are not sending any data to the model vendors.

s261
00:18:13.880 --> 00:18:19.560
And we also have our own inference core, with serverless inference, where you also get selection of open-source models.

s262
00:18:19.560 --> 00:18:30.680
And in this case, what I wanted to test is GLM 5.2, which was the latest model I tested, and also the best open-source model, open-weights model.

s263
00:18:30.680 --> 00:18:33.080
So, I just picked this for the demonstration.

s264
00:18:33.080 --> 00:18:36.320
You could choose any other, of course.

s265
00:18:36.560 --> 00:18:38.120
And now the thing is the settings.

s266
00:18:38.120 --> 00:18:39.280
There are a lot of settings.

s267
00:18:39.280 --> 00:18:42.160
Some are very familiar to people, some maybe not as much.

s268
00:18:42.160 --> 00:18:55.880
So, the context window size is also very important, because you always see this model has this context window size, but every provider has the Yeah, some providers choose to deliver a different size, a smaller size, for performance

s269
00:18:55.880 --> 00:18:57.080
performance reasons.

s270
00:18:57.080 --> 00:19:12.120
Like our own inference also doesn't offer the full 1 million of G GLM 5.2, only 256K, but that way we can serve it faster, and the quality of the model

s271
00:19:12.120 --> 00:19:14.800
de- degrades after a certain amount of tokens.

s272
00:19:14.800 --> 00:19:20.400
So, basically, half context is already very um very much noticeable.

s273
00:19:20.400 --> 00:19:24.160
And the thing is it costs a lot if you always go to the million and then you compact.

s274
00:19:24.160 --> 00:19:31.840
So, I personally think 256K, that's why I recommended it to our inference team to use this in that situation.

s275
00:19:31.840 --> 00:19:34.120
Um I also use the same with Codex.

s276
00:19:34.120 --> 00:19:40.680
Um basically, right now that gives you the best quality at the best price performance and speed.

s277
00:19:40.680 --> 00:19:41.800
Thinking and reasoning efforts.

s278
00:19:41.800 --> 00:19:53.680
This is a big factor and I have done different evaluations with different thinking modes and they really have a big effect on the model and sometimes a very surprising effect where if you raise the thinking,

s279
00:19:53.680 --> 00:19:56.720
it goes down, but I will tell about this when we get to the slide.

s280
00:19:56.720 --> 00:19:58.320
So, system prompt.

s281
00:19:58.320 --> 00:20:05.240
You don't always have the ability to change the system prompt or even inspect it, but it is a very important factor.

s282
00:20:05.240 --> 00:20:11.480
And if you think about an agent like Open Law Hermes agent, you have the sole MD where and put a lot of instructions.

s283
00:20:11.480 --> 00:20:14.400
And now imagine you are using this with the benchmark.

s284
00:20:14.400 --> 00:20:24.760
And if you give the instruction you are helpful assistant, it will be very different if you give it an instruction it is an autonomous agent doing stuff.

s285
00:20:24.760 --> 00:20:33.960
I have seen results where the model or the benchmark delivered a bad result because the model tried to ask the user, but there was no user and it didn't know.

s286
00:20:33.960 --> 00:20:40.240
So, if I change the system prompt and tell it you can't ask the user, it would behave differently and it would get a better score, I'm pretty sure.

s287
00:20:40.240 --> 00:20:44.280
And that would be completely changing what I'm testing.

s288
00:20:44.280 --> 00:20:46.640
So, there are so many variables here.

s289
00:20:46.640 --> 00:20:47.560
Also, tool config.

s290
00:20:47.560 --> 00:20:56.440
If you give the agent web access, I've seen where a model went out and looked up the solution for the benchmark because Terminal Bench is a public benchmark.

s291
00:20:56.440 --> 00:20:59.200
And you have to look in the traces to see what happened here.

s292
00:20:59.200 --> 00:21:04.280
It It's not really It's not even a bad thing if you want an agent, it should be smart.

s293
00:21:04.280 --> 00:21:09.360
And if there's a solution for the problem, why should it try to think of it if it there is a published solution?

s294
00:21:09.360 --> 00:21:14.120
So, it depends what you really want to measure and if you want to give it access or not.

s295
00:21:14.120 --> 00:21:16.280
And not all of those do that.

s296
00:21:16.280 --> 00:21:18.040
Or not every time.

s297
00:21:18.040 --> 00:21:19.480
Sandbox resources, all the thing.

s298
00:21:19.480 --> 00:21:28.480
If you give it less CPU or memory, it may run out or it be maybe slower, so it takes longer to do something or even crash and then you get a bad result.

s299
00:21:28.480 --> 00:21:32.040
And it's not the model's fault, it's because of the sandbox configuration.

s300
00:21:32.040 --> 00:21:34.880
Task timeout, every task gets a timeout.

s301
00:21:34.880 --> 00:21:40.160
By default, they range from I think 5 minutes 15 minutes to over 3 hours.

s302
00:21:40.160 --> 00:21:44.480
One is 3 hours something and another one is 2 hours or so.

s303
00:21:44.480 --> 00:21:48.400
And that means you have to decide how much time you want to invest.

s304
00:21:48.400 --> 00:22:00.840
When I started with Wolf Bench, I gave every task a fixed 2 hours because I didn't want to basically handicap open weights models which are often running on less powerful inference

s305
00:22:00.840 --> 00:22:05.040
endpoints than the big SOTA frontier lab models.

s306
00:22:05.040 --> 00:22:11.480
So, if you are running DeepSeek on a local system, it's usually not as fast as GPT on OpenAI systems.

s307
00:22:11.480 --> 00:22:15.000
And that would skew the score if you don't give it unlimited time.

s308
00:22:15.000 --> 00:22:23.400
But I noticed when I was going with 2 hours that if the model can't do the task, it sometimes just goes in circles and tries and tries and tries and fails and fails and fails.

s309
00:22:23.400 --> 00:22:34.080
So, it was very expensive and took a long time because if I do 2 hours per task and there are 89 tasks and I parallelize them depending on how much concurrency I'm using, which is the next point,

s310
00:22:34.080 --> 00:22:35.400
it takes so long.

s311
00:22:35.400 --> 00:22:41.160
It takes so long and it costs so much and you don't get a result because it just can't do the task.

s312
00:22:41.160 --> 00:22:44.160
So, I noticed 1 hour is a good mark.

s313
00:22:44.160 --> 00:22:49.080
If you use the default terminal bench the test duration will rise.

s314
00:22:49.080 --> 00:22:51.320
Concurrency, that is what I meant.

s315
00:22:51.320 --> 00:22:56.560
You don't want to wait 98 89 hours or even twice the amount of time.

s316
00:22:56.560 --> 00:23:01.640
You want to do concurrency, so you have multiple trials happening at the same time.

s317
00:23:01.640 --> 00:23:04.840
But then you have to decide how high can I go?

s318
00:23:04.840 --> 00:23:10.040
If If I go concurrency 89, I can do all the 89 tasks of terminal bench at once.

s319
00:23:10.040 --> 00:23:18.680
But, if my inference endpoint is not fast enough and I get rate limited, then I will get bad scores or even no scores at the the end.

s320
00:23:18.680 --> 00:23:19.800
The same with the sandboxes.

s321
00:23:19.800 --> 00:23:29.680
If I have sandbox provider, I get resources and if I use 8 GB per sandbox, yeah, it sums up quickly and maybe you get over the limit you are allowed.

s322
00:23:29.680 --> 00:23:31.800
Retry policy, how often to retry something.

s323
00:23:31.800 --> 00:23:38.320
That is also something if I don't retry a task and it fails because a sandbox crashed, I get a bad score for the model.

s324
00:23:38.320 --> 00:23:39.640
But, it wasn't the model's fault.

s325
00:23:39.640 --> 00:23:43.800
So, if there are errors that you should retry and there are errors you should not retry.

s326
00:23:43.800 --> 00:23:51.280
Like if the model causes a crash on its own by doing something stupid, that is actually a fail you want to measure.

s327
00:23:51.280 --> 00:23:53.600
Temperature, the usual stuff.

s328
00:23:53.600 --> 00:23:58.760
Um that is a difficult thing for a benchmark because which temperature do you use?

s329
00:23:58.760 --> 00:24:02.600
Some model providers they give you the recommended temperature to use with their model.

s330
00:24:02.600 --> 00:24:03.800
Some don't.

s331
00:24:03.800 --> 00:24:05.920
And some agents have a default.

s332
00:24:05.920 --> 00:24:08.920
Some agents decide based on the model.

s333
00:24:08.920 --> 00:24:15.000
And there are so many combinations that you have to decide, do you want to look for every model what is the recommended one?

s334
00:24:15.000 --> 00:24:17.000
If there is none, which do you use?

s335
00:24:17.000 --> 00:24:21.840
How do you make sure that people who are doing the same benchmark are using the same setting you chose?

s336
00:24:21.840 --> 00:24:23.480
It's difficult.

s337
00:24:23.480 --> 00:24:24.560
And top P as well.

s338
00:24:24.560 --> 00:24:30.120
Do you use 0.95 or do you use one or do you any value?

s339
00:24:30.120 --> 00:24:32.760
It's just one setting and there are so many others.

s340
00:24:32.760 --> 00:24:36.040
And max new tokens, how many tokens is the model allowed to generate?

s341
00:24:36.040 --> 00:24:44.000
You have to decide on this because there are recommendations by the providers, there are limitations by the by the the hosters.

s342
00:24:44.000 --> 00:24:47.280
You have to choose something and usually you have to configure it.

s343
00:24:47.280 --> 00:24:51.280
And if you can, you can also set a seed so the randomness is reduced.

s344
00:24:51.280 --> 00:24:55.720
We I when I started benchmark, I used temperature zero so there was less randomness.

s345
00:24:55.720 --> 00:24:57.480
There was never no randomness.

s346
00:24:57.480 --> 00:25:02.600
And I've seen things as small as a traffic driver update to change the actual results I got.

s347
00:25:02.600 --> 00:25:05.920
Even if everything was uh deterministic or looked like that.

s348
00:25:05.920 --> 00:25:14.720
Because we have floating point rounding errors, we have performance optimization, so many effects that change the result even if you use a temperature zero.

s349
00:25:14.720 --> 00:25:20.080
And nowadays, when the model is recommending temperature one, then you should not set zero, of course.

s350
00:25:20.080 --> 00:25:22.760
Otherwise, you would be going against the recommendation.

s351
00:25:22.760 --> 00:25:27.680
In that case, you could set a seed, but even if you choose a seed like 42, it's very popular.

s352
00:25:27.680 --> 00:25:28.760
You know why.

s353
00:25:28.760 --> 00:25:35.680
And the thing is, some model may be more have more luck with that seed than others have less luck with that seed.

s354
00:25:35.680 --> 00:25:38.440
It's all choices you have to make.

s355
00:25:38.440 --> 00:25:46.800
Now, for this choice, which inference we want to use, I have on the left side the normal inference core with serverless inference.

s356
00:25:46.800 --> 00:25:55.800
If you want to follow along and you want to use open router because you don't didn't go through the getting your own key here, you could use open router as well.

s357
00:25:55.800 --> 00:26:00.480
In either case, you just use an end file and put in the variable at the bottom here.

s358
00:26:00.480 --> 00:26:07.360
You just put the variable, the key in the end file, and then we will use that with the agent.

s359
00:26:07.360 --> 00:26:15.360
Um if you use open router, you have to set some special settings to use our provider, the One API provider.

s360
00:26:15.360 --> 00:26:18.520
This is very important if you are using open router and benchmarking a model.

s361
00:26:18.520 --> 00:26:21.720
There are so many providers, and by default, you get one of these.

s362
00:26:21.720 --> 00:26:23.720
They decide which is the best at the moment.

s363
00:26:23.720 --> 00:26:31.480
So, if you are running a single benchmark run 89 tasks, you may get half the task with this provider and half the task with the other provider.

s364
00:26:31.480 --> 00:26:36.280
So, you definitely have to set which one you want to use.

s365
00:26:36.280 --> 00:26:42.880
And after all this, you have chosen one part for each of the six layers we went through.

s366
00:26:42.880 --> 00:26:52.440
We have chosen Terminal Bench for our evaluation, we have chosen Harbor and a specific version of that, we have chosen our sandboxes, we have chosen Terminus 2 as the agent.

s367
00:26:52.440 --> 00:26:56.280
GLM 5.2 and we have specific settings.

s368
00:26:56.280 --> 00:26:59.640
If you are doing evaluation, you will do the same.

s369
00:26:59.640 --> 00:27:05.960
Whatever framework you use, whatever model you use, you have to make choices and you should be very clear what you are using.

s370
00:27:05.960 --> 00:27:12.560
This is important when you are publishing your results, so others know what you have actually used and measured.

s371
00:27:12.560 --> 00:27:14.680
So, here is another QR code.

s372
00:27:14.680 --> 00:27:21.040
This gives you the full configuration, so you don't have to copy stuff from the slides and here's a formatting that has been messed up a little.

s373
00:27:21.040 --> 00:27:22.040
Excuse this.

s374
00:27:22.040 --> 00:27:32.280
Um If you download this configuration, I will show you in the next slide how to run it and it will take about 5 minutes to actually get the results in this case.

s375
00:27:32.280 --> 00:27:34.000
And I will also do the same.

s376
00:27:34.000 --> 00:27:39.640
So, if you want to download the config, here's the QR code.

s377
00:27:39.640 --> 00:27:41.200
And I will go to the next one.

s378
00:27:41.200 --> 00:27:44.160
So, let me just not just show it, but actually run it.

s379
00:27:44.160 --> 00:27:45.800
So, take a look at this.

s380
00:27:45.800 --> 00:27:55.920
What we are doing, we are sourcing the end file to get the API key we find two slides back and then we are running it with the UV run harbor run config command

s381
00:27:55.920 --> 00:28:02.880
and giving it the configuration you downloaded or created and then we can actually get the results.

s382
00:28:02.880 --> 00:28:06.840
I will do this right now in the shell.

s383
00:28:06.840 --> 00:28:10.520
So, the first thing, installing the harbor framework.

s384
00:28:10.520 --> 00:28:11.880
Executing the command.

s385
00:28:11.880 --> 00:28:14.480
I will zoom in a bit.

s386
00:28:17.600 --> 00:28:24.320
So, if my Wi-Fi is working the way it should, it's full.

s387
00:28:24.320 --> 00:28:24.880
Okay.

s388
00:28:24.880 --> 00:28:26.360
We are letting this run.

s389
00:28:26.360 --> 00:28:29.800
I will go back in a minute.

s390
00:28:29.800 --> 00:28:33.600
So, continuing with the presentation.

s391
00:28:33.600 --> 00:28:38.240
So, we are running the thing and it will do just one test.

s392
00:28:38.240 --> 00:28:43.360
I didn't want to use 89 tests for this, otherwise we would be sitting an hour overtime.

s393
00:28:43.360 --> 00:28:49.240
And that is only the time I give every test, so afterwards the verifier has to run and check if it's correct.

s394
00:28:49.240 --> 00:28:52.240
That takes also some time, so it takes even longer.

s395
00:28:52.240 --> 00:28:59.600
But, um Harbor pulls the tasks, the Terminal Bench 2.0 data set, it uses that from the configuration.

s396
00:28:59.600 --> 00:29:08.160
So, this is the part where it has Terminal Bench version 2.0 in the config.

s397
00:29:08.160 --> 00:29:11.200
Then, it starts the sandboxes using a provider we are using.

s398
00:29:11.200 --> 00:29:15.880
So, it creates a sandbox per task, depending on the concurrency, how many it uses.

s399
00:29:15.880 --> 00:29:16.840
Then, it runs the agent.

s400
00:29:16.840 --> 00:29:21.600
It actually goes in the sandbox, installs the agent, and executes the task.

s401
00:29:21.600 --> 00:29:26.200
Give it Give it a system user prompt, basically, saying here, "That is what you have to do."

s402
00:29:26.200 --> 00:29:27.600
The agent does it.

s403
00:29:27.600 --> 00:29:34.680
And afterwards, it collects the results from the sandbox before it is destroyed.

s404
00:29:34.680 --> 00:29:37.160
And it gets the trajectories, really, what is the agent doing?

s405
00:29:37.160 --> 00:29:39.360
You get a lot of information that way.

s406
00:29:39.360 --> 00:29:49.880
And this small task, it's one of the smallest, it's just a Git repository where something has been lost, and it should be find it in the Git tree and restore it.

s407
00:29:49.880 --> 00:29:51.520
This takes about 3 to 5 minutes.

s408
00:29:51.520 --> 00:29:53.720
Let's see how far we are here.

s409
00:29:53.720 --> 00:29:55.320
It's installed now.

s410
00:29:55.320 --> 00:29:59.680
Then, we install our One B client.

s411
00:29:59.680 --> 00:30:07.200
And the next step, in this case, I'm just getting my API key from the One Password.

s412
00:30:11.280 --> 00:30:14.360
And then, I'm writing the configuration here.

s413
00:30:14.360 --> 00:30:15.640
This is the one you can download.

s414
00:30:15.640 --> 00:30:19.920
I'm just catting it into a file.

s415
00:30:21.440 --> 00:30:23.200
And loading the end file.

s416
00:30:23.200 --> 00:30:29.320
Now, the API key is in my environment variables, and I will just This is the run command.

s417
00:30:29.320 --> 00:30:30.960
Run the evaluation.

s418
00:30:30.960 --> 00:30:37.320
So, now it's starting to go out, starts the sandboxes, install the agent, run it.

s419
00:30:37.320 --> 00:30:40.280
And this is all happening here.

s420
00:30:41.480 --> 00:30:46.440
And while it's doing that, I will go back to the presentation.

s421
00:30:48.440 --> 00:30:50.080
Next part.

s422
00:30:50.080 --> 00:30:52.160
Run as evidence, not only a score.

s423
00:30:52.160 --> 00:30:53.560
The formatting is totally messed up here.

s424
00:30:53.560 --> 00:30:56.480
I will fix the presentation and we will upload it.

s425
00:30:56.480 --> 00:31:01.400
This is just the directory of what is in the in the evaluation folder.

s426
00:31:01.400 --> 00:31:04.000
We can create this live as soon as it's done.

s427
00:31:04.000 --> 00:31:07.240
Of course, it will take about 5 minutes should it take.

s428
00:31:07.240 --> 00:31:11.880
Right now, it's starting the environment, which is creating the sandboxes.

s429
00:31:11.880 --> 00:31:13.640
So, just letting it run.

s430
00:31:13.640 --> 00:31:14.360
Oh.

s431
00:31:14.360 --> 00:31:16.960
This is too early.

s432
00:31:18.480 --> 00:31:23.200
And when we have that, we get one task and it is scored.

s433
00:31:23.200 --> 00:31:25.840
It either It gets a score of one or zero.

s434
00:31:25.840 --> 00:31:27.720
Did it succeed or did it fail?

s435
00:31:27.720 --> 00:31:39.760
If you do that for the 89 tasks, you get an amount how many tasks it did and then you do that over five runs, for instance, and then you get a percentage and you know how many percent of Terminal Bench did the model get.

s436
00:31:39.760 --> 00:31:42.600
And then this is usually the score that the labs report.

s437
00:31:42.600 --> 00:31:47.200
Most people just report the average across five runs.

s438
00:31:47.280 --> 00:31:50.360
And this is the installation of the Wolf Bench Toolkit.

s439
00:31:50.360 --> 00:31:56.440
What we have done before, it was all the Harbor framework and Terminal Bench and this is what I programmed, what I created.

s440
00:31:56.440 --> 00:32:06.000
The Wolf Bench Toolkit is a way to work with this data because otherwise you now have a couple of files and you have some score, but how do you analyze it?

s441
00:32:06.000 --> 00:32:13.120
So, this is a collector, what Wolf Bench does and I will It's best if I show it when I open the program.

s442
00:32:13.160 --> 00:32:16.200
So, let me go back here.

s443
00:32:16.200 --> 00:32:20.960
While this is happening, I will just start it in a new shell.

s444
00:32:23.000 --> 00:32:26.240
And we just clone it and go in there.

s445
00:32:26.240 --> 00:32:28.680
So, it's open source, Apache 2 license.

s446
00:32:28.680 --> 00:32:30.640
You can use it for your own projects.

s447
00:32:30.640 --> 00:32:36.320
Of course, you have to adapt it if you are using something else than Terminal Bench, but it's easy to adapt nowadays.

s448
00:32:36.320 --> 00:32:38.600
Just tell your agent.

s449
00:32:38.600 --> 00:32:39.440
Quick question.

s450
00:32:39.440 --> 00:32:43.320
Who's using agents for stuff like that?

s451
00:32:43.960 --> 00:32:45.360
Not everybody?

s452
00:32:45.360 --> 00:32:46.640
Oh, I'm surprised.

s453
00:32:46.640 --> 00:32:49.680
I still I still see a lot of notebooks, by the way.

s454
00:32:49.680 --> 00:32:54.880
Because some people now just pull out their phone, take a screenshot, and say, "Okay, agent, go.

s455
00:32:54.880 --> 00:32:58.360
Do whatever he is showing on the stage."

s456
00:32:58.400 --> 00:33:00.320
Next year.

s457
00:33:00.320 --> 00:33:07.480
Okay, so it's downloading and I'm also downloading all the results I did with all the runs I did so far.

s458
00:33:07.480 --> 00:33:16.880
So, quick spoiler, I did uh 430 runs right now across um where is it?

s459
00:33:16.880 --> 00:33:22.360
Uh it was 430 runs.

s460
00:33:22.400 --> 00:33:26.360
But anyway, here are the agents I tested so far.

s461
00:33:26.360 --> 00:33:30.360
Cloud code, Hermes agent, open clock cursor, Codex, Klein is not published yet.

s462
00:33:30.360 --> 00:33:33.000
That is on my list and some other agents.

s463
00:33:33.000 --> 00:33:36.800
If you are making an agent, contact me if you want to appear as well.

s464
00:33:36.800 --> 00:33:42.200
And some models I tested, many on those W&B, that was on our own inference.

s465
00:33:42.200 --> 00:33:48.760
So, the the final cost was unfortunately not as high as it is displayed, almost 40K here.

s466
00:33:48.760 --> 00:33:54.800
So, this is some of this is on our own inference, but I just use the prices that have been published.

s467
00:33:54.800 --> 00:33:56.840
So, let's see where we are here.

s468
00:33:56.840 --> 00:33:58.600
It's downloaded.

s469
00:33:58.600 --> 00:34:01.320
So, the 13 megabytes, those are the results.

s470
00:34:01.320 --> 00:34:05.920
If you want to see all the trajectories, those have been uploaded to weave and you can just inspect them.

s471
00:34:05.920 --> 00:34:10.639
You uh um don't have to um download gigabytes of data.

s472
00:34:10.639 --> 00:34:19.440
Now, if you want to use the Wolf Bench, you can set some variables, so it is also an uploader, but I will show you if I just start it.

s473
00:34:19.720 --> 00:34:28.040
And this uses a Remeo notebook to um provide a dashboard, an easy way to handle all the evaluation data.

s474
00:34:28.040 --> 00:34:30.399
So, any moment now.

s475
00:34:30.399 --> 00:34:31.679
There it is.

s476
00:34:31.679 --> 00:34:32.639
So, this is a dashboard.

s477
00:34:32.639 --> 00:34:37.840
You can enter your information where you want to upload the traces if you have the We stuff.

s478
00:34:37.840 --> 00:34:47.360
Then you can discover VMs if you are using XE dev or Hetzner, it will find all your VMs on there and you can filter which VM names are evaluation VMs.

s479
00:34:47.360 --> 00:34:50.679
Because I don't run the Harbor Framework on my own laptop usually.

s480
00:34:50.679 --> 00:35:00.640
I run it in a VM so I can turn off my laptop or my Wi-Fi doesn't go off and the connection to the sandboxes fails because that could also affect something.

s481
00:35:00.640 --> 00:35:06.240
Um in this case, everything is local as soon as I download the stuff which I did.

s482
00:35:06.240 --> 00:35:10.840
So, I can just scan my local directory and it will find all the run data I already did.

s483
00:35:10.840 --> 00:35:14.240
So, here are the 443 unique runs.

s484
00:35:14.240 --> 00:35:21.560
And it shows you here a filter so I every run that has a score is is measured.

s485
00:35:21.560 --> 00:35:22.920
It has to have all the tests.

s486
00:35:22.920 --> 00:35:25.920
If some tests have not been run, it's not a valid res a run.

s487
00:35:25.920 --> 00:35:27.520
It was just a test run.

s488
00:35:27.520 --> 00:35:29.520
It shouldn't have too many errors or timeouts.

s489
00:35:29.520 --> 00:35:37.080
And then I get a list of all of this stuff and I can choose which models I want to create um uh nice chart for.

s490
00:35:37.080 --> 00:35:39.040
In this case, I will just use all of them.

s491
00:35:39.040 --> 00:35:48.960
Then you can also just change the names of the models and sometimes you also have to change the thinking because the parameter you give can be changed by the agent.

s492
00:35:48.960 --> 00:36:00.160
Like if you are using a thinking extra high or if you want to use a thinking max mode and your agent doesn't support it, maybe it is called a different way.

s493
00:36:00.160 --> 00:36:05.760
And then you have to change it so it's really reflecting what has been used actually.

s494
00:36:05.760 --> 00:36:11.480
Now, after all of this is done, I can just create my results file.

s495
00:36:11.480 --> 00:36:13.200
You can upload to We view if you have the key.

s496
00:36:13.200 --> 00:36:16.840
I do not have set the key so I right now don't have the button.

s497
00:36:16.840 --> 00:36:21.800
Now, I create a save file and this is one JSON file that contains all the results.

s498
00:36:21.800 --> 00:36:23.680
You can share that with other people.

s499
00:36:23.680 --> 00:36:28.480
And in the end, I generate a chart which is a artifact I generate.

s500
00:36:28.480 --> 00:36:32.000
So, now from all the results I did, I get a new chart.

s501
00:36:32.000 --> 00:36:36.680
And this is the same that I then in the next step, I have an uploader as well.

s502
00:36:36.680 --> 00:36:41.680
After the chart is generated, I just upload it to wolfbenchmark.ai and it's updated.

s503
00:36:41.680 --> 00:36:48.880
If I activate this option to replace the index.html and it is online immediately.

s504
00:36:49.760 --> 00:36:54.200
So, going back to the present Oh, let's just check how far the benchmark run.

s505
00:36:54.200 --> 00:36:55.040
Oh, it's done.

s506
00:36:55.040 --> 00:36:55.920
See, it's done.

s507
00:36:55.920 --> 00:37:01.160
Trials one, mean score 1.0. So, it had one task, it solved the task.

s508
00:37:01.160 --> 00:37:03.400
So, it worked.

s509
00:37:04.000 --> 00:37:07.360
It finished and it fixed the Git repository.

s510
00:37:07.360 --> 00:37:16.240
And this task is now in my directory and I could import it, but it's not a full run, so it is not displayed by default.

s511
00:37:16.240 --> 00:37:21.720
And once we have done that and uploaded this, we can look at what Wolf benchmark provides.

s512
00:37:21.720 --> 00:37:26.560
This is also the mindset thing again, what do we want to look at?

s513
00:37:27.480 --> 00:37:31.040
And in this case, this is what we just did.

s514
00:37:31.040 --> 00:37:33.040
I showed you the dashboard.

s515
00:37:33.040 --> 00:37:45.720
And now we have to take a look at the leaderboard and what it means, because the leaderboard itself, whatever it generates, it is not just an answer, it helps you to find a better question.

s516
00:37:45.720 --> 00:37:47.160
Like, what are you looking at?

s517
00:37:47.160 --> 00:37:51.480
Do you care about the capability of a model or the reliability of a model?

s518
00:37:51.480 --> 00:37:56.800
And I will show you what that means by going to wolfbenchmark.ai.

s519
00:38:00.000 --> 00:38:05.200
And the thing is, I will just make it a little smaller.

s520
00:38:05.200 --> 00:38:08.560
So, they are sorted from left to right and I have all of the information here.

s521
00:38:08.560 --> 00:38:13.520
Most benchmarks you look at, they just show the terminal's two score and they only show the average.

s522
00:38:13.520 --> 00:38:21.920
So, this is an interactive page, which is used not only to display results, but to create charts that you can then copy and paste and put somewhere else.

s523
00:38:21.920 --> 00:38:26.080
There's a screenshot symbol so I can show stuff on X for instance.

s524
00:38:26.080 --> 00:38:29.080
So if I want to show how did this model do, I can just mark it.

s525
00:38:29.080 --> 00:38:32.680
I can remove some of the bars or reorder the bars.

s526
00:38:32.680 --> 00:38:34.080
So it's interactive.

s527
00:38:34.080 --> 00:38:37.040
So I can create my report that way.

s528
00:38:37.040 --> 00:38:39.880
And what it also has it in the URL.

s529
00:38:39.880 --> 00:38:47.080
So everybody can if you give the URL to this, they see the same thing you had on your screen.

s530
00:38:47.080 --> 00:38:49.880
So I'm switching over here.

s531
00:38:51.960 --> 00:39:00.760
And it looks a bit overwhelming sometimes because there are so many information so so many information on the same bar.

s532
00:39:00.760 --> 00:39:08.520
Like I said, you can filter to different information to make it a simple bar, but I like to see this because the darker part at the bottom, that percentage

s533
00:39:08.520 --> 00:39:14.520
is how many percent of the whole benchmark has a model solved in every run of the five runs runs I did.

s534
00:39:14.520 --> 00:39:16.520
So that is what I call consistency.

s535
00:39:16.520 --> 00:39:23.320
This model and this agent, they have solved the task 60% of the whole benchmark every time.

s536
00:39:23.320 --> 00:39:32.240
And this means if that is a very high score, the model is very reliable because if it solves the task, you can expect it to solve the task all the time.

s537
00:39:32.240 --> 00:39:37.440
While the yellow score is the average, the one score that usually is reported.

s538
00:39:37.440 --> 00:39:43.280
And the up and down arrows around it, those are the best and the worst scores in one of the five runs.

s539
00:39:43.280 --> 00:39:47.200
So if those are very close to each other, then there's not a high variance.

s540
00:39:47.200 --> 00:39:49.080
So you can expect the same score.

s541
00:39:49.080 --> 00:39:56.720
If they are very far apart like we see here from 79% to 60%, that means there's a lot of variance.

s542
00:39:56.720 --> 00:40:01.680
And that is a sign that the model there's a lot of randomness involved here.

s543
00:40:01.680 --> 00:40:04.640
You can't really trust it to execute the the task.

s544
00:40:04.640 --> 00:40:07.920
Even if it has a high score, it's a bit random if it does it.

s545
00:40:07.920 --> 00:40:10.640
And the highest mark is the star, that is the ceiling.

s546
00:40:10.640 --> 00:40:15.440
How many percent of the benchmark has it solved at all, at least once.

s547
00:40:15.440 --> 00:40:24.720
The amount of task that tells you how Yeah, in an ideal case, that would be the score the model should get all the time if it's perfect and always doing it.

s548
00:40:24.720 --> 00:40:28.400
Although, that would also be an indicator that it is probably benchmarked for this.

s549
00:40:28.400 --> 00:40:31.480
Um but it shows you how far it can go.

s550
00:40:31.480 --> 00:40:39.160
So, what you want is a very very high ceiling, a very stable average, and a very high solid base.

s551
00:40:39.160 --> 00:40:46.320
And if a model has the same score and they are almost the same, you can still see that the solid base is different.

s552
00:40:46.320 --> 00:40:50.320
And then in that case, I would pick one that has a higher solid base.

s553
00:40:50.320 --> 00:40:53.160
Like here, it's 73% to 74%.

s554
00:40:53.160 --> 00:40:54.520
It's almost the same.

s555
00:40:54.520 --> 00:40:58.360
If you just look at the average score, it's almost exactly the same thing.

s556
00:40:58.360 --> 00:41:01.120
Could be a variance, just variance.

s557
00:41:01.120 --> 00:41:03.080
But this is a bigger difference.

s558
00:41:03.080 --> 00:41:06.480
It's 6% points, and that is a noticeable gap.

s559
00:41:06.480 --> 00:41:12.000
So, you see the extra high thinking is better than the medium thinking, as we expected.

s560
00:41:12.000 --> 00:41:16.040
But if we are looking at um This is also very interesting.

s561
00:41:16.040 --> 00:41:17.320
I will show it in the presentation.

s562
00:41:17.320 --> 00:41:20.640
Let me switch to the presentation.

s563
00:41:20.920 --> 00:41:23.760
So, these were the different ways to imagine the model.

s564
00:41:23.760 --> 00:41:24.960
Ah, risk and cost tolerance.

s565
00:41:24.960 --> 00:41:33.640
I have to show you this because this is something If you look at the second best model on my evaluation, it was Gemini 3.5 Flash.

s566
00:41:33.640 --> 00:41:39.160
So, this model, a flash model, is almost as good as GPT 5.5. So, what gives?

s567
00:41:39.160 --> 00:41:45.280
If I just look at terminus, so we don't have so many bars, and I can also say I just want to compare these two.

s568
00:41:45.280 --> 00:41:47.960
And yeah, it's a great model, Gemini 3.5 Flash.

s569
00:41:47.960 --> 00:41:54.880
But if I look at how did it achieve that, I have a 3D view, which shows you how many tokens it used to achieve that.

s570
00:41:54.880 --> 00:42:02.400
And it has been billions of tokens to get that score, while GPT 5.5 used much less.

s571
00:42:02.400 --> 00:42:04.960
Let me zoom in.

s572
00:42:06.240 --> 00:42:14.400
So, 136 million tokens for GPT 5.5, compared to 1.65 billion tokens for Gemini 3.5 Flash.

s573
00:42:14.400 --> 00:42:21.000
And the thing is, if we also look at the price, which is another 3D view, this is the cost.

s574
00:42:21.000 --> 00:42:24.040
And I can even show both at the same time.

s575
00:42:24.040 --> 00:42:25.600
The cost is now a shadow.

s576
00:42:25.600 --> 00:42:27.520
You see that the cost is over here.

s577
00:42:27.520 --> 00:42:37.320
It's overlap because it used so many tokens, but this shows that Gemini 3.5 Flash is very close to GBD 5.5, but it used so many tokens that it cost

s578
00:42:37.320 --> 00:42:40.440
even more than GBD 5.5 of course.

s579
00:42:40.440 --> 00:42:45.760
So, it costs in this case $620 compared to $497.

s580
00:42:45.760 --> 00:42:48.000
So, 500 to over 600.

s581
00:42:48.000 --> 00:42:51.280
So, it's not that you Yeah, you have to think about this.

s582
00:42:51.280 --> 00:42:54.160
Do you want to use a fast and cheap model?

s583
00:42:54.160 --> 00:42:59.560
But you have to compare is it really faster and cheaper because all those tokens also take time.

s584
00:42:59.560 --> 00:43:02.040
That is one thing.

s585
00:43:02.200 --> 00:43:04.840
Going back to the presentation.

s586
00:43:05.200 --> 00:43:07.440
And it's a decision surface.

s587
00:43:07.440 --> 00:43:08.800
It's not just a chart.

s588
00:43:08.800 --> 00:43:15.080
So, you load your snapshot and you can add your own if you did it and you execute the code, you would see them.

s589
00:43:15.080 --> 00:43:18.960
And then you can filter like I showed you different metrics and models.

s590
00:43:18.960 --> 00:43:21.920
You can reorder the bars and export your own.

s591
00:43:21.920 --> 00:43:28.240
And then you can show this and make a social post about this or write an article, whatever you do.

s592
00:43:28.240 --> 00:43:29.600
Um you can do this.

s593
00:43:29.600 --> 00:43:33.480
In this case, I was looking at which is the best model on this benchmark.

s594
00:43:33.480 --> 00:43:38.160
So, my number one right now is GBD 5.5 with Codex.

s595
00:43:38.160 --> 00:43:45.320
It is even better than Fable because if you are asking what happened to Fable, that is also very interesting to take a look at.

s596
00:43:45.320 --> 00:43:49.080
If I put Fable in here, it's had 13 refusals.

s597
00:43:49.080 --> 00:43:50.240
So, it refused.

s598
00:43:50.240 --> 00:43:54.480
And if you take a look at the cost, it goes so far.

s599
00:43:54.480 --> 00:43:56.760
It was my most expensive benchmark.

s600
00:43:56.760 --> 00:44:04.040
Don't tell anyone because it cost 11,000 just to test Fable and it failed because it refused so many tests.

s601
00:44:04.040 --> 00:44:05.320
Whoops.

s602
00:44:05.320 --> 00:44:09.360
And that is also something of course you have to decide.

s603
00:44:09.360 --> 00:44:13.760
That is when people say oh I don't trust this benchmark this model is in the first place.

s604
00:44:13.760 --> 00:44:23.120
Look at what happened here and the benchmark has to be transparent so you can look at the traces and see what has happened then it makes sense suddenly.

s605
00:44:23.120 --> 00:44:24.080
Okay.

s606
00:44:24.080 --> 00:44:33.080
Going back and yeah like I said the best model was Codex with GPT 5.5 but as you can see it was pretty very expensive.

s607
00:44:33.080 --> 00:44:34.160
Let's zoom in.

s608
00:44:34.160 --> 00:44:36.320
It cost almost a thousand bucks.

s609
00:44:36.320 --> 00:44:42.520
While Cursor with the same model cost only 282.

s610
00:44:42.520 --> 00:44:48.720
So that was very cheap compared to the other one and it used a little less tokens here.

s611
00:44:48.720 --> 00:44:59.840
But yeah that's you could say okay this is the best combination but maybe the other one is from a price and performance perspective the better one.

s612
00:44:59.840 --> 00:45:05.960
And Terminus also had the highest ceiling but if the scores are very close to each other that could be random.

s613
00:45:05.960 --> 00:45:16.800
You have would have to do 10 runs or even more to be very sure but you have to make a choice sometime and pick one and go with it.

s614
00:45:16.800 --> 00:45:28.080
This was also the thing that I just showed Gemini 3.5 Flash compared to GPT 5.5 lower score bigger bill so in that case it would make more sense to use the other model.

s615
00:45:28.080 --> 00:45:39.480
And this was also super interesting and surprised me a lot when I tested Opus 4.6 at the time I raised the thinking to max and the score decreased.

s616
00:45:39.480 --> 00:45:46.040
It really got worse and I did the same with Claude Code because I used Titanus as well and there I saw the same effect.

s617
00:45:46.040 --> 00:45:54.400
When I do that with GPT if I increase from thinking off to extra high or from medium to extra high if I raise the thinking level the model gets smarter as you would expect.

s618
00:45:54.400 --> 00:45:57.560
So why did it get worse with Opus?

s619
00:45:57.560 --> 00:46:04.040
That was surprising to me and the difference if If look at the traces is that you see the model it thinks too much.

s620
00:46:04.040 --> 00:46:05.560
It's overthinking.

s621
00:46:05.560 --> 00:46:07.880
And that means I give it 1 hour.

s622
00:46:07.880 --> 00:46:17.000
And if the model thinks for a minute or two and then decides I want to execute this command and the command doesn't exist on the machine yet, then it thinks another two or three minutes.

s623
00:46:17.000 --> 00:46:18.840
And how do I install this?

s624
00:46:18.840 --> 00:46:21.960
While an agent that is not even thinking, it's just doing stuff.

s625
00:46:21.960 --> 00:46:25.600
So the first impulse is I run this command, get an error message.

s626
00:46:25.600 --> 00:46:26.880
I run this command, get an error message.

s627
00:46:26.880 --> 00:46:29.040
I install it, get an error message, fix it.

s628
00:46:29.040 --> 00:46:31.480
And it gets more turns.

s629
00:46:31.480 --> 00:46:32.720
So it can do more.

s630
00:46:32.720 --> 00:46:39.280
That is the difference between thinking for an hour and not doing anything or just trying, trying, trying, trial and error.

s631
00:46:39.280 --> 00:46:45.360
And the interesting stuff is that the GPT models, they are by default they appeared not as smart as Opus.

s632
00:46:45.360 --> 00:46:47.440
Opus is very capable out of the box.

s633
00:46:47.440 --> 00:46:50.560
Even with thinking off, it already got a great score.

s634
00:46:50.560 --> 00:47:00.720
If you compare that here, it is already with off better than medium here on GPT 5.5. And even the solid base was higher.

s635
00:47:00.720 --> 00:47:04.920
So it did more tasks in every run.

s636
00:47:04.920 --> 00:47:05.280
Yeah.

s637
00:47:05.280 --> 00:47:07.360
So that is stuff you have to look at.

s638
00:47:07.360 --> 00:47:11.480
And if it's surprising, take a look at the traces to find out what is happening here.

s639
00:47:11.480 --> 00:47:13.160
Model overthinking.

s640
00:47:13.160 --> 00:47:19.640
I also saw the same with Fable, by the way, where the model was very smart and it seemed to know it was so smart.

s641
00:47:19.640 --> 00:47:22.800
So what it did is it decided to go this direction.

s642
00:47:22.800 --> 00:47:26.160
And even if the benchmark said it should be done that way, said, "No, I know better.

s643
00:47:26.160 --> 00:47:27.360
I will do it my way."

s644
00:47:27.360 --> 00:47:31.120
And then the verifier at the end, it ran, tested, and failed.

s645
00:47:31.120 --> 00:47:36.600
So the model was too smart for its own good in some cases.

s646
00:47:36.600 --> 00:47:39.840
And this is also an important feature I have to show you.

s647
00:47:39.840 --> 00:47:41.680
The bar is a claim, the trace is evidence.

s648
00:47:41.680 --> 00:47:43.640
I'm talking about traces all the time.

s649
00:47:43.640 --> 00:47:47.200
So we have the bar, which is what we see on Wolf Bench.

s650
00:47:47.200 --> 00:47:48.800
You have the nice bars.

s651
00:47:48.800 --> 00:47:53.120
And the thing is you can click any of the bars, any of these.

s652
00:47:53.120 --> 00:47:54.960
You just click it.

s653
00:47:54.960 --> 00:48:01.560
And you get taken to our One page, where we have weave, which allows you to inspect all the traces.

s654
00:48:01.560 --> 00:48:07.880
In this case, I just clicked one bar, and I'm seeing all the five runs that composed that score.

s655
00:48:07.880 --> 00:48:16.200
And I can now pick any of the runs, go in here, and uh this I should remove.

s656
00:48:16.200 --> 00:48:18.040
I'll show this later.

s657
00:48:18.040 --> 00:48:20.480
Uh go away.

s658
00:48:20.480 --> 00:48:21.880
So, we have 89 chords.

s659
00:48:21.880 --> 00:48:27.400
That are the 89 tasks that I have been doing, and you can see all of the traces here.

s660
00:48:27.400 --> 00:48:29.200
It's a lot, so it's loading a bit.

s661
00:48:29.200 --> 00:48:37.760
And then you can go into any of these, and you see the task name, and you see the reward, did it succeed or not, and you see all the steps.

s662
00:48:37.760 --> 00:48:42.120
You see the initial request, the user message that says, "You are an assistant.

s663
00:48:42.120 --> 00:48:43.960
You are tasked to do with this."

s664
00:48:43.960 --> 00:48:53.040
Then you see what the LLM did as a first command, what analysis it did, and what it wants to do, and then the command it sends, and so on.

s665
00:48:53.040 --> 00:48:57.160
And you can really inspect all of this what has happened, and look at it.

s666
00:48:57.160 --> 00:48:59.720
Or use your agent, and have this done.

s667
00:48:59.720 --> 00:49:03.560
And we have a new feature, which is also an agent built into the platform.

s668
00:49:03.560 --> 00:49:07.280
So, you have an agent inside of weave, and you can just tell it to do this.

s669
00:49:07.280 --> 00:49:14.160
There's a talk about this by my colleague, Tim Sweeney is doing it on July 1. There you can check it out.

s670
00:49:14.160 --> 00:49:18.760
I use agents a lot, because reading all the traces takes a long time.

s671
00:49:18.760 --> 00:49:24.160
And if the agent gives you a link to weave, and you see that it's true what it says, that is also very good.

s672
00:49:24.160 --> 00:49:28.360
So, to recap, there are so many variables relevant to benchmarks.

s673
00:49:28.360 --> 00:49:39.960
You have the benchmark version, the OneBench 2.0, 2.1, the agent version, the model service and route, even updates to the infer- inference endpoint can change the results and the settings.

s674
00:49:39.960 --> 00:49:49.320
Sandbox resources, like I said, if the if the agent runs out of resources, it will fail, and then you get a bad score for the model, although it wasn't the model's fault.

s675
00:49:49.320 --> 00:49:52.160
Sampling, temperature, all things we discussed.

s676
00:49:52.160 --> 00:50:00.360
Thinking defaults, like maximum not being higher if it thinks too long and a smaller, faster agent reaction could work better.

s677
00:50:00.360 --> 00:50:02.480
Time out policy, retry policy.

s678
00:50:02.480 --> 00:50:05.880
If you don't retry and there has been infrastructure error, you would get a bad score.

s679
00:50:05.880 --> 00:50:11.640
So, after every run I go through all of the errors and mistakes and check if there have been inference issues.

s680
00:50:11.640 --> 00:50:23.560
You can find that out if there are repeated characters or if prompt template um tokens leak into the actual response, then you know there's a tokenizer issue, an inference server issue.

s681
00:50:23.560 --> 00:50:25.840
And especially with new models, that is a big thing.

s682
00:50:25.840 --> 00:50:32.600
If you are benchmarking models as they come out, they are so new that the inference engine maybe has not the best support yet.

s683
00:50:32.600 --> 00:50:36.160
So, I have done evaluations where the new model wasn't that great.

s684
00:50:36.160 --> 00:50:42.520
A week later I did the same, everything was the same, and it was better because the inference engine has been improved.

s685
00:50:42.520 --> 00:50:44.680
That is also part of this.

s686
00:50:44.680 --> 00:50:47.240
So, use the message, bring your own everything.

s687
00:50:47.240 --> 00:50:54.800
You have the building blocks, check out the presentation, think about what you want to do, which evaluation to use, which agent to use, which model to test.

s688
00:50:54.800 --> 00:51:05.720
You have the mindset now to know what to look out for and apply this to make uh leaderboards and charts and evaluations that are more reliable than if you just run something and

s689
00:51:05.720 --> 00:51:07.440
publish a score.

s690
00:51:07.440 --> 00:51:10.440
So, run your own reproducible benchmark configurations.

s691
00:51:10.440 --> 00:51:12.000
Share the results.

s692
00:51:12.000 --> 00:51:13.960
If you want to work with Wolf Bench, contact me.

s693
00:51:13.960 --> 00:51:16.640
If you want to run some models, that would be great.

s694
00:51:16.640 --> 00:51:20.960
We could collaborate and publish the results so I don't have to run everything.

s695
00:51:20.960 --> 00:51:25.680
And uh if you want to make an off-spin, that's also great.

s696
00:51:25.680 --> 00:51:28.000
Use another framework, whatever you want.

s697
00:51:28.000 --> 00:51:32.760
And if you are building models or agents, feel free to contact me so I can benchmark them.

s698
00:51:32.760 --> 00:51:38.680
I work with some of the agent uh creators already and will publish results soon.

s699
00:51:38.680 --> 00:51:41.160
Yeah, that's basically the end of the presentation.

s700
00:51:41.160 --> 00:51:43.760
I hope you could get something out of it.

s701
00:51:43.760 --> 00:51:45.880
And if you have some questions, now is the time.

s702
00:51:45.880 --> 00:51:50.760
Otherwise, you can catch me at the booth and uh big shout out to Alex for the help.

s703
00:51:50.760 --> 00:51:58.160
The podcast Thursday I is every Thursday and in the morning the time zone I'm from another time zone.

s704
00:51:58.160 --> 00:52:03.080
8:30 um Pacific time.

s705
00:52:03.080 --> 00:52:10.480
Every week because it's so hard to keep track of what's happening and it's hard for us as well so we are forced to do it so we can report it to you.

s706
00:52:10.480 --> 00:52:13.120
Okay, thank you.

s707
00:52:27.966 --> 00:52:29.966
[music]