WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=_dTF3NpPzGo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/_dTF3NpPzGo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.960 --> 00:00:29.840
So, I'm here to talk about all these words that we call slub and more importantly, I want to try to help fixing what Karpathy termed as the the slub apocalypse

s3
00:00:29.840 --> 00:00:36.920
that he thought would start in 2026 and I think we've seen that on social media.

s4
00:00:36.920 --> 00:00:43.520
So, uh to try to fix this and to stop slub, we first need to define it.

s5
00:00:43.520 --> 00:00:45.360
So first, what is slub?

s6
00:00:45.360 --> 00:00:57.040
There are tons of different definitions from the various experts in the field, but what I think we can all agree is that it really sucks to read something that feels generated.

s7
00:00:57.040 --> 00:01:07.680
And it usually looks like this with uh the the words that we know of, the delve and uh the generaties and other things like these and the main problem

s8
00:01:07.680 --> 00:01:15.840
is that even as models get better with fable uh GPT-6.1 and all the the better models, slub doesn't seem to go go away.

s9
00:01:15.840 --> 00:01:18.600
It just keeps on scaling.

s10
00:01:18.600 --> 00:01:24.040
So, we need something different than just training the models more, it seems.

s11
00:01:24.040 --> 00:01:31.760
And I really want to try to at least first define what slub is so that we can understand why it's there.

s12
00:01:31.760 --> 00:01:35.480
And so to me, slub is made of six main thing.

s13
00:01:35.480 --> 00:01:42.960
First, there are tons of overused words by language models that we don't really use like delve that we really know well.

s14
00:01:42.960 --> 00:01:44.640
Second, it's usually too general.

s15
00:01:44.640 --> 00:01:48.120
It's always talking about most themes, most projects.

s16
00:01:48.120 --> 00:01:50.160
It doesn't share nothing specific.

s17
00:01:50.160 --> 00:01:59.560
And I'm myself guilty of using that on LinkedIn as well, which I guess it's the LinkedIn meta, so it's it's normal there.

s18
00:01:59.560 --> 00:02:04.400
And worse slop sloppy text usually says nothing.

s19
00:02:04.400 --> 00:02:09.479
It's just like nice words put together that we don't even usually use.

s20
00:02:09.479 --> 00:02:14.040
And even worse than that, it says nothing, but it also repeats itself a lot.

s21
00:02:14.040 --> 00:02:17.840
So it repeats nothing and it's too general.

s22
00:02:17.840 --> 00:02:22.320
And uh even worse, it hallucinates.

s23
00:02:22.320 --> 00:02:28.120
As we know, models hallucinate, so the sloppy text also we can't really fully trust it.

s24
00:02:28.120 --> 00:02:36.560
And lastly, the worst to me is that it usually has the wrong tone or at least it has no tone and like everyone's tone at the same time.

s25
00:02:36.560 --> 00:02:38.160
So it just doesn't really talk like you.

s26
00:02:38.160 --> 00:02:41.800
It doesn't share stuff that you would share.

s27
00:02:41.800 --> 00:02:50.440
So my goal here today and with this project that I've been working on is to first measure this sloppy writing.

s28
00:02:50.440 --> 00:02:51.640
Uh not just slop words.

s29
00:02:51.640 --> 00:02:54.080
I don't want to rebuild EQ-Bench slop score.

s30
00:02:54.080 --> 00:02:59.519
It's just it's really about measuring how sloppy is a writing.

s31
00:02:59.519 --> 00:03:06.400
So in order to do that, we are well today right now releasing a new benchmark that we called Tone-Bench.

s32
00:03:06.400 --> 00:03:16.440
Which uh the whole goal is the of the benchmark is to measure if the new models or models are good at creative writing, which I will try to define as well.

s33
00:03:16.440 --> 00:03:19.200
So here's the plan for today.

s34
00:03:19.200 --> 00:03:23.360
Uh the plan is to justify do we even need another benchmark?

s35
00:03:23.360 --> 00:03:33.120
Then how it works, what we measure, what how we built it, and I want to share like the cost of building it, our decisions, what we learned already from

s36
00:03:33.120 --> 00:03:39.560
for writing with AI from the benchmark, and share some more stuff about this.

s37
00:03:39.560 --> 00:03:42.640
So with that, let's delve in.

s38
00:03:42.640 --> 00:03:49.920
So the first First thing, do we even need another benchmark again?

s39
00:03:49.920 --> 00:03:55.000
And just looking at LinkedIn and X X replies, my replies on X, I think we do.

s40
00:03:55.000 --> 00:03:59.280
It's like people are spamming more and more generated content.

s41
00:03:59.280 --> 00:04:02.840
So, at least have them be decently good.

s42
00:04:02.840 --> 00:04:07.040
And uh more precisely, I also really need it.

s43
00:04:07.040 --> 00:04:14.320
We are a small team uh at Towards AI, my company, and we have one side of the company that is focused on education.

s44
00:04:14.320 --> 00:04:18.160
We build tons of educational content, mostly for AI engineering.

s45
00:04:18.160 --> 00:04:21.400
And as a small team, obviously we use AI as well for that.

s46
00:04:21.400 --> 00:04:26.800
And it's been extremely annoying to review this slop uh constantly and try to fix it.

s47
00:04:26.800 --> 00:04:29.440
We uh I published hundreds of videos.

s48
00:04:29.440 --> 00:04:31.520
We published thousands of articles.

s49
00:04:31.520 --> 00:04:32.560
We released a book.

s50
00:04:32.560 --> 00:04:36.413
We are working on a second one uh for AI engineering.

s51
00:04:36.413 --> 00:04:36.760
[snorts]

s52
00:04:36.760 --> 00:04:46.640
And so, we really need to have the best models possible that will help us write better uh and not be full of slop.

s53
00:04:46.640 --> 00:04:55.440
And another nice uh addition to that is that I've been sharing about this benchmark on on X for quite some time as I've been building it.

s54
00:04:55.440 --> 00:05:08.240
And it got really nice engagement, and people seem to confirm that we did need some sort of more creative uh writing benchmark like this one.

s55
00:05:08.440 --> 00:05:21.280
And lastly, just to prove that I think we are the right team to work on this benchmark, we well, I've been build evaluating AI models since my master's and then PhD.

s56
00:05:21.280 --> 00:05:25.440
And now at Towards AI, we do build evaluation pipelines for our clients.

s57
00:05:25.440 --> 00:05:32.080
We have 15 AI engineers with us, and we have creative writers, editors uh on board.

s58
00:05:32.080 --> 00:05:35.080
So, we already have that for our content.

s59
00:05:35.080 --> 00:05:42.520
So, I think uh well, I wanted to leverage that to build a benchmark and help the community even more.

s60
00:05:42.560 --> 00:05:44.600
So, what did we build exactly?

s61
00:05:44.600 --> 00:05:54.640
Hopefully, I I try to share as much as possible, and I'm happy to chat if we can help uh even build your own or if I can just share some some of our learnings.

s62
00:05:54.680 --> 00:05:57.680
So, what we built is super subjective.

s63
00:05:57.680 --> 00:06:01.880
It's a way to measure if a model can write like me.

s64
00:06:01.880 --> 00:06:13.440
And how I define that is the is just that if I can read it out loud like right now without cringing because of the the the the words that I wouldn't use myself.

s65
00:06:13.520 --> 00:06:15.360
So, the process is very simple.

s66
00:06:15.360 --> 00:06:20.240
The the benchmark the process benchmark the benchmark process is very simple.

s67
00:06:20.240 --> 00:06:32.680
We have a prompt that is the brief and the research that we send to a model, and hopefully it will generate entirely a video script that I could have written.

s68
00:06:32.680 --> 00:06:34.200
Uh obviously that doesn't happen.

s69
00:06:34.200 --> 00:06:45.880
It really sucks usually, but that's the whole goal, and we measure that that final script against our rubrics and um just our whole evaluation pipeline.

s70
00:06:45.880 --> 00:06:51.760
And the prompt that we send to every model in the benchmark is much more in-depth than that.

s71
00:06:51.760 --> 00:07:01.560
It it looks more like this where we have a lot of details on how I talk, how I write, all tons of examples of my previous content videos.

s72
00:07:01.560 --> 00:07:09.560
We have a lot of information on storytelling in general, active sentences, and just how to write well.

s73
00:07:09.560 --> 00:07:17.960
And we have uh directions on what to generate and what to do for this current task, which in the the model would generate one script.

s74
00:07:17.960 --> 00:07:19.880
And this is just one of them.

s75
00:07:19.880 --> 00:07:30.320
In the current benchmark, we have 10 different scripts uh that we worked really hard to make as good as possible, and we built rubrics for them to help um

s76
00:07:30.320 --> 00:07:35.120
as we will see LLM judges uh rate these these generations.

s77
00:07:35.120 --> 00:07:36.640
So, we have 10 different scripts.

s78
00:07:36.640 --> 00:07:38.320
Some of them are published, some are not.

s79
00:07:38.320 --> 00:07:44.560
We don't say which ones are the benchmarks so that we check for contamination as well.

s80
00:07:44.640 --> 00:07:49.520
And then what do we do with these 10 scripts?

s81
00:07:49.520 --> 00:07:55.680
We take all the models out there and we try to measure their generations.

s82
00:07:55.680 --> 00:08:05.240
What that means is that for every single model we will do 10 times this process, generate the article from the prompt.

s83
00:08:05.240 --> 00:08:06.600
In fact, we generate five of them.

s84
00:08:06.600 --> 00:08:08.840
I will very soon share why.

s85
00:08:08.840 --> 00:08:12.520
Then we have four different LLM judges analyze them.

s86
00:08:12.520 --> 00:08:14.800
I recently added Jeff as a test.

s87
00:08:14.800 --> 00:08:20.800
Not sure if I will keep that, but I I I added it just to see if it worked well.

s88
00:08:20.800 --> 00:08:25.520
And then we calculate in the end a final score for the leaderboard.

s89
00:08:25.520 --> 00:08:28.360
I will go through all of these steps from left to right.

s90
00:08:28.360 --> 00:08:31.720
So, first, why do we generate even five scripts?

s91
00:08:31.720 --> 00:08:43.680
It's to check for intra-model variability, which I highly suggest if you are building your own evaluations because especially I guess on a creative task, sometimes as we see here with Grok 4.7,

s92
00:08:43.680 --> 00:08:54.320
some of the generations are outliers where it's as we see for the from the benchmark it's just as good as Opus 5.5, but on just one generation out of five.

s93
00:08:54.320 --> 00:08:58.839
So, we want to measure for that and the worse the model is, the widest the variance.

s94
00:08:58.839 --> 00:09:03.280
So, uh we have some learnings about that that I'll share at the end as well.

s95
00:09:03.280 --> 00:09:14.560
But yeah, it's we we found that five drafts seems to be the good middle ground in terms of of cost for us to execute and to measure this this variability.

s96
00:09:14.800 --> 00:09:19.960
So, what do we measure exactly and how do you even quantify writing quality?

s97
00:09:19.960 --> 00:09:25.200
That's been the challenge and we worked on it for for quite some time.

s98
00:09:25.200 --> 00:09:30.320
Right now, it's all open on the benchmark and there are way more details on the benchmark page as well.

s99
00:09:30.320 --> 00:09:42.560
But what we measure is nine different metrics that we detailed that that we really detailed, but I can condense them in five main aspects.

s100
00:09:42.560 --> 00:09:44.200
First, we measure the tone.

s101
00:09:44.200 --> 00:09:45.600
So, does it sound like me?

s102
00:09:45.600 --> 00:09:49.920
Which means that like can the model adapt to a specific tone?

s103
00:09:49.920 --> 00:09:53.520
Second, we measure just pure is it good writing?

s104
00:09:53.520 --> 00:10:02.600
Uh does it have like good hooks, good emotion emotional transition, and all the the aspects that we're looking in a in a good story.

s105
00:10:02.680 --> 00:10:04.600
Then, we measure if it follows instructions.

s106
00:10:04.600 --> 00:10:09.680
Did it follow the the length the the expected length, the format expected?

s107
00:10:09.680 --> 00:10:15.560
Uh did it put the right images and right right things in the right place?

s108
00:10:15.720 --> 00:10:19.600
Fourth, we measure for research accuracy.

s109
00:10:19.800 --> 00:10:28.360
So, uh we have a research brief along with each script, and we just measure if it hallucinated or used the research correctly.

s110
00:10:28.360 --> 00:10:35.200
And lastly, we measure for slap using EQ-Bench slap score, but also our own measurements.

s111
00:10:35.200 --> 00:10:41.080
And what uh this looks in terms of what the judges see is more like that.

s112
00:10:41.080 --> 00:10:54.080
We have around a thousand words of examples and uh and instructions and directions around each of the metric that we have along with the script to analyze, my own

s113
00:10:54.080 --> 00:10:59.720
uh script as some sort of of guidance, and the brief and the research.

s114
00:10:59.720 --> 00:11:08.560
So, all in all, it's a 14,000 prompt that we send to to every judge to then get uh the feedback that we'll dive in next.

s115
00:11:08.560 --> 00:11:13.200
And uh so so the judge just analyzes the writing quality.

s116
00:11:13.200 --> 00:11:17.360
It doesn't focus on length uh and doesn't know which model is writing.

s117
00:11:17.360 --> 00:11:21.800
Uh we just do that after with with code itself.

s118
00:11:21.800 --> 00:11:35.560
On one metric like tone and voice, which is our main metric, it looks a bit like this where we have a lot of detailed descriptions of what is my tone and what to expect and how to analyze this.

s119
00:11:35.560 --> 00:11:45.720
Then the judge will categorize between six categories first and after having found a category, it will give a rating.

s120
00:11:45.720 --> 00:11:58.160
Uh we found that this is the best way to measure for this more qualitative task from uh just binary that we usually suggest to use or uh directly categories or directly percentage.

s121
00:11:58.160 --> 00:12:01.920
We found that this actually works really well.

s122
00:12:01.920 --> 00:12:11.400
And uh it has tons of examples of what each score means in terms of of writing for each of the task.

s123
00:12:11.400 --> 00:12:14.680
And what the judge sent back is something like this.

s124
00:12:14.680 --> 00:12:28.160
It has the nine different metrics scored with a justification for for the score and an example of like uh I I wrote this score because of this example in particular.

s125
00:12:28.160 --> 00:12:31.800
But the main thing is that all of this is automated.

s126
00:12:31.800 --> 00:12:38.160
Uh obviously because we with 120 models, we we generate five drafts each for 10 tasks.

s127
00:12:38.160 --> 00:12:44.040
It quickly goes to the thousands of scripts to review, so we cannot really review this ourselves.

s128
00:12:44.040 --> 00:12:49.320
And the problem with LLM judges is that they have preferences.

s129
00:12:49.320 --> 00:12:51.440
Uh some models prefer themselves.

s130
00:12:51.440 --> 00:12:53.640
Most models prefer themselves.

s131
00:12:53.640 --> 00:12:56.080
And uh particularly OpenAI.

s132
00:12:56.080 --> 00:13:08.480
And what I found quite funny when studying this is that it seems like Anthropic and OpenAI are really in a in a war here where they seem to have past training on hating each other,

s133
00:13:08.480 --> 00:13:10.760
which is quite funny.

s134
00:13:10.760 --> 00:13:20.120
But all this to say that you really want to check for when you have when you use LLM judges, like using GPT for instance, if you want to know which model is best for your task,

s135
00:13:20.120 --> 00:13:33.160
uh be careful and use different model family just to ensure cuz we see here that OpenAI really has a strong preference on its own family of models, not just one model, but all OpenAI models.

s136
00:13:33.960 --> 00:13:46.160
And so to fix also for that, we listened to a few comments that I got on X, so just thing to use human for a creative task, which would be ideal, but quite impractical.

s137
00:13:46.160 --> 00:14:02.040
We still tried to to do that in a scalable way, where right now we compared manually me and three of my teammates 1,000 AB pairs of degenerated scripts, not the full scripts, but like

s138
00:14:02.040 --> 00:14:05.960
a good chunk that is comparable between two scripts, two generations.

s139
00:14:05.960 --> 00:14:15.680
And hopefully we'll do more and I want to publicize this also for to get help from the community in the future, but that's not right now.

s140
00:14:15.680 --> 00:14:19.080
So, what did it cost us and what's the workload behind that?

s141
00:14:19.080 --> 00:14:31.880
For me, for us, it was a kickoff cost in terms of hours of around 500 hours to build the 10 scripts, define all the rubrics, find the examples, try to

s142
00:14:31.880 --> 00:14:36.200
quantify quality, which is very difficult.

s143
00:14:36.200 --> 00:14:48.520
And then I just iterated myself with four different 20X subscription for 4 months to build a benchmark and all the other things and experiment and change the the the metrics

s144
00:14:48.520 --> 00:14:51.200
ratio and all these things.

s145
00:14:51.200 --> 00:14:55.360
And so we just spent thousands of dollars there.

s146
00:14:55.360 --> 00:15:02.839
And if you want to do this a similar type of benchmark, you need to know that it cost quite a lot.

s147
00:15:02.839 --> 00:15:12.400
Right now, if I just change one thing in the rubrics or how the judge rate generations, it will cost us more than $2,000.

s148
00:15:12.400 --> 00:15:20.120
And if I just change one line in an input prompt for a model when it generates, it cost us more than $4,000.

s149
00:15:20.120 --> 00:15:24.280
So, it's very uh very expensive to experiment and change these things in the prompt.

s150
00:15:24.280 --> 00:15:28.000
So, you want to do this uh carefully.

s151
00:15:28.760 --> 00:15:33.280
Now, what can we get out of this benchmark, out of Tone Bench?

s152
00:15:33.280 --> 00:15:36.120
Uh you can get tons of things on the the website itself.

s153
00:15:36.120 --> 00:15:39.160
There's like a lot of details.

s154
00:15:39.160 --> 00:15:50.320
But, importantly, uh I want to share that what I've learned from the past 4 months is that you don't want to pick for for creative writing is that you don't want to pick the model that will generate a one-shot

s155
00:15:50.320 --> 00:15:51.760
amazing post.

s156
00:15:51.760 --> 00:15:57.840
You just want to find a model that will be the least annoying to you to get to where you want to be.

s157
00:15:57.840 --> 00:16:02.280
So, uh like there won't be any perfection anytime soon, I think.

s158
00:16:02.280 --> 00:16:10.400
And what that means right now on the benchmark, it means at scale GLM 5.3 flash is extremely good, extremely cheap.

s159
00:16:10.400 --> 00:16:21.600
It's 50 times cheaper than the best one, which is Opus 5.5 extra high, which is uh 2.5 times cheaper than Max thinking, but with same results.

s160
00:16:21.600 --> 00:16:24.920
So, you don't want to use Max for writing, usually.

s161
00:16:24.920 --> 00:16:36.440
Um And otherwise, for the best uh how to write with models that we got out of this is to first have an extremely long and detailed prompt.

s162
00:16:36.440 --> 00:16:40.280
It cannot be long enough, and the more the better.

s163
00:16:40.280 --> 00:16:45.240
You want to add every example you can think of of what to do and how to do it.

s164
00:16:45.240 --> 00:16:52.920
Same thing for negative examples of things you don't like and that doesn't sound right or the words you don't want, everything.

s165
00:16:52.960 --> 00:17:01.000
You you can share every If you write newsletters or whatever, you can share it in full, and it will just be better as well.

s166
00:17:01.000 --> 00:17:09.880
And what we've seen that really really helps is to use a high-thinking judge just one time, one to three times, otherwise it will loop into slap.

s167
00:17:09.880 --> 00:17:17.640
So, just one review of another model with the same instructions, directions that will give feedback to the writer.

s168
00:17:17.640 --> 00:17:24.360
Really like I don't have quantitative metrics on that, but it it helps a lot.

s169
00:17:24.360 --> 00:17:38.400
And for a company, if you work with APIs and want to do this at have the best results at scale possible, what I suggest is to generate several drafts, like 10 drafts of one specific thing

s170
00:17:38.400 --> 00:17:41.680
with a cheaper model like GLM 5.3 flash.

s171
00:17:41.680 --> 00:17:51.760
And then have a smarter high-thinking judge like Opus or or whatever you want pick and merge the best parts of each of this these drafts.

s172
00:17:51.760 --> 00:18:03.800
Which I think it will still land way way less expensive than just one generation by Opus 5.5. And I think will lead to better results from what I've seen.

s173
00:18:04.240 --> 00:18:05.920
And lastly, you want a human review.

s174
00:18:05.920 --> 00:18:08.680
You cannot really go around that.

s175
00:18:08.680 --> 00:18:17.200
But all of these benchmark measures obviously my voice, so it's super subjective, but I and it's just one that that point you don't want to fully trust it blindly.

s176
00:18:17.200 --> 00:18:20.160
Doesn't mean Opus 5.5 is the best model.

s177
00:18:20.160 --> 00:18:28.160
So, but ideally it's a strong signal to find which model is good at writing and which is less sloppy.

s178
00:18:28.160 --> 00:18:32.000
Um I suggest you to build your own evaluation pipelines.

s179
00:18:32.000 --> 00:18:33.320
If you want we can help.

s180
00:18:33.320 --> 00:18:35.280
You can chat or reach out.

s181
00:18:35.280 --> 00:18:37.080
And regarding the benchmark, it's live.

s182
00:18:37.080 --> 00:18:40.640
We just like right now published it and released it.

s183
00:18:40.640 --> 00:18:49.800
So, there are tons of graphs, tons of insights, tons of comparisons, and details on the implementation, full methodology.

s184
00:18:49.800 --> 00:18:53.560
So, just check it out and hopefully it's useful.

s185
00:18:53.560 --> 00:18:55.127
Thank you.

s186
00:18:55.127 --> 00:18:57.127
[applause]