Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI

AI Engineer · 19 min · 186 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12So, I'm here to talk about all these words that we call slub and more importantly, I want to try to help fixing what Karpathy termed as the the slub apocalypse
00:29that he thought would start in 2026 and I think we've seen that on social media.
00:36So, uh to try to fix this and to stop slub, we first need to define it.
00:43So first, what is slub?
00:45There are tons of different definitions from the various experts in the field, but what I think we can all agree is that it really sucks to read something that feels generated.
00:57And it usually looks like this with uh the the words that we know of, the delve and uh the generaties and other things like these and the main problem
01:07is that even as models get better with fable uh GPT-6.1 and all the the better models, slub doesn't seem to go go away.
01:15It just keeps on scaling.
01:18So, we need something different than just training the models more, it seems.
01:24And I really want to try to at least first define what slub is so that we can understand why it's there.
01:31And so to me, slub is made of six main thing.
01:35First, there are tons of overused words by language models that we don't really use like delve that we really know well.
01:42Second, it's usually too general.
01:44It's always talking about most themes, most projects.
01:48It doesn't share nothing specific.
01:50And I'm myself guilty of using that on LinkedIn as well, which I guess it's the LinkedIn meta, so it's it's normal there.
01:59And worse slop sloppy text usually says nothing.
02:04It's just like nice words put together that we don't even usually use.
02:09And even worse than that, it says nothing, but it also repeats itself a lot.
02:14So it repeats nothing and it's too general.
02:17And uh even worse, it hallucinates.
02:22As we know, models hallucinate, so the sloppy text also we can't really fully trust it.
02:28And lastly, the worst to me is that it usually has the wrong tone or at least it has no tone and like everyone's tone at the same time.
02:36So it just doesn't really talk like you.
02:38It doesn't share stuff that you would share.
02:41So my goal here today and with this project that I've been working on is to first measure this sloppy writing.
02:50Uh not just slop words.
02:51I don't want to rebuild EQ-Bench slop score.
02:54It's just it's really about measuring how sloppy is a writing.
02:59So in order to do that, we are well today right now releasing a new benchmark that we called Tone-Bench.
03:06Which uh the whole goal is the of the benchmark is to measure if the new models or models are good at creative writing, which I will try to define as well.
03:16So here's the plan for today.
03:19Uh the plan is to justify do we even need another benchmark?
03:23Then how it works, what we measure, what how we built it, and I want to share like the cost of building it, our decisions, what we learned already from
03:33for writing with AI from the benchmark, and share some more stuff about this.
03:39So with that, let's delve in.
03:42So the first First thing, do we even need another benchmark again?
03:49And just looking at LinkedIn and X X replies, my replies on X, I think we do.
03:55It's like people are spamming more and more generated content.
03:59So, at least have them be decently good.
04:02And uh more precisely, I also really need it.
04:07We are a small team uh at Towards AI, my company, and we have one side of the company that is focused on education.
04:14We build tons of educational content, mostly for AI engineering.
04:18And as a small team, obviously we use AI as well for that.
04:21And it's been extremely annoying to review this slop uh constantly and try to fix it.
04:26We uh I published hundreds of videos.
04:29We published thousands of articles.
04:31We released a book.
04:32We are working on a second one uh for AI engineering.
04:36[snorts]
04:36And so, we really need to have the best models possible that will help us write better uh and not be full of slop.
04:46And another nice uh addition to that is that I've been sharing about this benchmark on on X for quite some time as I've been building it.
04:55And it got really nice engagement, and people seem to confirm that we did need some sort of more creative uh writing benchmark like this one.
05:08And lastly, just to prove that I think we are the right team to work on this benchmark, we well, I've been build evaluating AI models since my master's and then PhD.
05:21And now at Towards AI, we do build evaluation pipelines for our clients.
05:25We have 15 AI engineers with us, and we have creative writers, editors uh on board.
05:32So, we already have that for our content.
05:35So, I think uh well, I wanted to leverage that to build a benchmark and help the community even more.
05:42So, what did we build exactly?
05:44Hopefully, I I try to share as much as possible, and I'm happy to chat if we can help uh even build your own or if I can just share some some of our learnings.
05:54So, what we built is super subjective.
05:57It's a way to measure if a model can write like me.
06:01And how I define that is the is just that if I can read it out loud like right now without cringing because of the the the the words that I wouldn't use myself.
06:13So, the process is very simple.
06:15The the benchmark the process benchmark the benchmark process is very simple.
06:20We have a prompt that is the brief and the research that we send to a model, and hopefully it will generate entirely a video script that I could have written.
06:32Uh obviously that doesn't happen.
06:34It really sucks usually, but that's the whole goal, and we measure that that final script against our rubrics and um just our whole evaluation pipeline.
06:45And the prompt that we send to every model in the benchmark is much more in-depth than that.
06:51It it looks more like this where we have a lot of details on how I talk, how I write, all tons of examples of my previous content videos.
07:01We have a lot of information on storytelling in general, active sentences, and just how to write well.
07:09And we have uh directions on what to generate and what to do for this current task, which in the the model would generate one script.
07:17And this is just one of them.
07:19In the current benchmark, we have 10 different scripts uh that we worked really hard to make as good as possible, and we built rubrics for them to help um
07:30as we will see LLM judges uh rate these these generations.
07:35So, we have 10 different scripts.
07:36Some of them are published, some are not.
07:38We don't say which ones are the benchmarks so that we check for contamination as well.
07:44And then what do we do with these 10 scripts?
07:49We take all the models out there and we try to measure their generations.
07:55What that means is that for every single model we will do 10 times this process, generate the article from the prompt.
08:05In fact, we generate five of them.
08:06I will very soon share why.
08:08Then we have four different LLM judges analyze them.
08:12I recently added Jeff as a test.
08:14Not sure if I will keep that, but I I I added it just to see if it worked well.
08:20And then we calculate in the end a final score for the leaderboard.
08:25I will go through all of these steps from left to right.
08:28So, first, why do we generate even five scripts?
08:31It's to check for intra-model variability, which I highly suggest if you are building your own evaluations because especially I guess on a creative task, sometimes as we see here with Grok 4.7,
08:43some of the generations are outliers where it's as we see for the from the benchmark it's just as good as Opus 5.5, but on just one generation out of five.
08:54So, we want to measure for that and the worse the model is, the widest the variance.
08:58So, uh we have some learnings about that that I'll share at the end as well.
09:03But yeah, it's we we found that five drafts seems to be the good middle ground in terms of of cost for us to execute and to measure this this variability.
09:14So, what do we measure exactly and how do you even quantify writing quality?
09:19That's been the challenge and we worked on it for for quite some time.
09:25Right now, it's all open on the benchmark and there are way more details on the benchmark page as well.
09:30But what we measure is nine different metrics that we detailed that that we really detailed, but I can condense them in five main aspects.
09:42First, we measure the tone.
09:44So, does it sound like me?
09:45Which means that like can the model adapt to a specific tone?
09:49Second, we measure just pure is it good writing?
09:53Uh does it have like good hooks, good emotion emotional transition, and all the the aspects that we're looking in a in a good story.
10:02Then, we measure if it follows instructions.
10:04Did it follow the the length the the expected length, the format expected?
10:09Uh did it put the right images and right right things in the right place?
10:15Fourth, we measure for research accuracy.
10:19So, uh we have a research brief along with each script, and we just measure if it hallucinated or used the research correctly.
10:28And lastly, we measure for slap using EQ-Bench slap score, but also our own measurements.
10:35And what uh this looks in terms of what the judges see is more like that.
10:41We have around a thousand words of examples and uh and instructions and directions around each of the metric that we have along with the script to analyze, my own
10:54uh script as some sort of of guidance, and the brief and the research.
10:59So, all in all, it's a 14,000 prompt that we send to to every judge to then get uh the feedback that we'll dive in next.
11:08And uh so so the judge just analyzes the writing quality.
11:13It doesn't focus on length uh and doesn't know which model is writing.
11:17Uh we just do that after with with code itself.
11:21On one metric like tone and voice, which is our main metric, it looks a bit like this where we have a lot of detailed descriptions of what is my tone and what to expect and how to analyze this.
11:35Then the judge will categorize between six categories first and after having found a category, it will give a rating.
11:45Uh we found that this is the best way to measure for this more qualitative task from uh just binary that we usually suggest to use or uh directly categories or directly percentage.
11:58We found that this actually works really well.
12:01And uh it has tons of examples of what each score means in terms of of writing for each of the task.
12:11And what the judge sent back is something like this.
12:14It has the nine different metrics scored with a justification for for the score and an example of like uh I I wrote this score because of this example in particular.
12:28But the main thing is that all of this is automated.
12:31Uh obviously because we with 120 models, we we generate five drafts each for 10 tasks.
12:38It quickly goes to the thousands of scripts to review, so we cannot really review this ourselves.
12:44And the problem with LLM judges is that they have preferences.
12:49Uh some models prefer themselves.
12:51Most models prefer themselves.
12:53And uh particularly OpenAI.
12:56And what I found quite funny when studying this is that it seems like Anthropic and OpenAI are really in a in a war here where they seem to have past training on hating each other,
13:08which is quite funny.
13:10But all this to say that you really want to check for when you have when you use LLM judges, like using GPT for instance, if you want to know which model is best for your task,
13:20uh be careful and use different model family just to ensure cuz we see here that OpenAI really has a strong preference on its own family of models, not just one model, but all OpenAI models.
13:33And so to fix also for that, we listened to a few comments that I got on X, so just thing to use human for a creative task, which would be ideal, but quite impractical.
13:46We still tried to to do that in a scalable way, where right now we compared manually me and three of my teammates 1,000 AB pairs of degenerated scripts, not the full scripts, but like
14:02a good chunk that is comparable between two scripts, two generations.
14:05And hopefully we'll do more and I want to publicize this also for to get help from the community in the future, but that's not right now.
14:15So, what did it cost us and what's the workload behind that?
14:19For me, for us, it was a kickoff cost in terms of hours of around 500 hours to build the 10 scripts, define all the rubrics, find the examples, try to
14:31quantify quality, which is very difficult.
14:36And then I just iterated myself with four different 20X subscription for 4 months to build a benchmark and all the other things and experiment and change the the the metrics
14:48ratio and all these things.
14:51And so we just spent thousands of dollars there.
14:55And if you want to do this a similar type of benchmark, you need to know that it cost quite a lot.
15:02Right now, if I just change one thing in the rubrics or how the judge rate generations, it will cost us more than $2,000.
15:12And if I just change one line in an input prompt for a model when it generates, it cost us more than $4,000.
15:20So, it's very uh very expensive to experiment and change these things in the prompt.
15:24So, you want to do this uh carefully.
15:28Now, what can we get out of this benchmark, out of Tone Bench?
15:33Uh you can get tons of things on the the website itself.
15:36There's like a lot of details.
15:39But, importantly, uh I want to share that what I've learned from the past 4 months is that you don't want to pick for for creative writing is that you don't want to pick the model that will generate a one-shot
15:50amazing post.
15:51You just want to find a model that will be the least annoying to you to get to where you want to be.
15:57So, uh like there won't be any perfection anytime soon, I think.
16:02And what that means right now on the benchmark, it means at scale GLM 5.3 flash is extremely good, extremely cheap.
16:10It's 50 times cheaper than the best one, which is Opus 5.5 extra high, which is uh 2.5 times cheaper than Max thinking, but with same results.
16:21So, you don't want to use Max for writing, usually.
16:24Um And otherwise, for the best uh how to write with models that we got out of this is to first have an extremely long and detailed prompt.
16:36It cannot be long enough, and the more the better.
16:40You want to add every example you can think of of what to do and how to do it.
16:45Same thing for negative examples of things you don't like and that doesn't sound right or the words you don't want, everything.
16:52You you can share every If you write newsletters or whatever, you can share it in full, and it will just be better as well.
17:01And what we've seen that really really helps is to use a high-thinking judge just one time, one to three times, otherwise it will loop into slap.
17:09So, just one review of another model with the same instructions, directions that will give feedback to the writer.
17:17Really like I don't have quantitative metrics on that, but it it helps a lot.
17:24And for a company, if you work with APIs and want to do this at have the best results at scale possible, what I suggest is to generate several drafts, like 10 drafts of one specific thing
17:38with a cheaper model like GLM 5.3 flash.
17:41And then have a smarter high-thinking judge like Opus or or whatever you want pick and merge the best parts of each of this these drafts.
17:51Which I think it will still land way way less expensive than just one generation by Opus 5.5. And I think will lead to better results from what I've seen.
18:04And lastly, you want a human review.
18:05You cannot really go around that.
18:08But all of these benchmark measures obviously my voice, so it's super subjective, but I and it's just one that that point you don't want to fully trust it blindly.
18:17Doesn't mean Opus 5.5 is the best model.
18:20So, but ideally it's a strong signal to find which model is good at writing and which is less sloppy.
18:28Um I suggest you to build your own evaluation pipelines.
18:32If you want we can help.
18:33You can chat or reach out.
18:35And regarding the benchmark, it's live.
18:37We just like right now published it and released it.
18:40So, there are tons of graphs, tons of insights, tons of comparisons, and details on the implementation, full methodology.
18:49So, just check it out and hopefully it's useful.
18:53Thank you.
18:55[applause]