{ "video": { "id": "_dTF3NpPzGo", "title": "Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI", "duration": 1142, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "_dTF3NpPzGo", "title": "Why AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI", "one_liner": "For creative writing, don't hunt for the model that one-shots a great draft; pick the one that is least annoying to get where you want. On Tone-Bench, that meant cheap GLM 5.3 flash drafts at scale, merged by one high-thinking judge (Opus), with a human review at the end.", "summary": "Bouchard argues that slop doesn't go away as models improve, so it has to be defined and measured instead of trained away. He breaks slop into six traits: overused words, too general, saying nothing, repetition, hallucination, and wrong or no tone. He then presents Tone-Bench, released by Towards AI during the talk. It measures whether a model can write a video script in his voice. Each model writes 5 drafts for each of 10 tasks, four LLM judges score them on 9 metrics, and the scores are checked against 1,000 human A/B comparisons. He also covers what it cost, judge bias, and practical advice on writing with LLMs.", "key_points": [ "Slop has six parts: overused LLM words (e.g. 'delve'), being too general ('most teams, most projects'), saying nothing, repeating itself, hallucinating, and having the wrong tone or no tone ('everyone's tone at the same time').", "Tone-Bench gives each model a long prompt (a brief, research, his style, examples of his past content, storytelling guidance) and asks for a full video script. There are 10 scripts in the benchmark. Some have been published and some haven't, and they don't say which, so they can check for contamination.", "Each model writes 5 drafts per task to measure variance within the model. Grok 4.7 matched Opus 5.5 on just one of five generations, and the worse the model, the wider the variance. Five drafts was their trade-off between cost and measuring variability.", "The 9 metrics fall into 5 groups: tone/voice (the main metric), writing quality (hooks, emotional transitions), instruction following (length, format, image placement), research accuracy against the brief, and slop (EQ-Bench slop score plus their own measures).", "The judge prompt is about 14,000 tokens (the talk says '14,000 prompt'). It includes about 1,000 words of examples and instructions per metric, his own script as a reference, and the brief and research. Judges don't know which model wrote the script, and length is checked separately in code. Judges first pick one of six categories, then give a rating. He found this worked better than binary, category-only or percentage scoring for this task.", "LLM judges prefer their own model family, OpenAI's most of all, and Anthropic and OpenAI models seem to 'hate each other'. To check the judges, he and three teammates hand-compared 1,000 A/B pairs of comparable script chunks.", "Building it took about 500 hours up front (10 scripts, rubrics, examples), then 4 months of iteration on four '20X' subscriptions, costing thousands of dollars. Now one change to a rubric or the judging costs more than $2,000 to rerun, and one changed line in the generation prompt costs more than $4,000.", "Results: GLM 5.3 flash is very good and 50x cheaper than the top model, Opus 5.5 extra high. Opus 5.5 extra high matches Max thinking at 2.5x lower cost, so 'you don't want to use Max for writing'." ], "takeaways": [ "Write very long, detailed prompts: every positive example you can find, negative examples and banned words, and whole newsletters or past pieces in your voice. 'It cannot be long enough.'", "Have one high-thinking model review the draft with the same instructions and give feedback to the writer. Do it once, at most three times, or it 'will loop into slop'.", "At scale over APIs, generate around 10 drafts with a cheap model like GLM 5.3 flash, then have a smarter high-thinking model like Opus merge the best parts. He expects this to cost much less than one Opus 5.5 generation and give better results.", "When you build your own evals, generate several outputs per prompt to measure variance, and use judges from different model families, because judges favour their own family.", "Keep a human review step, and build your own evaluation pipeline for your own voice. Tone-Bench measures his voice and is a signal, not proof of which model is best." ], "topics": [ "llm writing quality", "slop", "benchmarks", "evals", "llm-as-judge", "judge bias", "prompting", "cost" ], "tools": [ "Tone-Bench", "EQ-Bench slop score", "Towards AI", "Opus 5.5", "GLM 5.3 flash", "Grok 4.7", "GPT-6.1", "Fable", "OpenAI", "Anthropic", "LinkedIn", "X" ], "quotes": [ { "text": "if I can read it out loud like right now without cringing because of the the the the words that I wouldn't use myself.", "at": "06:03", "url": "https://www.youtube.com/watch?v=_dTF3NpPzGo&t=363s" }, { "text": "if I just change one line in an input prompt for a model when it generates, it cost us more than $4,000.", "at": "15:14", "url": "https://www.youtube.com/watch?v=_dTF3NpPzGo&t=914s" }, { "text": "you don't want to pick the model that will generate a one-shot amazing post. You just want to find a model that will be the least annoying to you to get to where you want to be.", "at": "15:47", "url": "https://www.youtube.com/watch?v=_dTF3NpPzGo&t=947s" }, { "text": "It cannot be long enough, and the more the better.", "at": "16:37", "url": "https://www.youtube.com/watch?v=_dTF3NpPzGo&t=997s" } ], "words": 3503 }, "summary_url": "/#_dTF3NpPzGo", "transcript": { "html": "/transcripts/_dTF3NpPzGo.html", "txt": "/transcripts/_dTF3NpPzGo.txt", "vtt": "/transcripts/_dTF3NpPzGo.vtt" } }