{ "video": { "id": "lCBf9slCanI", "title": "Ending AI Slop — Thais Castello Branco, Taste Labs", "duration": 990, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "lCBf9slCanI", "title": "Ending AI Slop — Thais Castello Branco, Taste Labs", "one_liner": "Taste Labs founder Thais Castello Branco argues AI slop is a measurability problem: models are good at code because code decomposes, verifies and executes — so to fix design and writing you must decompose fuzzy qualities like 'on brand' into codifiable ground truth, and route what's left to human judgment rather than collapsing it to the mean.", "summary": "The talk claims subjective domains (design, creative writing, personality, emotional intelligence) lag because of two properties: capability follows measurability, and models predict the most likely next outcome while in subjective work the optimal answer lives at the ends of the distribution. Castello Branco shows how to pull a fuzzy capability toward verification — 'make something great' is untrainable, but 'make something on brand' decomposes into colors, typography, motion, animation and texture, which becomes the ground truth for an RL environment where an agent must produce a net-new page graded against those components rather than against the original. Whatever resists that decomposition is a human-judgment problem better solved with data, and she describes deliberately forcing preference distributions across 1,000+ expert designers, attaching preference vectors to preference data, and QA'ing data by specificity and by tying expert commentary to the exact code component. The closing message is quality over quantity.", "key_points": [ "Code's verifiability is 'a fact about code', not a property of models — code decomposes, verifies and executes, which is why training on it is easy; design and writing offer no equivalent handle.", "Two reasons subjective domains are harder: capability follows measurability, and mode collapse to the mean — where in math the average answer is also the optimal one, but in writing and design it is not.", "'What is great design' is unanswerable because it is contextual (the same slide is amazing for a startup and inappropriate for a finance firm) and changes over time (good today ≠ good five years ago), unlike code or math.", "Brand is the worked example: 'create something great' is undefinable, but 'make something on brand' is decomposable into colors, typography, motion, animation and textures — codified elements you can verify against.", "LLM-as-a-judge is not always the right method for brand adherence — she cites reward hacking and 'interesting hallucination patterns'; instead the decomposition itself becomes the ground truth.", "The RL task design is the hardest part: the agent is asked to create a net-new page adhering to the brand, and the output is graded against the decomposed ground truth rather than against the original, so valid novel recombinations still score.", "Design is a spectrum and therefore a routing problem: vision, alignment and typography sit near objective and suit programmatic verification; style fit and creativity sit at the far end and need human judgment.", "Taste Labs works with a community of 1,000+ expert designers across different mediums and styles, deliberately forcing the distribution to prevent mode collapse; old-style preference data collapsed to the mean because it was collected without knowing who the raters were or why they liked something.", "Data-quality QA levers she considers controllable: specificity of the expert's reasoning language, and tying an expert's commentary to the exact code component it refers to — because models struggle to connect a piece of code to the visual.", "Expert disagreement is diagnostic, not automatically noise: disagreement on alignment (near-objective) signals a data flaw, while disagreement on style or aesthetics is good data showing a real preference distinction." ], "takeaways": [ "Before trying to train or evaluate a subjective capability, decompose it into named, observable components and treat that decomposition as the ground truth — pull the task toward verification instead of asking a judge 'is this good?'", "Route each sub-problem to the method that fits it: programmatically verifiable pieces to RL environments, contextual/time-dependent/preference pieces to human-judgment data, and some things to context and user-intent modelling at the application layer rather than the foundation model.", "Don't grade generative outputs by similarity to the reference; design the task so a net-new artifact is scored against the decomposed criteria, or you train replication instead of capability.", "Stop averaging preference data — capture who the rater is as a preference vector so pluralism survives, and deliberately force expert distribution across styles and mediums instead of collecting from an undifferentiated crowd.", "QA data on properties that correlate with signal (specificity of reasoning, commentary anchored to the exact code component), and use consensus checks to separate real taste disagreement from actual data flaws.", "Choose quality over quantity: a smaller set of high-quality data from genuinely high-taste domain experts outperforms a large noisy, unintentional collection." ], "topics": [ "ai-slop", "rl-environments", "evals", "post-training", "data-quality", "preference-data", "llm-as-judge", "design" ], "tools": [ "Taste Labs", "Reduct (brand used as the worked example)", "LLM-as-a-judge", "AI Engineer (conference/channel)" ], "quotes": [ { "text": "we treat for example the fact that code is verifiable and measurable as something that is a property about models and models are great at coding um because we've made them great at coding but realistically it's actually a fact about code. Code is something that decomposes, it verifies, it executes", "at": "02:29", "url": "https://www.youtube.com/watch?v=lCBf9slCanI&t=149s" }, { "text": "capability follows measurability. So if we can solve the measurability problem or at least part of it, then we can solve a big portion of these domains.", "at": "03:07", "url": "https://www.youtube.com/watch?v=lCBf9slCanI&t=187s" }, { "text": "a lot of greatness and creativity happens actually at the ends of the distribution. It's not the most likely outcome. It's when you actually actively break from rules and actively break from patterns that you can create things that are subjective and great.", "at": "08:11", "url": "https://www.youtube.com/watch?v=lCBf9slCanI&t=491s" }, { "text": "we need to fundamentally understand that the world is multi-preference and how do we do that matching accordingly", "at": "11:17", "url": "https://www.youtube.com/watch?v=lCBf9slCanI&t=677s" } ], "words": 3856 }, "summary_url": "/#lCBf9slCanI", "transcript": { "html": "/transcripts/lCBf9slCanI.html", "txt": "/transcripts/lCBf9slCanI.txt", "vtt": "/transcripts/lCBf9slCanI.vtt" } }