Training Taste — Thais Castello Branco, Taste Labs
https://www.youtube.com/watch?v=sDMGWK4wZ_w
AI Engineer · 15 min · 187 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] Test.
[00:13] Okay, amazing.
[00:15] It's great to meet everyone.
[00:16] I'm Taís.
[00:16] I'm the founder of Taste Labs.
[00:19] Uh for those of you who don't know us, we came out of stealth a few weeks ago and our whole mission is basically how do we end AI slop?
[00:25] I that's my personal enemy.
[00:27] Um and so we really believe that to solve this problem of slop, we have to like decode subjective domains.
[00:34] Uh there's been so much effort being put into getting models and agents amazing at things like coding and math.
[00:39] Uh and it's time that we put all that same effort into making them great at things like design uh and writing.
[00:44] And so design is this first pillar that we're starting with and it's been it's been incredibly exciting.
[00:49] Um We work primarily in two ways.
[00:51] So we work a lot with the frontier labs on how do we evaluate their models, understand where they're breaking, understand what could be better about them, and then construct the right either post-training data or RL environments to basically fix that problem.
[01:04] And part of this is like how do you take something as fuzzy and large as design and break it down to a level that you can identify what is best solved through each method.
[01:13] What are elements of design that are almost like once you kind of boil down the problem becomes so specific that they almost become deterministic.
[01:19] So for example, uh if you're trying to train a model to be good at selecting color palettes or have contrast or alignment, those are things that if you define the problem and the context in a specific enough way,
[01:29] uh you you can get to an answer that's like pretty objective or that at least most experts would agree to.
[01:33] But maybe other things like uh aesthetics, you naturally will see this expert disagreement.
[01:38] And so then you want to lean on to things that are closer to to data.
[01:41] So anyway, we spend a lot of time thinking about all those problems.
[01:44] Uh but on the other side is also without even touching the model layer, right?
[01:47] How do we actually help agents and app layer companies produce better things?
[01:52] And there's a lot that goes into that, right?
[01:53] You have these different sets of problems at the application layer because you're using an off-the-shelf model that tends to collapse in terms uh uh of style tends to collapse to the mean.
[02:01] So, how do we force that creativity back to the system?
[02:04] How do we avoid these patterns of slop, which we'll talk about a lot today?
[02:07] Uh how do you understand like user preferences or brand preferences preference so that you can uh maintain endurance to that style?
[02:15] Uh so, there's lots of things that are actually need to be solved as context or judgment or verification at the app layer, which is why we kind of work across both.
[02:25] Maybe I'll start with more of a a philosophical question of like how how do you define something that is great?
[02:30] Like how do you define greatness?
[02:32] And for something like math, it's easier, right?
[02:34] Because there's kind of one objective answer, and uh great is the same as correct.
[02:39] But then for something like writing or design, it's much harder, right?
[02:43] Like how do you define what's like a great tweet or what's a great art piece or what's a great website?
[02:49] Um I don't know what's the last time that you interacted with a poem or walked into a coffee shop and for some reason it kind of like hit different and it felt
[02:56] very special.
[02:58] Uh but probably it's a combination of things that it it felt very unique.
[03:01] It felt almost a little different.
[03:02] It kind of called your attention.
[03:03] Uh it felt like there it was made with a lot of care and attention to to detail and craft, and it almost had the sense of of like authenticity.
[03:11] Um and I think that's a lot of what AI is missing today.
[03:13] It's like how do we take uh things that are not necessarily average, right?
[03:16] How do we produce things that are purposely like out of distribution?
[03:19] Um and slop is kind of the opposite of that, right?
[03:22] I think it is hard to define what is great sometimes, but I think it's pretty pretty easy to define what is slop in the sense that most people would agree.
[03:28] I think the sense of like repetition of kind of soullessness is something that all of us feel right now when using AI, and I think it's quite magical, by the way, that
[03:36] AI has gotten to a point that any human on the planet that is not even a designer, that is not an engineer, can click a button and suddenly make an entire PowerPoint or make a website or make a web app.
[03:46] That's pretty cool.
[03:47] But it comes with consequences, right?
[03:49] Uh it comes with consequences of suddenly now the cost of generation is basically going to zero.
[03:54] Uh and But the average person hasn't necessarily honed their taste.
[03:58] Like I does think about the amount of effort and work that a designer puts in throughout their life to like build up their taste, right?
[04:04] Like there's all this process of like getting exposed to many things and learning to like spot patterns and learning to develop a point of view and like kind of do things in a in a courageous way that maybe are a little bit against the norm.
[04:15] Learning what not to do and how to like have restraint and that's very hard.
[04:19] Like the average person doesn't necessarily have the the time nor the skills to go and develop taste in everything, let's say in design.
[04:26] And so um I think it would be a bad case scenario for us to just like be like, "Okay, the way to fix slop is for everyone to have taste."
[04:32] cuz I don't think that's necessarily realistic.
[04:35] Um I think how do we how can we understand this better so that we can make even for the average person the ability to create something great and to understand maybe their own taste
[04:43] um easy more more more easy.
[04:45] So that's that's a lot of what we're we're focusing on.
[04:49] Um So yeah, I mean this phenomenon of slop, by the way, is not new.
[04:52] Uh if you were in the internet uh as social media emerged, you probably saw a lot of slop before that.
[04:58] But I do think that AI has been this kind of like accelerating force, right?
[05:01] Of like being able to create things very easily uh with a click of a button and that like thoughtlessness around it.
[05:05] And there's kind of these three characteristics that I I would say repeat in slop.
[05:10] Uh so A, repetition.
[05:11] So you start seeing the same thing many many many times.
[05:15] Um the second is lack of fit, which I actually think is is very related.
[05:18] So fit is kind of this ability for something to feel correct for a specific context, right?
[05:23] For a specific moment in time, for a specific person.
[05:25] Uh but suddenly if you have repetition and let's say one person asks for a website for their pet shop and the other one asks for a website for their finance firm and somehow those designs converge and look the same.
[05:37] That's quite odd, right?
[05:38] Like if that was in if you were actually crafting that with care, that wouldn't you wouldn't converge necessarily on those things.
[05:44] And so this lack of fit and lack of understanding of context is actually huge problem that like leads to slop.
[05:49] Um and the third is maybe low intent, which is probably a mix of Yeah, you're going to have a bunch of people prompting really quickly and maybe just wanting to one shot something.
[05:56] But I think there's actually this like intent interpretation piece that's missing in the systems that we're building.
[06:01] Like how can you help your user, right?
[06:02] Like how can you help them better understand the intent that they have um so that you can add more color and add more context on onto what you're trying to create.
[06:12] Okay.
[06:12] And I I'm a big believer by the way that you in order to fix something first have to measure it and you first have to understand it.
[06:19] I think that's exactly why we're so focused on like how do we uh turn these domains into something a bit more verifiable so that we can attach a measure to it.
[06:27] So uh you'll you'll go on a little bit of a research journey with me here now, but we basically wanted to figure out can we measure slop?
[06:34] Like can we actually measure this quantitatively and spot this and what does that like look like?
[06:41] So we analyzed over 2 million websites from the past like 10 years kind of like way back machine style to try to understand all the trends across like design,
[06:49] how is the internet changing, uh how are how is like design changing over time?
[06:53] And two things were interesting.
[06:55] And we also met by the way then kind of synthetically generated a set of uh design websites so we could kind of like compare like how does human-made sites compare to AI-generated
[07:06] ones?
[07:07] And there were a few things that were interesting.
[07:08] So one was that you already kind of saw a a bit of like a collapse uh on the internet before even AI.
[07:15] So you saw kind of the internet becoming more homogeneous, using more similar color palettes, using more similar layouts, uh which is probably a function of more uh I would say this like kind of trends spreading more and more and more quickly, let's say.
[07:27] Uh but with AI I think you saw this repetition happening a lot more and being almost more like um identified kind of regardless of context.
[07:34] So even in completely different buckets you saw patterns that were very similar.
[07:37] So we we built this I I call this probes, but basically we uh we did two things.
[07:42] So we did this like pattern mining on all this data to understand like what are features that we can extract from all these sites.
[07:47] What are all these characteristics that we can make more objective, right?
[07:50] Colors, typography, layout, audience.
[07:53] Like, how can we like distill this down into things that become almost like uh structured?
[07:57] And then how do we uh train up these like probes?
[07:59] So, think of these as like baby classifiers.
[08:01] Like, how do we uh train the ability to spot this one characteristic?
[08:05] And for all these slop sites, we identify we started identifying like what are the probes that basically mean the site is very likely to be AI slop.
[08:14] Um and especially when you start combining them use and you see the frequency of multiple of these happening at once, it became very likely that you could actually like measure
[08:23] uh and predict slop.
[08:24] And we saw a a super high basically ability to do that prediction, which was really cool to see.
[08:29] This performed better, by the way, than like most LLM as a judge methods of like asking an LLM to like judge if that uh is like great human quality versus like AI-generated
[08:37] slop.
[08:38] Uh so, that was pretty cool to see.
[08:39] And I think kind of shows this pattern that we see in AI really being uh an actually quantitative thing that we can see in slop, uh which I find really cool.
[08:49] But, obviously, we don't want to stop there, right?
[08:50] We don't want to just measure slop.
[08:52] We want to also solve it.
[08:53] And so, um there's a few I I think I mentioned this before, but like the as the cost of production basically goes to zero, I think the thing that becomes
[09:02] expensive and matters more than ever is judgment.
[09:05] Um I don't even want to use the word taste here.
[09:07] Is judgment.
[09:08] I think it's this ability to discern what's right.
[09:10] It's this ability to break down a problem so that you can actually understand it and create solutions for it.
[09:15] And so, yes, there's the side of judgment that is human judgment that I actually think is more valuable than ever.
[09:19] But, there's also the side of like how do we build the right tools and systems to like fix pieces of this problem, right?
[09:27] So, yeah, how do we how do we fight slop, my my enemy?
[09:30] Um And, by the way, I think there's there's a lot of conversation going around how do you fight slop at the model layer?
[09:37] Like, how do we make models better?
[09:39] How do we make models have a higher bar?
[09:40] Which don't get me wrong, it has to be solved and we're working very hard to solve that, too.
[09:44] But I actually think this problem of inference time is equally, if not even more important.
[09:49] Because that's actually when you interact with the end user.
[09:52] And this kind of back and forth of how do you understand this context and intent happens at the moment of inference time.
[09:57] So, I don't think that we can ignore and just make models better and not solve this, otherwise slop will keep existing.
[10:02] Um so, maybe breaking down a few of those pieces and kind of um a few of the ways that we've thought about solving this or a few solutions that we built to solve this.
[10:10] But I think, for example, for something like repetition, one of the things that we're working on is I I've nicknamed it, I don't know if that's going to be the official name, but like the creativity API.
[10:17] How can we create a system that almost becomes an inspiration machine for your agent?
[10:22] So that it can produce something that's actually out of distribution instead of something that is in that same average and kind of mean that we're seeing happen with like the slop sites.
[10:30] Um so, this is one of the ways that practically, if we can intentionally produce something that's out of distribution, you can improve this like overall uh quality.
[10:38] And by the way, I I don't think that this can be something just like randomness.
[10:43] So, it's not just about like turning up the temperature of the model and and kind of fingers crossed hoping for the best.
[10:47] I think it's much more like how do we understand um even like what are rules or expectations in specific domains?
[10:53] Like let's say that you asked for a slide deck for for the pitch of your startup.
[10:57] Like what is a what does a good pitch deck look like?
[11:00] And then how do you almost like intentionally break rules uh to create things that are more creative, right?
[11:05] Because usually creativity isn't like randomness, isn't doing something that completely feels off for that situation.
[11:11] It's like you intentionally maybe diverge on a couple of things while maintaining kind of um adherence to to expectations of that category, let's say for for others.
[11:20] So, that's one of the things we're working on.
[11:22] The second one of this problem of fit, I think um it's interesting, but brands, as probably a lot of you who are designers know, takes so much effort to create great brands.
[11:31] Like great brands are the work of dozens of designers putting in a lot of like craft and thought and care.
[11:39] And so we've almost like already pre-done the work of defining what is great for that specific company and then we're not using it well.
[11:44] So this like brand endurance actually think is a huge problem and one of the things that can very more easily let's say like raise that bar of quality.
[11:51] So I'll I'll touch on an example on this one specifically.
[11:54] And then same with like intent and judgment I think the baby classifiers was a good example.
[11:58] Um like how it how we can actually like use this to be even become a gate for slop and and not let your agent ship slop.
[12:06] But so the brand API is the first product that we're releasing to to the public.
[12:09] This is already in in beta testing with a bunch of our design partners.
[12:13] And essentially what it does is it can take let's say a brand URL and extract this into like very specific components that are good for an agent to follow.
[12:22] So basically how do we turn something as fuzzy as a brand into something so structured that it becomes easy to for your agent to follow that but also for you to judge against it, right?
[12:30] Because I think the piece that we can't forget here is this judgment and verification.
[12:35] So yes, this goes and helps your agent to produce something better.
[12:38] But how can we also add a way for you to judge okay, is the agent actually staying on track?
[12:43] Is it actually performing well to adhere to this brand or how is it failing or where is it failing?
[12:47] So this is the first flow I would say that we we're seeing that is really helping to improve quality.
[12:53] And what's cool is of course we're talking here about an example of a brand that already exists.
[12:56] But let's say you have an agent or you have an app and the person that is using your app actually doesn't have a brand.
[13:03] Let's say they're an average consumer.
[13:04] Can we actually One of the things that we're creating is basically like a repository, like an index of brands of pre almost like pre-created brand system so that if they want something that feels dreamy,
[13:14] why not retrieve a dreamy brand system that already has been thought out to be cohesive instead of doing like a generative approach the moment of that might end up not so great or might end up again in those pillars of slop.
[13:26] And I want to show you a real example of this in action.
[13:28] So, um there's this company that I think is awesome called the General Intelligence Company of New York.
[13:32] They have a sick website, you guys should check it out.
[13:34] Um but basically, if you ask Cloud Design to create a slide deck uh in their branding, the the middle one is basically what it comes up with.
[13:41] So, the one on the left is is the original brand.
[13:44] Uh this is the kind of the the default.
[13:46] And if you kind of use this extraction actually in the process, it creates something that's way more high fidelity with the original.
[13:52] Um and that even like in the details, I would say like feels right.
[13:56] So, this is just to show an example of it in in action.
[13:59] Um but yeah, I think we I think all of us would agree that like human human taste and kind of the peak of human craft is always going to be like deeply valuable.
[14:11] And that right now, I think the challenge is we are almost even not earning the right to debate this like how can we have uh models like reach this like pinnacle of taste.
[14:21] I don't think it's about that at all.
[14:22] It's like how do we first just like raise the bar.
[14:25] Like the bar is currently, I would say, on the ground.
[14:27] And so, I think all of this work that we're putting into like how do we decompose a problem and how do we measure it is exactly so that we can at least like improve this bar
[14:36] um of quality.
[14:36] And I think we have to start with that.
[14:41] That's it.
[14:41] Uh thank you very much for for the time.
[14:43] Uh this is this is awesome.
[14:46] [applause]
[15:03] [music]