WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=lIgdnF0s0kQ NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/lIgdnF0s0kQ.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.639 --> 00:00:13.920 Welcome everyone. s3 00:00:13.920 --> 00:00:17.680 Thank you for coming out to the LM Rexus track. s4 00:00:17.680 --> 00:00:21.359 Um I'll be sharing the first talk. s5 00:00:21.359 --> 00:00:27.439 Uh my talk is titled tokens and engagement out training LM recommenders. s6 00:00:27.439 --> 00:00:30.160 Um, and I want to make two big arguments today. s7 00:00:30.160 --> 00:00:38.719 The first is that recommendation systems scale just like LMS do and that the field is very early in that scaling curve. s8 00:00:38.719 --> 00:00:47.120 Um, and the second is that the LM recommener is going to be one of the biggest consumer applications of AI. s9 00:00:47.200 --> 00:00:48.480 Um, okay. s10 00:00:48.480 --> 00:00:49.680 So, quickly about me. s11 00:00:49.680 --> 00:00:52.960 Uh, I currently work at Meta on research and product. s12 00:00:52.960 --> 00:01:06.960 Uh I lead a team called Meta Recommendations Research uh which is this group that's training frontier models, LLMs and recommenders that power Instagram, Facebook ads, the meta family of apps. s13 00:01:06.960 --> 00:01:14.560 Uh before this I was at Google for a long time uh working on a lot of the key ML teams including deep mind and YouTube. s14 00:01:14.560 --> 00:01:27.280 Last year I gave a talk at AI engineer called teaching Gemini to speak YouTube about two ideas semantic IDs and generative retrieval which we also wrote two papers about which I've linked here. s15 00:01:27.280 --> 00:01:31.439 Um it was really fun and it led to a lot of discussions and collaborations. s16 00:01:31.439 --> 00:01:38.880 Uh the ideas behind semantic ID and generative retrieval have really taken off in the industry over the last year. s17 00:01:38.880 --> 00:01:43.439 Um and they've moved from more research to now scaled production systems. s18 00:01:43.439 --> 00:01:51.280 And I've seen exciting launches and papers from YouTube, Meta, Spotify, Door Dash across the industry. s19 00:01:51.280 --> 00:01:55.759 And we have a couple of examples of that later today. s20 00:01:55.759 --> 00:01:59.759 This year I want to talk about four uh sections. s21 00:01:59.759 --> 00:02:07.600 Recommendation scaling curves, a framework of four, recommendation paradigm scurves that we are climbing as an industry. s22 00:02:07.600 --> 00:02:15.360 um sharing the recommener recipe and finally this consumer AI app framework. s23 00:02:15.360 --> 00:02:17.280 Um let's start with scaling curves. s24 00:02:17.280 --> 00:02:23.680 So I wanted to start with this landmark scaling curve paper from 2020 which feels like a lifetime ago. s25 00:02:23.680 --> 00:02:27.599 This is when Daario was still at OpenAI and Anthropic didn't exist yet. s26 00:02:27.599 --> 00:02:31.680 But the core idea that this paper shared is the power law of scaling. s27 00:02:31.680 --> 00:02:42.720 As you increase model size, data, the amount of compute flops trained uh for model training, the loss falls on this log linear scale. s28 00:02:42.720 --> 00:02:52.959 And this clean and predictable curve is what really set off the race for the AI frontier because you can forecast what model quality and capability improvements will look like. s29 00:02:52.959 --> 00:02:59.280 And this is what's underwriting the massive capex investments and the AI buildout today. s30 00:02:59.280 --> 00:03:04.239 It turns out that recommendation systems follow a very similar scaling law. s31 00:03:04.239 --> 00:03:10.800 In fact, before this wave of LLMs, Rexs were the largest production ML models in big tech companies. s32 00:03:10.800 --> 00:03:17.440 And they're still some of the largest models that are served at a scale of a billion plus daily active users. s33 00:03:17.440 --> 00:03:20.239 And they follow this similar power loss scaling curve. s34 00:03:20.239 --> 00:03:23.840 On the x-axis, you have data, compute, and model size. s35 00:03:23.840 --> 00:03:31.920 And on the Y ais you would see falling loss or in this chart uh an improvement in recommendation quality. s36 00:03:31.920 --> 00:03:46.319 In offline evals it's net entropy or AU gains and then when it's translated to a real production launch it's engagement impact revenue impact at some of the biggest consumer app scale. s37 00:03:46.319 --> 00:03:52.959 Here's a real example from Meta that demonstrates these power loss scaling curves. s38 00:03:52.959 --> 00:04:10.640 uh there's a paper the first is a paper HSTU from 2024 and the second is a followup from this year both demonstrate that as we scale model size compute and data we see this clear improvement in offline eval of recommendation quality s39 00:04:11.599 --> 00:04:23.520 scaling curves aren't just academic research they're driving real product impact at scale for some of the biggest consumer businesses in the world here's a couple of examples I have from Meta's recent earnings reports. s40 00:04:23.520 --> 00:04:29.360 Um, Instagram reels had a strong quarter, 30% year-on-year watch time. s41 00:04:29.360 --> 00:04:44.400 And, uh, the optimizations we made to improve the quality of recommendations included simplifying our ranking architecture to enable efficient model scaling and longer interaction histories to identify a person's interests. s42 00:04:44.400 --> 00:04:53.680 uh we doubled the length of user interaction sequences used for training Instagram and increased the richness of each user interaction. s43 00:04:53.680 --> 00:04:59.360 So these are direct parallels to the power loss scaling curves for LLMs. s44 00:04:59.360 --> 00:05:08.000 And I want to introduce this idea of a flywheel of tokens in engagement out which is what's powering all of these Rex model scaling. s45 00:05:08.000 --> 00:05:09.360 You train a model. s46 00:05:09.360 --> 00:05:17.759 You then run inference on it which is the tokens in that recommendation model results in better content recommendations. s47 00:05:17.759 --> 00:05:21.680 It drives consumer engagement, daily active users time spent. s48 00:05:21.680 --> 00:05:28.160 It translates to monetization and ads or subscription which pays for the next model training run. s49 00:05:28.160 --> 00:05:33.199 And so every step on the scaling curve is one loop around this flywheel. s50 00:05:33.199 --> 00:05:39.039 And a lot of consumer apps are spinning this core flywheel at the heart of their business. s51 00:05:39.039 --> 00:05:41.440 So we have a long way to scale these recommener systems. s52 00:05:41.440 --> 00:05:46.320 I want to talk about the four paradigms that I see the industry progressing through. s53 00:05:46.320 --> 00:05:55.280 The first scurve was more traditional Rexus where this scurve focused more on feature engineering and user and content embeddings. s54 00:05:55.280 --> 00:05:57.759 Most production systems are still sitting on this curve. s55 00:05:57.759 --> 00:06:04.160 they're running some type of two tower sparse network rankers scaling the embedding models. s56 00:06:04.160 --> 00:06:13.680 I don't think this curve is going to go away but model development here will be accelerated with auto research and things like feature engineering will be handled by agents s57 00:06:13.680 --> 00:06:16.800 uh rather than real ML engineers. s58 00:06:16.800 --> 00:06:23.840 The next curve is kind of LM inspired models where you are scaling models ideally end to end. s59 00:06:23.840 --> 00:06:27.280 HSTU and one wreck papers are examples in this paradigm. s60 00:06:27.280 --> 00:06:34.720 I think this is where the leading recommener systems in the industry are largely operating today. s61 00:06:34.720 --> 00:06:46.160 I think the next scurve will be this paradigm of LLM native where you adapt a base model that understands and can reason uh and adapt it for recommendation tasks. s62 00:06:46.160 --> 00:06:50.080 The tiger and plum papers are some examples of this paradigm. s63 00:06:50.080 --> 00:06:58.319 And I think the final paradigm that I start to see emerging is agentic where LLMs will start to orchestrate Rex systems in a loop. s64 00:06:58.319 --> 00:07:00.880 I think there's a parallel here with coding agents. s65 00:07:00.880 --> 00:07:14.400 So the LM native models are like improving the core capabilities of the model going from opus 45 to 48 versus the agentic curve will be like improving the coding harness behind claw code or codeex. s66 00:07:14.400 --> 00:07:26.240 Um and so instead of just having a single forward pass through the recommener, you can imagine a loop where agents plan, retrieve, rank and then critique the recommendations. s67 00:07:26.240 --> 00:07:31.039 They can refine them by calling models again and finally deliver the recommendations. s68 00:07:31.039 --> 00:07:33.199 This I think is an interesting area of research. s69 00:07:33.199 --> 00:07:41.840 Now let me jump into so here here's kind of the framework of all the four Rex paradigms. s70 00:07:41.840 --> 00:07:48.160 I think companies are scaling across each of these curves in parallel. s71 00:07:48.160 --> 00:07:54.560 Most of recommendations I think lives in LM inspired today and is trying to graduate into LM native. s72 00:07:54.560 --> 00:08:01.759 Uh but then a lot of companies are still using traditional models uh and climbing that scurve. s73 00:08:02.400 --> 00:08:09.440 Let me shift gears a bit to share the recipe of how to actually build a LLM recommener. s74 00:08:10.479 --> 00:08:11.840 I think it's pretty simple. s75 00:08:11.840 --> 00:08:13.680 It's three steps. s76 00:08:13.680 --> 00:08:18.879 Uh you start with tokenizing your content and creating a language for your domain. s77 00:08:18.879 --> 00:08:27.599 Then you want to adapt the LLM so that it understands both English and your domain language and becomes this bilingual model. s78 00:08:27.599 --> 00:08:36.719 Finally, you can prompt this model with user information and it will directly decode recommendations from your content corpus. s79 00:08:36.719 --> 00:08:38.080 Let's go a bit deeper. s80 00:08:38.080 --> 00:08:41.279 This is the LLM recommener as a five layer cake. s81 00:08:41.279 --> 00:08:43.360 Um, we'll start at the bottom. s82 00:08:43.360 --> 00:08:51.360 Uh, that's semantic ID where you're converting your content corpus into tokens that the LLM can understand and reason over. s83 00:08:51.360 --> 00:08:54.880 Then you have kind of the base LLM foundation model. s84 00:08:54.880 --> 00:09:00.240 This can be an open weights model or an internal first party model. s85 00:09:00.240 --> 00:09:02.000 Then the core training stages. s86 00:09:02.000 --> 00:09:06.399 Pre-training is around bridging English and these recommener tokens. s87 00:09:06.399 --> 00:09:17.360 Post-raining is about steering the model towards recommendation tasks like predicting engagement or reasoning over recommendations. s88 00:09:17.360 --> 00:09:24.000 And then finally, you can just do some light surface specific fine-tuning to deploy it on a product surface. s89 00:09:24.000 --> 00:09:30.320 The exciting thing about this paradigm is most of the compute is shared across all of the product surfaces. s90 00:09:30.320 --> 00:09:36.080 So you don't have to train individual models from scratch for every product surface. s91 00:09:36.080 --> 00:09:39.839 I'll go a bit deeper into each stage for semantic ids. s92 00:09:39.839 --> 00:09:41.680 I think this has seen incredible adoption. s93 00:09:41.680 --> 00:09:48.800 A lot of teams are just replacing their hash ID with the SID in traditional models and seeing good impact. s94 00:09:48.800 --> 00:09:51.920 I think there's two big reasons to tokenize content. s95 00:09:51.920 --> 00:10:01.120 The first is it gives you the stable representation for models to learn over rather than a constantly shifting hash that the model can only memorize. s96 00:10:01.120 --> 00:10:03.040 And the second is compression. s97 00:10:03.040 --> 00:10:07.279 You want to be reasoning over these long sequences of user interactions. s98 00:10:07.279 --> 00:10:17.279 And if you don't compress the content, for example, a threeminut Instagram real video would be 10,000 tokens and it will just fill up the content context window too quickly. s99 00:10:17.279 --> 00:10:21.120 So you have to to compress it into about 10 tokens. s100 00:10:21.120 --> 00:10:24.399 And so here I have some examples of Instagram reels about tennis. s101 00:10:24.399 --> 00:10:30.959 You can see that the semantic token shares the prefix of the first three tokens because they're very similar reels. s102 00:10:30.959 --> 00:10:41.920 You can imagine the first token representing sports um and the second two tokens representing tennis and then the final token making these videos individual. s103 00:10:42.399 --> 00:10:49.680 Once you have a semantic ID, you can train it to understand both English and semantic ID. s104 00:10:49.680 --> 00:10:59.279 And so the task I have on the left for pre-training here is an example of where you prompt with a video with semantic ID ABC has the description blank s105 00:10:59.279 --> 00:11:03.120 and the output is a shot that was instantly iconic from Wimbledon. s106 00:11:03.120 --> 00:11:09.839 Here you're teaching the model to connect these semantic tokens with synthetic English natural language text. s107 00:11:09.839 --> 00:11:15.120 The example on the right is about reasoning over sequences of semantic IDs. s108 00:11:15.120 --> 00:11:27.760 And so in a user's interaction history, you can mask some parts of the sequence and the model learns to predict them and understand what videos are watched together in sequence. s109 00:11:27.760 --> 00:11:32.399 Here I have an example of post training where we're teaching the model how to rerank content. s110 00:11:32.399 --> 00:11:43.279 So the input is a bunch of user information and 30 candidate videos that are then ranked to be the top five recommendations from this LLM ranker. s111 00:11:43.279 --> 00:11:49.200 What's really interesting here is that you can see the chain of thought reasoning of this model. s112 00:11:49.200 --> 00:11:58.240 And because this model knows both English and recommendations, you can simply just inspect the model and understand why it made the decisions that it did. s113 00:11:58.240 --> 00:12:05.760 In this example, the model understands the user's topic interests like comedy, food, DIY, wellness. s114 00:12:05.760 --> 00:12:12.560 It understands the engagement style and what creators this user has an affinity towards. s115 00:12:12.560 --> 00:12:18.160 and then it reranks the content based on this chain of thought reasoning. s116 00:12:18.720 --> 00:12:32.000 I think this is super exciting because once you have a model that can understand both English and recommendations, it opens up new product surfaces and new experiences where users can steer their feed. s117 00:12:32.000 --> 00:12:40.639 Here's an example from your algorithm on Instagram where users can talk to the algorithm while they're consuming content. s118 00:12:40.639 --> 00:12:54.240 It's a screenshot from scrolling through reels or when you click in you can understand what the Instagram algorithm thinks about you and your interests and then you can add or remove interest and talk to it in natural language. s119 00:12:54.240 --> 00:13:06.959 And so we're going to see this shift I think from blackbox recommendations algorithms to giving users more control over their algorithm and algorithms becoming more interactive and steerable. s120 00:13:06.959 --> 00:13:16.480 I'm really excited that users can direct it towards their own goals that's expressed in language rather than just likes or comments. s121 00:13:16.480 --> 00:13:23.839 Um, and I think this can this foundation model can also start to explain its recommendations. s122 00:13:23.920 --> 00:13:29.360 And so for this example, I've added an interest that I want to follow the FIFA World Cup at this time. s123 00:13:29.360 --> 00:13:42.399 And the model would get both my user history and this new input and then be able to decode recommendations that are personalized to me like this free cake that Messi scored recently. s124 00:13:44.320 --> 00:13:52.000 I think these interactive recommenders are going to be a really interesting new product surface and we're seeing this across the industry. s125 00:13:52.000 --> 00:14:02.560 We have some examples of prompted playlists from Spotify, custom feeds from YouTube, Ask Door Dash, and we'll be hearing more from speakers about these. s126 00:14:03.199 --> 00:14:19.120 Um, finally, I want to talk about kind of the framework of tokens in engagement out this flywheel that I started with um of model training, inference, consumer engagement, and then monetization. s127 00:14:19.120 --> 00:14:26.480 This is actually the same flywheel that's shared by content feeds and the AI chat apps. s128 00:14:26.480 --> 00:14:30.320 And this is a lens that you can use to evaluate any consumer app. s129 00:14:30.320 --> 00:14:38.480 What'll make an app successful on the training ROI side is how well can it translate compute into a frontier model. s130 00:14:38.480 --> 00:14:46.000 On the inference side, how well can the inference tokens translate into engagement and then monetization. s131 00:14:46.000 --> 00:14:57.440 And so if you try to compare content feeds and AI chat apps, I think that LM recommenders are actually structurally more token efficient than the AI chat. s132 00:14:57.440 --> 00:15:01.920 So on the left you have content feeds like Instagram, Facebook, Tik Tok and YouTube. s133 00:15:01.920 --> 00:15:07.440 On the right you have the big AI chat apps like Gemini, ChatGpt and Claude. s134 00:15:07.440 --> 00:15:13.040 Content feeds are currently using models that are around 1 to 10 billion active parameters. s135 00:15:13.040 --> 00:15:17.839 The chat apps are serving much larger models 10 to 100 billion active parameters. s136 00:15:17.839 --> 00:15:30.480 Um the output for the content feed is a semantic ID token which is a pointer or an address to existing content because the content supplied on these content feeds is uploaded by creators. s137 00:15:30.480 --> 00:15:38.639 There's a very healthy creator economy and so the supply of content is effectively free or it's uploaded by creators. s138 00:15:38.639 --> 00:15:54.160 For the AI chat apps, they have to decode every token of content themselves and the amount of tokens output in every turn of an LM chat interaction is a few thousand tokens. s139 00:15:54.160 --> 00:16:01.120 Uh, and the the big difference here is that every token has to be manufactured by the app at inference time. s140 00:16:01.120 --> 00:16:15.519 And so what that means is if you compare these two apps on how much inference cost and compute is spent to generate an hour of consumer engagement, there's a huge structural gap where content feeds are significantly s141 00:16:15.519 --> 00:16:24.480 cheaper up to 100 times or more cheaper than AI chat apps because they're decoding pointers to content rather than content itself. s142 00:16:24.959 --> 00:16:33.680 Um, finally I want to kind of end with why I think LM Rexus is one of the most significant consumer AI applications. s143 00:16:33.680 --> 00:16:38.480 If you look at the top apps by daily active users, these are the top 10 apps. s144 00:16:38.480 --> 00:16:41.839 Four out of 10 of them are content feeds. s145 00:16:41.839 --> 00:16:47.920 And so this is a really significant consumer uh application. s146 00:16:47.920 --> 00:16:58.000 If you look at the content feeds, almost all of the consumer app growth on both the engagement and monetization side is driven by the recommener and ads models. s147 00:16:58.000 --> 00:17:02.880 And these are going to be entirely transformed by LLM recommenders. s148 00:17:02.880 --> 00:17:08.480 It's a very large and very token efficient application of AI for consumer apps. s149 00:17:08.480 --> 00:17:22.240 We're going to see a lot of new product experiences come through with steerable and interactive recommendations, explanation of recommendations, and just putting more users in control of their experience on these apps. s150 00:17:22.240 --> 00:17:29.760 I think we're going to see some really exciting research on consumer agents and recommendation agents that come out over the next year or so. s151 00:17:29.760 --> 00:17:40.080 Um and so this is why I think this is a super exciting area of both research and product uh at this intersection of LLM and recommendations. s152 00:17:41.120 --> 00:17:41.919 That's all. s153 00:17:41.919 --> 00:17:44.400 Thank you so much.