Why LLM Recommenders Will Be AI's Biggest Consumer App — Devansh Tandon, Meta

AI Engineer · 18 min · 153 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Welcome everyone.
00:13Thank you for coming out to the LM Rexus track.
00:17Um I'll be sharing the first talk.
00:21Uh my talk is titled tokens and engagement out training LM recommenders.
00:27Um, and I want to make two big arguments today.
00:30The first is that recommendation systems scale just like LMS do and that the field is very early in that scaling curve.
00:38Um, and the second is that the LM recommener is going to be one of the biggest consumer applications of AI.
00:47Um, okay.
00:48So, quickly about me.
00:49Uh, I currently work at Meta on research and product.
00:52Uh I lead a team called Meta Recommendations Research uh which is this group that's training frontier models, LLMs and recommenders that power Instagram, Facebook ads, the meta family of apps.
01:06Uh before this I was at Google for a long time uh working on a lot of the key ML teams including deep mind and YouTube.
01:14Last year I gave a talk at AI engineer called teaching Gemini to speak YouTube about two ideas semantic IDs and generative retrieval which we also wrote two papers about which I've linked here.
01:27Um it was really fun and it led to a lot of discussions and collaborations.
01:31Uh the ideas behind semantic ID and generative retrieval have really taken off in the industry over the last year.
01:38Um and they've moved from more research to now scaled production systems.
01:43And I've seen exciting launches and papers from YouTube, Meta, Spotify, Door Dash across the industry.
01:51And we have a couple of examples of that later today.
01:55This year I want to talk about four uh sections.
01:59Recommendation scaling curves, a framework of four, recommendation paradigm scurves that we are climbing as an industry.
02:07um sharing the recommener recipe and finally this consumer AI app framework.
02:15Um let's start with scaling curves.
02:17So I wanted to start with this landmark scaling curve paper from 2020 which feels like a lifetime ago.
02:23This is when Daario was still at OpenAI and Anthropic didn't exist yet.
02:27But the core idea that this paper shared is the power law of scaling.
02:31As you increase model size, data, the amount of compute flops trained uh for model training, the loss falls on this log linear scale.
02:42And this clean and predictable curve is what really set off the race for the AI frontier because you can forecast what model quality and capability improvements will look like.
02:52And this is what's underwriting the massive capex investments and the AI buildout today.
02:59It turns out that recommendation systems follow a very similar scaling law.
03:04In fact, before this wave of LLMs, Rexs were the largest production ML models in big tech companies.
03:10And they're still some of the largest models that are served at a scale of a billion plus daily active users.
03:17And they follow this similar power loss scaling curve.
03:20On the x-axis, you have data, compute, and model size.
03:23And on the Y ais you would see falling loss or in this chart uh an improvement in recommendation quality.
03:31In offline evals it's net entropy or AU gains and then when it's translated to a real production launch it's engagement impact revenue impact at some of the biggest consumer app scale.
03:46Here's a real example from Meta that demonstrates these power loss scaling curves.
03:52uh there's a paper the first is a paper HSTU from 2024 and the second is a followup from this year both demonstrate that as we scale model size compute and data we see this clear improvement in offline eval of recommendation quality
04:11scaling curves aren't just academic research they're driving real product impact at scale for some of the biggest consumer businesses in the world here's a couple of examples I have from Meta's recent earnings reports.
04:23Um, Instagram reels had a strong quarter, 30% year-on-year watch time.
04:29And, uh, the optimizations we made to improve the quality of recommendations included simplifying our ranking architecture to enable efficient model scaling and longer interaction histories to identify a person's interests.
04:44uh we doubled the length of user interaction sequences used for training Instagram and increased the richness of each user interaction.
04:53So these are direct parallels to the power loss scaling curves for LLMs.
04:59And I want to introduce this idea of a flywheel of tokens in engagement out which is what's powering all of these Rex model scaling.
05:08You train a model.
05:09You then run inference on it which is the tokens in that recommendation model results in better content recommendations.
05:17It drives consumer engagement, daily active users time spent.
05:21It translates to monetization and ads or subscription which pays for the next model training run.
05:28And so every step on the scaling curve is one loop around this flywheel.
05:33And a lot of consumer apps are spinning this core flywheel at the heart of their business.
05:39So we have a long way to scale these recommener systems.
05:41I want to talk about the four paradigms that I see the industry progressing through.
05:46The first scurve was more traditional Rexus where this scurve focused more on feature engineering and user and content embeddings.
05:55Most production systems are still sitting on this curve.
05:57they're running some type of two tower sparse network rankers scaling the embedding models.
06:04I don't think this curve is going to go away but model development here will be accelerated with auto research and things like feature engineering will be handled by agents
06:13uh rather than real ML engineers.
06:16The next curve is kind of LM inspired models where you are scaling models ideally end to end.
06:23HSTU and one wreck papers are examples in this paradigm.
06:27I think this is where the leading recommener systems in the industry are largely operating today.
06:34I think the next scurve will be this paradigm of LLM native where you adapt a base model that understands and can reason uh and adapt it for recommendation tasks.
06:46The tiger and plum papers are some examples of this paradigm.
06:50And I think the final paradigm that I start to see emerging is agentic where LLMs will start to orchestrate Rex systems in a loop.
06:58I think there's a parallel here with coding agents.
07:00So the LM native models are like improving the core capabilities of the model going from opus 45 to 48 versus the agentic curve will be like improving the coding harness behind claw code or codeex.
07:14Um and so instead of just having a single forward pass through the recommener, you can imagine a loop where agents plan, retrieve, rank and then critique the recommendations.
07:26They can refine them by calling models again and finally deliver the recommendations.
07:31This I think is an interesting area of research.
07:33Now let me jump into so here here's kind of the framework of all the four Rex paradigms.
07:41I think companies are scaling across each of these curves in parallel.
07:48Most of recommendations I think lives in LM inspired today and is trying to graduate into LM native.
07:54Uh but then a lot of companies are still using traditional models uh and climbing that scurve.
08:02Let me shift gears a bit to share the recipe of how to actually build a LLM recommener.
08:10I think it's pretty simple.
08:11It's three steps.
08:13Uh you start with tokenizing your content and creating a language for your domain.
08:18Then you want to adapt the LLM so that it understands both English and your domain language and becomes this bilingual model.
08:27Finally, you can prompt this model with user information and it will directly decode recommendations from your content corpus.
08:36Let's go a bit deeper.
08:38This is the LLM recommener as a five layer cake.
08:41Um, we'll start at the bottom.
08:43Uh, that's semantic ID where you're converting your content corpus into tokens that the LLM can understand and reason over.
08:51Then you have kind of the base LLM foundation model.
08:54This can be an open weights model or an internal first party model.
09:00Then the core training stages.
09:02Pre-training is around bridging English and these recommener tokens.
09:06Post-raining is about steering the model towards recommendation tasks like predicting engagement or reasoning over recommendations.
09:17And then finally, you can just do some light surface specific fine-tuning to deploy it on a product surface.
09:24The exciting thing about this paradigm is most of the compute is shared across all of the product surfaces.
09:30So you don't have to train individual models from scratch for every product surface.
09:36I'll go a bit deeper into each stage for semantic ids.
09:39I think this has seen incredible adoption.
09:41A lot of teams are just replacing their hash ID with the SID in traditional models and seeing good impact.
09:48I think there's two big reasons to tokenize content.
09:51The first is it gives you the stable representation for models to learn over rather than a constantly shifting hash that the model can only memorize.
10:01And the second is compression.
10:03You want to be reasoning over these long sequences of user interactions.
10:07And if you don't compress the content, for example, a threeminut Instagram real video would be 10,000 tokens and it will just fill up the content context window too quickly.
10:17So you have to to compress it into about 10 tokens.
10:21And so here I have some examples of Instagram reels about tennis.
10:24You can see that the semantic token shares the prefix of the first three tokens because they're very similar reels.
10:30You can imagine the first token representing sports um and the second two tokens representing tennis and then the final token making these videos individual.
10:42Once you have a semantic ID, you can train it to understand both English and semantic ID.
10:49And so the task I have on the left for pre-training here is an example of where you prompt with a video with semantic ID ABC has the description blank
10:59and the output is a shot that was instantly iconic from Wimbledon.
11:03Here you're teaching the model to connect these semantic tokens with synthetic English natural language text.
11:09The example on the right is about reasoning over sequences of semantic IDs.
11:15And so in a user's interaction history, you can mask some parts of the sequence and the model learns to predict them and understand what videos are watched together in sequence.
11:27Here I have an example of post training where we're teaching the model how to rerank content.
11:32So the input is a bunch of user information and 30 candidate videos that are then ranked to be the top five recommendations from this LLM ranker.
11:43What's really interesting here is that you can see the chain of thought reasoning of this model.
11:49And because this model knows both English and recommendations, you can simply just inspect the model and understand why it made the decisions that it did.
11:58In this example, the model understands the user's topic interests like comedy, food, DIY, wellness.
12:05It understands the engagement style and what creators this user has an affinity towards.
12:12and then it reranks the content based on this chain of thought reasoning.
12:18I think this is super exciting because once you have a model that can understand both English and recommendations, it opens up new product surfaces and new experiences where users can steer their feed.
12:32Here's an example from your algorithm on Instagram where users can talk to the algorithm while they're consuming content.
12:40It's a screenshot from scrolling through reels or when you click in you can understand what the Instagram algorithm thinks about you and your interests and then you can add or remove interest and talk to it in natural language.
12:54And so we're going to see this shift I think from blackbox recommendations algorithms to giving users more control over their algorithm and algorithms becoming more interactive and steerable.
13:06I'm really excited that users can direct it towards their own goals that's expressed in language rather than just likes or comments.
13:16Um, and I think this can this foundation model can also start to explain its recommendations.
13:23And so for this example, I've added an interest that I want to follow the FIFA World Cup at this time.
13:29And the model would get both my user history and this new input and then be able to decode recommendations that are personalized to me like this free cake that Messi scored recently.
13:44I think these interactive recommenders are going to be a really interesting new product surface and we're seeing this across the industry.
13:52We have some examples of prompted playlists from Spotify, custom feeds from YouTube, Ask Door Dash, and we'll be hearing more from speakers about these.
14:03Um, finally, I want to talk about kind of the framework of tokens in engagement out this flywheel that I started with um of model training, inference, consumer engagement, and then monetization.
14:19This is actually the same flywheel that's shared by content feeds and the AI chat apps.
14:26And this is a lens that you can use to evaluate any consumer app.
14:30What'll make an app successful on the training ROI side is how well can it translate compute into a frontier model.
14:38On the inference side, how well can the inference tokens translate into engagement and then monetization.
14:46And so if you try to compare content feeds and AI chat apps, I think that LM recommenders are actually structurally more token efficient than the AI chat.
14:57So on the left you have content feeds like Instagram, Facebook, Tik Tok and YouTube.
15:01On the right you have the big AI chat apps like Gemini, ChatGpt and Claude.
15:07Content feeds are currently using models that are around 1 to 10 billion active parameters.
15:13The chat apps are serving much larger models 10 to 100 billion active parameters.
15:17Um the output for the content feed is a semantic ID token which is a pointer or an address to existing content because the content supplied on these content feeds is uploaded by creators.
15:30There's a very healthy creator economy and so the supply of content is effectively free or it's uploaded by creators.
15:38For the AI chat apps, they have to decode every token of content themselves and the amount of tokens output in every turn of an LM chat interaction is a few thousand tokens.
15:54Uh, and the the big difference here is that every token has to be manufactured by the app at inference time.
16:01And so what that means is if you compare these two apps on how much inference cost and compute is spent to generate an hour of consumer engagement, there's a huge structural gap where content feeds are significantly
16:15cheaper up to 100 times or more cheaper than AI chat apps because they're decoding pointers to content rather than content itself.
16:24Um, finally I want to kind of end with why I think LM Rexus is one of the most significant consumer AI applications.
16:33If you look at the top apps by daily active users, these are the top 10 apps.
16:38Four out of 10 of them are content feeds.
16:41And so this is a really significant consumer uh application.
16:47If you look at the content feeds, almost all of the consumer app growth on both the engagement and monetization side is driven by the recommener and ads models.
16:58And these are going to be entirely transformed by LLM recommenders.
17:02It's a very large and very token efficient application of AI for consumer apps.
17:08We're going to see a lot of new product experiences come through with steerable and interactive recommendations, explanation of recommendations, and just putting more users in control of their experience on these apps.
17:22I think we're going to see some really exciting research on consumer agents and recommendation agents that come out over the next year or so.
17:29Um and so this is why I think this is a super exciting area of both research and product uh at this intersection of LLM and recommendations.
17:41That's all.
17:41Thank you so much.