WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ACPEpji5NV4 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ACPEpji5NV4.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.480 --> 00:00:17.680 Thank you everyone for coming to this talk and Devanch for organizing this. s2 00:00:17.680 --> 00:00:18.960 Uh I'm Ragav. s3 00:00:18.960 --> 00:00:21.680 I'm a staff uh machine learning engineer at Door Dash. s4 00:00:21.680 --> 00:00:35.360 I work on search and personalization and uh today I'm going to talk about uh the ways that we're integrating LLMs for you know our marketplace discovery uh specifically in four pieces and s5 00:00:35.360 --> 00:00:45.520 um how these primitives are kind of uh uh working towards our integration of LLMs into traditional uh Rexus as well. s6 00:00:45.520 --> 00:00:49.200 So I'll start off uh today uh with a simple claim. s7 00:00:49.200 --> 00:01:01.520 Uh the claim is that to build effective discovery in this case for a marketplace like Door Dash the real bottleneck is semantic understanding. s8 00:01:01.520 --> 00:01:17.439 So that is knowing what you know actually what items mean uh for users given some context and what a shopper actually intends to do on you know an app like Door Dash. s9 00:01:17.439 --> 00:01:23.759 And historically we have treated these uh as engagement optimization problems. s10 00:01:23.759 --> 00:01:30.720 Um however LLMs give us a genuinely new lever to extend this. s11 00:01:30.720 --> 00:01:41.040 So this talk is about uh you know the four primitives that have allowed us to start leveraging LLMs for problems at Door Dash. s12 00:01:41.040 --> 00:01:46.479 As you may know Door Dash is now grown uh well beyond restaurants. s13 00:01:46.479 --> 00:01:50.720 So we're in grocery, retail, pets, gifting and more. s14 00:01:50.720 --> 00:01:55.280 And our goal is to capture every shoppable moment. s15 00:01:55.280 --> 00:02:03.200 Uh and we know that across these verticals, the users frequently have very broad shopping missions, right? s16 00:02:03.200 --> 00:02:10.479 So for example, a consumer may say when they come to our app is, you know, I just adopted a puppy. s17 00:02:10.479 --> 00:02:13.360 What do I need to get started this week? s18 00:02:13.360 --> 00:02:20.640 And this consumer start may start on like searching for puppy food because they just adopted the puppy. s19 00:02:20.640 --> 00:02:25.760 That one query could unfold into further query reformulations. s20 00:02:25.760 --> 00:02:38.560 Um they land on collections for you know other shopping needs, complimentary items such as crates, leashes, uh training pads all the way to checkout. s21 00:02:38.560 --> 00:02:43.680 And this could happen over multiple sessions, mult multiple days, right? s22 00:02:43.680 --> 00:02:48.959 And capturing that whole arc is pretty hard because it spans multiple models. s23 00:02:48.959 --> 00:02:55.440 Um, and this is where the four primitives I will cover come in. s24 00:02:55.440 --> 00:03:03.120 Specifically, supervision, catalog semantics, semantic personalization, and steerable content generation. s25 00:03:03.120 --> 00:03:14.400 As we heard previously as well, it's a system of shared representations that help us map this customer's journey uh with the use of LLMs. s26 00:03:14.879 --> 00:03:25.200 Now, with supervision, the question we're asking or looking to address is how do we teach our retrieval and ranking systems what good means for different tasks? s27 00:03:25.200 --> 00:03:27.360 And here's a simple example, right? s28 00:03:27.360 --> 00:03:30.000 A shopper searches for gluten-free pasta. s29 00:03:30.000 --> 00:03:40.159 And if we only rank from engagement, very popular regular spaghettes might show up and because it sells well, you know, it may continue to show up. s30 00:03:40.159 --> 00:03:46.480 Even gluten-free white bread might be retrieved and, you know, show up because it overlaps with gluten-free. s31 00:03:46.480 --> 00:03:50.239 But neither of these are clearly the right answer. s32 00:03:50.239 --> 00:03:53.040 What we actually want is graded relevance. s33 00:03:53.040 --> 00:03:58.319 A true, you know, gluten-free pasta should be a high relevance item. s34 00:03:58.319 --> 00:04:07.200 Chickpea pasta might be a reasonable substitute and regular spaghetti while it's popular it fails the constraint that the user has. s35 00:04:07.200 --> 00:04:10.959 Now to solve this there are obviously two sources of truth. s36 00:04:10.959 --> 00:04:22.079 We could look at human annotation but that's expensive slow and becomes quickly stale because our catalog changes quite rapidly. s37 00:04:22.079 --> 00:04:23.440 Second is behavioral signals. s38 00:04:23.440 --> 00:04:33.600 They're abundant but obviously as I mentioned biased by exposure, position, price, promotions and you know previous model choices. s39 00:04:33.600 --> 00:04:37.520 So the missing signal is a scalable reasoning signal. s40 00:04:37.520 --> 00:04:44.639 This is where LLMs are very useful as they offer a way to produce that supervision for such tasks. s41 00:04:44.639 --> 00:04:50.080 So in our case, you know, the PIP pipeline starts with human ground truths. s42 00:04:50.080 --> 00:05:00.320 uh we build a high quality seed set of example query item pairs and on a three three uh level relevant scale 012. s43 00:05:00.320 --> 00:05:02.800 We audit the suspicious cases. s44 00:05:02.800 --> 00:05:14.080 For example, if a human label says you know that an item is irrelevant but that item performs very well on add to cart or conversions, we send that case to a stronger LLM s45 00:05:14.080 --> 00:05:17.520 uh to re-evaluate with more granular prompts. s46 00:05:17.520 --> 00:05:20.320 We also reconcile with other models. s47 00:05:20.320 --> 00:05:24.240 You know, for example, our query to taxonomy or category models. s48 00:05:24.240 --> 00:05:33.919 And if a query, you know, query maps to a set of valid categories and the items belong to those valid categories, we may adjust the label accordingly. s49 00:05:33.919 --> 00:05:45.440 And through this process, we're able to achieve a pretty high precision golden data set that we then fine-tune a lightweight LLM, let's say, for example, a GPD 40 mini s50 00:05:45.440 --> 00:05:46.560 on top of it. s51 00:05:46.560 --> 00:05:50.560 And that's when it, you know, really pays off. s52 00:05:50.560 --> 00:05:59.039 We are able to use that fine-tuned labeler offline across, you know, our full catalog to generate uh full graded query item pairs. s53 00:05:59.039 --> 00:06:04.800 And those labels become one shared target for both retrieval and ranking systems. s54 00:06:04.800 --> 00:06:10.000 This pattern has been pretty important for us over the past uh few years. s55 00:06:10.000 --> 00:06:21.520 um use you know expensive reasoning once offline then distill it into models that can you know be served cheaply and quickly. s56 00:06:22.160 --> 00:06:24.000 Now let's take retrieval first. s57 00:06:24.000 --> 00:06:32.479 The challenge is you know standard embeddings um at e-commerce scale quickly collapse the relevance uh distinction. s58 00:06:32.479 --> 00:06:44.319 items that are merely related sit close to items that actually match and the model uh may know two items are related but not be able to separate uh exact matches, substitutes, s59 00:06:44.319 --> 00:06:46.000 compliments. s60 00:06:46.000 --> 00:06:52.240 So our fix is a two-stage contrastive method um trained on uh the same graded labels. s61 00:06:52.240 --> 00:06:57.600 Stage one, which we call mining, does a global geometry shaping. s62 00:06:57.600 --> 00:07:02.720 We use two tower encoders uh with multi-level supervised contrastive loss. s63 00:07:02.720 --> 00:07:17.199 Um after that we use that base model to mine harder negatives and uh these are items that the model confuses hard negatives that are ranked too high or strong positives ranked too low. s64 00:07:17.199 --> 00:07:24.400 And the second stage after that is where we really put the model through curriculum training. s65 00:07:24.400 --> 00:07:33.680 uh use those hard negatives, relabel them and our uh you know relevance of the retrieval layer improves sharply after that. s66 00:07:33.680 --> 00:07:45.759 So after stage one we see overlaps between you know relevant and uh moderately relevant but after stage two there is a significant difference and this has been one of like our biggest s67 00:07:45.759 --> 00:07:50.479 uh levers in retrieval improvement uh across the board. s68 00:07:50.479 --> 00:07:54.879 um you know we've improved NDCG of relevance by 2.3%. s69 00:07:54.879 --> 00:08:07.199 And uh that's also been true with our downstream business metrics and we did the same with uh uh exercise with our ranking models as well distilling LM reason labels s70 00:08:07.199 --> 00:08:08.960 into our rankers. s71 00:08:08.960 --> 00:08:14.800 However, combining this with uh business and engagement objectives. s72 00:08:15.039 --> 00:08:29.199 Here we add a new tower uh in this case an ordinal uh relevance tower on top of the LLM graded labels and uh this prediction sits right alongside existing engagement towers s73 00:08:29.199 --> 00:08:40.159 uh click add to cart and conversion and since they share the same uh bottom layers the semantic fit and relevance fit gets distilled in the same backward pass and s74 00:08:40.159 --> 00:08:52.160 uh our model is able to predict uh you know probability of relevance across the different levels and we are able to blend uh through a value function on on top of this s75 00:08:52.160 --> 00:08:58.720 that allows us to uh fine-tune between engagement and relevance for different surfaces. s76 00:08:59.760 --> 00:09:07.440 Now these two examples are part of a larger theme of our work across similar projects uh where it seems right. s77 00:09:07.440 --> 00:09:17.040 We're not replacing the retrieval and ranking system with an LLM, but rather distilling the uh reasoning and understanding into some production ranking architecture. s78 00:09:17.040 --> 00:09:30.800 And this allows us to kind of quickly test and also scalably improve our models uh with LLMs while keeping uh you know the business and system objectives uh of these surfaces in mind. s79 00:09:31.279 --> 00:09:43.200 So once we have supervision, the next question is how do we represent a very large and constantly changing catalog uh in a way that every model be it language models or s80 00:09:43.200 --> 00:09:54.720 uh you know traditional retrieval and ranking models uh that they can understand and at you know Door Dash's scale uh you can imagine we're kind of surfacing items across stores across s81 00:09:54.720 --> 00:10:08.240 internationally actually uh now and you know that's easily a few billion items at the store item level and for these items you obviously have unique skew ids but it says nothing semantically. s82 00:10:08.240 --> 00:10:15.839 So we also have a human curated taxonomy that helps us classify and categorize these items into meaningful spaces. s83 00:10:15.839 --> 00:10:19.519 But often this is too coarse and too rigid. s84 00:10:19.519 --> 00:10:33.440 So for example something like hot sauces may just have like sauces hot sauces in the taxonomy uh but it doesn't tell you anything about the relationship between hoof or franks or tobasco. s85 00:10:33.680 --> 00:10:39.440 So that's where we've also adapted uh or adopted um semantic ids. s86 00:10:39.440 --> 00:10:41.839 Uh it's been introduced previously already. s87 00:10:41.839 --> 00:10:47.120 It's a backbone for a lot of our uh work uh now. s88 00:10:47.120 --> 00:11:00.399 And uh what it does is it gives us a you know a short hierarchical code that's analogous to our taxonomy but uh the ability to control the fine grain nature of it. s89 00:11:00.399 --> 00:11:15.120 And the result of this learn taxonomy is that each item gets this hierarchical code and the prefix captures broad neighborhoods and you know the later tokens capture finer distinctions. s90 00:11:15.120 --> 00:11:24.720 So for example in the map over on on the slide you'll see hot sauces and the structure emerging from the data with zero labels. s91 00:11:24.720 --> 00:11:32.640 each one share the same first and second prefix in this uh ID sequence but then split by specialtity. s92 00:11:32.640 --> 00:11:38.720 So now you've got Mexican, Caribbean, Korean hot sauces uh being split up. s93 00:11:38.720 --> 00:11:53.040 And the important property is that this code that's learned is you know comparable by as I said by prefix but also usable by all of our downstream models and that unlocks a few things for us. s94 00:11:53.040 --> 00:11:57.120 So the first is cross-category comparisons or relationships. s95 00:11:57.120 --> 00:12:08.800 You know chips, salsa, guacamole may live in different taxonomy branches but a shared semantic ID neighborhood can uh indicate that they belong together and maybe a shopping mission. s96 00:12:08.800 --> 00:12:10.880 Second cold start problems. s97 00:12:10.880 --> 00:12:24.880 uh oftentimes we get new catalog items added by stores and with techniques like engram and bite parent coding of these tokens we're able to scalably add these to our models as sparse ID features s98 00:12:24.880 --> 00:12:38.959 third tail coverage so sparse items uh and uh inherit a lot of signal from semantically related items and uh you know we don't have to wait for the volume or exposure s99 00:12:38.959 --> 00:12:42.959 uh to you um for to consumers for these items. s100 00:12:42.959 --> 00:12:48.000 And then lastly, nice to have is the fact that we can actually also do a reverse audit. s101 00:12:48.000 --> 00:12:55.760 So we can audit our catalog and how well our human labels agree with the semantically learned labels. s102 00:12:56.160 --> 00:13:02.079 So a couple of examples of where we're using it today uh just to get give some real metrics. s103 00:13:02.079 --> 00:13:06.160 Um it has been one of our biggest improvements to our ranker. s104 00:13:06.160 --> 00:13:18.480 Um in in this case it's actually improved our uh ranker by uh improving MRR between four to 5% and that's translated to pretty big conversion uh wins as well. s105 00:13:18.480 --> 00:13:23.760 The second one that I'm particularly fond of is query reformulation. s106 00:13:23.760 --> 00:13:36.720 So again going back to the hot sauce example, Sriracha can lead to garlic chili garlic sauce or sambell Ole and because these queries map onto like our catalog grounded s107 00:13:36.720 --> 00:13:44.800 semantic neighborhood uh that results in much more relevant queries that we're suggesting to users. s108 00:13:44.800 --> 00:13:51.519 Um ultimately if these are queries that we don't have inventory for on Door Dash, they're meaningless. s109 00:13:51.519 --> 00:14:02.480 And with the addition of semantic ids in this query graph, we were actually able to see pretty massive um MR gains as well uh for for this piece of work. s110 00:14:03.120 --> 00:14:10.079 So for now we've talked about you know item mapping and relevance and the next question is consumer context. s111 00:14:10.079 --> 00:14:17.600 What does the system know about the shopper and can that knowledge be reused across models again? s112 00:14:17.600 --> 00:14:22.480 And this is where uh our third primitive comes in memory. s113 00:14:22.480 --> 00:14:26.480 So memory in the agent context is pretty well understood now. s114 00:14:26.480 --> 00:14:33.040 Um we apply the same concepts to uh our recommendation systems. s115 00:14:33.040 --> 00:14:36.160 So user embeddings are clearly very useful. s116 00:14:36.160 --> 00:14:49.920 uh we have many representations of our consumers but they don't necessarily get to uh why a consumer may have certain intents um and why they might have certain preferences. s117 00:14:50.000 --> 00:14:54.639 Additionally, LLMs cannot uh readily use these uh embeddings. s118 00:14:54.639 --> 00:14:58.560 You could tokenize and like uh train these models to learn it. s119 00:14:58.560 --> 00:15:05.360 But oftentimes having some explicit LM native counterpart is very useful and that's what we found. s120 00:15:05.360 --> 00:15:22.399 So the idea of memory is to represent uh the consumer in multiple forms semantic in you know inspectable and reusable and uh the way we think about memory is uh in three time scales. s121 00:15:22.399 --> 00:15:31.040 So long-term memory captures durable preferences from orders, searches, browsing, support interactions. s122 00:15:31.040 --> 00:15:49.120 Real-time context captures uh agentic inter interactions um oh sorry in session interactions so cart state active searches and then we have stated preferences which come from agentic interactions so something like ask door dash for example s123 00:15:49.120 --> 00:16:02.240 where consumers are able to explicitly state their constraints and preferences as well and the way we represent consumers in uh in uh this long-term memory is through uh memory blocks. s124 00:16:02.240 --> 00:16:14.959 And these are structured in a way that allows uh us to add new dimensions of the user, you know, from dietary preferences to dining preferences to uh substitute pre uh preferences s125 00:16:14.959 --> 00:16:27.600 um as we learn more about the consumer and that is uh decoupled from like the downstream system that u that doesn't need to reinterpret uh this for their own use cases. s126 00:16:27.600 --> 00:16:31.600 So each consumer's memory is multi materialized in multiple forms. s127 00:16:31.600 --> 00:16:34.480 So first is text that's human readable. s128 00:16:34.480 --> 00:16:37.839 It's uh captures the consumer in a compact way. s129 00:16:37.839 --> 00:16:45.360 Second is are latent vectors embeddings of those memory blocks that can be fed into retrieval and ranking systems. s130 00:16:45.360 --> 00:16:49.120 And third are graph and tree or hierarchical approaches. s131 00:16:49.120 --> 00:17:07.839 So uh you know type relationships between consumers and brands taxonomies and memory uh revealed um preferences and these are assembled into you know composite memory objects that both ML models and LLM use downstream. s132 00:17:07.839 --> 00:17:21.199 So an example is you know this context graph uh where we connect consumers with extracted memory concepts and a graph needs fits neatly into shopping journeys because you know oftent times s133 00:17:21.199 --> 00:17:25.120 uh consumer item interactions are sparse and multihop. s134 00:17:25.120 --> 00:17:35.679 And in this case these context graphs are able to link consumers across memory concepts that we previously did not have uh relationships for. s135 00:17:35.679 --> 00:17:40.480 And it's particularly helpful uh at fine grain taxonomy levels. s136 00:17:40.480 --> 00:17:53.360 In our case, we're seeing uh for retrieval uh uh using uh graph uh based embeddings outperforming our uh existing taxonomy based embeddings. s137 00:17:54.480 --> 00:17:58.400 And today this memory framework shows up in three places. s138 00:17:58.400 --> 00:18:00.559 The first is personalized collections. s139 00:18:00.559 --> 00:18:02.880 I'll talk about that a bit later. s140 00:18:02.880 --> 00:18:04.880 Uh second is agentic personalization. s141 00:18:04.880 --> 00:18:10.640 So ask door dash for example uses this to personalize its sessions uh for you. s142 00:18:10.640 --> 00:18:20.960 And third in uh retrieval and ranking models as I said we encode these memory blocks and feed them as features in in downstream models as well. s143 00:18:22.000 --> 00:18:25.360 So the fourth primitive is steerable content generation. s144 00:18:25.360 --> 00:18:35.679 We heard quite a bit about it in the previous talk and this slide is my take on how we're p piecing all these uh primitives together. s145 00:18:35.679 --> 00:18:50.320 So all of these inputs, semantic ids, memory blocks um and graded relevance or um LLM supervision feed into multiple models you know be it LLMs or small language models or s146 00:18:50.320 --> 00:19:04.960 um uh you know traditional models and from these inputs we can generate different output shapes you know ranked semantic ID lists, carousel titles, subcopies and these show up across different surfaces today on Gord Dash s147 00:19:04.960 --> 00:19:11.520 and A direct example of this is uh our um personalized collections on store pages. s148 00:19:11.520 --> 00:19:26.960 So historically collections on store pages uh on Door Dash have been a fixed library or attribute-based um and we've been able to kind of expand that through uh consumer level s149 00:19:26.960 --> 00:19:28.799 collection generation. s150 00:19:28.799 --> 00:19:42.240 And today this is an offline uh LLM process where we take in consumer memory semantic ids and syni syn synthesize uh collections um all the way from title subtitle to the actual items. s151 00:19:42.240 --> 00:19:45.360 And the important part is that control and steerability. s152 00:19:45.360 --> 00:19:57.120 We're able to react to occasions and moments um and build those collections as needed uh for different consumers at serving time. s153 00:19:57.120 --> 00:20:10.960 uh because this is all batch and uh generated offline through LLMs, we are able to still use our uh existing retrieval and ranking stack for item hydration, collection ranking, etc. s154 00:20:11.039 --> 00:20:14.320 And you know this is for example what a shopper actually sees, right? s155 00:20:14.320 --> 00:20:19.600 For a shopper with plant-based affinity, the system can generate plant-based pantry rows. s156 00:20:19.600 --> 00:20:23.760 For cat-only households, it can generate cat dry food rows. s157 00:20:23.760 --> 00:20:30.880 And if a session indicates that you know a person is going through pantry restocking it can bias towards pantry staples. s158 00:20:30.880 --> 00:20:37.360 And our early tests show that you know consumers are feeling the benefits of these tailored experiences. s159 00:20:37.360 --> 00:20:48.720 Uh an example is uh within our pets vertical we've been able to drive uh close to 1% uh order rate increases and 6% in uh active users. s160 00:20:48.880 --> 00:20:58.799 So to conclude uh there are three takeaways I would love for you to take away from this talk and how LLM fit into search and recommendations. s161 00:20:58.799 --> 00:21:02.480 First discovery is a semantic understanding problem. s162 00:21:02.480 --> 00:21:04.559 It's not only about engagement. s163 00:21:04.559 --> 00:21:09.120 LMS give us a way to reason about item meaning and shopper intent. s164 00:21:09.120 --> 00:21:12.159 Uh in in Door Dash's case for example. s165 00:21:12.159 --> 00:21:16.159 Second uh distill LM reasoning into primitives. s166 00:21:16.159 --> 00:21:26.080 capture you know reasoning uh offline as labels, semantic ids, memory and then let smaller and faster models serve it. s167 00:21:26.080 --> 00:21:31.520 The online LLM call is often not the product architecture you need. s168 00:21:31.520 --> 00:21:35.200 Third, shared representations create many use cases. s169 00:21:35.200 --> 00:21:53.039 Uh once you have these primitives, they can power, you know, retrieval, ranking, content generation, etc. So last but not least, thank you and thank you to all the collaborators at Door Dash who have helped ship a lot of these things as well.