WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=2LRIAfng7eA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/2LRIAfng7eA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:13.519
Welcome everyone.

s3
00:00:13.519 --> 00:00:18.720
Um really glad and thank you Deven for inviting us as well to talk today.

s4
00:00:18.720 --> 00:00:31.359
Uh so um uh Jackie and I are going to present today about making LLMs speak Spotify and how we turned our recommendation system on its head to be LLM native.

s5
00:00:31.359 --> 00:00:33.520
Uh just a few quick words about myself.

s6
00:00:33.520 --> 00:00:35.600
Uh I joined Spotify a year ago.

s7
00:00:35.600 --> 00:00:44.399
Before that I was at Google working on personalization for Google search and prior to that I was uh working on personalization at Netflix.

s8
00:00:44.480 --> 00:00:47.760
Um all right so let's dive in.

s9
00:00:47.760 --> 00:00:54.480
Um first a couple of numbers to describe the scale of the problem that we have to solve.

s10
00:00:54.480 --> 00:01:02.719
Um Spotify has about I was a little bit sad not to see it on the DAU by chart because it should be up there but it has about 760

s11
00:01:02.719 --> 00:01:08.640
million uh active uh users monthly uh in about 184 markets.

s12
00:01:08.640 --> 00:01:15.119
But one thing that makes uh the Spotify personalization problem particularly challenging is the size of its catalog.

s13
00:01:15.119 --> 00:01:26.479
Uh so of course you know Spotify has basically all music ever published a bit more than 100 million uh music tracks but it also has a range of videos, podcasts and audio books as well.

s14
00:01:26.479 --> 00:01:32.479
So the matching problem uh is uh actually surprisingly uh complicated.

s15
00:01:32.479 --> 00:01:40.079
So today what we're going to talk about Jackie and I is basically the extent of this matching problem and how we uh how we are solving it.

s16
00:01:40.079 --> 00:01:50.880
I'm going to talk about a little bit of history, the new phase that we're entering and then Jackie is going to help us uh go into the guts of the modes as well to understand how those things are trained as well.

s17
00:01:50.880 --> 00:01:52.640
Uh so a little bit on history.

s18
00:01:52.640 --> 00:01:57.040
Uh initially Spotify personalization was really built around curation.

s19
00:01:57.040 --> 00:02:02.719
So uh people basically manually assembling playlists that target specific taste.

s20
00:02:02.719 --> 00:02:05.759
And that's still a very common uh use case on Spotify.

s21
00:02:05.759 --> 00:02:12.319
Right now we have about 10 billion playlists with many many uh created every every hour.

s22
00:02:12.319 --> 00:02:19.440
Uh then you know Spotify moved into taking these creation signals and order signals and moving into recommendations.

s23
00:02:19.440 --> 00:02:25.599
So basically being able to turn these uh creation signals to something that can be applied at scale.

s24
00:02:25.599 --> 00:02:36.160
Uh and a great example of that would be discover weekly for example that was launched in 2014 as one of the one of the early like recommendation use case on on Spotify.

s25
00:02:36.160 --> 00:02:47.280
But the phase that we entering now which uh uh which we're going to talk about in more details is something that we call generative personalization where we are not only solving a matching problem from the user to the content.

s26
00:02:47.280 --> 00:02:57.280
We're also solving the ability to generate an experience that is interactively and dynamically shaped around each user.

s27
00:02:58.480 --> 00:03:07.680
And that transition from recommendations to generative personalization involves a couple of different uh a couple of different big shifts.

s28
00:03:07.680 --> 00:03:18.239
One is move from moving moving from personalization as guessing uh where basically you have a ranking algorithms that spits out a brand ranked list of entities in your catalog

s29
00:03:18.239 --> 00:03:28.239
to personalization as reasoning that can introspect these results and really try to understand whether that's indeed the right match for this user in this particular context.

s30
00:03:28.239 --> 00:03:32.959
The other aspect as well is moving from blackbox algorithms.

s31
00:03:32.959 --> 00:03:42.879
uh it's very difficult to fully introspect a multi-stage ranking system for example to transparent and steerable personalization where the user is always fully in control.

s32
00:03:42.879 --> 00:03:50.640
So uh basically giving the ability to these mods to speak and understand English.

s33
00:03:50.720 --> 00:04:00.799
The other thing that these these systems can do is uh uh not stopping just at recommending but also generation of experiences and explaining as well.

s34
00:04:00.799 --> 00:04:04.560
uh and we're going to show a couple of examples of that.

s35
00:04:05.040 --> 00:04:12.400
So one example that uh we launched a couple of years ago uh so pretty early in that journey was the Spotify DJ.

s36
00:04:12.400 --> 00:04:26.400
Uh so the Spotify DJ is something that you can spin up that will start playing music for you of course personalized uh and but one interesting thing about it is that since last year you can tap that button on the bottom uh right and

s37
00:04:26.400 --> 00:04:28.720
steer it in whatever direction you see fit.

s38
00:04:28.720 --> 00:04:37.680
So at any point you can chime in and let the algorithm know what you want and it's going to steer the session in the in in that in that direction.

s39
00:04:37.919 --> 00:04:47.360
Uh another example of what we call generative personalization is uh showcased in a prompted playlist here that Deanch showed a little bit earlier as well.

s40
00:04:47.360 --> 00:04:54.720
Uh and here you can you basically have full unfettered access to the recommendation algorithm that Spotify has.

s41
00:04:54.720 --> 00:05:00.639
uh and you can uh prompt it with very high level prompts or very detailed prompts.

s42
00:05:00.639 --> 00:05:06.720
On the left hand side here, I have a prompt that tells me uh create me a playlist of bands that are playing in San Francisco tonight.

s43
00:05:06.720 --> 00:05:11.039
Uh so turns out there's a bunch of good shows if you're excited to check them out.

s44
00:05:11.039 --> 00:05:15.440
On the right hand side, you see a prompt that is asking for a playlist to accompany me on my run.

s45
00:05:15.440 --> 00:05:29.280
And what's interesting with the right hand side as well is that you will see that the experience itself gets dynamically shaped as a function of the request on the user to be able to introduce itself and different phases in my run as well.

s46
00:05:29.680 --> 00:05:39.120
Uh, another one that I'm really excited about, we launched it in New Zealand a couple of months ago and uh, it's coming soon in more markets is something called the taste profile.

s47
00:05:39.120 --> 00:05:49.280
And that basically gives you the ability to introspect in natural language what the Spotify algorithm has understood about you in a way that you can edit and refine.

s48
00:05:49.280 --> 00:05:52.479
Uh, so if you see something that's missing or something that's wrong.

s49
00:05:52.479 --> 00:06:02.240
Uh so for example one of my edit is that all Disney music are my kids because we have a bunch of shared devices at home but please don't recommend that to me that's not my taste please.

s50
00:06:02.240 --> 00:06:09.520
Uh and the algorithm would then take that into account uh making sure that we never recommend this this in the wrong context.

s51
00:06:09.520 --> 00:06:14.479
Uh and similarly you can also use the taste profile to share some more aspiration uh goals as well.

s52
00:06:14.479 --> 00:06:23.360
Uh so getting into a new genre, getting into a new topic, uh learning about a new language for example, all of these things can be can be done.

s53
00:06:23.360 --> 00:06:37.440
Uh another one that's coming uh soon uh which we uh announced very recently is something called personal podcast where the the the generative personalization system doesn't stop at just recommending and ascending experiences but also generating content as well.

s54
00:06:37.440 --> 00:06:43.680
In this particular example, I'm generating a daily brief that's uh that's generated on a cadence daily.

s55
00:06:43.680 --> 00:06:50.800
Uh and I'm going to uh let it uh tell me about what's happening in my community.

s56
00:06:51.280 --> 00:06:51.840
All right.

s57
00:06:51.840 --> 00:06:54.160
So now to go into the guts of it.

s58
00:06:54.160 --> 00:07:01.199
So there's one big system that controls like all of these different applications I mentioned and we call that internally some the large taste model.

s59
00:07:01.199 --> 00:07:04.240
We're not great at naming these internal things.

s60
00:07:04.240 --> 00:07:06.000
Um so it has a couple of properties.

s61
00:07:06.000 --> 00:07:10.479
One is that it understands every historical interaction piece of content on Spotify.

s62
00:07:10.479 --> 00:07:14.720
Uh it combines prediction and reasoning to the point that I mentioned earlier.

s63
00:07:14.720 --> 00:07:21.759
So not only guessing but also reasoning layered on top and it gives users the ability to shape and generate experiences in real time.

s64
00:07:21.759 --> 00:07:25.680
So it's fully steable and promptable by uh by users.

s65
00:07:25.680 --> 00:07:32.319
Um and uh as of today about one in four US premium subscribers interact with that system on a daily basis as well.

s66
00:07:32.319 --> 00:07:33.520
So that's pretty exciting.

s67
00:07:33.520 --> 00:07:42.960
Um, one thing that's exciting as well to uh is that deploying this system across existing recommendation surfaces as well also led to some gains.

s68
00:07:42.960 --> 00:07:47.199
Uh, we saw gains on autoplay, we saw gains on podcast discoveries.

s69
00:07:47.199 --> 00:07:50.160
Uh, we saw gains on users interacting with DJ messages as well.

s70
00:07:50.160 --> 00:07:55.680
So, we saw pretty pretty sizable gains across the board as well by deploying the system.

s71
00:07:55.680 --> 00:08:03.440
On that note, I'm going to hand it over to Jackie to talk to us about what's one of the core component that underpins this whole system.

s72
00:08:03.440 --> 00:08:09.280
Hi everyone, I'm Jackie or Jacqueline, a staff machine learning engineer at Spotify.

s73
00:08:09.280 --> 00:08:16.000
So let's dive a little bit deeper and talk about how these models are actually trained at least at Spotify.

s74
00:08:16.000 --> 00:08:27.680
So semantic IDs were presented in the previous talk, but that is how we are embedding these openweight LLMs with knowledge of Spotify's catalog.

s75
00:08:27.680 --> 00:08:42.159
They are created by taking existing content embeddings such as podcast episode embeddings and applying a quantization algorithm to convert them to a set of discrete tokens.

s76
00:08:42.159 --> 00:08:54.959
We then take a openweight LLM such as Quen and we modify its vocabulary to add these new special tokens and then we fine-tune the model to be able to understand

s77
00:08:54.959 --> 00:09:03.279
both natural language as well as these new special tokens semantic IDs that represent Spotify catalog entities.

s78
00:09:03.279 --> 00:09:11.519
So we can power experiences such as this where the user can ask in natural language for a podcast on morality.

s79
00:09:11.519 --> 00:09:18.399
And that is passed to the prompt along with their listening history represented as semantic IDs.

s80
00:09:18.399 --> 00:09:30.959
And the model responds both with a relevant semantic ID podcast episode as well as a natural language description of why they recommended that to this user.

s81
00:09:33.040 --> 00:09:35.440
So how is this model actually trained?

s82
00:09:35.440 --> 00:09:47.120
Um we published a paper linked here uh describing our training paradigm called NEO which consists of four distinct stages.

s83
00:09:47.120 --> 00:10:01.680
The first I already covered which is the semantic foundation stage where we construct meaningful semantic ID tokens and then add them to an openway LLM's vocabulary.

s84
00:10:01.680 --> 00:10:13.600
The second stage we call domain grounding in which we align these new semantic ID token embeddings in the original language embedding space.

s85
00:10:13.600 --> 00:10:24.560
We do this by learning a birectional mapping between semantic ids to text, text to semantic ids and any combination.

s86
00:10:26.000 --> 00:10:35.920
And we actually freeze the LLM backbone at this stage and only train the new semantic ID embeddings.

s87
00:10:35.920 --> 00:10:49.519
So the original model weights and embeddings are frozen and we just learn those new semantic ID tokens and this helps us to mitigate catastrophic forgetting of the pre-trained LLM's

s88
00:10:49.519 --> 00:10:52.079
core language abilities.

s89
00:10:52.079 --> 00:11:03.200
The third stage we call capability induction which is multitask instruction tuning on tasks that Spotify cares about such as the ones shown here.

s90
00:11:03.200 --> 00:11:21.839
next item recommendation retrieval etc. This is done by unfreezing the whole model all of its weights and embeddings and running either full parameter fine-tuning or Laura fine-tuning on the multiple Spotify tasks

s91
00:11:21.839 --> 00:11:36.640
and then there is an optional fourth stage to do post- training such as RL fine-tuning etc. So, how much of a difference does this four-stage training paradigm actually make?

s92
00:11:36.640 --> 00:11:42.000
I'll dive into a few of the abilations we've done to investigate this.

s93
00:11:42.000 --> 00:11:52.079
First, we assessed whether multitask training is actually hurting performance by comparing the multitask model against single task variance.

s94
00:11:52.079 --> 00:12:06.800
And we consistently saw that across our tasks, the multitask model can match or actually beat the single task performance indicating that there is some positive cross-learning happening across the tasks.

s95
00:12:06.800 --> 00:12:11.680
This is particularly noticeable for audiobook recommendations.

s96
00:12:11.680 --> 00:12:28.240
If you look here, which is a new newer content type at Spotify, demonstrating that these multitask models can help with cold start entities by learning from other items in the catalog such as podcast recommendations,

s97
00:12:28.240 --> 00:12:32.880
how to make meaningful audiobook recommendations.

s98
00:12:33.040 --> 00:12:36.480
Then we did some abilations on the actual training recipe.

s99
00:12:36.480 --> 00:12:53.279
We evaluated both dropping the frozen backbone domain grounding stage altogether as well as combining the domain grounding stage with the capability induction stage in a multitask instruction tuning stage

s100
00:12:53.279 --> 00:13:03.680
that those are rows A and B here in the middle and we saw for both of those that it degraded performance but actually the biggest drop in performance was from

s101
00:13:03.680 --> 00:13:11.519
using a randomly initialized backbone instead of the pre-trained openweight LLM that we are using.

s102
00:13:11.519 --> 00:13:17.680
We also evaluated using continuous pre-training for the domain grounding stage.

s103
00:13:17.680 --> 00:13:23.839
And as you can see, it's it's minimal actual degradation on the task specific performance.

s104
00:13:23.839 --> 00:13:34.399
But where continuous pre-training really hits us is on the natural language and world knowledge capabilities of the pre-trained backbone LLM we are using.

s105
00:13:34.399 --> 00:13:41.360
after we do continuous pre-training, it goes to essentially zero versus if we do the frozen backbone domain grounding.

s106
00:13:41.360 --> 00:13:47.680
We retain all of that core language ability and are still able to learn the semantic ids.

s107
00:13:47.680 --> 00:13:55.120
I want to call out that these abilations were done with Quen, but we also validated that these findings hold with Llama.

s108
00:13:55.120 --> 00:14:03.120
So, it is not specific to the model backbone, but actually the training paradigm itself.

s109
00:14:03.120 --> 00:14:13.279
And then lastly, we did some investigation on different inference strategies and their effect on accuracy versus latency.

s110
00:14:13.279 --> 00:14:18.720
We tested beam search with both constrained decoding and not.

s111
00:14:18.720 --> 00:14:27.360
And we saw that even without constrained decoding, we can generate valid semantic IDs 98% of the time.

s112
00:14:27.360 --> 00:14:40.079
Constrained decoding does add a little latency overhead, but it's also helpful for specific cases where you want to target specific types of content, such as only make new content recommendations,

s113
00:14:40.079 --> 00:14:42.079
for example.

s114
00:14:42.399 --> 00:14:50.720
We also compared top P sampling to beam search and saw that top P sampling pretty significantly hurts our accuracy.

s115
00:14:50.720 --> 00:15:00.160
So although beam search is a little more latency intensive, we decided that trade-off worked for us.

s116
00:15:02.160 --> 00:15:04.880
So what is meaningful about this?

s117
00:15:04.880 --> 00:15:21.760
There have been lots of work in the industry in the space on generative semantic ID retrieval, toolbased LLM recommenders, the plum paper, etc. But NEO is actually the first example of combining all these capabilities into one system

s118
00:15:21.760 --> 00:15:40.000
that understands grounded catalog items, is naturally language steerable, can do search, recommendation, explanation use cases, as well as low latency tool-free inference at an industrial scale.

s119
00:15:41.360 --> 00:15:45.199
And we are using this in production today.

s120
00:15:45.199 --> 00:15:52.959
Um there's a paper linked as well here for how we're using this to power podcast discovery.

s121
00:15:53.279 --> 00:16:04.800
What we saw is that using a model train like this, we can break users out of their habitual patterns and get them to listen to more unfamiliar content.

s122
00:16:04.800 --> 00:16:08.880
And we saw huge wins online with this.

s123
00:16:10.880 --> 00:16:14.320
So none of this works without meaningful evaluations.

s124
00:16:14.320 --> 00:16:23.279
So I want to talk about that a little bit as our recommendations are becoming more generative and explanatory.

s125
00:16:23.519 --> 00:16:32.240
We our original maybe traditional offline eval metrics are not sufficient.

s126
00:16:32.240 --> 00:16:44.560
Yes, they can tell us whether or not the user interacted with that content, but they can't tell us whether or not the user or that recommendation makes sense for the user,

s127
00:16:44.560 --> 00:16:52.800
whether the explanation is accurate, whether it aligns with the user's intent, etc. So, that's where LLM judges really shine.

s128
00:16:52.800 --> 00:17:01.680
But in our work at Spotify, we really find that you need to invest in grounding your LLM judges in meaningful data.

s129
00:17:01.680 --> 00:17:07.199
so that they can be reliable evaluators that align with human preferences.

s130
00:17:07.199 --> 00:17:27.919
So, a few examples um for evaluating the podcast recommendations that I just mentioned, we create textual user profiles summarizing the users's listening history and that is passed to the LLM as a judge and we saw that this corresponded with a 75%

s131
00:17:27.919 --> 00:17:32.000
alignment between the LLM judge and human preferences.

s132
00:17:32.000 --> 00:17:37.760
Similarly, you can use actual behavioral signals to ground these LLM judges.

s133
00:17:37.760 --> 00:17:50.000
So, for example, for a search task, you can for a given query, you can take similar queries and how the user has interacted with them in the past and pass that to the model.

s134
00:17:50.000 --> 00:18:01.600
And we saw that overall it increased alignment by 5% but on ambiguous queries it actually increased alignment by 91%.

s135
00:18:01.600 --> 00:18:13.120
Showcasing the value these this grounding of the LLM judge plays especially in ambiguous cases where LLM judges tend to struggle.

s136
00:18:13.120 --> 00:18:21.039
Lastly, we used grounded LM judges to scale up our Cranfield style collections.

s137
00:18:21.039 --> 00:18:33.600
These are evaluation sets that are constructed by taking candidates from multiple different sources, creating a pool, and then using a human to rank that pool.

s138
00:18:33.600 --> 00:18:37.360
However, that human ranking stage is expensive.

s139
00:18:37.360 --> 00:18:52.000
So, we invested in significant grounding for our LM as judge and are able to have an LLM judge that aligns with uh human system rankings with an agreement value of 0.87.

s140
00:18:56.000 --> 00:19:05.120
In summary, um like we're going to hear a lot about today in all the talks, there is a new era of personalization among us.

s141
00:19:05.120 --> 00:19:07.840
this generative personalization.

s142
00:19:07.840 --> 00:19:20.640
And if you want to power language steerable personalized recommendations for your user, this is how we taught openweight LLMs to speak Spotify.

s143
00:19:20.640 --> 00:19:22.638
Thank you.

s144
00:19:22.638 --> 00:19:24.638
[applause]

s145
00:19:37.048 --> 00:19:39.048
[music]