Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face

AI Engineer · 21 min · 160 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:13Good afternoon everyone.
3. 00:17I have a question.
4. 00:19How many of you knows hugging face?
5. 00:25Nice.
6. 00:27How many of you already use hugging face?
7. 00:33Amazing.
8. 00:34Almost everyone.
9. 00:36But I think we still have opportunity to grow our usage.
10. 00:41My name is Ar Borutki.
11. 00:43I work as machine learning platform and database engineer at hugging face.
12. 00:49Today I would like to walk you through how hugging faced scaled infrastructure and how we ended up serving 3 million models to developers around the world.
13. 01:09I would like to share architectural decisions we made, challenges we faced, and lessons we learned while scaling one of the fastest growing open-source AI communities in the world.
14. 01:29I hope you will enjoy it and let's get started.
15. 01:37Before I dive into technical details, let's talk about scale.
16. 01:44Today, Hugging Face serves more than 14 million users and this number is growing very fast, especially in the last couple of months.
17. 01:57We host three million public models, 1 million data sets, 50,000 organizations, and not only hobbyists or scientists.
18. 02:16More than 30% of Fortune 500 use hugging face as a part of AI workflows.
19. 02:27Just to give you some perspective, few years ago we had 20,000 models.
20. 02:36Today 3 million.
21. 02:39It is around 150x increase in just last couple of years.
22. 02:48And this grow is exactly why I'm here today talking about infrastructural decisions that keep the hub healthy at scale.
23. 03:03This is how fast the number of public models is growing on the hub.
24. 03:09Every big release like llama or deepseek generated thousands of new models on top and our infrastructure needs to handle that and it is not only models also data sets.
25. 03:31In 2022, we had 10K.
26. 03:35In 2024, 100K.
27. 03:39Less than a year ago, we had 500K.
28. 03:45Today, 1 million.
29. 03:49All this data must be stored, indexed, and also must be searchable.
30. 03:57And that's the hardest part.
31. 04:02And this is also the reason why we had to rethink our search.
32. 04:09At 20,000 models, any query is fast.
33. 04:14Even without an index, trust me, no one would notice.
34. 04:18At 3 million, same approach breaks.
35. 04:23Imagine what would you do if the hub search would be slow.
36. 04:30you would just leave and go somewhere else.
37. 04:32And this is also what user are doing.
38. 04:35They expect fast instant results.
39. 04:39With 14 million users, even 1% is a not small number.
40. 04:47It is 140,000 of people hitting slow search at scale.
41. 04:58P99 is much more important than P50 and we are paying a lots of attention to P99 and that's the reason why we invest in premputee tokens denormalize optimize for read
42. 05:20collection in MongoDB full text search based on Apache lucine Kubernetes autoscaling and soon in database sharding.
43. 05:34The next slides will show you how high level architecture when user interact with the hugging face hub his request flows from the front end to the hub API.
44. 05:54The hub is running on Kubernetes.
45. 05:58Currently we are using horizontal pot autoscaler.
46. 06:03During spikes new ports scale up automatically to handle the load and scale back down when traffic drops.
47. 06:14This help us to keep the hub healthy without manual intervention.
48. 06:22Next the request goes to MongoDB Atlas which is source of true for our metadata.
49. 06:34And there is one point that sometimes surprise people.
50. 06:39MongoDB does not store the models themselves.
51. 06:44It stores everything about the models.
52. 06:48What doesn't mean in practice in MongoDB we hold all the metadata users repositories models data sets buckets spaces information configuration data billing data access control and more.
53. 07:17The actual models artifacts, tokenizer files, cart assets and configuration files are stored separately in cloud object storage such as AWSS3.
54. 07:34This separation of concern let us scale metadata independently from binary storage and compute independently from both.
55. 07:46We can optimize each component individually for specific workload.
56. 07:55Now let's check how search works in details.
57. 07:59For example, someone wants to search the model on the hub and let's say that's llama.
58. 08:08So someone type llama into hugging face search bar.
59. 08:12His request flows through the hub to an optimized read collection on MongoDB.
60. 08:19And this is not our main repo collection when we keep all the data.
61. 08:26It's a separate denormalized copy only for reads and listings.
62. 08:35The key information is on the left.
63. 08:40We tokenize model names on insert time, not at query time.
64. 08:47For example, someone wants to publish model meta minus llama/ llama 3.18b.
65. 08:58We split the long model name into small tokens like meta lama 3.18b and we store them in an array in MongoDB document.
66. 09:12Next, Atlas search which is using Apache Lucin under the hood use autocomplete to find matching models instantly.
67. 09:27This is example of single document from our model collection.
68. 09:34In this example, I'm using find one method.
69. 09:37I want to find model ID meta minus llama/lama 3.1. So that's the model from the previous slide.
70. 09:47And I'm projecting only search token array.
71. 09:53And we see that all those premputee tokens are part of this array.
72. 09:57So we have metal lama 3.1 metal lama etc. Next there must be a query.
73. 10:07In the past we were using classical MongoDB find method on models collection with reax operator and this reax operator we're searching in search tokens arrays models which are equal to llama
74. 10:30and then we were s sorting results by trending score which is calculated every five minutes.
75. 10:39This is number of downloads and number of likes is as far as I remember from the last seven days.
76. 10:48This solution was working well as long as data set was small.
77. 10:55Reax doesn't scale well.
78. 10:57So when our data set started to grow very quickly, we started to having problems with latency.
79. 11:06So we decided to switch to Atlas search that's a feature which is using Apache lucine under the hood.
80. 11:18So MongoDB doesn't provide in core MongoDB server full text search there is additional process MongoD uh this MongoDBT process is a wrapper around Apache lucine for end user users this is transparent you are just using Unifi MongoDB
81. 11:42query API and when You use aggregation pip pipeline together with dollar search operator.
82. 11:55MongoDB will know that you would like to search Apache Lucin index.
83. 12:02Obviously you need to put the name of this index which is in this scenario model search autocomplete model equal to llama path search tokens and we still sort results by trending score
84. 12:24and this solution is much more efficient and is so far scale Well, so we don't have any more latency issues in our search bar.
85. 12:41First two results returned by previous query.
86. 12:44First metal lama has trending score 33.
87. 12:49Second one 14.
88. 12:56But hugging face hub is not only search.
89. 13:00We have hundreds of different services in hugging face which are utilizing which are using MongoDB to handle million of queries.
90. 13:11We use seven nodes MongoDB clusters cluster with multiple machines.
91. 13:20We can distribute queries across multiple nodes.
92. 13:24So no single node become read bottleneck.
93. 13:29This is how it works.
94. 13:31Application talk to the MongoDB cluster.
95. 13:34All inserts, deletions or updates goes to single primary because only primary can handle them.
96. 13:43However, we are distributing reads across multiple machines.
97. 13:51We also have one analytic hidden node.
98. 13:57What does it mean?
99. 13:59This mean this node is invisible from application.
100. 14:03MongoDB driver is not routing any queries to this hidden node.
101. 14:10This node is still replicating data from primary but is not interacting with interacting with production traffic.
102. 14:19We are connecting directly to this node and we use him for any kind of reporting traffic on or any kind of really heavy queries.
103. 14:35All secondaries continuously tail the oplog from primary keeping cluster in sync.
104. 14:45Now let's have a look what actually is running on secondaries.
105. 14:50First all queries which doesn't require the latest data goes to secondaries.
106. 14:58Only queries that must have strong consistency stay on primary and we are paying a lots of attention to this.
107. 15:09We are paying lots of attention to the queries which must run on primary.
108. 15:15Second, complex aggregations.
109. 15:18Aggregations pipelines that scan large amount of data, sort, group or transform the data should not go on primary.
110. 15:32They are heavy.
111. 15:33Secondaries are better placed for them.
112. 15:38Third, change streams.
113. 15:42We react to data changes in real time for several reasons.
114. 15:46For example, caching validation, sync to different data store technologies like for example AWS, Red Shift or for eventdriven workloads.
115. 16:00Those kind of operations are also not very light and they should stay impossible on secondaries.
116. 16:08Fourth, all ad hoc queries, reporting queries, maybe some experimental queries go to hidden MongoDB replicas set member which is isolated from production traffic.
117. 16:23The pattern is simple.
118. 16:25Primary should focus on what only primary can do.
119. 16:30Anything else can be pushed to different machines.
120. 16:38However, with 14 million users, 3 million models and our grow soon single MongoDB replica set will not be enough.
121. 16:52The next step is sharding.
122. 16:56Sharding means scaling your database horizontally.
123. 17:00Instead of putting full data set on one replic on one replica set cluster, we are going to cut data into pieces and put each piece on separate chart.
124. 17:16Each shard will have his own replication primary and secondary.
125. 17:21So we will keep replication just multiplied.
126. 17:26The key difference between replica set cluster and sharded cluster is replica set keep full data set on each node.
127. 17:35Sharded cluster keep only part of the data on each shard.
128. 17:39And then if you want to scale horizontally more you are just adding more shards and then MongoDB balancer will balance data across all those shards.
129. 17:49There is also shard key which must be selected.
130. 17:53This is not trivial operation but this talk is not about choosing short key.
131. 18:00This way we are going to scale everything CPU memory storage reads and writes.
132. 18:14Now let's have a look what is going on the hub level.
133. 18:19DHA is running on Kubernetes.
134. 18:22Currently we are using horizontal pot autoscaler.
135. 18:26When CPU or memory threshold go above target, Kubernetes adds new pot automatically to handle this spike and scale them back down when traffic drops.
136. 18:40Our deployment hub deployment can scale from 10 to 500 bots depends of on the traffic.
137. 18:49This is how we keep the hub healthy without manual interventions and without infrastructure overprovisioning.
138. 18:58So this is also cost effective solution.
139. 19:02However, what happens if horizontal pot autoscaler want to add new ports but kubernetes does not have free nodes anymore.
140. 19:17This is where second layer comes in.
141. 19:20We are using cast AI for Kubernetes note autoscaling when pods are pending because there is no capacity and Kubernetes scheduler is not able to schedule them.
142. 19:34Cast AAI is adding new nodes and then scheduler is able to schedule those spots.
143. 19:42So we have two layers of scaling.
144. 19:45First one is at deployment level.
145. 19:48Second one is at infrastructure level via castai.
146. 19:54But we are going to migrate horizontal port autoscaler to keta kubernetes eventdriven autoscaling.
147. 20:04The difference HPA scale only based on CPU and memory.
148. 20:11KDA scale on real application metrics like request per second or event loop utilization.
149. 20:21It means scaling is driven by actual workload not by resource utilization only.
150. 20:30For example, pot can have low CPU but high request Q. KDA can see it.
151. 20:39HPA not the best part of this architecture.
152. 20:46You never have to think about it.
153. 20:49When you pop when you push the model, search the hub or download the model, it just works.
154. 20:58This is what scaling medium models is really about.
155. 21:02Keeping the user experience simple no matter how complex it gets under the hood.
156. 21:10Thank you very much.
157. 21:11It was pleasure for me to be here today and I wish nice day for all of you.
158. 21:21Thank
159. 21:21[applause]
160. 21:21you.