WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=lyL5QhgIOxc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/lyL5QhgIOxc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.519 --> 00:00:16.400 Good afternoon everyone. s3 00:00:17.359 --> 00:00:19.520 I have a question. s4 00:00:19.520 --> 00:00:24.400 How many of you knows hugging face? s5 00:00:25.039 --> 00:00:27.039 Nice. s6 00:00:27.920 --> 00:00:33.119 How many of you already use hugging face? s7 00:00:33.840 --> 00:00:34.559 Amazing. s8 00:00:34.559 --> 00:00:36.880 Almost everyone. s9 00:00:36.880 --> 00:00:41.680 But I think we still have opportunity to grow our usage. s10 00:00:41.680 --> 00:00:43.520 My name is Ar Borutki. s11 00:00:43.520 --> 00:00:49.600 I work as machine learning platform and database engineer at hugging face. s12 00:00:49.600 --> 00:01:08.320 Today I would like to walk you through how hugging faced scaled infrastructure and how we ended up serving 3 million models to developers around the world. s13 00:01:09.119 --> 00:01:28.960 I would like to share architectural decisions we made, challenges we faced, and lessons we learned while scaling one of the fastest growing open-source AI communities in the world. s14 00:01:29.200 --> 00:01:35.600 I hope you will enjoy it and let's get started. s15 00:01:37.680 --> 00:01:43.680 Before I dive into technical details, let's talk about scale. s16 00:01:44.159 --> 00:01:57.600 Today, Hugging Face serves more than 14 million users and this number is growing very fast, especially in the last couple of months. s17 00:01:57.920 --> 00:02:15.680 We host three million public models, 1 million data sets, 50,000 organizations, and not only hobbyists or scientists. s18 00:02:16.000 --> 00:02:26.959 More than 30% of Fortune 500 use hugging face as a part of AI workflows. s19 00:02:27.599 --> 00:02:36.080 Just to give you some perspective, few years ago we had 20,000 models. s20 00:02:36.080 --> 00:02:39.519 Today 3 million. s21 00:02:39.599 --> 00:02:48.400 It is around 150x increase in just last couple of years. s22 00:02:48.400 --> 00:03:01.200 And this grow is exactly why I'm here today talking about infrastructural decisions that keep the hub healthy at scale. s23 00:03:03.760 --> 00:03:09.680 This is how fast the number of public models is growing on the hub. s24 00:03:09.680 --> 00:03:31.599 Every big release like llama or deepseek generated thousands of new models on top and our infrastructure needs to handle that and it is not only models also data sets. s25 00:03:31.599 --> 00:03:35.680 In 2022, we had 10K. s26 00:03:35.920 --> 00:03:39.760 In 2024, 100K. s27 00:03:39.760 --> 00:03:45.760 Less than a year ago, we had 500K. s28 00:03:45.760 --> 00:03:49.040 Today, 1 million. s29 00:03:49.120 --> 00:03:57.840 All this data must be stored, indexed, and also must be searchable. s30 00:03:57.840 --> 00:04:01.200 And that's the hardest part. s31 00:04:02.239 --> 00:04:08.239 And this is also the reason why we had to rethink our search. s32 00:04:09.120 --> 00:04:14.480 At 20,000 models, any query is fast. s33 00:04:14.480 --> 00:04:18.479 Even without an index, trust me, no one would notice. s34 00:04:18.479 --> 00:04:23.359 At 3 million, same approach breaks. s35 00:04:23.759 --> 00:04:30.400 Imagine what would you do if the hub search would be slow. s36 00:04:30.400 --> 00:04:32.479 you would just leave and go somewhere else. s37 00:04:32.479 --> 00:04:35.040 And this is also what user are doing. s38 00:04:35.040 --> 00:04:39.120 They expect fast instant results. s39 00:04:39.120 --> 00:04:47.360 With 14 million users, even 1% is a not small number. s40 00:04:47.360 --> 00:04:58.639 It is 140,000 of people hitting slow search at scale. s41 00:04:58.880 --> 00:05:20.160 P99 is much more important than P50 and we are paying a lots of attention to P99 and that's the reason why we invest in premputee tokens denormalize optimize for read s42 00:05:20.160 --> 00:05:33.919 collection in MongoDB full text search based on Apache lucine Kubernetes autoscaling and soon in database sharding. s43 00:05:34.160 --> 00:05:54.479 The next slides will show you how high level architecture when user interact with the hugging face hub his request flows from the front end to the hub API. s44 00:05:54.639 --> 00:05:57.919 The hub is running on Kubernetes. s45 00:05:58.160 --> 00:06:03.360 Currently we are using horizontal pot autoscaler. s46 00:06:03.360 --> 00:06:14.880 During spikes new ports scale up automatically to handle the load and scale back down when traffic drops. s47 00:06:14.880 --> 00:06:22.080 This help us to keep the hub healthy without manual intervention. s48 00:06:22.080 --> 00:06:34.400 Next the request goes to MongoDB Atlas which is source of true for our metadata. s49 00:06:34.400 --> 00:06:39.280 And there is one point that sometimes surprise people. s50 00:06:39.280 --> 00:06:44.080 MongoDB does not store the models themselves. s51 00:06:44.080 --> 00:06:48.720 It stores everything about the models. s52 00:06:48.960 --> 00:07:17.440 What doesn't mean in practice in MongoDB we hold all the metadata users repositories models data sets buckets spaces information configuration data billing data access control and more. s53 00:07:17.759 --> 00:07:34.240 The actual models artifacts, tokenizer files, cart assets and configuration files are stored separately in cloud object storage such as AWSS3. s54 00:07:34.240 --> 00:07:46.319 This separation of concern let us scale metadata independently from binary storage and compute independently from both. s55 00:07:46.319 --> 00:07:53.840 We can optimize each component individually for specific workload. s56 00:07:55.599 --> 00:07:59.759 Now let's check how search works in details. s57 00:07:59.759 --> 00:08:08.240 For example, someone wants to search the model on the hub and let's say that's llama. s58 00:08:08.240 --> 00:08:12.319 So someone type llama into hugging face search bar. s59 00:08:12.319 --> 00:08:19.840 His request flows through the hub to an optimized read collection on MongoDB. s60 00:08:19.840 --> 00:08:26.160 And this is not our main repo collection when we keep all the data. s61 00:08:26.160 --> 00:08:35.120 It's a separate denormalized copy only for reads and listings. s62 00:08:35.919 --> 00:08:39.919 The key information is on the left. s63 00:08:40.000 --> 00:08:47.760 We tokenize model names on insert time, not at query time. s64 00:08:47.760 --> 00:08:57.120 For example, someone wants to publish model meta minus llama/ llama 3.18b. s65 00:08:58.160 --> 00:09:12.720 We split the long model name into small tokens like meta lama 3.18b and we store them in an array in MongoDB document. s66 00:09:12.720 --> 00:09:25.200 Next, Atlas search which is using Apache Lucin under the hood use autocomplete to find matching models instantly. s67 00:09:27.839 --> 00:09:34.480 This is example of single document from our model collection. s68 00:09:34.480 --> 00:09:37.440 In this example, I'm using find one method. s69 00:09:37.440 --> 00:09:47.920 I want to find model ID meta minus llama/lama 3.1. So that's the model from the previous slide. s70 00:09:47.920 --> 00:09:53.279 And I'm projecting only search token array. s71 00:09:53.279 --> 00:09:57.680 And we see that all those premputee tokens are part of this array. s72 00:09:57.680 --> 00:10:07.839 So we have metal lama 3.1 metal lama etc. Next there must be a query. s73 00:10:07.839 --> 00:10:28.959 In the past we were using classical MongoDB find method on models collection with reax operator and this reax operator we're searching in search tokens arrays models which are equal to llama s74 00:10:30.000 --> 00:10:39.279 and then we were s sorting results by trending score which is calculated every five minutes. s75 00:10:39.279 --> 00:10:48.079 This is number of downloads and number of likes is as far as I remember from the last seven days. s76 00:10:48.079 --> 00:10:54.800 This solution was working well as long as data set was small. s77 00:10:55.120 --> 00:10:57.920 Reax doesn't scale well. s78 00:10:57.920 --> 00:11:06.480 So when our data set started to grow very quickly, we started to having problems with latency. s79 00:11:06.480 --> 00:11:18.160 So we decided to switch to Atlas search that's a feature which is using Apache lucine under the hood. s80 00:11:18.480 --> 00:11:42.399 So MongoDB doesn't provide in core MongoDB server full text search there is additional process MongoD uh this MongoDBT process is a wrapper around Apache lucine for end user users this is transparent you are just using Unifi MongoDB s81 00:11:42.399 --> 00:11:55.120 query API and when You use aggregation pip pipeline together with dollar search operator. s82 00:11:55.120 --> 00:12:02.320 MongoDB will know that you would like to search Apache Lucin index. s83 00:12:02.480 --> 00:12:24.720 Obviously you need to put the name of this index which is in this scenario model search autocomplete model equal to llama path search tokens and we still sort results by trending score s84 00:12:24.720 --> 00:12:38.079 and this solution is much more efficient and is so far scale Well, so we don't have any more latency issues in our search bar. s85 00:12:41.120 --> 00:12:44.959 First two results returned by previous query. s86 00:12:44.959 --> 00:12:49.519 First metal lama has trending score 33. s87 00:12:49.519 --> 00:12:53.120 Second one 14. s88 00:12:56.639 --> 00:13:00.399 But hugging face hub is not only search. s89 00:13:00.399 --> 00:13:11.760 We have hundreds of different services in hugging face which are utilizing which are using MongoDB to handle million of queries. s90 00:13:11.760 --> 00:13:20.240 We use seven nodes MongoDB clusters cluster with multiple machines. s91 00:13:20.240 --> 00:13:24.959 We can distribute queries across multiple nodes. s92 00:13:24.959 --> 00:13:29.680 So no single node become read bottleneck. s93 00:13:29.920 --> 00:13:31.440 This is how it works. s94 00:13:31.440 --> 00:13:34.160 Application talk to the MongoDB cluster. s95 00:13:34.160 --> 00:13:43.600 All inserts, deletions or updates goes to single primary because only primary can handle them. s96 00:13:43.600 --> 00:13:50.959 However, we are distributing reads across multiple machines. s97 00:13:51.839 --> 00:13:57.760 We also have one analytic hidden node. s98 00:13:57.760 --> 00:13:59.360 What does it mean? s99 00:13:59.360 --> 00:14:03.279 This mean this node is invisible from application. s100 00:14:03.279 --> 00:14:10.160 MongoDB driver is not routing any queries to this hidden node. s101 00:14:10.160 --> 00:14:19.760 This node is still replicating data from primary but is not interacting with interacting with production traffic. s102 00:14:19.760 --> 00:14:35.440 We are connecting directly to this node and we use him for any kind of reporting traffic on or any kind of really heavy queries. s103 00:14:35.760 --> 00:14:45.519 All secondaries continuously tail the oplog from primary keeping cluster in sync. s104 00:14:45.839 --> 00:14:50.560 Now let's have a look what actually is running on secondaries. s105 00:14:50.560 --> 00:14:58.079 First all queries which doesn't require the latest data goes to secondaries. s106 00:14:58.079 --> 00:15:09.519 Only queries that must have strong consistency stay on primary and we are paying a lots of attention to this. s107 00:15:09.519 --> 00:15:15.279 We are paying lots of attention to the queries which must run on primary. s108 00:15:15.519 --> 00:15:18.959 Second, complex aggregations. s109 00:15:18.959 --> 00:15:32.480 Aggregations pipelines that scan large amount of data, sort, group or transform the data should not go on primary. s110 00:15:32.480 --> 00:15:33.760 They are heavy. s111 00:15:33.760 --> 00:15:38.160 Secondaries are better placed for them. s112 00:15:38.399 --> 00:15:42.160 Third, change streams. s113 00:15:42.160 --> 00:15:46.720 We react to data changes in real time for several reasons. s114 00:15:46.720 --> 00:15:59.839 For example, caching validation, sync to different data store technologies like for example AWS, Red Shift or for eventdriven workloads. s115 00:16:00.320 --> 00:16:08.639 Those kind of operations are also not very light and they should stay impossible on secondaries. s116 00:16:08.639 --> 00:16:23.199 Fourth, all ad hoc queries, reporting queries, maybe some experimental queries go to hidden MongoDB replicas set member which is isolated from production traffic. s117 00:16:23.199 --> 00:16:25.120 The pattern is simple. s118 00:16:25.120 --> 00:16:30.399 Primary should focus on what only primary can do. s119 00:16:30.399 --> 00:16:37.040 Anything else can be pushed to different machines. s120 00:16:38.639 --> 00:16:52.399 However, with 14 million users, 3 million models and our grow soon single MongoDB replica set will not be enough. s121 00:16:52.399 --> 00:16:55.839 The next step is sharding. s122 00:16:56.000 --> 00:17:00.800 Sharding means scaling your database horizontally. s123 00:17:00.959 --> 00:17:16.959 Instead of putting full data set on one replic on one replica set cluster, we are going to cut data into pieces and put each piece on separate chart. s124 00:17:16.959 --> 00:17:21.919 Each shard will have his own replication primary and secondary. s125 00:17:21.919 --> 00:17:26.319 So we will keep replication just multiplied. s126 00:17:26.559 --> 00:17:35.039 The key difference between replica set cluster and sharded cluster is replica set keep full data set on each node. s127 00:17:35.039 --> 00:17:39.360 Sharded cluster keep only part of the data on each shard. s128 00:17:39.360 --> 00:17:49.840 And then if you want to scale horizontally more you are just adding more shards and then MongoDB balancer will balance data across all those shards. s129 00:17:49.840 --> 00:17:53.280 There is also shard key which must be selected. s130 00:17:53.280 --> 00:18:00.160 This is not trivial operation but this talk is not about choosing short key. s131 00:18:00.160 --> 00:18:13.360 This way we are going to scale everything CPU memory storage reads and writes. s132 00:18:14.880 --> 00:18:19.360 Now let's have a look what is going on the hub level. s133 00:18:19.360 --> 00:18:22.400 DHA is running on Kubernetes. s134 00:18:22.559 --> 00:18:26.559 Currently we are using horizontal pot autoscaler. s135 00:18:26.559 --> 00:18:40.640 When CPU or memory threshold go above target, Kubernetes adds new pot automatically to handle this spike and scale them back down when traffic drops. s136 00:18:40.640 --> 00:18:49.760 Our deployment hub deployment can scale from 10 to 500 bots depends of on the traffic. s137 00:18:49.760 --> 00:18:58.960 This is how we keep the hub healthy without manual interventions and without infrastructure overprovisioning. s138 00:18:58.960 --> 00:19:02.720 So this is also cost effective solution. s139 00:19:02.720 --> 00:19:17.200 However, what happens if horizontal pot autoscaler want to add new ports but kubernetes does not have free nodes anymore. s140 00:19:17.200 --> 00:19:20.640 This is where second layer comes in. s141 00:19:20.640 --> 00:19:34.799 We are using cast AI for Kubernetes note autoscaling when pods are pending because there is no capacity and Kubernetes scheduler is not able to schedule them. s142 00:19:34.799 --> 00:19:42.720 Cast AAI is adding new nodes and then scheduler is able to schedule those spots. s143 00:19:42.720 --> 00:19:45.760 So we have two layers of scaling. s144 00:19:45.760 --> 00:19:48.720 First one is at deployment level. s145 00:19:48.720 --> 00:19:54.160 Second one is at infrastructure level via castai. s146 00:19:54.480 --> 00:20:04.160 But we are going to migrate horizontal port autoscaler to keta kubernetes eventdriven autoscaling. s147 00:20:04.160 --> 00:20:11.600 The difference HPA scale only based on CPU and memory. s148 00:20:11.600 --> 00:20:21.600 KDA scale on real application metrics like request per second or event loop utilization. s149 00:20:21.600 --> 00:20:30.480 It means scaling is driven by actual workload not by resource utilization only. s150 00:20:30.480 --> 00:20:39.440 For example, pot can have low CPU but high request Q. KDA can see it. s151 00:20:39.440 --> 00:20:46.880 HPA not the best part of this architecture. s152 00:20:46.880 --> 00:20:49.200 You never have to think about it. s153 00:20:49.200 --> 00:20:58.720 When you pop when you push the model, search the hub or download the model, it just works. s154 00:20:58.720 --> 00:21:02.799 This is what scaling medium models is really about. s155 00:21:02.799 --> 00:21:10.720 Keeping the user experience simple no matter how complex it gets under the hood. s156 00:21:10.720 --> 00:21:11.919 Thank you very much. s157 00:21:11.919 --> 00:21:19.520 It was pleasure for me to be here today and I wish nice day for all of you. s158 00:21:21.120 --> 00:21:21.194 Thank s159 00:21:21.194 --> 00:21:21.360 [applause] s160 00:21:21.360 --> 00:21:23.360 you.