WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=FQwTqUmcbRg

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/FQwTqUmcbRg.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.800 --> 00:00:13.599
We're doing this.

s3
00:00:13.599 --> 00:00:16.400
Hey everyone, welcome to my talk.

s4
00:00:16.400 --> 00:00:17.119
I'm Bogdan.

s5
00:00:17.119 --> 00:00:19.680
I'm one of the co-founders and the CEO of Theology.

s6
00:00:19.680 --> 00:00:26.240
Today we're going to talk about the engineering lessons from scaling synthetic data for trillion token scale pre-training.

s7
00:00:26.240 --> 00:00:27.439
Um, let's see.

s8
00:00:27.439 --> 00:00:30.080
So why do we need synthetic data in the first place?

s9
00:00:30.080 --> 00:00:42.160
Well, turns out that with the scaling laws, we kind of hit a data wall um you need to put exponentially more compute and exponentially more data to get um models that are getting kind of linearly better and better.

s10
00:00:42.160 --> 00:00:47.120
So um we're we're at this point where pre-training runs are massive.

s11
00:00:47.120 --> 00:00:59.120
So in order to do the the kind of the ones that actually get to I don't know the mythos level um capabilities you need a lot of data and a lot of it uh cannot be found on the web.

s12
00:00:59.120 --> 00:01:05.920
So um internet has limited amount of data um we can supplement it with high quality synthetic data.

s13
00:01:05.920 --> 00:01:07.439
So what are we going to talk about today?

s14
00:01:07.439 --> 00:01:11.280
We're going to talk about um you know how do we do this in the first place?

s15
00:01:11.280 --> 00:01:15.680
We're going to talk about a bit about beyond web which is kind of our syntheta data recipe.

s16
00:01:15.680 --> 00:01:26.000
Um we're going to talk about the engineering lessons from our prior experience here at Ethology of running trillion tail trillion um token scale synthetic data runs.

s17
00:01:26.000 --> 00:01:29.520
Um we recently completed one for about 12 trillion tokens.

s18
00:01:29.520 --> 00:01:31.520
That's across web math and code.

s19
00:01:31.520 --> 00:01:36.320
And we're going to talk about what we learned from running absolutely massive synthetic runs.

s20
00:01:36.320 --> 00:01:41.439
Um and then we're going to talk about kind of what's the why does this matter in the first place.

s21
00:01:41.439 --> 00:01:43.119
Um awesome.

s22
00:01:43.119 --> 00:01:44.159
So let's get started.

s23
00:01:44.159 --> 00:01:50.799
Um, so the way that we solve synthetic data at theology and maybe let's start with a with a quick intro about it.

s24
00:01:50.799 --> 00:01:53.200
We're a kind of data curation service.

s25
00:01:53.200 --> 00:01:58.719
Think of us as the oil refinery for your kind of pre-training and mid-training data sets.

s26
00:01:58.719 --> 00:02:06.159
We work with customers both startups and enterprises that come to us to um kind of help them with their pre-training and mid-training data needs.

s27
00:02:06.159 --> 00:02:07.920
So in general they have a lot of data.

s28
00:02:07.920 --> 00:02:19.040
um they want to get to the best data sets possible and we help them identify the the highest quality data points and um once we do that we pass them through our synthetic data recipe the ones that we're going to talk about today

s29
00:02:19.040 --> 00:02:21.520
um to get absolutely massive data sets.

s30
00:02:21.520 --> 00:02:27.040
So um that's a bit about us uh in terms of beyond web is is our synthetic data recipe.

s31
00:02:27.040 --> 00:02:34.560
Um there's kind of many ways to do synthetic data but um kind of the the way that we approach it is um quite different.

s32
00:02:34.560 --> 00:02:37.760
So here's a few results from from um kind of how we do this.

s33
00:02:37.760 --> 00:02:39.120
So uh there's two plots.

s34
00:02:39.120 --> 00:02:40.480
Maybe let's start with the first one.

s35
00:02:40.480 --> 00:02:44.400
Um on the x-axis we have the model size and billions of parameters.

s36
00:02:44.400 --> 00:02:46.560
Um 1 billion, 3 billion, 8 billion.

s37
00:02:46.560 --> 00:02:49.599
And then on the y-axis we have um accuracy.

s38
00:02:49.599 --> 00:03:02.560
So um we've taken kind of a number of data sets like Neotron Synth, Cosmopedia, QA rap, um Red Pajama, and then we've trained models at kind of 1B, 3B and 8B scale together with um

s39
00:03:02.560 --> 00:03:07.440
kind of the the recipe we've put together for for uh synthetic data which we call beyond web.

s40
00:03:07.440 --> 00:03:18.560
Um and what we see is that um kind of our synthetic data recipe um kind of matches and and sometimes even um kind of in this case the the 3B

s41
00:03:18.560 --> 00:03:22.959
data point matches the 8B data point um for neatron synth.

s42
00:03:22.959 --> 00:03:32.080
So that means that you can train slightly smaller models for the same performance as you would train larger ones by using synthetic data.

s43
00:03:32.080 --> 00:03:37.040
So um that's a bit about kind of the the the quality of of our recipe.

s44
00:03:37.040 --> 00:03:39.200
Um same thing here when you when you talk about performance.

s45
00:03:39.200 --> 00:03:46.159
So on the x axis we have um kind of number of tokens in billions and then the y- axis we have uh the the average accuracy.

s46
00:03:46.159 --> 00:03:58.239
Um and what we show is that we match the the the performance of um kind of the the um Neotron synth and the Cosmopedia ones in about 2.7x and 5.3x

s47
00:03:58.239 --> 00:04:00.400
respectively uh less tokens.

s48
00:04:00.400 --> 00:04:08.480
So uh if you want to train a model to the same performance, you can do that um by using less data and and overall less compute.

s49
00:04:08.480 --> 00:04:14.000
Um so at a high level what what beyond web is is is kind of how we think about our synthetic data recipe.

s50
00:04:14.000 --> 00:04:16.160
What we do is we do rephrasings.

s51
00:04:16.160 --> 00:04:28.400
Um so where we we kind of see the rephrase the the synthetic data process by um kind of finding the highest quality data points in a customer's data set and then we rephrase or restructure this data points in in various ways.

s52
00:04:28.400 --> 00:04:30.800
Um this is results are actually a bit old.

s53
00:04:30.800 --> 00:04:41.360
I think the the origin we published this in uh in um I think late summer 25 and at this point we're we're much much better than this um and and we're continuing to improve quite a bit.

s54
00:04:41.360 --> 00:04:51.440
So how do you get synthetic data right in the in the simplest form you have some sort of target data set again we're doing rephrasings um or reframings of existing documents.

s55
00:04:51.440 --> 00:05:03.120
So the idea behind it is that um if you just prompt a model and say hey give me a synthetic data um data point you're going to just learn the modes of the distribution the data was was origin trained on.

s56
00:05:03.120 --> 00:05:09.440
So uh the way that we do it as mentioned previously is that we seed it with um some sort of existing documents.

s57
00:05:09.440 --> 00:05:16.320
So we identify high quality documents in the customer's um training data sets and then we we pass them through through our pipeline.

s58
00:05:16.320 --> 00:05:18.639
At the simplest set, we have a a target data set.

s59
00:05:18.639 --> 00:05:21.199
Imagine this can be text, this can be image text.

s60
00:05:21.199 --> 00:05:29.520
Um, in the simplest forms, it is just a bunch of prompts that that we prefix to um a number of documents that we've identified.

s61
00:05:29.520 --> 00:05:36.240
You have a bunch of GPUs and then you um kind of use various models to generate synthetic data.

s62
00:05:36.240 --> 00:05:41.919
Um so in practice, you have some sort of um expectations around how much time this is going to take.

s63
00:05:41.919 --> 00:05:49.360
But in practice, this is a lot more wavy and and it it really it's really hard to match the the kind of ideal throughput that you're trying to get of your

s64
00:05:49.360 --> 00:05:51.280
um out of your GPUs.

s65
00:05:51.280 --> 00:05:52.720
So, how do we manage that?

s66
00:05:52.720 --> 00:05:54.000
Well, well, where did we start?

s67
00:05:54.000 --> 00:06:04.080
We started where um synthetic data wasn't integrated in our product was something that we would do in a research setting uh on a cluster that that runs slurm that is away from our core kubernetes-based

s68
00:06:04.080 --> 00:06:04.800
pipeline.

s69
00:06:04.800 --> 00:06:11.039
Uh we had this two-track system where kind of the product and the research team would would not use the same type of codebase.

s70
00:06:11.039 --> 00:06:16.960
This was slow, errorprone, um, limited in velocity and kind of led to underutilizing our GPUs.

s71
00:06:16.960 --> 00:06:19.680
We have a set number of GPUs like everybody out there.

s72
00:06:19.680 --> 00:06:28.720
And in general, we want to be able to to maximize the the the use of our GPUs by running this in a in a cohesive way both with training and synthetic data runs.

s73
00:06:28.720 --> 00:06:30.080
Um, so where do we start?

s74
00:06:30.080 --> 00:06:33.680
In general, um, we'd like to run everything on on Kubernetes.

s75
00:06:33.680 --> 00:06:37.360
Um, a lot of our pipeline is either Ray or Spark.

s76
00:06:37.360 --> 00:06:46.240
Today we're going to talk about um kind of ray and cube and vlm um those are kind of the core components of our synthetic data library um or data

s77
00:06:46.240 --> 00:06:47.600
um kind of pipelines.

s78
00:06:47.600 --> 00:06:51.280
So today what we do is we have an orchestration layer in one kubernetes cluster.

s79
00:06:51.280 --> 00:07:03.440
We um run spark in another Kubernetes clusters and um we we keep the aster tracker which is our data catalog in the original control cluster where everything um kind of gets coordinated

s80
00:07:03.440 --> 00:07:11.280
um into um all the data is stored in S3 that's our storage layer um as long as it speaks kind of S3 compatible APIs we can run it.

s81
00:07:11.280 --> 00:07:16.720
So in some ways we everything needs to run in Kubernetes because our pipelines get deployed in all sorts of environments.

s82
00:07:16.720 --> 00:07:24.000
Um it can be AWS, it can be GCP, it can be onrem etc. We want to make sure that that um kind of we can run it in any

s83
00:07:24.000 --> 00:07:25.199
um Kubernetes environment.

s84
00:07:25.199 --> 00:07:27.599
And then separately we had this H100 cluster.

s85
00:07:27.599 --> 00:07:30.960
Um it ran synthetic, it run training, it ran eval kind of a number of things.

s86
00:07:30.960 --> 00:07:33.840
All of them originally orchestrated by slurm.

s87
00:07:33.840 --> 00:07:37.039
Um moving data in between these two clusters was manual.

s88
00:07:37.039 --> 00:07:43.120
So whenever you generated synthetic data in slurm, you had to backfill it into the the asset tracker into our data catalog.

s89
00:07:43.120 --> 00:07:44.080
So where are we now?

s90
00:07:44.080 --> 00:07:50.319
So um kind of same setup in some ways where we have the the original cluster um that that does the control.

s91
00:07:50.319 --> 00:07:57.120
We have the compute cluster that that we run spark jobs and now we have this other H100 compute cluster in in our case we run everything on AWS.

s92
00:07:57.120 --> 00:07:58.319
We run it on hyper pod.

s93
00:07:58.319 --> 00:08:01.280
Hyperpod has this product called um EKS on hyper pod.

s94
00:08:01.280 --> 00:08:07.599
So it's basically a bunch of H100 clusters on the same spine all of them um orchestrated by uh by Kubernetes.

s95
00:08:07.599 --> 00:08:23.440
So what we do is we continue to run spark and um ray jobs on demand capacity on demand CPUs and sometimes on demand GPUs um especially for for things that we don't require H100s we can do that um and then we orchestrate it in a similar way that that we've done before where kind of spark runs within

s96
00:08:23.440 --> 00:08:40.800
its Kubernetes cluster and then ray runs within um the same cluster for um GPUs that are not H100s and anything that we need to do for um H100 nodes we go to the this other Kubernetes cluster and we we kind of have a job scheduler that we've built internally that that orchestrates all of these things.

s97
00:08:40.800 --> 00:08:45.519
Um if we do this then this allows us to to run fast experiments.

s98
00:08:45.519 --> 00:09:00.240
Imagine that you can run kind of um curate where you run a bunch of spark jobs, you run a bunch of synthetic data, then you go and and launch a training job and then you run an eval job and you can orchestrate all of this together in one workflow which um for our research team is is quite

s99
00:09:00.240 --> 00:09:08.080
useful and this is puts the the the research flow and the engineering flow in in the same lines in the same infrastructure in the same code in the same path.

s100
00:09:08.080 --> 00:09:08.399
Okay.

s101
00:09:08.399 --> 00:09:11.120
So, um we where are we now, right?

s102
00:09:11.120 --> 00:09:12.240
Like where Oh, look.

s103
00:09:12.240 --> 00:09:15.440
I have a a oneonone with someone on this fine.

s104
00:09:15.440 --> 00:09:22.880
Um where where we're at is um kind of synthetic data is is a core thing to to our workflows.

s105
00:09:22.880 --> 00:09:25.279
Um we have we run experiments all the time.

s106
00:09:25.279 --> 00:09:27.279
We we do this crosscluster scheduling.

s107
00:09:27.279 --> 00:09:28.160
It's a lot more faster.

s108
00:09:28.160 --> 00:09:30.959
It's a lot more reliable and we can run it at scale.

s109
00:09:30.959 --> 00:09:34.720
So, um we've recently done a massive um synthetic data run.

s110
00:09:34.720 --> 00:09:38.640
Uh let's talk a bit about the the engineering lessons that that we learned from this right.

s111
00:09:38.640 --> 00:09:42.560
So um what are the bottlenecks at at trillion token scale?

s112
00:09:42.560 --> 00:09:54.720
Um well first and foremost um internet scale um for for text data set is is quite limited right on on the web we can find about 30 trillion tokens give or take for um

s113
00:09:54.720 --> 00:10:10.480
images there's about 2 12.8 8 billion in in things like dataccom for or for text we usually start from existing highquality data sets like fine web dcmotron um we can we can um kind of put them all together and we get this this large data sets

s114
00:10:10.480 --> 00:10:24.640
um but at this scale um usually everything gets stored in parquet or some sort of of format you have millions of data partitions um if you're trying to fetch all of them from s3 at a high rate you might get throttled and then if you especially if you run this on ray

s115
00:10:24.640 --> 00:10:32.320
um your your ray metadata uh is becomes and how you manage that becomes a bottleneck and that leads to the the worst thing possible which is that you get

s116
00:10:32.320 --> 00:10:33.680
idle GPUs.

s117
00:10:33.680 --> 00:10:41.760
So bottleneck um that that we initially seen is that we have um S3 we try to fetch things from our array head.

s118
00:10:41.760 --> 00:10:45.040
um this has some sort of metadata store.

s119
00:10:45.040 --> 00:10:57.200
um this is kind of quite slow and and if you do one request that takes 10 milliseconds and and um you're trying to to do I don't know millions and millions of them add I don't know let's say

s120
00:10:57.200 --> 00:11:08.800
um if you need to do um kind of the metadata fetches originally those are more expensive um and there's also a rate limit so if you need to do millions of them you're going to get rate limited then your workers are going to be quite slow

s121
00:11:08.800 --> 00:11:16.320
um so we kind of ran the the the back of the anvil of math just to populate the the metadata for um trillions and trillions in tokens.

s122
00:11:16.320 --> 00:11:20.000
It would take about 9 to 11 days, which is unacceptable, right?

s123
00:11:20.000 --> 00:11:23.120
You if you're running this, you're probably running on millions of dollars in compute.

s124
00:11:23.120 --> 00:11:27.680
You cannot have them um just waiting for you to fetch metadata for for 11 days.

s125
00:11:27.680 --> 00:11:38.880
So, the the um way that we've went past this is that um instead of of kind of um kind of going to S3, you can actually do um requests that are

s126
00:11:38.880 --> 00:11:40.079
um batched together.

s127
00:11:40.079 --> 00:11:43.200
So, instead of doing one at a time, you can batch them all together.

s128
00:11:43.200 --> 00:11:45.360
S3 supports this through their APIs.

s129
00:11:45.360 --> 00:11:58.320
And then this way um you can you can do this uh at at a massive scale by kind of increasing the the um the page size of your if your batches to to kind of the limit that that S3 gives you

s130
00:11:58.320 --> 00:12:09.040
um which is about a thousand um kind of requests in one list and that gets you to about um kind of two hours of API processing time to get your your metadata.

s131
00:12:09.040 --> 00:12:22.399
So, first lesson, um, make sure that you know how you manage your metadata and make sure that you have, um, kind of thought through just getting the state of your S3 buckets into whatever synthetic data system that you have in order for you to,

s132
00:12:22.399 --> 00:12:25.760
um, kind of be able to to even start doing this at scale.

s133
00:12:25.760 --> 00:12:28.240
So, 11 days to two hours.

s134
00:12:28.240 --> 00:12:30.880
Uh, that's that's the first one.

s135
00:12:30.880 --> 00:12:33.600
Um, awesome.

s136
00:12:33.600 --> 00:12:35.760
So, the second one is GPU instability.

s137
00:12:35.760 --> 00:12:42.720
So whenever you're running um these types of large scale synthetic data runs, of course you're going to hit some sort of bad GPUs, right?

s138
00:12:42.720 --> 00:12:44.800
So that means that the machine might go down.

s139
00:12:44.800 --> 00:12:46.320
Uh GPU might be bad.

s140
00:12:46.320 --> 00:12:48.160
Something around those lines.

s141
00:12:48.160 --> 00:12:51.279
Everything that you can expect when you try to run it at scale would happen.

s142
00:12:51.279 --> 00:13:00.160
So that leads to loss progress that leads to kind of wasted compute, partial outputs, um if you if you only parse through half of the file.

s143
00:13:00.160 --> 00:13:05.920
And of course at the end of the day you're going to uh need more cluster time which is always hard to get.

s144
00:13:05.920 --> 00:13:16.560
Um so the the the way that we do this in in our pipeline is that we have a Spark job that outputs the the the prompts together with the the documents that we need to rephrase that it's it stitches them together.

s145
00:13:16.560 --> 00:13:21.279
Um and then we start a rig job that that can actually uses the GPUs, right?

s146
00:13:21.279 --> 00:13:27.200
Um so the the problem is that there's some sort of onetoone mapping between input and output.

s147
00:13:27.200 --> 00:13:31.120
That's that's kind of how we we usually structure these types of jobs.

s148
00:13:31.120 --> 00:13:44.240
Um, in practice, if your partitions are um quite large, what happens is that um you're going to do a bunch of of of um kind of progress.

s149
00:13:44.240 --> 00:13:55.120
And then if you if if a refrazing job on one node for one file for one partition takes eight hours and the box goes down in the last five minutes of those eight hours, you're going to lose a lot of time.

s150
00:13:55.120 --> 00:14:09.519
So you need to think about um kind of you either have to checkpoint and whenever you do progress within the synthetic data um jobs you need to periodically say okay let me stop and like flash this back into S3

s151
00:14:09.519 --> 00:14:15.040
or um you need to have partitions that are small enough that you can tolerate that downtime.

s152
00:14:15.040 --> 00:14:20.800
So um you need to to think about either your partition size or in general kind of have some sort of checkpointing.

s153
00:14:20.800 --> 00:14:28.399
Um what we've done is is a bit of both where we've managed to right size our partition sizes and checkpoint as well when it comes to this things.

s154
00:14:28.399 --> 00:14:39.600
So um basically whenever um a job fails it gets retrieded but the amount of um kind of time that we lose is it's minimal and it makes sense for the setup that we have.

s155
00:14:39.600 --> 00:14:47.199
Um so from multi-day job failures now we have recoverable retries and it it it works it works as as expected.

s156
00:14:47.199 --> 00:14:59.199
So just to recap resumeum execution some sort of IDM potency where whenever you're checkpointing you can override the file that you're checkpointing into and then you need to have some sort of mechanism for for simless recovery

s157
00:14:59.199 --> 00:15:01.040
um cross infrastructure orchestration.

s158
00:15:01.040 --> 00:15:11.120
So in in our case the the the cluster where we origin run curation in general our pipeline runs in a centralized location and then customers might have synthetic data

s159
00:15:11.120 --> 00:15:25.360
um jobs that run in various GPU locations throughout the world right you might it's going to be hard to get especially for synthetic data where you don't need necessarily fast networking in between your GPU nodes to get a lot of capacity in one place so what happens then is that you need to be able to orchestrate

s160
00:15:25.360 --> 00:15:33.199
all of these jobs um in in various clusters so it's not just one GPU cluster that you're working with you're working with um kind of a number of them.

s161
00:15:33.199 --> 00:15:44.880
Um the problem here is that whenever you have um and you're trying to to to schedule um these jobs is that you might be able to um kind of schedule your

s162
00:15:44.880 --> 00:15:51.680
CPU requirements for your ray jobs but not necessarily your um kind of GPU requirements in the GPU pool.

s163
00:15:51.680 --> 00:15:57.199
So you need to think about kind of being able to schedule in some sort of fashion across these two things.

s164
00:15:57.199 --> 00:16:08.880
Now um once you have the the next job coming in um you have GPU pull capacity but you don't have CPU pull capacity right so in general whenever you're trying to do this this type of cross infra

s165
00:16:08.880 --> 00:16:21.120
or orchestration you need to think about scheduling and you need to think about um kind of being able to schedule this effectively um and in this case as I mentioned you you you will get to a point where you cannot schedule effectively

s166
00:16:21.120 --> 00:16:28.079
so what happens then is that um you you kind of need to think about the the ways in which you do this in a in a proper fashion.

s167
00:16:28.079 --> 00:16:31.759
So some sort of bin packing problem at the at the end of the day.

s168
00:16:31.759 --> 00:16:34.480
Uh so how did we solve this?

s169
00:16:34.480 --> 00:16:36.560
Um so same setup as before.

s170
00:16:36.560 --> 00:16:39.279
Um we have this orchestrator that runs in a different cluster.

s171
00:16:39.279 --> 00:16:47.040
We have the compute cluster that runs spark and the the ray jobs that mostly need CPU and then we have the the one that has H100s.

s172
00:16:47.040 --> 00:16:54.160
Um so whenever we schedule um we have dedicated pools for the specific resources that you need in those clusters.

s173
00:16:54.160 --> 00:17:04.720
So if you have a spark requirements then you spark you might have a a spark pool that that has CPUs and then you dedicate special pools for um kind of the the ray head.

s174
00:17:04.720 --> 00:17:12.880
So when you think about um some sort of atomic scheduling you need to think about both your CPU and your GPU requirements in order to do this optimally.

s175
00:17:12.880 --> 00:17:22.559
So in this case whenever we have more CPU um requirements that we need to schedule because we've decoupled these two things um and and then we can schedule the the

s176
00:17:22.559 --> 00:17:33.200
um head and the worker for array um in a in a more atomic fashion then we don't have the problems as as before um and then everything scales as expected.

s177
00:17:33.600 --> 00:17:44.400
So cross in front orchestration um you need to think about multiple providers multiple um clouds that you run this in you need to think about scheduling and u make sure that whenever you schedule

s178
00:17:44.400 --> 00:17:50.320
CPU and GPU resources especially for the same type of job you do that in a way that that makes sense.

s179
00:17:50.320 --> 00:17:53.120
Um last up is inference optimizations.

s180
00:17:53.120 --> 00:18:03.840
Um so whenever you run VLM or whenever you run SG lang um you need to think about the the hyperparameters that you use for your jobs right and uh what what we found is that you need to be

s181
00:18:03.840 --> 00:18:13.679
u quite disciplined about this and you need to be able to to benchmark this and optimize and have a harness that just focuses on the VLM parameters you can get um a lot of gains

s182
00:18:13.679 --> 00:18:25.360
um in the case in this case about 40% throughput gains just by tweaking the flats for VLM um and SG line right so in our case we we went with VLM Right now we're actually testing HGLang as well.

s183
00:18:25.360 --> 00:18:26.960
We do a lot of batch inference.

s184
00:18:26.960 --> 00:18:32.320
Um which is very much different from from the the normal online one that everybody is used to.

s185
00:18:32.320 --> 00:18:41.360
Um but whenever you you kind of orchestrate this through VLM, you need to make sure that you have the the right batch size, you have the right speculative decoding

s186
00:18:41.360 --> 00:18:50.880
um setup, etc. Um if you do kind of a bunch of grid searches over the parameters that you need to optimize, turns out you can get quite a lot of uh of results in terms of throughput.

s187
00:18:50.880 --> 00:18:58.559
So to recap, um whenever you you're trying to do things, make sure that um you think about kind of uh the scale that you're running at.

s188
00:18:58.559 --> 00:19:02.320
You have a good system for auto recovery whenever jobs go down.

s189
00:19:02.320 --> 00:19:06.720
You need to orchestrate in some sort of atomic fashion across CPU and GPU resources.

s190
00:19:06.720 --> 00:19:11.200
Um and you need to make sure that that you kind of sweep over your VLM parameters.

s191
00:19:11.200 --> 00:19:15.679
So what what this setup helped us achieve is is quite straightforward.

s192
00:19:15.679 --> 00:19:26.080
Um and in at the end of 2014, sorry, 2024, uh we can only output around 30 billion synthetic tokens in kind of limited runs within our SERM cluster.

s193
00:19:26.080 --> 00:19:29.280
Uh our requirements for synthetic data went up by by a bunch.

s194
00:19:29.280 --> 00:19:42.880
And then um kind of end of last year, we were able to to run this massive uh synthetic data jobs that that um generate in this case for web about 7 trillion tokens and then for math and code another five for a bunch of our customers.

s195
00:19:42.880 --> 00:19:47.039
uh we're continuing to ablate and and kind of search for better synthetic G recipes.

s196
00:19:47.039 --> 00:19:48.880
Um we're actively working towards this.

s197
00:19:48.880 --> 00:19:49.919
So keep an eye out.

s198
00:19:49.919 --> 00:19:50.960
We'll be publishing more.

s199
00:19:50.960 --> 00:19:53.360
And um I think domain coverage is quite important.

s200
00:19:53.360 --> 00:20:05.039
I think something that we haven't talked today is how does this differ in between web, multilingual, math code, uh legal data, etc. By the way, we're hiring um throughout the board for data infrastructure, cloud infrastructure,

s201
00:20:05.039 --> 00:20:09.120
normal infrastructure, program managers, etc. So hit me up afterwards.

s202
00:20:09.120 --> 00:20:14.720
Uh thank you for the team that worked on this and uh thanks everybody for attending my talk.

s203
00:20:28.139 --> 00:20:30.139
[music]