WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=-tviRdpmHvs

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/-tviRdpmHvs.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.120 --> 00:00:13.840
Okay.

s3
00:00:13.840 --> 00:00:15.759
So, uh I'm Sa.

s4
00:00:15.759 --> 00:00:16.640
I'm from Korea.

s5
00:00:16.640 --> 00:00:26.960
I'm going to be talking a little bit about uh how we recently trained uh our like image foundation model uh Korea 2 as well as and we also recently like open sourced

s6
00:00:26.960 --> 00:00:28.800
uh the medium version of our model.

s7
00:00:28.800 --> 00:00:33.200
So I'll be like talking about like how we like trained it mostly from research perspective.

s8
00:00:33.200 --> 00:00:43.840
Uh later in the day uh like my colleague will also give a little bit of details on like infrastructure and training infrastructure and like everything that needed to be like set up for that.

s9
00:00:43.840 --> 00:00:51.120
But today I'll mostly be talking about like research also public speaking is not one of my best abilities.

s10
00:00:51.120 --> 00:00:56.160
So take that in mind but anyways I'll go ahead and start.

s11
00:00:56.160 --> 00:01:05.760
So uh we rec as I said we recently open source our uh create 2 uh creative to medium uh variant of our like closed source model.

s12
00:01:05.760 --> 00:01:09.600
Now it's like open source and I think people are enjoying it quite a bit.

s13
00:01:09.600 --> 00:01:15.520
One thing we focused quite a bit was like stylistic uh diversity.

s14
00:01:15.680 --> 00:01:27.439
So how we are going to like go about this is that I'm going to like describe a little bit of the training like pipeline uh that we have like used to like curate data curate data actually like train the model

s15
00:01:27.439 --> 00:01:42.640
train the model and then I'll like talk a little bit about like what were like the most effective levers for actually improving model performance and then the last thing would be you know like what I think would be are like kind of exciting directions for the next

s16
00:01:42.640 --> 00:01:44.159
uh generation image model.

s17
00:01:44.159 --> 00:01:48.159
So I'll go ahead and then start talking about that.

s18
00:01:48.159 --> 00:02:00.719
So one of the trend that we've been like seeing recently is that you know existing models like chipt uh chach nanopar pro like the like the production grade uh models that are like from the big labs.

s19
00:02:00.719 --> 00:02:06.479
One thing is that they kind of like focus on like slower generation but like very reliable output.

s20
00:02:06.479 --> 00:02:13.280
So I don't know chachi BT2 or like nanobanara pro I think they take up to like a minute or like two to like give an output.

s21
00:02:13.280 --> 00:02:15.200
typically is like something very acceptable.

s22
00:02:15.200 --> 00:02:17.760
There's like hardly barely any like flaws.

s23
00:02:17.760 --> 00:02:25.520
But one problem is that in order to get like good consistency, they have like kind of like significantly mode collapse their models.

s24
00:02:25.520 --> 00:02:33.360
Because I don't know like if you're trying to render a person, the easiest and most reliable way to like render a person is render the most boring average person

s25
00:02:33.360 --> 00:02:35.760
that exists and then like put it in a center frame.

s26
00:02:35.760 --> 00:02:40.720
Uh and like this for example like this is one of the examples that I like to use.

s27
00:02:40.720 --> 00:02:56.400
So like if you type like burning skull ch2 like it's very consistent all the outputs are like fine but you know there's barely any diversity on the other end like kind of the thing that we kind of like focus on is like faster generation so that like people can like iterate with like different visual ideas and then

s28
00:02:56.400 --> 00:03:06.640
also have like little bit of knobs on kind of because you know when people like come in sometimes like they want very something very specific they want to generate like a poster or

s29
00:03:06.640 --> 00:03:17.120
like a birthday card in that case like chatb2 or nano bar pro excellent solution, but sometimes like when you're like a creative studio, you don't quite know what you want yet and

s30
00:03:17.120 --> 00:03:20.560
want to like slightly explore what kind of like visuals you want to make.

s31
00:03:20.560 --> 00:03:24.480
That's something that we wanted to uh focus on a little bit.

s32
00:03:24.480 --> 00:03:30.480
So, you know, with that in mind, like how do you actually like train a diffusion model?

s33
00:03:30.480 --> 00:03:37.760
Um I hope most of the people in the audience like know how diffusion models work but just to give a small recap.

s34
00:03:37.760 --> 00:03:51.840
How it works is that you have an image and you add a little bit of like noise and then you ask the model to like okay like how here is an image with some of the noise added in like how would you like remove this noise

s35
00:03:51.840 --> 00:03:54.239
remove this noise to get like a valid image.

s36
00:03:54.239 --> 00:04:05.360
So like at a high level this is how diffusion models are like trained and once you train a model to do this you can just like get a random noise and then tell the model to like progressively

s37
00:04:05.360 --> 00:04:08.560
remove the noise uh noise to generate like an image.

s38
00:04:08.560 --> 00:04:12.400
So at a high level like this is how a diffusion models are like trained.

s39
00:04:12.400 --> 00:04:22.079
But right now most of the production uh production diffusion models at least the open source ones that are very competitive including ours use older encoder.

s40
00:04:22.079 --> 00:04:35.520
So this is typically called a latent diffusion model where you use a autoenccoder to like instead of taking the raw pixels you first like spatially compress it and then decompress it uh later but the generation works in a spatially

s41
00:04:35.520 --> 00:04:39.280
compressed uh latent space so that it's a little bit more efficient to train.

s42
00:04:39.280 --> 00:04:48.240
One of the main reason is that most people use diffusion transformers uh transformers or like some variant of that to train diffusion models.

s43
00:04:48.240 --> 00:04:59.280
And as you know transformers at least the non-sparse ones uh they tend to have like n squared uh or n squared uh time complexity.

s44
00:04:59.280 --> 00:05:03.360
So if you try to like model every single pixel that's very expensive.

s45
00:05:03.360 --> 00:05:16.639
So now uh and that was the initial motivation for like why people kind of like started with like latent uh latent diffusion models because now they can actually like model this a little bit in a more efficient space.

s46
00:05:16.639 --> 00:05:26.000
So one thing that I'll say is that like really like data is like quite everything that goes into the model like typically you lock in your architecture and then you just

s47
00:05:26.000 --> 00:05:29.840
a lot of work just goes into like just feeding the model like what it wants.

s48
00:05:29.840 --> 00:05:33.440
Like I mean this sounds stupid but I cannot iterate this more.

s49
00:05:33.440 --> 00:05:46.720
That's why I put like again like really data is quite like everything like again like this this just comes up again and again and typically after you lock in your architecture like most of the work actually goes into like data curation

s50
00:05:46.720 --> 00:05:53.039
like making sure that the data is good and in our case like we wanted to focus on like stylistic diversity.

s51
00:05:53.039 --> 00:06:11.600
So making sure that like we don't necessarily like remove like remove like remove like data data unintentionally to like cut stylistic diversity like that was also very important focus for us for instance like some people like think I know like low resolution CRT videos are like

s52
00:06:11.600 --> 00:06:24.960
a bad image but some people like that kind of like aesthetic so making sure that we have like good coverage and don't just rely on like very standard like aesthetic scores or like image quality scores to like c like over sample

s53
00:06:24.960 --> 00:06:31.039
uh typically what are considered like conventionally good images was also something that we had to like take into account.

s54
00:06:31.039 --> 00:06:46.319
So with that in mind I'm going to describe some of the things that I think we did uh quite cool when we came to like data and kind of like things that we like kind of consider bad data are like duplicated samples over represented concepts.

s55
00:06:46.319 --> 00:07:00.080
So these are typically taken uh care of by like dduplication and and uh clustering uh based uh rebalancing and then you know there are certain samples that like vision language models we use to generate the captions.

s56
00:07:00.080 --> 00:07:06.880
They sometimes consistently fail to like capture important aspect of the image which leads to certain biases.

s57
00:07:06.880 --> 00:07:08.960
Uh which is also the third point.

s58
00:07:08.960 --> 00:07:18.800
And then like you know sometimes when you train on like very low resolution it doesn't make sense to like put train on an image that has like I don't know like 2000

s59
00:07:18.800 --> 00:07:28.319
like characters on a 256 by 256 pixels because that's just going to be like too hard for the model to like learn at least at like low resolution stages.

s60
00:07:28.319 --> 00:07:30.080
So that's one uh example.

s61
00:07:30.080 --> 00:07:44.960
And then obviously AI images uh distillation is like a very big like topic but we try very hard to like remove any AI images like at all because it does like provide you a shortcut to to get you like a good model but

s62
00:07:44.960 --> 00:07:55.919
synthetic data is like so sticky to the model that once you like start training on AI image data sure your model is good but you kind of lose the point because then you're going to get like very like chatbt

s63
00:07:55.919 --> 00:08:09.599
or like nanob aesthetic and at least for me like I can tell when like model has been very heavily trained or like distilled on like CHPD2 or like nanob pro and in the long run like that's just not how you want to like do like you don't I

s64
00:08:09.599 --> 00:08:15.120
and you know as a researcher it also slightly hurts my ego if all I'm doing is distillation.

s65
00:08:15.120 --> 00:08:20.000
So yeah and then you know captions are very important.

s66
00:08:20.000 --> 00:08:28.560
So like just to briefly describe our like captioning pipeline how we we take the images and then we run OCR because text rendering is quite important.

s67
00:08:28.560 --> 00:08:36.640
Uh so we make sure that we first extract out all the text that are like visible in the image and then we also add like optional metadata.

s68
00:08:36.640 --> 00:08:41.440
If we know this is like a picture of a famous person, we make sure that that's included.

s69
00:08:41.440 --> 00:08:47.920
And then we do a second pass with a visual language model to generate a very detailed caption for this image.

s70
00:08:47.920 --> 00:08:57.600
And once we have like a caption that sufficiently captures all the things that are relevant about the image, then we can like rewrite it to like what like JSON prompts or like other formats

s71
00:08:57.600 --> 00:09:00.640
you would like to like uh feed the model.

s72
00:09:00.640 --> 00:09:09.519
So like this our like captioning pipeline and this is actually one good example of what I consider like bad data.

s73
00:09:09.519 --> 00:09:10.959
It doesn't look that bad.

s74
00:09:10.959 --> 00:09:22.800
But one of the issues that we had with this kind of images is that we would like try many things with the captioner uh and it would say that oh this is a painting of blah blah

s75
00:09:22.800 --> 00:09:34.160
blah blah but it would not mention the fact that it's framed on the wall with like on and have a white background and I know sometimes the captioner would like consistently like fail to mention this fact.

s76
00:09:34.160 --> 00:09:41.040
So when you try to generate a painting of whatever it will be always hanged on a wall on a white wall which is probably not what the user wants.

s77
00:09:41.040 --> 00:09:56.000
So this is an example of like you know the image is fine it you can train on it but if it's a kind of image that you know LLMs or vision language models cannot like consistently like capture important aspect of the model like this is an example where we just had to like

s78
00:09:56.000 --> 00:10:12.320
design filters and then throw this kind of like data out or at least under sample it and you know dduplication we mostly use like hashbased solutions so we when we like train these models we need to use like like I don't know anywhere from two to like 10 billion images.

s79
00:10:12.320 --> 00:10:15.519
That's a lot of images to run like filters on.

s80
00:10:15.519 --> 00:10:29.920
So like first thing we do is like just tip calculate like P hash or like MD5 hash to do basic dduplication and then once we get to like a smaller like size that's when we bring in you know some of the embedding based uh dduplication method SSCD

s81
00:10:29.920 --> 00:10:42.399
like cichlip to do uh se semantic duplication or like near dup uh remove near duplicates and like typically how we like also like design filter is that we use like large

s82
00:10:42.399 --> 00:10:59.600
uh large vision language model I know for instance is you design a prompt to like get a large language model to like learn does this look like a AI image or not and then like we can after like we get a good like fine tune or a system from from a large uh vision language model

s83
00:10:59.600 --> 00:11:12.160
one thing that uh we can do is we can like distill this data this kind of like decision and like knowledge to like a very small like cichlip classifier and then you can bas you have like a very like cheap classifier

s84
00:11:12.160 --> 00:11:22.160
that is somewhat like reliable and you know like typically if you run a classifier over like a billion images you do need things to be like cycllip sized uh for instance.

s85
00:11:22.160 --> 00:11:31.120
So this is one of and also like this is if you guys are into like LM literature like this is one of the approaches that uh that essential web data is used.

s86
00:11:31.120 --> 00:11:43.920
They also used the big LM big LM to like come up with like some kind of taxonomy or like classifiers to like judge whether this text data is like good or not in this and then distilled that down to like I don't know like 500

s87
00:11:43.920 --> 00:11:55.120
million parameter model so that you can actually run this over like a pre-training level corpus otherwise it'll be somewhat expensive and frankly inefficient use of your GPUs to do so

s88
00:11:55.120 --> 00:12:02.480
and another thing which I'm a little bit proud of uh is that we actually use like sparse order encoders uh for some of our filtering.

s89
00:12:02.480 --> 00:12:12.959
Uh I don't know how many of you guys are still like into like sparse auto encoders but I we did some work or at least I did some work in doing

s90
00:12:12.959 --> 00:12:23.120
uh sparse auto encoder research on like clip or these kind of like image models vision models and one thing that you can actually get out of SAPE is unsupervised

s91
00:12:23.120 --> 00:12:23.920
tagging system.

s92
00:12:23.920 --> 00:12:36.320
So one thing you can do is that once you train a SAPE on a vision model what you can do is that you can feed an image and then it will give you like sparse features that get activated for instance

s93
00:12:36.320 --> 00:12:46.240
let's say you feed this image to a sparse odo vision sparse odo encoder and it will give you like it will get activated for like features like horse black and white

s94
00:12:46.240 --> 00:12:56.160
blur uh blurry like image blurry image so once so you can kind of use this as like offtheshelf of like unsupervised tagging system.

s95
00:12:56.160 --> 00:13:07.360
And if one of these like have like something that you want to like filter on or over sample on like easy things are like signatures or like watermarks or like some kind of like border artifacts that I was like talking about.

s96
00:13:07.360 --> 00:13:17.600
So if you have like a feature for that in your sparse auto encoder like this is one nice thing to remove uh to like remove like data that's kind of like undesirable

s97
00:13:17.600 --> 00:13:19.680
uh in your data set.

s98
00:13:19.680 --> 00:13:26.639
And then one thing that I think like people have quite liked about our open source create to model is like world knowledge.

s99
00:13:26.639 --> 00:13:39.040
Frankly, I don't know how much this has helped, but this is one of the things we did uh is that you know this is also something that like clip the original clip paper did is that you can actually take the Wikipedia

s100
00:13:39.040 --> 00:13:52.079
the entire Wikipedia and for each article or concept you can compute the page rank of each of the concepts and then if it's like if it has like quite high it's in the top 90%

s101
00:13:52.079 --> 00:13:56.240
tile of like page rank on Wikipedia is probably something quite important.

s102
00:13:56.240 --> 00:13:57.360
for the model to know.

s103
00:13:57.360 --> 00:14:11.760
And what we do is uh what we do is we kind of like take these keywords and then make sure that and then just run like I know like standard like plain text search or like embedding search just to make sure that like these kind of concepts are in our data set.

s104
00:14:11.760 --> 00:14:19.920
And during this little bit of exploration like one thing that we did find was that there's a Barack Obama the horse.

s105
00:14:19.920 --> 00:14:34.079
I actually don't know if this ended up in our data set, but you know, like there are many interesting things in the Wikipedia article, so something fun to share, but actually I don't think we had this specific bra Obama the horse, but it's it's a nice horse.

s106
00:14:34.079 --> 00:14:43.920
Anyways, uh yeah and and the day like we have like around like 30 to we I counted uh and then I think we ended up having around like 30 to 40 like

s107
00:14:43.920 --> 00:15:01.519
custom in-house classifier like different heristics and like filters that we've used and again like data is very important because once you lock in your I think now we're I mean you're probably going to train like a transformer you know what you're training and then like data is like what really determines the quality of your model.

s108
00:15:01.519 --> 00:15:06.240
So again, can't emphasize of how important data is.

s109
00:15:06.240 --> 00:15:17.120
And then like once your data is set up like we kind of go through this like very LLM uh inspired training pipeline where we do uh we progress we do like

s110
00:15:17.120 --> 00:15:31.279
uh low resolution to like high resolution pre-training mid-training supervised fine tuning preference optimization uh reinforcement learning and we also like train our uh prompt expander which takes in user prompt and then expands that to a long prompt.

s111
00:15:31.279 --> 00:15:42.079
So this is pretty straightforward you know at typically most people like train uh start training at low resolution because that's where the model actually learns like text to image capabilities

s112
00:15:42.079 --> 00:15:58.399
like you know like it needs to know like how a horse looks like you can do train that at low resolution and then you can progressively like scale up your training resolution so that it first learns semantics and then learns like structure detail like these kind of things that can be learned at high resolution later.

s113
00:15:58.399 --> 00:16:02.959
So we've trained from 256 to 1k resolution.

s114
00:16:02.959 --> 00:16:08.959
And then once you have your like pre-trained model, you kind of have this very malleable base to like train on.

s115
00:16:08.959 --> 00:16:14.639
And similar to like LMS like if you pre-train LM is just basically autocomplete.

s116
00:16:14.639 --> 00:16:20.399
But typically at least for LM you wanted to do like chat or like agent tech stuff tool calling.

s117
00:16:20.399 --> 00:16:24.240
So you actually need to like mold this into like things that are useful for you.

s118
00:16:24.240 --> 00:16:33.839
So in our case we curate like illustration, graphic design, photography, cinematics, you know, kind of like data you want you have in mind for your downstream use case.

s119
00:16:33.839 --> 00:16:40.959
So we curate some like large scale mid-train data and then SFT data to kind of mold your distribution.

s120
00:16:40.959 --> 00:16:51.199
And once this little bit of like molding is done, then we do like preference optimization, which is, you know, if you do CHP, they'll ask you like do you like this over this?

s121
00:16:51.199 --> 00:16:59.519
So we collect bunch of these pairs where we use this for doing a little bit of like preference optimization to just polish out the model a little bit.

s122
00:16:59.519 --> 00:17:06.400
This is where we get a little bit more opinionated need opinionated about like the kind of model we want to train.

s123
00:17:06.400 --> 00:17:12.720
And then you know reinforcement learning is also like kind of is now extremely standard in LMS.

s124
00:17:12.720 --> 00:17:14.559
That's what we do in diffusion land too.

s125
00:17:14.559 --> 00:17:31.760
So we have like a pretty much like a gpo inspired uh method where where we have model like generate images and then we send that to the reward servers and then based on this like feedback we teach the model like I don't know like how to like improve text rendering like

s126
00:17:31.760 --> 00:17:46.559
have like better anatomy structure these kind of things and then the last step this is kind of become a almost essential step for production grade diffusion models you actually need to train a small LM that takes in like user prompt and then outputs is like a very long detailed prompt.

s127
00:17:46.559 --> 00:18:02.960
Uh because typically longer detail prompt they're more in distribution with your models like training data that tends to like make better images and you know the kind of next step if you're into like uh LM literature is something that we are doing is uh doing like multi-expert

s128
00:18:02.960 --> 00:18:05.679
on policy distillation that we are like currently working on.

s129
00:18:05.679 --> 00:18:17.120
So we would like train experts that are like specialized in like photography, text rendering and uh different capabilities and then kind of like merge all of these capabilities into a single student.

s130
00:18:17.120 --> 00:18:30.559
So so that like we can have like kind of all we can have a student that can effectively match the capabilities of each of the expert in like whatever you want the expert to be uh good at.

s131
00:18:30.559 --> 00:18:44.320
And like in my experience like kind of like things that mattered for me was like infrastructure to iterate fast and also again like data is important like messes can like change every time code is something you can change very easily but you know data is like

s132
00:18:44.320 --> 00:18:51.440
eternal like you can if you have good data that's going to be valuable no matter what the hot new training paradigm is.

s133
00:18:51.440 --> 00:18:58.960
Uh simplicity and scalability I pref we very much prefer like methods that have like low number of hyperparameters that we need to tune.

s134
00:18:58.960 --> 00:18:59.520
efficiency.

s135
00:18:59.520 --> 00:19:03.360
Again, this goes ties back to just fast iteration like speed.

s136
00:19:03.360 --> 00:19:11.919
And then like you know, our thing that I like to do is steal a lot from LM research so that I can just reuse their kernels and like research and like literature.

s137
00:19:11.919 --> 00:19:16.559
So all of these are things that I kind of like found very useful to iterate.

s138
00:19:16.559 --> 00:19:25.600
I think I only have like one minute but kind of things that I find funny is that you know we started with like encoder and like decoder uh transformers.

s139
00:19:25.600 --> 00:19:32.160
Now we are going the opposite way because we have a prom expander which is all the regressive decoder and then diffusion which is the encoder.

s140
00:19:32.160 --> 00:19:40.400
So it's kind of getting reversed and with that in mind it's also starting to look a little bit like deli too where you know you have like a model

s141
00:19:40.400 --> 00:19:47.760
that would like generate conditioning using like the regressive model or like diffusion model and then we feed that to our diffusion model.

s142
00:19:47.760 --> 00:20:00.400
So it's kind of this prom expansion pipeline makes reminds me a little bit of like Dolly 2. So that's kind of a funny observation and you know like there's bunch of stuff that go into like diffusion model training.

s143
00:20:00.400 --> 00:20:09.200
I really like to you know simplify the stack so that we can get rid of VAE and then text encoders and then just train a single clean transformer.

s144
00:20:09.200 --> 00:20:12.080
Uh so this something I always look forward to working on.

s145
00:20:12.080 --> 00:20:29.120
And then you know vision language models has gotten like more powerful like this is from the original LDM paper but you know like bounding boxes before it was expensive to generate but now you can perfectly generate good bounding box for every image and you can condition the image model in this

s146
00:20:29.120 --> 00:20:42.240
uh this is something that ideoggram rev have also like been working on and then you know this is from actually 2017 from uh FA Lee uh Lee's lab like you could do like scene graph to like image generation

s147
00:20:42.240 --> 00:20:47.280
So again like image generation I think is really like a proxy for like BLM like progress.

s148
00:20:47.280 --> 00:20:58.799
So kind of things I'm like excited for is like what are other what are interesting like textual ways to like describe on image now that we have more like more like powerful visual

s149
00:20:58.799 --> 00:20:59.520
language models.

s150
00:20:59.520 --> 00:21:03.760
So this is something that I'm quite excited about like bounding boxes you know scene graphs.

s151
00:21:03.760 --> 00:21:06.960
I mean there's different things that could be like quite useful here.

s152
00:21:06.960 --> 00:21:10.960
And then yes and this uh my shameless recording slide.

s153
00:21:10.960 --> 00:21:16.880
So, if you enjoyed this talk, send me an email here, take a picture, and yeah, I think that's mostly it.

s154
00:21:16.880 --> 00:21:22.240
I don't know if there's time for Q& A, but sorry that I went slightly over, but hopefully you enjoyed it.

s155
00:21:22.240 --> 00:21:25.679
I'll stick around for under 30 minutes if anybody wants to talk to me.

s156
00:21:25.679 --> 00:21:28.000
But yeah,