WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ewtOo0scUh0

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ewtOo0scUh0.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.639 --> 00:00:21.199
Hey everyone, um today I'll be talking about data and uh environment curation for uh post- training LLMs.

s2
00:00:21.199 --> 00:00:23.439
And I am Mahesh Satyimi.

s3
00:00:23.439 --> 00:00:27.439
Um I'm co-founder and CEO of Bespoke Labs.

s4
00:00:27.439 --> 00:00:32.880
And previously I was a researcher and uh engineer at uh Google deep mind.

s5
00:00:32.880 --> 00:00:41.360
So very briefly I will tell you a little bit about uh bespoke and uh after that the talk will be mostly around uh opensource work we have done.

s6
00:00:41.360 --> 00:00:53.039
So bespoke is an applied data research lab with a mission to help enterprises and frontier labs access high quality data and RL environments for their post training needs.

s7
00:00:53.039 --> 00:01:03.680
So very briefly what we do and what we have done is that last year we put out something called curator which is a tool for curating uh synthetic data

s8
00:01:03.680 --> 00:01:15.360
for post training with basically SFT and right after that actually deepse landed and we started an effort to curate reasoning data and that's how we started something called bespoke stratos

s9
00:01:15.360 --> 00:01:26.799
which eventually formed something uh into the project called open thoughts which some of you hopefully know about and we have also been core contributors to terminal bench.

s10
00:01:26.799 --> 00:01:32.799
Um you know these days we we do a lot of research and build and ship RL environments.

s11
00:01:32.799 --> 00:01:43.920
So I was actually looking forward to the previous talk uh from Nick who is also you know uh doing something similar uh and and the other thing we do is we

s12
00:01:43.920 --> 00:01:49.759
do a lot of post training and help enterprises uh to get their own custom models.

s13
00:01:49.759 --> 00:01:49.920
Right?

s14
00:01:49.920 --> 00:01:56.320
That's the name of that's how we ended up with the bespoke uh t title for the company.

s15
00:01:56.320 --> 00:02:04.799
Uh the the other thing I want to kind of mention is there's there is this you know you know in in our industry there are a lot of people who create data

s16
00:02:04.799 --> 00:02:09.840
uh create RL environments and then there are the uh researchers who consume this.

s17
00:02:09.840 --> 00:02:18.640
But I feel like there is this slight mismatch and it's kind of beneficial for someone to kind of go do both at the same time.

s18
00:02:18.640 --> 00:02:30.160
And in fact uh as you're curating data you want to put yourself in the shoes of the researcher to see what what does it take to you know uh actually move the metrics on the models.

s19
00:02:30.160 --> 00:02:40.080
So that's one of the motivations of how we kind of think about the other thing I want to kind of talk about is you know how you know uh AI has evolved right.

s20
00:02:40.080 --> 00:02:46.000
So early on we used to think about and evaluate models on what they know.

s21
00:02:46.000 --> 00:02:59.760
Um for example this is a this was a very popular benchmark on uh testing LLMs on various kinds of STEM humanities and all that knowledge and these days we have all these benchmarks that test

s22
00:02:59.760 --> 00:03:10.239
uh how how agents are able to do things we have moved on from knowing to doing right so that's the idea of agents obviously and the one of the key

s23
00:03:10.239 --> 00:03:25.040
principles or one of the key things about agents is that they are autonomous And there are as I was saying there are many benchmarks including uh swb bench terminal bench and so on but ultimately for many people what they care about is

s24
00:03:25.040 --> 00:03:38.080
are these agents autonomous for long durations of time uh Nick had uh sorry uh Ross had a great talk on long horizon right so that's the goal is eventually we make these agents

s25
00:03:38.080 --> 00:03:48.560
autonomous for maybe few hours or you know few days or few weeks and what is it that's blocking the uh autonomy of agents.

s26
00:03:48.560 --> 00:03:50.879
It's basically reliability, right?

s27
00:03:50.879 --> 00:03:58.720
So, at some point something falls apart like either they called the wrong tool or they made a mistake and you know what what not, right?

s28
00:03:58.720 --> 00:04:02.480
And what what's one lever to improve reliability there?

s29
00:04:02.480 --> 00:04:14.239
There are of course many um obviously you can prompt your way to improving the agents u reliability or you can uh update the harness you know the tools and whatnot

s30
00:04:14.239 --> 00:04:29.280
but postraining is a very powerful tool to improve reliability or or maybe even pre-train good models right so if you think of frontier labs this is one of their primary mechanisms of improving agents over to to

s31
00:04:29.280 --> 00:04:46.479
uh get better [clears throat] u capability ities in uh various domains or you know or for better you know benchmark numbers or better u um autonomy for long longer and longer durations

s32
00:04:46.479 --> 00:04:59.280
and for post training one of the popular techniques as you know is reinforcement learning and that's kind of um something you know a lot of you are excited about is the you know notion of RL environments

s33
00:04:59.280 --> 00:05:16.400
but ultimately for post training be it SFT or or uh reinforcement learning data is the bottleneck right so when when I talk about data RLNs are also something I'm calling it as data it's just the data is now in a very different shape

s34
00:05:16.400 --> 00:05:33.360
u again here you know compute is kind of well definfined models you know uh good sort of models exist and the uh infrastructure to post train for example u the There are various providers like fireworks,

s35
00:05:33.360 --> 00:05:36.479
tinker or uh slime world and whatnot.

s36
00:05:36.479 --> 00:05:40.000
So all all of those are somewhat well defined.

s37
00:05:40.000 --> 00:05:55.039
Most of the places where people struggle especially enterprises is that they don't have access to good quality data and RLNs and this obviously also applies to frontier labs where they have all this infra setup and they are you know needing good quality RLMs

s38
00:05:55.039 --> 00:06:06.639
right uh beyond so that that's one of the this is kind of how we are thinking about why to invest time in you know doing data research and RLN research

s39
00:06:06.639 --> 00:06:18.639
and as a side note one of the um other there are many other benefits of post- training for example you can reduce latency or improve cost throughput and whatnot

s40
00:06:18.639 --> 00:06:31.039
and I'll give one concrete example of a post- training work we did uh with one of the enterprises so in this talk I will mostly uh cover some of the work we have done in the open-source

s41
00:06:31.039 --> 00:06:51.520
uh community so we did some work on curating reasoning data for reasoning models and for uh you curating trajectories and uh environments for agents and recently we had an engagement with post training which uh I I'll very briefly talk about and some tools on data curation

s42
00:06:51.680 --> 00:07:12.639
so open thoughts um is a reasoning data set as well as a paper right so we we started this effort last year as I was saying this uh we we after DC came out we realized that there is uh lack lack of very high quality reasoning data in the uh community.

s43
00:07:12.639 --> 00:07:26.560
Obviously the labs have access to good data but outside we didn't have access to data right so we we at bespoke started this effort called bespoke stratos and then we realized that this is actually quite useful so we joined

s44
00:07:26.560 --> 00:07:40.080
um together with various folks in uh Stanford UC Berkeley Udub and so on to create this consortium called open thoughts and we did lot of work on basically identifying the curation recipe

s45
00:07:40.080 --> 00:07:46.880
and we also published this as a paper in night of this year and this is the main figure of the paper.

s46
00:07:46.880 --> 00:07:52.400
So what it shows is like we we figured out a curation recipe and it shows the scaling law right.

s47
00:07:52.400 --> 00:08:01.039
So again this is last year when Amy uh and and uh live codebench and these these were some of the popular benchmarks.

s48
00:08:01.039 --> 00:08:12.639
What we showed is that with this recipe if you keep you know scaling up the data set size the the you know the the it's a scalable recipe right the the metrics also improve

s49
00:08:12.639 --> 00:08:24.479
uh it's actually very widely used as well for example this is um Microsoft cso tweeting about the work and this alle Alex is my uh co-founder he's a chief scientist

s50
00:08:24.479 --> 00:08:39.360
and also a professor at UC Berkeley and this is John Schulman talking about open thoughts that he as he and his uh colleagues have been using it internally at uh thinking machines right and some of their blog posts also reference this

s51
00:08:39.360 --> 00:08:56.320
so I'll uh talk about how we did the curation for open thoughts um this is the pipeline that we used so you start with curate you start with a bunch of source questions right so there are various data sets out there that have the

s52
00:08:56.320 --> 00:09:00.560
uh prompt response and we choose with the prompt so we start with the prompts.

s53
00:09:00.560 --> 00:09:03.519
These are various uh sources we have.

s54
00:09:03.519 --> 00:09:16.480
And then uh if you look at the paper, so if you look at this graph for any given data point, say if there are 10,000 samples that you want, the question is then how do you choose

s55
00:09:16.480 --> 00:09:24.320
uh the questions from all these different data set so that you have 10,000 uh for the data point.

s56
00:09:24.320 --> 00:09:28.640
So the the then there is the aspect around how do you mix these questions.

s57
00:09:28.640 --> 00:09:32.399
So uh you can use various methods.

s58
00:09:32.399 --> 00:09:40.080
So the paper talks about for example using LLMs to check for whether this is a good good question hardness of a question and so on.

s59
00:09:40.080 --> 00:09:44.720
And then you want to filter questions u and generate the answers.

s60
00:09:44.720 --> 00:09:47.200
Again this is all like driven by LLMs right?

s61
00:09:47.200 --> 00:09:55.600
So this is the curation recipe we did for uh creating this reasoning data set and the answer generation is using teacher models.

s62
00:09:55.600 --> 00:10:09.200
So you can take other reasoning data uh reasoning models such as deepseek or quenbased models or even gemini and whatnot and then you can also filter the answers once you have the answers for uh these questions.

s63
00:10:09.200 --> 00:10:16.000
Um and and then you can also you know given a question generate multiple answers or a single answer.

s64
00:10:16.000 --> 00:10:31.279
So these are various knobs in the curation recipe and the systematic way of doing this is like you run ablations and figure out which uh you know in each of these stages what works and you kind of proceed to the next.

s65
00:10:31.279 --> 00:10:34.720
So after doing all of this you get the final recipe right.

s66
00:10:34.720 --> 00:10:47.279
So this uh you can read this paper it has lots and lots of uh you know information about how we did the curation but here are some of the learnings that you know some of them are quite

s67
00:10:47.279 --> 00:10:54.959
uh counterintuitive and some of this was also covered in last year's uh AI uh AI engineer conference.

s68
00:10:54.959 --> 00:11:00.399
For example sampling um multiple answers per question works pretty well.

s69
00:11:00.399 --> 00:11:04.880
This is something that uh we it's it's kind of counterintuitive.

s70
00:11:04.880 --> 00:11:18.640
So as an example, something else we could have done is we could have had more much many more questions and then just answered them exactly once versus taking one question and answering them 16 times.

s71
00:11:18.640 --> 00:11:24.320
The I think the the reasoning is probably that it gives like a variety of how reasoning is done.

s72
00:11:24.320 --> 00:11:29.600
So the the during finetuning we also use the the reasoning traces, right?

s73
00:11:29.600 --> 00:11:32.000
So I think the diversity helps there.

s74
00:11:32.000 --> 00:11:42.160
And the other thing we saw is like the stronger teachers are not always the best uh uh stronger models are not always the better teachers.

s75
00:11:42.160 --> 00:11:54.560
And there were a few other counterintuitive aspects around like you know uh synthetic question generation or question answering working whereas answer filtering and other aspects not working very well.

s76
00:11:54.560 --> 00:12:05.519
And after the open thoughts work which was around uh data curation for reasoning models such as you know uh deepse kind of models, we moved on to open thoughts agents which is

s77
00:12:05.519 --> 00:12:15.360
um very similar but how do you curate these the the data and RL environments for uh training agents now right not models.

s78
00:12:15.360 --> 00:12:17.360
We we have a very similar figure here.

s79
00:12:17.360 --> 00:12:19.920
Again we want to establish scaling loss.

s80
00:12:19.920 --> 00:12:25.600
Um so as you increase the data set size we want to make sure that the curation recipe actually works.

s81
00:12:25.600 --> 00:12:38.160
Uh and again I'm I'm not going to go into details here but very similarly there are various ways of choosing different sources for example stack exchange and and whatnot.

s82
00:12:38.160 --> 00:12:39.279
How do you mix the test?

s83
00:12:39.279 --> 00:12:40.399
How do you filter?

s84
00:12:40.399 --> 00:12:44.880
Generating the rollouts uh choosing the teacher and so on.

s85
00:12:44.880 --> 00:13:01.120
And again these are some of the lessons learnings u as an example even here we saw that stronger models are not necessarily the uh best teachers right so we found out some some some of the I think uh

s86
00:13:01.120 --> 00:13:16.800
um quen models were better than for example um um um claude models I think and sampling multiple answers again helped in this case synthetic rewriting and task augmentation um

s87
00:13:16.800 --> 00:13:26.399
is something we thought will work but it didn't very work work very well and the other thing is like in in this whole process of building this open thoughts agent

s88
00:13:26.399 --> 00:13:42.800
SFT still contributed a lot to the gains um RL was kind of you know it's very comput inensive and for for the last few few percentages it really helped uh but but you know in many of the situations for example in enterprises

s89
00:13:42.800 --> 00:13:46.639
SFT actually works works pretty well, right?

s90
00:13:46.720 --> 00:13:54.399
And here is one concrete example I wanted to share on um uh actually deploying something to production, right?

s91
00:13:54.399 --> 00:13:55.200
By post training.

s92
00:13:55.200 --> 00:14:11.680
So we have seen a lot of people talk about post- training but in enterprise settings we haven't seen lot of successes at least haven't seen uh that here is a very concrete example of uh with intude there is a this app called credit karma which if you install there is a page

s93
00:14:11.680 --> 00:14:22.560
place where you can uh the the the app gives you a reasoning as to why a credit card has been recommended and this you can prompt a model to do this.

s94
00:14:22.560 --> 00:14:29.279
But one of the reasons one of the places where it fails is that the you know it it it's not always compliant.

s95
00:14:29.279 --> 00:14:37.519
So you have to have a long list of rules to make sure the responses are compliant and that actually blows up the latency.

s96
00:14:37.519 --> 00:14:57.680
So answer here is like you want to curate data and post train right seems kind of straightforward but one of the things that we ran into is um the data set can be quite impa imbalance and lot lots and lots of places for example you will have 0% APR and the model after fine-tuning

s97
00:14:57.680 --> 00:15:11.040
can kind of hallucinate the these uh numbers so this again kind of ties back to what Ross talked about some time back with respect to attacks And we kind uh we created this specific

s98
00:15:11.040 --> 00:15:28.639
uh curation recipe where instead of just having these uh question the the prompt response pairs in plain language, we added these uh tags which helped the model to focus on you know uh the the kind of form rather than the specific numbers itself

s99
00:15:28.639 --> 00:15:44.240
and that gave a big boost and uh we we saw that um the the overall the compliance metrics improved the latency improved, the throughput improved and eventually you know they they are able to own the model right as frontier models improve

s100
00:15:44.240 --> 00:15:55.360
they don't need to kind of go and u um uh update it and also as we see now the uh frontier [clears throat] models are also getting more and more expensive

s101
00:15:55.360 --> 00:16:04.480
and you know this kind of g gives them a very good way for owning the model and also um lowering the costs.

s102
00:16:04.720 --> 00:16:16.880
I think with that I want to briefly touch upon um uh you know curator the tooling that we had built last year um which is for curating reasoning data.

s103
00:16:16.880 --> 00:16:29.440
So uh what it does is you can basically uh you know um specify the you you can either go with say a hugging face data set where you have various prompts

s104
00:16:29.440 --> 00:16:36.639
or uh in many situations you may have collected logs and you want to get the responses and fine-tune a model.

s105
00:16:36.639 --> 00:16:52.560
So this curator kind of makes it pretty easy to do that and it comes with the integration with you know uh tinker and fireworks and this this is again the tool that we used um originally for curating open thoughts

s106
00:16:52.560 --> 00:17:04.319
and here is a very very detailed diagram of what we are building today but um this again connects back to um what Ross was talking about where he was talking about algorithms

s107
00:17:04.319 --> 00:17:22.640
uh environments and compute root right so it it feels like you know we are kind of converging on something very similar so if you think about uh the the stack that is needed to say not just curate these RL environments but to post train models

s108
00:17:22.640 --> 00:17:35.679
one of the things you need is obviously handle on like how do you build these RL environments how do you measure the quality how do you track the different versions and so on so that's one of the layers and below that you want various infrastructure

s109
00:17:35.679 --> 00:17:39.600
to uh um sand use sandboxes, right?

s110
00:17:39.600 --> 00:17:50.960
to spin up the rollouts to to spin up the sandboxes to generate rollouts and especially if you have long horizon rollouts then maybe at some points you need to do a checkpointing and then you need to be able to

s111
00:17:50.960 --> 00:18:05.679
snapshot or roll back to something else right so that's the other uh the lower level u you know compute and orchestration and at the top I have been giving examples on post training so there is all this uh layer around like how do you do SFT

s112
00:18:05.679 --> 00:18:12.880
how do you do RL and so on but there is also this method called Japa which is around uh which is on prompt optimization.

s113
00:18:12.880 --> 00:18:22.640
I don't know if you if you guys have heard of it but you can use LLMs itself to uh to to kind of optimize the prompts based on reflection.

s114
00:18:22.640 --> 00:18:28.240
Um so that also works pretty well for updating the system prompts and also the harnesses.

s115
00:18:28.240 --> 00:18:42.400
So this is kind of I feel like you know the the new architecture or the new reference uh stack for how how at least we are building and how many others are building um the the stack on how to build the RLMs

s116
00:18:42.400 --> 00:18:45.039
and then also post train agents.

s117
00:18:45.039 --> 00:18:52.080
I think with that uh I will uh end the talk and u you know happy to take questions offline.

s118
00:18:53.818 --> 00:18:55.818
[applause]