Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA

AI Engineer · 55 min · 562 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:12The workshop is about something more generic and more powerful.
2. 00:16It's about how do we build data pipelines um using the tool that we created Nemo uh data designer uh to be able to create or add any kind of functionality to a model or an agent.
3. 00:29So let's get started.
4. 00:31And by the way, I have a GitHub repo uh where I have some notebooks.
5. 00:34So if you're interested, you can scan the QR code or you can go to that GitHub repo.
6. 00:39Um and I'll have the same QR code on all the other slides as well.
7. 00:42So if you miss it, there'll be more chances to get it.
8. 00:48So what is the workshop about and what will we learn here?
9. 00:51Um the first thing we're going to learn is how do you actually take a target agent behavior that you want and turn it into a data problem right?
10. 01:00So basically can we create data for this targeted agentic behavior and then can we train a model to learn how to um uh do that task.
11. 01:11And so we built a tool called data designer to build uh synthetic data pipelines uh which does exactly that.
12. 01:19And the use case that we're going to target today is search.
13. 01:22Um, and by the way, this is a very reusable framework.
14. 01:25So if you want to create data for something apart from search, this is exactly how you would do it.
15. 01:31And there's two ways to participate in today's workshop.
16. 01:34So um, we have, you know, of course, you can follow live, try to run the notebooks, but since we just have like probably an hour or less than an hour,
17. 01:43um, you could also listen now and then run the notebooks later.
18. 01:47Um, so yeah, make sure you get the GitHub link.
19. 01:50And if you want the slides, I also have the slides on the on the uh GitHub repo.
20. 01:58So you're going to need four things if you're going to follow along either now or after the workshop.
21. 02:03Obviously, you need a GPU enabled machine.
22. 02:05So a cloud VM or a workstation.
23. 02:08Um, you also need a Tavi search API key.
24. 02:12Tavi is a company that does uh search for agents.
25. 02:16If you sign up on their website, they'll give you a thousand credits for free.
26. 02:20And you need data designer, which is our tool for creating data.
27. 02:23It's fully open source.
28. 02:25Anyone can download it, use it, do whatever they want with it.
29. 02:29And all the notebooks that I've provided um on the GitHub repo.
30. 02:37So here's the agenda for today.
31. 02:38So we'll first spend a few minutes talking about what is data designer and why do we actually need a framework like this and then we'll talk about uh the basics.
32. 02:49Um there's three notebooks I'm going to actually be showing you code.
33. 02:53U the first notebook is about the basics of data designer.
34. 02:57How do you actually create data that you want?
35. 02:59And then the second notebook is about model context protocol or MCP.
36. 03:04How do you actually uh create agentic data?
37. 03:06And then the third notebook which is what um is the super interesting use case of search.
38. 03:11How do you actually teach a model to do search and then we'll leave some time at the end for questions.
39. 03:19So I want to start off with a quote from Andre Karpati and he said internet data is not the data you want for your transformer.
40. 03:26So this is um a very uh valid point because internet data could be you know highquality sources like Wikipedia or it could be you know someone's opinion on Twitter or X or some other platform right so how do we actually know
41. 03:42um what data is useful for training AI models and what data is not useful so uh we need to process this internet data curate it and then uh create training data from it so that's the traditional approach approach.
42. 03:56What we're proposing is that we could actually synthetically generate data uh either from scratch or from a seed data set.
43. 04:06And so that's what data designer is for.
44. 04:08So data designer is a framework to create synthetic data.
45. 04:13And what do I mean by um seed data?
46. 04:16So seed data like you could use Wikipedia data as a seed and then try to create synthetic training data from Wikipedia.
47. 04:25So your synthetic data is not just made up by a model.
48. 04:29It is grounded in real world uh distribution and real world data.
49. 04:37So data design is a completely open-source framework.
50. 04:40We have a large team at NVIDIA that works on this.
51. 04:43Um and uh you can just install it by uh running pip install data designer.
52. 04:51And I think um what's really um what we're really proud of is data designer has been used to create six and a half trillion tokens of um training data and this is increasing every single day.
53. 05:06Um and yeah we're keeping track of the GitHub stars as well.
54. 05:11So if you like our repo give us a star.
55. 05:16So uh what is synthetic data generation right?
56. 05:18U well one way to do synthetic data generation is take a loop run it a million times and then if you want to create a product review data set just
57. 05:29you know continually prompt a LLM saying give me a product give me a review and then give me a customer profile that's one way to create a data set but you're going to run into a bunch of issues if you do that
58. 05:40so what's missing um well diversity diversity diversity that's the most important thing right if you continually prompt LLM after 10, 12, maybe 20 tries, it's going to start repeating stuff.
59. 05:53It's not going to give you what you need.
60. 05:55Um, you're also missing correlations between all the features that you're looking for in your data set.
61. 06:01Um, you don't have multimodality.
62. 06:04Um, you know, you're not really able to steer the creation of that data set and there's no validation, verification, uh, and so on.
63. 06:12And if I create a data set, I can't really share it with you because how do you actually take the uh process that I use to create my data set to create something similar.
64. 06:23So that's why we created data designer.
65. 06:26So how does it work?
66. 06:28Uh we have many different columns or features in data designer.
67. 06:34U these columns are essentially um a data designer abstraction.
68. 06:40In data designer, we generate data column by column.
69. 06:43So first you have your seed column which is completely optional.
70. 06:46So I gave the example of Wikipedia which could be a seed data set.
71. 06:50Um we could use any data set as a seed.
72. 06:53So anything which is a CSV, JSONL, parket file.
73. 06:57So you could use your own company data as seed.
74. 07:00And then you can try to uh add sampler columns to inject diversity.
75. 07:06So let's say you are trying to create a coding data set.
76. 07:09Well, a very simpler sampler column would be like easy, medium, and difficult coding questions.
77. 07:15So that's a sampler column.
78. 07:16You could add which is difficulty.
79. 07:18You could also add subsamplers.
80. 07:21So for easy, you could map it to a specific type of problem.
81. 07:25For medium, you could map it to a specific type of problem and so on.
82. 07:28So we'll get into all of this.
83. 07:29I know it feels a bit abstract, but we'll get into all of this once we look at the code.
84. 07:33Um and then we have columns for generating text, for generating structured data, for generating code.
85. 07:40Uh we also uh data designer is now multimodal.
86. 07:43So you can generate images as well.
87. 07:45Um and you can validate your generations too.
88. 07:51So the whole idea is we build a DAG or a graph which describes how your data set should be and then this DAG once you're ready to execute on it you can just hit one click and get a data set out of it.
89. 08:06So there's three different notebooks we'll walk through today.
90. 08:09Um I know some of you are thinking well in 2026 he's going to be showing us code right?
91. 08:14I mean now there's codeex and cloud code and all these amazing tools.
92. 08:19So I think if you understand the code, it'll be much easier to use codeex or cloud code to write your own uh synthetic data workflows for you.
93. 08:28Uh so we'll just take a brief look at each notebook now.
94. 08:33And by the way, this is our repo.
95. 08:35So this is our data designer repo.
96. 08:38It's fully open source.
97. 08:39Um and this is the GitHub repo that um I have created for this workshop where we have notebooks.
98. 08:48Um so you can try out notebook one 2 3 and then we also have um the slides here if you want to follow along or if you want access to the slides after the workshop.
99. 09:02So let's go to the first notebook.
100. 09:04So this is our intro to data designer notebook.
101. 09:06Can everyone see?
102. 09:07Okay.
103. 09:07or should I zoom in?
104. 09:11It's good.
105. 09:15Okay, so first things first, we set up our environment.
106. 09:21Um there's a bunch of uh prompts in here that we're importing.
107. 09:25And then what we do is we just, you know, import data designer.
108. 09:29That's it literally.
109. 09:31Uh and once you do that, you see in data designer there's a concept of model providers.
110. 09:37Um, so we're currently using uh VLLM, so that's local inference.
111. 09:42So I'm running this on a on a on a Nvidia GPU cluster and then we have access to different types of models, right?
112. 09:48So on my local cluster, I have access to GPTOSS 120 billion.
113. 09:52You can use any model or any open model to create synthetic data.
114. 09:57Um, some closed models prevent you because of licensing issues uh from creating data from it and using it for training models.
115. 10:05But open models are usually fair game.
116. 10:07So I'm using this model and um the goal here is can I actually take a data set of Wikipedia topics uh for this notebook.
117. 10:16So like you know different like uh what Wikipedia has so many different pages and um uh many different topics.
118. 10:24So we just made a list of those put it in a CSV and then we basically just call our config builder with seed data set.
119. 10:32And so essentially now we're um trying to create a data designer workflow but with Wikipedia uh topics.
120. 10:39Um so once you do that you can add question difficulty.
121. 10:43You can say easy, medium, hard, however you want your data set to be.
122. 10:46And then there's weights you can specify.
123. 10:48So you can have more medium questions than um you know easy or hard.
124. 10:53And then there's all kinds of sampler columns you can add about question difficulty.
125. 10:59So for example for easy you might do like direct lookup or simple fact or definition right but difficult might be you want something in more detail or obscure um and then once we get to the LLM generated columns
126. 11:13you just call this function add column and then you have your prompt which is essentially going to um look something like this.
127. 11:26Just trying to find it.
128. 11:35Yep.
129. 11:36So, you are an expert trivia question writer.
130. 11:39Um, so here's a topic difficulty question type.
131. 11:42So whenever you put these uh ginger templating for topic, what it's going to do is the topics you created in the previous um or in the seed that you had from your Wikipedia,
132. 11:53it's going to insert those at runtime into this prompt.
133. 11:57So essentially every prompt that the LM sees is now unique and diverse.
134. 12:02And so the model is going to give you diverse data when you actually hit generate.
135. 12:08Um and then um you you can also have like a structured column.
136. 12:13So a structured column is when instead of trying to generate like text data um you can try to generate like a JSON or markdown format or any custom format that you're interested in.
137. 12:24Um and uh we know that LLMs you know we have to be nice to them.
138. 12:28We have to talk to them but sometimes they may not give you that structured format that you want.
139. 12:32So then we just end up dropping those records.
140. 12:39So the way um data creation works is we have something called preview mode in data designer.
141. 12:48So essentially when you hit preview what's going to happen is it's going to try to create three records for you and then you can go and look at those records.
142. 12:55So let's look at some records.
143. 12:56So all of this will make a lot more sense.
144. 12:59So the topic here is the worldwide web.
145. 13:01Um, and the question or I guess some consortium of the worldwide web and the question is which W3C recommendation is identified by the same three-letter abbreviation that serves as a airport code for some city in Norway.
146. 13:16So there's no way I would know the answer to this question, right?
147. 13:19That's why these questions are essentially hard because you want to invoke the model to do search, right?
148. 13:24But in this first notebook, we're just trying to create question answer pairs.
149. 13:28So uh which is a very simple data set.
150. 13:30So I guess this is the answer the model came up with.
151. 13:32And then you can also add LLM as a judge to score how your model is doing.
152. 13:37So this is the question the model came up with.
153. 13:39This is the answer it came up with.
154. 13:41And then um this is the question quality.
155. 13:44So we can have another model judge what uh it thinks of the question that the uh previous model created um and so on.
156. 13:53So in the end you end up with a data set like this.
157. 13:55And by the way, we have a a function that you can just push your data sets directly to hugging face.
158. 14:01Um so it's super easy to use and um you can also visualize all of your questions here as you can see.
159. 14:10And then you can get some stats about your data set.
160. 14:12Obviously we generated only three records here.
161. 14:15So uh you know these stats are not very meaningful but if you generate like hundreds or thousands of records you could do that.
162. 14:21And then we have something called batch mode.
163. 14:23So which is data designer.create create and here instead of a 100 I could just you know put how many other records I need and then I can go and create all these records
164. 14:32and as I mentioned previously we generate data column by column so in the batch mode first you generate the questions and then you generate the answers right so this is obviously a very simple example
165. 14:46and then you have the LLM as a judge um and here's your final data set that's what it looks like so That was the first notebook.
166. 14:56I think it's quite straightforward once you look at it and try it out.
167. 15:00Um, we have a second notebook which focuses more on MCP and then we're going to build to the third notebook which I think everyone will find interesting because that's where we'll actually talk about search.
168. 15:12Sorry.
169. 15:15Yeah.
170. 15:18Oh yeah.
171. 15:19The all the GitHub links are the same.
172. 15:21I'm just putting them in every single slide.
173. 15:23So, if you missed the first one, you can you can get it.
174. 15:26But I'll pause here for a minute if you want to scan.
175. 15:37So, um let's talk about MCP.
176. 15:40So, MCP is a standard which allows um an agent to essentially connect to tools, right?
177. 15:48So the tool here is tavly which is nothing but a search tool for agents.
178. 15:53There are many other search tools as well.
179. 15:55Um so that's basically the um uh what what we need the mo what we're trying to teach the model here is how do you do search right so uh tavly
180. 16:06is a good tool for the model to use to learn how to do that search.
181. 16:11Um so going back to the second notebook.
182. 16:18So here what we'll do is we'll just build upon some simple concepts from the first notebook and we'll add um how to create agentic data.
183. 16:27So again you have the imports you have the the model configs and then the same seed data set the same sampler column.
184. 16:36So question difficulty, question type and so on.
185. 16:40And then you actually create the question and then at the end you get like a config, right?
186. 16:44I don't know why it's printing printed this in like uh light color but yeah this is essentially your graph um that you're creating and then you have the answer.
187. 16:54So it's the exact same thing except now when you actually create the answer you add this tool alias here.
188. 17:03And so the goal is instead of just trying to answer the question, the model is now going to use a tavly search tool to try to find the answer on the internet.
189. 17:15Right?
190. 17:16So that's we're build we're building upon the first notebook where we were just like okay please generate a question and then please generate an answer.
191. 17:23Now it's please generate a question and then hey you have this tool available.
192. 17:27Can you use this tool to go to try to find the answer rather than trying to generate the answer?
193. 17:33So that's that's the difference between the first and the second notebook.
194. 17:38And then again you have the preview mode where you can generate a few records and then you look at your records.
195. 17:45And by the way data creation is very iterative.
196. 17:48It's not like one and done.
197. 17:50So when you create some data you look at it.
198. 17:52If you're not happy with it, then probably you go change your workflow, change your prompts and then you know iterate on it and then once you're happy with it, you can send it off to batch execution.
199. 18:03So let's look at what um the second notebook produced.
200. 18:08So let's say the question is uh to which German city did Teleifan move its headquarters in 1949.
201. 18:16Does anyone know the answer?
202. 18:19No. Right?
203. 18:19I mean I don't know the answer.
204. 18:21So um what we want to do is create questions that are difficult so it invokes reasoning and tool calling from the model and so when the model is actually trying to respond to this
205. 18:35um it's as you can see here it's using tavly so the first search it made is tell if you can moved its headquarters in 1949 to which company so essentially it's using one search tool called here and then
206. 18:48um it's going to get an answer so this is the query from the model and then here are the results.
207. 18:53These are all the URLs and then for each URL it's going to give you a title and then it's also giving you the content.
208. 18:59So that's what Tavali gives us right if you make a query or that's what it gives the agent when the agent makes the query and then you end up with the response and then finally
209. 19:09at the end you can create your data set with batch mode just like we did last time.
210. 19:16And if you're thinking he just blazed through this whole first two notebooks, that's because the crux of it is in the third notebook, right?
211. 19:23So, um that's what I really want us to focus on today.
212. 19:27So, let's talk about search.
213. 19:29Um cuz we just went through a couple of notebooks showing us how to use data designer and that's fine, but what's the problem we're actually trying to solve?
214. 19:38So, um like I mentioned before, I'm a research scientist at NVIDIA and we're also part of the Neotron team.
215. 19:44So, Neimotron is a collection of found open-source foundation models um that we've created.
216. 19:50Uh we just released our latest model Ultra, but we also have two other models.
217. 19:54So, we have three different sizes of models.
218. 19:56And so, as part of that work, we actually had to teach Neotron how to do search.
219. 20:02And that's very important, right?
220. 20:04Because maybe this feels like a long time ago now, but maybe two or three years ago, I'm sure you all remember looking at like I don't know chat GPT Claude or Gemini and then looking at the
221. 20:15web UI and it says the knowledge cutoff date is December 2023.
222. 20:20So the reason it said that is because well you have to start training the model at some point.
223. 20:25So you have to stop collecting and creating data at some point, right?
224. 20:29So let's say they're training the model in January 24.
225. 20:32by December 23, they need to like um essentially freeze the data.
226. 20:37So, anything that the model knows after um December 23 is not part of its training data.
227. 20:45So, if you had to ask a question like what is the weather today, there's no way the model can answer that question from its training data, right?
228. 20:53Uh question.
229. 20:58Yeah.
230. 21:00So yeah, definitely scan the QR code.
231. 21:02So the idea is how do you actually um teach the model about current events because learning about current events seems as important if not more important than you know knowing about
232. 21:15uh things from the past, right?
233. 21:17So um that's why search exists and that's why models need to learn how to do search.
234. 21:24Um so coming back to the slide um so teaching neotron models how to use search tools that was our motivation and um um we essentially the model needs access to real-time information to be able to do useful things.
235. 21:40So how did we actually do this?
236. 21:42So before I get to how did we do this?
237. 21:44We need something to measure progress on.
238. 21:46Right?
239. 21:47So um in the foundation model community when you need to measure progress on something you use a benchmark which is like a standardized test set without the answers right like we don't have the answers so one of the benchmarks for search is browse comp
240. 22:02uh which is a really difficult benchmark and uh the FIFA World Cup is going on right now so I don't know how many soccer fans there are here but I'm I'm a big soccer fan but we don't call it soccer
241. 22:13but um between 1990 and 1994 four.
242. 22:16Which teams played in a soccer match with a Brazilian referee?
243. 22:20Had four yellow cards, two for each team, three in total, blah blah blah.
244. 22:25I won't read the rest of the question, but it's it's quite complicated question, right?
245. 22:29There's no way that any of us know the answer to that or maybe there's like, you know, someone who's um really wellversed in this who has the answer at the top of their head.
246. 22:39What we would do back in the day like you know three four years ago is we'd actually try to break a problem down if we had a complicated uh query like this and then we would reason about what queries to actually make.
247. 22:51We would go and search on Google or elsewhere uh get access to more information and then reason about what was our original goal, what information did we get access to and then proceed from there and then finally arrive at the answer.
248. 23:06Right?
249. 23:06So the answer is Ireland versus Romania.
250. 23:09I I didn't know that.
251. 23:11Um so how did we actually create a data set that allows us to train a model to go from 0% to 30% on this browsecom benchmark?
252. 23:22Um so that's what we're going to learn about today.
253. 23:25So to do that we took we did something um uh we we used a seed data set which I described earlier except our seed data set is not just Wikipedia
254. 23:36it's something that um is essentially a graph of Wikipedia topics so it's called wiki data and what we did is we did a few hops along this graph and that is our seed data set and I know the diagram on the right looks like a tree it's not a tree it's a graph
255. 23:55So um if it looks like a tree that's my fault.
256. 23:58Uh it should it should it should be a graph.
257. 24:01Um and essentially um you can see um on the right the first topic is Nvidia and then Nvidia is connected to Jensen who is our CEO and then that is connected to Oregon State University because Wiki Wiki data says that that's where he went to college
258. 24:19and then that college is located in Benton County and Benton County is loca is actually named after a person called Thomas Hart Benton.
259. 24:30So if you took um if you randomly sampled an entity in wiki data and then did five to eight hops along it and I'll explain why five to eight you'll end up
260. 24:43um with a hop like this like the green you know set of entities with like arrows in between.
261. 24:50So that is essentially our seed data set to create uh search riddles.
262. 24:55So, one search riddle that I could create from this, and I'm going to have to read this from my speaker notes, is um which company was founded by the person who went to college where the county was named after Thomas Hart Benton.
263. 25:13So, there's no way the LLM knows the answer to that in the training data, right?
264. 25:17Because if we give an agent the chance to cheat, it's going to cheat.
265. 25:22It's going to just give us the answer.
266. 25:24it's not going to use the tool calls or the tools sorry like search that we want it to use right so we want to make the questions as difficult and as I guess
267. 25:34um riddle like as possible so we can invoke the search behavior from the model so this is our seed so I I spoke about the first step which is create random walks with seed and why four to eight or five to eight hops sometimes it just you know if you do these hops it just goes completely off
268. 25:52topic like It will say um I don't know like u a person's name and then it will say well human being well sure that is true but you know it's not really related like basically your hops will go off topic so that's why we want to as a heristic stay between four and eight that's what worked well
269. 26:11for me and then you transform these seeds into search riddles and then um you execute the search via the search tool which is tavly so that's how you create a data set with a prompt and a with a search prompt and a search response, right?
270. 26:27And then we evaluate our data and then something that's not mentioned here is, you know, we train our models, right?
271. 26:33So once you have the data set, you train the model with it and then you measure performance on a standardized benchmark.
272. 26:40So let's look at the notebook and see how we can do this.
273. 26:45Any questions so far?
274. 26:49Yes.
275. 26:57So the reason uh it was important to do I guess it's not difficult questions it's questions that the model uh cannot find on the internet readily uh cannot find answers readily on the internet too.
276. 27:12So like if you took that full blob of text that we had in the browsecom benchmark and you put that on the web you're not going to get the answer.
277. 27:19I mean you will get the answer because of the AI search thing with Gemini now but that's what we're trying to teach the model essentially.
278. 27:26Yeah.
279. 27:30Okay.
280. 27:31So um so this is the tutorial for the end to end aentic search.
281. 27:40So we already spoke about the way we do this and how we create the data.
282. 27:45We already spoke about how we import all these um prompts and I wrote a function to like use Tavi with MCP.
283. 27:53You can go and look at the code later.
284. 27:55Um and here is where we have our seed entities.
285. 28:03So this is the example I showed you.
286. 28:05This is just one seed entity.
287. 28:07So the way uh my seed collection algorithm works is like you can run it a million times and you get million different hops along wiki data right so those are just topics we want to ground our search riddle creation on
288. 28:21so another topic is um we like modern art and like I'm not really an art buff but so I don't understand most of these words but essentially this is like another seed hop that you can
289. 28:36So the way to create this data set is um well you are an expert search agent that uses web search to answer questions accurately provide valid JSON with the final answer supporting URLs and a short justification
290. 28:52and here are the tools you have available.
291. 28:53You have one tool which is tavly search and then here's some usage guidelines for the tool.
292. 28:59So, like we're like, "Okay, please please don't use more than like, you know, don't don't like call the tool like a hundred times and spend, you know, $1,000 or something."
293. 29:09I guess those are like all kinds of instructions you can put in there, but essentially um uh jokes aside, what you want here is um your agent shouldn't go crazy and try to like you know spend way too much money answering a question that's very easy
294. 29:26or you know something like that.
295. 29:29So that's why we have all these instructions and we also have an example.
296. 29:33So this is um um something like in context learning, right?
297. 29:38So you put an example into the prompt and then the model can essentially um learn from that example how to actually call the search tool.
298. 29:48So here's your prompt that you want to create.
299. 29:51So here what we do is um we collected the wiki data hops.
300. 29:58We can insert those hops into the prompt here uh which is our um you know seed entity.
301. 30:05And then uh we ask the model to essentially come up with a question um using any or all of the seed entities from this hop.
302. 30:16And then we ask the model to create a response using uh this agent solution draw which I showed earlier.
303. 30:28So in the in the second notebook.
304. 30:32So once you do this, you'll end up with a preview data set.
305. 30:36And so here's our preview data set.
306. 30:38So Indian Council of Medical Research is the start entity for the seed.
307. 30:44And then apparently that's located in a city called Bhuaneshwar which is a capital of some state in India.
308. 30:51Uh so that's our seed path.
309. 30:54And then here is the question that the model came up with.
310. 31:00And then this Asian solution draw is not displayed in here.
311. 31:04But if you look at it, we can we can see what it looks like here.
312. 31:07So it did a bunch of tool calls.
313. 31:08It reasoned about the question and then it's trying to find the answer on the internet.
314. 31:15So that's how you create um a search tool called data set.
315. 31:21And I guess I'll pause here for questions and then we can move on to the next few slides after that.
316. 31:30Any questions?
317. 31:31If something was Yes.
318. 31:51Yeah,
319. 31:53absolutely.
320. 31:56Yes.
321. 31:56So we moved the bird benchmark from 26 to 41%.
322. 32:00So you asked the you know exact right question and we have a blog post on this as well.
323. 32:05Um you can read about it.
324. 32:08Um so all the data sets we've created with data designer um we have blog posts and you can see right here so I was involved in this work as well.
325. 32:20Uh so yeah this talks about SQL and I can also share the link with you after um but essentially data designer can be used to create any kind of data set
326. 32:30for training AI models possibly other data sets as well if you're looking for test data sets not related to model training and so on.
327. 32:37Um and we have a bunch of data tools that we've created in our team apart from data designer that I'll talk about in the end.
328. 32:55So the soda is 70 on bird is because it's agentic.
329. 32:59Our um current SQL data set is not agentic.
330. 33:04And um also a sort on benchmarks is very much model dependent.
331. 33:10We're talking about moving uh Neotron Nano which is our small size um LLM or SLM.
332. 33:17Okay.
333. 33:18I don't I don't know if you can call it an SLM because it's an MOE but it's an LLM.
334. 33:21And so um for that model size it's a pretty respectable performance on board.
335. 33:27Yeah.
336. 33:28question.
337. 33:47Yeah.
338. 33:47No, this is a deterministic algorithm to create the seed data and then the uh tool calls and the agentic stuff is in creating the actual like performing the actual search once you have the question
339. 34:00and creating the trajectory.
340. 34:02We call it a trajectory.
341. 34:04So creating the search trajectory this is deterministic.
342. 34:07This is just something I came up with to create seed data.
343. 34:11Yeah, you can have any kind of seed data.
344. 34:13Maybe like you know you can come up with a better way of creating seed data than this as well.
345. 34:26Oh great question.
346. 34:27So we could do that.
347. 34:29Um we have created data sets that can be used in pre-training and post- trainining.
348. 34:34My main focus is post- training um or you know creating data sets for post- training.
349. 34:40So this specific uh um I guess project was SFT uh in posting.
350. 34:48Yes,
351. 34:52there's no reason apart from the fact that uh I think someone at Nvidia did a deal with Tavly to get credits.
352. 34:58You can use Exa.
353. 34:59Exa is another search agent uh provider.
354. 35:01You can write your own search tool.
355. 35:03Um I guess so.
356. 35:05Yeah, there's that as well.
357. 35:08Yeah.
358. 35:11essentially.
359. 35:17So the golden responses are generated by um tail designer using the model that you choose and so um in the first notebook um yeah and sorry I think I think um you know I went a bit quick because it's we just have an hour for this workshop
360. 35:35uh previous version of this workshop was 2 hours.
361. 35:38So the model that I used is GPDOSS120B to create the data.
362. 35:43Um but data designer is not just a collection of LLMs that you can use to create data.
363. 35:48It has a bunch of LLM and nonLLM models as well working together to help you create diverse data.
364. 35:58Say that again.
365. 36:01This is SC.
366. 36:01Yeah.
367. 36:02Uh what do you mean?
368. 36:07Yes.
369. 36:10Yeah.
370. 36:20Yes.
371. 36:23Yes, you are distilling.
372. 36:24But in some cases, you can even go beyond distillation.
373. 36:27And that's the case that I'm trying to make because we have built this uh tool that allows you to create diverse data.
374. 36:33So for example, a completely different use case that I was working on.
375. 36:37GPTOSS gets some percentage on this uh benchmark called strructal which is a way to measure how models do on JSON, markdown, XML, YAML and CSV formats.
376. 36:50And Neotron actually distilled from GPTOSS but went beyond GPOSS's performance.
377. 36:56So it's distillation but it can also be more powerful than that because of all the things we've built into data designer.
378. 37:03Yes.
379. 37:04How can I use data designer?
380. 37:07You can use data designer for RL.
381. 37:10So let's say uh and by the way we're doing this for search right now or or the team at NVIDIA is doing this for search.
382. 37:17So instead of trying to create the golden solution like how he described it or or you know um executing the search riddles as written on my slide, you just don't do that step and then you essentially end up with a data set full of prompts and then we have a tool called Nemo Gym
383. 37:33which is essentially um an RL um reward function.
384. 37:39You know I guess it's an it's a it's a tool to basically teach your models how to do RL.
385. 37:44But the thing is in Nemo gym you write like a a function that essentially gives you a reward and then you can use our uh any RL training framework to
386. 37:56execute in real time with those tools and then provide a reward in RL.
387. 38:10It's a great point.
388. 38:11We're thinking about how to do this actually currently.
389. 38:13But to build something generic is the is the challenge, right?
390. 38:16Like we don't want to build something specific to one use case.
391. 38:20Yeah.
392. 38:22Yeah.
393. 38:30Is this reliable and accurate for ETL?
394. 38:42So I haven't tried it myself but I think it is because it it should be because um we've created a diverse set of data sets with data designer and I think at the start we showed about 6 and a half trillion tokens
395. 38:56and if it's not reliable you should come talk to us and see how we can help you.
396. 39:05I can't hear you.
397. 39:06Sorry.
398. 39:11Sorry, I'm not able to hear.
399. 39:17Oh, I I don't know about that.
400. 39:20I I'll come back to you at the end and we can we can chat more.
401. 39:23Yeah, sorry about that.
402. 39:25Okay.
403. 39:25Um, we have about 15 more minutes.
404. 39:28So let me move on and we can have more questions at the end to something really interesting which is evaluating your synthetic data right and that's not an easy thing to do
405. 39:40because without this evaluation step the synthetic data is not that useful.
406. 39:46So when I started just to give you a sense of how SDG as we call it works synthetic data generation I started with 50,000 seeds and in the end I ended [clears throat] up only with 7,000
407. 39:58high quality endtoend SFT records and then I used that to train the model and we moved the benchmark from 0 to 30 right so that's great but 50,000 to 7,000 that's a lot of
408. 40:12you know um um funneling and filter ing for high quality.
409. 40:17So what like I'm trying to focus on here is that data quality is the most important thing when you're trying to create data and um what do I mean by quality?
410. 40:27It can be task specific but it can also be generic like diversity is one way to um you know one axis of quality I would say.
411. 40:35Um so after that I created this function to help us um show like this is what the LLM as a judge scores for each trajectory um and what is like your
412. 40:51um you know number of words in your uh question or search query and then here is your keep percentage 66% and so on.
413. 41:00So what's really so you can take a look at that function later but what's really interesting is how do you actually check if the answers that the model got are correct or not.
414. 41:10So I came across this really interesting example and the search riddle was quite simple.
415. 41:16It said which comp which country owns US steel and I thought the answer is the US.
416. 41:23And so um I you know I I guess the agent predicted US in the training data set and um my verifier said that it was correct but then I went and googled it and apparently the US sold US steel to Japan I think if I remember correctly a couple of years ago.
417. 41:43So even my wiki data seed can become stale, right?
418. 41:48And so that's the real challenge is how do you actually evaluate and um uh tell if the data that you've created is um good, accurate, you know, relevant and so on.
419. 42:04Um so I guess if you go and implement a determinist I guess like a simple matching function like how I did you'll end up with like 24% and then you'll be like well my data is terrible
420. 42:17why did I do all of this but that's not entirely true because here low percentage I expected because I don't know how to actually create that function.
421. 42:25So this is an active area of research is how do you actually evaluate this data?
422. 42:28So I think evaluation from my perspective is a very multifaceted problem.
423. 42:33So you can't just take LLM as a judge and put it in there and hope that that's going to you know give you everything.
424. 42:42Evaluation is a problem where if you check for a 100 problems and if you see that those 100 problems do all you all you can tell from checking for a 100
425. 42:51problems is those 100 issues don't exist in your data set.
426. 42:54there could be a thousand other issues.
427. 42:56So that's the challenge is how do you actually evaluate the data?
428. 42:59Um so we have a few different ways that we do that.
429. 43:02Um one of them is like I mentioned earlier LLM as a judge which is generic.
430. 43:08Uh something I did really interesting is I actually did LLM as a judge with uh and I took the judgments and I sent them to human reviewers and I have those reviewers
431. 43:17um cross-check the judgments of the LLM and essentially give me confidence that the LLM is judging my responses correctly.
432. 43:27So that's one way to do it.
433. 43:29And then in search you could look at real-time information.
434. 43:32So you have to basically find a union of all these different different uh layers of evaluation that's going to essentially give you enough confidence in your data set.
435. 43:43So that's it for the code.
436. 43:44I know I blazed through this but hopefully I've you know um given access to all of this in my GitHub repo so you all can take a look.
437. 43:53Um so let's talk about a few other use cases.
438. 43:57So like I mentioned earlier, we are the um you know part of a much larger team that creates Neotron, right?
439. 44:02So um my focus or my team's focus is essentially creating data sets or creating a framework for creating data sets and or I guess both of those.
440. 44:12So there's all kinds of data sets we've created.
441. 44:15The first one that we spoke about today, search tool calling.
442. 44:18Um and then we also have a data set to do multi-turn tool calls.
443. 44:23So, I think the search one was very much single turn, right?
444. 44:27Like we're going from prompt to response.
445. 44:29But what if you had multiple turns or what if you had multiple tools um or you know not just search tools but like you might have a I don't know a weather tool or a coding tool and so on.
446. 44:42Um so we're working on multiple different projects here.
447. 44:47Another really interesting one is structured output generation.
448. 44:49I briefly spoke about it.
449. 44:50There's a benchmark called strructal which measures your performance on toml, json, yaml, csv, xml and all of these are super important because everyone wants to use um you know agents
450. 45:03to do everything.
451. 45:05Now, I'm sure you all have seen the the uh the blowtorrch meme uh on LinkedIn or something, but but uh if we're going to be using agents as um the
452. 45:17central piece in software applications, then they need to reliably give us um like a JSON format or a YAML format and so on.
453. 45:25So, that's what we were focused on there.
454. 45:27Um we've also worked on SQL um that someone asked about here and uh we've also worked on scientific reasoning.
455. 45:36Um there's also pre-training data sets that we've created using data designer for stem performance and so on.
456. 45:43Um and so I can talk about the best practices for generating data.
457. 45:49Um I won't go over this in detail.
458. 45:51There's because you can always look at the slides later but essentially diversity before volume.
459. 45:56I think the Kim K2 paper showed something similar which is 10 strategies one epoch is better than one epoch.
460. 46:03Sorry 10 strategies one epoch is better than 10 epochs one strategy.
461. 46:07And um you always want to use the cheapest correct column type.
462. 46:12So like using a LLM to create a sampler makes no sense, right?
463. 46:15It's a complete waste of resources.
464. 46:18But using LLM to generate text data makes a lot of sense.
465. 46:21So our samplers are written in Python of course.
466. 46:24Um and then you generate you overproduce data and then you filter down like I mentioned earlier and then you verify what you can uh iterate on the preview.
467. 46:33I think the best way to know if your data is any good is to actually look at the data.
468. 46:37That's the best way.
469. 46:38Uh if you can look review a few samples and um of course what is this idea of data designer?
470. 46:45Essentially you're taking the data creation process and making it um something you know version controllable uh sharable like code.
471. 46:53So essentially we're creating code that can create data.
472. 46:57So now you can share this code with anyone because you know we already built all this tooling for sharing code and having version control and all of these other things.
473. 47:06So if you're trying to do search but not exactly the way I did it and just slightly differently you take my code you modify it and you can create your own data set from it.
474. 47:15Um, so something that's super interesting.
475. 47:19I'm getting into the extra slides which I didn't think I would.
476. 47:22I think I have 10 more minutes.
477. 47:24Right.
478. 47:25So, um, this is something that's really interesting.
479. 47:29We've created uh something called personas.
480. 47:32So, Neotron Personas USA um it's also inspired from this other paper which is scaling synthetic data creation with 1 billion personas.
481. 47:40So, what is a persona?
482. 47:41Right.
483. 47:42A persona is essentially um so we're trying to solve the problem here of how do you know if the data you've created actually um adheres to like demographic information or you know grounded in some real world distributions
484. 47:59right that's very important u so if I were to create a data set of healthcare records in San Francisco California well I want to make sure that the uh the demograph the actual demographic of San Francisco is reflected in my data set,
485. 48:15right?
486. 48:15So like the number of uh you know there's all kinds of demographic features.
487. 48:20Um so we created a data set called Neimatron personas uh using data designer but is also now a seed in data designer to create more data sets and um
488. 48:31this essentially allows you to learn from user data without actually leaking any single person's information out there.
489. 48:42So now you can train AI models and you can learn from the aggregate of um you know people's information but you don't actually have to have like for example my healthcare records
490. 48:53you know directly um uh being used in training an AI model.
491. 48:58I I keep using the healthcare example because I want to find a sensitive industry which is like you know there's HIPPA compliance there's all kinds of regulations there right that we have to respect.
492. 49:08So, um, so this is a persona.
493. 49:11Um, this person is a 50-year-old pharmacist with a free spirit, balances her relaxed approach, blah, blah, blah.
494. 49:17But essentially, there's a persona, there's a professional persona, um, first name, last name, middle name, gender, age, um, all kinds of things.
495. 49:28And we have a million of these.
496. 49:31You can take this, you can use it, um, however you please as seed to create more data.
497. 49:39So yeah, I think with the personas um I will probably stop here.
498. 49:44So we have two more minutes.
499. 49:45I'll talk about these other tools and then we'll leave time for questions.
500. 49:48Oh, one question.
501. 49:49Yeah.
502. 49:50Yeah.
503. 49:54For
504. 49:58Yeah.
505. 49:59Yeah.
506. 49:59Yeah.
507. 50:00Yeah.
508. 50:00So essent that's essentially why we created it is to um have a way to learn from aggregate information without actually like you know having individual people.
509. 50:11So like you could create synthetic social security numbers.
510. 50:15Now, they're not actually going to be like your social security number, right?
511. 50:18But I think the idea is like if there's um for example um um I don't know what is a good example that I
512. 50:36Yeah.
513. 50:36Yeah.
514. 50:38Yeah.
515. 50:38Yeah.
516. 50:39Exactly.
517. 50:39So for example, we we actually created some tax data for u you know a customer and this tax data I realized there's exactly uh seven people in the US who are below 18 earning more than $10 million a year on their tax return but there's like way more people who are older and they're earning you know x
518. 51:01or y or z amount of money and so the tax data that we create let's say we want to create a synthetic W2 form we want to make sure that that form is not just random values from the LLM because the LLM is not going to give you
519. 51:15um if you have you asked LLM to tell a joke it's going to tell you the same joke every time.
520. 51:20So similarly, it's not going to give you like numbers that are realistic, right?
521. 51:24Like it's always going to come back to the average of things.
522. 51:27And so when you create data, you want to make sure that that synthetic W2 form or 1040 form or whatever is actually um adhering to real world distribution.
523. 51:37So that's why we created this persona data set.
524. 51:41So we have a bunch of tools uh just two minutes and then more questions.
525. 51:45Sorry about that.
526. 51:46Um so we have a bunch of tools that we've created.
527. 51:50Data designer is only the tip of the iceberg I guess.
528. 51:53Uh but it's our most successful tool so far because it's our oldest tool.
529. 51:57Uh we also have uh you know at NVIDIA we have data curation tools.
530. 52:04So which is Nemo curator which I'm sure there are other talks about this.
531. 52:08I I believe I saw on the schedule.
532. 52:10Um it is very interesting.
533. 52:11you know you have to take data uh real world data curate it modify it post-process it get it to a state where it's usable to train in LLMs or you can create synthetic data so I think it depends on your use case like I'm not going to stand here and say synthetic data is the best thing ever
534. 52:27for everything but I think there are specific use cases where you want to curate real world data or where you want to you know generate synthetic data um and so we have data uh curation tools we have data designer which is our synthetic data generation tool We have a new tool called anonymizer.
535. 52:43And by the way, all these tools are completely open source.
536. 52:46So everything that our org does at NVIDIA is fully open source, including Neotron, including all these tools.
537. 52:53Um, so you can use it however you want.
538. 52:56Um, so this data tool is called Nemo Anonymizer.
539. 53:00And the goal here is can we actually anonymize data?
540. 53:03So, I don't know about you guys, but I I don't know if I should be saying this when the talk is recorded, but I've been putting all kinds of things into my agents like you know, I accidentally put my API keys in there once or like you know some other sensitive information, right?
541. 53:17All this log data for like all your agents that you're using.
542. 53:21You need some way to anonymize it and before it's usable, right?
543. 53:24So, that's what anonymizer does.
544. 53:26So, that's just an example, but there's obviously many other ways to use anonymizer.
545. 53:30And then we have Nemo safe synthesizer which is allowing us to generate safe synthetic versions of sensitive data sets um as well.
546. 53:38So yeah come talk to me if you have any questions.
547. 53:42Um here's my LinkedIn if you want to stay in touch.
548. 53:45Uh again you can use data designer.
549. 53:48I would love for you guys to try out the notebooks.
550. 53:51I'm so sorry that we had to we had only one hour so I had to just like blaze through all the code but you can try out the notebooks.
551. 53:57We also have the blogs if you're interested in text to SQL or some some other use case.
552. 54:02So um this is this is showing us how to use the personas.
553. 54:07Um I think there's a blog somewhere on how we created the personas data set for the US.
554. 54:13We took the census information and we used a PGM, a probabilistic graphical model to essentially um create um like a connection between all basically take that demographic information, process it and then try to
555. 54:28uh uh create a data set from it uh essentially.
556. 54:32So that's what we did.
557. 54:34Um and so every country pro uh publishes this uh demographic information.
558. 54:40So, it's not really like um I'm looking at other slides now.
559. 54:44We we've done a bunch of these workshops recently.
560. 54:47Um but yeah, there's all kinds of personas if you're not if you're trying to build for something outside the US, if you're building something for Japan, India, Singapore, Brazil, I'm sure we have more now.
561. 54:59So, we we have all kinds of personas data sets.
562. 55:02So, um it's giving us um um a way to uh create customized AI for whichever country you're interested in.