WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=qFaKvbVc5cE NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/qFaKvbVc5cE.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.880 --> 00:00:16.640 The workshop is about something more generic and more powerful. s2 00:00:16.640 --> 00:00:29.840 It's about how do we build data pipelines um using the tool that we created Nemo uh data designer uh to be able to create or add any kind of functionality to a model or an agent. s3 00:00:29.840 --> 00:00:31.039 So let's get started. s4 00:00:31.039 --> 00:00:34.719 And by the way, I have a GitHub repo uh where I have some notebooks. s5 00:00:34.719 --> 00:00:39.200 So if you're interested, you can scan the QR code or you can go to that GitHub repo. s6 00:00:39.200 --> 00:00:42.559 Um and I'll have the same QR code on all the other slides as well. s7 00:00:42.559 --> 00:00:46.800 So if you miss it, there'll be more chances to get it. s8 00:00:48.000 --> 00:00:51.039 So what is the workshop about and what will we learn here? s9 00:00:51.039 --> 00:01:00.399 Um the first thing we're going to learn is how do you actually take a target agent behavior that you want and turn it into a data problem right? s10 00:01:00.399 --> 00:01:11.439 So basically can we create data for this targeted agentic behavior and then can we train a model to learn how to um uh do that task. s11 00:01:11.439 --> 00:01:19.040 And so we built a tool called data designer to build uh synthetic data pipelines uh which does exactly that. s12 00:01:19.040 --> 00:01:22.799 And the use case that we're going to target today is search. s13 00:01:22.799 --> 00:01:25.840 Um, and by the way, this is a very reusable framework. s14 00:01:25.840 --> 00:01:31.759 So if you want to create data for something apart from search, this is exactly how you would do it. s15 00:01:31.759 --> 00:01:34.560 And there's two ways to participate in today's workshop. s16 00:01:34.560 --> 00:01:43.520 So um, we have, you know, of course, you can follow live, try to run the notebooks, but since we just have like probably an hour or less than an hour, s17 00:01:43.520 --> 00:01:47.280 um, you could also listen now and then run the notebooks later. s18 00:01:47.280 --> 00:01:50.000 Um, so yeah, make sure you get the GitHub link. s19 00:01:50.000 --> 00:01:56.560 And if you want the slides, I also have the slides on the on the uh GitHub repo. s20 00:01:58.399 --> 00:02:03.280 So you're going to need four things if you're going to follow along either now or after the workshop. s21 00:02:03.280 --> 00:02:05.680 Obviously, you need a GPU enabled machine. s22 00:02:05.680 --> 00:02:08.720 So a cloud VM or a workstation. s23 00:02:08.720 --> 00:02:12.160 Um, you also need a Tavi search API key. s24 00:02:12.160 --> 00:02:16.160 Tavi is a company that does uh search for agents. s25 00:02:16.160 --> 00:02:20.239 If you sign up on their website, they'll give you a thousand credits for free. s26 00:02:20.239 --> 00:02:23.680 And you need data designer, which is our tool for creating data. s27 00:02:23.680 --> 00:02:25.280 It's fully open source. s28 00:02:25.280 --> 00:02:29.280 Anyone can download it, use it, do whatever they want with it. s29 00:02:29.280 --> 00:02:35.280 And all the notebooks that I've provided um on the GitHub repo. s30 00:02:37.360 --> 00:02:38.879 So here's the agenda for today. s31 00:02:38.879 --> 00:02:49.599 So we'll first spend a few minutes talking about what is data designer and why do we actually need a framework like this and then we'll talk about uh the basics. s32 00:02:49.599 --> 00:02:53.680 Um there's three notebooks I'm going to actually be showing you code. s33 00:02:53.680 --> 00:02:57.040 U the first notebook is about the basics of data designer. s34 00:02:57.040 --> 00:02:59.360 How do you actually create data that you want? s35 00:02:59.360 --> 00:03:04.319 And then the second notebook is about model context protocol or MCP. s36 00:03:04.319 --> 00:03:06.800 How do you actually uh create agentic data? s37 00:03:06.800 --> 00:03:11.920 And then the third notebook which is what um is the super interesting use case of search. s38 00:03:11.920 --> 00:03:18.640 How do you actually teach a model to do search and then we'll leave some time at the end for questions. s39 00:03:19.599 --> 00:03:26.319 So I want to start off with a quote from Andre Karpati and he said internet data is not the data you want for your transformer. s40 00:03:26.319 --> 00:03:42.560 So this is um a very uh valid point because internet data could be you know highquality sources like Wikipedia or it could be you know someone's opinion on Twitter or X or some other platform right so how do we actually know s41 00:03:42.560 --> 00:03:56.560 um what data is useful for training AI models and what data is not useful so uh we need to process this internet data curate it and then uh create training data from it so that's the traditional approach approach. s42 00:03:56.560 --> 00:04:06.239 What we're proposing is that we could actually synthetically generate data uh either from scratch or from a seed data set. s43 00:04:06.239 --> 00:04:08.959 And so that's what data designer is for. s44 00:04:08.959 --> 00:04:13.599 So data designer is a framework to create synthetic data. s45 00:04:13.599 --> 00:04:16.799 And what do I mean by um seed data? s46 00:04:16.799 --> 00:04:25.120 So seed data like you could use Wikipedia data as a seed and then try to create synthetic training data from Wikipedia. s47 00:04:25.120 --> 00:04:29.680 So your synthetic data is not just made up by a model. s48 00:04:29.680 --> 00:04:35.680 It is grounded in real world uh distribution and real world data. s49 00:04:37.120 --> 00:04:40.160 So data design is a completely open-source framework. s50 00:04:40.160 --> 00:04:43.440 We have a large team at NVIDIA that works on this. s51 00:04:43.440 --> 00:04:51.600 Um and uh you can just install it by uh running pip install data designer. s52 00:04:51.840 --> 00:05:06.880 And I think um what's really um what we're really proud of is data designer has been used to create six and a half trillion tokens of um training data and this is increasing every single day. s53 00:05:06.880 --> 00:05:11.199 Um and yeah we're keeping track of the GitHub stars as well. s54 00:05:11.199 --> 00:05:14.960 So if you like our repo give us a star. s55 00:05:16.080 --> 00:05:18.960 So uh what is synthetic data generation right? s56 00:05:18.960 --> 00:05:29.199 U well one way to do synthetic data generation is take a loop run it a million times and then if you want to create a product review data set just s57 00:05:29.199 --> 00:05:40.960 you know continually prompt a LLM saying give me a product give me a review and then give me a customer profile that's one way to create a data set but you're going to run into a bunch of issues if you do that s58 00:05:40.960 --> 00:05:53.120 so what's missing um well diversity diversity diversity that's the most important thing right if you continually prompt LLM after 10, 12, maybe 20 tries, it's going to start repeating stuff. s59 00:05:53.120 --> 00:05:55.280 It's not going to give you what you need. s60 00:05:55.280 --> 00:06:01.759 Um, you're also missing correlations between all the features that you're looking for in your data set. s61 00:06:01.759 --> 00:06:04.240 Um, you don't have multimodality. s62 00:06:04.240 --> 00:06:12.479 Um, you know, you're not really able to steer the creation of that data set and there's no validation, verification, uh, and so on. s63 00:06:12.479 --> 00:06:23.039 And if I create a data set, I can't really share it with you because how do you actually take the uh process that I use to create my data set to create something similar. s64 00:06:23.039 --> 00:06:26.800 So that's why we created data designer. s65 00:06:26.800 --> 00:06:28.319 So how does it work? s66 00:06:28.319 --> 00:06:34.000 Uh we have many different columns or features in data designer. s67 00:06:34.000 --> 00:06:40.000 U these columns are essentially um a data designer abstraction. s68 00:06:40.000 --> 00:06:43.520 In data designer, we generate data column by column. s69 00:06:43.520 --> 00:06:46.560 So first you have your seed column which is completely optional. s70 00:06:46.560 --> 00:06:50.319 So I gave the example of Wikipedia which could be a seed data set. s71 00:06:50.319 --> 00:06:53.759 Um we could use any data set as a seed. s72 00:06:53.759 --> 00:06:57.919 So anything which is a CSV, JSONL, parket file. s73 00:06:57.919 --> 00:07:00.560 So you could use your own company data as seed. s74 00:07:00.560 --> 00:07:06.560 And then you can try to uh add sampler columns to inject diversity. s75 00:07:06.560 --> 00:07:09.440 So let's say you are trying to create a coding data set. s76 00:07:09.440 --> 00:07:15.280 Well, a very simpler sampler column would be like easy, medium, and difficult coding questions. s77 00:07:15.280 --> 00:07:16.880 So that's a sampler column. s78 00:07:16.880 --> 00:07:18.639 You could add which is difficulty. s79 00:07:18.639 --> 00:07:21.360 You could also add subsamplers. s80 00:07:21.360 --> 00:07:25.280 So for easy, you could map it to a specific type of problem. s81 00:07:25.280 --> 00:07:28.560 For medium, you could map it to a specific type of problem and so on. s82 00:07:28.560 --> 00:07:29.840 So we'll get into all of this. s83 00:07:29.840 --> 00:07:33.680 I know it feels a bit abstract, but we'll get into all of this once we look at the code. s84 00:07:33.680 --> 00:07:40.000 Um and then we have columns for generating text, for generating structured data, for generating code. s85 00:07:40.000 --> 00:07:43.199 Uh we also uh data designer is now multimodal. s86 00:07:43.199 --> 00:07:45.199 So you can generate images as well. s87 00:07:45.199 --> 00:07:50.880 Um and you can validate your generations too. s88 00:07:51.120 --> 00:08:06.800 So the whole idea is we build a DAG or a graph which describes how your data set should be and then this DAG once you're ready to execute on it you can just hit one click and get a data set out of it. s89 00:08:06.800 --> 00:08:09.759 So there's three different notebooks we'll walk through today. s90 00:08:09.759 --> 00:08:14.800 Um I know some of you are thinking well in 2026 he's going to be showing us code right? s91 00:08:14.800 --> 00:08:19.520 I mean now there's codeex and cloud code and all these amazing tools. s92 00:08:19.520 --> 00:08:28.639 So I think if you understand the code, it'll be much easier to use codeex or cloud code to write your own uh synthetic data workflows for you. s93 00:08:28.639 --> 00:08:33.440 Uh so we'll just take a brief look at each notebook now. s94 00:08:33.680 --> 00:08:35.839 And by the way, this is our repo. s95 00:08:35.839 --> 00:08:38.000 So this is our data designer repo. s96 00:08:38.000 --> 00:08:39.919 It's fully open source. s97 00:08:39.919 --> 00:08:48.720 Um and this is the GitHub repo that um I have created for this workshop where we have notebooks. s98 00:08:48.720 --> 00:09:01.839 Um so you can try out notebook one 2 3 and then we also have um the slides here if you want to follow along or if you want access to the slides after the workshop. s99 00:09:02.000 --> 00:09:04.480 So let's go to the first notebook. s100 00:09:04.480 --> 00:09:06.800 So this is our intro to data designer notebook. s101 00:09:06.800 --> 00:09:07.519 Can everyone see? s102 00:09:07.519 --> 00:09:07.760 Okay. s103 00:09:07.760 --> 00:09:11.040 or should I zoom in? s104 00:09:11.680 --> 00:09:13.920 It's good. s105 00:09:15.279 --> 00:09:21.360 Okay, so first things first, we set up our environment. s106 00:09:21.360 --> 00:09:25.920 Um there's a bunch of uh prompts in here that we're importing. s107 00:09:25.920 --> 00:09:29.920 And then what we do is we just, you know, import data designer. s108 00:09:29.920 --> 00:09:31.440 That's it literally. s109 00:09:31.440 --> 00:09:37.360 Uh and once you do that, you see in data designer there's a concept of model providers. s110 00:09:37.360 --> 00:09:42.160 Um, so we're currently using uh VLLM, so that's local inference. s111 00:09:42.160 --> 00:09:48.959 So I'm running this on a on a on a Nvidia GPU cluster and then we have access to different types of models, right? s112 00:09:48.959 --> 00:09:52.800 So on my local cluster, I have access to GPTOSS 120 billion. s113 00:09:52.800 --> 00:09:57.440 You can use any model or any open model to create synthetic data. s114 00:09:57.440 --> 00:10:05.360 Um, some closed models prevent you because of licensing issues uh from creating data from it and using it for training models. s115 00:10:05.360 --> 00:10:07.680 But open models are usually fair game. s116 00:10:07.680 --> 00:10:16.720 So I'm using this model and um the goal here is can I actually take a data set of Wikipedia topics uh for this notebook. s117 00:10:16.720 --> 00:10:24.160 So like you know different like uh what Wikipedia has so many different pages and um uh many different topics. s118 00:10:24.160 --> 00:10:32.079 So we just made a list of those put it in a CSV and then we basically just call our config builder with seed data set. s119 00:10:32.079 --> 00:10:39.120 And so essentially now we're um trying to create a data designer workflow but with Wikipedia uh topics. s120 00:10:39.120 --> 00:10:43.200 Um so once you do that you can add question difficulty. s121 00:10:43.200 --> 00:10:46.800 You can say easy, medium, hard, however you want your data set to be. s122 00:10:46.800 --> 00:10:48.480 And then there's weights you can specify. s123 00:10:48.480 --> 00:10:53.519 So you can have more medium questions than um you know easy or hard. s124 00:10:53.519 --> 00:10:59.200 And then there's all kinds of sampler columns you can add about question difficulty. s125 00:10:59.200 --> 00:11:13.680 So for example for easy you might do like direct lookup or simple fact or definition right but difficult might be you want something in more detail or obscure um and then once we get to the LLM generated columns s126 00:11:13.680 --> 00:11:23.519 you just call this function add column and then you have your prompt which is essentially going to um look something like this. s127 00:11:26.880 --> 00:11:29.760 Just trying to find it. s128 00:11:35.440 --> 00:11:36.399 Yep. s129 00:11:36.399 --> 00:11:39.279 So, you are an expert trivia question writer. s130 00:11:39.279 --> 00:11:42.480 Um, so here's a topic difficulty question type. s131 00:11:42.480 --> 00:11:53.120 So whenever you put these uh ginger templating for topic, what it's going to do is the topics you created in the previous um or in the seed that you had from your Wikipedia, s132 00:11:53.120 --> 00:11:57.519 it's going to insert those at runtime into this prompt. s133 00:11:57.519 --> 00:12:02.399 So essentially every prompt that the LM sees is now unique and diverse. s134 00:12:02.399 --> 00:12:08.000 And so the model is going to give you diverse data when you actually hit generate. s135 00:12:08.240 --> 00:12:13.440 Um and then um you you can also have like a structured column. s136 00:12:13.440 --> 00:12:24.240 So a structured column is when instead of trying to generate like text data um you can try to generate like a JSON or markdown format or any custom format that you're interested in. s137 00:12:24.240 --> 00:12:28.880 Um and uh we know that LLMs you know we have to be nice to them. s138 00:12:28.880 --> 00:12:32.720 We have to talk to them but sometimes they may not give you that structured format that you want. s139 00:12:32.720 --> 00:12:36.399 So then we just end up dropping those records. s140 00:12:39.600 --> 00:12:48.160 So the way um data creation works is we have something called preview mode in data designer. s141 00:12:48.160 --> 00:12:55.600 So essentially when you hit preview what's going to happen is it's going to try to create three records for you and then you can go and look at those records. s142 00:12:55.600 --> 00:12:56.959 So let's look at some records. s143 00:12:56.959 --> 00:12:59.360 So all of this will make a lot more sense. s144 00:12:59.360 --> 00:13:01.920 So the topic here is the worldwide web. s145 00:13:01.920 --> 00:13:16.560 Um, and the question or I guess some consortium of the worldwide web and the question is which W3C recommendation is identified by the same three-letter abbreviation that serves as a airport code for some city in Norway. s146 00:13:16.560 --> 00:13:19.040 So there's no way I would know the answer to this question, right? s147 00:13:19.040 --> 00:13:24.320 That's why these questions are essentially hard because you want to invoke the model to do search, right? s148 00:13:24.320 --> 00:13:28.000 But in this first notebook, we're just trying to create question answer pairs. s149 00:13:28.000 --> 00:13:30.079 So uh which is a very simple data set. s150 00:13:30.079 --> 00:13:32.480 So I guess this is the answer the model came up with. s151 00:13:32.480 --> 00:13:37.519 And then you can also add LLM as a judge to score how your model is doing. s152 00:13:37.519 --> 00:13:39.760 So this is the question the model came up with. s153 00:13:39.760 --> 00:13:41.839 This is the answer it came up with. s154 00:13:41.839 --> 00:13:44.560 And then um this is the question quality. s155 00:13:44.560 --> 00:13:53.360 So we can have another model judge what uh it thinks of the question that the uh previous model created um and so on. s156 00:13:53.360 --> 00:13:55.760 So in the end you end up with a data set like this. s157 00:13:55.760 --> 00:14:01.440 And by the way, we have a a function that you can just push your data sets directly to hugging face. s158 00:14:01.440 --> 00:14:10.560 Um so it's super easy to use and um you can also visualize all of your questions here as you can see. s159 00:14:10.560 --> 00:14:12.720 And then you can get some stats about your data set. s160 00:14:12.720 --> 00:14:15.199 Obviously we generated only three records here. s161 00:14:15.199 --> 00:14:21.360 So uh you know these stats are not very meaningful but if you generate like hundreds or thousands of records you could do that. s162 00:14:21.360 --> 00:14:23.360 And then we have something called batch mode. s163 00:14:23.360 --> 00:14:32.880 So which is data designer.create create and here instead of a 100 I could just you know put how many other records I need and then I can go and create all these records s164 00:14:32.880 --> 00:14:46.639 and as I mentioned previously we generate data column by column so in the batch mode first you generate the questions and then you generate the answers right so this is obviously a very simple example s165 00:14:46.639 --> 00:14:56.240 and then you have the LLM as a judge um and here's your final data set that's what it looks like so That was the first notebook. s166 00:14:56.240 --> 00:15:00.560 I think it's quite straightforward once you look at it and try it out. s167 00:15:00.560 --> 00:15:12.720 Um, we have a second notebook which focuses more on MCP and then we're going to build to the third notebook which I think everyone will find interesting because that's where we'll actually talk about search. s168 00:15:12.720 --> 00:15:14.720 Sorry. s169 00:15:15.199 --> 00:15:17.199 Yeah. s170 00:15:18.959 --> 00:15:19.360 Oh yeah. s171 00:15:19.360 --> 00:15:21.279 The all the GitHub links are the same. s172 00:15:21.279 --> 00:15:23.040 I'm just putting them in every single slide. s173 00:15:23.040 --> 00:15:26.320 So, if you missed the first one, you can you can get it. s174 00:15:26.320 --> 00:15:30.160 But I'll pause here for a minute if you want to scan. s175 00:15:37.760 --> 00:15:40.800 So, um let's talk about MCP. s176 00:15:40.800 --> 00:15:48.639 So, MCP is a standard which allows um an agent to essentially connect to tools, right? s177 00:15:48.639 --> 00:15:53.440 So the tool here is tavly which is nothing but a search tool for agents. s178 00:15:53.440 --> 00:15:55.519 There are many other search tools as well. s179 00:15:55.519 --> 00:16:06.800 Um so that's basically the um uh what what we need the mo what we're trying to teach the model here is how do you do search right so uh tavly s180 00:16:06.800 --> 00:16:11.360 is a good tool for the model to use to learn how to do that search. s181 00:16:11.360 --> 00:16:15.360 Um so going back to the second notebook. s182 00:16:18.000 --> 00:16:27.759 So here what we'll do is we'll just build upon some simple concepts from the first notebook and we'll add um how to create agentic data. s183 00:16:27.759 --> 00:16:36.639 So again you have the imports you have the the model configs and then the same seed data set the same sampler column. s184 00:16:36.639 --> 00:16:40.000 So question difficulty, question type and so on. s185 00:16:40.000 --> 00:16:44.880 And then you actually create the question and then at the end you get like a config, right? s186 00:16:44.880 --> 00:16:54.959 I don't know why it's printing printed this in like uh light color but yeah this is essentially your graph um that you're creating and then you have the answer. s187 00:16:54.959 --> 00:17:03.519 So it's the exact same thing except now when you actually create the answer you add this tool alias here. s188 00:17:03.519 --> 00:17:15.679 And so the goal is instead of just trying to answer the question, the model is now going to use a tavly search tool to try to find the answer on the internet. s189 00:17:15.679 --> 00:17:16.400 Right? s190 00:17:16.400 --> 00:17:23.439 So that's we're build we're building upon the first notebook where we were just like okay please generate a question and then please generate an answer. s191 00:17:23.439 --> 00:17:27.760 Now it's please generate a question and then hey you have this tool available. s192 00:17:27.760 --> 00:17:33.280 Can you use this tool to go to try to find the answer rather than trying to generate the answer? s193 00:17:33.280 --> 00:17:38.320 So that's that's the difference between the first and the second notebook. s194 00:17:38.880 --> 00:17:45.200 And then again you have the preview mode where you can generate a few records and then you look at your records. s195 00:17:45.200 --> 00:17:48.240 And by the way data creation is very iterative. s196 00:17:48.240 --> 00:17:50.160 It's not like one and done. s197 00:17:50.160 --> 00:17:52.480 So when you create some data you look at it. s198 00:17:52.480 --> 00:18:03.679 If you're not happy with it, then probably you go change your workflow, change your prompts and then you know iterate on it and then once you're happy with it, you can send it off to batch execution. s199 00:18:03.679 --> 00:18:08.320 So let's look at what um the second notebook produced. s200 00:18:08.320 --> 00:18:16.799 So let's say the question is uh to which German city did Teleifan move its headquarters in 1949. s201 00:18:16.799 --> 00:18:19.280 Does anyone know the answer? s202 00:18:19.280 --> 00:18:19.840 No. Right? s203 00:18:19.840 --> 00:18:21.440 I mean I don't know the answer. s204 00:18:21.440 --> 00:18:35.039 So um what we want to do is create questions that are difficult so it invokes reasoning and tool calling from the model and so when the model is actually trying to respond to this s205 00:18:35.039 --> 00:18:48.640 um it's as you can see here it's using tavly so the first search it made is tell if you can moved its headquarters in 1949 to which company so essentially it's using one search tool called here and then s206 00:18:48.640 --> 00:18:53.200 um it's going to get an answer so this is the query from the model and then here are the results. s207 00:18:53.200 --> 00:18:59.520 These are all the URLs and then for each URL it's going to give you a title and then it's also giving you the content. s208 00:18:59.520 --> 00:19:09.919 So that's what Tavali gives us right if you make a query or that's what it gives the agent when the agent makes the query and then you end up with the response and then finally s209 00:19:09.919 --> 00:19:16.880 at the end you can create your data set with batch mode just like we did last time. s210 00:19:16.880 --> 00:19:23.840 And if you're thinking he just blazed through this whole first two notebooks, that's because the crux of it is in the third notebook, right? s211 00:19:23.840 --> 00:19:27.679 So, um that's what I really want us to focus on today. s212 00:19:27.679 --> 00:19:29.360 So, let's talk about search. s213 00:19:29.360 --> 00:19:38.480 Um cuz we just went through a couple of notebooks showing us how to use data designer and that's fine, but what's the problem we're actually trying to solve? s214 00:19:38.480 --> 00:19:44.400 So, um like I mentioned before, I'm a research scientist at NVIDIA and we're also part of the Neotron team. s215 00:19:44.400 --> 00:19:50.400 So, Neimotron is a collection of found open-source foundation models um that we've created. s216 00:19:50.400 --> 00:19:54.400 Uh we just released our latest model Ultra, but we also have two other models. s217 00:19:54.400 --> 00:19:56.960 So, we have three different sizes of models. s218 00:19:56.960 --> 00:20:02.720 And so, as part of that work, we actually had to teach Neotron how to do search. s219 00:20:02.720 --> 00:20:04.799 And that's very important, right? s220 00:20:04.799 --> 00:20:15.600 Because maybe this feels like a long time ago now, but maybe two or three years ago, I'm sure you all remember looking at like I don't know chat GPT Claude or Gemini and then looking at the s221 00:20:15.600 --> 00:20:20.320 web UI and it says the knowledge cutoff date is December 2023. s222 00:20:20.320 --> 00:20:25.679 So the reason it said that is because well you have to start training the model at some point. s223 00:20:25.679 --> 00:20:29.280 So you have to stop collecting and creating data at some point, right? s224 00:20:29.280 --> 00:20:32.559 So let's say they're training the model in January 24. s225 00:20:32.559 --> 00:20:37.360 by December 23, they need to like um essentially freeze the data. s226 00:20:37.360 --> 00:20:45.360 So, anything that the model knows after um December 23 is not part of its training data. s227 00:20:45.360 --> 00:20:53.280 So, if you had to ask a question like what is the weather today, there's no way the model can answer that question from its training data, right? s228 00:20:53.280 --> 00:20:55.679 Uh question. s229 00:20:58.480 --> 00:21:00.480 Yeah. s230 00:21:00.640 --> 00:21:02.960 So yeah, definitely scan the QR code. s231 00:21:02.960 --> 00:21:15.280 So the idea is how do you actually um teach the model about current events because learning about current events seems as important if not more important than you know knowing about s232 00:21:15.280 --> 00:21:17.280 uh things from the past, right? s233 00:21:17.280 --> 00:21:24.159 So um that's why search exists and that's why models need to learn how to do search. s234 00:21:24.159 --> 00:21:40.240 Um so coming back to the slide um so teaching neotron models how to use search tools that was our motivation and um um we essentially the model needs access to real-time information to be able to do useful things. s235 00:21:40.240 --> 00:21:42.320 So how did we actually do this? s236 00:21:42.320 --> 00:21:44.240 So before I get to how did we do this? s237 00:21:44.240 --> 00:21:46.400 We need something to measure progress on. s238 00:21:46.400 --> 00:21:47.039 Right? s239 00:21:47.039 --> 00:22:02.559 So um in the foundation model community when you need to measure progress on something you use a benchmark which is like a standardized test set without the answers right like we don't have the answers so one of the benchmarks for search is browse comp s240 00:22:02.559 --> 00:22:13.360 uh which is a really difficult benchmark and uh the FIFA World Cup is going on right now so I don't know how many soccer fans there are here but I'm I'm a big soccer fan but we don't call it soccer s241 00:22:13.360 --> 00:22:16.960 but um between 1990 and 1994 four. s242 00:22:16.960 --> 00:22:20.720 Which teams played in a soccer match with a Brazilian referee? s243 00:22:20.720 --> 00:22:25.440 Had four yellow cards, two for each team, three in total, blah blah blah. s244 00:22:25.440 --> 00:22:29.360 I won't read the rest of the question, but it's it's quite complicated question, right? s245 00:22:29.360 --> 00:22:39.120 There's no way that any of us know the answer to that or maybe there's like, you know, someone who's um really wellversed in this who has the answer at the top of their head. s246 00:22:39.120 --> 00:22:51.600 What we would do back in the day like you know three four years ago is we'd actually try to break a problem down if we had a complicated uh query like this and then we would reason about what queries to actually make. s247 00:22:51.600 --> 00:23:06.320 We would go and search on Google or elsewhere uh get access to more information and then reason about what was our original goal, what information did we get access to and then proceed from there and then finally arrive at the answer. s248 00:23:06.320 --> 00:23:06.799 Right? s249 00:23:06.799 --> 00:23:09.039 So the answer is Ireland versus Romania. s250 00:23:09.039 --> 00:23:11.280 I I didn't know that. s251 00:23:11.280 --> 00:23:22.240 Um so how did we actually create a data set that allows us to train a model to go from 0% to 30% on this browsecom benchmark? s252 00:23:22.240 --> 00:23:25.840 Um so that's what we're going to learn about today. s253 00:23:25.840 --> 00:23:36.720 So to do that we took we did something um uh we we used a seed data set which I described earlier except our seed data set is not just Wikipedia s254 00:23:36.720 --> 00:23:55.760 it's something that um is essentially a graph of Wikipedia topics so it's called wiki data and what we did is we did a few hops along this graph and that is our seed data set and I know the diagram on the right looks like a tree it's not a tree it's a graph s255 00:23:55.760 --> 00:23:58.400 So um if it looks like a tree that's my fault. s256 00:23:58.400 --> 00:24:01.200 Uh it should it should it should be a graph. s257 00:24:01.200 --> 00:24:19.440 Um and essentially um you can see um on the right the first topic is Nvidia and then Nvidia is connected to Jensen who is our CEO and then that is connected to Oregon State University because Wiki Wiki data says that that's where he went to college s258 00:24:19.440 --> 00:24:30.240 and then that college is located in Benton County and Benton County is loca is actually named after a person called Thomas Hart Benton. s259 00:24:30.240 --> 00:24:43.200 So if you took um if you randomly sampled an entity in wiki data and then did five to eight hops along it and I'll explain why five to eight you'll end up s260 00:24:43.200 --> 00:24:50.960 um with a hop like this like the green you know set of entities with like arrows in between. s261 00:24:50.960 --> 00:24:55.760 So that is essentially our seed data set to create uh search riddles. s262 00:24:55.760 --> 00:25:13.039 So, one search riddle that I could create from this, and I'm going to have to read this from my speaker notes, is um which company was founded by the person who went to college where the county was named after Thomas Hart Benton. s263 00:25:13.039 --> 00:25:17.279 So, there's no way the LLM knows the answer to that in the training data, right? s264 00:25:17.279 --> 00:25:22.559 Because if we give an agent the chance to cheat, it's going to cheat. s265 00:25:22.559 --> 00:25:24.240 It's going to just give us the answer. s266 00:25:24.240 --> 00:25:34.080 it's not going to use the tool calls or the tools sorry like search that we want it to use right so we want to make the questions as difficult and as I guess s267 00:25:34.080 --> 00:25:52.880 um riddle like as possible so we can invoke the search behavior from the model so this is our seed so I I spoke about the first step which is create random walks with seed and why four to eight or five to eight hops sometimes it just you know if you do these hops it just goes completely off s268 00:25:52.880 --> 00:26:11.279 topic like It will say um I don't know like u a person's name and then it will say well human being well sure that is true but you know it's not really related like basically your hops will go off topic so that's why we want to as a heristic stay between four and eight that's what worked well s269 00:26:11.279 --> 00:26:27.440 for me and then you transform these seeds into search riddles and then um you execute the search via the search tool which is tavly so that's how you create a data set with a prompt and a with a search prompt and a search response, right? s270 00:26:27.440 --> 00:26:33.520 And then we evaluate our data and then something that's not mentioned here is, you know, we train our models, right? s271 00:26:33.520 --> 00:26:40.960 So once you have the data set, you train the model with it and then you measure performance on a standardized benchmark. s272 00:26:40.960 --> 00:26:45.360 So let's look at the notebook and see how we can do this. s273 00:26:45.360 --> 00:26:48.480 Any questions so far? s274 00:26:49.039 --> 00:26:51.039 Yes. s275 00:26:57.520 --> 00:27:12.400 So the reason uh it was important to do I guess it's not difficult questions it's questions that the model uh cannot find on the internet readily uh cannot find answers readily on the internet too. s276 00:27:12.400 --> 00:27:19.760 So like if you took that full blob of text that we had in the browsecom benchmark and you put that on the web you're not going to get the answer. s277 00:27:19.760 --> 00:27:26.400 I mean you will get the answer because of the AI search thing with Gemini now but that's what we're trying to teach the model essentially. s278 00:27:26.400 --> 00:27:28.400 Yeah. s279 00:27:30.880 --> 00:27:31.919 Okay. s280 00:27:31.919 --> 00:27:40.480 So um so this is the tutorial for the end to end aentic search. s281 00:27:40.880 --> 00:27:45.600 So we already spoke about the way we do this and how we create the data. s282 00:27:45.600 --> 00:27:53.600 We already spoke about how we import all these um prompts and I wrote a function to like use Tavi with MCP. s283 00:27:53.600 --> 00:27:55.760 You can go and look at the code later. s284 00:27:55.760 --> 00:28:03.679 Um and here is where we have our seed entities. s285 00:28:03.679 --> 00:28:05.440 So this is the example I showed you. s286 00:28:05.440 --> 00:28:07.360 This is just one seed entity. s287 00:28:07.360 --> 00:28:21.679 So the way uh my seed collection algorithm works is like you can run it a million times and you get million different hops along wiki data right so those are just topics we want to ground our search riddle creation on s288 00:28:21.679 --> 00:28:36.399 so another topic is um we like modern art and like I'm not really an art buff but so I don't understand most of these words but essentially this is like another seed hop that you can s289 00:28:36.799 --> 00:28:52.000 So the way to create this data set is um well you are an expert search agent that uses web search to answer questions accurately provide valid JSON with the final answer supporting URLs and a short justification s290 00:28:52.000 --> 00:28:53.760 and here are the tools you have available. s291 00:28:53.760 --> 00:28:59.279 You have one tool which is tavly search and then here's some usage guidelines for the tool. s292 00:28:59.279 --> 00:29:09.440 So, like we're like, "Okay, please please don't use more than like, you know, don't don't like call the tool like a hundred times and spend, you know, $1,000 or something." s293 00:29:09.440 --> 00:29:26.880 I guess those are like all kinds of instructions you can put in there, but essentially um uh jokes aside, what you want here is um your agent shouldn't go crazy and try to like you know spend way too much money answering a question that's very easy s294 00:29:26.880 --> 00:29:29.120 or you know something like that. s295 00:29:29.120 --> 00:29:33.520 So that's why we have all these instructions and we also have an example. s296 00:29:33.520 --> 00:29:38.559 So this is um um something like in context learning, right? s297 00:29:38.559 --> 00:29:48.320 So you put an example into the prompt and then the model can essentially um learn from that example how to actually call the search tool. s298 00:29:48.320 --> 00:29:51.279 So here's your prompt that you want to create. s299 00:29:51.279 --> 00:29:58.480 So here what we do is um we collected the wiki data hops. s300 00:29:58.480 --> 00:30:05.520 We can insert those hops into the prompt here uh which is our um you know seed entity. s301 00:30:05.520 --> 00:30:16.320 And then uh we ask the model to essentially come up with a question um using any or all of the seed entities from this hop. s302 00:30:16.320 --> 00:30:28.240 And then we ask the model to create a response using uh this agent solution draw which I showed earlier. s303 00:30:28.960 --> 00:30:32.640 So in the in the second notebook. s304 00:30:32.640 --> 00:30:36.720 So once you do this, you'll end up with a preview data set. s305 00:30:36.720 --> 00:30:38.559 And so here's our preview data set. s306 00:30:38.559 --> 00:30:44.000 So Indian Council of Medical Research is the start entity for the seed. s307 00:30:44.000 --> 00:30:51.520 And then apparently that's located in a city called Bhuaneshwar which is a capital of some state in India. s308 00:30:51.520 --> 00:30:54.080 Uh so that's our seed path. s309 00:30:54.080 --> 00:31:00.720 And then here is the question that the model came up with. s310 00:31:00.720 --> 00:31:04.080 And then this Asian solution draw is not displayed in here. s311 00:31:04.080 --> 00:31:07.200 But if you look at it, we can we can see what it looks like here. s312 00:31:07.200 --> 00:31:08.880 So it did a bunch of tool calls. s313 00:31:08.880 --> 00:31:15.440 It reasoned about the question and then it's trying to find the answer on the internet. s314 00:31:15.760 --> 00:31:21.120 So that's how you create um a search tool called data set. s315 00:31:21.120 --> 00:31:30.559 And I guess I'll pause here for questions and then we can move on to the next few slides after that. s316 00:31:30.559 --> 00:31:31.360 Any questions? s317 00:31:31.360 --> 00:31:34.320 If something was Yes. s318 00:31:51.679 --> 00:31:53.679 Yeah, s319 00:31:53.760 --> 00:31:55.760 absolutely. s320 00:31:56.000 --> 00:31:56.960 Yes. s321 00:31:56.960 --> 00:32:00.640 So we moved the bird benchmark from 26 to 41%. s322 00:32:00.640 --> 00:32:05.679 So you asked the you know exact right question and we have a blog post on this as well. s323 00:32:05.679 --> 00:32:08.880 Um you can read about it. s324 00:32:08.880 --> 00:32:20.640 Um so all the data sets we've created with data designer um we have blog posts and you can see right here so I was involved in this work as well. s325 00:32:20.640 --> 00:32:30.159 Uh so yeah this talks about SQL and I can also share the link with you after um but essentially data designer can be used to create any kind of data set s326 00:32:30.159 --> 00:32:37.840 for training AI models possibly other data sets as well if you're looking for test data sets not related to model training and so on. s327 00:32:37.840 --> 00:32:46.480 Um and we have a bunch of data tools that we've created in our team apart from data designer that I'll talk about in the end. s328 00:32:55.760 --> 00:32:59.760 So the soda is 70 on bird is because it's agentic. s329 00:32:59.760 --> 00:33:04.000 Our um current SQL data set is not agentic. s330 00:33:04.000 --> 00:33:10.159 And um also a sort on benchmarks is very much model dependent. s331 00:33:10.159 --> 00:33:17.919 We're talking about moving uh Neotron Nano which is our small size um LLM or SLM. s332 00:33:17.919 --> 00:33:18.159 Okay. s333 00:33:18.159 --> 00:33:21.919 I don't I don't know if you can call it an SLM because it's an MOE but it's an LLM. s334 00:33:21.919 --> 00:33:27.679 And so um for that model size it's a pretty respectable performance on board. s335 00:33:27.679 --> 00:33:28.399 Yeah. s336 00:33:28.399 --> 00:33:30.399 question. s337 00:33:47.039 --> 00:33:47.519 Yeah. s338 00:33:47.519 --> 00:34:00.880 No, this is a deterministic algorithm to create the seed data and then the uh tool calls and the agentic stuff is in creating the actual like performing the actual search once you have the question s339 00:34:00.880 --> 00:34:02.960 and creating the trajectory. s340 00:34:02.960 --> 00:34:04.240 We call it a trajectory. s341 00:34:04.240 --> 00:34:07.360 So creating the search trajectory this is deterministic. s342 00:34:07.360 --> 00:34:11.440 This is just something I came up with to create seed data. s343 00:34:11.440 --> 00:34:13.599 Yeah, you can have any kind of seed data. s344 00:34:13.599 --> 00:34:19.599 Maybe like you know you can come up with a better way of creating seed data than this as well. s345 00:34:26.240 --> 00:34:27.599 Oh great question. s346 00:34:27.599 --> 00:34:29.119 So we could do that. s347 00:34:29.119 --> 00:34:34.639 Um we have created data sets that can be used in pre-training and post- trainining. s348 00:34:34.639 --> 00:34:40.079 My main focus is post- training um or you know creating data sets for post- training. s349 00:34:40.079 --> 00:34:48.240 So this specific uh um I guess project was SFT uh in posting. s350 00:34:48.240 --> 00:34:50.240 Yes, s351 00:34:52.000 --> 00:34:58.079 there's no reason apart from the fact that uh I think someone at Nvidia did a deal with Tavly to get credits. s352 00:34:58.079 --> 00:34:59.040 You can use Exa. s353 00:34:59.040 --> 00:35:01.200 Exa is another search agent uh provider. s354 00:35:01.200 --> 00:35:03.119 You can write your own search tool. s355 00:35:03.119 --> 00:35:05.839 Um I guess so. s356 00:35:05.839 --> 00:35:08.640 Yeah, there's that as well. s357 00:35:08.640 --> 00:35:10.640 Yeah. s358 00:35:11.920 --> 00:35:13.920 essentially. s359 00:35:17.599 --> 00:35:35.839 So the golden responses are generated by um tail designer using the model that you choose and so um in the first notebook um yeah and sorry I think I think um you know I went a bit quick because it's we just have an hour for this workshop s360 00:35:35.839 --> 00:35:38.480 uh previous version of this workshop was 2 hours. s361 00:35:38.480 --> 00:35:43.119 So the model that I used is GPDOSS120B to create the data. s362 00:35:43.119 --> 00:35:48.240 Um but data designer is not just a collection of LLMs that you can use to create data. s363 00:35:48.240 --> 00:35:57.040 It has a bunch of LLM and nonLLM models as well working together to help you create diverse data. s364 00:35:58.240 --> 00:36:00.640 Say that again. s365 00:36:01.040 --> 00:36:01.839 This is SC. s366 00:36:01.839 --> 00:36:02.079 Yeah. s367 00:36:02.079 --> 00:36:05.119 Uh what do you mean? s368 00:36:07.920 --> 00:36:09.920 Yes. s369 00:36:10.079 --> 00:36:12.079 Yeah. s370 00:36:20.560 --> 00:36:22.560 Yes. s371 00:36:23.359 --> 00:36:24.720 Yes, you are distilling. s372 00:36:24.720 --> 00:36:27.839 But in some cases, you can even go beyond distillation. s373 00:36:27.839 --> 00:36:33.599 And that's the case that I'm trying to make because we have built this uh tool that allows you to create diverse data. s374 00:36:33.599 --> 00:36:37.359 So for example, a completely different use case that I was working on. s375 00:36:37.359 --> 00:36:50.640 GPTOSS gets some percentage on this uh benchmark called strructal which is a way to measure how models do on JSON, markdown, XML, YAML and CSV formats. s376 00:36:50.640 --> 00:36:56.320 And Neotron actually distilled from GPTOSS but went beyond GPOSS's performance. s377 00:36:56.320 --> 00:37:03.359 So it's distillation but it can also be more powerful than that because of all the things we've built into data designer. s378 00:37:03.359 --> 00:37:04.800 Yes. s379 00:37:04.800 --> 00:37:07.599 How can I use data designer? s380 00:37:07.599 --> 00:37:10.800 You can use data designer for RL. s381 00:37:10.880 --> 00:37:17.119 So let's say uh and by the way we're doing this for search right now or or the team at NVIDIA is doing this for search. s382 00:37:17.119 --> 00:37:33.920 So instead of trying to create the golden solution like how he described it or or you know um executing the search riddles as written on my slide, you just don't do that step and then you essentially end up with a data set full of prompts and then we have a tool called Nemo Gym s383 00:37:33.920 --> 00:37:39.839 which is essentially um an RL um reward function. s384 00:37:39.839 --> 00:37:44.720 You know I guess it's an it's a it's a tool to basically teach your models how to do RL. s385 00:37:44.720 --> 00:37:56.079 But the thing is in Nemo gym you write like a a function that essentially gives you a reward and then you can use our uh any RL training framework to s386 00:37:56.079 --> 00:38:03.680 execute in real time with those tools and then provide a reward in RL. s387 00:38:10.480 --> 00:38:11.280 It's a great point. s388 00:38:11.280 --> 00:38:13.680 We're thinking about how to do this actually currently. s389 00:38:13.680 --> 00:38:16.640 But to build something generic is the is the challenge, right? s390 00:38:16.640 --> 00:38:20.400 Like we don't want to build something specific to one use case. s391 00:38:20.400 --> 00:38:22.160 Yeah. s392 00:38:22.160 --> 00:38:24.160 Yeah. s393 00:38:30.000 --> 00:38:37.359 Is this reliable and accurate for ETL? s394 00:38:42.320 --> 00:38:56.880 So I haven't tried it myself but I think it is because it it should be because um we've created a diverse set of data sets with data designer and I think at the start we showed about 6 and a half trillion tokens s395 00:38:56.880 --> 00:39:02.800 and if it's not reliable you should come talk to us and see how we can help you. s396 00:39:05.359 --> 00:39:06.240 I can't hear you. s397 00:39:06.240 --> 00:39:08.240 Sorry. s398 00:39:11.599 --> 00:39:14.640 Sorry, I'm not able to hear. s399 00:39:17.520 --> 00:39:20.240 Oh, I I don't know about that. s400 00:39:20.240 --> 00:39:23.280 I I'll come back to you at the end and we can we can chat more. s401 00:39:23.280 --> 00:39:25.040 Yeah, sorry about that. s402 00:39:25.040 --> 00:39:25.760 Okay. s403 00:39:25.760 --> 00:39:28.720 Um, we have about 15 more minutes. s404 00:39:28.720 --> 00:39:40.720 So let me move on and we can have more questions at the end to something really interesting which is evaluating your synthetic data right and that's not an easy thing to do s405 00:39:40.720 --> 00:39:46.320 because without this evaluation step the synthetic data is not that useful. s406 00:39:46.320 --> 00:39:58.320 So when I started just to give you a sense of how SDG as we call it works synthetic data generation I started with 50,000 seeds and in the end I ended [clears throat] up only with 7,000 s407 00:39:58.320 --> 00:40:12.240 high quality endtoend SFT records and then I used that to train the model and we moved the benchmark from 0 to 30 right so that's great but 50,000 to 7,000 that's a lot of s408 00:40:12.240 --> 00:40:17.520 you know um um funneling and filter ing for high quality. s409 00:40:17.520 --> 00:40:27.599 So what like I'm trying to focus on here is that data quality is the most important thing when you're trying to create data and um what do I mean by quality? s410 00:40:27.599 --> 00:40:35.839 It can be task specific but it can also be generic like diversity is one way to um you know one axis of quality I would say. s411 00:40:35.839 --> 00:40:51.359 Um so after that I created this function to help us um show like this is what the LLM as a judge scores for each trajectory um and what is like your s412 00:40:51.359 --> 00:41:00.480 um you know number of words in your uh question or search query and then here is your keep percentage 66% and so on. s413 00:41:00.480 --> 00:41:10.640 So what's really so you can take a look at that function later but what's really interesting is how do you actually check if the answers that the model got are correct or not. s414 00:41:10.640 --> 00:41:16.240 So I came across this really interesting example and the search riddle was quite simple. s415 00:41:16.240 --> 00:41:23.280 It said which comp which country owns US steel and I thought the answer is the US. s416 00:41:23.280 --> 00:41:43.359 And so um I you know I I guess the agent predicted US in the training data set and um my verifier said that it was correct but then I went and googled it and apparently the US sold US steel to Japan I think if I remember correctly a couple of years ago. s417 00:41:43.359 --> 00:41:48.079 So even my wiki data seed can become stale, right? s418 00:41:48.079 --> 00:42:04.079 And so that's the real challenge is how do you actually evaluate and um uh tell if the data that you've created is um good, accurate, you know, relevant and so on. s419 00:42:04.079 --> 00:42:17.839 Um so I guess if you go and implement a determinist I guess like a simple matching function like how I did you'll end up with like 24% and then you'll be like well my data is terrible s420 00:42:17.839 --> 00:42:25.280 why did I do all of this but that's not entirely true because here low percentage I expected because I don't know how to actually create that function. s421 00:42:25.280 --> 00:42:28.960 So this is an active area of research is how do you actually evaluate this data? s422 00:42:28.960 --> 00:42:33.920 So I think evaluation from my perspective is a very multifaceted problem. s423 00:42:33.920 --> 00:42:42.240 So you can't just take LLM as a judge and put it in there and hope that that's going to you know give you everything. s424 00:42:42.240 --> 00:42:51.200 Evaluation is a problem where if you check for a 100 problems and if you see that those 100 problems do all you all you can tell from checking for a 100 s425 00:42:51.200 --> 00:42:54.480 problems is those 100 issues don't exist in your data set. s426 00:42:54.480 --> 00:42:56.319 there could be a thousand other issues. s427 00:42:56.319 --> 00:42:59.760 So that's the challenge is how do you actually evaluate the data? s428 00:42:59.760 --> 00:43:02.720 Um so we have a few different ways that we do that. s429 00:43:02.720 --> 00:43:08.160 Um one of them is like I mentioned earlier LLM as a judge which is generic. s430 00:43:08.160 --> 00:43:17.920 Uh something I did really interesting is I actually did LLM as a judge with uh and I took the judgments and I sent them to human reviewers and I have those reviewers s431 00:43:17.920 --> 00:43:27.920 um cross-check the judgments of the LLM and essentially give me confidence that the LLM is judging my responses correctly. s432 00:43:27.920 --> 00:43:29.520 So that's one way to do it. s433 00:43:29.520 --> 00:43:32.480 And then in search you could look at real-time information. s434 00:43:32.480 --> 00:43:43.119 So you have to basically find a union of all these different different uh layers of evaluation that's going to essentially give you enough confidence in your data set. s435 00:43:43.119 --> 00:43:44.560 So that's it for the code. s436 00:43:44.560 --> 00:43:53.200 I know I blazed through this but hopefully I've you know um given access to all of this in my GitHub repo so you all can take a look. s437 00:43:53.200 --> 00:43:57.040 Um so let's talk about a few other use cases. s438 00:43:57.040 --> 00:44:02.960 So like I mentioned earlier, we are the um you know part of a much larger team that creates Neotron, right? s439 00:44:02.960 --> 00:44:12.880 So um my focus or my team's focus is essentially creating data sets or creating a framework for creating data sets and or I guess both of those. s440 00:44:12.880 --> 00:44:15.280 So there's all kinds of data sets we've created. s441 00:44:15.280 --> 00:44:18.640 The first one that we spoke about today, search tool calling. s442 00:44:18.640 --> 00:44:23.440 Um and then we also have a data set to do multi-turn tool calls. s443 00:44:23.440 --> 00:44:27.200 So, I think the search one was very much single turn, right? s444 00:44:27.200 --> 00:44:29.680 Like we're going from prompt to response. s445 00:44:29.680 --> 00:44:42.560 But what if you had multiple turns or what if you had multiple tools um or you know not just search tools but like you might have a I don't know a weather tool or a coding tool and so on. s446 00:44:42.560 --> 00:44:47.200 Um so we're working on multiple different projects here. s447 00:44:47.200 --> 00:44:49.599 Another really interesting one is structured output generation. s448 00:44:49.599 --> 00:44:50.800 I briefly spoke about it. s449 00:44:50.800 --> 00:45:03.920 There's a benchmark called strructal which measures your performance on toml, json, yaml, csv, xml and all of these are super important because everyone wants to use um you know agents s450 00:45:03.920 --> 00:45:05.599 to do everything. s451 00:45:05.599 --> 00:45:17.040 Now, I'm sure you all have seen the the uh the blowtorrch meme uh on LinkedIn or something, but but uh if we're going to be using agents as um the s452 00:45:17.040 --> 00:45:25.440 central piece in software applications, then they need to reliably give us um like a JSON format or a YAML format and so on. s453 00:45:25.440 --> 00:45:27.760 So, that's what we were focused on there. s454 00:45:27.760 --> 00:45:36.240 Um we've also worked on SQL um that someone asked about here and uh we've also worked on scientific reasoning. s455 00:45:36.240 --> 00:45:43.119 Um there's also pre-training data sets that we've created using data designer for stem performance and so on. s456 00:45:43.119 --> 00:45:49.200 Um and so I can talk about the best practices for generating data. s457 00:45:49.200 --> 00:45:51.680 Um I won't go over this in detail. s458 00:45:51.680 --> 00:45:56.319 There's because you can always look at the slides later but essentially diversity before volume. s459 00:45:56.319 --> 00:46:03.599 I think the Kim K2 paper showed something similar which is 10 strategies one epoch is better than one epoch. s460 00:46:03.599 --> 00:46:07.839 Sorry 10 strategies one epoch is better than 10 epochs one strategy. s461 00:46:07.839 --> 00:46:12.000 And um you always want to use the cheapest correct column type. s462 00:46:12.000 --> 00:46:15.920 So like using a LLM to create a sampler makes no sense, right? s463 00:46:15.920 --> 00:46:18.000 It's a complete waste of resources. s464 00:46:18.000 --> 00:46:21.119 But using LLM to generate text data makes a lot of sense. s465 00:46:21.119 --> 00:46:24.160 So our samplers are written in Python of course. s466 00:46:24.160 --> 00:46:33.760 Um and then you generate you overproduce data and then you filter down like I mentioned earlier and then you verify what you can uh iterate on the preview. s467 00:46:33.760 --> 00:46:37.760 I think the best way to know if your data is any good is to actually look at the data. s468 00:46:37.760 --> 00:46:38.960 That's the best way. s469 00:46:38.960 --> 00:46:45.040 Uh if you can look review a few samples and um of course what is this idea of data designer? s470 00:46:45.040 --> 00:46:53.599 Essentially you're taking the data creation process and making it um something you know version controllable uh sharable like code. s471 00:46:53.599 --> 00:46:57.200 So essentially we're creating code that can create data. s472 00:46:57.200 --> 00:47:06.240 So now you can share this code with anyone because you know we already built all this tooling for sharing code and having version control and all of these other things. s473 00:47:06.240 --> 00:47:15.839 So if you're trying to do search but not exactly the way I did it and just slightly differently you take my code you modify it and you can create your own data set from it. s474 00:47:15.839 --> 00:47:19.839 Um, so something that's super interesting. s475 00:47:19.839 --> 00:47:22.480 I'm getting into the extra slides which I didn't think I would. s476 00:47:22.480 --> 00:47:24.000 I think I have 10 more minutes. s477 00:47:24.000 --> 00:47:25.280 Right. s478 00:47:25.280 --> 00:47:29.040 So, um, this is something that's really interesting. s479 00:47:29.040 --> 00:47:32.480 We've created uh something called personas. s480 00:47:32.480 --> 00:47:40.560 So, Neotron Personas USA um it's also inspired from this other paper which is scaling synthetic data creation with 1 billion personas. s481 00:47:40.560 --> 00:47:41.680 So, what is a persona? s482 00:47:41.680 --> 00:47:42.240 Right. s483 00:47:42.240 --> 00:47:59.040 A persona is essentially um so we're trying to solve the problem here of how do you know if the data you've created actually um adheres to like demographic information or you know grounded in some real world distributions s484 00:47:59.040 --> 00:48:15.200 right that's very important u so if I were to create a data set of healthcare records in San Francisco California well I want to make sure that the uh the demograph the actual demographic of San Francisco is reflected in my data set, s485 00:48:15.200 --> 00:48:15.760 right? s486 00:48:15.760 --> 00:48:20.079 So like the number of uh you know there's all kinds of demographic features. s487 00:48:20.079 --> 00:48:31.520 Um so we created a data set called Neimatron personas uh using data designer but is also now a seed in data designer to create more data sets and um s488 00:48:31.520 --> 00:48:42.800 this essentially allows you to learn from user data without actually leaking any single person's information out there. s489 00:48:42.800 --> 00:48:53.359 So now you can train AI models and you can learn from the aggregate of um you know people's information but you don't actually have to have like for example my healthcare records s490 00:48:53.359 --> 00:48:58.640 you know directly um uh being used in training an AI model. s491 00:48:58.640 --> 00:49:08.400 I I keep using the healthcare example because I want to find a sensitive industry which is like you know there's HIPPA compliance there's all kinds of regulations there right that we have to respect. s492 00:49:08.400 --> 00:49:11.920 So, um, so this is a persona. s493 00:49:11.920 --> 00:49:17.839 Um, this person is a 50-year-old pharmacist with a free spirit, balances her relaxed approach, blah, blah, blah. s494 00:49:17.839 --> 00:49:28.720 But essentially, there's a persona, there's a professional persona, um, first name, last name, middle name, gender, age, um, all kinds of things. s495 00:49:28.720 --> 00:49:31.200 And we have a million of these. s496 00:49:31.200 --> 00:49:39.040 You can take this, you can use it, um, however you please as seed to create more data. s497 00:49:39.040 --> 00:49:44.160 So yeah, I think with the personas um I will probably stop here. s498 00:49:44.160 --> 00:49:45.119 So we have two more minutes. s499 00:49:45.119 --> 00:49:48.160 I'll talk about these other tools and then we'll leave time for questions. s500 00:49:48.160 --> 00:49:49.200 Oh, one question. s501 00:49:49.200 --> 00:49:50.240 Yeah. s502 00:49:50.240 --> 00:49:52.240 Yeah. s503 00:49:54.480 --> 00:49:56.480 For s504 00:49:58.640 --> 00:49:59.680 Yeah. s505 00:49:59.680 --> 00:49:59.920 Yeah. s506 00:49:59.920 --> 00:50:00.079 Yeah. s507 00:50:00.079 --> 00:50:00.559 Yeah. s508 00:50:00.559 --> 00:50:11.760 So essent that's essentially why we created it is to um have a way to learn from aggregate information without actually like you know having individual people. s509 00:50:11.760 --> 00:50:15.280 So like you could create synthetic social security numbers. s510 00:50:15.280 --> 00:50:18.960 Now, they're not actually going to be like your social security number, right? s511 00:50:18.960 --> 00:50:31.760 But I think the idea is like if there's um for example um um I don't know what is a good example that I s512 00:50:36.160 --> 00:50:36.800 Yeah. s513 00:50:36.800 --> 00:50:38.319 Yeah. s514 00:50:38.319 --> 00:50:38.640 Yeah. s515 00:50:38.640 --> 00:50:39.040 Yeah. s516 00:50:39.040 --> 00:50:39.599 Exactly. s517 00:50:39.599 --> 00:51:01.680 So for example, we we actually created some tax data for u you know a customer and this tax data I realized there's exactly uh seven people in the US who are below 18 earning more than $10 million a year on their tax return but there's like way more people who are older and they're earning you know x s518 00:51:01.680 --> 00:51:15.520 or y or z amount of money and so the tax data that we create let's say we want to create a synthetic W2 form we want to make sure that that form is not just random values from the LLM because the LLM is not going to give you s519 00:51:15.520 --> 00:51:20.559 um if you have you asked LLM to tell a joke it's going to tell you the same joke every time. s520 00:51:20.559 --> 00:51:24.720 So similarly, it's not going to give you like numbers that are realistic, right? s521 00:51:24.720 --> 00:51:27.839 Like it's always going to come back to the average of things. s522 00:51:27.839 --> 00:51:37.440 And so when you create data, you want to make sure that that synthetic W2 form or 1040 form or whatever is actually um adhering to real world distribution. s523 00:51:37.440 --> 00:51:41.119 So that's why we created this persona data set. s524 00:51:41.760 --> 00:51:45.599 So we have a bunch of tools uh just two minutes and then more questions. s525 00:51:45.599 --> 00:51:46.720 Sorry about that. s526 00:51:46.720 --> 00:51:50.079 Um so we have a bunch of tools that we've created. s527 00:51:50.079 --> 00:51:53.520 Data designer is only the tip of the iceberg I guess. s528 00:51:53.520 --> 00:51:57.040 Uh but it's our most successful tool so far because it's our oldest tool. s529 00:51:57.040 --> 00:52:04.400 Uh we also have uh you know at NVIDIA we have data curation tools. s530 00:52:04.400 --> 00:52:08.079 So which is Nemo curator which I'm sure there are other talks about this. s531 00:52:08.079 --> 00:52:10.319 I I believe I saw on the schedule. s532 00:52:10.319 --> 00:52:11.839 Um it is very interesting. s533 00:52:11.839 --> 00:52:27.680 you know you have to take data uh real world data curate it modify it post-process it get it to a state where it's usable to train in LLMs or you can create synthetic data so I think it depends on your use case like I'm not going to stand here and say synthetic data is the best thing ever s534 00:52:27.680 --> 00:52:43.599 for everything but I think there are specific use cases where you want to curate real world data or where you want to you know generate synthetic data um and so we have data uh curation tools we have data designer which is our synthetic data generation tool We have a new tool called anonymizer. s535 00:52:43.599 --> 00:52:46.559 And by the way, all these tools are completely open source. s536 00:52:46.559 --> 00:52:53.520 So everything that our org does at NVIDIA is fully open source, including Neotron, including all these tools. s537 00:52:53.520 --> 00:52:56.000 Um, so you can use it however you want. s538 00:52:56.000 --> 00:53:00.079 Um, so this data tool is called Nemo Anonymizer. s539 00:53:00.079 --> 00:53:03.200 And the goal here is can we actually anonymize data? s540 00:53:03.200 --> 00:53:17.280 So, I don't know about you guys, but I I don't know if I should be saying this when the talk is recorded, but I've been putting all kinds of things into my agents like you know, I accidentally put my API keys in there once or like you know some other sensitive information, right? s541 00:53:17.280 --> 00:53:21.440 All this log data for like all your agents that you're using. s542 00:53:21.440 --> 00:53:24.960 You need some way to anonymize it and before it's usable, right? s543 00:53:24.960 --> 00:53:26.800 So, that's what anonymizer does. s544 00:53:26.800 --> 00:53:30.880 So, that's just an example, but there's obviously many other ways to use anonymizer. s545 00:53:30.880 --> 00:53:38.880 And then we have Nemo safe synthesizer which is allowing us to generate safe synthetic versions of sensitive data sets um as well. s546 00:53:38.880 --> 00:53:42.160 So yeah come talk to me if you have any questions. s547 00:53:42.160 --> 00:53:45.760 Um here's my LinkedIn if you want to stay in touch. s548 00:53:45.760 --> 00:53:48.800 Uh again you can use data designer. s549 00:53:48.800 --> 00:53:51.040 I would love for you guys to try out the notebooks. s550 00:53:51.040 --> 00:53:57.599 I'm so sorry that we had to we had only one hour so I had to just like blaze through all the code but you can try out the notebooks. s551 00:53:57.599 --> 00:54:02.400 We also have the blogs if you're interested in text to SQL or some some other use case. s552 00:54:02.400 --> 00:54:07.520 So um this is this is showing us how to use the personas. s553 00:54:07.520 --> 00:54:13.040 Um I think there's a blog somewhere on how we created the personas data set for the US. s554 00:54:13.040 --> 00:54:28.960 We took the census information and we used a PGM, a probabilistic graphical model to essentially um create um like a connection between all basically take that demographic information, process it and then try to s555 00:54:28.960 --> 00:54:32.880 uh uh create a data set from it uh essentially. s556 00:54:32.880 --> 00:54:34.319 So that's what we did. s557 00:54:34.319 --> 00:54:40.079 Um and so every country pro uh publishes this uh demographic information. s558 00:54:40.079 --> 00:54:44.400 So, it's not really like um I'm looking at other slides now. s559 00:54:44.400 --> 00:54:47.359 We we've done a bunch of these workshops recently. s560 00:54:47.359 --> 00:54:59.200 Um but yeah, there's all kinds of personas if you're not if you're trying to build for something outside the US, if you're building something for Japan, India, Singapore, Brazil, I'm sure we have more now. s561 00:54:59.200 --> 00:55:02.079 So, we we have all kinds of personas data sets. s562 00:55:02.079 --> 00:55:14.559 So, um it's giving us um um a way to uh create customized AI for whichever country you're interested in.