WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=hd7TOvmyAxU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/hd7TOvmyAxU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:13.280 --> 00:00:14.080 Okay. s3 00:00:14.080 --> 00:00:17.039 Hello everyone and thank you for attending this session. s4 00:00:17.039 --> 00:00:21.760 My name is Tim Sweeney, a principal engineer at Weights and Biases and Coreweave. s5 00:00:21.760 --> 00:00:27.439 And for the next 20 minutes, we're going to talk about Arya, our new AI research and iteration agent. s6 00:00:27.439 --> 00:00:29.599 Let's go ahead and get started. s7 00:00:29.599 --> 00:00:37.440 So, uh, first off, just by way of making some noise, some clapping, uh, who here, um, identifies as an ML researcher? s8 00:00:37.440 --> 00:00:41.440 You're someone that trains models, trains the brain? s9 00:00:41.520 --> 00:00:42.480 I heard one. s10 00:00:42.480 --> 00:00:42.800 Wow. s11 00:00:42.800 --> 00:00:43.120 Okay. s12 00:00:43.120 --> 00:00:43.520 Great work. s13 00:00:43.520 --> 00:00:44.079 Great work. s14 00:00:44.079 --> 00:00:48.079 Uh, what about who here is the applied engineer, the namesake of this conference? s15 00:00:48.079 --> 00:00:50.960 Who here actually builds the bots? s16 00:00:50.960 --> 00:00:51.600 Okay, good. s17 00:00:51.600 --> 00:00:53.039 Expected much more. s18 00:00:53.039 --> 00:00:54.879 And who here is in AI management? s19 00:00:54.879 --> 00:00:58.160 You are helping fund this compute. s20 00:00:58.160 --> 00:00:58.640 Okay. s21 00:00:58.640 --> 00:00:59.039 Okay. s22 00:00:59.039 --> 00:00:59.600 Nice. s23 00:00:59.600 --> 00:01:00.640 From the back. s24 00:01:00.640 --> 00:01:01.359 Lovely. s25 00:01:01.359 --> 00:01:04.400 Um, well, now that I know a little bit about you, just a little bit about me. s26 00:01:04.400 --> 00:01:05.760 Uh, again, my name is Tim. s27 00:01:05.760 --> 00:01:09.680 I have a masters in machine learning, uh, and reinforcement learning from Georgia Tech. s28 00:01:09.680 --> 00:01:14.640 So, I've been that, uh, researcher currently building Weights and Biases agent, Arya. s29 00:01:14.640 --> 00:01:16.960 So, identify as that applied engineer. s30 00:01:16.960 --> 00:01:20.400 And in a previous life was the PM of Twitter's ML stack. s31 00:01:20.400 --> 00:01:23.840 So, I hope you hopefully can connect with you middle management as well. s32 00:01:23.840 --> 00:01:24.495 [laughter] s33 00:01:24.495 --> 00:01:24.960 [gasps] s34 00:01:24.960 --> 00:01:31.360 Um today's agenda is kind of broken into three sections and hopefully each of you personas walk away with something valuable. s35 00:01:31.360 --> 00:01:37.119 So first we're going to learn about Arya itself and how it can supercharge your AI and ML workflows. s36 00:01:37.119 --> 00:01:42.079 We're going to dive into auto research and see that live in a live demo in just a moment. s37 00:01:42.079 --> 00:01:54.399 Then we're going to pull back the curtain and learn how we use weights and biases and uh coreweave to actually build Arya because a lot of you in the audience are building agents yourself and we believe a lot of these components can help you in your endeavors. s38 00:01:54.399 --> 00:02:02.640 And then towards the end we'll just take a step back and identify a few key tips and tricks for making sure that you're able to productionize your systems effectively. s39 00:02:02.640 --> 00:02:07.920 For those of you who might not be familiar, Weights and Biases is the world's leading AI development platform. s40 00:02:07.920 --> 00:02:13.200 We've been in business now for nine years and have happily joined the core family about a year ago. s41 00:02:13.200 --> 00:02:25.760 Uh we have a number of products in our suite but are really known for our models training inference and weave stack which really helps collect data uh about the AI development and machine learning workflows and makes that information actionable s42 00:02:25.760 --> 00:02:30.560 and uh enables users to make the best decisions about what to do next. s43 00:02:30.560 --> 00:02:33.599 So without further ado, let's go ahead and dive into Arya, our agent. s44 00:02:33.599 --> 00:02:37.440 Uh we'll show a demo and then we'll get back to some slides. s45 00:02:40.160 --> 00:02:42.400 Okay, beautiful. s46 00:02:42.400 --> 00:02:43.519 Let's make this a bit bigger. s47 00:02:43.519 --> 00:02:45.440 Holler at me if you need it to be bigger. s48 00:02:45.440 --> 00:02:49.360 So, uh, what you're looking at here is a weights and biases workspace. s49 00:02:49.360 --> 00:02:55.440 For you, for anybody that isn't familiar, on the lefth hand side, I actually see a list of a bunch of different experiments. s50 00:02:55.440 --> 00:02:59.120 In this particular project, I have over 200 training jobs. s51 00:02:59.120 --> 00:03:04.560 And on the right hand side, I see a scatter plot of, in this case, declining metrics, which is good. s52 00:03:04.560 --> 00:03:06.800 means our loss is going down over time. s53 00:03:06.800 --> 00:03:10.319 And this view would be very familiar for anyone that uses our tool. s54 00:03:10.319 --> 00:03:20.879 Now, to ground this, we're actually uh uh using the Carpathy Auto Research project, which I'm sure many of you are familiar with, but if you're not, it's just a very simple project that trains an LLM, s55 00:03:20.879 --> 00:03:30.000 and it's a great foundation for auto research type demonstrations because it's a very simple codebase and allows us to improve iteratively over time. s56 00:03:30.000 --> 00:03:34.959 So, let's jump back to the project and open up Arya by clicking this blue button in the upper right. s57 00:03:34.959 --> 00:03:46.640 When I click this button, I'm uh presented with the familiar chat interface with, you know, how can I help you today, a few call to actions, and you know, I can add different context in my project or maybe add images, s58 00:03:46.640 --> 00:03:52.799 etc. Um, everyone here is agent builders, so I don't need to bore you with the details of what an agent interface looks like. s59 00:03:52.799 --> 00:03:56.720 But let's go ahead and just, you know, enter in a basic intro here. s60 00:03:56.720 --> 00:03:58.000 Let's say, "Hello, Arya. s61 00:03:58.000 --> 00:04:00.640 You're on stage at AI World's Fair 2026. s62 00:04:00.640 --> 00:04:02.000 Please introduce yourself." s63 00:04:02.000 --> 00:04:06.239 So, it's going to go ahead and chug along and hopefully emit some sort of nice emoji. s64 00:04:06.239 --> 00:04:06.640 Yay. s65 00:04:06.640 --> 00:04:07.920 He I'm Arya. s66 00:04:07.920 --> 00:04:09.280 I'm talking to the audience. s67 00:04:09.280 --> 00:04:10.000 Great. s68 00:04:10.000 --> 00:04:13.040 But now, let's dive into the meat of why you came here. s69 00:04:13.040 --> 00:04:15.200 So, I'm going to open up this chat here. s70 00:04:15.200 --> 00:04:21.199 And this is a longunning chat where I've been running again over 200 experiments using the auto research loop. s71 00:04:21.199 --> 00:04:30.320 Um, it helped me download the code, set up my launch jobs, set up my GPUs, and is able to autonomously iterate on the code itself and the hyperparameters. s72 00:04:30.320 --> 00:04:36.240 We'll take a look at what it's doing in a moment, but while we're doing this, I'm going to kick off a live iteration right here. s73 00:04:36.240 --> 00:04:40.240 So, what I'm going to say is please conduct another batch of experiments. s74 00:04:40.240 --> 00:04:46.080 You are on stage at the AI Engineer Worlds Fair 2026 and we're hoping to find the best model live. s75 00:04:46.080 --> 00:04:47.199 I believe in you. s76 00:04:47.199 --> 00:04:49.759 Uh, because we know we have to encourage our models. s77 00:04:49.759 --> 00:04:51.680 Um, so it's been doing this for a while. s78 00:04:51.680 --> 00:04:54.080 What it what it's doing here is it's saying, "Okay, great. s79 00:04:54.080 --> 00:04:56.479 Um, I don't want to make a big architecture swing. s80 00:04:56.479 --> 00:04:58.080 That feels a little bit too risky." s81 00:04:58.080 --> 00:05:02.400 So, it's probably going to go for uh some modifications to the hyperparameters. s82 00:05:02.400 --> 00:05:09.840 And then it's kicking off a shell call here that is actually um executing that uh executing that experimentation loop. s83 00:05:09.840 --> 00:05:13.600 And we're going to check in on this periodically throughout this presentation. s84 00:05:13.600 --> 00:05:16.639 But I want to help explain what's going on behind the scenes. s85 00:05:16.639 --> 00:05:20.639 So behind the scenes I have set up a weights and biases launch queue. s86 00:05:20.639 --> 00:05:31.280 Launch is our our product that allows you to connect your compute clusters and allows humans and agents to launch longunning experimentation jobs particularly by leveraging GPUs. s87 00:05:31.280 --> 00:05:38.639 Here I'm looking at a uh a terminal output of my Kubernetes cluster where we're actually seeing live execution of experiments happening. s88 00:05:38.639 --> 00:05:40.320 So this is happening live right here. s89 00:05:40.320 --> 00:05:42.479 This is not a fake demo. s90 00:05:42.479 --> 00:05:44.000 Um great. s91 00:05:44.000 --> 00:05:51.360 And if we jump back, we see that at this point it started the cues and now it is simply polling and waiting for our work to be complete. s92 00:05:51.360 --> 00:05:53.759 So we'll jump back to that in a in a moment. s93 00:05:53.759 --> 00:05:56.800 But before but let's dive into a few other examples. s94 00:05:56.800 --> 00:06:05.440 So uh something else that is interesting you can do is maybe you might want to ask it something like please summarize the highest performing runs in this project. s95 00:06:05.440 --> 00:06:11.919 This use case would be something like maybe a new user come or a new uh team member is joining your project and want to understand the research. s96 00:06:11.919 --> 00:06:16.880 Um or maybe you've uh someone's been doing some work while you were on PTO and you want to get caught up. s97 00:06:16.880 --> 00:06:19.120 We'll see what this comes up with in a moment. s98 00:06:19.120 --> 00:06:23.120 Some other pre-anned uh examples are finding patterns in your project. s99 00:06:23.120 --> 00:06:28.000 So here we can see that I asked it, hey, can you find some patterns in this research? s100 00:06:28.000 --> 00:06:35.520 And we see that um it identified that a new family of models emerged as the as the auto uh auto research was happening. s101 00:06:35.520 --> 00:06:41.360 Uh it identified that batch size seems to be a really high high uh lever uh parameter. s102 00:06:41.360 --> 00:06:50.240 It identified an architectural recipe that seemed to be quite promising and a number of other insights that would have taken me hours or days to discover on my own. s103 00:06:50.240 --> 00:06:55.360 And Arya is able to do it right for me directly in the interface that I already live. s104 00:06:55.360 --> 00:07:03.919 Not only is it able to emit text based uh textbased outputs, but it also deeply integrates with a number of weights and biases visualization utilities. s105 00:07:03.919 --> 00:07:11.440 So here I've actually asked it to emit a weights and biases report which for those who aren't familiar is essentially a markdown file on steroids. s106 00:07:11.440 --> 00:07:15.360 It's got uh embedded embedded plots, charts and and and graphics. s107 00:07:15.360 --> 00:07:19.120 And so here uh you know it's talked about the thesis of the project. s108 00:07:19.120 --> 00:07:22.400 It's it's emitted a number of of data panels. s109 00:07:22.400 --> 00:07:24.479 And uh I actually think it's quite interesting. s110 00:07:24.479 --> 00:07:35.919 It used um one of our more esoteric panels, the uh parameter importance chart to uh tell me the correlation of various different parameters within this uh within this training job. s111 00:07:35.919 --> 00:07:40.560 Uh in addition to uh reports, it's also great at working with workspaces. s112 00:07:40.560 --> 00:07:47.360 So if you're a weights and biases user, uh you spend a lot of your time uh designing and working with workspaces. s113 00:07:47.360 --> 00:08:02.080 Well, Arya is actually customtuned and prompted to really understand how to build workspaces, build plots, and complement that that data analytics with real live graphics using the built-in proprietary charts that weights and biases users know and love. s114 00:08:02.080 --> 00:08:05.280 Um, so with that, let's go ahead and check back on some of our our prompts. s115 00:08:05.280 --> 00:08:09.039 We can see that the please summarize this project prompt is cooking away. s116 00:08:09.039 --> 00:08:10.479 It's querying weights and biases. s117 00:08:10.479 --> 00:08:11.520 It's applying patches. s118 00:08:11.520 --> 00:08:12.560 It's writing its own code. s119 00:08:12.560 --> 00:08:14.800 So, we'll come back and check on that in a moment. s120 00:08:14.800 --> 00:08:18.879 and our longunning training job is uh still pulling for the results. s121 00:08:18.879 --> 00:08:21.759 We can see that we're cooking away on our GPUs. s122 00:08:21.759 --> 00:08:23.045 So, we're we're frying some GPUs s123 00:08:23.045 --> 00:08:23.440 [music] s124 00:08:23.440 --> 00:08:25.440 and doing some data science all live. s125 00:08:25.440 --> 00:08:28.160 And while that's cooking, let's go ahead and jump back to the presentation. s126 00:08:28.160 --> 00:08:30.560 We'll come back in a moment. s127 00:08:30.560 --> 00:08:35.440 Uh oh, no, we're not looking at a dictionary. s128 00:08:35.440 --> 00:08:36.719 We're looking at a PO. s129 00:08:36.719 --> 00:08:37.200 Great. s130 00:08:37.200 --> 00:08:38.959 Uh okay, so quick recap here. s131 00:08:38.959 --> 00:08:39.919 What did Arya show? s132 00:08:39.919 --> 00:08:41.760 What did we show in these last five minutes? s133 00:08:41.760 --> 00:08:54.640 First, we show that uh Arya can serve as your data science companion right inside of Weights and Biases, helping you discover insights that you wouldn't you wouldn't be able to discover as your experiments and as your team size grows. s134 00:08:55.040 --> 00:08:59.440 Next, we address the problem of complicated reporting and complicated plotting. s135 00:08:59.440 --> 00:09:04.640 Weights and biases users are are really want to turn their insights into visual communication tools. s136 00:09:04.640 --> 00:09:07.440 They want to communicate with their peers and their colleagues. s137 00:09:07.440 --> 00:09:21.279 So Arya's built from the ground up to understand those primitives and help co-pilot and drive right along right alongside in the UI and announcing now today for the first time we are releasing Arya on our iOS device or on our iOS app. s138 00:09:21.279 --> 00:09:27.360 So uh uh Arya released on Monday and our iOS app now has Arya built in. s139 00:09:27.360 --> 00:09:41.440 So if you're conducting hyperparameter tuning jobs, if you're training models, or if you're just researching within the weights and biases ecosystem, you can go touched grass at Yerba Buena uh gardens and steer your uh hyperparameter tuning jobs all from your mobile device. s140 00:09:41.440 --> 00:09:43.040 And what is this all building up to? s141 00:09:43.040 --> 00:09:51.440 This is building up to an a fully automated endto-end research platform where we're not seeking to replace uh RL researchers, but complement your workflows. s142 00:09:51.440 --> 00:10:04.000 Arya's great at orchestrating jobs, understanding GPU workloads, responding to events within the within the Wandi ecosystem, and listening to researchers, uh, uh, looking up archive papers, and collaborating on hypothesis. s143 00:10:04.000 --> 00:10:12.720 So, we can let Arya drive the mechanics that you don't want to deal with while you focus on the new ideas, new architectures, and new parameters that you wanted to try. s144 00:10:12.720 --> 00:10:13.920 Um, great. s145 00:10:13.920 --> 00:10:15.360 So, that's Arya in a nutshell. s146 00:10:15.360 --> 00:10:17.120 We're really hoping that you give it a shot. s147 00:10:17.120 --> 00:10:22.000 And uh we'll jump back to the auto research at the end and see if we got a new best record. s148 00:10:22.000 --> 00:10:27.200 But before we do that, let's talk about how we use weights and biases and coreweave to actually build Arya. s149 00:10:27.200 --> 00:10:33.680 So now speaking to a lot of the the AI agent builders in the room, here's a quick architecture on the lefth hand side. s150 00:10:33.680 --> 00:10:43.040 You see that we have a web client, iOS client that communicates with our API server that then dumps data into our turn database and is worked on by our harness, our our worker harness. s151 00:10:43.040 --> 00:10:49.279 This is sort of archetypical of probably what most of you are all building in the room and is exactly what we have on our back end. s152 00:10:49.279 --> 00:10:54.640 But that harness worker is a magic is a is a magic box and it connects to a number of important utilities. s153 00:10:54.640 --> 00:11:06.320 First is a sandbox where it can execute arbitrary shell calls uh do do Python data science etc. And we invite you to try coreweave weights and biases sandbox to fit into your architecture. s154 00:11:06.320 --> 00:11:13.040 Next up you need an LLM provider of course and so if you're maybe using GLM 5.2 two or one of your fine-tuned models. s155 00:11:13.040 --> 00:11:18.640 We invite you to use uh weights and biases inference and connect that to your worker as well. s156 00:11:18.640 --> 00:11:26.800 If you're like us, you need to run longunning workloads outside of the main loop of the agent where you're actually training for day for sometimes days at a time. s157 00:11:26.800 --> 00:11:33.680 Weights and biases launch can actually help facilitate that and coreweave GPUs can help make that compute even better. s158 00:11:33.680 --> 00:11:37.040 And then lastly, and really most importantly, we need an observability layer. s159 00:11:37.040 --> 00:11:45.680 It's critical that your agents are able to log out their what's going on with their sessions, their turns, their tool calls, any errors they're hap that that's happening, etc. s160 00:11:45.680 --> 00:11:52.079 Uh we have a product called Weights and Biases Weave that we log 100% of our traces to where us and our team can learn from. s161 00:11:52.079 --> 00:12:07.120 And that's where we move from production to offline where our team is able to use Weights and Biases Weave to drive insights and identify behaviors, implement tasks with tasks which are essentially unit tests for your models and evaluate those models in a loop. s162 00:12:07.120 --> 00:12:22.320 We have a model repository which you might choose to use weights and biases artifacts to store your agents or models and you we emit our evaluation results to weave where we have a common dashboard that we can make go no-go decisions on various prompt changes or architectural changes s163 00:12:22.320 --> 00:12:29.839 that then feeds into a research loop which we call our improvement loop where we form hypotheses implement candidate agents and analyze the evals. s164 00:12:29.839 --> 00:12:44.800 So we have two sort of complimentary yet adversarial research loops going on going on offline feeding data from weights and biases weave ultimately to identify the best model so that we can promote that to production through our registry and close the data flywheel. s165 00:12:44.800 --> 00:12:59.440 So in the next just uh three seven minutes or so we'll just talk about uh weights and biases weave and show how we as a team actually use weave to facilitate this workflow and we believe this is something that you would benefit from as well all of you agent builders in the room. s166 00:12:59.440 --> 00:13:00.480 Yes, another demo. s167 00:13:00.480 --> 00:13:02.480 Great. s168 00:13:03.040 --> 00:13:04.240 Okay. s169 00:13:04.240 --> 00:13:05.600 Okay, we have new responses. s170 00:13:05.600 --> 00:13:07.680 So, it's going to be exciting when we open this up later. s171 00:13:07.680 --> 00:13:10.639 See if uh we've got some better metrics. s172 00:13:10.639 --> 00:13:11.279 Um, okay. s173 00:13:11.279 --> 00:13:13.040 Let me zoom out just a little bit here. s174 00:13:13.040 --> 00:13:15.279 So, here I'm looking at the agent dashboard. s175 00:13:15.279 --> 00:13:21.360 This is the live weights and biases agent or Arya agent dashboard uh built in weave. s176 00:13:21.360 --> 00:13:24.320 Man, that is a lot of uh branded buzzwords there. s177 00:13:24.320 --> 00:13:26.880 This is the dashboard that you would get if you use our tool. s178 00:13:26.880 --> 00:13:36.480 and uh you have a you know uh span volume, conversation volume, token tracking, etc. Think of this as like a uh a bird's eye view of your agent. s179 00:13:36.480 --> 00:13:41.519 For me, however, I really like this conversations view, which I do have pre-loaded in this tab. s180 00:13:41.519 --> 00:13:48.800 This conversations view is a live feed of all of the conversations that are going through Arya, but it's filtered down to just the internal employees. s181 00:13:48.800 --> 00:13:51.760 So, it's a little bit of a of a reduced set here. s182 00:13:51.760 --> 00:13:58.720 Um what I what I love is this middle spans view which gives me a visual indicator of the topology of a trace. s183 00:13:58.720 --> 00:14:03.279 Different colors and and shapes indicate different things that are happening within the agent. s184 00:14:03.279 --> 00:14:11.920 So things like tool calls, LLM calls, thinking blocks, etc. which really help me understand again the shape and topology of that particular conversation. s185 00:14:11.920 --> 00:14:22.320 I can of course open up one of these conversations and view our our conversation view where I can see the system prompt, the user message, shell calls, reasoning blocks, etc. s186 00:14:22.320 --> 00:14:35.360 This is where my research lead, myself and my PM go to add notes, add feedback, add emojis, and talk about and discover those insights and those behavioral nuances we spoke about earlier so that we can turn them into tasks. s187 00:14:35.360 --> 00:14:38.160 Arya's built in to the weights and biases system as well. s188 00:14:38.160 --> 00:14:43.040 Here you'll see a summarize button and these are sprinkled throughout the weights and biases application. s189 00:14:43.040 --> 00:14:48.880 I simply click summarize and we start a new chat contextualized to the thing that I'm looking at. s190 00:14:48.880 --> 00:14:53.839 So it it sees this and says give me a brief summary of this particular conversation. s191 00:14:53.839 --> 00:15:03.600 So if you if you're paying a attention closely, you'll realize that what we're doing is using Arya to analyze Arya's own conversations to then make recommendations about how to improve Arya s192 00:15:03.600 --> 00:15:06.079 all within the UI. s193 00:15:06.079 --> 00:15:07.279 Um okay, great. s194 00:15:07.279 --> 00:15:13.279 While that's cooking away, I want to show you the last item uh within the Weave ecosystem here, and that's signals. s195 00:15:13.279 --> 00:15:17.519 We've heard a lot today about the value of evals and the value of LLM judges. s196 00:15:17.519 --> 00:15:20.959 Weave actually offers an integrated LLM judge experience. s197 00:15:20.959 --> 00:15:33.199 So here, if I zoom out a little bit, you'll see that I have a user frustration signal, a lowquality response signal, ask user signal, etc. These are LLM judges that run live against against our live traffic. s198 00:15:33.199 --> 00:15:38.160 And we can see various different signals like user frustration moments or lowquality responses. s199 00:15:38.160 --> 00:15:43.600 These help our team identify these clusters of behavior for us to go fix in next week's iteration. s200 00:15:43.600 --> 00:15:46.160 Let's go ahead and do a live look and see what it says. s201 00:15:46.160 --> 00:15:51.040 Um this says the user explicitly states that I'm not satisfied with the loss curve. s202 00:15:51.040 --> 00:15:54.560 It looks bad and it apparently that indicates frustration. s203 00:15:54.560 --> 00:16:01.920 So here we can see an LLM judges live reasoning for why that particular flag was uh indicated. s204 00:16:02.079 --> 00:16:03.519 Uh let's see, four minutes left. s205 00:16:03.519 --> 00:16:04.000 Perfect. s206 00:16:04.000 --> 00:16:07.839 Um so, uh with that, I've been using the term task a lot. s207 00:16:07.839 --> 00:16:14.399 And so what we're do, what I've showed so far is is this live production loop where we are are are are tracing our our prod logs. s208 00:16:14.399 --> 00:16:18.720 We're looking at them as humans, maybe even using LLMs to complement that analysis. s209 00:16:18.720 --> 00:16:21.839 And what we end up doing is transforming those into tasks. s210 00:16:21.839 --> 00:16:26.160 Now, this gets a bit technical here, but our tasks are all described as YAML files. s211 00:16:26.160 --> 00:16:29.600 You can think of a task as essentially a unit test for your model. s212 00:16:29.600 --> 00:16:35.199 So here we say we have a an example user prompt that says check this run and that run. s213 00:16:35.199 --> 00:16:36.880 Both of these are giving good results. s214 00:16:36.880 --> 00:16:37.920 What can we learn from this? s215 00:16:37.920 --> 00:16:38.880 What's the difference? s216 00:16:38.880 --> 00:16:43.759 So this is an example of something we want Arya to be good at for all of you. s217 00:16:43.759 --> 00:16:48.639 And after the uh requisite metadata we see that we've defined an LLM judge. s218 00:16:48.639 --> 00:16:53.839 So here we've defined what correctness means in the context of that question. s219 00:16:53.839 --> 00:17:00.501 And we've then we've defined a second LLM judge that determines if the insights are actually interesting. s220 00:17:00.501 --> 00:17:00.959 [laughter] s221 00:17:00.959 --> 00:17:12.240 And then we've uh defined a third rule-based judge that says were you able to actually generate a result within just six tool calls meaning it got there with some degree of expediency. s222 00:17:12.240 --> 00:17:15.919 These are all then clustered together into we have about like 200 of these. s223 00:17:15.919 --> 00:17:19.360 They're all clustered together into an eval suite that runs nightly. s224 00:17:19.360 --> 00:17:22.319 And again we use weave to track all those evals. s225 00:17:22.319 --> 00:17:28.000 So here, I know it's a bit small on this screen, but what you're looking at is a listing of every night's eval. s226 00:17:28.000 --> 00:17:40.720 This is literally two nights ago, the evaluation for our candidate model got 73% on our production or on our eval suite against the 72% that our prod model got, which means we're definitely going to push that forward uh this Friday. s227 00:17:40.720 --> 00:17:44.000 Uh and we can see a kind of a a performance plot on the right. s228 00:17:44.000 --> 00:17:49.679 So these utilities are what you would get out of the box if you're uh if you decide to pick up weave and use this tool. s229 00:17:49.679 --> 00:18:00.000 Um, jumping back to the last conversation we had where it asked me where we asked, uh, can you please give a quick summary of this trace, we see that it actually analyzed the conversation, s230 00:18:00.000 --> 00:18:06.880 understood what the user was doing, and then ultimately decided that this was a pretty strong trace. s231 00:18:06.880 --> 00:18:10.799 Um, let's see, we've got two and a half minutes left, so let's just quickly recap here. s232 00:18:10.799 --> 00:18:14.720 Uh, first off, uh, what we use weave to do is a, collect production traffic. s233 00:18:14.720 --> 00:18:19.039 Super critical to collect all of your production traffic so you can learn and iterate. s234 00:18:19.039 --> 00:18:22.400 Secondly, we use it to generate insights both as humans as well. s235 00:18:22.400 --> 00:18:23.600 We we do it as humans. s236 00:18:23.600 --> 00:18:28.240 We use Arya and we use LLM judges to identify those behavioral nuances. s237 00:18:28.240 --> 00:18:29.919 We then enrich our tasks. s238 00:18:29.919 --> 00:18:43.039 We implement models and we evaluate using weights and biases weave as a shared dashboard where we can make decisions together as a team that then ultimately allows us to promote the best model forward with confidence. s239 00:18:43.200 --> 00:18:48.480 So speaking of confident productionization, let me speak uh briefly to the managers in the room. s240 00:18:48.480 --> 00:18:51.039 So a few tips for being successful here. s241 00:18:51.039 --> 00:18:54.799 First is um invest in agent-oriented observability. s242 00:18:54.799 --> 00:18:56.080 Uh I'm a bit biased. s243 00:18:56.080 --> 00:19:00.080 I believe that weights and biases weave is the uh observability platform of the future. s244 00:19:00.080 --> 00:19:02.160 Uh but pick your favorite flavor. s245 00:19:02.160 --> 00:19:05.760 Whatever it is, log your sessions, log your turns, log your tools and feedback. s246 00:19:05.760 --> 00:19:10.640 This introduces an ability to catch a new class of bugs in our world called behavioral bugs. s247 00:19:10.640 --> 00:19:14.480 Not exceptions, not performance, but behavioral bugs. s248 00:19:14.480 --> 00:19:17.600 Next up, tasks and evals are the new world of CI. s249 00:19:17.600 --> 00:19:18.640 You've heard a lot about this. s250 00:19:18.640 --> 00:19:21.679 If you are a software engineer, you've written unit tests your whole life. s251 00:19:21.679 --> 00:19:31.200 You must develop a practice where your researchers are sitting on the same scrum team as you developing tasks and you're viewing the performance metrics as true go no-go decisions. s252 00:19:31.200 --> 00:19:35.200 But in order to complement that, you must use humans as a necessary judge. s253 00:19:35.200 --> 00:19:38.240 There are behavioral nuances that LLM will not catch. s254 00:19:38.240 --> 00:19:49.440 You must be using your product and you must be manually reviewing these traces as a team at the end of the week on a board looking at the best and worst traces to understand how your model is performing. s255 00:19:49.440 --> 00:19:54.720 And then lastly, um just maybe one one more tip is to add value through context and tools. s256 00:19:54.720 --> 00:20:00.799 It can be really tempting to uh try to overengineer the harness and do a bunch of creative stuff around memory and things like this. s257 00:20:00.799 --> 00:20:12.080 We found that a a lot of lowhanging fruit can be ascertained through simply giving your agent context about your business domain, the underlying uh primitives that you have available and your particular uh business data. s258 00:20:12.080 --> 00:20:20.960 Um so with that, let's go ahead and check in on our uh our our research agent here and let's go ahead and toggle our workspace. s259 00:20:20.960 --> 00:20:30.159 And what we should be seeing is yes indeed a little dot that uh oh okay our previous dot which was done at lunch was 5.83. s260 00:20:30.159 --> 00:20:31.120 831. s261 00:20:31.120 --> 00:20:32.960 This got 5.833. s262 00:20:32.960 --> 00:20:37.120 So we were right on the edge of having a live improvement, but pretty darn close. s263 00:20:37.120 --> 00:20:40.080 Uh so that's what the uh that's what the model was able to produce. s264 00:20:40.080 --> 00:20:43.360 It actually uh ran uh quite a few tests here. s265 00:20:43.360 --> 00:20:45.919 I see I'm over time, so I will click close pretty soon. s266 00:20:45.919 --> 00:20:50.799 But we ran 12 different experiments within that experiment batch and uh we'll be running more all night. s267 00:20:50.799 --> 00:20:54.240 So please try out Arya, scan the QR codes, check out the docs. s268 00:20:54.240 --> 00:20:58.400 Uh we really love to see what you do with it and um looking forward to serving you. s269 00:20:58.400 --> 00:21:00.880 Thank you very much. s270 00:21:13.334 --> 00:21:15.334 [music]