An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases

AI Engineer · 21 min · 270 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:13Okay.
  3. 00:14Hello everyone and thank you for attending this session.
  4. 00:17My name is Tim Sweeney, a principal engineer at Weights and Biases and Coreweave.
  5. 00:21And for the next 20 minutes, we're going to talk about Arya, our new AI research and iteration agent.
  6. 00:27Let's go ahead and get started.
  7. 00:29So, uh, first off, just by way of making some noise, some clapping, uh, who here, um, identifies as an ML researcher?
  8. 00:37You're someone that trains models, trains the brain?
  9. 00:41I heard one.
  10. 00:42Wow.
  11. 00:42Okay.
  12. 00:43Great work.
  13. 00:43Great work.
  14. 00:44Uh, what about who here is the applied engineer, the namesake of this conference?
  15. 00:48Who here actually builds the bots?
  16. 00:50Okay, good.
  17. 00:51Expected much more.
  18. 00:53And who here is in AI management?
  19. 00:54You are helping fund this compute.
  20. 00:58Okay.
  21. 00:58Okay.
  22. 00:59Nice.
  23. 00:59From the back.
  24. 01:00Lovely.
  25. 01:01Um, well, now that I know a little bit about you, just a little bit about me.
  26. 01:04Uh, again, my name is Tim.
  27. 01:05I have a masters in machine learning, uh, and reinforcement learning from Georgia Tech.
  28. 01:09So, I've been that, uh, researcher currently building Weights and Biases agent, Arya.
  29. 01:14So, identify as that applied engineer.
  30. 01:16And in a previous life was the PM of Twitter's ML stack.
  31. 01:20So, I hope you hopefully can connect with you middle management as well.
  32. 01:23[laughter]
  33. 01:24[gasps]
  34. 01:24Um today's agenda is kind of broken into three sections and hopefully each of you personas walk away with something valuable.
  35. 01:31So first we're going to learn about Arya itself and how it can supercharge your AI and ML workflows.
  36. 01:37We're going to dive into auto research and see that live in a live demo in just a moment.
  37. 01:42Then we're going to pull back the curtain and learn how we use weights and biases and uh coreweave to actually build Arya because a lot of you in the audience are building agents yourself and we believe a lot of these components can help you in your endeavors.
  38. 01:54And then towards the end we'll just take a step back and identify a few key tips and tricks for making sure that you're able to productionize your systems effectively.
  39. 02:02For those of you who might not be familiar, Weights and Biases is the world's leading AI development platform.
  40. 02:07We've been in business now for nine years and have happily joined the core family about a year ago.
  41. 02:13Uh we have a number of products in our suite but are really known for our models training inference and weave stack which really helps collect data uh about the AI development and machine learning workflows and makes that information actionable
  42. 02:25and uh enables users to make the best decisions about what to do next.
  43. 02:30So without further ado, let's go ahead and dive into Arya, our agent.
  44. 02:33Uh we'll show a demo and then we'll get back to some slides.
  45. 02:40Okay, beautiful.
  46. 02:42Let's make this a bit bigger.
  47. 02:43Holler at me if you need it to be bigger.
  48. 02:45So, uh, what you're looking at here is a weights and biases workspace.
  49. 02:49For you, for anybody that isn't familiar, on the lefth hand side, I actually see a list of a bunch of different experiments.
  50. 02:55In this particular project, I have over 200 training jobs.
  51. 02:59And on the right hand side, I see a scatter plot of, in this case, declining metrics, which is good.
  52. 03:04means our loss is going down over time.
  53. 03:06And this view would be very familiar for anyone that uses our tool.
  54. 03:10Now, to ground this, we're actually uh uh using the Carpathy Auto Research project, which I'm sure many of you are familiar with, but if you're not, it's just a very simple project that trains an LLM,
  55. 03:20and it's a great foundation for auto research type demonstrations because it's a very simple codebase and allows us to improve iteratively over time.
  56. 03:30So, let's jump back to the project and open up Arya by clicking this blue button in the upper right.
  57. 03:34When I click this button, I'm uh presented with the familiar chat interface with, you know, how can I help you today, a few call to actions, and you know, I can add different context in my project or maybe add images,
  58. 03:46etc. Um, everyone here is agent builders, so I don't need to bore you with the details of what an agent interface looks like.
  59. 03:52But let's go ahead and just, you know, enter in a basic intro here.
  60. 03:56Let's say, "Hello, Arya.
  61. 03:58You're on stage at AI World's Fair 2026.
  62. 04:00Please introduce yourself."
  63. 04:02So, it's going to go ahead and chug along and hopefully emit some sort of nice emoji.
  64. 04:06Yay.
  65. 04:06He I'm Arya.
  66. 04:07I'm talking to the audience.
  67. 04:09Great.
  68. 04:10But now, let's dive into the meat of why you came here.
  69. 04:13So, I'm going to open up this chat here.
  70. 04:15And this is a longunning chat where I've been running again over 200 experiments using the auto research loop.
  71. 04:21Um, it helped me download the code, set up my launch jobs, set up my GPUs, and is able to autonomously iterate on the code itself and the hyperparameters.
  72. 04:30We'll take a look at what it's doing in a moment, but while we're doing this, I'm going to kick off a live iteration right here.
  73. 04:36So, what I'm going to say is please conduct another batch of experiments.
  74. 04:40You are on stage at the AI Engineer Worlds Fair 2026 and we're hoping to find the best model live.
  75. 04:46I believe in you.
  76. 04:47Uh, because we know we have to encourage our models.
  77. 04:49Um, so it's been doing this for a while.
  78. 04:51What it what it's doing here is it's saying, "Okay, great.
  79. 04:54Um, I don't want to make a big architecture swing.
  80. 04:56That feels a little bit too risky."
  81. 04:58So, it's probably going to go for uh some modifications to the hyperparameters.
  82. 05:02And then it's kicking off a shell call here that is actually um executing that uh executing that experimentation loop.
  83. 05:09And we're going to check in on this periodically throughout this presentation.
  84. 05:13But I want to help explain what's going on behind the scenes.
  85. 05:16So behind the scenes I have set up a weights and biases launch queue.
  86. 05:20Launch is our our product that allows you to connect your compute clusters and allows humans and agents to launch longunning experimentation jobs particularly by leveraging GPUs.
  87. 05:31Here I'm looking at a uh a terminal output of my Kubernetes cluster where we're actually seeing live execution of experiments happening.
  88. 05:38So this is happening live right here.
  89. 05:40This is not a fake demo.
  90. 05:42Um great.
  91. 05:44And if we jump back, we see that at this point it started the cues and now it is simply polling and waiting for our work to be complete.
  92. 05:51So we'll jump back to that in a in a moment.
  93. 05:53But before but let's dive into a few other examples.
  94. 05:56So uh something else that is interesting you can do is maybe you might want to ask it something like please summarize the highest performing runs in this project.
  95. 06:05This use case would be something like maybe a new user come or a new uh team member is joining your project and want to understand the research.
  96. 06:11Um or maybe you've uh someone's been doing some work while you were on PTO and you want to get caught up.
  97. 06:16We'll see what this comes up with in a moment.
  98. 06:19Some other pre-anned uh examples are finding patterns in your project.
  99. 06:23So here we can see that I asked it, hey, can you find some patterns in this research?
  100. 06:28And we see that um it identified that a new family of models emerged as the as the auto uh auto research was happening.
  101. 06:35Uh it identified that batch size seems to be a really high high uh lever uh parameter.
  102. 06:41It identified an architectural recipe that seemed to be quite promising and a number of other insights that would have taken me hours or days to discover on my own.
  103. 06:50And Arya is able to do it right for me directly in the interface that I already live.
  104. 06:55Not only is it able to emit text based uh textbased outputs, but it also deeply integrates with a number of weights and biases visualization utilities.
  105. 07:03So here I've actually asked it to emit a weights and biases report which for those who aren't familiar is essentially a markdown file on steroids.
  106. 07:11It's got uh embedded embedded plots, charts and and and graphics.
  107. 07:15And so here uh you know it's talked about the thesis of the project.
  108. 07:19It's it's emitted a number of of data panels.
  109. 07:22And uh I actually think it's quite interesting.
  110. 07:24It used um one of our more esoteric panels, the uh parameter importance chart to uh tell me the correlation of various different parameters within this uh within this training job.
  111. 07:35Uh in addition to uh reports, it's also great at working with workspaces.
  112. 07:40So if you're a weights and biases user, uh you spend a lot of your time uh designing and working with workspaces.
  113. 07:47Well, Arya is actually customtuned and prompted to really understand how to build workspaces, build plots, and complement that that data analytics with real live graphics using the built-in proprietary charts that weights and biases users know and love.
  114. 08:02Um, so with that, let's go ahead and check back on some of our our prompts.
  115. 08:05We can see that the please summarize this project prompt is cooking away.
  116. 08:09It's querying weights and biases.
  117. 08:10It's applying patches.
  118. 08:11It's writing its own code.
  119. 08:12So, we'll come back and check on that in a moment.
  120. 08:14and our longunning training job is uh still pulling for the results.
  121. 08:18We can see that we're cooking away on our GPUs.
  122. 08:21So, we're we're frying some GPUs
  123. 08:23[music]
  124. 08:23and doing some data science all live.
  125. 08:25And while that's cooking, let's go ahead and jump back to the presentation.
  126. 08:28We'll come back in a moment.
  127. 08:30Uh oh, no, we're not looking at a dictionary.
  128. 08:35We're looking at a PO.
  129. 08:36Great.
  130. 08:37Uh okay, so quick recap here.
  131. 08:38What did Arya show?
  132. 08:39What did we show in these last five minutes?
  133. 08:41First, we show that uh Arya can serve as your data science companion right inside of Weights and Biases, helping you discover insights that you wouldn't you wouldn't be able to discover as your experiments and as your team size grows.
  134. 08:55Next, we address the problem of complicated reporting and complicated plotting.
  135. 08:59Weights and biases users are are really want to turn their insights into visual communication tools.
  136. 09:04They want to communicate with their peers and their colleagues.
  137. 09:07So Arya's built from the ground up to understand those primitives and help co-pilot and drive right along right alongside in the UI and announcing now today for the first time we are releasing Arya on our iOS device or on our iOS app.
  138. 09:21So uh uh Arya released on Monday and our iOS app now has Arya built in.
  139. 09:27So if you're conducting hyperparameter tuning jobs, if you're training models, or if you're just researching within the weights and biases ecosystem, you can go touched grass at Yerba Buena uh gardens and steer your uh hyperparameter tuning jobs all from your mobile device.
  140. 09:41And what is this all building up to?
  141. 09:43This is building up to an a fully automated endto-end research platform where we're not seeking to replace uh RL researchers, but complement your workflows.
  142. 09:51Arya's great at orchestrating jobs, understanding GPU workloads, responding to events within the within the Wandi ecosystem, and listening to researchers, uh, uh, looking up archive papers, and collaborating on hypothesis.
  143. 10:04So, we can let Arya drive the mechanics that you don't want to deal with while you focus on the new ideas, new architectures, and new parameters that you wanted to try.
  144. 10:12Um, great.
  145. 10:13So, that's Arya in a nutshell.
  146. 10:15We're really hoping that you give it a shot.
  147. 10:17And uh we'll jump back to the auto research at the end and see if we got a new best record.
  148. 10:22But before we do that, let's talk about how we use weights and biases and coreweave to actually build Arya.
  149. 10:27So now speaking to a lot of the the AI agent builders in the room, here's a quick architecture on the lefth hand side.
  150. 10:33You see that we have a web client, iOS client that communicates with our API server that then dumps data into our turn database and is worked on by our harness, our our worker harness.
  151. 10:43This is sort of archetypical of probably what most of you are all building in the room and is exactly what we have on our back end.
  152. 10:49But that harness worker is a magic is a is a magic box and it connects to a number of important utilities.
  153. 10:54First is a sandbox where it can execute arbitrary shell calls uh do do Python data science etc. And we invite you to try coreweave weights and biases sandbox to fit into your architecture.
  154. 11:06Next up you need an LLM provider of course and so if you're maybe using GLM 5.2 two or one of your fine-tuned models.
  155. 11:13We invite you to use uh weights and biases inference and connect that to your worker as well.
  156. 11:18If you're like us, you need to run longunning workloads outside of the main loop of the agent where you're actually training for day for sometimes days at a time.
  157. 11:26Weights and biases launch can actually help facilitate that and coreweave GPUs can help make that compute even better.
  158. 11:33And then lastly, and really most importantly, we need an observability layer.
  159. 11:37It's critical that your agents are able to log out their what's going on with their sessions, their turns, their tool calls, any errors they're hap that that's happening, etc.
  160. 11:45Uh we have a product called Weights and Biases Weave that we log 100% of our traces to where us and our team can learn from.
  161. 11:52And that's where we move from production to offline where our team is able to use Weights and Biases Weave to drive insights and identify behaviors, implement tasks with tasks which are essentially unit tests for your models and evaluate those models in a loop.
  162. 12:07We have a model repository which you might choose to use weights and biases artifacts to store your agents or models and you we emit our evaluation results to weave where we have a common dashboard that we can make go no-go decisions on various prompt changes or architectural changes
  163. 12:22that then feeds into a research loop which we call our improvement loop where we form hypotheses implement candidate agents and analyze the evals.
  164. 12:29So we have two sort of complimentary yet adversarial research loops going on going on offline feeding data from weights and biases weave ultimately to identify the best model so that we can promote that to production through our registry and close the data flywheel.
  165. 12:44So in the next just uh three seven minutes or so we'll just talk about uh weights and biases weave and show how we as a team actually use weave to facilitate this workflow and we believe this is something that you would benefit from as well all of you agent builders in the room.
  166. 12:59Yes, another demo.
  167. 13:00Great.
  168. 13:03Okay.
  169. 13:04Okay, we have new responses.
  170. 13:05So, it's going to be exciting when we open this up later.
  171. 13:07See if uh we've got some better metrics.
  172. 13:10Um, okay.
  173. 13:11Let me zoom out just a little bit here.
  174. 13:13So, here I'm looking at the agent dashboard.
  175. 13:15This is the live weights and biases agent or Arya agent dashboard uh built in weave.
  176. 13:21Man, that is a lot of uh branded buzzwords there.
  177. 13:24This is the dashboard that you would get if you use our tool.
  178. 13:26and uh you have a you know uh span volume, conversation volume, token tracking, etc. Think of this as like a uh a bird's eye view of your agent.
  179. 13:36For me, however, I really like this conversations view, which I do have pre-loaded in this tab.
  180. 13:41This conversations view is a live feed of all of the conversations that are going through Arya, but it's filtered down to just the internal employees.
  181. 13:48So, it's a little bit of a of a reduced set here.
  182. 13:51Um what I what I love is this middle spans view which gives me a visual indicator of the topology of a trace.
  183. 13:58Different colors and and shapes indicate different things that are happening within the agent.
  184. 14:03So things like tool calls, LLM calls, thinking blocks, etc. which really help me understand again the shape and topology of that particular conversation.
  185. 14:11I can of course open up one of these conversations and view our our conversation view where I can see the system prompt, the user message, shell calls, reasoning blocks, etc.
  186. 14:22This is where my research lead, myself and my PM go to add notes, add feedback, add emojis, and talk about and discover those insights and those behavioral nuances we spoke about earlier so that we can turn them into tasks.
  187. 14:35Arya's built in to the weights and biases system as well.
  188. 14:38Here you'll see a summarize button and these are sprinkled throughout the weights and biases application.
  189. 14:43I simply click summarize and we start a new chat contextualized to the thing that I'm looking at.
  190. 14:48So it it sees this and says give me a brief summary of this particular conversation.
  191. 14:53So if you if you're paying a attention closely, you'll realize that what we're doing is using Arya to analyze Arya's own conversations to then make recommendations about how to improve Arya
  192. 15:03all within the UI.
  193. 15:06Um okay, great.
  194. 15:07While that's cooking away, I want to show you the last item uh within the Weave ecosystem here, and that's signals.
  195. 15:13We've heard a lot today about the value of evals and the value of LLM judges.
  196. 15:17Weave actually offers an integrated LLM judge experience.
  197. 15:20So here, if I zoom out a little bit, you'll see that I have a user frustration signal, a lowquality response signal, ask user signal, etc. These are LLM judges that run live against against our live traffic.
  198. 15:33And we can see various different signals like user frustration moments or lowquality responses.
  199. 15:38These help our team identify these clusters of behavior for us to go fix in next week's iteration.
  200. 15:43Let's go ahead and do a live look and see what it says.
  201. 15:46Um this says the user explicitly states that I'm not satisfied with the loss curve.
  202. 15:51It looks bad and it apparently that indicates frustration.
  203. 15:54So here we can see an LLM judges live reasoning for why that particular flag was uh indicated.
  204. 16:02Uh let's see, four minutes left.
  205. 16:03Perfect.
  206. 16:04Um so, uh with that, I've been using the term task a lot.
  207. 16:07And so what we're do, what I've showed so far is is this live production loop where we are are are are tracing our our prod logs.
  208. 16:14We're looking at them as humans, maybe even using LLMs to complement that analysis.
  209. 16:18And what we end up doing is transforming those into tasks.
  210. 16:21Now, this gets a bit technical here, but our tasks are all described as YAML files.
  211. 16:26You can think of a task as essentially a unit test for your model.
  212. 16:29So here we say we have a an example user prompt that says check this run and that run.
  213. 16:35Both of these are giving good results.
  214. 16:36What can we learn from this?
  215. 16:37What's the difference?
  216. 16:38So this is an example of something we want Arya to be good at for all of you.
  217. 16:43And after the uh requisite metadata we see that we've defined an LLM judge.
  218. 16:48So here we've defined what correctness means in the context of that question.
  219. 16:53And we've then we've defined a second LLM judge that determines if the insights are actually interesting.
  220. 17:00[laughter]
  221. 17:00And then we've uh defined a third rule-based judge that says were you able to actually generate a result within just six tool calls meaning it got there with some degree of expediency.
  222. 17:12These are all then clustered together into we have about like 200 of these.
  223. 17:15They're all clustered together into an eval suite that runs nightly.
  224. 17:19And again we use weave to track all those evals.
  225. 17:22So here, I know it's a bit small on this screen, but what you're looking at is a listing of every night's eval.
  226. 17:28This is literally two nights ago, the evaluation for our candidate model got 73% on our production or on our eval suite against the 72% that our prod model got, which means we're definitely going to push that forward uh this Friday.
  227. 17:40Uh and we can see a kind of a a performance plot on the right.
  228. 17:44So these utilities are what you would get out of the box if you're uh if you decide to pick up weave and use this tool.
  229. 17:49Um, jumping back to the last conversation we had where it asked me where we asked, uh, can you please give a quick summary of this trace, we see that it actually analyzed the conversation,
  230. 18:00understood what the user was doing, and then ultimately decided that this was a pretty strong trace.
  231. 18:06Um, let's see, we've got two and a half minutes left, so let's just quickly recap here.
  232. 18:10Uh, first off, uh, what we use weave to do is a, collect production traffic.
  233. 18:14Super critical to collect all of your production traffic so you can learn and iterate.
  234. 18:19Secondly, we use it to generate insights both as humans as well.
  235. 18:22We we do it as humans.
  236. 18:23We use Arya and we use LLM judges to identify those behavioral nuances.
  237. 18:28We then enrich our tasks.
  238. 18:29We implement models and we evaluate using weights and biases weave as a shared dashboard where we can make decisions together as a team that then ultimately allows us to promote the best model forward with confidence.
  239. 18:43So speaking of confident productionization, let me speak uh briefly to the managers in the room.
  240. 18:48So a few tips for being successful here.
  241. 18:51First is um invest in agent-oriented observability.
  242. 18:54Uh I'm a bit biased.
  243. 18:56I believe that weights and biases weave is the uh observability platform of the future.
  244. 19:00Uh but pick your favorite flavor.
  245. 19:02Whatever it is, log your sessions, log your turns, log your tools and feedback.
  246. 19:05This introduces an ability to catch a new class of bugs in our world called behavioral bugs.
  247. 19:10Not exceptions, not performance, but behavioral bugs.
  248. 19:14Next up, tasks and evals are the new world of CI.
  249. 19:17You've heard a lot about this.
  250. 19:18If you are a software engineer, you've written unit tests your whole life.
  251. 19:21You must develop a practice where your researchers are sitting on the same scrum team as you developing tasks and you're viewing the performance metrics as true go no-go decisions.
  252. 19:31But in order to complement that, you must use humans as a necessary judge.
  253. 19:35There are behavioral nuances that LLM will not catch.
  254. 19:38You must be using your product and you must be manually reviewing these traces as a team at the end of the week on a board looking at the best and worst traces to understand how your model is performing.
  255. 19:49And then lastly, um just maybe one one more tip is to add value through context and tools.
  256. 19:54It can be really tempting to uh try to overengineer the harness and do a bunch of creative stuff around memory and things like this.
  257. 20:00We found that a a lot of lowhanging fruit can be ascertained through simply giving your agent context about your business domain, the underlying uh primitives that you have available and your particular uh business data.
  258. 20:12Um so with that, let's go ahead and check in on our uh our our research agent here and let's go ahead and toggle our workspace.
  259. 20:20And what we should be seeing is yes indeed a little dot that uh oh okay our previous dot which was done at lunch was 5.83.
  260. 20:30831.
  261. 20:31This got 5.833.
  262. 20:32So we were right on the edge of having a live improvement, but pretty darn close.
  263. 20:37Uh so that's what the uh that's what the model was able to produce.
  264. 20:40It actually uh ran uh quite a few tests here.
  265. 20:43I see I'm over time, so I will click close pretty soon.
  266. 20:45But we ran 12 different experiments within that experiment batch and uh we'll be running more all night.
  267. 20:50So please try out Arya, scan the QR codes, check out the docs.
  268. 20:54Uh we really love to see what you do with it and um looking forward to serving you.
  269. 20:58Thank you very much.
  270. 21:13[music]