WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XyV6bSMyq-I

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XyV6bSMyq-I.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:16.640 --> 00:00:17.920
Okay.

s2
00:00:17.920 --> 00:00:18.560
Hello.

s3
00:00:18.560 --> 00:00:19.920
Thank you for showing up.

s4
00:00:19.920 --> 00:00:21.520
Uh my name is Zuben Isola.

s5
00:00:21.520 --> 00:00:23.199
Uh I work at Weights and Biases.

s6
00:00:23.199 --> 00:00:28.560
I'm building the Arya agent which is something that we released to general availability on Monday.

s7
00:00:28.560 --> 00:00:34.800
You might have seen Tim's talk uh one of the pe my peers at Weights and Biases who I work with uh on the main stage yesterday.

s8
00:00:34.800 --> 00:00:47.520
So I'm going to give you a little bit of a deep dive into how we evaluate and build the Arya agent uh and in particular uh the evaluation framework that we use and how we use Arya to sort of self-reinforce itself in the loop of doing research for itself

s9
00:00:47.520 --> 00:00:49.440
uh and talk about some weights and biases things.

s10
00:00:49.440 --> 00:00:57.520
So yeah, this is a rip from some of the slides that I presented at Nurups, but uh I think there's some interesting things about like using uh agentic harnesses.

s11
00:00:57.520 --> 00:01:05.280
I I imagine most people in the audience or at this conference are all excited to build agent harnesses or maybe use cloud code with skills in a variety of different packages,

s12
00:01:05.280 --> 00:01:12.159
but there's, you know, pretty strong differences in the way that you package these AI agents if you use them with different tool calls, etc. on a variety of different benchmarks.

s13
00:01:12.159 --> 00:01:18.880
And so inspired by that, we're doing some work at Weights and Biases to build our own agent harness to do research for you in the weights and biases platform.

s14
00:01:18.880 --> 00:01:29.119
Um, you know, and then there's probably some other things that I don't need to convince people of as to like why we want software agents and why a software harness as opposed to like just letting the, you know, LM run free and call tools or whatever else.

s15
00:01:29.119 --> 00:01:31.119
Uh, but that's not for here.

s16
00:01:31.119 --> 00:01:42.000
Uh, but one big problem presents itself, which is basically the thing that I go to bed thinking about every day, which is, you know, benchmarks, evaluations, the agents, and how you configure them are all covariant.

s17
00:01:42.000 --> 00:01:51.200
And so like if you're trying to apply principled evaluations to any sort of system that dynamically changes, you need to have good measurement to see how the system actually performs.

s18
00:01:51.200 --> 00:01:56.079
And you really want to see how the system performs in both your production environment and your offline environment.

s19
00:01:56.079 --> 00:02:05.200
Uh you know if you're RLP build from a long time ago, you might think about like the sim tore gap uh which is like basically how do we translate simulation environments into the real world environments

s20
00:02:05.200 --> 00:02:18.640
etc. Uh and so there's sort of two things that I'll talk about uh and then give a live demo of which is weights and biases weave for agents which is basically in my opinion at least the best observability platform for both production and offline tracing of agents.

s21
00:02:18.640 --> 00:02:27.599
So I use it predominantly from the offline side where I build simulation environments run the Arya agent over them and then track them in weave and my entire team does that as well.

s22
00:02:27.599 --> 00:02:29.840
So I can see all of our performance in offline metrics.

s23
00:02:29.840 --> 00:02:41.120
And then the other half of my team works on the production side of things, deploying the agent, logging the same things in the exact same format so that I can rip those production traces into our environments and then hill climb on them or resolve our errors.

s24
00:02:41.120 --> 00:02:42.400
And it's a pretty nice flywheel.

s25
00:02:42.400 --> 00:02:53.280
And then obviously the other thing that I'm here to talk about is Arya, the weights and devices agent itself, which is basically the thing that we use to now build itself because it's sophisticated enough that it can actually do that offline hill climbing uh by itself.

s26
00:02:53.280 --> 00:03:00.000
So I'm going to just jump into a super quick live demo and then we'll talk through some other slides uh about how we built the offline evaluation framework.

s27
00:03:00.000 --> 00:03:05.680
So if you hop into the weights and biases platform, this is a demo project that I called Arya researches Arya here.

s28
00:03:05.680 --> 00:03:10.000
Uh and I'm just going to chat to Arya and ask it to basically do auto research for itself.

s29
00:03:10.000 --> 00:03:14.159
And so this is a relatively long prompt that I had it generate in actually one of these other sessions.

s30
00:03:14.159 --> 00:03:22.159
Um but if we look at it, what it's going to do is it's going to take my codebase that is logged as a weights and biases artifact uh which we can see over here.

s31
00:03:22.159 --> 00:03:27.040
It's going to launch training jobs uh against this codebase which is our offline evaluation framework.

s32
00:03:27.040 --> 00:03:32.560
It's going to review production traces that we have, add new tasks to hill climb against uh and do stuff like that.

s33
00:03:32.560 --> 00:03:34.799
And so, you know, this thing's going to cook for a little bit.

s34
00:03:34.799 --> 00:03:36.640
The evaluations take a little bit of time to run.

s35
00:03:36.640 --> 00:03:41.200
And so, we'll see Arya uh basically try to write a new variant for itself.

s36
00:03:41.200 --> 00:03:44.239
The other side of this equation, right, is the production side of things.

s37
00:03:44.239 --> 00:03:49.599
And so, this is a sanitized view of internal customer traces or internal traces on Arya.

s38
00:03:49.599 --> 00:03:56.959
So we'll actually see very quickly uh that Arya will actually pop up here the exact conversation that we're running um right here.

s39
00:03:56.959 --> 00:04:05.680
But in this trace I'm going to use this as an opportunity to take one of these production traces that are logged in weave and put them into my offline evaluation framework so that I can hill climb on them.

s40
00:04:05.680 --> 00:04:06.879
And so I'm just going to do that live.

s41
00:04:06.879 --> 00:04:09.040
We'll ask you know for a relatively simple task.

s42
00:04:09.040 --> 00:04:14.159
Um this is going to be a little interesting because it's going to be slightly self-referential but I'll go here.

s43
00:04:14.159 --> 00:04:26.240
I'll pop up in a new Arya tab and I'll just ask it hey take this trace log it into our offline evals I don't think spelling matters too much and then run the candidate

s44
00:04:26.240 --> 00:04:43.520
and production agent on it and we'll have it cook for that as well and we'll see those evaluations get logged into weave as they're running um so that's basically the tight version of what I do basically every day which is look at production traces in our weights and biases weave platform run evaluations in our codebase which is

s45
00:04:43.520 --> 00:04:49.280
our offline client evaluation sandbox uh and then deploy new versions of the agent uh and work with my team to do that.

s46
00:04:49.280 --> 00:04:59.360
And if we look at our production project, this is, you know, a bunch of traces that we have over the last seven weeks of our nightly uh like CI jobs that run where we evaluate the agent in its production format and other candidate variants that we cut.

s47
00:04:59.360 --> 00:05:01.600
And so we can see our relative performance over time.

s48
00:05:01.600 --> 00:05:11.039
Uh like you know, for a while the CI broke, we had Arya have to fix itself last night, but you know, we're getting about like 66% performance on some of the tasks, etc. And so we have a variety of tasks

s49
00:05:11.039 --> 00:05:12.800
um that we train the agent to do.

s50
00:05:12.800 --> 00:05:28.960
And so now I'm just going to jump into a material version of the presentation to go a little bit deeper into the theory behind building a simulation environment for offline hill climbing for basically building a software agent where traditionally you might expect that you are really robust if you're doing reinforcement learning but I mostly prompt engineer these

s51
00:05:28.960 --> 00:05:37.039
days because the sophisticated models are relatively good at performing tasks at weights and biases and so we're really working on building skills for the software agent more than doing reinforcement learning.

s52
00:05:37.039 --> 00:05:40.400
But I think applying the same methodology to how you build the agent.

s53
00:05:40.400 --> 00:05:44.960
So the same robustness of how you simulate environments uh is really helpful for our work.

s54
00:05:44.960 --> 00:05:50.560
So we benchmark a bite-wise identical version of the agent in our production environment and our simulated environment.

s55
00:05:50.560 --> 00:05:53.199
There's a lot of text on these slides that I and Claude sort of slop out.

s56
00:05:53.199 --> 00:05:55.440
Uh but you know we'll just go for the graphics.

s57
00:05:55.440 --> 00:06:03.840
Um so I think the exciting part here is that both using the logging framework from weave as well as just in the way that we've designed the system our research and production code are exactly the same.

s58
00:06:03.840 --> 00:06:13.600
And in fact, there's like a 4-hour sync job that happens between production to our research environment so that we don't get any drift when you, you know, researchers are cutting new variants of the agent, new skills, etc., and doing their hill climbing.

s59
00:06:13.600 --> 00:06:21.039
And so, we have this nice little tight loop that basically mirror each other on two sides of the stack from our deployment layer and our offline benchmarking layer.

s60
00:06:21.039 --> 00:06:27.520
Uh, and then, you know, we expose internally as a team a bunch of run eval commands to generate score trajectories.

s61
00:06:27.520 --> 00:06:37.360
And so that's what we were talking about with this like large project where we have a bunch of trajectory examples where if I click into one of these, you know, we can see like our relative scores of a variety of different metrics when it loads.

s62
00:06:37.360 --> 00:06:42.720
Um, but essentially what we get is these super robust traces of the agent rolling out over time.

s63
00:06:42.720 --> 00:06:44.400
And there's a lot of signal to reap from this.

s64
00:06:44.400 --> 00:06:52.960
And I think the thesis behind this is generate a ton of traces and then decide what you're going to do with said ton of traces as you try to figure out how the agent performs.

s65
00:06:52.960 --> 00:06:57.600
you're measuring emergent properties and then trying to align the agent in particular directions.

s66
00:06:57.600 --> 00:07:11.520
And so the two ways that that happens is I look manually at the tasks or like we would do in that first example, I would ask Arya to review the roll out that it generated for itself or review some other rollout and decide what went wrong or what went well and try to reinforce that behavior through prompting

s67
00:07:11.520 --> 00:07:13.440
or anything else.

s68
00:07:13.440 --> 00:07:17.919
Uh and then you know the actual method behind the agent is relatively simple.

s69
00:07:17.919 --> 00:07:34.960
We want to test on a variety of the different models both models offered through corewave inference models offered through you know the foundation model players etc. And so we define you know a relatively agnostic software stack for how we treat compaction and how we prepare context and how we assemble UI payloads etc. And the again mental pattern

s70
00:07:34.960 --> 00:07:41.280
behind this is to really try to make it very very simple to have lots of mutations of the exact same configuration.

s71
00:07:41.280 --> 00:07:52.160
You want to basically YAML define different configurations of the agent to sort of get multiple parallel uh variants and then test them all and see what happens because ultimately you can have insight into how to make the problem better.

s72
00:07:52.160 --> 00:07:59.520
But if we take an adage from old reinforcement learning training or just simple model training, it's just better to run more experiments than fewer.

s73
00:07:59.520 --> 00:08:02.960
And so that's the thesis behind the agent harness itself too.

s74
00:08:02.960 --> 00:08:07.039
And so then you know we talk about like the sandbox stack behind the agent.

s75
00:08:07.039 --> 00:08:11.520
And I think that you know there's a lot of mentions of code mode here at this conference and other things like that.

s76
00:08:11.520 --> 00:08:18.960
And one thing that we really want to see happen because we want to build the best product for our customers is to just have a sandbox environment that they can do anything they wanted.

s77
00:08:18.960 --> 00:08:26.560
So for example prior to yesterday I wasn't entirely certain that Arya was going to be able to do a bunch of parallel executions of itself.

s78
00:08:26.560 --> 00:08:36.560
And so I asked it both while I was writing this presentation and preparing for this to just create the sandbox environment uh for itself to run a bunch of parallel executions of its own research loop.

s79
00:08:36.560 --> 00:08:36.959
Right.

s80
00:08:36.959 --> 00:08:44.080
Uh, and so like having that unconstrained environment is really useful for getting the agent to do emergent things.

s81
00:08:44.399 --> 00:08:55.600
And so, you know, I I'm not really sure what the six phrases per record uh means, but I think from a theory standpoint, we want to build then a good pattern for generating these simulation environments.

s82
00:08:55.600 --> 00:09:03.040
And so we have a relatively agnostic DAG that you might think about in a traditional machine learning context, which is like we take a configuration, which are these YAML files.

s83
00:09:03.040 --> 00:09:03.839
We hydrate them.

s84
00:09:03.839 --> 00:09:05.680
So we load the live data that we need to.

s85
00:09:05.680 --> 00:09:13.440
We set up an environment and the environments and weights and biases are relatively expensive because we have lots of production data that we might want to test against or other things like that.

s86
00:09:13.440 --> 00:09:15.200
These are full machine learning training logs.

s87
00:09:15.200 --> 00:09:22.320
We might have to simulate uh GPU executions if you're doing auto research etc. That's a relatively intensive process and so you might want to parallelize that.

s88
00:09:22.320 --> 00:09:31.360
Then you would rehydrate because oftentimes there's runtime configurations that you can't encode in a YAML specification and so you hot patch that data back into the config config.

s89
00:09:31.360 --> 00:09:33.760
Uh and then you run the agent and running the agent is trivial.

s90
00:09:33.760 --> 00:09:39.120
You run it, as I mentioned at the beginning, in the bite-wise identical version that you have in production, and then you score it.

s91
00:09:39.120 --> 00:09:44.880
And scoring is where I spend a lot of my time, which is basically trying to think about the robustness of how you measure results.

s92
00:09:44.880 --> 00:09:48.160
Uh, and I imagine most people here are pretty good at thinking about things like that.

s93
00:09:48.160 --> 00:09:54.720
And I think it's really worth spending the time once you have a configuration of how to run evaluations, thinking about the robustness of our emails.

s94
00:09:54.720 --> 00:10:05.200
uh a Slack message that I probably can't show live on the screen that I got from one of my teammates today made me very happy where he basically just spent all of yesterday thinking about the health of our evaluations and the drift between our evaluations and production.

s95
00:10:05.200 --> 00:10:09.839
Uh and I think that kind of mental model of like you know why are things working well why are things not working well?

s96
00:10:09.839 --> 00:10:12.880
What are the gaps that we see in these two patterns is really useful.

s97
00:10:12.880 --> 00:10:21.200
Um in our case weights and biases Arya scores itself in two patterns which is normatively which gives us basically did we pass a task or not and then relativistically

s98
00:10:21.200 --> 00:10:30.560
where we can set styles based on one variant where it asks questions to the user and one variant where it doesn't ask questions to the user and we can sort of see which one behaves better uh with a relative scoring.

s99
00:10:30.560 --> 00:10:34.079
So a pretty traditional formulation from a reinforcement learning standpoint.

s100
00:10:34.079 --> 00:10:45.519
And then of course if you're running this in parallel you got to tear it down because you want to start it again and you don't want to clobber your teammates's work etc. Uh and so we'll see what happens in this project because a bunch of my teammates are running eval right now and so we'll see how Arya

s101
00:10:45.519 --> 00:10:47.360
performs.

s102
00:10:47.360 --> 00:10:57.120
Uh and then you know more traditionally on tasks I think the eval tasks are just YAML specifications that we define as a starting condition of an environment with a bunch of user configurations

s103
00:10:57.120 --> 00:10:59.760
as well as then an ending condition that we sort of want to get to.

s104
00:10:59.760 --> 00:11:02.320
Uh in our case tasks are flows from users.

s105
00:11:02.320 --> 00:11:04.079
So we simulate them in three ways.

s106
00:11:04.079 --> 00:11:07.839
Uh we do just simple text which is like hey here's an instruction to Arya.

s107
00:11:07.839 --> 00:11:17.920
You might imagine that the task that it generates in this chat over here um where it's logging a specific trace from production uh into our local framework actually looks like

s108
00:11:17.920 --> 00:11:35.760
you know just a simple question that I asked it basically which was that question of Arya doing auto research itself um we can just ask it to continue um as it does that and that task is just a single question to it there's other variants of the tasks that exist as well where we simulate a language model

s109
00:11:35.760 --> 00:11:44.000
that has a persona from a user and we ask it, hey, pretend to be this user, ask certain questions in a particular order so that we can simulate multi-turn environments and interactions with our agent.

s110
00:11:44.000 --> 00:11:45.279
Uh, and then you just run a bunch of these.

s111
00:11:45.279 --> 00:11:46.880
So, we have 886 tasks.

s112
00:11:46.880 --> 00:11:48.160
We categorize them by levels.

s113
00:11:48.160 --> 00:11:57.360
We expose them to our product team so that they can decide whether or not the tasks are good enough or they reflect things that we care about from our benchmarks, etc. And then we just run them a bunch of times.

s114
00:11:57.360 --> 00:11:57.680
Yeah.

s115
00:11:57.680 --> 00:12:02.560
And then, you know, the ultimate thing here is that the trajectory is like the meat and the data that I live behind.

s116
00:12:02.560 --> 00:12:07.040
And so I look at our production traces that exist in our traffic and our offline traces that exist.

s117
00:12:07.040 --> 00:12:10.320
And I really try to exploit the behavior patterns from both of those.

s118
00:12:10.320 --> 00:12:13.600
And so we build tooling ourselves to understand our traces.

s119
00:12:13.600 --> 00:12:19.279
Uh we have like this behavior trace project etc that we run in both prod and offline.

s120
00:12:19.519 --> 00:12:21.360
And yeah that's the eval flywheel.

s121
00:12:21.360 --> 00:12:29.279
So you know fundamentally every production miss or every production goodness as well because I think it's useful to hill climb in a positive direction uh becomes a task for our agent framework.

s122
00:12:29.279 --> 00:12:33.760
And so we use that to sort of define the agent and make it better.

s123
00:12:33.760 --> 00:12:36.320
And these are, you know, our real scores straight from weave.

s124
00:12:36.320 --> 00:12:39.200
So we have a bunch of different categories that we measure ourselves on.

s125
00:12:39.200 --> 00:12:41.680
I think the agent is really good at conceptually guiding you.

s126
00:12:41.680 --> 00:12:44.240
We want to make it better at doing some error analysis for projects.

s127
00:12:44.240 --> 00:12:50.560
You know, we try to make these as difficult as possible for the agent to really test the metal of how this thing performs.

s128
00:12:50.560 --> 00:12:52.160
And so I'll hop back to Arya now.

s129
00:12:52.160 --> 00:12:56.160
There's sort of five minutes left um with sort of a demo of what it's actually doing.

s130
00:12:56.160 --> 00:13:00.639
So right now it's going to run a bunch of code as it's trying to rip this production trace and log it into the framework.

s131
00:13:00.639 --> 00:13:07.120
If I hop into this uh live demo that I just triggered with that nice long prompt, you know, Arya is going to tell me the research runs that it did.

s132
00:13:07.120 --> 00:13:08.720
So we can click through some of these links.

s133
00:13:08.720 --> 00:13:14.000
Uh I asked it to write a report using weights and biases reports from the uh run that it just had.

s134
00:13:14.000 --> 00:13:17.600
If we scrub this, we can see that this is the most recent production trace that it just logged.

s135
00:13:17.600 --> 00:13:20.720
It wrote itself a new task, ran it, and then scored it.

s136
00:13:20.720 --> 00:13:23.760
And so this is like the production variant getting benchmarked on our specific task.

s137
00:13:23.760 --> 00:13:27.360
If we look at the report that it generated, this is the demo that I asked it to do.

s138
00:13:27.360 --> 00:13:33.120
Um, so you know, it ran a real womb agent trace, turned it into a WBF regression task.

s139
00:13:33.120 --> 00:13:37.279
WBAF is the factory that I build, which is the weights and biases agent factory.

s140
00:13:37.279 --> 00:13:39.600
That's the principle behind our offline benchmarking.

s141
00:13:39.600 --> 00:13:44.880
It identified that the problem was that we weren't calling weave.log, one of our SDK calls properly in the sandbox.

s142
00:13:44.880 --> 00:13:50.720
It replicates the source trace, which we can click into here, uh, and then runs a bunch of variants of the agent to see how it makes itself better.

s143
00:13:50.720 --> 00:13:56.160
And so, you know, it added a hill climb target uh which was basically like here's what to do to fix it.

s144
00:13:56.160 --> 00:14:01.600
Ran the trace and then we can see if it gives us a little explanation of exactly what happened at the end.

s145
00:14:01.600 --> 00:14:09.440
Uh how did the prod versus candidate variant perform and you know this is now what I live in.

s146
00:14:09.440 --> 00:14:23.680
And so I think the flywheel here is exceptionally compelling for me where instead of going back to my cloud code and writing offline benchmarks and trying to think about what I want the agent to do, I'm just going to live in this platform instead and hop between my production tracing project where I'm looking at traces roll in

s147
00:14:23.680 --> 00:14:32.880
as they come through over time where AR is, you know, building itself and then hop back into my offline evaluation project to sort of see what my team is doing, what I'm doing, and what changes I want to make.

s148
00:14:32.880 --> 00:14:43.360
what change did we make to the prod skill or to the agent variant etc. And you know to sort of like watch this thing roll out.

s149
00:14:43.360 --> 00:14:49.839
I think a big challenge that we have is it's really easy to go like auto mode for some of these tasks where you want to just see the agent do everything.

s150
00:14:49.839 --> 00:14:54.160
And I haven't written a line of code in maybe eight months because I just tell Claude to write all my code for me.

s151
00:14:54.160 --> 00:14:59.199
And that's a really nice pattern, but it doesn't absolve yourself of the thought behind how we want to make improvements.

s152
00:14:59.199 --> 00:15:07.839
And I think using these tools to improve themselves is very valuable because you get to spend more of your time in the gray of actually trying to think how we make this system better, right?

s153
00:15:07.839 --> 00:15:15.120
And so if you're building an agent, I implore you to use weights and biases to both track that agent etc. Uh we have a really excellent model tracking platform as well.

s154
00:15:15.120 --> 00:15:25.519
So some of the other demos that we've shown here are Arya training machine learning models on H200's running on core infrastructure and doing auto research for Karpathy's nano chat etc or full-scale production projects.

s155
00:15:25.519 --> 00:15:30.959
But I think the exciting thing in this case for me is that I don't need to leave this platform in order to do my job now right.

s156
00:15:30.959 --> 00:15:33.040
I have full observability from a production standpoint.

s157
00:15:33.040 --> 00:15:35.120
I have full observability from an offline standpoint.

s158
00:15:35.120 --> 00:15:40.880
And if we look at, you know, the things that Arya is saying, it's running lots of evaluations on my behalf to improve itself.

s159
00:15:40.880 --> 00:15:48.079
And that auto mode that I just mentioned is really exciting because now I get to spend all of my time thinking about how I can make this system reinforce itself better.

s160
00:15:48.079 --> 00:15:55.199
The tight little guard rails that I can put around it to make it a more useful tool for all of you and then inherently for me.

s161
00:15:55.199 --> 00:15:57.120
Um, but yeah, that's a that's a quick demo.

s162
00:15:57.120 --> 00:16:00.320
We can we can look at the production call trace that we have here from Arya.

s163
00:16:00.320 --> 00:16:01.759
This is the evaluation that it just ran.

s164
00:16:01.759 --> 00:16:05.920
So if we look into like the actual trace, we'll see that, you know, AR did a bunch of work.

s165
00:16:05.920 --> 00:16:07.279
It ran this predict and score column.

s166
00:16:07.279 --> 00:16:09.199
We have a bunch of tool calls that ran.

s167
00:16:09.199 --> 00:16:12.480
Uh and you know, if we hop back into the chat, we'll see the actual variant that it ran.

s168
00:16:12.480 --> 00:16:21.199
Um it looks like the candidate variant just had a tight little prompt that we injected into the system prompt or in one of the skills to sort of solve that exact SDK error uh and mitigate it.

s169
00:16:21.199 --> 00:16:30.560
And that kind of replication from production to simulation to agent then defining the improvement pattern is the thing that I really would like to leave everybody with.

s170
00:16:30.800 --> 00:16:32.480
So that's coreweave Arya.

s171
00:16:32.480 --> 00:16:34.399
Uh welcome to weights and biases.

s172
00:16:34.399 --> 00:16:45.040
I hope that was an interesting talk in sort of how we decide to build uh our offline evaluation metrics, how we reap data from production and the flywheel of trying to build these agents and the things that I think about on a day-to-day basis.

s173
00:16:45.040 --> 00:16:47.040
Uh and yeah, let's get started.

s174
00:16:47.040 --> 00:16:49.680
Thank you all so much.

s175
00:17:04.431 --> 00:17:06.431
[music]