WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=CvRngaQZQ3Y

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/CvRngaQZQ3Y.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:13.840
Hey everyone.

s3
00:00:13.840 --> 00:00:27.520
Uh I'm Vivek and I lead applied research at LangChain and I'm going to talk about something that I think is sexy, which is data mining, but it's not as sexy as LLM, so we're going to try to like make it sexy together.

s4
00:00:27.520 --> 00:00:35.640
And the problem that we're going to talk about today is how do we continuously improve agents, but how do we do that via data?

s5
00:00:35.640 --> 00:00:46.640
So, to start, I'm going to tell a little story that I think maybe a lot of us have felt before, like I ran my agent, it did a bunch of things,

s6
00:00:46.640 --> 00:00:49.160
it made some mistakes.

s7
00:00:49.200 --> 00:00:51.640
Now, I ask someone like, what do I actually do about that?

s8
00:00:51.640 --> 00:00:56.480
Like, I have all this data, made some mistakes, what now?

s9
00:00:56.720 --> 00:01:11.520
Basically, what we're going to do today is we're going to motivate a recipe for what we should do to continuously improve agents over time, and then I'm going to talk from some lived experience and like some stuff that we help customers do to run this over large-scale trace data.

s10
00:01:11.520 --> 00:01:16.600
So, the first step in building a successful agent is shipping it.

s11
00:01:16.600 --> 00:01:23.480
So, if you put it out into the real world, then it can operate in environments and then you can get feedback from what it's doing.

s12
00:01:23.480 --> 00:01:27.120
The second step is collect a ton of traces.

s13
00:01:27.120 --> 00:01:35.840
So, agents operate in the environment every single time they operate, they do tool calls, they have output messages, they call APIs, they use CLIs.

s14
00:01:35.840 --> 00:01:42.120
All of that generates data and we want to store all of that so we can like do stuff with it.

s15
00:01:42.200 --> 00:01:52.200
The next thing is the data mining in this talk, which is once we have tons of trace data, maybe gigabytes, maybe terabytes, depending on like how many agents you're shipping,

s16
00:01:52.200 --> 00:01:54.280
we're going to do data mining over that.

s17
00:01:54.280 --> 00:02:00.440
And I promise I will tell you exactly what data mining we're going do, uh but we're going to do some over it.

s18
00:02:00.440 --> 00:02:14.120
And then the fun part, which is I collected that data, I read it, I curated it, and now we actually need to run the experiments in a data-driven way to see, "Hey, is this new prompt, or is this new tool, or is this new orchestration,

s19
00:02:14.120 --> 00:02:21.040
or is this new loop, is it actually improving things based on the previous traces that I've seen?"

s20
00:02:21.800 --> 00:02:27.080
And this is maybe a bit of a hot take, but continual learning is super hot right now.

s21
00:02:27.080 --> 00:02:30.800
I'm talking about it, this whole room is going to hear about it for the next like five, six hours.

s22
00:02:30.800 --> 00:02:37.200
Um but there's a very tight coupling between what observability is and what continual learning is.

s23
00:02:37.200 --> 00:02:50.680
And the main reason for that is that agents that operate in environments, they produce trace data, and what continual learning for agents and continual learning for humans basically is is I do a bunch of stuff in the world,

s24
00:02:50.680 --> 00:03:00.040
I think about what I did, and then I need to update my definition, like my knowledge, stuff I write down, in order to respond to the feedback from the environment.

s25
00:03:00.040 --> 00:03:08.000
And if you're continual learning company, you need traces, and if you have traces, then you can try to do continual learning over your agents.

s26
00:03:09.600 --> 00:03:18.440
I had to put in a meme because if you look at your data, then you can be like Will Hunting, if anyone's seen the movie, where like everything is super, super easy,

s27
00:03:18.440 --> 00:03:25.640
and you can like improve over time, and I promised Halima I would put this in there, so putting it in there.

s28
00:03:25.880 --> 00:03:26.519
Cool.

s29
00:03:26.519 --> 00:03:30.519
So, why am I talking a bunch about traces anyway?

s30
00:03:30.519 --> 00:03:43.280
So, I'm sure a ton of us were software engineers before, we're software engineers now, and on the left we have a code block, and we can sort of like read the code, and in my head, I can almost reason over what this code does.

s31
00:03:43.280 --> 00:03:46.600
I can see the functions, I can see like how they call each other.

s32
00:03:46.600 --> 00:03:49.040
I can roughly understand the logic in Python.

s33
00:03:49.040 --> 00:04:01.360
Um that doesn't exactly exist in agent world because agents have prompts, they have tools, they have skills, they have hooks, they have middlewares, some agents call other agents and I orchestrate them in swarms.

s34
00:04:01.360 --> 00:04:09.480
It's really, really hard for humans to reason about how certain prompts that they change are actually going to affect agent behavior at scale.

s35
00:04:09.480 --> 00:04:13.400
And this also varies between the different domains that you're doing it on.

s36
00:04:13.400 --> 00:04:19.600
So, a prompt change and you're using for the medical domain is going to be like completely different than a prompt change that you want to do for the law domain.

s37
00:04:19.600 --> 00:04:28.360
And in general, over the last four years since the ChatGPT moment, we've started trading determinism for autonomy.

s38
00:04:28.360 --> 00:04:40.240
And in that shift, sort of what we need to do is create tools and create systems to still understand agents when they're autonomously operating in environments.

s39
00:04:40.960 --> 00:04:42.680
So, I talked about traces.

s40
00:04:42.680 --> 00:04:45.080
Um, why like why should you read them?

s41
00:04:45.080 --> 00:04:49.480
And at LangChain, what do we actually do when we're reading traces?

s42
00:04:49.480 --> 00:04:58.919
So, we centralize a bunch of our data, so we put everything in a tracing project and this is usually either like per agent or like centralized across all of our agents.

s43
00:04:58.919 --> 00:05:05.240
And then what we do is we send agents to read traces from other agents, right?

s44
00:05:05.240 --> 00:05:07.760
And then we look for a bunch of different things.

s45
00:05:07.760 --> 00:05:15.720
And we might ask for, hey, like find a bunch of like good and bad interactions where like users got upset or like users are like really happy.

s46
00:05:15.720 --> 00:05:22.680
Um, another question I might ask is uh this is a technical question like agents now run for millions of tokens.

s47
00:05:22.680 --> 00:05:25.640
Does the agent get really dumb after the first compaction?

s48
00:05:25.640 --> 00:05:27.120
After the second compaction?

s49
00:05:27.120 --> 00:05:28.440
Does it never get dumb?

s50
00:05:28.440 --> 00:05:29.840
Like how do we actually answer these questions?

s51
00:05:29.840 --> 00:05:32.720
We need to do it by actually looking at the traces.

s52
00:05:32.720 --> 00:05:44.520
And then the the other thing is like if I look at the traces, then I can try to prove some counterfactuals, which is hey, like I ran GPT 5.5 for this and I heard like GLM is really good.

s53
00:05:44.520 --> 00:05:49.280
What happens if I run GLM 5.2 for this task, and how do I compare them?

s54
00:05:49.280 --> 00:05:50.760
Metrics, awesome.

s55
00:05:50.760 --> 00:05:54.200
The The trace level captures the actual like behavior that users see.

s56
00:05:54.200 --> 00:05:59.000
So, that's also like very helpful for seeing behavior like fine-grained scales.

s57
00:05:59.560 --> 00:06:14.920
And the way that we sort of think about the data that's being generated by agents is that the data that we see today is going to be the smallest that humans have ever seen in their entire lives because we're in this massive exponential

s58
00:06:14.920 --> 00:06:18.480
shift to our agents are doing more and more work in the economy.

s59
00:06:18.480 --> 00:06:32.040
And what that means is like the amount of data that humans have produced in our entire lifetime will soon be eclipsed by agents running on like year scales, and then 6-month scales, and 3-month scales, and then maybe every day, right?

s60
00:06:32.040 --> 00:06:38.919
Um and to understand a ton of that data, roughly what we need to do is contend with a couple problems.

s61
00:06:38.919 --> 00:06:41.200
There's more, but these are the two that I'm going to focus on.

s62
00:06:41.200 --> 00:06:52.040
So, one, uh reading traces at scale is super expensive, uh especially if you have millions of traces, and if you have millions of tokens per trace, right?

s63
00:06:52.040 --> 00:06:54.880
Um think of it as like an input token cost.

s64
00:06:54.880 --> 00:07:02.880
You can like literally multiply the input token cost uh times the number of traces times like how big each trace is on average, right?

s65
00:07:02.880 --> 00:07:15.000
Um the the other thing is [clears throat] if I have a super long interaction with a coding agent like Cloud Code or Codex or like deep agents, um I can't even read that trace with another agent because that

s66
00:07:15.000 --> 00:07:17.040
that context like doesn't fit in memory, right?

s67
00:07:17.040 --> 00:07:23.880
So, it's like we we need to develop systems so I can sort of treat that context as like an external object, and then I can sort of query into it, right?

s68
00:07:23.880 --> 00:07:36.560
So, we we need to build agents to efficiently mine data from other agents, and it's it's no longer as simple as just like feeding the data into context, and there's like tricks that we'll sort of talk about uh to to do that well.

s69
00:07:36.840 --> 00:07:37.120
Great.

s70
00:07:37.120 --> 00:07:51.720
So, one of the things that I think is really, really cool in the last 6 months is that open models have basically hit an inflection point in intelligence that we at LangChain don't reach for the frontier models for every single use case.

s71
00:07:51.720 --> 00:07:58.760
We're quite conscious about what is the minimum level of intelligence that I need to do any given task.

s72
00:07:58.760 --> 00:08:06.720
And like practically speaking, honestly, yes, we start with Opus, we start with 55 because we just want to know if the task is even possible.

s73
00:08:06.720 --> 00:08:15.400
But then once we reach that sort of like waterline, then we like look back at those traces and we see, "Hey, can we use an open model to do the same thing?"

s74
00:08:15.400 --> 00:08:20.280
So, this is a bunch of work that we did with Harvey and then their lab legal benchmark.

s75
00:08:20.280 --> 00:08:28.360
Basically, what we're looking at is can I match the trace judging capability of Opus with an open cheaper model?

s76
00:08:28.360 --> 00:08:35.000
And the answer is roughly yes at like an order or like two orders of magnitude cheaper.

s77
00:08:35.000 --> 00:08:43.000
And like the way we do that is we try a bunch of models, we do a bunch of like harness engineering, and the harness engineering is informed by a bunch of the traces that we read.

s78
00:08:43.000 --> 00:08:46.200
So, it's like, "Hey, like Opus reasons about things in this way.

s79
00:08:46.200 --> 00:08:48.080
Maybe that's because of the prompt.

s80
00:08:48.080 --> 00:08:56.760
Maybe Opus is just smarter, which it is than a bunch of the open models, but that might mean I need to give it a little bit more guidance so it can reach the sort of same intelligence level at like a much

s81
00:08:56.760 --> 00:08:58.839
much lower cost."

s82
00:08:58.839 --> 00:09:04.440
And the the other thing that we sort of look at is like harness engineering is amazing.

s83
00:09:04.440 --> 00:09:17.280
You get instant feedback and you can sort of like run on your evals, but eventually what we find is you hit a threshold of intelligence where it's like, "If I keep tweaking this prompt, I'm not going to get too much more out of it."

s84
00:09:17.280 --> 00:09:24.920
And once we reach that point, we sort of look at, "Okay, can I actually like fine-tune the model on my domain-specific task?"

s85
00:09:24.920 --> 00:09:27.280
And can I like make it better on those tasks?

s86
00:09:27.280 --> 00:09:35.000
And what we find is if we take like base models and we tune them on like very specific vertical tasks, which is what a lot of our customers do,

s87
00:09:35.000 --> 00:09:37.760
they don't really care about the entire variance of tasks.

s88
00:09:37.760 --> 00:09:39.560
like they care about what their customers care about.

s89
00:09:39.560 --> 00:09:47.640
So, if we focus on that narrow set of tasks, then we can fine-tune base models to sort of like reach and then also go beyond frontier performance.

s90
00:09:47.640 --> 00:09:59.640
And I think one sort of like small thing I'll mention as a lot of people are getting into fine-tuning is that another sort of like economic decision is that you can move from token costs

s91
00:09:59.640 --> 00:10:01.240
to hardware costs.

s92
00:10:01.240 --> 00:10:04.360
And this is like can be a really big change, right?

s93
00:10:04.360 --> 00:10:08.040
Cuz like you're very used to, "Hey, like a million tokens cost this much."

s94
00:10:08.040 --> 00:10:10.720
Not as much like this cluster sort of costs this much.

s95
00:10:10.720 --> 00:10:18.800
But for like very high inference workloads, we find it to be way cheaper just to like run a cluster and I get like unlimited inference on that cluster.

s96
00:10:18.800 --> 00:10:27.200
I don't have to worry about tokens, but I can just do the calculation of like, "Hey, um this will end up being cheaper and then I can spin it down when I don't need it."

s97
00:10:27.920 --> 00:10:28.240
Cool.

s98
00:10:28.240 --> 00:10:32.480
And I said all of this um so we obviously like built a product to do that.

s99
00:10:32.480 --> 00:10:36.000
Uh I won't shill it too much, but it's LangSwith engine.

s100
00:10:36.000 --> 00:10:48.800
Uh basically, this product is trying to automate this loop for you, which is if you have any volume of trace data and you're looking for something that trace data or you want to generate e-vals from that trace data

s101
00:10:48.800 --> 00:10:53.920
or you want to like generate feedback for like humans to read from that trace data, it will go read all of it.

s102
00:10:53.920 --> 00:10:54.960
It'll like find issues.

s103
00:10:54.960 --> 00:10:56.560
It'll agentically search over it.

s104
00:10:56.560 --> 00:11:00.680
They can like prepare data sets for you to do something after.

s105
00:11:00.680 --> 00:11:08.120
And a bit of a leader, um what that something basically is is the outputs of this trace mining exercise.

s106
00:11:08.120 --> 00:11:14.040
So, there's like three things that I mentioned here uh which we see a bunch and we kind of put into the product.

s107
00:11:14.040 --> 00:11:27.240
So, one is distillation and fine-tuning, which is let's say I'm running GLM 5.2. It's doing great, but I think that I can run this task like way cheaper with like a 9B or 13B model.

s108
00:11:27.240 --> 00:11:37.920
Then what I'll do is like I'll take the good traces and the good examples from the GLM 5.2 runs, I'll prepare them in a data set, and then I'll try to fine-tune a small model on that data set to like mimic behavior,

s109
00:11:37.920 --> 00:11:38.520
essentially, right?

s110
00:11:38.520 --> 00:11:41.080
And this is like distillation, SFT.

s111
00:11:41.080 --> 00:11:45.360
The The other one is generating evals in environments.

s112
00:11:45.360 --> 00:11:53.480
So, maybe another slightly hot take, I think you can basically define agent behavior by showing the evals that you ran on it, right?

s113
00:11:53.480 --> 00:12:03.960
Like, if someone showed me all the things that they're trying to test their agent on, I think I would have a rough idea about how that agent is going to behave because it literally like hill climbs those evals,

s114
00:12:03.960 --> 00:12:08.280
and you you alter the behavior of the agent to make the evals pass, right?

s115
00:12:08.280 --> 00:12:12.080
Like, the purpose of evals is roughly to try to make them pass, right?

s116
00:12:12.080 --> 00:12:15.080
So, I update my agent so that they essentially pass.

s117
00:12:15.080 --> 00:12:19.240
And then the the other thing is um like humans are still in the loop.

s118
00:12:19.240 --> 00:12:22.240
Like, I need to know that customers are happy.

s119
00:12:22.240 --> 00:12:24.560
I also want to know what my agents are doing.

s120
00:12:24.560 --> 00:12:27.720
I just don't have the bandwidth to read a bunch of traces.

s121
00:12:27.720 --> 00:12:37.040
So, preparing content for humans is still like really, really valuable today, especially in like high-trust domains like legal and medical.

s122
00:12:37.040 --> 00:12:45.560
Like, some human needs to review this, um but they can't read it all, so we try to make it easy for them to process all that data.

s123
00:12:46.160 --> 00:12:46.480
Great.

s124
00:12:46.480 --> 00:12:48.480
This is um maybe a bit of a throwback.

s125
00:12:48.480 --> 00:12:51.720
Like, how many people here know what like scikit-learn is?

s126
00:12:51.720 --> 00:12:54.360
Uh maybe put your uh psych This crowd is just awesome.

s127
00:12:54.360 --> 00:12:55.600
Um cool.

s128
00:12:55.600 --> 00:13:05.000
So, uh when I was like first doing my PhD, uh my PhD was like kind of trying to do this, but like add new algorithms to scikit-learn.

s129
00:13:05.000 --> 00:13:15.680
And like what scikit-learn basically is uh an abstract level, it's a bunch of helpers to fit learning systems to data.

s130
00:13:15.680 --> 00:13:15.839
Right?

s131
00:13:15.839 --> 00:13:26.080
And like classical machine learning, I had like a data set, I tried to fit it to it, but I think the same principles that we use in modern in I got I call it classical machine learning, it's like 6 years ago.

s132
00:13:26.080 --> 00:13:31.880
Um that we do in classical machine learning uh definitely still apply to this agent-first world.

s133
00:13:31.880 --> 00:13:38.160
Um, the way that they apply is what I like to call model harness task fit.

s134
00:13:38.160 --> 00:13:51.320
So, we still have this sort of like fit function that I'm going to try to like take my data, take a harness, take a model, and I'm going to try to fit it all together to make sure that all of my tasks pass, right?

s135
00:13:51.320 --> 00:14:05.680
The algorithms look slightly different, uh but the overall process of machine learning doesn't really look that different, and we'll talk about maybe roughly what our job becomes in this data-first,

s136
00:14:05.680 --> 00:14:07.280
agent-first, fit-first world.

s137
00:14:07.280 --> 00:14:12.720
So, a couple of our main jobs now are find good fit functions.

s138
00:14:12.720 --> 00:14:14.480
So, these are like auto research.

s139
00:14:14.480 --> 00:14:22.240
This is tons of great work that's being done in RL on different methods like OPD, OPSD, trySFT.

s140
00:14:22.680 --> 00:14:24.200
And also find good data, right?

s141
00:14:24.200 --> 00:14:33.440
So, if you put those two things together, then that is basically the applied or just overall research question that every team has to make their agents better.

s142
00:14:33.440 --> 00:14:40.640
I like some some examples that we've seen that are like very popular that we're pretty bullish on are just generally auto research.

s143
00:14:40.640 --> 00:14:47.839
So, if you have some sort of score that you can make number go up, uh agents are pretty good at making that number go up.

s144
00:14:47.839 --> 00:14:50.680
They might cheat a little bit and you need to like check them on some stuff.

s145
00:14:50.680 --> 00:14:59.240
Um, but this sort of like general feedback loop of do something, read the results, read the traces, and then do an update ends up being pretty useful.

s146
00:14:59.240 --> 00:15:02.440
And then I talked about like model fine-tuning a bunch as well.

s147
00:15:02.440 --> 00:15:05.800
Um, so we we just like went and did this.

s148
00:15:05.800 --> 00:15:12.280
Uh, this was I think even before the term auto research came out, but a lot of people were doing it, which is hey, like terminal benches like really hard.

s149
00:15:12.280 --> 00:15:19.120
Uh, what would happen if an agent just like read its traces, uh proposed experiments, and then tried to do fixes?

s150
00:15:19.120 --> 00:15:26.400
Um, I think one like really key thing here is uh giving agents dense feedback signals.

s151
00:15:26.400 --> 00:15:29.640
So, like terminal bench, the output is just a number, right?

s152
00:15:29.640 --> 00:15:31.560
Like did you pass or did you not pass?

s153
00:15:31.560 --> 00:15:38.720
Uh that's like kind of helpful, but if I give you like a super random task, like you just did a bunch of stuff, and then I just said like you failed or you passed,

s154
00:15:38.720 --> 00:15:43.800
uh if you failed, like you wouldn't really have a good signal to figure out what you should do next, right?

s155
00:15:43.800 --> 00:15:51.800
So, densifying feedback is uh really good way to improve agents, and like traces are the substrate that hold that feedback.

s156
00:15:51.800 --> 00:15:56.680
And then agents are very good at like reading those uh those traces and like figuring out like what to do next.

s157
00:15:56.680 --> 00:16:08.920
Um and then this sort of question always comes up, which is when should I like harness eng, when should I fine-tune, uh should I do more harness eng after it?

s158
00:16:08.920 --> 00:16:20.160
I'm like pretty bullish on the idea of if you need to do something for improving your agent, the best thing that you can do is collect feedback as quickly as possible,

s159
00:16:20.160 --> 00:16:23.360
like either from humans labeling or just letting the agents run.

s160
00:16:23.360 --> 00:16:27.240
So, like harness engineering gives you feedback in maybe 2 minutes.

s161
00:16:27.240 --> 00:16:37.839
Um once you sort of saturate the harness engineering ceiling, right, then you can maybe try to do like fine-tuning after that, but we find a lot of teams are happy with harness engineering

s162
00:16:37.839 --> 00:16:41.760
and uh it solves their customer use case, so like we always sort of sort of recommend it.

s163
00:16:41.760 --> 00:16:51.600
And then we have this like sort of sandwich, which is like try harness engineering, try to do fine-tuning to sort of like break through that ceiling, and then do more harness engineering again if you need to.

s164
00:16:51.600 --> 00:17:04.800
And then I'll sort of end on the the idea generally of continual learning is that there's an agent taking actions in the environment, and then it needs to use that information,

s165
00:17:04.800 --> 00:17:11.000
sorry guys, needs to use that information to update information about itself, right?

s166
00:17:11.000 --> 00:17:17.160
So, it's like I did a bunch of these tasks, and like I need to update my prompts to make sure I do them more efficiently.

s167
00:17:17.160 --> 00:17:28.000
Or users are users keep asking to search for these types of things, I should maybe tell like tell my creator that like they're they're doing this sort of stuff, right?

s168
00:17:28.000 --> 00:17:32.520
It's like taking action in the environment kind of like humans do and updating ourselves.

s169
00:17:32.520 --> 00:17:43.560
What that looks like today, slightly unclear, but we think that you're going to have to do it across all three axes, which is one, collect a bunch of training data, which is like observational

s170
00:17:43.560 --> 00:17:46.360
data from agents taking actions.

s171
00:17:46.360 --> 00:18:00.600
The The other one is like harness updates generally, like you know, the the Codex harness and the Cloud Code harness and like our harness and everyone's harness, like they look a certain way because like models are are trained in them and they look a certain way because of the tasks that they do in the real world and

s172
00:18:00.600 --> 00:18:06.840
we think like evolving those over time is going to be super important in in order to make them work.

s173
00:18:06.840 --> 00:18:10.120
And the the last thing is like memory.

s174
00:18:10.120 --> 00:18:17.920
So, uh we humans are like really good at like remembering stuff over time, but we are not append-only logs of information.

s175
00:18:17.920 --> 00:18:28.080
And if agents are going to be working with us over like year, 5-year, decade, lifetime time scales, we cannot just append everything to like a really big file and then search over it.

s176
00:18:28.080 --> 00:18:34.240
There's a ton of stuff that needs to happen with like updating those files over time and then just making memory like really efficient.

s177
00:18:34.240 --> 00:18:40.880
But, we think a lot of that actually comes from this idea of scaling sleep time compute and and dreaming generally.

s178
00:18:40.880 --> 00:18:45.302
So, it's like read all of the traces over the entire agent life cycle and then like do

s179
00:18:45.302 --> 00:18:45.440
[music]

s180
00:18:45.440 --> 00:18:48.760
things to update agent state.

s181
00:18:49.160 --> 00:18:49.520
Awesome.

s182
00:18:49.520 --> 00:18:55.480
So, like quick quick takeaways, uh mining traces gives you signals to hill climb on.

s183
00:18:55.480 --> 00:19:03.480
I would say like if you have an agent, just turn on tracing and point an agent at it and that's like the easiest thing that you can do to see like

s184
00:19:03.480 --> 00:19:06.160
to basically understand what your agents are doing.

s185
00:19:06.160 --> 00:19:08.320
We're very excited about open models.

s186
00:19:08.320 --> 00:19:11.080
We want to help you fine-tune open models.

s187
00:19:11.080 --> 00:19:12.640
We provide them as a service as well.

s188
00:19:12.640 --> 00:19:18.520
So, if you're interested in that, would would love to chat how you can use open models to make everything smarter and cheaper.

s189
00:19:18.520 --> 00:19:24.800
Um, continual learning is about operating environments and then integrating that data back into agent state.

s190
00:19:24.800 --> 00:19:31.880
And then finally, I think this is so cool that like we have systems that's going to pre- that are going to produce more data than we ever have before.

s191
00:19:31.880 --> 00:19:39.400
Uh, we need to all come up with like interesting research directions to learn how to like manage that at scale and like make all of our agents better.

s192
00:19:39.400 --> 00:19:41.600
And with that, thank you.

s193
00:19:41.600 --> 00:19:42.943
Thank you all for coming.

s194
00:19:42.943 --> 00:19:44.943
[applause]