WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=jRCpXUjz4CI

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/jRCpXUjz4CI.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:01.309
Awesome.

s2
00:00:01.309 --> 00:00:03.309
[music]

s3
00:00:13.100 --> 00:00:15.100
[applause]

s4
00:00:16.960 --> 00:00:18.800
Thank you so much.

s5
00:00:18.800 --> 00:00:22.160
Uh so yeah, like you said, my name is Alex Shaw.

s6
00:00:22.160 --> 00:00:23.840
I work at Lot Institute.

s7
00:00:23.840 --> 00:00:31.119
Um, and I'll be speaking today about Harbor, which is an agent evaluation and uh, RL environment framework.

s8
00:00:31.119 --> 00:00:34.239
And the title of my talk is everything is a rollout.

s9
00:00:34.239 --> 00:00:39.440
And I think you'll see as I get into it why we titled the talk that way.

s10
00:00:39.440 --> 00:00:46.480
Um, but first I want everybody to come travel back in time with me to the year 2018.

s11
00:00:46.480 --> 00:00:48.559
So eight eight years ago.

s12
00:00:48.559 --> 00:00:53.199
Um, and we're going to talk about some of the things that were going on in 2018.

s13
00:00:53.199 --> 00:00:58.239
So, uh, probably you're you're going to see Avengers Infinity War later today.

s14
00:00:58.239 --> 00:01:01.039
The second one is just getting released.

s15
00:01:01.039 --> 00:01:08.960
Uh, GPT1 was was just released and you probably didn't even notice, although maybe some people did.

s16
00:01:08.960 --> 00:01:17.119
Um, musically, this random startup was about to rebrand to a product called Tik Tok.

s17
00:01:17.119 --> 00:01:23.200
And you might have just learned about AirPods when you saw somebody walking around with headphones that had no cord.

s18
00:01:23.200 --> 00:01:26.000
Um, so what about what about software engineering?

s19
00:01:26.000 --> 00:01:30.080
What did software engineering look like in 2018?

s20
00:01:30.080 --> 00:01:35.040
Uh, we're going to read this tweet from Stay Sassy Sassy with two two A's.

s21
00:01:35.040 --> 00:01:38.320
Um, about what it was like to write code in 2018.

s22
00:01:38.320 --> 00:01:43.280
So it says, "It's 2018 and your coworker just sent you a 400 line pull request.

s23
00:01:43.280 --> 00:01:46.000
You get a cup of coffee and sit down to review it.

s24
00:01:46.000 --> 00:01:46.880
It's beautiful.

s25
00:01:46.880 --> 00:01:50.159
Elegant micro refactors, crispy method names.

s26
00:01:50.159 --> 00:01:51.920
You catch a few things, but that's okay.

s27
00:01:51.920 --> 00:01:53.280
It's part of the dance.

s28
00:01:53.280 --> 00:01:56.240
They didn't consider extensibility on part of their API.

s29
00:01:56.240 --> 00:01:57.600
Here's a comment, buddy.

s30
00:01:57.600 --> 00:02:00.640
And this is actually just part of the tweet, so it keeps going.

s31
00:02:00.640 --> 00:02:02.560
You should look it up if you want.

s32
00:02:02.560 --> 00:02:05.680
And it's obviously written humorously.

s33
00:02:05.680 --> 00:02:10.720
Um, but the thing is, it does feel a little bit nostalgic.

s34
00:02:10.720 --> 00:02:15.040
Um, just like some of these other things that were going on in 2018.

s35
00:02:15.040 --> 00:02:23.280
Now, the thing is, uh, this only changed maybe six or 12 or 18 if you're a very early adopter months ago.

s36
00:02:23.280 --> 00:02:27.599
Um, but it but it already feels like the distant past in some ways.

s37
00:02:27.599 --> 00:02:34.319
Uh, and I think it's time to start talking about well, what will the history book say about software engineering?

s38
00:02:34.319 --> 00:02:39.280
And when I say software engineer, I mean that style of 2018 software engineering.

s39
00:02:39.280 --> 00:02:44.160
Um, it will probably say the the the history books will probably say a lot of things.

s40
00:02:44.160 --> 00:02:51.920
Um, but I think one thing for sure that they'll say is software engineering was when you knew what the code would do before you ran it.

s41
00:02:51.920 --> 00:03:02.239
Um, so and that brings me then to a different tweet from France Chalet uh where he says, "Agentic coding is a form of machine learning.

s42
00:03:02.239 --> 00:03:12.319
Generated code is best treated as a blackbox artifact whose behavior and generalization should be managed via empirical evaluation like with any ML model."

s43
00:03:12.319 --> 00:03:28.959
Um, and that kind of brings us to the next part in this talk, which is to compare and contrast what agent development looks like versus what more traditional software engineering development looks like and why it demands a new set of tools to really understand what's going on and have confidence

s44
00:03:28.959 --> 00:03:30.080
and trust.

s45
00:03:30.080 --> 00:03:34.319
Um, so here's a here's a 2018 program right here.

s46
00:03:34.319 --> 00:03:41.360
Uh, so you can tell already the purpose of the program is to extract phone numbers from text.

s47
00:03:41.360 --> 00:03:45.599
And we have a rejax right here that uh looks for the phone number.

s48
00:03:45.599 --> 00:03:53.760
And I can say with 100% confidence what will happen if I run this program 1 million times in a row.

s49
00:03:53.760 --> 00:03:59.280
Um so now let's update it to the 2026 version.

s50
00:03:59.280 --> 00:04:09.360
So uh I swap out my reax and instead obviously I throw in my model call instead and I say extract this phone number.

s51
00:04:09.360 --> 00:04:16.880
Um, so in some ways this is actually a more powerful program because uh the reax was actually a little bit brittle.

s52
00:04:16.880 --> 00:04:22.560
It would have missed any phone number that wasn't formatted exactly like how it was specified.

s53
00:04:22.560 --> 00:04:29.919
Whereas I'm pretty confident that this program with GPT 5.5 will catch a lot of the phone numbers that are formatted weirdly.

s54
00:04:29.919 --> 00:04:43.520
Um, however, if I ran this exact program one million times, I'm not 100% confident that it will print the same thing every single time or that I know exactly what it will print.

s55
00:04:43.520 --> 00:04:46.560
Um, it probably gets it right almost every time.

s56
00:04:46.560 --> 00:04:48.320
This is a pretty simple task.

s57
00:04:48.320 --> 00:04:54.080
Um, but, uh, this is obviously far simpler than the things we're asking these models to do.

s58
00:04:54.080 --> 00:04:59.600
And the uncertainty only increases as the complexity of the task increases.

s59
00:04:59.600 --> 00:05:05.199
Uh so now let's come back to Francois's tweet and let's update it a little bit.

s60
00:05:05.199 --> 00:05:06.160
We'll generalize it.

s61
00:05:06.160 --> 00:05:12.479
So he says agentic coding and my claim is well just agents in general are a form of machine learning.

s62
00:05:12.479 --> 00:05:21.919
And then he says generated code is best treated as a blackbox artifact and I say agent performance itself is best treated as a blackbox artifact.

s63
00:05:21.919 --> 00:05:24.800
Uh and that brings us to our to our new paradigm.

s64
00:05:24.800 --> 00:05:32.080
So uh we know now agent development is more similar to machine learning than it is to software engineering.

s65
00:05:32.080 --> 00:05:43.520
So what are the things that we should keep an eye out for the the tools that you use for machine learning and also the pitfalls of machine learning and what are the analoges for those with agent development.

s66
00:05:43.520 --> 00:05:46.800
So we have machine learning on the left agent building on the right.

s67
00:05:46.800 --> 00:06:02.880
Uh so training data that now looks like environments and then your test and your validation set that looks like a val which if you take off the mask is also actually just environments at least the style that we'll be talking about today.

s68
00:06:02.880 --> 00:06:13.039
Um weights your model weights are now skills prompts tools the model whether you're picking between models or actually updating the model yourself.

s69
00:06:13.039 --> 00:06:18.800
uh your loss function looks now more like environment rewards and feedback.

s70
00:06:18.800 --> 00:06:29.199
Your backrop or optimizer is now some context textbased optimization algorithm like Jeepa or even just running a coding agent in a loop.

s71
00:06:29.199 --> 00:06:34.319
Uh your gradient descent step looks like a pull request into your repo.

s72
00:06:34.319 --> 00:06:42.639
And overfitting looks like reward hacking or also just uh overfitting that's also possible for agent development.

s73
00:06:42.639 --> 00:06:51.280
Um, so we have these things on the left, there's, you know, products and libraries and platforms that were built for these purposes.

s74
00:06:51.280 --> 00:06:54.160
And then for everything on the right, we're just getting started.

s75
00:06:54.160 --> 00:06:57.520
So, uh, we built Harbor to answer some of these questions.

s76
00:06:57.520 --> 00:07:08.720
We're building it to answer more of these questions, but other people are also building interesting products and interesting frameworks to help people tighten this loop of agent development.

s77
00:07:08.720 --> 00:07:21.599
Um and then something else to consider is that uh as popular as machine learning was and it spawned some of the largest companies in the world, you have like data bricks which are worth uh hundreds of billions of dollars.

s78
00:07:21.599 --> 00:07:35.840
Um already the number of people using and building agents is probably magnitudes larger than the number of people that ever uh were doing machine learning and that trend will will only continue.

s79
00:07:35.840 --> 00:07:48.560
Uh and now let's look at the last piece of Francois's tweet that I want to call out uh which is uh these these agent agent performance uh dut should be managed via empirical

s80
00:07:48.560 --> 00:07:49.919
evaluation.

s81
00:07:49.919 --> 00:07:55.360
Uh and that brings us to our next question which is how do I actually evaluate an agent?

s82
00:07:55.360 --> 00:08:01.360
Uh and this is when we start to get into uh what what harbor does.

s83
00:08:02.240 --> 00:08:05.440
So the answer as we already gave away earlier is environments.

s84
00:08:05.440 --> 00:08:08.800
That's your aval and your training data.

s85
00:08:08.800 --> 00:08:11.120
And what is an environment in this case?

s86
00:08:11.120 --> 00:08:12.720
Well, we need an instruction.

s87
00:08:12.720 --> 00:08:16.000
We need some way to tell the agent what it's supposed to do.

s88
00:08:16.000 --> 00:08:20.240
And then we need some place for the agent to try to do this thing.

s89
00:08:20.240 --> 00:08:24.319
And right now all of the agents run on computers and they do stuff on computers.

s90
00:08:24.319 --> 00:08:27.759
So we'll put it into a computer, but we'll put it into a virtual computer.

s91
00:08:27.759 --> 00:08:29.199
So a sandbox.

s92
00:08:29.199 --> 00:08:40.000
Um, and then we need some way of telling whether or not the agent actually did the thing that we told it to do in the sandbox within some amount of time or other stopping condition.

s93
00:08:40.000 --> 00:08:48.160
And that's your verifier, which is like some programmatic tests or rubrics um or agent that comes in to see what happened.

s94
00:08:48.160 --> 00:08:52.560
So in Harbor, you specify this as a file directory.

s95
00:08:52.560 --> 00:09:00.240
And this uh specific directory layout has become relatively standard in a lot of the environment space.

s96
00:09:00.240 --> 00:09:09.279
So lots of people have adopted it and use it as a way to specify environments which is useful because then it can easily pass between hands and becomes interoperable.

s97
00:09:09.279 --> 00:09:14.480
Um and then okay cool we have a bunch of data now we've implemented a bunch of these environments.

s98
00:09:14.480 --> 00:09:21.360
Now how do I actually use it to start understanding what my agent can and can't do and how to make it better.

s99
00:09:21.360 --> 00:09:33.279
Um so that's a harbor roll out and uh so what you do is you start with your tasks and then you take it you start up your sandbox and then step one you pass that sandbox to your agent.

s100
00:09:33.279 --> 00:09:43.360
So you're either running your agent outside the sandbox and executing commands into it or you're running it inside the sandbox and it's calling whatever its commands are as part of its program.

s101
00:09:43.360 --> 00:09:51.920
Uh, and then it runs for some amount of time until it hits a stopping condition, produces some trajectory, and we'll come back to that later because that's important.

s102
00:09:51.920 --> 00:09:57.680
And then you pass that sandbox to the verifier, which uh runs some verification process.

s103
00:09:57.680 --> 00:09:59.360
And then finally, you stop the sandbox.

s104
00:09:59.360 --> 00:10:10.399
The verifier produces a reward or a set of rewards and then you take those, you aggregate them a bunch across a bunch of rollouts in a data set with some agent, and that becomes your valid result.

s105
00:10:10.399 --> 00:10:20.160
So this process here which looks relatively simple is actually extremely universal and uh it's also a little bit over overly simplified.

s106
00:10:20.160 --> 00:10:23.680
So Harbor by now allows for a lot of different flavors of rollout.

s107
00:10:23.680 --> 00:10:30.959
So you can do multi-step, you can run your verification in a separate sandbox, you can collect artifacts, you can simulate a user.

s108
00:10:30.959 --> 00:10:37.120
Um but this is kind of like the the the bare bones approach uh that they're all built off of.

s109
00:10:37.120 --> 00:10:39.760
So what is Harbor more specifically?

s110
00:10:39.760 --> 00:10:43.440
One, it's a format for specifying agentic environments.

s111
00:10:43.440 --> 00:10:52.560
Two, it's an open-source framework for performing rollouts in parallel using any agent with any model in any sandbox on any task.

s112
00:10:52.560 --> 00:10:55.600
And it's a registry of popular training and a val environment sets.

s113
00:10:55.600 --> 00:10:58.160
I think we have like three or 400 Aval sets by now.

s114
00:10:58.160 --> 00:11:02.320
And in fact, I think two or three benchmarks even came out today that run with Harbor.

s115
00:11:02.320 --> 00:11:03.760
Um, so we're excited about that.

s116
00:11:03.760 --> 00:11:07.120
And in general, it is trying to be a common language for environments.

s117
00:11:07.120 --> 00:11:17.200
So a way for people to specify things that are extremely interoperable and it allows you to maximize data velocity and just increases progress in the industry.

s118
00:11:17.200 --> 00:11:18.880
Uh so who needs a val?

s119
00:11:18.880 --> 00:11:22.480
Uh now that we understand how to make them, we understand how to use them.

s120
00:11:22.480 --> 00:11:24.720
Well, now who should actually be doing this?

s121
00:11:24.720 --> 00:11:28.480
And the answer is every single company that uses computers.

s122
00:11:28.480 --> 00:11:34.000
And I think that's probably close to all of the companies in the world.

s123
00:11:34.000 --> 00:11:44.640
And the reason is because if you're doing something on a computer, then you should be seeing if you can use AI to automate part or all of that process that you're currently performing on a computer.

s124
00:11:44.640 --> 00:11:51.040
Uh because that will increase the productivity of your company and therefore increase the value that your company generates.

s125
00:11:51.040 --> 00:11:56.959
Uh so I like this quote from Satcha Nadella from the applied compute podcast he did last week.

s126
00:11:56.959 --> 00:12:07.440
He says if you want to build an agentic system start with the aval that matters and your ability to grade the outcome and then say I welcome all models.

s127
00:12:07.440 --> 00:12:15.920
So I like that last line because what he goes to say is that as soon as you have an aval the power is now in your hands.

s128
00:12:15.920 --> 00:12:18.079
You can consider every single model.

s129
00:12:18.079 --> 00:12:19.360
You don't have to trust brand.

s130
00:12:19.360 --> 00:12:21.200
You don't have to trust somebody else's aval.

s131
00:12:21.200 --> 00:12:22.800
You don't have to trust a public aval.

s132
00:12:22.800 --> 00:12:30.160
Um and you can kind of skate the paro however you desire to uh balance that cost performance tradeoff.

s133
00:12:30.160 --> 00:12:35.200
So step one build the eval step two kind of optimize against it.

s134
00:12:35.200 --> 00:12:46.000
Um so what will people evaluate and uh I'm going to list four things and these four things are based off of what we actually see people using harbor to build a val for right now.

s135
00:12:46.000 --> 00:12:53.279
Uh so so the first type of a val is people evaluating how well agents build their products.

s136
00:12:53.279 --> 00:12:58.800
So everybody that's building a software product has some internal codebase or set of internal code bases.

s137
00:12:58.800 --> 00:13:14.639
Um I think ramp recently announced like rampbench which is aubench built off of their internal codebase and what that allows them to do is pick the coding agent or the model that performs best on their internal use cases and they don't have to you know maximize token spend.

s138
00:13:14.639 --> 00:13:19.839
instead they can make informed and educated decisions about how they build their products with agents.

s139
00:13:19.839 --> 00:13:27.760
The second type of AVL that we see people build is how to evaluate uh how well agents use your product.

s140
00:13:27.760 --> 00:13:34.800
So anybody that builds a software product is probably moving towards a world where they offer some sort of headless mode.

s141
00:13:34.800 --> 00:13:38.560
So you see some companies have obvious have always been this way.

s142
00:13:38.560 --> 00:13:43.440
They've been developer first like uh HubSpot and Stripe and things like that.

s143
00:13:43.440 --> 00:13:51.200
And the idea is uh if you can make an aval to see how well agents use your product and then iterate on your product to make it more usable for agents,

s144
00:13:51.200 --> 00:13:57.519
you're going to get more usage and therefore your product uh will become more valuable to agents.

s145
00:13:57.519 --> 00:14:06.079
And then uh three evaluate how well agents power product features and then four evaluate how well agents automate internal processes.

s146
00:14:06.079 --> 00:14:15.920
So depending on what type of company company you are, uh one or more of these might apply to you, but everybody should be considering right now how they can do one of these things.

s147
00:14:15.920 --> 00:14:20.880
Um okay, so what are the different use cases of the harbor rollout?

s148
00:14:20.880 --> 00:14:24.160
So we've talked about aval that's the most popular right now.

s149
00:14:24.160 --> 00:14:26.639
Um oh oh here we go.

s150
00:14:26.639 --> 00:14:30.560
I actually have a video even of us doing an eval in harbor.

s151
00:14:30.560 --> 00:14:34.800
So this is uh how you can run it from the command line or have your agent run it from the command line.

s152
00:14:34.800 --> 00:14:36.959
So you can see here it says harbor run.

s153
00:14:36.959 --> 00:14:46.240
We're running in this case terminal bench 2.1 with codeex and GPT 5.5. And we're saying let's run 64 in parallel.

s154
00:14:46.240 --> 00:15:02.000
And then we actually type in the launch command right here which is something we're rolling out soon which allows you to launch the rollouts on harbor servers which means you can just fire and forget and go to sleep and then come back the next day and 10,000 rollouts are done for you.

s155
00:15:02.000 --> 00:15:06.399
And you can kind of see in here what it looks like to look at those rollouts happening.

s156
00:15:06.399 --> 00:15:11.600
Um, but they don't complete in the time span of this video because rollouts can take a while.

s157
00:15:11.600 --> 00:15:20.800
Uh, which is another case for parallelizing as much as you possibly can to tighten that uh that loop and maximize your throughput.

s158
00:15:20.800 --> 00:15:26.399
Um, so there are people who create tasks and sell those as data.

s159
00:15:26.399 --> 00:15:33.440
So there's actually like a multibillion dollar market right now that exists probably around harbor data and also other types of data.

s160
00:15:33.440 --> 00:15:47.920
Um but it's uh very uh I guess lucrative right now where experts can specify certain types of tasks that they would like automated and then sell those to one of the labs that's training models to uh improve on that capability.

s161
00:15:47.920 --> 00:15:51.440
Uh you can also do what we call prod rollouts.

s162
00:15:51.440 --> 00:15:56.800
So remember harbor is evaluate any agent with any model.

s163
00:15:56.800 --> 00:16:04.320
Uh I guess not even evaluate we'll just say uh run any agent with any model in any sandbox on any task.

s164
00:16:04.320 --> 00:16:07.920
And that actually doesn't mean you have to do it for evaluation.

s165
00:16:07.920 --> 00:16:10.320
It also doesn't mean you have to do it for training.

s166
00:16:10.320 --> 00:16:19.199
It could literally be that you want to do what we've been calling agentic map reduce which is you just want to run a ton of agents on distributed compute.

s167
00:16:19.199 --> 00:16:25.440
of sandboxes in parallel and then somehow aggregate those results probably also intelligently.

s168
00:16:25.440 --> 00:16:37.360
So things like looking over a bunch of trajectories to detect reward hacking or I don't know like processing a bunch of receipts for reimbursements or searching over your obsidian files to figure out

s169
00:16:37.360 --> 00:16:43.040
uh semantically what where you wrote some note or processing a bunch of PRs to ask a question about it.

s170
00:16:43.040 --> 00:16:44.880
So we see a bunch of different use cases.

s171
00:16:44.880 --> 00:16:46.079
This is actually an emergent use case.

s172
00:16:46.079 --> 00:16:50.240
So, we didn't build Harbor for this, but we see people doing it a lot.

s173
00:16:50.240 --> 00:16:53.199
So, we actually built this feature and launched it just for this.

s174
00:16:53.199 --> 00:16:54.720
So, it's Harbor Exec.

s175
00:16:54.720 --> 00:17:01.120
Uh, it's going to go away kind of soon, but you can see in this scenario, actually, let me see if I can pause this.

s176
00:17:01.120 --> 00:17:06.400
So, uh, I'm actually going to go back a little bit before we kicked it off.

s177
00:17:06.400 --> 00:17:14.079
So, I'm saying Harbor Exec input is all of my codec sessions from the June 20s.

s178
00:17:14.079 --> 00:17:20.559
And then I say prompt if I corrected the agent write an analysis.json with mistake reason and correction.

s179
00:17:20.559 --> 00:17:26.640
And then my reduce prompt is summarize recurring mistakes and failure categories into a concise feedback.mmd file.

s180
00:17:26.640 --> 00:17:28.720
And you can see in this case I'm running on modal.

s181
00:17:28.720 --> 00:17:33.520
I do the map step with cursor CLI because it's cheap and fast.

s182
00:17:33.520 --> 00:17:40.559
And then I do the reduce step with fable 5 because I wanted to have like an accurate summary and I care about intelligence in that scenario.

s183
00:17:40.559 --> 00:17:46.799
So, and then I'm limiting it to 32 because I didn't actually want to process all of my sessions for this demo.

s184
00:17:46.799 --> 00:17:54.000
But you could there's no reason you couldn't do like 10,000 sessions or a thousand sessions and then maybe you have to go and understand the results more deeply.

s185
00:17:54.000 --> 00:17:57.039
But um Oh, shoot.

s186
00:17:57.039 --> 00:18:01.840
But uh yeah, you can kind of see here I think.

s187
00:18:01.840 --> 00:18:03.200
So we kick this off.

s188
00:18:03.200 --> 00:18:04.960
A bunch of these rollouts start in parallel.

s189
00:18:04.960 --> 00:18:09.919
This one is running locally on my computer with like cloud sandboxing but orchestrated locally.

s190
00:18:09.919 --> 00:18:16.480
And then you can see here all of the cursor rollouts running right now.

s191
00:18:16.480 --> 00:18:18.880
Uh but those will take a second to finish.

s192
00:18:18.880 --> 00:18:26.080
And then you can see the reduce step here where Fable recommended me like recurring mistakes.

s193
00:18:26.080 --> 00:18:36.400
And like these recurring mistakes, for example, could be used to inform the next batch of Harbor tasks that I create to evaluate my agent and pick like a better agent or train up a scale or something like that.

s194
00:18:36.400 --> 00:18:39.200
So uh another use case of Harbor.

s195
00:18:39.200 --> 00:18:42.960
And then we see people taking the trajectories and doing SFT.

s196
00:18:42.960 --> 00:18:51.520
And then we also see people taking the reward or rewards the trajectory in the form of tokens and doing actual reinforcement learning.

s197
00:18:51.520 --> 00:18:55.280
Uh for example, Tinker launched an integration with Harbor.

s198
00:18:55.280 --> 00:18:58.400
And then uh people also do other types of optimization.

s199
00:18:58.400 --> 00:19:10.880
So we mentioned JEA but any of these like evolutionary methods you take the text feedback in the form of a trajectory and the Aval and you can do some sort of like auto hill climbing with a harness or a scale.

s200
00:19:10.880 --> 00:19:15.760
Um so in the last couple minutes I just want to talk about some cool things people have built with Harbor.

s201
00:19:15.760 --> 00:19:17.280
I'll try to breeze through this.

s202
00:19:17.280 --> 00:19:29.440
Uh so uh Frontier Suite and ultra long horizon software engineering benchmark built on Harbor Banker toolbench and investment banking benchmark built by handshake on Harbor.

s203
00:19:29.440 --> 00:19:34.880
Um Swix saying that his team at Cognition migrated all their valves to harbor.

s204
00:19:34.880 --> 00:19:36.880
This Rune bench this is one of my favorites.

s205
00:19:36.880 --> 00:19:40.640
It's like a benchmark to measure how well agents can play Runescape.

s206
00:19:40.640 --> 00:19:45.679
Um Scale launched their whole Sweet Atlas suite on Harbor.

s207
00:19:45.679 --> 00:19:50.559
Uh, pool side uses Harbor to do all of their evaluations for model training.

s208
00:19:50.559 --> 00:19:58.160
Uh, Kevin Goo created auto agent which is a self-optimizing agent that all you have to plug in is a Harboral.

s209
00:19:58.160 --> 00:20:10.240
Uh, after query post train a model using Harbor Cognition released Frontier Code recently which is a Harbor benchmark and lang chain just integrated deep agents and their sandboxes into Harbor.

s210
00:20:10.240 --> 00:20:22.240
Uh and then actually just today, Snorkel released Senior Sweet Bench, which is a benchmark for measuring how well agents can function under ambiguity with behavioral feedback.

s211
00:20:22.240 --> 00:20:27.120
So, uh that's kind of a brief overview of Harbor and the different things you can do with it.

s212
00:20:27.120 --> 00:20:29.200
I would encourage everybody here to check it out.

s213
00:20:29.200 --> 00:20:32.240
And then this is a link to my Twitter.

s214
00:20:32.240 --> 00:20:39.200
And the reason I'm linking my Twitter, usually I link Harbor Docs, but we're actually actively trying to grow the Harbor team right now.

s215
00:20:39.200 --> 00:20:48.480
So if you want to get involved then shoot me a DM on Twitter and if you just want to see the docs then click the docs link on my Twitter and that's the best way to get there.

s216
00:20:48.480 --> 00:20:51.064
Uh so thank you very much

s217
00:20:51.064 --> 00:20:53.064
[applause]

s218
00:21:07.439 --> 00:21:09.439
[music]