WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=kRJjSOYqW0I

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/kRJjSOYqW0I.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.719 --> 00:00:13.840
I'll go ahead and get started.

s3
00:00:13.840 --> 00:00:16.240
Uh, my name is Xander Song.

s4
00:00:16.240 --> 00:00:25.760
I'm an engineer at AriseAI where I work on Phoenix, our open-source observability experimentation and evaluation platform.

s5
00:00:25.760 --> 00:00:38.160
And today I'm going to introduce you to Harbor, which is an open- source framework for evaluating agents on real tasks with real data in isolated sandboxed environments.

s6
00:00:38.160 --> 00:00:43.200
The theme of today's talk is shipping based on evidence rather than vibes.

s7
00:00:43.200 --> 00:00:51.520
And my goal is to give you a conceptual overview of how Harbor works as a framework and to share some of the ways in which my team is using Harbor

s8
00:00:51.520 --> 00:00:58.800
to iterate more quickly and with more confidence on our agents, harnesses, skills, prompts, and tools.

s9
00:00:58.800 --> 00:01:02.879
This talk is going to be a discussion rather than a interactive workshop.

s10
00:01:02.879 --> 00:01:16.159
But my hope is that you walk away with a concrete sense of how Harbor works as a framework and also a sense of how you might use it in your day-to-day work to make your life and your agents a little bit better.

s11
00:01:16.159 --> 00:01:17.360
A disclaimer before I begin.

s12
00:01:17.360 --> 00:01:19.520
I am not a maintainer of Harbor.

s13
00:01:19.520 --> 00:01:25.759
Um I do however consider myself to be an avid power user at this point.

s14
00:01:28.560 --> 00:01:30.000
So, so what is harbor?

s15
00:01:30.000 --> 00:01:39.200
At a high level, Harbor is a framework that allows you to define realistic tasks in the form of natural language instructions to an agent.

s16
00:01:39.200 --> 00:01:44.479
These are tasks that you would like your agent to be able to accomplish.

s17
00:01:44.479 --> 00:01:54.640
Harbor feeds those tasks to your agent running in a fresh sandboxed environment with real data relevant to the given task.

s18
00:01:54.640 --> 00:01:57.840
And then it enables you to define what are called verifiers.

s19
00:01:57.840 --> 00:02:08.560
These verifiers are specific to each task and they just check that your agent produced the intended side effects inside of the environment.

s20
00:02:08.560 --> 00:02:10.319
Verifiers produce rewards.

s21
00:02:10.319 --> 00:02:16.640
Examples of rewards are binary scores such as zero or one or pass fail.

s22
00:02:16.640 --> 00:02:19.280
So this picture is like really really simple, right?

s23
00:02:19.280 --> 00:02:24.239
But I want to just take a moment to appreciate why it's powerful.

s24
00:02:24.239 --> 00:02:38.160
Because if my task and my data are realistic and if I've done a good job of defining verifiers that accurately describe what constitutes success or failure on the given task,

s25
00:02:38.160 --> 00:02:46.000
what that means is that I can hold my task and my verifier constant while I change the implementation of my agent.

s26
00:02:46.000 --> 00:02:58.080
And by simply executing this pipeline from start to finish, I can quickly and efficiently gain a signal telling me whether or not the change I made to my agent

s27
00:02:58.080 --> 00:03:02.879
produced an improvement or a regression for this particular task.

s28
00:03:02.879 --> 00:03:06.400
Um, and not only that, I can take it a step further.

s29
00:03:06.400 --> 00:03:15.440
I can compile not just one task, but many tasks that cover the broad surface area of the capabilities of my agent.

s30
00:03:15.440 --> 00:03:17.519
And this lets me close the loop here.

s31
00:03:17.519 --> 00:03:29.200
So that now I'm actually able to start iterating with a degree of confidence that I might not be able to um if I was shipping based on vibes alone.

s32
00:03:30.560 --> 00:03:35.120
So that's really the idea of harbor in a nutshell.

s33
00:03:35.680 --> 00:03:37.120
Let's break it down a little bit further.

s34
00:03:37.120 --> 00:03:40.959
I'm going to talk about like the architectural details of how Harbor works.

s35
00:03:40.959 --> 00:03:48.080
So on this slide here, I have the task that I was talking about on the previous slide.

s36
00:03:48.080 --> 00:03:52.879
Each task specifies a set of instructions in the form of a markdown file.

s37
00:03:52.879 --> 00:03:58.239
And once again, that is just a piece of natural language that you're going to feed to your agent.

s38
00:03:58.879 --> 00:04:03.040
The task also describes an environment in which the agent will run.

s39
00:04:03.040 --> 00:04:05.920
And that typically takes the form of a Docker file.

s40
00:04:05.920 --> 00:04:10.640
And that Docker file typically contains data that is relevant to your task at hand.

s41
00:04:10.640 --> 00:04:13.040
And we'll give examples of this in just a minute.

s42
00:04:13.040 --> 00:04:20.799
But the thing to realize here is that um Harbor's actually agnostic with respect to what kind of data you give it.

s43
00:04:20.799 --> 00:04:23.520
So some examples could be like a SQL light file.

s44
00:04:23.520 --> 00:04:29.600
A very common example for coding agents would be like a git repository at a particular commit.

s45
00:04:30.400 --> 00:04:34.960
And Harbor also lets you define a set of verifiers inside of a test directory.

s46
00:04:34.960 --> 00:04:37.040
And as I said, verifiers are just scripts.

s47
00:04:37.040 --> 00:04:45.759
They're just scripts that run on your environment after your agent is done running that verify that the intended side effect took place.

s48
00:04:46.800 --> 00:04:54.800
And like I said, typically we don't have just one task, but we have many collected together in what is called a data set.

s49
00:04:54.800 --> 00:05:02.160
And for each task in the data set, Harbor allows you to run a configurable number of trials.

s50
00:05:02.160 --> 00:05:08.880
These trials are just independent executions of the task to account for the fact that LLMs are stochastic.

s51
00:05:08.880 --> 00:05:10.160
Agents are stochastic.

s52
00:05:10.160 --> 00:05:12.639
They don't behave the same every time.

s53
00:05:12.639 --> 00:05:20.639
So each trial is going to build a fresh environment in the form of a brand new Docker container seated with that data we were just talking about.

s54
00:05:20.639 --> 00:05:24.800
Harbor is going to drop your agent of choice into this environment.

s55
00:05:24.800 --> 00:05:29.600
It's going to feed it the instruction markdown file and it's going to let it do its thing.

s56
00:05:29.600 --> 00:05:35.520
And of course, the agent under test is going to produce side effects in the environment usually.

s57
00:05:35.520 --> 00:05:39.120
So if it's a coding agent, it's going to make changes to source files.

s58
00:05:39.120 --> 00:05:41.759
Um, it might make changes to a database if you put it in there.

s59
00:05:41.759 --> 00:05:44.560
It could do any number of things, right?

s60
00:05:44.560 --> 00:05:49.600
And after the agent finishes, the verifiers are going to run on the environment.

s61
00:05:49.600 --> 00:05:55.199
And again, a verifier is just a script that checks the environment.

s62
00:05:55.520 --> 00:06:06.560
Those verifiers produce rewards which get written to a reward.ext file and that that's the most important thing if you take away nothing else.

s63
00:06:06.560 --> 00:06:08.880
This is like the architecture of how harbor works.

s64
00:06:08.880 --> 00:06:20.479
There's another thing which is kind of an additional optional thing you can do which is you can configure your agents to emit so-called trajectories in the form of ATIF.

s65
00:06:20.479 --> 00:06:24.400
ATIF stands for agent trajectory interchange format.

s66
00:06:24.400 --> 00:06:38.000
ATIF is Harbor's specification for a JSON based tracing format which basically lets you peer into the internal details and the internal steps that your agent took to produce the side effect on the system.

s67
00:06:38.000 --> 00:06:44.160
So it's basically there for debugging purposes and for logs and and recording and all that.

s68
00:06:45.919 --> 00:06:56.800
this whole pipeline executing end to end where we take all the tasks in a data set run a certain number of trials for those tasks run our verifiers to produce rewards and collect trajectories

s69
00:06:56.800 --> 00:07:07.199
this whole thing all taking together is what Harper calls a job and there's one more thing I want to point out while I'm on this slide which is you'll notice right here

s70
00:07:07.199 --> 00:07:17.520
um I said I said agent/oracle um so Harbor also has this notion of an oracle you can drop not only an agent agent into the environment, but also an oracle.

s71
00:07:17.520 --> 00:07:20.880
And an oracle is just a solution to your task.

s72
00:07:20.880 --> 00:07:24.560
It's just a script that solves your task.

s73
00:07:24.560 --> 00:07:36.560
And we'll talk about oracles more in a minute, but oracles turn out to be like this really useful thing because oracles basically let you work out all the the the kinks in your pipeline before you ever throw an agent into the mix.

s74
00:07:36.560 --> 00:07:47.759
It lets you make sure that your data is good, that your environment is configured properly, that your verifiers work as expected, and you get to do all that before you spend any money on tokens for an agent.

s75
00:07:49.039 --> 00:07:50.960
All right, so that is like the high level.

s76
00:07:50.960 --> 00:07:58.720
Um, before I pause for questions, I will I will just continue on for just a second to kind of illustrate it with a concrete example.

s77
00:07:58.720 --> 00:08:01.520
So, I'm going to break down a concrete task.

s78
00:08:01.520 --> 00:08:12.960
Let's suppose that our agent that we care about is a coding agent um or a coding agent with a particular set of skills or a coding agent configured with [clears throat] like a custom MCP server that we're running.

s79
00:08:12.960 --> 00:08:17.759
And let's suppose that the task is to fix a failing migration.

s80
00:08:17.759 --> 00:08:25.520
So if that's the case, the instruction markdown file would you know it would it would be some set of instructions that you would pass to the coding agent as a user.

s81
00:08:25.520 --> 00:08:28.479
So it might say, "Hey, I'm noticing that this migration is failing.

s82
00:08:28.479 --> 00:08:29.840
Could you please try to fix it?"

s83
00:08:29.840 --> 00:08:30.639
Right?

s84
00:08:30.639 --> 00:08:37.039
And in the environment, the environment would contain perhaps an actual um SQL file.

s85
00:08:37.039 --> 00:08:38.800
Maybe it would contain some DDL.

s86
00:08:38.800 --> 00:08:43.039
Um and maybe it would contain some source code containing an up migration.

s87
00:08:43.039 --> 00:08:46.880
And we've, you know, we've seated the data so that the up migration fails.

s88
00:08:46.880 --> 00:08:51.680
Um in addition, we'll have our tests.

s89
00:08:51.680 --> 00:08:59.600
And again, the tests, they could be just scripts that run the up migration that ensures that the up migration exit with status code zero.

s90
00:08:59.600 --> 00:09:01.760
And then it maybe it checks like some tables.

s91
00:09:01.760 --> 00:09:10.160
Maybe it just checks the tables to make sure that the columns are there that we expect, the indexes are there that we expect to make sure that the um migration

s92
00:09:10.160 --> 00:09:14.399
has actually had the intended side effects on the system.

s93
00:09:15.279 --> 00:09:18.560
And you'll also notice that here optionally we have the solution, right?

s94
00:09:18.560 --> 00:09:27.760
So the solution is just going to be the literal solution, the literal uh script that you could run um that would fix the the migration.

s95
00:09:27.760 --> 00:09:30.560
And again, why would why would you go through the trouble of doing this?

s96
00:09:30.560 --> 00:09:40.240
Um, the reason again is it kind of lets you quickly and confidently build a little bit of assurance that you've got your system set up, that your verifiers are correct,

s97
00:09:40.240 --> 00:09:49.120
and you can it kind of lets you um vibe a little harder, I think, with the system because you can gain a certain amount of confidence in your verifiers

s98
00:09:49.120 --> 00:09:52.399
without necessarily like reviewing them line by line.

s99
00:09:52.399 --> 00:09:55.839
At least that's my experience with it.

s100
00:10:00.560 --> 00:10:13.519
And so after the the agent runs the task in the environment, um it's going to edit that broken migration file, it's going to leave a couple modifications on the file system

s101
00:10:13.519 --> 00:10:16.800
and then the verifiers are going to run.

s102
00:10:16.800 --> 00:10:28.880
And typically if you're running um Harbor in Docker, which is like the default local setting, those verifiers are going to mount in the Docker container after the agent finishes.

s103
00:10:28.880 --> 00:10:30.959
And that's to prevent the agent from cheating.

s104
00:10:30.959 --> 00:10:38.320
So the the agent doesn't have access to the verifiers and the solutions um during the actual execution.

s105
00:10:38.959 --> 00:10:41.600
Those verifiers are going to run the sort of checks that we talked about.

s106
00:10:41.600 --> 00:10:43.680
Maybe it literally runs the up migration.

s107
00:10:43.680 --> 00:10:48.160
Maybe it um you know checks the tables, checks the indexes.

s108
00:10:48.160 --> 00:10:57.440
And I think the thing to really take away from this slide is um the verifiers are agnostic of the implementation of your agent.

s109
00:10:57.440 --> 00:11:00.480
They actually don't know anything about the agent at all.

s110
00:11:00.480 --> 00:11:04.880
All they care about is what the agent leaves behind.

s111
00:11:04.880 --> 00:11:14.399
So it is possible to like write some verifiers that are not like that that peak into the ATIF trajectory that kind of look at the implementation or the execution of the agent

s112
00:11:14.399 --> 00:11:16.079
and evaluate it that way.

s113
00:11:16.079 --> 00:11:21.040
But that's not really the canonical kind of verifier.

s114
00:11:21.040 --> 00:11:32.000
The verifiers, you know, canonically treat the agent as a little bit of a black box and they're grading the agent solely based on its side effects and its output in the environment

s115
00:11:32.000 --> 00:11:37.440
rather than the exact manner in which it produced those side effects and outcome.

s116
00:11:37.440 --> 00:11:47.120
And there's kind of a natural parallel here with integration tests because you can imagine that if you have some integration tests for an interface or an API, those integration tests

s117
00:11:47.120 --> 00:11:53.839
um if they're really well written are hopefully agnostic of the implementation that is behind that interface or API.

s118
00:11:53.839 --> 00:12:06.079
And in the same way, if we have really good verifiers, our verifiers are kind of describing what we expect the agent to do, the side effects we expected to produce in the environment.

s119
00:12:06.079 --> 00:12:09.519
And they're doing so in a way that is decoupled from the implementation.

s120
00:12:09.519 --> 00:12:16.720
And that's a really nice feature to have for a set of tests because it allows you to refactor under green, for example.

s121
00:12:16.720 --> 00:12:30.480
It allows you to hold your tasks constant while you like completely change the implementation of your agent and then it allows you to like fire this pipeline really quickly to figure out if everything still works.

s122
00:12:30.480 --> 00:12:37.120
That's kind of what we're talking about when we're talking about iterating quickly um and shipping confidently.

s123
00:12:37.360 --> 00:12:46.079
So before I go any further, let me just pause right here for questions or comments or um Yeah, let me just pause before I go any further.

s124
00:12:46.079 --> 00:12:46.240
Yeah.

s125
00:12:46.240 --> 00:12:46.399
Yeah.

s126
00:12:46.399 --> 00:12:46.639
Yeah.

s127
00:12:46.639 --> 00:12:47.279
You just Yeah.

s128
00:12:47.279 --> 00:12:48.959
Again, it's just a Docker container.

s129
00:12:48.959 --> 00:12:50.943
So, you just pop whatever you want in there, right?

s130
00:12:50.943 --> 00:12:51.120
[laughter]

s131
00:12:51.120 --> 00:12:54.399
It's a primitive that you're you're proposing.

s132
00:12:54.399 --> 00:12:54.560
Yeah.

s133
00:12:54.560 --> 00:12:54.720
Yeah.

s134
00:12:54.720 --> 00:12:54.880
Yeah.

s135
00:12:54.880 --> 00:12:55.120
Yeah.

s136
00:12:55.120 --> 00:12:55.279
Yeah.

s137
00:12:55.279 --> 00:12:55.440
Yeah.

s138
00:12:55.440 --> 00:12:57.760
So, you know, it could be it could be any framework, right?

s139
00:12:57.760 --> 00:12:59.360
Uh it could be any language.

s140
00:12:59.360 --> 00:13:01.600
It could be any piece of code, any executable.

s141
00:13:01.600 --> 00:13:05.440
Concretely, like, you know, I'll give you a little sneak sneak peek or sneak preview, right?

s142
00:13:05.440 --> 00:13:11.440
Like, we've had to like change tools like um you know, adjust adjust prompts, right?

s143
00:13:11.440 --> 00:13:17.279
Um, I'm going through a process right now where I'm trying to consolidate our skills, for example.

s144
00:13:17.279 --> 00:13:22.560
So, we kind of have like uh several different agent experiences that we try to support.

s145
00:13:22.560 --> 00:13:29.200
Um, there's a lot of overlap and then there's like certain differences and we're trying to like basically get our skills all kind of like reusable, right?

s146
00:13:29.200 --> 00:13:38.560
Um, and agnostic of the particular like direction that the agent is coming from, whether it's MCP or a CLI or via like our our own like server side agent.

s147
00:13:38.560 --> 00:13:48.000
we're trying to get it agnostic and and so those are the kinds of things where it kind of helps to be able to specify this set of tests that is like completely agnostic

s148
00:13:48.000 --> 00:14:00.800
of the underlying implementation, you know, and we we kind of we came to this like maybe um through the pain of doing it the other way where we were like kind of like had a set of tests that were um that were coupled

s149
00:14:00.800 --> 00:14:11.920
and it was becoming difficult to maintain because you know I would change a tool and the tests would break right even though the not even though the true functionality of the agent was still intact, right?

s150
00:14:11.920 --> 00:14:12.720
Things like that.

s151
00:14:12.720 --> 00:14:14.240
Um, we'll get to more of these, right?

s152
00:14:14.240 --> 00:14:15.920
We'll get to more of these at the end.

s153
00:14:15.920 --> 00:14:19.600
Um, any other questions before I go on?

s154
00:14:19.680 --> 00:14:21.680
Okay.

s155
00:14:21.839 --> 00:14:22.480
Okay.

s156
00:14:22.480 --> 00:14:25.760
So, uh, this this slide is going to talk about scale.

s157
00:14:25.760 --> 00:14:34.959
Um, so as we as we already talked about, right, each task and each trial, uh, is independent, runs in its own environment, its own docker container.

s158
00:14:34.959 --> 00:14:39.680
Um, and that makes this whole pipeline completely horizontally scalable.

s159
00:14:39.680 --> 00:14:46.959
And that means that it's very compatible with a sandbox provider such as Daytona or Modal or E2B or any of those.

s160
00:14:46.959 --> 00:14:52.560
Um, and it means you can kind of rip through these tests like really fast if you want to, which is kind of fun.

s161
00:14:52.560 --> 00:15:00.959
So in this diagram arrayed vertically, we have a set of four tasks in a data set, including the fixed migration task that we were just imagining.

s162
00:15:00.959 --> 00:15:04.320
Um, we've got a couple of different coding agents that we might want to test out.

s163
00:15:04.320 --> 00:15:10.800
claude code with opus 5 codeex with gbt5 and then we've got um a couple of different trials.

s164
00:15:10.800 --> 00:15:20.959
Let's suppose that we're doing three trials for k equals 1 2 3. Again, the point of that is just to kind of handle the handle the fact that these things are stocastic and they they're not perfect.

s165
00:15:20.959 --> 00:15:23.760
They don't behave the same every time.

s166
00:15:23.760 --> 00:15:32.880
Um so because all these are independent um you can run them all in parallel on these sandbox providers.

s167
00:15:32.880 --> 00:15:39.040
So we use Daytona um and it runs quite fast with like 200 concurrent workers.

s168
00:15:39.040 --> 00:15:42.560
So even if you have a large test suite you can rip through it quickly.

s169
00:15:42.560 --> 00:15:51.920
Um and then from these individual trials you are able to compute aggregate metrics for the kinds of things that you care about.

s170
00:15:51.920 --> 00:16:00.160
So for example you could compute a pass at Kmetric that just tells you in this case did I pass after three attempts at the at the task.

s171
00:16:00.160 --> 00:16:11.199
And so I can see that oh for like this particular task the fixed migration task both of my agents like passed at K right passed at three uh for this one only one of my two agents passed at three.

s172
00:16:11.199 --> 00:16:27.360
You can actually also compute those kind of aggregate metrics over the entire data set right and so this kind of like gives you this very quick picture of how your change is performing because again if you have like a a good data set that covers the set of functionality that you care about you can get these aggregate

s173
00:16:27.360 --> 00:16:29.199
metrics pretty pretty rapidly.

s174
00:16:29.199 --> 00:16:29.440
Yeah.

s175
00:16:29.440 --> 00:16:30.240
Go ahead.

s176
00:16:30.240 --> 00:16:31.600
Each Yeah, that's a great question.

s177
00:16:31.600 --> 00:16:33.360
So, this is a really important point, right?

s178
00:16:33.360 --> 00:16:46.240
Is each of these boxes is a totally separate Docker container, identical in the beginning, but then you you put an agent, a separate agent into each one, and you let it go and do its thing, and they do different things, right?

s179
00:16:46.240 --> 00:16:48.880
Um, so each one is totally independent.

s180
00:16:48.880 --> 00:16:50.480
They're completely isolated, right?

s181
00:16:50.480 --> 00:16:55.120
They're independent trials of the same task with the same agent.

s182
00:16:55.120 --> 00:17:04.720
Um you're basically run yeah you're basically running claude code three times and saying hey try to fix this migration and then you're saying oh after three tries did claude code fix the migration.

s183
00:17:04.720 --> 00:17:08.720
Okay so now now I've now we've talked about harbor at a at a high level.

s184
00:17:08.720 --> 00:17:13.439
Um but I want to switch gears and I want to talk about how Harbor plugs into Phoenix.

s185
00:17:13.439 --> 00:17:15.360
So again I'm not a maintainer of Harbor.

s186
00:17:15.360 --> 00:17:16.799
Um I'm just a fan.

s187
00:17:16.799 --> 00:17:24.400
Uh but I am a maintainer of of Phoenix uh which is our which is ARIS's open source AI observability tool.

s188
00:17:24.400 --> 00:17:33.840
And the point of this slide is just to basically convey that we have analogous concepts for the primitives in Harbor.

s189
00:17:33.840 --> 00:17:37.760
So we're able to map your data sets to our notion of a data set.

s190
00:17:37.760 --> 00:17:40.160
We're able to track your experiments.

s191
00:17:40.160 --> 00:17:43.440
We're able to um record your trials.

s192
00:17:43.440 --> 00:17:50.640
Uh we're able to convert your rewards into our notion of reward which we call an evaluation.

s193
00:17:50.640 --> 00:18:00.240
Um, and I guess the other thing I I want to take away from this slide is also we we do something where we convert those trajectories that I talked about into

s194
00:18:00.240 --> 00:18:02.960
so-called open inference format.

s195
00:18:02.960 --> 00:18:24.080
Um so open inference is a open standard of semantic conventions so-called semantic conventions which is built on top of open telemetry which is an open source framework for telemetry data such as logs metrics and traces

s196
00:18:24.080 --> 00:18:41.280
and these conventions basically define data structures that uh your AI I application can emit which then downstream consumers such as Phoenix can ingest and we can use those to build certain kinds of UI

s197
00:18:41.280 --> 00:18:44.559
experiences and workflows that wouldn't otherwise be possible.

s198
00:18:44.559 --> 00:18:47.039
So full disclosure, I am a maintainer of both imprints.

s199
00:18:47.039 --> 00:18:51.280
Um we recently donated this the spec to cloud native, right?

s200
00:18:51.280 --> 00:18:53.520
Um and to the hotel working group.

s201
00:18:53.520 --> 00:18:55.919
So it has some traction outside of arise.

s202
00:18:55.919 --> 00:19:00.559
And so that's maybe a reason you might want to um use open inference.

s203
00:19:00.559 --> 00:19:07.039
Um so this diagram on the left, you can see the ATIF trajectory for that fixed migration task.

s204
00:19:07.039 --> 00:19:14.640
you can see the individual steps that the agent might take um including an LLM call and including a tool call.

s205
00:19:14.640 --> 00:19:20.320
And over on the right, uh this is kind of what it winds up looking like in open inference.

s206
00:19:21.760 --> 00:19:31.440
And in the product, like I said, open inference allows us to build out the kinds of opinionated UIs and workflows that you see here.

s207
00:19:31.440 --> 00:19:38.720
So we can do things like we can render your trajectories with some opinionated UI, some specialized UI for LLMs and tools.

s208
00:19:38.720 --> 00:19:52.240
We can version your tasks and your data sets so that at any given time you know exactly what the task look like for a particular job in Harbor.

s209
00:19:52.320 --> 00:19:57.840
We track your experiments and we help you visualize metrics and rewards as you iterate.

s210
00:19:57.840 --> 00:20:07.200
So in this case the reward you could imagine it being pass fail um zero or one and you can run a sequence of experiments and you can kind of see over time

s211
00:20:07.200 --> 00:20:10.400
is my reward moving in the right direction.

s212
00:20:10.400 --> 00:20:18.969
We also have skills CLIs and an MCP server so that claude and codecs and cursor can easily get access to this data.

s213
00:20:18.969 --> 00:20:25.360
[clears throat] And at this point I do want to note that Harbor does have its own open- source UI um that you can run locally on your machine.

s214
00:20:25.360 --> 00:20:27.200
It also has a hosted web service.

s215
00:20:27.200 --> 00:20:30.640
Um, so you you definitely don't have to use Phoenix to use Harbor.

s216
00:20:30.640 --> 00:20:36.640
Um, but if you're interested in some of the features I just mentioned, that would be a reason to give Phoenix a try.

s217
00:20:37.760 --> 00:20:43.360
Okay, so hopefully at this point I've given you kind of a bird's eye view of how everything works.

s218
00:20:43.360 --> 00:20:47.120
Now I want to talk about some of the ways in which our team has been using Harbor.

s219
00:20:47.120 --> 00:20:52.960
And hopefully this answers um answers your point over here just around getting some concrete examples.

s220
00:20:52.960 --> 00:20:59.520
Um so in order to do that I just want to give kind of this high level of like what our architecture works with.

s221
00:20:59.520 --> 00:21:01.360
So again this is the Phoenix architecture.

s222
00:21:01.360 --> 00:21:08.720
This is particular to our team right this is the this is what our team is kind of like dealing with on a day-to-day basis.

s223
00:21:08.720 --> 00:21:16.320
Um on the left here you have an enduser's AI application or agent.

s224
00:21:16.320 --> 00:21:22.080
And again this application is going to have been instrumented with open telemetry and open inference.

s225
00:21:22.080 --> 00:21:28.880
And it's going to be exporting and emitting traces to the Phoenix server over OTLP, which is the open telemetry protocol.

s226
00:21:28.880 --> 00:21:31.840
So, it's basically sending data to Phoenix, right?

s227
00:21:31.840 --> 00:21:38.080
Saying, "Hey, I'm doing this and this is like what my these are the sequence of steps that I did to accomplish this task.

s228
00:21:38.080 --> 00:21:39.679
This could be like a production application.

s229
00:21:39.679 --> 00:21:42.960
It could be an application that you're testing out.

s230
00:21:42.960 --> 00:21:50.640
Those traces are ingested by Phoenix and they are written to one of our SQL backends, either SQLite or Postgres.

s231
00:21:50.640 --> 00:22:00.720
And once the data is in the system, we have a set of core services for things like traces, spans, sessions, data sets, experiments, and evaluations.

s232
00:22:00.720 --> 00:22:06.799
And these core services are exposed via various kinds of APIs, REST or GraphQL.

s233
00:22:06.960 --> 00:22:18.240
But our goal is not only to like make those programmatic interfaces, but it's also to make it easy for our end users and for their coding agents and other kinds of agents to get access to the data and use the data.

s234
00:22:18.240 --> 00:22:23.919
And so we support several different kinds of agent experiences.

s235
00:22:23.919 --> 00:22:27.840
Um the first of which are coding agents.

s236
00:22:27.840 --> 00:22:35.200
So for coding agents, we have a CLI that the coding agent can use which will invoke REST or it will invoke GraphQL.

s237
00:22:35.200 --> 00:22:44.080
We also have an MCP server that you can configure your coding agent to use um which is going to do the same things.

s238
00:22:44.080 --> 00:22:57.200
And we have a third agent interface that we maintain for the team which is a server side agent that we call Pixie which also has access to the MCP server and therefore has access to REST and GraphQL.

s239
00:22:57.200 --> 00:22:59.679
Um it also has like a bash emulator.

s240
00:22:59.679 --> 00:23:03.039
So it's it's kind of like a coding agent but running server side.

s241
00:23:03.039 --> 00:23:10.799
Um so these are the three surfaces that we that we want to support and that we have to test and ensure are working.

s242
00:23:10.799 --> 00:23:19.760
the coding agent with the CLI, the coding agent with an MCP server, and a server side agent, which is our own harness.

s243
00:23:20.960 --> 00:23:27.679
And when it comes time to actually like test all of this out, architecturally, we have a pretty big advantage.

s244
00:23:27.679 --> 00:23:31.120
We have this really nice feature, which is that this server runs on a single node.

s245
00:23:31.120 --> 00:23:33.679
It's kind of like a graphana- like architecture.

s246
00:23:33.679 --> 00:23:36.240
Um, and that means it's very easy to dockerize.

s247
00:23:36.240 --> 00:23:39.440
So, you just pop the whole thing in a Docker container.

s248
00:23:39.440 --> 00:23:44.799
What we do is we just take a SQLite file which contains some realistic data that we've tested on.

s249
00:23:44.799 --> 00:23:49.440
Um we replace the browser with a test client.

s250
00:23:49.440 --> 00:23:53.120
Um and we just put it in Docker.

s251
00:23:54.159 --> 00:24:01.280
And once it's in Docker, we can start using it in Harbor.

s252
00:24:01.280 --> 00:24:06.720
So we can feed the system the kinds of tasks that our particular users care about.

s253
00:24:06.720 --> 00:24:19.360
So, our particular users might ask a coding agent or they might ask our Pixie server agent, find some hallucinated LLM calls from the last week of my traces and add them to a data set.

s254
00:24:19.360 --> 00:24:22.480
That's a realistic query that one of our users might ask.

s255
00:24:22.480 --> 00:24:32.400
And we can go ahead and let any of these three kinds of agents go wild inside of this environment.

s256
00:24:32.400 --> 00:24:35.840
And at the end, we can run our verifiers.

s257
00:24:35.840 --> 00:24:46.640
And the kinds of verifiers that we can run for us in this particular setup are really quite simple because we have a server running inside of here with the SQL life file.

s258
00:24:46.640 --> 00:24:58.480
So we could either hit a rest endpoint to verify that the intended side effect has changed or we could literally like query the SQL life file if we wanted to because it's just in the container.

s259
00:24:58.480 --> 00:25:07.600
Um and just verify for example that the particular tupil for this data sets table has been written in the way that we expect.

s260
00:25:07.600 --> 00:25:07.840
Yeah.

s261
00:25:07.840 --> 00:25:08.080
Yeah.

s262
00:25:08.080 --> 00:25:10.720
No, thanks for stopping me because I think there's a lot going on on this slide.

s263
00:25:10.720 --> 00:25:14.080
So maybe I should pause and and kind of like make sure it's clear before I go on.

s264
00:25:14.080 --> 00:25:14.400
Right.

s265
00:25:14.400 --> 00:25:19.200
Um, so I think there were a couple of questions there and I'll I'll try and address address them.

s266
00:25:19.200 --> 00:25:26.480
Um, so in this slide there are three different agents in in any given trial.

s267
00:25:26.480 --> 00:25:29.440
I would only be testing one of them if that makes sense.

s268
00:25:29.440 --> 00:25:38.799
So you could imagine maybe I'm only testing Claude code with the CLI, maybe I'm only testing Claude code with MCP or maybe I'm only testing you know our Pixie server

s269
00:25:38.799 --> 00:25:39.760
server agent.

s270
00:25:39.760 --> 00:25:45.600
So just may let's imagine like it's just one at a time in terms of how does harbor feed at the prompt.

s271
00:25:45.600 --> 00:25:49.840
Harbor and this is maybe a nuance that I haven't really talked about, right?

s272
00:25:49.840 --> 00:25:53.679
Harbor itself um does not Harbor is a program.

s273
00:25:53.679 --> 00:25:55.200
It's a command line tool.

s274
00:25:55.200 --> 00:26:02.240
Um and it doesn't actually run in the environment per se in the Docker environment per se.

s275
00:26:02.240 --> 00:26:11.039
It runs on a host machine which could be your laptop if you're running it locally or it runs in like a GitHub action runner, you know, if you're running it in CI.

s276
00:26:11.039 --> 00:26:15.840
Um, and you can run the environment in a Docker container locally.

s277
00:26:15.840 --> 00:26:17.600
That's like the default behavior.

s278
00:26:17.600 --> 00:26:22.960
But if you're trying to do something more scalable, you would be running it on a sandbox provider in all likelihood.

s279
00:26:22.960 --> 00:26:25.279
Um, so you'd be running it on Daytona.

s280
00:26:25.279 --> 00:26:34.799
And so Harbor is this orchestrator um which is running on your laptop or running in GitHub actions which is um kicking off the run.

s281
00:26:34.799 --> 00:26:38.720
It's basically saying, you know, hey, start start the agent, right?

s282
00:26:38.720 --> 00:26:49.919
Um, and the way it does that is, and I can show you what this looks like on the file system, but it basically, you know, um, expects a particular script.

s283
00:26:49.919 --> 00:26:53.840
Harbor expects a particular script to be written for the task, right?

s284
00:26:53.840 --> 00:26:55.520
It's all driven via scripts.

s285
00:26:55.520 --> 00:27:02.640
So, you like when you're when you're setting up Harbor, you have to kind of make sure that all these scripts are in the right place in the Docker image.

s286
00:27:02.640 --> 00:27:05.039
Um, so there's going to be a script to start the task.

s287
00:27:05.039 --> 00:27:06.880
There's going to be a script to run the verifiers.

s288
00:27:06.880 --> 00:27:08.799
There's going to be a script to run the oracles.

s289
00:27:08.799 --> 00:27:10.000
It's all driven by scripts.

s290
00:27:10.000 --> 00:27:12.640
So the scripts are in the Docker container.

s291
00:27:12.640 --> 00:27:20.240
Um, but doc but Harborself is a program that lives on a on a host machine and just kind of orchestrates things and kicks things off.

s292
00:27:20.240 --> 00:27:20.799
That's right.

s293
00:27:20.799 --> 00:27:21.760
And the interf Yeah.

s294
00:27:21.760 --> 00:27:39.279
So maybe I'll just repeat what he what you just said because I think it's well put is like Harbor is a CLI um and it has certain conventions for if you like make your docker container or your docker image look this way it will be able to run it um and those conventions are almost all like

s295
00:27:39.279 --> 00:27:52.559
is there a script in the right place so that's kind of like the contract right if there was a a contract that you had to honor with harbor so that your your image would run it would be like did I put my scripts in the right place right That's kind of the contract.

s296
00:27:52.559 --> 00:27:53.120
Got it.

s297
00:27:53.120 --> 00:27:53.520
Yeah.

s298
00:27:53.520 --> 00:27:54.960
Does that Does that clarify?

s299
00:27:54.960 --> 00:27:55.360
Yes.

s300
00:27:55.360 --> 00:27:55.760
Okay.

s301
00:27:55.760 --> 00:27:56.080
Yeah.

s302
00:27:56.080 --> 00:27:58.399
So again on this one on this slide, right?

s303
00:27:58.399 --> 00:28:09.919
Basically I think the takeaway is um the takeaway on this slide is like you have a great deal of freedom um and flexibility when it comes to customizing your harbor environment, right?

s304
00:28:09.919 --> 00:28:18.240
Um the the simplest like the simplest example that people always get started with is the example of a coding agent operating on a git repo.

s305
00:28:18.240 --> 00:28:20.240
But this example is different, right?

s306
00:28:20.240 --> 00:28:23.919
This example is like I literally put some data from a SQL life file in there.

s307
00:28:23.919 --> 00:28:29.919
I threw my whole application in there with it along with its APIs along with its own serverside agent.

s308
00:28:29.919 --> 00:28:33.600
It's all it's a running server in the environment in the Docker container.

s309
00:28:33.600 --> 00:28:37.360
Um something you know there's a CLI in the environment.

s310
00:28:37.360 --> 00:28:45.279
So what's not even included here is like there's actually a set of skills literally in the environment on the file system that the agent has access to.

s311
00:28:45.279 --> 00:28:52.000
Um, so all that is to say is like you have a lot of flexibility in setting up your environment.

s312
00:28:52.000 --> 00:29:01.520
Um, you know, it has to fit in a Docker image, but if you can fit it in a Docker image, um, you know, you can you can make Harbor run it basically.

s313
00:29:04.399 --> 00:29:04.720
All right.

s314
00:29:04.720 --> 00:29:08.320
And now any other any other questions before I move on?

s315
00:29:08.320 --> 00:29:09.760
Maybe because this is a complex slide.

s316
00:29:09.760 --> 00:29:18.320
I think this is and and you know this is like again this slide is very much like particular to to our project right this is like what our project

s317
00:29:18.320 --> 00:29:32.640
looks like your project may not look like this right if you if you're not running like a single node server or something but yeah I mean that's such a good question and I haven't had to deal with that problem so it would require a lot of thought I think right but basically the question is like

s318
00:29:32.640 --> 00:29:44.799
my system is not a single node it's not a single you know I can't just like throw in a SQL single SQLite file I have like you know, some distributed system with like S3 and all this other stuff that I'm trying to maintain.

s319
00:29:44.799 --> 00:29:46.399
How would I test harbor out with that?

s320
00:29:46.399 --> 00:29:48.640
The answer is like that's probably harder.

s321
00:29:48.640 --> 00:29:50.000
Yeah, that's probably harder.

s322
00:29:50.000 --> 00:29:59.600
So, in that case, in that case, my initial thought is like, you know, you have to perhaps carve out your agent a little more, right?

s323
00:29:59.600 --> 00:30:05.039
So, here we've had this advantage where like it's simple for us to throw the whole damn thing in there in the Docker container.

s324
00:30:05.039 --> 00:30:06.880
And so, we did.

s325
00:30:06.880 --> 00:30:20.559
If that wasn't simple to do, if you can modularize your agent somehow into some kind of like interface, hide hide the implementation behind an interface um and if you can like if there's some like stable API boundary or interface

s326
00:30:20.559 --> 00:30:22.720
that is relatively stable, right?

s327
00:30:22.720 --> 00:30:28.480
Where you know um I'm feeding like this agent data in this format, right?

s328
00:30:28.480 --> 00:30:34.480
Whatever that format may be, like eventually this agent has access to data in this format, right?

s329
00:30:34.480 --> 00:30:39.600
Um, that would be one approach, right?

s330
00:30:39.600 --> 00:30:50.240
Um, and I realize that that is maybe not a satisfactory answer depending on what your exact setup looks like because maybe your agent like literally queries S3 and then like what are you to do, right?

s331
00:30:50.240 --> 00:30:52.080
Are you going to mock out all those S3 calls?

s332
00:30:52.080 --> 00:30:58.080
And then once you start mocking out S3 calls, you're kind of in a world of pain because now you're in the world of mocking stuff again, right?

s333
00:30:58.080 --> 00:31:01.360
And and that's really like what we're trying to get away from with Harbor.

s334
00:31:01.360 --> 00:31:03.120
We were trying to get away from that world.

s335
00:31:03.120 --> 00:31:08.320
Um, so I don't know that there's like I actually don't know because I haven't had to explore this problem yet.

s336
00:31:08.320 --> 00:31:19.919
Um, I do know that you know I've been kind of like eliding this particular feature of Harbor, but Harbor does have the ability to um like run multiple Docker images, right?

s337
00:31:19.919 --> 00:31:21.120
Spin up multiple things.

s338
00:31:21.120 --> 00:31:28.720
So like you could you could wonder I would wonder about that is like oh is there a way where I could somehow encapsulate my application or something?

s339
00:31:28.720 --> 00:31:29.760
Yeah, go ahead.

s340
00:31:29.760 --> 00:31:31.840
The question was like what scale do you use?

s341
00:31:31.840 --> 00:31:32.000
Right?

s342
00:31:32.000 --> 00:31:44.480
if you have some massive SQL life file or some you know really large set of traces do you like actually evaluate on the whole thing we don't currently do that and I think it would probably be okay for us to do that um

s343
00:31:44.480 --> 00:31:57.919
right now we you know right now the kind of format is like oh like we have some data here I'm going to have Claude kind of help me clean it up a little bit for this particular set of tasks and again the kind of the nice thing about Harbor is you know you you can

s344
00:31:57.919 --> 00:32:02.960
um use the exact same data for many different tasks but you don't have to.

s345
00:32:02.960 --> 00:32:13.679
Um, and so if there's a particular question that you care about and it only requires a small subset of the data to answer, you don't have to like upload all of the data to answer that particular question.

s346
00:32:13.679 --> 00:32:19.360
Um, but yeah, I mean it it just depends too like on what you're trying to test, I think, right?

s347
00:32:19.360 --> 00:32:29.679
If part of what you were trying to test was the performance of your agent when the system is full, when the database is like quite full, then I think it would totally make sense to um

s348
00:32:29.679 --> 00:32:30.880
put in a lot of data.

s349
00:32:30.880 --> 00:32:32.399
We we haven't done that yet.

s350
00:32:32.399 --> 00:32:34.240
Well, I tried to Yeah, that's such a good question.

s351
00:32:34.240 --> 00:32:37.919
I think the question is like what is what what do you get by using Phoenix with Harbor?

s352
00:32:37.919 --> 00:32:39.760
Why not just use Harbor maybe?

s353
00:32:39.760 --> 00:32:43.279
Um and the answer is like you totally can use just Harbor, right?

s354
00:32:43.279 --> 00:32:49.919
I'll I'll give you a little demo um in just a second where hopefully some of some of what we offer is is on display, right?

s355
00:32:49.919 --> 00:33:00.080
Because I I do think that there are some genuinely unique things around like data set versioning, around the experiments, around the visualizations, around the the agent experiences that you get.

s356
00:33:00.080 --> 00:33:04.480
um the ability to like actually inspect and drill down quickly, right?

s357
00:33:04.480 --> 00:33:12.320
So, we've been we've been solving those kinds of problems um for several years and and our UI is like really dialed in for that sort of thing.

s358
00:33:12.320 --> 00:33:14.080
And I'll give you a demo in just a sec.

s359
00:33:14.080 --> 00:33:14.799
Yeah.

s360
00:33:14.799 --> 00:33:18.240
And then you can ask me again, see if I answered the question.

s361
00:33:18.240 --> 00:33:18.880
All right.

s362
00:33:18.880 --> 00:33:21.679
Uh All right.

s363
00:33:21.679 --> 00:33:22.880
I'm going to go to the next slide.

s364
00:33:22.880 --> 00:33:23.440
Right.

s365
00:33:23.440 --> 00:33:30.320
So this particular example that we're going to think about is um this problem that we encountered recently.

s366
00:33:30.320 --> 00:33:35.760
Uh the problem was you know in this picture that I've shown you I talked about rest and I talked about GraphQL.

s367
00:33:35.760 --> 00:33:45.519
So our MCP server our CLIs they had access to these APIs but we found that people were asking questions that are not easily answerable via REST or GraphQL.

s368
00:33:45.519 --> 00:33:48.320
So they were asking aggregate kind of questions, right?

s369
00:33:48.320 --> 00:33:54.640
They were asking, you know, what was the busiest day that I had in the last few months or something, right?

s370
00:33:54.640 --> 00:33:56.880
Um how many traces did I get on that day?

s371
00:33:56.880 --> 00:34:01.440
Or they were asking questions like how much am I spending on uh each of these models, right?

s372
00:34:01.440 --> 00:34:05.600
Like which model is actually costing me the most out of all my LLM calls, right?

s373
00:34:05.600 --> 00:34:07.200
Where am I spending the most?

s374
00:34:07.200 --> 00:34:17.280
Um and so what the agents would try to do is they would uh you know when possible they would uh look for the the rest route or the uh

s375
00:34:17.280 --> 00:34:18.960
the GraphQL resolver.

s376
00:34:18.960 --> 00:34:20.879
Um but these were not aggregate resolvers.

s377
00:34:20.879 --> 00:34:26.720
So they would try to like basically pageionate through stuff and and do the aggregation themselves.

s378
00:34:26.720 --> 00:34:34.159
And as you can imagine that results in a problem because if there's enough data you wind up uh you wind up doing a lot of a lot of pageionations

s379
00:34:34.159 --> 00:34:35.679
right and a lot of tool calls.

s380
00:34:35.679 --> 00:34:42.879
Um, and so part of what we did was we actually added this this line right here.

s381
00:34:42.879 --> 00:34:57.200
So this line uh is meant to represent a tool that we added to our MCP server um, which basically allows the agent to write a readonly SQL query and execute it against the database.

s382
00:34:57.200 --> 00:35:01.440
Um, and a lot went into making that tool.

s383
00:35:01.440 --> 00:35:13.839
I won't really go into the implementation details but the thing that we noticed even after we made that tool was that oh sometimes sometimes the agent isn't calling the right tool because now the agent has this other problem right

s384
00:35:13.839 --> 00:35:26.960
now the agent is here and it's like well I got rest I got graphql and I got SQL which one which one should I be using the agent doesn't really know and so we realized oh yeah well naturally we have to like somehow update a prompt we got to tell it what to do right

s385
00:35:26.960 --> 00:35:31.920
um but even after you tell it what to do how do you verify that it's actually doing it Okay.

s386
00:35:31.920 --> 00:35:36.560
Um, you know, it's one thing to change the prompt, but again, we're trying to like ship based on some evidence, right?

s387
00:35:36.560 --> 00:35:38.240
And not just on vibes.

s388
00:35:38.240 --> 00:35:41.839
Um, so this is this is the change that we we wound up testing out.

s389
00:35:41.839 --> 00:35:45.200
It's like a small little tweak to a to a prompt, right?

s390
00:35:45.200 --> 00:35:59.359
So before it it only mentioned uh GraphQL, but after it says, "Hey, if you're dealing with like a some kind of aggregate like a count or group, um why don't you use our SQL tool rather than reaching for GraphQL first?"

s391
00:35:59.359 --> 00:36:00.320
Right?

s392
00:36:00.320 --> 00:36:08.720
Um, and of course the most important the most important thing that we need in order to answer this question is first of all some like realistic data.

s393
00:36:08.720 --> 00:36:19.280
So we throw in a you know semi-realistic um SQL life file that we've been kind of just like collecting as we develop ourselves as we work ourselves and like dog food the product ourselves.

s394
00:36:19.280 --> 00:36:32.320
Um, but then here's the really the part that I found quite cool or novel about Harbor, which is like um I literally just asked Claude like can you look at like the domains of my application,

s395
00:36:32.320 --> 00:36:37.200
you know, traces, spans, um, data sets, experiments.

s396
00:36:37.200 --> 00:36:38.480
Can you look at all these domains?

s397
00:36:38.480 --> 00:36:45.520
Can you brainstorm a list of user queries, some of which would be aggregate, some of which would not be aggregate, right?

s398
00:36:45.520 --> 00:36:49.760
And then can you write an Oracle solution to those queries, right?

s399
00:36:49.760 --> 00:36:58.640
Um the Oracle solution literally just being a REST call or perhaps a database query and the coding agent will just do that.

s400
00:36:58.640 --> 00:37:02.560
So we generated just this data set of synthetic questions.

s401
00:37:02.560 --> 00:37:04.320
Um I spot check the questions, right?

s402
00:37:04.320 --> 00:37:06.400
Just to make sure they sort of make sense.

s403
00:37:06.400 --> 00:37:14.000
Um and I have a certain degree of confidence in the actual verifiers which again are just scripts that are that are running in the environment.

s404
00:37:14.000 --> 00:37:26.800
But because those verifiers um are passing against an oracle again those verifiers are just like you know kind of passing against this oracle I have a certain amount of confidence in them right some of them were LLM as a judge

s405
00:37:26.800 --> 00:37:37.520
some of them were SQL queries or or REST calls themselves but you know just the ability to kind of work out those kinks before the agent ever gets to the system it gives you a certain level of confidence

s406
00:37:37.520 --> 00:37:50.240
um and then you know I was able to test out this particular change and I was able to see like, oh, you know, as you might expect, making a change like this, it causes the system to use SQL more frequently and more aggressively,

s407
00:37:50.240 --> 00:37:51.599
which is what I wanted it to do, right?

s408
00:37:51.599 --> 00:37:59.760
I wanted to overuse SQL rather than underuse it because I don't want it to like start trying to pageionate through trying to do like aggregate queries.

s409
00:37:59.760 --> 00:38:00.960
That would be bad.

s410
00:38:00.960 --> 00:38:12.079
And then the other interesting thing is just by changing this one prompt I can also see the number of tool calls in each in each uh trajectory the average number of tool calls dropping which again is like also

s411
00:38:12.079 --> 00:38:19.839
you know the reason that we did this in the first place is I wanted to reduce the number of tool calls and this data set it wasn't even like a big

s412
00:38:19.839 --> 00:38:27.280
data it wasn't a big data set so it didn't have to do a lot of pageionations I think it these numbers like these modest drops they kind of understate

s413
00:38:27.280 --> 00:38:34.800
the issue because the data was not large and So let me let me show you now what this looks like inside of Phoenix.

s414
00:38:34.800 --> 00:38:41.920
So the first thing that you can see here um are the examples in our data set, the tasks in our data set.

s415
00:38:41.920 --> 00:38:43.359
And again, these are versioned.

s416
00:38:43.359 --> 00:38:48.160
So we actually have a whole version history here every time there's a change.

s417
00:38:48.160 --> 00:38:51.280
Um and we can inspect what some of these look like.

s418
00:38:51.280 --> 00:38:55.440
So again, some of these are aggregate queries, some of them are not.

s419
00:38:55.440 --> 00:38:58.400
So here's an example of an aggregate query that I might care about.

s420
00:38:58.400 --> 00:39:01.440
What are the three most expensive models by total cost?

s421
00:39:01.440 --> 00:39:02.480
How much does each cost?

s422
00:39:02.480 --> 00:39:09.920
So this question is more easily answered with a SQL than than by pageionating through like a REST API.

s423
00:39:10.160 --> 00:39:15.920
And down here you can see the answer and again this answer um it comes from an oracle.

s424
00:39:15.920 --> 00:39:24.880
Um and so that's part of why I have a certain level of faith in it is because it's it's coming from a SQL query run against a concrete set of data.

s425
00:39:24.880 --> 00:39:37.680
It's not something that I feel um a strong need to check actually for this particular example because the answer is deterministic and I'm I'm sort of allow I'm I'm placing a certain amount of trust in Claude I guess to like write a good SQL query.

s426
00:39:37.680 --> 00:39:50.160
Um so we got all our examples here and then we got our um we got our experiments and our notion of an experiment that's that's what I've been calling a job in Harvard.

s427
00:39:50.160 --> 00:39:52.400
So just imagine this to be all the jobs that we've run.

s428
00:39:52.400 --> 00:39:55.119
I've run a lot of jobs in preparation for this talk.

s429
00:39:55.119 --> 00:40:01.280
Um, these charts show you the kinds of metrics that you might care about, whether those are cost or run latency.

s430
00:40:01.280 --> 00:40:05.280
Uh, the reward, as you can see, for this particular task is quite high.

s431
00:40:05.280 --> 00:40:10.640
Um, and that's because I haven't really done anything that's going to like hopefully break the system.

s432
00:40:10.640 --> 00:40:13.040
It's not like something was truly genuinely broken.

s433
00:40:13.040 --> 00:40:15.520
The system was getting the right answer in the beginning.

s434
00:40:15.520 --> 00:40:17.359
So, the reward was always high.

s435
00:40:17.359 --> 00:40:20.800
Um, it's just that it was just taking too many tool calls, right?

s436
00:40:20.800 --> 00:40:24.400
So this was more of an optimization rather than a correctness fix.

s437
00:40:24.400 --> 00:40:25.680
You can see tool call count.

s438
00:40:25.680 --> 00:40:30.560
You can see um whether or not we use SQL.

s439
00:40:30.560 --> 00:40:32.640
How do you calculate the reward?

s440
00:40:32.640 --> 00:40:39.599
So the reward if we go back in the slide, the reward is just coming from the verifier.

s441
00:40:39.599 --> 00:40:44.160
And the verifier is something that you the user have in your control.

s442
00:40:44.160 --> 00:40:44.880
Right?

s443
00:40:44.880 --> 00:40:55.280
So when you're defining your task, for each task, you get to write a script that says what constitutes success or failure.

s444
00:40:55.280 --> 00:40:59.440
And that script is going to emit a zero or a one, a pass or a fail.

s445
00:40:59.440 --> 00:41:01.200
You get to name the reward, right?

s446
00:41:01.200 --> 00:41:04.160
So the reward could have a different name other than pass fail.

s447
00:41:04.160 --> 00:41:06.240
Um that's all a reward is though.

s448
00:41:06.240 --> 00:41:15.200
you as the developer, you as the maintainer, you think about for this particular task, what what does it mean to succeed and what does it mean to fail?

s449
00:41:15.200 --> 00:41:24.480
How could I figure out just by looking at the side effects in the environment, looking at the file system or database file for example, how could I figure out whether the agent succeeded

s450
00:41:24.480 --> 00:41:27.520
and that's what the reward is and so that's what we're tracking over time.

s451
00:41:27.520 --> 00:41:31.040
There are the aggregate rewards for the entire job.

s452
00:41:31.040 --> 00:41:32.800
That's what we're tracking in this view.

s453
00:41:32.800 --> 00:41:34.079
Yeah, go ahead.

s454
00:41:34.079 --> 00:41:35.599
I mean this is such a good question, right?

s455
00:41:35.599 --> 00:41:41.200
I kind of get it kind of like gets to one of these big big qu big big picture questions about like what do you use Harbor for?

s456
00:41:41.200 --> 00:41:55.359
Actually the question was around like well hey you said that you know typically your verifiers are agnostic with respect to the the trajectory or the execution path of the agent but hey like look some of your verifiers actually like kind of it looks like they're peeking in to see whether the agent use SQL.

s457
00:41:55.359 --> 00:41:56.240
Isn't that cheating?

s458
00:41:56.240 --> 00:41:57.599
Um the answer is yes.

s459
00:41:57.599 --> 00:42:10.400
So it's not I think the times when you would do what I said in the purest form of like your verifiers are truly agnostic and they really don't look at the trajectory and they really just look at the file system you know or the the side effects

s460
00:42:10.400 --> 00:42:24.079
the situation for that is really like um when you care about correctness right so may you know one of the other things that I'm not showing here which is a different example that we've done is like we've worked on some regression tests and those tests are really like

s461
00:42:24.079 --> 00:42:30.400
does it get to the right answer in the end and for Those tests I think our verifiers are quite pure.

s462
00:42:30.400 --> 00:42:32.000
They don't really look at the trajectory.

s463
00:42:32.000 --> 00:42:34.640
They just look at like oh did it get the right answer right?

s464
00:42:34.640 --> 00:42:38.800
For this one for this particular problem I was concerned with something maybe subtly different.

s465
00:42:38.800 --> 00:42:41.200
Like I wasn't really just trying to get the right answer.

s466
00:42:41.200 --> 00:42:43.280
I was trying to optimize a metric.

s467
00:42:43.280 --> 00:42:43.599
Right?

s468
00:42:43.599 --> 00:42:46.319
I was trying to optimize tool calling in particular tool calls.

s469
00:42:46.319 --> 00:42:48.880
Harbor is one of these things where you can use it for many things.

s470
00:42:48.880 --> 00:42:51.920
You could use it for uh you know I I talked about integration testing.

s471
00:42:51.920 --> 00:43:06.480
You can use it for like kind of endto-end testing um for assuring correctness um for running in CI right or you could also use it because you're like oh right now I'm trying to solve a really particular problem which is like oh I kind of noticed that it sucks on these aggregate queries like how can I make

s472
00:43:06.480 --> 00:43:15.280
it better how could I in a quick and dirty way kind of convince myself that it's actually doing what I hope it's doing right um so like at least

s473
00:43:15.280 --> 00:43:23.920
for for us like what I found so far is it just depends on what you're trying to do um sometimes sometimes it is worth peing inside the trajectory I think

s474
00:43:23.920 --> 00:43:24.309
you know

s475
00:43:24.309 --> 00:43:24.880
[laughter]

s476
00:43:24.880 --> 00:43:36.720
yeah even though even though it makes the test less robust to change perhaps that that's a really good question you know my team is so small I don't know that we can I can really speak from personal experience

s477
00:43:36.720 --> 00:43:44.160
um that being said like one of the reasons that you you might use a tool like Phoenix is because it allows you know it has like the ability to

s478
00:43:44.160 --> 00:43:55.359
share data right to curate data from different sources to add it you know to to like add traces to a data set um it has the ability to like store data as a you know in an external

s479
00:43:55.359 --> 00:43:59.040
file and like upsert it to the database and then run it has those kinds of abilities.

s480
00:43:59.040 --> 00:44:06.079
So like that might help you in terms of the the uh logistics of like you know actually like dealing with this in a larger team.

s481
00:44:06.079 --> 00:44:10.240
I I don't I don't know because really it is just me and a handful of people working on this right now.

s482
00:44:10.240 --> 00:44:13.599
All right, I'm going to show you one last little demo and then I'll just take questions.

s483
00:44:13.599 --> 00:44:15.920
I think we got only a handful of minutes here left.

s484
00:44:15.920 --> 00:44:20.400
So you know back to back to your question around like what are some of the things that you might get out of Phoenix.

s485
00:44:20.400 --> 00:44:23.839
I think this is like kind of the the one of the bigger ones, right?

s486
00:44:23.839 --> 00:44:28.800
Is like the ability to compare individual jobs at the task level, right?

s487
00:44:28.800 --> 00:44:31.599
So right here I have two two particular jobs selected.

s488
00:44:31.599 --> 00:44:36.240
I have opus 5 before and after making that single prompt change.

s489
00:44:36.240 --> 00:44:38.000
This column is opus 5 before.

s490
00:44:38.000 --> 00:44:41.359
This column is opus 5 after these these metrics at the top.

s491
00:44:41.359 --> 00:44:49.920
They're telling me in aggregate how did these like metrics that I care about, how did these rewards change after I made this tweak to the prompt, right?

s492
00:44:49.920 --> 00:44:53.200
And I can see that like GraphQL tool calls went down.

s493
00:44:53.200 --> 00:44:57.760
So it's calling GraphQL in aggre in it's calling GraphQL less than it was.

s494
00:44:57.760 --> 00:45:01.680
Um the reward is staying the same which means everything is still passing.

s495
00:45:01.680 --> 00:45:03.040
Right?

s496
00:45:03.040 --> 00:45:05.200
SQL calls are going up.

s497
00:45:05.200 --> 00:45:07.040
Tool counts are going down.

s498
00:45:07.040 --> 00:45:17.440
The number of turns the agent is taking is going down because it's encapsulating a lot of that a lot of those turns that would have been multi-turn steps into a single turn because it's using SQL.

s499
00:45:17.440 --> 00:45:20.640
uh it's using more SQL, it's using less GraphQL, right?

s500
00:45:20.640 --> 00:45:26.560
So you get those kind of like top level metrics which are the ones that in my mind verify, okay, this is a decent change.

s501
00:45:26.560 --> 00:45:31.839
Um and then I'm able to see in this table like four particular tasks, right?

s502
00:45:31.839 --> 00:45:37.200
Here's the here's that one we looked at like what are the three most expensive models by by total cost.

s503
00:45:37.200 --> 00:45:40.480
I'm able to see what I expected the answer to be, right?

s504
00:45:40.480 --> 00:45:43.280
I'm able to see the actual answer in each job.

s505
00:45:43.280 --> 00:45:45.359
I'm able to see the metrics and how they changed.

s506
00:45:45.359 --> 00:45:57.920
So for this one um yeah for this one the number of tools went down the number of turns went down interestingly enough the number of SQL calls actually went down too which is kind of unexpected

s507
00:45:57.920 --> 00:46:08.160
and I can do things like I can I can filter by these right so I can here's a fun little feature that Miko built recently um which is I can say show me the runs where

s508
00:46:08.160 --> 00:46:25.040
the second experiment had um where the second experiment ment had lower tool call count and I'll use the exact the exact metric to help it out.

s509
00:46:25.040 --> 00:46:30.960
Lower tool count than the first experiment.

s510
00:46:30.960 --> 00:46:31.839
We'll see if this works.

s511
00:46:31.839 --> 00:46:42.480
So, what this is doing is it's actually running a a local model in the browser to convert that piece of natural language into a query DSL.

s512
00:46:42.480 --> 00:46:45.440
I think it messed that up though.

s513
00:46:46.800 --> 00:46:49.520
We gota we got to use a more powerful one.

s514
00:46:49.520 --> 00:46:52.160
The prompt API is a little bit

s515
00:46:52.160 --> 00:47:00.400
I'll just I'll just copy in what I know to be the correct uh correct expression here.

s516
00:47:02.079 --> 00:47:03.520
So I'm able to filter down.

s517
00:47:03.520 --> 00:47:14.160
So now I'm just looking at the um now I'm looking at the rows where it improved basically like I'm looking at the ones where it didn't use SQL before where now it's using SQL.

s518
00:47:14.160 --> 00:47:25.440
Um and you can see here this one used SQL it went up by one and if we actually like look at the uh look at the trace we can actually see that right so those

s519
00:47:25.440 --> 00:47:37.040
ATIF trajectories that get converted into open inference traces here you can see it you can see it was using bash before um and when it's using bash under the hood it is actually using graphql

s520
00:47:37.040 --> 00:47:46.400
it's calling has a bash emulator which is calling graphql underneath the hood right so I'm able to see that right here I'm able to see in the second job

s521
00:47:46.400 --> 00:48:00.240
the same trace or the analogous trace and I'm able to see oh instead of instead of calling bash and using graphql here it actually issued a SQL query directly to the database so I'm able to like visually inspect and say oh okay

s522
00:48:00.240 --> 00:48:08.880
in this particular example it did the right thing um and I can even I can even ask you know the agent itself so this is kind of like what we do

s523
00:48:08.880 --> 00:48:18.079
um you know I can literally ask the agent itself like hey look at a particular um set of experiments, you know, look at this particular task and it will tell me

s524
00:48:18.079 --> 00:48:19.440
exactly what happened.

s525
00:48:19.440 --> 00:48:27.920
And so this is kind of like how we get some of our data in the first place is we're kind of like using using the agent um recording its traces

s526
00:48:27.920 --> 00:48:31.440
uh getting its data and using the agent to like iterate on itself.

s527
00:48:31.440 --> 00:48:34.800
Yeah, I'd say with respect to the regression stuff, we're still really early days.

s528
00:48:34.800 --> 00:48:37.760
I mean, we're trying to get like happy paths going, I would say.

s529
00:48:37.760 --> 00:48:51.680
Um, and I think like the way that the way that we're structuring it is is around skills because I think in my mind when you're doing those regression tests, they are sort of analogous to like playright in a certain way where it's like an endto-end

s530
00:48:51.680 --> 00:48:54.880
thing and it just includes like an agent in the loop.

s531
00:48:54.880 --> 00:48:57.200
It includes like LLM calls in the loop.

s532
00:48:57.200 --> 00:49:08.319
Um, and so I think skills are a really good like thing to write those kinds of tests for because at least in our case like the skills are describing workflows.

s533
00:49:08.319 --> 00:49:11.040
They're saying, "Oh, if this happens to this, if that happens to this, right?

s534
00:49:11.040 --> 00:49:20.319
It's describing a workflow that you want to have happen and and so right now we're just kind of in the process of trying to nail down happy paths on our most important skills.

s535
00:49:20.319 --> 00:49:28.960
um and ensure that the happy path passes across like the the broad array of like models and providers that we're trying to support.

s536
00:49:28.960 --> 00:49:29.200
Right.

s537
00:49:29.200 --> 00:49:29.359
Right.

s538
00:49:29.359 --> 00:49:29.599
Right.

s539
00:49:29.599 --> 00:49:29.839
Yeah.

s540
00:49:29.839 --> 00:49:44.400
So this this particular comment is around like the ability of uh you could make the the greatest skill in the world, but like if somebody else makes another skill that kind of like where the front matter of the skill kind of treads on your feet a little bit, it's going to it may not fire anymore, right?

s541
00:49:44.400 --> 00:49:46.400
And then it's no good if it never fires.

s542
00:49:46.400 --> 00:49:52.720
Um, and for that sort of thing, it's an interesting question, right?

s543
00:49:52.720 --> 00:49:56.559
It's like there's multiple ways of approaching it, right?

s544
00:49:56.559 --> 00:50:07.200
Like one thing that you could do would you you could just say, well, I don't care which one fired as long as I have a good test and I know that like, oh well, like if this skill

s545
00:50:07.200 --> 00:50:10.000
didn't fire, it's probably not going to pass this test, right?

s546
00:50:10.000 --> 00:50:11.599
Um, that would be one approach.

s547
00:50:11.599 --> 00:50:24.640
Another thing you can do and maybe this is the appropriate thing right is you can always as we said reach into the trajectory with your verifiers you can always look in you can always peek you know you can cheat a little bit you can say oh well like given this question

s548
00:50:24.640 --> 00:50:39.040
did it call this particular skill and we have had tests like that um I think the challenge is maintainability right it's always around maintainability um if you change the skill name or something the test is going to flake

s549
00:50:39.040 --> 00:50:44.319
um but we've had tests where the thing we were trying to test was skillfiring, right?

s550
00:50:44.319 --> 00:50:48.000
And those tests are not they're not the hardest to write, right?

s551
00:50:48.000 --> 00:50:53.839
I mean, don't even because if if you're only testing one turn and all you want to know is like, did the right skill fire?

s552
00:50:53.839 --> 00:50:56.240
Um, you don't even really need great data for that.

s553
00:50:56.240 --> 00:50:59.920
You just ask it a question and and see did the skillfire.

s554
00:50:59.920 --> 00:51:05.680
Um, and like you can totally, you know, maintain a suite of tests that that do that, right?

s555
00:51:05.680 --> 00:51:10.240
And and you know, then it becomes less of a end to end test.

s556
00:51:10.240 --> 00:51:22.480
it becomes more of like a almost like it's like almost like you know on the spectrum between you know fully decoupled from the implementation and coupled to the implementation is a little bit more coupled

s557
00:51:22.480 --> 00:51:36.880
um and yet it could still be useful to you right um so there's no reason to not write a test like that um there's no reason to not write a test like that I think so it really just depends on like you know getting getting

s558
00:51:36.880 --> 00:51:40.800
a broad level of coverage um on the things that you care about.

s559
00:51:40.800 --> 00:51:44.000
We are collecting traces in production.

s560
00:51:44.000 --> 00:51:46.559
So when you deploy Phoenix in production, this is the picture.

s561
00:51:46.559 --> 00:51:51.599
The traces are coming into the system from an instrumented application or agent.

s562
00:51:51.599 --> 00:51:55.119
We're collecting those traces and we're writing them to the database.

s563
00:51:55.119 --> 00:52:00.319
When we're running things in harbor, you can see [clears throat] that this this thing disappears, right?

s564
00:52:00.319 --> 00:52:10.880
the OTOP ingest, the AI agent, which is the user's agent, those disappear and we just seed the database with data.

s565
00:52:10.880 --> 00:52:18.480
So when you're running when we evaluate in harbor, um we don't we don't pretend or emulate an end user's application.

s566
00:52:18.480 --> 00:52:25.359
Um we assume that the end user's application exists and has resulted in the data in the in the SQL life file.

s567
00:52:25.359 --> 00:52:25.599
Yeah.

s568
00:52:25.599 --> 00:52:32.319
So the question is around um Harbor is canonically used for evaluating coding agents.

s569
00:52:32.319 --> 00:52:47.760
That's the canonical use case and we're doing something similar on the one hand but the unit that we're trying to evaluation to to evaluate here is not the coding agent in isolation because we don't work on cloud code, right?

s570
00:52:47.760 --> 00:52:49.599
We don't we don't tweak cloud code.

s571
00:52:49.599 --> 00:52:51.520
Uh we don't tweak codecs, right?

s572
00:52:51.520 --> 00:53:05.599
The thing that we're trying to optimize for is this hybrid system which is cloud code plus our particular bespoke MCP server or cloud code plus our particular bespoke set of skills

s573
00:53:05.599 --> 00:53:21.599
that we maintain or cloud code plus our um cloud code plus uh our CLI or cloud code um interacting with our APIs.

s574
00:53:21.839 --> 00:53:24.000
And that is a different problem, right?

s575
00:53:24.000 --> 00:53:32.400
And the fact that that is the thing that we're trying to optimize this hybrid entity which includes cloud code, it includes the coding agent, but it's more than just the coding agent.

s576
00:53:32.400 --> 00:53:42.640
It's all of the um supporting infrastructure around the coding agent which we maintain which helps cloud code be better at interacting with our product.

s577
00:53:42.640 --> 00:53:55.920
Um, that's the reason that we've shoved all of this stuff into an environment together is because all of that stuff has to change um and be evaluated together um as a unit.

s578
00:53:55.920 --> 00:54:02.000
I think it took a second to kind of conceptually land on and understand Harbor well enough to land on this picture.

s579
00:54:02.000 --> 00:54:05.440
Um, and then it took a second.

s580
00:54:05.440 --> 00:54:14.559
it didn't take super long and again like I think part of that is part of that is due to our advantage which I mentioned which is that Phoenix runs on a single node

s581
00:54:14.559 --> 00:54:22.319
um and that makes it easy to dockerize right and so as as some of these other folks suggested it's like you know if you're working on a distributed system

s582
00:54:22.319 --> 00:54:29.040
you know it's it's tougher your your life is going to be a little tougher right and so we had a certain advantage that made it easier for us to set this picture up

s583
00:54:29.040 --> 00:54:44.800
um but I would say like the coding agents are Um, Harbor's tough in the sense that like there's no because the contract that Harbor tries to honor is all based on like scripts in certain directories.

s584
00:54:44.800 --> 00:54:49.280
You know, I need to put my my task my task initiation script here.

s585
00:54:49.280 --> 00:54:51.280
I need to put my verifier script here.

s586
00:54:51.280 --> 00:54:56.640
Um, there's no like llinters that the coding agents can rely on to make sure that that works.

s587
00:54:56.640 --> 00:55:02.160
Um, the best that you can do uh is you can download Harbor's skills.

s588
00:55:02.160 --> 00:55:05.680
Um and those will help you set up your project correctly.

s589
00:55:05.680 --> 00:55:06.000
Yeah.

s590
00:55:06.000 --> 00:55:09.200
So the question is uh I have a SQLite file here.

s591
00:55:09.200 --> 00:55:10.319
Where do I put it?

s592
00:55:10.319 --> 00:55:10.960
Right.

s593
00:55:10.960 --> 00:55:12.559
Um cloud storage.

s594
00:55:12.559 --> 00:55:12.960
Yeah.

s595
00:55:12.960 --> 00:55:18.559
So basically um it's kind of a one-off process right now.

s596
00:55:18.559 --> 00:55:26.400
It's like we have a collection of data that we generally use and if that thing works for the task at hand, I tend to reuse that file.

s597
00:55:26.400 --> 00:55:30.713
And if it if I don't need the whole file, I'll say Claude, can you clean this up?

s598
00:55:30.713 --> 00:55:30.960
[laughter]

s599
00:55:30.960 --> 00:55:35.119
can you can you cut out the stuff I don't need um and push it to cloud storage for me?

s600
00:55:35.119 --> 00:55:38.720
Um that data though, the data is the hard part, right?

s601
00:55:38.720 --> 00:55:43.440
I think the data getting the data is the hard part um for for any of these tasks.

s602
00:55:43.440 --> 00:55:52.160
And the more realistic your data, the better the system is going to run uh the better it's going to like uncover realistic issues for you.

s603
00:55:52.160 --> 00:55:55.599
So it's always the hard part getting data