How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI

AI Engineer · 56 min · 603 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12I'll go ahead and get started.
00:13Uh, my name is Xander Song.
00:16I'm an engineer at AriseAI where I work on Phoenix, our open-source observability experimentation and evaluation platform.
00:25And today I'm going to introduce you to Harbor, which is an open- source framework for evaluating agents on real tasks with real data in isolated sandboxed environments.
00:38The theme of today's talk is shipping based on evidence rather than vibes.
00:43And my goal is to give you a conceptual overview of how Harbor works as a framework and to share some of the ways in which my team is using Harbor
00:51to iterate more quickly and with more confidence on our agents, harnesses, skills, prompts, and tools.
00:58This talk is going to be a discussion rather than a interactive workshop.
01:02But my hope is that you walk away with a concrete sense of how Harbor works as a framework and also a sense of how you might use it in your day-to-day work to make your life and your agents a little bit better.
01:16A disclaimer before I begin.
01:17I am not a maintainer of Harbor.
01:19Um I do however consider myself to be an avid power user at this point.
01:28So, so what is harbor?
01:30At a high level, Harbor is a framework that allows you to define realistic tasks in the form of natural language instructions to an agent.
01:39These are tasks that you would like your agent to be able to accomplish.
01:44Harbor feeds those tasks to your agent running in a fresh sandboxed environment with real data relevant to the given task.
01:54And then it enables you to define what are called verifiers.
01:57These verifiers are specific to each task and they just check that your agent produced the intended side effects inside of the environment.
02:08Verifiers produce rewards.
02:10Examples of rewards are binary scores such as zero or one or pass fail.
02:16So this picture is like really really simple, right?
02:19But I want to just take a moment to appreciate why it's powerful.
02:24Because if my task and my data are realistic and if I've done a good job of defining verifiers that accurately describe what constitutes success or failure on the given task,
02:38what that means is that I can hold my task and my verifier constant while I change the implementation of my agent.
02:46And by simply executing this pipeline from start to finish, I can quickly and efficiently gain a signal telling me whether or not the change I made to my agent
02:58produced an improvement or a regression for this particular task.
03:02Um, and not only that, I can take it a step further.
03:06I can compile not just one task, but many tasks that cover the broad surface area of the capabilities of my agent.
03:15And this lets me close the loop here.
03:17So that now I'm actually able to start iterating with a degree of confidence that I might not be able to um if I was shipping based on vibes alone.
03:30So that's really the idea of harbor in a nutshell.
03:35Let's break it down a little bit further.
03:37I'm going to talk about like the architectural details of how Harbor works.
03:40So on this slide here, I have the task that I was talking about on the previous slide.
03:48Each task specifies a set of instructions in the form of a markdown file.
03:52And once again, that is just a piece of natural language that you're going to feed to your agent.
03:58The task also describes an environment in which the agent will run.
04:03And that typically takes the form of a Docker file.
04:05And that Docker file typically contains data that is relevant to your task at hand.
04:10And we'll give examples of this in just a minute.
04:13But the thing to realize here is that um Harbor's actually agnostic with respect to what kind of data you give it.
04:20So some examples could be like a SQL light file.
04:23A very common example for coding agents would be like a git repository at a particular commit.
04:30And Harbor also lets you define a set of verifiers inside of a test directory.
04:34And as I said, verifiers are just scripts.
04:37They're just scripts that run on your environment after your agent is done running that verify that the intended side effect took place.
04:46And like I said, typically we don't have just one task, but we have many collected together in what is called a data set.
04:54And for each task in the data set, Harbor allows you to run a configurable number of trials.
05:02These trials are just independent executions of the task to account for the fact that LLMs are stochastic.
05:08Agents are stochastic.
05:10They don't behave the same every time.
05:12So each trial is going to build a fresh environment in the form of a brand new Docker container seated with that data we were just talking about.
05:20Harbor is going to drop your agent of choice into this environment.
05:24It's going to feed it the instruction markdown file and it's going to let it do its thing.
05:29And of course, the agent under test is going to produce side effects in the environment usually.
05:35So if it's a coding agent, it's going to make changes to source files.
05:39Um, it might make changes to a database if you put it in there.
05:41It could do any number of things, right?
05:44And after the agent finishes, the verifiers are going to run on the environment.
05:49And again, a verifier is just a script that checks the environment.
05:55Those verifiers produce rewards which get written to a reward.ext file and that that's the most important thing if you take away nothing else.
06:06This is like the architecture of how harbor works.
06:08There's another thing which is kind of an additional optional thing you can do which is you can configure your agents to emit so-called trajectories in the form of ATIF.
06:20ATIF stands for agent trajectory interchange format.
06:24ATIF is Harbor's specification for a JSON based tracing format which basically lets you peer into the internal details and the internal steps that your agent took to produce the side effect on the system.
06:38So it's basically there for debugging purposes and for logs and and recording and all that.
06:45this whole pipeline executing end to end where we take all the tasks in a data set run a certain number of trials for those tasks run our verifiers to produce rewards and collect trajectories
06:56this whole thing all taking together is what Harper calls a job and there's one more thing I want to point out while I'm on this slide which is you'll notice right here
07:07um I said I said agent/oracle um so Harbor also has this notion of an oracle you can drop not only an agent agent into the environment, but also an oracle.
07:17And an oracle is just a solution to your task.
07:20It's just a script that solves your task.
07:24And we'll talk about oracles more in a minute, but oracles turn out to be like this really useful thing because oracles basically let you work out all the the the kinks in your pipeline before you ever throw an agent into the mix.
07:36It lets you make sure that your data is good, that your environment is configured properly, that your verifiers work as expected, and you get to do all that before you spend any money on tokens for an agent.
07:49All right, so that is like the high level.
07:50Um, before I pause for questions, I will I will just continue on for just a second to kind of illustrate it with a concrete example.
07:58So, I'm going to break down a concrete task.
08:01Let's suppose that our agent that we care about is a coding agent um or a coding agent with a particular set of skills or a coding agent configured with [clears throat] like a custom MCP server that we're running.
08:12And let's suppose that the task is to fix a failing migration.
08:17So if that's the case, the instruction markdown file would you know it would it would be some set of instructions that you would pass to the coding agent as a user.
08:25So it might say, "Hey, I'm noticing that this migration is failing.
08:28Could you please try to fix it?"
08:29Right?
08:30And in the environment, the environment would contain perhaps an actual um SQL file.
08:37Maybe it would contain some DDL.
08:38Um and maybe it would contain some source code containing an up migration.
08:43And we've, you know, we've seated the data so that the up migration fails.
08:46Um in addition, we'll have our tests.
08:51And again, the tests, they could be just scripts that run the up migration that ensures that the up migration exit with status code zero.
08:59And then it maybe it checks like some tables.
09:01Maybe it just checks the tables to make sure that the columns are there that we expect, the indexes are there that we expect to make sure that the um migration
09:10has actually had the intended side effects on the system.
09:15And you'll also notice that here optionally we have the solution, right?
09:18So the solution is just going to be the literal solution, the literal uh script that you could run um that would fix the the migration.
09:27And again, why would why would you go through the trouble of doing this?
09:30Um, the reason again is it kind of lets you quickly and confidently build a little bit of assurance that you've got your system set up, that your verifiers are correct,
09:40and you can it kind of lets you um vibe a little harder, I think, with the system because you can gain a certain amount of confidence in your verifiers
09:49without necessarily like reviewing them line by line.
09:52At least that's my experience with it.
10:00And so after the the agent runs the task in the environment, um it's going to edit that broken migration file, it's going to leave a couple modifications on the file system
10:13and then the verifiers are going to run.
10:16And typically if you're running um Harbor in Docker, which is like the default local setting, those verifiers are going to mount in the Docker container after the agent finishes.
10:28And that's to prevent the agent from cheating.
10:30So the the agent doesn't have access to the verifiers and the solutions um during the actual execution.
10:38Those verifiers are going to run the sort of checks that we talked about.
10:41Maybe it literally runs the up migration.
10:43Maybe it um you know checks the tables, checks the indexes.
10:48And I think the thing to really take away from this slide is um the verifiers are agnostic of the implementation of your agent.
10:57They actually don't know anything about the agent at all.
11:00All they care about is what the agent leaves behind.
11:04So it is possible to like write some verifiers that are not like that that peak into the ATIF trajectory that kind of look at the implementation or the execution of the agent
11:14and evaluate it that way.
11:16But that's not really the canonical kind of verifier.
11:21The verifiers, you know, canonically treat the agent as a little bit of a black box and they're grading the agent solely based on its side effects and its output in the environment
11:32rather than the exact manner in which it produced those side effects and outcome.
11:37And there's kind of a natural parallel here with integration tests because you can imagine that if you have some integration tests for an interface or an API, those integration tests
11:47um if they're really well written are hopefully agnostic of the implementation that is behind that interface or API.
11:53And in the same way, if we have really good verifiers, our verifiers are kind of describing what we expect the agent to do, the side effects we expected to produce in the environment.
12:06And they're doing so in a way that is decoupled from the implementation.
12:09And that's a really nice feature to have for a set of tests because it allows you to refactor under green, for example.
12:16It allows you to hold your tasks constant while you like completely change the implementation of your agent and then it allows you to like fire this pipeline really quickly to figure out if everything still works.
12:30That's kind of what we're talking about when we're talking about iterating quickly um and shipping confidently.
12:37So before I go any further, let me just pause right here for questions or comments or um Yeah, let me just pause before I go any further.
12:46Yeah.
12:46Yeah.
12:46Yeah.
12:46You just Yeah.
12:47Again, it's just a Docker container.
12:48So, you just pop whatever you want in there, right?
12:50[laughter]
12:51It's a primitive that you're you're proposing.
12:54Yeah.
12:54Yeah.
12:54Yeah.
12:54Yeah.
12:55Yeah.
12:55Yeah.
12:55So, you know, it could be it could be any framework, right?
12:57Uh it could be any language.
12:59It could be any piece of code, any executable.
13:01Concretely, like, you know, I'll give you a little sneak sneak peek or sneak preview, right?
13:05Like, we've had to like change tools like um you know, adjust adjust prompts, right?
13:11Um, I'm going through a process right now where I'm trying to consolidate our skills, for example.
13:17So, we kind of have like uh several different agent experiences that we try to support.
13:22Um, there's a lot of overlap and then there's like certain differences and we're trying to like basically get our skills all kind of like reusable, right?
13:29Um, and agnostic of the particular like direction that the agent is coming from, whether it's MCP or a CLI or via like our our own like server side agent.
13:38we're trying to get it agnostic and and so those are the kinds of things where it kind of helps to be able to specify this set of tests that is like completely agnostic
13:48of the underlying implementation, you know, and we we kind of we came to this like maybe um through the pain of doing it the other way where we were like kind of like had a set of tests that were um that were coupled
14:00and it was becoming difficult to maintain because you know I would change a tool and the tests would break right even though the not even though the true functionality of the agent was still intact, right?
14:11Things like that.
14:12Um, we'll get to more of these, right?
14:14We'll get to more of these at the end.
14:15Um, any other questions before I go on?
14:19Okay.
14:21Okay.
14:22So, uh, this this slide is going to talk about scale.
14:25Um, so as we as we already talked about, right, each task and each trial, uh, is independent, runs in its own environment, its own docker container.
14:34Um, and that makes this whole pipeline completely horizontally scalable.
14:39And that means that it's very compatible with a sandbox provider such as Daytona or Modal or E2B or any of those.
14:46Um, and it means you can kind of rip through these tests like really fast if you want to, which is kind of fun.
14:52So in this diagram arrayed vertically, we have a set of four tasks in a data set, including the fixed migration task that we were just imagining.
15:00Um, we've got a couple of different coding agents that we might want to test out.
15:04claude code with opus 5 codeex with gbt5 and then we've got um a couple of different trials.
15:10Let's suppose that we're doing three trials for k equals 1 2 3. Again, the point of that is just to kind of handle the handle the fact that these things are stocastic and they they're not perfect.
15:20They don't behave the same every time.
15:23Um so because all these are independent um you can run them all in parallel on these sandbox providers.
15:32So we use Daytona um and it runs quite fast with like 200 concurrent workers.
15:39So even if you have a large test suite you can rip through it quickly.
15:42Um and then from these individual trials you are able to compute aggregate metrics for the kinds of things that you care about.
15:51So for example you could compute a pass at Kmetric that just tells you in this case did I pass after three attempts at the at the task.
16:00And so I can see that oh for like this particular task the fixed migration task both of my agents like passed at K right passed at three uh for this one only one of my two agents passed at three.
16:11You can actually also compute those kind of aggregate metrics over the entire data set right and so this kind of like gives you this very quick picture of how your change is performing because again if you have like a a good data set that covers the set of functionality that you care about you can get these aggregate
16:27metrics pretty pretty rapidly.
16:29Yeah.
16:29Go ahead.
16:30Each Yeah, that's a great question.
16:31So, this is a really important point, right?
16:33Is each of these boxes is a totally separate Docker container, identical in the beginning, but then you you put an agent, a separate agent into each one, and you let it go and do its thing, and they do different things, right?
16:46Um, so each one is totally independent.
16:48They're completely isolated, right?
16:50They're independent trials of the same task with the same agent.
16:55Um you're basically run yeah you're basically running claude code three times and saying hey try to fix this migration and then you're saying oh after three tries did claude code fix the migration.
17:04Okay so now now I've now we've talked about harbor at a at a high level.
17:08Um but I want to switch gears and I want to talk about how Harbor plugs into Phoenix.
17:13So again I'm not a maintainer of Harbor.
17:15Um I'm just a fan.
17:16Uh but I am a maintainer of of Phoenix uh which is our which is ARIS's open source AI observability tool.
17:24And the point of this slide is just to basically convey that we have analogous concepts for the primitives in Harbor.
17:33So we're able to map your data sets to our notion of a data set.
17:37We're able to track your experiments.
17:40We're able to um record your trials.
17:43Uh we're able to convert your rewards into our notion of reward which we call an evaluation.
17:50Um, and I guess the other thing I I want to take away from this slide is also we we do something where we convert those trajectories that I talked about into
18:00so-called open inference format.
18:02Um so open inference is a open standard of semantic conventions so-called semantic conventions which is built on top of open telemetry which is an open source framework for telemetry data such as logs metrics and traces
18:24and these conventions basically define data structures that uh your AI I application can emit which then downstream consumers such as Phoenix can ingest and we can use those to build certain kinds of UI
18:41experiences and workflows that wouldn't otherwise be possible.
18:44So full disclosure, I am a maintainer of both imprints.
18:47Um we recently donated this the spec to cloud native, right?
18:51Um and to the hotel working group.
18:53So it has some traction outside of arise.
18:55And so that's maybe a reason you might want to um use open inference.
19:00Um so this diagram on the left, you can see the ATIF trajectory for that fixed migration task.
19:07you can see the individual steps that the agent might take um including an LLM call and including a tool call.
19:14And over on the right, uh this is kind of what it winds up looking like in open inference.
19:21And in the product, like I said, open inference allows us to build out the kinds of opinionated UIs and workflows that you see here.
19:31So we can do things like we can render your trajectories with some opinionated UI, some specialized UI for LLMs and tools.
19:38We can version your tasks and your data sets so that at any given time you know exactly what the task look like for a particular job in Harbor.
19:52We track your experiments and we help you visualize metrics and rewards as you iterate.
19:57So in this case the reward you could imagine it being pass fail um zero or one and you can run a sequence of experiments and you can kind of see over time
20:07is my reward moving in the right direction.
20:10We also have skills CLIs and an MCP server so that claude and codecs and cursor can easily get access to this data.
20:18[clears throat] And at this point I do want to note that Harbor does have its own open- source UI um that you can run locally on your machine.
20:25It also has a hosted web service.
20:27Um, so you you definitely don't have to use Phoenix to use Harbor.
20:30Um, but if you're interested in some of the features I just mentioned, that would be a reason to give Phoenix a try.
20:37Okay, so hopefully at this point I've given you kind of a bird's eye view of how everything works.
20:43Now I want to talk about some of the ways in which our team has been using Harbor.
20:47And hopefully this answers um answers your point over here just around getting some concrete examples.
20:52Um so in order to do that I just want to give kind of this high level of like what our architecture works with.
20:59So again this is the Phoenix architecture.
21:01This is particular to our team right this is the this is what our team is kind of like dealing with on a day-to-day basis.
21:08Um on the left here you have an enduser's AI application or agent.
21:16And again this application is going to have been instrumented with open telemetry and open inference.
21:22And it's going to be exporting and emitting traces to the Phoenix server over OTLP, which is the open telemetry protocol.
21:28So, it's basically sending data to Phoenix, right?
21:31Saying, "Hey, I'm doing this and this is like what my these are the sequence of steps that I did to accomplish this task.
21:38This could be like a production application.
21:39It could be an application that you're testing out.
21:42Those traces are ingested by Phoenix and they are written to one of our SQL backends, either SQLite or Postgres.
21:50And once the data is in the system, we have a set of core services for things like traces, spans, sessions, data sets, experiments, and evaluations.
22:00And these core services are exposed via various kinds of APIs, REST or GraphQL.
22:06But our goal is not only to like make those programmatic interfaces, but it's also to make it easy for our end users and for their coding agents and other kinds of agents to get access to the data and use the data.
22:18And so we support several different kinds of agent experiences.
22:23Um the first of which are coding agents.
22:27So for coding agents, we have a CLI that the coding agent can use which will invoke REST or it will invoke GraphQL.
22:35We also have an MCP server that you can configure your coding agent to use um which is going to do the same things.
22:44And we have a third agent interface that we maintain for the team which is a server side agent that we call Pixie which also has access to the MCP server and therefore has access to REST and GraphQL.
22:57Um it also has like a bash emulator.
22:59So it's it's kind of like a coding agent but running server side.
23:03Um so these are the three surfaces that we that we want to support and that we have to test and ensure are working.
23:10the coding agent with the CLI, the coding agent with an MCP server, and a server side agent, which is our own harness.
23:20And when it comes time to actually like test all of this out, architecturally, we have a pretty big advantage.
23:27We have this really nice feature, which is that this server runs on a single node.
23:31It's kind of like a graphana- like architecture.
23:33Um, and that means it's very easy to dockerize.
23:36So, you just pop the whole thing in a Docker container.
23:39What we do is we just take a SQLite file which contains some realistic data that we've tested on.
23:44Um we replace the browser with a test client.
23:49Um and we just put it in Docker.
23:54And once it's in Docker, we can start using it in Harbor.
24:01So we can feed the system the kinds of tasks that our particular users care about.
24:06So, our particular users might ask a coding agent or they might ask our Pixie server agent, find some hallucinated LLM calls from the last week of my traces and add them to a data set.
24:19That's a realistic query that one of our users might ask.
24:22And we can go ahead and let any of these three kinds of agents go wild inside of this environment.
24:32And at the end, we can run our verifiers.
24:35And the kinds of verifiers that we can run for us in this particular setup are really quite simple because we have a server running inside of here with the SQL life file.
24:46So we could either hit a rest endpoint to verify that the intended side effect has changed or we could literally like query the SQL life file if we wanted to because it's just in the container.
24:58Um and just verify for example that the particular tupil for this data sets table has been written in the way that we expect.
25:07Yeah.
25:07Yeah.
25:08No, thanks for stopping me because I think there's a lot going on on this slide.
25:10So maybe I should pause and and kind of like make sure it's clear before I go on.
25:14Right.
25:14Um, so I think there were a couple of questions there and I'll I'll try and address address them.
25:19Um, so in this slide there are three different agents in in any given trial.
25:26I would only be testing one of them if that makes sense.
25:29So you could imagine maybe I'm only testing Claude code with the CLI, maybe I'm only testing Claude code with MCP or maybe I'm only testing you know our Pixie server
25:38server agent.
25:39So just may let's imagine like it's just one at a time in terms of how does harbor feed at the prompt.
25:45Harbor and this is maybe a nuance that I haven't really talked about, right?
25:49Harbor itself um does not Harbor is a program.
25:53It's a command line tool.
25:55Um and it doesn't actually run in the environment per se in the Docker environment per se.
26:02It runs on a host machine which could be your laptop if you're running it locally or it runs in like a GitHub action runner, you know, if you're running it in CI.
26:11Um, and you can run the environment in a Docker container locally.
26:15That's like the default behavior.
26:17But if you're trying to do something more scalable, you would be running it on a sandbox provider in all likelihood.
26:22Um, so you'd be running it on Daytona.
26:25And so Harbor is this orchestrator um which is running on your laptop or running in GitHub actions which is um kicking off the run.
26:34It's basically saying, you know, hey, start start the agent, right?
26:38Um, and the way it does that is, and I can show you what this looks like on the file system, but it basically, you know, um, expects a particular script.
26:49Harbor expects a particular script to be written for the task, right?
26:53It's all driven via scripts.
26:55So, you like when you're when you're setting up Harbor, you have to kind of make sure that all these scripts are in the right place in the Docker image.
27:02Um, so there's going to be a script to start the task.
27:05There's going to be a script to run the verifiers.
27:06There's going to be a script to run the oracles.
27:08It's all driven by scripts.
27:10So the scripts are in the Docker container.
27:12Um, but doc but Harborself is a program that lives on a on a host machine and just kind of orchestrates things and kicks things off.
27:20That's right.
27:20And the interf Yeah.
27:21So maybe I'll just repeat what he what you just said because I think it's well put is like Harbor is a CLI um and it has certain conventions for if you like make your docker container or your docker image look this way it will be able to run it um and those conventions are almost all like
27:39is there a script in the right place so that's kind of like the contract right if there was a a contract that you had to honor with harbor so that your your image would run it would be like did I put my scripts in the right place right That's kind of the contract.
27:52Got it.
27:53Yeah.
27:53Does that Does that clarify?
27:54Yes.
27:55Okay.
27:55Yeah.
27:56So again on this one on this slide, right?
27:58Basically I think the takeaway is um the takeaway on this slide is like you have a great deal of freedom um and flexibility when it comes to customizing your harbor environment, right?
28:09Um the the simplest like the simplest example that people always get started with is the example of a coding agent operating on a git repo.
28:18But this example is different, right?
28:20This example is like I literally put some data from a SQL life file in there.
28:23I threw my whole application in there with it along with its APIs along with its own serverside agent.
28:29It's all it's a running server in the environment in the Docker container.
28:33Um something you know there's a CLI in the environment.
28:37So what's not even included here is like there's actually a set of skills literally in the environment on the file system that the agent has access to.
28:45Um, so all that is to say is like you have a lot of flexibility in setting up your environment.
28:52Um, you know, it has to fit in a Docker image, but if you can fit it in a Docker image, um, you know, you can you can make Harbor run it basically.
29:04All right.
29:04And now any other any other questions before I move on?
29:08Maybe because this is a complex slide.
29:09I think this is and and you know this is like again this slide is very much like particular to to our project right this is like what our project
29:18looks like your project may not look like this right if you if you're not running like a single node server or something but yeah I mean that's such a good question and I haven't had to deal with that problem so it would require a lot of thought I think right but basically the question is like
29:32my system is not a single node it's not a single you know I can't just like throw in a SQL single SQLite file I have like you know, some distributed system with like S3 and all this other stuff that I'm trying to maintain.
29:44How would I test harbor out with that?
29:46The answer is like that's probably harder.
29:48Yeah, that's probably harder.
29:50So, in that case, in that case, my initial thought is like, you know, you have to perhaps carve out your agent a little more, right?
29:59So, here we've had this advantage where like it's simple for us to throw the whole damn thing in there in the Docker container.
30:05And so, we did.
30:06If that wasn't simple to do, if you can modularize your agent somehow into some kind of like interface, hide hide the implementation behind an interface um and if you can like if there's some like stable API boundary or interface
30:20that is relatively stable, right?
30:22Where you know um I'm feeding like this agent data in this format, right?
30:28Whatever that format may be, like eventually this agent has access to data in this format, right?
30:34Um, that would be one approach, right?
30:39Um, and I realize that that is maybe not a satisfactory answer depending on what your exact setup looks like because maybe your agent like literally queries S3 and then like what are you to do, right?
30:50Are you going to mock out all those S3 calls?
30:52And then once you start mocking out S3 calls, you're kind of in a world of pain because now you're in the world of mocking stuff again, right?
30:58And and that's really like what we're trying to get away from with Harbor.
31:01We were trying to get away from that world.
31:03Um, so I don't know that there's like I actually don't know because I haven't had to explore this problem yet.
31:08Um, I do know that you know I've been kind of like eliding this particular feature of Harbor, but Harbor does have the ability to um like run multiple Docker images, right?
31:19Spin up multiple things.
31:21So like you could you could wonder I would wonder about that is like oh is there a way where I could somehow encapsulate my application or something?
31:28Yeah, go ahead.
31:29The question was like what scale do you use?
31:31Right?
31:32if you have some massive SQL life file or some you know really large set of traces do you like actually evaluate on the whole thing we don't currently do that and I think it would probably be okay for us to do that um
31:44right now we you know right now the kind of format is like oh like we have some data here I'm going to have Claude kind of help me clean it up a little bit for this particular set of tasks and again the kind of the nice thing about Harbor is you know you you can
31:57um use the exact same data for many different tasks but you don't have to.
32:02Um, and so if there's a particular question that you care about and it only requires a small subset of the data to answer, you don't have to like upload all of the data to answer that particular question.
32:13Um, but yeah, I mean it it just depends too like on what you're trying to test, I think, right?
32:19If part of what you were trying to test was the performance of your agent when the system is full, when the database is like quite full, then I think it would totally make sense to um
32:29put in a lot of data.
32:30We we haven't done that yet.
32:32Well, I tried to Yeah, that's such a good question.
32:34I think the question is like what is what what do you get by using Phoenix with Harbor?
32:37Why not just use Harbor maybe?
32:39Um and the answer is like you totally can use just Harbor, right?
32:43I'll I'll give you a little demo um in just a second where hopefully some of some of what we offer is is on display, right?
32:49Because I I do think that there are some genuinely unique things around like data set versioning, around the experiments, around the visualizations, around the the agent experiences that you get.
33:00um the ability to like actually inspect and drill down quickly, right?
33:04So, we've been we've been solving those kinds of problems um for several years and and our UI is like really dialed in for that sort of thing.
33:12And I'll give you a demo in just a sec.
33:14Yeah.
33:14And then you can ask me again, see if I answered the question.
33:18All right.
33:18Uh All right.
33:21I'm going to go to the next slide.
33:22Right.
33:23So this particular example that we're going to think about is um this problem that we encountered recently.
33:30Uh the problem was you know in this picture that I've shown you I talked about rest and I talked about GraphQL.
33:35So our MCP server our CLIs they had access to these APIs but we found that people were asking questions that are not easily answerable via REST or GraphQL.
33:45So they were asking aggregate kind of questions, right?
33:48They were asking, you know, what was the busiest day that I had in the last few months or something, right?
33:54Um how many traces did I get on that day?
33:56Or they were asking questions like how much am I spending on uh each of these models, right?
34:01Like which model is actually costing me the most out of all my LLM calls, right?
34:05Where am I spending the most?
34:07Um and so what the agents would try to do is they would uh you know when possible they would uh look for the the rest route or the uh
34:17the GraphQL resolver.
34:18Um but these were not aggregate resolvers.
34:20So they would try to like basically pageionate through stuff and and do the aggregation themselves.
34:26And as you can imagine that results in a problem because if there's enough data you wind up uh you wind up doing a lot of a lot of pageionations
34:34right and a lot of tool calls.
34:35Um, and so part of what we did was we actually added this this line right here.
34:42So this line uh is meant to represent a tool that we added to our MCP server um, which basically allows the agent to write a readonly SQL query and execute it against the database.
34:57Um, and a lot went into making that tool.
35:01I won't really go into the implementation details but the thing that we noticed even after we made that tool was that oh sometimes sometimes the agent isn't calling the right tool because now the agent has this other problem right
35:13now the agent is here and it's like well I got rest I got graphql and I got SQL which one which one should I be using the agent doesn't really know and so we realized oh yeah well naturally we have to like somehow update a prompt we got to tell it what to do right
35:26um but even after you tell it what to do how do you verify that it's actually doing it Okay.
35:31Um, you know, it's one thing to change the prompt, but again, we're trying to like ship based on some evidence, right?
35:36And not just on vibes.
35:38Um, so this is this is the change that we we wound up testing out.
35:41It's like a small little tweak to a to a prompt, right?
35:45So before it it only mentioned uh GraphQL, but after it says, "Hey, if you're dealing with like a some kind of aggregate like a count or group, um why don't you use our SQL tool rather than reaching for GraphQL first?"
35:59Right?
36:00Um, and of course the most important the most important thing that we need in order to answer this question is first of all some like realistic data.
36:08So we throw in a you know semi-realistic um SQL life file that we've been kind of just like collecting as we develop ourselves as we work ourselves and like dog food the product ourselves.
36:19Um, but then here's the really the part that I found quite cool or novel about Harbor, which is like um I literally just asked Claude like can you look at like the domains of my application,
36:32you know, traces, spans, um, data sets, experiments.
36:37Can you look at all these domains?
36:38Can you brainstorm a list of user queries, some of which would be aggregate, some of which would not be aggregate, right?
36:45And then can you write an Oracle solution to those queries, right?
36:49Um the Oracle solution literally just being a REST call or perhaps a database query and the coding agent will just do that.
36:58So we generated just this data set of synthetic questions.
37:02Um I spot check the questions, right?
37:04Just to make sure they sort of make sense.
37:06Um and I have a certain degree of confidence in the actual verifiers which again are just scripts that are that are running in the environment.
37:14But because those verifiers um are passing against an oracle again those verifiers are just like you know kind of passing against this oracle I have a certain amount of confidence in them right some of them were LLM as a judge
37:26some of them were SQL queries or or REST calls themselves but you know just the ability to kind of work out those kinks before the agent ever gets to the system it gives you a certain level of confidence
37:37um and then you know I was able to test out this particular change and I was able to see like, oh, you know, as you might expect, making a change like this, it causes the system to use SQL more frequently and more aggressively,
37:50which is what I wanted it to do, right?
37:51I wanted to overuse SQL rather than underuse it because I don't want it to like start trying to pageionate through trying to do like aggregate queries.
37:59That would be bad.
38:00And then the other interesting thing is just by changing this one prompt I can also see the number of tool calls in each in each uh trajectory the average number of tool calls dropping which again is like also
38:12you know the reason that we did this in the first place is I wanted to reduce the number of tool calls and this data set it wasn't even like a big
38:19data it wasn't a big data set so it didn't have to do a lot of pageionations I think it these numbers like these modest drops they kind of understate
38:27the issue because the data was not large and So let me let me show you now what this looks like inside of Phoenix.
38:34So the first thing that you can see here um are the examples in our data set, the tasks in our data set.
38:41And again, these are versioned.
38:43So we actually have a whole version history here every time there's a change.
38:48Um and we can inspect what some of these look like.
38:51So again, some of these are aggregate queries, some of them are not.
38:55So here's an example of an aggregate query that I might care about.
38:58What are the three most expensive models by total cost?
39:01How much does each cost?
39:02So this question is more easily answered with a SQL than than by pageionating through like a REST API.
39:10And down here you can see the answer and again this answer um it comes from an oracle.
39:15Um and so that's part of why I have a certain level of faith in it is because it's it's coming from a SQL query run against a concrete set of data.
39:24It's not something that I feel um a strong need to check actually for this particular example because the answer is deterministic and I'm I'm sort of allow I'm I'm placing a certain amount of trust in Claude I guess to like write a good SQL query.
39:37Um so we got all our examples here and then we got our um we got our experiments and our notion of an experiment that's that's what I've been calling a job in Harvard.
39:50So just imagine this to be all the jobs that we've run.
39:52I've run a lot of jobs in preparation for this talk.
39:55Um, these charts show you the kinds of metrics that you might care about, whether those are cost or run latency.
40:01Uh, the reward, as you can see, for this particular task is quite high.
40:05Um, and that's because I haven't really done anything that's going to like hopefully break the system.
40:10It's not like something was truly genuinely broken.
40:13The system was getting the right answer in the beginning.
40:15So, the reward was always high.
40:17Um, it's just that it was just taking too many tool calls, right?
40:20So this was more of an optimization rather than a correctness fix.
40:24You can see tool call count.
40:25You can see um whether or not we use SQL.
40:30How do you calculate the reward?
40:32So the reward if we go back in the slide, the reward is just coming from the verifier.
40:39And the verifier is something that you the user have in your control.
40:44Right?
40:44So when you're defining your task, for each task, you get to write a script that says what constitutes success or failure.
40:55And that script is going to emit a zero or a one, a pass or a fail.
40:59You get to name the reward, right?
41:01So the reward could have a different name other than pass fail.
41:04Um that's all a reward is though.
41:06you as the developer, you as the maintainer, you think about for this particular task, what what does it mean to succeed and what does it mean to fail?
41:15How could I figure out just by looking at the side effects in the environment, looking at the file system or database file for example, how could I figure out whether the agent succeeded
41:24and that's what the reward is and so that's what we're tracking over time.
41:27There are the aggregate rewards for the entire job.
41:31That's what we're tracking in this view.
41:32Yeah, go ahead.
41:34I mean this is such a good question, right?
41:35I kind of get it kind of like gets to one of these big big qu big big picture questions about like what do you use Harbor for?
41:41Actually the question was around like well hey you said that you know typically your verifiers are agnostic with respect to the the trajectory or the execution path of the agent but hey like look some of your verifiers actually like kind of it looks like they're peeking in to see whether the agent use SQL.
41:55Isn't that cheating?
41:56Um the answer is yes.
41:57So it's not I think the times when you would do what I said in the purest form of like your verifiers are truly agnostic and they really don't look at the trajectory and they really just look at the file system you know or the the side effects
42:10the situation for that is really like um when you care about correctness right so may you know one of the other things that I'm not showing here which is a different example that we've done is like we've worked on some regression tests and those tests are really like
42:24does it get to the right answer in the end and for Those tests I think our verifiers are quite pure.
42:30They don't really look at the trajectory.
42:32They just look at like oh did it get the right answer right?
42:34For this one for this particular problem I was concerned with something maybe subtly different.
42:38Like I wasn't really just trying to get the right answer.
42:41I was trying to optimize a metric.
42:43Right?
42:43I was trying to optimize tool calling in particular tool calls.
42:46Harbor is one of these things where you can use it for many things.
42:48You could use it for uh you know I I talked about integration testing.
42:51You can use it for like kind of endto-end testing um for assuring correctness um for running in CI right or you could also use it because you're like oh right now I'm trying to solve a really particular problem which is like oh I kind of noticed that it sucks on these aggregate queries like how can I make
43:06it better how could I in a quick and dirty way kind of convince myself that it's actually doing what I hope it's doing right um so like at least
43:15for for us like what I found so far is it just depends on what you're trying to do um sometimes sometimes it is worth peing inside the trajectory I think
43:23you know
43:24[laughter]
43:24yeah even though even though it makes the test less robust to change perhaps that that's a really good question you know my team is so small I don't know that we can I can really speak from personal experience
43:36um that being said like one of the reasons that you you might use a tool like Phoenix is because it allows you know it has like the ability to
43:44share data right to curate data from different sources to add it you know to to like add traces to a data set um it has the ability to like store data as a you know in an external
43:55file and like upsert it to the database and then run it has those kinds of abilities.
43:59So like that might help you in terms of the the uh logistics of like you know actually like dealing with this in a larger team.
44:06I I don't I don't know because really it is just me and a handful of people working on this right now.
44:10All right, I'm going to show you one last little demo and then I'll just take questions.
44:13I think we got only a handful of minutes here left.
44:15So you know back to back to your question around like what are some of the things that you might get out of Phoenix.
44:20I think this is like kind of the the one of the bigger ones, right?
44:23Is like the ability to compare individual jobs at the task level, right?
44:28So right here I have two two particular jobs selected.
44:31I have opus 5 before and after making that single prompt change.
44:36This column is opus 5 before.
44:38This column is opus 5 after these these metrics at the top.
44:41They're telling me in aggregate how did these like metrics that I care about, how did these rewards change after I made this tweak to the prompt, right?
44:49And I can see that like GraphQL tool calls went down.
44:53So it's calling GraphQL in aggre in it's calling GraphQL less than it was.
44:57Um the reward is staying the same which means everything is still passing.
45:01Right?
45:03SQL calls are going up.
45:05Tool counts are going down.
45:07The number of turns the agent is taking is going down because it's encapsulating a lot of that a lot of those turns that would have been multi-turn steps into a single turn because it's using SQL.
45:17uh it's using more SQL, it's using less GraphQL, right?
45:20So you get those kind of like top level metrics which are the ones that in my mind verify, okay, this is a decent change.
45:26Um and then I'm able to see in this table like four particular tasks, right?
45:31Here's the here's that one we looked at like what are the three most expensive models by by total cost.
45:37I'm able to see what I expected the answer to be, right?
45:40I'm able to see the actual answer in each job.
45:43I'm able to see the metrics and how they changed.
45:45So for this one um yeah for this one the number of tools went down the number of turns went down interestingly enough the number of SQL calls actually went down too which is kind of unexpected
45:57and I can do things like I can I can filter by these right so I can here's a fun little feature that Miko built recently um which is I can say show me the runs where
46:08the second experiment had um where the second experiment ment had lower tool call count and I'll use the exact the exact metric to help it out.
46:25Lower tool count than the first experiment.
46:30We'll see if this works.
46:31So, what this is doing is it's actually running a a local model in the browser to convert that piece of natural language into a query DSL.
46:42I think it messed that up though.
46:46We gota we got to use a more powerful one.
46:49The prompt API is a little bit
46:52I'll just I'll just copy in what I know to be the correct uh correct expression here.
47:02So I'm able to filter down.
47:03So now I'm just looking at the um now I'm looking at the rows where it improved basically like I'm looking at the ones where it didn't use SQL before where now it's using SQL.
47:14Um and you can see here this one used SQL it went up by one and if we actually like look at the uh look at the trace we can actually see that right so those
47:25ATIF trajectories that get converted into open inference traces here you can see it you can see it was using bash before um and when it's using bash under the hood it is actually using graphql
47:37it's calling has a bash emulator which is calling graphql underneath the hood right so I'm able to see that right here I'm able to see in the second job
47:46the same trace or the analogous trace and I'm able to see oh instead of instead of calling bash and using graphql here it actually issued a SQL query directly to the database so I'm able to like visually inspect and say oh okay
48:00in this particular example it did the right thing um and I can even I can even ask you know the agent itself so this is kind of like what we do
48:08um you know I can literally ask the agent itself like hey look at a particular um set of experiments, you know, look at this particular task and it will tell me
48:18exactly what happened.
48:19And so this is kind of like how we get some of our data in the first place is we're kind of like using using the agent um recording its traces
48:27uh getting its data and using the agent to like iterate on itself.
48:31Yeah, I'd say with respect to the regression stuff, we're still really early days.
48:34I mean, we're trying to get like happy paths going, I would say.
48:37Um, and I think like the way that the way that we're structuring it is is around skills because I think in my mind when you're doing those regression tests, they are sort of analogous to like playright in a certain way where it's like an endto-end
48:51thing and it just includes like an agent in the loop.
48:54It includes like LLM calls in the loop.
48:57Um, and so I think skills are a really good like thing to write those kinds of tests for because at least in our case like the skills are describing workflows.
49:08They're saying, "Oh, if this happens to this, if that happens to this, right?
49:11It's describing a workflow that you want to have happen and and so right now we're just kind of in the process of trying to nail down happy paths on our most important skills.
49:20um and ensure that the happy path passes across like the the broad array of like models and providers that we're trying to support.
49:28Right.
49:29Right.
49:29Right.
49:29Yeah.
49:29So this this particular comment is around like the ability of uh you could make the the greatest skill in the world, but like if somebody else makes another skill that kind of like where the front matter of the skill kind of treads on your feet a little bit, it's going to it may not fire anymore, right?
49:44And then it's no good if it never fires.
49:46Um, and for that sort of thing, it's an interesting question, right?
49:52It's like there's multiple ways of approaching it, right?
49:56Like one thing that you could do would you you could just say, well, I don't care which one fired as long as I have a good test and I know that like, oh well, like if this skill
50:07didn't fire, it's probably not going to pass this test, right?
50:10Um, that would be one approach.
50:11Another thing you can do and maybe this is the appropriate thing right is you can always as we said reach into the trajectory with your verifiers you can always look in you can always peek you know you can cheat a little bit you can say oh well like given this question
50:24did it call this particular skill and we have had tests like that um I think the challenge is maintainability right it's always around maintainability um if you change the skill name or something the test is going to flake
50:39um but we've had tests where the thing we were trying to test was skillfiring, right?
50:44And those tests are not they're not the hardest to write, right?
50:48I mean, don't even because if if you're only testing one turn and all you want to know is like, did the right skill fire?
50:53Um, you don't even really need great data for that.
50:56You just ask it a question and and see did the skillfire.
50:59Um, and like you can totally, you know, maintain a suite of tests that that do that, right?
51:05And and you know, then it becomes less of a end to end test.
51:10it becomes more of like a almost like it's like almost like you know on the spectrum between you know fully decoupled from the implementation and coupled to the implementation is a little bit more coupled
51:22um and yet it could still be useful to you right um so there's no reason to not write a test like that um there's no reason to not write a test like that I think so it really just depends on like you know getting getting
51:36a broad level of coverage um on the things that you care about.
51:40We are collecting traces in production.
51:44So when you deploy Phoenix in production, this is the picture.
51:46The traces are coming into the system from an instrumented application or agent.
51:51We're collecting those traces and we're writing them to the database.
51:55When we're running things in harbor, you can see [clears throat] that this this thing disappears, right?
52:00the OTOP ingest, the AI agent, which is the user's agent, those disappear and we just seed the database with data.
52:10So when you're running when we evaluate in harbor, um we don't we don't pretend or emulate an end user's application.
52:18Um we assume that the end user's application exists and has resulted in the data in the in the SQL life file.
52:25Yeah.
52:25So the question is around um Harbor is canonically used for evaluating coding agents.
52:32That's the canonical use case and we're doing something similar on the one hand but the unit that we're trying to evaluation to to evaluate here is not the coding agent in isolation because we don't work on cloud code, right?
52:47We don't we don't tweak cloud code.
52:49Uh we don't tweak codecs, right?
52:51The thing that we're trying to optimize for is this hybrid system which is cloud code plus our particular bespoke MCP server or cloud code plus our particular bespoke set of skills
53:05that we maintain or cloud code plus our um cloud code plus uh our CLI or cloud code um interacting with our APIs.
53:21And that is a different problem, right?
53:24And the fact that that is the thing that we're trying to optimize this hybrid entity which includes cloud code, it includes the coding agent, but it's more than just the coding agent.
53:32It's all of the um supporting infrastructure around the coding agent which we maintain which helps cloud code be better at interacting with our product.
53:42Um, that's the reason that we've shoved all of this stuff into an environment together is because all of that stuff has to change um and be evaluated together um as a unit.
53:55I think it took a second to kind of conceptually land on and understand Harbor well enough to land on this picture.
54:02Um, and then it took a second.
54:05it didn't take super long and again like I think part of that is part of that is due to our advantage which I mentioned which is that Phoenix runs on a single node
54:14um and that makes it easy to dockerize right and so as as some of these other folks suggested it's like you know if you're working on a distributed system
54:22you know it's it's tougher your your life is going to be a little tougher right and so we had a certain advantage that made it easier for us to set this picture up
54:29um but I would say like the coding agents are Um, Harbor's tough in the sense that like there's no because the contract that Harbor tries to honor is all based on like scripts in certain directories.
54:44You know, I need to put my my task my task initiation script here.
54:49I need to put my verifier script here.
54:51Um, there's no like llinters that the coding agents can rely on to make sure that that works.
54:56Um, the best that you can do uh is you can download Harbor's skills.
55:02Um and those will help you set up your project correctly.
55:05Yeah.
55:06So the question is uh I have a SQLite file here.
55:09Where do I put it?
55:10Right.
55:10Um cloud storage.
55:12Yeah.
55:12So basically um it's kind of a one-off process right now.
55:18It's like we have a collection of data that we generally use and if that thing works for the task at hand, I tend to reuse that file.
55:26And if it if I don't need the whole file, I'll say Claude, can you clean this up?
55:30[laughter]
55:30can you can you cut out the stuff I don't need um and push it to cloud storage for me?
55:35Um that data though, the data is the hard part, right?
55:38I think the data getting the data is the hard part um for for any of these tasks.
55:43And the more realistic your data, the better the system is going to run uh the better it's going to like uncover realistic issues for you.
55:52So it's always the hard part getting data