WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Ib5t2RLtxvM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Ib5t2RLtxvM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.960 --> 00:00:13.600
Okay.

s3
00:00:13.600 --> 00:00:13.840
Oh, yeah.

s4
00:00:13.840 --> 00:00:15.000
Thanks everyone for coming.

s5
00:00:15.000 --> 00:00:17.600
And I know this is the last session before lunch.

s6
00:00:17.600 --> 00:00:19.080
So, thanks for staying here.

s7
00:00:19.080 --> 00:00:21.280
Let's make it smooth and with good vibes.

s8
00:00:21.280 --> 00:00:22.720
Just as that said.

s9
00:00:22.720 --> 00:00:25.000
And thanks that for introduction.

s10
00:00:25.000 --> 00:00:26.280
And for inviting me.

s11
00:00:26.280 --> 00:00:27.000
So, yeah.

s12
00:00:27.000 --> 00:00:27.920
My name is Rustam.

s13
00:00:27.920 --> 00:00:31.600
I'm leading AI platform team uh at Snorkel.

s14
00:00:31.600 --> 00:00:42.280
And uh today I want to tell you how to turn agent traces into agent simulations and why this uh becomes the next stage for agent evaluations.

s15
00:00:42.640 --> 00:00:49.160
So, three main things that I want you to take away from my talk is uh every company needs a benchmark.

s16
00:00:49.160 --> 00:00:54.360
It's the only way to reliably evaluate, release, and improve your agents.

s17
00:00:54.360 --> 00:00:57.320
It has to be as close to production as possible.

s18
00:00:57.320 --> 00:01:04.480
Uh it has to mimic your real tools, real API services, policies, and workflows.

s19
00:01:04.760 --> 00:01:09.440
And finally, it has to be part of your agentic life cycle.

s20
00:01:09.440 --> 00:01:11.680
It's not a static benchmark.

s21
00:01:11.680 --> 00:01:17.360
It's a constantly populated data set from your production traces.

s22
00:01:17.440 --> 00:01:21.040
So, why is Snorkel AI giving this talk?

s23
00:01:21.040 --> 00:01:27.520
We are uh data as a service company and we uh basically selling uh benchmarks.

s24
00:01:27.520 --> 00:01:29.240
And we're producing benchmarks at scale.

s25
00:01:29.240 --> 00:01:33.040
And for us, benchmark construction is an engineering discipline.

s26
00:01:33.040 --> 00:01:35.720
We run millions of agent simulations per month.

s27
00:01:35.720 --> 00:01:41.320
And uh we learned how to do like uh environment built at scale.

s28
00:01:41.320 --> 00:01:51.560
Working using both agents uh and subject matter experts to build reliable benchmarks that are close to production and uh specific domains.

s29
00:01:52.120 --> 00:01:56.920
So, a lot of the time when people say about agent evaluation, they're focused on traces.

s30
00:01:56.920 --> 00:01:58.400
And traces are very useful.

s31
00:01:58.400 --> 00:02:01.640
The usual traces like you can see an example on the screen.

s32
00:02:01.640 --> 00:02:04.640
It basically shows, "Okay, here is the input prompt.

s33
00:02:04.640 --> 00:02:09.200
Here are the actions that agent took, and here is the agent output."

s34
00:02:09.200 --> 00:02:13.520
And then relation can analyze it and and say, "Okay, was agent successful or not?

s35
00:02:13.520 --> 00:02:16.000
Was there any edge case?"

s36
00:02:16.000 --> 00:02:22.800
Uh so, it is useful to find failures in production, but it's hard to test different variants.

s37
00:02:22.800 --> 00:02:35.040
You can run AB testing, and that's one way of checking different agent configurations, but it's hard to make sure that everything is repeatable because you will get different database state, different tool versions, and so on.

s38
00:02:35.040 --> 00:02:38.600
So, never fully compare apples to apples.

s39
00:02:38.840 --> 00:02:42.200
Offline simulation turns traces into repeatable experiments.

s40
00:02:42.200 --> 00:02:53.280
Now you take production traces, you construct tasks, and then you can run simulation benchmark um with different agent configuration offline.

s41
00:02:53.280 --> 00:03:00.680
And you can compare agents using different metrics, not just success rate, but cost, latency, and retries.

s42
00:03:00.680 --> 00:03:03.560
And you can run those in parallel.

s43
00:03:05.519 --> 00:03:07.519
But you can ask, "Okay, but why do we need it?

s44
00:03:07.519 --> 00:03:09.320
Like we already have public benchmarks."

s45
00:03:09.320 --> 00:03:14.440
The challenge with public benchmarks is that usually they are focused on a very specific domains.

s46
00:03:14.440 --> 00:03:24.160
For example, SweepBench is focused on like uh fixing GitHub issues, TerminalBench will focus on agent running in terminal, and Co- Bench will focus on computer use agent.

s47
00:03:24.160 --> 00:03:41.320
In your case, you want your benchmark to be focused on your company's domain both from perspective of use cases and in terms of tooling that your agent has, whether it follows the policies that your company uses, and whether you get full production environment.

s48
00:03:41.720 --> 00:03:52.280
Basically, public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship.

s49
00:03:52.560 --> 00:03:57.680
And a lot of the time public benchmarks they're specifically focused on pass rate.

s50
00:03:57.680 --> 00:04:03.320
Every time you see new model release, you see performance like pass rate on different benchmarks.

s51
00:04:03.320 --> 00:04:10.280
Which makes sense because it tells us like about the frontier of how good is the like new model.

s52
00:04:10.280 --> 00:04:14.840
But when you release agent to production, you also care about more metrics.

s53
00:04:14.840 --> 00:04:17.400
You care about cost for solving the task.

s54
00:04:17.400 --> 00:04:18.480
You care about latency.

s55
00:04:18.480 --> 00:04:20.840
You care about number of retries.

s56
00:04:20.840 --> 00:04:31.040
And by running evaluation offline with in simulations, you can effectively compare apples to apples and and iterate on agent.

s57
00:04:34.120 --> 00:04:38.200
In this case, you can test the full stack of your agent.

s58
00:04:38.200 --> 00:04:45.280
You're not just checking, okay, is model one is one model performs better than the other model.

s59
00:04:45.280 --> 00:04:48.520
You check even like thinking talk like thinking level.

s60
00:04:48.520 --> 00:04:49.720
You can change the prompt.

s61
00:04:49.720 --> 00:05:00.640
You can tune the full hardness and skills tools available to agent because in your production, you don't care about the model, you care about the full system.

s62
00:05:00.640 --> 00:05:10.760
And here you can configure and test the full system while keeping environment and evaluators the same between runs.

s63
00:05:12.560 --> 00:05:17.160
And that raise the point about like what benchmark is in this case.

s64
00:05:17.160 --> 00:05:20.720
Because first of all, you can use it to first release the agent.

s65
00:05:20.720 --> 00:05:22.400
You can make sure that it works.

s66
00:05:22.400 --> 00:05:24.040
You can handle edge cases well.

s67
00:05:24.040 --> 00:05:27.560
You can select the optimal model, and you can debug the traces.

s68
00:05:27.560 --> 00:05:30.120
That's how you make sure that it works.

s69
00:05:30.120 --> 00:05:31.600
Next, you can make it right.

s70
00:05:31.600 --> 00:05:41.480
You can put it as a release gate for your agent and verify that any change to agent stack in didn't reduce regression suddenly.

s71
00:05:41.480 --> 00:05:42.840
And iterate on hardness.

s72
00:05:42.840 --> 00:05:44.680
And finally, you can optimize it.

s73
00:05:44.680 --> 00:05:47.200
You can tune it for better cost or latency.

s74
00:05:47.200 --> 00:05:51.800
Or you can use the traces to even do error training.

s75
00:05:51.800 --> 00:06:05.040
Uh later I will say I will share link to our website where we have example how we used simulation environments to fine-tune small plan model to match performance of large plan model for specific tasks.

s76
00:06:05.040 --> 00:06:09.400
Here effectively, that becomes like trifecta of use cases.

s77
00:06:09.400 --> 00:06:23.480
Like for you, benchmark becomes part part of agent evaluation, becomes part of integration test for agent for release, and it becomes also training set for agent to improve it.

s78
00:06:24.160 --> 00:06:27.000
So, I hope I explained why you need benchmark.

s79
00:06:27.000 --> 00:06:30.480
So, now let's take a look like how can you actually construct it at scale for your company?

s80
00:06:30.480 --> 00:06:33.680
Like what is the anatomy of the benchmark task?

s81
00:06:33.680 --> 00:06:38.120
If you take a step back, like what is the sequence of running the benchmark?

s82
00:06:38.120 --> 00:06:39.640
It's straightforward.

s83
00:06:39.640 --> 00:06:48.600
Agent gets input prompt, it interacts with environment trying to solve the task with APIs, MCP tools, database, files.

s84
00:06:48.600 --> 00:06:57.440
Then it produces the output, trace, final state of the environment, and artifacts, basically output files.

s85
00:06:57.440 --> 00:07:05.919
Then we run verifiers, and we can produce the metrics how was agent able to solve the task, how well did it do, and so on.

s86
00:07:05.919 --> 00:07:09.400
The important second part of the task is Oracle.

s87
00:07:09.400 --> 00:07:23.160
When Oracle solution runs, it runs through the whole sequence, but just instead of running the real agent, it runs Oracle, and it and when we construct Oracle ourselves to make sure that task is solvable in the first place.

s88
00:07:23.160 --> 00:07:26.080
Because if it's not solvable, agent won't be able to solve it.

s89
00:07:26.080 --> 00:07:30.120
So, Oracle is important part of the task.

s90
00:07:30.120 --> 00:07:40.320
If we look at the anatomy of the like of the benchmark task, how it looks in terms of files, we can look at one of the most popular formats nowadays, Harbor format,

s91
00:07:40.320 --> 00:07:45.640
which was done by the same team who uh maintains Terminal Bench.

s92
00:07:45.640 --> 00:07:49.080
In this case, basically, it's just three set of files.

s93
00:07:49.080 --> 00:08:00.560
So, basically, what agent sees and interacts with, instruction.md, classic markdown file, environment, which you can see Dockerfile, could be Docker Compose in case you have multiple Docker containers.

s94
00:08:00.560 --> 00:08:07.160
You have something that agent doesn't see, which are which is Oracle solution and verifiers.

s95
00:08:07.160 --> 00:08:09.880
And finally, you have some metadata.

s96
00:08:09.880 --> 00:08:12.120
It may uh look very straightforward.

s97
00:08:12.120 --> 00:08:16.240
Yeah, I'm saying like, "Okay, simulation environment is just Dockerfile and a bunch of stuff."

s98
00:08:16.240 --> 00:08:23.040
But, it is useful because now you have repeatable way of running experiments in agent simulation.

s99
00:08:23.440 --> 00:08:26.800
So, now let's dive deeper into main parts of benchmark.

s100
00:08:26.800 --> 00:08:28.919
First is environment.

s101
00:08:28.919 --> 00:08:37.440
The main challenge with the environment is that effectively it has to be mini production, but you don't want to run full production for every experiment.

s102
00:08:37.440 --> 00:08:50.120
So, you want to make sure that uh your database, API service, tools, and files match production, just as like previous speaker mentioned like that you don't want your agent to know that it's running within simulation.

s103
00:08:50.120 --> 00:08:52.520
So, it has to be real.

s104
00:08:52.520 --> 00:08:56.240
One thing, though, you cannot put real user in your simulation task.

s105
00:08:56.240 --> 00:08:58.400
So, you can simulate the user.

s106
00:08:58.400 --> 00:09:10.360
In this case, that becomes effectively LLM with its own prompt, which with additional context that can mimic human behavior and interaction with your system.

s107
00:09:10.960 --> 00:09:18.360
So, in the port system that all these things exist in the environment, and verifiers just interact with the environment afterwards.

s108
00:09:18.600 --> 00:09:24.320
There are certain patterns to make sure that you can organize environment this way.

s109
00:09:24.320 --> 00:09:27.680
And think about it as like how you construct integration tests.

s110
00:09:27.680 --> 00:09:31.080
Basically, how you give effectively diff environment to your agent.

s111
00:09:31.080 --> 00:09:32.800
You don't run the full production database.

s112
00:09:32.800 --> 00:09:34.520
You have a certain snapshot.

s113
00:09:34.520 --> 00:09:38.320
You can run side containers sidecars in this case.

s114
00:09:38.320 --> 00:09:48.960
So, your uh agent runs in one main uh environment, but there are other containers which contain API services, databases, MCP tools, and so on, available to your agent.

s115
00:09:48.960 --> 00:09:52.720
You don't need to have like full production API services, you can mock them.

s116
00:09:52.720 --> 00:09:55.160
I already mentioned simulated users.

s117
00:09:55.160 --> 00:10:03.280
Uh and uh one important piece here is multi-step to handle long horizon task, if your agent needs to handle task that span hours.

s118
00:10:03.280 --> 00:10:14.600
You want to ensure that you have intermediate like steps, and for each step you have separate prompt, separate verifiers, and uh you can uh finish simulation early if you see agent failing.

s119
00:10:14.600 --> 00:10:20.960
And basically that enables you to simulate long-running horizon tasks.

s120
00:10:20.960 --> 00:10:23.560
So, next part is verifiers.

s121
00:10:23.560 --> 00:10:30.920
Like in traditional sense, usually when people speak about verifiers, you just verify the output.

s122
00:10:30.920 --> 00:10:33.280
You get agent output, you verify it, that's it.

s123
00:10:33.280 --> 00:10:34.960
That's why, let's say, how coding works.

s124
00:10:34.960 --> 00:10:37.240
It's like we just verify the output code.

s125
00:10:37.240 --> 00:10:39.000
We test, and so on.

s126
00:10:39.000 --> 00:10:40.839
Here it's more complex.

s127
00:10:40.839 --> 00:10:45.160
The way agent interacts with the simulation, we get a lot of different data.

s128
00:10:45.160 --> 00:10:49.400
We get the world, basically how environment changed in the final environment state.

s129
00:10:49.400 --> 00:10:51.040
What is your database state?

s130
00:10:51.040 --> 00:10:52.640
What are the API responses?

s131
00:10:52.640 --> 00:10:53.880
What were user replies?

s132
00:10:53.880 --> 00:10:55.280
And so on.

s133
00:10:55.280 --> 00:10:59.560
And your verifier analyzes final state, trace, and artifacts.

s134
00:10:59.560 --> 00:11:02.200
So, how can you analyze it?

s135
00:11:02.200 --> 00:11:04.040
Effectively, there are multiple ways to do it.

s136
00:11:04.040 --> 00:11:06.040
You can have deterministic checks.

s137
00:11:06.040 --> 00:11:14.920
Basically, uh and that can work really well for things like final output or tool calls, where like it's very easy to check whether it was correct or not.

s138
00:11:14.920 --> 00:11:24.520
It Sometimes you can use LLM as a judge, or even harness as a judge, or agent as a judge to evaluate basically whether the uh trace quality was successful,

s139
00:11:24.520 --> 00:11:29.000
whether um planning of the agent was correct, and so on.

s140
00:11:29.000 --> 00:11:31.640
In this case, it really depends on the use case.

s141
00:11:31.640 --> 00:11:37.120
So, you can use one or another or both depending on what's what works better.

s142
00:11:37.120 --> 00:11:44.000
And finally, it's important to keep in mind that you can use sub- uh subject matter expert to review some of the traces, some of the outputs.

s143
00:11:44.000 --> 00:11:53.920
Not for everything, but for cases where you see discrepancy in agent behavior and where you want human involvement.

s144
00:11:54.880 --> 00:12:05.880
So, final piece, how can we kind of organize everything together as part of agent release and agent improvement process?

s145
00:12:05.880 --> 00:12:08.120
So, can we just start?

s146
00:12:08.120 --> 00:12:08.640
Not just yet.

s147
00:12:08.640 --> 00:12:10.600
Can something go wrong with benchmark task?

s148
00:12:10.600 --> 00:12:12.400
100%.

s149
00:12:12.400 --> 00:12:19.360
Agent can try to reward hack simulation environment because it can understand that it's in simulation and it can hack it.

s150
00:12:19.360 --> 00:12:26.640
Task could be too simple and like our verifiers could be too broad and in this case agent will always pass even if it does something incorrectly.

s151
00:12:26.640 --> 00:12:30.680
It could be that agent always fail because verifiers are incorrect.

s152
00:12:30.680 --> 00:12:37.760
Or it could be that agents not perform in a stable way and like you have high variation of agent success.

s153
00:12:38.080 --> 00:12:47.680
So, all of these are effectively edge cases that you need to catch during your benchmark development because benchmark development is an art on its own.

s154
00:12:47.680 --> 00:12:59.040
We saw already like hundreds of benchmarks appear over the course of last years, but this is something that the card culture and engineering discipline that needs to be built

s155
00:12:59.040 --> 00:13:04.880
in each uh engineering team that needs to ship AI agents to production.

s156
00:13:06.640 --> 00:13:10.960
Because as you saw, effectively benchmark is software.

s157
00:13:10.960 --> 00:13:11.960
It's code.

s158
00:13:11.960 --> 00:13:12.880
It's files.

s159
00:13:12.880 --> 00:13:14.000
You need to treat it as such.

s160
00:13:14.000 --> 00:13:16.920
You need to have a separate CI pipeline for it.

s161
00:13:16.920 --> 00:13:28.680
E- And you can check pretty obvious things like for example, making sure that all dependencies are pinned or like your base image is correct or you don't have any missing fixtures,

s162
00:13:28.680 --> 00:13:33.160
then you can run Oracle uh solution and make sure that it passes.

s163
00:13:33.160 --> 00:13:43.360
Or like if you don't run Oracle that verifies fail, you can run several agent runs uh on the task and verify that okay, it is solvable and it is hard for agent.

s164
00:13:43.360 --> 00:13:51.440
You can tag the task whether it's simple, medium, or hard depending on how much time how many times the agent is successful.

s165
00:13:51.440 --> 00:13:56.240
And finally, you can approve it to make it part of your benchmark.

s166
00:13:57.520 --> 00:14:01.320
In this case, the process for improving agent becomes straightforward.

s167
00:14:01.320 --> 00:14:14.600
You establish baseline in like on your benchmark, you run evaluation data set, you see the failures where it doesn't perform well, you change one thing, you rerun experiment and where you can use something like Arize

s168
00:14:14.600 --> 00:14:15.960
to record your experiments.

s169
00:14:15.960 --> 00:14:20.160
And then once you fix it, you rerun the full experimentation again.

s170
00:14:20.160 --> 00:14:23.200
And then you can finally release to production.

s171
00:14:24.040 --> 00:14:29.080
What it unlocks is to make sure that you fix issues correctly.

s172
00:14:29.080 --> 00:14:34.839
There is a bit of an anti-pattern in the industry where like folks try to fix things in the prompt.

s173
00:14:34.839 --> 00:14:39.120
And they populate the prompt with things like, "Never do this or only do that.

s174
00:14:39.120 --> 00:14:42.080
Never output this critical that."

s175
00:14:42.080 --> 00:14:46.760
Which is one way of handling it for sure, but with simulation, you control the full stack.

s176
00:14:46.760 --> 00:14:48.640
You can evaluate the full stack.

s177
00:14:48.640 --> 00:14:51.680
And you can make sure that fix lives in the correct place.

s178
00:14:51.680 --> 00:14:54.120
You don't push everything to the prompt.

s179
00:14:54.120 --> 00:14:58.520
You fix hardness if you want if you have context overload.

s180
00:14:58.520 --> 00:15:02.160
Or you can if there is a missing procedure, you put it into skill.

s181
00:15:02.160 --> 00:15:08.440
Or like if you need to have specific output schema, you put it as part of structured output.

s182
00:15:09.560 --> 00:15:14.839
So, once agent is in production, effectively, you will have two loops.

s183
00:15:14.839 --> 00:15:19.000
One loop for benchmark expansion.

s184
00:15:19.000 --> 00:15:25.680
You take uh, observability traces uh, traces from observability basically using something like a rise.

s185
00:15:25.680 --> 00:15:30.480
You record failures and use failures to build your benchmark further.

s186
00:15:30.480 --> 00:15:38.320
Then you have simulation runner that will run experiments on your on extended benchmark on a new or on new agent config.

s187
00:15:38.320 --> 00:15:40.520
Uh, we'll record these experiments.

s188
00:15:40.520 --> 00:15:48.680
Then you can use it for as a release gate whether, uh, agent performs significantly well and finally release it to production.

s189
00:15:48.680 --> 00:16:00.800
Uh, in this case it's important that you have system like it's important to under- to make sure that your observability piece and experimentation piece are connected.

s190
00:16:00.800 --> 00:16:05.280
Because they're part they're two sides of the same coin.

s191
00:16:07.000 --> 00:16:13.680
Um, so yeah, to summarize my talk, everyone needs every company needs a benchmark.

s192
00:16:13.680 --> 00:16:20.960
Traces are useful for like the finding edge cases in production, but simulation helps to, uh, test what would happen.

s193
00:16:20.960 --> 00:16:29.200
And finally, uh, you want, uh, your, um, benchmark to be part of your, uh, agent ops, uh, loop.

s194
00:16:29.280 --> 00:16:40.851
Thank you very much for for your time and, uh, I think maybe I will have answer for one question, but will be happy to answer any questions outside and please check our booth, uh, if you have any questions about benchmarks.

s195
00:16:40.851 --> 00:16:42.851
[applause]

s196
00:16:48.520 --> 00:16:49.400
All right.

s197
00:16:49.400 --> 00:16:51.640
A couple questions.

s198
00:16:51.640 --> 00:16:53.000
All right.

s199
00:16:53.000 --> 00:16:53.960
A great talk.

s200
00:16:53.960 --> 00:16:55.960
I have a question about structuring the benchmark.

s201
00:16:55.960 --> 00:16:58.920
How many examples you should include ideally?

s202
00:16:58.920 --> 00:17:04.040
And do you split them into, uh, like a train test split?

s203
00:17:04.040 --> 00:17:11.280
If so, like how do you recommend structuring examples, um, across those two splits?

s204
00:17:11.280 --> 00:17:11.880
Great question.

s205
00:17:11.880 --> 00:17:17.240
So, uh, the question was basically how can we structure to for training and validation?

s206
00:17:17.240 --> 00:17:22.920
And just as previous speakers uh shared, basically they also had a pattern of like having train validation split.

s207
00:17:22.920 --> 00:17:28.520
This is very close to traditional machine learning where we want to have standalone data set that agent didn't see.

s208
00:17:28.520 --> 00:17:31.000
But basically where we can verify agent config.

s209
00:17:31.000 --> 00:17:35.200
I think the classic approach applies when like you want to have 80/20%.

s210
00:17:35.200 --> 00:17:36.960
Always depends on the use case.

s211
00:17:36.960 --> 00:17:44.960
But uh you do want to have a standout data set that agent didn't see through the experimentation process.

s212
00:17:45.000 --> 00:17:45.600
Awesome.

s213
00:17:45.600 --> 00:17:47.760
Next question.

s214
00:17:47.760 --> 00:17:54.440
Um so when you create the benchmarks, um what data do you include in the benchmark?

s215
00:17:54.440 --> 00:17:58.920
Do you include production runs that you've handpicked into the benchmark?

s216
00:17:58.920 --> 00:18:08.000
Or if your agent is not yet in production, do you create uh data sets or problems to solve for the agent in the benchmark?

s217
00:18:08.000 --> 00:18:19.440
And if you're if you're like handpicking problems for the agent to solve, um how do you make sure you have like enough coverage uh similar to what you'd see in production?

s218
00:18:19.440 --> 00:18:20.480
Great question.

s219
00:18:20.480 --> 00:18:24.840
So the question is how to ensure coverage and distribution of benchmark is a very important piece.

s220
00:18:24.840 --> 00:18:27.960
So effectively you want to keep keep make sure like you have both.

s221
00:18:27.960 --> 00:18:34.880
You have bread and butter use cases, basically covering all main use cases that work, but also making sure you have edge cases.

s222
00:18:34.880 --> 00:18:41.360
Basically how you can make sure like your agent can handle edge case when tools fail or if like there is problem with database and so on.

s223
00:18:41.360 --> 00:18:45.760
So you want to have both uh it basically think about it as integration test.

s224
00:18:45.760 --> 00:18:49.400
You you have happy path, but you also have edge cases.

s225
00:18:49.400 --> 00:18:54.160
Would you simulate agent with LLMs or would you create hand-crafted business?

s226
00:18:54.160 --> 00:18:55.680
Yeah, uh great question.

s227
00:18:55.680 --> 00:19:00.880
Basically similar to us handcrafting, uh I think basically I mean people use agents to write code.

s228
00:19:00.880 --> 00:19:02.880
So a lot of things here can be automated.

s229
00:19:02.880 --> 00:19:05.800
So the what is handcrafted kind of changes.

s230
00:19:05.800 --> 00:19:15.360
Like the the most important piece that you can provide is like the build environment once and then build the context uh for basically that mimics your production.

s231
00:19:15.480 --> 00:19:24.400
Um you mentioned sometimes you use LLM or sometimes use human expert to build verifiers.

s232
00:19:24.400 --> 00:19:25.440
Can you elaborate on that?

s233
00:19:25.440 --> 00:19:28.880
What's the best practice over there?

s234
00:19:28.880 --> 00:19:29.760
Great question.

s235
00:19:29.760 --> 00:19:35.160
So, in our case basically, we have a lot of subject matter experts, so we do things at scale.

s236
00:19:35.160 --> 00:19:49.640
So, the important piece here is to you don't need subject matter experts to review everything, but you specifically want to find cases where there's disagreement between agent and different verifiers.

s237
00:19:49.640 --> 00:20:00.000
If you you think that basically task was supposed to be solved, but agent somehow marks it as like not solved correctly or like it marks that trace wasn't optimal enough.

s238
00:20:00.000 --> 00:20:07.000
And this is where you want to have subject matter expert that can basically tune the agent that does the review.

s239
00:20:21.589 --> 00:20:23.589
[music]