From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

AI Engineer · 20 min · 239 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:01[music]
2. 00:12Okay.
3. 00:13Oh, yeah.
4. 00:13Thanks everyone for coming.
5. 00:15And I know this is the last session before lunch.
6. 00:17So, thanks for staying here.
7. 00:19Let's make it smooth and with good vibes.
8. 00:21Just as that said.
9. 00:22And thanks that for introduction.
10. 00:25And for inviting me.
11. 00:26So, yeah.
12. 00:27My name is Rustam.
13. 00:27I'm leading AI platform team uh at Snorkel.
14. 00:31And uh today I want to tell you how to turn agent traces into agent simulations and why this uh becomes the next stage for agent evaluations.
15. 00:42So, three main things that I want you to take away from my talk is uh every company needs a benchmark.
16. 00:49It's the only way to reliably evaluate, release, and improve your agents.
17. 00:54It has to be as close to production as possible.
18. 00:57Uh it has to mimic your real tools, real API services, policies, and workflows.
19. 01:04And finally, it has to be part of your agentic life cycle.
20. 01:09It's not a static benchmark.
21. 01:11It's a constantly populated data set from your production traces.
22. 01:17So, why is Snorkel AI giving this talk?
23. 01:21We are uh data as a service company and we uh basically selling uh benchmarks.
24. 01:27And we're producing benchmarks at scale.
25. 01:29And for us, benchmark construction is an engineering discipline.
26. 01:33We run millions of agent simulations per month.
27. 01:35And uh we learned how to do like uh environment built at scale.
28. 01:41Working using both agents uh and subject matter experts to build reliable benchmarks that are close to production and uh specific domains.
29. 01:52So, a lot of the time when people say about agent evaluation, they're focused on traces.
30. 01:56And traces are very useful.
31. 01:58The usual traces like you can see an example on the screen.
32. 02:01It basically shows, "Okay, here is the input prompt.
33. 02:04Here are the actions that agent took, and here is the agent output."
34. 02:09And then relation can analyze it and and say, "Okay, was agent successful or not?
35. 02:13Was there any edge case?"
36. 02:16Uh so, it is useful to find failures in production, but it's hard to test different variants.
37. 02:22You can run AB testing, and that's one way of checking different agent configurations, but it's hard to make sure that everything is repeatable because you will get different database state, different tool versions, and so on.
38. 02:35So, never fully compare apples to apples.
39. 02:38Offline simulation turns traces into repeatable experiments.
40. 02:42Now you take production traces, you construct tasks, and then you can run simulation benchmark um with different agent configuration offline.
41. 02:53And you can compare agents using different metrics, not just success rate, but cost, latency, and retries.
42. 03:00And you can run those in parallel.
43. 03:05But you can ask, "Okay, but why do we need it?
44. 03:07Like we already have public benchmarks."
45. 03:09The challenge with public benchmarks is that usually they are focused on a very specific domains.
46. 03:14For example, SweepBench is focused on like uh fixing GitHub issues, TerminalBench will focus on agent running in terminal, and Co- Bench will focus on computer use agent.
47. 03:24In your case, you want your benchmark to be focused on your company's domain both from perspective of use cases and in terms of tooling that your agent has, whether it follows the policies that your company uses, and whether you get full production environment.
48. 03:41Basically, public benchmark is useful to orient and build your prior, but your private benchmark is useful to ship.
49. 03:52And a lot of the time public benchmarks they're specifically focused on pass rate.
50. 03:57Every time you see new model release, you see performance like pass rate on different benchmarks.
51. 04:03Which makes sense because it tells us like about the frontier of how good is the like new model.
52. 04:10But when you release agent to production, you also care about more metrics.
53. 04:14You care about cost for solving the task.
54. 04:17You care about latency.
55. 04:18You care about number of retries.
56. 04:20And by running evaluation offline with in simulations, you can effectively compare apples to apples and and iterate on agent.
57. 04:34In this case, you can test the full stack of your agent.
58. 04:38You're not just checking, okay, is model one is one model performs better than the other model.
59. 04:45You check even like thinking talk like thinking level.
60. 04:48You can change the prompt.
61. 04:49You can tune the full hardness and skills tools available to agent because in your production, you don't care about the model, you care about the full system.
62. 05:00And here you can configure and test the full system while keeping environment and evaluators the same between runs.
63. 05:12And that raise the point about like what benchmark is in this case.
64. 05:17Because first of all, you can use it to first release the agent.
65. 05:20You can make sure that it works.
66. 05:22You can handle edge cases well.
67. 05:24You can select the optimal model, and you can debug the traces.
68. 05:27That's how you make sure that it works.
69. 05:30Next, you can make it right.
70. 05:31You can put it as a release gate for your agent and verify that any change to agent stack in didn't reduce regression suddenly.
71. 05:41And iterate on hardness.
72. 05:42And finally, you can optimize it.
73. 05:44You can tune it for better cost or latency.
74. 05:47Or you can use the traces to even do error training.
75. 05:51Uh later I will say I will share link to our website where we have example how we used simulation environments to fine-tune small plan model to match performance of large plan model for specific tasks.
76. 06:05Here effectively, that becomes like trifecta of use cases.
77. 06:09Like for you, benchmark becomes part part of agent evaluation, becomes part of integration test for agent for release, and it becomes also training set for agent to improve it.
78. 06:24So, I hope I explained why you need benchmark.
79. 06:27So, now let's take a look like how can you actually construct it at scale for your company?
80. 06:30Like what is the anatomy of the benchmark task?
81. 06:33If you take a step back, like what is the sequence of running the benchmark?
82. 06:38It's straightforward.
83. 06:39Agent gets input prompt, it interacts with environment trying to solve the task with APIs, MCP tools, database, files.
84. 06:48Then it produces the output, trace, final state of the environment, and artifacts, basically output files.
85. 06:57Then we run verifiers, and we can produce the metrics how was agent able to solve the task, how well did it do, and so on.
86. 07:05The important second part of the task is Oracle.
87. 07:09When Oracle solution runs, it runs through the whole sequence, but just instead of running the real agent, it runs Oracle, and it and when we construct Oracle ourselves to make sure that task is solvable in the first place.
88. 07:23Because if it's not solvable, agent won't be able to solve it.
89. 07:26So, Oracle is important part of the task.
90. 07:30If we look at the anatomy of the like of the benchmark task, how it looks in terms of files, we can look at one of the most popular formats nowadays, Harbor format,
91. 07:40which was done by the same team who uh maintains Terminal Bench.
92. 07:45In this case, basically, it's just three set of files.
93. 07:49So, basically, what agent sees and interacts with, instruction.md, classic markdown file, environment, which you can see Dockerfile, could be Docker Compose in case you have multiple Docker containers.
94. 08:00You have something that agent doesn't see, which are which is Oracle solution and verifiers.
95. 08:07And finally, you have some metadata.
96. 08:09It may uh look very straightforward.
97. 08:12Yeah, I'm saying like, "Okay, simulation environment is just Dockerfile and a bunch of stuff."
98. 08:16But, it is useful because now you have repeatable way of running experiments in agent simulation.
99. 08:23So, now let's dive deeper into main parts of benchmark.
100. 08:26First is environment.
101. 08:28The main challenge with the environment is that effectively it has to be mini production, but you don't want to run full production for every experiment.
102. 08:37So, you want to make sure that uh your database, API service, tools, and files match production, just as like previous speaker mentioned like that you don't want your agent to know that it's running within simulation.
103. 08:50So, it has to be real.
104. 08:52One thing, though, you cannot put real user in your simulation task.
105. 08:56So, you can simulate the user.
106. 08:58In this case, that becomes effectively LLM with its own prompt, which with additional context that can mimic human behavior and interaction with your system.
107. 09:10So, in the port system that all these things exist in the environment, and verifiers just interact with the environment afterwards.
108. 09:18There are certain patterns to make sure that you can organize environment this way.
109. 09:24And think about it as like how you construct integration tests.
110. 09:27Basically, how you give effectively diff environment to your agent.
111. 09:31You don't run the full production database.
112. 09:32You have a certain snapshot.
113. 09:34You can run side containers sidecars in this case.
114. 09:38So, your uh agent runs in one main uh environment, but there are other containers which contain API services, databases, MCP tools, and so on, available to your agent.
115. 09:48You don't need to have like full production API services, you can mock them.
116. 09:52I already mentioned simulated users.
117. 09:55Uh and uh one important piece here is multi-step to handle long horizon task, if your agent needs to handle task that span hours.
118. 10:03You want to ensure that you have intermediate like steps, and for each step you have separate prompt, separate verifiers, and uh you can uh finish simulation early if you see agent failing.
119. 10:14And basically that enables you to simulate long-running horizon tasks.
120. 10:20So, next part is verifiers.
121. 10:23Like in traditional sense, usually when people speak about verifiers, you just verify the output.
122. 10:30You get agent output, you verify it, that's it.
123. 10:33That's why, let's say, how coding works.
124. 10:34It's like we just verify the output code.
125. 10:37We test, and so on.
126. 10:39Here it's more complex.
127. 10:40The way agent interacts with the simulation, we get a lot of different data.
128. 10:45We get the world, basically how environment changed in the final environment state.
129. 10:49What is your database state?
130. 10:51What are the API responses?
131. 10:52What were user replies?
132. 10:53And so on.
133. 10:55And your verifier analyzes final state, trace, and artifacts.
134. 10:59So, how can you analyze it?
135. 11:02Effectively, there are multiple ways to do it.
136. 11:04You can have deterministic checks.
137. 11:06Basically, uh and that can work really well for things like final output or tool calls, where like it's very easy to check whether it was correct or not.
138. 11:14It Sometimes you can use LLM as a judge, or even harness as a judge, or agent as a judge to evaluate basically whether the uh trace quality was successful,
139. 11:24whether um planning of the agent was correct, and so on.
140. 11:29In this case, it really depends on the use case.
141. 11:31So, you can use one or another or both depending on what's what works better.
142. 11:37And finally, it's important to keep in mind that you can use sub- uh subject matter expert to review some of the traces, some of the outputs.
143. 11:44Not for everything, but for cases where you see discrepancy in agent behavior and where you want human involvement.
144. 11:54So, final piece, how can we kind of organize everything together as part of agent release and agent improvement process?
145. 12:05So, can we just start?
146. 12:08Not just yet.
147. 12:08Can something go wrong with benchmark task?
148. 12:10100%.
149. 12:12Agent can try to reward hack simulation environment because it can understand that it's in simulation and it can hack it.
150. 12:19Task could be too simple and like our verifiers could be too broad and in this case agent will always pass even if it does something incorrectly.
151. 12:26It could be that agent always fail because verifiers are incorrect.
152. 12:30Or it could be that agents not perform in a stable way and like you have high variation of agent success.
153. 12:38So, all of these are effectively edge cases that you need to catch during your benchmark development because benchmark development is an art on its own.
154. 12:47We saw already like hundreds of benchmarks appear over the course of last years, but this is something that the card culture and engineering discipline that needs to be built
155. 12:59in each uh engineering team that needs to ship AI agents to production.
156. 13:06Because as you saw, effectively benchmark is software.
157. 13:10It's code.
158. 13:11It's files.
159. 13:12You need to treat it as such.
160. 13:14You need to have a separate CI pipeline for it.
161. 13:16E- And you can check pretty obvious things like for example, making sure that all dependencies are pinned or like your base image is correct or you don't have any missing fixtures,
162. 13:28then you can run Oracle uh solution and make sure that it passes.
163. 13:33Or like if you don't run Oracle that verifies fail, you can run several agent runs uh on the task and verify that okay, it is solvable and it is hard for agent.
164. 13:43You can tag the task whether it's simple, medium, or hard depending on how much time how many times the agent is successful.
165. 13:51And finally, you can approve it to make it part of your benchmark.
166. 13:57In this case, the process for improving agent becomes straightforward.
167. 14:01You establish baseline in like on your benchmark, you run evaluation data set, you see the failures where it doesn't perform well, you change one thing, you rerun experiment and where you can use something like Arize
168. 14:14to record your experiments.
169. 14:15And then once you fix it, you rerun the full experimentation again.
170. 14:20And then you can finally release to production.
171. 14:24What it unlocks is to make sure that you fix issues correctly.
172. 14:29There is a bit of an anti-pattern in the industry where like folks try to fix things in the prompt.
173. 14:34And they populate the prompt with things like, "Never do this or only do that.
174. 14:39Never output this critical that."
175. 14:42Which is one way of handling it for sure, but with simulation, you control the full stack.
176. 14:46You can evaluate the full stack.
177. 14:48And you can make sure that fix lives in the correct place.
178. 14:51You don't push everything to the prompt.
179. 14:54You fix hardness if you want if you have context overload.
180. 14:58Or you can if there is a missing procedure, you put it into skill.
181. 15:02Or like if you need to have specific output schema, you put it as part of structured output.
182. 15:09So, once agent is in production, effectively, you will have two loops.
183. 15:14One loop for benchmark expansion.
184. 15:19You take uh, observability traces uh, traces from observability basically using something like a rise.
185. 15:25You record failures and use failures to build your benchmark further.
186. 15:30Then you have simulation runner that will run experiments on your on extended benchmark on a new or on new agent config.
187. 15:38Uh, we'll record these experiments.
188. 15:40Then you can use it for as a release gate whether, uh, agent performs significantly well and finally release it to production.
189. 15:48Uh, in this case it's important that you have system like it's important to under- to make sure that your observability piece and experimentation piece are connected.
190. 16:00Because they're part they're two sides of the same coin.
191. 16:07Um, so yeah, to summarize my talk, everyone needs every company needs a benchmark.
192. 16:13Traces are useful for like the finding edge cases in production, but simulation helps to, uh, test what would happen.
193. 16:20And finally, uh, you want, uh, your, um, benchmark to be part of your, uh, agent ops, uh, loop.
194. 16:29Thank you very much for for your time and, uh, I think maybe I will have answer for one question, but will be happy to answer any questions outside and please check our booth, uh, if you have any questions about benchmarks.
195. 16:40[applause]
196. 16:48All right.
197. 16:49A couple questions.
198. 16:51All right.
199. 16:53A great talk.
200. 16:53I have a question about structuring the benchmark.
201. 16:55How many examples you should include ideally?
202. 16:58And do you split them into, uh, like a train test split?
203. 17:04If so, like how do you recommend structuring examples, um, across those two splits?
204. 17:11Great question.
205. 17:11So, uh, the question was basically how can we structure to for training and validation?
206. 17:17And just as previous speakers uh shared, basically they also had a pattern of like having train validation split.
207. 17:22This is very close to traditional machine learning where we want to have standalone data set that agent didn't see.
208. 17:28But basically where we can verify agent config.
209. 17:31I think the classic approach applies when like you want to have 80/20%.
210. 17:35Always depends on the use case.
211. 17:36But uh you do want to have a standout data set that agent didn't see through the experimentation process.
212. 17:45Awesome.
213. 17:45Next question.
214. 17:47Um so when you create the benchmarks, um what data do you include in the benchmark?
215. 17:54Do you include production runs that you've handpicked into the benchmark?
216. 17:58Or if your agent is not yet in production, do you create uh data sets or problems to solve for the agent in the benchmark?
217. 18:08And if you're if you're like handpicking problems for the agent to solve, um how do you make sure you have like enough coverage uh similar to what you'd see in production?
218. 18:19Great question.
219. 18:20So the question is how to ensure coverage and distribution of benchmark is a very important piece.
220. 18:24So effectively you want to keep keep make sure like you have both.
221. 18:27You have bread and butter use cases, basically covering all main use cases that work, but also making sure you have edge cases.
222. 18:34Basically how you can make sure like your agent can handle edge case when tools fail or if like there is problem with database and so on.
223. 18:41So you want to have both uh it basically think about it as integration test.
224. 18:45You you have happy path, but you also have edge cases.
225. 18:49Would you simulate agent with LLMs or would you create hand-crafted business?
226. 18:54Yeah, uh great question.
227. 18:55Basically similar to us handcrafting, uh I think basically I mean people use agents to write code.
228. 19:00So a lot of things here can be automated.
229. 19:02So the what is handcrafted kind of changes.
230. 19:05Like the the most important piece that you can provide is like the build environment once and then build the context uh for basically that mimics your production.
231. 19:15Um you mentioned sometimes you use LLM or sometimes use human expert to build verifiers.
232. 19:24Can you elaborate on that?
233. 19:25What's the best practice over there?
234. 19:28Great question.
235. 19:29So, in our case basically, we have a lot of subject matter experts, so we do things at scale.
236. 19:35So, the important piece here is to you don't need subject matter experts to review everything, but you specifically want to find cases where there's disagreement between agent and different verifiers.
237. 19:49If you you think that basically task was supposed to be solved, but agent somehow marks it as like not solved correctly or like it marks that trace wasn't optimal enough.
238. 20:00And this is where you want to have subject matter expert that can basically tune the agent that does the review.
239. 20:21[music]