WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=KMR_RBoCa4M

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/KMR_RBoCa4M.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.639 --> 00:00:14.320
Hi everybody, my name is Shrea.

s3
00:00:14.320 --> 00:00:21.279
I am the CEO of Snow Globe and we have with us Ammon who is a principal machine learning engineer at new.

s4
00:00:21.279 --> 00:00:31.359
And this talk is going to be about simulation maxing and how you can ship or how new bank ships uh agents 20x faster using simulations.

s5
00:00:32.719 --> 00:00:34.160
Hey everyone, I'm Aman.

s6
00:00:34.160 --> 00:00:37.040
Uh so let me talk about New Bank at a glance.

s7
00:00:37.040 --> 00:00:40.239
We are the lead leading digital bank in Latin America.

s8
00:00:40.239 --> 00:00:47.280
We have 135 million customers uh in Brazil, Mexico, and Colombia and we are launching in the US real soon.

s9
00:00:47.280 --> 00:00:51.039
uh our quarterly revenue crossed 5 billion in Q1 2026.

s10
00:00:51.039 --> 00:01:08.560
Our NPS customer love is very high and we are the perfect company for using AI agents for customer support where human Xpers are fanatical customer care people and AI agents together solve customer issues in a fast empathetic and correct manner.

s11
00:01:08.560 --> 00:01:11.520
AI handles a lot of routines end to end.

s12
00:01:11.520 --> 00:01:19.840
uh humans focus on the hardest and longtail cases and together we aim to delight our customers.

s13
00:01:19.840 --> 00:01:24.880
So as Shria said this talk is only about uh one thing really.

s14
00:01:24.880 --> 00:01:34.400
If you generate your eval data in sim instead of waiting on production data you can ship agents 20x faster and we'll give you evidence for that.

s15
00:01:34.400 --> 00:01:36.240
So let's start with the results directly.

s16
00:01:36.240 --> 00:01:51.600
Uh uh so this is you know the average of TNPS which is a measure of customer satisfaction for five of our AI agents in production and at the beginning uh they were not so great but now in a with a few months of

s17
00:01:51.600 --> 00:02:03.439
uh of work and a few quarters worth of effort we've been able to massively increase the TNPS and customer love for our AI agents and many of them are approaching human quality and this data is a bit stale

s18
00:02:03.439 --> 00:02:05.520
many of them are exceeding human quality.

s19
00:02:05.520 --> 00:02:11.039
So we are at the stage where we are actually able to show proof that this actually works in production.

s20
00:02:11.039 --> 00:02:14.239
Uh here's a QR code for our KD paper in case you want to check it out.

s21
00:02:14.239 --> 00:02:19.440
It's going to be presented uh in Korea uh in August.

s22
00:02:21.280 --> 00:02:21.920
Uh awesome.

s23
00:02:21.920 --> 00:02:31.040
So we open with results and you know it's really about this journey of how do you implement the right systems uh for evaluation in order to you know be able to achieve those results.

s24
00:02:31.040 --> 00:02:34.640
So in this talk we basically split it up into these three sections.

s25
00:02:34.640 --> 00:02:44.800
The first is why evals are so important and essential but why they're also the bottleneck from being able to you know do a lot of high throughput experimentation to get the results that Amen was showing earlier

s26
00:02:44.800 --> 00:03:00.480
and then why the second part of this talk is about why simulated data uh works as well as collected data and helps you circumvent a lot of this bottleneck that we're going to talk about and then the third part is really digging deep into the systems and the findings that we had by implementing

s27
00:03:00.480 --> 00:03:05.440
uh you know this framework at scale in new Uh, awesome.

s28
00:03:05.440 --> 00:03:10.720
So, eval there this there's like a whole talk track dedicated at this conference to eval.

s29
00:03:10.720 --> 00:03:16.879
Evals are absolutely critical for building good agents and evals are really only about two things, right?

s30
00:03:16.879 --> 00:03:19.040
There's metrics and there's data.

s31
00:03:19.040 --> 00:03:35.599
uh metrics again I hope you attended many of the amazing talks yesterday on the evals track but metrics are really while they're challenging we have a playbook for how to build metrics that are really well aligned with you know the rubrics that we care about right which is you essentially use um you know LM as a judge

s32
00:03:35.599 --> 00:03:46.879
style uh you know classifiers and you align it with human judgment and and getting human data and you can iteratively build on it using you know auto optimization and autoprompt tuning uh techniques

s33
00:03:46.879 --> 00:03:59.120
the thing that's a bottleneck And that still remains very challenging and unsolved is what is the data that you're actually computing these metrics on and that process is very timeconuming and very expensive

s34
00:03:59.120 --> 00:04:01.439
specifically so for agents.

s35
00:04:01.439 --> 00:04:14.560
So once again people have been talking about you know if if you're around like machine learning era circa you know 2018 it would be like ML work is 85% data work right so data has always been challenging but with agents the level of sophistication

s36
00:04:14.560 --> 00:04:17.519
that data requires is just so much more expensive.

s37
00:04:17.519 --> 00:04:32.000
So here's examples you know of what structured ML data looked like uh and what even early era of AI uh data for you know chat bots or single turn QA which was you know much more manageable and tractable and you could still think of it as these structured rows.

s38
00:04:32.000 --> 00:04:48.800
Uh but now for multi-turn agents each data point is a trajectory with a lot of internal tool calls that all need state etc. And that becomes very even one data point is expensive to generate and annotate right but scaling it up on a sample set becomes even harder.

s39
00:04:49.680 --> 00:04:54.800
Um so how teams primarily get eval data today is two methods.

s40
00:04:54.800 --> 00:05:05.360
Uh one is either manual or the second is production traces and both of those are painful but for different reasons right manual authoring I mean this one's obvious it is so timeconuming

s41
00:05:05.360 --> 00:05:08.240
especially the kind of data that I was showing you on the previous slide.

s42
00:05:08.240 --> 00:05:10.639
You have to hand plan you know state updates.

s43
00:05:10.639 --> 00:05:26.160
You have to hand plan the trajectory that you want to take this uh you know agent data point in and synthetic uh the the state for these data points has to stay consistent across all tool calls etc that you make right production traces in comparison are almost free.

s44
00:05:26.160 --> 00:05:27.360
You don't have to pay for them.

s45
00:05:27.360 --> 00:05:28.960
You're going to get them anyway.

s46
00:05:28.960 --> 00:05:33.919
But the cost is that you're testing on real live users every time you're testing it.

s47
00:05:33.919 --> 00:05:44.400
So for example, if you want to run, you know, many parallel experiments at scale, which is often something you want to do in a machine learning setting is it becomes very hard to do that in prod.

s48
00:05:46.160 --> 00:05:53.280
So for those of you who build agents, you know how difficult building a new agent and maintaining uh an existing agent is.

s49
00:05:53.280 --> 00:06:02.720
So if you want to change the agent harness, you want to change the prompt, the tools, anything around the the LLM that powers your agent, that can take a few hours and in in the best case.

s50
00:06:02.720 --> 00:06:08.080
But as Fria mentioned, running offline evals on hand curated data can take a few days.

s51
00:06:08.080 --> 00:06:17.680
And then the the worst part is if you run an AB test in production and monitor regressions, see if you get a statistically significant improvement of of a previous version

s52
00:06:17.680 --> 00:06:29.039
that can take forever because customer feedback is through some kind of a feedback form and it can be sparse, it can be noisy and can take forever to know am I moving in the right direction or not.

s53
00:06:29.039 --> 00:06:41.840
So I'm here to tell you that uh we have verified in production that yes simulations circuit this timeline short circuit this timeline from a few weeks you can go to less than a day sometimes even a few hours a few minutes

s54
00:06:41.840 --> 00:06:50.960
and uh as I mentioned earlier uh you know with simulations it takes a few hours uh without simulation it takes a few hours to change the harness and offline evals

s55
00:06:50.960 --> 00:07:02.560
and all of that's become super fast now with simulation the agent harness chain still takes a few hours running offline evals becomes a lot easier because you have a simulated user to fall back on.

s56
00:07:02.560 --> 00:07:04.880
You're not waiting on hand curated data.

s57
00:07:04.880 --> 00:07:07.759
You're not waiting on you know people to create data for you.

s58
00:07:07.759 --> 00:07:12.080
And the best part if you want to know that am I doing well in production?

s59
00:07:12.080 --> 00:07:14.639
Is this version going to do well?

s60
00:07:14.639 --> 00:07:15.840
That can take just a few hours.

s61
00:07:15.840 --> 00:07:19.919
So this has been a real gamecher for us at New Bank.

s62
00:07:20.560 --> 00:07:21.360
Um all right.

s63
00:07:21.360 --> 00:07:39.360
So we talked about you know like simulations shortening the release cycle but what is mechanically a simulation you know it's existed in robotics it's existed in you know a lot of different domains but simulations for agent eval you know mean a specific thing and this is really where snow globe enters and you know sets up a lot

s64
00:07:39.360 --> 00:07:40.960
of that core infrastructure for you.

s65
00:07:40.960 --> 00:07:57.599
So simulations are essentially you know it works by first like wrapping your agent you know using a um uh like pointing the snow globe SDK at your agent without requiring any code changes and you know you can gro what are the tools that you really need in to mock in order to make the simulation really work.

s66
00:07:57.599 --> 00:08:02.080
The second input is about how you want to drive or steer your your simulation.

s67
00:08:02.080 --> 00:08:07.440
So you know who are the personas what are the use cases what are the data points that we're really simulating.

s68
00:08:07.440 --> 00:08:15.520
Once you have those inputs ready, snow globe essentially runs the simulation and on the output you end up getting you know first just a data set.

s69
00:08:15.520 --> 00:08:19.759
So you get like thousands of multi-turn conversations against your real agents.

s70
00:08:19.759 --> 00:08:36.320
These conversations have you know tools mocked etc. have all of the bells and whistles needed to be actually you know useful for evaluating your agent already ready and then you run like judges on top of it so that on this data set you get you know per turn information about how your agent is really behaving and it

s71
00:08:36.320 --> 00:08:42.800
then like pipes directly into you know your evaluation pipeline so that you can really get data on demand.

s72
00:08:42.800 --> 00:08:56.640
Um this is an example of what a sim what the what what a concrete output of simulation is like right so this is as an example like one simulated persona and then the corresponding conversation or data point for that simulated persona

s73
00:08:56.640 --> 00:09:07.600
so here for example this is totally synthetic a user named Maria Souza uh you know she wants to order a credit card this is who she is she's 34 a designer first-time credit card customer

s74
00:09:07.600 --> 00:09:20.160
in addition to you know this basic information about this synthetic user we also will generate, you know, this essential grounding data as an example, this fake address, a fake credit card, etc. that stays consistent

s75
00:09:20.160 --> 00:09:24.480
across your agent execution so that you know your agent can actually be evaluated against that.

s76
00:09:24.480 --> 00:09:28.560
And then you also have things like tone, voice, how do they really interact?

s77
00:09:28.560 --> 00:09:39.360
And then oops, and then when you actually run this persona through the conver through the agent, you end up getting, you know, this uh conversation that's consistent with the simulated intent, right?

s78
00:09:39.360 --> 00:09:48.399
So you see for example that this user has very curt oneline messages in terms of how they interact because this is how the persona was set up or that you know when

s79
00:09:48.399 --> 00:10:03.360
the agent wants to when the agent runs its tool when Maria enters the chat the agent is actually able to get a valid output mocked by the simulator that you know is able to identify who the user is where the address is the address can be verified

s80
00:10:03.360 --> 00:10:09.680
etc. Um and then finally all of this is also grounded in like account data as well.

s81
00:10:10.880 --> 00:10:15.200
So for those of us building real agents, you know how hard things are.

s82
00:10:15.200 --> 00:10:18.320
Uh you don't know if your agent is good, you don't know if your evals are good.

s83
00:10:18.320 --> 00:10:19.600
You have no idea what's happening.

s84
00:10:19.600 --> 00:10:21.920
And that was me a few quarters ago.

s85
00:10:21.920 --> 00:10:26.399
Uh since the theme of uh this year's conference is self-improving agents.

s86
00:10:26.399 --> 00:10:29.120
So we are also kind of jumping on the bandwagon.

s87
00:10:29.120 --> 00:10:31.600
So this is our self-improvement loop.

s88
00:10:31.600 --> 00:10:35.279
You ship your agent, you observe, you create robust evals.

s89
00:10:35.279 --> 00:10:41.600
You spend a lot of time creating robust evals using automated prompt optimization with algorithms like Japa.

s90
00:10:41.600 --> 00:10:43.680
And then you also do simulation.

s91
00:10:43.680 --> 00:10:49.040
You run simulation and you pipe the simulation data and real data through your evals.

s92
00:10:49.040 --> 00:10:53.200
And that gives you rich signal to then optimize your agent harness.

s93
00:10:53.200 --> 00:10:57.839
And once you verify the agent harness optimization is good, you ship.

s94
00:10:57.839 --> 00:11:01.839
And this has enabled us to do many many many AB tests for a given agent.

s95
00:11:01.839 --> 00:11:04.000
And we have many many such agents across the company.

s96
00:11:04.000 --> 00:11:14.320
So this has been a real gamecher for us in terms of improving our agents and the proof as I already showed you is that you know the quality of our agents is going up uh pretty dramatically.

s97
00:11:14.320 --> 00:11:18.880
This is some data on how simulation really tracks real production data.

s98
00:11:18.880 --> 00:11:25.360
So when you run uh simulations you want to compare is my simulation comparable to production data.

s99
00:11:25.360 --> 00:11:28.720
So we looked at sim quality via evals.

s100
00:11:28.720 --> 00:11:39.519
So far uh you know quality of uh with evals from sim and real data the correlation is pretty high and we had human review done where 80% of our domain expert labels confirmed

s101
00:11:39.519 --> 00:11:45.760
that sims give us usable data not just for uh mature agents but also for green field agents.

s102
00:11:45.760 --> 00:11:55.040
So this has been uh pretty useful for us in bringing up a new agent from scratch and also taking an existing agent and and massively improving it.

s103
00:11:55.120 --> 00:12:00.880
And as as we as I mentioned, you know, simulation is not just about improving uh and sh also talked about this.

s104
00:12:00.880 --> 00:12:04.000
We are derisking a lot of things that could have made to production.

s105
00:12:04.000 --> 00:12:09.680
For instance, we caught a regression uh with simulation that could have made it to production, but simulation caught it.

s106
00:12:09.680 --> 00:12:16.959
And at the same time, we also caught in another agent an issue uh which could have lowered our self-service rate making it less efficient.

s107
00:12:16.959 --> 00:12:20.079
And all of this has resulted in pretty large improvements.

s108
00:12:20.079 --> 00:12:33.600
Uh for instance, one of our agents, the TNPS, has 2xed uh thanks to simulation, robust evals, and investing in something super principled rather than, you know, throwing something at the wall and see what sticks.

s109
00:12:33.600 --> 00:12:37.600
Um it's been pretty awesome to see simulation saving us so much time.

s110
00:12:37.600 --> 00:12:45.200
So now the team uses snow globe and tries different ideas and they're able to short circuit launching AB test.

s111
00:12:45.200 --> 00:12:47.920
They don't launch until they're happy with the same output.

s112
00:12:47.920 --> 00:12:53.360
So imagine if you're running 10 AB tests a quarter for your agent or probably more if you have more data.

s113
00:12:53.360 --> 00:13:01.440
Now you can just run a bunch of stuff through SIM and launch just one AB test which kind of shortcircuits launching the first five or six.

s114
00:13:01.440 --> 00:13:08.800
And at the same time uh this this is often discussed in customer service circles that TNPS can come at the cost of self-service rate.

s115
00:13:08.800 --> 00:13:14.240
Sometimes there's a trade-off but I'm happy to report that we are not compromising all self- service rate with SIM.

s116
00:13:14.240 --> 00:13:18.160
We are also able to pretty dramatically improve SSR across the board.

s117
00:13:18.160 --> 00:13:22.000
But in one particular case, we were able to improve it by 4%.

s118
00:13:22.000 --> 00:13:33.120
All of this makes us a much more efficient company and makes sure that we keep our customers really happy and we keep our SSR rates and TPS rates very very high.

s119
00:13:33.120 --> 00:13:35.760
This is one example that I am most excited about.

s120
00:13:35.760 --> 00:13:40.560
As you know, all of you know open source models have been on a tear recently.

s121
00:13:40.560 --> 00:13:42.000
So many of them have come out.

s122
00:13:42.000 --> 00:13:48.079
So, we've been trying to test various open source models in our agent harnesses and see which ones are the best.

s123
00:13:48.079 --> 00:13:59.199
And with SIM, we have the perfect recipe to just throw a bunch of ideas, open source models at our agent harness and eval setup and see which model versions

s124
00:13:59.199 --> 00:14:02.160
uh you know really really work for us.

s125
00:14:02.160 --> 00:14:06.000
Uh this has saved us multiple weeks of effort and probably underelling it.

s126
00:14:06.000 --> 00:14:17.519
It's it's been it's been pretty awesome and uh we have AB test running as I speak uh figuring out are these open source models as good as you know uh the frontier models.

s127
00:14:17.519 --> 00:14:26.000
So SIM is is is been super super helpful in helping us figure out uh kind of you know spreading our cards uh a bit.

s128
00:14:26.720 --> 00:14:27.519
Um awesome.

s129
00:14:27.519 --> 00:14:36.560
So um the core three takeaways from this talk right is about um where eval is today and how you can really remove a lot of bottlenecks to it.

s130
00:14:36.560 --> 00:14:40.320
So we lied when we said earlier that there's just one thing you should take away from it.

s131
00:14:40.320 --> 00:14:50.560
That one thing is still essential but there's a few key downstream things that you can unlock if you know you adopt it which is the first is if you generate your evaluation data in simulation

s132
00:14:50.560 --> 00:15:04.079
rather than solely relying on production to get signal on how you know different agents are performing uh you'll be able to undercut or you'll be able to short circuit a lot of the uh bottleneck in in releasing you know versions of your agents much faster.

s133
00:15:04.079 --> 00:15:11.839
The second is in order for any of these gains to really be unlocked uh you know you really need to close out the sim toreal gap.

s134
00:15:11.839 --> 00:15:25.120
So you need to you know set up like offline online human review kind of metrics to really understand how sim performs visav real production data that you've seen uh so that you you you are able to kind of like trust the results of these simulations.

s135
00:15:25.120 --> 00:15:38.079
Um and the third is again there's so much excitement around you know like auto research self-improving agents RSI etc. uh in an enterprise setting when you're building an agent, it really does come down to two things, data and metrics.

s136
00:15:38.079 --> 00:15:55.360
If you have align metrics that are able to really catch the signals you care about and you have a reliable way of generating data that those metrics can give you signal on, it's then very easy to put together a loop of an agent that you know continuously improves itself from like feedback it receives from all of these

s137
00:15:55.360 --> 00:15:56.000
places.

s138
00:15:56.000 --> 00:16:00.880
Uh which you know again is like where the future of this field is heading.

s139
00:16:01.199 --> 00:16:02.399
Um, awesome.

s140
00:16:02.399 --> 00:16:03.279
That's that's us.

s141
00:16:03.279 --> 00:16:04.880
Uh, thank you for coming for the talk.

s142
00:16:04.880 --> 00:16:13.440
Uh, there's few like contact links and details for us, you know, if you guys want to get in touch and otherwise we'll be around here if you guys want to ask us any questions.

s143
00:16:27.096 --> 00:16:29.096
[music]