WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=xLxhT2ZI7UM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/xLxhT2ZI7UM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.880 --> 00:00:14.639
Good afternoon everyone.

s3
00:00:14.639 --> 00:00:16.000
Um, thank you for coming.

s4
00:00:16.000 --> 00:00:18.080
Hope you've had a great expo so far.

s5
00:00:18.080 --> 00:00:22.160
We're in the last stretch, but I know it's been a wonderful, wonderful expo, at least for me.

s6
00:00:22.160 --> 00:00:25.439
So hopefully you're learning a lot and we can teach you a little bit more.

s7
00:00:25.439 --> 00:00:27.199
Um, so I'm Miguel.

s8
00:00:27.199 --> 00:00:30.560
I'm the tech lead for uh our agent platform at Browserbase.

s9
00:00:30.560 --> 00:00:38.320
For those of you who don't know, browserbase is a infrastructure company for deploying agents on the web.

s10
00:00:38.320 --> 00:00:51.520
Um it basically unlocks all of the web, not just what is easily accessible uh by managing browsers in the cloud and all the runtime that uh requires to do agentic web automation

s11
00:00:51.520 --> 00:00:52.559
at scale.

s12
00:00:52.559 --> 00:00:54.480
So um let

s13
00:00:54.480 --> 00:00:55.440
and I'm Corby.

s14
00:00:55.440 --> 00:01:02.320
I'm a researcher at Microsoft and I've been collaborating with with browser base to help build really good verifiers.

s15
00:01:02.320 --> 00:01:07.360
So today we're here to talk about some research that we published about a month ago.

s16
00:01:07.360 --> 00:01:11.040
Um and it is regarding evalance.

s17
00:01:11.040 --> 00:01:16.080
It is regarding RL environments and LM judges as verifiers.

s18
00:01:16.080 --> 00:01:23.759
So early on so I've been working on this open source framework called stchen for about a year and a half now.

s19
00:01:23.759 --> 00:01:35.840
And from the beginning as any good agentic product needs to have we were building evals to make sure that the changes that we were introducing in the framework and that the model capabilities

s20
00:01:35.840 --> 00:01:43.040
continue to improve and hill climb on those evals and at the beginning all those evals were deterministic environments.

s21
00:01:43.040 --> 00:01:44.560
um it was relatively simple.

s22
00:01:44.560 --> 00:01:55.759
Model capabilities were some somewhat constrained and so we were able to build a lot of deterministic environments as model capabilities continue to improve as we continue to make improvements on the agent harness

s23
00:01:55.759 --> 00:01:57.439
that just didn't scale.

s24
00:01:57.439 --> 00:02:10.479
We found ourselves constantly patching or generating static sites um coming up with longer trajectories with a lot of checkpoints to make sure that the evals were deterministically verifiable.

s25
00:02:10.479 --> 00:02:19.760
And so there's many different project problems with automating the web at scale and verifying that what you intended to do actually succeeded.

s26
00:02:19.760 --> 00:02:21.440
And it's that the web is very open-ended.

s27
00:02:21.440 --> 00:02:24.959
There's not just one path to correctness.

s28
00:02:24.959 --> 00:02:27.599
Um there is no ground truth.

s29
00:02:27.599 --> 00:02:32.800
Sometimes the web changes and the product that you were checking for before is no longer there.

s30
00:02:32.800 --> 00:02:37.760
So that breaks the whole determinism of your environment and there's blockages.

s31
00:02:37.760 --> 00:02:46.000
there's a lot of the environment errors that you face that don't let you get sort of a verifiable deterministic signal.

s32
00:02:46.000 --> 00:02:56.480
So we recurred to what most people do which is trying to use an LLM judge to help improve um and scale this operation.

s33
00:02:56.720 --> 00:03:10.640
But we quickly found out that a lot of the leading benchmarks and some of you may know about OS World uh for computer use but there's some analogous benchmarks for web use called online minor web and web voyager.

s34
00:03:10.879 --> 00:03:14.800
Those come with LLM judges as verifiers.

s35
00:03:14.800 --> 00:03:24.480
And what we noticed when we were using our sort of human expert annotators to verify the verifier is that in many instances they're very confidently wrong.

s36
00:03:24.480 --> 00:03:29.840
And that leads to results that you can't really trust.

s37
00:03:30.879 --> 00:03:42.080
So very quickly we realize that even if you were using this as an RL reward or as a signal to improve your hardness and auto research, you're not really training a better agent.

s38
00:03:42.080 --> 00:03:45.680
You're just training a more competent liar.

s39
00:03:47.360 --> 00:03:54.640
And so in a real world use case at Microsoft, we trained this FARS 7B model which is a a web browser agent.

s40
00:03:54.640 --> 00:04:02.400
And the same model on the same benchmark, we judged it according to the official web voyager judge, which is GPT40.

s41
00:04:02.400 --> 00:04:06.319
Uh and it said that it had 74% uh success rate.

s42
00:04:06.319 --> 00:04:16.799
But there's a huge gap between the real truth which is when we used our new universal verifier uh which has high agreement with human labels that that number quickly becomes 30 like 38%.

s43
00:04:16.799 --> 00:04:22.639
So there's a huge gap between what the existing verifiers say and what is actually the ground truth.

s44
00:04:22.639 --> 00:04:25.600
And so we needed to find a way to to close this gap.

s45
00:04:25.600 --> 00:04:28.560
The existing verifiers have lots of weaknesses.

s46
00:04:28.560 --> 00:04:33.680
Um they're they use smaller and like much dumber models.

s47
00:04:33.680 --> 00:04:48.240
uh as the LM as a judge they use 04 mini or GBT40 they don't even use rubrics so rubrics are the most critical thing that you need to have because you need to assign credit to where credit is due they don't look at the the relevant screenshots or they or they try to look at all the screenshots

s48
00:04:48.240 --> 00:04:58.880
and they quickly get lost and overflow the context window of the LLM as a judge and sometimes they don't even look at the final answer or the action history of the model so our verifier checks all of these boxes

s49
00:04:58.880 --> 00:05:13.039
um and this is kind of how it works uh at a very high level you And you can look in the paper for more details, but given a task like book a cheapest flight from Seattle to Boston, the first thing we do is we generate a really good rubric and I'll give some examples of that.

s50
00:05:13.039 --> 00:05:18.000
And a rubric has like maybe 10 different criteria of what success looks like.

s51
00:05:18.000 --> 00:05:23.680
And then given that that criteria in the rubric, we look at the agents trajectory.

s52
00:05:23.680 --> 00:05:28.240
All of the screenshots in this case are the the evidence of the ground truth.

s53
00:05:28.240 --> 00:05:33.280
and we rank what are the most relevant screenshots in the trajectory for each criterion.

s54
00:05:33.280 --> 00:05:48.080
And then we use that group of of top K evidence to determine whether that criterion was met or not and if there's any like differences or contradictions between what the agent said it did and what the state of the environment actually showed.

s55
00:05:48.080 --> 00:05:56.639
And then given that we output uh a scored rubric which we we call like a process score because it gives partial credit in some cases.

s56
00:05:56.639 --> 00:06:08.639
and we give an we output a uh an outcome boolean uh value to turn to basically say whether the agent accomplished the task according to what a reasonable user was would expect.

s57
00:06:08.639 --> 00:06:14.240
Um and so this is an example of what the rubric uh the rubric output is.

s58
00:06:14.240 --> 00:06:24.800
It's basically a list of criteria um as well as the outcome verifier is basically a true false flag with an explanation as to why a reasonable user would expect

s59
00:06:24.800 --> 00:06:28.319
um this trajectory to have succeeded or not.

s60
00:06:28.319 --> 00:06:33.280
There were four guiding principles when we created this universal verifier.

s61
00:06:33.280 --> 00:06:40.080
One is around rubric creation which is we wanted to grade only what was asked and not any extraneous criterion.

s62
00:06:40.080 --> 00:06:45.120
Uh we also did not want errors to cascade from one rubric uh criterion to another.

s63
00:06:45.120 --> 00:06:46.240
We wanted them to be isolated.

s64
00:06:46.240 --> 00:06:48.160
And I'll give an example of that.

s65
00:06:48.160 --> 00:06:50.479
You also wanted to look at the ground truth screenshots.

s66
00:06:50.479 --> 00:06:51.840
Like this is a must.

s67
00:06:51.840 --> 00:06:56.880
The agents will often overconfidently claim that they did something when in fact they did not do it.

s68
00:06:56.880 --> 00:06:58.960
And so you have to look at the ground truth state.

s69
00:06:58.960 --> 00:07:04.880
And then we also needed to separate um what the agent could control versus what the agent could not control.

s70
00:07:04.880 --> 00:07:06.400
The agent operates in a web browser.

s71
00:07:06.400 --> 00:07:12.720
The web browser is an environment that it doesn't always have ability to control and I'll show some examples of that too.

s72
00:07:12.720 --> 00:07:16.080
So this is an example of what a good and bad rubric looks like.

s73
00:07:16.080 --> 00:07:29.120
The task here is to uh find a cheap uh hotel in Jakarta for these dates and then use the hotel's address to search for the closest coffee shop and then output the name and address of that coffee shop.

s74
00:07:29.120 --> 00:07:37.280
Now a bad rubric which we did see happen would output a criteria saying like please tell me the total price for the stay at the hotel.

s75
00:07:37.280 --> 00:07:45.680
This is actually an extraneous criterion that was not asked for in the task but we saw a lot of rubrics naively generated do something like this and they would artificially

s76
00:07:45.680 --> 00:07:50.720
deflate the scores because the agent didn't do something it wasn't asked to do.

s77
00:07:50.720 --> 00:08:02.160
Um when it comes to not cascading errors, um there are very subtle mistakes because a lot of tasks build on each other uh if they are like multi-step tasks.

s78
00:08:02.160 --> 00:08:13.759
So in this example here, the task was to um determine the net worth of the individual with the longest last name from among the members of NSync and Backstreet Boys.

s79
00:08:13.759 --> 00:08:14.479
Okay.

s80
00:08:14.479 --> 00:08:23.919
And so the agent in this case mistakenly thought that Timberlake had the longest last name which is only 10 letters whereas Kirkpatrick actually had the longest last name.

s81
00:08:23.919 --> 00:08:35.599
So the model made one arithmetic mistake in one criteria of the rubric and that should not cascade to the next criteria of the rubric which is report the net worth of of the person that you chose.

s82
00:08:35.599 --> 00:08:35.839
Right?

s83
00:08:35.839 --> 00:08:42.320
So if you're able to report the net worth of Timberlake uh accurately then you shouldn't get penalized in this criteria.

s84
00:08:42.320 --> 00:08:44.880
you should only get penalized in that one.

s85
00:08:44.880 --> 00:08:49.279
Hallucinations is the biggest one uh that we really wanted to focus on.

s86
00:08:49.279 --> 00:09:01.120
So in this task, this is an example of a very subtle hallucination uh where you wanted to find some information about some image captioning model and the agent said that this model had plus 6.2%

s87
00:09:01.120 --> 00:09:08.640
in some cider score, but in reality the abstract of the underlying paper said it was only 2.8% in cider score.

s88
00:09:08.640 --> 00:09:15.600
And so this was a like a very subtle mistake that even humans will not catch unless an LLM will flag it.

s89
00:09:15.600 --> 00:09:23.279
Um so this kind of gives uh an overview of of how process scores and outcome scores differ.

s90
00:09:23.279 --> 00:09:30.000
Um in the case we mentioned one of the principles was we wanted to separate controllable versus uncontrollable um failures.

s91
00:09:30.000 --> 00:09:41.760
So, if the task is to buy, say, this uh plushy toy from Amazon, and the agent accurately searched for the plushy toy, but found that it was out of stock, it could not continue.

s92
00:09:41.760 --> 00:09:54.720
And so, here we say that the agent gets full marks for doing its best effort uh to achieve the goal, but the outcome was still not met because it couldn't buy the thing that it wanted to buy because it was out of stock.

s93
00:09:54.720 --> 00:10:05.120
Um, so we basically enumerated a bunch of scenarios as to what could happen and how do we assign credit if the agent was able to find a similar alternative plushy toy in a different way.

s94
00:10:05.120 --> 00:10:11.600
Um, it would still get success there uh for both for both cases and it would get penalized if it made a controllable mistake.

s95
00:10:11.600 --> 00:10:15.120
Uh, it would get penalized if it made hallucinations and so on and so forth.

s96
00:10:15.120 --> 00:10:25.600
So we basically enumerated a a schema of like all the possible failure modes and how we would assign credit to that and the universal verifier adheres to those to those schema.

s97
00:10:25.600 --> 00:10:26.079
So

s98
00:10:26.079 --> 00:10:41.519
basically in in order to evaluate the verifier and really tell whether or not we were able to hill climb the accuracy of that verifier, we started designing a lot of experiments with human experts and built a whole platform

s99
00:10:41.519 --> 00:10:49.040
to collect data on what we call the ground labels that we've also released under cool verify bench for others to train upon.

s100
00:10:49.040 --> 00:10:53.839
But the process that we followed was the following.

s101
00:10:53.839 --> 00:11:05.519
At first, we would show the full trajectory of evidence to the human verifiers and the annotators would judge based on the evidence and the result of all the actions that the agent took.

s102
00:11:05.519 --> 00:11:19.279
After that, they would be prompted to a seeing the judgment of the universal verifier to say whether they agree or disagree with a human with a universal verifier.

s103
00:11:19.279 --> 00:11:32.079
And we found this to be a very reliable signal to tell whether or not we were working in the right direction because I'll jump to this one straight up, but

s104
00:11:32.079 --> 00:11:34.959
the verifier started correcting humans at one point.

s105
00:11:34.959 --> 00:11:38.959
They it would identify things that the humans were missing.

s106
00:11:38.959 --> 00:11:57.200
And so ultimately the result of the of the universal verifier on these golden labels was that we were able to reduce false positives from almost half to zero and coins cappa isometric to measure inner annotator inner annotator agreement.

s107
00:11:57.200 --> 00:12:04.800
Um and what we can see here is that the h universal verifier agrees with humans as often as humans agree with one another.

s108
00:12:04.800 --> 00:12:19.519
So we can see that 0.58 coins scalpa score uh versus the two annotators per task um where we computed coins scappa as well to see how much they agreed with one another.

s109
00:12:20.880 --> 00:12:31.519
Another way another way to uh verify the another way to verify the quality of our verifier is to actually train on data filtered by it.

s110
00:12:31.519 --> 00:12:37.600
So what we did was we did ran an experiment where we held constant the number of training examples.

s111
00:12:37.600 --> 00:12:46.320
In this case it was 3k or 9k training trajectories, but we filtered those training trajectories based on whether our verifier said they were pass or not.

s112
00:12:46.320 --> 00:12:54.160
And so when you're doing SFT, which is what we did here, you want to train only on SFT examples where they were true uh like successful trajectories.

s113
00:12:54.160 --> 00:13:08.639
And so if you train on 3,000 trajectories filtered by the process score of our universal verifier, you will get a much higher quality model than if you were to train on a worse verifier like our own a baseline verifier.

s114
00:13:08.639 --> 00:13:10.800
And this held at larger scales as well.

s115
00:13:10.800 --> 00:13:22.079
And so this basically this experiment told us that uh if you have a higher quality verifier, it means you can filter higher quality data uh and training on that data leads to a higher quality model.

s116
00:13:22.079 --> 00:13:33.120
And so this was like the ultimate experimental proof in addition to the um uh to the to the human agreements uh that this model this verifier is working and is and is reliable

s117
00:13:33.120 --> 00:13:36.639
um in a real production uh scenario.

s118
00:13:36.639 --> 00:13:47.920
Um the last thing we did kind of as a side project is we wanted to determine whether AI in auto research can build the same verifier that we built.

s119
00:13:47.920 --> 00:13:56.959
So basically what I did and what Miguel and I did is we sat down for three weeks um to build the universal verifier and to tweak the prompts and to write the code for it.

s120
00:13:56.959 --> 00:14:05.519
And we basically ran about 30 different experiments over the course of three weeks to determine whether the verifier we were building agrees with human labels.

s121
00:14:05.519 --> 00:14:05.839
Okay.

s122
00:14:05.839 --> 00:14:09.440
And so that's what the Coen Kappa score here on the Y-axis is measuring.

s123
00:14:09.440 --> 00:14:15.680
It measures whether an individual verifier system is agreeing with the human labels.

s124
00:14:15.680 --> 00:14:25.760
And so the blue line here is the model Miguel and I, the system Miguel and I were were creating this auto research uh this uh universal verifier.

s125
00:14:25.760 --> 00:14:36.720
And then we wanted to see if if we stripped everything that we did away, could AI um in an auto research loop build the same verifier that we built with the same level of quality and fidelity?

s126
00:14:36.720 --> 00:14:38.880
And that's what the red and green lines here are.

s127
00:14:38.880 --> 00:14:44.399
The red line is if we took away all of our code and prompts, could AI replicate that?

s128
00:14:44.399 --> 00:14:52.560
The conclusion that we got from this is that while we took about three weeks to build this verifier, uh an auto research loop could do it in about one day.

s129
00:14:52.560 --> 00:14:55.360
It ran the same number of experiments in about one day.

s130
00:14:55.360 --> 00:15:02.079
However, it only reached about 70% of the agreements that our verifier uh was able to reach.

s131
00:15:02.079 --> 00:15:03.920
So there's still a gap there.

s132
00:15:03.920 --> 00:15:16.160
And so the kind of the conclusion that I would draw here is that uh you can use auto research to build uh metrics and verifiers um at least to help you speed up uh experimentation.

s133
00:15:16.160 --> 00:15:19.680
Um but you probably still need some level of human intervention here.

s134
00:15:19.680 --> 00:15:25.279
Um then again, these these results were from Opus 4.6. I haven't tried it with Fable yet.

s135
00:15:25.279 --> 00:15:30.959
Maybe it will do better, but it's a pretty powerful baseline and it got pretty far um along the way.

s136
00:15:30.959 --> 00:15:39.279
So, not only can auto research help you build a model, but in this case, auto research can help you build a verifier as well, which we thought was a pretty cool

s137
00:15:39.279 --> 00:15:39.839
um thing.

s138
00:15:39.839 --> 00:15:45.839
And and building verifiers is as important as building the models themselves is one of the takeaways I want to leave you with.

s139
00:15:45.839 --> 00:15:47.839
So

s140
00:15:47.839 --> 00:15:59.839
yeah, and just to double emphasize um the green light is the auto research verifier um primed with a lot of the findings that we had gotten from the three weeks of experimentation

s141
00:15:59.839 --> 00:16:06.240
and in that optimization it was able to reach higher percentages than we were ever before.

s142
00:16:06.240 --> 00:16:15.279
So combining that human intuition with auto research loops has proven to be the most powerful recipe for many research tasks.

s143
00:16:15.279 --> 00:16:20.639
Um so the all of the work is open source.

s144
00:16:20.639 --> 00:16:26.639
The paper is published under this uh as a preprint um on archive.

s145
00:16:26.639 --> 00:16:33.440
Um and the Microsoft repo contains the golden labels on kua verifier bench.

s146
00:16:33.440 --> 00:16:37.600
It contains the code to run the experiments.

s147
00:16:37.600 --> 00:16:43.519
But beyond that um this research also yielded an entire new benchmark.

s148
00:16:43.519 --> 00:16:47.440
So we've talked briefly about online minor web and web voyager.

s149
00:16:47.440 --> 00:16:55.279
I spend a lot of time working with labs helping make sure that we can provide them quantitative signal to improve their models.

s150
00:16:55.279 --> 00:17:09.039
A lot of those open source benchmarks are very quickly getting saturated and it is very difficult to provide any sort of signal because it feels like the the test data is already in the distribution of the training data.

s151
00:17:09.039 --> 00:17:21.679
And so this new benchmark has proven to have the biggest gap to complete success and it's my daily driver to make sure that I can tell whether model capabilities

s152
00:17:21.679 --> 00:17:26.319
are strong, whether they lack and how to continue to close the gap.

s153
00:17:26.319 --> 00:17:27.679
So

s154
00:17:27.679 --> 00:17:28.799
did you have more slides?

s155
00:17:28.799 --> 00:17:29.280
No.

s156
00:17:29.280 --> 00:17:30.080
Okay,

s157
00:17:30.080 --> 00:17:36.000
we can take a little bit of I think questions, but uh we're good on time.

s158
00:17:36.559 --> 00:17:38.559
Yes.

s159
00:17:44.080 --> 00:17:44.400
Yeah.

s160
00:17:44.400 --> 00:17:51.200
So the question is the question is whether we've used the uh the universal verifier for other kinds of tasks beyond just web web tasks.

s161
00:17:51.200 --> 00:18:02.080
So one thing we're working on in Microsoft right now is to apply a version of this verifier for desktop tasks like things that you would do on your laptop like kind of enterprise workflows.

s162
00:18:02.080 --> 00:18:09.520
And so we hope to publish another benchmark uh of not just web data but also like enterprise style desktop data.

s163
00:18:09.520 --> 00:18:13.440
Uh and we would have basically the similar the same kind of verifier for that too.

s164
00:18:13.440 --> 00:18:20.640
It would look at a little bit more information because on desktops you have the terminal, you have more telemetry, more logs than just the browser.

s165
00:18:20.640 --> 00:18:23.919
So the verifier would take those things into account as well.

s166
00:18:23.919 --> 00:18:25.760
Yeah.

s167
00:18:25.760 --> 00:18:28.240
Any other questions?

s168
00:18:28.240 --> 00:18:29.679
Yeah.

s169
00:18:29.679 --> 00:18:47.840
So I personally never when you present I was thinking about like how do you think about that show?

s170
00:19:00.880 --> 00:19:02.880
Yeah.

s171
00:19:03.200 --> 00:19:03.440
Yeah.

s172
00:19:03.440 --> 00:19:08.400
So, the question is about how do you make sure you're not overfitting the verifier to the human labels because that is a problem.

s173
00:19:08.400 --> 00:19:13.120
Uh Miguel and I spent a lot of time uh working on this question.

s174
00:19:13.120 --> 00:19:20.400
And so like one thing that we did was we uh kind of held out uh two sets of labels.

s175
00:19:20.400 --> 00:19:24.720
One we we basically had a set of like 150 trajectories.

s176
00:19:24.720 --> 00:19:31.039
um on about 50 of those trajectories, I labeled them myself and I used them to hill climb in those 30 experiments.

s177
00:19:31.039 --> 00:19:34.799
But for the other hundred trajectories, they were basically held out to us.

s178
00:19:34.799 --> 00:19:39.919
They were done by humans uh that we had paid uh with like 2x overlap.

s179
00:19:39.919 --> 00:19:53.440
So once our verifier was done being built, we gave it to those human annotators and each human labeled each trajectory, sorry, two humans labeled each trajectory and then that's what we published in KUA verifier bench.

s180
00:19:53.440 --> 00:19:56.320
And so those are the numbers that we report here.

s181
00:19:56.320 --> 00:20:03.280
Um so we're pretty confident that we didn't overfit because we held out at least twothirds of the labels.

s182
00:20:03.280 --> 00:20:08.320
Um like we did not train like iterate on them.

s183
00:20:08.320 --> 00:20:08.960
Yeah.

s184
00:20:08.960 --> 00:20:21.360
So that's very important is you don't want to train on like to be able to

s185
00:20:24.320 --> 00:20:25.280
Yeah.

s186
00:20:25.280 --> 00:20:25.520
Yeah.

s187
00:20:25.520 --> 00:20:25.760
Yeah.

s188
00:20:25.760 --> 00:20:35.280
The gold standard is uh is to hire humans um and train them to do the the verification task themsel and then see whether your system agrees with it.

s189
00:20:35.280 --> 00:20:37.200
Yeah.

s190
00:20:37.200 --> 00:20:39.200
Any other questions?

s191
00:20:39.200 --> 00:20:42.559
Anyone want to help us build more benchmarks?

s192
00:20:43.440 --> 00:20:44.960
Yeah.

s193
00:20:44.960 --> 00:20:45.840
Oh yeah.

s194
00:20:45.840 --> 00:20:47.280
Okay.

s195
00:20:47.280 --> 00:20:47.679
Okay.

s196
00:20:47.679 --> 00:20:47.919
Yeah.

s197
00:20:47.919 --> 00:20:48.080
Yeah.

s198
00:20:48.080 --> 00:20:49.760
Yeah.

s199
00:20:49.760 --> 00:20:50.159
Awesome.

s200
00:20:50.159 --> 00:20:51.120
Well, thanks a lot.

s201
00:20:51.120 --> 00:20:53.520
Uh hopefully you enjoyed the rest of the expo.

s202
00:20:53.520 --> 00:20:54.559
It was a pleasure.

s203
00:20:54.559 --> 00:20:56.000
Thank you so much.

s204
00:20:56.000 --> 00:20:56.240
Yeah.

s205
00:20:56.240 --> 00:20:58.240
Yeah.