WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Z-c11pV_uvU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Z-c11pV_uvU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.400 --> 00:00:15.160
Okay, I'm getting rolling and uh welcome aboard.

s3
00:00:15.160 --> 00:00:18.360
We just had a little technical issues, but uh we resolved them.

s4
00:00:18.360 --> 00:00:20.320
So, my name is Frank Coyle.

s5
00:00:20.320 --> 00:00:23.640
Uh I am a computer science guy.

s6
00:00:23.640 --> 00:00:28.680
I've been teaching computer science for over 30 years, and I'm now teaching at Berkeley.

s7
00:00:28.680 --> 00:00:40.600
And one of the problems that uh all my students, past and present, are having is AI, because computer science is no longer the magic pathway to a job.

s8
00:00:40.600 --> 00:00:49.320
So, I've been trying to figure out ways to uh help them come up with schemes to help them get ready for this world of agentic AI.

s9
00:00:49.320 --> 00:01:02.040
And one of the things that sort of uh dropped into my uh plate was the something called the Claude Certified Architect exam, which I will be talking about today, and it has

s10
00:01:02.040 --> 00:01:03.840
um a number of aspects to it.

s11
00:01:03.840 --> 00:01:14.320
And I think if you're interested in a career in agentic AI, then certainly take a look at least what the exam is about, because I feel that um Anthropic

s12
00:01:14.320 --> 00:01:19.520
knows how people are using their system and what the issues are going to be.

s13
00:01:19.520 --> 00:01:26.040
So, before we jump into that, I want to give a little bit of my uh my philosophy.

s14
00:01:26.880 --> 00:01:36.360
bop bop bop bop May have to do this manually, getting stuck.

s15
00:01:36.360 --> 00:01:42.320
So, this is a quote from uh a woman named Sister Corita Kent.

s16
00:01:42.320 --> 00:01:44.320
Nothing is a mistake.

s17
00:01:44.320 --> 00:01:46.320
There's no win and no fail.

s18
00:01:46.320 --> 00:01:48.480
There's only make.

s19
00:01:48.480 --> 00:01:52.440
Bottom line here is experiment, experiment, experiment.

s20
00:01:52.440 --> 00:01:54.880
Not only should you read, but you should do.

s21
00:01:54.880 --> 00:01:56.080
You should make stuff.

s22
00:01:56.080 --> 00:01:59.360
Now, what happens when you make stuff?

s23
00:01:59.360 --> 00:02:03.720
A lot of times things don't work.

s24
00:02:03.720 --> 00:02:07.040
Thomas Edison said, "I have not failed.

s25
00:02:07.040 --> 00:02:11.240
I've only found 10,000 ways that don't work."

s26
00:02:11.240 --> 00:02:24.360
And what I want to emphasize here is that what this shows us are something that in the design patterns movement, which came around in the early 1990s with object-oriented programming,

s27
00:02:24.360 --> 00:02:27.000
we had patterns for objects.

s28
00:02:27.000 --> 00:02:31.400
We now have patterns for agents, but there's also anti-patterns.

s29
00:02:31.400 --> 00:02:44.360
And I think anti-patterns are a key to understanding what you should not do because understanding what you should not do is the key to leading you to what you should do.

s30
00:02:44.800 --> 00:02:50.800
So, a little bit about the Claude Certified Exam, released in March, so it's brand new.

s31
00:02:50.800 --> 00:02:55.040
It is uh it is based on scenarios.

s32
00:02:55.040 --> 00:02:55.840
It is timed.

s33
00:02:55.840 --> 00:02:57.360
It is proctored.

s34
00:02:57.360 --> 00:03:11.360
It is available to companies in the Claude ecosystem, the Anthropic ecosystem, but individuals can pay $99 and take the exam once every once every 6 months.

s35
00:03:11.360 --> 00:03:15.040
And it's not just multiple-choice questions.

s36
00:03:15.040 --> 00:03:24.080
It is multiple-choice, but they're they are based on uh realistic constraints and realistic scenarios.

s37
00:03:24.519 --> 00:03:26.680
The five domains.

s38
00:03:26.680 --> 00:03:30.080
There are five domains that are covered and they give you the percentages of each.

s39
00:03:30.080 --> 00:03:33.280
So, agentic architecture, 27%.

s40
00:03:33.280 --> 00:03:39.440
Claude code, how to configure the Claude code system and workflow, 20%.

s41
00:03:39.440 --> 00:03:47.480
How to doing prompt engineering, structuring your output, using JSON all over the place.

s42
00:03:47.480 --> 00:03:49.120
Tool design.

s43
00:03:49.120 --> 00:03:51.720
Model context protocol integration.

s44
00:03:51.720 --> 00:03:56.040
These are topics that you should understand and know whether you're going to take the exam or not.

s45
00:03:56.040 --> 00:04:02.680
This is going to help you get ready for whatever the agentic world is going to throw at you.

s46
00:04:02.680 --> 00:04:06.480
And then there's going to be contact management and reliability.

s47
00:04:06.480 --> 00:04:12.959
So these are the areas of of the kind of questions you're going to run into.

s48
00:04:13.160 --> 00:04:29.680
Then there are and they they provide you with six production scenarios and your the exam will randomly choose four and all the questions will be centered around the four that they choose.

s49
00:04:29.680 --> 00:04:45.640
And what I'm going to do is walk you through um the production scenarios and give you some anti-patterns to be aware of because there's a number of ways you can solve the problem but one of the big things is what not to do and that often can be the key

s50
00:04:45.640 --> 00:04:47.560
to getting these questions right.

s51
00:04:47.560 --> 00:04:50.680
So, number one customer support resolution agent.

s52
00:04:50.680 --> 00:04:58.160
So we have agentic loops, control, something called stop reason which is uh what Cloud Code has.

s53
00:04:58.160 --> 00:05:05.640
Every time something happens, there's a stop reason and you need to take a look at that because that can give you a lot of information about what's going on.

s54
00:05:05.640 --> 00:05:08.240
Uh scenario two, code generation.

s55
00:05:08.240 --> 00:05:12.880
Three, multi-agent research system which we'll look at.

s56
00:05:12.880 --> 00:05:15.960
How do you How do you distribute your agents?

s57
00:05:15.960 --> 00:05:17.480
Hub and spoke.

s58
00:05:17.480 --> 00:05:18.600
Who's the orchestrator?

s59
00:05:18.600 --> 00:05:20.600
How much information should they know?

s60
00:05:20.600 --> 00:05:22.960
All these are important factors.

s61
00:05:22.960 --> 00:05:27.640
Um scenario four, developer productivity with code.

s62
00:05:27.640 --> 00:05:30.160
So how do you do subtask isolation?

s63
00:05:30.160 --> 00:05:33.040
Keep your tasks in their little universes.

s64
00:05:33.040 --> 00:05:39.000
And this hearkens back to what we learn in computer science from doing multi-threaded programming.

s65
00:05:39.000 --> 00:05:45.240
When you have multiple threads operating and sharing memory, then you get into issues with synchronization.

s66
00:05:45.240 --> 00:05:50.120
You You to put locks Keep the little threads independent.

s67
00:05:50.120 --> 00:05:52.840
Keep your agents independent.

s68
00:05:52.840 --> 00:05:58.000
Um and then some cloud code for continuous integration.

s69
00:05:58.000 --> 00:06:03.240
And then we'll look at some patterns for structured data extraction.

s70
00:06:03.240 --> 00:06:06.720
Okay, that's kind of where we're going to go.

s71
00:06:06.720 --> 00:06:10.240
Now, here's something that I I I like to point out.

s72
00:06:10.240 --> 00:06:12.240
Everybody's talking about loops, right?

s73
00:06:12.240 --> 00:06:14.600
Every The loop is the new thing.

s74
00:06:14.600 --> 00:06:22.080
Um uh Boris Cherney says he doesn't write code, but his job is to write loops.

s75
00:06:22.080 --> 00:06:28.640
And Peter Steinberger master of Open Claw says, "I don't I don't uh I don't code anymore.

s76
00:06:28.640 --> 00:06:32.200
I just design loops that prompt your agents."

s77
00:06:32.200 --> 00:06:34.720
So, loops are the new big thing, right?

s78
00:06:34.720 --> 00:06:36.520
Well, no, they're not.

s79
00:06:36.520 --> 00:06:37.400
Okay?

s80
00:06:37.400 --> 00:06:45.480
Um back in the day uh early days of computing, we had programming languages were exploding.

s81
00:06:45.480 --> 00:06:49.360
We had Fortran, we had COBOL, and there were big fights.

s82
00:06:49.360 --> 00:06:53.600
My program My programming language is better than yours.

s83
00:06:53.600 --> 00:06:54.400
It can do more.

s84
00:06:54.400 --> 00:06:55.120
No, it can't.

s85
00:06:55.120 --> 00:06:56.520
We can do this.

s86
00:06:56.520 --> 00:07:13.240
Böhm and Jacopini, 1966 proved that if you want a language to be Turing complete, which means can compute anything that computers are possibly able to compute, then you need only three things.

s87
00:07:13.240 --> 00:07:18.720
The ability to to to write statements sequentially, okay?

s88
00:07:18.720 --> 00:07:24.520
To have if-then conditionals, and the third piece is the loop.

s89
00:07:24.520 --> 00:07:28.840
If you add the loop, you have Turing computability.

s90
00:07:28.840 --> 00:07:38.560
And now we are seeing this being resurrected in the agentic world with the focus on loops, cuz up to now we've had sort of sequences.

s91
00:07:38.560 --> 00:07:42.320
You have prompts, you have maybe if-then, but now we have a loop.

s92
00:07:42.320 --> 00:07:44.280
And now this is what's giving us the power.

s93
00:07:44.280 --> 00:07:48.800
This is where the agentic stuff is getting very exciting.

s94
00:07:48.800 --> 00:07:50.720
Okay.

s95
00:07:50.720 --> 00:07:56.080
I'm start with uh with scenario one, customer support resolution.

s96
00:07:56.080 --> 00:08:04.160
So here we have a loop operating and the I'm going to jump to the anti-pattern.

s97
00:08:04.160 --> 00:08:12.520
What you don't want is just to let the agent go and do something and get the response back and use it, okay?

s98
00:08:12.520 --> 00:08:16.600
What you want to do is you want to loop with something called the stop reason.

s99
00:08:16.600 --> 00:08:19.440
So I'm going to show you a little code here.

s100
00:08:19.440 --> 00:08:20.720
So here we have while loop.

s101
00:08:20.720 --> 00:08:22.960
It's a while true, it's a loop.

s102
00:08:22.960 --> 00:08:24.840
We're looping right here, okay?

s103
00:08:24.840 --> 00:08:29.919
So the first little block is where we call uh we call the model, okay?

s104
00:08:29.919 --> 00:08:31.360
And we pass it the messages.

s105
00:08:31.360 --> 00:08:38.840
The messages are essentially the sequence of prompts that exist in the context window, okay?

s106
00:08:38.840 --> 00:08:47.920
And we are asking the and we have a we have a prompt and we have we have the context and we have a tool.

s107
00:08:47.920 --> 00:08:53.440
And we're asking the LLM to do something with this tool and help us out.

s108
00:08:53.440 --> 00:08:56.960
The problem is the LLM can't do anything.

s109
00:08:56.960 --> 00:09:01.720
It is just a probabilistic next word predictor.

s110
00:09:01.720 --> 00:09:03.360
It can't execute tools.

s111
00:09:03.360 --> 00:09:16.920
So what it does though is it can figure out if you point it to a tool, it can figure out how to set things up so that you or your code can execute it.

s112
00:09:16.920 --> 00:09:20.160
So it's important to understand that the LLM is not executing these tools.

s113
00:09:20.160 --> 00:09:26.560
It can't do anything except talk back to you, very intelligently sometimes, but all it can do is talk back to you.

s114
00:09:26.560 --> 00:09:39.520
So when it finishes this task and has a result which is basically here is I've I know what you want.

s115
00:09:39.520 --> 00:09:41.360
I know what the tool can do.

s116
00:09:41.360 --> 00:09:48.640
Here's how I It sets up the parameters that can then be or that then used to actually execute the tool.

s117
00:09:48.640 --> 00:09:55.560
So, the second block you see why did the LLM come back to us?

s118
00:09:55.560 --> 00:09:57.360
That's our stop reason.

s119
00:09:57.360 --> 00:09:58.080
Tool use.

s120
00:09:58.080 --> 00:09:59.320
Oh, okay.

s121
00:09:59.320 --> 00:10:03.480
We've stopped because the LLM it wants to use the tool.

s122
00:10:03.480 --> 00:10:05.480
So, let's just run the tool.

s123
00:10:05.480 --> 00:10:07.200
So, that's what the second block is.

s124
00:10:07.200 --> 00:10:17.200
Run tool, the response is what the LLM said, and it's basically the parameters that it has extracted from the data that you provided it.

s125
00:10:17.200 --> 00:10:17.760
Okay?

s126
00:10:17.760 --> 00:10:19.520
Then it executes that.

s127
00:10:19.520 --> 00:10:20.920
Then it goes back.

s128
00:10:20.920 --> 00:10:22.240
That then it continues.

s129
00:10:22.240 --> 00:10:26.800
Continues means the LLM sees it and says, "Oh, successful run.

s130
00:10:26.800 --> 00:10:28.600
So, okay."

s131
00:10:28.600 --> 00:10:31.000
Come back down.

s132
00:10:31.040 --> 00:10:32.520
We're not running a tool anymore.

s133
00:10:32.520 --> 00:10:34.280
We're end the end of our loop.

s134
00:10:34.280 --> 00:10:35.480
Bingo.

s135
00:10:35.480 --> 00:10:43.240
Now, then we take the answer, and this is an opportunity for you to have a human in the loop potentially.

s136
00:10:43.240 --> 00:10:44.839
You check the confidence.

s137
00:10:44.839 --> 00:10:46.560
If it looks good, you keep it.

s138
00:10:46.560 --> 00:10:49.960
If you don't, then you escalate to a human.

s139
00:10:49.960 --> 00:10:55.320
So, now there's another reason why you need to make sure you check your stop reason.

s140
00:10:55.320 --> 00:11:06.400
One of the stop reasons may be you have run out of tokens, and this response is based on partial when the LLM had to stop.

s141
00:11:06.400 --> 00:11:12.520
And it's going to give you a response, but if you have run out of tokens, then you need to take action.

s142
00:11:12.520 --> 00:11:13.400
Okay.

s143
00:11:13.400 --> 00:11:17.120
Um Next scenario.

s144
00:11:17.120 --> 00:11:19.080
Uh code generation with Claude.

s145
00:11:19.080 --> 00:11:27.839
So, Claude code has this has this concept of the Claude MD file, a markdown file, where you put all the things you wanted to know.

s146
00:11:27.839 --> 00:11:34.720
What Anthropic recommends is you have three levels of Claude.

s147
00:11:34.720 --> 00:11:48.080
One that you have at the top level of your project, the other that you have in inside your sort of the project folder, and then within directories you can also specify.

s148
00:11:48.080 --> 00:11:58.920
So, the idea is to have a hierarchical set of rules that that can then control how the system is going to respond.

s149
00:11:58.960 --> 00:12:00.720
Okay.

s150
00:12:00.720 --> 00:12:05.000
Moving right along, uh we have a multi-agent research system.

s151
00:12:05.000 --> 00:12:16.280
So, here we're going to have uh the problem is how do I how do I get my agents to to go off and do stuff and bring the answers back in a reasonable way?

s152
00:12:16.280 --> 00:12:22.840
The anti-pattern you have one agent and you load it up with tools, all right?

s153
00:12:22.840 --> 00:12:31.400
So, I like to think about you you know, you hire somebody to come to your house, you hire a carpenter to come to the house, and the guy shows up with uh

s154
00:12:31.400 --> 00:12:34.680
plumbing tools, carpenter tools, electrical tools.

s155
00:12:34.680 --> 00:12:35.720
He says, "I can do anything."

s156
00:12:35.720 --> 00:12:39.400
Well, maybe you don't want this guy, maybe you want a a professional carpenter.

s157
00:12:39.400 --> 00:12:41.160
So, that's the kind of idea.

s158
00:12:41.160 --> 00:12:52.080
And this kind of back takes us back to some of the the functional programming uh ideas that functions should be do one thing.

s159
00:12:52.080 --> 00:13:03.120
And if you can get your agents to do one thing, you with maybe one or two tools available to it, then that's going to be a win, and that's going to help you with this exam.

s160
00:13:03.120 --> 00:13:07.200
So, specialize, don't overload.

s161
00:13:07.200 --> 00:13:24.440
The other part of this is don't let your agents context spill over into the main context because context means tokens, tokens mean money, and the more context you have, the more confused

s162
00:13:24.440 --> 00:13:27.320
the LLM is going to be in giving you an answer.

s163
00:13:27.320 --> 00:13:32.240
So, even though oh, a million token context window, I can put everything in there.

s164
00:13:32.240 --> 00:13:34.600
No, no, don't put everything in there.

s165
00:13:34.600 --> 00:13:41.120
Limit what's going to go in there because then you're going to get a much more accurate system.

s166
00:13:41.880 --> 00:13:48.160
So, here's a Here's an example of a specialized sub agents.

s167
00:13:48.160 --> 00:13:52.080
You're giving it So, this would be the critic.

s168
00:13:52.080 --> 00:13:54.120
So, let's say you've run some stuff.

s169
00:13:54.120 --> 00:13:57.880
Now, you want to get an agent to look at what's happened.

s170
00:13:57.880 --> 00:14:03.760
What you want to do is just give it what it needs to solve that critic problem.

s171
00:14:03.760 --> 00:14:10.680
I'm only giving it here the we're passing it the claim and the evidence.

s172
00:14:10.680 --> 00:14:14.040
So, this is your claim is sort of how we're going to solve the problem.

s173
00:14:14.040 --> 00:14:24.400
Here's Here's the evidence, but we're not giving it the the thought processes that went in to creating this claim.

s174
00:14:24.400 --> 00:14:25.840
Why?

s175
00:14:25.840 --> 00:14:35.880
When you When you get a bunch of agents together collaborating and talking to each other, there's a tendency to have group think.

s176
00:14:35.880 --> 00:14:41.720
And all the agents seem to kind of devolve into one idea.

s177
00:14:41.720 --> 00:14:53.240
I mean, it's it's like, you know, you're in a group, you know, you're at a party, and everybody wants pizza except you, but then people talk you into you you know, you don't want to be uh

s178
00:14:53.240 --> 00:14:55.120
you don't want to spoil the party, so you'll go along.

s179
00:14:55.120 --> 00:14:58.000
And it seems that agents kind of work in the same way.

s180
00:14:58.000 --> 00:15:04.520
So, you're going to return Basically, you're going to give each agent only a slice.

s181
00:15:04.520 --> 00:15:07.200
I didn't think about the pizza analogy, but yes.

s182
00:15:07.200 --> 00:15:12.760
Every agent gets its own slice, and and it it should come through.

s183
00:15:12.760 --> 00:15:14.760
Okay.

s184
00:15:17.120 --> 00:15:21.600
Fourth scenario, developer productivity.

s185
00:15:21.600 --> 00:15:24.200
So, the anti-pattern.

s186
00:15:25.080 --> 00:15:30.440
Let every subtask dump its full output into the primary thread, crowding out the context.

s187
00:15:30.440 --> 00:15:33.360
Again, this is what we're I was just talking about.

s188
00:15:33.360 --> 00:15:34.640
This is bad.

s189
00:15:34.640 --> 00:15:37.520
Let the context grow unbounded.

s190
00:15:37.520 --> 00:15:38.720
Bad, right?

s191
00:15:38.720 --> 00:15:40.760
For the reasons we just talked about.

s192
00:15:40.760 --> 00:15:47.400
You want to isolate your subtask output, and you want to compact long sessions.

s193
00:15:47.400 --> 00:15:49.960
I'm going to take a second to talk about that.

s194
00:15:49.960 --> 00:15:54.840
So, here's here's a an example of a pattern.

s195
00:15:54.840 --> 00:16:06.680
Uh you want to have your agent uh look at the logs and create a summary of where the problems are in the log.

s196
00:16:06.680 --> 00:16:10.400
So, here's your task, scan all the logs for error.

s197
00:16:10.400 --> 00:16:12.200
Context fork.

s198
00:16:12.200 --> 00:16:25.600
So, you're forking the agent into a like a separate thread where whatever the agent does and thinks and adds tokens to does not come back and pollute the main

s199
00:16:25.600 --> 00:16:28.400
uh the main context.

s200
00:16:28.400 --> 00:16:40.960
Now, you see here what happens, then you take this summation, and then you add that summation without all the other stuff into the overriding context.

s201
00:16:40.960 --> 00:16:45.160
Now, this last little block is kind of interesting, I think.

s202
00:16:45.160 --> 00:16:53.960
Because you can check your token count, and you can determine how big the token count is.

s203
00:16:53.960 --> 00:17:03.240
And if you can set some limit and you know, if if you have more than 150,000 tokens, then what you want to do is you can run a compact.

s204
00:17:03.240 --> 00:17:12.760
So, Anthropic and Claude have these compaction algorithms that take this giant context and and compact it in some way, shape, or form.

s205
00:17:12.760 --> 00:17:17.920
Not quite sure how the implementation is of that, but there is compaction.

s206
00:17:17.920 --> 00:17:26.720
Now, a little side effect a little side channel I've been walking around when you walk outside, you see see these guys handing out these books.

s207
00:17:26.720 --> 00:17:27.480
Okay?

s208
00:17:27.480 --> 00:17:30.360
Anybody see these guys handing out these but take them.

s209
00:17:30.360 --> 00:17:32.160
This is this is actually a pretty good little book.

s210
00:17:32.160 --> 00:17:40.640
In fact, I was looking at it last night and one of the things it had in it was this is by this guy Sam Sam Bagwell.

s211
00:17:40.640 --> 00:17:46.280
I have no connection I didn't even know Sam, but it there's a online page 32.

s212
00:17:46.280 --> 00:17:53.280
It says uh his company provides custom logic for compression of context.

s213
00:17:53.280 --> 00:17:55.120
So, he's got an and you can write your own.

s214
00:17:55.120 --> 00:18:02.760
He's got a he's got he you can extend his base class and have your own compression of your data, whatever you think is important.

s215
00:18:02.760 --> 00:18:07.680
So, I think that's kind of an interesting spin on this whole thing.

s216
00:18:07.680 --> 00:18:09.680
Okay.

s217
00:18:09.920 --> 00:18:20.440
Cloud code for uh uh continuous integration uh anti-pattern Always have interactive modes in a pipeline.

s218
00:18:20.440 --> 00:18:27.880
Well, no no no cuz interactive modes mean uh Cloud will stop and ask you, "You want to do this?

s219
00:18:27.880 --> 00:18:28.480
You want to do that?

s220
00:18:28.480 --> 00:18:29.640
Can I have permission for that?"

s221
00:18:29.640 --> 00:18:34.960
So, there are ways to set it up so that it'll just run straight through, okay?

s222
00:18:34.960 --> 00:18:46.400
The other uh the other tip that I'll give you here is there's something called the uh the batch.

s223
00:18:46.400 --> 00:18:58.400
So, you can take your prompts, you can take your work, and you can put them in a batch and for 50% fewer token cost you will get the result they promise

s224
00:18:58.400 --> 00:19:00.560
in at at least 24 hours.

s225
00:19:00.560 --> 00:19:09.280
So, if you're going to go take a nap, you're going to go on vacation, you're going to go out, take a a day off, run your stuff in batch mode, and you're going to have a a

s226
00:19:09.280 --> 00:19:11.640
less to pay.

s227
00:19:13.800 --> 00:19:15.760
Where am I here?

s228
00:19:15.760 --> 00:19:22.480
All right, I've only got a few few minutes left, few seconds left, but I want to conclude with this.

s229
00:19:22.480 --> 00:19:24.920
Remember, nothing is a mistake.

s230
00:19:24.920 --> 00:19:29.200
There's no win, there's no fail, there's no exam, only make.

s231
00:19:29.200 --> 00:19:32.920
You do it and you make it and you're going to succeed.

s232
00:19:32.920 --> 00:19:38.160
If you want to reach out to me, reach out to me uh coil at Berkeley, look at my websites.

s233
00:19:38.160 --> 00:19:40.440
I got a website co-supreme AI.

s234
00:19:40.440 --> 00:19:45.880
I'm a big jazz fan and I named this website after John Coltrane, Love Supreme, if you know that song, great.

s235
00:19:45.880 --> 00:19:50.240
Anyway, that's my story and I'm sticking to it and I'm about to zero time.

s236
00:19:50.240 --> 00:19:50.803
Okay,

s237
00:19:50.803 --> 00:19:51.000
[applause]

s238
00:19:51.000 --> 00:19:53.200
thank you.