WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=JGtkTjn4120

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/JGtkTjn4120.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:12.960
I'm Mitch.

s3
00:00:12.960 --> 00:00:15.720
I'm one of the two co-founders of Basis.

s4
00:00:15.720 --> 00:00:30.320
We are based right here in New York City and the thing that we focus on is how can we get agents to do production-scale work reliably enough so that we are trusted

s5
00:00:30.320 --> 00:00:35.440
and used end-to-end by accounting firms across the country.

s6
00:00:35.440 --> 00:00:37.560
So, a lot of our research really focuses on this.

s7
00:00:37.560 --> 00:00:41.960
How can you get these agents to do real work in the real economy?

s8
00:00:41.960 --> 00:00:54.520
So, I want to maybe start with a question, which is if you're in the at this conference, honestly, if you're watching this talk, you've probably increased your own productivity a lot.

s9
00:00:54.520 --> 00:00:59.720
You're using co-work or uh Codex or cloud code, whatever it is.

s10
00:00:59.720 --> 00:01:10.520
And yet, if you go and look at the average company, the average enterprise, the average firm, their productivity hasn't gone up 10x, 100x.

s11
00:01:10.520 --> 00:01:13.960
I know my productivity has gone up maybe 100x.

s12
00:01:13.960 --> 00:01:18.840
But, if I were working inside of a company, it wouldn't be anywhere near that order of magnitude.

s13
00:01:18.840 --> 00:01:21.400
And so, why is that?

s14
00:01:22.160 --> 00:01:28.200
At Basis, one of the things we have is that humans are already used to working with non-deterministic systems.

s15
00:01:28.200 --> 00:01:32.720
It's just those systems are normally their co-workers, not their computers.

s16
00:01:32.720 --> 00:01:41.560
And so, I actually think to answer this question, it's useful to maybe look at the existing way we organize intelligent systems, which are companies.

s17
00:01:41.560 --> 00:01:47.560
So, how do organizations scale their own productivity, right?

s18
00:01:47.640 --> 00:02:00.440
Organizations need to trust that the people doing the work at every level are doing it in a way that is going to produce good outputs consistently at scale, right?

s19
00:02:00.440 --> 00:02:07.412
If look at this image here, the uh here I'm showing an accounting firm, but this could honestly be any company, traditional company.

s20
00:02:07.412 --> 00:02:15.572
The person who is the head of the firm, they have literally no idea about the exact day-to-day of the people at the very bottom.

s21
00:02:15.572 --> 00:02:19.212
And yet somehow they trust the work is going to be done.

s22
00:02:19.212 --> 00:02:21.532
How does that happen?

s23
00:02:21.852 --> 00:02:24.812
It doesn't happen through deterministic systems, right?

s24
00:02:24.812 --> 00:02:26.252
People are not deterministic.

s25
00:02:26.252 --> 00:02:28.212
They're intelligent entities.

s26
00:02:28.212 --> 00:02:38.092
And instead there are processes that are put in place, and those processes, if we believe the people who, you know, put them in place, think that those processes will generalize

s27
00:02:38.092 --> 00:02:49.292
to good outcomes, then they can be comfortable that even though the people executing the processes are not deterministic, you will generally get to good outcomes if you follow the process.

s28
00:02:49.372 --> 00:02:51.092
This is true, by the way, even in the smallest steps.

s29
00:02:51.092 --> 00:03:03.852
So, in this example, earlier I was showing the process of maybe doing the monthly accounting work for a client at an accounting firm, but even within that process, say you wanted to see, how do I reconcile the cash?

s30
00:03:03.852 --> 00:03:09.132
Well, I need to get the bank data, and even that has a process.

s31
00:03:09.132 --> 00:03:10.572
The process could be really basic.

s32
00:03:10.572 --> 00:03:17.092
It could just be use the approved software to extract the data, but there's still some process that is put in place.

s33
00:03:17.092 --> 00:03:27.612
You wouldn't expect the firm to just let anyone Google, you know, extract PDF and then just use whatever they want to go extract it, because now you wouldn't have process.

s34
00:03:29.532 --> 00:03:33.012
Now let's talk about agents.

s35
00:03:33.932 --> 00:03:41.772
As I said earlier, everyone already has co-work, ChatGPT, Codex, Cloud Code, but where is that productivity?

s36
00:03:42.012 --> 00:03:52.972
Our opinion is there are two main problems that come with what I'll call coding-style agent products, where you're working with an agent and you just tell it to go and do something.

s37
00:03:52.972 --> 00:04:04.014
The first problem is that there is a competency gap, because agents, as you think of them as you think of them today, they outsource the complexity to the user.

s38
00:04:04.014 --> 00:04:14.534
Right, if I'm the user, I now have to manage my skill and memories and think about how to when I should use the agent.

s39
00:04:14.534 --> 00:04:16.454
I have to have like theory of mind over it.

s40
00:04:16.454 --> 00:04:18.614
This is really hard.

s41
00:04:18.614 --> 00:04:29.094
And for those who are here, maybe you guys find it easy, but I promise for the wide majority of knowledge workers, they have no idea how to do any of this.

s42
00:04:29.574 --> 00:04:31.254
And there's a second problem.

s43
00:04:31.254 --> 00:04:33.374
Which is there is a process gap.

s44
00:04:33.374 --> 00:04:39.254
Right, if you are saying, "Hey user, here's your agent, do what you want."

s45
00:04:39.254 --> 00:04:47.614
Well, now suddenly the way the work is being done is essentially up to whoever the user is to customize the way the agent is behaving.

s46
00:04:47.614 --> 00:04:51.934
Which goes entirely against the way that organizations scale.

s47
00:04:51.934 --> 00:05:00.134
The process should be coming from senior leadership down, not from any individual employee deciding how they're going to do their job.

s48
00:05:00.134 --> 00:05:10.534
Right, if you going back to my example, if every employee just made up their own way of extracting data, you know, even back in 2018, that wouldn't be a good outcome.

s49
00:05:10.534 --> 00:05:13.093
And so, we believe strongly that the key

s50
00:05:13.093 --> 00:05:13.094
[snorts]

s51
00:05:13.094 --> 00:05:17.934
to building production agents that actually do work end-to-end is solving these two problems.

s52
00:05:17.934 --> 00:05:22.414
You have to solve the consistency gap by making it simple.

s53
00:05:22.414 --> 00:05:25.374
To the user, you shouldn't even have to know it's an agent.

s54
00:05:25.374 --> 00:05:28.534
Or you probably know, but you don't have to really feel it.

s55
00:05:28.534 --> 00:05:31.654
It should be as simple as just working with a co-worker.

s56
00:05:31.654 --> 00:05:35.974
You just tell it what you want it to do and all the things you expect to work should work.

s57
00:05:35.974 --> 00:05:37.294
And the second is the process gap.

s58
00:05:37.294 --> 00:05:40.694
You need agents to be reliable.

s59
00:05:40.694 --> 00:05:44.214
Not having a stroke of genius one day and forgetting something the other day.

s60
00:05:44.214 --> 00:05:47.534
It's not about being perfect.

s61
00:05:47.534 --> 00:05:49.174
Humans are not perfect.

s62
00:05:49.174 --> 00:05:51.414
Right, nobody is looking for perfect.

s63
00:05:51.414 --> 00:05:56.934
What you're looking for is reliability and consistency and trustworthiness.

s64
00:05:57.494 --> 00:06:00.334
So, what does that actually look like?

s65
00:06:00.614 --> 00:06:08.855
We think that to be simple and reliable where you know it's you don't have to think about what to do.

s66
00:06:08.855 --> 00:06:13.255
The agent can just do it for you without the user knowing the internals.

s67
00:06:13.255 --> 00:06:21.575
To do it the same way consistently, you have to be you have to be opinionated.

s68
00:06:21.575 --> 00:06:23.975
Let me just show you actually one slide back here.

s69
00:06:23.975 --> 00:06:30.495
I want to point out that getting it right isn't actually the main task, because again, humans are imperfect themselves.

s70
00:06:30.495 --> 00:06:32.735
The point is to do it in a similar way.

s71
00:06:32.735 --> 00:06:35.615
It's okay if you you can't do something.

s72
00:06:35.615 --> 00:06:40.055
But you should be consistently able to not do something, and then you get better, just like a person does.

s73
00:06:40.055 --> 00:06:48.255
If you suddenly one day are an expert at something, and then you miss something very basic that would be expected, now you're not reliable.

s74
00:06:48.815 --> 00:06:51.215
So you have to be opinionated.

s75
00:06:51.295 --> 00:06:54.055
What's an example of an opinion?

s76
00:06:54.335 --> 00:06:58.695
Let's say we have an agent, and we need to go and read a PDF.

s77
00:06:58.695 --> 00:07:02.095
So in this case, what is the best way to read a PDF?

s78
00:07:02.095 --> 00:07:02.815
There's many ways.

s79
00:07:02.815 --> 00:07:13.415
You can throw it into a VLM, you can go write some Python code and use an open-source OCR library, you can work with one of the many OCR-specific model providers.

s80
00:07:13.415 --> 00:07:17.615
You could do an OCR, and then you could have the VLM check its work.

s81
00:07:17.615 --> 00:07:19.415
You could have five VLMs check its work.

s82
00:07:19.415 --> 00:07:21.735
You could do 100 VLMs and take the best of them.

s83
00:07:21.735 --> 00:07:24.815
There's literally infinite possibilities here.

s84
00:07:24.815 --> 00:07:33.935
And so if you just said, "Agent, extract Excel extract PDF," you might get good outcomes, but they're not going to be reliable.

s85
00:07:33.935 --> 00:07:34.055
Right?

s86
00:07:34.055 --> 00:07:39.415
Not because the process it did was bad, but because the process it did was not opinionated.

s87
00:07:39.935 --> 00:07:50.575
So the way we develop opinions is you use initial intuition, right?

s88
00:07:50.575 --> 00:07:53.295
Some things you know what the opinions are from expertise.

s89
00:07:53.295 --> 00:08:04.575
For example, if I'm trying to get an agent to do tax work well, I know that if the agent is going goes and reads the primary source before returning to the user,

s90
00:08:04.575 --> 00:08:07.255
it is more likely to be accurate.

s91
00:08:07.255 --> 00:08:09.297
I don't need a set of evals to tell me that.

s92
00:08:09.297 --> 00:08:11.817
I don't need to divine that from first principles.

s93
00:08:11.817 --> 00:08:13.697
We know that, right?

s94
00:08:13.697 --> 00:08:22.137
I know that if I have an agent make a PowerPoint, if it takes a picture of the PowerPoint before it returns to user, it is less likely to have a formatting issue.

s95
00:08:22.137 --> 00:08:24.457
That is obvious.

s96
00:08:24.817 --> 00:08:27.617
And so, there are lots of different ways to develop opinions.

s97
00:08:27.617 --> 00:08:34.137
And one of the key points here, I'll just make before I move on to talking about how to use the opinions, is that the opinions are not just about accuracy,

s98
00:08:34.137 --> 00:08:34.297
right?

s99
00:08:34.297 --> 00:08:37.817
We just talked about the example of taking a picture of a PowerPoint.

s100
00:08:37.817 --> 00:08:40.057
Well, that adds latency, right?

s101
00:08:40.057 --> 00:08:41.097
You may not want that.

s102
00:08:41.097 --> 00:08:46.737
Maybe the agent only makes formatting mistakes one out of a hundred times, and you're okay with that.

s103
00:08:46.737 --> 00:08:50.097
And so, therefore, you don't have to take a picture of the PowerPoint.

s104
00:08:50.097 --> 00:08:52.297
The point isn't exactly what the opinion is.

s105
00:08:52.297 --> 00:09:02.257
And this talk, we're going to focus not on how to develop opinions, but why opinions matter, and then how to leverage those opinions as you scale in production.

s106
00:09:02.257 --> 00:09:09.337
So, say that you have a bunch of outcome-based evals.

s107
00:09:09.337 --> 00:09:13.457
And when I say outcome-based evals, I mean what we traditionally think of as evals.

s108
00:09:13.457 --> 00:09:15.337
It has a right answer.

s109
00:09:15.337 --> 00:09:18.097
We have some rubric we're trying to judge what the agent did.

s110
00:09:18.097 --> 00:09:22.857
How do you know that the evals generalize to production?

s111
00:09:22.857 --> 00:09:23.057
Right?

s112
00:09:23.057 --> 00:09:25.457
You might have a thousand cases or a hundred cases.

s113
00:09:25.457 --> 00:09:30.417
But in production, if you're doing real work, you're going to have hundreds of thousands of situations.

s114
00:09:30.417 --> 00:09:46.657
And that's only going to increase as the trajectories get much longer, because the amount of decisions that are made in a trajectory uh just grow exponentially as the like long horizon abilities of agents increase.

s115
00:09:47.137 --> 00:09:54.737
And so, I can't prove if I If you're looking at all these distribution of uh trajectories, you can't prove every one of them is correct.

s116
00:09:54.737 --> 00:09:57.777
You don't have a ground truth for every trajectory.

s117
00:09:57.897 --> 00:10:06.297
But you can potentially look at the trajectories to decide if the behavior, the opinion that you have, is actually happening.

s118
00:10:06.297 --> 00:10:12.338
And you can then infer that if it's actually happening, it's more likely to generalize to production.

s119
00:10:12.338 --> 00:10:13.018
It's not perfect.

s120
00:10:13.018 --> 00:10:18.578
You don't know for sure, but at least you are having an opinion.

s121
00:10:18.658 --> 00:10:20.418
Let's run through a real example.

s122
00:10:20.418 --> 00:10:34.978
So, let's say that I uh for a behavior, this is a an accounting example where if there is a uh mistake for uh in in the bank reconciliation, um

s123
00:10:34.978 --> 00:10:40.338
uh I don't want the agent to like post an entry to make it tie.

s124
00:10:40.338 --> 00:10:44.898
I want it to uh it sort of explain the difference instead of putting a plug.

s125
00:10:44.898 --> 00:10:46.098
Again, this is subjective.

s126
00:10:46.098 --> 00:10:48.338
You might want the other way.

s127
00:10:48.338 --> 00:10:53.218
But, what you don't want is a customer getting one way one time and another way the other time.

s128
00:10:53.218 --> 00:10:54.378
And so, what would you do?

s129
00:10:54.378 --> 00:10:58.018
Well, first, let's understand what governs the behavior.

s130
00:10:58.018 --> 00:10:58.538
Right?

s131
00:10:58.538 --> 00:11:03.098
The behavior's not just you write the two sentences into your system prompt and it just works.

s132
00:11:03.098 --> 00:11:05.058
That might work for very simple narrow agents.

s133
00:11:05.058 --> 00:11:16.498
But, when you're building these big agent systems, the behavior's actually a combination of the system prompt, your progressive disclosures, uh potentially tool descriptions, obviously the training of the model.

s134
00:11:16.498 --> 00:11:19.658
There's a lot of things that govern the behavior.

s135
00:11:19.658 --> 00:11:30.218
And one tip, by the way, I have for when you think about building production agents is really think of the context the agent is going to see as it's working as runtime training data.

s136
00:11:30.218 --> 00:11:32.778
The word prompt, I think, trivializes it a bit, right?

s137
00:11:32.778 --> 00:11:37.058
At the end of the day, this is the data that the agent is being trained on at runtime.

s138
00:11:37.058 --> 00:11:40.058
Not literally trained on, but effectively trained on.

s139
00:11:40.058 --> 00:11:48.378
And so, you want to care about every single token and understand how that data causes the agent to behave in certain ways.

s140
00:11:48.378 --> 00:12:01.338
So, when you're putting uh behaviors into production, it it might be very difficult for one behavior to put it in exactly one, you know, paragraph that you know exactly because again, the behavior is all of the data at once,

s141
00:12:01.338 --> 00:12:03.618
not just one thing.

s142
00:12:03.618 --> 00:12:11.058
And one of the things that we developed at Basis was this concept of a behavior spec.

s143
00:12:11.058 --> 00:12:20.859
And the idea of a behavior spec is essentially saying before we update the context or write the context, uh what do we actually want?

s144
00:12:20.859 --> 00:12:22.379
What is our intent?

s145
00:12:22.379 --> 00:12:26.939
You can kind of think of this a little bit as like a product spec for the agent era.

s146
00:12:26.939 --> 00:12:29.299
And you can't write You shouldn't be writing every detail down.

s147
00:12:29.299 --> 00:12:34.699
You should be taking the most important opinions and putting it because if you put everything, you'll be duplicating a lot of state.

s148
00:12:34.699 --> 00:12:37.459
So, you define the ones that matter the most.

s149
00:12:37.459 --> 00:12:46.419
And then you can use them as a way to align your team from a spec perspective, but then you can also use them as a rubric.

s150
00:12:46.419 --> 00:12:58.339
And you can say in this particular trajectory, whether that is in any trajectory that doesn't have a ground truth, did the agent get into a situation in which this behavior was relevant?

s151
00:12:58.339 --> 00:13:02.939
And then if so, did it actually exhibit the behavior?

s152
00:13:02.939 --> 00:13:13.339
And this way you can scale the amount of signal that you get on your trajectories far larger than trying to increase the amount of outcome-based evals, ground truths, and rubrics that you have.

s153
00:13:13.339 --> 00:13:23.499
Those are really important, but you cannot scale those anywhere near as fast as you can scale the ability to supervise what the behaviors of the agent are.

s154
00:13:24.099 --> 00:13:26.779
Let's just take a quick example here.

s155
00:13:26.779 --> 00:13:35.419
Um this is so this is actually the second concept, um which is called forking, which we can combine here with behavior specs to think about.

s156
00:13:35.419 --> 00:13:48.539
Forking, if you've ever used uh cloud code or or Codex the web apps, there's a little little fork icon uh under each message, uh where you can essentially create a copy of the agent using the same thing, which works really well on your local computer.

s157
00:13:48.539 --> 00:13:59.059
It's hard to build a system that does this in the cloud perfectly with recreating the database environment and recreating the uh the prompts, uh all the all the different environments.

s158
00:13:59.059 --> 00:14:02.259
But forking is another tool that we use to check the behavior.

s159
00:14:02.259 --> 00:14:14.579
So, a flow that you can imagine is saying we had a trajectory, could be in an eval set, it could be in testing or in production, and it failed the behavior eval.

s160
00:14:14.579 --> 00:14:16.581
And now you're trying to understand why.

s161
00:14:16.581 --> 00:14:19.661
Why did it fail failed the behavior eval?

s162
00:14:19.661 --> 00:14:20.941
Why did it not exhibit the behavior?

s163
00:14:20.941 --> 00:14:31.221
And then you can go into the trajectory, go right before the point in which the agent made the decision that was in contrast to the behavior, assuming again your behavior is good, and

s164
00:14:31.221 --> 00:14:33.661
it's actually producing signal.

s165
00:14:33.661 --> 00:14:35.341
And then you can fork it.

s166
00:14:35.341 --> 00:14:41.901
And when you fork it, you can ask the agent uh or you would fork it right after it made the decision.

s167
00:14:41.901 --> 00:14:44.141
You could ask it why did you do that?

s168
00:14:44.141 --> 00:14:50.301
Or you can go and update the uh context of the agent and see if that improves the behavior.

s169
00:14:50.301 --> 00:15:00.421
And so with forking you can start to actually take the signal and start to figure out, okay, how can we then improve the harness to now uh leverage the signal?

s170
00:15:00.421 --> 00:15:09.221
Whereas if you're just looking at signal from an outcome-based eval, you can't do that anywhere near as easily because it failed the outcome, how do you know why?

s171
00:15:09.221 --> 00:15:15.621
You have potentially a thousand inference calls in your 5-hour long agent trajectory run.

s172
00:15:15.621 --> 00:15:17.261
And then we have the last tool that I'd like to talk about.

s173
00:15:17.261 --> 00:15:20.581
This is actually my favorite, which is called simulations.

s174
00:15:20.581 --> 00:15:33.621
Uh this is going to sound silly, but I promise it's very effective, which is that you can, when working with coding agents, ask the coding agent to make a sub agent that pretends to be your production agent.

s175
00:15:33.621 --> 00:15:38.461
And you have to tell it to not cheat, and you give it the system prompt and the skills, and you render everything perfectly.

s176
00:15:38.461 --> 00:15:41.901
And what you can do is you can ask it what it would do in some scenario.

s177
00:15:41.901 --> 00:15:46.861
You can give it the situation, and see if you can recreate the situation that you saw in your eval or in production.

s178
00:15:46.861 --> 00:15:54.861
And if you can recreate it, now you can tweak your context to see if the sub agent would change its behavior based on your tweaks.

s179
00:15:54.861 --> 00:16:03.381
And the reason to do this is obviously far worse than forking, but you do this because this is much cheaper, much faster, and allows you as an agent builder

s180
00:16:03.381 --> 00:16:12.021
to really easily iterate on ideas without needing to go through the expensive process of actually forking an entire environment.

s181
00:16:12.261 --> 00:16:14.581
And you can do this today without building anything.

s182
00:16:14.581 --> 00:16:23.222
Um I did this for literally year year and a half plus before we uh built forking into our internal systems.

s183
00:16:23.622 --> 00:16:36.542
And the main idea here is understanding that if you look at the the distribution from earlier, you're trying to understand is the is the uh agent generalizing to that distribution.

s184
00:16:36.542 --> 00:16:38.942
And so you don't know purely based on the outcome eval.

s185
00:16:38.942 --> 00:16:43.662
So you can try to choose the best outcome evals that should represent the curve, but you don't really know.

s186
00:16:43.662 --> 00:16:52.702
And so supervising the process through some of these methods gives you some additional signal in order to understand if it's generalizing properly.

s187
00:16:53.062 --> 00:17:01.702
And what's really cool is as you start to get more signal, you can actually start to close the loop.

s188
00:17:01.702 --> 00:17:03.742
And an opinion is what lets you do that, right?

s189
00:17:03.742 --> 00:17:08.942
If you don't have opinions about your system, you're not going to be able to develop that signal.

s190
00:17:10.262 --> 00:17:21.862
And now if we go back to the firm, we talk about talk about supervision, we talked about how the agent uh if he's our opinionated, he can be more reliable and he actually can be simpler.

s191
00:17:21.862 --> 00:17:31.182
What that means is the agent maybe it starts doing something narrow, but because people will work with it, they see that it's easy to use, and they see that it's reliable.

s192
00:17:31.182 --> 00:17:33.462
As it gets better and better, they will trust it more.

s193
00:17:33.462 --> 00:17:35.382
It's just like a co-worker.

s194
00:17:35.382 --> 00:17:36.862
You have a co-worker day one.

s195
00:17:36.862 --> 00:17:39.942
Imagine someone sends you a PR.

s196
00:17:39.942 --> 00:17:43.622
You're not going to if it's you're looking at two axes whenever someone sends you a PR.

s197
00:17:43.622 --> 00:17:45.102
It's these two axes.

s198
00:17:45.102 --> 00:17:50.022
What's the black greatest of the PR, and what is your perception of how competent that person is?

s199
00:17:50.022 --> 00:17:51.142
Right?

s200
00:17:51.142 --> 00:17:58.022
If the least competent person in your company is sending you a PR that's editing a core service, you're going to look at it.

s201
00:17:58.022 --> 00:18:04.542
If the most competent person is sending you a PR that's editing an internal document, you may not look at it as much.

s202
00:18:04.542 --> 00:18:10.142
And so this idea is true with humans working with agents just like it is with humans working with humans.

s203
00:18:10.382 --> 00:18:17.582
And I think I just wanted to make one last point to close here, which is that one of the things that people say when they hear this is, "Well, wait a second.

s204
00:18:17.582 --> 00:18:21.023
If you let the agents just do what they want, won't they find a better way?

s205
00:18:21.023 --> 00:18:23.703
Won't they move 37 your work?

s206
00:18:23.703 --> 00:18:26.103
And the answer is they might.

s207
00:18:26.103 --> 00:18:30.663
But that's not how you scale GDP and scale output.

s208
00:18:30.663 --> 00:18:35.423
Nobody wants their agent to move 37 their tax return.

s209
00:18:35.423 --> 00:18:42.623
They want it done reliably, at scale, in a way that they can trust, and in a way that their people know how to work with.

s210
00:18:42.623 --> 00:18:47.583
And so the thing I will leave you with is for agents to diffuse, they need to be simple.

s211
00:18:47.583 --> 00:18:50.903
For teams to trust them, they need to be reliable.

s212
00:18:50.903 --> 00:19:03.103
The way you get to simple and reliable is you develop opinions on how the agent behaves, and then opinions allow you to build that additional signal that helps you start to close the loop so that you can achieve self-improving

s213
00:19:03.103 --> 00:19:04.583
agents in production.

s214
00:19:04.583 --> 00:19:06.263
Thank you very much.

s215
00:19:06.263 --> 00:19:08.263
[applause]