WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=5xi_S1f9sDU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/5xi_S1f9sDU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:01.309
All

s2
00:00:01.309 --> 00:00:03.309
[music]

s3
00:00:12.960 --> 00:00:14.960
right.

s4
00:00:16.720 --> 00:00:19.279
Deploying browser agents.

s5
00:00:19.279 --> 00:00:24.080
Uh running a browser agent demo once is easy.

s6
00:00:24.080 --> 00:00:25.760
Uh it's on one site.

s7
00:00:25.760 --> 00:00:31.359
It's one run on a good day with you watching and your fingers crossed.

s8
00:00:31.359 --> 00:00:34.719
Uh, but we know production is vastly different.

s9
00:00:34.719 --> 00:00:41.600
Thousands of runs unattended while the site can change or your model can just have an off day.

s10
00:00:41.600 --> 00:00:44.320
So that's why we're here.

s11
00:00:44.320 --> 00:00:48.079
What separates a demo from production?

s12
00:00:48.079 --> 00:00:49.600
My name is Derek Megan.

s13
00:00:49.600 --> 00:00:52.239
I'm a software engineer at Browserbase.

s14
00:00:52.239 --> 00:00:55.520
And today we'll break up this talk into four parts.

s15
00:00:55.520 --> 00:01:02.480
First, we'll bring order to chaos, and we'll understand concretely how agents interact with the web.

s16
00:01:02.480 --> 00:01:08.640
Second, we'll define the impossible and explore why running them at scale is so hard.

s17
00:01:08.640 --> 00:01:11.119
Third, we'll take a step back.

s18
00:01:11.119 --> 00:01:16.400
We'll think about what's important and the value proposition that agents propose.

s19
00:01:16.400 --> 00:01:23.680
And then last, which is hopefully the reason why we're all here, we'll generate some shareholder value.

s20
00:01:23.680 --> 00:01:28.320
So first how agents interact with the web.

s21
00:01:28.320 --> 00:01:32.240
I like to think about the output of a model as a probability distribution.

s22
00:01:32.240 --> 00:01:34.320
You receive input tokens.

s23
00:01:34.320 --> 00:01:40.079
They represent the page state, the overarching goal or the steps that have already been taken.

s24
00:01:40.079 --> 00:01:46.159
Then the model explores a distribution of possible actions proposing the most likely one.

s25
00:01:46.159 --> 00:01:53.920
Then that action is transposed into a structured tool call and deterministically executed to interact with the browser.

s26
00:01:54.640 --> 00:01:57.759
But the interface of the browser is complicated.

s27
00:01:57.759 --> 00:02:00.320
You have several distinct layers.

s28
00:02:00.320 --> 00:02:03.439
You have requests flowing in and out of the browser.

s29
00:02:03.439 --> 00:02:07.520
You have the document object model or the HTML representation of the page.

s30
00:02:07.520 --> 00:02:09.520
You can screenshot the browser.

s31
00:02:09.520 --> 00:02:25.760
the accessibility tree which is a semantic textual representation of the page and the browser state storage cookies console URL tabs etc. Underneath all of this, you have a dynamic code execution runtime

s32
00:02:25.760 --> 00:02:31.680
where you can interact with the browser by writing arbitrary do JavaScript or CDP.

s33
00:02:32.480 --> 00:02:34.720
How do you wire these together?

s34
00:02:34.720 --> 00:02:38.400
Typically in industry, there's three main strategies.

s35
00:02:38.400 --> 00:02:41.760
Number one is creating a textual representation of the browser.

s36
00:02:41.760 --> 00:02:49.040
This typically involves taking the HTML state of the page in the accessibility tree and creating a hybrid representation.

s37
00:02:49.040 --> 00:02:54.800
Number two is computer use where you represent the browser in a series of screenshots of the page.

s38
00:02:54.800 --> 00:03:08.560
And finally, and becoming increasingly popular, is deharing your agent, removing tailor made tools in favor of dynamic execution environments where the agent can write arbitrary code against the browser.

s39
00:03:08.560 --> 00:03:17.519
In practice, most production systems lean on the first, the second, or a combination of the first and the second two strategies.

s40
00:03:18.000 --> 00:03:21.840
Now let's think about the type of agentic trajectories on the browser.

s41
00:03:21.840 --> 00:03:27.519
And here I define three spanning across a spectrum of agenticness.

s42
00:03:27.519 --> 00:03:36.799
From least agentic, we have purposebuilt browser trajectories that are specially made to complete a single task.

s43
00:03:36.799 --> 00:03:46.400
To the most agentic, we have just in time browser automations where we don't know what the task is beforehand and the user provides us an arbitrary task to complete.

s44
00:03:46.400 --> 00:03:50.319
In the middle, we have something sort of between both.

s45
00:03:50.319 --> 00:03:59.519
We're using the browser as an implementation detail to serve a larger goal, such as performing deep research or competitor analysis.

s46
00:03:59.920 --> 00:04:04.159
At scale, I really want to hone in on these transactional workflows.

s47
00:04:04.159 --> 00:04:08.400
Not because it makes things easy, but that's because we typically see at scale.

s48
00:04:08.400 --> 00:04:14.720
These workflows are transactional by nature where we want to complete a unit of work end to end.

s49
00:04:14.720 --> 00:04:22.799
On the other side, browser agents provide us a flexible execution environment to handle ambiguity in production.

s50
00:04:22.800 --> 00:04:32.880
So, we know how agents interact with the web and the different types of trajectories we may see and we're honing in specifically on transactional workflows.

s51
00:04:32.880 --> 00:04:36.000
Why is it so hard in production?

s52
00:04:36.000 --> 00:04:41.040
And I think it comes down to a key interaction.

s53
00:04:41.040 --> 00:04:46.639
Cost accumulation is continuous while value realization is terminal.

s54
00:04:46.639 --> 00:04:50.960
Each step in your trajectory, you're incurring some incremental cost.

s55
00:04:50.960 --> 00:04:54.880
Additionally, you're incurring some incremental risk of failure.

s56
00:04:54.880 --> 00:05:01.360
Yet, I only receive value from this browser trajectory when all steps of the task have been completed.

s57
00:05:01.360 --> 00:05:05.759
That's all to say that with browser agents, there is no partial credit.

s58
00:05:05.759 --> 00:05:10.400
And the vast majority of web automations fall into this category.

s59
00:05:10.560 --> 00:05:15.440
So you say, "Okay, well each step there's more risk and there's more cost.

s60
00:05:15.440 --> 00:05:20.479
If I use a highly capable model, I can sidestep this conundrum."

s61
00:05:20.479 --> 00:05:22.960
Take this thought experiment for an example.

s62
00:05:22.960 --> 00:05:28.320
You have a browser agent where each step has an independent success rate of 99%.

s63
00:05:28.320 --> 00:05:37.600
If your browser trajectory spans 100 steps, the overall success rate at scale starts to look like 36%.

s64
00:05:37.600 --> 00:05:40.560
only about a third of the time.

s65
00:05:40.560 --> 00:05:46.479
That's not super great and it's certainly not how we deliver reliable outcomes for customers in production.

s66
00:05:46.479 --> 00:05:58.240
And so you say, okay, well, we're continuously accumulating costs, we're continuously accumulating risk, and the percentage chance of value realization is very low.

s67
00:05:58.240 --> 00:06:01.919
That would suggest to never use a browser agent.

s68
00:06:01.919 --> 00:06:05.120
Uh, thankfully, that's not the subject of my talk today.

s69
00:06:05.120 --> 00:06:09.919
I think this dynamic is real, but the math isn't the whole story.

s70
00:06:09.919 --> 00:06:13.520
And so, it's time to take a step back.

s71
00:06:13.520 --> 00:06:19.199
Let's think about what's important and how agents actually deliver value.

s72
00:06:19.199 --> 00:06:26.639
And to illustrate the value that I wish agents to deliver, uh, I will propose a simple equation.

s73
00:06:26.639 --> 00:06:30.880
Um, perhaps not used enough by many of us.

s74
00:06:30.880 --> 00:06:34.960
And this equation is profit equals revenue minus costs.

s75
00:06:34.960 --> 00:06:37.039
Yes, very simple.

s76
00:06:37.039 --> 00:06:40.400
Um, but perhaps overlooked when thinking about browser agents.

s77
00:06:40.400 --> 00:06:47.520
And this is all to say that your agent is just another line item and that your agent should give you more than it takes.

s78
00:06:47.520 --> 00:06:54.800
But when we think about what your agent should give you, that begs the question of what does a browser agent actually give you?

s79
00:06:54.800 --> 00:07:00.240
And so to answer this question, I like to think about the context of the customer.

s80
00:07:00.240 --> 00:07:06.240
At browserbase, we help our customers deploy browser automations for their users.

s81
00:07:06.240 --> 00:07:14.080
Oftent times, they're not automating the web for their own needs, but instead performing actions on the web on their customer's behalf.

s82
00:07:14.080 --> 00:07:19.599
And so, a browser agent actually gives you a task completed for your customer repeatedly.

s83
00:07:19.599 --> 00:07:22.160
Well, it gives you that ideally.

s84
00:07:22.160 --> 00:07:36.639
And the question is how you can create a dynamic system that affords the benefit of being able to wade through ambiguous problems while also completing the task repeatedly every time when the environment stays the same.

s85
00:07:36.639 --> 00:07:41.680
And so we'll look across three dimensions and in order of importance.

s86
00:07:41.680 --> 00:07:43.599
First is performance.

s87
00:07:43.599 --> 00:07:47.039
Second is cost and third is maintainability.

s88
00:07:47.039 --> 00:07:52.880
And when we think about performance, this is the primary measure of success that we want to focus on.

s89
00:07:52.880 --> 00:07:58.160
Because once we know a browser agent is performant, it can complete the task we ask it to do.

s90
00:07:58.160 --> 00:08:02.639
Then cost and maintainability simply become optimization problems.

s91
00:08:02.639 --> 00:08:07.840
And I like optimization problems because those can be addressed with engineering.

s92
00:08:07.840 --> 00:08:13.759
So the first thing we need to ask is what success actually looks like.

s93
00:08:13.759 --> 00:08:21.680
And success in these transactional browser trajectories look like some kind of concrete artifact that the run leaves behind.

s94
00:08:21.680 --> 00:08:25.759
If you're paying a bill, that might look like a confirmation email.

s95
00:08:25.759 --> 00:08:30.400
If you're placing an order, that may look like an order ID or a receipt.

s96
00:08:30.400 --> 00:08:38.320
Or if you're submitting a form, that may look like a new record in a system that you can deterministically query.

s97
00:08:38.320 --> 00:08:39.919
Second,

s98
00:08:39.919 --> 00:08:43.360
how often is an agent correct?

s99
00:08:43.360 --> 00:08:46.720
And there's two ways to measure that.

s100
00:08:46.720 --> 00:08:52.080
Naively, you may measure the success of an agent on a perr run basis.

s101
00:08:52.080 --> 00:08:54.080
Assume you have a browser agent.

s102
00:08:54.080 --> 00:08:58.560
It's probability of success on any run is 50%.

s103
00:08:58.640 --> 00:09:10.320
Yet, if you permit that agent to retry for a particular transaction, maybe up to four retries, your per transaction success rate quickly climbs to 94%.

s104
00:09:10.320 --> 00:09:16.640
And so when you think about what metric matters, again, I'd implore that we ground ourselves in the customer.

s105
00:09:16.640 --> 00:09:25.680
The customer does not care necessarily how many retries you perform, but rather that the workflow was completed successfully and reliably.

s106
00:09:25.680 --> 00:09:33.360
In short, permit retries and measure success against a per transaction basis.

s107
00:09:34.160 --> 00:09:39.360
Next, we'll explore cost and we'll break cost down into two dimensions.

s108
00:09:39.360 --> 00:09:44.320
The first is per run, which today is predominantly model costs.

s109
00:09:44.320 --> 00:09:53.040
But as engineering evolves in the open source landscape for models, there's an incredible downward price pressure on the cost of intelligence.

s110
00:09:53.040 --> 00:10:00.560
And I believe over time, model cost will become a smaller and smaller component of deploying browser agent trajectories.

s111
00:10:00.560 --> 00:10:02.880
Next is infrastructure and compute.

s112
00:10:02.880 --> 00:10:07.200
What are the real resources you need to run your browser agent trajectories?

s113
00:10:07.200 --> 00:10:10.320
And last is additional integration and tooling.

s114
00:10:10.320 --> 00:10:12.720
Then we think about maintenance over time.

s115
00:10:12.720 --> 00:10:20.640
We need to invest in observability not only in how the agent made decisions but what actually transpired in the browser.

s116
00:10:20.640 --> 00:10:22.880
Second is developer time.

s117
00:10:22.880 --> 00:10:31.519
When things break or things are not working optimally, people need to come in and reconfigure the harness or augment its capabilities.

s118
00:10:31.519 --> 00:10:33.760
And last is re-evaluation.

s119
00:10:33.760 --> 00:10:45.600
As the problem changes, sites change, models progress, and we need to re-evaluate our point in time solution to last the test of time.

s120
00:10:46.399 --> 00:10:53.200
So, we talked about performance, we talked about unit costs, we also talked about maintainability.

s121
00:10:53.200 --> 00:10:56.640
What are the risk factors to the durability of these over time?

s122
00:10:56.640 --> 00:11:01.440
Well, first and foremost, the environment can fight back.

s123
00:11:01.440 --> 00:11:07.760
Antibbots and just generally we understand the web as not the friendliest place for your agent.

s124
00:11:07.760 --> 00:11:10.240
Second, the task changes.

s125
00:11:10.240 --> 00:11:17.519
The underlying job shifts beneath the a shifts beneath the agent and we ask the agent to wade through ambiguity.

s126
00:11:17.519 --> 00:11:26.079
Luckily, this also benefits from increased model capability as models are more able to intuitively understand the intention of the user.

s127
00:11:26.079 --> 00:11:32.399
and therefore in the face of something go wrong course correct and ultimately complete the goal.

s128
00:11:32.399 --> 00:11:39.839
Third is a better methods better methods appear and this is sort of a canonical conundrum with automating the web.

s129
00:11:39.839 --> 00:11:50.959
What happens if I'm automating the web today and that same website creates an API tomorrow but for the vast majority of browser agent use cases uh an API isn't coming anytime soon.

s130
00:11:50.959 --> 00:11:54.000
And finally the model strays off tasks.

s131
00:11:54.000 --> 00:11:56.320
models are inherently indeterministic.

s132
00:11:56.320 --> 00:12:04.000
So from run to run it may not ide follow the ideal path or the critical path to perform the action.

s133
00:12:04.000 --> 00:12:09.040
So finally we've come to the meat of the talk.

s134
00:12:09.040 --> 00:12:12.079
How do we concretely generate shareholder value?

s135
00:12:12.079 --> 00:12:13.279
This is the question.

s136
00:12:13.279 --> 00:12:19.120
Uh we understand that if a model performs well then cost and maintainability are an optimization problem.

s137
00:12:19.120 --> 00:12:22.480
So let's explore that in practice together.

s138
00:12:22.480 --> 00:12:27.839
We're going to go through a highle architecture of a browser agent for real world tasks.

s139
00:12:27.839 --> 00:12:32.399
This task is automating an health insurance portal.

s140
00:12:32.399 --> 00:12:38.480
Every step we take in creating this dynamic system should influence one of these key levers.

s141
00:12:38.480 --> 00:12:42.720
Cost, performance or maintainability.

s142
00:12:42.720 --> 00:12:46.639
So first we'll start with a lowly agent.

s143
00:12:46.639 --> 00:12:51.040
The goal of this agent is to log in and download an explanation of benefits.

s144
00:12:51.040 --> 00:12:59.680
A request comes in, the agent performs the action on the browser, and this is the minimum necessary system you need to just do something.

s145
00:12:59.680 --> 00:13:08.560
Next, you realize that when you go to download that explanation of benefits, that downloading involves programmatically interacting with the page.

s146
00:13:08.560 --> 00:13:15.519
And then once you perform that download, you need to retrieve that file from storage and bring it into the agent's runtime.

s147
00:13:15.519 --> 00:13:19.519
So you create a tool that does this all in one transaction.

s148
00:13:19.519 --> 00:13:27.040
Instead of the model needing to make decisions on the fly, you encapsulate a complex operation into a single tool call.

s149
00:13:27.040 --> 00:13:33.360
Next, you want to give the agent an opportunity to verify its work real time.

s150
00:13:33.360 --> 00:13:36.079
This can be done by giving it another tool.

s151
00:13:36.079 --> 00:13:44.639
This tool is an OCR tool that deterministically extracts entities from the document and compares that against the system of record.

s152
00:13:44.639 --> 00:13:45.360
Great.

s153
00:13:45.360 --> 00:13:54.000
Now the agent knows if it's did it if it really did its job right or not because it has a deterministic mechanism to verify that.

s154
00:13:54.000 --> 00:14:07.279
Next, you realize that while this portal may have ambiguity in your business logic that the authentication workflow is never changing and you want to reuse the same authentication mechanism

s155
00:14:07.279 --> 00:14:10.639
for many business operations on the same portal.

s156
00:14:10.639 --> 00:14:15.279
So you pull that authentication logic out from the responsibility of the model.

s157
00:14:15.279 --> 00:14:21.040
Again, we're reducing the number of steps that the agent has to take to complete the task successfully.

s158
00:14:21.040 --> 00:14:35.360
This means lower unit cogs, higher performance, and easier maintainability as well as we can reuse this authentication function across multiple business logic workflows on the same portal.

s159
00:14:35.519 --> 00:14:44.639
Last, we've contained the responsibility of the agent to only the area of ambiguity necessary.

s160
00:14:44.639 --> 00:14:46.720
So, we draft a skill.

s161
00:14:46.720 --> 00:15:01.040
These skills look very similar to a standard operating procedure that we would have humans use in the past, except this standard operating procedure is for an agent, and it allows the agent to follow the critical path more closely.

s162
00:15:01.040 --> 00:15:08.560
At the beginning of the talk, we discussed how the output of an agent can be thought of like a probability distribution.

s163
00:15:08.560 --> 00:15:18.399
So if we introduce the critical path to the agent, it knows it's at step two and knows that it needs to take step three because we have outlined it in this skill,

s164
00:15:18.399 --> 00:15:21.199
then it's more likely to take that step three.

s165
00:15:21.199 --> 00:15:26.959
It knows exactly where it needs to go and we remove ambiguity from the agents decision process.

s166
00:15:26.959 --> 00:15:30.399
In the end, this is the final result.

s167
00:15:30.399 --> 00:15:32.959
We have actually a pretty complex system.

s168
00:15:32.959 --> 00:15:38.320
A request comes in, a serverless authentication function authenticates the browser.

s169
00:15:38.320 --> 00:15:41.360
That browser is provided to the agent runtime.

s170
00:15:41.360 --> 00:15:57.360
The agent navigates the portal using the skill provided, calls a deterministic download function, provides that downloaded file to a verify tool, and can know authoritatively whether its trajectory succeeded or failed.

s171
00:15:57.360 --> 00:16:07.040
In the end, this system takes steps out of the model's responsibility that simply do not need to be in the responsibility of the agent.

s172
00:16:07.040 --> 00:16:13.040
We reduce the number of steps it needs to take while still being able to achieve the desired outcome.

s173
00:16:13.040 --> 00:16:21.839
Thus, the system's more maintainable and it's more performant and we can overcome the browser agent conundrum.

s174
00:16:21.839 --> 00:16:23.279
So,

s175
00:16:23.279 --> 00:16:25.120
here we are.

s176
00:16:25.120 --> 00:16:28.000
Welcome to production and congrats.

s177
00:16:28.000 --> 00:16:29.600
Now you're on call.

s178
00:16:29.600 --> 00:16:31.839
Thank you.

s179
00:16:46.746 --> 00:16:48.746
[music]