Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase

AI Engineer · 16 min · 179 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00All
00:01[music]
00:12right.
00:16Deploying browser agents.
00:19Uh running a browser agent demo once is easy.
00:24Uh it's on one site.
00:25It's one run on a good day with you watching and your fingers crossed.
00:31Uh, but we know production is vastly different.
00:34Thousands of runs unattended while the site can change or your model can just have an off day.
00:41So that's why we're here.
00:44What separates a demo from production?
00:48My name is Derek Megan.
00:49I'm a software engineer at Browserbase.
00:52And today we'll break up this talk into four parts.
00:55First, we'll bring order to chaos, and we'll understand concretely how agents interact with the web.
01:02Second, we'll define the impossible and explore why running them at scale is so hard.
01:08Third, we'll take a step back.
01:11We'll think about what's important and the value proposition that agents propose.
01:16And then last, which is hopefully the reason why we're all here, we'll generate some shareholder value.
01:23So first how agents interact with the web.
01:28I like to think about the output of a model as a probability distribution.
01:32You receive input tokens.
01:34They represent the page state, the overarching goal or the steps that have already been taken.
01:40Then the model explores a distribution of possible actions proposing the most likely one.
01:46Then that action is transposed into a structured tool call and deterministically executed to interact with the browser.
01:54But the interface of the browser is complicated.
01:57You have several distinct layers.
02:00You have requests flowing in and out of the browser.
02:03You have the document object model or the HTML representation of the page.
02:07You can screenshot the browser.
02:09the accessibility tree which is a semantic textual representation of the page and the browser state storage cookies console URL tabs etc. Underneath all of this, you have a dynamic code execution runtime
02:25where you can interact with the browser by writing arbitrary do JavaScript or CDP.
02:32How do you wire these together?
02:34Typically in industry, there's three main strategies.
02:38Number one is creating a textual representation of the browser.
02:41This typically involves taking the HTML state of the page in the accessibility tree and creating a hybrid representation.
02:49Number two is computer use where you represent the browser in a series of screenshots of the page.
02:54And finally, and becoming increasingly popular, is deharing your agent, removing tailor made tools in favor of dynamic execution environments where the agent can write arbitrary code against the browser.
03:08In practice, most production systems lean on the first, the second, or a combination of the first and the second two strategies.
03:18Now let's think about the type of agentic trajectories on the browser.
03:21And here I define three spanning across a spectrum of agenticness.
03:27From least agentic, we have purposebuilt browser trajectories that are specially made to complete a single task.
03:36To the most agentic, we have just in time browser automations where we don't know what the task is beforehand and the user provides us an arbitrary task to complete.
03:46In the middle, we have something sort of between both.
03:50We're using the browser as an implementation detail to serve a larger goal, such as performing deep research or competitor analysis.
03:59At scale, I really want to hone in on these transactional workflows.
04:04Not because it makes things easy, but that's because we typically see at scale.
04:08These workflows are transactional by nature where we want to complete a unit of work end to end.
04:14On the other side, browser agents provide us a flexible execution environment to handle ambiguity in production.
04:22So, we know how agents interact with the web and the different types of trajectories we may see and we're honing in specifically on transactional workflows.
04:32Why is it so hard in production?
04:36And I think it comes down to a key interaction.
04:41Cost accumulation is continuous while value realization is terminal.
04:46Each step in your trajectory, you're incurring some incremental cost.
04:50Additionally, you're incurring some incremental risk of failure.
04:54Yet, I only receive value from this browser trajectory when all steps of the task have been completed.
05:01That's all to say that with browser agents, there is no partial credit.
05:05And the vast majority of web automations fall into this category.
05:10So you say, "Okay, well each step there's more risk and there's more cost.
05:15If I use a highly capable model, I can sidestep this conundrum."
05:20Take this thought experiment for an example.
05:22You have a browser agent where each step has an independent success rate of 99%.
05:28If your browser trajectory spans 100 steps, the overall success rate at scale starts to look like 36%.
05:37only about a third of the time.
05:40That's not super great and it's certainly not how we deliver reliable outcomes for customers in production.
05:46And so you say, okay, well, we're continuously accumulating costs, we're continuously accumulating risk, and the percentage chance of value realization is very low.
05:58That would suggest to never use a browser agent.
06:01Uh, thankfully, that's not the subject of my talk today.
06:05I think this dynamic is real, but the math isn't the whole story.
06:09And so, it's time to take a step back.
06:13Let's think about what's important and how agents actually deliver value.
06:19And to illustrate the value that I wish agents to deliver, uh, I will propose a simple equation.
06:26Um, perhaps not used enough by many of us.
06:30And this equation is profit equals revenue minus costs.
06:34Yes, very simple.
06:37Um, but perhaps overlooked when thinking about browser agents.
06:40And this is all to say that your agent is just another line item and that your agent should give you more than it takes.
06:47But when we think about what your agent should give you, that begs the question of what does a browser agent actually give you?
06:54And so to answer this question, I like to think about the context of the customer.
07:00At browserbase, we help our customers deploy browser automations for their users.
07:06Oftent times, they're not automating the web for their own needs, but instead performing actions on the web on their customer's behalf.
07:14And so, a browser agent actually gives you a task completed for your customer repeatedly.
07:19Well, it gives you that ideally.
07:22And the question is how you can create a dynamic system that affords the benefit of being able to wade through ambiguous problems while also completing the task repeatedly every time when the environment stays the same.
07:36And so we'll look across three dimensions and in order of importance.
07:41First is performance.
07:43Second is cost and third is maintainability.
07:47And when we think about performance, this is the primary measure of success that we want to focus on.
07:52Because once we know a browser agent is performant, it can complete the task we ask it to do.
07:58Then cost and maintainability simply become optimization problems.
08:02And I like optimization problems because those can be addressed with engineering.
08:07So the first thing we need to ask is what success actually looks like.
08:13And success in these transactional browser trajectories look like some kind of concrete artifact that the run leaves behind.
08:21If you're paying a bill, that might look like a confirmation email.
08:25If you're placing an order, that may look like an order ID or a receipt.
08:30Or if you're submitting a form, that may look like a new record in a system that you can deterministically query.
08:38Second,
08:39how often is an agent correct?
08:43And there's two ways to measure that.
08:46Naively, you may measure the success of an agent on a perr run basis.
08:52Assume you have a browser agent.
08:54It's probability of success on any run is 50%.
08:58Yet, if you permit that agent to retry for a particular transaction, maybe up to four retries, your per transaction success rate quickly climbs to 94%.
09:10And so when you think about what metric matters, again, I'd implore that we ground ourselves in the customer.
09:16The customer does not care necessarily how many retries you perform, but rather that the workflow was completed successfully and reliably.
09:25In short, permit retries and measure success against a per transaction basis.
09:34Next, we'll explore cost and we'll break cost down into two dimensions.
09:39The first is per run, which today is predominantly model costs.
09:44But as engineering evolves in the open source landscape for models, there's an incredible downward price pressure on the cost of intelligence.
09:53And I believe over time, model cost will become a smaller and smaller component of deploying browser agent trajectories.
10:00Next is infrastructure and compute.
10:02What are the real resources you need to run your browser agent trajectories?
10:07And last is additional integration and tooling.
10:10Then we think about maintenance over time.
10:12We need to invest in observability not only in how the agent made decisions but what actually transpired in the browser.
10:20Second is developer time.
10:22When things break or things are not working optimally, people need to come in and reconfigure the harness or augment its capabilities.
10:31And last is re-evaluation.
10:33As the problem changes, sites change, models progress, and we need to re-evaluate our point in time solution to last the test of time.
10:46So, we talked about performance, we talked about unit costs, we also talked about maintainability.
10:53What are the risk factors to the durability of these over time?
10:56Well, first and foremost, the environment can fight back.
11:01Antibbots and just generally we understand the web as not the friendliest place for your agent.
11:07Second, the task changes.
11:10The underlying job shifts beneath the a shifts beneath the agent and we ask the agent to wade through ambiguity.
11:17Luckily, this also benefits from increased model capability as models are more able to intuitively understand the intention of the user.
11:26and therefore in the face of something go wrong course correct and ultimately complete the goal.
11:32Third is a better methods better methods appear and this is sort of a canonical conundrum with automating the web.
11:39What happens if I'm automating the web today and that same website creates an API tomorrow but for the vast majority of browser agent use cases uh an API isn't coming anytime soon.
11:50And finally the model strays off tasks.
11:54models are inherently indeterministic.
11:56So from run to run it may not ide follow the ideal path or the critical path to perform the action.
12:04So finally we've come to the meat of the talk.
12:09How do we concretely generate shareholder value?
12:12This is the question.
12:13Uh we understand that if a model performs well then cost and maintainability are an optimization problem.
12:19So let's explore that in practice together.
12:22We're going to go through a highle architecture of a browser agent for real world tasks.
12:27This task is automating an health insurance portal.
12:32Every step we take in creating this dynamic system should influence one of these key levers.
12:38Cost, performance or maintainability.
12:42So first we'll start with a lowly agent.
12:46The goal of this agent is to log in and download an explanation of benefits.
12:51A request comes in, the agent performs the action on the browser, and this is the minimum necessary system you need to just do something.
12:59Next, you realize that when you go to download that explanation of benefits, that downloading involves programmatically interacting with the page.
13:08And then once you perform that download, you need to retrieve that file from storage and bring it into the agent's runtime.
13:15So you create a tool that does this all in one transaction.
13:19Instead of the model needing to make decisions on the fly, you encapsulate a complex operation into a single tool call.
13:27Next, you want to give the agent an opportunity to verify its work real time.
13:33This can be done by giving it another tool.
13:36This tool is an OCR tool that deterministically extracts entities from the document and compares that against the system of record.
13:44Great.
13:45Now the agent knows if it's did it if it really did its job right or not because it has a deterministic mechanism to verify that.
13:54Next, you realize that while this portal may have ambiguity in your business logic that the authentication workflow is never changing and you want to reuse the same authentication mechanism
14:07for many business operations on the same portal.
14:10So you pull that authentication logic out from the responsibility of the model.
14:15Again, we're reducing the number of steps that the agent has to take to complete the task successfully.
14:21This means lower unit cogs, higher performance, and easier maintainability as well as we can reuse this authentication function across multiple business logic workflows on the same portal.
14:35Last, we've contained the responsibility of the agent to only the area of ambiguity necessary.
14:44So, we draft a skill.
14:46These skills look very similar to a standard operating procedure that we would have humans use in the past, except this standard operating procedure is for an agent, and it allows the agent to follow the critical path more closely.
15:01At the beginning of the talk, we discussed how the output of an agent can be thought of like a probability distribution.
15:08So if we introduce the critical path to the agent, it knows it's at step two and knows that it needs to take step three because we have outlined it in this skill,
15:18then it's more likely to take that step three.
15:21It knows exactly where it needs to go and we remove ambiguity from the agents decision process.
15:26In the end, this is the final result.
15:30We have actually a pretty complex system.
15:32A request comes in, a serverless authentication function authenticates the browser.
15:38That browser is provided to the agent runtime.
15:41The agent navigates the portal using the skill provided, calls a deterministic download function, provides that downloaded file to a verify tool, and can know authoritatively whether its trajectory succeeded or failed.
15:57In the end, this system takes steps out of the model's responsibility that simply do not need to be in the responsibility of the agent.
16:07We reduce the number of steps it needs to take while still being able to achieve the desired outcome.
16:13Thus, the system's more maintainable and it's more performant and we can overcome the browser agent conundrum.
16:21So,
16:23here we are.
16:25Welcome to production and congrats.
16:28Now you're on call.
16:29Thank you.
16:46[music]