Bringing agents onto the world wide web — Paul Klein IV, Browserbase

AI Engineer · 18 min · 226 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello.
00:14Very sleepy crowd in the computer use room.
00:16Have we all given up at this point?
00:18Like, what's going on?
00:19Uh thank you for coming in to my talk.
00:22My name is Paul Klein.
00:23I'm the founder of Browserbase, and I'm going to talk about bringing agents onto the World Wide Web.
00:28If you're in this audience in this track, you've done computer use, you tried operator when it came out, and you're probably like, "Why isn't this happening yet?
00:36This this seems obvious."
00:38Well, we'll address some of the high-level needs of computer use to really serve what I think is the largest category of AI agents, the agents that actually go out and do work on your behalf in the real world.
00:50We'll talk through some of the technical stuff, but really trying to focus on there's a huge model capabilities overhang in this category specifically, and you all here can hopefully solve it.
01:00So, thank you for coming.
01:03Well, of course, as we all know, the web wasn't built for agents.
01:06It was built for people, and that becomes very challenging as we're building systems to try and interact with it and automate it.
01:13So, when we're thinking about building agents that interact with people for systems, we have to really wonder, "Why wasn't it, you know, why was it built for us?
01:21And and why is it struggle?"
01:23When you've done any sort of automation in the past, you've run into so many roadblocks.
01:27You know, the pages change, the web was built in a very context inefficient way.
01:31It's a lot of text, a lot of tokens.
01:33And when you're running any sort of browser agent or web agent right now, you get a broken browser that doesn't spin up.
01:38You know, you have pages that don't work.
01:39You have blockers or other sort of problems that really limit you.
01:43And I actually started my career doing web automation, maintaining these scripts every single day.
01:47It was very painful.
01:48So, in my world, agents have made a huge advancement and allowed me to write durable web automation scripts, but we still haven't gone to agents yet.
01:57And the question I want to ask is why?
01:59We we're sitting here in this room we're thinking computer use we saw a year ago has progress stalled?
02:05Like why why are web agents and browser extensions not as big as it could be?
02:09And I think it's really a comes down to a few things.
02:12You know, until recently the the bottom neck was the models.
02:16The models one year ago really weren't good at long context horizon tasks.
02:21But that's clearly been you know, solved in a major way.
02:25Models can do more and more complex tasks than ever and of course in AI you always have to update your priors.
02:31It's clear to me that anything I believe 6 months ago I have to revisit every single week because these models are progressing at an insanely fast pace.
02:39So I don't think it's the models.
02:41And especially models are now much better at using interfaces.
02:45You know, we've seen this kind of capabilities improvement in computer use models.
02:49In the last year a lot of investment was made in RL environments for coding.
02:54And in the last 6 months months just as much investment has been made in RL environments for computer use.
03:00And computer use models are getting better and you can see this in the evals.
03:03When you train things on human trajectories in RL environments that model our real world, the real web, you can make better models.
03:10So the models are getting there, I promise.
03:13But the kind of so okay, so the models are good.
03:15Why why do agents still struggle to use the web?
03:18What are the what are the problems here?
03:20If if it's a model problem, Docus says if the models were good enough diffusion would just happen.
03:24Uh there's still a lot of work to be done here.
03:27And and to me it's it's no longer just the models.
03:30I'd argue that agents are missing the right harness and tools.
03:33And if you aren't familiar with an agent harness, if you haven't been on Twitter in the last few weeks, it's the it's the scaffolding and systems around your model
03:39that enable it to actually interact with the world.
03:41Uh a lot of talks and talk about harness engineering we're going to spend a lot of time on today, but I really think you can invest a lot in a harness and get a lot more out of the models and extract that overhang out of the models.
03:52You know, Karpathy actually tweeted this back in November 2023 and I thought it was just so forward-looking that this what he described, these systems around an LLM, it's the harness.
04:02It's It's the the tools that it can access.
04:04And if you fast-forward 3 years later, a lot of what we're doing every single day is building towards this.
04:09A code interpreter for the LLM, audio and video input like screenshots, a browser, and other LLMs as subagents.
04:15All of these principles have held true.
04:16So, if you're ever wondering what should I build next, just go look at Karpathy's old talks.
04:20He's a pretty good predictor of the future.
04:23And you know, applying this to coding, we know that harnesses work really, really well with coding.
04:30Uh on the the graphic on the right, you can see Factory when it compared to Claude Code using the same model but using their kind of custom uh harness.
04:38And it turns out when you build a harness optimized for the domain that your agent is operating in, it can actually achieve, you know, above model results in that domain.
04:47Harness engineering is a real thing.
04:49And I'd say Cursor actually started this.
04:51Cursor was the first one that was doing a model engineering or harness engineering on top of the original LLMs.
04:58And a lot of what we've done at Browserbase with browser models has been, you know, harness engineering.
05:03But and I think that like building a good harness is an engineering problem.
05:06You don't have to be a lab to build a good harness.
05:09And And a lot of us in the room maybe aren't working at labs.
05:11Your company can make a great harness for your domain and actually improve model results.
05:15You don't just have to wait for the models to catch up.
05:16And once again, I believe the models are quite capable now.
05:20And And if you look at this, you can see that it's not just Cursor, it's not just Factory.
05:25You know, many, many different types of companies are building coding harnesses on top of models and over performing on the models' capabilities.
05:32Now, it's not clear yet if custom harnesses are going to beat out durable, you know, RL'd models.
05:38Uh but we're not going to debate that today.
05:39We know that adding a harness on a model improves results, whether or not, you know, Claude Code will be the best harness ever or not.
05:45I think that's a different conversation.
05:46But you should still have some sort of harness on your model and measure the performance versus baseline model.
05:52And what I really want to get back to is that there is a massive capabilities overhang in computer use.
05:56The models are good enough, but we haven't done the engineering work to solve it.
06:00And I love this great Brokman tweet where he says, "Whenever I don't use Codex for a task, I I ask myself why?"
06:06And it feels like the task is an outside of the capabilities of the model.
06:09The overhang is there.
06:10The actual work we can do is missing.
06:13And when you look at the amount of like task completion you can get with coding, it's so much higher than COA because we haven't actually really pushed the models far enough and given it the right tools.
06:24So, to me not only is this important because I think that non-coding is a much bigger opportunity than it is coding.
06:33If you look at this in recent slide, there's so many use cases that are in the non-coding domain that can benefit from computer use.
06:39It's a problem worth investing in, and the wrong answer is to sit around and just wait for the models to get better.
06:44You can actually solve this today.
06:47Solving overhang is an engineering problem, and this is the work that we can do within our companies and within our agents, especially within the computer use domain, to build reliable web agents.
06:57And when I think about browser agents that work, it it really comes down to three different types of things.
07:03They're multimodal, they're harness engineered, and they have reliable infrastructure.
07:07And I'll go through each of these three.
07:10First, they're multimodal.
07:11You no longer have to use a single model to actually interact with the task.
07:16And we see this with coding agents all the time.
07:18Sometimes you'll use a smarter model for a more complex page.
07:21Sometimes a a dumber model for a simpler page.
07:24And maybe you're using a combination of coding and computer use to actually power your agent.
07:28This is a really important insight.
07:30It turns out automating the web isn't always just clicking the button on the screen.
07:34It might be intercepting the network requests and writing a coding agent or having coding write a script to actually replay those network requests.
07:41The most reliable browser agents that we see in production right now are often writing code alongside using the browser to actually automate a task.
07:49If you've done any sort of personal automation work in your life, you might see Cloud code output a script more often than using, you know, Cloud in Chrome because that's a very context-efficient
07:58way to automate a repeatable task.
08:01There's also harness engineering.
08:03It turns out that sure we can, you know, write scripts and use models, but doing these things repeatedly, you want to benefit from things like memory and skills.
08:11We launched something if we so-called browser.sh, which actually publishes skills for websites.
08:16So, before your agent even goes to the website, it can observe what types of tasks it can do.
08:21WebMCP is very useful for this.
08:22It's part of pulling in existing knowledge to optimize a website.
08:26Your agent doesn't have to discover something in the first place if it's done it before.
08:29It can use its memory and its skills to actually make it better.
08:33And you should think about trying to build in skills to your agents.
08:36If your agent is using CLIs to control websites like the Playwright CLI, you could actually give it skills and context to be more effective there.
08:44And this results in much more optimized token usage.
08:47As if you're throwing everything on the page to a model, you're going to get sub-par results and it's going to cost you a lot.
08:52The right harness should not only present the right tools, but present an optimized amount of tokens that are compressed to get exactly the right repeatable result every single time.
09:02And finally, the infrastructure here is extremely important because when you're running browser agents in production, you want an environment that's going to work everywhere, every time.
09:13And I think a lot of work still needs to be done here.
09:15This is a lot of what our company does because computer use environments are pretty complex to scale up.
09:20You know, it's funny when Open Cloud came out, everyone started buying Mac minis, which to me feels like an infrastructure problem, right?
09:26You're running Open Cloud on a Mac mini in your house because that's the best way to run Mac OS that you can SSH into and then end up like solving the CAPTCHAs cuz of your home IP address.
09:37That is not something you can do when you're building thousands of agents for customers in production.
09:41I've yet to see a SOC 2 compliant Mac mini setup at scale, but please tell me afterwards if you found one.
09:46I'm very curious about it.
09:47The infrastructure problem that needs to be solved here is also an engineering problem.
09:51And most importantly, the consistency in this infrastructure is important.
09:55When your agent is running across a website multiple times, you want it to see the same inputs and outputs, the same page layout, the same size.
10:03If your infrastructure renders a page in a mobile layout one time and then like in a desktop layout the second time, it's going to have inconsistent results.
10:10Consistency in the infrastructure is the nice base layer on top of your harness and on top of your models to actually get good results with this.
10:19I also think we have to improve the web itself.
10:21So, there's a whole other side of this problem that's very interesting, which is like, how are we going to make it so the web works well with agents?
10:26And I think this is arguably the harder challenge because we're not just engineering on our own systems anymore.
10:32We have to be evangelists to the web and to the broader world that, "Hey, you want agents to come to your website."
10:38So, accessibility is the first thing I want to talk about.
10:42There's been a lot of really cool stuff here.
10:43Now, when you look at what best-in-class browser agents are doing, they're not just consuming the raw DOM and HTML of the page anymore.
10:50They're looking at sub, you know, subsections of that like the accessibility tree, the area tags.
10:55These are labeled components of a page that can help show your agent where it needs to click and why.
11:01Chrome just added WebMCP, which I think is really, really cool.
11:04Websites can now publish MCP servers within their page that your agent can take advantage of without pre-installing the actual MCP.
11:12It can now issue tool calls to a website like submit the registration form in a way that's not only context context context-efficient, but is website approved and blessed.
11:22More and more work can go into accessibility and we've seen things like LLMs.text, skills.md, agents.md all being published alongside our websites.
11:32We need to see more of that to build the agent-first web.
11:37I think authentication is actually an even bigger problem here, too, because once your agent can actually go to a website, how can it log in on your behalf?
11:45There's been a lot of different paradigms here.
11:46Maybe you're just giving your agent your password, but doing that securely can be very challenging.
11:51Maybe you're creating a service account for your agent where it has some limited access and you constantly have to give it new permissions.
11:57You know, authentication for agents is the next thing to be solved once you solve the harnessing capability problems, and doing that securely where you can have a human loop approve certain actions on a website
12:08is going to be a major challenge for unlocking computers for the enterprise.
12:11The biggest gate to building agents that actually can work in prod is going to be the systems it has access to.
12:17And authentication is something that needs to be solved in our industry to make this possible.
12:21I've seen a lot of really cool stuff come out.
12:22WorkOS just launched OffMD, which is a new way for your agent that goes to a website to find how to sign up on that website and get its own accounts.
12:30And if you're building software now, you should think about what is my agent-first sign up and login flow look like because agents are going to be using your software whether you like it or not.
12:38It's best to let them use it securely.
12:41Finally, I want to talk about trust.
12:43The web was built to stop bad bots, but now there's good agents and bad bots.
12:49How do we delineate between the two?
12:51And the CAPTCHA has been the tool in our tool chest for a very long time, but as we all know, CAPTCHAs are not as effective as we think against agents.
12:59And trying to identify these good agents is very important.
13:02There's been a lot of cool frameworks and work done on things like web bot off and in more authenticated ways to say this is my agent, it's coming from me, and you can follow me along on the web, but I still don't think we've solved the issue yet.
13:14And a big unlock to agents accessing the web alongside authentication is actually how can we trust these agents?
13:21And I think there needs to be almost like a Verisign moment for web agents where who can be the certificate issuer in saying my agent is trusted in this agent vendor is trusted?
13:30Nobody's come out and done that yet.
13:32I think those are really really big opportunities.
13:35So, building reliable browser agents is is not a model problem.
13:40It's an engineering problem that all of us can solve, but doing that engineering is is a full-time job.
13:48And if you are working in the space, I'd love to meet you.
13:51But if you aren't and you just want to build something that works, I might feel ideas.
13:56You really don't have to reinvent the wheel here.
13:58There's been a lot of stuff happening and it's a consortium of companies that are continuing to push the world forward on what's possible when you want to automate the web.
14:06And I think there's like a few things here that are really important for a great solution, right?
14:10It has to be a scalable platform that serves your infrastructure needs.
14:13You know, you can want to run run one agent, but also thousands of agents.
14:16And And the challenge is that those different levels of scale is very very important.
14:20You want browser agents that are model agnostic.
14:22As a developer, I don't want to be locked into a single model provider.
14:26As models continually change and get better, I want to be able to move my agent around.
14:30That's why you need model agnostic infrastructure.
14:33You need somebody to solve agent identity.
14:35Somebody who's going to go out and negotiate with the, you know, anti-bot providers of the world and say, "We are the platform for trusted agents and we are the ones that can help broker the access for your agents as you use the web."
14:47And finally, you need observability.
14:49When you're building these agents that go to any website in the world, you need to see where they're going and why and how that you can make sure that it's improving every iteration.
14:57Every agent you run should get better every single time.
14:59You need screen recordings, logs, network activity.
15:02And you need to feed that back into your agent so it can self-improve.
15:05We published something called Auto Browse earlier this year.
15:08That's a really interesting way to see how is my agent able to improve itself over multiple loops.
15:13And the the feed-in of data to that from observability is extremely important to make your agents get better over time.
15:20And And that's, you know, what we're building here at BrowserBase.
15:22You know, we power browser agents, web data extraction, and really all these use cases across the entire web to make your agents work well.
15:30And what I've been extremely surprised by in building this company is the plethora of use cases.
15:35Of course, there are the, you know, large AI native companies that use companies like BrowserBase to power the browser agents, but there's also all these little companies across the world that can benefit from automation.
15:47And my core belief with this company is that solving computer use accelerates the diffusion of AI to the real economy.
15:54And as much as I love our bubble here in San Francisco, the real economy is companies like the logistics company in Singapore, the bank in South Africa, or the lumber factory in Mexico.
16:03These people are built on PHP websites with forums and human beings clicking buttons every single day.
16:09That's a huge opportunity for you to go solve to build browser agents for them, and hopefully you can use the right infrastructure to power those things.
16:18And and that's why we built BrowserBase agents, by the way.
16:20This is our new product we launched yesterday, because we want to give everyone a battery-included agent and harness for everything they need to automate the web.
16:27The goal here is that you shouldn't reinvent the wheel.
16:29You shouldn't have to figure all this out and optimize it.
16:31You should benefit from the platform scale that we've seen millions and millions of sessions every single month, and understand how we've solved the edge cases for you, so you don't have to solve them on your own.
16:42I have a quick little demo here.
16:43The way it works is instead of having to pull our tools together, you can actually put in a prompt, and we will stand up the harness, the runtime, the sandbox, the code execution,
16:52the fetch, the search tools, and the models to actually accomplish a task for you.
16:56And what's beautiful is as this agent is running, it's looking at its steps, and it's remembering what it can do, and learning from it, so it can do them again in the future.
17:06The future for you is not having to reinvent the wheel every single time.
17:09It's actually being able to use an agent that's purpose-built for browsing the web, and pull it in as a sub-agent of your larger agentic system.
17:16This is not the main thing you should be focusing your time on.
17:19You should be focusing your time on actually solving customer problems, not trying to rebuild the best in class browser agents.
17:25The optimization feature is quite cool.
17:27It's going to look back and actually understand, hey, how can I do this better after looking back at this this is this data feedback loop that I've talked about before.
17:34And I think it's what makes agents really, really special.
17:38I kind of want to end with this last note.
17:41You know, based on the attendance in the room, I do think a lot of people have stepped back from computers because they've had so much challenges over the past year making browser agents work in production,
17:51but I can tell you first hand from our customers we see it working.
17:54And actually, I think 1 year from now this room is going to be overfilled with people because the models are getting better, the techniques are getting better, and the tools are getting better.
18:01It's just on us to build better things.
18:03Thank you all for having me today.
18:04I really appreciate it.
18:06[applause]
18:23[music]