Computer-use models will agentify the web, not APIs — Dhruv Batra, Yutori

AI Engineer · 21 min · 270 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Um welcome.
00:14Let's get started.
00:15So, my name is Drupad I want to talk to you about an argument.
00:19There's an argument online that says, roughly speaking, AI agents will be the drivers of actions on the web, not humans.
00:28That there will be more agents clicking buttons, reading things, buying things for us than human eyeballs on the web.
00:35Um you drive one step deeper into the argument and you ask, "How will this happen?"
00:40Because today the web is extremely hostile to automated traffic.
00:45And the answer usually is, "The web will be agentified."
00:49Um you drive one step deeper into, "What does that mean and how will that happen?"
00:55And the answer usually is, "With APIs that agents will call via some set of protocols."
01:01And the set of protocols grows over time.
01:03It was initially supposed to be MCP servers, then web MCP, and for payments there are 20 different competing protocols.
01:11And you know, every single company wants to introduce their own protocol along the way.
01:15But okay, there will be APIs is the is the argument.
01:19My claim today and argument and what I hope to convince you today is that this last bit is wrong.
01:24I think the first two I generally agree with.
01:27This last bit that suddenly the web will provide you APIs for accessing things is is just delusional.
01:34And my claim is computer use agents and computer use models will agentify the web, not APIs.
01:41And more specifically, the long tail of the web.
01:44Um the head of the distribution, the most popular website, perhaps will give you the API, but the long tail will not.
01:51Um and the story starts uh usually with these sorts of use cases.
01:55I I agree that they also frustrate me when people show a use case of find me business class flights from Miami to Palma de Mallorca for August and somehow they expect that behind the scenes it'll be a computer use agent clicking buttons on flights.google.com.
02:11I find this example, this is my own example.
02:13I I've not dunked you on everybody else.
02:15I find this example bizarre.
02:17Why would you possibly do it this way?
02:19Aren't you aware that there are already aggregators?
02:22You send them a variable, they will send you a JSON.
02:26Your LLMs can do tool calling.
02:28Why would you click buttons?
02:30The purpose of those clicking buttons is to generate this behind the scenes.
02:34So, we're on the same page there.
02:36The next step usually is, okay, let's roll this out to some other case.
02:40Um, I want to know, does my favorite restaurant or does this restaurant have gluten-free items on the restaurant's menu?
02:50And so, I assume that there's going to be a similar endpoint somewhere for that restaurant website or via an aggregator.
02:59Oops.
03:02Um, that accepts a query that I can filter through, where I can ask for give me your menu items that are gluten-free.
03:11Um, I want to for those of you who can already see, I want to rid you of the delusion that such an endpoint exists.
03:19Uh, and I want to show you just for the sake of, you know, being on the same page, what two restaurant pages look like.
03:26Um, there are three that are on the screen.
03:29This is the first one.
03:30This is what we imagine a prototypical, this is easy mode.
03:33You go to a web page, there's no API, but at least it's text.
03:38It's in German, sure.
03:40Different part of the world, models can speak languages.
03:43I see text and I see pricings.
03:46Maybe this is easily scrapable.
03:47Okay.
03:48This is easy mode.
03:50The medium mode is a uh uh uh a page like this where of course the thing that you're looking for is slightly hidden away.
03:59You press the menu button, it actually takes you to a PDF.
04:02Okay, fine.
04:03If your agent has to do this, it needs a PDF reader mode, fine.
04:08Like, you know, I can highlight these.
04:10Okay, so this is at least text, no problem.
04:12Like, you know, I will dump this into ChatGPT or whatever and it'll do it.
04:17Here's the hard mode.
04:19This is what a hard mode website for a restaurant looks like where it's just pictures of the people who created the restaurant, um of where they're located, of menu items,
04:34and you're like, "Where is the menu?
04:37Uh does anybody speak Spanish?"
04:40Nuestra carta, see?
04:42Okay, let's let's check that's our menu.
04:45Um you click there.
04:47Oh my god, these are What am I staring at?
04:51Okay, uh you click on it.
04:52Oh, it's a gallery of of individual What is this?
04:56This is a This is a pixelated This is There's no text here.
05:02I am looking at JPEGs embedded in a gallery which contain the PDF items.
05:08You download this and you put it into ChatGPT and it's having It's struggling doing OCR on this thing.
05:15Okay, so this is what the web looks like.
05:19You're telling me this will give you an API endpoint that you can pass gluten-free items and it'll tell you what that is.
05:28That's an example number one.
05:30Let's take a look at another examples.
05:33Uh this is much more, you know, enterprise-focused.
05:37Um um I am a business.
05:40I want to sell to a school district.
05:42There's about 15 to 20,000 school districts in the US.
05:45And I want to ask the question, is this school district where I give you, you know, mydistrict.gov, is this school district procuring a laptop right now?
05:56Right?
05:56It's a simple question, and I want a similar endpoint.
06:00I want to say RFP status open and topic laptop.
06:04And you think this will exist.
06:07Uh let me tell you what these websites look like again.
06:11Um this is what easy mode looks like.
06:15This is, you know, Ithaca public schools.
06:18Um okay, there's some portal.
06:20Um maybe there's, you know, some, you know, enrollment is maybe the wrong place to go.
06:26This is what slightly harder mode looks like.
06:29Um this is a different public school.
06:32You look at the menu, under finance, there's purchasing, under purchasing, there's certain solicitations.
06:41Um scan PDF again.
06:42Okay, very nice.
06:43Uh so, no text in here, but this is how they tell you about what they have have purchased.
06:49Um and for ultimate boss level, I want to show you a different school district where in order to find information, you have to file a request for access of info or information under Freedom of Information Act.
07:06And then what they do is they scan your email that you sent to them, and they will scan that email, put it on a Google Drive, and then attach PDFs
07:17associated with your request.
07:19These are the people you're telling me will give you an MCP server?
07:25The the amount of delusion here is off the charts.
07:29Like Like if there was ever a time to to say go touch grass, like
07:35[laughter]
07:36I think this is it.
07:38Okay.
07:39Uh so, this is what My claim here is that the web we forget is massive.
07:49It is extremely big.
07:51The the number of websites out there somewhere, you know, the active websites are somewhere 200 million.
07:57The total number of websites are sitting in a in a billion.
08:00Infrastructure changes very slowly.
08:03Um you can imagine as an engineer you know, getting unfettered access to software systems and letting your favorite coding agent rip and generate an API endpoint.
08:16Even if that technological problem is solved, which I agree, seems like on a horizon it should eventually be solved, you're not going to get unfettered access to these institutions
08:25and these institutions change very slowly.
08:28There are still places that are faxing each other.
08:31You're you're not going to be able to suddenly change this overnight.
08:35Okay.
08:36So at this point of time you might be thinking, "Fine.
08:39I'm stuck with infra as it is.
08:41Um so there's not going to be APIs available.
08:44But I have coding agents.
08:45Why don't I just throw them at the HTML?
08:48There is after all, if the browser's doing it, it's a piece of code.
08:52My coding agent should be able to do it as well."
08:55Fair fair point.
08:56I you know, I also thought this way two years ago.
08:59Let me tell you what what the web actually looks like.
09:03Um If I ask you the question, what was the final score of this game between Minnesota Timberwolves and the Brooklyn Nets?
09:11Here's what the web page looks like.
09:12This is a modern website.
09:14You know, it's nba.com.
09:16You and I can see the score, 125 versus 109.
09:19Okay.
09:21Uh behind the scenes, when you actually load the page and read it, this is what initially gets loaded.
09:28There's an empty placeholder initially.
09:30And you wait a few hundred milliseconds to a few seconds, depending on your network connection, and your browser makes an asynchronous call later to fetch the information.
09:41So, the browser makes a call to an endpoint that it's extracting that information from.
09:47That endpoint responds with the JSON that contains the answer.
09:51If you just read the HTML when you load it, the answer is not in the HTML, and so your your chatbot doesn't have access to that, either.
10:00So, okay, you go, "Fine, these are just details.
10:02I just have to add some weights and sleeps, and sure."
10:05Okay, fair enough.
10:06I will take you to another example.
10:09Um I ask you the question on a product webpage, "Is the 25-mm osmium cube in stock or out of stock?"
10:15You're on a on this product website.
10:18Um you scroll down, there is a drop down.
10:21Uh that drop down is telling you and I as humans, you know, three things are sold out, one thing is in stock, even though it doesn't actually say in stock.
10:28Like, there's no text there that says in stock, but you understand that grayed out means sold out, and you know, sometimes the sold out won't actually be there as a text, sometimes it'll just be unclickable grayed out.
10:40Okay, surely this information must be in the code somewhere.
10:43You go and read the HTML, and it turns out there is an option selector.
10:47It actually doesn't say any of the things that I'm seeing on screen.
10:50It doesn't say sold out.
10:53It doesn't say available.
10:54So, what's going on?
10:56Uh turns out behind the scenes, uh the browser makes a call, gets a JSON object, which is the variable, which is product count.
11:05That contains a variable called quantity.
11:07That quantity is, you know, 10 sometimes, zero sometimes.
11:10That's just, you know, how many things can the back end support right now.
11:15Some of those quantities are zero, and there's a different rendering script that, anytime they're zero, grays it out and makes it unclickable.
11:23Fundamentally, what is happening here is this information that you are seeing on screen is not written somewhere as pure text.
11:33It is calculated.
11:35It is rendered.
11:36Um and for people who work in the browser industry, they understand this.
11:40But, you know, often people who are coming from an AI background like me, uh we didn't always understand this.
11:46The browser is a rendering engine.
11:49You are seeing pixels on screen.
11:51It's Think of it as a game engine.
11:54Um and you're asking, "Can I not read the source code of the game and predict exactly what the pixels are going to be?"
12:00Well, yes, eventually, but right now you're asking for an exact inversion of that process.
12:08Fundamentally, the web was built for human eyes.
12:12Pixels are the source of the truth because the consumers of the websites are humans.
12:17That is what it was built for.
12:20And so, there is a implication here that we have to we have to grapple with, which is machines will need vision to operate those things because the web was was built for for human consumption.
12:31Um in a way, this is the bitter lesson uh for web agents that the more you end up writing scaffolds around existing websites, the it doesn't actually generalize to the long tail of the web.
12:44The thing that generalizes is the thing that it was designed for, which is the most general solution, just pixels in.
12:52[snorts]
12:53This is what um you know, we and some others in the in the area have been working on.
12:58Um we have a model called Navigator.
13:01Um the first version of the model went out in November last year.
13:05The first version acted purely like a human.
13:07Screenshot in, button clicks and scrolls out.
13:11Um I will tell you in a second that is not where you should settle on, but it is a general solution.
13:16It lets you do things like this.
13:18Um I give you an e-commerce website, and I tell you there is this discount code.
13:23Please go The discount code is applicable under certain constraints.
13:27Maybe it's only on a product, maybe it's only on a certain set of dates, maybe it's only um with this minimum cart threshold.
13:33I describe that in natural language and I tell you, "Tell me if this discount code is valid or not."
13:38There is no API for this.
13:40The reason The way to do it is just like a human can, you go to that website, you find the product that is described, you add it to cart, you apply the discount code, and you check whether the claim of 22%
13:52off was met or not.
13:54And that is, you know, screenshot in, button click out.
13:59This trajectory took, you know, 20, 30, 40 steps, depending on the sophistication of the task.
14:05If you can do it on your browser, uh this model can accomplish it in principle.
14:10In practice, of course, there are accuracy gaps and so on, but in principle, this task is solvable, whereas in a lot of earlier cases, even in principle, that task may not be solvable.
14:21[snorts]
14:22So, my claim is the web was built for human eyes, machines will need vision, but of course, they do not need to be limited to human ways.
14:29Just because for the long tail, you need to have a capability, it does not mean that is the only way you should do it.
14:36Um here is an example showing that.
14:38The next version of the model that we trained can also write JavaScript on demand.
14:43So, this is a Chrome extension.
14:45On the right, you see an action that says execute JS, value default, text select.
14:50On the left, you saw that the model filled out multiple form fields simultaneously.
14:55The reason why it could do that is because it wrote a little bit of a function.
14:58It can read the code when necessary, it can write code when necessary, because the browser, after all, is an engine that can execute code, but it has a sort of formal verification system built in.
15:11It It is seeing the screenshot, that is the source of the truth, so it knows whether it succeeded or not.
15:16So, click buttons when you have to, write code when you have to, and look at the result uh through pixels, because that is the that is the source of of truth.
15:25[snorts]
15:26And of course because these things are are machines, you can string them into multi-agent systems.
15:31So, you can have an orchestrator that is launching multiple navigators in parallel each with a cloud cloud sandbox instance.
15:38They are clicking buttons on multiple websites.
15:40So, you can accomplish things that would be superhuman because no human would be able to paralyze over that many instances.
15:48Um around here usually, you know, in this flow of an argument is when people start asking are computer use models actually good enough for these tasks?
15:58There's actually a perception online that it's not clear whether progress on computer use has been fast and there are questions about why has progress been slow?
16:08That's not the reality I'm seeing and that's not the reality that the numbers back up.
16:14This is a popular benchmark online mind to web.
16:18No benchmark is perfect.
16:19The point isn't that this is the the right solution, but on the x-axis are release times of different models.
16:27On the y-axis is performance which is human eval on on this.
16:31So, a human went in, looked at the trajectory, decided whether it was correct or not.
16:36And basically this particular version of the benchmark is saturated.
16:39The last the model that we just released navigator N1.5 is sitting at 97% human eval.
16:45Eight trajectories out of 300 are incorrect.
16:48At this point of time, you should just retire the benchmark, build something harder.
16:51There's about, you know, 30 to 50 steps of interaction that are happening.
16:55The next step is to go for something larger.
16:57So, at least in numbers what I'm seeing, we're seeing steady progress in computer use agents becoming being able to do more and more things.
17:05And this is the model that I showed pixels in button clicks and code out.
17:09[snorts]
17:10Around this time usually is when people start asking questions, okay, so they're getting better, but aren't these things slow?
17:18Cuz after all, you're looking at a screen, you're you're clicking a button, there are lots of buttons to click, and you know, these things are expensive if you're running them for hundreds of of things.
17:28Um that claim I think is largely true.
17:31There is some truth to it.
17:33Um but I think people forget how how much you can optimize these things out.
17:39Um so this is our results compared to the frontier models, Opus 4.7, GPT 5.5, on a couple of Brazilians benchmarks.
17:48Um we're slightly better, but I think that's within statistical uh noise in terms of accuracy.
17:54That improvement I wouldn't beat the drum on.
17:57What I would emphasize is latency per step and cost per task.
18:01In terms of latency, this is a smaller footprint model, that's why it's a lot faster than some of the trillion parameter plus models, and there are corresponding cost savings.
18:10So if you have something like on these data sets, something like 20, 30 steps of interaction, you're looking at 80 cents per task versus $2.30, and that makes a big difference.
18:21And so the models are getting cheaper uh in that sense, and you can launch them at at scale.
18:27Um so hopefully at a at a high level, I've sort of convinced you that there is something off with this argument.
18:36You know, this is where where what my goal was, this is where I started.
18:39AI agents are going to be the dry primary drivers of action.
18:44Um that seems sort of an uncontestable statement because, you know, the underlying intelligence of the models is becoming larger and larger.
18:53Um there is a certain gain of productivity and efficiency and, you know, just ease of life that you get, so it makes sense.
18:59But I think this hypothesis that suddenly overnight 30 years of infrastructure that was built layer upon layer for human consumption will, in what, two, 5, 10 years be reinvented
19:15is I think a a fantasy.
19:18Um But this is ultimately what you want, right?
19:21Ultimately, you want an endpoint that um you know, some higher-level entity can go to and I say, I want you to do X. There is some task.
19:32Maybe I give you uh that task description in natural language.
19:35Maybe I have some programmatic description with parameters.
19:39And there's you know, some endpoint I should be able to do it.
19:41So, how will the web be agentified?
19:44My answer is just extrapolate this trend.
19:46This is where we stand today.
19:48Accuracies are getting higher.
19:50Benchmarks are falling.
19:52Latencies are getting smaller.
19:53Costs are falling.
19:55And that trend, I think, will continue.
19:57And the way we will agentify the web and in in in some sense, this is depressing, but this is true.
20:04We will just pile on another layer of mess on top of the mess that the web is.
20:10Which Which is to say, you know, you will issue a task.
20:14Behind the scenes, there will be hundreds of browsers that are pretending and clicking buttons just like humans would.
20:21Um And they will give you a result in a structured format.
20:25It will cost less than a penny.
20:27It will run in your browser sometimes.
20:29It will, you know, run in less than 100 milliseconds.
20:33And at some point, you will say, "Yeah, that's an API.
20:35Like, why do I care?"
20:37And that's how we'll we'll agentify the web.
20:39So, thank you.
20:41[applause]