How to Build Financial Agents That Work Across the Web - Jay Sahnan & Arik Bird, Browserbase

AI Engineer · 19 min · 295 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hello.
00:13Hello everyone.
00:15We are here to talk about how to build financial agents that work across the entire web.
00:20I hope you've been having a great day so far in New York.
00:25I am Jay.
00:26I'm Amarik and I think the place to start is what even is a browser agent, right?
00:33So if we think about it and first if we want to go back to the previous slide, quick show of hands, who is building agents today?
00:39Show of hands.
00:41Okay, we're in the right place.
00:42And who works on like finance operations?
00:45I heard about some conference going on today.
00:47Anyone in finance?
00:49Okay, couple more engineers.
00:51If you also want to ask us any questions throughout, you can scan this QR code and you'll be able to ask away.
01:00Cool.
01:00Back to the show.
01:01So what is a browser agent?
01:03An a browser agent if you think about it, if an agent needs to access the web, interact with the web, read information from the web, take action on the web,
01:12it needs a browser.
01:13Whether it is booking a plane ticket, booking a flight, planning a date night for you and your wife, getting some pricing information on a competitor, it typically needs a browser to be able to access those hardest to reach sites that don't have an API or don't have an MCP.
01:28And so in a nutshell, anything that you need to do on the web, your agent should have the tool called the browser.
01:37Show of hands, who knows these three companies?
01:40Ramp, Bill, Stripe.
01:41They familiar with the crowd?
01:42Cool.
01:43When you think about what these three companies have in common, well first they're all financial institutions, they're dealing with highly sensitive information, but when you think about it for like Ramp, Bill, and Stripe, a lot of this information
01:55lives on the hardest to reach parts of the web, whether that is behind a portal where you need to log in, whether it's on a website that is heavily protected by antibot.
02:05It is extremely challenging to get access to this information, and the answer is using browser agents.
02:13So, if we want to start off, we'll go with Ramp first.
02:16So, Ramp, I'm sure everyone in this room is familiar with Ramp.
02:19We actually are big users and fans of Ramp.
02:22Uh they make it incredibly easy to for a company to manage spend.
02:26When they were growing and scaling, they started having to interact with hardest to reach portals, and more portals than than ever before.
02:34And so, the problem or the gap was, how can we fetch these receipts efficiently at scale?
02:41They didn't know about the browser agent.
02:43So, instead, they're like, let's just put together a team of humans, and they will log onto these portals, and they'll go fetch these receipts.
02:49Well, that can that can be good for a series A, series B company, but when you start growing and scaling to the to the masses that uh Ramp is at, you need a more efficient way.
02:59You can't continue to hire humans.
03:01You're just going to be bottlenecked by the limitation of scale.
03:05So, in comes browser agents.
03:08And now, again, we are big users of Ramp.
03:11I booked my flight to this conference using my Ramp card.
03:15Immediately after booking my flight on that Ramp card, Ramp spins up a browser agent.
03:19That browser agent goes to Delta.
03:21It logs in.
03:22It captures that receipt, and it matches it to the transaction.
03:26Um so, once they do that, that's that's all good and dandy, but when you're doing it at scale, you're going to run into things like antibot.
03:34So, how do you reduce that friction?
03:37How do you eliminate getting blocked?
03:40And to understand this, I really want to go back to how does the browser work on your laptop?
03:47Think about Chromium or Safari or dear.
03:52Chromium allows you to access the web.
03:54It's an executable binary.
03:56And right now from this slide we can see that we are booking a flight to New York.
04:02And in the background, your browser has an entire fingerprint.
04:10It knows your browser version, uh there's OS, there's fonts, there's screen size, and even time zone.
04:16The website also sees a familiar visitor.
04:18Maybe you visit the site before, maybe you've got some cookies.
04:21You've logged in or you've booked too many trips on the same account.
04:26There's a familiar pattern as well.
04:28Each request will go through a net You've got a network reputation.
04:33And all the navigation, you are clearly a human.
04:37And now the problem comes when you try and put that same version of Chromium into a data center.
04:45And that's when we scale beyond the laptop.
04:48We need to replicate it.
04:50For companies like Ramp, they are not going to have thousands of laptops open running a different human version of Chromium.
04:56They're not going to have one person with a cron job running this either.
05:00They need scaled infra that lets them get to sites reliably.
05:05The problems start to appear though when you put that same Chromium in the data center.
05:10Um let's say we do this.
05:13Let's say we've put that in.
05:15The problem now is when we launch a called out browser, that headless browser in an an an EC2 instance somewhere.
05:22Every single time we launch it, it's going to have a different fingerprint.
05:26It's going to have fresh cookies.
05:27You may need to log in again.
05:29And proxies.
05:31It's going to clear indicator that you are coming from a data center instead of a residential IP.
05:39And sure, you could trick the website by using a proxy service and changing your proxy.
05:47And that means is you might not be presented as many captures or you can fake your location from that data center, but that still doesn't change the underlying problems with Chromium.
05:59You're using that same Chromium on a laptop.
06:03And if we just step one step down and go a little bit deeper and think about what makes Chromium so good is if you look at the browser process, right?
06:13Inside you have a page engine.
06:14This is a very simplified version.
06:17And where it renders the content and it has all the HTML, JavaScript.
06:22Um then you have your network and if you ever like inspected before you go on a network tab and you can see all the requests coming in and out.
06:29And you have your device and graphics.
06:32The engine is so great that they can make a change every 2 weeks, which makes it terrible for us as a company trying to automate the web.
06:39So we forked it.
06:41And I could go on forever on how many changes we made to the Chromium binary.
06:46Um but we forked it and came up with a verified browser that we run in the data center.
06:52And what we did with three things I'm going to share now is we first removed the automation flag.
06:56An obvious one, right?
06:57When you're driving a headless browser um through an automation, you will pass this webdriver equals to true and we just swap that to false everywhere.
07:08Then we aligned on the identity.
07:10Instead of now being called headless Chrome, we're calling it Chrome.
07:14Now these may seem like simple fixes, but because how complicated Chromium is, this goes deep into the code and it's in so many different places that we need to find.
07:24And every 2 weeks is a new change.
07:25That's why we have our own version.
07:27We also [clears throat] keep the process alive.
07:28What I mean by that is a lot of scraping jobs or automation jobs or workflows that people are doing as developers locally or even in the hosted infra will start and stop.
07:39They kill the browser and they kill a Chromium instantly as soon as the task is done.
07:43Instead, we leave chromium running and close the windows out.
07:49So, why don't we put it to the test?
07:52We've got a snippet here and I'm going to ask Aric, can we launch a browser?
07:58Okay.
08:00And bear in mind the Wi-Fi has been terrible today.
08:03So, it's launch is going to take a second to render and we are now looking at an iframe of a live view of a hosted browser on BrowserStack.
08:12That's great.
08:12We can go to Rome.
08:13Okay, nothing special.
08:16Let's stop that.
08:18And let's go to the next slide.
08:20Why don't we launch 100 browsers?
08:22Do the exact same thing but we'll launch them concurrently.
08:26Okay.
08:28And to the next slide.
08:30Um now what's going to happen in the background is I'm calling the same thing with a bunch of different websites and you will be able to Aric scroll through
08:39the um some of the sites and we're not automating at the moment.
08:43We're just going and visiting the sites as 100 browsers.
08:46Now I'm going to try something for the next minute.
08:48Um I set this up very quickly and the Wi-Fi is a little sucks a little bit.
08:53And it's all through an Engrok portal on my laptop.
08:56So, if it doesn't work, uh we'll move on.
08:58Um I want everyone to scan this QR code and you will be given a random website out of one of these 100.
09:06Um someone tell me if they can see a browser.
09:09I'm trying to prove to you that these browsers in our cloud and you have access.
09:16Someone shout out when they're when it's loaded.
09:17Hopefully the demo gods are with us.
09:22Yeah, you got a browser?
09:23Awesome.
09:25Yeah?
09:26Got another one?
09:27Great.
09:28Cool.
09:29Sweet.
09:30Nice.
09:31You can actually interact with this as well.
09:33Um through the iframe you can click, control, move and do things.
09:37And that takes me on to the next thing.
09:39I'm going to uh stop that now.
09:40Thanks, Eric.
09:41Um Um let's take me on to the next thing is control.
09:45How do we control these browsers?
09:48And just to sum this up, Ramp now can deploy thousands of browser agents.
09:53They have saved north of 30,000 manual hours in data entry that is done that used to be done by humans that is now done by agents, and their scale has gone up 5x.
10:03So, just by switching from humans to agents, they were able to accelerate their business and save a bunch of those hours, which now they can dedicate to other initiatives across Ramp.
10:13When we talk about Bill, so Ramp scale and access.
10:16Now, it's about control.
10:17Everyone's familiar with Bill.
10:19Um Bill, you know, processes a ton of payments across thousands of different vendors.
10:24When you think about using agents to do that, the biggest challenge is well, every portal Next slide, please.
10:31Every portal is different.
10:33The checkout flow is different.
10:34The process to fill out the payment is different.
10:37The UI changes.
10:39So, in reality, the way Bill thinks about this is it's impossible to automate with agents.
10:44That's why we need a team of humans to go click through portals and do this on behalf of our customers.
10:50Um and again, the answer to this is an agent that can use a browser.
10:55Um and so, for Bill, the notion was how can we build a single agent that can interact and act upon thousands of different portals.
11:04One agent, thousands of different portals.
11:08Um that brings us to Stage Hand.
11:10Stage Hand is our open-source SDK that lets you control a browser.
11:16Think about Playwright, but built way on top of it.
11:20[snorts]
11:20Your agent will sit and use Stage Hand to control the browser.
11:23I'm going to show you a little bit how.
11:25Go to the next slide.
11:28I want to start with understanding again what is wrong with Playwright.
11:31Playwright's great.
11:32I first learned how to code by using Selenium and beautiful soup and writing Playwright scripts by hand.
11:39This was before AI.
11:41Now, I can't remember last time I coded.
11:44And so, with Playwright, what the problem is is as soon as that website changes, the button moves, the text box changes like from first name to just name.
11:57In the in the back, like when you're looking for that XPath in Playwright or you're trying to click on an element or a selector, your script's also going to break.
12:05And now, that's fine if you're doing one-off tasks and you're running everything locally, but when you try to scale like a company like Ramp or Bill, you need reliability every single time.
12:16The reason we like built Stage Hand was because we wanted to give the ability to agents to navigate the web and control them.
12:23Next slide.
12:25So, how does it actually know?
12:26What's different?
12:28Well, if you play play Sorry.
12:31What's different?
12:32So, Stage Hand actually works a little bit like Playwright in some sense.
12:36You can use deterministic rules and go page.click or everything you'd see in a Playwright script.
12:44But, when it gets a little bit more complex, you can actually send natural language prompts to go and do actions on the site.
12:51Pay page.click or page.act, go fill out this form, or enter my name, or enter password, find the red button.
13:03Stage Hand will then be able to take the DOM and accessibility tree and everything it sees on the website, pass it to an LM, pass it to an agent,
13:10and understand where to click and go and click it.
13:14Now, you may be thinking it might get expensive, but that's where we have our excellent engineers behind Stage Hand.
13:23And to demonstrate this quickly, Eric, if you could hit that compare button.
13:27If you were to use an agent with playwright, you'd be sending everything on the left.
13:33Uh, that's 122k tokens.
13:35And that's a raw HTML.
13:37And you ask your agent, "Where do I click?
13:39Where do I go to?"
13:40What Stage Hand does before it hits that agent is it shrinks the DOM to a really cool accessibility tree, where we're only sending like 4,000 tokens for this same example.
13:49And it'll give direct selectors.
13:51So, Stage Hand will go, "Click that."
13:53Save a lot of money on um the amount of tokens that you're sending, but also reduce the context and get quicker results.
14:00And if we go next, another reason why we use um Stage Hand and with Browser Base is because there's fewer net There's fewer round trips.
14:11Let's say we are currently controlling a browser in a data center.
14:13We've got to talk back.
14:14We send the HTML back and forth every single time.
14:17What we do at Browser Base is we put Stage Hand next to the browser.
14:22And that What that means is Stage Hand is a Chrome extension on a hosted browser.
14:27So, those actions become a whole lot quicker, and there's no round trip latency from those actions and those like instructions from the agent of where to click and where to go next.
14:37And so, when you bring these two things together, you need scale from the browsers, Stage Hand gives you the control, and we're going to show you an example that use both.
14:52This is our last example.
14:53So, with Ramp, we saw that browser agents solve the issue of scale and access to heavily protected sites.
15:01With Bill, it was adaptive automation.
15:03How can browser agents enable you to interact with thousands of of different portals with ha- without having to manage thousands of different playwright scripts.
15:13Now, with Stripe, everyone's familiar with Stripe in this room.
15:15Stripe onboards thousands, tens of thousands of merchants on a monthly, quarterly, quarterly, yearly basis.
15:22And again, they knew that the only way to access the information to verify that these businesses were real and they were the right business to work with is through a browser.
15:32And this was before the age of browser agents.
15:34So, they had a team of humans going on and going to Secretary of State portals and using the browser to get that information.
15:41So, a quick switch is being able to use the browser agent to go and automate this KYC verification.
15:51Um let's dig a bit more into Stage Hand here.
15:55Um so, we have instead of using a direct agent like Bill, you could use act, extract, and observe.
16:01Those
16:02[snorts]
16:02are three key primitives that Stage Hand we give we expose to you as developer.
16:08Um one is observe, find the learn more button.
16:12Um this is taking action.
16:13Next slide.
16:14Act, click the login button.
16:16Next slide.
16:17And then the extract.
16:19Now, this is my favorite one because you can extract anything into a Zod schema, which is great for engineers because we're taking really messy unstructured data on the web
16:27and converting it to a really easy-to-use data and readable and send it to an agent.
16:33Next slide.
16:35Now, I'm going to show a couple more demos.
16:37We're going to start with one company website and we're going to go to Chipotle and find out what it sells using Stage Hand.
16:43Um and next slide then.
16:45Okay.
16:45So, this browser will now spin up on a hosted browser and browse page and it's going to use Stage Hand um extract to extract it.
16:58And there we go.
16:58We've got Chipotle Mexican Grill.
17:00Um it's selling some lime chips.
17:02Cool.
17:03Uh if we press stop.
17:06Now, this is where it's going to get fun.
17:07Let's go for the next one.
17:09And now we're going to research four different sources for Chipotle.
17:13We're going to go to their website, investor page, SEC filing, and New York Stock Exchange listing.
17:19Okay?
17:20So, we're running and all right, you can also scroll once they load.
17:28Okay, great.
17:28So, we've got the SEC filing down at the bottom, and we've got New York Stock Exchange, and up top I think we've got the website that will appear in a second.
17:36But, there we go, they got investor relations, as well.
17:39So, these are actually happening quicker than they are on the screen.
17:41This is already being happened in in the back.
17:43It's just the live view that's streaming back to us.
17:46And obviously, when you're running so many browsers at scale, you're not always going to be checking the live view.
17:49This is more for development.
17:51Okay, cool.
17:52Let's stop that.
17:54And let's go to the next one.
17:56Now, we're going to do 20 companies for the same three sources.
18:00And we're going to hit run.
18:02And bear in mind, this is the same code that's running on every single website.
18:06Then, we go to the next slide.
18:07And we can see Chipotle will come up at the top.
18:10I'm getting hungry now.
18:12Um JP Morgan.
18:14Um there we go.
18:15All right, keep on scrolling.
18:16Let's see what else is there.
18:17Golden Sachs.
18:20Going and doing the same tasks across every single site without interruption, unblocked, all using Stage Hand as well.
18:30And it's really cool cuz it's a verified browser that I mentioned at the very start, which means that you don't get hit with as many CAPTCHAs.
18:36And if you do, we solve them.
18:38And the Stage Hand allows us to navigate without anyone looking at this.
18:42Okay?
18:43And we can stop there.
18:44And I think we can see some results on the right-hand side.
18:46Um next slide.
18:49Okay, cool.
18:49We've got some of the results here.
18:51Um SEC filing data.
18:52Keep on scrolling.
18:57I honestly have no idea what the SEC stuff means.
19:00I'm not a finance guy.
19:02But, hopefully it's useful.
19:03Cool.
19:04Um let's go to the next slide.
19:06And so, that SEC template I was something I built a while ago.
19:10If you want access to that particular, just the SEC one, you can do.
19:14Um, but we also have if you want to get started with anything browser-based or anything I showed, we have if you go to our website, we have a copy prompt
19:22which you just send it to your agent and that'll do a lot of the work as well.
19:24Uh, next slide.
19:26Okay.
19:26Thank you so much.
19:27That's our talk.
19:29[applause]