Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase

AI Engineer · 51 min · 528 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00All right.
00:13Hello.
00:14Thank you everybody for being here.
00:16Uh I know it's the last workshop of the day.
00:18I appreciate all of you turning up here.
00:20U I am show uh I'm here to talk about hill climbing skills.
00:26uh specifically how can you improve your agents without really going into the models without really going in the weights.
00:34Uh I have a slido uh up here as well.
00:37I'll just uh flash the QR code in case any of you want to uh put in any questions and we can address them towards the end as well.
00:45U a little bit about me.
00:47I'm Sh.
00:48I'm uh on the sales engineering team at browserbase.
00:51Uh I essentially work with our customers deploying browser agents in production.
00:58Uh I'm not sure maybe we'll do a show of hands.
01:01Uh how many of you know what browser agents are?
01:06Anybody?
01:07All right, a few.
01:08Amazing.
01:09Uh have any of you built browser agents before?
01:13Have any of you used these tools?
01:15Show hands.
01:16Okay, a few of them.
01:18Amazing.
01:19Nice.
01:20uh maybe I'll I'll talk a bit about what browser agents are before diving into the whole concept of hill climbing for skills uh so that it becomes clear what we're doing towards the end of the talk we'll also look into
01:33how you can do this in your domain in your uh field as well.
01:39All right.
01:39Uh so quick primer on what browser agents are.
01:45If you have an agent that needs to drive a browser, you basically call that a browser agent, right?
01:51Browser becomes the tool to your agent.
01:53And how do you control it?
01:55You control it with some sort of CLI, some sort of MCP, some sort of method where you can control the browser and say, "Hey, go and click on this red button.
02:05Go and click on the refresh button.
02:07Go and uh take a screenshot or take a snapshot of the accessibility tree of this website."
02:14Right?
02:14So these are this low-level tools that your agent can have access to to control the browser.
02:20Uh and browserbase is essentially a platform for you to build and run these browser agents.
02:27We provide the infrastructure and the SDKs, the skills, the CLI that go with building these browser agents.
02:35Uh and and we're going to be using all of these tools to see what hill climbing does look like.
02:41All right.
02:42Uh I want to talk a bit about the biggest problem with agents today essentially reliability right uh people are frustrated with browser agents getting stuck or getting into an anti-border
02:58capture issue uh or maybe just forgetting or being lazy and not completing the task.
03:04Uh and we see reliability essentially is an engineering problem.
03:08I think the recent models the Opus and the Fable when we when it came out for those three days it was really really smart right uh but we see that a lot of the
03:18work that has to be done for reliability is the hardness around it is the tools that you give it it is the skills that you give here u and and that's what we're going to be sort of looking at how do we improve the reliability
03:31of these browser agents I wanted to give you all a few examples of like where these agents suck today, right?
03:38If I zoom in a bit, you'll see this browser agent is trying to like go on Google flights and click on the uh it's trying to drag the time filter and it's unable to figure out how do I drag that.
03:52Dragging is a tricky operation for these agents to do and it's getting super confused.
03:56It doesn't know what the HTML element for that is, what the JavaScript for that is.
04:00It's not really able to do this and it gets stuck and it's not able to book my flight tickets.
04:05Uh, another example I wanted to show uh is a browser agent on open table.
04:11Let's say I have my open claw or my Hermes agent and I need to book a date night and I tell hey go to open table and and find me some and and look at the anti-bot that it runs into, right?
04:22Uh, and it's just like access denied.
04:24I don't know what happens here.
04:26Uh, but like can my browser agent get around these issues?
04:29Can they be more reliable?
04:30And can they book my date night restaurants is the biggest question I wanted to ask here.
04:35uh and this is where we sort of look at different ideas to improve the reliability of these agents and one of them uh oh actually let's let's dig into the root cause here right like why this is happening
04:51u we have like these opus models these five five the super nice general purpose you might have used codeex computer use very nice but when you want to sort of do
05:01uh longunning tricky websites like the open tables or the Google flights it sort of becomes very difficult to run these in production and these sightspecific tricks or these javascript that you need to run on your browser
05:16the agent doesn't know about them right and the most important thing is it's every run starts from zero you're not really having any sort of learning baked into your runs I think Darkh talked about this a couple days ago it's like why is progress on computer use still so slow today.
05:34Uh although computer use seems like a verifiable problem, why is it still not solved?
05:40Essentially, there's no learning happening.
05:42Every agent run is a start from scratch run and it's not learning from its production traces.
05:49U and these browser agents, I call them geniuses that never learn.
05:53Um and some people might say, oh, why don't you fine-tune on something some like open table or restaurant websites.
06:00Uh but it it's not really scalable, right?
06:03uh if you're thinking of building agents, you can't really be fine-tuning for every new task that comes up.
06:07You want some you want a a modality where it can learn from its production traces uh and get better over time.
06:17This is where we sort of started thinking about and this was like six four to six months ago where Karpadi came up with this idea of auto research.
06:25This is essentially a loop, right?
06:27Uh and this loop sort of takes a verifiable problem statement and goes at it, tries it again and again, looks at its own traces and tries to improve itself.
06:40And we wondered how can we apply this to browser agents?
06:43How can we apply this uh to these complicated workflows, these complicated websites which are not deterministic, they change every day, they have different captures every time.
06:53Uh how can we learn from these?
06:55Right?
06:56Uh this is where I want to sort of switch into a demo and sort of show this live.
07:02Uh and I'm going to start with uh a prompt over here.
07:09So I want to maybe do something like uh go to Google flights and uh find me a business class flight from SF to New York for this Saturday.
07:22I want to have one carry-on bag.
07:25Uh and this flight should be in the morning.
07:29I hope my Okay, this is correct.
07:35Just paste this and run this.
07:36Right.
07:37So starting a browser based session here.
07:40Uh so what you see in the center is a remote browser has just spun up and has navigated to Google flights.
07:47If I expand my agent here, uh it's doing all of these things.
07:53This is the reasoning traces that you can see live and it's trying to sort of apply the business class filter.
07:59So it it'll do its job and and and we'll come back to this.
08:02Uh but I wanted to talk about what's really happening here.
08:05Uh u and the architecture behind this auto research loop.
08:15So we we fired the Google flights task.
08:18Uh and and and the architecture here is this auto browse loop that we call this.
08:24We took auto research and we took browsing.
08:26We call it auto browse.
08:27And this loop essentially is two clawed agent SDK loop uh agents.
08:32Uh you can call one as the doer agent and one as the teacher agent.
08:36This doer agent its job is to actually go and do the browsing.
08:40It has access to the browse CLI that we talked about.
08:43It has access to the browser uh and it goes and sort of clicks on buttons, looks at the screenshot, tries to get me information and the teacher teacher's job is to look at the traces of the doer and learn what went wrong and what went right
08:58and they keep working together until they find a very good solution that converges.
09:05Right?
09:06Uh we don't tell it how many loops you need to go for.
09:10We don't tell it what's the success factor success uh rate that we want.
09:15We don't tell it hey we need to do this within x number of tokens or x number of seconds.
09:21We just give it a broad prompt and we let the bitter lesson we embrace the bitter lesson and we're like okay let the doer and teacher go at it.
09:28Uh and the key thing that happens is there is a strategy.md file.
09:34uh that is essentially the artifact across multiple runs.
09:39After every iteration, the teacher thinks about what happened, looks at the screenshot, looks at the logs, and then thinks, "Oh, you did not click on this button.
09:49U you should actually click on this button instead, and it puts that in the strategy.mmd file."
09:54And that artifact carries over to the next iteration, and it improves from there.
09:59Maybe we'll go back to our demo and see what happened.
10:02All right.
10:03So, iteration number one, uh, there was a failure here.
10:07Uh, and the learning from this iteration was skip the entire form calendar widget because you can just do a query parameter and you don't really need to click around, right?
10:18The agent was smart enough in this instance to think, oh, I'm wasting so many tokens and I'm wasting so much time clicking around where I could just use Google Flight's query parameter.
10:29Now this is obvious in hindsight but maybe as a human I wouldn't have thought of this and I wouldn't have written this complicated query parameter myself right uh so this is the learning from the first iteration now it's going and doing things in iteration number two it'll keep going on till it feels like it has maxed out
10:49all the tricks it has maxed out all JavaScript hacks that it could do uh and and and then uh think about okay this is the skill that I can use from here.
11:03I think the key the key thing uh that I wanted to also talk about is at the end of this converged loop the output that I want is a skill.mmd
11:15file.
11:15This skill.md file would be what I give to my future agents.
11:20Uh and it would have a collection of all the hacks or all the tricks or all the learnings from previous runs.
11:27We'll we'll look at what that scale might look like.
11:30Uh it could look like, hey, a pre-filled URL is mandatory.
11:34Don't go around and waste your clicks and tokens clicking on these buttons.
11:40There is a reopen bug in the calendar.
11:43Don't do this.
11:44Uh if you want to move the drag uh if you want to drag your time stamp thing, uh use JavaScript injection.
11:51Don't try to do it with the mouse.
11:53Right?
11:54These are the kind of tricks that are now persisted for every future run and gives me so many more savings uh from a performance, reliability and cost perspective.
12:07Now uh we'll go back to our uh run and see if it has passed.
12:13Okay, so iteration number two has passed and if you can see uh the savings in in cost are like 4x.
12:21Uh this chart is being created on the fly.
12:24Uh so it started from $12 and the second one was $3.
12:27And keep in mind my agents here are Opus 48 both for the doer and uh for the uh teacher.
12:37Now iteration three just came in.
12:40Uh the number of turns decreased.
12:42The latency is also better.
12:44Uh but there was a failure in iteration number three.
12:47there's something that it sort of overcorrected on and iteration number four should probably be learning from that and we'll see if it does that.
12:55Uh while it's doing its own thing, I wanted to show some previous runs that I did uh just in preparation for this uh demo.
13:03U so for example, I went to uh auto browse and I was like hey go to the AI engineers website and tell me which talks and workshops are talking about hill climbing,
13:12right?
13:13uh and it did that across multiple iterations and you can see the replay of each of these.
13:17These are replays that are coming from browserbase super helpful important for uh observability perspective and as you can see uh it sort of uh got better in ter in terms of cost 7.4x 4x
13:31uh even though it was able to pass in all the iterations, I still got improvements in cost.
13:38Now I wanted to actually see what are the learnings from this skill, right?
13:42Uh how do you sort of uh persist all of these learnings?
13:46It says go straight to the shed schedule page.
13:49If there are open errors, don't try retry blindly.
13:53Run this.
13:54Uh type the keyword into the session search input.
13:58Wait briefly.
13:58So like have these weights because sometimes websites take a lot of time to load.
14:02So you you can have these kind of tricks uh and then read the results.
14:06Uh you know like there there's a lot of learning that is happening in these skills.
14:11Maybe I wanted to show some interesting ones.
14:14I wanted to show this open table one because I started with this blocker of like access denied when we started this demo.
14:22Uh and if we go to the uh screenshot, this was the access denied thing that we got right in in iteration number one.
14:30Let's see how it learned from there.
14:32Uh it realized that the uh block is environmental because the website doesn't want this agent to do something.
14:40uh in iteration number two, it figured that if I do a query parameter and if I put a latitude longitude metro ID, I automatically get that uh UI that works.
14:52Do not open /r that's blocked and do not reload / s. These are the places where the blocks kick in.
15:00If you don't do that and you follow this technique instead, your agent will work, right?
15:05Uh and as you can see, uh it sort of went from $6.22 22 to 0.51.
15:12The time taken went from $189 seconds to 27 seconds.
15:17Uh and the scale it came up with was here.
15:19Uh build the search URL, do the date math like this.
15:25Uh this is the known metro pins for San Francisco.
15:29Uh and and follow these steps, right?
15:31Never call / r because that will return access denied.
15:35Now these are the kind of learnings where a human could never have done but you just sort of embrace the bitter lesson and a let agents do the job uh and they will find these uh tricks for you.
15:48I want to uh also show uh a few examples of like the kind of improvements we see.
15:54Uh the reason why this problem is so close to my heart is because I from a sales perspective am deploying these browser agents into production.
16:04uh the sale is only done when the browser agent is successful.
16:09So reliability is like the most important thing for me and this is what I this is the key tool that I take to my customers and I'm like just fire your agents
16:18let them do this auto research thing and your agents will be successful in production.
16:22Uh and these are the kind of u success metrics that people are looking for up to 16x cheaper 12x cheaper 3x cheaper uh on different tasks.
16:32Of course, every task is different.
16:34The bottlenecks on each task are different.
16:35So, uh you might see variance there.
16:38But the idea is you can hill climb on these uh techniques.
16:46I wanted to go back to our live demo and see if there's any uh improvement.
16:53All right.
16:53So, we have a graduated skill u and the skill applied end to end.
16:58No further learning.
16:59The results are showing correctly.
17:00one-way business class, San Francisco to New York, July the 4th, one carry-on bag included.
17:06Uh, and there is 8 a.m. $649.
17:09Right?
17:09So, it was able to get all of this and I can sort of watch the entire replay to see if that is accurate.
17:15Uh, it was able to sort of figure out this entire skill and the fast path is encode the whole query in the URL.
17:22strongly preferred the again like I know I'm I'm showing these things but when I first started looking at the the results here this was like my moment of like this is the mythos moment for
17:35browser agents you know like just reverse engineering the hell out of things uh I I look at this with some of my customers the customers are mind-blown uh earlier they would be writing scripts
17:46for each of their websites they would be like hey I need to spend so many engineering hours on like just reversing these websites.
17:55How can I scale this to 100 a thousand websites and now our customers have actually built these entire pipelines of uh these skills that get generated on the fly pull requests that get created automatically.
18:09If there is a failure in production, an agent kicks in and starts healing the skill as well.
18:15So there's a lot of uh engineering that we can do around this to make leverage to leverage this technique.
18:22Uh I wanted to also show what we've done with this right we actually went ahead and created this product called browse.sh uh we were like if we can do this for one website we can do this for five why not do this for like 500 websites and give it for free for anybody to do this.
18:40Uh and like if you remember in the 19 or wherever like in the '9s before internet you had the yellow pages uh you had details about accessing people, businesses
18:53etc. This is yellow pages for browser agents right it has details about how to do any task on any website on the web.
19:02U I wanted to actually show this to you all.
19:05uh you can actually install browse CLI and use all of these skills completely for free.
19:13We've eaten up the token costs ourselves and we're making this available for you all.
19:19Uh so if I go to let's say search products on Amazon uh I can see the entire scale is over here.
19:26When should you use the scale?
19:28How should you look at like pricing and availability and whatever right?
19:33uh this is how you use this lot of details about Amazon the screenshots u and I just give this browse skills CLI to my openclaw and it just knows when to pull the alltrails.com
19:47skill when to pull the weather.gov skill and and this makes my life so much easier.
19:53uh we essentially are running the same auto research loop behind the scenes uh and making this uh available for you all.
20:01In in fact if you would want to you can go here and you can be like hey go and make a skill for me to uh look up this uh events in San Francisco for this week.
20:13Uh you do this generate a skill and this will sort of do this for you.
20:18This will send you an email and you can then use your skill uh in whatever agent you want to, right?
20:24Uh so we've done this for browser agents.
20:28Uh and I wanted to talk about like how can you potentially do this for any other domain, right?
20:35Uh I think coding agents is like one place where I have seen this frustration where the agent is relearning the repo conventions every time.
20:45it's not applying the linting rules correctly and I have to tell it every time and maybe you can have your cloud MD and agents MD and improve that but
20:54the the fundamental premise is like the agent should not be relearning your tasks from zero right uh I've seen this in research agents we run a bunch of like deep research agents for our internal workflows
21:09uh and every time it struggles and it tries to relearn oh I pulled this information from Wikipedia IA uh but I did not pull it from this other website.
21:20Uh I was like why not?
21:22Uh like we always do this other website you should learn from your previous traces.
21:27Uh and you should pull information.
21:29Uh internal ops you have all these dashboards all these databases all these internal tools but your agents don't know which ones work and which ones are preferred.
21:38Ideally you should be applying this all research technique.
21:42Uh a and the two conditions that are required is that a you should have a replayable environment of some sorts so that you can run these loops and b you should have some sort of verifiable outcome
21:54some sort of quant quantifiable metric for you to run these loops.
21:58Yes, I see a question there.
22:18I realized that Yeah.
22:43Yeah.
22:44Yeah.
22:45Exactly.
22:45Uh so I think the question is like how do you know that the auto research loop is accurate whether it is converging and when it says it's converging it's actually converging.
22:55When it says it's good it's actually good.
22:57When it says it's bad it's actually bad.
22:59like how do I have confidence uh on what the agent is saying and what the loop is doing.
23:05I think that's where I want to sort of talk about the verifiable outcome here.
23:09Uh for Karpathi, it was quite easy because he was looking at the log log loss of the experiment when he was training neural networks.
23:17Right?
23:17So it's very easy to see whether your log loss has gone down or not.
23:21Uh but in many real world scenarios that is not the case for browser agents.
23:25As I explained, uh, whether you book a flight ticket or not, that might be straightforward, but did you find the best flight options for me might be slightly fuzzy
23:37uh and might not be the best uh Yeah, if if I understand this correctly, you're saying like one mistake in the loop can sort of throw off the entire auto research run.
24:13Uh and how do you prevent against that?
24:15I I I have actually seen where some uh iterations actually fail.
24:22uh for example if we uh look at the previous slide uh there's like failures happening after there have been successes right uh so it's not always that once you succeed all the future iterations will always be successful but I think it's the design of your loop where you can make it resilient
24:44to u failures uh where it doesn't overindex on those uh and still sort of correct protects itself uh from any mistake that happened in previous runs.
24:55Uh I think we're trying to sort of prevent like uh what do they call like exploding sort of loss sort of scenarios.
25:03Uh I think you could do that.
25:05Uh in fact Karpathi talks about this like this is one way of doing auto research.
25:10You can actually auto research on different ways of doing auto research itself.
25:14Uh it's like is the teacher doer loop the best?
25:17Should there be a multi-agent architecture to do this or is there any other approach that might actually work better for research tasks in neural networks versus browser agents?
25:27Right?
25:27You could have different designs of your loops to prevent situations like those.
25:32Uh and I think going back to my point, as long as you have a verifiable outcome or some quantifiable metric, I think you should be able to design uh a good uh loop here.
25:45uh replayable environment I think sometimes is under discussed uh but extremely important.
25:52I think he was talking about Darkh he said having a verifiable outcome is important but also you should have uh extremely grindable environments or extremely grindable uh use cases as well.
26:05Right?
26:06If your agent cannot uh grind through failures, it will be difficult for it to uh go for multiple iterations and fix itself.
26:17Uh so do you have like coding agents is a good example because you can start from a docker image or you can start from uh a particular EC2 and and you can replay from different parts of the uh stack.
26:29Uh but maybe uh knowledge work is probably not that easy to replay.
26:35uh maybe there's like a lot of different human in the loop actions that make it very hard to replay from a particular point in time.
26:42So investing in your environment also can be super uh rewarding in these cases.
26:52Amazing.
26:53Uh I just wanted to share the uh URLs of uh the auto browse loop that I uh demoed today.
27:02So if you scan the QR you'll be able to pull the uh GitHub for that.
27:05Uh and also the second QR here uh is the skill itself.
27:10You can just do /auto browse on your claude code or codeex and be like hey /auto browse use opus48 on both of these uh teacher and doer and go create a skill for me to do this.
27:22Uh I'm looking at one of these questions.
27:28Uh and also if any of you have ideas of something we want to sort of put the agent against feel free to type it in the slido and we can actually try it live.
27:40Uh but let's look at this question.
27:41Is there a way to optimize for convergence?
27:43Uh I think this is the same question that we were just talking about.
27:46Uh high cost is interesting because you might think that u doing longer iterations might become costlier over time.
27:57Uh I have run a bunch of evals on this where I can throw a sonnet at this problem but run it for longer iterations versus throwing opus for lesser iterations.
28:08I think you can get the same job done by throwing sonnet larger number of iterations but still cheaper overall.
28:16Uh also this cost is amotized right like one auto browse loop that we probably run is like maybe $30 $40 if it's super complicated $100 at most.
28:26Uh so this is amotized as long as those costs make sense for you uh to deploy this in production uh just go for it.
28:33Uh but there's a lot of interesting things uh that you could do uh with evaling the kind of loops that you're generating.
28:40Uh yes I will share the slides as well.
28:42Uh I can do this on the GitHub itself.
28:47Uh can you share how you thought about uh auto research into auto browse?
28:54Uh I mean the process was fairly straightforward.
28:58Uh took the GitHub repo that Karpathi put out uh and then there's all this context about my customers uh that I have uh where I was like hey these customers face these kinds of reliability problems
29:12uh the way I solve it for them is I go into their office I sit with them very forward deployed uh and I try to sort of just brute force and reverse engineer this.
29:22This is what I do.
29:23So I am the teacher agent.
29:25Uh can you automate me out of the process and create a loop borrowing from uh Karpathi and and and and build the same thing?
29:34And of course there's a lot of iterations on top of that.
29:36Uh there's a lot of engineering as well uh to simplify this.
29:41Uh but yes, that was the process of converting this.
29:48Can you use this in a proxy?
29:49Can you be logged into a site?
29:50Yes.
29:51Uh again this goes back to the environment that you design.
29:54Uh browserbased the infrastructure that all of this is running on uh gives you the ability to log into websites uh save the cookies uh reuse those cookies in future runs so you don't have to log in every time.
30:07Uh makes it super easy for you to uh replicate real world workflows.
30:12Uh and yes you can use proxies as well.
30:15If you have uh particular IP addresses, residential IP addresses, you can bring those in as well.
30:21Uh a lot of success is definitely how good your infrastructure and environment is, especially in these adversarial scenarios, uh like captures and antibots.
30:32Uh so making the infrastructure as good as possible is definitely of value.
30:42Amazing.
30:42any any asks for something that we might want to run live?
30:48Yes.
30:48Question.
30:56Great, great question.
30:57Actually, I wanted to sort of show the code as well.
30:59Uh so if you uh look at the GitHub, you'll be able to find this evaluate.mjs file.
31:05Uh essentially, it lists out the entire loop.
31:09Uh what's happening here?
31:10Uh but to answer your question, the teacher and the doer are u working one after the other for every loop.
31:22So the teacher is finding learnings in every iteration.
31:26Uh maybe if I show the demo uh over here uh let's go to the one we just did.
31:33Iteration one had these learnings and then iteration two had some other learnings.
31:38Right?
31:39you should include the explicit ISO date in this particular format.
31:43Iteration three would have some other learnings.
31:45The date resolution still opens the bare page and clicks wasting three turns uh reading today's date uh or having the executor compute today.
31:54So it's figuring out something about the date time.
31:56So in every iteration the teacher is learning something new and putting this back into the strategy.mmd.
32:03Maybe I'll make this real by looking at the traces for one of these runs.
32:07Uh so over here these are all my runs.
32:14Uh so for example this was the open table reservation example u and uh this was the strategy.mmd that the teacher sort of appended to every time.
32:26So uh step zero you should do this so on and so forth.
32:30Uh this was the task this was the uh prompt that I gave it.
32:33Find me reservations for the big B bigs restaurant.
32:36uh and these were the kind of traces that uh it was sort of learning from.
32:40It would take screenshots would take these sort of traces at every step is doing browse open this is what the output is.
32:48Then turn number two browse snapshot.
32:51Snapshot essentially takes the accessibility tree and this is the snapshot.
32:55So it's reading this entire thing and at every step thinking of like what went right and what went wrong.
33:01U yeah I mean this is like machine code.
33:04uh I won't show this entire thing.
33:06Uh but yeah, this is the summary that it also writes uh at every iteration.
33:11Uh so yeah, this there's a lot lot of like engineering tweaks that you can do to suit this to your domain as well.
33:18U and uh like the easiest way to uh read this is we put it out here on skills.sh as well uh in auto browse u and you can sort of do something like this.
33:54Great question.
34:03Such a good question.
34:04I think that goes back to like how do you increase the reliability of your uh agents.
34:12Uh reliability is an engineering problem.
34:15Uh you can design your harness to sort of go for one skill for each task or you can be like hey one skill for every task that you will ever see.
34:25Uh I think it entirely depends how you do the harness engineering entirely depends on the use case that you have u and you run evals with all these different strategies
34:37uh and see which one works for you.
34:39for us.
34:39This was Yeah.
34:55Yes.
35:00Yeah.
35:01Yeah.
35:01And this is where again the harness comes into the picture.
35:05If the harness has the intelligence to bring in skills from Open Table and from Resi and Google res Google reservations, maybe it can bring all of those in to figure out what is the best way to book me a date night.
35:21That's your solution, right?
35:23Like the harness should be able to uh figure out which tools to use to get the job done.
35:29That's like how I would go about it.
35:31uh figure out what are the most common use cases that your customers have for those use cases.
35:37What's the best scale files, prompts, tools, maybe you want to do like specific tools for specific use cases, right?
35:45Maybe you want to do generic tools for all use cases.
35:48Like that's the same argument here is like do you do one skill for each or one skill for all uh and eval across all of these strategies and and let the hardness figure it out.
36:01Yes.
36:02Yes.
36:03Yeah.
36:03I mean it's a very similar concept as this, right?
36:06Uh so DSPY.
36:09Yeah.
36:09I I think those guys were on to something really cool like they were around like 3 four years ago where they started building DSP and Japa all of that.
36:19I think uh we have been doing a lot of research uh along these lines as well.
36:24Browserbase works with uh Microsoft.
36:26We actually released a paper uh about how to build the right sort of verifiers uh for your agents.
36:33Uh I think running evals is straightforward until you start questioning okay how should I eval my eval right like do I need to put all these humans to verify all these runs or is there a good verifier that I can build here and all of that u uses the uh learnings from
36:52uh these papers as well.
36:54I can share all of that in the GitHub as well.
37:00I'll also see if there's any new questions here.
37:05Where and how were these skills stored retrieved at scale.
37:08Uh right now for this repo uh all of these get stored in my cloud over here.
37:14So each of these is a scale file.
37:16This was my recipe searcher.
37:18Uh I wanted to build some I wanted to cook some dinner yesterday.
37:21So I was like, "Hey, make me some recipes."
37:24Uh, so all of these are currently in my local uh skills folder.
37:30Uh, but if you want to ship this uh in production, of course, you can uh bake this into your agent loop.
37:38Uh, or if you want to I think for browse.sh the public uh publicly available uh skills that we have, I think we use versel blob storage to store all these skills uh and make them on demand.
37:51Hope that helps.
37:52Uh, websites change frequently.
37:55What's your STR strategy for monitoring that and keeping the website skill updated?
38:00Yeah, this is such a great question.
38:02Uh, and this is what I was alluding to earlier is like how do you keep these skills up to date because the website's going to change.
38:11The the current working theory that we have right now is you don't want to waste tokens running auto research every day.
38:20you want to probably trigger or if it's failing at any particular point in production, right?
38:26So, we build these loops or like systems where as soon as we run into any production failure, we run an auto research loop to see what went wrong to heal it,
38:38create a pull request, that goes directly to a human engineer and they fix it automatically.
38:44The skill gets updated.
38:46Uh so yes there is that notion of like production failure when this might happen.
38:51Uh but it's a trade-off.
38:53You can run an auto research or you can run a sample run every 2 hours every 3 hours to see if the skill is working uh perfectly.
39:03So there's different ways of having the uh cron jobs uh check for the accuracy of these skills.
39:10This is actually one thing that I'm actively working on with the customer right now to implement.
39:15And there's all these questions, right?
39:16People don't want 100% accuracy.
39:19People might want like 10x cheaper cost.
39:20So like uh that's the uh question that comes up.
39:24What's your teachers criteria here in the example?
39:27Separate from reaching the goal seems to do some optimizations for turns.
39:32What other criteria do you use?
39:33Uh so again this is the design of the auto research loop.
39:39Uh there's one way of telling your order research that hey I want to focus on just success.
39:45As soon as you get me success I'm fine just give that to me.
39:49Uh that's typically not what everybody wants to do.
39:52Uh people want to minimize on uh latency.
39:56Uh because maybe this is a real time run where you're focused on how quickly can you get the job done.
40:02U so if that is the optimizing metric and you focus on getting it as fast as possible.
40:07What I've seen sometimes these auto research loops are so smart they realize that browsers are not needed at all and they just reverse engineer the API and they fire the API and that's the fastest that you'll ever get to.
40:19Right?
40:20Uh so sometimes uh I've seen situations where uh it finds a solution that works in 500 milliseconds.
40:28U sometimes it is the number of turns, right?
40:32you want to minimize your context loop uh and you want to minimize the number of uh turns it takes to complete the task.
40:39Uh in my instance, I think what I have done is it's your decision to pass or converge whenever you want to pause it.
40:48I'm not really concerned about the costs.
40:50Uh I care most about reliability.
40:53So keep doing it as long as you feel it's relevant and then it's the teacher agent's job to uh call it day when it wants to.
41:03But yeah, you can uh try all different strategies.
41:07Uh if a website supports WebMP, will this use it?
41:10WebMCP is so cool.
41:12U it's essentially this concept.
41:14I don't know if everybody knows this, but it's this concept of like making your website programmable.
41:21It's it's come out in this new Google Chrome release recently where you tell your u website that if anybody wants to book a reservation they can do xyz on the website.
41:34It's kind of like an API but it's on the browser.
41:37Uh right now I don't think our oral research loop supports this.
41:41Uh but I don't think it would be uh too hard to do that.
41:47Uh in fact we recently shipped support for WebMCP with our SDKs that we launched.
41:53Uh we also made some tools for website owners to ship web MCPS at scale.
41:58Uh but yeah this is an interesting one.
42:00We've not really seen a lot of adoption with Web MCP as yet.
42:06Uh ground truth for pass.
42:09That's interesting.
42:10Uh I think as long as the job was done as long as for example in Google flights uh if it found the cheapest flights in open table if it found a reservation
42:21uh so it it evaluates itself.
42:24It's typically a binary uh and yeah again I I I fully embrace the bitter lesson.
42:31I try not to put many guardrails in uh these evaluations.
42:38Uh, have you considered a verifier as a service for browsing tasks, including something that allows us to mock certain side effects like purchases?
42:47Uh, I might have to ask uh a followup on that.
42:54What do you mean by verifier as a service?
42:56Ravish, if you're here.
42:57Yes.
43:29Uh so uh do you mean that yeah if I understand this correctly like you have like different environments that you can use for RL training and you want to like mock what happens
43:44if I am replicating Amazon if I click on purchase what happens after Uh no because uh we try to rely more on real world websites.
44:00Uh browser base we have the privilege of being able to access most of the web with our uh access to different websites uh and capture solvers etc. Uh so we try to mimic the real world as much as possible but we've been seeing a lot of these RL environments where people are mimicking websites to train
44:20uh and build browser agents that are uh really good at like browsing Salesforce, really good at like browsing SAP.
44:28Uh we try to do that on real world portals as much as possible.
44:33[snorts]
44:34Are edge captures easy for the skill to move past or is it token incentive?
44:39uh what is the point of edge captures for website owners?
44:41Edge capture if I'm understanding this is like a particular kind of capture.
44:45It's kind of like Google recapture etc. uh yeah I would probably talk about this separately.
44:54Uh I don't think we can talk about like what it what captures we can solve or not.
44:58Uh but it's not a token inensive job per se because it's an inf like it's thing that's done at the infrastructure layer, right?
45:06uh we don't really need to use an AI to solve a capture.
45:12Uh it's solved at the Chrome binary layer.
45:14Uh and we do that out of the box.
45:19How does the teacher evaluate the doer and when is the skill markers graduated?
45:23Yes.
45:23So uh it evaluates the doer by essentially reading these traces.
45:29Uh for every iteration there is this folder that gets created.
45:34uh there are these uh JSON files.
45:36Uh it reads these uh it has a clawed agent SDK loop uh and it essentially figures out what the learnings from that loop are.
45:45Uh actually I think I have a slide that talks about this at the end.
45:53Yeah.
45:53Uh this is kind of like the prompt for this uh where it says that you're a browser automation agent.
46:00You can use these tools uh and the strategy.md gets injected at every iteration and this is how the uh uh doer the teacher evaluates the doer's job.
46:14Uh so yeah it reads the traces it has the system prompt it has the previous strategy and it comes up with a better strategy.
46:21[snorts]
46:27Amazing.
46:28Uh I hope we've covered all the questions.
46:34Uh any other questions live that we might want to talk about?
46:39Yes.
46:44Yeah, it can work with the stage hand SDK.
46:46It can work in your cloud code.
46:48Uh it's essentially a markdown, right?
46:50Uh I took uh a skill and I told my clot code, can you find events in SF for this Friday, July the 3rd?
46:57and went on and did this.
46:59So it it need not use browser base.
47:01It need not use stage hand.
47:02It is essentially a skill.
47:03It is a portable artifact you can use in any agent or any SDK.
47:30Yeah.
47:31Yeah.
47:31Great question.
47:31I think if I understand this correctly, how do you prevent against prompt injection?
47:36Uh because websites could trick my auto research loop in learning something bad that can do some malicious behavior in in a production setting.
47:48Uh there's a few ways we do that.
47:51One is at the infrastructure layer itself.
47:53Uh we have a bunch of uh protections around our browsers itself.
48:00Uh we have this browserbased safe mode where you can sort of control which URLs it can go to.
48:07You can control what kind of actions it can take on different websites.
48:11So at the infrastructure layer, at the network firewall, we control things.
48:15Now you might say that it is possible that Amazon.com gets hacked someday.
48:19How do you prevent against like malicious actions on those portals as well?
48:23Uh I think we also try to uh run our skills that are generated against some security uh reviewer agent of sorts uh before we actually publish this on browser.sh.
48:36Uh so that's like one way of doing it.
48:39uh we encourage our customers to have uh a good level of uh prompt injection guardrail uh surface area uh if they're sort of giving this out to c consumers
48:53uh for example some customers of ours uh might want to automate something on Amazon and what if somebody puts uh a listing on Amazon which has some hidden instructions and how do you prevent against those kind of things uh and that's where uh we work with our customers to build the right kind of uh guardrails
49:11before the uh instruction even reaches the hardness.
49:16Um so yeah I think I would put like three layers here.
49:19One is as the as the query is even entering our surface area we control that at the skill layer we make sure that our skill is not uh tricked and at the infrastructure
49:32layer we have network restrictions and action restrictions.
49:35You can even do something like what codeex or plot code does u like always allow or allow once or allow any action.
49:42Uh you can this is what like browserbased safe mode offers out of the box.
49:47I know browser is a crazy threat vector when you think about it like it has access to anything on the wild web.
49:53Uh and take that super seriously from a prompt injection POV.
50:02Amazing.
50:03I'll wait for 10 more seconds if there's any other questions.
50:08This was fun.
50:09I really enjoyed this.
50:10I love the kind of questions as well.
50:13Uh and yeah, I will put the QR code back on screen in case any of you want to grab this.
50:22Yes.
50:33Yeah.
50:44Yeah.
50:45Yeah.
50:46Exactly.
50:46Yeah.
50:46I think we talked about it a while back.
50:48What you could do is like I used uh entropic managed agents quite a bit.
50:52Uh so I would run a chron job on managed agents where I would say use this skill every day at 9:00 am to do this job and if this is a success all good if this is a failure send me a notification here that will trigger a healing loop
51:07uh so that's how I sort of can bake in uh checks and balances around websites changing antibbots changing etc. I'm going to hang around here.
51:24Uh also uh I'll be at the browserbased booths all days of the conference as well.
51:29So uh in case you're curious and want to chat more uh love to do that.
51:34Thank you.
51:50[music]