WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=p7gaxmCUiIU

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/p7gaxmCUiIU.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:13.840
All right.

s2
00:00:13.840 --> 00:00:14.559
Hello.

s3
00:00:14.559 --> 00:00:16.320
Thank you everybody for being here.

s4
00:00:16.320 --> 00:00:18.480
Uh I know it's the last workshop of the day.

s5
00:00:18.480 --> 00:00:20.720
I appreciate all of you turning up here.

s6
00:00:20.720 --> 00:00:26.640
U I am show uh I'm here to talk about hill climbing skills.

s7
00:00:26.640 --> 00:00:34.000
uh specifically how can you improve your agents without really going into the models without really going in the weights.

s8
00:00:34.000 --> 00:00:37.120
Uh I have a slido uh up here as well.

s9
00:00:37.120 --> 00:00:45.360
I'll just uh flash the QR code in case any of you want to uh put in any questions and we can address them towards the end as well.

s10
00:00:45.360 --> 00:00:47.520
U a little bit about me.

s11
00:00:47.520 --> 00:00:48.399
I'm Sh.

s12
00:00:48.399 --> 00:00:51.840
I'm uh on the sales engineering team at browserbase.

s13
00:00:51.840 --> 00:00:58.000
Uh I essentially work with our customers deploying browser agents in production.

s14
00:00:58.000 --> 00:01:01.520
Uh I'm not sure maybe we'll do a show of hands.

s15
00:01:01.520 --> 00:01:06.320
Uh how many of you know what browser agents are?

s16
00:01:06.320 --> 00:01:07.360
Anybody?

s17
00:01:07.360 --> 00:01:08.880
All right, a few.

s18
00:01:08.880 --> 00:01:09.520
Amazing.

s19
00:01:09.520 --> 00:01:13.040
Uh have any of you built browser agents before?

s20
00:01:13.040 --> 00:01:15.840
Have any of you used these tools?

s21
00:01:15.840 --> 00:01:16.640
Show hands.

s22
00:01:16.640 --> 00:01:18.000
Okay, a few of them.

s23
00:01:18.000 --> 00:01:19.200
Amazing.

s24
00:01:19.200 --> 00:01:20.159
Nice.

s25
00:01:20.159 --> 00:01:33.439
uh maybe I'll I'll talk a bit about what browser agents are before diving into the whole concept of hill climbing for skills uh so that it becomes clear what we're doing towards the end of the talk we'll also look into

s26
00:01:33.439 --> 00:01:39.200
how you can do this in your domain in your uh field as well.

s27
00:01:39.200 --> 00:01:39.920
All right.

s28
00:01:39.920 --> 00:01:45.119
Uh so quick primer on what browser agents are.

s29
00:01:45.119 --> 00:01:51.119
If you have an agent that needs to drive a browser, you basically call that a browser agent, right?

s30
00:01:51.119 --> 00:01:53.920
Browser becomes the tool to your agent.

s31
00:01:53.920 --> 00:01:55.600
And how do you control it?

s32
00:01:55.600 --> 00:02:05.840
You control it with some sort of CLI, some sort of MCP, some sort of method where you can control the browser and say, "Hey, go and click on this red button.

s33
00:02:05.840 --> 00:02:07.920
Go and click on the refresh button.

s34
00:02:07.920 --> 00:02:14.000
Go and uh take a screenshot or take a snapshot of the accessibility tree of this website."

s35
00:02:14.000 --> 00:02:14.239
Right?

s36
00:02:14.239 --> 00:02:20.000
So these are this low-level tools that your agent can have access to to control the browser.

s37
00:02:20.000 --> 00:02:27.280
Uh and browserbase is essentially a platform for you to build and run these browser agents.

s38
00:02:27.280 --> 00:02:35.040
We provide the infrastructure and the SDKs, the skills, the CLI that go with building these browser agents.

s39
00:02:35.040 --> 00:02:41.599
Uh and and we're going to be using all of these tools to see what hill climbing does look like.

s40
00:02:41.599 --> 00:02:42.239
All right.

s41
00:02:42.239 --> 00:02:58.239
Uh I want to talk a bit about the biggest problem with agents today essentially reliability right uh people are frustrated with browser agents getting stuck or getting into an anti-border

s42
00:02:58.239 --> 00:03:04.000
capture issue uh or maybe just forgetting or being lazy and not completing the task.

s43
00:03:04.000 --> 00:03:08.159
Uh and we see reliability essentially is an engineering problem.

s44
00:03:08.159 --> 00:03:18.800
I think the recent models the Opus and the Fable when we when it came out for those three days it was really really smart right uh but we see that a lot of the

s45
00:03:18.800 --> 00:03:31.519
work that has to be done for reliability is the hardness around it is the tools that you give it it is the skills that you give here u and and that's what we're going to be sort of looking at how do we improve the reliability

s46
00:03:31.519 --> 00:03:38.560
of these browser agents I wanted to give you all a few examples of like where these agents suck today, right?

s47
00:03:38.560 --> 00:03:52.400
If I zoom in a bit, you'll see this browser agent is trying to like go on Google flights and click on the uh it's trying to drag the time filter and it's unable to figure out how do I drag that.

s48
00:03:52.400 --> 00:03:56.959
Dragging is a tricky operation for these agents to do and it's getting super confused.

s49
00:03:56.959 --> 00:04:00.879
It doesn't know what the HTML element for that is, what the JavaScript for that is.

s50
00:04:00.879 --> 00:04:05.599
It's not really able to do this and it gets stuck and it's not able to book my flight tickets.

s51
00:04:05.599 --> 00:04:11.040
Uh, another example I wanted to show uh is a browser agent on open table.

s52
00:04:11.040 --> 00:04:22.320
Let's say I have my open claw or my Hermes agent and I need to book a date night and I tell hey go to open table and and find me some and and look at the anti-bot that it runs into, right?

s53
00:04:22.320 --> 00:04:24.479
Uh, and it's just like access denied.

s54
00:04:24.479 --> 00:04:26.000
I don't know what happens here.

s55
00:04:26.000 --> 00:04:29.600
Uh, but like can my browser agent get around these issues?

s56
00:04:29.600 --> 00:04:30.800
Can they be more reliable?

s57
00:04:30.800 --> 00:04:35.360
And can they book my date night restaurants is the biggest question I wanted to ask here.

s58
00:04:35.360 --> 00:04:51.360
uh and this is where we sort of look at different ideas to improve the reliability of these agents and one of them uh oh actually let's let's dig into the root cause here right like why this is happening

s59
00:04:51.360 --> 00:05:01.040
u we have like these opus models these five five the super nice general purpose you might have used codeex computer use very nice but when you want to sort of do

s60
00:05:01.040 --> 00:05:16.880
uh longunning tricky websites like the open tables or the Google flights it sort of becomes very difficult to run these in production and these sightspecific tricks or these javascript that you need to run on your browser

s61
00:05:16.880 --> 00:05:34.080
the agent doesn't know about them right and the most important thing is it's every run starts from zero you're not really having any sort of learning baked into your runs I think Darkh talked about this a couple days ago it's like why is progress on computer use still so slow today.

s62
00:05:34.080 --> 00:05:40.479
Uh although computer use seems like a verifiable problem, why is it still not solved?

s63
00:05:40.479 --> 00:05:42.800
Essentially, there's no learning happening.

s64
00:05:42.800 --> 00:05:49.919
Every agent run is a start from scratch run and it's not learning from its production traces.

s65
00:05:49.919 --> 00:05:53.759
U and these browser agents, I call them geniuses that never learn.

s66
00:05:53.759 --> 00:06:00.400
Um and some people might say, oh, why don't you fine-tune on something some like open table or restaurant websites.

s67
00:06:00.400 --> 00:06:03.039
Uh but it it's not really scalable, right?

s68
00:06:03.039 --> 00:06:07.759
uh if you're thinking of building agents, you can't really be fine-tuning for every new task that comes up.

s69
00:06:07.759 --> 00:06:16.960
You want some you want a a modality where it can learn from its production traces uh and get better over time.

s70
00:06:17.039 --> 00:06:25.280
This is where we sort of started thinking about and this was like six four to six months ago where Karpadi came up with this idea of auto research.

s71
00:06:25.280 --> 00:06:27.600
This is essentially a loop, right?

s72
00:06:27.600 --> 00:06:40.080
Uh and this loop sort of takes a verifiable problem statement and goes at it, tries it again and again, looks at its own traces and tries to improve itself.

s73
00:06:40.080 --> 00:06:43.840
And we wondered how can we apply this to browser agents?

s74
00:06:43.840 --> 00:06:53.440
How can we apply this uh to these complicated workflows, these complicated websites which are not deterministic, they change every day, they have different captures every time.

s75
00:06:53.440 --> 00:06:55.680
Uh how can we learn from these?

s76
00:06:55.680 --> 00:06:56.240
Right?

s77
00:06:56.240 --> 00:07:02.240
Uh this is where I want to sort of switch into a demo and sort of show this live.

s78
00:07:02.240 --> 00:07:09.599
Uh and I'm going to start with uh a prompt over here.

s79
00:07:09.599 --> 00:07:22.560
So I want to maybe do something like uh go to Google flights and uh find me a business class flight from SF to New York for this Saturday.

s80
00:07:22.560 --> 00:07:25.360
I want to have one carry-on bag.

s81
00:07:25.360 --> 00:07:29.120
Uh and this flight should be in the morning.

s82
00:07:29.120 --> 00:07:34.880
I hope my Okay, this is correct.

s83
00:07:35.440 --> 00:07:36.880
Just paste this and run this.

s84
00:07:36.880 --> 00:07:37.440
Right.

s85
00:07:37.440 --> 00:07:40.240
So starting a browser based session here.

s86
00:07:40.240 --> 00:07:47.360
Uh so what you see in the center is a remote browser has just spun up and has navigated to Google flights.

s87
00:07:47.360 --> 00:07:53.120
If I expand my agent here, uh it's doing all of these things.

s88
00:07:53.120 --> 00:07:59.520
This is the reasoning traces that you can see live and it's trying to sort of apply the business class filter.

s89
00:07:59.520 --> 00:08:02.560
So it it'll do its job and and and we'll come back to this.

s90
00:08:02.560 --> 00:08:05.599
Uh but I wanted to talk about what's really happening here.

s91
00:08:05.599 --> 00:08:12.639
Uh u and the architecture behind this auto research loop.

s92
00:08:15.759 --> 00:08:18.960
So we we fired the Google flights task.

s93
00:08:18.960 --> 00:08:24.240
Uh and and and the architecture here is this auto browse loop that we call this.

s94
00:08:24.240 --> 00:08:26.080
We took auto research and we took browsing.

s95
00:08:26.080 --> 00:08:27.440
We call it auto browse.

s96
00:08:27.440 --> 00:08:32.399
And this loop essentially is two clawed agent SDK loop uh agents.

s97
00:08:32.399 --> 00:08:36.719
Uh you can call one as the doer agent and one as the teacher agent.

s98
00:08:36.719 --> 00:08:40.560
This doer agent its job is to actually go and do the browsing.

s99
00:08:40.560 --> 00:08:43.519
It has access to the browse CLI that we talked about.

s100
00:08:43.519 --> 00:08:58.800
It has access to the browser uh and it goes and sort of clicks on buttons, looks at the screenshot, tries to get me information and the teacher teacher's job is to look at the traces of the doer and learn what went wrong and what went right

s101
00:08:58.800 --> 00:09:05.760
and they keep working together until they find a very good solution that converges.

s102
00:09:05.760 --> 00:09:06.399
Right?

s103
00:09:06.399 --> 00:09:10.000
Uh we don't tell it how many loops you need to go for.

s104
00:09:10.000 --> 00:09:15.760
We don't tell it what's the success factor success uh rate that we want.

s105
00:09:15.760 --> 00:09:21.040
We don't tell it hey we need to do this within x number of tokens or x number of seconds.

s106
00:09:21.040 --> 00:09:28.959
We just give it a broad prompt and we let the bitter lesson we embrace the bitter lesson and we're like okay let the doer and teacher go at it.

s107
00:09:28.959 --> 00:09:34.480
Uh and the key thing that happens is there is a strategy.md file.

s108
00:09:34.480 --> 00:09:39.440
uh that is essentially the artifact across multiple runs.

s109
00:09:39.600 --> 00:09:49.440
After every iteration, the teacher thinks about what happened, looks at the screenshot, looks at the logs, and then thinks, "Oh, you did not click on this button.

s110
00:09:49.440 --> 00:09:54.720
U you should actually click on this button instead, and it puts that in the strategy.mmd file."

s111
00:09:54.720 --> 00:09:59.920
And that artifact carries over to the next iteration, and it improves from there.

s112
00:09:59.920 --> 00:10:02.959
Maybe we'll go back to our demo and see what happened.

s113
00:10:02.959 --> 00:10:03.279
All right.

s114
00:10:03.279 --> 00:10:07.360
So, iteration number one, uh, there was a failure here.

s115
00:10:07.360 --> 00:10:18.880
Uh, and the learning from this iteration was skip the entire form calendar widget because you can just do a query parameter and you don't really need to click around, right?

s116
00:10:18.880 --> 00:10:29.200
The agent was smart enough in this instance to think, oh, I'm wasting so many tokens and I'm wasting so much time clicking around where I could just use Google Flight's query parameter.

s117
00:10:29.200 --> 00:10:49.279
Now this is obvious in hindsight but maybe as a human I wouldn't have thought of this and I wouldn't have written this complicated query parameter myself right uh so this is the learning from the first iteration now it's going and doing things in iteration number two it'll keep going on till it feels like it has maxed out

s118
00:10:49.279 --> 00:11:01.839
all the tricks it has maxed out all JavaScript hacks that it could do uh and and and then uh think about okay this is the skill that I can use from here.

s119
00:11:03.519 --> 00:11:15.040
I think the key the key thing uh that I wanted to also talk about is at the end of this converged loop the output that I want is a skill.mmd

s120
00:11:15.040 --> 00:11:15.760
file.

s121
00:11:15.760 --> 00:11:20.560
This skill.md file would be what I give to my future agents.

s122
00:11:20.560 --> 00:11:27.600
Uh and it would have a collection of all the hacks or all the tricks or all the learnings from previous runs.

s123
00:11:27.600 --> 00:11:30.720
We'll we'll look at what that scale might look like.

s124
00:11:30.720 --> 00:11:34.959
Uh it could look like, hey, a pre-filled URL is mandatory.

s125
00:11:34.959 --> 00:11:40.399
Don't go around and waste your clicks and tokens clicking on these buttons.

s126
00:11:40.399 --> 00:11:43.440
There is a reopen bug in the calendar.

s127
00:11:43.440 --> 00:11:44.720
Don't do this.

s128
00:11:44.720 --> 00:11:51.920
Uh if you want to move the drag uh if you want to drag your time stamp thing, uh use JavaScript injection.

s129
00:11:51.920 --> 00:11:53.680
Don't try to do it with the mouse.

s130
00:11:53.680 --> 00:11:54.240
Right?

s131
00:11:54.240 --> 00:12:07.120
These are the kind of tricks that are now persisted for every future run and gives me so many more savings uh from a performance, reliability and cost perspective.

s132
00:12:07.680 --> 00:12:13.120
Now uh we'll go back to our uh run and see if it has passed.

s133
00:12:13.120 --> 00:12:21.040
Okay, so iteration number two has passed and if you can see uh the savings in in cost are like 4x.

s134
00:12:21.040 --> 00:12:24.160
Uh this chart is being created on the fly.

s135
00:12:24.160 --> 00:12:27.839
Uh so it started from $12 and the second one was $3.

s136
00:12:27.839 --> 00:12:37.760
And keep in mind my agents here are Opus 48 both for the doer and uh for the uh teacher.

s137
00:12:37.760 --> 00:12:40.000
Now iteration three just came in.

s138
00:12:40.000 --> 00:12:42.320
Uh the number of turns decreased.

s139
00:12:42.320 --> 00:12:44.399
The latency is also better.

s140
00:12:44.399 --> 00:12:47.360
Uh but there was a failure in iteration number three.

s141
00:12:47.360 --> 00:12:55.600
there's something that it sort of overcorrected on and iteration number four should probably be learning from that and we'll see if it does that.

s142
00:12:55.600 --> 00:13:03.120
Uh while it's doing its own thing, I wanted to show some previous runs that I did uh just in preparation for this uh demo.

s143
00:13:03.120 --> 00:13:12.800
U so for example, I went to uh auto browse and I was like hey go to the AI engineers website and tell me which talks and workshops are talking about hill climbing,

s144
00:13:12.800 --> 00:13:13.360
right?

s145
00:13:13.360 --> 00:13:17.920
uh and it did that across multiple iterations and you can see the replay of each of these.

s146
00:13:17.920 --> 00:13:31.839
These are replays that are coming from browserbase super helpful important for uh observability perspective and as you can see uh it sort of uh got better in ter in terms of cost 7.4x 4x

s147
00:13:31.839 --> 00:13:38.160
uh even though it was able to pass in all the iterations, I still got improvements in cost.

s148
00:13:38.160 --> 00:13:42.079
Now I wanted to actually see what are the learnings from this skill, right?

s149
00:13:42.079 --> 00:13:46.560
Uh how do you sort of uh persist all of these learnings?

s150
00:13:46.560 --> 00:13:49.680
It says go straight to the shed schedule page.

s151
00:13:49.680 --> 00:13:53.200
If there are open errors, don't try retry blindly.

s152
00:13:53.200 --> 00:13:54.240
Run this.

s153
00:13:54.240 --> 00:13:58.000
Uh type the keyword into the session search input.

s154
00:13:58.000 --> 00:13:58.800
Wait briefly.

s155
00:13:58.800 --> 00:14:02.639
So like have these weights because sometimes websites take a lot of time to load.

s156
00:14:02.639 --> 00:14:06.560
So you you can have these kind of tricks uh and then read the results.

s157
00:14:06.560 --> 00:14:11.680
Uh you know like there there's a lot of learning that is happening in these skills.

s158
00:14:11.680 --> 00:14:14.480
Maybe I wanted to show some interesting ones.

s159
00:14:14.480 --> 00:14:22.399
I wanted to show this open table one because I started with this blocker of like access denied when we started this demo.

s160
00:14:22.399 --> 00:14:30.480
Uh and if we go to the uh screenshot, this was the access denied thing that we got right in in iteration number one.

s161
00:14:30.480 --> 00:14:32.720
Let's see how it learned from there.

s162
00:14:32.720 --> 00:14:40.880
Uh it realized that the uh block is environmental because the website doesn't want this agent to do something.

s163
00:14:40.880 --> 00:14:52.720
uh in iteration number two, it figured that if I do a query parameter and if I put a latitude longitude metro ID, I automatically get that uh UI that works.

s164
00:14:52.720 --> 00:15:00.079
Do not open /r that's blocked and do not reload / s. These are the places where the blocks kick in.

s165
00:15:00.079 --> 00:15:05.440
If you don't do that and you follow this technique instead, your agent will work, right?

s166
00:15:05.440 --> 00:15:12.639
Uh and as you can see, uh it sort of went from $6.22 22 to 0.51.

s167
00:15:12.639 --> 00:15:17.199
The time taken went from $189 seconds to 27 seconds.

s168
00:15:17.199 --> 00:15:19.839
Uh and the scale it came up with was here.

s169
00:15:19.839 --> 00:15:25.279
Uh build the search URL, do the date math like this.

s170
00:15:25.279 --> 00:15:29.120
Uh this is the known metro pins for San Francisco.

s171
00:15:29.120 --> 00:15:31.680
Uh and and follow these steps, right?

s172
00:15:31.680 --> 00:15:35.760
Never call / r because that will return access denied.

s173
00:15:35.760 --> 00:15:46.959
Now these are the kind of learnings where a human could never have done but you just sort of embrace the bitter lesson and a let agents do the job uh and they will find these uh tricks for you.

s174
00:15:48.160 --> 00:15:54.320
I want to uh also show uh a few examples of like the kind of improvements we see.

s175
00:15:54.320 --> 00:16:04.480
Uh the reason why this problem is so close to my heart is because I from a sales perspective am deploying these browser agents into production.

s176
00:16:04.480 --> 00:16:09.120
uh the sale is only done when the browser agent is successful.

s177
00:16:09.120 --> 00:16:18.079
So reliability is like the most important thing for me and this is what I this is the key tool that I take to my customers and I'm like just fire your agents

s178
00:16:18.079 --> 00:16:22.480
let them do this auto research thing and your agents will be successful in production.

s179
00:16:22.480 --> 00:16:32.399
Uh and these are the kind of u success metrics that people are looking for up to 16x cheaper 12x cheaper 3x cheaper uh on different tasks.

s180
00:16:32.399 --> 00:16:34.079
Of course, every task is different.

s181
00:16:34.079 --> 00:16:35.920
The bottlenecks on each task are different.

s182
00:16:35.920 --> 00:16:38.399
So, uh you might see variance there.

s183
00:16:38.399 --> 00:16:45.040
But the idea is you can hill climb on these uh techniques.

s184
00:16:46.399 --> 00:16:53.120
I wanted to go back to our live demo and see if there's any uh improvement.

s185
00:16:53.120 --> 00:16:53.440
All right.

s186
00:16:53.440 --> 00:16:58.160
So, we have a graduated skill u and the skill applied end to end.

s187
00:16:58.160 --> 00:16:59.360
No further learning.

s188
00:16:59.360 --> 00:17:00.880
The results are showing correctly.

s189
00:17:00.880 --> 00:17:06.720
one-way business class, San Francisco to New York, July the 4th, one carry-on bag included.

s190
00:17:06.720 --> 00:17:09.280
Uh, and there is 8 a.m. $649.

s191
00:17:09.280 --> 00:17:09.600
Right?

s192
00:17:09.600 --> 00:17:15.360
So, it was able to get all of this and I can sort of watch the entire replay to see if that is accurate.

s193
00:17:15.360 --> 00:17:22.880
Uh, it was able to sort of figure out this entire skill and the fast path is encode the whole query in the URL.

s194
00:17:22.880 --> 00:17:35.200
strongly preferred the again like I know I'm I'm showing these things but when I first started looking at the the results here this was like my moment of like this is the mythos moment for

s195
00:17:35.200 --> 00:17:46.640
browser agents you know like just reverse engineering the hell out of things uh I I look at this with some of my customers the customers are mind-blown uh earlier they would be writing scripts

s196
00:17:46.640 --> 00:17:55.840
for each of their websites they would be like hey I need to spend so many engineering hours on like just reversing these websites.

s197
00:17:55.840 --> 00:18:09.440
How can I scale this to 100 a thousand websites and now our customers have actually built these entire pipelines of uh these skills that get generated on the fly pull requests that get created automatically.

s198
00:18:09.440 --> 00:18:15.280
If there is a failure in production, an agent kicks in and starts healing the skill as well.

s199
00:18:15.280 --> 00:18:22.240
So there's a lot of uh engineering that we can do around this to make leverage to leverage this technique.

s200
00:18:22.240 --> 00:18:40.880
Uh I wanted to also show what we've done with this right we actually went ahead and created this product called browse.sh uh we were like if we can do this for one website we can do this for five why not do this for like 500 websites and give it for free for anybody to do this.

s201
00:18:40.880 --> 00:18:53.679
Uh and like if you remember in the 19 or wherever like in the '9s before internet you had the yellow pages uh you had details about accessing people, businesses

s202
00:18:53.679 --> 00:19:02.240
etc. This is yellow pages for browser agents right it has details about how to do any task on any website on the web.

s203
00:19:02.240 --> 00:19:05.200
U I wanted to actually show this to you all.

s204
00:19:05.200 --> 00:19:13.679
uh you can actually install browse CLI and use all of these skills completely for free.

s205
00:19:13.679 --> 00:19:19.039
We've eaten up the token costs ourselves and we're making this available for you all.

s206
00:19:19.039 --> 00:19:26.320
Uh so if I go to let's say search products on Amazon uh I can see the entire scale is over here.

s207
00:19:26.320 --> 00:19:28.400
When should you use the scale?

s208
00:19:28.400 --> 00:19:33.280
How should you look at like pricing and availability and whatever right?

s209
00:19:33.280 --> 00:19:47.840
uh this is how you use this lot of details about Amazon the screenshots u and I just give this browse skills CLI to my openclaw and it just knows when to pull the alltrails.com

s210
00:19:47.840 --> 00:19:53.600
skill when to pull the weather.gov skill and and this makes my life so much easier.

s211
00:19:53.600 --> 00:20:01.280
uh we essentially are running the same auto research loop behind the scenes uh and making this uh available for you all.

s212
00:20:01.280 --> 00:20:13.039
In in fact if you would want to you can go here and you can be like hey go and make a skill for me to uh look up this uh events in San Francisco for this week.

s213
00:20:13.039 --> 00:20:18.320
Uh you do this generate a skill and this will sort of do this for you.

s214
00:20:18.320 --> 00:20:24.480
This will send you an email and you can then use your skill uh in whatever agent you want to, right?

s215
00:20:24.480 --> 00:20:28.080
Uh so we've done this for browser agents.

s216
00:20:28.080 --> 00:20:35.520
Uh and I wanted to talk about like how can you potentially do this for any other domain, right?

s217
00:20:35.520 --> 00:20:45.840
Uh I think coding agents is like one place where I have seen this frustration where the agent is relearning the repo conventions every time.

s218
00:20:45.840 --> 00:20:54.480
it's not applying the linting rules correctly and I have to tell it every time and maybe you can have your cloud MD and agents MD and improve that but

s219
00:20:54.480 --> 00:21:09.840
the the fundamental premise is like the agent should not be relearning your tasks from zero right uh I've seen this in research agents we run a bunch of like deep research agents for our internal workflows

s220
00:21:09.840 --> 00:21:20.559
uh and every time it struggles and it tries to relearn oh I pulled this information from Wikipedia IA uh but I did not pull it from this other website.

s221
00:21:20.559 --> 00:21:22.080
Uh I was like why not?

s222
00:21:22.080 --> 00:21:27.280
Uh like we always do this other website you should learn from your previous traces.

s223
00:21:27.280 --> 00:21:29.679
Uh and you should pull information.

s224
00:21:29.679 --> 00:21:38.159
Uh internal ops you have all these dashboards all these databases all these internal tools but your agents don't know which ones work and which ones are preferred.

s225
00:21:38.159 --> 00:21:42.080
Ideally you should be applying this all research technique.

s226
00:21:42.080 --> 00:21:54.799
Uh a and the two conditions that are required is that a you should have a replayable environment of some sorts so that you can run these loops and b you should have some sort of verifiable outcome

s227
00:21:54.799 --> 00:21:58.400
some sort of quant quantifiable metric for you to run these loops.

s228
00:21:58.400 --> 00:22:01.280
Yes, I see a question there.

s229
00:22:18.960 --> 00:22:41.039
I realized that Yeah.

s230
00:22:43.200 --> 00:22:44.880
Yeah.

s231
00:22:44.880 --> 00:22:45.280
Yeah.

s232
00:22:45.280 --> 00:22:45.760
Exactly.

s233
00:22:45.760 --> 00:22:55.760
Uh so I think the question is like how do you know that the auto research loop is accurate whether it is converging and when it says it's converging it's actually converging.

s234
00:22:55.760 --> 00:22:57.679
When it says it's good it's actually good.

s235
00:22:57.679 --> 00:22:59.280
When it says it's bad it's actually bad.

s236
00:22:59.280 --> 00:23:05.520
like how do I have confidence uh on what the agent is saying and what the loop is doing.

s237
00:23:05.520 --> 00:23:09.200
I think that's where I want to sort of talk about the verifiable outcome here.

s238
00:23:09.200 --> 00:23:17.280
Uh for Karpathi, it was quite easy because he was looking at the log log loss of the experiment when he was training neural networks.

s239
00:23:17.280 --> 00:23:17.600
Right?

s240
00:23:17.600 --> 00:23:21.440
So it's very easy to see whether your log loss has gone down or not.

s241
00:23:21.440 --> 00:23:25.600
Uh but in many real world scenarios that is not the case for browser agents.

s242
00:23:25.600 --> 00:23:37.520
As I explained, uh, whether you book a flight ticket or not, that might be straightforward, but did you find the best flight options for me might be slightly fuzzy

s243
00:23:37.520 --> 00:24:13.039
uh and might not be the best uh Yeah, if if I understand this correctly, you're saying like one mistake in the loop can sort of throw off the entire auto research run.

s244
00:24:13.039 --> 00:24:15.919
Uh and how do you prevent against that?

s245
00:24:15.919 --> 00:24:22.400
I I I have actually seen where some uh iterations actually fail.

s246
00:24:22.400 --> 00:24:44.400
uh for example if we uh look at the previous slide uh there's like failures happening after there have been successes right uh so it's not always that once you succeed all the future iterations will always be successful but I think it's the design of your loop where you can make it resilient

s247
00:24:44.400 --> 00:24:55.360
to u failures uh where it doesn't overindex on those uh and still sort of correct protects itself uh from any mistake that happened in previous runs.

s248
00:24:55.360 --> 00:25:03.520
Uh I think we're trying to sort of prevent like uh what do they call like exploding sort of loss sort of scenarios.

s249
00:25:03.520 --> 00:25:05.840
Uh I think you could do that.

s250
00:25:05.840 --> 00:25:10.559
Uh in fact Karpathi talks about this like this is one way of doing auto research.

s251
00:25:10.559 --> 00:25:14.480
You can actually auto research on different ways of doing auto research itself.

s252
00:25:14.480 --> 00:25:17.360
Uh it's like is the teacher doer loop the best?

s253
00:25:17.360 --> 00:25:27.440
Should there be a multi-agent architecture to do this or is there any other approach that might actually work better for research tasks in neural networks versus browser agents?

s254
00:25:27.440 --> 00:25:27.919
Right?

s255
00:25:27.919 --> 00:25:32.720
You could have different designs of your loops to prevent situations like those.

s256
00:25:32.720 --> 00:25:45.679
Uh and I think going back to my point, as long as you have a verifiable outcome or some quantifiable metric, I think you should be able to design uh a good uh loop here.

s257
00:25:45.679 --> 00:25:52.559
uh replayable environment I think sometimes is under discussed uh but extremely important.

s258
00:25:52.559 --> 00:26:05.440
I think he was talking about Darkh he said having a verifiable outcome is important but also you should have uh extremely grindable environments or extremely grindable uh use cases as well.

s259
00:26:05.440 --> 00:26:06.080
Right?

s260
00:26:06.080 --> 00:26:17.039
If your agent cannot uh grind through failures, it will be difficult for it to uh go for multiple iterations and fix itself.

s261
00:26:17.039 --> 00:26:29.840
Uh so do you have like coding agents is a good example because you can start from a docker image or you can start from uh a particular EC2 and and you can replay from different parts of the uh stack.

s262
00:26:29.840 --> 00:26:35.440
Uh but maybe uh knowledge work is probably not that easy to replay.

s263
00:26:35.440 --> 00:26:42.080
uh maybe there's like a lot of different human in the loop actions that make it very hard to replay from a particular point in time.

s264
00:26:42.080 --> 00:26:49.360
So investing in your environment also can be super uh rewarding in these cases.

s265
00:26:52.640 --> 00:26:53.200
Amazing.

s266
00:26:53.200 --> 00:27:02.159
Uh I just wanted to share the uh URLs of uh the auto browse loop that I uh demoed today.

s267
00:27:02.159 --> 00:27:05.840
So if you scan the QR you'll be able to pull the uh GitHub for that.

s268
00:27:05.840 --> 00:27:10.320
Uh and also the second QR here uh is the skill itself.

s269
00:27:10.320 --> 00:27:22.480
You can just do /auto browse on your claude code or codeex and be like hey /auto browse use opus48 on both of these uh teacher and doer and go create a skill for me to do this.

s270
00:27:22.480 --> 00:27:28.480
Uh I'm looking at one of these questions.

s271
00:27:28.480 --> 00:27:40.000
Uh and also if any of you have ideas of something we want to sort of put the agent against feel free to type it in the slido and we can actually try it live.

s272
00:27:40.000 --> 00:27:41.440
Uh but let's look at this question.

s273
00:27:41.440 --> 00:27:43.520
Is there a way to optimize for convergence?

s274
00:27:43.520 --> 00:27:46.480
Uh I think this is the same question that we were just talking about.

s275
00:27:46.480 --> 00:27:57.360
Uh high cost is interesting because you might think that u doing longer iterations might become costlier over time.

s276
00:27:57.360 --> 00:28:08.480
Uh I have run a bunch of evals on this where I can throw a sonnet at this problem but run it for longer iterations versus throwing opus for lesser iterations.

s277
00:28:08.480 --> 00:28:16.159
I think you can get the same job done by throwing sonnet larger number of iterations but still cheaper overall.

s278
00:28:16.159 --> 00:28:26.480
Uh also this cost is amotized right like one auto browse loop that we probably run is like maybe $30 $40 if it's super complicated $100 at most.

s279
00:28:26.480 --> 00:28:33.919
Uh so this is amotized as long as those costs make sense for you uh to deploy this in production uh just go for it.

s280
00:28:33.919 --> 00:28:40.799
Uh but there's a lot of interesting things uh that you could do uh with evaling the kind of loops that you're generating.

s281
00:28:40.799 --> 00:28:42.799
Uh yes I will share the slides as well.

s282
00:28:42.799 --> 00:28:47.679
Uh I can do this on the GitHub itself.

s283
00:28:47.919 --> 00:28:54.960
Uh can you share how you thought about uh auto research into auto browse?

s284
00:28:54.960 --> 00:28:58.880
Uh I mean the process was fairly straightforward.

s285
00:28:58.880 --> 00:29:12.399
Uh took the GitHub repo that Karpathi put out uh and then there's all this context about my customers uh that I have uh where I was like hey these customers face these kinds of reliability problems

s286
00:29:12.399 --> 00:29:22.559
uh the way I solve it for them is I go into their office I sit with them very forward deployed uh and I try to sort of just brute force and reverse engineer this.

s287
00:29:22.559 --> 00:29:23.520
This is what I do.

s288
00:29:23.520 --> 00:29:25.600
So I am the teacher agent.

s289
00:29:25.600 --> 00:29:34.000
Uh can you automate me out of the process and create a loop borrowing from uh Karpathi and and and and build the same thing?

s290
00:29:34.000 --> 00:29:36.799
And of course there's a lot of iterations on top of that.

s291
00:29:36.799 --> 00:29:41.039
Uh there's a lot of engineering as well uh to simplify this.

s292
00:29:41.039 --> 00:29:46.320
Uh but yes, that was the process of converting this.

s293
00:29:48.080 --> 00:29:49.440
Can you use this in a proxy?

s294
00:29:49.440 --> 00:29:50.799
Can you be logged into a site?

s295
00:29:50.799 --> 00:29:51.360
Yes.

s296
00:29:51.360 --> 00:29:54.559
Uh again this goes back to the environment that you design.

s297
00:29:54.559 --> 00:30:07.279
Uh browserbased the infrastructure that all of this is running on uh gives you the ability to log into websites uh save the cookies uh reuse those cookies in future runs so you don't have to log in every time.

s298
00:30:07.279 --> 00:30:12.559
Uh makes it super easy for you to uh replicate real world workflows.

s299
00:30:12.559 --> 00:30:15.120
Uh and yes you can use proxies as well.

s300
00:30:15.120 --> 00:30:21.520
If you have uh particular IP addresses, residential IP addresses, you can bring those in as well.

s301
00:30:21.520 --> 00:30:32.159
Uh a lot of success is definitely how good your infrastructure and environment is, especially in these adversarial scenarios, uh like captures and antibots.

s302
00:30:32.159 --> 00:30:38.480
Uh so making the infrastructure as good as possible is definitely of value.

s303
00:30:42.240 --> 00:30:42.880
Amazing.

s304
00:30:42.880 --> 00:30:48.159
any any asks for something that we might want to run live?

s305
00:30:48.159 --> 00:30:48.559
Yes.

s306
00:30:48.559 --> 00:30:50.559
Question.

s307
00:30:56.320 --> 00:30:57.200
Great, great question.

s308
00:30:57.200 --> 00:30:59.840
Actually, I wanted to sort of show the code as well.

s309
00:30:59.840 --> 00:31:05.600
Uh so if you uh look at the GitHub, you'll be able to find this evaluate.mjs file.

s310
00:31:05.600 --> 00:31:09.440
Uh essentially, it lists out the entire loop.

s311
00:31:09.440 --> 00:31:10.799
Uh what's happening here?

s312
00:31:10.799 --> 00:31:22.399
Uh but to answer your question, the teacher and the doer are u working one after the other for every loop.

s313
00:31:22.399 --> 00:31:26.960
So the teacher is finding learnings in every iteration.

s314
00:31:26.960 --> 00:31:33.200
Uh maybe if I show the demo uh over here uh let's go to the one we just did.

s315
00:31:33.200 --> 00:31:38.880
Iteration one had these learnings and then iteration two had some other learnings.

s316
00:31:38.880 --> 00:31:39.360
Right?

s317
00:31:39.360 --> 00:31:43.679
you should include the explicit ISO date in this particular format.

s318
00:31:43.679 --> 00:31:45.760
Iteration three would have some other learnings.

s319
00:31:45.760 --> 00:31:54.799
The date resolution still opens the bare page and clicks wasting three turns uh reading today's date uh or having the executor compute today.

s320
00:31:54.799 --> 00:31:56.720
So it's figuring out something about the date time.

s321
00:31:56.720 --> 00:32:03.840
So in every iteration the teacher is learning something new and putting this back into the strategy.mmd.

s322
00:32:03.840 --> 00:32:07.919
Maybe I'll make this real by looking at the traces for one of these runs.

s323
00:32:07.919 --> 00:32:14.880
Uh so over here these are all my runs.

s324
00:32:14.880 --> 00:32:26.559
Uh so for example this was the open table reservation example u and uh this was the strategy.mmd that the teacher sort of appended to every time.

s325
00:32:26.559 --> 00:32:30.159
So uh step zero you should do this so on and so forth.

s326
00:32:30.159 --> 00:32:33.600
Uh this was the task this was the uh prompt that I gave it.

s327
00:32:33.600 --> 00:32:36.559
Find me reservations for the big B bigs restaurant.

s328
00:32:36.559 --> 00:32:40.240
uh and these were the kind of traces that uh it was sort of learning from.

s329
00:32:40.240 --> 00:32:48.720
It would take screenshots would take these sort of traces at every step is doing browse open this is what the output is.

s330
00:32:48.720 --> 00:32:51.120
Then turn number two browse snapshot.

s331
00:32:51.120 --> 00:32:55.360
Snapshot essentially takes the accessibility tree and this is the snapshot.

s332
00:32:55.360 --> 00:33:01.760
So it's reading this entire thing and at every step thinking of like what went right and what went wrong.

s333
00:33:01.760 --> 00:33:04.720
U yeah I mean this is like machine code.

s334
00:33:04.720 --> 00:33:06.960
uh I won't show this entire thing.

s335
00:33:06.960 --> 00:33:11.600
Uh but yeah, this is the summary that it also writes uh at every iteration.

s336
00:33:11.600 --> 00:33:18.240
Uh so yeah, this there's a lot lot of like engineering tweaks that you can do to suit this to your domain as well.

s337
00:33:18.240 --> 00:33:34.640
U and uh like the easiest way to uh read this is we put it out here on skills.sh as well uh in auto browse u and you can sort of do something like this.

s338
00:33:54.880 --> 00:33:57.200
Great question.

s339
00:34:03.360 --> 00:34:04.320
Such a good question.

s340
00:34:04.320 --> 00:34:12.960
I think that goes back to like how do you increase the reliability of your uh agents.

s341
00:34:12.960 --> 00:34:15.919
Uh reliability is an engineering problem.

s342
00:34:15.919 --> 00:34:25.919
Uh you can design your harness to sort of go for one skill for each task or you can be like hey one skill for every task that you will ever see.

s343
00:34:25.919 --> 00:34:37.280
Uh I think it entirely depends how you do the harness engineering entirely depends on the use case that you have u and you run evals with all these different strategies

s344
00:34:37.280 --> 00:34:39.119
uh and see which one works for you.

s345
00:34:39.119 --> 00:34:39.599
for us.

s346
00:34:39.599 --> 00:34:42.560
This was Yeah.

s347
00:34:55.919 --> 00:34:57.919
Yes.

s348
00:35:00.560 --> 00:35:01.280
Yeah.

s349
00:35:01.280 --> 00:35:01.599
Yeah.

s350
00:35:01.599 --> 00:35:05.920
And this is where again the harness comes into the picture.

s351
00:35:05.920 --> 00:35:21.920
If the harness has the intelligence to bring in skills from Open Table and from Resi and Google res Google reservations, maybe it can bring all of those in to figure out what is the best way to book me a date night.

s352
00:35:21.920 --> 00:35:23.200
That's your solution, right?

s353
00:35:23.200 --> 00:35:29.040
Like the harness should be able to uh figure out which tools to use to get the job done.

s354
00:35:29.040 --> 00:35:31.920
That's like how I would go about it.

s355
00:35:31.920 --> 00:35:37.839
uh figure out what are the most common use cases that your customers have for those use cases.

s356
00:35:37.839 --> 00:35:45.440
What's the best scale files, prompts, tools, maybe you want to do like specific tools for specific use cases, right?

s357
00:35:45.440 --> 00:35:48.079
Maybe you want to do generic tools for all use cases.

s358
00:35:48.079 --> 00:35:59.440
Like that's the same argument here is like do you do one skill for each or one skill for all uh and eval across all of these strategies and and let the hardness figure it out.

s359
00:36:01.440 --> 00:36:02.640
Yes.

s360
00:36:02.640 --> 00:36:03.359
Yes.

s361
00:36:03.359 --> 00:36:03.760
Yeah.

s362
00:36:03.760 --> 00:36:06.079
I mean it's a very similar concept as this, right?

s363
00:36:06.079 --> 00:36:08.880
Uh so DSPY.

s364
00:36:09.280 --> 00:36:09.760
Yeah.

s365
00:36:09.760 --> 00:36:19.119
I I think those guys were on to something really cool like they were around like 3 four years ago where they started building DSP and Japa all of that.

s366
00:36:19.119 --> 00:36:24.880
I think uh we have been doing a lot of research uh along these lines as well.

s367
00:36:24.880 --> 00:36:26.960
Browserbase works with uh Microsoft.

s368
00:36:26.960 --> 00:36:33.839
We actually released a paper uh about how to build the right sort of verifiers uh for your agents.

s369
00:36:33.839 --> 00:36:52.480
Uh I think running evals is straightforward until you start questioning okay how should I eval my eval right like do I need to put all these humans to verify all these runs or is there a good verifier that I can build here and all of that u uses the uh learnings from

s370
00:36:52.480 --> 00:36:54.400
uh these papers as well.

s371
00:36:54.400 --> 00:36:58.560
I can share all of that in the GitHub as well.

s372
00:37:00.400 --> 00:37:04.480
I'll also see if there's any new questions here.

s373
00:37:05.599 --> 00:37:08.960
Where and how were these skills stored retrieved at scale.

s374
00:37:08.960 --> 00:37:14.560
Uh right now for this repo uh all of these get stored in my cloud over here.

s375
00:37:14.560 --> 00:37:16.800
So each of these is a scale file.

s376
00:37:16.800 --> 00:37:18.400
This was my recipe searcher.

s377
00:37:18.400 --> 00:37:21.920
Uh I wanted to build some I wanted to cook some dinner yesterday.

s378
00:37:21.920 --> 00:37:24.720
So I was like, "Hey, make me some recipes."

s379
00:37:24.720 --> 00:37:30.480
Uh, so all of these are currently in my local uh skills folder.

s380
00:37:30.480 --> 00:37:38.079
Uh, but if you want to ship this uh in production, of course, you can uh bake this into your agent loop.

s381
00:37:38.079 --> 00:37:51.359
Uh, or if you want to I think for browse.sh the public uh publicly available uh skills that we have, I think we use versel blob storage to store all these skills uh and make them on demand.

s382
00:37:51.440 --> 00:37:52.960
Hope that helps.

s383
00:37:52.960 --> 00:37:55.680
Uh, websites change frequently.

s384
00:37:55.680 --> 00:38:00.480
What's your STR strategy for monitoring that and keeping the website skill updated?

s385
00:38:00.480 --> 00:38:02.720
Yeah, this is such a great question.

s386
00:38:02.720 --> 00:38:11.040
Uh, and this is what I was alluding to earlier is like how do you keep these skills up to date because the website's going to change.

s387
00:38:11.040 --> 00:38:20.000
The the current working theory that we have right now is you don't want to waste tokens running auto research every day.

s388
00:38:20.000 --> 00:38:26.960
you want to probably trigger or if it's failing at any particular point in production, right?

s389
00:38:26.960 --> 00:38:38.079
So, we build these loops or like systems where as soon as we run into any production failure, we run an auto research loop to see what went wrong to heal it,

s390
00:38:38.079 --> 00:38:44.960
create a pull request, that goes directly to a human engineer and they fix it automatically.

s391
00:38:44.960 --> 00:38:46.320
The skill gets updated.

s392
00:38:46.320 --> 00:38:51.599
Uh so yes there is that notion of like production failure when this might happen.

s393
00:38:51.599 --> 00:38:53.119
Uh but it's a trade-off.

s394
00:38:53.119 --> 00:39:03.119
You can run an auto research or you can run a sample run every 2 hours every 3 hours to see if the skill is working uh perfectly.

s395
00:39:03.119 --> 00:39:10.880
So there's different ways of having the uh cron jobs uh check for the accuracy of these skills.

s396
00:39:10.880 --> 00:39:15.040
This is actually one thing that I'm actively working on with the customer right now to implement.

s397
00:39:15.040 --> 00:39:16.800
And there's all these questions, right?

s398
00:39:16.800 --> 00:39:19.040
People don't want 100% accuracy.

s399
00:39:19.040 --> 00:39:20.720
People might want like 10x cheaper cost.

s400
00:39:20.720 --> 00:39:24.880
So like uh that's the uh question that comes up.

s401
00:39:24.880 --> 00:39:27.839
What's your teachers criteria here in the example?

s402
00:39:27.839 --> 00:39:32.000
Separate from reaching the goal seems to do some optimizations for turns.

s403
00:39:32.000 --> 00:39:33.680
What other criteria do you use?

s404
00:39:33.680 --> 00:39:39.359
Uh so again this is the design of the auto research loop.

s405
00:39:39.359 --> 00:39:45.920
Uh there's one way of telling your order research that hey I want to focus on just success.

s406
00:39:45.920 --> 00:39:49.440
As soon as you get me success I'm fine just give that to me.

s407
00:39:49.440 --> 00:39:52.880
Uh that's typically not what everybody wants to do.

s408
00:39:52.880 --> 00:39:56.480
Uh people want to minimize on uh latency.

s409
00:39:56.480 --> 00:40:02.320
Uh because maybe this is a real time run where you're focused on how quickly can you get the job done.

s410
00:40:02.320 --> 00:40:07.359
U so if that is the optimizing metric and you focus on getting it as fast as possible.

s411
00:40:07.359 --> 00:40:19.920
What I've seen sometimes these auto research loops are so smart they realize that browsers are not needed at all and they just reverse engineer the API and they fire the API and that's the fastest that you'll ever get to.

s412
00:40:19.920 --> 00:40:20.400
Right?

s413
00:40:20.400 --> 00:40:28.320
Uh so sometimes uh I've seen situations where uh it finds a solution that works in 500 milliseconds.

s414
00:40:28.320 --> 00:40:32.160
U sometimes it is the number of turns, right?

s415
00:40:32.160 --> 00:40:39.280
you want to minimize your context loop uh and you want to minimize the number of uh turns it takes to complete the task.

s416
00:40:39.280 --> 00:40:48.320
Uh in my instance, I think what I have done is it's your decision to pass or converge whenever you want to pause it.

s417
00:40:48.320 --> 00:40:50.640
I'm not really concerned about the costs.

s418
00:40:50.640 --> 00:40:53.839
Uh I care most about reliability.

s419
00:40:53.839 --> 00:41:03.599
So keep doing it as long as you feel it's relevant and then it's the teacher agent's job to uh call it day when it wants to.

s420
00:41:03.599 --> 00:41:07.920
But yeah, you can uh try all different strategies.

s421
00:41:07.920 --> 00:41:10.800
Uh if a website supports WebMP, will this use it?

s422
00:41:10.800 --> 00:41:12.400
WebMCP is so cool.

s423
00:41:12.400 --> 00:41:14.160
U it's essentially this concept.

s424
00:41:14.160 --> 00:41:21.760
I don't know if everybody knows this, but it's this concept of like making your website programmable.

s425
00:41:21.760 --> 00:41:34.240
It's it's come out in this new Google Chrome release recently where you tell your u website that if anybody wants to book a reservation they can do xyz on the website.

s426
00:41:34.240 --> 00:41:37.119
It's kind of like an API but it's on the browser.

s427
00:41:37.119 --> 00:41:41.680
Uh right now I don't think our oral research loop supports this.

s428
00:41:41.680 --> 00:41:47.839
Uh but I don't think it would be uh too hard to do that.

s429
00:41:47.839 --> 00:41:53.440
Uh in fact we recently shipped support for WebMCP with our SDKs that we launched.

s430
00:41:53.440 --> 00:41:58.480
Uh we also made some tools for website owners to ship web MCPS at scale.

s431
00:41:58.480 --> 00:42:00.720
Uh but yeah this is an interesting one.

s432
00:42:00.720 --> 00:42:05.280
We've not really seen a lot of adoption with Web MCP as yet.

s433
00:42:06.640 --> 00:42:09.760
Uh ground truth for pass.

s434
00:42:09.760 --> 00:42:10.400
That's interesting.

s435
00:42:10.400 --> 00:42:21.440
Uh I think as long as the job was done as long as for example in Google flights uh if it found the cheapest flights in open table if it found a reservation

s436
00:42:21.440 --> 00:42:24.319
uh so it it evaluates itself.

s437
00:42:24.319 --> 00:42:31.920
It's typically a binary uh and yeah again I I I fully embrace the bitter lesson.

s438
00:42:31.920 --> 00:42:38.160
I try not to put many guardrails in uh these evaluations.

s439
00:42:38.480 --> 00:42:47.359
Uh, have you considered a verifier as a service for browsing tasks, including something that allows us to mock certain side effects like purchases?

s440
00:42:47.359 --> 00:42:54.000
Uh, I might have to ask uh a followup on that.

s441
00:42:54.000 --> 00:42:56.240
What do you mean by verifier as a service?

s442
00:42:56.240 --> 00:42:57.760
Ravish, if you're here.

s443
00:42:57.760 --> 00:42:59.760
Yes.

s444
00:43:29.200 --> 00:43:44.800
Uh so uh do you mean that yeah if I understand this correctly like you have like different environments that you can use for RL training and you want to like mock what happens

s445
00:43:44.800 --> 00:44:00.079
if I am replicating Amazon if I click on purchase what happens after Uh no because uh we try to rely more on real world websites.

s446
00:44:00.079 --> 00:44:20.560
Uh browser base we have the privilege of being able to access most of the web with our uh access to different websites uh and capture solvers etc. Uh so we try to mimic the real world as much as possible but we've been seeing a lot of these RL environments where people are mimicking websites to train

s447
00:44:20.560 --> 00:44:28.079
uh and build browser agents that are uh really good at like browsing Salesforce, really good at like browsing SAP.

s448
00:44:28.079 --> 00:44:33.085
Uh we try to do that on real world portals as much as possible.

s449
00:44:33.085 --> 00:44:34.400
[snorts]

s450
00:44:34.400 --> 00:44:39.040
Are edge captures easy for the skill to move past or is it token incentive?

s451
00:44:39.040 --> 00:44:41.839
uh what is the point of edge captures for website owners?

s452
00:44:41.839 --> 00:44:45.839
Edge capture if I'm understanding this is like a particular kind of capture.

s453
00:44:45.839 --> 00:44:54.560
It's kind of like Google recapture etc. uh yeah I would probably talk about this separately.

s454
00:44:54.560 --> 00:44:58.800
Uh I don't think we can talk about like what it what captures we can solve or not.

s455
00:44:58.800 --> 00:45:06.640
Uh but it's not a token inensive job per se because it's an inf like it's thing that's done at the infrastructure layer, right?

s456
00:45:06.640 --> 00:45:12.079
uh we don't really need to use an AI to solve a capture.

s457
00:45:12.079 --> 00:45:14.880
Uh it's solved at the Chrome binary layer.

s458
00:45:14.880 --> 00:45:18.160
Uh and we do that out of the box.

s459
00:45:19.440 --> 00:45:23.280
How does the teacher evaluate the doer and when is the skill markers graduated?

s460
00:45:23.280 --> 00:45:23.920
Yes.

s461
00:45:23.920 --> 00:45:29.200
So uh it evaluates the doer by essentially reading these traces.

s462
00:45:29.200 --> 00:45:34.079
Uh for every iteration there is this folder that gets created.

s463
00:45:34.079 --> 00:45:36.560
uh there are these uh JSON files.

s464
00:45:36.560 --> 00:45:45.359
Uh it reads these uh it has a clawed agent SDK loop uh and it essentially figures out what the learnings from that loop are.

s465
00:45:45.359 --> 00:45:51.839
Uh actually I think I have a slide that talks about this at the end.

s466
00:45:53.359 --> 00:45:53.839
Yeah.

s467
00:45:53.839 --> 00:46:00.000
Uh this is kind of like the prompt for this uh where it says that you're a browser automation agent.

s468
00:46:00.000 --> 00:46:14.079
You can use these tools uh and the strategy.md gets injected at every iteration and this is how the uh uh doer the teacher evaluates the doer's job.

s469
00:46:14.079 --> 00:46:21.161
Uh so yeah it reads the traces it has the system prompt it has the previous strategy and it comes up with a better strategy.

s470
00:46:21.161 --> 00:46:23.161
[snorts]

s471
00:46:27.520 --> 00:46:28.800
Amazing.

s472
00:46:28.800 --> 00:46:34.079
Uh I hope we've covered all the questions.

s473
00:46:34.079 --> 00:46:39.680
Uh any other questions live that we might want to talk about?

s474
00:46:39.680 --> 00:46:41.680
Yes.

s475
00:46:44.720 --> 00:46:46.720
Yeah, it can work with the stage hand SDK.

s476
00:46:46.720 --> 00:46:48.480
It can work in your cloud code.

s477
00:46:48.480 --> 00:46:50.400
Uh it's essentially a markdown, right?

s478
00:46:50.400 --> 00:46:57.760
Uh I took uh a skill and I told my clot code, can you find events in SF for this Friday, July the 3rd?

s479
00:46:57.760 --> 00:46:59.119
and went on and did this.

s480
00:46:59.119 --> 00:47:01.680
So it it need not use browser base.

s481
00:47:01.680 --> 00:47:02.720
It need not use stage hand.

s482
00:47:02.720 --> 00:47:03.920
It is essentially a skill.

s483
00:47:03.920 --> 00:47:09.119
It is a portable artifact you can use in any agent or any SDK.

s484
00:47:30.400 --> 00:47:31.040
Yeah.

s485
00:47:31.040 --> 00:47:31.359
Yeah.

s486
00:47:31.359 --> 00:47:31.920
Great question.

s487
00:47:31.920 --> 00:47:36.960
I think if I understand this correctly, how do you prevent against prompt injection?

s488
00:47:36.960 --> 00:47:48.160
Uh because websites could trick my auto research loop in learning something bad that can do some malicious behavior in in a production setting.

s489
00:47:48.160 --> 00:47:51.280
Uh there's a few ways we do that.

s490
00:47:51.280 --> 00:47:53.680
One is at the infrastructure layer itself.

s491
00:47:53.680 --> 00:48:00.880
Uh we have a bunch of uh protections around our browsers itself.

s492
00:48:00.880 --> 00:48:07.359
Uh we have this browserbased safe mode where you can sort of control which URLs it can go to.

s493
00:48:07.359 --> 00:48:11.440
You can control what kind of actions it can take on different websites.

s494
00:48:11.440 --> 00:48:15.200
So at the infrastructure layer, at the network firewall, we control things.

s495
00:48:15.200 --> 00:48:19.760
Now you might say that it is possible that Amazon.com gets hacked someday.

s496
00:48:19.760 --> 00:48:23.680
How do you prevent against like malicious actions on those portals as well?

s497
00:48:23.680 --> 00:48:36.960
Uh I think we also try to uh run our skills that are generated against some security uh reviewer agent of sorts uh before we actually publish this on browser.sh.

s498
00:48:36.960 --> 00:48:39.520
Uh so that's like one way of doing it.

s499
00:48:39.520 --> 00:48:53.040
uh we encourage our customers to have uh a good level of uh prompt injection guardrail uh surface area uh if they're sort of giving this out to c consumers

s500
00:48:53.040 --> 00:49:11.680
uh for example some customers of ours uh might want to automate something on Amazon and what if somebody puts uh a listing on Amazon which has some hidden instructions and how do you prevent against those kind of things uh and that's where uh we work with our customers to build the right kind of uh guardrails

s501
00:49:11.680 --> 00:49:16.079
before the uh instruction even reaches the hardness.

s502
00:49:16.079 --> 00:49:19.680
Um so yeah I think I would put like three layers here.

s503
00:49:19.680 --> 00:49:32.000
One is as the as the query is even entering our surface area we control that at the skill layer we make sure that our skill is not uh tricked and at the infrastructure

s504
00:49:32.000 --> 00:49:35.520
layer we have network restrictions and action restrictions.

s505
00:49:35.520 --> 00:49:42.960
You can even do something like what codeex or plot code does u like always allow or allow once or allow any action.

s506
00:49:42.960 --> 00:49:47.200
Uh you can this is what like browserbased safe mode offers out of the box.

s507
00:49:47.200 --> 00:49:53.440
I know browser is a crazy threat vector when you think about it like it has access to anything on the wild web.

s508
00:49:53.440 --> 00:49:58.640
Uh and take that super seriously from a prompt injection POV.

s509
00:50:02.960 --> 00:50:03.839
Amazing.

s510
00:50:03.839 --> 00:50:08.720
I'll wait for 10 more seconds if there's any other questions.

s511
00:50:08.720 --> 00:50:09.440
This was fun.

s512
00:50:09.440 --> 00:50:10.720
I really enjoyed this.

s513
00:50:10.720 --> 00:50:13.359
I love the kind of questions as well.

s514
00:50:13.359 --> 00:50:22.480
Uh and yeah, I will put the QR code back on screen in case any of you want to grab this.

s515
00:50:22.480 --> 00:50:24.480
Yes.

s516
00:50:33.760 --> 00:50:35.760
Yeah.

s517
00:50:44.800 --> 00:50:45.760
Yeah.

s518
00:50:45.760 --> 00:50:46.000
Yeah.

s519
00:50:46.000 --> 00:50:46.400
Exactly.

s520
00:50:46.400 --> 00:50:46.559
Yeah.

s521
00:50:46.559 --> 00:50:48.960
I think we talked about it a while back.

s522
00:50:48.960 --> 00:50:52.960
What you could do is like I used uh entropic managed agents quite a bit.

s523
00:50:52.960 --> 00:51:07.280
Uh so I would run a chron job on managed agents where I would say use this skill every day at 9:00 am to do this job and if this is a success all good if this is a failure send me a notification here that will trigger a healing loop

s524
00:51:07.280 --> 00:51:24.240
uh so that's how I sort of can bake in uh checks and balances around websites changing antibbots changing etc. I'm going to hang around here.

s525
00:51:24.240 --> 00:51:29.920
Uh also uh I'll be at the browserbased booths all days of the conference as well.

s526
00:51:29.920 --> 00:51:34.800
So uh in case you're curious and want to chat more uh love to do that.

s527
00:51:34.800 --> 00:51:37.040
Thank you.

s528
00:51:50.630 --> 00:51:52.630
[music]