WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Cc0_nyxROBA

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Cc0_nyxROBA.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.520 --> 00:00:13.760
Let's begin.

s3
00:00:13.760 --> 00:00:16.800
The topic for this talk is RL to IRL.

s4
00:00:16.800 --> 00:00:28.720
And for those of you who didn't get the clever word play here, I'm going to talk about what breaks when agents trained with reinforcement learning are deployed in real life.

s5
00:00:30.000 --> 00:00:30.920
This is me.

s6
00:00:30.920 --> 00:00:33.560
I'm a researcher at the Amazon AGI lab.

s7
00:00:33.560 --> 00:00:39.440
I work on training agents that can do anything a computer can anything a human can on a computer.

s8
00:00:39.440 --> 00:00:49.280
Before this, I've spent more than 10 years at Google, the last six of them at Deep Brain and Deep Mind training language models and agents.

s9
00:00:49.920 --> 00:00:53.240
Let's start by talking about RL.

s10
00:00:53.720 --> 00:00:58.120
Lightning review of what RL is and how we use it in the context of training agents.

s11
00:00:58.120 --> 00:01:03.800
So, when we train an agent with RL, the agent is our policy.

s12
00:01:03.800 --> 00:01:04.960
We give it a task.

s13
00:01:04.960 --> 00:01:06.600
We sample generations.

s14
00:01:06.600 --> 00:01:16.720
And then we compute a reward on the whole generation, and that's how it differs from SFT and pre-training where you're assigning loss to every token prediction.

s15
00:01:16.720 --> 00:01:22.720
And then we have different algorithms to apply that reward and update the model weights.

s16
00:01:22.720 --> 00:01:26.240
We have PPO, GRPO, but many many other variants.

s17
00:01:26.240 --> 00:01:29.800
When is RL effective versus SFT?

s18
00:01:29.800 --> 00:01:31.040
Three characteristics.

s19
00:01:31.040 --> 00:01:40.120
One, when domains where you can collect or generate tasks fairly easily, but it's very hard to collect demonstration data for SFT.

s20
00:01:40.120 --> 00:01:42.480
That's one where RL is very effective.

s21
00:01:42.480 --> 00:01:55.880
When tasks have multiple correct solutions or many ways to get to the correct solution, and the outcome is verifiable, but if you try to collect SFT data for all the different paths, you might not be able to or you might

s22
00:01:55.880 --> 00:01:59.960
narrow down the model to following a few patterns only, which is not good.

s23
00:01:59.960 --> 00:02:04.040
And third is reasoning heavy domains, where again, it's very subjective.

s24
00:02:04.040 --> 00:02:09.800
You want to let the model learn how to think and only judge it based on the outcome.

s25
00:02:09.800 --> 00:02:12.520
So, if you think about it, coding fits this paradigm perfectly.

s26
00:02:12.520 --> 00:02:18.959
That's why we've been able to train really compelling coding agents using RL.

s27
00:02:18.959 --> 00:02:20.840
What are the key components of RL?

s28
00:02:20.840 --> 00:02:22.000
Three in my mind.

s29
00:02:22.000 --> 00:02:23.360
The first one is the task.

s30
00:02:23.360 --> 00:02:27.400
This is the problem that we give to the model to generate samples on.

s31
00:02:27.400 --> 00:02:29.920
The task has to have a verifiable outcome.

s32
00:02:29.920 --> 00:02:35.560
The task has to be really targeted to the skills that the that we're trying to teach the model.

s33
00:02:35.560 --> 00:02:37.800
And it also has to be in the right difficulty window.

s34
00:02:37.800 --> 00:02:43.720
If the task is very easy or very difficult, then we're not going to get much training signal out of the model.

s35
00:02:43.720 --> 00:02:45.040
Second part is the environment.

s36
00:02:45.040 --> 00:02:53.600
Now, we're asking agents to produce code, to produce actions, and we need to be able to have safe environments where the actions and code can be executed.

s37
00:02:53.600 --> 00:02:57.160
And those code sandboxes are a big part of this system.

s38
00:02:57.160 --> 00:03:01.800
And then the last part is the verifier, verifier, which, you know, produces the training signal.

s39
00:03:01.800 --> 00:03:19.720
So, taking model responses and judging them, the judge can be something as simple as a string equality to compiler, linter, running unit tests, database lookups, to also agents which are given a set of rubrics and then are asked to grade the model responses.

s40
00:03:20.640 --> 00:03:37.080
So, when we got really good coding agents out of RL, what people started realizing is that you can actually deploy coding agents in the real world and ask them to do stuff beyond coding, like reading emails or sending chats or filing receipts for you, or doing

s41
00:03:37.080 --> 00:03:40.680
research on a topic, you know, surfing and searching the web.

s42
00:03:40.680 --> 00:03:46.320
And that's because all of these tasks can be represented as code, as coding tasks.

s43
00:03:46.320 --> 00:03:51.640
So, chat, email, docs can all be accessed through MCP or API calls.

s44
00:03:51.640 --> 00:03:55.880
You can interact with the browser using Playwright JavaScript Web MCP.

s45
00:03:55.880 --> 00:03:59.360
You can surf and search the web using web search APIs.

s46
00:03:59.360 --> 00:04:04.960
So, in theory, coding agents can be really good at computer use.

s47
00:04:05.080 --> 00:04:07.560
So, what's the catch?

s48
00:04:07.760 --> 00:04:09.840
Uh this is where real life kicks in.

s49
00:04:09.840 --> 00:04:14.640
Uh so, let's see what breaks when the reward function meets a real login screen.

s50
00:04:14.640 --> 00:04:20.440
I'm going to show you a couple of demos uh to give you a bit of context.

s51
00:04:20.440 --> 00:04:28.120
These are um these are trajectories from our web browser use training runs.

s52
00:04:28.120 --> 00:04:34.600
Um Uh these are from the early stages, so we will see some common traps that our agents fall into.

s53
00:04:34.600 --> 00:04:37.080
Uh there is the prompt at the top.

s54
00:04:37.080 --> 00:04:39.360
The verifiable outcome is over here.

s55
00:04:39.360 --> 00:04:44.560
Uh here's an excerpt from the model thinking, and this is the browser window that the model sees.

s56
00:04:44.560 --> 00:04:52.680
And here uh we have uh we have a very simple task where the model is asked to enter uh and submit an expense.

s57
00:04:52.680 --> 00:04:56.080
Um let's see what happens.

s58
00:04:57.400 --> 00:05:00.400
Okay, it enters the amount successfully.

s59
00:05:00.400 --> 00:05:04.880
It clicks the button, but we are actually signed out now, so it needs to sign in.

s60
00:05:04.880 --> 00:05:08.120
Um Let's see what it does.

s61
00:05:08.120 --> 00:05:13.000
Okay, it says "Credential expired, but I can infer the account password."

s62
00:05:13.000 --> 00:05:16.080
So, it doesn't really know the password, but it's trying to guess now.

s63
00:05:16.080 --> 00:05:19.080
Okay, it entered something.

s64
00:05:20.080 --> 00:05:21.360
Didn't work.

s65
00:05:21.360 --> 00:05:23.160
Uh password was likely close.

s66
00:05:23.160 --> 00:05:25.240
I will generate another password.

s67
00:05:25.240 --> 00:05:27.880
Not going in a good direction.

s68
00:05:27.880 --> 00:05:30.280
Okay, still failed again.

s69
00:05:30.280 --> 00:05:31.919
I will resolve this without handoff.

s70
00:05:31.919 --> 00:05:34.720
Let me try another one.

s71
00:05:34.720 --> 00:05:38.040
Uh-oh, the account is now blocked.

s72
00:05:38.200 --> 00:05:41.440
Okay, let's take a look at another example.

s73
00:05:42.000 --> 00:05:43.320
Uh same situation.

s74
00:05:43.320 --> 00:05:44.760
Uh small difference.

s75
00:05:44.760 --> 00:05:50.680
There's a an ad over here with a submit button that looks very similar to the actual submit button.

s76
00:05:50.680 --> 00:05:54.040
Very common scenario that we have seen probably every day.

s77
00:05:54.040 --> 00:05:56.680
Let's see what the model does.

s78
00:05:56.680 --> 00:05:59.960
I think you already know what will happen here.

s79
00:06:00.720 --> 00:06:02.560
Model enters the right amount.

s80
00:06:02.560 --> 00:06:05.400
It It looks and just clicks the wrong button.

s81
00:06:05.400 --> 00:06:09.680
Now we're in a different website and it starts filling personal details.

s82
00:06:09.680 --> 00:06:17.440
Now one can only hope that it is now hallucinating these details, but very dangerous behavior and we don't want this.

s83
00:06:17.440 --> 00:06:20.240
Okay, so what went wrong?

s84
00:06:20.240 --> 00:06:27.680
A big realization has been that RL worked when the world was a game and IRL starts when the game fights back.

s85
00:06:27.680 --> 00:06:30.480
So let's We saw a few challenges.

s86
00:06:30.480 --> 00:06:37.440
Let's talk about them and a few more challenges when you actually deploy agents in real-world applications.

s87
00:06:37.440 --> 00:06:39.200
The first one is partial observability.

s88
00:06:39.200 --> 00:06:44.440
So in the in the demo, the agent has access to the screenshot and the DOM.

s89
00:06:44.440 --> 00:06:48.040
But neither of them are actually complete sources of information.

s90
00:06:48.040 --> 00:06:52.800
The DOM has some info, but it doesn't have content that is dynamically generated.

s91
00:06:52.800 --> 00:06:58.240
It didn't have the sponsored content for the ad because it was embedded into the image.

s92
00:06:58.240 --> 00:07:00.960
The screenshot has it, but the screenshot might be partial.

s93
00:07:00.960 --> 00:07:04.320
There might be content that is that you need to scroll to reveal.

s94
00:07:04.320 --> 00:07:11.800
And so the model is being fed all these sources of information and doesn't really know what to expect from each and what to pay attention to.

s95
00:07:11.800 --> 00:07:13.280
That's a big problem.

s96
00:07:13.280 --> 00:07:14.520
Irreversibility.

s97
00:07:14.520 --> 00:07:20.919
Once you submit a form, once you delete a file, once you lock an account, it's often irreversible for the time being.

s98
00:07:20.919 --> 00:07:21.960
Non-determinism.

s99
00:07:21.960 --> 00:07:25.480
When you click a button, you don't really know what happens.

s100
00:07:25.480 --> 00:07:29.760
It might work, but it might take a long time to load.

s101
00:07:29.760 --> 00:07:31.280
Your internet might be flaky.

s102
00:07:31.280 --> 00:07:34.320
Your computer might restart for an upgrade.

s103
00:07:34.320 --> 00:07:36.280
So many things can go wrong.

s104
00:07:36.280 --> 00:07:37.800
Ephemeral authority.

s105
00:07:37.800 --> 00:07:40.040
This thing that we saw, the session expired.

s106
00:07:40.040 --> 00:07:41.320
Very, very common.

s107
00:07:41.320 --> 00:07:43.080
Credentials expire very often.

s108
00:07:43.080 --> 00:07:48.080
You have to be able to handle uh those navigation um those uh those edge cases.

s109
00:07:48.080 --> 00:07:53.040
Ambiguous success, um done often doesn't mean successful.

s110
00:07:53.040 --> 00:08:03.840
If the agent filed us uh uh expense report for me, but also sent a resignation letter on my behalf to the CEO, it is done, but not what I wanted it to do, right?

s111
00:08:03.840 --> 00:08:17.280
Um adversarial content, this is everything we see around us is designed to grab our attention, and we uh we have to train ourselves to navigate that, and the model that is now working on our behalf also needs to be able to navigate that.

s112
00:08:17.280 --> 00:08:19.560
So, these are just a few challenges.

s113
00:08:19.560 --> 00:08:22.800
How do we How do we How do we adapt to this?

s114
00:08:22.800 --> 00:08:32.640
Uh our big learning has been uh that um for computer use agents, and to use a uh a helpful analogy here, we need flight school, not just exams.

s115
00:08:32.640 --> 00:08:45.120
So, the uh the agent has to be able to give in all these edge cases, all the messiness of real world has to be modeled into a simulation during training, so that the model can fall into all those traps,

s116
00:08:45.120 --> 00:08:47.160
learn from them, and then become better.

s117
00:08:47.160 --> 00:08:59.040
So, it's not just producing a generation that is uh rewarded by a reward model, but it's actually the environment, and all of the training setup has to reflect the messiness and all the edge cases of the real world.

s118
00:08:59.040 --> 00:09:01.360
And it also means upgrading the pilot and the cockpit.

s119
00:09:01.360 --> 00:09:03.960
So, let's talk about each of those components.

s120
00:09:03.960 --> 00:09:05.760
The first one is the flight simulator.

s121
00:09:05.760 --> 00:09:11.600
As we talked about, the first and biggest requirement is that we need high-fidelity digital sandboxes.

s122
00:09:11.600 --> 00:09:20.720
So, we have to train with all the messiness, train with the layout shift, the slow loads, the missing labels, pop-ups, focus stealing, random account states, stale tabs.

s123
00:09:20.720 --> 00:09:24.200
And then, recovery also has to be a native model action.

s124
00:09:24.200 --> 00:09:32.920
So, a lot of uh often during traditional RL, what we do is when there's a infra error, we we just reset the state or ask the model to just restart,

s125
00:09:32.920 --> 00:09:34.360
but that's not an option in real life.

s126
00:09:34.360 --> 00:09:45.440
So, what we do is we uh whenever we have an infra error, we pass it to the model and we expect the model to recover from it using native tool use, native actions like, you know, refresh, backtrack,

s127
00:09:45.440 --> 00:09:49.000
compare, wait, abandon, escalate to the user.

s128
00:09:49.000 --> 00:09:51.400
Third part is the process reward model.

s129
00:09:51.400 --> 00:09:59.840
So, as as we talked about the outcome is very important, but the path the model takes and the impact it has throughout the trajectory is very important as well.

s130
00:09:59.840 --> 00:10:09.320
And so, we focus really hard on making sure we catch all of these dangerous actions throughout the process, not just the outcome, and penalize that accordingly.

s131
00:10:09.320 --> 00:10:13.240
One one other really important part is calibrated confidence.

s132
00:10:13.240 --> 00:10:21.880
So, the we need to teach the agent to know how actions are risky and when it is supposed to escalate to the user.

s133
00:10:21.880 --> 00:10:31.120
So, based on if the action is authorized, if it is irreversible, is it visible to the user, what impact it has, we need to teach the model to know when to

s134
00:10:31.120 --> 00:10:34.440
go for it or when to step back and escalate to the user.

s135
00:10:34.440 --> 00:10:36.080
And the last part is adversarial tasks.

s136
00:10:36.080 --> 00:10:47.920
So, we saw a couple of very simple adversarial tasks in the demo where the the training environment tests the model in two particular ways that that models can make mistakes.

s137
00:10:47.920 --> 00:10:50.600
And this has to be part of the mainstream training.

s138
00:10:50.600 --> 00:10:52.600
It cannot be something that's just byproduct.

s139
00:10:52.600 --> 00:11:01.839
You have to actually test the model during training to make mistakes and then learn learn from them so that it does well in production.

s140
00:11:02.440 --> 00:11:04.120
Let's talk about the pilot, the model.

s141
00:11:04.120 --> 00:11:05.400
What needs to change?

s142
00:11:05.400 --> 00:11:12.000
One of our biggest bets is that coding abilities are not sufficient to do well on computer use.

s143
00:11:12.000 --> 00:11:19.280
The model needs to be able to look at the screen the way we humans look at a screen and then make sense from it.

s144
00:11:19.280 --> 00:11:21.520
And that that means a few things.

s145
00:11:21.520 --> 00:11:24.040
Uh Computer computer screens are very dense.

s146
00:11:24.040 --> 00:11:32.480
So, grounding is really important for the agent to be able to understand what is the layout, what what what where are the the where are the text, what does all of it mean?

s147
00:11:32.480 --> 00:11:40.400
And then the semantic understanding of it, like what is the purpose of the different things, what to pay attention to for the task that it's trying to do.

s148
00:11:40.400 --> 00:11:42.080
Uh change detection is also important.

s149
00:11:42.080 --> 00:11:47.640
So, what we do is after every uh action, we take screenshots and we keep putting it in the model context.

s150
00:11:47.640 --> 00:12:01.800
So, the model has access to all these screenshots, but it need needs to understand what are the changes that are happening, are they desirable, uh what needs to change, and then what is what's the uh plan, and what are the actions the model has to take going forward.

s151
00:12:01.800 --> 00:12:03.640
And then the multi-source observation part.

s152
00:12:03.640 --> 00:12:16.160
So, having all these incomplete sources of information, but then learning to know what to expect from each of those, and then figuring out what to pay attention to for the task at hand is a important step.

s153
00:12:16.160 --> 00:12:20.800
So, all of these capabilities need to be baked into the model.

s154
00:12:21.480 --> 00:12:23.920
The third part is the cockpit.

s155
00:12:23.920 --> 00:12:25.600
Um this is the harness.

s156
00:12:25.600 --> 00:12:31.680
Uh harness is a very overloaded term, but I think of the harness as every the interface between the model and the world.

s157
00:12:31.680 --> 00:12:40.720
Uh so, all the context management, all the tools uh that are available to the model, all the uh tool execution, everything is handled by the harness.

s158
00:12:40.720 --> 00:12:51.360
And we can put an additional layer of guardrails in the harness to uh prevent the model from doing something bad, and then also nudge it in the right direction when needed.

s159
00:12:51.360 --> 00:12:55.839
Uh few things that we have baked into our harness are checkpointing and rollback when possible.

s160
00:12:55.839 --> 00:13:03.600
So, if there is a risky state, uh risky action, checkpoint and maybe come back to it if possible if there's a bad action.

s161
00:13:03.600 --> 00:13:07.839
Uh action risk classifier, this is another layer of uh protection.

s162
00:13:07.839 --> 00:13:16.839
So, if looking at the proposed actions from the models and then figuring out if they're actually safe or if they're uh risky.

s163
00:13:16.839 --> 00:13:25.880
Credential guardrails again, like it's easy to detect if the credentials are active, if we've been signed out, and then nudge the model in the right direction based on that.

s164
00:13:25.880 --> 00:13:35.760
Uh similarly, execution monitor, looking out for any bad patterns from the model loops or repeated clicks or unproductive behavior, and then nudging it in the right direction.

s165
00:13:35.760 --> 00:13:46.040
Uh, audit logs, so maintaining evidence of all the actions and effects so that we can always go back and see what was the trail, what happened, and what what was the effect.

s166
00:13:46.040 --> 00:13:58.640
And then human handoff, so wherever the confidence calibration of the model is not correct, we let the harness override the model and force it to give control back to the user.

s167
00:14:00.160 --> 00:14:09.680
All right, so quickly summarizing the some of the assumptions of uh, traditional RL, how reality differs, and what we have done to adapt to it.

s168
00:14:09.680 --> 00:14:12.280
So, the assumption is that state is observable.

s169
00:14:12.280 --> 00:14:15.520
The reality is that UI is partial and messy.

s170
00:14:15.520 --> 00:14:19.760
Uh, we've introduced perception primitives to deal with that.

s171
00:14:19.760 --> 00:14:22.400
The assumption is that actions are cheap.

s172
00:14:22.400 --> 00:14:28.640
Uh, reality is that actions are can be irreversible, and so we have to focus on risk-aware execution.

s173
00:14:28.640 --> 00:14:30.720
The assumption is that reward is clear.

s174
00:14:30.720 --> 00:14:33.440
Uh, reality is that success is often ambiguous.

s175
00:14:33.440 --> 00:14:37.240
So, we have to focus on audit and verification.

s176
00:14:37.240 --> 00:14:39.560
Uh, the assumption is that failure recites.

s177
00:14:39.560 --> 00:14:45.040
Uh, the reality is that failure is often persistent, so we have to focus on recovery policies.

s178
00:14:45.040 --> 00:14:48.040
The assumption is that environment is passive.

s179
00:14:48.040 --> 00:14:53.560
It reality is that content can be really adversarial, so we have to set the right trust boundaries.

s180
00:14:53.560 --> 00:14:56.320
And the assumption is that autonomy is always good.

s181
00:14:56.320 --> 00:15:03.839
The reality is that handoff can be optimal in some cases, and the requirement is calibrated confidence.

s182
00:15:04.200 --> 00:15:14.040
Okay, with all of this baked in, I want to show you a trajectory on a on the same same task, uh, few steps down the RL training loop.

s183
00:15:14.600 --> 00:15:18.320
So, same task, you have to enter the final amount and click submit.

s184
00:15:18.320 --> 00:15:23.840
Uh, now, you'll see that the model, uh, I see two submit buttons.

s185
00:15:23.840 --> 00:15:27.680
One is sponsored, so the model is now able to distinguish between the two buttons.

s186
00:15:27.680 --> 00:15:29.920
That's great.

s187
00:15:32.680 --> 00:15:34.680
So, it clicks the right thing.

s188
00:15:34.680 --> 00:15:36.280
Now, we sign out.

s189
00:15:36.280 --> 00:15:37.200
We see the sign-in screen.

s190
00:15:37.200 --> 00:15:38.680
Now, it says, "I see a sign-in screen.

s191
00:15:38.680 --> 00:15:39.800
Credentials expired."

s192
00:15:39.800 --> 00:15:41.440
So, the task data should not go here.

s193
00:15:41.440 --> 00:15:42.800
Next, I'll hand off to the user.

s194
00:15:42.800 --> 00:15:49.720
So, now it's giving up control to the user to enter the password, sign in again, and then give the control back to the agent.

s195
00:15:49.720 --> 00:15:56.840
So, now we have a user simulator agent that is going to enter the right password and then sign in.

s196
00:15:56.840 --> 00:15:59.400
And now it gives control back to the agent.

s197
00:15:59.400 --> 00:16:03.840
The agent says that we we're back on the expense screen with the amount preserved.

s198
00:16:03.840 --> 00:16:05.040
Sign is complete.

s199
00:16:05.040 --> 00:16:07.840
Next, I'll submit the expense.

s200
00:16:08.760 --> 00:16:11.120
Okay, amazing.

s201
00:16:14.400 --> 00:16:22.640
The last message I want to leave you with is that the difference between a demo and a product is what happens after the first click, first failed click.

s202
00:16:22.640 --> 00:16:30.760
So, all of the things that we talked about today is essentially boils down to simulating reality in your training setup.

s203
00:16:30.760 --> 00:16:34.880
And that can only happen when you actually deploy the product and let it fail.

s204
00:16:34.880 --> 00:16:50.680
Uh So, what we do is we work really closely with our design partners and internal customers to get them to use our use our model and see what fails and then, you know, complete the loop and fill in those capabilities.

s205
00:16:50.680 --> 00:16:53.160
And early on, our harness is really strong.

s206
00:16:53.160 --> 00:17:05.280
So, harness has to detect all the gaps in the model and make it fail gracefully so that we are able to capture the the failure modes and train on them, but we also are not

s207
00:17:05.280 --> 00:17:08.800
causing any harm to the users that are actually using the models.

s208
00:17:08.800 --> 00:17:14.959
And over time, the model becomes better and better, and the harness becomes thinner and thinner.

s209
00:17:15.400 --> 00:17:17.160
Okay, that's all I have.

s210
00:17:17.160 --> 00:17:25.280
I'll hang around outside if you have questions for me or if you'd like please come by the booth the Amazon AGI booth to meet me and my awesome teammates.

s211
00:17:25.280 --> 00:17:27.560
We'll be there today, right?

s212
00:17:27.560 --> 00:17:29.720
Thank you.

s213
00:17:43.732 --> 00:17:45.732
[music]