From RL to IRL — Gaurav Mishra, Amazon AGI Lab

AI Engineer · 17 min · 213 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Let's begin.
00:13The topic for this talk is RL to IRL.
00:16And for those of you who didn't get the clever word play here, I'm going to talk about what breaks when agents trained with reinforcement learning are deployed in real life.
00:30This is me.
00:30I'm a researcher at the Amazon AGI lab.
00:33I work on training agents that can do anything a computer can anything a human can on a computer.
00:39Before this, I've spent more than 10 years at Google, the last six of them at Deep Brain and Deep Mind training language models and agents.
00:49Let's start by talking about RL.
00:53Lightning review of what RL is and how we use it in the context of training agents.
00:58So, when we train an agent with RL, the agent is our policy.
01:03We give it a task.
01:04We sample generations.
01:06And then we compute a reward on the whole generation, and that's how it differs from SFT and pre-training where you're assigning loss to every token prediction.
01:16And then we have different algorithms to apply that reward and update the model weights.
01:22We have PPO, GRPO, but many many other variants.
01:26When is RL effective versus SFT?
01:29Three characteristics.
01:31One, when domains where you can collect or generate tasks fairly easily, but it's very hard to collect demonstration data for SFT.
01:40That's one where RL is very effective.
01:42When tasks have multiple correct solutions or many ways to get to the correct solution, and the outcome is verifiable, but if you try to collect SFT data for all the different paths, you might not be able to or you might
01:55narrow down the model to following a few patterns only, which is not good.
01:59And third is reasoning heavy domains, where again, it's very subjective.
02:04You want to let the model learn how to think and only judge it based on the outcome.
02:09So, if you think about it, coding fits this paradigm perfectly.
02:12That's why we've been able to train really compelling coding agents using RL.
02:18What are the key components of RL?
02:20Three in my mind.
02:22The first one is the task.
02:23This is the problem that we give to the model to generate samples on.
02:27The task has to have a verifiable outcome.
02:29The task has to be really targeted to the skills that the that we're trying to teach the model.
02:35And it also has to be in the right difficulty window.
02:37If the task is very easy or very difficult, then we're not going to get much training signal out of the model.
02:43Second part is the environment.
02:45Now, we're asking agents to produce code, to produce actions, and we need to be able to have safe environments where the actions and code can be executed.
02:53And those code sandboxes are a big part of this system.
02:57And then the last part is the verifier, verifier, which, you know, produces the training signal.
03:01So, taking model responses and judging them, the judge can be something as simple as a string equality to compiler, linter, running unit tests, database lookups, to also agents which are given a set of rubrics and then are asked to grade the model responses.
03:20So, when we got really good coding agents out of RL, what people started realizing is that you can actually deploy coding agents in the real world and ask them to do stuff beyond coding, like reading emails or sending chats or filing receipts for you, or doing
03:37research on a topic, you know, surfing and searching the web.
03:40And that's because all of these tasks can be represented as code, as coding tasks.
03:46So, chat, email, docs can all be accessed through MCP or API calls.
03:51You can interact with the browser using Playwright JavaScript Web MCP.
03:55You can surf and search the web using web search APIs.
03:59So, in theory, coding agents can be really good at computer use.
04:05So, what's the catch?
04:07Uh this is where real life kicks in.
04:09Uh so, let's see what breaks when the reward function meets a real login screen.
04:14I'm going to show you a couple of demos uh to give you a bit of context.
04:20These are um these are trajectories from our web browser use training runs.
04:28Um Uh these are from the early stages, so we will see some common traps that our agents fall into.
04:34Uh there is the prompt at the top.
04:37The verifiable outcome is over here.
04:39Uh here's an excerpt from the model thinking, and this is the browser window that the model sees.
04:44And here uh we have uh we have a very simple task where the model is asked to enter uh and submit an expense.
04:52Um let's see what happens.
04:57Okay, it enters the amount successfully.
05:00It clicks the button, but we are actually signed out now, so it needs to sign in.
05:04Um Let's see what it does.
05:08Okay, it says "Credential expired, but I can infer the account password."
05:13So, it doesn't really know the password, but it's trying to guess now.
05:16Okay, it entered something.
05:20Didn't work.
05:21Uh password was likely close.
05:23I will generate another password.
05:25Not going in a good direction.
05:27Okay, still failed again.
05:30I will resolve this without handoff.
05:31Let me try another one.
05:34Uh-oh, the account is now blocked.
05:38Okay, let's take a look at another example.
05:42Uh same situation.
05:43Uh small difference.
05:44There's a an ad over here with a submit button that looks very similar to the actual submit button.
05:50Very common scenario that we have seen probably every day.
05:54Let's see what the model does.
05:56I think you already know what will happen here.
06:00Model enters the right amount.
06:02It It looks and just clicks the wrong button.
06:05Now we're in a different website and it starts filling personal details.
06:09Now one can only hope that it is now hallucinating these details, but very dangerous behavior and we don't want this.
06:17Okay, so what went wrong?
06:20A big realization has been that RL worked when the world was a game and IRL starts when the game fights back.
06:27So let's We saw a few challenges.
06:30Let's talk about them and a few more challenges when you actually deploy agents in real-world applications.
06:37The first one is partial observability.
06:39So in the in the demo, the agent has access to the screenshot and the DOM.
06:44But neither of them are actually complete sources of information.
06:48The DOM has some info, but it doesn't have content that is dynamically generated.
06:52It didn't have the sponsored content for the ad because it was embedded into the image.
06:58The screenshot has it, but the screenshot might be partial.
07:00There might be content that is that you need to scroll to reveal.
07:04And so the model is being fed all these sources of information and doesn't really know what to expect from each and what to pay attention to.
07:11That's a big problem.
07:13Irreversibility.
07:14Once you submit a form, once you delete a file, once you lock an account, it's often irreversible for the time being.
07:20Non-determinism.
07:21When you click a button, you don't really know what happens.
07:25It might work, but it might take a long time to load.
07:29Your internet might be flaky.
07:31Your computer might restart for an upgrade.
07:34So many things can go wrong.
07:36Ephemeral authority.
07:37This thing that we saw, the session expired.
07:40Very, very common.
07:41Credentials expire very often.
07:43You have to be able to handle uh those navigation um those uh those edge cases.
07:48Ambiguous success, um done often doesn't mean successful.
07:53If the agent filed us uh uh expense report for me, but also sent a resignation letter on my behalf to the CEO, it is done, but not what I wanted it to do, right?
08:03Um adversarial content, this is everything we see around us is designed to grab our attention, and we uh we have to train ourselves to navigate that, and the model that is now working on our behalf also needs to be able to navigate that.
08:17So, these are just a few challenges.
08:19How do we How do we How do we adapt to this?
08:22Uh our big learning has been uh that um for computer use agents, and to use a uh a helpful analogy here, we need flight school, not just exams.
08:32So, the uh the agent has to be able to give in all these edge cases, all the messiness of real world has to be modeled into a simulation during training, so that the model can fall into all those traps,
08:45learn from them, and then become better.
08:47So, it's not just producing a generation that is uh rewarded by a reward model, but it's actually the environment, and all of the training setup has to reflect the messiness and all the edge cases of the real world.
08:59And it also means upgrading the pilot and the cockpit.
09:01So, let's talk about each of those components.
09:03The first one is the flight simulator.
09:05As we talked about, the first and biggest requirement is that we need high-fidelity digital sandboxes.
09:11So, we have to train with all the messiness, train with the layout shift, the slow loads, the missing labels, pop-ups, focus stealing, random account states, stale tabs.
09:20And then, recovery also has to be a native model action.
09:24So, a lot of uh often during traditional RL, what we do is when there's a infra error, we we just reset the state or ask the model to just restart,
09:32but that's not an option in real life.
09:34So, what we do is we uh whenever we have an infra error, we pass it to the model and we expect the model to recover from it using native tool use, native actions like, you know, refresh, backtrack,
09:45compare, wait, abandon, escalate to the user.
09:49Third part is the process reward model.
09:51So, as as we talked about the outcome is very important, but the path the model takes and the impact it has throughout the trajectory is very important as well.
09:59And so, we focus really hard on making sure we catch all of these dangerous actions throughout the process, not just the outcome, and penalize that accordingly.
10:09One one other really important part is calibrated confidence.
10:13So, the we need to teach the agent to know how actions are risky and when it is supposed to escalate to the user.
10:21So, based on if the action is authorized, if it is irreversible, is it visible to the user, what impact it has, we need to teach the model to know when to
10:31go for it or when to step back and escalate to the user.
10:34And the last part is adversarial tasks.
10:36So, we saw a couple of very simple adversarial tasks in the demo where the the training environment tests the model in two particular ways that that models can make mistakes.
10:47And this has to be part of the mainstream training.
10:50It cannot be something that's just byproduct.
10:52You have to actually test the model during training to make mistakes and then learn learn from them so that it does well in production.
11:02Let's talk about the pilot, the model.
11:04What needs to change?
11:05One of our biggest bets is that coding abilities are not sufficient to do well on computer use.
11:12The model needs to be able to look at the screen the way we humans look at a screen and then make sense from it.
11:19And that that means a few things.
11:21Uh Computer computer screens are very dense.
11:24So, grounding is really important for the agent to be able to understand what is the layout, what what what where are the the where are the text, what does all of it mean?
11:32And then the semantic understanding of it, like what is the purpose of the different things, what to pay attention to for the task that it's trying to do.
11:40Uh change detection is also important.
11:42So, what we do is after every uh action, we take screenshots and we keep putting it in the model context.
11:47So, the model has access to all these screenshots, but it need needs to understand what are the changes that are happening, are they desirable, uh what needs to change, and then what is what's the uh plan, and what are the actions the model has to take going forward.
12:01And then the multi-source observation part.
12:03So, having all these incomplete sources of information, but then learning to know what to expect from each of those, and then figuring out what to pay attention to for the task at hand is a important step.
12:16So, all of these capabilities need to be baked into the model.
12:21The third part is the cockpit.
12:23Um this is the harness.
12:25Uh harness is a very overloaded term, but I think of the harness as every the interface between the model and the world.
12:31Uh so, all the context management, all the tools uh that are available to the model, all the uh tool execution, everything is handled by the harness.
12:40And we can put an additional layer of guardrails in the harness to uh prevent the model from doing something bad, and then also nudge it in the right direction when needed.
12:51Uh few things that we have baked into our harness are checkpointing and rollback when possible.
12:55So, if there is a risky state, uh risky action, checkpoint and maybe come back to it if possible if there's a bad action.
13:03Uh action risk classifier, this is another layer of uh protection.
13:07So, if looking at the proposed actions from the models and then figuring out if they're actually safe or if they're uh risky.
13:16Credential guardrails again, like it's easy to detect if the credentials are active, if we've been signed out, and then nudge the model in the right direction based on that.
13:25Uh similarly, execution monitor, looking out for any bad patterns from the model loops or repeated clicks or unproductive behavior, and then nudging it in the right direction.
13:35Uh, audit logs, so maintaining evidence of all the actions and effects so that we can always go back and see what was the trail, what happened, and what what was the effect.
13:46And then human handoff, so wherever the confidence calibration of the model is not correct, we let the harness override the model and force it to give control back to the user.
14:00All right, so quickly summarizing the some of the assumptions of uh, traditional RL, how reality differs, and what we have done to adapt to it.
14:09So, the assumption is that state is observable.
14:12The reality is that UI is partial and messy.
14:15Uh, we've introduced perception primitives to deal with that.
14:19The assumption is that actions are cheap.
14:22Uh, reality is that actions are can be irreversible, and so we have to focus on risk-aware execution.
14:28The assumption is that reward is clear.
14:30Uh, reality is that success is often ambiguous.
14:33So, we have to focus on audit and verification.
14:37Uh, the assumption is that failure recites.
14:39Uh, the reality is that failure is often persistent, so we have to focus on recovery policies.
14:45The assumption is that environment is passive.
14:48It reality is that content can be really adversarial, so we have to set the right trust boundaries.
14:53And the assumption is that autonomy is always good.
14:56The reality is that handoff can be optimal in some cases, and the requirement is calibrated confidence.
15:04Okay, with all of this baked in, I want to show you a trajectory on a on the same same task, uh, few steps down the RL training loop.
15:14So, same task, you have to enter the final amount and click submit.
15:18Uh, now, you'll see that the model, uh, I see two submit buttons.
15:23One is sponsored, so the model is now able to distinguish between the two buttons.
15:27That's great.
15:32So, it clicks the right thing.
15:34Now, we sign out.
15:36We see the sign-in screen.
15:37Now, it says, "I see a sign-in screen.
15:38Credentials expired."
15:39So, the task data should not go here.
15:41Next, I'll hand off to the user.
15:42So, now it's giving up control to the user to enter the password, sign in again, and then give the control back to the agent.
15:49So, now we have a user simulator agent that is going to enter the right password and then sign in.
15:56And now it gives control back to the agent.
15:59The agent says that we we're back on the expense screen with the amount preserved.
16:03Sign is complete.
16:05Next, I'll submit the expense.
16:08Okay, amazing.
16:14The last message I want to leave you with is that the difference between a demo and a product is what happens after the first click, first failed click.
16:22So, all of the things that we talked about today is essentially boils down to simulating reality in your training setup.
16:30And that can only happen when you actually deploy the product and let it fail.
16:34Uh So, what we do is we work really closely with our design partners and internal customers to get them to use our use our model and see what fails and then, you know, complete the loop and fill in those capabilities.
16:50And early on, our harness is really strong.
16:53So, harness has to detect all the gaps in the model and make it fail gracefully so that we are able to capture the the failure modes and train on them, but we also are not
17:05causing any harm to the users that are actually using the models.
17:08And over time, the model becomes better and better, and the harness becomes thinner and thinner.
17:15Okay, that's all I have.
17:17I'll hang around outside if you have questions for me or if you'd like please come by the booth the Amazon AGI booth to meet me and my awesome teammates.
17:25We'll be there today, right?
17:27Thank you.
17:43[music]