Perception Agents — Antje Barth, Amazon AGI Lab

AI Engineer · 21 min · 218 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Joining us on stage is a member of technical staff at Amazon AGI lab
00:17[music]
00:17onjab.
00:17bar.
00:36Good morning.
00:37It's so great to be back here at the AI Engineer Worlds Fair.
00:43Just a year ago, the hard problem was getting an agent to find a button and click it on a screen, especially screens it had never seen before.
00:57Now, agents can drive browsers and they're starting to also drive desktop apps.
01:06But what we figured out, click clicking was actually the easy part.
01:15What we didn't solve is the actual work.
01:20And what do I mean with this?
01:22Let's take a very simple example.
01:25A new team member starts on Monday.
01:28And maybe your job is to set up their accounts, add them to your Slack channel, book intros with colleagues, order the laptops, etc. And nobody really owns this end to end process in the company
01:46and it might be also touching five different systems.
01:51Now, agents can most likely perform each single individual individual step of this workflow, but agents still struggle to do this end to end because the real work lives within the seams of all of those different applications, of all of those different steps you have to take.
02:12And this is mostly where it all falls apart.
02:17The agent can use every single tool you give it, but it still can't do the full work.
02:26So why do we see this gap?
02:30Think about for a minute what we actually build.
02:34We taught computers to use computers.
02:38So what do I mean with this?
02:40We started building out the basics.
02:43We taught them clicking, scrolling, typing, calling an API, filling out a form, and we got those steps, these steps really reliable, and you can string them together in a workflow.
02:57And agents these days are fairly good at like operating those workflows.
03:03So, why can't you not just hand them more of your work and then literally just walk away and trust it to be completed?
03:14So all the things I talked about like using a tool models itself, tool use, stringing agents together, this is all capabilities and we mostly figured out how to add capabilities
03:31to models.
03:32Now the next hard part is really reliability and without reliability we cannot really build up trust in those systems.
03:45So here's a quick gut check and maybe all of you can just think about an agent doing work in an end toend workflow.
03:54How often do you think that actually succeeds these days?
03:59Maybe 60 maybe 80% of the time.
04:03And it sounds really fine, but if you look into this, if your agent one in four times deletes a database, you will never touch that agent again, right?
04:19So when you need this reliability, you really need to be it in the nines.
04:27You need to have the trust that it actually can do the work successfully.
04:36Now there's actually one place where we made enormous progress on reliability and trust and this is coding right.
04:47Think about how fast coding evolved.
04:52I still remember the first time when it started autocompleting for you, right?
04:56You just tapped autocomplete.
04:58Amazing.
05:00Then short time later, it started to write functions.
05:04And we thought that is amazing.
05:07And now look at these days.
05:09Coding agents write the code.
05:12They open up the pull requests themselves.
05:14And we had it earlier this week.
05:16Code keeps flying by.
05:19So once in a time we were able to just every single line that it generated we felt like the urge we need to really read it and make sure it's correct right I think most in the audience here can still relate to that
05:33these days I think hardly anyone is still doing that like we cannot even do that right code is generated at such a pace at the same time coding made that jump so why is that
05:47because we were able to bring it from just being capable the coding agents to actually be reliable and then trusted.
05:59So why is that?
06:00Why was coding first solved?
06:04It's because code is verifiable.
06:08You can run it, you can test it, you can check it and you can be for sure that it worked.
06:16So reliability showed up in the first place you can actually verify the answer.
06:27But here's the catch.
06:30Most of the work we do if you look at the broader knowledge work areas is not like that.
06:37Knowledge work is messy and heck the whole real world is really messy.
06:44Did the report I created land?
06:47Is the design on brand?
06:50Did it get it what I actually meant?
06:53So there is no unit test that can answer those questions.
06:59So verification really hits the wall right where most of our work lives.
07:05It's living in the seams of all of those applications we're using on a day-by-day basis.
07:11And nobody really has corrected this part yet.
07:15How do you make an agent reliable when there's no way to verify the answer that easily?
07:23And that's a field that is still wide open.
07:29So, how can we solve this?
07:32Well, so how do humans handle messy work?
07:35I mean, we're successful at it, right?
07:37Each of us like every day we work across different systems.
07:41We manage out how to onboard a new colleague.
07:44We do this.
07:47Well, we're doing it by figuring things out together.
07:52You grab a colleague, you jump on a Zoom meeting, you're discussing things, you're looking at the problem to solve, you're discussing p pointing at systems, and maybe two minutes later, you solved it.
08:06You're done.
08:07But none of this work is actually directly verifiable.
08:12And we do this all day.
08:15So one of the things is we're looking mostly at the same screen, right?
08:19If you're jumping on a meeting with a colleague, you see the same screen, both of you, and you can actually like figure out really quickly what needs to be done.
08:31So this is what the agent these days is missing.
08:35You don't necessarily need a bigger brain.
08:38What you need is this shared context.
08:41Because if we're looking the agent and myself at the same screen, I probably have much less explanating to do.
08:53So what kind of agent do we really need to build to achieve this?
08:59And today's agent, as I said, they can already see a screen, right?
09:03and they can click and take actions in it.
09:06That part works.
09:08But if they fire off actions, what they usually do, they move on.
09:13They don't watch what happens or recover if one step didn't succeed or something goes sideways.
09:22And we need an agent that can actually work like you do, like humans work.
09:28And one example is robotics.
09:31If you just look for a moment as how robotics do it, a robot perceives what's around it and it plans what to do and then acts.
09:42So this loop here from perceiving to planning to acting, this is actually what we also would need on a screen.
09:53And it starts here really with the first word which is perceive.
09:58The agent has to take in the screen the way you do, not scrape the code behind the page, but what's actually rendered, the layout, the state, what just changed
10:12the work, what we're doing, and then do it.
10:20And it would also have to keep up in real time.
10:24Think about how we as humans work together.
10:28You jump in, you react to build on top of what each other you say.
10:35And today agents can still don't do it.
10:39What we're doing is we're sending a prompt.
10:41We're waiting.
10:42It goes away and at one point the agent come back and we might have to take a couple of turns, right?
10:47Because what the agent come back with is not exactly what we might want to do.
10:51So we're sending another prompt say, "Hey, go back do this, do this differently."
10:55And we have this long back and forth which we got so used to from our chatbot experience and from this rhythm taking turns.
11:05But what we actually would need, think about it, is an agent that can react while you're still working.
11:13Wouldn't that be really cool, right?
11:15Like at the same time you're working, it can also come up with suggestions, can help you, and there is no waiting time.
11:25So basically an agent that perceives what you perceive and understands what you mean.
11:36We call them perception agents.
11:42So why perception agents?
11:44Why do they matter?
11:46So first they complete the loop on computer use.
11:52Today's agents again they can act, they can click, they can type, they can scroll, but what they can't do well is looking at the results and whether it actually worked out.
12:06A perception agent can read the rendered screen so it can confirm its own output instead of just firing off those actions and then hoping.
12:19Second, it doesn't need an API or backend process.
12:24And that's important because it works off the rendered interface.
12:28It sees the same pixels and the structure you see.
12:32And most of today's software people use every day don't expose APIs at all.
12:41And then third, the input also goes the other way here.
12:45Instead of writing a long paragraph to describe what you want to change, let's say you're working on a website and you want to describe all the changes you want to apply.
12:56Instead of writing this really long description, wouldn't it be great if you can just point to it and say, "Hey, here this heading needs to change.
13:05Hey, can you update this section?"
13:09This is a much more precise signal and less lossy than text.
13:15and the agent can act exactly on what you marked.
13:23So this is where we started and I'm happy to share that we just recently launched the first two pieces of our perception agent harness open source.
13:38There's two pieces.
13:39There is annotation which you can use to tell it what you want.
13:45And then the second piece, the verification part gives the agent the capability to check its own work.
13:56So let me show you the first one.
13:59So here's a very quick demo on our annotation tool.
14:05This one is a Chrome extension, so it's super easy to use.
14:09And I'm going to play here this quick video demo.
14:13So you have the extension installed and then you can just select different elements on a screen.
14:20So this example we're just drawing around the heading there marking the section.
14:25And maybe you want to change it.
14:26Why not?
14:27Let's change it to red.
14:30You could also select the elements on this page.
14:33You see how if I hover over it finds the right element.
14:36You click it, you select it and say something maybe double the font size.
14:41And you see also how the agent here captures on the screen exactly the feedback, the location, the style elements and it creates this complete summary which you can then use
14:53and then give your agent to implement.
14:55So there is no back and forth anymore because you captured exactly what you saw on screen and the agent can see the same thing.
15:07Now let's have a very brief look at the second one at verification.
15:14So the idea of verification is that you can describe let's stay in this case of the web development.
15:22You can describe in a design MD file what your design rules are for this.
15:29And then what happens if I play this video here, the act the agent can actually check its own work against those design specs.
15:38So it will take what you defined, the colors, the components, your layout, and it turns it into those rules if you don't have it written before yet.
15:48And it does two kinds of checks.
15:50Then it does a visual check, which is really cool.
15:54So everything is on brand, for example.
15:57it's the right layout.
16:00The other part is also checking user flows.
16:04So what it does there, it actually walks through this experience through the app for example depending on the tasks available.
16:12It might add a task, it might delete a task like a real user would.
16:17So it helps you walk through those user flows as well in an automated fashion.
16:23And then once it's done, it's writing a report which you can review and it's going to call out which tests passed and it's going to tell you anything that didn't.
16:34So ultimately, you're the one that doesn't have to click through this at midnight at the end of the day because great work.
16:43The agent already did this job for you.
16:48Now there might not always be a screen, right?
16:53So I talked a lot right now.
16:54I called it perception.
16:56I talked about the agent sees what you see on a screen.
17:00But there are times in your day where you don't have a screen.
17:05Maybe you're in the office.
17:07You're walking into a meeting with a colleague.
17:11So I did a fun experiment yesterday at the conference here.
17:16So I grabbed my colleague Giovanni who is also here and actually on the second floor there's a great like little meeting booth.
17:22We found that by coincidence.
17:24So we went in there and we had our design meeting.
17:28And the goal here is really kind of show you how perception is so much more than just the visual part.
17:35So in this example, what we want to show you is perception can also be listening in the room to what you're discussing.
17:43And you can see here on the picture, both of us are wearing our B devices.
17:48Big shout out to B for sponsoring these.
17:51Um, so we're sitting there.
17:53We have our B devices that can do a transcript.
17:55They're listening to what we're saying.
17:57And then we have this design meeting and I had a couple of great ideas how to change this website.
18:03Um, you will see them in a in a second here.
18:06So let's have a quick look how this changed the same workflow on this website using this device.
18:15So we had the discussion the be did the transcript and you can see here on the right we're pulling this meeting transcript right in there is a whole detailed summary of the meeting.
18:29There is what we discussed and then it basically captures those insights.
18:35We have them right here and we can click apply.
18:39So what this apply button does is it sends it straight to the agent.
18:43And you can see here my crazy ideas to turn the background to yellow, turn the heading to red, and also change an emoji directly applied.
18:53And it also straight kicks off the verification right away.
18:56So it creates this report and and luckily this color scheme was apparently into in the approved rules.
19:03Otherwise, this would have liked like you did some weird things here.
19:07But again, you could change those rules if you don't want to have yellow backgrounds and it will make sure um that we still adhere to those guidelines.
19:15It would flag anything that's off.
19:17So, you have the judgment call if you want to either update the design specs because you actually like yellow or you take an action and say, "No, um fix this violation."
19:30But this is really the very first step.
19:34These two pieces are the very first beginning.
19:36And we're building out the rest in the open because these patterns can only get better if more people are using them, building on top of them, breaking things.
19:48So my ask here to you is go and try them out.
19:52They're on our GitHub repos, open source.
19:55Tell us what we're missing.
19:57give us the feedback what you would like to see where this should go next because ultimately none of us get smart alone and that's the whole point.
20:08We want to build AI that makes all of us smarter together.
20:16Now, if you're interested in a little bit more on human agent interactions and how we see those patterns changing, I would highly recommend this podcast by my colleague Danielle
20:28Persik.
20:29She is a cognitive scientist and runs our AGI ACI team at the lab and discusses a lot about human computer interaction patterns with experts in the industry.
20:41You can find the podcast on on any popular podcast platform.
20:46We also have more sessions this week.
20:49Um, so check them out.
20:50We have a booth down there.
20:51We have expert talks.
20:53We also have another computer use track talk coming up with my colleague Gaf Mishra at 1:30 in the computer use track.
21:02Highly recommend checking out his talk from RL to IRL.
21:08And then ultimately come find us.
21:09We have a huge presence down at the expo hall.
21:12We would love to continue the conversation with you all.
21:16If you're not here in person, you can also check out our code on our GitHub repo and check out our website.
21:24And with that, thank you very much.