WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=2JX6JYyQG4Y NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/2JX6JYyQG4Y.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.639 --> 00:00:17.030 Joining us on stage is a member of technical staff at Amazon AGI lab s3 00:00:17.030 --> 00:00:17.440 [music] s4 00:00:17.440 --> 00:00:17.840 onjab. s5 00:00:17.840 --> 00:00:19.840 bar. s6 00:00:36.320 --> 00:00:37.520 Good morning. s7 00:00:37.520 --> 00:00:42.800 It's so great to be back here at the AI Engineer Worlds Fair. s8 00:00:43.760 --> 00:00:57.440 Just a year ago, the hard problem was getting an agent to find a button and click it on a screen, especially screens it had never seen before. s9 00:00:57.440 --> 00:01:06.479 Now, agents can drive browsers and they're starting to also drive desktop apps. s10 00:01:06.640 --> 00:01:14.240 But what we figured out, click clicking was actually the easy part. s11 00:01:15.920 --> 00:01:20.720 What we didn't solve is the actual work. s12 00:01:20.720 --> 00:01:22.560 And what do I mean with this? s13 00:01:22.560 --> 00:01:25.280 Let's take a very simple example. s14 00:01:25.280 --> 00:01:28.479 A new team member starts on Monday. s15 00:01:28.479 --> 00:01:46.159 And maybe your job is to set up their accounts, add them to your Slack channel, book intros with colleagues, order the laptops, etc. And nobody really owns this end to end process in the company s16 00:01:46.159 --> 00:01:50.799 and it might be also touching five different systems. s17 00:01:51.200 --> 00:02:12.879 Now, agents can most likely perform each single individual individual step of this workflow, but agents still struggle to do this end to end because the real work lives within the seams of all of those different applications, of all of those different steps you have to take. s18 00:02:12.879 --> 00:02:17.440 And this is mostly where it all falls apart. s19 00:02:17.440 --> 00:02:25.680 The agent can use every single tool you give it, but it still can't do the full work. s20 00:02:26.800 --> 00:02:30.239 So why do we see this gap? s21 00:02:30.400 --> 00:02:34.160 Think about for a minute what we actually build. s22 00:02:34.160 --> 00:02:38.239 We taught computers to use computers. s23 00:02:38.239 --> 00:02:40.640 So what do I mean with this? s24 00:02:40.640 --> 00:02:43.040 We started building out the basics. s25 00:02:43.040 --> 00:02:57.280 We taught them clicking, scrolling, typing, calling an API, filling out a form, and we got those steps, these steps really reliable, and you can string them together in a workflow. s26 00:02:57.280 --> 00:03:03.440 And agents these days are fairly good at like operating those workflows. s27 00:03:03.760 --> 00:03:12.959 So, why can't you not just hand them more of your work and then literally just walk away and trust it to be completed? s28 00:03:14.879 --> 00:03:31.120 So all the things I talked about like using a tool models itself, tool use, stringing agents together, this is all capabilities and we mostly figured out how to add capabilities s29 00:03:31.120 --> 00:03:32.959 to models. s30 00:03:32.959 --> 00:03:44.640 Now the next hard part is really reliability and without reliability we cannot really build up trust in those systems. s31 00:03:45.200 --> 00:03:54.720 So here's a quick gut check and maybe all of you can just think about an agent doing work in an end toend workflow. s32 00:03:54.720 --> 00:03:59.519 How often do you think that actually succeeds these days? s33 00:03:59.519 --> 00:04:03.760 Maybe 60 maybe 80% of the time. s34 00:04:03.760 --> 00:04:19.440 And it sounds really fine, but if you look into this, if your agent one in four times deletes a database, you will never touch that agent again, right? s35 00:04:19.440 --> 00:04:27.360 So when you need this reliability, you really need to be it in the nines. s36 00:04:27.360 --> 00:04:33.120 You need to have the trust that it actually can do the work successfully. s37 00:04:36.000 --> 00:04:47.360 Now there's actually one place where we made enormous progress on reliability and trust and this is coding right. s38 00:04:47.360 --> 00:04:52.080 Think about how fast coding evolved. s39 00:04:52.080 --> 00:04:56.880 I still remember the first time when it started autocompleting for you, right? s40 00:04:56.880 --> 00:04:58.400 You just tapped autocomplete. s41 00:04:58.400 --> 00:05:00.160 Amazing. s42 00:05:00.160 --> 00:05:04.560 Then short time later, it started to write functions. s43 00:05:04.560 --> 00:05:07.199 And we thought that is amazing. s44 00:05:07.199 --> 00:05:09.199 And now look at these days. s45 00:05:09.199 --> 00:05:12.160 Coding agents write the code. s46 00:05:12.160 --> 00:05:14.880 They open up the pull requests themselves. s47 00:05:14.880 --> 00:05:16.560 And we had it earlier this week. s48 00:05:16.560 --> 00:05:19.199 Code keeps flying by. s49 00:05:19.199 --> 00:05:33.360 So once in a time we were able to just every single line that it generated we felt like the urge we need to really read it and make sure it's correct right I think most in the audience here can still relate to that s50 00:05:33.360 --> 00:05:47.680 these days I think hardly anyone is still doing that like we cannot even do that right code is generated at such a pace at the same time coding made that jump so why is that s51 00:05:47.680 --> 00:05:58.240 because we were able to bring it from just being capable the coding agents to actually be reliable and then trusted. s52 00:05:59.680 --> 00:06:00.720 So why is that? s53 00:06:00.720 --> 00:06:04.800 Why was coding first solved? s54 00:06:04.800 --> 00:06:08.560 It's because code is verifiable. s55 00:06:08.560 --> 00:06:16.560 You can run it, you can test it, you can check it and you can be for sure that it worked. s56 00:06:16.560 --> 00:06:25.360 So reliability showed up in the first place you can actually verify the answer. s57 00:06:27.919 --> 00:06:30.639 But here's the catch. s58 00:06:30.720 --> 00:06:37.360 Most of the work we do if you look at the broader knowledge work areas is not like that. s59 00:06:37.360 --> 00:06:44.800 Knowledge work is messy and heck the whole real world is really messy. s60 00:06:44.800 --> 00:06:47.520 Did the report I created land? s61 00:06:47.520 --> 00:06:50.479 Is the design on brand? s62 00:06:50.479 --> 00:06:53.600 Did it get it what I actually meant? s63 00:06:53.600 --> 00:06:58.880 So there is no unit test that can answer those questions. s64 00:06:59.039 --> 00:07:05.599 So verification really hits the wall right where most of our work lives. s65 00:07:05.599 --> 00:07:11.599 It's living in the seams of all of those applications we're using on a day-by-day basis. s66 00:07:11.599 --> 00:07:15.199 And nobody really has corrected this part yet. s67 00:07:15.199 --> 00:07:23.199 How do you make an agent reliable when there's no way to verify the answer that easily? s68 00:07:23.199 --> 00:07:26.960 And that's a field that is still wide open. s69 00:07:29.440 --> 00:07:32.319 So, how can we solve this? s70 00:07:32.319 --> 00:07:35.840 Well, so how do humans handle messy work? s71 00:07:35.840 --> 00:07:37.919 I mean, we're successful at it, right? s72 00:07:37.919 --> 00:07:41.759 Each of us like every day we work across different systems. s73 00:07:41.759 --> 00:07:44.560 We manage out how to onboard a new colleague. s74 00:07:44.560 --> 00:07:46.960 We do this. s75 00:07:47.840 --> 00:07:52.880 Well, we're doing it by figuring things out together. s76 00:07:52.880 --> 00:08:06.000 You grab a colleague, you jump on a Zoom meeting, you're discussing things, you're looking at the problem to solve, you're discussing p pointing at systems, and maybe two minutes later, you solved it. s77 00:08:06.000 --> 00:08:07.440 You're done. s78 00:08:07.440 --> 00:08:12.240 But none of this work is actually directly verifiable. s79 00:08:12.720 --> 00:08:15.759 And we do this all day. s80 00:08:15.840 --> 00:08:19.919 So one of the things is we're looking mostly at the same screen, right? s81 00:08:19.919 --> 00:08:31.199 If you're jumping on a meeting with a colleague, you see the same screen, both of you, and you can actually like figure out really quickly what needs to be done. s82 00:08:31.199 --> 00:08:35.039 So this is what the agent these days is missing. s83 00:08:35.039 --> 00:08:38.479 You don't necessarily need a bigger brain. s84 00:08:38.479 --> 00:08:41.680 What you need is this shared context. s85 00:08:41.680 --> 00:08:51.200 Because if we're looking the agent and myself at the same screen, I probably have much less explanating to do. s86 00:08:53.600 --> 00:08:59.440 So what kind of agent do we really need to build to achieve this? s87 00:08:59.519 --> 00:09:03.440 And today's agent, as I said, they can already see a screen, right? s88 00:09:03.440 --> 00:09:06.240 and they can click and take actions in it. s89 00:09:06.240 --> 00:09:08.560 That part works. s90 00:09:08.560 --> 00:09:13.519 But if they fire off actions, what they usually do, they move on. s91 00:09:13.519 --> 00:09:22.000 They don't watch what happens or recover if one step didn't succeed or something goes sideways. s92 00:09:22.160 --> 00:09:28.959 And we need an agent that can actually work like you do, like humans work. s93 00:09:28.959 --> 00:09:31.360 And one example is robotics. s94 00:09:31.360 --> 00:09:42.880 If you just look for a moment as how robotics do it, a robot perceives what's around it and it plans what to do and then acts. s95 00:09:42.880 --> 00:09:53.200 So this loop here from perceiving to planning to acting, this is actually what we also would need on a screen. s96 00:09:53.279 --> 00:09:58.160 And it starts here really with the first word which is perceive. s97 00:09:58.160 --> 00:10:12.959 The agent has to take in the screen the way you do, not scrape the code behind the page, but what's actually rendered, the layout, the state, what just changed s98 00:10:12.959 --> 00:10:18.160 the work, what we're doing, and then do it. s99 00:10:20.320 --> 00:10:24.160 And it would also have to keep up in real time. s100 00:10:24.160 --> 00:10:28.240 Think about how we as humans work together. s101 00:10:28.240 --> 00:10:34.880 You jump in, you react to build on top of what each other you say. s102 00:10:35.360 --> 00:10:39.200 And today agents can still don't do it. s103 00:10:39.200 --> 00:10:41.680 What we're doing is we're sending a prompt. s104 00:10:41.680 --> 00:10:42.320 We're waiting. s105 00:10:42.320 --> 00:10:47.360 It goes away and at one point the agent come back and we might have to take a couple of turns, right? s106 00:10:47.360 --> 00:10:51.279 Because what the agent come back with is not exactly what we might want to do. s107 00:10:51.279 --> 00:10:55.519 So we're sending another prompt say, "Hey, go back do this, do this differently." s108 00:10:55.519 --> 00:11:05.360 And we have this long back and forth which we got so used to from our chatbot experience and from this rhythm taking turns. s109 00:11:05.360 --> 00:11:13.279 But what we actually would need, think about it, is an agent that can react while you're still working. s110 00:11:13.279 --> 00:11:15.519 Wouldn't that be really cool, right? s111 00:11:15.519 --> 00:11:25.440 Like at the same time you're working, it can also come up with suggestions, can help you, and there is no waiting time. s112 00:11:25.440 --> 00:11:34.480 So basically an agent that perceives what you perceive and understands what you mean. s113 00:11:36.399 --> 00:11:40.160 We call them perception agents. s114 00:11:42.800 --> 00:11:44.640 So why perception agents? s115 00:11:44.640 --> 00:11:46.640 Why do they matter? s116 00:11:46.640 --> 00:11:52.240 So first they complete the loop on computer use. s117 00:11:52.240 --> 00:12:06.240 Today's agents again they can act, they can click, they can type, they can scroll, but what they can't do well is looking at the results and whether it actually worked out. s118 00:12:06.240 --> 00:12:19.120 A perception agent can read the rendered screen so it can confirm its own output instead of just firing off those actions and then hoping. s119 00:12:19.600 --> 00:12:24.480 Second, it doesn't need an API or backend process. s120 00:12:24.639 --> 00:12:28.399 And that's important because it works off the rendered interface. s121 00:12:28.399 --> 00:12:32.800 It sees the same pixels and the structure you see. s122 00:12:32.800 --> 00:12:40.639 And most of today's software people use every day don't expose APIs at all. s123 00:12:41.040 --> 00:12:45.920 And then third, the input also goes the other way here. s124 00:12:45.920 --> 00:12:56.880 Instead of writing a long paragraph to describe what you want to change, let's say you're working on a website and you want to describe all the changes you want to apply. s125 00:12:56.880 --> 00:13:05.519 Instead of writing this really long description, wouldn't it be great if you can just point to it and say, "Hey, here this heading needs to change. s126 00:13:05.519 --> 00:13:09.120 Hey, can you update this section?" s127 00:13:09.440 --> 00:13:15.519 This is a much more precise signal and less lossy than text. s128 00:13:15.519 --> 00:13:21.120 and the agent can act exactly on what you marked. s129 00:13:23.600 --> 00:13:38.320 So this is where we started and I'm happy to share that we just recently launched the first two pieces of our perception agent harness open source. s130 00:13:38.320 --> 00:13:39.440 There's two pieces. s131 00:13:39.440 --> 00:13:45.279 There is annotation which you can use to tell it what you want. s132 00:13:45.279 --> 00:13:54.320 And then the second piece, the verification part gives the agent the capability to check its own work. s133 00:13:56.720 --> 00:13:59.199 So let me show you the first one. s134 00:13:59.199 --> 00:14:05.360 So here's a very quick demo on our annotation tool. s135 00:14:05.360 --> 00:14:09.440 This one is a Chrome extension, so it's super easy to use. s136 00:14:09.440 --> 00:14:13.440 And I'm going to play here this quick video demo. s137 00:14:13.440 --> 00:14:20.240 So you have the extension installed and then you can just select different elements on a screen. s138 00:14:20.240 --> 00:14:25.360 So this example we're just drawing around the heading there marking the section. s139 00:14:25.360 --> 00:14:26.720 And maybe you want to change it. s140 00:14:26.720 --> 00:14:27.279 Why not? s141 00:14:27.279 --> 00:14:30.320 Let's change it to red. s142 00:14:30.320 --> 00:14:33.120 You could also select the elements on this page. s143 00:14:33.120 --> 00:14:36.399 You see how if I hover over it finds the right element. s144 00:14:36.399 --> 00:14:41.040 You click it, you select it and say something maybe double the font size. s145 00:14:41.040 --> 00:14:53.120 And you see also how the agent here captures on the screen exactly the feedback, the location, the style elements and it creates this complete summary which you can then use s146 00:14:53.120 --> 00:14:55.920 and then give your agent to implement. s147 00:14:55.920 --> 00:15:05.839 So there is no back and forth anymore because you captured exactly what you saw on screen and the agent can see the same thing. s148 00:15:07.839 --> 00:15:14.240 Now let's have a very brief look at the second one at verification. s149 00:15:14.399 --> 00:15:22.160 So the idea of verification is that you can describe let's stay in this case of the web development. s150 00:15:22.160 --> 00:15:29.040 You can describe in a design MD file what your design rules are for this. s151 00:15:29.040 --> 00:15:38.800 And then what happens if I play this video here, the act the agent can actually check its own work against those design specs. s152 00:15:38.800 --> 00:15:48.240 So it will take what you defined, the colors, the components, your layout, and it turns it into those rules if you don't have it written before yet. s153 00:15:48.240 --> 00:15:50.639 And it does two kinds of checks. s154 00:15:50.639 --> 00:15:54.720 Then it does a visual check, which is really cool. s155 00:15:54.720 --> 00:15:57.519 So everything is on brand, for example. s156 00:15:57.519 --> 00:16:00.399 it's the right layout. s157 00:16:00.720 --> 00:16:04.399 The other part is also checking user flows. s158 00:16:04.399 --> 00:16:12.320 So what it does there, it actually walks through this experience through the app for example depending on the tasks available. s159 00:16:12.320 --> 00:16:17.759 It might add a task, it might delete a task like a real user would. s160 00:16:17.759 --> 00:16:23.360 So it helps you walk through those user flows as well in an automated fashion. s161 00:16:23.360 --> 00:16:34.880 And then once it's done, it's writing a report which you can review and it's going to call out which tests passed and it's going to tell you anything that didn't. s162 00:16:34.880 --> 00:16:43.120 So ultimately, you're the one that doesn't have to click through this at midnight at the end of the day because great work. s163 00:16:43.120 --> 00:16:46.959 The agent already did this job for you. s164 00:16:48.079 --> 00:16:53.360 Now there might not always be a screen, right? s165 00:16:53.360 --> 00:16:54.639 So I talked a lot right now. s166 00:16:54.639 --> 00:16:56.160 I called it perception. s167 00:16:56.160 --> 00:17:00.959 I talked about the agent sees what you see on a screen. s168 00:17:00.959 --> 00:17:05.839 But there are times in your day where you don't have a screen. s169 00:17:05.839 --> 00:17:07.439 Maybe you're in the office. s170 00:17:07.439 --> 00:17:11.679 You're walking into a meeting with a colleague. s171 00:17:11.679 --> 00:17:16.000 So I did a fun experiment yesterday at the conference here. s172 00:17:16.000 --> 00:17:22.640 So I grabbed my colleague Giovanni who is also here and actually on the second floor there's a great like little meeting booth. s173 00:17:22.640 --> 00:17:24.319 We found that by coincidence. s174 00:17:24.319 --> 00:17:28.160 So we went in there and we had our design meeting. s175 00:17:28.160 --> 00:17:35.280 And the goal here is really kind of show you how perception is so much more than just the visual part. s176 00:17:35.280 --> 00:17:43.600 So in this example, what we want to show you is perception can also be listening in the room to what you're discussing. s177 00:17:43.600 --> 00:17:48.320 And you can see here on the picture, both of us are wearing our B devices. s178 00:17:48.320 --> 00:17:51.679 Big shout out to B for sponsoring these. s179 00:17:51.679 --> 00:17:53.440 Um, so we're sitting there. s180 00:17:53.440 --> 00:17:55.919 We have our B devices that can do a transcript. s181 00:17:55.919 --> 00:17:57.919 They're listening to what we're saying. s182 00:17:57.919 --> 00:18:03.679 And then we have this design meeting and I had a couple of great ideas how to change this website. s183 00:18:03.679 --> 00:18:06.720 Um, you will see them in a in a second here. s184 00:18:06.720 --> 00:18:15.520 So let's have a quick look how this changed the same workflow on this website using this device. s185 00:18:15.840 --> 00:18:29.520 So we had the discussion the be did the transcript and you can see here on the right we're pulling this meeting transcript right in there is a whole detailed summary of the meeting. s186 00:18:29.520 --> 00:18:35.919 There is what we discussed and then it basically captures those insights. s187 00:18:35.919 --> 00:18:39.360 We have them right here and we can click apply. s188 00:18:39.360 --> 00:18:43.919 So what this apply button does is it sends it straight to the agent. s189 00:18:43.919 --> 00:18:53.280 And you can see here my crazy ideas to turn the background to yellow, turn the heading to red, and also change an emoji directly applied. s190 00:18:53.280 --> 00:18:56.880 And it also straight kicks off the verification right away. s191 00:18:56.880 --> 00:19:03.440 So it creates this report and and luckily this color scheme was apparently into in the approved rules. s192 00:19:03.440 --> 00:19:07.200 Otherwise, this would have liked like you did some weird things here. s193 00:19:07.200 --> 00:19:15.120 But again, you could change those rules if you don't want to have yellow backgrounds and it will make sure um that we still adhere to those guidelines. s194 00:19:15.120 --> 00:19:17.200 It would flag anything that's off. s195 00:19:17.200 --> 00:19:28.880 So, you have the judgment call if you want to either update the design specs because you actually like yellow or you take an action and say, "No, um fix this violation." s196 00:19:30.080 --> 00:19:34.400 But this is really the very first step. s197 00:19:34.400 --> 00:19:36.880 These two pieces are the very first beginning. s198 00:19:36.880 --> 00:19:48.720 And we're building out the rest in the open because these patterns can only get better if more people are using them, building on top of them, breaking things. s199 00:19:48.720 --> 00:19:52.799 So my ask here to you is go and try them out. s200 00:19:52.799 --> 00:19:55.760 They're on our GitHub repos, open source. s201 00:19:55.760 --> 00:19:57.520 Tell us what we're missing. s202 00:19:57.520 --> 00:20:08.080 give us the feedback what you would like to see where this should go next because ultimately none of us get smart alone and that's the whole point. s203 00:20:08.080 --> 00:20:14.559 We want to build AI that makes all of us smarter together. s204 00:20:16.240 --> 00:20:28.799 Now, if you're interested in a little bit more on human agent interactions and how we see those patterns changing, I would highly recommend this podcast by my colleague Danielle s205 00:20:28.799 --> 00:20:29.840 Persik. s206 00:20:29.840 --> 00:20:41.600 She is a cognitive scientist and runs our AGI ACI team at the lab and discusses a lot about human computer interaction patterns with experts in the industry. s207 00:20:41.600 --> 00:20:46.559 You can find the podcast on on any popular podcast platform. s208 00:20:46.720 --> 00:20:49.200 We also have more sessions this week. s209 00:20:49.200 --> 00:20:50.320 Um, so check them out. s210 00:20:50.320 --> 00:20:51.520 We have a booth down there. s211 00:20:51.520 --> 00:20:53.120 We have expert talks. s212 00:20:53.120 --> 00:21:02.080 We also have another computer use track talk coming up with my colleague Gaf Mishra at 1:30 in the computer use track. s213 00:21:02.080 --> 00:21:08.159 Highly recommend checking out his talk from RL to IRL. s214 00:21:08.400 --> 00:21:09.919 And then ultimately come find us. s215 00:21:09.919 --> 00:21:12.640 We have a huge presence down at the expo hall. s216 00:21:12.640 --> 00:21:16.240 We would love to continue the conversation with you all. s217 00:21:16.240 --> 00:21:24.559 If you're not here in person, you can also check out our code on our GitHub repo and check out our website. s218 00:21:24.559 --> 00:21:28.320 And with that, thank you very much.