WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ZGzthw4BZiA NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ZGzthw4BZiA.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.960 --> 00:00:17.520 Hello everyone and thank you for coming to our workshop building with perception agents. s2 00:00:17.520 --> 00:00:19.040 Uh my name is Emil Basel. s3 00:00:19.040 --> 00:00:23.720 I lead a team of solutions architects uh within Amazon AGI Lab. s4 00:00:23.720 --> 00:00:25.560 And our team is focused on two things. s5 00:00:25.560 --> 00:00:34.080 The first is working with some of our earliest adopter customers who want to work with some of our early models and products to help them get into production. s6 00:00:34.080 --> 00:00:46.200 And the second part is closing the feedback loop from our customer engagements, uh building evals for our research team to train our models on, and bringing product uh and engineering feedback s7 00:00:46.200 --> 00:00:48.120 to those teams as well. s8 00:00:48.120 --> 00:00:55.480 And I'm joined by one of our stellar solutions architects, Shruti Arora, who's going to be running the workshop today. s9 00:00:56.040 --> 00:00:58.360 Uh just a bit about what we're going to be talking about today. s10 00:00:58.360 --> 00:01:06.920 I'll be giving a quick overview of the Amazon AGI Lab, and talking about the first product that we shipped last year, Amazon Nova Act. s11 00:01:06.920 --> 00:01:14.960 And then we'll get into perception agents, and that's what the focus of today's workshop is, and then we'll get into the workshop. s12 00:01:16.600 --> 00:01:17.520 So a bit about the lab. s13 00:01:17.520 --> 00:01:27.360 So we are a frontier research and product team that combines startup speed with the scale and the resources of Amazon. s14 00:01:27.360 --> 00:01:38.400 And we are really focused on building knowledge work agents that can perceive, reason, and act in a variety of different environments to complete real-world tasks. s15 00:01:38.400 --> 00:01:41.800 Um our offices are in San Francisco and Boston. s16 00:01:41.800 --> 00:01:44.440 We've got a few other talks going on throughout the week. s17 00:01:44.440 --> 00:01:52.600 We have a booth at the expo, so stop by to collect some swag, meet some folks from the team, and hear some of the talks there. s18 00:01:52.840 --> 00:01:57.600 So I want to share a bit about our approach and how we think about agents. s19 00:01:58.040 --> 00:01:59.080 Across three different ways. s20 00:01:59.080 --> 00:02:00.560 So, the first is around collaboration. s21 00:02:00.560 --> 00:02:05.640 So, we really think about building agents that are teammates much more than just a black box. s22 00:02:05.640 --> 00:02:17.000 And when we say teammates, we really want an agent that can understand what our goal is, our objective is, and can share how they're thinking about achieving that goal. s23 00:02:17.000 --> 00:02:25.320 And and then grounding all of that in actual customer uh use cases. s24 00:02:25.320 --> 00:02:32.800 And so, we work very very closely with a large number of customers to really help ground all the work that we're doing with agents. s25 00:02:32.800 --> 00:02:35.040 And the second is around perception. s26 00:02:35.040 --> 00:02:37.880 So, agents need to understand their environment. s27 00:02:37.880 --> 00:02:44.120 And so, for us, because it's a perception-based agent, the environment spans multiple devices. s28 00:02:44.120 --> 00:02:53.040 So, we started with browsers last year, but you can imagine any visual screen is an environment for our agent to carry out knowledge work in. s29 00:02:53.040 --> 00:02:59.120 So, mobile devices, and also computer use is the other one. s30 00:02:59.120 --> 00:03:01.200 And the third one is reliability. s31 00:03:01.200 --> 00:03:06.080 And we have a great customer quote that said, "If my agent works 50% of the time, it doesn't work at all." s32 00:03:06.080 --> 00:03:16.720 And that's because you're either not going to run it at all, or if you do run it, you're going to end up uh debugging it and spending all this time, and you're going to basically do away with all the gains that you're getting from that agent. s33 00:03:16.720 --> 00:03:23.560 And so, we spend a lot of time building and training our models in uh simulated environments for reinforcement learning. s34 00:03:23.560 --> 00:03:36.840 Um and again, a lot of those are based on real-world uh use cases, which we get from not only our external customers, but a lot of internal Amazon teams that are carrying out a lot of uh knowledge work. s35 00:03:38.200 --> 00:03:38.440 Cool. s36 00:03:38.440 --> 00:03:42.440 So, with that backdrop, uh I want to talk about the first product that we shipped last year. s37 00:03:42.440 --> 00:03:54.560 So, Amazon Nova Act is a first-class AWS service that allows developers to build and run reliable browser-based workflows at scale. s38 00:03:54.560 --> 00:04:08.959 So, using natural language, you can drive a browser workflow to populate a browser, navigate, search with it, fill out forms, and call a human in the loop if you need a human for a expense report approval, s39 00:04:08.959 --> 00:04:13.320 or if you get stuck, or other things like that. s40 00:04:13.360 --> 00:04:17.600 And there's kind of four primary use cases which we've seen customers adopt this for. s41 00:04:17.600 --> 00:04:19.440 The first is form filling. s42 00:04:19.440 --> 00:04:27.040 Uh so, if you have to do that at scale across a large number of different websites, and you don't want to write a prompt for each single website, or write like a s43 00:04:27.040 --> 00:04:33.960 like a like a a deterministic script for each single site, you can use Nova Act for that. s44 00:04:33.960 --> 00:04:35.320 Uh search and extract. s45 00:04:35.320 --> 00:04:42.520 If you want to do research and get data from a lot of different websites, shopping, and QA. s46 00:04:42.520 --> 00:04:45.080 So, a bit more on what we launched. s47 00:04:45.080 --> 00:04:48.640 Uh the Let me just get through all these. s48 00:04:48.640 --> 00:04:51.160 Uh so, it's a first-class AWS service. s49 00:04:51.160 --> 00:04:55.760 So, if you go into the AWS console, type in Nova Act, you will see Nova Act there. s50 00:04:55.760 --> 00:04:58.960 And from the console itself, you can see all of the all of the trajectory. s51 00:04:58.960 --> 00:05:09.360 So, it's a very easy way to see everything which your agent has done, the bounding boxes around all the elements which it's looking at, and it's trying to take actions on there. s52 00:05:09.360 --> 00:05:16.080 And then behind the scenes, we have a custom computer use model which is doing all of the all of the inference there. s53 00:05:16.080 --> 00:05:24.480 This year, we launched an MCP server as well as a skill, and you're going to be using those in the workshop, and that's for your agents to be able to use s54 00:05:24.480 --> 00:05:26.080 Nova Act very easily. s55 00:05:26.080 --> 00:05:28.120 And then we've got human in the loop and tool use. s56 00:05:28.120 --> 00:05:29.400 And we also have a playground. s57 00:05:29.400 --> 00:05:37.360 So, if you go to the Nova Act homepage, it's a no-code way to just see Nova Act in action there. s58 00:05:37.360 --> 00:05:43.000 Now, looking ahead, and the topic of today's workshop is perception agents. s59 00:05:43.000 --> 00:05:46.440 Now, why do we feel there's a need for perception agents? s60 00:05:46.440 --> 00:05:55.400 So, uh for those of you who are not familiar with the term of computer use, uh that's basically the way for an agent to take an action on a computer screen, right? s61 00:05:55.400 --> 00:06:00.520 So, you see a blue button in the right corner, and you tell your agent, "Hey, go click this blue button over here." s62 00:06:00.520 --> 00:06:10.920 Very important, very powerful, it's very important for uh automating workflows, but we feel that there's another half of that, which is the agent actually understanding what it's seeing on the screen itself, s63 00:06:10.920 --> 00:06:18.440 not just using that as a sort of blind uh screen that it can just go and click on elements there. s64 00:06:18.440 --> 00:06:22.560 And we really want this agent to be able to understand screen just like a human would, right? s65 00:06:22.560 --> 00:06:39.720 So, if I showed someone a uh email client from like the late '90s, that was like a Yahoo email client or Hotmail client or something which you've never seen before, you would probably be able to figure out how to send an email from that, even though you've never used that client before, cuz we've s66 00:06:39.720 --> 00:06:51.440 got this collective understanding of thousands and thousands of hours of using these screens that we kind of intuitively understand how to use these things, and that's really where we're trying to get to with these s67 00:06:51.440 --> 00:06:53.800 with these agents. s68 00:06:53.800 --> 00:06:58.400 Uh and the second one is these agents don't require APIs in order to work, right? s69 00:06:58.400 --> 00:07:07.480 So, if you think about a lot of knowledge work, um it's a lot of humans using screens that just don't have APIs, and that's why there are people who are doing these things. s70 00:07:07.480 --> 00:07:13.960 And we want these agents to be able to carry out the same work across a variety of sites as well. s71 00:07:13.960 --> 00:07:19.520 And we want the models to see the screens the same way that we do, because that really facilitates the collaboration, right? s72 00:07:19.520 --> 00:07:30.040 So, rather than typing up paragraphs for what you want the agent to do uh on this screen and describing the screen, uh you can just point, click, and describe s73 00:07:30.040 --> 00:07:32.600 changes that you want, and that's what you're going to do in the workshop today. s74 00:07:32.600 --> 00:07:36.800 It's you're going to click on a couple elements and say, "Hey, I want to change the background to red." s75 00:07:36.800 --> 00:07:37.360 And that's it. s76 00:07:37.360 --> 00:07:41.800 And then you just have the agent go go and do that. s77 00:07:43.240 --> 00:07:53.160 And uh to to start with we've open-sourced two perception agent primitives, and you're going to be using these in depth in the workshop today. s78 00:07:53.160 --> 00:07:55.160 Uh the first is visual annotation. s79 00:07:55.160 --> 00:08:02.000 And so what this lets you do is on any webpage, you can point, click, and just describe changes that you want to make. s80 00:08:02.000 --> 00:08:10.120 And when you're done with that, you click export and it exports out a JSON file with all the changes that you want and a prompt, which you can give to your coding agent s81 00:08:10.120 --> 00:08:14.800 along with that JSON file to say, "Hey, go and make these changes and do that." s82 00:08:14.800 --> 00:08:17.800 So important is uh to to note, this has nothing to do with inference. s83 00:08:17.800 --> 00:08:19.520 This is purely deterministic. s84 00:08:19.520 --> 00:08:25.560 It just outputs JSON, doesn't have It's not tied to any model or anything else like that. s85 00:08:25.560 --> 00:08:30.480 The second part uh the second primitive which we've launched is visual verification. s86 00:08:30.480 --> 00:08:39.080 And if you think about a a site that you have, you've got certain CSS styles and designs which you want your your site to adhere to. s87 00:08:39.080 --> 00:08:41.919 Uh you've got different workflows which you want your site to support. s88 00:08:41.919 --> 00:08:45.680 Maybe have like a checkout flow or a login flow or something else. s89 00:08:45.680 --> 00:08:48.800 And you can describe those as Gherkin files. s90 00:08:48.800 --> 00:09:00.560 And so you can have visual verification, this this skill, um run by your coding agent against these design artifacts that you have, against these Gherkin files, and make sure that your site is adhering s91 00:09:00.560 --> 00:09:01.200 to those. s92 00:09:01.200 --> 00:09:05.080 And again, this is something which you're going to use uh in the workshop today. s93 00:09:05.080 --> 00:09:08.360 And again, these are all open-sourced, available today. s94 00:09:08.360 --> 00:09:12.320 And uh yeah, that's what we're going to do in the workshop. s95 00:09:12.320 --> 00:09:20.280 So I think at this point I've spoken enough, and I am going to hand it off to Shruti, who is going to kick us off on the workshop. s96 00:09:23.720 --> 00:09:24.920 Do you just press that? s97 00:09:24.920 --> 00:09:27.080 Thank you. s98 00:09:27.160 --> 00:09:28.040 All right. s99 00:09:28.040 --> 00:09:29.320 Can you all hear me okay? s100 00:09:29.320 --> 00:09:29.839 Yes. s101 00:09:29.839 --> 00:09:31.120 Okay. s102 00:09:31.120 --> 00:09:32.320 Welcome. s103 00:09:32.320 --> 00:09:44.960 Uh thanks Emil for that wonderful introduction to our uh Amazon AGI labs and the products that we launched, we released earlier in May this this year. s104 00:09:44.960 --> 00:09:59.040 So, as he mentioned, we are going to be doing hands-on workshop, but hands-on on like whatever we released the UI and the visual annotator and the visual verification. s105 00:09:59.760 --> 00:10:08.840 I have a short demo here, but before we dive into the demo, I just want all of you to like step into this role play here. s106 00:10:08.840 --> 00:10:11.560 Just just bear with me. s107 00:10:11.560 --> 00:10:18.200 Imagine you are a front-end engineer and you are building a web application. s108 00:10:18.200 --> 00:10:23.400 You have a design team and you are taking inspirations. s109 00:10:23.400 --> 00:10:27.240 They have these Figma files that you are building against. s110 00:10:27.240 --> 00:10:37.320 You are also collaborating with a sister team that they want to put in a piece of their information on the web page that you are building. s111 00:10:37.320 --> 00:10:47.839 So, if you are if you think about it that you have the designs, you'll probably take like four to five weeks to like build the designs and build everything code base and s112 00:10:47.839 --> 00:10:50.520 actually build that website. s113 00:10:50.520 --> 00:11:03.440 Now, in the third week or like fourth week around that time of your building process, you actually think, "Oh, maybe I should like go back and like verify if this looks fine, if everything looks okay with the rest of the team." s114 00:11:03.440 --> 00:11:05.360 And you are constantly collaborating, right? s115 00:11:05.360 --> 00:11:12.960 So, you are checking if if the work that you have done is something that is that is what you want to ship. s116 00:11:12.960 --> 00:11:28.560 Um so, you send your code base to you share the GitHub or however you are doing your GitLab or whatever you have your code in and you they run that this this web server locally and they open up the website s117 00:11:28.560 --> 00:11:31.280 and they say, "Oh, this this feels off. s118 00:11:31.280 --> 00:11:35.240 You know, the head and heading should be probably not in red. s119 00:11:35.240 --> 00:11:37.720 I should I want to switch it back to something. s120 00:11:37.720 --> 00:11:40.440 Um, and they provide this feedbacks. s121 00:11:40.440 --> 00:11:46.920 Probably they will do it in like a Slack message or in a shared doc, for example. s122 00:11:46.920 --> 00:11:55.560 Uh, and then once you get those feedback on that on your website, you would um, then like go over one of them. s123 00:11:55.560 --> 00:12:03.520 You'll disagree, agree go to some of them, come to a middle ground, and finalize this is the feedback you want to incorporate. s124 00:12:03.680 --> 00:12:08.320 You have to go through all of them and one by one then and start making the changes. s125 00:12:08.320 --> 00:12:19.280 Uh, now with coding agents, I guess it'll take maybe over which used to take maybe a week or it can like reduce to like maybe uh, few few days. s126 00:12:19.280 --> 00:12:33.000 Now, think about if you had the ability to take their feedback directly and pass it to the coding agent without you your supervision and just like once the changes have been done, that's when you interfere and say, "Oh, this looks good. s127 00:12:33.000 --> 00:12:35.280 All the changes have been incorporated." s128 00:12:35.280 --> 00:12:37.480 How would that reduce the time? s129 00:12:37.480 --> 00:12:38.800 Maybe few hours. s130 00:12:38.800 --> 00:12:48.040 You just start uh, collect the feedbacks, send it to your agent, and go for a coffee break, come back, and all the changes have been merged. s131 00:12:48.040 --> 00:12:52.360 So, this is exactly what uh, Emil was mentioning. s132 00:12:52.360 --> 00:12:57.920 Uh, that we have a Chrome extension that we are calling as visual annotator. s133 00:12:57.920 --> 00:13:00.000 Um, there are three ways to annotate. s134 00:13:00.000 --> 00:13:10.680 You can see here, it's either you draw on the page or you pick the element or you drop a pin as you would drop a pin on a Google Maps, say, and provide your annotations. s135 00:13:10.680 --> 00:13:20.720 And once you have your annotations, you can download these and send it to your team who needs to make the changes or even if it's you who's make the changes who's making the changes, s136 00:13:20.720 --> 00:13:30.680 then we provide a AI-friendly prompt that you can send it to your agent and it's going to change automatic like well, you have to instruct the agent to like change s137 00:13:30.680 --> 00:13:33.000 the good base. s138 00:13:33.600 --> 00:13:36.600 So, this was the visual annotator. s139 00:13:36.600 --> 00:13:41.960 The next um Oh. s140 00:13:44.320 --> 00:13:47.840 primitive that we launched is the visual verification skill. s141 00:13:47.840 --> 00:13:54.720 And when I say skill, it is actually a skill that you can install on Claude or Keynote Keynote power. s142 00:13:54.720 --> 00:14:05.840 And um you would see these verification as I may mention, these are the design files that have against which you will be running your visual like um verification against. s143 00:14:05.840 --> 00:14:09.000 Like if does the design colors look fine? s144 00:14:09.000 --> 00:14:11.360 Is the font size right? s145 00:14:11.360 --> 00:14:13.160 Is the user flow right? s146 00:14:13.160 --> 00:14:20.920 But, one thing that will come to your mind like these design specs, these flows, how are these generated? s147 00:14:20.920 --> 00:14:23.240 How are how how you going to build that? s148 00:14:23.240 --> 00:14:26.440 Not to worry, we have a solution for that as well. s149 00:14:26.440 --> 00:14:39.040 When you run your verification for the first time, the there will be a a deep um one level surface crawl that will be done by the UI verification tool s150 00:14:39.040 --> 00:14:51.080 on your website to go and check what are the links, what are the content, and based on that it's going to create these user flows that are in terms of s151 00:14:51.080 --> 00:14:59.440 in Gherkin for syntax that the UI UI verification is going to use when it's going to run the verification next. s152 00:14:59.440 --> 00:15:03.240 So, let me just play that play the demo here. s153 00:15:03.240 --> 00:15:06.480 Um so, you can see you just like call the tool. s154 00:15:06.480 --> 00:15:13.000 It's going to go through the design docs specs specs specs and the user flows. s155 00:15:13.000 --> 00:15:14.560 Uh go through it. s156 00:15:14.560 --> 00:15:22.880 It's going to open up the browser, run through the design flow user flows, and then generate a report. s157 00:15:22.880 --> 00:15:28.680 So, this these are the two tools that you will get hands-on in this workshop. s158 00:15:28.680 --> 00:15:35.520 Uh so, I'm going to jump to what we have how the workshop has been laid out as of now. s159 00:15:35.520 --> 00:15:38.360 First, as I mentioned, two uh hands-on. s160 00:15:38.360 --> 00:15:43.520 The second thing that's what we are trying to do is these are the two primitives, of course. s161 00:15:43.520 --> 00:15:47.000 Perception is not just these two primitives. s162 00:15:47.000 --> 00:15:51.840 How would you wire these two primitives up for a particular use case? s163 00:15:51.840 --> 00:15:56.520 There is a scenario that has been uh given in the workshop. s164 00:15:56.520 --> 00:16:04.360 We are trying to make this fun, so please try it out yourself uh and see like it's like a game day. s165 00:16:04.360 --> 00:16:09.960 You would see there are like certain hints for the to solve the problem, what the problem is, what the end goal is. s166 00:16:09.960 --> 00:16:11.960 So, everything is laid out in the workshop. s167 00:16:11.960 --> 00:16:12.840 Uh it's a GitHub link. s168 00:16:12.840 --> 00:16:15.320 We will share that with you in a bit. s169 00:16:15.320 --> 00:16:19.000 Uh so, how will you basically wire these two things up? s170 00:16:19.000 --> 00:16:25.720 Uh the sec- the next thing we have are um sponsors for the prices. s171 00:16:25.720 --> 00:16:28.120 We have certain contests that run at the end. s172 00:16:28.120 --> 00:16:35.839 And um thank you the B team for sponsoring the prices for the uh for the uh event. s173 00:16:35.839 --> 00:16:54.800 And um we will have a way how you can instead of typing or like annotating on the screen, what are some other ways cool ways of making sure that there are some annotations that you want to do or like feedback that you want to provide about a web page and its designs. s174 00:16:54.800 --> 00:17:02.120 So, B team will show you how you can do that beyond just typing in your feedback. s175 00:17:02.120 --> 00:17:10.439 Finally, there are Well, before that, we have After you finish the workshop, we also have like three different options for bonus challenge. s176 00:17:10.439 --> 00:17:20.959 We will have plenty of time for you to try out these tools in a particular use case that you want to maybe some you want to try wire it up with B s177 00:17:20.959 --> 00:17:30.800 and see how that how you would probably wire it with any other form of feedback that you would essentially get from your team. s178 00:17:30.800 --> 00:17:34.280 Or if you think that hey, this is really cool. s179 00:17:34.280 --> 00:17:39.520 I want to build something that I want to show to my team on Monday like the following Monday. s180 00:17:39.520 --> 00:17:49.920 So, go for it and show if you're brave enough come in front come at the stage show your use case demo your use case and you'll get a B device. s181 00:17:49.920 --> 00:18:05.040 The final part is we're going to do a little fun exercise where once you have a good understanding of these two primitives you can think of like multiple ways of how you would use it on a daily s182 00:18:05.040 --> 00:18:17.360 on your day to day day to day and we will you will submit these ideas of like hey, I think this will be really cool for me to use this and we will have people vote on for these ideas s183 00:18:17.360 --> 00:18:25.400 and the top three ideas will get a B device and I think we also have one for crowd favorite. s184 00:18:25.400 --> 00:18:30.720 So, with that going to jump into the workshop. s185 00:18:30.720 --> 00:18:34.840 This is the link for the workshop. s186 00:18:34.840 --> 00:18:41.880 You can type in it's pretty it's it's available on GitHub so you can type in perception agents workshop. s187 00:18:41.880 --> 00:18:45.080 You will probably get the first GitHub link. s188 00:18:45.080 --> 00:18:54.760 Um in the workshop we also need you to have use a Nova Act API key. s189 00:18:54.760 --> 00:19:01.840 So, there is a piece where in the workshop we have obscured the first three letters. s190 00:19:01.840 --> 00:19:04.280 So, that is on the screen here. s191 00:19:04.280 --> 00:19:15.000 So, you can like copy paste these three letters ACE that stands for ace and you're good to go with the Novact API key. s192 00:19:17.120 --> 00:19:24.200 So, let me just pause this. s193 00:19:24.200 --> 00:19:30.040 So, in the workshop, you would the readme will basic point you to the workshop with MD file. s194 00:19:30.040 --> 00:19:33.320 And here is the scenario that I was talking about. s195 00:19:33.320 --> 00:19:34.560 I'm going to read it out from here. s196 00:19:34.560 --> 00:19:40.160 You are a front-end developer preparing to ship a landing page for a podcast. s197 00:19:40.160 --> 00:19:46.000 Yesterday you demoed the page to a few peers and they provided some feedback. s198 00:19:46.000 --> 00:19:48.560 The hero title feels too big. s199 00:19:48.560 --> 00:19:51.560 The accent color doesn't pop up the way you it should. s200 00:19:51.560 --> 00:19:58.000 Some buttons are oversized and a few small things that you slightly just off. s201 00:19:58.000 --> 00:20:00.840 So, you are on board with those feedback. s202 00:20:00.840 --> 00:20:03.320 And today you come in fresh. s203 00:20:03.320 --> 00:20:07.160 And I'm ready to incorporate those changes. s204 00:20:07.160 --> 00:20:09.720 So, what do you have at your disposal? s205 00:20:09.720 --> 00:20:18.000 The Novact Chrome extension, the UI verification scale, you have the Novact SDK, and the Novact MCP server. s206 00:20:18.000 --> 00:20:31.400 These are the tools that you have at your disposal to basically implement the solution where you would essentially build build this entire loop by itself. s207 00:20:31.400 --> 00:20:42.080 Like, you just annotate and automatically you would just hit a button in the Chrome extension and all the changes will be applied. s208 00:20:42.080 --> 00:20:47.760 The UI verification will run and you have a report by the end of it. s209 00:20:47.760 --> 00:20:52.760 So, this is the challenge and there are a few prerequisites. s210 00:20:52.760 --> 00:21:01.120 So, um you need to have an macOS or a Linux tool because Novact MCP server runs on macOS and Linux. s211 00:21:01.120 --> 00:21:03.200 You should have Node.js installed. s212 00:21:03.200 --> 00:21:05.000 If not, you can install them. s213 00:21:05.000 --> 00:21:15.120 uh NPM, Python, uh Chrome browser to basically test it out because it's a Chrome extension, and then um an AI coding agent. s214 00:21:15.120 --> 00:21:17.880 So, if you have Claude, that works great. s215 00:21:17.880 --> 00:21:19.440 We also have Kiro integration. s216 00:21:19.440 --> 00:21:26.600 So, if you don't have any co- coding um agent, you can install Kiro and start playing around with it. s217 00:21:26.600 --> 00:21:31.120 Um and lastly, we have the UV um that's UVX. s218 00:21:31.120 --> 00:21:32.840 That's basically to install. s219 00:21:32.840 --> 00:21:39.880 If you don't want to install UV or UVX, then please raise your hand and we can help you install using pip. s220 00:21:39.880 --> 00:21:44.520 Um So, that's that's that. s221 00:21:44.520 --> 00:21:50.840 And the workshop is laid out for I think the setup should be about 5 to 7 minutes. s222 00:21:50.840 --> 00:21:54.880 Uh then you can like start playing around with those skills. s223 00:21:54.880 --> 00:22:01.480 That gets a good understanding of how would you use the Chrome extension, how would you use the uh UI verification skill. s224 00:22:01.480 --> 00:22:03.680 That's what has been explained here. s225 00:22:03.680 --> 00:22:13.760 Uh to solve the challenge, you have um about 15 minutes in if you want to go that route, like I want to go step by step. s226 00:22:13.760 --> 00:22:16.520 And I'm just going to go scroll down real quick. s227 00:22:16.520 --> 00:22:16.960 Yeah. s228 00:22:16.960 --> 00:22:18.000 So, choose your path. s229 00:22:18.000 --> 00:22:21.680 I have this workshop designed in two levels. s230 00:22:21.680 --> 00:22:24.960 Uh you you remember I said like I want you to have fun. s231 00:22:24.960 --> 00:22:28.640 So, two levels, if you want to go for a challenge, go for the medium route. s232 00:22:28.640 --> 00:22:34.920 If you feel that uh I'm just going to like uh follow the instructions and get the workshop done. s233 00:22:34.920 --> 00:22:36.040 You'll feel good about it. s234 00:22:36.040 --> 00:22:46.280 So, yeah, we have an easy mode where you'll have exact instructions what you what should be done, and you will have these two primitives wired up by the end of the hands-on. s235 00:22:46.280 --> 00:22:53.680 Um so, 15 minutes we are expecting the easy path, um and then 30 to 35 minutes for the medium. s236 00:22:53.680 --> 00:23:06.400 Once we have allocated the 15 minutes time, I will come up and show you where you would like essentially um add your code so that uh you can wire these two primitives up s237 00:23:06.400 --> 00:23:11.200 and you can see the entire thing come into life in action. s238 00:23:11.840 --> 00:23:14.080 And we also have a bonus challenge. s239 00:23:14.080 --> 00:23:17.280 If you finish early, you can start with the bonus challenge. s240 00:23:17.280 --> 00:23:26.440 Uh but before that we will have the B team Giovanni um in some time to like present. s241 00:23:27.360 --> 00:23:28.240 After Yeah. s242 00:23:28.240 --> 00:23:29.280 Yeah, sure thing. s243 00:23:29.280 --> 00:23:34.040 So I saw that you had as I was installing these uh s244 00:23:34.040 --> 00:23:35.200 Mhm. s245 00:23:35.200 --> 00:23:39.200 What is it the Nova Act UI verification skills? s246 00:23:39.200 --> 00:23:46.840 Um it it said that there were some defaults clients that are always supported and it included like Codex and Cloud Code. s247 00:23:46.840 --> 00:23:51.920 But then below that there are these additional skills and one of them is also Cloud Code. s248 00:23:51.920 --> 00:23:57.960 So I'm just curious like are there two different things going on there or are those the same? s249 00:23:57.960 --> 00:24:00.760 The the fact that there are two Cloud Code s250 00:24:00.760 --> 00:24:02.960 Yes, so there are two skills. s251 00:24:02.960 --> 00:24:05.480 One is the Nova Act skill. s252 00:24:05.480 --> 00:24:07.600 One is the Nova Act UI verification skills. s253 00:24:07.600 --> 00:24:09.560 So that's why you must be seeing both of them. s254 00:24:09.560 --> 00:24:11.560 Yeah. s255 00:24:13.560 --> 00:24:21.640 So I'm going to give you the time here to I'm going to bring up the slide back. s256 00:24:21.640 --> 00:24:23.920 I'm going to ask Emil to pop up here. s257 00:24:23.920 --> 00:24:29.440 Slide back on the GitHub repo and so that you have the A API key handy if you have to change. s258 00:24:29.440 --> 00:24:33.320 Feel free to like raise your hand if you have any questions. s259 00:24:33.320 --> 00:24:37.640 We have our great awesome team supporting the workshop. s260 00:24:37.640 --> 00:24:40.080 There you find them, see them waving. s261 00:24:40.080 --> 00:24:47.280 So reach out to either of us and we will pop by you to help. s262 00:24:47.280 --> 00:24:49.560 Have fun. s263 00:24:49.840 --> 00:25:00.960 Okay, I'm just going to try to walk through the solution so that Okay. s264 00:25:12.360 --> 00:25:15.040 So, this is the step-by-step instructions. s265 00:25:15.040 --> 00:25:18.280 You can see that we have the Chrome extension. s266 00:25:18.280 --> 00:25:22.920 We have your agent uh AI agent running. s267 00:25:22.920 --> 00:25:33.040 Uh if you want your coding agent to like automatically apply the changes from the extension, you need a bridge because there's no bridge between them. s268 00:25:33.040 --> 00:25:37.760 How would the extension like send the prompt to the coding uh coding agent? s269 00:25:37.760 --> 00:25:50.960 So, for that, we have built uh something called agent bridge that will automatically uh take the annotations and up- give prompted to the coding agent. s270 00:25:50.960 --> 00:25:56.680 That's where it will start applying those changes and then running the UI verification skill. s271 00:25:56.680 --> 00:26:12.920 So, um essentially in the extension, we don't have the apply changes button in the in the primitive mode, but I for this workshop, I have built in we have built this um apply changes button s272 00:26:12.920 --> 00:26:14.640 in the Chrome extension. s273 00:26:14.640 --> 00:26:18.240 Um I'll walk you through that in a bit. s274 00:26:18.240 --> 00:26:26.120 Uh so, for the Chrome for for this entire thing to work, you need to make changes in like two files. s275 00:26:26.120 --> 00:26:42.920 The first file is um the manifest.json file in the Chrome extension, where you essentially add the this line for the host permissions to start accepting requests from the local host. s276 00:26:42.920 --> 00:26:47.560 Uh and then you add this this piece of code. s277 00:26:47.560 --> 00:26:52.240 It's a huge like piece of chunk of code here. s278 00:26:52.240 --> 00:27:05.680 Um, and you can what this part does in the extension is creates and apply changes button, wires up with the extension, uh, make sure makes makes the UI verification s279 00:27:05.680 --> 00:27:17.920 run uh, in using the Nova at SDK instead of the skill, uh, and starts generates a report that you can click on and then view the report. s280 00:27:17.920 --> 00:27:35.200 Um, so this piece of code is goes in this file that is the, um, let's see, content.js in the extension and it needs to be added around like line 261. s281 00:27:35.200 --> 00:27:48.160 Uh, between this append child export button just before and I think I have hash, uh, there's a comment there add code here in that file and that's where you would like paste this code. s282 00:27:48.240 --> 00:27:59.640 So once you have that running now you would see this apply change button here in the extension. s283 00:27:59.640 --> 00:28:07.520 Um, I think, uh, Emil was testing this so he changed this to like FIFA World Cup, why not? s284 00:28:07.520 --> 00:28:18.440 Um, so I'm going to select one of these and I'm going to refresh the extension again. s285 00:28:20.080 --> 00:28:22.520 Let's see. s286 00:28:25.080 --> 00:28:28.080 Annotate current page. s287 00:28:28.080 --> 00:28:32.800 Uh, I'm going to try to change the published date. s288 00:28:34.080 --> 00:28:42.440 Change to today's June 20 9th. s289 00:28:42.440 --> 00:28:43.840 Okay. s290 00:28:43.840 --> 00:28:46.360 And save it. s291 00:28:46.360 --> 00:28:50.720 And this is the extension. s292 00:28:50.720 --> 00:28:52.600 Apply changes. s293 00:28:52.600 --> 00:28:56.560 So, in the background, it's going to open up your coding agent. s294 00:28:56.560 --> 00:29:00.400 It's been tested on Cloud Code and Kiro as I mentioned. s295 00:29:00.400 --> 00:29:14.200 So, if you're trying Cursor or Codex, the agent bridge is something that you would have to edit so that it can accept these apply changes button like requests. s296 00:29:14.200 --> 00:29:24.240 Um So, in a bit it'll start applying the changes and you will see certain logs that pop up as well in the repository. s297 00:29:24.240 --> 00:29:36.800 Um that is the key the CLI's logs as well as when the verification runs Nova Act has these logs uh that gives you exactly what the AI model was thinking s298 00:29:36.800 --> 00:29:40.920 and what part of the screen it was trying to make a decision on. s299 00:29:40.920 --> 00:29:44.480 And you can see that the button changed to run verification now. s300 00:29:44.480 --> 00:29:47.720 So, it's running the verification step. s301 00:29:48.400 --> 00:29:51.680 And in a bit it'll also generate a report that you can see. s302 00:29:51.680 --> 00:30:04.440 So, this is where you basically apply the changes and go grab a lunch or like break and then come back and just see uh the verification report once you're back. s303 00:30:05.240 --> 00:30:13.280 Going to give it a minute because it's going to go through all the design flows, all the user flows, and then generate a report. s304 00:30:13.280 --> 00:30:16.480 So, it takes a bit of a time. s305 00:30:19.440 --> 00:30:22.880 Let's see if I can pull up the repo. s306 00:30:24.520 --> 00:30:25.640 Oh. s307 00:30:25.640 --> 00:30:27.640 Shoot. s308 00:31:35.360 --> 00:31:52.440 I think the Wi-Fi is super slow, so But, I I think maybe in the meantime I can have Giovanni come over on the stage and show you a cool way to annotate your webs web pages s309 00:31:52.440 --> 00:31:59.840 instead of like just going and typing stuff on the web on the web page that you're working on. s310 00:31:59.840 --> 00:32:00.520 Hello. s311 00:32:00.520 --> 00:32:01.400 Oh, the prime. s312 00:32:01.400 --> 00:32:02.640 Okay. s313 00:32:02.640 --> 00:32:04.200 Can we have s314 00:32:04.200 --> 00:32:05.240 Uh I'm done. s315 00:32:05.240 --> 00:32:07.080 Can I see the slides? s316 00:32:07.080 --> 00:32:09.560 The slides, yes. s317 00:32:11.360 --> 00:32:13.360 Hello. s318 00:32:14.840 --> 00:32:26.840 Hi everyone, I'm Giovanni and I was thinking to imagine you are in to do your design meeting just talking about the design modification you want to do to that website. s319 00:32:26.840 --> 00:32:32.400 And then automatically your perception agent agent is able to modify the website. s320 00:32:32.400 --> 00:32:35.440 This is something that you can do right now with the workshop. s321 00:32:35.440 --> 00:32:37.600 And this why, how? s322 00:32:37.600 --> 00:32:40.520 Using the B wearable devices. s323 00:32:40.520 --> 00:32:43.640 So, B is a device we have in Amazon. s324 00:32:43.640 --> 00:32:47.920 And you can buy it like in the b.computer website. s325 00:32:47.920 --> 00:32:50.600 It's a wearable this is your AI assistant. s326 00:32:50.600 --> 00:32:53.640 So, it just have all it has a microphone. s327 00:32:53.640 --> 00:33:00.040 So, you can activate this the button and then it's going to listen for your conversation. s328 00:33:00.040 --> 00:33:11.800 No worries, there is a l l i d that's will be on if it's listening to your conversation and then all your conversation on your private cloud. s329 00:33:11.800 --> 00:33:21.000 After the AI agent that is running B is able to gives you suggestion about whatever conversation you had. s330 00:33:21.000 --> 00:33:28.960 So, what I did is like integrating with this this agent the B agent with the perception agent that we had for the workshop. s331 00:33:28.960 --> 00:33:36.080 So, getting the input listening from a AI conversation from a design conversation. s332 00:33:36.080 --> 00:33:38.760 Uh, I don't see the video. s333 00:33:38.760 --> 00:33:40.000 Yeah. s334 00:33:40.000 --> 00:33:40.720 Ah, this one. s335 00:33:40.720 --> 00:33:41.480 Yeah. s336 00:33:41.480 --> 00:33:43.720 Thank you. s337 00:33:43.960 --> 00:33:46.600 You can achieve this with three simple step. s338 00:33:46.600 --> 00:33:54.080 So, first of all, yes, you have to have a B. And I want with me that you can use here and now or we can have a test a test account s339 00:33:54.080 --> 00:33:58.680 to download that you can use to get the conversation as you want the conversation. s340 00:33:58.680 --> 00:34:06.080 Then you need to install the B C L I. It's a C L I that will enable that is going to allow you to get the data for your conversation s341 00:34:06.080 --> 00:34:11.440 and it's in the workshop the instruction to download the B C L I by just a npm command. s342 00:34:11.440 --> 00:34:23.480 Then you start the server of the workshop and then you start the bridge listening from B conversation and applying that to the perception agent. s343 00:34:23.480 --> 00:34:25.800 Then you just enable your B device. s344 00:34:25.800 --> 00:34:27.240 The LED is on. s345 00:34:27.240 --> 00:34:30.919 You say something like, "Oh, I want to change the title color to red. s346 00:34:30.919 --> 00:34:35.240 I want to change the emoji instead of our brain emoji, I want a robot emoji. s347 00:34:35.240 --> 00:34:42.919 And on the right side, you will see your conversation, your transcript of the conversation, and even the most important points. s348 00:34:42.919 --> 00:34:50.639 Here you can see the response API, the the API response coming from the B. You can open the panel and check. s349 00:34:50.639 --> 00:34:53.120 And then you can just select apply. s350 00:34:53.120 --> 00:34:57.600 And here is going to be the normal flow of the perception agent workshop. s351 00:34:57.600 --> 00:35:03.920 So, it's going to change the color, the color that you just said, it's going to change the emoji. s352 00:35:03.920 --> 00:35:16.640 And after the modification of the website are applied, it's going to run even for you automatically the verification phase of no act. s353 00:35:16.640 --> 00:35:23.400 So, we have a test account that you can use with a preloaded conversation, preloaded design conversation. s354 00:35:23.400 --> 00:35:27.320 It is like just talking about, "Oh, we change the color, we change the emoji." s355 00:35:27.320 --> 00:35:29.120 Something like that. s356 00:35:29.120 --> 00:35:38.480 And you can after you install the CLI and you log in with the B account with a test device with a test account, you just click on the bottom s357 00:35:38.480 --> 00:35:41.760 button, the the button at the bottom. s358 00:35:41.760 --> 00:35:44.440 Uh this is the verification page you can check. s359 00:35:44.440 --> 00:35:51.080 But otherwise, if you want to test the B device, we can like you do we can actually just use mine and I will show you around if you want it. s360 00:35:51.080 --> 00:36:01.400 Like we can actually see like have a conversation about the design and you can see that it's actually applying this after like two or three minutes of elaborating the conversation. s361 00:36:01.400 --> 00:36:04.520 The Wi-Fi is a little bit slow, so that's why I didn't do the live demo. s362 00:36:04.520 --> 00:36:09.360 Uh and yeah, and then in the end there is there is the verification report that you can check. s363 00:36:09.360 --> 00:36:13.880 The B devices are also our prizes for the for the challenge. s364 00:36:13.880 --> 00:36:16.920 So, yeah, you're going to have one if you win the challenge. s365 00:36:16.920 --> 00:36:21.920 And we also are at the booth today and all the week. s366 00:36:22.080 --> 00:36:26.760 So, yeah, let me know if you want to integrate B and I'm here for you. s367 00:36:26.760 --> 00:36:40.480 So, B device, if you want to come up on the stage, we have 10 minutes for you to like explore these two primitives, think of an idea that you want to probably use it. s368 00:36:40.480 --> 00:36:43.960 And if you want to come up, you'll get one of these devices. s369 00:36:43.960 --> 00:36:46.000 Um so, yeah. s370 00:36:46.000 --> 00:36:52.200 The if just get creative, I guess. s371 00:36:52.240 --> 00:36:59.160 Hey everyone, I have Jed with me who built something really cool and he would like to show it to everyone. s372 00:36:59.160 --> 00:37:01.840 Jed, come up on the stage. s373 00:37:02.720 --> 00:37:04.320 Yeah. s374 00:37:04.320 --> 00:37:05.000 Jed. s375 00:37:05.000 --> 00:37:05.800 Oh. s376 00:37:05.800 --> 00:37:06.240 What am I doing? s377 00:37:06.240 --> 00:37:07.000 Where am I going? s378 00:37:07.000 --> 00:37:08.400 Oh, I called Jed, but yeah. s379 00:37:08.400 --> 00:37:09.960 Oh, I I My name is Jeff, so I thought s380 00:37:09.960 --> 00:37:12.000 You are Sorry. s381 00:37:12.000 --> 00:37:15.880 I'll have Jeffrey next to show something cool as well. s382 00:37:15.880 --> 00:37:17.240 Oh, okay. s383 00:37:17.240 --> 00:37:32.600 All right, this this was just goofy, but it's actually an an an interesting little funnel here is is with this animators going through and Oh, you can't see it. s384 00:37:33.600 --> 00:37:36.440 Change it to a duck podcast. s385 00:37:40.800 --> 00:37:43.080 There we go. s386 00:37:43.960 --> 00:37:48.000 So, just turn asked it to turn it retro NES. s387 00:37:48.000 --> 00:37:55.120 Uh do some duck jokes and went back and did the verification funnel. s388 00:37:55.120 --> 00:38:08.600 Um it's probably as what I was telling them is probably really interesting to me here is to watch this this great AI said I made it a duck you know, I I made a duck icon. s389 00:38:08.600 --> 00:38:22.120 But uh often as everyone here likely knows is you don't actually see that happen, but then it it went through and did the the entire verification funnel, came back with the reports s390 00:38:22.120 --> 00:38:23.480 saying, yep, it's done. s391 00:38:23.480 --> 00:38:24.160 I did it. s392 00:38:24.160 --> 00:38:25.120 I actually did it. s393 00:38:25.120 --> 00:38:27.920 So, that was cool. s394 00:38:29.720 --> 00:38:31.920 Oh, yeah. s395 00:38:33.920 --> 00:38:38.720 I had it I had it just do asked it to do some jokes. s396 00:38:39.840 --> 00:38:42.600 Which is pretty bad at. s397 00:38:45.520 --> 00:38:47.720 There you go. s398 00:38:52.960 --> 00:38:57.160 Oh, because you came up with a cool demo. s399 00:38:57.160 --> 00:38:59.560 Do you have to step? s400 00:39:01.280 --> 00:39:03.400 As promised, you win this. s401 00:39:03.400 --> 00:39:04.240 Oh, thank you. s402 00:39:04.240 --> 00:39:06.400 P device. s403 00:39:10.240 --> 00:39:17.080 UP NEXT, WE HAVE Jeff Lee who wants to come and present what he did. s404 00:40:00.880 --> 00:40:10.680 Yeah, so I'm still building a test harness to actually be able to show this cuz the default B account has a uh sample conversation that isn't relevant to the things. s405 00:40:10.680 --> 00:40:14.960 So, this is just this is just something trying to uh bring the B to life. s406 00:40:14.960 --> 00:40:28.360 So, the idea was to have access the BCLI um and collect the summary of a of a conversation which was a discussion of of a backlog features that would be added to a s407 00:40:28.360 --> 00:40:30.280 uh repo that you're working on, right? s408 00:40:30.280 --> 00:40:41.360 So, you're working on a specific uh GitHub repository that's that's that that's cloned and then um uh what's going to happen is your conversation is going to be used in context with s409 00:40:41.360 --> 00:40:43.280 uh all of the code that's that's there. s410 00:40:43.280 --> 00:40:50.720 It's going to build a backlog based on the the summary and it's going to use the Atlassian MCP to put that into Jira. s411 00:40:50.720 --> 00:40:54.320 So, that's the uh that's the uh idea here. s412 00:40:54.320 --> 00:41:02.640 And so, the key thing that I'm just building right now in the background is uh the uh the test harness to insert a new conversation that's relevant to the the repo. s413 00:41:02.640 --> 00:41:11.160 So, that's Pretty simple and uh yeah. s414 00:41:11.160 --> 00:41:12.280 Anyway. s415 00:41:12.280 --> 00:41:13.000 That's awesome. s416 00:41:13.000 --> 00:41:13.960 Thank you so much. s417 00:41:13.960 --> 00:41:15.960 Thanks. s418 00:41:17.160 --> 00:41:17.480 You're welcome. s419 00:41:17.480 --> 00:41:19.280 So, here you win your new device. s420 00:41:19.280 --> 00:41:22.040 I know you wanted to go and get it, but you won it. s421 00:41:22.040 --> 00:41:24.320 Thank you very much. s422 00:41:29.960 --> 00:41:34.760 So, You have a question? s423 00:41:37.480 --> 00:41:41.240 Ryan, who's going to present next? s424 00:41:43.640 --> 00:41:45.040 Hello, everyone. s425 00:41:45.040 --> 00:41:47.480 Excited to be here. s426 00:41:53.840 --> 00:41:54.720 Let me see. s427 00:41:54.720 --> 00:41:56.280 Start mirroring. s428 00:41:56.280 --> 00:42:02.200 So, I was I was late to this session uh because the other session ended at 3:00. s429 00:42:02.200 --> 00:42:07.600 So, um I came up to uh Giovanni. s430 00:42:07.600 --> 00:42:11.360 Let me shift the key right here. s431 00:42:12.360 --> 00:42:13.640 Oh, perfect. s432 00:42:13.640 --> 00:42:14.840 That's fine. s433 00:42:14.840 --> 00:42:16.080 So, I gave it to Giovanni. s434 00:42:16.080 --> 00:42:18.800 Um currently wearing a bee hat. s435 00:42:18.800 --> 00:42:22.240 I currently attend uh Wharton uh business school. s436 00:42:22.240 --> 00:42:24.120 My cluster's mascot is a bee. s437 00:42:24.120 --> 00:42:28.240 I am one of three people who care that the mascot to be. s438 00:42:28.240 --> 00:42:34.520 Um so, you can imagine my excitement when I came to this conference and saw we were presenting about uh bee device. s439 00:42:34.520 --> 00:42:40.520 So, naturally, being late, being a Wharton grad, I went and asked if I could have a free bee device. s440 00:42:40.520 --> 00:42:43.520 Uh Giovanni said, "You have an idea to share?" s441 00:42:43.520 --> 00:42:46.920 So, I I currently working at an AI upskilling startup. s442 00:42:46.920 --> 00:42:49.800 Um I'm not a huge fan of Linear. s443 00:42:49.800 --> 00:42:52.440 Apologies to anyone who works at Linear. s444 00:42:52.440 --> 00:43:03.640 I just needed a better way of understanding what should I be doing anytime during the day, going from capturing an idea, discussing with my co-founder, and then updating a project repository. s445 00:43:03.640 --> 00:43:06.440 So, this is a natural use case for the bee device. s446 00:43:06.440 --> 00:43:09.360 So, we have a internal tool called Sidekick. s447 00:43:09.360 --> 00:43:13.440 Emphasis on an internal because there is no design. s448 00:43:13.440 --> 00:43:21.120 Um in my co-founder's defense, he did tell me to uh tell Claude to make it look pretty, no mistakes, and this is the best it did. s449 00:43:21.120 --> 00:43:29.040 So, um but how it works is we have a a joint inbox. s450 00:43:29.040 --> 00:43:31.760 Um you can ignore the Go Touch Crash SF. s451 00:43:31.760 --> 00:43:35.320 I just see all these billboards and um gets me concerned. s452 00:43:35.320 --> 00:43:39.840 Uh that we send emails jointly to uh capture@sidekick.ausin.org. s453 00:43:41.000 --> 00:43:46.560 And these are all the different inputs we have um thoughts throughout the week, newsletters. s454 00:43:46.560 --> 00:43:49.720 I don't know if anyone here is subscribed to an AI newsletter. s455 00:43:49.720 --> 00:43:53.000 Um see something interesting, we send it to inbox. s456 00:43:53.000 --> 00:44:03.080 And we are big believers in getting things done, which was a uh mode for knowledge workers of organizing a projects task developed by David Allen in 2010. s457 00:44:03.080 --> 00:44:07.400 And oh, if the Wi-Fi works, you could be able to see it. s458 00:44:07.400 --> 00:44:12.520 Um Sorry, that did just work off the screen. s459 00:44:12.920 --> 00:44:13.840 One second. s460 00:44:13.840 --> 00:44:27.760 So, the getting the getting things done tree shows all the projects, and under each project it has uh the strategy, has what the what completion looks like, tasks, next steps, references. s461 00:44:27.760 --> 00:44:34.520 And we've built an MCP so we can sit and talk to Claude and strategize on, "Hey, this is what we're thinking on this project. s462 00:44:34.520 --> 00:44:37.040 What do you think we should be doing?" s463 00:44:37.040 --> 00:44:43.640 All to solve the need of when you're working after you have a meeting of like, "What should I be doing right now to tie to my ultimate strategy?" s464 00:44:43.640 --> 00:44:58.520 So, how we imagine this to work with the uh B uh device, as Giovanni uh informed me there's an MCP, would be capture different meetings or different ideas, have it sent to our capture inbox, and then use that to update our strategy. s465 00:44:58.520 --> 00:45:02.760 Um I wish this was was loading because, let me see. s466 00:45:02.760 --> 00:45:04.800 Oh, that's why. s467 00:45:04.800 --> 00:45:06.400 my phones. s468 00:45:06.400 --> 00:45:07.920 See that worked. s469 00:45:07.920 --> 00:45:11.720 Um Yes. s470 00:45:11.720 --> 00:45:15.120 You know, I'm now 0 for 11 for live demos. s471 00:45:15.120 --> 00:45:19.080 Um but that is that is my concept of the B device. s472 00:45:19.080 --> 00:45:24.400 I'm looking forward to using it and sharing on LinkedIn how much I love the bees. s473 00:45:24.400 --> 00:45:25.920 Uh go bees. s474 00:45:25.920 --> 00:45:28.120 Buzz buzz. s475 00:45:28.480 --> 00:45:30.680 Thank you. s476 00:45:34.040 --> 00:45:41.040 How would you uh go and integrate the perception agents into your uh s477 00:45:41.040 --> 00:45:42.760 Oh, I actually I have an answer for that, too. s478 00:45:42.760 --> 00:45:43.720 Okay. s479 00:45:43.720 --> 00:45:55.680 So, as I've actually emailed myself inside kick to discuss later with my co-founder, is uh we developed a lot of product offerings but such as sidekick, um different tools for learning. s480 00:45:55.680 --> 00:46:01.680 Um the whole mission of making sure everyone knows how to adjust to the upcoming workforce displacement. s481 00:46:01.680 --> 00:46:09.120 But the bottleneck right now is product design of building something that users can understand exactly what to do. s482 00:46:09.120 --> 00:46:22.120 So, we would use the annotation feature of the perception agent to then um send that description either through sidekick through our own like agent we have running to to make improvements, but then to close that loop of development. s483 00:46:22.120 --> 00:46:22.480 Oh. s484 00:46:22.480 --> 00:46:23.600 So. s485 00:46:23.600 --> 00:46:28.840 Yeah, so if AGI would love to sponsor Ozzie and I will give you my card and we can talk later. s486 00:46:28.840 --> 00:46:29.480 So. s487 00:46:29.480 --> 00:46:30.280 Thank you. s488 00:46:30.280 --> 00:46:30.800 Awesome. s489 00:46:30.800 --> 00:46:35.920 Thank you so much for coming up and demoing and here as promised a B device. s490 00:46:35.920 --> 00:46:36.160 Thank you. s491 00:46:36.160 --> 00:46:38.560 Appreciate it. s492 00:46:39.960 --> 00:46:51.480 So, as we are wrapping this up, um just Emil, I'm just going to We are going to put up a survey on the board on the screen. s493 00:46:51.480 --> 00:46:58.680 Uh it's something that we Yes. s494 00:46:59.560 --> 00:47:01.840 You want to Oh, okay. s495 00:47:01.840 --> 00:47:08.240 Okay, there is uh Jonathan who wants to present as well. s496 00:47:11.440 --> 00:47:16.280 Uh so, mine's more of like a concept that I didn't have time to build everything. s497 00:47:16.280 --> 00:47:20.000 Um but I'll just kind of walk through it. s498 00:47:30.240 --> 00:47:30.520 All right. s499 00:47:30.520 --> 00:47:39.160 So, I was kind of looking at the uh the proxy for the B and trying to figure out how are we getting the information from the CLI. s500 00:47:39.160 --> 00:47:41.560 Oh, it's not. s501 00:47:41.560 --> 00:47:42.200 It's not sharing. s502 00:47:42.200 --> 00:47:42.800 Oh, there we go. s503 00:47:42.800 --> 00:47:43.600 All right. s504 00:47:43.600 --> 00:47:56.360 So, I was trying to figure out uh how the proxy actually works and how we're getting information from the B and uh besides just doing transcripts uh transcription could you pull different data from different sources? s505 00:47:56.360 --> 00:47:59.560 Could you pull up from an IoT device for whatever reason? s506 00:47:59.560 --> 00:48:07.400 Um And what I started to come up with is uh at the company I'm at, which is a finance company, we have a lot of non-technical people. s507 00:48:07.400 --> 00:48:19.640 And not only do we have to get our uh criteria in, we also have like a lot of UI UX um that we have to try and capture, and we don't always get it in like the best s508 00:48:19.640 --> 00:48:23.120 details uh for acceptance criteria. s509 00:48:23.120 --> 00:48:35.720 So, I've been pushing the team to use uh Jam MCP, which basically allows you to do a recording, captures all the console logs, captures the network requests, and then uh feed that information. s510 00:48:35.720 --> 00:48:41.160 Um we also have I built an AR AI harness so we can pipe this information to it. s511 00:48:41.160 --> 00:48:53.560 So, really the architecture kind of looks like this, where very similar to like the um the CLI, you could actually take the Jam MCP, pipe that information into the proxy, s512 00:48:53.560 --> 00:49:00.240 have the LLM go out to you um Nova, and then actually take the UI uh implementation. s513 00:49:00.240 --> 00:49:11.960 So, if you're saying things like, "Oh, I wish this uh would say, you know, uh this investor schedule needs to go like this um or this score should be s514 00:49:11.960 --> 00:49:16.680 uh a research level score and be in a different location." s515 00:49:16.680 --> 00:49:25.320 Uh we'd be able to actually pull that information from the MCP, send it over to Nova, and actually make the changes on the AI harness that is running uh locally. s516 00:49:25.320 --> 00:49:30.800 So, you could do the proxy the same way, um but through a different device. s517 00:49:30.840 --> 00:49:32.520 Um did some initial tests. s518 00:49:32.520 --> 00:49:41.240 It looks like it works even through just some mocking, uh but wasn't able with the Wi-Fi to get uh the fig or the Jam MCP up and running. s519 00:49:41.240 --> 00:49:43.920 But, that'll be my test when I get home. s520 00:49:43.920 --> 00:49:45.160 So. s521 00:49:45.160 --> 00:49:47.160 Thanks. s522 00:49:54.520 --> 00:49:58.440 Thank you so much and here you go. s523 00:49:58.920 --> 00:50:06.520 So, right now we have 15 more minutes that will close the workshop. s524 00:50:06.520 --> 00:50:12.360 Um we will put up a QR code on screen. s525 00:50:12.360 --> 00:50:25.280 If you feel that you have a great idea and you don't want to come up and talk to us about it you can submit these ideas here and we'll do a s526 00:50:26.560 --> 00:50:33.520 we will sub we will vote on those ideas and the top three ideas will get a TB device. s527 00:50:33.520 --> 00:50:43.000 So, if you have an idea please put it in those the poll. s528 00:50:43.480 --> 00:50:46.680 Just give me a second. s529 00:50:47.680 --> 00:50:48.240 There you go. s530 00:50:48.240 --> 00:50:49.080 You guys see that? s531 00:50:49.080 --> 00:50:51.080 Yeah. s532 00:50:52.960 --> 00:50:59.720 So, we want them to scan the QR code and enter an innovative idea using what we've learned. s533 00:50:59.720 --> 00:51:07.120 What would you use this perception, be all the things that you saw today, all the things you played around with today? s534 00:51:07.120 --> 00:51:17.120 What do you think would be a good use case for you that will help you make your life easier on a everyday basis? s535 00:51:19.160 --> 00:51:22.240 We saw a bunch of ideas. s536 00:51:22.240 --> 00:51:32.320 Now, and if you think you had one you're feeling a little shy there's a chance to win another device. s537 00:51:33.560 --> 00:51:34.000 So, what do you think? s538 00:51:34.000 --> 00:51:35.560 5 minutes and then we'll s539 00:51:35.560 --> 00:51:39.920 5 minutes and then we will Yep. s540 00:51:44.600 --> 00:51:47.040 Let me fix that. s541 00:51:49.240 --> 00:51:53.360 Okay, I stopped the poll now. s542 00:51:53.360 --> 00:51:56.360 So, all of them great ideas. s543 00:51:56.360 --> 00:52:01.800 I was reading through them and I was discussing with the mail there like so hard to pick. s544 00:52:01.800 --> 00:52:04.080 Is Isn't it a mail? s545 00:52:04.080 --> 00:52:16.640 So, we have the one with pixel judge. s546 00:52:17.040 --> 00:52:20.720 Can someone raise their hand who submitted that idea? s547 00:52:22.760 --> 00:52:24.920 No one? s548 00:52:25.200 --> 00:52:27.560 Pixel judge? s549 00:52:27.680 --> 00:52:31.320 Uh What's that? s550 00:52:33.080 --> 00:52:34.400 Okay. s551 00:52:34.400 --> 00:52:41.440 Um the next one was the agentic end-to-end testing for a product. s552 00:52:41.440 --> 00:52:42.520 Okay. s553 00:52:42.520 --> 00:52:44.960 Come on up here. s554 00:52:48.760 --> 00:52:54.560 Do you want to have Yeah. s555 00:52:54.560 --> 00:52:56.560 Awesome. s556 00:52:56.720 --> 00:52:59.040 Thank you. s557 00:53:00.920 --> 00:53:09.680 The reframe a life thinking canvas for brainstorming and planning. s558 00:53:09.880 --> 00:53:10.600 That's you? s559 00:53:10.600 --> 00:53:13.040 Come on up here. s560 00:53:19.760 --> 00:53:28.520 So, we had decided we'll give away three and one was not claimed. s561 00:53:28.520 --> 00:53:33.560 So, we have we did actually circle down to four. s562 00:53:33.960 --> 00:53:42.840 The last one is build a canvas app where human perception agent work on a UI design together in real time. s563 00:53:42.840 --> 00:53:45.280 Go on, up here. s564 00:53:48.440 --> 00:53:51.120 Thank one. s565 00:53:52.880 --> 00:53:58.040 A huge round of applause for all the winners here. s566 00:54:12.280 --> 00:54:20.080 Okay, so I just want to wrap the session up with uh something like a key takeaway. s567 00:54:20.080 --> 00:54:30.840 The first the most important thing of this workshop, you can all imagine, is the way human and agent collaboration is happening is changing. s568 00:54:30.840 --> 00:54:45.040 So, we need tools that will make the visual piece of how the um human and agents are collaborating, and this is what a step next to building perception perception agent is. s569 00:54:45.040 --> 00:54:51.240 So, you can get started with Nova Act if you haven't already, you know, looked at the GitHub repo. s570 00:54:51.240 --> 00:54:54.320 It's available, you can scan this QR code. s571 00:54:54.320 --> 00:55:06.640 It's an SDK, and also we have a blog post by our team that you can look it up explaining how what we are thinking about perception agent at large. s572 00:55:06.680 --> 00:55:13.320 So, with that, thank you everyone, and we also have a feedback um QR code up. s573 00:55:13.320 --> 00:55:22.120 If you think you like the session, you like Emil, uh you liked me, you liked Giovanni here who presented, please leave us a feedback. s574 00:55:22.120 --> 00:55:24.200 We appreciate all your presence here. s575 00:55:24.200 --> 00:55:26.680 Thank you so much. s576 00:55:42.640 --> 00:55:44.640 Mhm.