Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab

AI Engineer · 55 min · 576 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hello everyone and thank you for coming to our workshop building with perception agents.
00:17Uh my name is Emil Basel.
00:19I lead a team of solutions architects uh within Amazon AGI Lab.
00:23And our team is focused on two things.
00:25The first is working with some of our earliest adopter customers who want to work with some of our early models and products to help them get into production.
00:34And the second part is closing the feedback loop from our customer engagements, uh building evals for our research team to train our models on, and bringing product uh and engineering feedback
00:46to those teams as well.
00:48And I'm joined by one of our stellar solutions architects, Shruti Arora, who's going to be running the workshop today.
00:56Uh just a bit about what we're going to be talking about today.
00:58I'll be giving a quick overview of the Amazon AGI Lab, and talking about the first product that we shipped last year, Amazon Nova Act.
01:06And then we'll get into perception agents, and that's what the focus of today's workshop is, and then we'll get into the workshop.
01:16So a bit about the lab.
01:17So we are a frontier research and product team that combines startup speed with the scale and the resources of Amazon.
01:27And we are really focused on building knowledge work agents that can perceive, reason, and act in a variety of different environments to complete real-world tasks.
01:38Um our offices are in San Francisco and Boston.
01:41We've got a few other talks going on throughout the week.
01:44We have a booth at the expo, so stop by to collect some swag, meet some folks from the team, and hear some of the talks there.
01:52So I want to share a bit about our approach and how we think about agents.
01:58Across three different ways.
01:59So, the first is around collaboration.
02:00So, we really think about building agents that are teammates much more than just a black box.
02:05And when we say teammates, we really want an agent that can understand what our goal is, our objective is, and can share how they're thinking about achieving that goal.
02:17And and then grounding all of that in actual customer uh use cases.
02:25And so, we work very very closely with a large number of customers to really help ground all the work that we're doing with agents.
02:32And the second is around perception.
02:35So, agents need to understand their environment.
02:37And so, for us, because it's a perception-based agent, the environment spans multiple devices.
02:44So, we started with browsers last year, but you can imagine any visual screen is an environment for our agent to carry out knowledge work in.
02:53So, mobile devices, and also computer use is the other one.
02:59And the third one is reliability.
03:01And we have a great customer quote that said, "If my agent works 50% of the time, it doesn't work at all."
03:06And that's because you're either not going to run it at all, or if you do run it, you're going to end up uh debugging it and spending all this time, and you're going to basically do away with all the gains that you're getting from that agent.
03:16And so, we spend a lot of time building and training our models in uh simulated environments for reinforcement learning.
03:23Um and again, a lot of those are based on real-world uh use cases, which we get from not only our external customers, but a lot of internal Amazon teams that are carrying out a lot of uh knowledge work.
03:38Cool.
03:38So, with that backdrop, uh I want to talk about the first product that we shipped last year.
03:42So, Amazon Nova Act is a first-class AWS service that allows developers to build and run reliable browser-based workflows at scale.
03:54So, using natural language, you can drive a browser workflow to populate a browser, navigate, search with it, fill out forms, and call a human in the loop if you need a human for a expense report approval,
04:08or if you get stuck, or other things like that.
04:13And there's kind of four primary use cases which we've seen customers adopt this for.
04:17The first is form filling.
04:19Uh so, if you have to do that at scale across a large number of different websites, and you don't want to write a prompt for each single website, or write like a
04:27like a like a a deterministic script for each single site, you can use Nova Act for that.
04:33Uh search and extract.
04:35If you want to do research and get data from a lot of different websites, shopping, and QA.
04:42So, a bit more on what we launched.
04:45Uh the Let me just get through all these.
04:48Uh so, it's a first-class AWS service.
04:51So, if you go into the AWS console, type in Nova Act, you will see Nova Act there.
04:55And from the console itself, you can see all of the all of the trajectory.
04:58So, it's a very easy way to see everything which your agent has done, the bounding boxes around all the elements which it's looking at, and it's trying to take actions on there.
05:09And then behind the scenes, we have a custom computer use model which is doing all of the all of the inference there.
05:16This year, we launched an MCP server as well as a skill, and you're going to be using those in the workshop, and that's for your agents to be able to use
05:24Nova Act very easily.
05:26And then we've got human in the loop and tool use.
05:28And we also have a playground.
05:29So, if you go to the Nova Act homepage, it's a no-code way to just see Nova Act in action there.
05:37Now, looking ahead, and the topic of today's workshop is perception agents.
05:43Now, why do we feel there's a need for perception agents?
05:46So, uh for those of you who are not familiar with the term of computer use, uh that's basically the way for an agent to take an action on a computer screen, right?
05:55So, you see a blue button in the right corner, and you tell your agent, "Hey, go click this blue button over here."
06:00Very important, very powerful, it's very important for uh automating workflows, but we feel that there's another half of that, which is the agent actually understanding what it's seeing on the screen itself,
06:10not just using that as a sort of blind uh screen that it can just go and click on elements there.
06:18And we really want this agent to be able to understand screen just like a human would, right?
06:22So, if I showed someone a uh email client from like the late '90s, that was like a Yahoo email client or Hotmail client or something which you've never seen before, you would probably be able to figure out how to send an email from that, even though you've never used that client before, cuz we've
06:39got this collective understanding of thousands and thousands of hours of using these screens that we kind of intuitively understand how to use these things, and that's really where we're trying to get to with these
06:51with these agents.
06:53Uh and the second one is these agents don't require APIs in order to work, right?
06:58So, if you think about a lot of knowledge work, um it's a lot of humans using screens that just don't have APIs, and that's why there are people who are doing these things.
07:07And we want these agents to be able to carry out the same work across a variety of sites as well.
07:13And we want the models to see the screens the same way that we do, because that really facilitates the collaboration, right?
07:19So, rather than typing up paragraphs for what you want the agent to do uh on this screen and describing the screen, uh you can just point, click, and describe
07:30changes that you want, and that's what you're going to do in the workshop today.
07:32It's you're going to click on a couple elements and say, "Hey, I want to change the background to red."
07:36And that's it.
07:37And then you just have the agent go go and do that.
07:43And uh to to start with we've open-sourced two perception agent primitives, and you're going to be using these in depth in the workshop today.
07:53Uh the first is visual annotation.
07:55And so what this lets you do is on any webpage, you can point, click, and just describe changes that you want to make.
08:02And when you're done with that, you click export and it exports out a JSON file with all the changes that you want and a prompt, which you can give to your coding agent
08:10along with that JSON file to say, "Hey, go and make these changes and do that."
08:14So important is uh to to note, this has nothing to do with inference.
08:17This is purely deterministic.
08:19It just outputs JSON, doesn't have It's not tied to any model or anything else like that.
08:25The second part uh the second primitive which we've launched is visual verification.
08:30And if you think about a a site that you have, you've got certain CSS styles and designs which you want your your site to adhere to.
08:39Uh you've got different workflows which you want your site to support.
08:41Maybe have like a checkout flow or a login flow or something else.
08:45And you can describe those as Gherkin files.
08:48And so you can have visual verification, this this skill, um run by your coding agent against these design artifacts that you have, against these Gherkin files, and make sure that your site is adhering
09:00to those.
09:01And again, this is something which you're going to use uh in the workshop today.
09:05And again, these are all open-sourced, available today.
09:08And uh yeah, that's what we're going to do in the workshop.
09:12So I think at this point I've spoken enough, and I am going to hand it off to Shruti, who is going to kick us off on the workshop.
09:23Do you just press that?
09:24Thank you.
09:27All right.
09:28Can you all hear me okay?
09:29Yes.
09:29Okay.
09:31Welcome.
09:32Uh thanks Emil for that wonderful introduction to our uh Amazon AGI labs and the products that we launched, we released earlier in May this this year.
09:44So, as he mentioned, we are going to be doing hands-on workshop, but hands-on on like whatever we released the UI and the visual annotator and the visual verification.
09:59I have a short demo here, but before we dive into the demo, I just want all of you to like step into this role play here.
10:08Just just bear with me.
10:11Imagine you are a front-end engineer and you are building a web application.
10:18You have a design team and you are taking inspirations.
10:23They have these Figma files that you are building against.
10:27You are also collaborating with a sister team that they want to put in a piece of their information on the web page that you are building.
10:37So, if you are if you think about it that you have the designs, you'll probably take like four to five weeks to like build the designs and build everything code base and
10:47actually build that website.
10:50Now, in the third week or like fourth week around that time of your building process, you actually think, "Oh, maybe I should like go back and like verify if this looks fine, if everything looks okay with the rest of the team."
11:03And you are constantly collaborating, right?
11:05So, you are checking if if the work that you have done is something that is that is what you want to ship.
11:12Um so, you send your code base to you share the GitHub or however you are doing your GitLab or whatever you have your code in and you they run that this this web server locally and they open up the website
11:28and they say, "Oh, this this feels off.
11:31You know, the head and heading should be probably not in red.
11:35I should I want to switch it back to something.
11:37Um, and they provide this feedbacks.
11:40Probably they will do it in like a Slack message or in a shared doc, for example.
11:46Uh, and then once you get those feedback on that on your website, you would um, then like go over one of them.
11:55You'll disagree, agree go to some of them, come to a middle ground, and finalize this is the feedback you want to incorporate.
12:03You have to go through all of them and one by one then and start making the changes.
12:08Uh, now with coding agents, I guess it'll take maybe over which used to take maybe a week or it can like reduce to like maybe uh, few few days.
12:19Now, think about if you had the ability to take their feedback directly and pass it to the coding agent without you your supervision and just like once the changes have been done, that's when you interfere and say, "Oh, this looks good.
12:33All the changes have been incorporated."
12:35How would that reduce the time?
12:37Maybe few hours.
12:38You just start uh, collect the feedbacks, send it to your agent, and go for a coffee break, come back, and all the changes have been merged.
12:48So, this is exactly what uh, Emil was mentioning.
12:52Uh, that we have a Chrome extension that we are calling as visual annotator.
12:57Um, there are three ways to annotate.
13:00You can see here, it's either you draw on the page or you pick the element or you drop a pin as you would drop a pin on a Google Maps, say, and provide your annotations.
13:10And once you have your annotations, you can download these and send it to your team who needs to make the changes or even if it's you who's make the changes who's making the changes,
13:20then we provide a AI-friendly prompt that you can send it to your agent and it's going to change automatic like well, you have to instruct the agent to like change
13:30the good base.
13:33So, this was the visual annotator.
13:36The next um Oh.
13:44primitive that we launched is the visual verification skill.
13:47And when I say skill, it is actually a skill that you can install on Claude or Keynote Keynote power.
13:54And um you would see these verification as I may mention, these are the design files that have against which you will be running your visual like um verification against.
14:05Like if does the design colors look fine?
14:09Is the font size right?
14:11Is the user flow right?
14:13But, one thing that will come to your mind like these design specs, these flows, how are these generated?
14:20How are how how you going to build that?
14:23Not to worry, we have a solution for that as well.
14:26When you run your verification for the first time, the there will be a a deep um one level surface crawl that will be done by the UI verification tool
14:39on your website to go and check what are the links, what are the content, and based on that it's going to create these user flows that are in terms of
14:51in Gherkin for syntax that the UI UI verification is going to use when it's going to run the verification next.
14:59So, let me just play that play the demo here.
15:03Um so, you can see you just like call the tool.
15:06It's going to go through the design docs specs specs specs and the user flows.
15:13Uh go through it.
15:14It's going to open up the browser, run through the design flow user flows, and then generate a report.
15:22So, this these are the two tools that you will get hands-on in this workshop.
15:28Uh so, I'm going to jump to what we have how the workshop has been laid out as of now.
15:35First, as I mentioned, two uh hands-on.
15:38The second thing that's what we are trying to do is these are the two primitives, of course.
15:43Perception is not just these two primitives.
15:47How would you wire these two primitives up for a particular use case?
15:51There is a scenario that has been uh given in the workshop.
15:56We are trying to make this fun, so please try it out yourself uh and see like it's like a game day.
16:04You would see there are like certain hints for the to solve the problem, what the problem is, what the end goal is.
16:09So, everything is laid out in the workshop.
16:11Uh it's a GitHub link.
16:12We will share that with you in a bit.
16:15Uh so, how will you basically wire these two things up?
16:19Uh the sec- the next thing we have are um sponsors for the prices.
16:25We have certain contests that run at the end.
16:28And um thank you the B team for sponsoring the prices for the uh for the uh event.
16:35And um we will have a way how you can instead of typing or like annotating on the screen, what are some other ways cool ways of making sure that there are some annotations that you want to do or like feedback that you want to provide about a web page and its designs.
16:54So, B team will show you how you can do that beyond just typing in your feedback.
17:02Finally, there are Well, before that, we have After you finish the workshop, we also have like three different options for bonus challenge.
17:10We will have plenty of time for you to try out these tools in a particular use case that you want to maybe some you want to try wire it up with B
17:20and see how that how you would probably wire it with any other form of feedback that you would essentially get from your team.
17:30Or if you think that hey, this is really cool.
17:34I want to build something that I want to show to my team on Monday like the following Monday.
17:39So, go for it and show if you're brave enough come in front come at the stage show your use case demo your use case and you'll get a B device.
17:49The final part is we're going to do a little fun exercise where once you have a good understanding of these two primitives you can think of like multiple ways of how you would use it on a daily
18:05on your day to day day to day and we will you will submit these ideas of like hey, I think this will be really cool for me to use this and we will have people vote on for these ideas
18:17and the top three ideas will get a B device and I think we also have one for crowd favorite.
18:25So, with that going to jump into the workshop.
18:30This is the link for the workshop.
18:34You can type in it's pretty it's it's available on GitHub so you can type in perception agents workshop.
18:41You will probably get the first GitHub link.
18:45Um in the workshop we also need you to have use a Nova Act API key.
18:54So, there is a piece where in the workshop we have obscured the first three letters.
19:01So, that is on the screen here.
19:04So, you can like copy paste these three letters ACE that stands for ace and you're good to go with the Novact API key.
19:17So, let me just pause this.
19:24So, in the workshop, you would the readme will basic point you to the workshop with MD file.
19:30And here is the scenario that I was talking about.
19:33I'm going to read it out from here.
19:34You are a front-end developer preparing to ship a landing page for a podcast.
19:40Yesterday you demoed the page to a few peers and they provided some feedback.
19:46The hero title feels too big.
19:48The accent color doesn't pop up the way you it should.
19:51Some buttons are oversized and a few small things that you slightly just off.
19:58So, you are on board with those feedback.
20:00And today you come in fresh.
20:03And I'm ready to incorporate those changes.
20:07So, what do you have at your disposal?
20:09The Novact Chrome extension, the UI verification scale, you have the Novact SDK, and the Novact MCP server.
20:18These are the tools that you have at your disposal to basically implement the solution where you would essentially build build this entire loop by itself.
20:31Like, you just annotate and automatically you would just hit a button in the Chrome extension and all the changes will be applied.
20:42The UI verification will run and you have a report by the end of it.
20:47So, this is the challenge and there are a few prerequisites.
20:52So, um you need to have an macOS or a Linux tool because Novact MCP server runs on macOS and Linux.
21:01You should have Node.js installed.
21:03If not, you can install them.
21:05uh NPM, Python, uh Chrome browser to basically test it out because it's a Chrome extension, and then um an AI coding agent.
21:15So, if you have Claude, that works great.
21:17We also have Kiro integration.
21:19So, if you don't have any co- coding um agent, you can install Kiro and start playing around with it.
21:26Um and lastly, we have the UV um that's UVX.
21:31That's basically to install.
21:32If you don't want to install UV or UVX, then please raise your hand and we can help you install using pip.
21:39Um So, that's that's that.
21:44And the workshop is laid out for I think the setup should be about 5 to 7 minutes.
21:50Uh then you can like start playing around with those skills.
21:54That gets a good understanding of how would you use the Chrome extension, how would you use the uh UI verification skill.
22:01That's what has been explained here.
22:03Uh to solve the challenge, you have um about 15 minutes in if you want to go that route, like I want to go step by step.
22:13And I'm just going to go scroll down real quick.
22:16Yeah.
22:16So, choose your path.
22:18I have this workshop designed in two levels.
22:21Uh you you remember I said like I want you to have fun.
22:24So, two levels, if you want to go for a challenge, go for the medium route.
22:28If you feel that uh I'm just going to like uh follow the instructions and get the workshop done.
22:34You'll feel good about it.
22:36So, yeah, we have an easy mode where you'll have exact instructions what you what should be done, and you will have these two primitives wired up by the end of the hands-on.
22:46Um so, 15 minutes we are expecting the easy path, um and then 30 to 35 minutes for the medium.
22:53Once we have allocated the 15 minutes time, I will come up and show you where you would like essentially um add your code so that uh you can wire these two primitives up
23:06and you can see the entire thing come into life in action.
23:11And we also have a bonus challenge.
23:14If you finish early, you can start with the bonus challenge.
23:17Uh but before that we will have the B team Giovanni um in some time to like present.
23:27After Yeah.
23:28Yeah, sure thing.
23:29So I saw that you had as I was installing these uh
23:34Mhm.
23:35What is it the Nova Act UI verification skills?
23:39Um it it said that there were some defaults clients that are always supported and it included like Codex and Cloud Code.
23:46But then below that there are these additional skills and one of them is also Cloud Code.
23:51So I'm just curious like are there two different things going on there or are those the same?
23:57The the fact that there are two Cloud Code
24:00Yes, so there are two skills.
24:02One is the Nova Act skill.
24:05One is the Nova Act UI verification skills.
24:07So that's why you must be seeing both of them.
24:09Yeah.
24:13So I'm going to give you the time here to I'm going to bring up the slide back.
24:21I'm going to ask Emil to pop up here.
24:23Slide back on the GitHub repo and so that you have the A API key handy if you have to change.
24:29Feel free to like raise your hand if you have any questions.
24:33We have our great awesome team supporting the workshop.
24:37There you find them, see them waving.
24:40So reach out to either of us and we will pop by you to help.
24:47Have fun.
24:49Okay, I'm just going to try to walk through the solution so that Okay.
25:12So, this is the step-by-step instructions.
25:15You can see that we have the Chrome extension.
25:18We have your agent uh AI agent running.
25:22Uh if you want your coding agent to like automatically apply the changes from the extension, you need a bridge because there's no bridge between them.
25:33How would the extension like send the prompt to the coding uh coding agent?
25:37So, for that, we have built uh something called agent bridge that will automatically uh take the annotations and up- give prompted to the coding agent.
25:50That's where it will start applying those changes and then running the UI verification skill.
25:56So, um essentially in the extension, we don't have the apply changes button in the in the primitive mode, but I for this workshop, I have built in we have built this um apply changes button
26:12in the Chrome extension.
26:14Um I'll walk you through that in a bit.
26:18Uh so, for the Chrome for for this entire thing to work, you need to make changes in like two files.
26:26The first file is um the manifest.json file in the Chrome extension, where you essentially add the this line for the host permissions to start accepting requests from the local host.
26:42Uh and then you add this this piece of code.
26:47It's a huge like piece of chunk of code here.
26:52Um, and you can what this part does in the extension is creates and apply changes button, wires up with the extension, uh, make sure makes makes the UI verification
27:05run uh, in using the Nova at SDK instead of the skill, uh, and starts generates a report that you can click on and then view the report.
27:17Um, so this piece of code is goes in this file that is the, um, let's see, content.js in the extension and it needs to be added around like line 261.
27:35Uh, between this append child export button just before and I think I have hash, uh, there's a comment there add code here in that file and that's where you would like paste this code.
27:48So once you have that running now you would see this apply change button here in the extension.
27:59Um, I think, uh, Emil was testing this so he changed this to like FIFA World Cup, why not?
28:07Um, so I'm going to select one of these and I'm going to refresh the extension again.
28:20Let's see.
28:25Annotate current page.
28:28Uh, I'm going to try to change the published date.
28:34Change to today's June 20 9th.
28:42Okay.
28:43And save it.
28:46And this is the extension.
28:50Apply changes.
28:52So, in the background, it's going to open up your coding agent.
28:56It's been tested on Cloud Code and Kiro as I mentioned.
29:00So, if you're trying Cursor or Codex, the agent bridge is something that you would have to edit so that it can accept these apply changes button like requests.
29:14Um So, in a bit it'll start applying the changes and you will see certain logs that pop up as well in the repository.
29:24Um that is the key the CLI's logs as well as when the verification runs Nova Act has these logs uh that gives you exactly what the AI model was thinking
29:36and what part of the screen it was trying to make a decision on.
29:40And you can see that the button changed to run verification now.
29:44So, it's running the verification step.
29:48And in a bit it'll also generate a report that you can see.
29:51So, this is where you basically apply the changes and go grab a lunch or like break and then come back and just see uh the verification report once you're back.
30:05Going to give it a minute because it's going to go through all the design flows, all the user flows, and then generate a report.
30:13So, it takes a bit of a time.
30:19Let's see if I can pull up the repo.
30:24Oh.
30:25Shoot.
31:35I think the Wi-Fi is super slow, so But, I I think maybe in the meantime I can have Giovanni come over on the stage and show you a cool way to annotate your webs web pages
31:52instead of like just going and typing stuff on the web on the web page that you're working on.
31:59Hello.
32:00Oh, the prime.
32:01Okay.
32:02Can we have
32:04Uh I'm done.
32:05Can I see the slides?
32:07The slides, yes.
32:11Hello.
32:14Hi everyone, I'm Giovanni and I was thinking to imagine you are in to do your design meeting just talking about the design modification you want to do to that website.
32:26And then automatically your perception agent agent is able to modify the website.
32:32This is something that you can do right now with the workshop.
32:35And this why, how?
32:37Using the B wearable devices.
32:40So, B is a device we have in Amazon.
32:43And you can buy it like in the b.computer website.
32:47It's a wearable this is your AI assistant.
32:50So, it just have all it has a microphone.
32:53So, you can activate this the button and then it's going to listen for your conversation.
33:00No worries, there is a l l i d that's will be on if it's listening to your conversation and then all your conversation on your private cloud.
33:11After the AI agent that is running B is able to gives you suggestion about whatever conversation you had.
33:21So, what I did is like integrating with this this agent the B agent with the perception agent that we had for the workshop.
33:28So, getting the input listening from a AI conversation from a design conversation.
33:36Uh, I don't see the video.
33:38Yeah.
33:40Ah, this one.
33:40Yeah.
33:41Thank you.
33:43You can achieve this with three simple step.
33:46So, first of all, yes, you have to have a B. And I want with me that you can use here and now or we can have a test a test account
33:54to download that you can use to get the conversation as you want the conversation.
33:58Then you need to install the B C L I. It's a C L I that will enable that is going to allow you to get the data for your conversation
34:06and it's in the workshop the instruction to download the B C L I by just a npm command.
34:11Then you start the server of the workshop and then you start the bridge listening from B conversation and applying that to the perception agent.
34:23Then you just enable your B device.
34:25The LED is on.
34:27You say something like, "Oh, I want to change the title color to red.
34:30I want to change the emoji instead of our brain emoji, I want a robot emoji.
34:35And on the right side, you will see your conversation, your transcript of the conversation, and even the most important points.
34:42Here you can see the response API, the the API response coming from the B. You can open the panel and check.
34:50And then you can just select apply.
34:53And here is going to be the normal flow of the perception agent workshop.
34:57So, it's going to change the color, the color that you just said, it's going to change the emoji.
35:03And after the modification of the website are applied, it's going to run even for you automatically the verification phase of no act.
35:16So, we have a test account that you can use with a preloaded conversation, preloaded design conversation.
35:23It is like just talking about, "Oh, we change the color, we change the emoji."
35:27Something like that.
35:29And you can after you install the CLI and you log in with the B account with a test device with a test account, you just click on the bottom
35:38button, the the button at the bottom.
35:41Uh this is the verification page you can check.
35:44But otherwise, if you want to test the B device, we can like you do we can actually just use mine and I will show you around if you want it.
35:51Like we can actually see like have a conversation about the design and you can see that it's actually applying this after like two or three minutes of elaborating the conversation.
36:01The Wi-Fi is a little bit slow, so that's why I didn't do the live demo.
36:04Uh and yeah, and then in the end there is there is the verification report that you can check.
36:09The B devices are also our prizes for the for the challenge.
36:13So, yeah, you're going to have one if you win the challenge.
36:16And we also are at the booth today and all the week.
36:22So, yeah, let me know if you want to integrate B and I'm here for you.
36:26So, B device, if you want to come up on the stage, we have 10 minutes for you to like explore these two primitives, think of an idea that you want to probably use it.
36:40And if you want to come up, you'll get one of these devices.
36:43Um so, yeah.
36:46The if just get creative, I guess.
36:52Hey everyone, I have Jed with me who built something really cool and he would like to show it to everyone.
36:59Jed, come up on the stage.
37:02Yeah.
37:04Jed.
37:05Oh.
37:05What am I doing?
37:06Where am I going?
37:07Oh, I called Jed, but yeah.
37:08Oh, I I My name is Jeff, so I thought
37:09You are Sorry.
37:12I'll have Jeffrey next to show something cool as well.
37:15Oh, okay.
37:17All right, this this was just goofy, but it's actually an an an interesting little funnel here is is with this animators going through and Oh, you can't see it.
37:33Change it to a duck podcast.
37:40There we go.
37:43So, just turn asked it to turn it retro NES.
37:48Uh do some duck jokes and went back and did the verification funnel.
37:55Um it's probably as what I was telling them is probably really interesting to me here is to watch this this great AI said I made it a duck you know, I I made a duck icon.
38:08But uh often as everyone here likely knows is you don't actually see that happen, but then it it went through and did the the entire verification funnel, came back with the reports
38:22saying, yep, it's done.
38:23I did it.
38:24I actually did it.
38:25So, that was cool.
38:29Oh, yeah.
38:33I had it I had it just do asked it to do some jokes.
38:39Which is pretty bad at.
38:45There you go.
38:52Oh, because you came up with a cool demo.
38:57Do you have to step?
39:01As promised, you win this.
39:03Oh, thank you.
39:04P device.
39:10UP NEXT, WE HAVE Jeff Lee who wants to come and present what he did.
40:00Yeah, so I'm still building a test harness to actually be able to show this cuz the default B account has a uh sample conversation that isn't relevant to the things.
40:10So, this is just this is just something trying to uh bring the B to life.
40:14So, the idea was to have access the BCLI um and collect the summary of a of a conversation which was a discussion of of a backlog features that would be added to a
40:28uh repo that you're working on, right?
40:30So, you're working on a specific uh GitHub repository that's that's that that's cloned and then um uh what's going to happen is your conversation is going to be used in context with
40:41uh all of the code that's that's there.
40:43It's going to build a backlog based on the the summary and it's going to use the Atlassian MCP to put that into Jira.
40:50So, that's the uh that's the uh idea here.
40:54And so, the key thing that I'm just building right now in the background is uh the uh the test harness to insert a new conversation that's relevant to the the repo.
41:02So, that's Pretty simple and uh yeah.
41:11Anyway.
41:12That's awesome.
41:13Thank you so much.
41:13Thanks.
41:17You're welcome.
41:17So, here you win your new device.
41:19I know you wanted to go and get it, but you won it.
41:22Thank you very much.
41:29So, You have a question?
41:37Ryan, who's going to present next?
41:43Hello, everyone.
41:45Excited to be here.
41:53Let me see.
41:54Start mirroring.
41:56So, I was I was late to this session uh because the other session ended at 3:00.
42:02So, um I came up to uh Giovanni.
42:07Let me shift the key right here.
42:12Oh, perfect.
42:13That's fine.
42:14So, I gave it to Giovanni.
42:16Um currently wearing a bee hat.
42:18I currently attend uh Wharton uh business school.
42:22My cluster's mascot is a bee.
42:24I am one of three people who care that the mascot to be.
42:28Um so, you can imagine my excitement when I came to this conference and saw we were presenting about uh bee device.
42:34So, naturally, being late, being a Wharton grad, I went and asked if I could have a free bee device.
42:40Uh Giovanni said, "You have an idea to share?"
42:43So, I I currently working at an AI upskilling startup.
42:46Um I'm not a huge fan of Linear.
42:49Apologies to anyone who works at Linear.
42:52I just needed a better way of understanding what should I be doing anytime during the day, going from capturing an idea, discussing with my co-founder, and then updating a project repository.
43:03So, this is a natural use case for the bee device.
43:06So, we have a internal tool called Sidekick.
43:09Emphasis on an internal because there is no design.
43:13Um in my co-founder's defense, he did tell me to uh tell Claude to make it look pretty, no mistakes, and this is the best it did.
43:21So, um but how it works is we have a a joint inbox.
43:29Um you can ignore the Go Touch Crash SF.
43:31I just see all these billboards and um gets me concerned.
43:35Uh that we send emails jointly to uh capture@sidekick.ausin.org.
43:41And these are all the different inputs we have um thoughts throughout the week, newsletters.
43:46I don't know if anyone here is subscribed to an AI newsletter.
43:49Um see something interesting, we send it to inbox.
43:53And we are big believers in getting things done, which was a uh mode for knowledge workers of organizing a projects task developed by David Allen in 2010.
44:03And oh, if the Wi-Fi works, you could be able to see it.
44:07Um Sorry, that did just work off the screen.
44:12One second.
44:13So, the getting the getting things done tree shows all the projects, and under each project it has uh the strategy, has what the what completion looks like, tasks, next steps, references.
44:27And we've built an MCP so we can sit and talk to Claude and strategize on, "Hey, this is what we're thinking on this project.
44:34What do you think we should be doing?"
44:37All to solve the need of when you're working after you have a meeting of like, "What should I be doing right now to tie to my ultimate strategy?"
44:43So, how we imagine this to work with the uh B uh device, as Giovanni uh informed me there's an MCP, would be capture different meetings or different ideas, have it sent to our capture inbox, and then use that to update our strategy.
44:58Um I wish this was was loading because, let me see.
45:02Oh, that's why.
45:04my phones.
45:06See that worked.
45:07Um Yes.
45:11You know, I'm now 0 for 11 for live demos.
45:15Um but that is that is my concept of the B device.
45:19I'm looking forward to using it and sharing on LinkedIn how much I love the bees.
45:24Uh go bees.
45:25Buzz buzz.
45:28Thank you.
45:34How would you uh go and integrate the perception agents into your uh
45:41Oh, I actually I have an answer for that, too.
45:42Okay.
45:43So, as I've actually emailed myself inside kick to discuss later with my co-founder, is uh we developed a lot of product offerings but such as sidekick, um different tools for learning.
45:55Um the whole mission of making sure everyone knows how to adjust to the upcoming workforce displacement.
46:01But the bottleneck right now is product design of building something that users can understand exactly what to do.
46:09So, we would use the annotation feature of the perception agent to then um send that description either through sidekick through our own like agent we have running to to make improvements, but then to close that loop of development.
46:22Oh.
46:22So.
46:23Yeah, so if AGI would love to sponsor Ozzie and I will give you my card and we can talk later.
46:28So.
46:29Thank you.
46:30Awesome.
46:30Thank you so much for coming up and demoing and here as promised a B device.
46:35Thank you.
46:36Appreciate it.
46:39So, as we are wrapping this up, um just Emil, I'm just going to We are going to put up a survey on the board on the screen.
46:51Uh it's something that we Yes.
46:59You want to Oh, okay.
47:01Okay, there is uh Jonathan who wants to present as well.
47:11Uh so, mine's more of like a concept that I didn't have time to build everything.
47:16Um but I'll just kind of walk through it.
47:30All right.
47:30So, I was kind of looking at the uh the proxy for the B and trying to figure out how are we getting the information from the CLI.
47:39Oh, it's not.
47:41It's not sharing.
47:42Oh, there we go.
47:42All right.
47:43So, I was trying to figure out uh how the proxy actually works and how we're getting information from the B and uh besides just doing transcripts uh transcription could you pull different data from different sources?
47:56Could you pull up from an IoT device for whatever reason?
47:59Um And what I started to come up with is uh at the company I'm at, which is a finance company, we have a lot of non-technical people.
48:07And not only do we have to get our uh criteria in, we also have like a lot of UI UX um that we have to try and capture, and we don't always get it in like the best
48:19details uh for acceptance criteria.
48:23So, I've been pushing the team to use uh Jam MCP, which basically allows you to do a recording, captures all the console logs, captures the network requests, and then uh feed that information.
48:35Um we also have I built an AR AI harness so we can pipe this information to it.
48:41So, really the architecture kind of looks like this, where very similar to like the um the CLI, you could actually take the Jam MCP, pipe that information into the proxy,
48:53have the LLM go out to you um Nova, and then actually take the UI uh implementation.
49:00So, if you're saying things like, "Oh, I wish this uh would say, you know, uh this investor schedule needs to go like this um or this score should be
49:11uh a research level score and be in a different location."
49:16Uh we'd be able to actually pull that information from the MCP, send it over to Nova, and actually make the changes on the AI harness that is running uh locally.
49:25So, you could do the proxy the same way, um but through a different device.
49:30Um did some initial tests.
49:32It looks like it works even through just some mocking, uh but wasn't able with the Wi-Fi to get uh the fig or the Jam MCP up and running.
49:41But, that'll be my test when I get home.
49:43So.
49:45Thanks.
49:54Thank you so much and here you go.
49:58So, right now we have 15 more minutes that will close the workshop.
50:06Um we will put up a QR code on screen.
50:12If you feel that you have a great idea and you don't want to come up and talk to us about it you can submit these ideas here and we'll do a
50:26we will sub we will vote on those ideas and the top three ideas will get a TB device.
50:33So, if you have an idea please put it in those the poll.
50:43Just give me a second.
50:47There you go.
50:48You guys see that?
50:49Yeah.
50:52So, we want them to scan the QR code and enter an innovative idea using what we've learned.
50:59What would you use this perception, be all the things that you saw today, all the things you played around with today?
51:07What do you think would be a good use case for you that will help you make your life easier on a everyday basis?
51:19We saw a bunch of ideas.
51:22Now, and if you think you had one you're feeling a little shy there's a chance to win another device.
51:33So, what do you think?
51:345 minutes and then we'll
51:355 minutes and then we will Yep.
51:44Let me fix that.
51:49Okay, I stopped the poll now.
51:53So, all of them great ideas.
51:56I was reading through them and I was discussing with the mail there like so hard to pick.
52:01Is Isn't it a mail?
52:04So, we have the one with pixel judge.
52:17Can someone raise their hand who submitted that idea?
52:22No one?
52:25Pixel judge?
52:27Uh What's that?
52:33Okay.
52:34Um the next one was the agentic end-to-end testing for a product.
52:41Okay.
52:42Come on up here.
52:48Do you want to have Yeah.
52:54Awesome.
52:56Thank you.
53:00The reframe a life thinking canvas for brainstorming and planning.
53:09That's you?
53:10Come on up here.
53:19So, we had decided we'll give away three and one was not claimed.
53:28So, we have we did actually circle down to four.
53:33The last one is build a canvas app where human perception agent work on a UI design together in real time.
53:42Go on, up here.
53:48Thank one.
53:52A huge round of applause for all the winners here.
54:12Okay, so I just want to wrap the session up with uh something like a key takeaway.
54:20The first the most important thing of this workshop, you can all imagine, is the way human and agent collaboration is happening is changing.
54:30So, we need tools that will make the visual piece of how the um human and agents are collaborating, and this is what a step next to building perception perception agent is.
54:45So, you can get started with Nova Act if you haven't already, you know, looked at the GitHub repo.
54:51It's available, you can scan this QR code.
54:54It's an SDK, and also we have a blog post by our team that you can look it up explaining how what we are thinking about perception agent at large.
55:06So, with that, thank you everyone, and we also have a feedback um QR code up.
55:13If you think you like the session, you like Emil, uh you liked me, you liked Giovanni here who presented, please leave us a feedback.
55:22We appreciate all your presence here.
55:24Thank you so much.
55:42Mhm.