Tell the Robot What You Want — Sandhya Subramani, AWS

AI Engineer · 17 min · 210 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13Hello everyone.
00:14My name is Sandy and meet my co-host today, Scout.
00:20This is my friendly rover.
00:23And one would think that rovers can't really think for themselves, right?
00:29We have to tell them what to do or we have to very specifically program them on how to think.
00:34But this little guy here actually has a brain and he can think for himself.
00:41Let me show you my screen.
00:43Oh no, it's going to the wrong screen.
00:44I'm going to see how I can stop this and I'm going to see how I can move to my screen.
00:50Um, give me just a second.
00:53I'm going to end show and then we get to this.
00:56So this is what Scout here is looking at.
01:01And Scout here has a small little brain.
01:04And Scout can understand what I'm saying in natural language.
01:09For example, if you can see my screen here, if I say, "Hey Scout," I'm going to type to him.
01:15Say, "Hey Scout, turn on your headlights and say hi to everyone.
01:27he is actually going to be able to understand and do those actions.
01:32But how is he able to do these things, right?
01:34It's going to take him a couple of seconds to think about it because how this has been set up is he is working on with a Raspberry Pi installed
01:44and that Raspberry Pi is actually in a small little location in New York City and he's got a SIM card here which is then connecting to that to that Raspberry Pi and then he's getting signals.
01:56So, this is via 4G connection.
01:58And so, he's like, "Oh, hey, on it."
02:00And if you can see, you know what?
02:01I'm going to tell him, "Spin around."
02:04And oh, no, he's actually speaking to me.
02:06He's like, "I see the stage and it's looking great.
02:09Big presentation screen to the left, bright stage lights, and rows of seats for the audience.
02:15How How many people do you see?
02:21I've never trained him to answer this question, but he's Let's see if he's get getting it.
02:28He's thinking for like a second or two.
02:32Let me take a good look from the front camera.
02:35I see two people.
02:36One near the speaker monitor set up at the back of the stage and another person further right.
02:42Nah.
02:43spin 360 and he's also saying is this session about to start right so he's going to spin 360 any second now and then he's probably going to be like oh wow I did
03:01so this little robot here is a next generation of robot where ah there we go he is spinning 360 now and he's probably going to tell me what he's seeing
03:18and he's saying let's spin.
03:19Right?
03:20So this new generation of robots is to it's different from our traditional robot training because I have given this guy a little brain.
03:30And what do I mean by I've given him a brain?
03:34I've given this robot an agentic layer and I've given it it's called strands agents which is an open-source framework which was built by AWS and I'm going to quickly go back to my slide deck
03:49we can see it right and so here what happens is we have these existing tools that the robot can do he can take certain actions by himself but only those actions by himself
04:02so what we can do is we can add a layer of LLM or even better add a layer of agent to it so that the agent orchestrates which tool to call and how to really get the robot to start doing the things we want.
04:17So in traditional software with traditional AI machine uh like AI engineering we can give agents software tools.
04:26Similarly, we can give the same AI agent a hardware tool called a robot which has access to preset functions or programmable policies and then the agent can decide which policy to implement when.
04:42So all it takes is one robot agent for us to be able to do new innumerous tasks and have it understand what we're teaching it in natural language.
04:54So how do we get started with it?
04:56All it takes is five lines of code.
05:00This is through uh the agent harness called strands.
05:04And all we have to do is import the strands agent and call the robot tool.
05:10And we say ro tools equals the robot and then we say pick up the red cube and should be able to pick up a red cube assuming that the robot has that capability.
05:19Yeah.
05:20Now he's seen someone and he's like oh let me go towards that person.
05:23So he gets pretty excited.
05:25This guy is pretty special because he doesn't have just one agent.
05:29He's got three different agents.
05:30All three of them are strands and all three of them are working simultaneously.
05:35One of them is the thinker agent and that's the part of him that's constantly thinking and assessing the environment and like what do I do next?
05:42And that guy's that part of his brain is constantly thinking.
05:46Then there's the other communication part of it where and I'm going to show you that in a bit, right?
05:51and I've connected him to my telegram app as well as to my web app.
05:56And so he is able to have a conversation with me in natural language and then take actions based on what I am telling him to do.
06:03Apart from him just perceiving and thinking and figuring out what he wants to do.
06:08And the third agent, the third type of agent that he's got access to is a voice agent.
06:13I did have to disable it because every time I speak, he's going to think I'm speaking to him and so he's going to keep chatting away with me and it's just not going to be fun because we're going to have our co-host interrupting me all the time.
06:24So, I've disabled that feature for the time being.
06:28But essentially all three of these agents work in tandem with this one robot and thereby this gives him the ability to do way more than what just what he's been trained to do
06:42more than just the policies that he's learned.
06:45Now what is a quick overview on this trans package itself?
06:51This turns package has more than supports more than 40 different robots under eight categories.
06:58And all of these are just simple robot tool calls.
07:02And how is this all set up?
07:04Four different layers.
07:05The first one is the agent layer, the topmost one.
07:09And there are two parts to this.
07:10One is how the actions go in and the second is how it observes and the observations go up.
07:17So if you notice it's very birectional.
07:20So first when we give it an instruction we would be talking to this trans agent which is the agentic layer that would then decide which policy to call and the policy provider again
07:33stands agent supports a bunch of different policy providers and we can then train our policy based on our traditional robot training.
07:41So in our policies we would collect data and then we would train on it and we would sim create more simulation data and that policy then becomes a VLA
07:50model which then the robot would have access to strand agents would have access to and then it would invoke that specific policy based on the question that we're asking it or the command that we're giving it and that policy needs to sit somewhere right so that sits in the back end which could be your simulation environment or
08:10it could be a real hardware chip, your hardware environment.
08:14That is the back end on which that is the interface on which the policy is running.
08:19And finally, the output actually takes place in the physical hardware which is the robot.
08:25And so the robot ah see so now it's responding this even if he falls down he's supposed to be fine.
08:31He technically shouldn't um he technically shouldn't uh get hurt.
08:37He should be able to pick back up from where he um stops.
08:41Ah, okay.
08:42So, I'm telling him to go back a bit.
08:44Back off.
08:46Let's see if he actually backs off.
08:48Um, so that is the four layers of how to get started with building this, right?
08:55And what's happening under the hood, like a more picturesic view of what's the architecture of what's going on under the hood.
09:03We want everything is basically strands agents on the edge as well as on the cloud.
09:09We want to be able to train the VLA and the policies on with using agent core.
09:15Um and we want that to happen on the cloud but we also want to be able to call it directly on edge so that our robot can uh execute functions and policies faster.
09:28So this is sort of like a hybrid model where a part of it happens on the cloud and another part of it happens on the edge and strands can decide when to call which part of it.
09:39And so this helps with massive amounts of training as well when it's constantly collecting information and it's train able to train on that information and learn from itself but also just execute at runtime
09:51really really quickly.
09:53Now, like I said, the agent decides what to do and the policy decides how it should be done.
10:01But he's pretty smart.
10:02He should be able to pick himself back up if he's not fully fallen down.
10:06And he should be able to continue moving along.
10:10So, I think he's okay.
10:12Now, where does this leave us?
10:15And why is this so special?
10:17We started off with very traditional robots.
10:21Robots have existed since forever, right?
10:25And they've always just been programmed, pre-programmed to do to autom be automated and do a certain set of tasks autonomously.
10:34But there is a future in this world where this these robot policies, these VLA models could be so advanced that we wouldn't even need to do this.
10:47They could be as large as our large language models.
10:51So that ah wait hang on he's falling back again.
10:55I'm gonna see if I can get him to move back up.
10:59Good boy.
11:00Stop.
11:02Then he's fallen off again.
11:04Um we get to a point where these large language the the VA models could be as large and as amazing as our larger language models and they know they have all the information in the world and we wouldn't even have to do this.
11:18we might just have to feed in one simple model and then we could give it to him and then he would know exactly what to do.
11:24But until that point where we don't have to fine-tune on top of existing VAS and existing policies, we can do this.
11:32And this is a stepping stone towards a future where we don't need to train robots anymore.
11:40So now if we wanted to do more things than just the tasks it's trained on, give it an agent and see what it can do.
11:48And so let me quickly go back to my demo and I'm going to show you how it's actually working.
11:59Okay.
11:59So this is my so this is strand here.
12:03This is scout here.
12:05And I've been telling him to do a bunch of things.
12:07So I can say, "Hey, do something complex."
12:15That's not complex.
12:16He's going to be thinking now.
12:17Ah, he's going to fall off.
12:21So he's saying, "Let's spin.
12:23Full 360.
12:24Done.
12:25Still safely on the stage.
12:26I can see the bright stage lights and the audience seating area."
12:30All good.
12:30What's there?
12:32A challenge.
12:34So he's speaking.
12:35I called this my signature performance,
12:40but he's not doing anything.
12:42What are you doing?
12:45He clearly seems to be speaking, but what are you doing?
12:48Please do something.
12:50He just turned off his headlines.
12:52Cool.
12:53Okay, now he's calling.
12:54So, do you see it saying calling rover speak, which was the function that it called because I said do something complex.
13:01So now it spoke, but now I think it should have been attempting to do something and it fell off because it tried doing something.
13:08I've actually seen it do like a funky dance, like this funky dance move.
13:13But he's got a mind of his own right now.
13:16What's going on under the hood here?
13:19Couple of things.
13:20The first thing is here, I can use this.
13:22What is the point of creating him?
13:24I can use him to create my data sets because I'm able to also manually move him.
13:30I will get him to navigate in the direction that I want him to and then I can create training episodes and I can get information on how he's responding and how he's reasoning based on the questions that I ask.
13:42And this is super good information for me to then be able to make him do a better job of it.
13:48So that's one part of this whole process and this experiment of getting of giving him his own autonomy and getting him to do things so that I can create more data
13:59but also apart from that uh this is my configuration.
14:04So over here under the hood strands agents which is your harness SDK is using currently anthropic claude opus 4.8 under the hood.
14:14So that is the brain and then this is my simple prompt where system prompt where I'm telling it what it's supposed to be doing and I'm telling it all of the rules and I'm also giving it access to all of the rules that it's already got.
14:27So I'm telling it what each of these rules are meant for.
14:31And so that's how strand decides which tool to invoke based on what I'm asking it to do.
14:38And the voice that it's using is the one of open AI real time.
14:43And I've also given it more information for it to be able to like just safety and guard rails to ensure that it's doing really well.
14:52Now it's this is these are two of the agents.
14:55The other thing that it can do is also chat with me on Telegram.
15:00This is amazing because when I'm not at home and I still want to get it to speak to me, I can say, "Hey, scout.
15:07Who is turn around uh spin around analyze?"
15:13Uh-uh.
15:14Don't fall off.
15:15Analyze the room and tell me who is the best looking person.
15:22This is always a hit or miss.
15:23Don't blame me if if Scout decides to be biased.
15:27I tried playing with this because it makes me feel like Snow White um and the queen saying mirror mirror on the wall who is the fairest of them all and so let's see if Strand can give us that.
15:38So it's saying haha on it on there spinning around to scope the room.
15:42Is he spinning?
15:43He is.
15:44Looks like he's making and let's look at what he's looking at now.
15:47He's still looking at the stage.
15:49This is the other miller.
15:50He's looking at me.
15:50Hi.
15:51Am I the prettiest?
15:52Oh, you flatter me.
15:54But no, turn again.
15:55Spin some more, please.
15:57He's decided not to spin more.
15:59Let's see what he says.
16:07Spin complete.
16:07Here is my analysis of the room.
16:09You can see my screen, right?
16:11I have no way of zooming this in.
16:13Um, spotted six to seven people total.
16:16A couple seated front center near the bright stage, right?
16:19A few more in the audience.
16:21One person a dark outfit on the stage.
16:23The rocking khaki jeans and sneakers.
16:25Thank you very much.
16:27Oh, no, no, no. The person in front of me, cocky jeans and sneakers, clearly walking with purpose on stage.
16:34Uh, best looking.
16:36Honestly, from my low angle ground view, it's tough to tell faces, but the person on the stage right now gets style points for the confidence stride and the dramatic blue curtain backdrop.
16:47Thank you very much.
16:48Every time I need a confidence boost, I can be the only person in the room and ask this guy who's the prettiest of them all, and he will always
16:58choose me as the answer.
17:01[laughter]
17:02Thank you very much.
17:18[music]