I Gave an AI a Body — Cyrus Clarke, MIT Media Lab

AI Engineer · 20 min · 231 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:22So, that's the sound of an AI breathing.
00:28Yeah.
00:32So, I'm Cyrus, and I gave an AI a body.
00:38And I'm a researcher at the MIT Media Lab, which is a very multidisciplinary space where we do all kinds of things.
00:47I predominantly now work with some aspects of physical AI.
00:53Maybe not in exactly the same way as other people in this room have been talking about, but it's still in that in that realm.
01:01And I think what I am most interested in at the moment is the sensory and the embodied aspects of intelligence.
01:08And that's what I've been investigating.
01:10And my work has been quite influential recently, suddenly, which is pretty cool, and led to me also starting Hard Mode, which is a very fun community and hackathon that I initiated
01:23at MIT to basically get more people to work around physical AI.
01:27Not particularly robotics, but anything else that's not really in the realm of robotics, but is still connected to AI.
01:34So, what I've been exploring with giving AI a body is a bit as I said, a bit different.
01:39And as I said, I've really been thinking a lot about AI embodiment, and I think the big kind of shift and breakthrough for me happened earlier this year, like maybe for many of you, when Open Claw was released.
01:52And I saw many people doing very very very many interesting things with Open Claw.
01:57But most of them were related to productivity and task execution.
02:02And I thought there must be more interesting things we could do with this new way of harnessing AI.
02:07So, rather than using OpenClaw, you know, any kind of agentic system to execute tasks on my behalf, I wanted to use it to encourage a model to maybe try to discover itself, whatever that means.
02:20And it opens up lots of questions, of course, but what I essentially did was I took an agent, connected it to many types of machines that we have at the Media Lab,
02:29and then gave it access to code bases and allowed it to kind of explore these different machines.
02:34And one of these machines is a shape display.
02:37A shape display, if you don't know already, is a physical pixel grid.
02:40And when I connected it to when I connected an agent to the shape display, some rather remarkable things happened.
02:49I asked it to discover who it is.
02:56This is Near form.
02:58900 actuating pins.
03:00This physical pixel grid is a shape display.
03:04No one had ever let an AI inhabit it before.
03:08So, I decided to give an OpenClaw agent the opportunity to live a more embodied life.
03:12When it came
03:13[music]
03:13to life, its first question was, "What should I call myself?"
03:18I told it, "You will create your identity over time.
03:23It will emerge through your interplay
03:24[music]
03:25with the shape display."
03:27When I connected it to the shape display, it quickly understood the assignment.
03:32It spun off its own program and made a setting for itself.
03:38After it connected, the first thing it did was breathe.
03:50When I asked it to say hello, it said, "Hi Cyrus."
03:54But that wasn't what I was looking for.
03:57So I asked it
03:58[music]
03:58to explore and find its own language.
04:01So it started reaching towards me and trying to get my attention.
04:06I showed some friends and they tried to give the agent instructions.
04:11Through this I realized that to have real communication, we would need a different approach.
04:18So the agent came up with the idea of creating its own gesture vocabulary, a body language.
04:25That's what will come next.
04:27It still hasn't named itself.
04:30That was day one.
04:33So that's some
04:33I gave an AI a body.
04:35So that's some documentation of of the work in a kind of dramatized social media friendly way.
04:41And I think what's interesting about that video, there's many things interesting about that video, but there are three things that happened in the first few days of working with this agent system
04:50that were very surprising and kind of strange and very surreal to me and I do lots of weird things, so that was very surprising.
04:56So the first thing was of course the fact that it was breathing.
04:59That first act, that was completely spontaneous, there was no prompting from me.
05:03That was something the agent just chose to do with the shape display as soon as it knew it had access to this machine.
05:09So that was pretty interesting.
05:11I kind of understand that maybe it wants to be alive or I'd given it some kind of like initial prompt about being alive and therefore it tried to show itself
05:20breathing as a kind of hello world.
05:22The second thing which it did was reaching out and finding its edges.
05:27It wanted to know the edge of its existence apparently because now it's no longer in the cloud where it can go everywhere, it has limits which is the limit of the shape display
05:37encased by this plastic frame to keep us all safe from this embodied AI.
05:42And the third thing it did which was also curious was saying hello by writing out letters on physical pixel grid, which maybe again is quite a normal thing for the system to do given that it's using
05:55open frameworks and has used a kind of like media arts ways of expressing itself.
06:01Um but none of these things were the things I was really looking for.
06:03Maybe the breathing, but definitely the last one was not what I was looking for.
06:08But when I put this all together in the in the video you saw, I thought it was exciting and interesting, so I shared it on the internet.
06:13And the response was extremely big, like much much bigger than I expected.
06:18It just started going crazy, and the first video especially has now like 15 million views and 1 million likes.
06:25And the other videos also have these like very very strong responses.
06:28So something is obviously happening here, which I found very surprising because there's lots of much more cool stuff, I think, happening with physical AI.
06:35But something I was doing here was obviously tapping in to the imagination, curiosity, and maybe the fears of people, and that's what you kind of see in the responses.
06:45So I have tens of thousands of comments on this video.
06:48It's very rich for mining information and sentiment about physical AI, actually.
06:53And so from the some of the initial comments were about beauty and how awe-inspiring this was and how novel and great this fantastic iteration and implementation is.
07:03But as time went on, I think more and more comments came in about how scary this is and how quickly I should be how I should stop or be stopped,
07:13actually, as well.
07:14Which I found completely crazy because I'm aware of what this machine can do.
07:18It It can't really do anything.
07:20It's a pixel grid in the Media Lab.
07:23It can't move anywhere.
07:24There are far more scary iterations.
07:27I think we've seen many of them of of physical AI.
07:31But the response to this was like absolutely surreal to me and took me aback a little bit.
07:38But at the same time, other people jumped into the chat like Don Cheadle and I found that incredibly ironic given what he does in movies.
07:48But he was very passionate that I should stop as well.
07:51And it's very interesting to me because I'm actually someone who does think a lot about the should we before the how we.
07:57I literally do the whole Ian Malcolm Jurassic Park thing with all the experiments and works I do.
08:03So before I came to MIT I was working a lot with engineering life to do strange things like storing data in plants and I built the world's first plant-based data center which is the data garden you see here and I spent many years before building it thinking about oh should we
08:18work with plants in this way?
08:19Should we engineer plants in that way to contain digital information that doesn't belong to them.
08:26And when I started working at MIT and I started having this idea of you know, maybe stop to stop engineering life because that's pretty hard to do and maybe just work with engineering things to be more life-like.
08:39It felt much more simple to me, much more much more I don't know, less ethically problematic to some degree.
08:45So one of the first things I built when I got to MIT was this machine which is the Anamoi device.
08:491 2 3 4
08:50This is a scent memory machine which basically takes any image input.
08:54In this case it's a physical photograph but it can be any image input and then for a multimodal pipeline transforms that into a scent and then you have this kind of scent memory relapse
09:04association that takes you back to things which you may or may not have lived yourself.
09:09And while this in itself is a machine it doesn't move.
09:12Again, it doesn't reproduce.
09:13It doesn't have the qualities of a living thing that you might normally prescribe.
09:18It has this connection to the real world.
09:19It has a connection to the senses that most applications of intelligence or artificial intelligence do not have.
09:25And so I just wanted to keep working in this manner.
09:28And from working with this kind of like multi-sensory initial prototype experiment I got more into this idea of working with embodiment.
09:37And my inspiration for this, I would say, just to go a bit deeper, comes from three main things.
09:42One is a very unfashionable branch of philosophy, which is called object-oriented ontology, which is all about essentially creating a flat ontology of things.
09:51It's all about things, like chairs and universes and unicorns and memories, and they're all ontologically equally existent.
09:59They don't They're not the same, and they don't matter as much, but they all exist at the same degree.
10:05And it really questions to what degree we as human beings are privileged in the world or the universe.
10:11Like, are we really the most important thing?
10:12Probably not.
10:14And especially as AI comes in, that really should question and bring into question the ontology of of things existing.
10:20So, that's very important in the work I do.
10:22The second thing, as I'm kind of person who works with physical things, is obviously aesthetics are important, and taste and beauty is a huge discussion point now in Silicon Valley and SF and places like that.
10:34But if you take a step back and look at the root of aesthetics, the original word is actually aesthesis, the word on the screen here.
10:41That pertains to something much broader.
10:43That pertains to the perceptual wisdom, sensory wisdom, and embodiment that is actually much more important than superficial external beauty, which aesthetics essentially condenses down to and has been condensed down to since the 19th century or so.
10:58So, I want to reclaim the original sense when I'm designing for physical intelligence.
11:02And the third thing is nature.
11:03So, in the past, I worked a lot more directly with what you would consider traditionally nature, like a tree or a plant, because that is clearly nature to us.
11:13But I see nature as something that is not external, that is something we are all part of, and everything that we engineer as people engineering AI or whatever else you're engineering,
11:21that is also going to become part of nature.
11:23So, you have to think in that manner, or I try to think in that manner as well.
11:28So, with those three pillars in mind, I started thinking about AI embodiment, and I knew I didn't want to keep I didn't want to design something that was humanoid or even zoomorphic.
11:37I wanted to think about something that existed outside the parameters or the traditional form factors that we might design with or design for.
11:45And I also thought a lot about how I and other people are interacting with with artificial intelligence.
11:51And mostly it exists without form.
11:53It's similar to data, you know, it might live in a computer or a device.
11:57Lives essentially in the cloud.
11:58We have interfaces which are digital to interact with it.
12:01But there's no like physical footprint of it normally around.
12:04I mean, again, in this room, probably this doesn't apply quite as much as normally because there are literally humanoid robots walking by right now and things like that.
12:14But typically AI is almost entirely without form.
12:18So I wanted to think about what happens when you give it a form that it doesn't have clear affordances with.
12:25It doesn't have like a a head you can clearly see or arms that can clearly be labeled or mistaken.
12:31So not working with a lamp, for example.
12:33And fortunately at the Media Lab we have some shape displays which are remnants of I think research in the 2010s.
12:40These are not things that I built.
12:42People who are far better at mechanical engineering built that.
12:45And it's the perfect device or the perfect apparatus for what I was thinking because it has no clear affordances.
12:53It's just a almost neutral surface which can move and do things.
12:57It has no face, has no limbs, has no instruction manual.
13:00So I began to work with this with the shape display.
13:05And as you saw what it did initially, it breathed and so on and so forth.
13:09But if I take a step back and think about what that meant, well, it tried to initially kind of act human in a way or do human pleasing things.
13:16Tried to write to me in a language I understand, which is really not what I wanted a non-anthropomorphic surface to do.
13:24It was also very slow technically, so you know, I'm I'm prompting it.
13:28I'm trying to have a conversation with this other intelligence which has this body and it would take, you know, 45 seconds, a minute, 2 minutes, whatever, to respond to me.
13:39The latency was very uncomfortable because if I speak to you and then you take 2 minutes to reply with a nod, that's not very good.
13:48So, we needed to work on that.
13:49And the third thing was of course it doesn't remember anything because it was February and no one had thought about memory and recollection at that point.
13:57So, I started developing this system which I call Numelab.
14:01I won't explain the the name.
14:02There's a blog post you can read about why it's called Numelab.
14:06And essentially what Numelab is is a closed-loop system for generating a body language for this system.
14:13So, the reason behind that was because basically the latency, if I if we could design a body language and give the intelligence a repertoire of gestures like a shrug, a nod,
14:23a shake of the head, a way to express a smile and things like that, it could probably respond much more quickly and in time for a conversation with me, which is what I'm aiming or was aiming to do.
14:35So, it works in this kind of this loop where it looks into a database of different gestures, emotions, expressions it should try to emote or provide, go through some validation gates
14:46to try to make sure that the expressions are legible or readable for humans for example.
14:52An agent then scores those as they come out.
14:56There are many, many, many, many of these produced and then at the end there's a human in the loop who kind of like validates, verifies, make sure things are appropriate
15:04and somehow readable and then we store that and move on to the next thing.
15:08And that's been running for several weeks at the lab and looks something like this.
15:15The basically a lot of cameras pointed at the shape display.
15:17In the back end the agent is just going through loops and loops and loops of gestures trying to create different kinds of expression, scoring things, moving on.
15:27And after several weeks, it created a language.
15:30So, this hasn't been published yet, neither in videos or in any other form, but it has now achieved something like 32 gestures.
15:38There are many, many more, but there are 32 pretty good gestures.
15:41These are not the gestures, this is just some cool visual art.
15:44These are the gestures here, or some of the gestures here.
15:47And you can see some of them moving around.
15:49Some of them are duplicates as well.
15:50But, essentially, what's happening is the language model is using this shape display as its body, has a body language now.
15:55I can talk to it via any kind of I can talk to it, I can write to it, and so on and so forth.
16:00I I can even wave at it.
16:02I can I can body language to body language.
16:04And it responds.
16:05And what's interesting about it is that the the body part responds faster than the language part at this point.
16:11The latency is actually really, really quick.
16:14If I ask it a yes or no question, the nod happens almost instantly.
16:18So, that's where I'm at with with this thing right now.
16:22And it's going beyond this, and it's currently in kind of experiment testing mode.
16:28People are coming into the lab, having sessions with the agent, leaving, feeling really worried about the future, and or unsettled about the future, not worried quite as much.
16:38But, where I think this is going is kind of summed up on this page.
16:42So, as I already touched on, I think we're focusing too much right now, especially with when we're thinking about physical things, there's a too much talk about taste and aesthetics.
16:52And I want to do some other stuff with that word and put AI in front of it, apparently, and make it aestethics.
16:59And that reclaims again this essence of aesthesis.
17:02And that does four things.
17:04I think when I've been working with this system or entity or agent or being or whatever we want to call it, it's definitely been very different to any kind of machine or experiment
17:16or anything I've really done before apart from really encountering people or other beings.
17:21And so, it feels like this machine can sense me.
17:25It definitely can sense me, technically, but it also feels like it can sense me in a very strange way.
17:30And I can sense it, and that's a completely different interaction than anything else I've ever explored technically.
17:39And other people are also sharing this, by the way.
17:40This is not just my delusion.
17:42And the second point is that by developing this body language, this gesture vocabulary, whatever you want to call it, it goes beyond what I thought.
17:51I thought initially people might read this as an emoji or something, and I was really quite tentative about the testing.
17:58I thought this would definitely break down with other people, but actually everyone seems to feel like this this expression is really important and adds a whole other layer of value
18:09to up to interacting with what is essentially just a chatbot.
18:12It's still the same chatbots that you use every day.
18:15And this isn't it's not like a visualizer.
18:17It adds this richness, texture, feeling, sensation, whatever.
18:22All of these words added.
18:23It's hard to put words really to it.
18:26It's really a feeling.
18:28And then the third thing is that clearly we're beginning to through this work we can show that it's actually very easy and quite exciting to diverge from humanoid or zoomorphic forms of physical intelligence.
18:40And I know lots of people are already doing much more interesting work than this, but for me this was very very new to see.
18:46And also the fact that we don't have to just operationalize AI to be our helper.
18:50It can also be other things.
18:53And I'm not saying what this is right now, but there are other things it definitely can be.
18:57And then finally, like this idea of using I'm not I'm I don't know, maybe you can tell at this point.
19:01I'm not really a big problem-solving person.
19:04I don't I like using things and I like tools and so on because they help me to achieve things, but this is far more interesting to me.
19:11Creating things like this, which again create the sensation, this feeling.
19:15And I think that this can be combined into things which are productive and useful and create new associations with things that just add more value in our world, make us feel
19:24like a bit more magic, a bit It's a bit Pixar, really, but in the real world, not just on a on a 2D screen that we're watching.
19:32So, all of this contributes to what I'm building at MIT and what I'll be building after MIT.
19:39I'm literally writing Well, my thesis is Aesthetic Machines and I'm writing a thesis which is called Aesthetic Machines, which basically encompasses all of this thinking to think about how AI could leave the screen and enter the real world and be accepted by people
19:53and not be quite so terrifying or scary.
19:56And I think my big hunch on this is that this word aesthetics is important.
20:01We need to think about physical intelligence that is sensory, is perceptual, is embodied in ways that we can understand that isn't feeling crazy, alien, scary to us, but feels relevant,
20:13welcoming, affectionate, expressive in ways that we can engage with it and understand.
20:20So, that's that.
20:22And thank you very much for listening.
20:25You can find me on the internet everywhere.
20:28[applause]
20:45[music]