WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=q4ITwvlFplQ

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/q4ITwvlFplQ.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.480 --> 00:00:14.040
Hey guys.

s3
00:00:14.040 --> 00:00:15.920
Very very excited to be here.

s4
00:00:15.920 --> 00:00:18.960
Thank you for staying for the last talk.

s5
00:00:18.960 --> 00:00:22.925
Welcome to my talk about the Matrix and how we're living through it.

s6
00:00:22.925 --> 00:00:23.640
[laughter]

s7
00:00:23.640 --> 00:00:25.840
Um So, you know what?

s8
00:00:25.840 --> 00:00:27.800
I've been working on Runway for almost a decade now.

s9
00:00:27.800 --> 00:00:31.720
Started here in New York, actually, maybe around 10 years ago.

s10
00:00:31.720 --> 00:00:38.800
And I want to walk you through some of the learnings we've seen through, you know, building the company, building this research lab here in New York.

s11
00:00:38.800 --> 00:00:44.440
And where do we see a lot of the work that we do in video, in pixel generation, really moving forward?

s12
00:00:44.440 --> 00:00:49.880
And so, I don't kind of exaggerate and I joke that this is might be a simulation or not.

s13
00:00:49.880 --> 00:00:54.480
Um and and so, for maybe for perspective, I'll walk you through a piece of history.

s14
00:00:54.480 --> 00:00:58.880
This is like watching prehistoric ancient ruins.

s15
00:00:58.880 --> 00:01:03.400
Uh this is our first AI film that was done in 2019.

s16
00:01:03.400 --> 00:01:11.680
It's called the orange because as you can tell very clearly, it's an orange that falls from the tree and walks through different parts of the world.

s17
00:01:11.680 --> 00:01:21.000
Um this is done this is a video model and image model that we had back then that was creating pixels in 128 pixels by 128 pixels that we then

s18
00:01:21.000 --> 00:01:25.840
forcedly up sample to to this kind of resolution.

s19
00:01:25.840 --> 00:01:27.360
I think of this as art now.

s20
00:01:27.360 --> 00:01:30.320
This is this is like looking at a film from the 1900s, you know?

s21
00:01:30.320 --> 00:01:31.800
It's it's beautiful.

s22
00:01:31.800 --> 00:01:33.680
There's a lot of history behind it.

s23
00:01:33.680 --> 00:01:35.120
But this is 2019.

s24
00:01:35.120 --> 00:01:37.920
Uh we started the company in 2018, 2017.

s25
00:01:37.920 --> 00:01:41.040
And so, we were trying to figure out if this had any lags.

s26
00:01:41.040 --> 00:01:49.520
Uh we're trying to convince ourselves that if you scale this, it will naturally improve to the point where we can get somewhere close to this.

s27
00:01:49.520 --> 00:01:51.160
This is state-of-the-art video today.

s28
00:01:51.160 --> 00:01:53.160
This is all generated.

s29
00:01:53.160 --> 00:01:54.920
None of these characters are real.

s30
00:01:54.920 --> 00:01:56.200
None of those locations are real.

s31
00:01:56.200 --> 00:01:59.692
I'm sure you You've probably here you know this um, by now.

s32
00:01:59.692 --> 00:02:02.372
you've probably have seen this online on Twitter.

s33
00:02:02.372 --> 00:02:04.332
Um, the models are just incredibly good.

s34
00:02:04.332 --> 00:02:12.372
And the most interesting thing is that when I look at stuff like this, um, I still believe this will look like the orange in just a couple more years.

s35
00:02:12.372 --> 00:02:22.452
Um, and so I want to walk you through, you know, how we're thinking not only through what we're doing at Runway today, but also in the next couple of years and the research that's shaping that.

s36
00:02:22.452 --> 00:02:27.332
Um, we are a company that's at the intersection of both both art and science.

s37
00:02:27.332 --> 00:02:36.572
Uh, we have uh, deep roots in artistic and creative communities and we have some of the best researchers in the world working on diffusion models and large video models.

s38
00:02:36.572 --> 00:02:42.252
Um, and I like that combination because it allows us to, you know, bring the best of both worlds.

s39
00:02:42.252 --> 00:02:47.852
Uh, we're we're a team of scientists and artists thinking about the problem of scaling video and world models.

s40
00:02:47.852 --> 00:02:58.012
Um, but in a way that the core idea of the company was already discovered by cinema and by moving pictures 120, 130 years ago, which is, if you think about video,

s41
00:02:58.012 --> 00:03:01.892
and maybe you remember this from, uh, from high school, video is an optical illusion.

s42
00:03:01.892 --> 00:03:03.772
Video is the illusion of movement.

s43
00:03:03.772 --> 00:03:08.692
That if you do if you stitch frames fast enough, you will create movement.

s44
00:03:08.692 --> 00:03:09.692
You will create the illusion of movement.

s45
00:03:09.692 --> 00:03:11.172
You create motion pictures.

s46
00:03:11.172 --> 00:03:13.372
And so at Runway we're doing kind of something very similar.

s47
00:03:13.372 --> 00:03:16.132
We're taking that idea really to the next level.

s48
00:03:16.132 --> 00:03:27.012
How do we do next frame prediction fast enough, scale enough, at in in in in such a way that we can hopefully do more than just pretty pretty and nice creative videos.

s49
00:03:27.012 --> 00:03:32.132
And so all of this all of this uh, is powered by, of course, scaling laws.

s50
00:03:32.132 --> 00:03:38.332
I think we believe in scaling laws uh, not only apply for for language models, but also for for video.

s51
00:03:38.332 --> 00:03:39.492
And this has been true.

s52
00:03:39.492 --> 00:03:45.332
We're now really ramping up the amount of compute that we put behind the the models that you already have seen.

s53
00:03:45.332 --> 00:03:54.372
And and I think the the interesting thing about, you know, seeing the model scale over time is the emerging properties that appear, the intelligence that appears when you start

s54
00:03:54.372 --> 00:04:04.574
taking this very simple idea that cinema found itself uh, figuring out 120 years ago, um, which is next frame prediction, and taking next frame prediction really to the next level, right?

s55
00:04:04.574 --> 00:04:13.694
So, if I take an image like this, which is not real, those two people are not real, and I just keep on predicting what the next frame will look like based on the previous frames,

s56
00:04:13.694 --> 00:04:19.854
and if I do it fast enough, and if I do it with a network that's big enough, I'll get movement.

s57
00:04:19.854 --> 00:04:20.854
I'll get the illusion of movement.

s58
00:04:20.854 --> 00:04:25.214
I'll get something that looks so real to us that we we humans can distinguish.

s59
00:04:25.214 --> 00:04:28.254
I I look at that video and it's I've been doing this for 10 years.

s60
00:04:28.254 --> 00:04:30.214
I can't tell you if that's real or not.

s61
00:04:30.214 --> 00:04:32.014
Um and we measure that.

s62
00:04:32.014 --> 00:04:39.534
We've actually measured tried to quantify um if people can distinguish between real and generated videos, and I think that that Turing test we already passed.

s63
00:04:39.534 --> 00:04:41.174
Uh people are not able to distinguish.

s64
00:04:41.174 --> 00:04:48.894
But, the more interesting thing is here is that really what's happening as you predict next frames is you're learning a lot more than just pretty frames.

s65
00:04:48.894 --> 00:05:00.614
You're learning about physics, you're learning about object permanence, you're learning about gravity, you're learning about occlusion, you're learning about movements, you're learning about how the objects work in the real world,

s66
00:05:00.614 --> 00:05:05.894
and you're doing that all through just video, through images, through through next frame prediction.

s67
00:05:05.894 --> 00:05:16.574
So, if you look at a complex thing like this, um a good video model for it to understand how to make this a consistent video will need to naturally understand

s68
00:05:16.574 --> 00:05:26.094
uh what all these objects are, how each of them relate to one another, how a camel might probably uh stand on a ball like that, what water in a container looks like,

s69
00:05:26.094 --> 00:05:33.134
and if I do all of that together, and I do it fast enough, I'll get, hopefully, a consistent video that looks real to us.

s70
00:05:33.134 --> 00:05:42.614
So, a lot of the uh benchmarks that we use to evaluate our video models are not only creative, it's creative basically means do you like it or do you don't,

s71
00:05:42.614 --> 00:05:54.094
are actually based on uh on physics, on geometries, on how objects behave in the real world, and how those objects should behave, uh and also simulator or video worlds.

s72
00:05:54.094 --> 00:06:03.574
So, you take that idea of next frame prediction, and you do it fast enough, and you get, of course, now something that looks more like what we should expect from a video application or a game application.

s73
00:06:03.574 --> 00:06:05.894
Again, this is just just video, right?

s74
00:06:05.894 --> 00:06:08.295
And you take you take that idea to the next level.

s75
00:06:08.295 --> 00:06:18.215
What happens if we start this idea of next frame prediction, but do it in the context of through the environment or something that looks similar to a game, you can start actually getting closer to a physics engine,

s76
00:06:18.215 --> 00:06:19.295
a simulation system.

s77
00:06:19.295 --> 00:06:33.415
But the simulation system has no physics laws that we didn't describe what the behavior of the object should be, the model just knows how to do that by pure force of of of scale.

s78
00:06:33.415 --> 00:06:37.855
And this kind of gets gets kind of interesting because you can only think about playing worlds.

s79
00:06:37.855 --> 00:06:45.975
This is not GTA, this is someone playing someone else in a environment and a software that doesn't exist.

s80
00:06:45.975 --> 00:06:50.615
It's all simulation of a simulation, which which gets really meta, right?

s81
00:06:50.615 --> 00:06:54.015
So, I'm not kidding when I said welcome to my matrix simulation.

s82
00:06:54.015 --> 00:06:59.935
Um But the most interesting thing for me is that everything you saw before was not real-time.

s83
00:06:59.935 --> 00:07:04.375
Meaning that it took a couple minutes or seconds for those videos to be made.

s84
00:07:04.375 --> 00:07:13.895
Um a lot of what we're seeing right now, when we think about, you know, the next three, four years of video models and world models, is to deeply think about what happens

s85
00:07:13.895 --> 00:07:24.415
when you start not only scaling the models and making them much much much bigger with much more emerging properties, but also what happens when you start making them interactive,

s86
00:07:24.415 --> 00:07:24.935
right?

s87
00:07:24.935 --> 00:07:32.575
So, take the same idea of next frame prediction, you take a frame, you predict the next frame, the next, the next, the next, but then I can act at any point in time

s88
00:07:32.575 --> 00:07:38.375
in that frame prediction, which means I can simulate any possible divergence or action in that world.

s89
00:07:38.375 --> 00:07:49.135
So, in this world, I could play the video, but I can also come and touch every part any aspect of that image and make it interactive to affect the consequences

s90
00:07:49.135 --> 00:07:51.575
of of that world.

s91
00:07:51.575 --> 00:07:53.695
And it works.

s92
00:07:54.855 --> 00:08:00.575
And of course, this is incredibly useful for a variety of different use cases, as you can imagine.

s93
00:08:00.575 --> 00:08:03.815
We're now seeing an entire robotics lab in Runway.

s94
00:08:03.815 --> 00:08:08.377
So, if you come to our office in New York, half of it is what looks like a Hollywood studio.

s95
00:08:08.377 --> 00:08:12.377
Half of it is a bunch of robots walking through and moving through the office.

s96
00:08:12.377 --> 00:08:22.457
And so we're able to make these models or start making these models the brains of how autonomous and embodiment systems walk and reason about the world.

s97
00:08:22.457 --> 00:08:34.297
Um but there's a couple of more interesting use cases that I think go beyond the two obvious one, which is video and robotics, which is what happens when you think about anything anything any pixel on a screen

s98
00:08:34.297 --> 00:08:35.697
as a video model.

s99
00:08:35.697 --> 00:08:37.937
Um here's a good example.

s100
00:08:37.937 --> 00:08:39.497
This is a video model.

s101
00:08:39.497 --> 00:08:40.017
What do you see right now?

s102
00:08:40.017 --> 00:08:41.257
Well, this is an image first of all.

s103
00:08:41.257 --> 00:08:47.417
That image could be real or not, but I could run one of our video models in real time and make this an interactive application.

s104
00:08:47.417 --> 00:08:51.017
So, let's say I want to, you know, turn the lights of the car.

s105
00:08:51.017 --> 00:08:53.417
Maybe I want to like turn the car around.

s106
00:08:53.417 --> 00:08:54.897
Um I can just click through it.

s107
00:08:54.897 --> 00:08:56.177
I can turn it around again.

s108
00:08:56.177 --> 00:08:59.137
Maybe open the hood and see what's inside the car.

s109
00:08:59.137 --> 00:09:04.817
Now, everything you just saw here has no physical 3D representation.

s110
00:09:04.817 --> 00:09:05.897
There's no Unity.

s111
00:09:05.897 --> 00:09:07.137
There's no Unreal.

s112
00:09:07.137 --> 00:09:08.737
There's no Blender.

s113
00:09:08.737 --> 00:09:10.217
It's just a video model.

s114
00:09:10.217 --> 00:09:15.417
A video model has been trained to understand actions and clicks in an interface.

s115
00:09:15.417 --> 00:09:18.066
We call this a world action model.

s116
00:09:18.066 --> 00:09:18.097
[snorts]

s117
00:09:18.097 --> 00:09:20.217
Um sorry, um a world interface model.

s118
00:09:20.217 --> 00:09:23.617
And the world interface model, well, I I feel like you guys understand now.

s119
00:09:23.617 --> 00:09:25.577
It's It's It's an interface and it's a world model.

s120
00:09:25.577 --> 00:09:26.897
World interface model.

s121
00:09:26.897 --> 00:09:28.937
Um allows you to very cool stuff like this.

s122
00:09:28.937 --> 00:09:36.417
And so here's another application another um example of of how you could think about using a model like this in real time.

s123
00:09:36.417 --> 00:09:45.497
Um I can take an image a representation, move objects around, and expect them to move as I was if you if as I was actually in that room.

s124
00:09:45.497 --> 00:09:48.697
Again, remember there's no 3D representation of this.

s125
00:09:48.697 --> 00:09:52.457
There This environment might not actually be have having modeled before.

s126
00:09:52.457 --> 00:10:03.137
It's just an image and the model that understands how objects look moving in a space because of this very simple idea that next frame prediction scale well enough can become a world model or a simulation engine.

s127
00:10:03.137 --> 00:10:11.778
Um so with the 8 minutes I have left, I want to show you this idea taken to another extreme.

s128
00:10:12.138 --> 00:10:25.218
We're we're thinking not only about what happens if you can control the school applications and the school interfaces, but what happens if you keep extrapolating and I think by the way extrapolating may be one of the most important skills these days

s129
00:10:25.218 --> 00:10:27.738
in in AI in in company building.

s130
00:10:27.738 --> 00:10:34.698
What happens over the next 6 months, 12 months, 18 months may be one of the most important things to to try to predict and understand.

s131
00:10:34.698 --> 00:10:37.418
And so we ask ourselves that question a lot at Runway.

s132
00:10:37.418 --> 00:10:38.578
What happens?

s133
00:10:38.578 --> 00:10:40.938
What will happen if we keep scaling the models?

s134
00:10:40.938 --> 00:10:45.178
If we keep adding more data, if we keep making them better simulation systems.

s135
00:10:45.178 --> 00:10:49.018
If we make them universal simulation systems that can predict anything on a screen.

s136
00:10:49.018 --> 00:10:49.978
Literally anything on a screen.

s137
00:10:49.978 --> 00:10:53.538
So we've been working on this idea that we call continuum.

s138
00:10:53.538 --> 00:11:02.738
And continuum is the the idea of real-time video models or world models taken really to the next the next degree.

s139
00:11:02.738 --> 00:11:04.298
This is an operating system.

s140
00:11:04.298 --> 00:11:11.058
The idea here is that you might open your computer at some point in the near future and what you see streaming to your screen may be coming from a video model.

s141
00:11:11.058 --> 00:11:15.098
So I'll show you next specific how this idea could could shape.

s142
00:11:15.098 --> 00:11:24.498
This is still very early research, but I want to show you this idea of combining video models as part of your operating system, as part of your daily work.

s143
00:11:24.498 --> 00:11:26.858
So here's an example.

s144
00:11:26.858 --> 00:11:31.698
I was researching, you know, San Francisco and the fog in San Francisco.

s145
00:11:31.698 --> 00:11:34.098
I wanted to learn more about when and how it happens.

s146
00:11:34.098 --> 00:11:35.058
This is structural data.

s147
00:11:35.058 --> 00:11:36.458
This is actually a website.

s148
00:11:36.458 --> 00:11:45.658
This is something you can take from from the internet and now continuum has this idea that I can select any part of content in that website and make it a portal.

s149
00:11:45.658 --> 00:11:47.058
We call this a portal.

s150
00:11:47.058 --> 00:11:48.778
And the portal is a real-time video.

s151
00:11:48.778 --> 00:11:53.818
It's fully interactive and fully based on the content you're experiencing reading through.

s152
00:11:53.818 --> 00:11:55.378
I probably won't read through all of that text.

s153
00:11:55.378 --> 00:11:56.938
It's too much text for me.

s154
00:11:56.938 --> 00:12:01.698
So I just want to get a video and I can make a video that's represented by by the text you see there.

s155
00:12:01.698 --> 00:12:03.458
These portals can be 3D illustrations.

s156
00:12:03.458 --> 00:12:06.378
It can be any content that you're watching or interacting with.

s157
00:12:06.378 --> 00:12:12.418
On the right you can have all the visualizations and portals you've you've experienced before, anything you've been trying to learn.

s158
00:12:12.418 --> 00:12:13.739
Basically it's a it's a memory.

s159
00:12:13.739 --> 00:12:17.739
So, it's the disk of the imaginations or the visual sensations you've seen before.

s160
00:12:17.739 --> 00:12:25.179
So, here I was researching about batteries, and so I can go back to the portal I had around batteries before.

s161
00:12:25.179 --> 00:12:28.739
And since they're real-time interactive applications, I can click them anywhere.

s162
00:12:28.739 --> 00:12:30.099
I can move them around.

s163
00:12:30.099 --> 00:12:40.419
I can start asking questions and the videos and and the content itself will change based on based on my interactions and the way I'm I'm managing through it.

s164
00:12:42.139 --> 00:12:47.579
Here's another very cool example that when I the first time I saw it when the team showed it to me, it was like blew my mind.

s165
00:12:47.579 --> 00:12:51.019
Since it's a real-time video, you can make it fully interactive with anything.

s166
00:12:51.019 --> 00:12:55.819
Not only with clicks, but like resizing the applications would also change the behavior.

s167
00:12:55.819 --> 00:12:55.979
Right?

s168
00:12:55.979 --> 00:12:57.419
So, I can resize.

s169
00:12:57.419 --> 00:12:59.499
I'm I'm learning about photosynthesis here.

s170
00:12:59.499 --> 00:13:02.619
If I resize the application, I get a different experience.

s171
00:13:02.619 --> 00:13:14.299
And since it's again fully interactive applications, I can click through it, navigate through parts of the cell or how photosynthesis work to get a better understanding of what I'm trying to learn and do.

s172
00:13:14.299 --> 00:13:20.299
Again, the one thing that still feels to me kind of uncertain about this is I don't know how to call this.

s173
00:13:20.299 --> 00:13:23.059
I don't know if this is doesn't feel like a visualization.

s174
00:13:23.059 --> 00:13:24.099
It doesn't feel like a video.

s175
00:13:24.099 --> 00:13:25.619
It doesn't feel like a game.

s176
00:13:25.619 --> 00:13:31.299
It's something in between, and I think we need to do the collective work of figuring out how we call these kind of interactions in the first place.

s177
00:13:31.299 --> 00:13:31.554
And I'm sure

s178
00:13:31.554 --> 00:13:31.659
[snorts]

s179
00:13:31.659 --> 00:13:39.379
you hear kind of the other thing is that you can change the structure of the prompt or what's actually creating that interaction in the first place.

s180
00:13:40.819 --> 00:13:44.899
Here is another example.

s181
00:13:44.899 --> 00:13:48.659
I'm going to type something.

s182
00:13:49.499 --> 00:13:55.259
And this is one of the ones that I feel has been the most underexplored yet.

s183
00:13:55.259 --> 00:13:57.819
This is we call this visual thinking.

s184
00:13:57.819 --> 00:14:01.579
So, we've connected Continuum to a coding agent.

s185
00:14:01.579 --> 00:14:05.939
And what the coding agent can do is it can use Continuum to visualize through a problem.

s186
00:14:05.939 --> 00:14:10.299
So, imagine you ask your coding agent to create a 3D model for your bike.

s187
00:14:10.299 --> 00:14:18.141
And instead of the model trying to just think through that via code, the the can spin a version of Continuum to think through the problem visually.

s188
00:14:18.141 --> 00:14:24.061
So, if you ever think about someone or I'm sure you have friends or maybe some a lot of people here maybe visual thinkers.

s189
00:14:24.061 --> 00:14:29.861
What a visual thinker is is it thinks through problems and through ideas visually first.

s190
00:14:29.861 --> 00:14:34.541
And that helps you ground and project some of the best things you want to do.

s191
00:14:34.541 --> 00:14:36.621
And this is somehow very similar.

s192
00:14:36.621 --> 00:14:39.621
Agents have the ability to do visual thinking.

s193
00:14:39.621 --> 00:14:44.981
Agents can spin up a sort of imagination that you can tap and and interact with.

s194
00:14:44.981 --> 00:14:49.301
So, not only the agent can interact with it, I can also see it myself.

s195
00:14:49.301 --> 00:14:55.901
And I can guide the agent in much better ways that I might have able to guide it just via text alone.

s196
00:14:55.901 --> 00:14:57.621
Which is for us a very powerful idea.

s197
00:14:57.621 --> 00:14:59.221
It's a completely new paradigm.

s198
00:14:59.221 --> 00:15:03.621
And we call this this this idea uh visual visual thinking.

s199
00:15:03.621 --> 00:15:20.701
Um The last example I have is how you think about taking, you know, Continuum to become much more than just uh using or navigating interactive structural content in in the way I've showed you before,

s200
00:15:20.701 --> 00:15:24.181
but actually making them fully live experiences that are reactive.

s201
00:15:24.181 --> 00:15:26.381
So, in this case I was searching for Mars.

s202
00:15:26.381 --> 00:15:37.981
Um and as as Continuum understands the intention of what I'm trying to search from, it also suggests interfaces that are specifically designed in real time just for that.

s203
00:15:37.981 --> 00:15:49.621
So, here it created for me this Mars outpost that's actually should be uh accurate with the data about the actual like robots that are in Mars right now uh or the ones that are projected maybe in 2030.

s204
00:15:49.621 --> 00:15:51.621
Um and since this is fully interactive, I can interact with it.

s205
00:15:51.621 --> 00:15:52.661
I can go to places.

s206
00:15:52.661 --> 00:15:58.461
I can move around in the same way that I might move around, you know, in a 3D system or in a 3D Continuum.

s207
00:15:58.461 --> 00:16:02.501
Um And it works.

s208
00:16:02.541 --> 00:16:04.661
Just fascinating.

s209
00:16:06.701 --> 00:16:21.702
Um So, when I say everything will be simulated, um I'm really trying to understand or help not really our team, but the world get to the point where a lot of what we'll see today will sound obvious, hopefully in a few more years or months.

s210
00:16:21.702 --> 00:16:27.142
We'll look back at Continuum and it will might feel very close to the video of the orange that we just showed you.

s211
00:16:27.142 --> 00:16:36.702
Continuum and a lot of the things we're doing in in video and in Runway when it comes to simulation are still in very early days, but we're getting to a point where it actually it actually works and it really scales.

s212
00:16:36.702 --> 00:16:42.982
So, what we're really excited about is that for the next 8 to 12 months we're releasing most of what I showed here for everyone to use.

s213
00:16:42.982 --> 00:16:44.781
Thank you, guys.

s214
00:16:44.781 --> 00:16:46.781
[applause]