You Can't Tell What's Real Anymore. Here's What Comes Next - Cristóbal Valenzuela, Runway

AI Engineer · 16 min · 214 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hey guys.
00:14Very very excited to be here.
00:15Thank you for staying for the last talk.
00:18Welcome to my talk about the Matrix and how we're living through it.
00:22[laughter]
00:23Um So, you know what?
00:25I've been working on Runway for almost a decade now.
00:27Started here in New York, actually, maybe around 10 years ago.
00:31And I want to walk you through some of the learnings we've seen through, you know, building the company, building this research lab here in New York.
00:38And where do we see a lot of the work that we do in video, in pixel generation, really moving forward?
00:44And so, I don't kind of exaggerate and I joke that this is might be a simulation or not.
00:49Um and and so, for maybe for perspective, I'll walk you through a piece of history.
00:54This is like watching prehistoric ancient ruins.
00:58Uh this is our first AI film that was done in 2019.
01:03It's called the orange because as you can tell very clearly, it's an orange that falls from the tree and walks through different parts of the world.
01:11Um this is done this is a video model and image model that we had back then that was creating pixels in 128 pixels by 128 pixels that we then
01:21forcedly up sample to to this kind of resolution.
01:25I think of this as art now.
01:27This is this is like looking at a film from the 1900s, you know?
01:30It's it's beautiful.
01:31There's a lot of history behind it.
01:33But this is 2019.
01:35Uh we started the company in 2018, 2017.
01:37And so, we were trying to figure out if this had any lags.
01:41Uh we're trying to convince ourselves that if you scale this, it will naturally improve to the point where we can get somewhere close to this.
01:49This is state-of-the-art video today.
01:51This is all generated.
01:53None of these characters are real.
01:54None of those locations are real.
01:56I'm sure you You've probably here you know this um, by now.
01:59you've probably have seen this online on Twitter.
02:02Um, the models are just incredibly good.
02:04And the most interesting thing is that when I look at stuff like this, um, I still believe this will look like the orange in just a couple more years.
02:12Um, and so I want to walk you through, you know, how we're thinking not only through what we're doing at Runway today, but also in the next couple of years and the research that's shaping that.
02:22Um, we are a company that's at the intersection of both both art and science.
02:27Uh, we have uh, deep roots in artistic and creative communities and we have some of the best researchers in the world working on diffusion models and large video models.
02:36Um, and I like that combination because it allows us to, you know, bring the best of both worlds.
02:42Uh, we're we're a team of scientists and artists thinking about the problem of scaling video and world models.
02:47Um, but in a way that the core idea of the company was already discovered by cinema and by moving pictures 120, 130 years ago, which is, if you think about video,
02:58and maybe you remember this from, uh, from high school, video is an optical illusion.
03:01Video is the illusion of movement.
03:03That if you do if you stitch frames fast enough, you will create movement.
03:08You will create the illusion of movement.
03:09You create motion pictures.
03:11And so at Runway we're doing kind of something very similar.
03:13We're taking that idea really to the next level.
03:16How do we do next frame prediction fast enough, scale enough, at in in in in such a way that we can hopefully do more than just pretty pretty and nice creative videos.
03:27And so all of this all of this uh, is powered by, of course, scaling laws.
03:32I think we believe in scaling laws uh, not only apply for for language models, but also for for video.
03:38And this has been true.
03:39We're now really ramping up the amount of compute that we put behind the the models that you already have seen.
03:45And and I think the the interesting thing about, you know, seeing the model scale over time is the emerging properties that appear, the intelligence that appears when you start
03:54taking this very simple idea that cinema found itself uh, figuring out 120 years ago, um, which is next frame prediction, and taking next frame prediction really to the next level, right?
04:04So, if I take an image like this, which is not real, those two people are not real, and I just keep on predicting what the next frame will look like based on the previous frames,
04:13and if I do it fast enough, and if I do it with a network that's big enough, I'll get movement.
04:19I'll get the illusion of movement.
04:20I'll get something that looks so real to us that we we humans can distinguish.
04:25I I look at that video and it's I've been doing this for 10 years.
04:28I can't tell you if that's real or not.
04:30Um and we measure that.
04:32We've actually measured tried to quantify um if people can distinguish between real and generated videos, and I think that that Turing test we already passed.
04:39Uh people are not able to distinguish.
04:41But, the more interesting thing is here is that really what's happening as you predict next frames is you're learning a lot more than just pretty frames.
04:48You're learning about physics, you're learning about object permanence, you're learning about gravity, you're learning about occlusion, you're learning about movements, you're learning about how the objects work in the real world,
05:00and you're doing that all through just video, through images, through through next frame prediction.
05:05So, if you look at a complex thing like this, um a good video model for it to understand how to make this a consistent video will need to naturally understand
05:16uh what all these objects are, how each of them relate to one another, how a camel might probably uh stand on a ball like that, what water in a container looks like,
05:26and if I do all of that together, and I do it fast enough, I'll get, hopefully, a consistent video that looks real to us.
05:33So, a lot of the uh benchmarks that we use to evaluate our video models are not only creative, it's creative basically means do you like it or do you don't,
05:42are actually based on uh on physics, on geometries, on how objects behave in the real world, and how those objects should behave, uh and also simulator or video worlds.
05:54So, you take that idea of next frame prediction, and you do it fast enough, and you get, of course, now something that looks more like what we should expect from a video application or a game application.
06:03Again, this is just just video, right?
06:05And you take you take that idea to the next level.
06:08What happens if we start this idea of next frame prediction, but do it in the context of through the environment or something that looks similar to a game, you can start actually getting closer to a physics engine,
06:18a simulation system.
06:19But the simulation system has no physics laws that we didn't describe what the behavior of the object should be, the model just knows how to do that by pure force of of of scale.
06:33And this kind of gets gets kind of interesting because you can only think about playing worlds.
06:37This is not GTA, this is someone playing someone else in a environment and a software that doesn't exist.
06:45It's all simulation of a simulation, which which gets really meta, right?
06:50So, I'm not kidding when I said welcome to my matrix simulation.
06:54Um But the most interesting thing for me is that everything you saw before was not real-time.
06:59Meaning that it took a couple minutes or seconds for those videos to be made.
07:04Um a lot of what we're seeing right now, when we think about, you know, the next three, four years of video models and world models, is to deeply think about what happens
07:13when you start not only scaling the models and making them much much much bigger with much more emerging properties, but also what happens when you start making them interactive,
07:24right?
07:24So, take the same idea of next frame prediction, you take a frame, you predict the next frame, the next, the next, the next, but then I can act at any point in time
07:32in that frame prediction, which means I can simulate any possible divergence or action in that world.
07:38So, in this world, I could play the video, but I can also come and touch every part any aspect of that image and make it interactive to affect the consequences
07:49of of that world.
07:51And it works.
07:54And of course, this is incredibly useful for a variety of different use cases, as you can imagine.
08:00We're now seeing an entire robotics lab in Runway.
08:03So, if you come to our office in New York, half of it is what looks like a Hollywood studio.
08:08Half of it is a bunch of robots walking through and moving through the office.
08:12And so we're able to make these models or start making these models the brains of how autonomous and embodiment systems walk and reason about the world.
08:22Um but there's a couple of more interesting use cases that I think go beyond the two obvious one, which is video and robotics, which is what happens when you think about anything anything any pixel on a screen
08:34as a video model.
08:35Um here's a good example.
08:37This is a video model.
08:39What do you see right now?
08:40Well, this is an image first of all.
08:41That image could be real or not, but I could run one of our video models in real time and make this an interactive application.
08:47So, let's say I want to, you know, turn the lights of the car.
08:51Maybe I want to like turn the car around.
08:53Um I can just click through it.
08:54I can turn it around again.
08:56Maybe open the hood and see what's inside the car.
08:59Now, everything you just saw here has no physical 3D representation.
09:04There's no Unity.
09:05There's no Unreal.
09:07There's no Blender.
09:08It's just a video model.
09:10A video model has been trained to understand actions and clicks in an interface.
09:15We call this a world action model.
09:18[snorts]
09:18Um sorry, um a world interface model.
09:20And the world interface model, well, I I feel like you guys understand now.
09:23It's It's It's an interface and it's a world model.
09:25World interface model.
09:26Um allows you to very cool stuff like this.
09:28And so here's another application another um example of of how you could think about using a model like this in real time.
09:36Um I can take an image a representation, move objects around, and expect them to move as I was if you if as I was actually in that room.
09:45Again, remember there's no 3D representation of this.
09:48There This environment might not actually be have having modeled before.
09:52It's just an image and the model that understands how objects look moving in a space because of this very simple idea that next frame prediction scale well enough can become a world model or a simulation engine.
10:03Um so with the 8 minutes I have left, I want to show you this idea taken to another extreme.
10:12We're we're thinking not only about what happens if you can control the school applications and the school interfaces, but what happens if you keep extrapolating and I think by the way extrapolating may be one of the most important skills these days
10:25in in AI in in company building.
10:27What happens over the next 6 months, 12 months, 18 months may be one of the most important things to to try to predict and understand.
10:34And so we ask ourselves that question a lot at Runway.
10:37What happens?
10:38What will happen if we keep scaling the models?
10:40If we keep adding more data, if we keep making them better simulation systems.
10:45If we make them universal simulation systems that can predict anything on a screen.
10:49Literally anything on a screen.
10:49So we've been working on this idea that we call continuum.
10:53And continuum is the the idea of real-time video models or world models taken really to the next the next degree.
11:02This is an operating system.
11:04The idea here is that you might open your computer at some point in the near future and what you see streaming to your screen may be coming from a video model.
11:11So I'll show you next specific how this idea could could shape.
11:15This is still very early research, but I want to show you this idea of combining video models as part of your operating system, as part of your daily work.
11:24So here's an example.
11:26I was researching, you know, San Francisco and the fog in San Francisco.
11:31I wanted to learn more about when and how it happens.
11:34This is structural data.
11:35This is actually a website.
11:36This is something you can take from from the internet and now continuum has this idea that I can select any part of content in that website and make it a portal.
11:45We call this a portal.
11:47And the portal is a real-time video.
11:48It's fully interactive and fully based on the content you're experiencing reading through.
11:53I probably won't read through all of that text.
11:55It's too much text for me.
11:56So I just want to get a video and I can make a video that's represented by by the text you see there.
12:01These portals can be 3D illustrations.
12:03It can be any content that you're watching or interacting with.
12:06On the right you can have all the visualizations and portals you've you've experienced before, anything you've been trying to learn.
12:12Basically it's a it's a memory.
12:13So, it's the disk of the imaginations or the visual sensations you've seen before.
12:17So, here I was researching about batteries, and so I can go back to the portal I had around batteries before.
12:25And since they're real-time interactive applications, I can click them anywhere.
12:28I can move them around.
12:30I can start asking questions and the videos and and the content itself will change based on based on my interactions and the way I'm I'm managing through it.
12:42Here's another very cool example that when I the first time I saw it when the team showed it to me, it was like blew my mind.
12:47Since it's a real-time video, you can make it fully interactive with anything.
12:51Not only with clicks, but like resizing the applications would also change the behavior.
12:55Right?
12:55So, I can resize.
12:57I'm I'm learning about photosynthesis here.
12:59If I resize the application, I get a different experience.
13:02And since it's again fully interactive applications, I can click through it, navigate through parts of the cell or how photosynthesis work to get a better understanding of what I'm trying to learn and do.
13:14Again, the one thing that still feels to me kind of uncertain about this is I don't know how to call this.
13:20I don't know if this is doesn't feel like a visualization.
13:23It doesn't feel like a video.
13:24It doesn't feel like a game.
13:25It's something in between, and I think we need to do the collective work of figuring out how we call these kind of interactions in the first place.
13:31And I'm sure
13:31[snorts]
13:31you hear kind of the other thing is that you can change the structure of the prompt or what's actually creating that interaction in the first place.
13:40Here is another example.
13:44I'm going to type something.
13:49And this is one of the ones that I feel has been the most underexplored yet.
13:55This is we call this visual thinking.
13:57So, we've connected Continuum to a coding agent.
14:01And what the coding agent can do is it can use Continuum to visualize through a problem.
14:05So, imagine you ask your coding agent to create a 3D model for your bike.
14:10And instead of the model trying to just think through that via code, the the can spin a version of Continuum to think through the problem visually.
14:18So, if you ever think about someone or I'm sure you have friends or maybe some a lot of people here maybe visual thinkers.
14:24What a visual thinker is is it thinks through problems and through ideas visually first.
14:29And that helps you ground and project some of the best things you want to do.
14:34And this is somehow very similar.
14:36Agents have the ability to do visual thinking.
14:39Agents can spin up a sort of imagination that you can tap and and interact with.
14:44So, not only the agent can interact with it, I can also see it myself.
14:49And I can guide the agent in much better ways that I might have able to guide it just via text alone.
14:55Which is for us a very powerful idea.
14:57It's a completely new paradigm.
14:59And we call this this this idea uh visual visual thinking.
15:03Um The last example I have is how you think about taking, you know, Continuum to become much more than just uh using or navigating interactive structural content in in the way I've showed you before,
15:20but actually making them fully live experiences that are reactive.
15:24So, in this case I was searching for Mars.
15:26Um and as as Continuum understands the intention of what I'm trying to search from, it also suggests interfaces that are specifically designed in real time just for that.
15:37So, here it created for me this Mars outpost that's actually should be uh accurate with the data about the actual like robots that are in Mars right now uh or the ones that are projected maybe in 2030.
15:49Um and since this is fully interactive, I can interact with it.
15:51I can go to places.
15:52I can move around in the same way that I might move around, you know, in a 3D system or in a 3D Continuum.
15:58Um And it works.
16:02Just fascinating.
16:06Um So, when I say everything will be simulated, um I'm really trying to understand or help not really our team, but the world get to the point where a lot of what we'll see today will sound obvious, hopefully in a few more years or months.
16:21We'll look back at Continuum and it will might feel very close to the video of the orange that we just showed you.
16:27Continuum and a lot of the things we're doing in in video and in Runway when it comes to simulation are still in very early days, but we're getting to a point where it actually it actually works and it really scales.
16:36So, what we're really excited about is that for the next 8 to 12 months we're releasing most of what I showed here for everyone to use.
16:42Thank you, guys.
16:44[applause]