The Next Medium: Why Real-Time Interactive Video Changes Everything — Ahmed Ahres, Reactor

AI Engineer · 17 min · 223 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:12Hi everyone.
  3. 00:13Uh welcome to the talk.
  4. 00:14First of all, thank you all for making the time.
  5. 00:16I know it's the last talk of the day probably or I think the last one is at uh 3:45, but uh yeah, thank you all for your time.
  6. 00:22I I know you're all probably very busy.
  7. 00:24[snorts]
  8. 00:24Um, today I'm going to be talking about something that is a little bit slightly futuristic, though not for San Francisco, and that's world models.
  9. 00:31Um, I know here it's written real-time interactive video, but the way we think about world models is really in real time interactive video.
  10. 00:38And I'll explain why.
  11. 00:39Um, and in today's world, I think world models is a little bit of a marketing term that people think about it from a gosh and splatting standpoint, others from video.
  12. 00:47Uh, but the way we define world models is really real- time interactive video.
  13. 00:51And I have strong evidence or like beliefs that this will actually change everything in how we produce and consume content.
  14. 00:59Oops.
  15. 01:01What happened?
  16. 01:03Sorry.
  17. 01:06Sorry about that.
  18. 01:09Cool.
  19. 01:10[clears throat] So if you think about video, video has always been something passive.
  20. 01:14Before people used to produce videos and movies and then we would watch it.
  21. 01:19And in today's world we have these models.
  22. 01:21the V3, the C dense 2 and what they do is you prompt them, you get back a file, you watch and good luck.
  23. 01:28It's a slot machine.
  24. 01:28You cannot change it, you cannot do anything about it.
  25. 01:31So there's a question that um we like to think about in our company is what happens when video becomes programmable like software and what happens when pixels can be generated in real time.
  26. 01:43Once this happens, it actually changes completely how we think about consuming content and even producing content.
  27. 01:49and I'll be talking about how in history we've seen that real time has always been the future and the kind of applications that it unfolded and how you can get started today.
  28. 02:00Quick quick background here.
  29. 02:02Um so my name is Ahmed.
  30. 02:03I'm the head of go to market at reactor.
  31. 02:05My background is in computer vision and machine learning.
  32. 02:07Um and I state machine learning because this is the time where we used to actually train our models.
  33. 02:12Um I built and shipped games on iOS uh on iOS and Android for fun.
  34. 02:16Um I was a founder.
  35. 02:18Um and today I'm the head of go to market at reactor and who we are quickly is we're a series A company building the platform for these real-time world models.
  36. 02:28So so far the main most of them have been still models but we are building the infrastructure and the developer platform to make them usable so that anyone can integrate these real-time interactive video and we can democratize access to this technology.
  37. 02:43I'll start with a problem.
  38. 02:44We can today generate pretty much anything, but we just can't change it, right?
  39. 02:48A generator video, as I mentioned earlier, is still a recording.
  40. 02:52You get it back, you can't do anything about it.
  41. 02:55And real time changes what the medium is.
  42. 02:57It doesn't just make it faster.
  43. 02:59And I'm going to talk about two examples that actually show us what real time has unlocked in the past.
  44. 03:04Before, in the 1950s and even before, we used to look at a map to know where we are.
  45. 03:10Someone produces a map, you look at where you are, and that's it.
  46. 03:12You cannot do anything about it.
  47. 03:14Then GPS came.
  48. 03:15GPS made it real time.
  49. 03:17Suddenly I can know where where I am at the instant and I can you know track it.
  50. 03:21Now you think GPS has just made it a bit faster to know where I am.
  51. 03:24But actually Uber would not exist if we did not have the GPS.
  52. 03:30Another example which is even a bit more powerful.
  53. 03:32I'd say before we used to to use the film to produce content, right?
  54. 03:37We had a film.
  55. 03:38Someone shoots something they can't see what they're shooting.
  56. 03:41They go somewhere, they produce that film, and then you can see it.
  57. 03:45And then it became digital.
  58. 03:47You can start seeing what you're shooting.
  59. 03:48If today you pick up your iPhone and you start recording a video, then you can see what's going on.
  60. 03:54And that's why we can produce high quality content.
  61. 03:56It's because you are able to see what's going on in the screen and adapt accordingly.
  62. 04:01That gave rise to in Instagram and Tik Tok.
  63. 04:04Instagram and Tik Tok would not exist if we could not produce high quality content.
  64. 04:08And the only reason why we're able to produce high quality content among many reasons is because we can see in real time what's happening.
  65. 04:15It's not a slot machine.
  66. 04:16We can actually just edit and see.
  67. 04:19And that's what unlocks all of these new use cases.
  68. 04:24So when video becomes programmable, you can address it.
  69. 04:27You can condition it.
  70. 04:28You can change it.
  71. 04:29I can show you on the screen whatever I want to show you on the screen.
  72. 04:32And it becomes programmable a little bit like software or like anything that is programmable in the world.
  73. 04:39And in in the market today we are seeing three kinds of models that do this.
  74. 04:44Some of them you will be familiar with others you will not be familiar with.
  75. 04:48The first one is these are VO think about VO or Sora but real time and interactive meaning that first of all they're infinite.
  76. 04:56So they don't stop after 5 10 or 30 seconds.
  77. 05:00They actually continue forever.
  78. 05:02They're interactive, meaning you can change what's happening on the screen, and they're in real time, so you don't need to wait to see what's going on.
  79. 05:09And assuming this works, so this is an example of a of a video that I passed an image with a dog, and this was all generated in real time.
  80. 05:17And at some point, I'm going to prompt a cat shows up and you will see that a cat showed up in the in the video.
  81. 05:22This would not be possible in the existing batch regular video generation models because you would get back the video and you cannot do anything about it.
  82. 05:30I could have added anything.
  83. 05:31I could have gone on to create an entire story with it.
  84. 05:33I could have said the cat the dog starts running, starts jumping, a dragon shows up, it goes to I don't know to the world cup.
  85. 05:40All of this would have happened in front of you.
  86. 05:43These type these types of models pro like unlock a few things.
  87. 05:47First of all, control.
  88. 05:49So if you think about generative media today, the big problem that content creators all have is I don't have the control I need.
  89. 05:55Like yes, it's great to use Cense 2 or VO3 to generate videos, but I just don't have the control.
  90. 06:01And this is always the thing that any filmmaker or movie producer or any content creator will tell you.
  91. 06:07And so real time actually ends the slot machine type of mentality and actually gives you the the control that you need.
  92. 06:14And a big saying I like to say is instant feedback is the ultimate level of control.
  93. 06:19And you we will never be able to have this level of control if we don't have real time.
  94. 06:23The second thing it unlocks, which you know, among other things, which is a field I'm not particularly fond of, but I think is going to be big, is advertising.
  95. 06:33If I can know what you looked for a minute ago, why can't I insert the logo of whatever you've been looking for?
  96. 06:39Why can't I produce an ad in real time in front of you?
  97. 06:41We don't need to produce pre-produce anything.
  98. 06:44Now granted, this is going to take some time because you know brands afraid of AI, afraid of uh you know if their logo has one pixel that is white instead of dark
  99. 06:55but it will happen eventually and so and I think at the moment that happens we will not be to need to produce any ads anymore.
  100. 07:02Everything will be happening in front of you in real time.
  101. 07:06The second type of model is the one that probably you're most familiar with.
  102. 07:09It's the Genie3 like from Google.
  103. 07:12These are models where you can pass an image and a text typically and you can control a character.
  104. 07:18They're fun.
  105. 07:19The first thing that you think about when you think about this is games, right?
  106. 07:22It's a character.
  107. 07:23It's a world.
  108. 07:24You can generate anything.
  109. 07:26But actually, it actually goes way far way beyond games.
  110. 07:32It creates entire new interactive experiences where combined with the first types of models that I talked about, we've already been seeing people in our community building a mix of games and and movies.
  111. 07:42If you've ever watched Benders Nash from Netflix, which is the game, the movie that you can pick your next scene, this is one of those things that gets uh that that becomes possible that you can control a character, you can control what's happening and create entire new types of interactive experiences
  112. 07:57that were not possible.
  113. 08:00The second thing is robotics.
  114. 08:03So because you can simulate and you can control, you can actually create as much training data as you want.
  115. 08:07And robotics, world models in robotics is actually a ginormous market today.
  116. 08:12Um I cannot tell you the number of robotics labs that um are training and building these models.
  117. 08:17But because you can control whatever you want to control in any environment, this creates a new opportunity to generate infinite amount of data for robotics.
  118. 08:26And finally, something I like to think about, this is more maybe a passionate thing that I have is education.
  119. 08:32Um, because you can step into anything who with today's world in AI, I don't actually believe that educ in the future of education is LLM based or textbook based.
  120. 08:42If you can put any any kid in the situation, for example, in a history lesson, that enables entire new types of educ experiences that uh can be educational.
  121. 08:56The third type of model is probably the type of model that you know is more, let's say, um, something that we've been seeing before, which is avatars, but live and interactive.
  122. 09:06The thing with avatars, though, is it hasn't actually been cracked.
  123. 09:10They're still all kind of weird.
  124. 09:11If you speak to an avatar in any customer support or anything, it's still kind of off, right?
  125. 09:16Um, and these types of models, and we're seeing a rise of these live and interactive avatar models in research preview, that combined with model one and model two is actually going to be,
  126. 09:26I believe, a a big change in what we've been seeing so far.
  127. 09:30And this will be applied to things like customer support, training, sales, gaming, streaming services, etc. And so just just to give you a glimpse of what our users are building today at Reactor with these types of models,
  128. 09:44some of them are building interactive live stream, right?
  129. 09:47A live stream where people are watching and then the users can type what happens next and then they vote.
  130. 09:52Why?
  131. 09:53Because pixels can be generated in real time.
  132. 09:55So there's no reason why I cannot put a live stream on X, YouTube or Twitch and enable users to pick what happens next.
  133. 10:02Something that was surprising to me is a little bit on on the medical simulation.
  134. 10:06So we've seen users create applications where you generate a world and then you simulate what happens next.
  135. 10:12What if I put this medicine?
  136. 10:13What if I remove this medicine?
  137. 10:15Right?
  138. 10:15And this creates can be a training playground for people wanting to become doctors.
  139. 10:21The third one which is also kind of surprising is cooking simulation.
  140. 10:25So people are building applications where you can simulate cooking and what happens if you put this ingredient.
  141. 10:32And finally, video editing.
  142. 10:34With videotovideo models, video editing becomes very interesting because I'm able to just add visual effects in real time.
  143. 10:42And we've seen people build entire video edit editing platforms.
  144. 10:46Now, granted, they're not very good yet just because of the quality of the models, but it's a new paradigm when you're able to edit videos just via prompting or via talking to it or via clicking.
  145. 10:58And now for the final part, how do you actually do all of this?
  146. 11:02And this is why I like to save the world behind an API.
  147. 11:07At at Reactor, we have four types of models today.
  148. 11:09First one is called Helios, which is the interactive video model that I talked about.
  149. 11:13This one is from Bite Dance.
  150. 11:15Lingbot, which is a world model like Genie 3 trained by Alibaba.
  151. 11:20Long live 2 from Nvidia, which is multi-shot film.
  152. 11:24You can prompt things in advance and create a consistent story over time and sound streaming also from Nvidia which is a model that does video to video editing.
  153. 11:34So people are already using this for example by shooting something or creating something on CDE 2 uploading it and then adding visual effects removing people adding background and this be gets very interesting in previsualization
  154. 11:47for example for Hollywood movies and under the hood when we talk about infrastructure the thing that I think I like to drive home is building infrastructure for regular video generation models is very different from real time because in regular video generation models, you're talking about requests.
  155. 12:06You just send a request, a job gets run in the cloud, and I'm oversimplifying here, but you know, a job runs in the cloud and it gives you back a file.
  156. 12:14With real time, it's a different ballgame.
  157. 12:17Uh you cannot just take what works for batch inference and apply to real-time inference.
  158. 12:22For example, you need to think about streaming, right?
  159. 12:25Once you need to think about streaming um pixels uh from a server to the client um it adds entire new entire complexities that batch generation does not have to think about.
  160. 12:37The second one is that everything is a live session.
  161. 12:39So everything runs constantly and there's memory to be kept into account.
  162. 12:43Now granted one of the things that live vid live realtime models struggle with is memory.
  163. 12:49If you've seen demos from Genie, for example, we've all seen that the character can look back and then not remember what what's going on.
  164. 12:56So, there is a lot of work that needs to be going into maintaining that context window so that you can remember what happened if you turned your character left and right.
  165. 13:06And finally, global scale.
  166. 13:08If you if you're think about real time, it needs to be sub 100 millisecond latency anywhere you are.
  167. 13:13And if you're deploying applications in the world, then someone based in India or someone based in Japan should be routed to a GPU that is based in India or Japan or as close as possible to it.
  168. 13:26If not, if you don't have the compute worldwide, then the experiences are not real time anymore and it breaks completely the medium.
  169. 13:35And with Reactor, this is as easy as it gets to integrate these real-time mods.
  170. 13:39Okay, I kind of maybe oversimplified it a little bit here, but it really is maybe 10 lines of code.
  171. 13:44Um, and we have docs to do all of this, but essentially you can just load the model with an API key and just start integrating it in whatever video, image, plugin,
  172. 13:54anything that you're building.
  173. 13:56And if you want to get started, there's a QR code there.
  174. 13:59And I've added a promo code AIE 2026, which will give you $75 off.
  175. 14:05Well, $75 worth of credits, which is um significant amount of compute uh in our case because we make the models extremely cheap.
  176. 14:15Thank you very much.
  177. 14:18[applause]
  178. 14:19I'm happy to take questions if anybody has a question.
  179. 14:26Okay.
  180. 14:40uh well one thing that actually we do is multiGPUs.
  181. 14:44So we use multiple GPUs.
  182. 14:46Um optim optimizing the model weights applying quantization techniques.
  183. 14:51So there are ways it's just a matter of priorities but it is uh there are ways around it.
  184. 14:56Yeah good question though.
  185. 15:06Hey.
  186. 15:07Yeah.
  187. 15:15Sorry.
  188. 15:16Are you aware of what the
  189. 15:18uh No.
  190. 15:21Okay.
  191. 15:27I wasn't, but now I'm going to look at it.
  192. 15:29Yeah.
  193. 15:29[laughter]
  194. 15:31Yeah.
  195. 15:35How do you feel like?
  196. 15:41Yeah.
  197. 15:45Yeah.
  198. 15:46Do you guys experiment with that?
  199. 15:49When you say deterministic engines, what do you mean exactly?
  200. 15:53any kind of like deterministic rule set that you check against maybe so that the simulation stays
  201. 16:03so we don't do any of that today um the reason why also we build a developer platform and but we've seen people build that on top of us um
  202. 16:12building the infra for this is already a lot of work um and I think we we were seeing already developers building and then open sourcing and then we reuse it
  203. 16:21um but so the community is doing it for which is even better.
  204. 16:24Yeah.
  205. 16:26Question.
  206. 16:26Yeah.
  207. 16:33Uh you're asking a question that the entire research community in world models has not answered.
  208. 16:38You're evolves for real time and consistency is uh and fidelity.
  209. 16:42Well, fidelity is easy.
  210. 16:43It's just like pixels, right?
  211. 16:45But um evaluation for these real-time models is an unsolved problem.
  212. 16:50So today it's literally just look at it and human judgment.
  213. 16:55That's what it is today.
  214. 16:57And this is including by the way J deep mind and everything.
  215. 17:00Nobody has solved this problem yet.
  216. 17:04We're working on it.
  217. 17:05We have a research team.
  218. 17:06Yes.
  219. 17:06[laughter]
  220. 17:11Awesome.
  221. 17:11Cool.
  222. 17:11Well, thank you everyone.
  223. 17:12Thanks for your time.