Building an Agentic Video Editor for Mass Consumer — Ekaterina Deyneka, Reelful

AI Engineer · 12 min · 132 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:13I think we can start.
00:15Uh but before we start, I want to ask you a couple of questions.
00:19So, first of all, how many of you took a photo or video during this conference?
00:24Please raise your hands.
00:27Okay, and how many of you actually posted any video content from it online?
00:36Not that many.
00:38And um to be honest, that was me.
00:41I I was recording a lot of content during conferences, events, trips, meetups, uh and I never posted them online because video editing is hard.
00:53Uh it sounds and it is a lot of work.
00:56Uh it's tedious and it's largely still manual.
01:01So, and and it also feels like an art and not really automated.
01:07And that's why we're building RealFull and we're trying to tackle agentic video editing problem from uh video editing problem from the agentic standpoint.
01:18I'm Kate.
01:19I'm founder and CEO at RealFull.
01:22But let's first talk about what's agentic video editing is.
01:27So, as a user, you just drop in your media, photos and videos, and provide some context.
01:34It can be uh the context what happened in these media files, or it can be some directions.
01:39For example, like add captions, add music, add voiceover, and something like that.
01:46And then, the agent will go, understand your media, uh find the right moments, assemble everything together, generate captions, music, voiceover, b-rolls, uh and give you a ready-to-share clip.
02:02Uh and um yeah, so basically, uh that's uh video agentic video editing.
02:09So, agent does everything by itself.
02:12Or another example, you recorded a speak-to-camera video and you have a lot of pauses, unsuccessful shots, and you expect an agent to figure it out to remove unsuccessful shots, remove pauses, and give you a ready-to-share
02:30clip.
02:31Um and uh the the interesting thing is that a lot of actually a lot of the things inside this pipeline can be automated.
02:42And this is exactly what we're doing at Real Fall.
02:45Oh, this is um the example of uh video edited.
02:52And um since we're at AI Engineering Conference, I wanted to talk a little bit about infrastructure.
03:00And from the infrastructure standpoint, agentic uh video editor is very similar to agentic app builder.
03:08Uh sorry, there is a typo on the slide.
03:10So, the uh second column is agentic video editor.
03:14So, both of them have a prompt uh a UI for prompt uh and in the video editor case, it's a media plus prompt.
03:23And usually on back end, what's happening?
03:26There is a remote machine which is called sandbox, uh which is spinning up, and inside this machine, there is an agent with tools and skills, which is working on
03:38uh what you're uh you're you're asking it to do.
03:40Uh in the case of the agentic uh app builder, it's a code base.
03:44Uh in the case of the agentic video editor, it's a video video composition.
03:50And as a result, in the Argentic builder user get an app preview and for the video editor users user get rendered a rendered video.
04:03And but yes, the the infrastruc- from infrastructural standpoint, it's pretty similar, but there are a couple of differences.
04:11Uh and this is actually the most interesting to me, generating versus editing.
04:17At RealFull, we're focusing on editing real footage.
04:22So, we do not generate a lot of content.
04:24We are expecting you to provide your real life, your personal content, and we will edit it for you.
04:33And actually, this is a more complex problem because if the agent has a blank blank canvas, it can do whatever they can.
04:43But in the editing case, the agent has to figure out which moments are the best.
04:50Uh what to omit, what to use, how to organize everything together.
04:55And also, um sometimes footage can be messy or incomplete, and agent still has to deliver a very polished result, professionally made, so that ideally the viewers of this content
05:11don't get if it is like AI or human edited.
05:17So, let's actually have a look how we do it at RealFull.
05:22So, we start, as I already mentioned, with your media plus a prompt, some directions like how you want it to be edited, and we need to get a polished
05:34clip.
05:35So, let's go through it step by step.
05:37So, we are doing first media understanding.
05:41We need to understand what's what's actually happening uh, on those clips and photos.
05:47And we also need to transcribe transcribe speech, for example, in the case if you have speak-to-camera videos.
05:54Then, we are providing a creative plan for the user so that they can approve if they like it or not, what they want to change or maybe regenerate, uh, and we
06:06create this plan before actually starting editing.
06:10Once the user approved this plan, uh, we spin up a sandbox, the remote uh, remote machine that we already discussed, and this is an environment for the agent to,
06:22uh, execute everything.
06:24So, the agent comes with the skills, and in our case, in in the case of, uh, video editing, our skills are, for example, cut rules, how, for example, how to select
06:35the best moments, uh, also font pairs, which fonts are, uh, more suitable for this use case, which are not, for example, how to generate B-rolls, and this is where our taste and craft,
06:48uh, live, actually.
06:50Um, and then also agent, uh, can, um, can initiate some other sub-processes, for example, generating music that will fit this exact composition, generating voice-over, adding sounds, animating images.
07:05Yes, this is actually what we do, uh, if you provide photos, we can animate your photos to make them more, uh, dynamic and engaging.
07:15And then comes Remotion composition.
07:18So, here a little bit of background.
07:19What's Remotion?
07:20Remotion is a framework, open-source open-source framework, uh, to create videos as code, as React code.
07:30Uh, so basically, it's just like a a file with the order with all your assets and tracks and how they're following each other.
07:39And, why it is important?
07:42Because, uh, agents are really good at writing code and therefore we can use them to create videos with this remotion framework.
07:51And then the last thing is the verification layer.
07:55Of course agent can make mistakes and that's why we develop this verification layer to make sure that all the the composition is clean, is well defined, everything will be rendered
08:08and if there are there are some problems then the agent will reiterate on the composition.
08:15And this is how we got to a polished clip.
08:19[sighs and gasps]
08:19So it's a lot, right?
08:21It's like very complex workflow and ideally we don't want our users to even know anything about it.
08:31And this is even maybe a bigger problem how to deliver this complex agentic workflow to mass consumer.
08:40And this is how we're tackling that at Real Flow.
08:43So we decided to go mobile first so that users can edit videos videos while driving, walking or maybe lifting weights.
08:53Also, I know that prompting videos can sometimes be also challenging.
08:59That's why we create directional templates.
09:03For example, like speak to camera in videos or maybe you want to add B-rolls or voiceover so that users can just select these directional templates, drop their media and that's it.
09:15Even without any prompt it will it will work.
09:19And the third thing is a building editor.
09:23Why?
09:23Because we want to make this experience convenient and familiar for users.
09:28So a lot of people are already sort of using regular video editors and that's why we want to provide this experience as well.
09:38So, how it works?
09:39User first generates a video agentically, but if they want to tweak it, for example, remove a second or maybe correct some word in the captions, they can go into building editor and
09:53edit it a little bit.
09:55Um Yeah, and actually I have a couple of examples here that I recently created with Real Fill.
10:03I will play them just maybe one of that.
10:05Oh, sorry.
10:08Last week I was invited
10:09Do do you hear?
10:12A little bit.
10:12Okay, you just can enjoy the video.
10:15because of course our creators dinner and honestly it was one of the best event experiences I've had.
10:20First of all, the venue was stunning.
10:22It was a custom event space transformed into this tropical sunset feast.
10:26The whole atmosphere felt so warm and cinematic.
10:29Second, the food was way beyond my expectations.
10:31They brought in private chefs from LA and we had lobster, blue crab, and this incredible ice cream that I'm still thinking about.
10:38But most importantly, the conversations were so much fun.
10:42It was kind of atmosphere.
10:43It felt really easy to connect with people, talk about what they're working on, and just enjoy the community.
10:48And lastly, we got gifts.
10:50One of the highlights
10:52So, yeah, basically all these videos they were assembled only using agent, no regular video editor, and I already posting them on social media.
11:04And yeah, it I I have a lot of fun with that.
11:08And oh, sorry.
11:11And exclusively for this conference, we are giving our better, which is our new new second version.
11:22Please give it a try and let me know if you have any feedback.
11:28Here is my email.
11:30Please Uh free to reach out.
11:33Uh we're still early.
11:34We're actively working on it.
11:37Uh so we will have uh we will be happy to hear any feedback and also curious how you use it.
11:44And also um some exciting news.
11:46We recently got funded by A16Z speed run.
11:50Uh so I'm very excited to uh continue working.
11:55Um
11:55[applause]
11:57Yeah, that's it.
11:58Thank you so much.
11:59And because it's a presentation about content and how to edit videos, I have to uh film a video with you all.
12:12Second.
12:25Yay!
12:42[music]