Evaling Video Slop — Maor Bril, Character.ai

AI Engineer · 23 min · 259 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12So, hi.
00:13I'm Mayur.
00:14I've been with Character for a bit over 2 years and we'll talk about AI slop, right?
00:21I think that, you know, when we look at video generations as a whole, right?
00:27We have like two kind of uh parallel tracks.
00:30One is the the video generation, which became insanely good from from models like Kling and SeaDance and VEO and Sora.
00:40We still remember Sora.
00:43But but but the part that got left behind is how we evaluate the quality of the video that was generated, right?
00:51So, on on the one hand, um we we we still kind of squint at it and decide whether or not it's it's good, but on the other hand, we know that the generation has gotten a lot better.
01:01And when when we when we we look at at X or whatever social you're you're you're consuming your content on, there are a lot of guides on how to create amazing videos with
01:13um this model or or that.
01:16So, the hard part was never how to make video.
01:19The hard part was how do we generate um good enough video and how do we judge if the video is good enough?
01:28So, so now we've gotten to a world where the the generation of video is basically free, right?
01:34Free as especially when you compare it to how much studios would charge.
01:38Um and but but the problem is the most the grand majority of videos that is generated is not that good, right?
01:45We have like a lot of hallucinations like a third limb, uh opening and closing the the door at the same time, hovering, physics, um etc. So, unfortunately, in order to get high high high-quality
01:57content, we need a human to judge.
02:00And I I know when was the last time you've seen how someone is creating these long-form um generated video.
02:08It's usually a lot of shorter generations and a lot of editing.
02:13The problem is because we're using a lot of the tools that we built for the ticks for the text era, for the image era, for videos, right?
02:22We're using things like clip score, which is is great to to to judge a single frame.
02:28Things like LP IPS will help us kind of detect the drift between frames, but we don't have I mean but the problem is when you kind of combine all these together, all these tools are good at watching the individual frames.
02:44They're good at checking this this this this does this specific frame does it match the the prompt that generated it, right?
02:54It will check consistency between frames and it will check whether or not it match the prompt that the drove it.
02:59But but what it won't do it doesn't tell you if if it's if you told if you told the story that you you meant to tell, right?
03:07If you think about what is video, video is a storytelling medium.
03:11Video is just another form on how we tell a story, right?
03:14From for any type of story.
03:17So so one of one of the things we have to look at, does it tell the actual story?
03:21Does the physics make sense?
03:22Like for example, if you want a video of a character walking downstairs, does it actually walk or or hover?
03:29Does the character stay the same character across multiple shots?
03:35Does the pacing make sense?
03:36Like you know, for example, people take time going from one place to another.
03:41We need to make sure that the pacing makes sense as well.
03:44And especially when we add audio, we want to make sure that the audio is kind of synced with the imagery.
03:48Like for example, if someone is slamming a door, we want that that that that that that sound of the door being slammed to be exactly when the door is actually being slammed.
03:59Now, the the next iteration we all went to a while ago.
04:03We started using LLM as a judge for everything and we have amazing foundational models that we just throw videos at them.
04:11The problem with them is that A, they're slow.
04:14B, they're only as good as your prompt and multiple people will prompt multiple ways and the same model may respond in a very very different way.
04:22And sometimes the prompt we use like is it consistent?
04:26Does this match the the prompt?
04:28But but then the question we really care about is it good?
04:31And the answer varies.
04:34So, oops, sorry about that.
04:36So, our first iteration is like let's take all these things and build a repeatable benchmark on how we test video that we can rerun over and over and over again.
04:47So, so that combines both metrics as I said earlier that that knows how to view individual frames, but also consistent LLM as a judge, right?
04:56Where we also use human annotation to to calibrate the the LLM as a judge.
05:01So, for every report that we generate with that harness, we're able to have humans annotate it and basically feed that feedback back in into the the the LLM as a judge prompt to make sure that it's it's aligned with what I think or what the annotator thought is good.
05:19And and we use it to to score the videos.
05:22The problem with this approach, it's very slow, it's very expensive and especially when we we want to bring it in for our users to be able to generate a lot of video because creation is a very hard process.
05:35And so the problem as I said, the problem is when when So, this is a slow process and we need to bring it as close to the users as possible and also earlier into the process.
05:46The reason for that is if we take a look at all the metrics and there's there mistakes that we can find earlier than than later than it's a lot cheaper to correct that that particular
06:00mistake.
06:01So, for example, right?
06:03On on the left we have two starting frames of different different shots, right?
06:12But it's easy to to correct to to view it at this point and see did the character drift between frame one and and frame two because those frames will be used as starting frames to generate videos.
06:23So, if you can correct catch the drift at this point and correct it, then then it's much cheaper to generate the video as a whole because we we can correct it at a much
06:33cheaper cost.
06:34And the same thing applies when we look at longer form video, right?
06:39When we see all these three, four, five-minute long videos, they're usually a collection of a lot of shorter videos.
06:45And being able to catch a six-second generation that drifted and and regenerated that regenerate that before we combine the whole video will end up being a better result as a whole.
06:58Um
07:00[clears throat]
07:01And now the the other problem what we're trying to solve is some of these axes, right?
07:06Only exist across time, right?
07:09So, for example, when we look at the right?
07:13We we mentioned the story, right?
07:16So, does does the story that we're trying to tell with that video, does it hold in that video?
07:21Does the video tell the exact story?
07:23Does the pacing make sense, right?
07:25And we mentioned the the the the sound.
07:28So, as I said, right?
07:30The the you know, the underlying goal is to bring that evals closer to to the to the the online generation because the sooner we're able to catch those mistakes, we were to
07:41the sooner we're able to catch that drift, right?
07:43Then it's um it's much easier, much much cheaper to fix.
07:49Now, so so that now now the problem is that as I said, this is a very slow process.
07:56So the the solution is actually actually to take all these committee of experts and distill it into one small model that is also very very fast, but it is able to give us
08:10a response that is not whether or not this video is slap or not, but why is it slap?
08:15Right?
08:15Why Why is that video scored low versus the other?
08:19Because for example, it added an extra limb, because it didn't obey physics, because the audio was was out of sync.
08:27So so the the goal was a build it on top of a small VLM and why is it a VLM?
08:33VLM because we needed the video the model to be able to see the image, but also we needed to work fast, right?
08:40Because we bought it closer to the generation, where in fact it takes about the model we have trained, it takes about 3 seconds to score a three 15 second video.
08:55Now, the we we also tested a bigger model and the results were better, but it was significantly slower.
09:03And and the decision was to go with the the the smaller model because the the added value from the bigger model didn't justify the the the the slowness.
09:16The other very interesting realization we came to is don't score compare.
09:22What does that mean?
09:23For example, if I'll ask any person in this room to look at a particular video and and rank it from 1 to 10 on storytelling, right?
09:32I'm pretty sure that you know, what will be a six for you will be a five for you, will be a four for you and and and an and an eight for you.
09:38Right?
09:38But if I show you two videos and I'll ask you which one of them is telling a better story, the grand majority will probably agree that B is telling a better story than A, right?
09:47And if you do it enough times, then it it uh it's easy to um uh generalize the model at um towards detecting what's better uh uh versus uh not.
10:00So,
10:01[clears throat]
10:02we trained on pairs, right?
10:04Um uh A versus B as opposed to 1 through 10.
10:08Now, we manufactured badness.
10:10So, luckily, the internet is full of very high-quality videos, and it's very very easy to get good videos, and it was very fun to create bad videos A by either corrupting
10:23good videos or by, you know, just generating random slop.
10:29[clears throat]
10:29Um Now, we shipped V1, and it was so wrong.
10:35It It was wrong, but it was wrong in a very confident way.
10:38So, uh uh for for for example, the the frame you see here is from from a video that the model scored 9.2 on the camera work, and the camera didn't move.
10:48For for 4 seconds, it was like a still image of the same character, but but but the model was was very very happy with with, you know, the cinematography.
10:59Um so, the physics in in in in some other videos, which I'm not showing the cuz of uh time limitations, it says that the physics look great, but it said it on on ghosts hovering and people flying, etc.
11:15So, the I mean, so the so then the question is like, why was it wrong?
11:20The The reason it was wrong is because how we generated that data, right?
11:25It It um it scored the vibe as opposed to the the the axes.
11:29So, it it learned how to how to detect um cohe- coherent videos and it learned how to detect the the the the the the artificial artifacts.
11:38Basically, the gloss of the video as opposed to uh whether or not the video actually um told the the the the the Sorry.
11:48The videos actually told the the the story.
11:51And um And so so so the solution was to fix the data set.
11:59And so the way we we fixed the the the the the the data set, we actually I uh um I started pairing real footage versus um AI footage.
12:10Now, the risk with that and that that's the reason why I voted I voted I avoided doing it at first is because I didn't didn't want to create an AI detector,
12:18right?
12:18Because if you start creating pairs of good is is is human-generated video and bad is AI uh uh video, then then there's a very big chance of of the model overfitting
12:29and becoming an AI detector as opposed to a um uh uh video quality detector.
12:34So, the There are two things I did in order to avoid that.
12:38A, I made sure that the encoding is is consistent across both sides of of the equation.
12:45So, uh there there there there's no um artificial artifacts for for video A versus video B. Uh and I I used the exact same method of annotating both videos.
12:58So, so both the axes So, all the axes in those videos were annotated in the same way.
13:04Uh and surprise, it it turned out pretty awesome.
13:09Um and so so so now what what what we're able to do, especially when we you looking at at at videos, A, we we changed from a very complex pipeline,
13:21right?
13:22To an agentic workflow.
13:26The reason behind this is, a, the pipelines work great if you have a very very unique use case.
13:31But one once you put put put it in front of users, they'll have a very very distinct story that they want to tell with their own characters, with with their own
13:39images, and their own voice.
13:40So that's where it starts to drift.
13:42But by providing the agents with tools to validate the quality of of the outputs it's creating, it's able to adapt to changes better.
13:51It's also able to verify its own work and and and fix things as they go along.
13:57So if you're going to steal from from these from the stock of a few things, one, go relative, not absolute, right?
14:05As I explained earlier, the the value of comparing video A versus video B will always give you a better result going forward.
14:16Be score the real axis that you care about.
14:19So if you care about storytelling, if you care about pacing, if you care about physics, score those axes.
14:27Don't expect them to miraculously appear.
14:31And put eval inside the generation loop, right?
14:36Especially if your goal is is is is to have a higher quality of generation, get the the eval as close to to the generation loop loop as possible.
14:47Eventually evaluate it as a story.
14:49Videos are stories.
14:51Videos are just another way for us to tell stories to others.
14:55And Thank thank you very much.
14:59[applause]
15:05All right.
15:06Any questions?
15:06Okay, down here.
15:08Awesome.
15:09All right, I got two down here.
15:11Here you go.
15:16Hi.
15:17How do you eval sound?
15:18Sound and video matching.
15:21I'm I'm I'm Can you you repeat?
15:22How do you evaluate sound?
15:24Sound effects and matching with the video.
15:26oh yeah.
15:28That's a fantastic question.
15:29So, um so so sound is actually a a combination of few things.
15:34One, I'm using Atmos to to to make sure that that that the the the sound quality is high enough and is understandable.
15:47B, the model will will will will learn to learn to identify key frames.
15:53Right?
15:53And and and especially because when I feed something into the model, it's it can be just a video or it can be the video plus the prompt that generated
16:02that video.
16:03So, for example, if the prompt will say the door slammed, right?
16:07It will look for a door being slammed and and and will will will match the sound at that same frame.
16:18I Did I answer your question?
16:20How does the model recognize sound?
16:22Uh so, it's both by using um um uh Atmos, but and also to to correlate the So, for example, when it's looking at the frames, right?
16:35It's making sure that that for example, the door being slammed at frame six, frame six has a a specific timestamp.
16:43So, it's looking for that that spike in the sound at that timestamp.
16:50It doesn't know that it is that sound, but it's looking for a specific spike of sound at at that timestamp.
16:57What about lip syncing?
16:58Lip syncing is an unsolved problem yet.
17:02[laughter]
17:03One question.
17:04We're trying though.
17:06Yeah.
17:09Yeah, I said the the question was what about lip syncing?
17:14Oh, Well, wasn't me, but I guess the lip syncing answer would be interesting before I ask my question.
17:21Uh yeah, um as I said, it is an unsolved problem still.
17:25We're still working through it.
17:29Especially for us, some some of the characters that we're trying to do a talking head are humans, right?
17:35Which, you know, we can look at the different techniques to trying to identify the the lips, but some of them are just talking, you know, talking animations that have no real correlation between, you know, the the movement of of of the mouth and and and speech.
17:50So, unfortunately, I don't have a solution for that yet.
17:54So, I'm So, I'm curious about, for example, if you wanted to further enrich the data set with human evaluation.
18:02Yes.
18:02Um the question of of taste in what is good, because I think there is a big question mark about is that is that going to remain the domain of humans?
18:13But, I've also seen people say that, well, most humans they have terrible taste anyway in in videos and games and books.
18:20Fair.
18:20Um so, how would you construct and align sort of like any human judges?
18:28Yeah.
18:28So, So, So, So, this is actually solved at first at the the the Judge Judy part, where every report it will generate a human can go and annotate it.
18:39And we actually we we do that.
18:42We we will periodically have sessions where everyone spends 10 to 15 minutes just annotating videos.
18:48And and that usually happens on multiple axes.
18:53I won't ask everyone to annotate the same video on on 10 different things.
18:56It'll It'll It'll be random.
18:57And I use the data to to calibrate the the AI judges.
19:03And and the results from that is actually being served as as a data set for for training for the next version of of that model.
19:10So, it's a process that does take a little bit of time and and hopefully and and it does evolve over time.
19:16Uh but it's not immediate you know, because also taste is very subjective and and things that are great for me, you know, some that I think are fantastics some people that come and say
19:28are you sure they're great because you know So, yeah, it's it's it's a process and and and and I use the human feedback to calibrate the models all the time.
19:39Um how did you land on the Quan small VLM?
19:42Did you try any others?
19:44Uh I did.
19:45So, so the intent I had was to A you know, find a small enough model.
19:51The reason I went with Quan is because we also had a very good experience with post training Quan on other use cases.
19:59So, it I mean yes, I could have I did try a few others, but it just you know, everything was just there and it was good enough.
20:09So, my question is about scale.
20:10So, obviously Character AI produces thousands, millions, bajillion videos.
20:15What scale does this become reasonable for my domain that is not Character AI?
20:19So, my domain has hundreds, maybe a thousand videos.
20:22Mhm.
20:23Sure.
20:24So, if you're happy with with the cohort of experts and and and and it and you don't need right so I'll rephrase that.
20:34The scale is both for speed, right?
20:37As well as capacity because I can serve this model as one instance on one GPU or I can serve it as you know, 100 instances, right?
20:45So, so that's that determines my scale.
20:48The reason I chose to go towards the model is because I wanted to to speed up the creation process, right?
20:56It would work would have worked just as well if I didn't have this particular model, I would have used like the cohort of experts, right?
21:03From metrics that are available both on CPU and GPU as well as the frontier models.
21:12Right?
21:12So it's it was a balance as you know, A, how long did it take me to to to train this model and to curate the the data set and and get it to a working set, right?
21:23And and how much does it cost to serve it versus how much it would have cost me to do this A slower.
21:29Now potentially it is better, right?
21:32I mean like I assume that if you're going to use Fable which came back today, right?
21:36It will probably give you a a better result, but at what cost, right?
21:41If you do it for one or two, that's probably fine.
21:42If if you do it for thousands or tens of thousands per day, it adds up.
21:48So it's it's it's a matter of your your your unit economics.
21:54Cool.
21:55Over here.
21:56On your right, there you go.
22:00Last question.
22:00It's very bright.
22:01I'm sorry.
22:02No worries.
22:02No worries.
22:03Um My question is I looked a bit at the repo.
22:07You guys don't export all tail traces of the LMS judges yet.
22:11Correct.
22:12Is that something are you open to that so you can connect to other platforms?
22:17Sure.
22:17So so the the the repo itself it's a harness and you can connect any any agents or any LLMs you want.
22:29We actually have an internal version of this which is running it as a service, right?
22:33We we have an agentic harness on top of it that has all the the the the metrics we we care about, but I I I I do accept your feature request and I'll be adding
22:44hotel telemetry to the the harness.
22:49Awesome.
22:49Thank you very much.
22:50A warm welcome or round of applause for Mayor.
22:53Thank you.
22:54Thank you all.
22:55Thanks.
23:12Oh.