{
  "video": {
    "id": "b_PmGocP4rc",
    "title": "Evaling Video Slop — Maor Bril, Character.ai",
    "duration": 1393,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "b_PmGocP4rc",
    "title": "Evaling Video Slop — Maor Bril, Character.ai",
    "one_liner": "Character.ai distilled a slow committee of frame metrics, LLM judges and human annotation into a small Qwen VLM that scores a 15-second video in about 3 seconds on pairwise A/B comparisons — so quality eval can sit inside the generation loop instead of after it.",
    "summary": "Video generation got very good very fast (Kling, SeaDance, VEO, Sora), but evaluation was left behind: CLIP score and LPIPS judge individual frames and frame-to-frame drift, and tell you nothing about whether the video told the story you meant to tell. The talk walks through Character.ai's path from a slow, expensive benchmark harness (frame metrics plus human-calibrated LLM-as-a-judge) to a distilled small VLM that is fast enough to run close to generation, trained on pairwise comparisons rather than 1–10 scores, and that explains why a video is slop rather than just scoring it. V1 shipped confidently wrong — 9.2 on camera work for a shot where the camera never moved, 'physics look great' on hovering ghosts — because the training data taught it to score gloss and coherence rather than the axes; the fix was rebuilding the dataset by pairing real footage against AI footage with identical encoding and identical annotation on both sides.",
    "key_points": [
      "The frame-era toolkit is the wrong shape for video: CLIP score judges a single frame, LPIPS detects drift between frames, and both check prompt-match and consistency — none of them answer 'is it good?' or 'did it tell the story?'",
      "The axes that actually matter only exist across time: does it tell the story, does the physics make sense (does a character walk downstairs or hover), does the character stay the same character across shots, does the pacing make sense, and is the audio synced to the imagery (the door slam sound landing exactly when the door slams).",
      "LLM-as-a-judge on foundation models is slow, and only as good as the prompt — multiple people prompt multiple ways and the same model answers very differently.",
      "Eval must move earlier and closer to the user, because mistakes are far cheaper to fix early: catching character drift between the two starting frames of adjacent shots costs almost nothing, and catching a drifted 6-second generation before it is combined into a 3–5 minute video avoids regenerating the whole thing.",
      "The solution was distilling the committee of experts into one small VLM — a VLM because the model must see the imagery and still be fast; it takes about 3 seconds to score a 15-second video, and it outputs why the video scored low (extra limb, physics not obeyed, audio out of sync), not just whether it is slop. A bigger model scored better but the added value did not justify the slowness.",
      "'Don't score, compare': ask a room to rate storytelling 1–10 and you get 4, 5, 6 and 8 for the same video, but show two videos and most people agree which tells the better story — so the model was trained on A-vs-B pairs, not 1–10 ratings.",
      "Training data was built by manufacturing badness — the internet supplies plenty of good video, and bad video came from corrupting good videos or generating random slop. V1 learned the vibe and the gloss instead of the axes: it gave 9.2 on camera work to a 4-second still frame and praised the physics of ghosts hovering and people flying.",
      "The dataset fix was pairing real footage against AI footage, avoided at first for fear of training an AI detector; the two guards were keeping encoding consistent across both sides so there are no artifactual tells, and annotating every axis on both videos with the exact same method.",
      "The complex pipeline was replaced with an agentic workflow: pipelines work for one very specific use case, but users bring their own stories, characters, images and voices, and an agent given quality-validation tools adapts, verifies its own work and fixes problems as it goes."
    ],
    "takeaways": [
      "Go relative, not absolute — train and evaluate on pairwise A/B comparisons instead of 1–10 scores, because absolute ratings do not agree across annotators.",
      "Score the real axes you care about — storytelling, pacing, physics — explicitly; don't expect them to emerge from a generic quality score, or the model will learn gloss and coherence instead.",
      "Put eval inside the generation loop, not after it: validate starting frames before generating, and validate each short clip before assembling long-form video.",
      "If you distill a judge, check what your dataset actually teaches — pair real against synthetic only with identical encoding and identical annotation on both sides, or you build an AI detector rather than a quality detector.",
      "Choose model size on unit economics: at hundreds or a thousand videos the cohort of metrics plus frontier models is fine; a small self-served VLM only pays off at thousands to tens of thousands of scorings per day, where it can be scaled from one GPU instance to a hundred.",
      "Keep humans in the loop cheaply — short 10–15 minute annotation sessions over randomized axes, used both to calibrate the AI judges and as the training set for the next model version."
    ],
    "topics": [
      "video-generation",
      "evals",
      "llm-as-judge",
      "vlm",
      "pairwise-comparison",
      "distillation",
      "agents",
      "multimodal"
    ],
    "tools": [
      "Character.ai",
      "Kling",
      "SeaDance",
      "VEO",
      "Sora",
      "CLIP score",
      "LPIPS",
      "Qwen (the small VLM that was post-trained)",
      "Atmos",
      "Fable",
      "OpenTelemetry"
    ],
    "quotes": [
      {
        "text": "So the hard part was never how to make video.",
        "at": "01:16",
        "url": "https://www.youtube.com/watch?v=b_PmGocP4rc&t=76s"
      },
      {
        "text": "we shipped V1, and it was so wrong. It was wrong, but it was wrong in a very confident way.",
        "at": "10:34",
        "url": "https://www.youtube.com/watch?v=b_PmGocP4rc&t=634s"
      },
      {
        "text": "the model scored 9.2 on the camera work, and the camera didn't move.",
        "at": "10:44",
        "url": "https://www.youtube.com/watch?v=b_PmGocP4rc&t=644s"
      },
      {
        "text": "Eventually evaluate it as a story. Videos are stories. Videos are just another way for us to tell stories to others.",
        "at": "14:47",
        "url": "https://www.youtube.com/watch?v=b_PmGocP4rc&t=887s"
      }
    ],
    "words": 4381
  },
  "summary_url": "/#b_PmGocP4rc",
  "transcript": {
    "html": "/transcripts/b_PmGocP4rc.html",
    "txt": "/transcripts/b_PmGocP4rc.txt",
    "vtt": "/transcripts/b_PmGocP4rc.vtt"
  }
}