{
  "video": {
    "id": "jNE8No-wvok",
    "title": "Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta",
    "duration": 1292,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "jNE8No-wvok",
    "title": "Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta",
    "one_liner": "Meta's approach to catching mid-video content switches and unoriginal reposts in short-form video at 100M+ scale: three agents (perceiver, retriever, reviewer) running on small, domain-tuned and distilled VLMs, plus metadata pruning, caching and frame compression so most videos never enter the pipeline.",
    "summary": "Aditya Gautam describes two problems on short-form video surfaces. The first is intra-modality misalignment, where a video suddenly cuts to off-topic content such as an ad or political material. The second is unoriginal content: copied or AI-transformed videos that cause attribution-credit problems, ecosystem imbalance and user fatigue. He argues the problem is complex enough to need a multi-agent system: a reviewer orchestrator, a perceiver that breaks the video into clips, and a retriever that finds similar content. Each agent runs on a specialized small VLM built through in-house pre-training, instruction fine-tuning, DPO on production failures, and distillation plus quantization. He also calls for evaluation that goes beyond precision/recall to cover every node, the reasoning budget, robustness, cost and LLM-as-judge drift.",
    "key_points": [
      "The data is messy: 100M+ scale, viral and adversarial content, multilingual on-screen text, constant drift as new AI tools appear month to month, and no clear ground truth.",
      "Inter-modality alignment is largely solved by comparing CLIP-style embeddings with cosine similarity. The hard problem is intra-modality: spotting a segment, e.g. seconds 6 to 6.5, where a sports video suddenly turns political.",
      "Architecture: the reviewer agent is the orchestrator, described as an 'API gateway'. The perceiver is a fine-tuned VLM expert that splits the video wherever the content changes over time, not at a fixed frame rate, and emits per-clip JSON (embeddings, tags, OCR, natural-language description, frame timestamps). The retriever indexes the metadata adaptively: an inverted index for topics, a vector DB for embeddings, a graph DB for entities. It then reranks candidates using tools such as a spam classifier score.",
      "The retriever's indexing and periodic clustering run offline at scale (e.g. on a Ray cluster). Online, it returns the top-N similar clips and authors to the reviewer, which can go back for more. The reviewer also takes in real-time engagement signals: reports, likes and dislikes, and comment sentiment.",
      "Model training pipeline: pre-train the vision transformer on in-house image tokens, because frontier VLMs were trained on clean web data. Then instruction-tune (vision encoder → projector → LLM) on in-house labels with a JSON output schema and chain-of-thought. Then run DPO on production failures.",
      "Continuous improvement loop: sample production data daily, score it with an in-house LLM-as-judge trained on human labels, and send it to a human review queue against a threshold such as 95%. Humans inspect the traces (LLM calls, MCP, tool calls, retrieval) to find where each failure came from, which yields positive and negative samples for retraining.",
      "Frontier-size VLMs don't scale for this ('I really don't care if the model can solve a coding problem'). Instead they use off-policy and on-policy knowledge distillation plus quantization experiments (4-bit vs bfloat16) to build a table of size vs performance vs cost.",
      "Optimizations: spatio-temporal reduction merges similar frames; caching lets viral near-duplicates with high similarity scores skip the multi-agent system; metadata pruning filters out creators with high authenticity scores, good records and strong engagement before they ever reach the pipeline."
    ],
    "takeaways": [
      "Decompose into multiple agents only when one agent or one LLM can't do the job, and give each agent a clear role (perceive, retrieve, review).",
      "Evaluate the whole pipeline, not just end-task precision/recall/F1. Measure retrieval latency and recall, reasoning quality and budget (adapt it up or down), error rates per node, tool, hook and retrieval step, token cost, and per-agent latency.",
      "Keep checking your LLM-as-judge against the human queue for drift, and retrain it on fresh labels from the labeling team.",
      "For an internal, domain-specific task, distill and quantize small VLMs, choosing the operating point from a size/precision vs performance table rather than serving frontier models.",
      "Cut the candidate set before running the expensive agentic workflow: merge similar frames, cache viral near-duplicates, and prune using creator and topic metadata."
    ],
    "topics": [
      "multi-agent systems",
      "vision language models",
      "video understanding",
      "content moderation",
      "knowledge distillation",
      "evals",
      "llm-as-judge",
      "retrieval"
    ],
    "tools": [
      "Meta",
      "CLIP",
      "Ray",
      "MCP",
      "DPO",
      "vector database",
      "graph database",
      "inverted index"
    ],
    "quotes": [
      {
        "text": "if you can solve a problem with one agent one LLM you we don't need to actually do it on multi- aent systems",
        "at": "04:08",
        "url": "https://www.youtube.com/watch?v=jNE8No-wvok&t=248s"
      },
      {
        "text": "I really don't care if the model can solve a coding problem. I only care about my domain specific problem.",
        "at": "16:25",
        "url": "https://www.youtube.com/watch?v=jNE8No-wvok&t=985s"
      },
      {
        "text": "Good evaluation is paramount. Everything in and out depends on this.",
        "at": "20:55",
        "url": "https://www.youtube.com/watch?v=jNE8No-wvok&t=1255s"
      }
    ],
    "words": 4647
  },
  "summary_url": "/#jNE8No-wvok",
  "transcript": {
    "html": "/transcripts/jNE8No-wvok.html",
    "txt": "/transcripts/jNE8No-wvok.txt",
    "vtt": "/transcripts/jNE8No-wvok.vtt"
  }
}