{
  "video": {
    "id": "CDqzWpwkSls",
    "title": "Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo",
    "duration": 1553,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "CDqzWpwkSls",
    "title": "Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo",
    "one_liner": "Duolingo secretly fed fake AI cheating flags to its expert human proctors and found they upheld 50% of them — then fixed it with a copy change to the review guidelines rather than a better model or UI.",
    "summary": "Angel Ortmann Lee, a security engineer on the Duolingo English Test, argues that human-in-the-loop AI fails not because of bad models or bad people but because the interaction loop invites approval instead of discernment. She presents Duolingo's 'When Machines Mislead' study, where proctors who score above 90% on accuracy calibration accepted half of deliberately fabricated copy-typing flags — a coin-flip rate she reads as automation bias. Rewriting the proctoring guideline to say the AI signal is only a preliminary alert and that independent video evidence is required before upholding a flag raised rejection of the fake signals from 50% to 71%, with no model or UI change. She generalises this into a cyclical model→interaction→human behaviour→data→model flywheel and a set of design principles for eliciting real judgment.",
    "key_points": [
      "A Wharton study named the failure mode 'cognitive surrender': when the AI was right, human performance rose 25 percentage points; when it was wrong, it fell 15; and 80% of participants accepted AI answers even when those answers were wrong.",
      "The Duolingo English Test is fully online and remotely proctored, trusted by 6,000 programs worldwide, with identity verification, a lockdown testing environment, AI cheating detection, and human proctors reviewing all video footage plus the AI results.",
      "The copy-typing detector is a custom model on keystroke-pattern anomalies (transcribing vs composing), tuned highly conservative for test-taker fairness, with a 1% false positive rate.",
      "In the experiment, historical sessions with no cheating whatsoever were dressed up with a fake 'check for copy typing at this moment' signal and slipped into proctors' normal workflow; proctors upheld 50% of them, effectively falsely accusing people half the time, in an exam that influences college admissions and visa decisions. No real test takers were affected.",
      "The diagnosis was explicitly not the model and not the people but the interface — so the fix was two lines of guideline copy: the AI signal is a preliminary alert and you are the final decision-maker; you must find independent evidence in the video footage before upholding a flag. Rejection of fake flags went from 50% to 71%, matching production rates more closely, with no model or UI change.",
      "Headphone detection hid two questions in one yes/no CTA: did the model correctly detect headphones, and is this a violation? A test taker wearing a hearing aid is a true detection but should not be flagged — previously proctors clicked 'no', poisoning the model's training signal. Splitting the question yields more and better data.",
      "A generic LLM asked to act as a writing tutor returned 400 lines for a short paragraph — praise, then feedback, then an unrequested full rewrite. The Duolingo-style tutor instead marks up the text inline (green = good, yellow = awkward, red = mistake), ties every piece of feedback to a span, shows it on hover, and lets you accept suggestions in place — same AI output, different surfacing.",
      "Out-of-the-box coding agents fall into two rubber-stamp patterns: one giant multi-file diff you approve and review later on GitHub, or a notification per file/function where you click yes, yes, yes. Both produce only binary accept/reject labels skewed toward accept; an agent that plans, asks good questions, documents design decisions and breaks up PRs yields structured data about bad assumptions, tradeoffs and stylistic preferences."
    ],
    "takeaways": [
      "Before changing the model or the UI, audit the copy: tell the reviewer the AI output is a preliminary alert, that they are the final decision-maker, and require independent corroborating evidence before they can uphold it.",
      "Red-team your own human-in-the-loop by injecting known-clean cases carrying a fake AI signal into the normal review workflow, and measure the rubber-stamp rate — calibration accuracy above 90% will not predict it.",
      "Split any review control that hides two questions (was the model's perception correct? does it warrant the action?) so a correct detection that shouldn't be actioned isn't logged as a model error.",
      "Match friction to stakes: add structured review gates and deliberate speed bumps where decisions are consequential, and keep low-oversight, delightful interactions seamless.",
      "Design the interaction around the data you need before you build: treat every accept, reject, edit and follow-up question as a label, capture the diff when a human accepts then silently changes the output, and collect explicit feedback at meaningful touch points rather than a thumbs up/down."
    ],
    "topics": [
      "human-in-the-loop",
      "automation-bias",
      "interaction-design",
      "evals",
      "coding-agents",
      "trust-and-safety",
      "data-labeling",
      "proctoring"
    ],
    "tools": [
      "Duolingo",
      "Duolingo English Test",
      "Wharton",
      "Gemini",
      "Google Search",
      "CDC",
      "WHO",
      "GitHub"
    ],
    "quotes": [
      {
        "text": "When a human foregoes deliberation and adopts AI output as their own with minimal scrutiny.",
        "at": "02:20",
        "url": "https://www.youtube.com/watch?v=CDqzWpwkSls&t=140s"
      },
      {
        "text": "And this coin flip rate is something that is a strong suggestion of automation bias.",
        "at": "06:40",
        "url": "https://www.youtube.com/watch?v=CDqzWpwkSls&t=400s"
      },
      {
        "text": "Either way, in both of these cases, you're just becoming a rubber stamp.",
        "at": "16:29",
        "url": "https://www.youtube.com/watch?v=CDqzWpwkSls&t=989s"
      },
      {
        "text": "Sometimes the fix is not a better model or more oversight, it's just engineering the interaction itself.",
        "at": "25:46",
        "url": "https://www.youtube.com/watch?v=CDqzWpwkSls&t=1546s"
      }
    ],
    "words": 5043
  },
  "summary_url": "/#CDqzWpwkSls",
  "transcript": {
    "html": "/transcripts/CDqzWpwkSls.html",
    "txt": "/transcripts/CDqzWpwkSls.txt",
    "vtt": "/transcripts/CDqzWpwkSls.vtt"
  }
}