{
  "video": {
    "id": "2JX6JYyQG4Y",
    "title": "Perception Agents — Antje Barth, Amazon AGI Lab",
    "duration": 1305,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "2JX6JYyQG4Y",
    "title": "Perception Agents — Antje Barth, Amazon AGI Lab",
    "one_liner": "Amazon AGI Lab's Antje Barth argues that agents have solved capability but not reliability — because reliability only arrived where answers are verifiable (code) — and introduces \"perception agents\" that read the rendered screen to check their own work, with two open-source pieces launched: an annotation Chrome extension and a design-spec verification tool.",
    "summary": "Barth's framing is that clicking turned out to be the easy part: agents can perform each individual step of a workflow like onboarding a new hire, but the real work lives \"in the seams\" between five different systems and nobody owns it end to end. Coding made the jump from capable to reliable to trusted because code is verifiable — you can run it, test it, check it — while most knowledge work has no unit test for \"is the design on brand?\" or \"did it get what I meant?\". Her proposed answer borrows the robotics perceive→plan→act loop for screens: an agent that takes in what's actually rendered (not the code behind the page), reacts in real time instead of in prompt-and-wait turns, and confirms its own output. She demos the two open-source pieces of Amazon AGI Lab's perception agent harness shipped so far — annotation and verification — plus an experiment where a wearable meeting transcript feeds design changes straight into the agent.",
    "key_points": [
      "The onboarding example: a new team member starts Monday, and setting up accounts, adding them to Slack, booking intros and ordering laptops touches five different systems that nobody owns end to end — agents can do each step but not the whole thing.",
      "Reliability gut check: end-to-end agent workflows succeed maybe 60–80% of the time, which \"sounds really fine\" until you notice that an agent that deletes a database one in four times is one you will never touch again; you need reliability \"in the nines\".",
      "Coding was solved first — autocomplete → writing functions → agents opening their own pull requests — because code is verifiable, so reliability showed up first where you can check the answer; nobody now reads every generated line.",
      "Verification hits a wall for knowledge work: there is no unit test for \"did the report land\", \"is the design on brand\", \"did it get what I actually meant\" — a field she calls still wide open.",
      "Humans handle messy work by grabbing a colleague and looking at the same screen; the missing ingredient for agents is not a bigger brain but shared context.",
      "Three reasons perception agents matter: (1) they close the loop on computer use by reading the rendered screen to confirm their own output instead of firing actions and hoping; (2) they need no API or backend, which matters because most everyday software exposes no APIs at all; (3) pointing at an element is a more precise, less lossy input signal than a long written description.",
      "Annotation demo (a Chrome extension): draw around a heading or hover to auto-select the right element, say \"change it to red\" or \"double the font size\", and it captures the feedback, location and style elements into a complete summary to hand to the agent — no back and forth.",
      "Verification demo: design rules live in a design MD file (generated from your colors, components and layout if you haven't written them), then the agent runs a visual check for on-brand/layout plus an automated user-flow walkthrough — adding and deleting a task like a real user — and writes a report calling out what passed and what didn't.",
      "Off-screen perception experiment: Barth and colleague Giovanni wore B devices (sponsored) in a meeting booth at the conference; the meeting transcript produced a summary and captured insights with an \"apply\" button that sent the changes (yellow background, red heading, emoji swap) straight to the agent and kicked off verification, which passed because yellow was in the approved rules."
    ],
    "takeaways": [
      "Judge agents on end-to-end reliability, not capability — a 60–80% success rate is not a product, and the bar for unattended work is \"in the nines\".",
      "Manufacture verifiability where it doesn't exist: write your design/brand rules into a design MD file and have the agent check its own rendered output and walk the user flows, so nobody is clicking through the app at midnight.",
      "Replace long written change descriptions with annotation on the rendered page — mark the element and state the change, capturing location and style as a precise signal the agent can act on exactly.",
      "Build against the rendered interface rather than assuming an API exists, since most software people use daily exposes none.",
      "Go try the two open-source pieces on Amazon AGI Lab's GitHub and report what's missing — the harness is explicitly a first step being built in the open."
    ],
    "topics": [
      "agents",
      "computer-use",
      "reliability",
      "verification",
      "human-agent-interaction",
      "perception",
      "open-source",
      "browser-automation"
    ],
    "tools": [
      "Amazon AGI Lab",
      "Chrome extension",
      "B (wearable meeting-transcription devices)",
      "GitHub",
      "Slack",
      "Zoom",
      "design.md / design MD design-spec file"
    ],
    "quotes": [
      {
        "text": "But what we figured out, click clicking was actually the easy part.",
        "at": "01:06",
        "url": "https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=66s"
      },
      {
        "text": "The agent can use every single tool you give it, but it still can't do the full work.",
        "at": "02:17",
        "url": "https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=137s"
      },
      {
        "text": "if your agent one in four times deletes a database, you will never touch that agent again, right?",
        "at": "04:09",
        "url": "https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=249s"
      },
      {
        "text": "You don't necessarily need a bigger brain. What you need is this shared context.",
        "at": "08:35",
        "url": "https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=515s"
      }
    ],
    "words": 3320
  },
  "summary_url": "/#2JX6JYyQG4Y",
  "transcript": {
    "html": "/transcripts/2JX6JYyQG4Y.html",
    "txt": "/transcripts/2JX6JYyQG4Y.txt",
    "vtt": "/transcripts/2JX6JYyQG4Y.vtt"
  }
}