{
  "video": {
    "id": "ZGzthw4BZiA",
    "title": "Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab",
    "duration": 3343,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "ZGzthw4BZiA",
    "title": "Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab",
    "one_liner": "Amazon AGI Lab argues that computer-use agents also need perception, meaning they understand a screen the way a human does. They open-sourced two primitives that close the front-end feedback loop: a visual annotator Chrome extension and a Gherkin-driven visual verification skill. Wired together through an agent bridge, they let feedback go from annotation to code changes to a verification report with no supervision.",
    "summary": "Emile Baizel introduces Amazon AGI Lab's approach to agents, built on collaboration, perception and reliability, and its first product, Nova Act, an AWS service for reliable browser workflows at scale. He then argues for 'perception agents' that understand what is on screen rather than just clicking on it. These agents work without APIs and let humans point, click and describe changes. Shruti Arora runs a hands-on workshop that wires the two open-sourced primitives, visual annotation and visual verification, into one loop: annotate a podcast landing page, click 'Apply changes', and a coding agent edits the code, after which Nova Act runs UI verification and produces a report. Giovanni shows voice-driven design feedback from Amazon's Bee wearable feeding the same pipeline, and attendees demo their own integrations.",
    "key_points": [
      "Amazon AGI Lab's agent approach rests on three pillars: collaboration (agents as teammates who share how they are thinking, not black boxes), perception (any visual screen is an environment: browsers first, then mobile and computer use) and reliability (heavy RL training in simulated environments based on real customer and internal Amazon use cases).",
      "Nova Act is a first-class AWS service for building browser workflows in natural language. It runs on a custom computer-use model, and the AWS console shows the agent's full trajectory with bounding boxes. This year it gained an MCP server and a skill, plus human-in-the-loop, tool use and a no-code playground. The four main customer use cases are form filling, search and extract, shopping, and QA.",
      "Perception agents mean understanding what is on the screen, not just clicking on it, the way a human could work out how to send an email in an unfamiliar late-'90s Hotmail client. They need no APIs, and because they see screens as humans do, people can point and click instead of writing paragraphs describing the screen.",
      "Primitive 1, visual annotation, is a Chrome extension with three modes: draw on the page, pick an element, or drop a pin. It exports a JSON file of changes plus an AI-friendly prompt for a coding agent. It is purely deterministic and not tied to any model.",
      "Primitive 2, visual verification, is a skill (for Claude Code, Kiro and others) that checks a site against design specs and user flows written as Gherkin files. On first run it does a one-level surface crawl of the site to generate those Gherkin flows, then opens a browser, runs the flows and produces a report.",
      "In the workshop, an 'agent bridge' connects the extension to the coding agent. You add localhost host_permissions to manifest.json and paste code into content.js around line 261 to add an 'Apply changes' button, which runs UI verification through the Nova Act SDK. The setup has been tested on Claude Code and Kiro; Cursor and Codex need edits to the bridge.",
      "Giovanni's Bee integration: the Bee wearable records a design conversation, the Bee CLI (installed via npm) fetches the transcript and key points, and a bridge applies them through the perception agent pipeline, followed by automatic Nova Act verification. A real conversation takes about 2-3 minutes to process.",
      "Attendee demos: a 'duck podcast' retro NES restyle whose verification report confirmed the change was actually made; Bee conversation summaries turned into a Jira backlog via the Atlassian MCP; an internal GTD tool called Sidekick fed by Bee; and Jam MCP recordings (console logs, network requests) piped through the proxy to Nova for UI changes."
    ],
    "takeaways": [
      "Feed reviewer feedback to your coding agent as structured visual annotations (element selection plus JSON export plus prompt) instead of Slack messages or shared docs. This can shrink the feedback-incorporation loop from days to hours.",
      "Encode design specs and critical user flows (checkout, login) as Gherkin files, then have your coding agent run visual verification against them after every change. That way 'done' is checked by an agent looking at the rendered page, not just claimed.",
      "Use reliability as the bar for agents: 'If my agent works 50% of the time, it doesn't work at all.' Unreliable agents just shift the work into debugging.",
      "Use perception-based browser agents like Nova Act for work on sites without APIs, such as form filling, search and extract, or QA across many sites, instead of writing a prompt or deterministic script for each site.",
      "Treat feedback capture as pluggable: the same bridge-and-proxy pattern can take input from voice wearables (Bee), screen recordings (Jam MCP) or other sources."
    ],
    "topics": [
      "perception agents",
      "computer use",
      "browser automation",
      "visual verification",
      "coding agents",
      "front-end development",
      "human-agent collaboration",
      "mcp"
    ],
    "tools": [
      "Amazon Nova Act",
      "Nova Act SDK",
      "Nova Act MCP server",
      "Nova Act UI verification skill",
      "Visual annotator Chrome extension",
      "Agent bridge",
      "AWS",
      "Amazon AGI Lab",
      "Claude Code",
      "Kiro",
      "Codex",
      "Cursor",
      "Bee (b.computer)",
      "Bee CLI",
      "Gherkin",
      "Figma",
      "Chrome",
      "Node.js",
      "npm",
      "Python",
      "uv/uvx",
      "pip",
      "GitHub",
      "GitLab",
      "Slack",
      "Atlassian MCP",
      "Jira",
      "Linear",
      "Jam MCP",
      "Sidekick"
    ],
    "quotes": [
      {
        "text": "If my agent works 50% of the time, it doesn't work at all.",
        "at": "03:02",
        "url": "https://www.youtube.com/watch?v=ZGzthw4BZiA&t=182s"
      },
      {
        "text": "this has nothing to do with inference. This is purely deterministic.",
        "at": "08:17",
        "url": "https://www.youtube.com/watch?v=ZGzthw4BZiA&t=497s"
      },
      {
        "text": "collect the feedbacks, send it to your agent, and go for a coffee break, come back, and all the changes have been merged.",
        "at": "12:41",
        "url": "https://www.youtube.com/watch?v=ZGzthw4BZiA&t=761s"
      },
      {
        "text": "You know, I'm now 0 for 11 for live demos.",
        "at": "45:11",
        "url": "https://www.youtube.com/watch?v=ZGzthw4BZiA&t=2711s"
      }
    ],
    "words": 9398
  },
  "summary_url": "/#ZGzthw4BZiA",
  "transcript": {
    "html": "/transcripts/ZGzthw4BZiA.html",
    "txt": "/transcripts/ZGzthw4BZiA.txt",
    "vtt": "/transcripts/ZGzthw4BZiA.vtt"
  }
}