{
  "video": {
    "id": "ZSQb5fzRFPw",
    "title": "Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua",
    "duration": 1001,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "ZSQb5fzRFPw",
    "title": "Computer-Use 2.0: Agents Just Got Multi-Cursor — Francesco Bonacci, Cua",
    "one_liner": "Cua's team shows how computer-use agents move from screenshot-driven takeover of your screen to a background driver (Cua Driver) that works via accessibility trees per-window — lifting pass rate from 62% to 80% with 34% fewer tokens — plus Cua Bench for verifiable GUI evals and Cua Fleet's warm sandbox pool to stop RL GPUs idling.",
    "summary": "Francesco Bonacci frames 'computer use 1.0' as the old human-imitating loop — screenshot, reason, click/type/scroll — where the agent takes over your screen, and presents Cua Driver, open-sourced two months ago, which runs agents in the background across macOS, Windows and Linux using undocumented Apple framework APIs and accessibility trees, falling back to pixel-level background clicks. CTO Dylan then argues that giving an agent hands raises the trust question, answered by Cua Bench: tasks built from setup, oracle (a golden GUI trajectory) and evaluator functions, a catalog of 130+ verifiable tasks across 42 environments and five platforms, and a Snorkel AI collaboration testing agents on real electrical-engineering software. Results are humbling — the top agent fully passed 6 of 25 KiCad tasks, all of them edits to an existing schematic, 0% from a blank one, and no model exceeds 30% reward. Chief infra officer Robert closes on Cua Fleet: a demand-based autoscaled warm sandbox pool so RL training GPUs never sit idle waiting for a 40GB environment to boot.",
    "key_points": [
      "Computer use 1.0 is defined as the human-imitating loop: take a screenshot, reason and plan, act in a click/type/scroll action space — and the agent takes over your screen while doing it.",
      "Cua Driver, released open source ~2 months ago and hacked together over a weekend after Codex shipped its computer-use model, runs agents in the background so they never seize your cursor; it relies on undocumented APIs in the Apple framework that ships with the laptop, and spans macOS, Windows and Linux.",
      "The driver's loop is: call get_window_state for an accessibility-tree representation plus a screenshot, attempt background execution through the accessibility tree, and if that fails fall back to a pixel background click. About eight application harnesses guard against regressions between releases; early adopters named include Clicky, H Company and Droid Factory.",
      "A Cua Bench task has three pieces — a setup function that sets the machine's initial state, an oracle function giving a golden trajectory (GUI actions, unlike Terminal Bench), and an evaluator that probes the environment for success. The SDK collapses a GUI task working across 5+ desktop platforms into a single Python file, and anyone (or any agent) can author one.",
      "The catalog is 130+ verifiable tasks, 42 environments, five platforms, reproducible via their CLI. The newest dataset, built with Snorkel AI, tests agents on electrical-engineering tasks in software used by real professionals, with evaluator functions that actually simulate the circuits.",
      "Benchmark results: the top agent fully passed only 6 of 25 EE tasks; 100% of those six involved editing an existing schematic, and starting from a blank schematic the success rate is 0%. Across all models tested the leaderboard is flat — no model above 30% reward.",
      "On Cua Bench basic scaled to 4K resolution, swapping the agent's built-in computer tool for Cua Driver lifts pass rate from ~62% to 80% while using 34% fewer tokens, mainly because the driver focuses on a window rather than the entire desktop.",
      "Eval integrity is enforced by first trying to break the environment: a matrix of agents attempts reward hacking and environment breakage, the data is compiled into a CodeRabbit-style code review, and only surviving tasks enter the dataset — 'evals all the way down.'",
      "World-model measurement: any recorded run can be forked at any moment of its trajectory to recover the computer's state; the model is then probed to predict reward, internal state or another observation and scored against the fork.",
      "Cua Fleet attacks idle-GPU cost in RL training — GPUs sit unused while a sandbox spins up or resets, and a researcher's 40GB environment can't always be made to boot fast. A demand-based autoscaler detects how many GPUs currently need a sandbox and grows a warm pool to match, so warm-pool size needn't be known upfront and can shift over a multi-day run; sandboxes can be 2–4x cheaper than GPUs, so redundancy still nets savings. Instant sandboxes exist for Windows, Linux and Android, with macOS coming."
    ],
    "takeaways": [
      "If you're running a computer-use agent on a real desktop, use a per-window background driver rather than a full-desktop screenshot tool — the same agent scored 62% → 80% at 4K with 34% fewer tokens purely from the tool swap.",
      "Structure GUI evals as setup + oracle (golden GUI trajectory) + evaluator that probes environment state, and write the task once in a single Python file so it runs across desktop platforms.",
      "Adversarially attack your own eval before trusting it: run a matrix of agents that try to reward-hack or break the environment and gate dataset entry on surviving that review.",
      "Don't judge an agent only on action success — fork recorded trajectories at arbitrary points and ask the model to predict reward or internal state, comparing against the fork to make its world model measurable.",
      "For RL training on computer-use environments, pay sandbox startup on the infrastructure side: run a demand-based autoscaled warm pool instead of trying to make a 40GB environment boot fast, since sandboxes cost 2–4x less than the GPUs they keep busy."
    ],
    "topics": [
      "computer-use agents",
      "gui-agents",
      "evals",
      "benchmarks",
      "rl-training",
      "infrastructure",
      "sandboxes",
      "world-models"
    ],
    "tools": [
      "Cua",
      "Cua Driver",
      "Cua Bench",
      "Cua Fleet",
      "Terminal Bench",
      "Harbor",
      "Codex",
      "Snorkel AI",
      "KiCad (CuaBench KiCad EE dataset)",
      "CodeRabbit",
      "macOS",
      "Windows",
      "Linux",
      "Android",
      "Microsoft",
      "Arm",
      "Clicky",
      "H Company",
      "Droid Factory"
    ],
    "quotes": [
      {
        "text": "with cua driver we gave an agent hands but then the question becomes how can you trust the agent to use those hands correctly and not leave anything broken behind",
        "at": "06:37",
        "url": "https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=397s"
      },
      {
        "text": "And when we start the task from a blank schematic, the success rate drops to 0%.",
        "at": "08:48",
        "url": "https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=528s"
      },
      {
        "text": "And if you ask us how we trust that agent, the answer is that it's just evals all the way down.",
        "at": "10:06",
        "url": "https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=606s"
      },
      {
        "text": "And that prediction is the world model of the agent made measurable.",
        "at": "10:42",
        "url": "https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=642s"
      },
      {
        "text": "your GPU really isn't doing anything useful here and you know I don't know if you've heard but GPU time is pretty expensive right now.",
        "at": "12:04",
        "url": "https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=724s"
      }
    ],
    "words": 2989
  },
  "summary_url": "/#ZSQb5fzRFPw",
  "transcript": {
    "html": "/transcripts/ZSQb5fzRFPw.html",
    "txt": "/transcripts/ZSQb5fzRFPw.txt",
    "vtt": "/transcripts/ZSQb5fzRFPw.vtt"
  }
}