{
  "video": {
    "id": "tyFPqtyOK5w",
    "title": "The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular",
    "duration": 4379,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "tyFPqtyOK5w",
    "title": "The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular",
    "one_liner": "Simular argues that reliable computer-use agents have to be neurosymbolic: a vision-language agent works out a task once, writes down what it did as deterministic code (their open-source Simulan), and replays that code next time, falling back to the VLM only when the environment changes, so repeat runs become cheap and repeatable.",
    "summary": "Ang Li splits the 'autonomous computer' into two problems: autonomy and the computer it runs on. For autonomy, he traces Simular's Agent S line on OSWorld: the first version used the accessibility tree and scored 20% in late 2024, the second added a manager/worker setup with a 7B open-source grounding model, and the third used test-time scaling to beat human performance in December. He then argues that benchmark success is not reliability. Agents degrade on pass^k, meaning the same task must succeed on every one of k runs, so Simular combines the adaptive VLM agent with the deterministic but brittle code layer through record, replay and a self-healing loop. For the computer side, he presents cloud desktops (Sidebox, with a newly released API) across Windows, macOS, and coming-soon Linux and Android, so that one person can run many agents in parallel.",
    "key_points": [
      "Agent S history on OSWorld: the first version (late 2024) used accessibility trees with an LLM and ID-based grounding and reached 20%, when the benchmark had started early 2024 at about 5%; it won best paper at the ICLR 2025 agentic AI workshop. The second version (early 2025) was vision-only with manager/worker layers, and showed that two foundation models, one of them a 7B open-source model, beat one. The third version (late 2025) used test-time scaling with many workers and broke the human-performance record in December, before Anthropic and OpenAI reached it in March.",
      "OSWorld 2 was just released, and its best model scores only about 20%, which Li calls the second phase of computer-use agents.",
      "Pass^k (the run must succeed every one of k times) is what customers actually care about, and no benchmark captures it. Running the top OSWorld agents three times dropped success by almost 10%, and it got worse when the query or environment was perturbed.",
      "A simple reliability trick from their paper: have the agent ask clarification questions about ambiguous parts of the instruction. The prompt asks for minimal clarification while 'still preserving the nature' of the task. It improved Claude, and Agent S with GPT-5 gained the most.",
      "Simulan is an open-source, JavaScript-based scripting language for computer use, positioned like Playwright for desktops and filling a gap left by RPA, which has no shared standard. It installs with npm and ships a Claude Code skill. In the demo, Claude Code wrote Simulan code, hit errors and revised the code itself, and chose an accessibility snapshot over a screenshot for a summarization task.",
      "Neurosymbolic mode: the VLM agent writes Simulan code line by line as it works, and a successful run can be saved as a skill. Later runs replay the code with changed parameters (Tuesday changed to Wednesday in the demo), verify as they go, and drop to the VLM agent when something breaks. Li frames this as the stability–plasticity dilemma and a search for the Pareto frontier.",
      "Repeat runs can be about 50% code with no LLM calls, which is why Simular is experimenting with a $500 unlimited-task plan instead of usage-based pricing. The stated goal is for token consumption on repeated tasks to eventually reach zero.",
      "Infrastructure: Sai is the interface, Agent S provides the autonomy, and Sidebox is the cloud computer with a SAPI API and CLI released that day. Windows and macOS are supported; Linux and Android are coming soon. Android demos showed Angry Birds dragging and an authenticator app solving a 2FA challenge. There is also a Microsoft Windows 365 for Agents collaboration filling forms in a legacy ERP."
    ],
    "takeaways": [
      "Measure pass^k, not just pass@k, for any workflow that has to run unattended every day.",
      "Have agents ask clarifying questions about ambiguous instructions before acting; it measurably improves reliability.",
      "Turn successful agent runs into deterministic code and run that code first, falling back to the VLM and a healing loop only when the code fails. This cuts both variance and token cost.",
      "For accuracy-critical work such as healthcare data entry, add verification loops (have the model check the filled form and redo the task if it is wrong), rely on accessibility-tree text matching for deterministic replay, and put guardrails in code first, then an agent, then a human, before any irreversible action.",
      "Use several input signals (screenshots, accessibility tree, DOM), the 'Waymo approach' rather than vision only. For legacy software where exploring costs nothing, let agents explore freely to map out successful trajectories."
    ],
    "topics": [
      "computer-use agents",
      "neurosymbolic ai",
      "reliability",
      "benchmarks",
      "agent infrastructure",
      "cloud desktops",
      "rpa",
      "test-time scaling"
    ],
    "tools": [
      "Simular",
      "Agent S",
      "Simulan",
      "Sai",
      "Sidebox",
      "SAPI",
      "OSWorld",
      "OSWorld 2",
      "GPT-4V",
      "GPT-5",
      "Claude",
      "Claude Code",
      "Codex",
      "OpenAI",
      "Anthropic",
      "DeepMind",
      "Microsoft Windows 365 for Agents",
      "Playwright",
      "GLM",
      "OpenClaw",
      "npm",
      "VS Code",
      "FaceTime",
      "LinkedIn",
      "Cal Hacks",
      "Waymo",
      "Tesla"
    ],
    "quotes": [
      {
        "text": "you don't have to spend the same amount of tokens each time anymore",
        "at": "42:12",
        "url": "https://www.youtube.com/watch?v=tyFPqtyOK5w&t=2532s"
      },
      {
        "text": "verification is is much easier than generation",
        "at": "59:01",
        "url": "https://www.youtube.com/watch?v=tyFPqtyOK5w&t=3541s"
      },
      {
        "text": "we want the token consumption to eventually become zero",
        "at": "63:12",
        "url": "https://www.youtube.com/watch?v=tyFPqtyOK5w&t=3792s"
      },
      {
        "text": "LinkedIn is is the Turing test for AGI.",
        "at": "56:34",
        "url": "https://www.youtube.com/watch?v=tyFPqtyOK5w&t=3394s"
      }
    ],
    "words": 12789
  },
  "summary_url": "/#tyFPqtyOK5w",
  "transcript": {
    "html": "/transcripts/tyFPqtyOK5w.html",
    "txt": "/transcripts/tyFPqtyOK5w.txt",
    "vtt": "/transcripts/tyFPqtyOK5w.vtt"
  }
}