{
  "video": {
    "id": "CTLa_p6iOiY",
    "title": "Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma Labs",
    "duration": 1048,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "CTLa_p6iOiY",
    "title": "Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma Labs",
    "one_liner": "A \"replay agent\" — a sub-megabyte script that blindly replays recorded action sequences — matches or beats the frontier model it was extracted from on OSWorld and MobileWorld, which is the clearest proof that computer-use benchmarks need randomized, verified environments and honest confidence intervals.",
    "summary": "Pierluca D'Oro (Programma Labs, work done at Meta Superintelligence Labs) shows that because most computer-use benchmarks are static and deterministic, you can compile one successful frontier-model trace per task into a blind replay script that scores the same or better than the model itself — and proves formally in the accompanying paper that pass@k on a deterministic environment is literally measuring that replay agent's success rate. He proposes fixing both halves of the problem: environments built to the \"PRISM\" principles (multifactorial variation, verified-valid combinations, sandboxing, verifiers/privileged information, realism), demonstrated in a mobile benchmark of 15 Android apps, 387 verified scenarios and 3.2 million verified configurations generated and validated by a compiler-like pipeline; and metrics that respect the benchmark's hierarchy so confidence intervals actually cover. The practical stake is money: overconfident intervals lead to deploying the wrong model, which at a million tasks and a 4% real performance gap costs hundreds of thousands of dollars a month.",
    "key_points": [
      "A \"replay agent\" is built by running a frontier model on a benchmark, keeping one successful trajectory per task, and compiling those tap/type/scroll sequences into a script — under a megabyte for a benchmark with hundreds of tasks — that replays blindly; on OSWorld and MobileWorld its success rate is the same as or better than the frontier model it came from.",
      "The paper proves formally that on a deterministic environment, pass@k is equivalent to evaluating the success rate of that replay agent — \"pass at K is sort of a metrification of that exploit.\"",
      "The two failure classes are environments with exploitable structure and metrics built on fragile statistics; the PRISM design principles answer the first — multifactorial variation (data, appearance, initial state), a system that checks every generated combination is valid, sandboxing, support for verifiers with privileged information, and faithful realism. Existing benchmarks each hit some boxes; none hit all.",
      "The benchmark they built is 15 Android apps across domains, 387 verified scenarios, and 3.2 million verified configurations, varying the instance (e.g. the exact amount of money to send), the data profile (which contacts or emails exist), the theme, and the starting screen (login page vs. another valid page).",
      "Coding agents can mass-produce the variations, but \"a lot of software is not the same as an effective environment\" — the scaling key is a rejection-based verification strategy: generate all combinations, throw away the broken ones, keep the valid ones. They built a compiler that takes a parameterized task template, its matching verifier, and mock data, plus a base data and UI state, and emits a valid configuration.",
      "On the resulting benchmark the replay agent gets only a little performance — roughly what you'd want, since some tasks are repeatable by nature.",
      "The same variation axes let you measure robustness: in the worst case frontier models are \"pretty bad\" at being robust to changes that shouldn't matter, like which screen the task starts from or the app's theme.",
      "Uncertainty has two sources — stochastic actions across rollouts and variability from the environment itself — and only the second reflects what you meet in the real world. Using rollouts on the base case alone, a nominal 95% confidence interval achieves only about 17–20% coverage in realistic cases; computing it hierarchically restores 90–95%.",
      "Worked cost example: tiny-but-overconfident intervals make you pick model B over model A; with a million tasks, a real 4% performance gap and a mistake costing ~$12–20 on average, that's hundreds of thousands of dollars in a single month. A reliable interval instead tells you \"I'm not confident enough to make an informed decision,\" so you can buy more evaluation."
    ],
    "takeaways": [
      "Build the replay agent for your own benchmark as a diagnostic: compile one successful trace per task into a blind script and score it. If it matches your frontier model, your benchmark is deterministic and gameable.",
      "Stop reporting pass@k on deterministic computer-use environments — on a static benchmark it measures replayability, not capability.",
      "Randomize initial state across runs, not just data — varying the starting screen is rare in existing benchmarks and is one of the axes frontier models are least robust to. Pair every generated variation with a verifier that rejects invalid combinations; diversity without validity is just more software.",
      "Compute confidence intervals that respect the benchmark's hierarchy (variation across configurations as well as across rollouts), and treat a \"not confident enough to decide\" result as a signal to spend more on evaluation rather than shipping the decision.",
      "Price the eval against the deployment: multiply your task volume by the plausible performance gap and the per-mistake cost before deciding that a cheap, narrow-looking eval is good enough."
    ],
    "topics": [
      "evals",
      "benchmarks",
      "computer-use",
      "agents",
      "statistics",
      "robustness",
      "mobile-agents",
      "environment-design"
    ],
    "tools": [
      "Programma Labs",
      "Meta Superintelligence Labs",
      "OSWorld",
      "MobileWorld",
      "DigiWorld (rendered \"DG World\" in the transcript)",
      "Android",
      "coding agents",
      "pass@k"
    ],
    "quotes": [
      {
        "text": "This is a kind of a weird but maybe trivial phenomenon, but I would argue that we shouldn't accept this kind of blind scripts beating the frontier models.",
        "at": "01:52",
        "url": "https://www.youtube.com/watch?v=CTLa_p6iOiY&t=112s"
      },
      {
        "text": "In other words, pass at K is sort of a metrification of that exploit of the replay agent.",
        "at": "03:06",
        "url": "https://www.youtube.com/watch?v=CTLa_p6iOiY&t=186s"
      },
      {
        "text": "Coding agents can generate a lot of software, but a lot of software is not the same as an effective environment.",
        "at": "07:38",
        "url": "https://www.youtube.com/watch?v=CTLa_p6iOiY&t=458s"
      },
      {
        "text": "A non-rigorous benchmark is misleading.",
        "at": "16:12",
        "url": "https://www.youtube.com/watch?v=CTLa_p6iOiY&t=972s"
      }
    ],
    "words": 3266
  },
  "summary_url": "/#CTLa_p6iOiY",
  "transcript": {
    "html": "/transcripts/CTLa_p6iOiY.html",
    "txt": "/transcripts/CTLa_p6iOiY.txt",
    "vtt": "/transcripts/CTLa_p6iOiY.vtt"
  }
}