{
  "video": {
    "id": "2aS7aKoXn64",
    "title": "Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software",
    "duration": 1275,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "2aS7aKoXn64",
    "title": "Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software",
    "one_liner": "Two Theta Software co-founders argue that \"long horizon\" is a moving scalar, not a category, and that the hard part of building RL environments for it is the verifier — which has to be an agent with its own harness and read-only access to the environment, not an LLM call with the trajectory stuffed in its context.",
    "summary": "The talk first attacks the definition problem: long horizon can be measured against human time (METR's 50%-success thresholds, e.g. a 16-hour task) or against model units (tokens, steps, tool calls), and both are noisy and incomplete, so you need both. It then lays out what actually makes an environment measure capability — tool-coordination complexity, state changes where early decisions cascade into later ones (sequential rather than parallelizable), and deliberate ambiguity that forces exploration. The second half is about verifiers: as work moves out of hard-verifiable domains into ones where no deterministic checker exists, judge/critic models grading both final environment state and trajectory become the reward signal, which means the judge needs the same harness and environment access the agent had, plus a queryable trajectory. It closes by claiming existing finance benchmarks (GDPval, ToolBench, APEX Agents) are too short, already saturated, too narrow, and too coarse in reward signal, versus Theta's own data at 15 human-hours per task.",
    "key_points": [
      "Long horizon is a scalar, not a binary category — what counted as long horizon a year ago isn't today, so it's only useful for relative comparison between tasks.",
      "Two measurement frames: METR-style human horizon (a model 'reaching 16 hours' means 50% success on tasks that take a human 16 hours) and model-native units (tokens, steps, tool calls). Token counts are noisy across models and harnesses — Codex models are seen as more token-efficient than Claude models — but they define the technical frontier, e.g. the same model generation going from ~500k to a million-token trajectory via a bigger context window or better compaction.",
      "Human-time estimates break down at the frontier: tasks only the top 10%/1%/0.1% of humans can do produce very noisy estimates, and expert quality plus methodology differences make '16-hour tasks' from one lab incomparable with another's '20-hour tasks'.",
      "Human difficulty ≠ model difficulty: reformatting a huge Excel file might take an analyst days but a model writes a Python script, because the analyst can't write the script.",
      "Chaining unrelated independent tasks makes a task artificially long-horizon without measuring anything — what matters is that earlier decisions influence later ones through environment state changes. Parallelizable complexity (fan out sub-agents over a codebase) is weaker than sequential complexity (a bad early Grafana/log query cascading downstream).",
      "Ambiguity in the starting instructions and artifacts is deliberate — it mirrors how humans work and tests exploration — but the trade-off is that standardized evaluation gets much, much harder because there are many correct paths.",
      "Judges are agents too: for a task like 'diagnose a deployment failure from GitHub CI/CD and CloudWatch logs, patch the code, open a PR, trigger a redeploy', the judge must re-check GitHub and AWS logs itself rather than trust the agent's tool calls, reusing the agent's harness but with read-only permissions so it can't mutate the environment or kick off a deploy.",
      "You can't stuff a long trajectory into a judge's context window — the trajectory has to be made queryable: stored in a database, enriched by sub-agents, segmented into phases (log reading / code writing / verification), tagged with metadata so the judge can find failure points.",
      "The judge is the main defense against reward hacking — sandbox escapes, reading privileged info like a hidden test suite — which is why the trajectory, not just the final state, gets graded.",
      "Rubric density is a learnability trade-off: overload it and judges apply it inconsistently, especially on frontier problems models can't yet do, so QA on rubric application is needed or you waste compute on unlearnable problems.",
      "Benchmark critique: GDPval, ToolBench and APEX Agents have average human-hours per task far below current frontier model thresholds, are already reasonably saturated (APEX Agents' IB section pass@1 means ~57% of cases are 100% solved), and are narrow — GDPval a small set of Excel finance tasks, APEX largely investment banking, leaving credit, debt and risk uncovered.",
      "Theta's own finance data: 15 human-hours per task on average across a 50-task sample, rubrics with 20 criteria and 10 sub-criteria each, and models still struggling significantly across finance domains."
    ],
    "takeaways": [
      "Report both human-horizon and model-unit metrics for your tasks, and publish the methodology (expert quality, how times were measured) — a bare 'N-hour task' number isn't comparable across teams.",
      "Design environments so early tool use changes state that later steps depend on; if your long task is just independent subtasks chained together, it isn't measuring long-horizon capability.",
      "Give the judge the same harness and environment observability as the agent, with read-only permissions, and have it verify by inspecting real environment state (logs, deploys, DB) rather than the agent's self-reported tool calls.",
      "Stop grading against a single reference answer or sample trajectory — it collapses the space of valid solutions on open-ended tasks; build robust rubrics that admit multiple correct paths instead.",
      "Preprocess trajectories into something queryable (database, sub-agent enrichment, phase segmentation, step metadata) instead of dumping them into one LLM call.",
      "Keep deterministic verifiers in the loop alongside judges — e.g. have them emit metrics or artifacts for the judge to grade — and consider dynamic evaluation-time rubrics that grant partial credit by assuming an earlier wrong step was correct, like grading the rest of a test question.",
      "QA your rubrics with gold, no-op and variance tests plus coverage and expert-agreement checks — more so as AI helps author the rubrics and as tasks get longer."
    ],
    "topics": [
      "rl-environments",
      "long-horizon-agents",
      "evals",
      "llm-as-judge",
      "rubrics",
      "reward-hacking",
      "benchmarks",
      "agent-harnesses"
    ],
    "tools": [
      "Theta Software",
      "METR",
      "GDPval",
      "ToolBench",
      "APEX Agents",
      "Grafana",
      "GitHub (CI/CD)",
      "AWS CloudWatch",
      "Codex models",
      "Claude models",
      "GPT-5.5",
      "Python",
      "Excel",
      "Deep Silken (ternary models research)"
    ],
    "quotes": [
      {
        "text": "long horizon is really kind of a scalar metric. Uh, it's useful for kind of measuring relative tasks like one task might be more long than another, but it's really hard to define into kind of a binary category of this task is long and this task is not.",
        "at": "01:33",
        "url": "https://www.youtube.com/watch?v=2aS7aKoXn64&t=93s"
      },
      {
        "text": "one task can you know maybe be made by artificially long horizon by chaining together unrelated independent tasks. However, that doesn't actually tell us or meaningfully measure the model capabilities.",
        "at": "08:04",
        "url": "https://www.youtube.com/watch?v=2aS7aKoXn64&t=484s"
      },
      {
        "text": "if you have to use a dashboard or logs, a bad early query or a misread can cascade into these downstream steps that really start to have major consequences later on",
        "at": "08:55",
        "url": "https://www.youtube.com/watch?v=2aS7aKoXn64&t=535s"
      },
      {
        "text": "part of the reason we also need the judge to be an agent is that you can't just use this really basic approach of taking the trajectory and stuffing it in the context window of the judge and kind of have it be a basic LM call.",
        "at": "15:36",
        "url": "https://www.youtube.com/watch?v=2aS7aKoXn64&t=936s"
      }
    ],
    "words": 5492
  },
  "summary_url": "/#2aS7aKoXn64",
  "transcript": {
    "html": "/transcripts/2aS7aKoXn64.html",
    "txt": "/transcripts/2aS7aKoXn64.txt",
    "vtt": "/transcripts/2aS7aKoXn64.vtt"
  }
}