{
  "video": {
    "id": "GJX19pNhmSw",
    "title": "FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft",
    "duration": 1284,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer"
  },
  "analysis": {
    "video_id": "GJX19pNhmSw",
    "title": "FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft",
    "one_liner": "Two Microsoft engineers argue that agent cost control belongs at the agent-run layer, not the model-request layer, and demo \"TokenOps\" — an out-of-band control plane that steers a running agent (compaction, tool-output trimming, injected \"be succinct\" instructions) before ever killing it, benchmarked at ~78% lower average spend with completion rising from 67% to ~96% versus simple throttling.",
    "summary": "Tisha Chawla and Susheem Koul frame the industry's current \"token maxing\" culture — people proud to be \"token billionaires\" — as needing to shift to value maxing, and note that the agentic era has no real control plane where code calls the model. They argue existing tools (LiteLLM, Portkey, Cloudflare) govern at the request/model-gateway level with hard caps and model routing, which cannot control the agent-run loop, sub-agent spawning, or runaway context growth. Their proposal, TokenOps, is an out-of-band plane with instrumentation (OpenTelemetry, cost in microns, attribution), accounting (a ledger of runs), and enforcement, where in-call-path \"steer\" actions are exhausted before a budget-cap halt is used as a last resort. The demo shows preview mode, halt mode, and a \"cost guard\" that predicts budget exhaustion from consumption and velocity and injects instructions to make outputs more succinct.",
    "key_points": [
      "The control surface has moved each era: SaaS billed per UI seat with usage caps and tier policies; cloud went pay-as-you-go with autoprovisioning/autoscaling; the agentic era bills per model call but has no control plane at the point where the code calls the model.",
      "Unbounded consumption is already biting — they cite news of Uber's AI budget being exhausted within 4 months, and companies hitting hundreds of millions of dollars within months or days from runaway loops with no mechanism to stop them.",
      "First principles: the token is the unit of cost so value must also be measured in tokens; cost is created at the LLM call boundary; without attribution (which agent, which run) you cannot narrow the problem down; policies should fix the condition in place, and a hard halt from a budget cap is only the last resort.",
      "Existing tools — LiteLLM, Portkey, Cloudflare — do halting and model routing at the request layer; the missing piece is governance at the agent-run layer, covering the agent↔tool loop, sub-agent spawning from a main agent, and growing context.",
      "TokenOps architecture is deliberately an out-of-band plane that doesn't touch your code: instrumentation (OpenTelemetry, cost in microns, enrichment, attribution) → accounting (a ledger of runs) → enforcement (steer, then halt).",
      "Three layers: your agent runtime, a bridge, and a control plane that runs in your own tenant. The bridge's heart is a `boundary` annotation you put on any method regardless of framework (LangChain or whatever) — it flights inputs/outputs up as ledger entries AND acts as a downward channel for control-plane actions; `wrap_complete` applies the same to LLM objects rather than methods. A `governor` node, configured by the developer, declares which actions are allowed and applies them non-destructively so the control plane can't do random things to your agent.",
      "Control plane primitives: segments (cohorts built from attribution dimensions, e.g. tag `cohort = AIE 2026`, so budgets can be applied at cohort, agent or run level), ledger, budgets (static thresholds over a time window), actions (halt: kill; steer: allow, mutate, inject), and policies that group budgets and actions against segments or runs.",
      "Demo on a two-agent research→summarizer workflow showed three scenarios: preview mode (policies execute, enforcement off — so you can test and tune thresholds in production safely), governance on (pre-call cost cap exceeded, agent killed — a circuit breaker), and steer, where \"cost guard\" reads how much of the budget is consumed plus the velocity of consumption, predicts overrun, and injects into the system instructions to make LLM outputs more succinct.",
      "Benchmarked across multiple iterations, stress tests, and simple/hard scenarios on the open-source repos browser-use and MetaGPT: average spend down ~78% with the full policy suite, and completion up from 67% (simple throttling, which kills agent runs regardless) to roughly 96%.",
      "The policy catalog covers researched real-world failure modes: spend management, context management (context compaction, tool output reduction), loop detection and progress detection.",
      "Envisioned end state is a self-learning module inside the control plane that reads the ledger, asks which failure modes it is still not catching, then generates new policies on the fly or refines parameters of existing ones."
    ],
    "takeaways": [
      "Stop assuming a model gateway is cost control — hard caps and model routing at the request layer can't see the agent loop, sub-agent fan-out, or context growth that actually produce runaway bills.",
      "Instrument at the run level with attribution on every call, so an unexplained bill can be traced to a specific agent and run rather than just a broad number.",
      "Order your enforcement: exhaust in-place remedies (compaction, caching, tool-output reduction, injected succinctness instructions) before letting a budget cap kill the run — killing is the crude option and simple throttling tanks completion rates.",
      "Roll cost governance into production in preview mode first: let policies execute without enforcement so you can watch what they'd do and tune thresholds before they can break a live agent.",
      "Attach budgets to cohorts/segments derived from attribution dimensions, not only to individual agents or runs, so a shared preview agent given to a whole room of users can be capped as a group.",
      "Constrain what the control plane may do to your agent via an explicit governor config of allowed actions, so remote steering stays non-destructive."
    ],
    "topics": [
      "agents",
      "finops",
      "cost-control",
      "observability",
      "governance",
      "tokens",
      "context-management",
      "benchmarks"
    ],
    "tools": [
      "TokenOps",
      "LiteLLM",
      "Portkey",
      "Cloudflare",
      "OpenTelemetry",
      "LangChain",
      "browser-use",
      "MetaGPT",
      "Microsoft",
      "Uber"
    ],
    "quotes": [
      {
        "text": "people are proud to call themselves token billionaires and um I think that's all right but this talk is you know the shift from token maxing to value maxing",
        "at": "00:55",
        "url": "https://www.youtube.com/watch?v=GJX19pNhmSw&t=55s"
      },
      {
        "text": "if we don't have proper attribution like if we don't know what agent want run made that particular call we we can't you know control it",
        "at": "04:37",
        "url": "https://www.youtube.com/watch?v=GJX19pNhmSw&t=277s"
      },
      {
        "text": "We do not have a single directional highway. We want the control plane to be able to tweak the behavior of the agent on the fly to ensure that we are able to squeeze in more runs inside our budget cap.",
        "at": "12:41",
        "url": "https://www.youtube.com/watch?v=GJX19pNhmSw&t=761s"
      },
      {
        "text": "the average spend goes down by almost 78% with token ops enabled with the full policy suit that we have today... you get an uplift in that completion percentage from 67% to roughly 96%",
        "at": "19:01",
        "url": "https://www.youtube.com/watch?v=GJX19pNhmSw&t=1141s"
      }
    ],
    "words": 4240
  },
  "summary_url": "/#GJX19pNhmSw",
  "transcript": {
    "html": "/transcripts/GJX19pNhmSw.html",
    "txt": "/transcripts/GJX19pNhmSw.txt",
    "vtt": "/transcripts/GJX19pNhmSw.vtt"
  }
}