{
  "video": {
    "id": "aGIUg8zKI8A",
    "title": "How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver",
    "duration": 1403,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "aGIUg8zKI8A",
    "title": "How a Trading Firm Ships New AI Models the Same Day - Noyan Tokgozoglu, Optiver",
    "one_liner": "Optiver, a market maker that sees the P&L effect of a change within hours, built an on-prem AI platform that doesn't depend on any one model or harness. All traffic goes through its own inference gateway and every new model is benchmarked on internal golden sets, so a new frontier model can be used in production flows the day it is released.",
    "summary": "Optiver's head of AI engineering says trading changes how AI decisions get made. The balance sheet is zeroed at the start of each day, so the P&L effect of a change shows up within hours, and a missed opportunity is expensive. Optiver built its own modular, self-service AI platform around a few principles: move fast, measure at every layer, give traders good defaults plus forward-deployed engineers, stay safe, and run on-prem. The platform has three main parts: an inference gateway, an agent platform with an agent gateway, and its own user surfaces such as a web portal, an agent workbench and a personal assistant. The talk also covers how they brought AI spend back under control after it spiked with the summer frontier-model releases, and how they built polished versions of the hacky tools traders had already made for themselves.",
    "key_points": [
      "Scale: 10M+ trades a day, 1M+ instruments priced and petabytes of data processed daily, across 10+ locations. The balance sheet is zeroed at the top of each day, so P&L impact shows within hours, not over the months or quarters a typical tech company waits.",
      "Same-day model adoption: when a new frontier model is released publicly, they run it on golden-set benchmarks collected from key trading teams, because their data distribution differs from external benchmarks. They make it available in the gateway with APIs and cost management, update the harnesses, share a recommendation internally, and support it with round-the-clock regional on-call rotations.",
      "Everything runs on-prem (own GPUs and infrastructure) because the industry is regulated and they value their IP. The only outside calls are third-party model inference under zero-data-retention agreements, and some workloads run in a fully isolated DMZ on-prem zone.",
      "Architecture: third-party, self-hosted and open-source models sit behind an inference gateway that handles routing, per-model spend tracking, budgets and provider failover. Every interaction is logged and fed automatically into the eval system. A 'meta harness' layer sits over several supported harnesses: teams define their own harness configs, which plug into a team knowledge base that is harvested automatically from traces. An agent gateway acts as a single ledger where agents and humans communicate and orchestrate with shared identity management.",
      "Cost: fresh sessions were starting with 200k+ tokens of context because people had loaded lots of MCPs and knowledge into them. Fixes were MCP tool search, CLIs that wrap MCPs (much more efficient), tuning cache invalidation and TTL parameters, moving material out of huge system prompts into knowledge bases, and per-team harness optimisation. These savings scale with token price.",
      "Giving teams visibility of their spend, alerting, and the ability to set their own limits worked: many teams set reasonable limits for themselves. Other spend levers: open-source models that are competitive on token cost for some use cases, routing prompts to cheaper models, and having a frontier model write the plan while smaller models implement it.",
      "Traders had built bespoke harness setups and hacky web UIs, so the platform team built a proper web portal with sessions, chat, artifacts and collaborative document editing. Traders now send PRs to it. It is growing into 'agent multiplayer' rooms that mix people's own agents with deployed agents, plus a Slack client.",
      "Q&A: their knowledge base is hybrid (markdown plus vector search) and the speaker is not happy with it; he wants to look at graph databases. Replaying agent evals is hard because tools depend on world state at the time. They snapshot that state where they can (trading backtest databases are easy to point-in-time; Slack is hard), give each tool call a confidence score for how well it can be replayed, and put a proxy around MCPs. Rather than merging agents into one bloated agent, they scope agents narrowly and version them, and let them call each other through the agent gateway."
    ],
    "takeaways": [
      "Build internal golden-set benchmarks from your real users' workloads, and keep the gateway, cost controls and harness updates ready so a new model can be evaluated and shipped the day it is released.",
      "Route all model traffic through your own gateway and stay independent of any one model or harness: it gives you failover and spend tracking, and you avoid moving 'at the speed that they let us move' when you depend on one vendor.",
      "Audit the context a fresh session starts with. Use MCP tool search, wrap MCPs as CLIs, tune caching, and move bulky system prompts into retrievable knowledge bases. Show teams what they spend and let them set their own limits.",
      "When high-agency users build hacky tools of their own, build a better shared version and accept their PRs.",
      "Keep agents narrowly scoped, versioned and composable through an agent registry or gateway rather than adding features to one agent, so each stays measurable."
    ],
    "topics": [
      "ai-platform",
      "inference-gateway",
      "evals",
      "cost-optimization",
      "agents",
      "on-prem",
      "knowledge-bases",
      "trading"
    ],
    "tools": [
      "Claude Code",
      "Codex",
      "MCP",
      "Slack",
      "OpenTelemetry (transcribed as 'hotel logging')",
      "Optiver",
      "Apple",
      "Spotify"
    ],
    "quotes": [
      {
        "text": "we roll out that change to our traders and our our quants, and literally within hours you can see the P&L impact",
        "at": "01:29",
        "url": "https://www.youtube.com/watch?v=aGIUg8zKI8A&t=89s"
      },
      {
        "text": "if a new frontier model is made available publicly, uh, we want to be able to leverage it in production flows the same day",
        "at": "07:11",
        "url": "https://www.youtube.com/watch?v=aGIUg8zKI8A&t=431s"
      },
      {
        "text": "sometimes people have like 200k tokens worth of context like by default minimum, sometimes even more. That's That's ridiculous, right?",
        "at": "10:51",
        "url": "https://www.youtube.com/watch?v=aGIUg8zKI8A&t=651s"
      },
      {
        "text": "if our infrastructure is overly invested in one particular provider, we move at the speed that they let us move",
        "at": "18:08",
        "url": "https://www.youtube.com/watch?v=aGIUg8zKI8A&t=1088s"
      }
    ],
    "words": 5415
  },
  "summary_url": "/#aGIUg8zKI8A",
  "transcript": {
    "html": "/transcripts/aGIUg8zKI8A.html",
    "txt": "/transcripts/aGIUg8zKI8A.txt",
    "vtt": "/transcripts/aGIUg8zKI8A.vtt"
  }
}