{
  "video": {
    "id": "YIhvrvHtonk",
    "title": "We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI",
    "duration": 1206,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "YIhvrvHtonk",
    "title": "We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI",
    "one_liner": "After two years of rebuilding Arize's in-product agent Alex, nearly everything that survived was harness code rather than model choice: move the plan, the stopping rule and the memory into the harness, and keep them simple. The long-term memory, for example, is one markdown file per user, capped at 8,000 characters.",
    "summary": "Fuad Ali, an AI PM at Arize, traces Alex from a late-2023 prototype to today. In late 2023 it was a router over skills with about 50% hallucination. It is now a single agent that loads skills on demand and has long-term memory. He argues that 'we didn't train these models. Everything we rebuilt was harness code'. Arize uses no agent framework, so the loop, plan, stopping rule and memory are all their own. He walks through the hard decisions (who owns the plan, what ends a turn, how to handle oversized context, one agent or many, which default model) and shows how they debug Alex with Alex and with Signal, a trace-reading agent. Production sessions become their regression tests.",
    "key_points": [
      "Timeline: prototype in late 2023 (about 50% hallucination); maths moved out of the model and into harness code early on; launched as a 'copilot' that was a router over skills, a decision tree with rails; rewritten in August 2025 with iterative tool calling and a to-do tool; Alex 2.0 relaunched in February 2026 with 'Sonar 4.5' (as transcribed) as the default model; 'One Alex' in June merged the specialists into one agent that loads skills on demand; then long-term memory.",
      "Planning: when the model was given a to-do list it spent most of its LLM calls checking items off, lost focus after several tool calls, and the to-do tools took 20+ iterations. They moved the plan into the harness step by step. First they added an 'in progress' marker. Then the harness rendered the plan as a message on every call, with the rules next to the data. Then they dropped 'in progress', because the first unfinished item in ranked order is the next task anyway. They are now testing with no to-do tool at all: the harness holds the state and checks completion.",
      "Ending a turn: treating 'text with no tool calls' as done broke on models that announce a step in one turn and make the call in the next. That matters because customers bring their own provider. The fix is an explicit finish call that bounces back up to 3 times while plan items are open, a warning one step before the iteration limit, and a forced finish.",
      "Context and memory, explained with a 'books on your desk' analogy: treat a huge tool result like a file. Truncate inside large JSON trace blobs, leave a pointer (a 'sticky note'), and let the agent use jq to fetch the full content when it needs it. When the agent's own trajectory gets too long, the harness makes the model write a checkpoint and drops what the checkpoint replaced. History has a fixed token budget.",
      "Long-term memory: they tried vector retrieval, episodic and semantic stores, and temporal knowledge graphs. One markdown file per user, capped at 8,000 characters, beat all of them. It is loaded on every request and written by a single background job after the session ends. It is small, exact, easy for a human to audit, and safe against leaking one user's memory into another's file.",
      "Specialists: one trace agent in 2025 grew to 10 page-specific specialists (for example a dataset sub-agent for synthetic data), which were then rolled back into one agent that loads skill instructions and tools on demand.",
      "Model choice: the newest model is not the best by default. They default to 'fast and good enough' and put the effort into the harness. They are exploring per-task auto-routing and small post-trained SLMs for narrow surfaces. Some surfaces, such as 'filter with AI', need a single model call rather than the full loop.",
      "Testing: everything is traced. Production sessions become regression datasets and 'living fixtures', session datasets can replay a specific turn, and Arize experiments A/B test harness changes. Agent bugs rarely surface as errors: a status can say OK while there are 43 identical tool calls in one turn, or a 227-second latency. Signal automates finding these, including Alex's retry loops. Judges are aligned to a human baseline of gold labels."
    ],
    "takeaways": [
      "Have the harness own the plan state and the completion check, and render the plan into each model call, instead of giving the model a to-do tool it will over-use.",
      "End turns with an explicit finish gate: bounce back while plan items are open (capped at a few retries), warn before the iteration limit, and force a finish. Don't infer 'done' from text with no tool calls, especially if users can swap models.",
      "Don't put oversized tool output into context. Store it, leave a pointer, and let the agent pull slices with code such as jq. For long-term memory, start with one small capped file per user, written by a background job, before trying vector stores or graphs.",
      "Keep maths and data aggregation in code exposed as tools, so code computes and the model reasons and explains.",
      "Turn production traces into regression and golden datasets in CI, align LLM judges to human gold labels, and leave speed-versus-quality trade-offs as an explicit human decision."
    ],
    "topics": [
      "agents",
      "agent harness",
      "planning",
      "context management",
      "long-term memory",
      "evals",
      "observability",
      "model selection"
    ],
    "tools": [
      "Arize",
      "Arize AX (transcribed as 'Arize X' / 'Aux')",
      "Alex (Arize's in-product agent)",
      "Signal (Arize's trace-reading agent)",
      "Cursor",
      "Anthropic",
      "OpenAI",
      "jq",
      "Sonar 4.5 (as transcribed)"
    ],
    "quotes": [
      {
        "text": "we didn't train these models. Everything we rebuilt was harness code",
        "at": "05:43",
        "url": "https://www.youtube.com/watch?v=YIhvrvHtonk&t=343s"
      },
      {
        "text": "We don't use an agent framework. Um The loop, the plan, the stopping rule, and the memory are all ours.",
        "at": "06:05",
        "url": "https://www.youtube.com/watch?v=YIhvrvHtonk&t=365s"
      },
      {
        "text": "models can be very eager with uh lists.",
        "at": "08:26",
        "url": "https://www.youtube.com/watch?v=YIhvrvHtonk&t=506s"
      },
      {
        "text": "in agents, bugs rarely show up as explicit errors.",
        "at": "16:37",
        "url": "https://www.youtube.com/watch?v=YIhvrvHtonk&t=997s"
      }
    ],
    "words": 4461
  },
  "summary_url": "/#YIhvrvHtonk",
  "transcript": {
    "html": "/transcripts/YIhvrvHtonk.html",
    "txt": "/transcripts/YIhvrvHtonk.txt",
    "vtt": "/transcripts/YIhvrvHtonk.vtt"
  }
}