{
  "video": {
    "id": "kZsf_Sfm7RU",
    "title": "The Missing Layer After Launch - Raphael Kalandadze, Wandero AI",
    "duration": 1173,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "kZsf_Sfm7RU",
    "title": "The Missing Layer After Launch - Raphael Kalandadze, Wandero AI",
    "one_liner": "Most agent talks stop at ship day; this one argues the real work is the feedback loop after launch, and shows the three agents Wandero runs on their own production agent — an hourly log-monitoring agent that opens PRs, a session analyzer that scores every conversation, and a computer-use agent that plays the customer in a browser.",
    "summary": "Raphael Kalandadze calls post-launch observability and self-improvement \"the missing layer\": with today's models writing 100k lines of code is the easy part, but once an agent is live you lose the feel for your own system because coverage is endless, LLMs are non-deterministic, and failures hide behind lucky recoveries and technically-successful-but-wrong outcomes. Because separating a real bug from noise, and a symptom from a root cause, needs reasoning, he argues that operating an agent is itself an agent problem. He walks through Wandero's \"meta harness\": a fast loop (log-monitoring agent over trajectories plus codebase access → PR → adversarial review agent with fresh context → human), a slow loop (session analyzer scoring every conversation into a health dashboard), and a computer-use agent that exercises the real UI.",
    "key_points": [
      "Unit tests, regex, rule-based checks and scripted customer-conversation simulations were all tried and are \"only one slice of the whole problem\" — production is where you learn what you needed to test in the first place.",
      "The scariest failure mode is the silent one: an agent struggles mid-task, gets lucky, finds a workaround, finishes — no red alert, no dashboard problem — and reliable agents shouldn't depend on luck. He cites an Anthropic blog post on agents marking features complete without checking they actually worked.",
      "\"Finished\" ≠ helpful: in Wandero's travel use case, the agent built an itinerary but used a different service and miscalculated the price — technically successful, still failing the task.",
      "Fast loop: a log-monitoring agent runs every hour (or every 15 minutes) over a one-hour window of logs with access to trajectories and the codebase, then opens a PR or fires a Slack alert; a PR can be ready in half an hour.",
      "PRs carry a description, metadata, mermaid diagrams or ASCII tables and HTML artifacts so a human can grasp the problem at a glance; static analysis flags critical vs. heads-up issues.",
      "A separate review agent with fresh context is treated as critical: it is unbiased toward the fix, criticizes and scores the PR from a different angle, runs focused tests, requests changes or closes the PR outright — because fixing agents are \"pretty eager to send the PR.\"",
      "The PR agent and review agent together open 10x more PRs per day than the three-person team, making the human the bottleneck — his position is close the loop first, become the bottleneck, then remove yourself.",
      "Slow loop: a session analyzer scores every conversation (many stateful agents, lots of tokens) into a dashboard of system health, sessions analyzed, cost, average score, success rate, trends, score distribution, sentiment, entities, tool-call analytics and rejection reasons — plus AI insights naming root cause, sessions affected and a recommended fix. Run it once or twice a week; it's for patterns, not specific bugs.",
      "Third angle is the user's: a computer-use agent opens the browser, logs in, sends messages and checks the UI. Codex driving the browser was \"pretty slow\"; a purpose-built skill that knows their site's DOM is much faster — but it burns a lot of tokens.",
      "Every one of these agents must be given the same access a human would need — trajectories, metrics, database, UI — so its conclusions track the real problem instead of guessing."
    ],
    "takeaways": [
      "Treat monitoring your agent as an agent-building problem, not a dashboard problem: wire trajectories, logs and codebase access into an agent that diagnoses and opens PRs on a 15-minute-to-hourly cadence.",
      "Always pair the fixing agent with a separate review agent on fresh context whose job is to criticize, score and close bad PRs — it is the filter that makes the loop trustworthy enough to run unattended.",
      "Run two loops at different zoom levels: a fast one for local fixes, and a weekly session analyzer that scores every conversation to give you the system's health and cross-session patterns.",
      "Add a user-perspective check — a computer-use agent that logs in and drives the real UI — since code and logs can't see UI breakage; build a site-specific skill rather than generic browser driving for speed.",
      "Keep the human in the loop initially and invest in calibration; deliberately become the bottleneck, then automate yourself out, rather than skipping the loop entirely."
    ],
    "topics": [
      "agents",
      "observability",
      "production-monitoring",
      "evals",
      "agent-harness",
      "computer-use",
      "code-review-agents",
      "feedback-loops"
    ],
    "tools": [
      "Wandero AI",
      "Claude Code",
      "Codex",
      "Anthropic",
      "Slack",
      "Mermaid"
    ],
    "quotes": [
      {
        "text": "But I think the shipping is the moment when the real work begins. Because you need to close the loop as soon as possible.",
        "at": "00:58",
        "url": "https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=58s"
      },
      {
        "text": "So actually, you'll find out that the operating agent itself is an agent problem.",
        "at": "06:43",
        "url": "https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=403s"
      },
      {
        "text": "So technically it's successful but still failing the task.",
        "at": "05:34",
        "url": "https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=334s"
      },
      {
        "text": "In our case the PR agent and the review agent send 10 times more PR than the three of us every day.",
        "at": "10:56",
        "url": "https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=656s"
      }
    ],
    "words": 3892
  },
  "summary_url": "/#kZsf_Sfm7RU",
  "transcript": {
    "html": "/transcripts/kZsf_Sfm7RU.html",
    "txt": "/transcripts/kZsf_Sfm7RU.txt",
    "vtt": "/transcripts/kZsf_Sfm7RU.vtt"
  }
}