{
  "video": {
    "id": "0RNNfxpdbQk",
    "title": "Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest",
    "duration": 681,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "0RNNfxpdbQk",
    "title": "Medic for Apache Spark - First Aid for Failing Jobs - Drasko Profirovic, Pinterest",
    "one_liner": "Pinterest's Drasko Profirovic walks through how \"Medic for Apache Spark\" evolved from a single ReAct agent with one overloaded prompt into a multi-agent LangGraph deep-agents system that answers \"why did this job fail?\" with an evidence-backed root-cause report — and why log classification, image-based metrics analysis, and a record/playback eval harness were the changes that actually made it work.",
    "summary": "The talk argues that data-platform support rotations are an unbounded stream of hard distributed-systems questions, and that LLMs can scale that knowledge on demand where humans cannot. Medic's vision was to ask \"why did a job fail?\" and get back a deep research document with evidence on the root cause plus fixes grounded in the job's context, available on the surfaces users already use (Slack, the Airflow UI). The prototype — MCP-exposed data resources plus a single ReAct agent with one giant prompt — worked but hit unsustainable prompt tuning, inconsistent response quality, no controls to keep the agent on track, and context-window blowups from large log tool outputs. The fixes were observability (OpenTelemetry traces to Langfuse), a record/playback end-to-end test harness with offline evals, an exception classifier pipeline, metrics-as-images in a quarantined sub-agent, and a decomposition into specialized agents on LangGraph's deep agents library.",
    "key_points": [
      "The single-ReAct-agent prototype failed in specific ways: one prompt had to do everything so adding detail in one area degraded another; analysis was sometimes shallow and sometimes too verbose; there were no controls to keep the agent on track; and large log tool outputs consumed tokens and halted reasoning on production jobs.",
      "End-to-end testing originally relied on manual tests against production, which felt anecdotal because production data gets retentioned away — so it was hard to know whether a change broke earlier wins.",
      "The test harness has a record mode (agent calls real downstream systems, tool responses captured as fixtures, saved to the filesystem and checked in as code) and a playback mode (agent runs against fixtures, still performs analysis and generates the report, which the suite then grades against offline evals).",
      "A concrete offline eval example: check for a limit of three suggested fixes, scoring lower if the agent offers too many, to manage the verbosity of the final report.",
      "Logs are noisy and many exceptions are benign, so focusing on the last exception isn't reliable. A regex heuristic filter didn't scale; instead an exception classifier pipeline learns which exceptions commonly appear in *successful* jobs, treats those as likely red herrings, and filters them. Exceptions are fingerprinted, clustered, and ranked by content relevance and recency relative to job termination.",
      "The agent stopped consuming logs directly and was given two MCP tools instead: get the top-K truncated exceptions, or get full log details for a specific exception — improving signal-to-noise and reducing anchoring on a misleading exception.",
      "Raw time-series metrics are not context-window friendly and are token inefficient, so a quarantined sub-agent converts them into graphs collaged into a single annotated image (min/max called out, like a Grafana dashboard) and the model reasons over the picture — guaranteeing a fixed input-token cost regardless of job duration. Signals spotted include executors dropping to zero or near zero, long plateaus, and bottlenecks.",
      "The production architecture: intent classification (simple question vs. deep diagnostic) → triage agent determines the job's lifecycle state → if failed, generates failure hypotheses → each hypothesis researched in parallel, returning a score and root cause → supervisor picks the highest-confidence root cause → healer agent offers remediations from runbooks ingested into a vector database → supervisor assembles the final formatted report.",
      "Because each agent is a dedicated prompt plus a subset of MCP tools, expanding scope was as simple as adding a new prompt — which is how Medic was extended to help users optimize Spark SQL jobs.",
      "LangGraph's workflows were trialed to make the agent more deterministic, but proved brittle compared to the reasoning-and-acting agent paradigm."
    ],
    "takeaways": [
      "Don't feed raw logs or raw time-series to the agent. Put a classifier in front of logs (learn benign exceptions from successful runs and filter them) and expose top-K truncated exceptions plus a drill-down tool; render metrics as an annotated image so token cost is constant regardless of job length.",
      "Build a record/playback fixture harness before you tune prompts — snapshot production tool responses as checked-in fixtures so evals are reproducible after production data is retentioned away, and grade reports with offline evals so quality is quantified rather than intuited.",
      "Stop scaling a single ReAct prompt. When adding detail in one area degrades another, decompose into specialized agents each with a dedicated prompt and a narrow subset of MCP tools; LangGraph's deep agents library supplies the to-do list and virtual file system that keep the agent on track.",
      "Instrument agent runs with OpenTelemetry into Langfuse and read executions as waterfall diagrams — that's how you find the cause of low-quality responses.",
      "Put sub-agents in quarantine for token-heavy analysis: have them summarize findings back to the parent so the parent's context window stays healthy.",
      "Prefer ReAct-style agency over hard-coded deterministic workflow graphs for diagnostics; the deterministic version was brittle."
    ],
    "topics": [
      "agents",
      "multi-agent",
      "evals",
      "observability",
      "mcp",
      "data-platform",
      "spark",
      "context-engineering"
    ],
    "tools": [
      "Apache Spark",
      "Pinterest",
      "Model Context Protocol (MCP)",
      "LangGraph",
      "LangGraph deep agents library",
      "Langfuse",
      "OpenTelemetry",
      "Slack",
      "Airflow",
      "Grafana",
      "Claude Code",
      "Codex",
      "Spark SQL",
      "Apache Flink",
      "Trino"
    ],
    "quotes": [
      {
        "text": "One prompt had to do everything, and adding detail in one area degraded the behavior in another.",
        "at": "03:03",
        "url": "https://www.youtube.com/watch?v=0RNNfxpdbQk&t=183s"
      },
      {
        "text": "The core idea was we would learn which exceptions commonly appear in successful jobs, treat those as likely red herrings, and filter them out in the future analysis.",
        "at": "06:02",
        "url": "https://www.youtube.com/watch?v=0RNNfxpdbQk&t=362s"
      },
      {
        "text": "Images worked better because we could guarantee how many input tokens would be used for analyzing any given Spark job irrespective of its duration.",
        "at": "07:42",
        "url": "https://www.youtube.com/watch?v=0RNNfxpdbQk&t=462s"
      },
      {
        "text": "We trialed using LangGraph's workflows to make the agent more deterministic, but this approach proved to be brittle compared to the reasoning and acting agent paradigm.",
        "at": "10:40",
        "url": "https://www.youtube.com/watch?v=0RNNfxpdbQk&t=640s"
      }
    ],
    "words": 1863
  },
  "summary_url": "/#0RNNfxpdbQk",
  "transcript": {
    "html": "/transcripts/0RNNfxpdbQk.html",
    "txt": "/transcripts/0RNNfxpdbQk.txt",
    "vtt": "/transcripts/0RNNfxpdbQk.vtt"
  }
}