{
  "video": {
    "id": "CvRngaQZQ3Y",
    "title": "Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain",
    "duration": 1202,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "CvRngaQZQ3Y",
    "title": "Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain",
    "one_liner": "LangChain's applied research lead argues that improving agents is a data mining problem: ship the agent, collect every trace, point other agents at those traces to mine them, and turn what you find into evals, distillation datasets and harness changes.",
    "summary": "Vivek Trivedy lays out a four-step recipe — ship it, collect a ton of traces, data-mine the traces, then run data-driven experiments — and argues observability and continual learning are the same problem, because traces are the substrate that holds an agent's feedback. Because agents now run for millions of tokens, you can't read traces by hand or even fit one into another agent's context, so LangChain sends agents to read traces from other agents and treats long context as an external object to query into. He shows work with Harvey's legal benchmark where an open model matched Opus's trace-judging quality one to two orders of magnitude cheaper, and recommends a sandwich: harness engineering first, fine-tuning to break its ceiling, then more harness engineering.",
    "key_points": [
      "The recipe for continuous improvement: (1) ship the agent so it operates in a real environment, (2) collect all trace data — tool calls, output messages, API and CLI calls, (3) data-mine the gigabytes or terabytes of traces, (4) run experiments against prior traces to check whether a new prompt, tool, orchestration or loop actually helps.",
      "Observability and continual learning are tightly coupled: 'if you're a continual learning company, you need traces, and if you have traces, then you can try to do continual learning over your agents.'",
      "You can read Python and reason about what it does; you can't do that for agents with prompts, tools, skills, hooks, middlewares and swarms — and a prompt change for the medical domain behaves completely differently from one for law.",
      "LangChain centralizes traces in a tracing project (per-agent or across all agents) and sends agents to read other agents' traces — to find good and bad user interactions, to answer technical questions like 'does the agent get really dumb after the first compaction? after the second?', and to test counterfactuals like swapping GPT 5.5 for GLM 5.2.",
      "Two scaling problems: reading traces costs roughly input-token-cost × number of traces × average trace size; and a long coding-agent trace (Claude Code, Codex, deep agents) doesn't even fit in the reading agent's context, so context must be treated as an external object you query into.",
      "With Harvey on their legal benchmark, an open cheaper model roughly matched Opus's trace-judging capability at one to two orders of magnitude lower cost — achieved by trying several models and doing harness engineering informed by reading Opus's traces to see where the open model needs more guidance.",
      "Three concrete outputs of trace mining, all built into LangSmith Engine: distillation/SFT (take good GLM 5.2 traces, fine-tune a 9B or 13B model to mimic the behavior), generating evals in environments, and preparing curated content for humans to review — still essential in high-trust domains like legal and medical where nobody can read it all.",
      "On Terminal Bench, an agent read its own traces, proposed experiments and applied fixes; the key lesson was densifying feedback — a bare pass/fail number gives an agent no signal about what to do next.",
      "Continual learning has to happen across three axes: observational training data from agents acting, evolving the harness itself (Codex's, Claude Code's, everyone's), and memory — which can't be an append-only file you search over, and likely comes from scaling sleep-time compute and 'dreaming' over the whole trace lifecycle."
    ],
    "takeaways": [
      "Turn on tracing and point an agent at your traces — Trivedy calls it the single easiest thing you can do to understand what your agents are actually doing.",
      "Optimize for feedback speed: harness engineering returns a signal in about two minutes, so exhaust it first, fine-tune only to break through its ceiling, then go back to harness engineering. Many teams never need to leave step one.",
      "Start on a frontier model (Opus, 5.5) only to establish that the task is possible, then go back to those traces and test whether a cheaper open model can hit the same waterline with more guidance in the harness.",
      "Densify your feedback signals — replace bare pass/fail scores with richer trace-level signal, because agents are good at reading traces and figuring out what to do next (though they'll cheat on a score sometimes and need checking).",
      "For high inference volume, compare hardware cost against token cost: running your own cluster gives unlimited inference and can be spun down, which they find much cheaper than per-million-token pricing.",
      "Treat your eval suite as your agent's behavior spec — the agent hill-climbs whatever you test, so choose evals knowing they define what it becomes."
    ],
    "topics": [
      "agents",
      "observability",
      "traces",
      "continual-learning",
      "evals",
      "fine-tuning",
      "open-models",
      "distillation"
    ],
    "tools": [
      "LangChain",
      "LangSmith Engine",
      "Opus",
      "GPT 5.5",
      "GLM 5.2",
      "Harvey",
      "Terminal Bench",
      "scikit-learn",
      "Claude Code",
      "Codex",
      "deep agents",
      "ChatGPT"
    ],
    "quotes": [
      {
        "text": "if you have an agent, just turn on tracing and point an agent at it and that's like the easiest thing that you can do to basically understand what your agents are doing",
        "at": "18:56",
        "url": "https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1136s"
      },
      {
        "text": "I think you can basically define agent behavior by showing the evals that you ran on it",
        "at": "11:48",
        "url": "https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=708s"
      },
      {
        "text": "the data that we see today is going to be the smallest that humans have ever seen in their entire lives",
        "at": "06:07",
        "url": "https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=367s"
      },
      {
        "text": "over the last four years since the ChatGPT moment, we've started trading determinism for autonomy",
        "at": "04:19",
        "url": "https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=259s"
      }
    ],
    "words": 4347
  },
  "summary_url": "/#CvRngaQZQ3Y",
  "transcript": {
    "html": "/transcripts/CvRngaQZQ3Y.html",
    "txt": "/transcripts/CvRngaQZQ3Y.txt",
    "vtt": "/transcripts/CvRngaQZQ3Y.vtt"
  }
}