{
  "video": {
    "id": "ImnZ7s68jw8",
    "title": "What Does an AI Engineer Do Now? - Aparna Dhinakaran, Arize AI",
    "duration": 928,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": ""
  },
  "analysis": {
    "video_id": "ImnZ7s68jw8",
    "title": "What Does an AI Engineer Do Now? - Aparna Dhinakaran, Arize AI",
    "one_liner": "The AI engineer's job has shifted from building AI features to building the loop that does it: trace everything, including the coding agents, write evals that agree with human judgment, and set up always-on agents that read telemetry and open PRs.",
    "summary": "Dhinakaran starts from Karpathy's funnel: 6B+ people have touched an LLM, but only about 5,000 at the frontier labs run thousands of agents at once. She argues the AI engineer's role is to close that gap by building the loop (build, eval, ship, monitor, improve) rather than doing each step by hand. As a worked example she walks through Arize's Signal, an always-on agent that reads traces, finds issues, remembers what it found before and can open PRs. Its design covers triggers, sandboxed runs behind a single proxy, durable state and support for any model or harness. She ends by saying the core skill is writing evals and checking them against human judgment, run both offline and online.",
    "key_points": [
      "Karpathy's funnel: 6B+ users have touched an LLM, 1-2B are casual free-tier users, 20M are professionals cutting weeks of work to days, and only about 5,000 at the frontier labs run thousands of agents at once.",
      "Then vs now: the AI engineer used to write the feature, read the diff and set up metrics. Now they build the loop that writes the feature, set up agents and evals to do code review, and have agents read the observability data to find patterns.",
      "Agents, not humans, should be the main readers of telemetry, and observability data becomes the trigger that wakes always-on agents.",
      "Arize Signal is an always-on agent. It reads traces, finds patterns, keeps a memory of issues it has seen before (including ones that become critical as more data arrives) and can open a fix or PR if it has repo access.",
      "Triggers can be a schedule (e.g. every 3 or 6 hours, pull the last X traces), an event (e.g. a Slack message about a user complaint) or a metric.",
      "Each run gets a fresh sandbox with the traces, third-party APM logs and metrics, repo access and keys. A single proxy is the only way in or out, and everything inside is traced. Issues, traces and learned skills are kept (skills are saved to the repo). The sandbox, credentials and repo access are torn down.",
      "Signal is durable: run state lives in memory outside the sandbox, so if the sandbox or harness fails, a new one picks up from the failed step. It works with Claude Code (the first harness), Cursor and Codex, with open-source harnesses planned. It can use any Anthropic model, which lets Arize evaluate quality and weigh cost against quality.",
      "Arize evaluates its own evaluators: it builds golden datasets, compares judge output with human output and only ships a new evaluator once the two agree. Evals run offline before shipping and online over production telemetry."
    ],
    "takeaways": [
      "Build the loop rather than the feature: design always-on agents with clear triggers, sandboxed runs, memory kept between runs and durable state.",
      "Set up telemetry with agents as the main readers, and use observability data as the trigger for automated investigation.",
      "Trace everything, including the coding agents (Claude Code, Cursor, Codex) that built your agents.",
      "Spend your time defining what good looks like: look at traces with your agents, find the common errors, and check evaluators against golden datasets labelled by humans before trusting them.",
      "Run the same evals offline before shipping and online on production traces, then feed the results back to agents so they can find gaps."
    ],
    "topics": [
      "ai engineering",
      "agents",
      "evals",
      "observability",
      "llm-as-judge",
      "sandboxing",
      "coding agents",
      "always-on agents"
    ],
    "tools": [
      "Arize AI",
      "Arize Signal",
      "Claude Code",
      "Cursor",
      "Codex",
      "Anthropic",
      "Claude skills",
      "Andrej Karpathy"
    ],
    "quotes": [
      {
        "text": "As an AI engineer, your job's no longer to just write the feature, write an AI feature, it's about how do you build the loop to write that feature.",
        "at": "02:33",
        "url": "https://www.youtube.com/watch?v=ImnZ7s68jw8&t=153s"
      },
      {
        "text": "how do you set up your telemetry data so that agents are the primary consumers of that data.",
        "at": "05:04",
        "url": "https://www.youtube.com/watch?v=ImnZ7s68jw8&t=304s"
      },
      {
        "text": "we actually evaluate the evaluators.",
        "at": "12:06",
        "url": "https://www.youtube.com/watch?v=ImnZ7s68jw8&t=726s"
      },
      {
        "text": "tracing everything now means not just tracing the agents, but also tracing the coding agents behind the agents that built this.",
        "at": "13:07",
        "url": "https://www.youtube.com/watch?v=ImnZ7s68jw8&t=787s"
      }
    ],
    "words": 3046
  },
  "summary_url": "/#ImnZ7s68jw8",
  "transcript": {
    "html": "/transcripts/ImnZ7s68jw8.html",
    "txt": "/transcripts/ImnZ7s68jw8.txt",
    "vtt": "/transcripts/ImnZ7s68jw8.vtt"
  }
}