{
  "video": {
    "id": "XLEYtv3cMlw",
    "title": "Autonomous Agents for Scientific Tasks - Sina Shahandeh, Radicait",
    "duration": 1163,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "Online"
  },
  "analysis": {
    "video_id": "XLEYtv3cMlw",
    "title": "Autonomous Agents for Scientific Tasks - Sina Shahandeh, Radicait",
    "one_liner": "Coding agents plateau on open-ended scientific problems because they run out of hypotheses, not implementation ability — Sina Shahandeh's fix is to make the agent first explicitly decompose the codebase into a linked hierarchy of components, then generate and critique radical ideas against every node of that scaffold.",
    "summary": "Shahandeh starts from Karpathy's auto-researcher setup — a coding agent hill-climbing a metric — and shows that on long-horizon scientific work the agents saturate: they implement and run experiments well but run out of ideas, what people call research taste. Using Radicait's in-silico PET problem (generating a synthetic PET scan from a CT scan to judge whether a lung nodule is cancerous), he demonstrates a trick analogous to chain-of-thought: prompt the coding agent to walk the codebase and emit a hierarchy of linked documents (data, architecture, training loss, metrics, scripts, data prep) down to the actual code, then ask it to generate improvement ideas against that hierarchy. With that scaffold the agent proposes genuinely radical changes — e.g. going from a 2.5D stacked-slice generator to 3D convolutions — instead of only fiddling with hyperparameters. He also equips the loop with extra models as skills: a multimodal model to eyeball image-registration QC, and a bigger reasoning model for hypothesis generation and critique.",
    "key_points": [
      "Karpathy's auto-research repo works well on coding tasks, but on open-ended, long-horizon scientific problems agents saturate at a level while good humans — and especially the top 1% — keep climbing; the gap is hypothesis generation, not implementation.",
      "The learning/memory half of the scientific loop he considers 'quite solved by just simply organizing patterns'; coming up with a good hypothesis is the hard part and the focus of the talk.",
      "Radicait's problem: in-silico PET — image translation from a CT scan to a synthetic PET scan, so a lung nodule can be assessed without the difficult, time-consuming real PET; an encoder-decoder GAN, with metrics on fidelity of synthetic PET to real PET, and ~80% of the work being getting a good dataset.",
      "In a real Codex run, iterations spread into dead ends and then desaturate at a plateau; the speaker had to interrupt mid-loop with 'what about this idea' or go read papers to unstick it.",
      "The trick: an explicit decomposition step, inducible with one prompt — the coding agent walks the codebase and writes a linked hierarchy of documents (level 1 problem statement → level 2/3 components: data, model architecture, training loss, operations, metrics and evidence, peripheral scripts, data preparation) terminating in the actual code; shown as a graph in Obsidian.",
      "Given that hierarchy, asking for improvements can yield ~100 different solutions because there is a process to modify every component — a much more comprehensive search, and it surfaces radical moves like 3D convolutions instead of the original 2.5D (2D convs over stacked slice channels) that the agent would otherwise miss.",
      "Second example — CT/PET image registration, misaligned because scans happen at different times, with respiration, patient movement and different scanners; metrics here are partly qualitative and it's hard to tell which issue caused a failure.",
      "He built a skill that calls a stronger multimodal model (Gemini) to review generated images as a QC metric — is the lung mask on the lungs, are the scans properly truncated or cropped — and uses Peter Spinberger's Oracle CLI to package code and data to the GPT-5.5 Pro API for hypothesis generation and critique ('this is what we implemented, this is the outcome, does this make sense, what would you do next').",
      "The stated bottleneck to a 'full scientist in a data center' is observation: no LLM today reliably spots a tiny nodule because they aren't trained on scientific images and data, so models can't close the loop by observing the way a trained scientist does."
    ],
    "takeaways": [
      "Before running an optimization loop, make the agent emit an explicit hierarchical decomposition of the codebase as linked documents down to the code — then generate hypotheses against each component rather than asking it to 'optimize this repo'.",
      "When your agent plateaus, treat it as an idea shortage, not an implementation problem: the scaffold is what lets it propose architecture-level changes (2.5D → 3D) instead of hyperparameter tweaks.",
      "Review generated plans adversarially or collaboratively with a second agent or model, and feed each loop's implementation-plus-outcome back for critique and a 'what would you do next'.",
      "Equip the loop with additional skills that call out to specialist models: a multimodal model for qualitative visual QC metrics, and a high-test-time-compute reasoning model (via something like the Oracle CLI packaging code and data to the API) for hypothesis generation.",
      "Expect these hierarchy tricks to be temporary scaffolding — as models get post-trained to compartmentalize and break down problems, less of this will be needed, the same way chain-of-thought was bolted onto GPT-4o-era models."
    ],
    "topics": [
      "agents",
      "auto-research",
      "hypothesis-generation",
      "medical-imaging",
      "test-time-compute",
      "multimodal",
      "prompting",
      "coding-agents"
    ],
    "tools": [
      "Codex",
      "Obsidian",
      "Gemini",
      "Oracle CLI (Peter Spinberger)",
      "GPT-5.5 Pro",
      "GPT-4o",
      "Radicait",
      "GitHub",
      "GAN"
    ],
    "quotes": [
      {
        "text": "the problem is they ran out of ideas uh or you know what people call them research taste",
        "at": "01:15",
        "url": "https://www.youtube.com/watch?v=XLEYtv3cMlw&t=75s"
      },
      {
        "text": "what is much more difficult is it coming up with a hypothesis",
        "at": "02:31",
        "url": "https://www.youtube.com/watch?v=XLEYtv3cMlw&t=151s"
      },
      {
        "text": "So basically creates a much more radical change that it's otherwise would have not come up or or would have missed.",
        "at": "11:16",
        "url": "https://www.youtube.com/watch?v=XLEYtv3cMlw&t=676s"
      },
      {
        "text": "no LLM today is able to identify these very well because they're just not simply trained on scientific images and scientific data",
        "at": "17:29",
        "url": "https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1049s"
      }
    ],
    "words": 3635
  },
  "summary_url": "/#XLEYtv3cMlw",
  "transcript": {
    "html": "/transcripts/XLEYtv3cMlw.html",
    "txt": "/transcripts/XLEYtv3cMlw.txt",
    "vtt": "/transcripts/XLEYtv3cMlw.vtt"
  }
}