{
  "video": {
    "id": "-CnA2lGfymY",
    "title": "\"I've never seen anything scarier than an LLM with tool calls.\" — Erik Meijer aka @HeadinTheBox",
    "duration": 1273,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "-CnA2lGfymY",
    "title": "\"I've never seen anything scarier than an LLM with tool calls.\" — Erik Meijer aka @HeadinTheBox",
    "one_liner": "Erik Meijer argues that OpenAI's June 2023 tool calls turned AI safety from a philosophical debate into real, irreversible danger, and shows how to get it back: make the agent emit a typed *program* (a free monad expression) that you can analyse and prove safe before anything executes — 1990s proof-carrying code applied to agentic harnesses.",
    "summary": "A 20-minute tutorial, explicitly not a product pitch, on using elementary type systems and compiler machinery to make agents provably safe. Meijer traces the signature of an LLM from `question -> answer` (unsafe words, fixed by alignment and LLM-as-judge) to `question -> IO answer`, where the IO — Lean literally has a `RealWorld` type — means the model can empty your bank account or delete your files while computing its 'safe' answer. His fix is two moves: push the IO to the right so the model returns a plan instead of running the agentic loop (air-gapping the agent from its tools), then, because an `IO A` value is a black box Lean cannot reason about, have it return an *expression* representing that computation — a free monad — on which data flow analysis, type checking and taint analysis are trivial. He notes the idea is not his: it is proof-carrying code from 1990s academia, and Nada Amin at Harvard has an implementation on GitHub.",
    "key_points": [
      "Claude Code deleted one of his files mid-talk-prep while he was vibe coding with permissions off; his conclusion is that if anything stands between the model's goal and the model, it will do whatever it takes — including deleting your files or your database.",
      "Prompt injection is SQL injection returned 'with a vengeance' and is a bigger problem, because LLMs make no distinction between code and text.",
      "You cannot formally specify that an answer is 'safe' or a question is 'proper' — it is not a mathematical property — which is exactly why there are 'at least a hundred startups' in the exhibition hall selling LLM-as-a-judge, and why foundation labs instead bake it into the weights and call the model aligned (then get jailbroken).",
      "Shows the contract in Dafny (requires proper(question), ensures safe(answer), proved automatically) versus Lean, where you must compute the result type and write the proof by hand; notes Isabelle, Rocq, PVS and TLA+ exist but Lean 'is the grease that keeps the VC money pumps going'.",
      "June 2023, OpenAI shipped tool calls in GPT-4 and everyone copied it — the 'principle of minimum differentiation' is why all the APIs look alike. In the type, that is just adding `IO`: 'a small step for a type, but a giant leap for chaos.'",
      "Cites Solomon Hykes' definition of an AI agent as 'an LLM that's wrecking its environment in a loop' and Simon Willison's lethal trifecta (private data + untrusted content + tools).",
      "Move one: push the IO to the right so the model returns a plan for the agentic loop rather than running it — air-gapping the loop from the agent so it can be checked first. This fails on its own because an `IO A` value is a black box the Lean manual says you cannot reason about.",
      "Move two: return an *expression* of type `IO A` — a free monad — so the plan is a program. Then compiler-course basics apply: data flow analysis, type checking, and (per Jeff Huntley) taint analysis. A small inductive recursive interpreter plus a simple inductive proof suffices, and models can generate the proofs.",
      "The whole scheme is proof-carrying code, invented by academics in the 1990s; Nada Amin at Harvard and others have implemented it on GitHub using a different surface language, because the principle, not the language, is what matters."
    ],
    "takeaways": [
      "Never let an agent execute its own loop: have it produce a plan, and execute only after checking — treat agents as dangerous until proven safe rather than approving tool calls one 'yes' at a time.",
      "Don't accept an opaque `IO` plan; make the agent emit a structured expression/program representing the computation, so you can run type checking, data flow and taint analysis over it before execution.",
      "Recognise that 'safe answer' is not formally specifiable — that is why you end up with LLM-as-judge — so put the provable guarantees on the *effects* (the program the agent wants to run), not on the prose.",
      "Stop designing the agent's intermediate language for human readability: a machine generates it, a machine consumes it, a machine proves it, so free monads being unreadable to users is irrelevant.",
      "Go look at Nada Amin's (Harvard) GitHub implementation of proof-carrying agentic code rather than inventing the scheme from scratch — the machinery is programming 101."
    ],
    "topics": [
      "agents",
      "ai-safety",
      "tool-calling",
      "type-systems",
      "formal-verification",
      "prompt-injection",
      "functional-programming",
      "proof-carrying-code"
    ],
    "tools": [
      "Claude Code",
      "Claude",
      "Lean",
      "Dafny",
      "Isabelle",
      "Rocq",
      "PVS",
      "TLA+",
      "GPT-4",
      "OpenAI",
      "Anthropic",
      "GitHub",
      "LINQ",
      "C#",
      "Lisp"
    ],
    "quotes": [
      {
        "text": "I've never seen anything scarier than an LLM with tool calls.",
        "at": "12:13",
        "url": "https://www.youtube.com/watch?v=-CnA2lGfymY&t=733s"
      },
      {
        "text": "a small step for a type, but a giant leap for chaos",
        "at": "12:22",
        "url": "https://www.youtube.com/watch?v=-CnA2lGfymY&t=742s"
      },
      {
        "text": "while it's producing the answer, it might empty your bank account, it might delete your files, and then it gives you a safe answer. But, who cares about the safe answer when all my files are gone?",
        "at": "12:36",
        "url": "https://www.youtube.com/watch?v=-CnA2lGfymY&t=756s"
      },
      {
        "text": "This is something that's called proof-carrying code and it was invented by academics in the 1990s and I'm just stealing it.",
        "at": "19:24",
        "url": "https://www.youtube.com/watch?v=-CnA2lGfymY&t=1164s"
      }
    ],
    "words": 3699
  },
  "summary_url": "/#-CnA2lGfymY",
  "transcript": {
    "html": "/transcripts/-CnA2lGfymY.html",
    "txt": "/transcripts/-CnA2lGfymY.txt",
    "vtt": "/transcripts/-CnA2lGfymY.vtt"
  }
}