{
  "video": {
    "id": "1EZdpEhwmNc",
    "title": "Through the AI Fog: The Architectural Decision Agentic Security Depends On — Manoj Nair, Snyk",
    "duration": 1409,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "1EZdpEhwmNc",
    "title": "Through the AI Fog: The Architectural Decision Agentic Security Depends On — Manoj Nair, Snyk",
    "one_liner": "Snyk's CTO argues from customer telemetry and fresh benchmarks that the generator and the validator of code cannot be the same model — LLMs find the same vulnerability in only 50% of five repeated runs and catch 75% of what a boring deterministic check finds — so agentic security needs independent, deterministic verification plus governance over skills, MCP servers and agent behaviour.",
    "summary": "Manoj Nair opens the first security track at the AI Engineer World's Fair by naming the one architectural decision he thinks agentic security depends on: separating the generator from the validator. He backs it with real data from Snyk's ~5,000 enterprise customers — vulnerability backlogs up 108% quarter over quarter across 4,800+ customers, three times more agentic components than models per repo, red-team results where one popular open model leaked PII 100% of the time — and with a just-published benchmark showing the latest frontier models are both inconsistent and incomplete at finding vulnerabilities. The fix he shows is not a better model but security context injected into the agent loop (Snyk Studio), automated remediation, skill/MCP risk assessment, and an OODA-loop system called Evo aimed at making AI security engineers 10x.",
    "key_points": [
      "Central claim: 'can the generator and the validator be the same?' — a question security never had to ask before, and the data says no; don't expect probabilistic systems to validate themselves.",
      "New benchmark published the day before the talk (deliberately one no model has trained on): asked the latest models — including ones not publicly available — to find the same vulnerability five times; only 50% were found consistently across the five runs, only 75% of issues were found versus a 'good old boring deterministic check', and the F1 score was 40%.",
      "Across 4,800+ customers over the last year, actual vulnerability backlog grew 108% quarter over quarter — the backlog is growing despite the best agents, and AI-generated code quality is 'a little worse' than human-generated.",
      "Snyk's red-team risk DB run from that weekend: the two leading frontier models resisted PII extraction entirely with their attacks, while the 'hot new model in Silicon Valley... rhymes with LLM' leaked PII 100% of the time — but on decision-override attacks the result flipped, with the frontier models doing worse and the open model overridden 0% of the time. Which model is safe depends on the test, and it changes constantly.",
      "The environment is poisoned, not just the code: research found more than a third of agent skills (across Claude Code, Codex and others) contain malware or vulnerabilities — 'three lines of English are able to now bring a system down' — and the GitHub MCP server exploit Snyk disclosed a year ago made customers shut down all MCP servers and then lose their dev screens.",
      "Agent behaviour creates unknown attack surface: in Fortune 100 environments, agents 'squirrel away' copies of PII into untrusted databases just in case they need it later, outside any enterprise security coverage.",
      "Discovery gap: across 3,000+ customers, for every model found in a repo there are three times more agentic components — agents, tools, everything they use — and 'you can't govern what you don't know exists.'",
      "Attacks are the other half: frontier labs have all been used in automated attacks, and you can chain low-severity vulnerabilities into working exploits without a frontier model, which kills contextual risk management ('I fixed my criticals and highs, I'm fine'). The LiteLLM exploit is existing vulnerabilities chained across a new surface, not a novel one. Five Eyes intelligence leaders said last week AI will bypass cybersecurity systems 'in months, not years'.",
      "Shipping answers: Snyk Studio puts package health/vulnerability/malware context into the agent loop so it won't pick a bad package or write SQL injection; a remediation agent that understands 'breakability' took Labelbox to zero backlog and let a Mag-7 company remediate 16,000 critical issues last week; agentic dev security GA'd the day before, covering environment, output, skills, MCP servers and the behaviour of Cursor, Claude and Codex.",
      "Live demo (Ezra): Claude asked to build a QR-code CLI calls the Snyk package health tool on two candidate dependencies — both qrcode and qrimage are CVE-free today, but qrimage was first released 10 years ago and is unmaintained, so a future vulnerability would take far longer to patch. A second demo ran risk assessment on a shared 'competitive analysis' skill and returned four findings: it echoes the authorization header, pulls live content from Reddit and Twitter, and — worst — fetches its monitoring instructions and classification rules from a YAML file hosted on the internet, so the skill's logic can change without the skill file changing at all."
    ],
    "takeaways": [
      "Don't let the model that generated the code be the only thing that validates it — pair LLMs with deterministic checks and use them for the surface deterministic checks can't reach, not as a replacement.",
      "Don't hardcode a 'safe model' choice: test candidate models yourself against both PII extraction and decision override, because a model that aces one fails the other and the rankings change week to week.",
      "Treat skills and MCP servers as untrusted supply chain — assess them before use, and specifically flag skills that echo auth headers or fetch their own logic/config from a third-party URL, since that logic can be swapped without the skill file changing.",
      "Make the package-health check deterministic: rather than prompting the agent verbosely, wire it in as a skill or hook so every dependency choice is checked automatically, and judge packages on maintenance and patch latency, not just on whether they have a CVE today.",
      "Inventory the agentic layer, not just models — expect roughly 3x more agents, tools and components than models per repo — and enforce AI governance as real-time policy in the dev/agent loop instead of a Confluence page or PDF."
    ],
    "topics": [
      "agentic security",
      "appsec",
      "mcp",
      "agent skills",
      "supply chain",
      "red teaming",
      "benchmarks",
      "ai governance"
    ],
    "tools": [
      "Snyk",
      "Snyk Studio",
      "Snyk Evo",
      "Claude",
      "Claude Code",
      "Codex",
      "Cursor",
      "GitHub MCP server",
      "MCP",
      "LiteLLM",
      "Labelbox",
      "qrcode / qrimage npm packages",
      "Five Eyes",
      "AI Engineer World's Fair"
    ],
    "quotes": [
      {
        "text": "Can, you know, the generator and the validator be the same?",
        "at": "03:22",
        "url": "https://www.youtube.com/watch?v=1EZdpEhwmNc&t=202s"
      },
      {
        "text": "The question nobody is answering is, can you trust what your agents just shipped and how they did it, right?",
        "at": "04:12",
        "url": "https://www.youtube.com/watch?v=1EZdpEhwmNc&t=252s"
      },
      {
        "text": "Three lines of English are able to now bring a system down.",
        "at": "09:14",
        "url": "https://www.youtube.com/watch?v=1EZdpEhwmNc&t=554s"
      },
      {
        "text": "That AI governance cannot live in a confluence page or PDF.",
        "at": "15:14",
        "url": "https://www.youtube.com/watch?v=1EZdpEhwmNc&t=914s"
      }
    ],
    "words": 4917
  },
  "summary_url": "/#1EZdpEhwmNc",
  "transcript": {
    "html": "/transcripts/1EZdpEhwmNc.html",
    "txt": "/transcripts/1EZdpEhwmNc.txt",
    "vtt": "/transcripts/1EZdpEhwmNc.vtt"
  }
}