{
  "video": {
    "id": "iKQ78wyJEXU",
    "title": "We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank",
    "duration": 984,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "iKQ78wyJEXU",
    "title": "We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank",
    "one_liner": "Nubank's product security team built \"Skill Vector\", a CI gate that scans every AI skill before it reaches their internal marketplace — 2,000+ skills scanned, 1,500+ risks found — on the premise that skills look like configuration but behave like supply chain dependencies.",
    "summary": "Lucas Palma (LP), product security manager at Nubank, argues that AI skills, plugins, MCP servers and agent rules are now part of the software supply chain: a skill author is effectively guiding the code another engineer's agent generates, inside a regulated financial environment that demands auditability and credential safety. Nubank made an internal marketplace the single canonical distribution channel for skills, then inserted a CI step called Skill Vector that runs deterministic regex checks first, escalates to an LLM review for context, reports findings back on the PR, and emits SARIF into the existing vulnerability management program. Over a 2,000-skill baseline it surfaced more than 1,500 risks, roughly 1,000 remediated immediately and a few blocked outright, plus a retroactive scan of skills created before the tool existed. The closing lesson: protect the whole developer workflow, not just the generated code.",
    "key_points": [
      "Skills \"look like configuration\" but \"behave like supply chain dependencies\" — the AI-era supply chain now includes skills, plugins, MCP servers and agent rules alongside packages, containers and models.",
      "The core danger is transitive authorship: when one person shares a skill, they are \"guiding over the code that's being generated by the other person\" — concrete failure modes cited are skills that fetch and hardcode a token that then leaks into logs, skills that instruct the agent to run destructive shell commands or modify files on someone else's machine, over-broad permissions, and even a typo being dangerous depending on who runs the skill.",
      "Nubank funnels all skill sharing through one internal marketplace (skills bundled with plugins), making it the security boundary; a skill reaches it via pull request, and Skill Vector runs as a CI step before that PR merges.",
      "Skill Vector is deliberately hybrid: deterministic checks (regular expressions) for the easy risks, then LLM review where more context is needed — because LLM verdicts vary with temperature, \"sometimes it will check that it's a risk sometimes it might not.\"",
      "It runs in three places: locally so engineers can iterate before uploading, again in the PR (you can't assume the engineer ran the latest version locally), and again after upload. Findings are posted as PR comments and exported as SARIF so existing security tooling consumes them and they enter the vulnerability management program.",
      "Checks include unsafe instructions, behavioral drift from the agent's intended behavior, destructive shell commands, unwanted file modifications, how credentials are requested, unintentional data exposure, over-broad permissions and risky MCP usage.",
      "Numbers on the baseline: over 2,000 skills scanned (\"now there is much more than that\"), more than 1,500 risks identified (one skill can carry several), ~1,000 remediated right after, a few genuinely risky ones blocked before reaching the marketplace. A separate historical scan of pre-Skill-Vector skills found further risks and fed them into vulnerability management.",
      "The prompt-level confirmation trap: an author writes \"ask for confirmation\", but the AI may ask confirmation of itself — \"from your perspective there is a human in the loop but for the AI perspective there has been a confirmation.\" Skill Vector scans for real human-in-the-loop, checking which tool executes and whether it passes through approval gates and hooks.",
      "Honest list of what needed improvement: treating all shell commands as equally risky when they aren't, weak low-context signals that were \"more troublesome than helpful\", warnings that look harmless locally but are impactful in production, findings shipped without clear remediation guidance, and the fact that anyone can spin up a second marketplace outside the gate.",
      "Next steps generalize the pattern: MCP Vector and rules checks with their own risk sets, gates varying by policy and marketplace, tool-level enforcement with audit logs and trusted gateways, and a trusted AI marketplace that also ingests third-party skills — download something external and you should upload it to the marketplace so it gets scanned before use."
    ],
    "takeaways": [
      "Treat skills, plugins, MCP servers, rules and hooks as supply chain dependencies with review gates, not as configuration files that merge without scrutiny.",
      "Make one internal marketplace the canonical distribution path and put the scanner in the PR/CI step in front of it — including for third-party skills people download from outside.",
      "Combine deterministic pattern checks with LLM review rather than choosing one: regex for the cheap, stable risks; LLM for behavioral context, with the deterministic layer compensating for temperature-driven nondeterminism.",
      "Let engineers run the same scanner locally to iterate, but re-run it in CI and after upload — never trust that the local run used the latest version.",
      "Deliver findings where the work is (PR comments) and in a machine format (SARIF) so they flow into existing vulnerability management instead of living in a separate tool; and make sure every finding carries clear remediation guidance.",
      "Don't trust a prompt-level \"ask for confirmation\" as human-in-the-loop — verify the executing tool actually routes through approval gates and hooks, or the agent will just confirm to itself."
    ],
    "topics": [
      "ai-security",
      "supply-chain",
      "agent-skills",
      "mcp",
      "ci-cd",
      "code-review",
      "fintech",
      "human-in-the-loop"
    ],
    "tools": [
      "Nubank",
      "Skill Vector (internal)",
      "MCP Vector (internal)",
      "SARIF",
      "MCP servers",
      "internal skill/plugin marketplace"
    ],
    "quotes": [
      {
        "text": "although they look like configuration they behave like supply chain dependence like uh for example libraries and others",
        "at": "01:37",
        "url": "https://www.youtube.com/watch?v=iKQ78wyJEXU&t=97s"
      },
      {
        "text": "we should be protecting the whole workflow not only the code that's being generated",
        "at": "02:03",
        "url": "https://www.youtube.com/watch?v=iKQ78wyJEXU&t=123s"
      },
      {
        "text": "when we do that this first person is guiding over the code that's being generated by the other person and then that can be dangerous",
        "at": "03:36",
        "url": "https://www.youtube.com/watch?v=iKQ78wyJEXU&t=216s"
      },
      {
        "text": "people sometimes will add the instruction like you need to ask for confirmation but the AI may ask confirmation for itself. So from your perspective there is a human in the loop but for the AI perspective there is has been a confirmation and that's okay another has confirmed then let's go",
        "at": "13:03",
        "url": "https://www.youtube.com/watch?v=iKQ78wyJEXU&t=783s"
      }
    ],
    "words": 2487
  },
  "summary_url": "/#iKQ78wyJEXU",
  "transcript": {
    "html": "/transcripts/iKQ78wyJEXU.html",
    "txt": "/transcripts/iKQ78wyJEXU.txt",
    "vtt": "/transcripts/iKQ78wyJEXU.vtt"
  }
}