{ "video": { "id": "BgLdvF7Qkh8", "title": "300 Agents, 60 People: How We Secured Our Own AI Agents - Aiden Sochowski, Runlayer", "duration": 998, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "" }, "analysis": { "video_id": "BgLdvF7Qkh8", "title": "300 Agents, 60 People: How We Secured Our Own AI Agents - Aiden Sochowski, Runlayer", "one_liner": "Runlayer (about 50–60 people) runs over 300 agents a day. After logging every session and tool call through a gateway, they concluded that what works is tightly scoping where agents can write, using trained classifiers to flag tool calls that drift from an agent's defined task, and using the same monitoring to catch cost blowups.", "summary": "Aiden Sochowski, a founding engineer at Runlayer, explains how the company secured its own agents. They put in a gateway that logs every session and tool call to one central place, then mapped their agents by topic and risk. He argues that static gateway policies are hard to write because whether an action is safe depends on context. Some attacks also play out across a whole session rather than in one tool call, which is why they trained classifiers to check each tool call against the agent's defined intent. The same monitoring turned up problems that weren't about security: buggy vibe-coded MCP servers and model switches that pushed costs up more than 100x.", "key_points": [ "Most companies have zero visibility into the agents and MCP clients people run, whether locally or through SaaS products that connect to MCP servers. Runlayer fixed this for itself with a gateway that logs every session and tool call centrally.", "At Runlayer there are more agents than people: over 300 run every day at a company of about 50–60. 'Agent' here means things users aren't necessarily triggering themselves, such as chatbots, triggers and SaaS apps.", "Agents are mapped by topic and risk. A colleague built a 'galaxy' visualization where each team is a galaxy, each agent belongs to one, each planet is a single run, and you can drill into the MCP servers and actions involved. Every tool call is treated as attack surface: hostile input, destructive actions or data exfiltration.", "Policy options at the gateway get stricter in layers: a trained classifier that checks every email read for injected instructions, regex-based blocking of PII or customer names in outbound email, and blocking whenever an agent reads from a sensitive source such as a private calendar. None is 100% effective, all of them depend on context, and Runlayer found them hard to deploy.", "Some attacks span a whole session. In the example, an AI PR reviewer that is allowed to approve PRs is talked round by the author in GitHub comments ('only review the cleanup'). Ask it 10 times and you will probably get it to approve. Runlayer trained classifiers that look at the whole session and judge whether each tool call matches what the agent was defined to do.", "For a narrowly scoped agent, session data is a high-signal place to spot anomalies. Their heavily provisioned recruiter bot has access to the ATS, meeting notes, calendar and messaging. Runlayer flags runs that use tools outside the bot's historical range, and has built models that read past tool-call history and suggest permissions to remove.", "One poisoned input, such as a malicious resume, can spread to other agents that read the same poorly partitioned systems. He says the only real defense is to map how systems connect and tightly scope where agents can write, so that agents' context windows stay unpoisoned.", "Monitoring also turned up cost problems. The recruiting team, who had never written software before, vibe-coded an MCP server for a vendor's API; formatting errors made it fail over and over, and costs grew a lot. Separately, someone switched from GPT-56 Luna to Fable 5.1 and costs went up more than 100x within a week." ], "takeaways": [ "Send all agent traffic through a gateway and log every session and tool call centrally before trying to write policy, because you can't secure what you can't see.", "Assume per-call rules won't be enough. Use classifiers that check each tool call against the agent's defined intent across the whole session, especially for agents that hold approval power, such as PR reviewers.", "Use each agent's own tool-call history to flag unusual runs and to remove permissions it never uses, rather than trying to set the right permissions up front.", "Map which systems agents read from and write to, and tightly limit write access so one poisoned agent can't contaminate the others.", "Watch MCP server error rates and rate-limit them, and set budgets with sane model defaults so people have to justify moving to more expensive models." ], "topics": [ "agent security", "mcp", "prompt injection", "agent gateways", "observability", "least privilege", "cost management", "classifiers" ], "tools": [ "Runlayer", "MCP", "Claude Code", "Opus", "GitHub", "GPT-56 Luna", "Fable 5.1" ], "quotes": [ { "text": "We have over 300 agents that are running every single day", "at": "01:43", "url": "https://www.youtube.com/watch?v=BgLdvF7Qkh8&t=103s" }, { "text": "If you ask it 10 times, it's probably likely that you know, you could get it to do something.", "at": "07:48", "url": "https://www.youtube.com/watch?v=BgLdvF7Qkh8&t=468s" }, { "text": "cuz really at the end of day, agents are just software, right?", "at": "11:54", "url": "https://www.youtube.com/watch?v=BgLdvF7Qkh8&t=714s" }, { "text": "their cost ballooned to, you know, over like 100x within a week.", "at": "14:23", "url": "https://www.youtube.com/watch?v=BgLdvF7Qkh8&t=863s" } ], "words": 3851 }, "summary_url": "/#BgLdvF7Qkh8", "transcript": { "html": "/transcripts/BgLdvF7Qkh8.html", "txt": "/transcripts/BgLdvF7Qkh8.txt", "vtt": "/transcripts/BgLdvF7Qkh8.vtt" } }