{
  "video": {
    "id": "s67bE2Ur3bY",
    "title": "Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi",
    "duration": 1149,
    "upload_date": null,
    "channel": "AI Engineer",
    "source": "AI Engineer",
    "event_format": "In person"
  },
  "analysis": {
    "video_id": "s67bE2Ur3bY",
    "title": "Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi",
    "one_liner": "Eight months of running a group-chat/glasses agent called Judith convinced Sai Krishna Rallabandi that single-user agent engineering doesn't transfer: group agents need a guard at the action surface (not the input), memory built from atomic facts, and per-user routing of who hears what.",
    "summary": "Rallabandi argues that we've essentially won at building agents, but every agent we build has 'the customer of size one' — while the next generation will serve a group, live on wearables like glasses, and be on all day. Drawing on Judith, an agent he has run among friends and family for 8 months, he walks through the three things a group harness owes that a personal one doesn't: a security layer (which he names Jatayu) placed at the action boundary rather than the input, a memory layer that extracts and continually rescores atomic facts instead of compacting transcripts, and a routing/privacy layer that decides which member of the group hears a given answer and whether the agent should speak at all. His framing throughout: the surface area an agentic system touches is far richer than an LLM's, and it is amplified in a group.",
    "key_points": [
      "An agent is just a model (the brain), some orchestration harness, and tools — buildable 'in an afternoon', useful 'by the evening'; the interesting question has already moved to group and always-on wearable deployment.",
      "Production examples from Judith (8 months, friends and family): it refused to answer a venue-directions question in the conference group chat and DM'd him instead for privacy; it synced calendars across him and his wife to find a shared event slot; while he was driving it chose to speak through his glasses rather than the car's media system; it curated an evolving multi-week group thread down to what stayed relevant; and it tutors his three-year-old daughter on capitals and numbers while reporting her progress back to the parents.",
      "'We can't secure an agentic system like we secure a large language model' — an LLM hallucination can be inspected at the output, but an agent acts, so its exposed surface is every web page, group message, uploaded screenshot, GitHub issue and promotional email it reads, all day.",
      "Two cited papers make the case that static review is insufficient: a runtime skill audit found code that survives a static scan can still break at runtime, and 'when safe skills collide' showed two individually benign skills — an OCR module and a reporting module, both with pristine static scans — turn malignant together, with the reporter exfiltrating PII to a third party alongside the OCR'd content; around 90% of observed attacks have this shape.",
      "His defence: let the agent read everything, and put a fast deterministic guard at the action surface instead — on reading bash variables, exporting, reading secret config variables — with a naive three-class categorisation (benign/allow, needs approval, block).",
      "Judith goes further with a LoRA fine-tuned SLM trained to separate the data channel from the instruction channel and pass through only benign content, so 'summarize this page for me' still gets done even when the crawled page carries a prompt injection.",
      "Benchmarked on Inspect agent, the naive approach sits at 50% — 'a coin flip' — and the learned model improves on it; the motivating caveat was an attack that intersperses text with dots ('L.I.K.E'), which regex and static approaches miss and a learned model can catch.",
      "On memory: 'the model is just the engine... the memory is what the agent becomes', evolving in real time. Store extracted atomic facts rather than storing everything and compacting; build autoraters for whether high-value atomic facts were extracted, whether they're relevant, hierarchically related, temporally weighted, retrieved correctly, and retrievable intelligently (e.g. via a graph) rather than by crawling everything.",
      "Forgetting should be learned: the paper 'learning what not to forget' trains a relevance scorer that keeps scoring *continually*, because in a group setting different things matter at different times — giving knowledge-based compaction and real token savings. Serving these memories needs a KV-cache-aware injection engine, since KV caches break when you do something cute.",
      "Privacy is a new social contract, not a shared model: a grocery list is benign for everyone, salary or health information is not — 'the data has not changed, the room in which it is deployed has changed'. His proposal, inspired by the human brain, is one common shared memory layer with a different LoRA adapter per user, baking permissions in with machine learning rather than code (reference: the 'user as an engram' paper).",
      "Group-deployed agents tend to be over-proactive and over-articulative when nobody asked them anything; his in-progress fix is a model — possibly as simple as a RoBERTa classifier — that determines role and when the agent is allowed to speak."
    ],
    "takeaways": [
      "Move your guard from the input to the action surface: gating everything the agent reads is impossible and slow, so let it read and instead put fast deterministic checks (regex/traditional NLP) on the actions — secret and bash variable reads, exports — classified into allow / needs-approval / block.",
      "Don't trust per-skill static scans. Audit skills at runtime and in combination, since benign-in-isolation pairs (OCR + reporting) are the dominant attack shape, and expect regex evasions like dot-interspersed text — that's the case for training a small model you can hill-climb against.",
      "Rebuild memory for groups: extract atomic facts instead of storing-and-compacting, write autoraters for extraction quality, relevance, hierarchy, temporal decay and retrieval, and add a continually-updating relevance scorer to decide what to forget. Make the injection engine KV-cache aware.",
      "Treat per-user permissions as a modelling problem: one shared memory layer with a per-user LoRA adapter bakes in who can see what, rather than enforcing it in application code.",
      "Build an explicit routing decision — which channel and which person gets the answer (DM vs group, glasses vs car speakers) — and a separate when-to-speak classifier, because the default failure mode of a group agent is talking too much when unasked."
    ],
    "topics": [
      "agents",
      "group-chat agents",
      "agent memory",
      "prompt injection",
      "agent security",
      "privacy",
      "wearables",
      "lora"
    ],
    "tools": [
      "Judith (the speaker's group agent)",
      "Jatayu (his agent security layer)",
      "Inspect (agent benchmark)",
      "LoRA / supervised fine-tuned SLM",
      "RoBERTa",
      "smart glasses",
      "papers: runtime skill audit, 'when safe skills collide', 'learning what not to forget', 'user as an engram'"
    ],
    "quotes": [
      {
        "text": "almost every agent we build today has the customer of size one",
        "at": "01:31",
        "url": "https://www.youtube.com/watch?v=s67bE2Ur3bY&t=91s"
      },
      {
        "text": "we can't secure an agentic system like we secure a large language model",
        "at": "06:06",
        "url": "https://www.youtube.com/watch?v=s67bE2Ur3bY&t=366s"
      },
      {
        "text": "The memory is what the agent becomes and it's becoming in real time it's evolving in real time.",
        "at": "12:47",
        "url": "https://www.youtube.com/watch?v=s67bE2Ur3bY&t=767s"
      },
      {
        "text": "the data has not changed the room in which it is deployed has changed",
        "at": "17:10",
        "url": "https://www.youtube.com/watch?v=s67bE2Ur3bY&t=1030s"
      }
    ],
    "words": 3536
  },
  "summary_url": "/#s67bE2Ur3bY",
  "transcript": {
    "html": "/transcripts/s67bE2Ur3bY.html",
    "txt": "/transcripts/s67bE2Ur3bY.txt",
    "vtt": "/transcripts/s67bE2Ur3bY.vtt"
  }
}