{ "video": { "id": "vSx5IULvBns", "title": "Always-on agents run production without the on-call tax — Justin Smith, Resolve AI", "duration": 1496, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "In person" }, "analysis": { "video_id": "vSx5IULvBns", "title": "Always-on agents run production without the on-call tax — Justin Smith, Resolve AI", "one_liner": "Justin Smith of Resolve AI argues that since ~70% of an engineer's time goes to running production rather than writing code, the next thing to hand to AI is the operational long tail — background agents that watch deploys, run scheduled health checks, write on-call handoffs and answer Slack questions — and that what makes them work is production context, not execution ability.", "summary": "Smith frames the first wave of AI as a coding-productivity wave whose net effect is bigger PRs, faster change and more production issues — while coding was never the bottleneck. He splits agent work into execution (load the dashboard, call the tool) and production context (know that the metric smells off), and argues the second is where the value is, which is why Resolve invests in a learning/knowledge system that tracks how your environment and its causal chains evolve. The bulk of the talk covers background agents for work that has no page and no incident bridge: deployment monitoring, scheduled health and anomaly checks, operational reports and handoffs, and a Slack first responder. Live demos show an agent spotting a GitHub release tag in Slack and building a release-specific check plan, and an on-call handoff report being permanently shortened by replying to it in-thread.", "key_points": [ "A survey study cited in the talk found 70% of an engineer's time is not spent writing code but running it: maintaining platforms, scaling infra, debugging incidents, being on call, shipping hot fixes, handling alerts, updating runbooks, restoring services and answering other teams — 'coding was never the big bottleneck.'", "The AI coding wave makes this worse: more and larger PRs, including from non-developers who don't know the code or its operating principles, landing in systems whose structures weren't built for that change rate — while 'unlimited tokens' ends, prices rise and companies clamp down on AI spend.", "A task = execution + production context, and production context is the harder, more valuable half: execution can load the dashboard; context is what says 'this feels wrong' and digs to the next layer. Models got very capable over the last ~6 months, so the gap is a learning system that captures how your services interact, where the hotspots are, and how that changes as the system evolves.", "Background agents are defined along three axes: when they run (schedule, recurring events like a Thursday on-call handover, event streams such as CI/CD deploys or Slack messages, or direct instruction), how they run (always-on in the cloud so closing your laptop is fine, in a sandbox with a filesystem, feeding a shared knowledge/memory loop across tasks), and how they know what to do (a task system, imported skills, integrations).", "Four proven workloads: deployment monitoring, scheduled health/anomaly checks, operational reports and handoffs, and a first responder to engineering questions.", "Deployment-monitoring gap: CI/CD checks are good baselines but not exhaustive, every rollout is unique, and change systems like feature flags and infra changes often bypass CI/CD entirely and get no monitoring — you just trust an alert fires and an on-caller asks who changed what.", "Demo: an agent sees a GitHub release tag posted in Slack, reads what actually changed, and builds a customized check plan for that release — for a checkout service replacing a currency service it watches checkout latency and error rates plus the Kafka pipeline as part of the causal chain. Nothing is hard-coded: it can decide to wait an hour, or come back in 3 days to ask whether the deploy is still healthy and whether the expected effect showed up. Explicitly not a CI/CD replacement, but a patch for the less robust parts.", "Demo: an over-long on-call handoff digest was fixed by replying 'this is too verbose, make it shorter' in the Slack thread — the agent updates its underlying task so the next firing is shorter. A separate Slack agent passively watches channels with no @mention, decides whether it has enough confidence to answer, and can DM a human to confirm an answer before posting — emergent behavior Smith calls fun.", "Everything shown is reachable over MCP servers, so Resolve can be grafted into your own agent harness for production context or the learning layer, and you can bring your own skills rather than duplicating them." ], "takeaways": [ "Audit the operational long tail that has no page and no incident bridge — deploy watching, morning digests, a P99 drift nobody owns, capacity reports, recurring health checks — and hand those to scheduled or event-triggered agents instead of absorbing them as unwritten job description.", "Cover the change paths your CI/CD pipeline never sees: feature flags and infra changes are real deploys with no rollout monitoring. Point an agent at those event streams rather than extending the pipeline.", "Make deploy checks change-aware rather than fixed: derive which telemetry and which downstream causal chain to watch from what the release actually changed, and keep watching on a variable horizon (an hour, three days) instead of a hard-coded 15-minute window.", "Put the agent where people already work — Slack or MS Teams as a first-party surface, not a dashboard you remember to open — and let people correct its tasks conversationally in-thread so the fix persists to the next run.", "Invest in the knowledge/context layer over raw execution: build or buy a system that keeps learning your environment as it evolves, because that's what makes an agent able to tell important from unimportant." ], "topics": [ "agents", "background-agents", "observability", "incident-response", "on-call", "devops", "mcp", "production-context" ], "tools": [ "Resolve AI", "Splunk", "VMware", "Granola", "Slack", "Microsoft Teams", "GitHub", "Kafka", "MCP (Model Context Protocol) servers" ], "quotes": [ { "text": "So, really coding was never the the the big bottleneck, right? A lot of it was really around like how do we actually run these things sort of in production.", "at": "02:22", "url": "https://www.youtube.com/watch?v=vSx5IULvBns&t=142s" }, { "text": "It's one thing to go check a dashboard. It's another thing to say that metric smells off. And the execution is can load the dashboard. It's the production context that's going to say, this feels wrong.", "at": "10:41", "url": "https://www.youtube.com/watch?v=vSx5IULvBns&t=641s" }, { "text": "Often times you have change systems that you're not piping through a CICD system like a feature flag or maybe some infra changes that might happen which maybe don't get any monitoring at all. And you're sort of just trusting that an alert might fire and an on-caller will wake up and say, \"Who changed what?\"", "at": "14:35", "url": "https://www.youtube.com/watch?v=vSx5IULvBns&t=875s" }, { "text": "It would be great if you had a single engineer just focused on like watching all the things on every release, but that's really expensive. There's a lot of cognitive load. You'd rather have them doing other things.", "at": "18:41", "url": "https://www.youtube.com/watch?v=vSx5IULvBns&t=1121s" } ], "words": 5970 }, "summary_url": "/#vSx5IULvBns", "transcript": { "html": "/transcripts/vSx5IULvBns.html", "txt": "/transcripts/vSx5IULvBns.txt", "vtt": "/transcripts/vSx5IULvBns.vtt" } }