WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=4kYl2_mqmnQ

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/4kYl2_mqmnQ.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:01.040
Hello, I'm Kyle.

s2
00:00:01.040 --> 00:00:08.320
I run a fleet of AI coding agents across three machines every day, and this is the story of what broke, starting with me.

s3
00:00:08.320 --> 00:00:12.040
This is my fleet, three machines I run every single day.

s4
00:00:12.040 --> 00:00:18.640
The MacBook is where I do my heavy coding and my personal projects, and it sleeps because, well, it's a laptop.

s5
00:00:18.640 --> 00:00:21.960
Then two Linux boxes, both headless and always on.

s6
00:00:21.960 --> 00:00:24.960
Linux A takes the long-running coding tasks.

s7
00:00:24.960 --> 00:00:28.240
Linux B runs my short-lived personal side projects.

s8
00:00:28.240 --> 00:00:31.960
Two operating systems, one control plane tying it all together.

s9
00:00:31.960 --> 00:00:35.280
And to be clear, this isn't a demo I spun up for the talk.

s10
00:00:35.280 --> 00:00:39.000
This is what I actually use all day, every day.

s11
00:00:39.000 --> 00:00:42.040
And here it is live, the whole thing running on one screen.

s12
00:00:42.040 --> 00:00:43.480
Real agents, real work.

s13
00:00:43.480 --> 00:00:45.760
This is a normal Tuesday for me.

s14
00:00:45.760 --> 00:00:49.880
So, let's go back to the beginning because before any machine broke, I broke.

s15
00:00:49.880 --> 00:00:53.080
When I started, it was just me and a terminal.

s16
00:00:53.080 --> 00:00:56.800
A few tmux panes, a few agents, then a few more.

s17
00:00:56.800 --> 00:01:02.680
And really fast, I'm sitting in front of four, five, six live contexts at the same time.

s18
00:01:02.680 --> 00:01:04.640
And here's what nobody warns you about.

s19
00:01:04.640 --> 00:01:07.040
At that point, I'm not running agents anymore.

s20
00:01:07.040 --> 00:01:09.760
I've become the scheduler, deciding who does what.

s21
00:01:09.760 --> 00:01:14.760
I'm the memory, holding what every one of them is doing, and I'm the reviewer, checking all of it.

s22
00:01:14.760 --> 00:01:17.480
One human, three roles, six contexts.

s23
00:01:17.480 --> 00:01:19.040
It does not scale.

s24
00:01:19.040 --> 00:01:22.520
This This is the mess that broke me.

s25
00:01:22.520 --> 00:01:25.800
I just couldn't hold what six agents were doing at once.

s26
00:01:25.800 --> 00:01:28.480
My own attention was the bottleneck.

s27
00:01:28.480 --> 00:01:30.920
So, I asked myself kind of a strange question.

s28
00:01:30.920 --> 00:01:34.480
How does a handful of executives run a company of thousands of people?

s29
00:01:34.480 --> 00:01:36.440
They don't hold all of it in their heads.

s30
00:01:36.440 --> 00:01:37.880
They separate context.

s31
00:01:37.880 --> 00:01:40.480
Each person only ever sees their own slice.

s32
00:01:40.480 --> 00:01:41.680
And that was the unlock.

s33
00:01:41.680 --> 00:01:43.880
What if my agents weren't a flat pile?

s34
00:01:43.880 --> 00:01:46.240
What if they were an organization?

s35
00:01:46.240 --> 00:01:50.600
So, that's what I built, a hierarchy, CEO, VP, manager, worker.

s36
00:01:50.600 --> 00:01:51.800
And please hear me on this.

s37
00:01:51.800 --> 00:01:54.000
These are real entity types in the system.

s38
00:01:54.000 --> 00:01:55.480
It is not a cute metaphor.

s39
00:01:55.480 --> 00:02:00.560
Each one is its own agent with its own scoped context and its own approval boundary.

s40
00:02:00.560 --> 00:02:02.120
Context flows down.

s41
00:02:02.120 --> 00:02:04.640
Each layer only gets the slice it needs.

s42
00:02:04.640 --> 00:02:08.399
Results flow back up and I only review what reaches the very top.

s43
00:02:08.399 --> 00:02:12.720
So, instead of holding six contexts in my head, I hold exactly one.

s44
00:02:12.720 --> 00:02:16.080
Next problem, where does an agent's state actually live?

s45
00:02:16.080 --> 00:02:20.360
Normally, it lives inside the model's context window and that window fills up.

s46
00:02:20.360 --> 00:02:21.400
So, I moved it out.

s47
00:02:21.400 --> 00:02:24.120
Every entity gets its own workspace on disk.

s48
00:02:24.120 --> 00:02:28.120
Shared context that everyone has to honor lives in shared.

s49
00:02:28.120 --> 00:02:30.959
Machine-bound state lives under machines.

s50
00:02:30.959 --> 00:02:38.600
And inside each workspace, the mission, the current status, and a handoff folder, the actual work product that gets passed along.

s51
00:02:38.600 --> 00:02:40.160
The state lives in files.

s52
00:02:40.160 --> 00:02:42.959
It is not trapped inside one model.

s53
00:02:42.959 --> 00:02:46.400
And this is the single most practical thing I learned all year.

s54
00:02:46.400 --> 00:02:52.160
When that window fills up, the built-in move is to compact, summarize the history, make room.

s55
00:02:52.160 --> 00:02:53.240
I stopped doing it.

s56
00:02:53.240 --> 00:02:54.040
It's slow.

s57
00:02:54.040 --> 00:02:55.800
I can't choose what survives.

s58
00:02:55.800 --> 00:02:57.800
And whatever it throws away is just gone.

s59
00:02:57.800 --> 00:03:00.959
So, instead, I don't compact, I reset.

s60
00:03:00.959 --> 00:03:05.920
And by reset, I mean right inside Claude, I clear the context completely.

s61
00:03:05.920 --> 00:03:13.040
And then it just reads back the handoff and the history files it wrote for itself and picks up exactly where it left off.

s62
00:03:13.040 --> 00:03:20.239
The context can get wiped, the machine can even crash, and the work still survives because it was never only in the model.

s63
00:03:20.239 --> 00:03:22.880
But there was still one problem, alignment.

s64
00:03:22.880 --> 00:03:26.040
Plans flow down the hierarchy, but they drift.

s65
00:03:26.040 --> 00:03:29.280
And I'd only catch it by walking into each pane by hand.

s66
00:03:29.280 --> 00:03:31.000
So, I built a review gateway.

s67
00:03:31.000 --> 00:03:34.880
Any layer that wants to act submits its plan and then it blocks.

s68
00:03:34.880 --> 00:03:35.720
It waits.

s69
00:03:35.720 --> 00:03:37.320
Nothing runs until I approve.

s70
00:03:37.320 --> 00:03:40.920
And the second I approve, a hook fires the work off automatically.

s71
00:03:40.920 --> 00:03:45.720
One web inbox, one control point, I never walk into the work windows anymore.

s72
00:03:45.720 --> 00:03:47.160
Here's one full cycle.

s73
00:03:47.160 --> 00:03:50.519
A plan comes in, I approve, the work resumes.

s74
00:03:50.519 --> 00:03:51.680
That's the whole loop.

s75
00:03:51.680 --> 00:03:54.680
And honestly, I didn't build this gateway by hand.

s76
00:03:54.680 --> 00:03:57.000
An infra team inside the fleet built it.

s77
00:03:57.000 --> 00:03:59.920
Agents building the tools that run the agents.

s78
00:03:59.920 --> 00:04:02.280
I think that's kind of the whole point.

s79
00:04:02.280 --> 00:04:05.240
So on one machine, all of this works beautifully.

s80
00:04:05.240 --> 00:04:06.400
And then it didn't.

s81
00:04:06.400 --> 00:04:08.800
Then one machine wasn't enough.

s82
00:04:08.800 --> 00:04:10.040
Five things broke.

s83
00:04:10.040 --> 00:04:10.959
Failure.

s84
00:04:10.959 --> 00:04:15.959
One, my agents kept doing the work themselves instead of dispatching it down to a worker.

s85
00:04:15.959 --> 00:04:18.160
The orchestrator is supposed to delegate.

s86
00:04:18.160 --> 00:04:21.680
Instead, it just rolls up its sleeves and does the task itself.

s87
00:04:21.680 --> 00:04:22.440
Wrong.

s88
00:04:22.440 --> 00:04:24.000
So I forced its hand.

s89
00:04:24.000 --> 00:04:27.520
A CLI harness with skills that call those CLIs.

s90
00:04:27.520 --> 00:04:30.720
So dispatching becomes the only path it can take.

s91
00:04:30.720 --> 00:04:33.560
Failure two, and this one's almost funny.

s92
00:04:33.560 --> 00:04:44.080
As I kept throwing more tasks at my managers, they kept spinning up more and more worker panes inside a single window until it was so crowded the panes were too small to read.

s93
00:04:44.080 --> 00:04:51.320
Even tmux capture pane, the thing I used to read a pane programmatically, couldn't pull anything meaningful out of them anymore.

s94
00:04:51.320 --> 00:04:55.040
I'd literally run out of room to see what my own fleet was doing.

s95
00:04:55.040 --> 00:04:56.880
Failure three, out of memory.

s96
00:04:56.880 --> 00:04:57.640
Look at this.

s97
00:04:57.640 --> 00:05:02.720
That's my activity monitor, completely buried under cloud code and MCP processes.

s98
00:05:02.720 --> 00:05:03.880
Swap almost full.

s99
00:05:03.880 --> 00:05:05.640
Basically no free memory left.

s100
00:05:05.640 --> 00:05:09.240
The sessions just stacked up until the machine couldn't breathe.

s101
00:05:09.240 --> 00:05:11.480
Failure four, get credentials.

s102
00:05:11.480 --> 00:05:13.240
The expectation is clean.

s103
00:05:13.240 --> 00:05:18.480
Credential A to workspace A, credential B to workspace B, one to one.

s104
00:05:18.480 --> 00:05:23.160
The reality, they collided, crossed over, bound to the wrong workspaces.

s105
00:05:23.160 --> 00:05:27.800
The fix was a clean, fully separated environment for each workspace.

s106
00:05:27.800 --> 00:05:31.480
And failure five, the one that really hurt.

s107
00:05:31.480 --> 00:05:38.720
The MacBook is a laptop, so the moment it loses power or drops off the network, every in progress job dies with it.

s108
00:05:38.720 --> 00:05:43.440
And one time I'd pushed so many workers onto it that the whole machine just gave out.

s109
00:05:43.440 --> 00:05:48.000
I came back later, opened the lid, and it had already restarted itself.

s110
00:05:48.000 --> 00:05:50.120
Everything in flight just gone.

s111
00:05:50.120 --> 00:05:59.240
So, the very first thing I did was build a boot command, one overlord boot, and the whole fleet comes straight back up because all the state was sitting in files.

s112
00:05:59.240 --> 00:06:04.520
But a machine that can just vanish on you like that, that's when I knew one machine wasn't going to cut it.

s113
00:06:04.520 --> 00:06:07.280
Every single one of these pointed at the same answer.

s114
00:06:07.280 --> 00:06:09.200
So, I added machines.

s115
00:06:09.200 --> 00:06:10.080
Here's the split.

s116
00:06:10.080 --> 00:06:14.160
The MacBook was carrying everything, heavy coding and my personal projects.

s117
00:06:14.160 --> 00:06:15.240
So, I offloaded it.

s118
00:06:15.240 --> 00:06:21.320
Long-running coding tasks went to Linux A, short-lived personal projects went to Linux B, both always on.

s119
00:06:21.320 --> 00:06:27.880
More resources, the long-running work finally off the laptop, and still one control plane over all of it.

s120
00:06:27.880 --> 00:06:31.120
But now I've got context on one machine that I need on another.

s121
00:06:31.120 --> 00:06:32.120
How do I move it?

s122
00:06:32.120 --> 00:06:32.560
Git.

s123
00:06:32.560 --> 00:06:34.560
I commit the context files and push.

s124
00:06:34.560 --> 00:06:39.120
Then I poke the other machine with tmux send-keys over SSH and tell it to pull.

s125
00:06:39.120 --> 00:06:44.400
It pulls, the agent there reads the files, and picks up exactly where the first one left off.

s126
00:06:44.400 --> 00:06:45.919
Of course, it wasn't that clean.

s127
00:06:45.919 --> 00:06:50.280
When two machines point at the same directory and both change it, you get conflicts.

s128
00:06:50.280 --> 00:06:53.720
The same context quietly diverged, so I separated it.

s129
00:06:53.720 --> 00:07:00.000
Per-machine directories for machine-specific state, and the shared stuff only changes through a pull request.

s130
00:07:00.000 --> 00:07:04.840
It's boring, but boring is what stops the two machines from silently disagreeing.

s131
00:07:04.840 --> 00:07:07.200
Then the gateway came back to bite me.

s132
00:07:07.200 --> 00:07:12.320
I now had one on every machine, so I was checking multiple inboxes again.

s133
00:07:12.320 --> 00:07:13.600
So, I collapsed them.

s134
00:07:13.600 --> 00:07:23.760
Every machine sends its review requests over SSH into one main gateway, and that main one lives on an always-on Linux box because, remember, the Mac sleeps.

s135
00:07:23.760 --> 00:07:27.280
Your one point of control can't be a thing that falls asleep.

s136
00:07:27.280 --> 00:07:30.800
And then, my absolute favorite failure of the whole project.

s137
00:07:30.800 --> 00:07:40.560
I was developing across all of these, bouncing back and forth, and at one point I went looking for a feature I'd built, and I genuinely could not remember which machine I'd built it on.

s138
00:07:40.560 --> 00:07:42.080
And that's when it clicked.

s139
00:07:42.080 --> 00:07:44.080
I need one place to root from.

s140
00:07:44.080 --> 00:07:46.320
So, Discord became my single router.

s141
00:07:46.320 --> 00:07:47.640
One bot per machine.

s142
00:07:47.640 --> 00:07:53.320
Mac, Linux A, Linux B. My phone is now the remote control for the entire fleet.

s143
00:07:53.320 --> 00:07:54.400
So, where am I now?

s144
00:07:54.400 --> 00:07:56.880
Honestly, with a pile of things I have not solved.

s145
00:07:56.880 --> 00:07:57.560
Four of them.

s146
00:07:57.560 --> 00:08:08.880
Consistency across machines, abstracting away the local only tools that are still stuck on my Mac, the MCP servers, the browser, secure credential handoff between instances, and resource management.

s147
00:08:08.880 --> 00:08:12.520
I really want to stop being the one who decides what runs where.

s148
00:08:12.520 --> 00:08:16.919
And when I lined those four up, I realized these aren't new questions at all.

s149
00:08:16.919 --> 00:08:20.440
An agent should just declare what it needs, not where it runs.

s150
00:08:20.440 --> 00:08:24.640
Above it sits the orchestrator, the review gate, the logical hierarchy.

s151
00:08:24.640 --> 00:08:27.440
Below it, compute, secrets, tools.

s152
00:08:27.440 --> 00:08:31.480
A scheduler places it, and the machines just disappear underneath.

s153
00:08:31.480 --> 00:08:35.400
These are the exact questions Kubernetes already answers.

s154
00:08:35.400 --> 00:08:36.760
So, that's where I'm headed.

s155
00:08:36.760 --> 00:08:39.960
I'm not going to reinvent compute, secrets, and tools.

s156
00:08:39.960 --> 00:08:41.560
Kubernetes already nailed those.

s157
00:08:41.560 --> 00:08:45.480
I'm stacking them underneath, and building my orchestration manager on top.

s158
00:08:45.480 --> 00:08:48.920
Task orchestration, review flow, context management.

s159
00:08:48.920 --> 00:08:52.920
Reuse what exists, build the new part on tops.

s160
00:08:52.920 --> 00:08:54.440
And that's the honest state of it.

s161
00:08:54.440 --> 00:08:56.160
One machine, I solved.

s162
00:08:56.160 --> 00:08:58.800
Across machines, still rough, still building.

s163
00:08:58.800 --> 00:09:05.400
If you're running agents at any kind of scale, I'd genuinely love to compare notes, because this is the part nobody has figured out yet.

s164
00:09:05.400 --> 00:09:08.800
I'm on LinkedIn and on X. Links are in the description.

s165
00:09:08.800 --> 00:09:11.280
Thank you so much.