WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7P0elyLIxXo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7P0elyLIxXo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:01.880
An agent opens a pull request.

s2
00:00:01.880 --> 00:00:03.080
It passes the tests.

s3
00:00:03.080 --> 00:00:04.640
It updates the documentation.

s4
00:00:04.640 --> 00:00:07.640
It closes the issue and comments, "Looks done to me."

s5
00:00:07.640 --> 00:00:09.360
But is it actually done?

s6
00:00:09.360 --> 00:00:10.920
Is it done enough to merge?

s7
00:00:10.920 --> 00:00:12.600
Is it done enough to deploy?

s8
00:00:12.600 --> 00:00:15.000
Is it done enough to announce to your customers?

s9
00:00:15.000 --> 00:00:21.480
These are fundamentally different operational claims, and most agent systems just flatten it to a single green check mark.

s10
00:00:21.480 --> 00:00:22.040
I'm Dota.

s11
00:00:22.040 --> 00:00:28.680
I'm the creator of Paperclip, and I'm going to give you some hard-earned lessons that we've learned in creating Paperclip's liveness model.

s12
00:00:28.680 --> 00:00:30.440
What does done even mean?

s13
00:00:30.440 --> 00:00:31.040
Here's the thing.

s14
00:00:31.040 --> 00:00:38.920
Programming is solved, and agents can now produce more code and documentation faster than any human can ever verify.

s15
00:00:38.920 --> 00:00:45.600
And this actually gives us a new failure mode, is that agents can actually create more work than humans have time to verify.

s16
00:00:45.600 --> 00:00:50.160
So we need a way to verify that our agents are done more than just letting them check a checkbox.

s17
00:00:50.160 --> 00:00:54.880
Done doesn't mean that an agent just changed the status of a task being done.

s18
00:00:54.880 --> 00:00:57.920
Saying that something is done is actually a bundle of claims.

s19
00:00:57.920 --> 00:01:07.640
You're saying that an artifact was produced, that you have evidence that the task is actually complete, and you have a rubric in which you can verify against.

s20
00:01:07.640 --> 00:01:14.040
You know exactly who the owner is for the next step, and you know exactly what the next step is.

s21
00:01:14.040 --> 00:01:16.520
There's different levels to how done something is.

s22
00:01:16.520 --> 00:01:23.680
The producer might claim something is complete, but you need to have a reviewer, another party that looks at it and finds no obvious issues.

s23
00:01:23.680 --> 00:01:28.840
You want to verify and make sure that the evidence actually meets a specified standard.

s24
00:01:28.840 --> 00:01:34.520
You want to make sure that a person who is authorized to approve it actually approves that the work is done.

s25
00:01:34.520 --> 00:01:37.960
And you want to make sure that there's somebody who actually stands behind the decision.

s26
00:01:37.960 --> 00:01:44.880
And ideally, what you want is that the outcome has actually survived real-world conditions.

s27
00:01:44.880 --> 00:01:49.680
Because exhaustive human verification fails at high volume.

s28
00:01:49.680 --> 00:02:03.600
You might be able to verify a few tasks per day, but essentially, if you have humans verifying all the tasks and they have to sign off on it, you eventually what you just get is a form of verification theater.

s29
00:02:03.600 --> 00:02:09.840
What you need is a protocol for defining how tasks actually progress through a system.

s30
00:02:09.840 --> 00:02:16.120
You want to make sure that tasks are always kept moving, but they don't get stuck into invalid states.

s31
00:02:16.120 --> 00:02:30.160
You need a control plane that actually has the execution of the tasks being tied to specific contracts and constraints about what the system will do and what agents it will hand off your next task to.

s32
00:02:30.160 --> 00:02:37.520
Because really what you're trying to play against is this idea around keeping work moving, but also having it verified.

s33
00:02:37.520 --> 00:02:41.800
When a task has been reviewed by a human, you get the assurance that it's correct.

s34
00:02:41.800 --> 00:02:46.640
But having a human verify it means that the task is dead in its tracks.

s35
00:02:46.640 --> 00:02:48.320
You also want to keep liveliness.

s36
00:02:48.320 --> 00:02:52.360
Liveliness means that the work is continuing with no blockers.

s37
00:02:52.360 --> 00:02:54.680
And you're always trying to keep these two things in balance.

s38
00:02:54.680 --> 00:03:04.800
If you have tasks that are completely alive with no approvals, then what you get is this classic AI slop because you're producing a lot of things with kind of no quality control

s39
00:03:04.800 --> 00:03:08.680
and it's worse than creating nothing after a long period of time.

s40
00:03:08.680 --> 00:03:17.400
But if you have peer review, um then you have this enormous review queue where humans can't actually review it by hand anyway.

s41
00:03:17.400 --> 00:03:27.600
These agents will be creating far more than you can ever actually review, and so we have to find a way to tease apart the bundle of claims that are involved in saying a task is done.

s42
00:03:27.600 --> 00:03:31.840
With Paperclip, we have a number of mechanisms to keep this going.

s43
00:03:31.840 --> 00:03:39.000
You might think that you can easily just write a for loop over your task manager and have your agents work, but quickly you'll find that falls apart.

s44
00:03:39.000 --> 00:03:55.360
As soon as you start integrating task dependency trees, blockers, multiple agents, item potent checkouts, like locks on checkouts, you find that this tension between liveliness and verification actually gets quite complicated.

s45
00:03:55.360 --> 00:04:03.520
There's really three invariants that are extremely important when you're thinking about what you want out of a control plane for your agentic work.

s46
00:04:03.520 --> 00:04:05.960
You want to ensure that productive work continues.

s47
00:04:05.960 --> 00:04:09.240
You want to make sure that only real blockers stop work.

s48
00:04:09.240 --> 00:04:11.760
And you want to make sure that infinite loops are bounded.

s49
00:04:11.760 --> 00:04:16.920
In Paperclip, we have built a number of mechanisms to deal with this problem.

s50
00:04:16.920 --> 00:04:23.000
So, for example, every time you have a task, there's clear transitions to what the next state could be.

s51
00:04:23.000 --> 00:04:28.480
We have first-class blockers between tasks, and the control plane enforces those blockers.

s52
00:04:28.480 --> 00:04:33.800
We have moments of interactive human approval, where human choices leave an audit trail.

s53
00:04:33.800 --> 00:04:41.480
You can set reviewers and approvers on tasks explicitly, meaning when this task completes, another agent can review it.

s54
00:04:41.480 --> 00:04:50.720
We also have the idea of watchdogs, which is this maximizer mode, which says, um, try as hard as you can to make sure that this happens.

s55
00:04:50.720 --> 00:04:58.800
When you have a watchdog, it's another agent, um, who is given a goal, and it enforces that all of your agents continue to work until that goal has been achieved.

s56
00:04:58.800 --> 00:05:04.560
The important thing here is that the watchdog within Paperclip is harness agnostic.

s57
00:05:04.560 --> 00:05:15.160
You can use it with Pi, OpenGL, Hermes, Claude Code, Codex, whatever you're using, you have one consistent interface for ensuring that goal is complete.

s58
00:05:15.160 --> 00:05:21.760
So, one of the best pieces of advice we have is that you stop treating done as a Boolean and treat it more like an object.

s59
00:05:21.760 --> 00:05:23.680
This isn't specific to Paperclip.

s60
00:05:23.680 --> 00:05:27.040
It's just advice on how you think about what is done.

s61
00:05:27.040 --> 00:05:36.760
Humans automatically paper over these details, but when we're building agentic systems, it's important that your agents can distinguish between the different pieces of what they're claiming when they say something is done.

s62
00:05:36.760 --> 00:05:47.360
The artifact that they're saying is complete, the scope, the rubric or the standard, the evidence that it's done, who verified the work, who has the authority to sign off on the work,

s63
00:05:47.360 --> 00:05:48.920
And what risk might be left?

s64
00:05:48.920 --> 00:05:51.600
And really, what's the next action going to be?

s65
00:05:51.600 --> 00:05:56.240
So, you want to make sure that when you define done, it's not just a checkbox.

s66
00:05:56.240 --> 00:06:00.480
So, if you want to get 100 times more work done, you should steal this checklist.

s67
00:06:00.480 --> 00:06:05.320
You need to define exactly what does done mean for this task.

s68
00:06:05.320 --> 00:06:08.440
You definitely want to separate the verifier from the author.

s69
00:06:08.440 --> 00:06:10.160
Often, this means you're using a different model.

s70
00:06:10.160 --> 00:06:14.040
So, if you're coding using Claude, have Codex verify.

s71
00:06:14.040 --> 00:06:16.640
You want to ask your agents to provide evidence.

s72
00:06:16.640 --> 00:06:19.080
Don't just ask them to say, "Is this done?"

s73
00:06:19.080 --> 00:06:22.680
But, give them the tools they need to verify that the work is done.

s74
00:06:22.680 --> 00:06:25.520
Write the code to have the custom browser harness.

s75
00:06:25.520 --> 00:06:27.640
Write the code to take the screenshots.

s76
00:06:27.640 --> 00:06:29.800
Make sure they have access to a browser.

s77
00:06:29.800 --> 00:06:39.880
Make sure that they have custom agent hooks or custom agent tooling to actually run through and click the buttons and try it out themselves and verify that the work is truly done.

s78
00:06:39.880 --> 00:06:48.760
Make sure you have a clear chain of custody, that every agent knows that as soon as they're done, who they're supposed to give the work to next.

s79
00:06:48.760 --> 00:06:54.560
It can be easy to just fire off a single-line instruction and vibe with whatever comes back.

s80
00:06:54.560 --> 00:07:04.080
But, if you have serious work that you're accountable for, it's very important that you define what done really means in as much detail as possible, and that you have a structure

s81
00:07:04.080 --> 00:07:11.200
for your agents, so that way they can verify that all of the claims that are involved with something being done are actually met.

s82
00:07:11.200 --> 00:07:13.360
Thank you.