WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=dYh7TQkuYFE

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/dYh7TQkuYFE.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:00.000 --> 00:00:03.440
I believe everything my agents tell me.

s2
00:00:03.440 --> 00:00:07.800
And yet, my agents get things wrong all the time.

s3
00:00:07.800 --> 00:00:11.920
I've put over 107 billion tokens through agents this year.

s4
00:00:11.920 --> 00:00:17.600
So, I analyzed my agent sessions from 2026 year-to-date to see where they went wrong.

s5
00:00:17.600 --> 00:00:23.320
In about four in 10 sessions, an agent got something wrong that mattered.

s6
00:00:23.320 --> 00:00:26.240
That's how often agents get things wrong in the first place.

s7
00:00:26.240 --> 00:00:30.000
It isn't how often those errors made it into work outputs.

s8
00:00:30.000 --> 00:00:40.240
Catching those errors before they reach me is the job of the agent-based review method that I've been running for most of this year, and that's the method that I'm sharing with you today.

s9
00:00:40.280 --> 00:00:44.400
This analysis identified three ways agents keep failing.

s10
00:00:44.400 --> 00:00:47.360
Number one, they overstate what they did.

s11
00:00:47.360 --> 00:00:51.360
That is, they say it's done or checked when it isn't.

s12
00:00:51.360 --> 00:00:53.880
Number two, they get the facts wrong.

s13
00:00:53.880 --> 00:00:56.760
They misread the source, for example.

s14
00:00:56.760 --> 00:00:59.880
Or number three, they get the reasoning wrong.

s15
00:00:59.880 --> 00:01:04.120
So, the facts are right, but the conclusion isn't.

s16
00:01:05.760 --> 00:01:12.680
The method is comprised of three levels of review, one to address each of the three ways that agents keep failing.

s17
00:01:12.680 --> 00:01:15.560
I built each level as an agent skill.

s18
00:01:15.560 --> 00:01:22.240
Here's how each one works, and then I'll show you how each was applied in three real cases.

s19
00:01:22.240 --> 00:01:26.720
Level one prevents agents overstating what they did.

s20
00:01:26.720 --> 00:01:32.040
Before an agent starts, the job spells out exactly what it has to hand back.

s21
00:01:32.040 --> 00:01:36.760
The file, where it goes, and a command that checks it.

s22
00:01:36.760 --> 00:01:41.960
It points out the shortcuts agents have taken before, so this one doesn't repeat them.

s23
00:01:41.960 --> 00:01:47.200
The agent performing the review has to end its review with one of two things.

s24
00:01:47.200 --> 00:01:57.080
Artifact verified with a fingerprint of the file or files documenting its work, or artifact missing with the reason.

s25
00:01:57.080 --> 00:02:01.040
Then, the agent that sent the job runs the check itself.

s26
00:02:01.040 --> 00:02:05.120
Saying done on its own isn't a permissible answer.

s27
00:02:05.120 --> 00:02:09.160
Level two prevents agents from getting the facts wrong.

s28
00:02:09.160 --> 00:02:15.560
When correctness matters, first find something in the source that the answer has to agree with.

s29
00:02:15.560 --> 00:02:21.400
Then, two agents get an identical set of instructions and do the same job independently.

s30
00:02:21.400 --> 00:02:24.080
Neither one sees the other's work.

s31
00:02:24.080 --> 00:02:30.240
Each one notes where every fact came from and checks its result against the facts.

s32
00:02:30.240 --> 00:02:41.040
Then, a third agent with the original source in front of it compares the work of the two independent agents line by line and settles every difference by going back to the source.

s33
00:02:41.040 --> 00:02:44.040
It never forces a match.

s34
00:02:44.080 --> 00:02:48.240
Level three prevents agents getting the reasoning wrong.

s35
00:02:48.240 --> 00:02:55.240
When clarity of thought really matters, we perform adversarial review in three stages.

s36
00:02:55.240 --> 00:02:58.080
First, the agent audits its own work.

s37
00:02:58.080 --> 00:03:07.120
It lists every claim it's making and marks each one as measured, checked against the source, inferred, or assumed.

s38
00:03:07.120 --> 00:03:12.360
Anything it can't back up gets tested or labeled as a guess.

s39
00:03:12.360 --> 00:03:22.160
Second, reviewers with none of the original agent's context, each acting as a persona with reasons to find fault, read it cold.

s40
00:03:22.160 --> 00:03:31.920
Each one plays a real reader and gets the work and the sources, but no history, no drafts, and no conclusions to agree with.

s41
00:03:31.920 --> 00:03:37.120
Their findings get checked against the source, too, before anything changes.

s42
00:03:37.120 --> 00:03:45.040
Third, once the work is revised, brand new reviewers who've never seen it check the final version.

s43
00:03:45.040 --> 00:03:48.680
Each level adds to the one before as the stakes go up.

s44
00:03:48.680 --> 00:03:55.640
In these cases, each review took minutes, not hours, and the agents do the checking, not me.

s45
00:03:55.640 --> 00:03:59.240
At the scale we operate, checking it myself isn't an option.

s46
00:03:59.240 --> 00:04:03.440
So, let's see each review level applied in three real cases.

s47
00:04:03.440 --> 00:04:05.720
First, a memory leak.

s48
00:04:05.720 --> 00:04:09.440
An open-source dictation app I use was leaking memory.

s49
00:04:09.440 --> 00:04:11.760
It grew to 9 GB.

s50
00:04:11.760 --> 00:04:18.359
We set out to find the leak, fix it, and get the fix accepted by the project's maintainer.

s51
00:04:18.359 --> 00:04:22.600
And to fix it, we had to find out what was really causing it.

s52
00:04:22.600 --> 00:04:28.240
My agents and I went after it, and three times an agent told me something that wasn't true.

s53
00:04:28.240 --> 00:04:39.280
First, a sub-agent I sent to measure the leak with one tool used another and reported its work as done without saying it hadn't used the tool as instructed.

s54
00:04:39.280 --> 00:04:43.640
Nobody caught it at first, including the agent that dispatched the work.

s55
00:04:43.640 --> 00:04:46.320
It came out when I questioned the results.

s56
00:04:46.320 --> 00:04:52.160
The file the right tool would have produced didn't exist, and nobody had noticed.

s57
00:04:52.160 --> 00:04:54.520
That's what level one of the review is for.

s58
00:04:54.520 --> 00:05:02.640
In this case, we dispatched a sub-agent to build a specially instrumented version of the app and capture telemetry while it was running.

s59
00:05:02.640 --> 00:05:04.919
However, the build failed.

s60
00:05:04.919 --> 00:05:14.919
Because we instructed the sub-agent to provide documentation of all of its work, instead of just done, it said the main deliverable wasn't delivered and explained why.

s61
00:05:14.919 --> 00:05:26.960
It reported that an artifact was missing, the software build, and that there were 25 compiler [clears throat] errors, and it correctly labeled the what it could deliver as partial.

s62
00:05:26.960 --> 00:05:35.480
Dispatching the sub-agent with level one review tasking included took 19 minutes and about 16 million tokens.

s63
00:05:35.480 --> 00:05:46.080
The other two significant inaccuracies by agents were about to go public, and the stakes were higher, so for those we borrowed from level three, reviewers with none of the agents' context.

s64
00:05:46.080 --> 00:05:54.560
We'll see all three stages of a level three review in the last example I explain shortly, but here's how it worked for the memory leak fix.

s65
00:05:54.560 --> 00:05:58.080
In the second case, an agent blamed the wrong code.

s66
00:05:58.080 --> 00:06:03.400
To find the leak source, it wrote a small test that used the same library the app does.

s67
00:06:03.400 --> 00:06:07.720
The test leaked, too, so it concluded that the library was to blame.

s68
00:06:07.720 --> 00:06:10.760
It said, "This is solid enough to file."

s69
00:06:10.760 --> 00:06:17.160
Reviewers reproduced the test and found that it had skipped an important step that the library requires.

s70
00:06:17.160 --> 00:06:20.200
Used properly, the library didn't leak.

s71
00:06:20.200 --> 00:06:23.200
The problem was in how the app used the library.

s72
00:06:23.200 --> 00:06:33.560
In the third significant error by agents working on the memory leak, the agent wrote up our fix for the pull request and said that it eliminated the leak completely,

s73
00:06:33.560 --> 00:06:37.120
but it had measured the leak in a way that missed most of it.

s74
00:06:37.120 --> 00:06:39.480
Reviewers caught that, also.

s75
00:06:39.480 --> 00:06:42.919
The fix reduced the leak, but it didn't fully eliminate it.

s76
00:06:42.919 --> 00:06:48.320
So, the pull request went up with honest numbers, and the maintainer accepted the fix and merged it.

s77
00:06:48.320 --> 00:06:54.600
Each of these level three reviews took about 15 minutes and, together, cost about 6 million tokens.

s78
00:06:54.600 --> 00:06:59.080
For the next example, we'll look at portfolio company accounting.

s79
00:06:59.080 --> 00:07:01.680
Buying and building businesses can be messy.

s80
00:07:01.680 --> 00:07:14.400
For one of our portfolio companies, we needed a year of scanned statements from a source accounting system turned into a complete, accurate record of every business transaction for financial reporting purposes.

s81
00:07:14.400 --> 00:07:22.680
An agent extracted the year and handed it back with, "Hey, I recommend spot-checking a few of these transactions."

s82
00:07:22.680 --> 00:07:28.240
This put the burden of reviewing the agent's work back on us, which is a non-starter.

s83
00:07:28.560 --> 00:07:34.919
Extracting data from scans often goes wrong in ways that are tedious and difficult to detect.

s84
00:07:34.919 --> 00:07:41.880
Transactions are missed entirely, transactions are put in the wrong months, or they're recorded with the wrong details.

s85
00:07:41.880 --> 00:07:47.400
One agent simply reporting done can't tell you whether any of that's happened.

s86
00:07:47.400 --> 00:07:50.720
And that's what level two of the review method is for.

s87
00:07:50.720 --> 00:07:56.840
Here, we use the balance printed on the statements as the control total for each month.

s88
00:07:56.840 --> 00:08:04.360
Two agents extracted every month independently, and each one went back to the statements until every month balanced.

s89
00:08:04.360 --> 00:08:12.600
Then, a third agent compared the two extractions line by line and settled each difference against the statements themselves.

s90
00:08:12.600 --> 00:08:20.240
Finally, two more agents, tasked identically, checked each of the final transaction records before they were accepted.

s91
00:08:20.240 --> 00:08:28.919
This fan-out, fan-in pattern of identically tasked agents is highly effective in driving accuracy and reliability.

s92
00:08:28.919 --> 00:08:37.039
When we ran this level two review, we caught missed transactions, incorrect transactions, and transactions in the wrong months.

s93
00:08:37.039 --> 00:08:43.080
Comparing the two extractions caught 56 differences in descriptions, dates, and notes.

s94
00:08:43.080 --> 00:08:47.240
But, ultimately, the whole year balanced to the penny.

s95
00:08:47.240 --> 00:08:53.160
This entire review took about 20 minutes and cost 7.5 million tokens.

s96
00:08:53.160 --> 00:08:56.280
Last, the highest-stakes example.

s97
00:08:56.280 --> 00:08:59.360
More messy portfolio company business here.

s98
00:08:59.360 --> 00:09:06.320
One of ours was in a billing dispute with a vendor, and this amounted to thousands of dollars with legal implications.

s99
00:09:06.320 --> 00:09:16.440
We needed to send a demand letter clearly stating our position and requirements, and every claim in it had to stand up against the contract and each invoice.

s100
00:09:16.440 --> 00:09:21.080
An agent drafted the letter on the premise that the vendor's contract had expired.

s101
00:09:21.080 --> 00:09:29.280
Anything with legal or financial stakes gets a level three review before it goes out, and here's how level three ran on this letter.

s102
00:09:29.280 --> 00:09:31.440
First, the self-audit.

s103
00:09:31.440 --> 00:09:40.120
The agent checked the claims in the letter against the contract and the invoices, including the dates, and reported them verified.

s104
00:09:40.120 --> 00:09:42.160
Stage one passed.

s105
00:09:42.160 --> 00:09:54.840
Second, cold reviewers, each in a role with a reason to find fault, so in this case our own litigator, the vendor's counsel, and an auditor who trusts none of the numbers.

s106
00:09:54.840 --> 00:10:00.480
They each got the letter, the contract, and the invoices, and nothing else.

s107
00:10:00.480 --> 00:10:06.680
The auditor recomputed the financial demand from the invoices, and it was correct down to the cent.

s108
00:10:06.680 --> 00:10:11.800
But both lawyers independently found something buried deep in the contract's fine print.

s109
00:10:11.800 --> 00:10:14.880
A three-year auto-renewal clause.

s110
00:10:14.880 --> 00:10:18.600
The contract had auto-renewed less than three weeks earlier.

s111
00:10:18.600 --> 00:10:26.480
The letter's demands would have had us walking away from a contract that was still in force and owing the vendor thousands of dollars for leaving early.

s112
00:10:26.480 --> 00:10:35.760
The two lawyer agents also learned that, while revising, the original agent had even deleted a line that had the renewal correctly identified.

s113
00:10:35.760 --> 00:10:40.200
The original agent's own check had agreed with its own error.

s114
00:10:40.200 --> 00:10:44.240
Just the sort of thing this review method is designed to catch.

s115
00:10:44.240 --> 00:10:48.360
Our litigator's verdict, though, was do not send the letter.

s116
00:10:48.360 --> 00:10:54.640
That three-part cold review process took 15 minutes and cost 13 million tokens.

s117
00:10:54.640 --> 00:10:59.839
So, the letter was rewritten and handed to the third stage of level three review.

s118
00:10:59.839 --> 00:11:04.160
Brand new reviewers who'd never seen it checked the final version.

s119
00:11:04.160 --> 00:11:06.520
The flawed version never went out.

s120
00:11:06.520 --> 00:11:11.280
So, that's three review stories and three levels of review.

s121
00:11:11.280 --> 00:11:18.120
An agent that couldn't build a customized version of the app said so instead of incorrectly saying that it was done.

s122
00:11:18.120 --> 00:11:24.760
Two groups of two agents extracted and accounted for a year of business transactions down to the penny.

s123
00:11:24.760 --> 00:11:35.280
And cold persona-based reviewers with none of the original agent's context revised a letter that the drafting agents' own check had called verified.

s124
00:11:35.280 --> 00:11:40.800
I've packaged the three review levels as agent skills, and I've released them as open source.

s125
00:11:40.800 --> 00:11:43.520
The write-up is at gallon.me.

s126
00:11:43.520 --> 00:11:50.320
Point your agent at the write-up, and it will know how to check its own work using this review method, so you can rely on it.

s127
00:11:50.320 --> 00:11:57.880
To learn more of the best ways to do real work with AI agents, subscribe to my newsletter and connect with me here.

s128
00:11:58.360 --> 00:12:01.920
So, I believe everything my agents tell me.

s129
00:12:01.920 --> 00:12:03.440
Not because they're right.

s130
00:12:03.440 --> 00:12:05.720
They get things wrong all the time.

s131
00:12:05.720 --> 00:12:11.800
I believe them because everything I rely on has been thoroughly reviewed and validated first.

s132
00:12:11.800 --> 00:12:16.080
And most importantly, not by me.