I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore

AI Engineer · 12 min · 132 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00I believe everything my agents tell me.
00:03And yet, my agents get things wrong all the time.
00:07I've put over 107 billion tokens through agents this year.
00:11So, I analyzed my agent sessions from 2026 year-to-date to see where they went wrong.
00:17In about four in 10 sessions, an agent got something wrong that mattered.
00:23That's how often agents get things wrong in the first place.
00:26It isn't how often those errors made it into work outputs.
00:30Catching those errors before they reach me is the job of the agent-based review method that I've been running for most of this year, and that's the method that I'm sharing with you today.
00:40This analysis identified three ways agents keep failing.
00:44Number one, they overstate what they did.
00:47That is, they say it's done or checked when it isn't.
00:51Number two, they get the facts wrong.
00:53They misread the source, for example.
00:56Or number three, they get the reasoning wrong.
00:59So, the facts are right, but the conclusion isn't.
01:05The method is comprised of three levels of review, one to address each of the three ways that agents keep failing.
01:12I built each level as an agent skill.
01:15Here's how each one works, and then I'll show you how each was applied in three real cases.
01:22Level one prevents agents overstating what they did.
01:26Before an agent starts, the job spells out exactly what it has to hand back.
01:32The file, where it goes, and a command that checks it.
01:36It points out the shortcuts agents have taken before, so this one doesn't repeat them.
01:41The agent performing the review has to end its review with one of two things.
01:47Artifact verified with a fingerprint of the file or files documenting its work, or artifact missing with the reason.
01:57Then, the agent that sent the job runs the check itself.
02:01Saying done on its own isn't a permissible answer.
02:05Level two prevents agents from getting the facts wrong.
02:09When correctness matters, first find something in the source that the answer has to agree with.
02:15Then, two agents get an identical set of instructions and do the same job independently.
02:21Neither one sees the other's work.
02:24Each one notes where every fact came from and checks its result against the facts.
02:30Then, a third agent with the original source in front of it compares the work of the two independent agents line by line and settles every difference by going back to the source.
02:41It never forces a match.
02:44Level three prevents agents getting the reasoning wrong.
02:48When clarity of thought really matters, we perform adversarial review in three stages.
02:55First, the agent audits its own work.
02:58It lists every claim it's making and marks each one as measured, checked against the source, inferred, or assumed.
03:07Anything it can't back up gets tested or labeled as a guess.
03:12Second, reviewers with none of the original agent's context, each acting as a persona with reasons to find fault, read it cold.
03:22Each one plays a real reader and gets the work and the sources, but no history, no drafts, and no conclusions to agree with.
03:31Their findings get checked against the source, too, before anything changes.
03:37Third, once the work is revised, brand new reviewers who've never seen it check the final version.
03:45Each level adds to the one before as the stakes go up.
03:48In these cases, each review took minutes, not hours, and the agents do the checking, not me.
03:55At the scale we operate, checking it myself isn't an option.
03:59So, let's see each review level applied in three real cases.
04:03First, a memory leak.
04:05An open-source dictation app I use was leaking memory.
04:09It grew to 9 GB.
04:11We set out to find the leak, fix it, and get the fix accepted by the project's maintainer.
04:18And to fix it, we had to find out what was really causing it.
04:22My agents and I went after it, and three times an agent told me something that wasn't true.
04:28First, a sub-agent I sent to measure the leak with one tool used another and reported its work as done without saying it hadn't used the tool as instructed.
04:39Nobody caught it at first, including the agent that dispatched the work.
04:43It came out when I questioned the results.
04:46The file the right tool would have produced didn't exist, and nobody had noticed.
04:52That's what level one of the review is for.
04:54In this case, we dispatched a sub-agent to build a specially instrumented version of the app and capture telemetry while it was running.
05:02However, the build failed.
05:04Because we instructed the sub-agent to provide documentation of all of its work, instead of just done, it said the main deliverable wasn't delivered and explained why.
05:14It reported that an artifact was missing, the software build, and that there were 25 compiler [clears throat] errors, and it correctly labeled the what it could deliver as partial.
05:26Dispatching the sub-agent with level one review tasking included took 19 minutes and about 16 million tokens.
05:35The other two significant inaccuracies by agents were about to go public, and the stakes were higher, so for those we borrowed from level three, reviewers with none of the agents' context.
05:46We'll see all three stages of a level three review in the last example I explain shortly, but here's how it worked for the memory leak fix.
05:54In the second case, an agent blamed the wrong code.
05:58To find the leak source, it wrote a small test that used the same library the app does.
06:03The test leaked, too, so it concluded that the library was to blame.
06:07It said, "This is solid enough to file."
06:10Reviewers reproduced the test and found that it had skipped an important step that the library requires.
06:17Used properly, the library didn't leak.
06:20The problem was in how the app used the library.
06:23In the third significant error by agents working on the memory leak, the agent wrote up our fix for the pull request and said that it eliminated the leak completely,
06:33but it had measured the leak in a way that missed most of it.
06:37Reviewers caught that, also.
06:39The fix reduced the leak, but it didn't fully eliminate it.
06:42So, the pull request went up with honest numbers, and the maintainer accepted the fix and merged it.
06:48Each of these level three reviews took about 15 minutes and, together, cost about 6 million tokens.
06:54For the next example, we'll look at portfolio company accounting.
06:59Buying and building businesses can be messy.
07:01For one of our portfolio companies, we needed a year of scanned statements from a source accounting system turned into a complete, accurate record of every business transaction for financial reporting purposes.
07:14An agent extracted the year and handed it back with, "Hey, I recommend spot-checking a few of these transactions."
07:22This put the burden of reviewing the agent's work back on us, which is a non-starter.
07:28Extracting data from scans often goes wrong in ways that are tedious and difficult to detect.
07:34Transactions are missed entirely, transactions are put in the wrong months, or they're recorded with the wrong details.
07:41One agent simply reporting done can't tell you whether any of that's happened.
07:47And that's what level two of the review method is for.
07:50Here, we use the balance printed on the statements as the control total for each month.
07:56Two agents extracted every month independently, and each one went back to the statements until every month balanced.
08:04Then, a third agent compared the two extractions line by line and settled each difference against the statements themselves.
08:12Finally, two more agents, tasked identically, checked each of the final transaction records before they were accepted.
08:20This fan-out, fan-in pattern of identically tasked agents is highly effective in driving accuracy and reliability.
08:28When we ran this level two review, we caught missed transactions, incorrect transactions, and transactions in the wrong months.
08:37Comparing the two extractions caught 56 differences in descriptions, dates, and notes.
08:43But, ultimately, the whole year balanced to the penny.
08:47This entire review took about 20 minutes and cost 7.5 million tokens.
08:53Last, the highest-stakes example.
08:56More messy portfolio company business here.
08:59One of ours was in a billing dispute with a vendor, and this amounted to thousands of dollars with legal implications.
09:06We needed to send a demand letter clearly stating our position and requirements, and every claim in it had to stand up against the contract and each invoice.
09:16An agent drafted the letter on the premise that the vendor's contract had expired.
09:21Anything with legal or financial stakes gets a level three review before it goes out, and here's how level three ran on this letter.
09:29First, the self-audit.
09:31The agent checked the claims in the letter against the contract and the invoices, including the dates, and reported them verified.
09:40Stage one passed.
09:42Second, cold reviewers, each in a role with a reason to find fault, so in this case our own litigator, the vendor's counsel, and an auditor who trusts none of the numbers.
09:54They each got the letter, the contract, and the invoices, and nothing else.
10:00The auditor recomputed the financial demand from the invoices, and it was correct down to the cent.
10:06But both lawyers independently found something buried deep in the contract's fine print.
10:11A three-year auto-renewal clause.
10:14The contract had auto-renewed less than three weeks earlier.
10:18The letter's demands would have had us walking away from a contract that was still in force and owing the vendor thousands of dollars for leaving early.
10:26The two lawyer agents also learned that, while revising, the original agent had even deleted a line that had the renewal correctly identified.
10:35The original agent's own check had agreed with its own error.
10:40Just the sort of thing this review method is designed to catch.
10:44Our litigator's verdict, though, was do not send the letter.
10:48That three-part cold review process took 15 minutes and cost 13 million tokens.
10:54So, the letter was rewritten and handed to the third stage of level three review.
10:59Brand new reviewers who'd never seen it checked the final version.
11:04The flawed version never went out.
11:06So, that's three review stories and three levels of review.
11:11An agent that couldn't build a customized version of the app said so instead of incorrectly saying that it was done.
11:18Two groups of two agents extracted and accounted for a year of business transactions down to the penny.
11:24And cold persona-based reviewers with none of the original agent's context revised a letter that the drafting agents' own check had called verified.
11:35I've packaged the three review levels as agent skills, and I've released them as open source.
11:40The write-up is at gallon.me.
11:43Point your agent at the write-up, and it will know how to check its own work using this review method, so you can rely on it.
11:50To learn more of the best ways to do real work with AI agents, subscribe to my newsletter and connect with me here.
11:58So, I believe everything my agents tell me.
12:01Not because they're right.
12:03They get things wrong all the time.
12:05I believe them because everything I rely on has been thoroughly reviewed and validated first.
12:11And most importantly, not by me.