WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=LlgiOCmFG_w

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/LlgiOCmFG_w.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.223 --> 00:00:02.046
OK.

s2
00:00:02.046 --> 00:00:16.564
I'm here to talk about solving the PR bottleneck, and that's a bit of a big name for trying to fix something that I think most organizations struggle with, and that they've historically struggled with since before AI came along.

s3
00:00:16.564 --> 00:00:28.746
We've always had a huge amount of PR that was just sitting around with no one checking it, and now that workload has massively increased due to the new challenges that AI is putting before us , imposing all these strange limitations.

s4
00:00:28.746 --> 00:00:35.241
And to do that, I'll use a rubric of my skills that you may have heard of or even used.

s5
00:00:35.241 --> 00:00:46.553
I have a couple of new skills I want to announce that I hope will improve your approach to PR, improve the speed at which you review and execute them.

s6
00:00:46.553 --> 00:00:48.553
Speed.

s7
00:00:48.883 --> 00:00:59.491
Now we are being forced to do more with fewer resources, or essentially, more PR, more work, etc. And that's the main promise of AI: we'll be able to use

s8
00:00:59.491 --> 00:01:02.879
these agents to scale and do more work.

s9
00:01:02.879 --> 00:01:07.878
This led to the emergence of the “software factory”—perhaps the biggest buzzword today.

s10
00:01:07.878 --> 00:01:12.021
Everyone I talk to talks about the software factory.

s11
00:01:12.021 --> 00:01:19.908
I think of a software factory as something where instead of a human initiating all of this work, we outsource some of these processes.

s12
00:01:19.908 --> 00:01:23.326
Part of the initiation will be performed by agents.

s13
00:01:23.326 --> 00:01:56.563
And I think about it this way: Maybe from there you have a classifier like Jeff who says, " Okay, let's turn this into a fix or a replay."

s14
00:01:56.563 --> 00:01:58.666
Or maybe I'll notify people right away.

s15
00:01:58.666 --> 00:02:00.355
Maybe you have something else.

s16
00:02:00.355 --> 00:02:05.179
You may have Planet Scale connected to get reports on slow database queries.

s17
00:02:05.179 --> 00:02:09.776
Maybe this triggers another link in your software factory.

s18
00:02:09.776 --> 00:02:13.553
All this is not run by people, but by deterministic code, right?

s19
00:02:13.553 --> 00:02:16.808
So, they speed up your software factory.

s20
00:02:16.808 --> 00:02:19.282
They push more code through it .

s21
00:02:19.282 --> 00:02:21.346
But then you need brakes, right?

s22
00:02:21.346 --> 00:02:27.807
If you have constant acceleration pushing things through a factory, you get a " garbage cannon," right?

s23
00:02:27.807 --> 00:02:35.447
You'll just get a bunch of worthless, terrible PR that you won't even be able to touch, review, or even look at.

s24
00:02:35.447 --> 00:02:36.946
So you need brakes.

s25
00:02:36.946 --> 00:02:48.728
These are mechanisms that slow down, improve quality, and ensure that your codebase doesn't turn into a nightmare of software entropy, because code is the environment in which your agent runs.

s26
00:02:48.728 --> 00:02:54.152
And if you have bad code in your codebase, it will generate even more bad code.

s27
00:02:54.152 --> 00:03:02.260
So, in this talk, I'll talk about these three brakes and how we can use them to, paradoxically, move faster.

s28
00:03:02.260 --> 00:03:03.925
So , automated checks.

s29
00:03:03.925 --> 00:03:11.408
These are deterministic checks in your repository that have been around for thousands of years, well, or since the 50s.

s30
00:03:11.408 --> 00:03:19.879
Deterministic checks, where we have linting, tests, type checking, and code quality—all working together and the same way every time.

s31
00:03:19.879 --> 00:03:23.680
On top of that, we have an automated review.

s32
00:03:23.680 --> 00:03:32.829
So we have agents that look at our code and say, "Okay, this is for things that the tests didn't catch, or this is an analysis of the structure of the codebase in general ."

s33
00:03:32.829 --> 00:03:37.076
And then on top of that , the third level, the final level—is human verification.

s34
00:03:37.076 --> 00:03:39.054
That is, people who view PR.

s35
00:03:39.054 --> 00:03:45.292
And these three levels form a kind of "cake" that we get when we get to human verification.

s36
00:03:45.292 --> 00:03:48.481
So higher speed, of course, means more PR.

s37
00:03:48.481 --> 00:03:54.627
So the goal here is to make human verification faster by building on these first two steps.

s38
00:03:54.627 --> 00:03:57.082
And we are going to stop the negligence.

s39
00:03:57.082 --> 00:04:09.836
This is the first principle: if you improve the quality of the code you release, you will spend less time on human review because it is simply a better job, and therefore you will need to do less intervention.

s40
00:04:09.836 --> 00:04:12.137
So, automated checks.

s41
00:04:12.137 --> 00:04:14.847
Nowadays, checks are cheap.

s42
00:04:14.847 --> 00:04:22.466
The cool thing about them is that they don't cost tokens like an automated review, they don't require human effort, they only cost processor time.

s43
00:04:22.466 --> 00:04:25.629
For example, you run tests for every code change.

s44
00:04:25.629 --> 00:04:34.770
Perhaps these tests require some tokens, because let's say the agent created a bug and the test detected it—then you'll have to spend some tokens to fix it.

s45
00:04:34.770 --> 00:04:38.322
But, in my opinion, these are tokens spent quite appropriately.

s46
00:04:38.322 --> 00:04:39.796
So, inspections are cheap.

s47
00:04:39.796 --> 00:04:43.352
This means you can add a lot of them to your repositories.

s48
00:04:43.352 --> 00:04:49.145
And you're probably not using them enough or creatively enough.

s49
00:04:49.145 --> 00:04:51.392
But tests can lie.

s50
00:04:51.392 --> 00:04:55.811
Does a green CI status mean that the code is ready for merging ?

s51
00:04:55.811 --> 00:04:56.999
No, it doesn't mean.

s52
00:04:56.999 --> 00:05:05.503
So we always needed an extra layer on top of these checks to find out if there was something catastrophically wrong with the code before we released it .

s53
00:05:05.503 --> 00:05:10.610
So all the other stages, human verification and automated verification—they're lie detectors.

s54
00:05:10.610 --> 00:05:13.935
They are needed to detect lies in automated checks.

s55
00:05:13.935 --> 00:05:24.276
Now I want to show you some examples of such lies, because it helps us , when we think about code and automated testing, to understand how bad it is and what it can lead to.

s56
00:05:24.276 --> 00:05:26.532
The first is tautological tests.

s57
00:05:26.532 --> 00:05:29.535
A test that simply confirms the implementation.

s58
00:05:29.535 --> 00:05:31.244
Opus 5 got hooked on them.

s59
00:05:31.244 --> 00:05:33.122
I don't quite understand why.

s60
00:05:33.122 --> 00:05:37.213
For example, it would be: the character limit in post X is 280.

s61
00:05:37.213 --> 00:05:40.922
Can anyone guess the test written to check for this behavior?

s62
00:05:40.922 --> 00:05:41.608
Truth?

s63
00:05:41.608 --> 00:05:43.205
You've probably seen this a thousand times.

s64
00:05:43.205 --> 00:05:47.244
This is real code from agents or what I found out agents do.

s65
00:05:47.244 --> 00:05:51.808
It says: " We expect the character limit for post X to be 280."

s66
00:05:51.808 --> 00:05:54.088
So, the implementation looked exactly like this.

s67
00:05:54.088 --> 00:05:56.900
And the test essentially just repeated the implementation.

s68
00:05:56.900 --> 00:06:00.443
This is the tautological test.

s69
00:06:00.443 --> 00:06:05.293
Tautological tests are bad because they are extremely sensitive to structure.

s70
00:06:05.293 --> 00:06:09.538
They are very dependent on the internal workings of the system.

s71
00:06:09.538 --> 00:06:14.324
This means I can't change the constant without failing the test.

s72
00:06:14.324 --> 00:06:17.739
Moreover, I can't rename a constant without failing the test.

s73
00:06:17.739 --> 00:06:21.472
This is so closely tied to the structure of my system.

s74
00:06:21.472 --> 00:06:25.640
I found another one that I would say is even more blatant.

s75
00:06:25.640 --> 00:06:29.870
This is an incredible e-e test.

s76
00:06:29.870 --> 00:06:35.355
It essentially checks whether two elements in the interface appear in the correct order.

s77
00:06:35.355 --> 00:06:42.109
So, it checks whether the pitch details page , or more specifically, the video section, comes after the content plan.

s78
00:06:42.109 --> 00:06:45.478
It doesn't render it to the screen.

s79
00:06:45.478 --> 00:06:51.659
It simply reads the file, the module into its own memory, and then finds the right thing.

s80
00:06:51.659 --> 00:06:58.944
So, it finds the content plan, finds the video, and then expects the videos to come after it in the source material.

s81
00:06:58.944 --> 00:07:05.217
Which is crazy when you think about it, because I can just change the look of the source material and this test will fail.

s82
00:07:05.217 --> 00:07:07.945
It is too sensitive to the structure of my codebase.

s83
00:07:07.945 --> 00:07:11.987
So this is another way that automated checks can go wrong.

s84
00:07:11.987 --> 00:07:13.441
And there's also Oh , sorry.

s85
00:07:13.441 --> 00:07:15.417
Automated checks can lie.

s86
00:07:15.417 --> 00:07:18.773
There are also tests that literally cannot fail.

s87
00:07:18.773 --> 00:07:24.006
This will sound familiar to you if you have abused mochas or used different things before.

s88
00:07:24.006 --> 00:07:28.037
For example , here we have the use audio boost function.

s89
00:07:28.037 --> 00:07:34.041
And this function use audio boost, its internal logic uses the audio context API in the DOM.

s90
00:07:34.041 --> 00:07:36.325
Don't worry if you don't know any of this .

s91
00:07:36.325 --> 00:07:40.914
But here we are simply creating " stubs" with dummy methods.

s92
00:07:40.914 --> 00:07:47.097
It turns out that the audio context has complex error modes and crashes under strange conditions.

s93
00:07:47.097 --> 00:07:56.187
Therefore, such actions mean that our test cannot fail in these modes, and we will encounter strange errors in operation that the tests will not fix.

s94
00:07:57.138 --> 00:08:03.698
So the question arises: is it possible to bypass automated checks, even with the best of intentions?

s95
00:08:03.698 --> 00:08:07.810
AI doesn't try to write bad tests.

s96
00:08:07.810 --> 00:08:14.733
It simply follows our instructions and writes tests that are too tied to the structure, instead of actually running the code.

s97
00:08:14.733 --> 00:08:18.707
So how do you make automated checks harder to bypass?

s98
00:08:18.707 --> 00:08:24.500
If we do this, we will raise the quality of inspections, and therefore our overall quality standard.

s99
00:08:24.500 --> 00:08:28.998
And the first thing I really like here is the code-based design .

s100
00:08:28.998 --> 00:08:33.200
You can literally design solutions to avoid these bad checks.

s101
00:08:33.200 --> 00:08:36.092
So what does good code-based design look like?

s102
00:08:36.092 --> 00:08:40.248
I've talked about this before in previous talks for AI engineers—these are deep modules.

s103
00:08:40.248 --> 00:08:43.909
These are modules that hide complex behavior behind simple interfaces.

s104
00:08:43.909 --> 00:08:48.825
This is an idea from John Osterhout on software design philosophy .

s105
00:08:48.825 --> 00:08:56.748
If we look at these two modules, we see A, where the large implementation is hidden behind a small interface on top.

s106
00:08:56.748 --> 00:08:57.452
Of course?

s107
00:08:57.452 --> 00:09:03.960
And module B has a large interface, many functions that can be called, but each of them does very little.

s108
00:09:03.960 --> 00:09:05.458
Does this make sense?

s109
00:09:05.458 --> 00:09:06.318
Yes?

s110
00:09:06.318 --> 00:09:17.213
If you have a deep module like A, you will have fewer structure-sensitive tests because you are hiding more of the implementation behind the interface.

s111
00:09:17.213 --> 00:09:21.979
If testing only happens through this interface, you will get better tests.

s112
00:09:21.979 --> 00:09:30.467
Your task here is to get agents to use this little interface, not to dig into the implementation to test weird details.

s113
00:09:30.467 --> 00:09:35.466
I have a skill for this.

s114
00:09:35.466 --> 00:09:43.666
You can take the weirdest, worst codebase you've ever seen, apply this skill, and it becomes better.

s115
00:09:43.666 --> 00:09:47.701
It essentially gives you opportunities to deepen the modules.

s116
00:09:47.701 --> 00:09:50.006
It looks something like this .

s117
00:09:50.006 --> 00:09:52.695
Raise your hands if you've used this skill before.

s118
00:09:52.695 --> 00:09:54.679
Not sure how many of my people are in this room.

s119
00:09:54.679 --> 00:09:55.107
Yes, okay.

s120
00:09:55.107 --> 00:09:57.403
This is really really cool.

s121
00:09:57.403 --> 00:09:58.915
Essentially, it gives you an HTML document.

s122
00:09:58.915 --> 00:10:00.960
I'll step aside so you can see.

s123
00:10:00.960 --> 00:10:04.772
With all the different, um, potential possibilities she sees.

s124
00:10:04.772 --> 00:10:12.755
So here we see a “before” and “after” state where we reduce duplication and create a beautiful, deep, and tested module.

s125
00:10:12.755 --> 00:10:15.976
And then you can safely implement it.

s126
00:10:15.976 --> 00:10:28.039
This is also coupled with a language that I developed to describe modules, because if you ever try to learn how to structure a codebase, you'll find 20 different approaches, and they'll all be called

s127
00:10:28.039 --> 00:10:33.793
DDD, and you need a consistent language that you can use within a team to discuss these things.

s128
00:10:33.793 --> 00:10:41.168
So I have a small set of codebase design skills that define what locality is, what leverage is, and what seam is.

s129
00:10:41.168 --> 00:10:45.595
I used joints before it became mainstream.

s130
00:10:45.595 --> 00:10:51.027
Locality means how well all the code is placed together.

s131
00:10:51.027 --> 00:10:56.548
How can you change one small part in a module without causing a chain reaction?

s132
00:10:56.548 --> 00:11:06.046
And leverage is what you get by having a deep module, because the caller of that module gets a lot of benefit from calling a simple function.

s133
00:11:06.046 --> 00:11:09.233
Both of these concepts are very useful in codebases.

s134
00:11:09.233 --> 00:11:12.114
It turns out they are also useful for agents.

s135
00:11:12.114 --> 00:11:20.055
But I'm describing all these high coding standards, but how do we actually get an agent to adhere to them?

s136
00:11:20.055 --> 00:11:30.957
How do you make sure that the agent creates deep modules, writes good tests, and doesn't create those worthless tautological or structure-dependent tests?

s137
00:11:30.957 --> 00:11:34.681
Well, I think most people do it wrong.

s138
00:11:34.681 --> 00:11:42.014
And my first piece of advice: don't implement coding standards directly into implementing agents .

s139
00:11:42.014 --> 00:11:43.070
Of course?

s140
00:11:43.070 --> 00:11:45.100
Let me explain.

s141
00:11:45.100 --> 00:11:52.681
If you imagine that the implementing agent looks something like this, then these are all the tasks that it has to perform within one context window.

s142
00:11:52.681 --> 00:11:56.853
He must be able to investigate, that is, look for code that needs to be changed.

s143
00:11:56.853 --> 00:12:01.336
Then he has to modify it directly, i.e. update the files marked in green.

s144
00:12:01.336 --> 00:12:04.721
And then he needs the time to adjust himself.

s145
00:12:04.721 --> 00:12:10.436
So, to run automated checks, to actually check and confirm that everything is working.

s146
00:12:10.436 --> 00:12:13.759
As you can see, this is quite a large amount of work.

s147
00:12:13.759 --> 00:12:20.986
And if you try to put coding standards enforcement on it , it will perform worse.

s148
00:12:20.986 --> 00:12:22.143
So, the implementation is overloaded.

s149
00:12:22.143 --> 00:12:25.715
This is the mental model I want you to adopt.

s150
00:12:25.715 --> 00:12:30.313
So can we find a way to implement these coding standards in a way that doesn't overload the agent?

s151
00:12:30.313 --> 00:12:32.299
Well, this is my attempt to do so.

s152
00:12:32.299 --> 00:12:34.514
This is my code review skill.

s153
00:12:34.514 --> 00:12:36.494
She gets a diff file.

s154
00:12:36.494 --> 00:12:43.295
It reads a file in the repository called " coding standards" which you can write and customize.

s155
00:12:43.295 --> 00:12:47.851
Then it checks whether the code meets these standards.

s156
00:12:47.851 --> 00:12:52.782
So, if we look at the review agent, he also importantly works as a subagent.

s157
00:12:52.782 --> 00:12:55.999
Therefore, it has its own context window for processing tasks.

s158
00:12:55.999 --> 00:12:57.485
He has his own budget.

s159
00:12:57.485 --> 00:13:01.046
He needs to do some research, right?

s160
00:13:01.046 --> 00:13:10.292
Because of course, he gets a diff file, so he knows exactly where the code is, but he should explore the system a little for a broader context to better understand the code.

s161
00:13:10.292 --> 00:13:12.703
However, he does not need to deal with implementation.

s162
00:13:12.703 --> 00:13:15.534
He doesn't need to do any debugging.

s163
00:13:15.534 --> 00:13:19.887
So while the implementation is overloaded, the view is actually underloaded.

s164
00:13:19.887 --> 00:13:22.044
So, he doesn't have much work to do.

s165
00:13:22.044 --> 00:13:29.190
This means you can add a bunch of coding standards to it , and it will handle it much better than if you tried to do it during implementation.

s166
00:13:29.190 --> 00:13:32.785
I think about it this way, and it's a little uncomfortable, right?

s167
00:13:32.785 --> 00:13:36.875
Because we all want to get quality code the first time .

s168
00:13:36.875 --> 00:13:45.949
But I see it as a two-step process of writing good code: implementation—to make it work, and code review—to make it good quality.

s169
00:13:45.949 --> 00:13:48.592
You apply your own coding standards.

s170
00:13:48.592 --> 00:13:53.422
And for the “ old school” developers among us, this is essentially a red-green-refactor approach.

s171
00:13:53.422 --> 00:14:00.399
We use one context window to make the code work by going through the red-green stage, and then another for refactoring.

s172
00:14:00.399 --> 00:14:03.921
At least that's how it works in my head, and it's what has brought me success.

s173
00:14:03.921 --> 00:14:08.953
This means that when you have coding standards, you don't add them to the global scope .

s174
00:14:08.953 --> 00:14:13.860
You don't put them in agents.md because then they overload your implementation tool.

s175
00:14:13.860 --> 00:14:16.264
He may or may not read them .

s176
00:14:16.264 --> 00:14:21.844
You place them in coding standards.md, and that way they are only processed by the code review agent .

s177
00:14:21.844 --> 00:14:26.306
Now I have another idea: I talked to a lot of people today.

s178
00:14:26.306 --> 00:14:31.642
Many say I was talking about review, automated review, and eliminating bottlenecks in the PR process.

s179
00:14:31.642 --> 00:14:35.031
So many people say, "Oh, yeah, we just use a third-party service."

s180
00:14:35.031 --> 00:14:36.726
We use Cursor Bug Bot.

s181
00:14:36.726 --> 00:14:38.375
We use Code Rabbit.

s182
00:14:38.375 --> 00:14:39.790
Or something like that.

s183
00:14:39.790 --> 00:14:49.583
I think I was really trying to create a universal code review skill that would find all the bugs, perform security audits, etc. before.

s184
00:14:49.583 --> 00:14:59.000
It turns out that this is very, very difficult because you either make it too general and get false positives that are irrelevant to your case, or you make it too specific.

s185
00:14:59.000 --> 00:15:01.773
You say, "Okay, find everything related to TypeScript."

s186
00:15:01.773 --> 00:15:04.160
And then Rust developers can't take advantage of it.

s187
00:15:04.160 --> 00:15:07.748
Therefore, I would say: you should not outsource automated testing.

s188
00:15:07.748 --> 00:15:08.903
Create your own.

s189
00:15:08.903 --> 00:15:11.360
Develop your own coding standards over time.

s190
00:15:11.360 --> 00:15:12.974
Share them with your team.

s191
00:15:12.974 --> 00:15:21.574
And if you have the ability to implement coding standards, and you have some documents that no one reads, this is exactly the place to put them.

s192
00:15:21.574 --> 00:15:32.249
Also , when you have an automated reviewer like this, many people have a natural desire to say, "Oh, yeah, my code review agent reads the code and then leaves comments

s193
00:15:32.249 --> 00:15:34.964
on the PR."

s194
00:15:34.964 --> 00:15:39.801
So your code review agent in this case is just adding work to the human reviewer.

s195
00:15:39.801 --> 00:15:46.357
A human reviewer then has to read all these lengthy comments and decide: do we implement this or do we implement that?

s196
00:15:46.357 --> 00:15:48.065
The reviewer must make commits.

s197
00:15:48.065 --> 00:15:50.115
He has to make corrections.

s198
00:15:50.115 --> 00:15:52.785
That is, he must actually change what he finds.

s199
00:15:52.785 --> 00:15:57.339
Because when a person comes, he or she checks a finished, high-quality artifact.

s200
00:15:57.339 --> 00:16:03.615
If he finds something that raises questions, he can of course leave a comment, but the default should be a commit.

s201
00:16:03.615 --> 00:16:06.819
Stop trying to get the perfect code the first time.

s202
00:16:06.819 --> 00:16:13.126
Stop trying to make the executive agent a mere tool and thinking, “Okay, I’ll make it work perfectly.”

s203
00:16:13.126 --> 00:16:19.523
It takes some time, you know, to make sense of it, but once you get it, it's just incredible.

s204
00:16:19.523 --> 00:16:19.912
So, okay.

s205
00:16:19.912 --> 00:16:27.918
With this whole process: we've done our automated checks, now we've done an automated audit to make sure the checks aren't lying.

s206
00:16:27.918 --> 00:16:31.844
How do we maximize the quality of PR from a human verification perspective?

s207
00:16:31.844 --> 00:16:34.842
How to make it work as best as possible?

s208
00:16:34.842 --> 00:16:36.838
We need human-friendly PR.

s209
00:16:36.838 --> 00:16:45.048
And this is a new skill that's coming to the repository, it's currently in development, but I'll be releasing it soon—it's the PR skill.

s210
00:16:45.048 --> 00:16:47.407
I thought about this for a long, long time.

s211
00:16:47.407 --> 00:16:50.005
I still don't fully understand what the modern standard is.

s212
00:16:50.005 --> 00:16:56.182
I realized that the best way to master the skill of creating PR is to simply borrow all the best ideas that everyone has.

s213
00:16:56.182 --> 00:16:57.693
And that's very good.

s214
00:16:57.693 --> 00:17:01.648
So, what does a good PR description look like?

s215
00:17:01.648 --> 00:17:04.541
How can you recreate this skill yourself?

s216
00:17:04.541 --> 00:17:09.250
Well, the first principle is that some checks are more important than others.

s217
00:17:09.250 --> 00:17:12.740
Not every inspection is mandatory, right?

s218
00:17:12.740 --> 00:17:20.790
And once you understand that, you realize: okay, this means I can focus my energy on the really important checks.

s219
00:17:20.790 --> 00:17:23.262
But how do we classify them?

s220
00:17:23.262 --> 00:17:30.607
You need to think about PR using AWS terminology: is it a one-way door or a two-way door?

s221
00:17:30.607 --> 00:17:34.220
Most of your PR will be a two-way door.

s222
00:17:34.220 --> 00:17:37.987
You can merge a PR and then you can always roll it back.

s223
00:17:37.987 --> 00:17:42.849
That's a great advantage of working as a software engineer instead of a civil engineer, right?

s224
00:17:42.849 --> 00:17:45.960
Usually when you're a civil engineer, it's always a one-way door, right?

s225
00:17:45.960 --> 00:17:54.082
If you make a mistake , the bridge will collapse, but if you have doors on both sides, it means you can easily undo the changes.

s226
00:17:54.082 --> 00:17:57.007
This may be a little more complicated than you expect.

s227
00:17:57.007 --> 00:18:04.181
It could happen that a very simple change accidentally sends an email to 60,000 people or something like that, in which case it's a one-way door.

s228
00:18:04.181 --> 00:18:06.826
You have to check this very, very carefully.

s229
00:18:06.826 --> 00:18:10.570
If it involves expensive migrations or data loss, it's a one-way door .

s230
00:18:10.570 --> 00:18:13.440
Check such PR as carefully as possible.

s231
00:18:13.440 --> 00:18:17.312
But, in addition, we need to understand the radius of this PR.

s232
00:18:17.312 --> 00:18:18.653
What could go wrong?

s233
00:18:18.653 --> 00:18:22.421
And if things go wrong, how serious is it?

s234
00:18:22.421 --> 00:18:28.186
This means that at the end of all my PRs there is a nice short summary—the level of merger risk.

s235
00:18:28.186 --> 00:18:30.937
I see that this PR is a two-way door .

s236
00:18:30.937 --> 00:18:35.966
Its radius of effect is local, so I see: great, I don't need to pay much attention to it.

s237
00:18:35.966 --> 00:18:38.028
I'll just take a look at it.

s238
00:18:38.028 --> 00:18:40.346
This is very important.

s239
00:18:40.346 --> 00:18:43.952
Next, we need to understand what this PR is actually doing, right?

s240
00:18:43.952 --> 00:18:50.720
I've tried many ways to figure this out, and the best I've come up with is using pseudocode.

s241
00:18:50.720 --> 00:18:58.382
I want to express my great gratitude for the "show me" skill from the human layer skills repository by Dex Hadley.

s242
00:18:58.382 --> 00:19:06.978
This is a phenomenal skill that essentially eliminates most of the text and shows everything through images and diagrams.

s243
00:19:06.978 --> 00:19:13.384
This makes it much easier to understand what exactly is changing and why it is happening.

s244
00:19:13.761 --> 00:19:20.185
So, you get a standard set of mermaid and UML diagrams to show the sequence of events.

s245
00:19:20.185 --> 00:19:23.152
You also get great simple examples like this one.

s246
00:19:23.152 --> 00:19:26.557
We can look at this and say, "Okay, we're working in the command line interface.

s247
00:19:26.557 --> 00:19:27.895
We see that a new command has been added ."

s248
00:19:27.895 --> 00:19:31.290
And we have two new things, these little flags at the top.

s249
00:19:31.290 --> 00:19:32.696
Just a brief summary.

s250
00:19:32.696 --> 00:19:34.433
This really makes a huge difference.

s251
00:19:34.433 --> 00:19:36.888
It is difficult to overestimate its importance.

s252
00:19:36.888 --> 00:19:42.915
So, you try to make understanding the reasons as fast as possible.

s253
00:19:42.915 --> 00:19:49.747
And I think the third principle here is what you have to think about when you're doing a human verification.

s254
00:19:49.747 --> 00:20:03.810
Because we don't do like...Because our processes are so streamlined now, and we all collaborate around these same skill files, these same management files.

s255
00:20:03.810 --> 00:20:09.576
We are all building an environment in which our agents will work together.

s256
00:20:09.576 --> 00:20:15.364
You should think of the process that creates your code as just as important as the code itself.

s257
00:20:15.364 --> 00:20:21.388
In other words, when you do human verification, you are not only checking the code, but also the system that generates it.

s258
00:20:21.388 --> 00:20:26.006
And the theory here is that you never want to write the same comment twice.

s259
00:20:26.006 --> 00:20:27.074
Is n't that right?

s260
00:20:27.074 --> 00:20:31.831
You never want to see an agent doing the same thing in two different merge requests (PRs).

s261
00:20:31.831 --> 00:20:35.891
So, what mechanism allows you to make your verification of a person significant?

s262
00:20:35.891 --> 00:20:37.176
Well, it's a new skill.

s263
00:20:37.176 --> 00:20:38.943
It's called retro.

s264
00:20:38.943 --> 00:20:40.567
Retro— from "retrospective."

s265
00:20:40.567 --> 00:20:42.973
You are essentially taking the session you had.

s266
00:20:42.973 --> 00:20:54.465
It could be either a single agent session, or a PR along with a session, or you could just have her review it: "Okay, look at all the PRs we've done in the last week, all the reviews, put them together."

s267
00:20:54.465 --> 00:20:56.727
Let's take a look back at them.

s268
00:20:56.727 --> 00:21:03.430
And it will offer automated checks and coding standards to make next time better.

s269
00:21:03.430 --> 00:21:12.734
So, it's a cumulative effect, where by doing a human review, you actually increase the quality of the next review and reduce the amount of work for yourself in the future.

s270
00:21:12.734 --> 00:21:15.160
And retro is a really smart skill.

s271
00:21:15.160 --> 00:21:17.032
She adds a bunch of everything.

s272
00:21:17.032 --> 00:21:19.558
Of course, it offers automated checks.

s273
00:21:19.558 --> 00:21:22.693
It suggests updating the coding standards.md file.

s274
00:21:22.693 --> 00:21:24.447
She does other smart things too.

s275
00:21:24.447 --> 00:21:26.936
For example, it analyzes navigation indicators.

s276
00:21:26.936 --> 00:21:29.538
How easily did the agent find the information?

s277
00:21:29.538 --> 00:21:33.363
Can we provide a hint in agents.md to help him next time?

s278
00:21:33.363 --> 00:21:35.642
It analyzes the effectiveness of using tools.

s279
00:21:35.642 --> 00:21:41.040
Are there other tools we use in the session that could be more effective with tokens?

s280
00:21:41.040 --> 00:21:45.707
It's amazing how much this reveals, because such things are often very difficult to debug from the outside.

s281
00:21:45.707 --> 00:21:47.565
This also checks the system for redundancy.

s282
00:21:47.565 --> 00:21:49.894
So, are there overloaded control files?

s283
00:21:49.894 --> 00:21:53.175
Are there excessive skills that lead to such poor results?

s284
00:21:53.175 --> 00:21:56.385
Can we make them more orderly?

s285
00:21:56.385 --> 00:21:57.180
So, that's the goal.

s286
00:21:57.180 --> 00:22:00.458
This is to make human verification faster.

s287
00:22:00.458 --> 00:22:10.153
We achieve this by overlaying automated verification to make human review as painless, simple, and optional as possible.

s288
00:22:10.153 --> 00:22:13.771
You really don't need to test every decision that can be changed.

s289
00:22:13.771 --> 00:22:16.764
But every irreversible decision is mandatory.

s290
00:22:16.764 --> 00:22:18.586
So, these are my skills.

s291
00:22:18.586 --> 00:22:25.068
I'm here at hero.dev/skills, and this week we're releasing version 1.3. It was great spending time with you.

s292
00:22:25.068 --> 00:22:26.791
It was a really enjoyable conference.

s293
00:22:26.791 --> 00:22:30.087
I'll be in the lobby if anyone wants to chat.

s294
00:22:30.087 --> 00:22:32.111
Thank you very much for the invitation.

s295
00:22:32.111 --> 00:22:34.643
Thank you, Paris.