Does Code Quality Still Matter? It Depends on This — Max Kanat-Alexander, Capital One

AI Engineer · 16 min · 223 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12So, today I figured I'd just pick a nice uncontroversial topic that nobody has any opinions about.
00:18Right?
00:19Nobody nobody said anything on about on this about social media, you know, and of course every take about this on social media is like really deeply nuanced, you know, like people have lots of like depth and like, you know, really big thoughts.
00:32You know, maybe there's a little bit more to it than like yes or no. And you know, of course if you ever ask a senior engineer a question, we always have the same answer to every question, every senior engineer's favorite answer to every question,
00:45which is it depends.
00:48Now, that's not a very helpful answer though, frankly, because what does it depend on?
00:55So, let's start with this.
00:57Why did code quality ever matter in the first place?
01:00Like I see a lot of people be like, well, it made me feel good.
01:02That's not really a sufficient justification.
01:05So, let's see if we can inject some nuance into this conversation and help everybody at least be able to have a basis for reasoning about it while we're discussing it.
01:13So, let's see if we can make a framework here and that's what we're going to try to do with this talk.
01:17Okay.
01:18So, we're going to start off with everything you want to ever accomplish with a piece of software uh depends on two factors and two factors alone.
01:25And that is understanding and executing on that understanding, whether that's aligning a model, scaling a system, driving adoption, anything.
01:32All software success is determined by these two factors.
01:36This is why we have things like debuggers and observability and evals and even tests.
01:40All those things actually are there to improve your understanding or ability to execute on that understanding.
01:44People think that they're there for other reasons, but fundamentally they're there for that reason.
01:48Everything we have ever done in the field of code quality is exclusively and only for these two things.
01:56It's not for anything else.
01:57Not doesn't have something to do with correctness or something else.
02:00It has to do only with these two things.
02:02Now, there's two understandings that are the most important to have if you're going to be a software engineer.
02:07There are other understandings to have about software if you're going to be a product, if you're going to be in sales, but if you're a software engineer, there's the two that are the most important.
02:14And the first is what is the behavior of the system?
02:17Like con- the concretely, not what does a doc say the behavior of the system is, but what is the actual behavior of the system?
02:23And the second one is how do I correctly and safely modify that behavior, okay?
02:30On a very small system, if you want to understand what the system does, I just tell you what it does.
02:36It prints hello world.
02:38You say, "I want it to say hi world."
02:39I say, "Okay, I will make it say hi world now."
02:42There's very little to understand about it other than purely what it does.
02:46That's all of the understanding that you need.
02:49However, any real software system is enormously complex.
02:53This is a real map of the code base of Chromium, which is the open-source project behind Chrome.
03:01And if you think about all of the different ways that these components can interact, they have essentially infinite possible behaviors.
03:09Like, I could not describe to you in words the behavior of Chromium as a whole as a system.
03:17However, what I can describe to you, and as we all know, everybody's been a software engineer for a while, I can describe to you components of the system.
03:25I can successfully describe to you how a component of the system behaves.
03:30So, this is the URL parser, the actual functions and stuff for the URL parser of Chromium.
03:35It figures out all the different parts of the URL.
03:37And this is the whole reason that code exists.
03:44The whole reason that we have code at all in the first place is for structure.
03:50It Because it What do I mean by that?
03:53Okay.
03:54So, both of these two things on the left and on the right will print Hello World if run on a Linux machine.
04:01However, on the left, you can see that it is Hello World.
04:05It has the identity and structure of Hello World.
04:09On the right, you just see a series of instructions with no particular identity, recognizable or understandable identity.
04:16It's not a component, it's just a bunch of instructions.
04:18While that's understandable there, if that was millions or billions of lines of that, it would be much harder to grok.
04:25In the past, we've talked about things like readability or testability or things that seemed like human concepts about code, but what actually mattered the most always was that the system had some sort of understandable structure.
04:38So, it's the structure that actually matters.
04:42And when this modifies when this matters the most is when you want to modify the system, the second uh understanding that we talked about.
04:51All modifications to a system carry two categories of risk.
04:54The first is that you will introduce some sort of problem.
04:57And a problem isn't just a bug, it's anything bad that could happen.
05:00Like your project could take much longer than you expect.
05:03Uh it could not behave in the way that you wanted it to for your users.
05:06The project could fail because you have budget overruns.
05:09Any kind of problem could happen from just you trying to go through the process of making this change.
05:14And the second is you could make it harder to make future changes.
05:19In particular, what that means is you can make it more likely to have future problems.
05:22You could increase future risk of the system.
05:25These two things are caused by under problems with understanding.
05:31So, the first has to do with the difficulty of understanding the system in the first place, and the second is you make the system harder to understand.
05:40I can reduce every postmortem I have ever read for any incident to these two factors in one way or The the production configuration was too hard to understand, somebody made something impossible to understand, there were emergent properties of the system that were impossible to understand, whatever.
05:56Point two feeds into point one in a vicious cycle that compounds in a non-linear fashion.
06:05So, the difficulty of understanding does not grow linearly, it compounds over time.
06:11You will eventually reach a point, and all systems eventually tend to reach this point, where every modification involves some sort of problem.
06:21Uh and you know, some some systems get to the point where like you can't modify them at like if you think of like, you know, some of like your giant 100 million 500 million line mainframe code bases and that
06:33everything, every change is some sort of problem.
06:35You know, you'll see it in organizational issues, you'll see it all over the place, you'll see it in timelines, whatever.
06:41And look, there's a lot of arguments about maybe future models will fix this.
06:44Look, maybe they will.
06:45Maybe future models will have infinite ability to deal with complexity somehow.
06:49Uh but we don't have to discuss that today because I think models are doing something else really interesting right now.
06:56They are creating a new kind of software, a new category of software.
07:01This category has always existed, but it's about to become a hundred thousand times more common than any other form of software.
07:11And that is disposable software.
07:15So, what is that?
07:17Okay.
07:18So, imagine a world in which the consequences of failure for your system are very minimal or non-existent.
07:24Imagine that there's very few people that depend on it or and almost no other systems depend on that piece of software.
07:31Change management, very minimal, non-existent.
07:34Imagine the cost of replacement went to near zero.
07:38You could replace the whole thing in hours or days.
07:41And it had basically no cost of maintenance.
07:44In that world, code quality does not matter.
07:51Because it all of those things we were trying to prevent about the compounding risks over time don't exist.
08:00Traditionally, we've always built durable software.
08:04You know, it lives for years.
08:05It's depended on by millions or thousands of people.
08:07There's huge consequences of failure, potentially business catastrophic consequences of failure.
08:12And it requires enormous maintenance.
08:14Most of the software engineers in the world spend most of their time maintaining these giant durable systems.
08:21I expect this to continue to exist.
08:24So, this is one of the areas where I think sometimes the conversation goes a little astray, okay?
08:29Durable systems will continue to probably be the most important systems in the world.
08:36And there will be probably more of them.
08:39But the most numerous systems will be disposable software.
08:45They will be replaceable in hours.
08:47They will be depended on by a few or change management just won't matter.
08:50And look, if it doesn't work, you will give everybody agents.
08:53They'll make a version of it themselves.
08:56You know, maintenance will be unimportant or nearly free.
08:58Don't like the calculator on your OS?
09:00Make your own.
09:02Want to plan a party for your kids?
09:04Write a party planner software just for that one party.
09:08Throw it away.
09:09Code quality doesn't matter.
09:10Nobody cares.
09:12In the future, we will apply engineering rigor to the degree of disposability of the software.
09:19On the left, nothing.
09:21None of our practices of engineering rigor matter at all.
09:25On the right, all the practices of engineering that we have built up over the last many decades matter greatly.
09:31We can determine where you are on this scale by those four factors that I have been talking about.
09:37Consequences, cost of replacement, dependencies, and cost of maintenance.
09:41If they're all low, it's disposable software.
09:43If they're all high, it's durable software.
09:45And look, any person who's written software for a long time knows you should err on the side of it being durable.
09:51You should err on these all being higher than you expect them to be.
09:54But, there's also a new thing you can do in the world.
10:00You can make software more disposable.
10:03So, like, what do What does that involve?
10:05Well, come up with better isolated environments, right?
10:09So, hey, you can only deploy to this environment that has access to only your data.
10:13No customer data, no personally identifiable information, whatever.
10:17You can enforce independence.
10:19You can say, "Okay, no inbound API calls from other production systems are allowed to this system in this environment, right?"
10:26And so, everything deployed to that environment is disposable.
10:29Redundancy.
10:30You can have an old clunky backup system that is like hard to use.
10:35And then, like, look, if people want to vibe code their own like more efficient front ends, you let them vibe code their more own more efficient front ends because there is an old durable clunky backup system that they can use if their, you know, crappy little vibe coded thing fails.
10:47Which Which it will fail, right?
10:48That's fine.
10:48But, it doesn't matter if it fails.
10:50Uh and you can limit scope, just like I talked about, right?
10:53It's just a calculator for this one person.
10:55It just plans this one party.
10:58There's so much more we could do here and talk about ways of increasing disposability.
11:04In fact, I think this is a whole new discipline of software engineering, not to think about how we improve the rigor of durable systems, but how do we take our systems and how do we encourage disposability
11:17or allow more disposability, allow people to throw these things away.
11:21Interesting thought.
11:22Now, I want to re-emphasize that durable software engineering is not going away because this is the biggest confusion that I see on social media today.
11:32In fact, I believe the absolute amount of durable software in the world will increase because AI coding tools make it easier and easier to make more durable software.
11:44And another important thing to understand is that disposable and durable are not black and white.
11:49There's a spectrum, like I showed before, but that spectrum what changes across the spectrum is how much effort you're going to put into engineering rigor.
11:59Okay?
11:59And that's both immediate effort and effort over time, just like we've always talked about for software.
12:05However, what you want to do, and I think what you want to care about the most, is reducing the cost of effort, reducing the cost of rigor.
12:15Like, if rigor is cheaper, if it's the the easier and easier you make it to be rigor, the less you have to worry about where you are on the scale.
12:23So, this is another approach you can take.
12:26If you change your mind, it's much easier to go like you're like, "Oh, I thought this was going to be disposable and now it needs to be durable."
12:33The cheaper you make it to make it durable, the easier it is to change your mind or to be wrong because you're never going to be 100% right all the time on that estimate.
12:43So, that involves a different set of practices that are not engineering rigor themselves, but that are about how to decrease the cost of engineering rigor.
12:53Now, I could talk about this for hours.
12:54I have spent a tremendous amount of my career doing this.
12:58Uh I don't have 10 hours.
12:59I have about 5 minutes left on that timer.
13:01So, I'll tell I'll talk about what I can, okay?
13:05Let's talk about standardization.
13:07Why do we do standardization?
13:09People are like, "Well, I don't know because the boss said so."
13:11Well, now, we do standardization because it removes possibilities.
13:17If you don't have any PHP in your system, you don't have to figure out what are all the linters for PHP, what are all the practices, what are all the security holes I have to fix.
13:26You just eliminate possibilities through standardization and it just reduces the cost of rigor.
13:32Uh same thing with like frameworks.
13:33Like why don't you want to have every front-end framework or every back-end framework in the in existence?
13:38Because it makes rigor more expensive.
13:40Like yes, rigor is still possible.
13:41Yes, you could still have the agent code up all your tools, but you have to do it every single time.
13:46Uh you know, proto- uh protocols.
13:47Why do you want to have standardized RPC protocols?
13:49All the same thing.
13:50Standards are all for this reason.
13:52Okay?
13:53Uh what I call inward flow tooling.
13:55So, inward flow tooling is tooling that just occurs.
13:57The programmer doesn't have to choose to run it.
13:59They don't have to understand how it runs.
14:01It just happens.
14:02If it's there, it it's feels and seems free and it is reliable.
14:07Reliability is very, very important.
14:08Like it doesn't have a ton of false positives.
14:10It's fast.
14:11It just happens.
14:12And the more that's true, like you can apply it to your disposable software, too.
14:16If it's free and fast, who cares?
14:18Uh making testing easier.
14:21So, this is not just about like, "Hey, how do we do better testing?"
14:24Look, how do you make it so that nobody has to think about how to do better testing?
14:28How do you make it so that every test is fast and deterministic without anybody having to be a deep expert in testing?
14:35There's a ton of work that you have to put into test infrastructure to make that possible.
14:41For example, you know, one of the great dreams that I've always had as a as a test owner is that everybody would provide in-memory fake versions of their services and I could just run those in-memory fake versions of their services.
14:51Guess what?
14:52The coding agent can do that.
14:53So, you can just do that now, right?
14:55But you have to choose to do it and make it available to everybody.
14:58And another thing is, look, maximum strictness.
15:02You just turn the strictness up everywhere.
15:03You turn You set all your lint rules to maximum.
15:06You You never compromise on anything and it just eliminates It's very similar to standardization.
15:11It eliminates a ton of conversations.
15:12Sometimes the lint is wrong.
15:13Like as long as the false positive rate is relatively low and it doesn't have catastrophic consequences, it doesn't matter because all of a sudden there's a bunch of future lint errors or future issues that you don't have to deal with.
15:23Like, ask anybody who's been like, "Oh, I suddenly want to turn on my linter in this million-line code base."
15:28And they're like, "Oh, wow.
15:29Now here's my 100,000-line change that I need to make, right?"
15:32Much easier to just do that up front and have it on for everybody.
15:35Now, look.
15:36Like I said, I could literally stand here and talk to you about this straight for 10 hours.
15:40There is much, much more you can do here.
15:43This is just a few areas, but also a whole area of software engineering discipline that you can apply on the other axis.
15:49So, this lets us now answer our question, not just it depends, but yes or no. So, does code quality even matter?
15:59For disposable software, no. For durable software, yes.
16:06And we get to answer a third and even more interesting question, which is does it have to be expensive?
16:12No. Thank you very much.
16:15[applause]
16:17[music]