Why a One-Line Change Can Break Your Whole Stack - Cole Gannaway, Qodo

AI Engineer · 14 min · 175 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Well, thanks so much for showing up to this discussion here.
00:16I'll probably actually ask for a little audience participation as well.
00:20We'd love to get to know how you guys are thinking about things.
00:22The goal of today is to talk a little bit about Codo, which is the organization I'm representing, as well as how software factories have evolved.
00:32This is sort of what we're seeing in the industry where a lot of AI native or AI first companies are starting to head.
00:40I think everyone has a grand vision of where these factories can go, and we believe Codo is a critical part of that, not the complete picture of that.
00:47So, there's an element of what we do, where we specialize, and then areas that are outside and we'll make sure to be clear on some of that.
00:54I'll I'll kind of hop into the first one that's worth kind of walking through.
00:59I'm what's called a solutions engineer, so I spend time with organizations talking through how our product or other products might fit into their workflows and teaching about Codo as a platform.
01:11What I often hear from customers is that they're super excited about going from ticket all the way to production.
01:17From somebody generating code or somebody coming up with a ticket and being able to push that all the way through, and what they're finding is at the pull request level or at the review stage,
01:29they're struggling.
01:30This has become a huge bottleneck.
01:32Code generation, I wouldn't say is solved, but it's definitely moving at a rate that we've never seen before.
01:38And so, this has been a big bottleneck.
01:41This is also where it's critical that we provide the human or the coding agent or the review agent with as much context as we can to make sure that we're making the right decisions.
01:52This is really the trust and verification layer.
01:55All the other components are still valuable, but just worth noting that's where we'll focus a lot today is on that review section here.
02:04I want to give a a practical example.
02:06Um, as I mentioned, code generation's not necessarily solved, but when we're looking at code and when we're creating features and when we're trying to generate new code, new features,
02:18we can often miss things that we're not trying to.
02:20So, this is an example where we have just a one-line change.
02:23This is on Bitwarden, which is a open-source password manager, and you'll notice that we have just one line change to our data model.
02:31If I was a developer and I saw this, it seems good to me, right?
02:36I'm going to say this seems like the right change.
02:38I wanted it to rename it.
02:40I would probably merge it.
02:41I would probably look and see that the build passed, that unit tests passed, maybe even my linters passed.
02:48It seems like this is a change that we needed.
02:51This is a feature that we wanted.
02:53Let's Let's approve it.
02:55So, I'm curious, maybe just from like a show of hands, but like how are you guys when you're talking about code review or review changes?
03:02I would imagine you have a bunch of tools.
03:04When they all give you green checks, does that mean it's good to go or what does that What does that look like for you guys?
03:10Maybe a show of hands, like how many have merged a PR that they were like, "Uh, it seems good to me.
03:16Not 100% sure.
03:16I I didn't look at all the lines of code."
03:18Uh, maybe just a raise of hands.
03:20Yeah.
03:21Right.
03:21So, every everyone here, I think, just about everyone here is like, "We can't look at all the lines of code that have changed.
03:26So, we rely on these tools."
03:28There's a bunch of great tools out there.
03:30Worth noting, not all tools are built the same and not all uh context is built the same.
03:35In this example that we would have just merged, we would have missed the blast radius or the effect that this can happen.
03:42This one line of code change can affect multiple repos.
03:46In this example, again, I'm using Bitwarden as as a sort of an illustrative example, we have a back-end server that could be connected to web clients, that could be connected to mobile or iOS devices,
03:56it could be connected to MCP servers or CLI.
03:59And you'll notice that all of these components are affected on that one line of change.
04:03So, this is typically found in production, right?
04:06Like maybe a bug or a user story or the web app crashes.
04:10And so, we'd love to be able to catch this as early as possible.
04:15Context is king here.
04:17Um one of the things that uh Codo will do and I think uh is is really important is build out a software graph of the repositories that your team is working on today.
04:27This will build it out automatically and it'll also build out an integration surface.
04:31So, not just uh you'd be disappointed or uh it wouldn't be as valuable if we said just Bitwarden server was connected to Bitwarden Android, but not what the actual components are that are connected.
04:42And so, this is a huge area of if we're building the software factory, if we're not looking at every line of code, how do we get this to my agent to my review agent
04:51uh to be able to take this into account?
04:56Another thing that's worth uh calling out is like not everything is uh is is purely functional.
05:03Some are going to be a bit more senior-level reviewers, custom rules, maybe a bit of taste that's involved, and how are we learning from some of our past mistakes?
05:14Um so, when we go back to like the software uh factory for example here, we have a human that reviews it, but is there a way that we can bring that human reviewer
05:23back into the code intelligence, back into the review layer, rather than it just being every time that's the bottleneck, if that makes sense?
05:33So, in this example here, we have a custom rule that we can add, which says, "Hey, if we're going to change anything about public APIs, we want to make sure that we add either a deprecation flag
05:43or we review this a bit more."
05:47Additionally, we'd love to have this information pre-PR, pre-code generation.
05:52So, is there an opportunity for us to bring this information in before we've ever generated line of code knowing that we have potential downstream effects if we make some of these changes.
06:02These are some of the areas you can see that in this example we've got an MCP server, we've got code base wisdom, we're getting those custom rules or or tribal knowledge that might exist,
06:12bring all that into into our coding agent long before we're pushing it up.
06:17That doesn't guarantee that it's not going to make mistakes, right?
06:20I think any anybody would say that we want to double check, but this gives us a much better opportunity and a much better shot at making it right on the first try.
06:30Worth calling out here, I think many organizations are thinking should I just use cloud code or my implementation agent to just do the review.
06:39This review stage has some information, but some of this is actually valuable in the actual generation stage.
06:45It's also worth noting that an implementation agent has a bunch of other things that it's doing that are not related to review.
06:51And context windows are getting bigger, but that doesn't mean that things don't get buried, that doesn't mean things don't get hallucinated.
06:58And so there's debugging, there's implementation, there's exploring the code base versus a review agent has a specific set of context that we want to use.
07:07And you'll notice here on the right we potentially have some room to focus on other things as well.
07:13So, not just the code base wisdom that we saw or the graph, we can even bring in more information that might be custom to this specific run or this specific review
07:23that we have potentially some room for as well.
07:30So, this kind of goes into the agent that wrote the code, we don't need him grading the code.
07:35One, we have more context window available.
07:38The other is we want to make sure that these are independent.
07:42So, that can be things like uh the LLM that's being used under the hood, whether it's Anthropic, whether it's OpenAI, whether it's open weight models.
07:50Can we leverage multiple models to try to get rid of some of that bias?
07:53Another is the harness.
07:55If the harnesses are the same, we could have some bias there.
07:59We would love to be able to swap out all of these or even make adversarial reviewers um that have the ability to look at the same code and see if there's differences in what we're finding.
08:09If we find the same thing, probably high signal that that's that's true.
08:13If we find completely different things, then it might be worth further investigation.
08:19Awesome.
08:20And then lastly, how do we take this information from this specific example and make sure that it doesn't happen again?
08:27Uh I know a lot of organizations that I work with will do postmortem reviews and these are valuable.
08:33They'll we learn from it.
08:34Often we learn things that don't have anything to do with code, but have to do with organizational changes or process changes that we need to do.
08:41But for the code level changes, how do we bring that back in and make that actual tribal knowledge a organizational knowledge that everyone can use goes in automated fashion.
08:51And this is a this is a huge area as well.
08:53You can see in this example here uh Codo, for example, suggesting a rule based on a comment on a PR.
09:00So, that way we can go back and now apply that rule to every PR in the future.
09:05Uh you'll also see that because we have that software graph information, we can see the blast radius of that one line of code change was large, we can flag that to the reviewer,
09:15whether that's a human or another agent, to make sure that we're looking at this a bit closer.
09:22So, wanted to show this at least from a a uh you know, high-level view of what we're talking through.
09:27There's a bunch of things involved in each software factory.
09:31Context is going to be king.
09:32So, can we get the right data, the right information?
09:35Can we apply those rules or standards that we may already have or ones that we want to learn from?
09:41And then can we check this pre-PR using MCP CLI like there's great things to catch at the PR level.
09:47However, if we could have caught it earlier, then maybe we could have caught it at the code gen stage or have ability to remediate and go from there.
09:55Um I'll show a little bit of this live, but I just wanted to at least show some of what we're thinking, how we've seen customers building out their software factories.
10:05Um curious if there's any questions so far.
10:08I'm I'm happy to pause, but um what what does this look like for you guys?
10:14Were you Were you raising your hand?
10:16Yeah.
10:16Yeah, go for it.
10:17So uh during the PR stage
10:22Yeah.
10:23Uh we go through a review.
10:25Or I go through a review with the agent.
10:28I tell the agent, "Hey, I want to change XYZ.
10:30I want to load some questions about the code base, what was Um and then I go in and I fill in some documentation.
10:41Basically what you're saying where it'll get applied on the next run of the factory.
10:48Yes.
10:49Uh so you're you're doing all that then, right?
10:52Yep.
10:53Yeah, we're able to to bring in So I'm I'm showing just some some examples of what we do and I'm kind of focusing on the software graph and the custom rules.
11:01Uh we can bring in a bunch of other things like ticketing systems, things like that as well.
11:04Do you find that most of your clients have problems with intent on before the code gets even written?
11:12Yeah.
11:12So if we go back to this section here on the uh the ticket right here, right?
11:17Maybe the ticket isn't written super well.
11:19Is that Is that kind of what you're describing?
11:21Maybe or
11:23Okay.
11:23whatever context is produced to generate the implementation aspect.
11:30Um the linkage between those two is missing the real intent and so it's learning the wrong the wrong thing.
11:40Guiding it.
11:41Guiding it.
11:41So you're saying like the agent can then start going like I think one of the that maybe a lot of people experience is the over-engineering of some of these agents where
11:50I asked it to do one thing and it's now like recreating everything from scratch.
11:55Is that kind of what you're describing there?
11:56And maybe even
11:58We're approaching the problem incorrectly.
12:01It just could be
12:02Totally incorrect.
12:03So not even over-engineering just just just wrong.
12:07Yeah.
12:08So Coda I can maybe even show a little bit of this live here.
12:12If I go in we've got integrations with things like Jira tickets or linear tickets and then these will go into every PR review.
12:21So if I go to like this one for example, you can see here we'll link in tickets directly into it.
12:29Wow.
12:30When you go when you go from dark mode to light mode you really get you get blasted there.
12:34You can see there's acceptance criteria in here and then if I go back to the review if I go back here you'll see that we actually will highlight requirements gaps based on that ticket that we just saw.
12:46So this is when we're when we're talking Coda we're not looking at just those things but we're looking at are we completing the ticket?
12:53So are we actually going down that that rabbit hole?
12:55And because we can do this review PR pre-PR we can actually guide the agent.
13:01So typically what we'll do when we work with clients is we'll say run a review periodically.
13:06So we'll often put a skill in there which says as we're generating code about at your 25% your 50% your 75% mark go ahead and run a review make sure we're on the right path.
13:16As I mentioned that context window earlier starts to shrink and so like this is one of those things where if I go to this one this specialized review agent can act as almost a manager agent in this way of guiding the agent
13:32from doing the wrong thing or over engineering it or things like that.
13:37Did that help answer that question there?
13:40And then maybe maybe further that we need to talk through it.
13:44Okay.
13:45Cool.
13:47Any other questions, comments?
13:49Just curious here.
13:53Okay.
13:55Awesome.
13:56We'll be at the booth downstairs the the code of booth.
14:01I'm happy to talk through it with you guys further.
14:04I think a lot of our our clients are getting towards this how do we make this software factory run and the critical section that at least our platform is solving for is that review stage.
14:14How do we give the human as much information?
14:17How do we bring this context layer this code intelligence and wisdom layer into the review agent to make it as accurate And so then I'd say that the next layer of where we're going is
14:31what ones can be merged without a human looking at it.
14:34I think today people are already doing that.
14:36We'd love to know which ones that we should which ones that we should not.
14:40And so that that blast radius information is available there.