WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=YqxpPI7YUjE NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/YqxpPI7YUjE.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.560 --> 00:00:16.880 Well, thanks so much for showing up to this discussion here. s3 00:00:16.880 --> 00:00:20.480 I'll probably actually ask for a little audience participation as well. s4 00:00:20.480 --> 00:00:22.880 We'd love to get to know how you guys are thinking about things. s5 00:00:22.880 --> 00:00:32.520 The goal of today is to talk a little bit about Codo, which is the organization I'm representing, as well as how software factories have evolved. s6 00:00:32.520 --> 00:00:40.320 This is sort of what we're seeing in the industry where a lot of AI native or AI first companies are starting to head. s7 00:00:40.320 --> 00:00:47.240 I think everyone has a grand vision of where these factories can go, and we believe Codo is a critical part of that, not the complete picture of that. s8 00:00:47.240 --> 00:00:54.000 So, there's an element of what we do, where we specialize, and then areas that are outside and we'll make sure to be clear on some of that. s9 00:00:54.000 --> 00:00:59.120 I'll I'll kind of hop into the first one that's worth kind of walking through. s10 00:00:59.120 --> 00:01:11.040 I'm what's called a solutions engineer, so I spend time with organizations talking through how our product or other products might fit into their workflows and teaching about Codo as a platform. s11 00:01:11.040 --> 00:01:17.800 What I often hear from customers is that they're super excited about going from ticket all the way to production. s12 00:01:17.800 --> 00:01:29.960 From somebody generating code or somebody coming up with a ticket and being able to push that all the way through, and what they're finding is at the pull request level or at the review stage, s13 00:01:29.960 --> 00:01:30.560 they're struggling. s14 00:01:30.560 --> 00:01:32.400 This has become a huge bottleneck. s15 00:01:32.400 --> 00:01:38.560 Code generation, I wouldn't say is solved, but it's definitely moving at a rate that we've never seen before. s16 00:01:38.560 --> 00:01:41.200 And so, this has been a big bottleneck. s17 00:01:41.200 --> 00:01:52.240 This is also where it's critical that we provide the human or the coding agent or the review agent with as much context as we can to make sure that we're making the right decisions. s18 00:01:52.240 --> 00:01:55.760 This is really the trust and verification layer. s19 00:01:55.760 --> 00:02:03.280 All the other components are still valuable, but just worth noting that's where we'll focus a lot today is on that review section here. s20 00:02:04.080 --> 00:02:06.560 I want to give a a practical example. s21 00:02:06.560 --> 00:02:18.120 Um, as I mentioned, code generation's not necessarily solved, but when we're looking at code and when we're creating features and when we're trying to generate new code, new features, s22 00:02:18.120 --> 00:02:20.680 we can often miss things that we're not trying to. s23 00:02:20.680 --> 00:02:23.680 So, this is an example where we have just a one-line change. s24 00:02:23.680 --> 00:02:31.600 This is on Bitwarden, which is a open-source password manager, and you'll notice that we have just one line change to our data model. s25 00:02:31.600 --> 00:02:36.120 If I was a developer and I saw this, it seems good to me, right? s26 00:02:36.120 --> 00:02:38.160 I'm going to say this seems like the right change. s27 00:02:38.160 --> 00:02:40.040 I wanted it to rename it. s28 00:02:40.040 --> 00:02:41.400 I would probably merge it. s29 00:02:41.400 --> 00:02:48.640 I would probably look and see that the build passed, that unit tests passed, maybe even my linters passed. s30 00:02:48.640 --> 00:02:51.160 It seems like this is a change that we needed. s31 00:02:51.160 --> 00:02:53.959 This is a feature that we wanted. s32 00:02:53.959 --> 00:02:55.280 Let's Let's approve it. s33 00:02:55.280 --> 00:03:02.480 So, I'm curious, maybe just from like a show of hands, but like how are you guys when you're talking about code review or review changes? s34 00:03:02.480 --> 00:03:04.959 I would imagine you have a bunch of tools. s35 00:03:04.959 --> 00:03:10.480 When they all give you green checks, does that mean it's good to go or what does that What does that look like for you guys? s36 00:03:10.480 --> 00:03:16.000 Maybe a show of hands, like how many have merged a PR that they were like, "Uh, it seems good to me. s37 00:03:16.000 --> 00:03:16.880 Not 100% sure. s38 00:03:16.880 --> 00:03:18.840 I I didn't look at all the lines of code." s39 00:03:18.840 --> 00:03:20.400 Uh, maybe just a raise of hands. s40 00:03:20.400 --> 00:03:21.320 Yeah. s41 00:03:21.320 --> 00:03:21.560 Right. s42 00:03:21.560 --> 00:03:26.440 So, every everyone here, I think, just about everyone here is like, "We can't look at all the lines of code that have changed. s43 00:03:26.440 --> 00:03:28.200 So, we rely on these tools." s44 00:03:28.200 --> 00:03:30.160 There's a bunch of great tools out there. s45 00:03:30.160 --> 00:03:35.840 Worth noting, not all tools are built the same and not all uh context is built the same. s46 00:03:35.840 --> 00:03:42.680 In this example that we would have just merged, we would have missed the blast radius or the effect that this can happen. s47 00:03:42.680 --> 00:03:46.680 This one line of code change can affect multiple repos. s48 00:03:46.680 --> 00:03:56.560 In this example, again, I'm using Bitwarden as as a sort of an illustrative example, we have a back-end server that could be connected to web clients, that could be connected to mobile or iOS devices, s49 00:03:56.560 --> 00:03:59.520 it could be connected to MCP servers or CLI. s50 00:03:59.520 --> 00:04:03.720 And you'll notice that all of these components are affected on that one line of change. s51 00:04:03.720 --> 00:04:06.760 So, this is typically found in production, right? s52 00:04:06.760 --> 00:04:10.959 Like maybe a bug or a user story or the web app crashes. s53 00:04:10.959 --> 00:04:15.400 And so, we'd love to be able to catch this as early as possible. s54 00:04:15.400 --> 00:04:17.120 Context is king here. s55 00:04:17.120 --> 00:04:27.200 Um one of the things that uh Codo will do and I think uh is is really important is build out a software graph of the repositories that your team is working on today. s56 00:04:27.200 --> 00:04:31.480 This will build it out automatically and it'll also build out an integration surface. s57 00:04:31.480 --> 00:04:42.680 So, not just uh you'd be disappointed or uh it wouldn't be as valuable if we said just Bitwarden server was connected to Bitwarden Android, but not what the actual components are that are connected. s58 00:04:42.680 --> 00:04:51.440 And so, this is a huge area of if we're building the software factory, if we're not looking at every line of code, how do we get this to my agent to my review agent s59 00:04:51.440 --> 00:04:54.800 uh to be able to take this into account? s60 00:04:56.320 --> 00:05:03.480 Another thing that's worth uh calling out is like not everything is uh is is purely functional. s61 00:05:03.480 --> 00:05:14.240 Some are going to be a bit more senior-level reviewers, custom rules, maybe a bit of taste that's involved, and how are we learning from some of our past mistakes? s62 00:05:14.240 --> 00:05:23.800 Um so, when we go back to like the software uh factory for example here, we have a human that reviews it, but is there a way that we can bring that human reviewer s63 00:05:23.800 --> 00:05:33.080 back into the code intelligence, back into the review layer, rather than it just being every time that's the bottleneck, if that makes sense? s64 00:05:33.560 --> 00:05:43.840 So, in this example here, we have a custom rule that we can add, which says, "Hey, if we're going to change anything about public APIs, we want to make sure that we add either a deprecation flag s65 00:05:43.840 --> 00:05:46.760 or we review this a bit more." s66 00:05:47.360 --> 00:05:52.280 Additionally, we'd love to have this information pre-PR, pre-code generation. s67 00:05:52.280 --> 00:06:02.480 So, is there an opportunity for us to bring this information in before we've ever generated line of code knowing that we have potential downstream effects if we make some of these changes. s68 00:06:02.480 --> 00:06:12.640 These are some of the areas you can see that in this example we've got an MCP server, we've got code base wisdom, we're getting those custom rules or or tribal knowledge that might exist, s69 00:06:12.640 --> 00:06:17.720 bring all that into into our coding agent long before we're pushing it up. s70 00:06:17.720 --> 00:06:20.040 That doesn't guarantee that it's not going to make mistakes, right? s71 00:06:20.040 --> 00:06:30.000 I think any anybody would say that we want to double check, but this gives us a much better opportunity and a much better shot at making it right on the first try. s72 00:06:30.000 --> 00:06:39.320 Worth calling out here, I think many organizations are thinking should I just use cloud code or my implementation agent to just do the review. s73 00:06:39.320 --> 00:06:45.360 This review stage has some information, but some of this is actually valuable in the actual generation stage. s74 00:06:45.360 --> 00:06:51.440 It's also worth noting that an implementation agent has a bunch of other things that it's doing that are not related to review. s75 00:06:51.440 --> 00:06:58.080 And context windows are getting bigger, but that doesn't mean that things don't get buried, that doesn't mean things don't get hallucinated. s76 00:06:58.080 --> 00:07:07.640 And so there's debugging, there's implementation, there's exploring the code base versus a review agent has a specific set of context that we want to use. s77 00:07:07.640 --> 00:07:13.760 And you'll notice here on the right we potentially have some room to focus on other things as well. s78 00:07:13.760 --> 00:07:23.680 So, not just the code base wisdom that we saw or the graph, we can even bring in more information that might be custom to this specific run or this specific review s79 00:07:23.680 --> 00:07:27.400 that we have potentially some room for as well. s80 00:07:30.320 --> 00:07:35.919 So, this kind of goes into the agent that wrote the code, we don't need him grading the code. s81 00:07:35.919 --> 00:07:38.800 One, we have more context window available. s82 00:07:38.800 --> 00:07:42.120 The other is we want to make sure that these are independent. s83 00:07:42.120 --> 00:07:50.440 So, that can be things like uh the LLM that's being used under the hood, whether it's Anthropic, whether it's OpenAI, whether it's open weight models. s84 00:07:50.440 --> 00:07:53.920 Can we leverage multiple models to try to get rid of some of that bias? s85 00:07:53.920 --> 00:07:55.240 Another is the harness. s86 00:07:55.240 --> 00:07:59.040 If the harnesses are the same, we could have some bias there. s87 00:07:59.040 --> 00:08:09.280 We would love to be able to swap out all of these or even make adversarial reviewers um that have the ability to look at the same code and see if there's differences in what we're finding. s88 00:08:09.280 --> 00:08:13.200 If we find the same thing, probably high signal that that's that's true. s89 00:08:13.200 --> 00:08:17.919 If we find completely different things, then it might be worth further investigation. s90 00:08:19.440 --> 00:08:20.320 Awesome. s91 00:08:20.320 --> 00:08:27.320 And then lastly, how do we take this information from this specific example and make sure that it doesn't happen again? s92 00:08:27.320 --> 00:08:33.080 Uh I know a lot of organizations that I work with will do postmortem reviews and these are valuable. s93 00:08:33.080 --> 00:08:34.400 They'll we learn from it. s94 00:08:34.400 --> 00:08:41.560 Often we learn things that don't have anything to do with code, but have to do with organizational changes or process changes that we need to do. s95 00:08:41.560 --> 00:08:51.520 But for the code level changes, how do we bring that back in and make that actual tribal knowledge a organizational knowledge that everyone can use goes in automated fashion. s96 00:08:51.520 --> 00:08:53.880 And this is a this is a huge area as well. s97 00:08:53.880 --> 00:09:00.320 You can see in this example here uh Codo, for example, suggesting a rule based on a comment on a PR. s98 00:09:00.320 --> 00:09:05.880 So, that way we can go back and now apply that rule to every PR in the future. s99 00:09:05.880 --> 00:09:15.640 Uh you'll also see that because we have that software graph information, we can see the blast radius of that one line of code change was large, we can flag that to the reviewer, s100 00:09:15.640 --> 00:09:21.440 whether that's a human or another agent, to make sure that we're looking at this a bit closer. s101 00:09:22.000 --> 00:09:27.920 So, wanted to show this at least from a a uh you know, high-level view of what we're talking through. s102 00:09:27.920 --> 00:09:31.280 There's a bunch of things involved in each software factory. s103 00:09:31.280 --> 00:09:32.680 Context is going to be king. s104 00:09:32.680 --> 00:09:35.520 So, can we get the right data, the right information? s105 00:09:35.520 --> 00:09:41.080 Can we apply those rules or standards that we may already have or ones that we want to learn from? s106 00:09:41.080 --> 00:09:47.960 And then can we check this pre-PR using MCP CLI like there's great things to catch at the PR level. s107 00:09:47.960 --> 00:09:55.560 However, if we could have caught it earlier, then maybe we could have caught it at the code gen stage or have ability to remediate and go from there. s108 00:09:55.560 --> 00:10:05.280 Um I'll show a little bit of this live, but I just wanted to at least show some of what we're thinking, how we've seen customers building out their software factories. s109 00:10:05.280 --> 00:10:08.080 Um curious if there's any questions so far. s110 00:10:08.080 --> 00:10:14.040 I'm I'm happy to pause, but um what what does this look like for you guys? s111 00:10:14.960 --> 00:10:16.080 Were you Were you raising your hand? s112 00:10:16.080 --> 00:10:16.400 Yeah. s113 00:10:16.400 --> 00:10:17.000 Yeah, go for it. s114 00:10:17.000 --> 00:10:22.760 So uh during the PR stage s115 00:10:22.760 --> 00:10:23.320 Yeah. s116 00:10:23.320 --> 00:10:25.320 Uh we go through a review. s117 00:10:25.320 --> 00:10:28.040 Or I go through a review with the agent. s118 00:10:28.040 --> 00:10:30.320 I tell the agent, "Hey, I want to change XYZ. s119 00:10:30.320 --> 00:10:41.600 I want to load some questions about the code base, what was Um and then I go in and I fill in some documentation. s120 00:10:41.600 --> 00:10:48.720 Basically what you're saying where it'll get applied on the next run of the factory. s121 00:10:48.720 --> 00:10:49.520 Yes. s122 00:10:49.520 --> 00:10:52.520 Uh so you're you're doing all that then, right? s123 00:10:52.520 --> 00:10:53.000 Yep. s124 00:10:53.000 --> 00:11:01.040 Yeah, we're able to to bring in So I'm I'm showing just some some examples of what we do and I'm kind of focusing on the software graph and the custom rules. s125 00:11:01.040 --> 00:11:04.640 Uh we can bring in a bunch of other things like ticketing systems, things like that as well. s126 00:11:04.640 --> 00:11:12.480 Do you find that most of your clients have problems with intent on before the code gets even written? s127 00:11:12.480 --> 00:11:12.839 Yeah. s128 00:11:12.839 --> 00:11:17.839 So if we go back to this section here on the uh the ticket right here, right? s129 00:11:17.839 --> 00:11:19.880 Maybe the ticket isn't written super well. s130 00:11:19.880 --> 00:11:21.960 Is that Is that kind of what you're describing? s131 00:11:21.960 --> 00:11:23.040 Maybe or s132 00:11:23.040 --> 00:11:23.720 Okay. s133 00:11:23.720 --> 00:11:30.360 whatever context is produced to generate the implementation aspect. s134 00:11:30.360 --> 00:11:40.360 Um the linkage between those two is missing the real intent and so it's learning the wrong the wrong thing. s135 00:11:40.360 --> 00:11:41.040 Guiding it. s136 00:11:41.040 --> 00:11:41.680 Guiding it. s137 00:11:41.680 --> 00:11:50.080 So you're saying like the agent can then start going like I think one of the that maybe a lot of people experience is the over-engineering of some of these agents where s138 00:11:50.080 --> 00:11:55.160 I asked it to do one thing and it's now like recreating everything from scratch. s139 00:11:55.160 --> 00:11:56.400 Is that kind of what you're describing there? s140 00:11:56.400 --> 00:11:58.520 And maybe even s141 00:11:58.520 --> 00:12:01.120 We're approaching the problem incorrectly. s142 00:12:01.120 --> 00:12:02.960 It just could be s143 00:12:02.960 --> 00:12:03.760 Totally incorrect. s144 00:12:03.760 --> 00:12:07.000 So not even over-engineering just just just wrong. s145 00:12:07.000 --> 00:12:08.160 Yeah. s146 00:12:08.160 --> 00:12:12.000 So Coda I can maybe even show a little bit of this live here. s147 00:12:12.000 --> 00:12:21.720 If I go in we've got integrations with things like Jira tickets or linear tickets and then these will go into every PR review. s148 00:12:21.720 --> 00:12:29.320 So if I go to like this one for example, you can see here we'll link in tickets directly into it. s149 00:12:29.320 --> 00:12:30.000 Wow. s150 00:12:30.000 --> 00:12:34.240 When you go when you go from dark mode to light mode you really get you get blasted there. s151 00:12:34.240 --> 00:12:46.240 You can see there's acceptance criteria in here and then if I go back to the review if I go back here you'll see that we actually will highlight requirements gaps based on that ticket that we just saw. s152 00:12:46.240 --> 00:12:53.320 So this is when we're when we're talking Coda we're not looking at just those things but we're looking at are we completing the ticket? s153 00:12:53.320 --> 00:12:55.920 So are we actually going down that that rabbit hole? s154 00:12:55.920 --> 00:13:01.040 And because we can do this review PR pre-PR we can actually guide the agent. s155 00:13:01.040 --> 00:13:06.560 So typically what we'll do when we work with clients is we'll say run a review periodically. s156 00:13:06.560 --> 00:13:16.920 So we'll often put a skill in there which says as we're generating code about at your 25% your 50% your 75% mark go ahead and run a review make sure we're on the right path. s157 00:13:16.920 --> 00:13:32.080 As I mentioned that context window earlier starts to shrink and so like this is one of those things where if I go to this one this specialized review agent can act as almost a manager agent in this way of guiding the agent s158 00:13:32.080 --> 00:13:37.080 from doing the wrong thing or over engineering it or things like that. s159 00:13:37.080 --> 00:13:40.240 Did that help answer that question there? s160 00:13:40.880 --> 00:13:44.840 And then maybe maybe further that we need to talk through it. s161 00:13:44.840 --> 00:13:45.960 Okay. s162 00:13:45.960 --> 00:13:47.240 Cool. s163 00:13:47.240 --> 00:13:49.840 Any other questions, comments? s164 00:13:49.840 --> 00:13:52.200 Just curious here. s165 00:13:53.720 --> 00:13:55.280 Okay. s166 00:13:55.280 --> 00:13:56.320 Awesome. s167 00:13:56.320 --> 00:14:01.640 We'll be at the booth downstairs the the code of booth. s168 00:14:01.640 --> 00:14:04.240 I'm happy to talk through it with you guys further. s169 00:14:04.240 --> 00:14:14.800 I think a lot of our our clients are getting towards this how do we make this software factory run and the critical section that at least our platform is solving for is that review stage. s170 00:14:14.800 --> 00:14:17.400 How do we give the human as much information? s171 00:14:17.400 --> 00:14:31.760 How do we bring this context layer this code intelligence and wisdom layer into the review agent to make it as accurate And so then I'd say that the next layer of where we're going is s172 00:14:31.760 --> 00:14:34.160 what ones can be merged without a human looking at it. s173 00:14:34.160 --> 00:14:36.080 I think today people are already doing that. s174 00:14:36.080 --> 00:14:40.120 We'd love to know which ones that we should which ones that we should not. s175 00:14:40.120 --> 00:14:45.360 And so that that blast radius information is available there.