How to Build Quality Gates into Agentic Coding Workflows — Nnenna Ndukwe, Qodo AI

AI Engineer · 57 min · 646 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:13Um, so my name is Nana Andukquay and I am uh developer relations lead at Kodto.
00:20We're we used to be just like an AI code uh code review platform but now we're kind of expanding beyond that.
00:26that's all about code quality and governance across the software development life cycle.
00:31And I've spent a lot of time um thinking about and working on like what does a structured AI coding uh process look like.
00:40There are many different ways that we're all doing it.
00:42I would love to hear from y'all especially towards the end of this.
00:46But I'm just going to have what I want is to walk through what that can look like.
00:51There's a repo I have that everyone can fork.
00:54And if you do nothing else from this, ask your AI to analyze and grab some tips from the structure of this the repo and try to see what you can apply to your own workflow.
01:04All right.
01:06All right.
01:07So, we're going to walk through just a little bit of the beginning, who I am, Nana.
01:12Um, definitely passionate about uh AI for developers.
01:16This is my thing as a Devril and as a former uh full-time software engineer.
01:21And I live in Boston.
01:23And this is my cat, Pippen.
01:25He's named after Lord of the Rings Hobbit, not the basketball player, which people ask me a lot about.
01:31And you know, that's what this is what I'm here for, implementing structured AI coding workflow.
01:36Um we're going to be using a COD.
01:38Um you'll be able to sign into the platform.
01:40If you don't want to use it for um some of the earlier stages of the SDLC we'll walk through, you can just use it for uh PR code reviews.
01:50And uh first things first uh if you want to you you'll want to have access to the slides and I think the best way to do that or the easiest way to do that is to go to github.commyame
02:04and then it should be the repo should be at the top of the repository or um those are the slides um that will link to my GitHub or my um my Google slides.
02:18So hopefully everybody can I'll give everyone a minute to do that.
02:25So yeah, slides are here.
02:27The repository should be at the top of that.
02:31And somebody anybody raise their hand if they don't see it or can't get access.
02:43Thumbs up if I'm good to move on or or I'll wait.
02:48Okay.
02:54Um, so in general, where and how code quality can break down?
02:59Uh, who who here is, you know, using AI for coding?
03:02I'm assuming everybody.
03:05We have do we have Claude code fans here and Codeex?
03:12Wow.
03:13Wait, less hands for Codeex.
03:15Okay.
03:15I thought they like blew up in the past couple months.
03:17I've been using codecs for a lot of stuff.
03:20But um where can code quality break down in the agentic development life cycle?
03:26Uh everywhere.
03:28Planning and design, development, code review, testing, deployment.
03:32There are so many potential failure modes for um for using AI for coding just to ship high quality software.
03:40And that is where we come into this process where we kind of have to overdo it with steering AI coding agents to get what we need out of it just so that we can get behind or stand behind the quality of the code and be like, okay, you know what, I can put my name on that.
03:56This is good to go.
03:57We can move this forward through the pipeline or I'm I feel safe to have another developer review my code at this point.
04:03[snorts]
04:03And so I really believe that we should be building qualitydriven systems.
04:08I think that there needs to be a verification um aspect to agentic engineering and it really starts with number one defining what code quality means to you uh and your team if you're working on a team of developers
04:23that needs to be defined and then that needs to be that needs to live somewhere.
04:27it needs to be codified so that our AI, no matter what interface you're using, can extract that information and do its best to adhere to the code standards that you have in place.
04:40And then once you've decided where those standards live, whether they're locally in your specific individual repos or if they're in a context management layer, which we'll get into, then you can decide and pull that into the different interfaces.
04:55If that's your CLI, if that's your IDE, um your your CI workflows, it should be uh distributable.
05:06And uh we do not have to go through all of that.
05:10Um I already explained that we might just be able to Okay.
05:14Well, here's what we're going to do.
05:15We are going to operationalize our context.
05:19Um I have a repo that has rules.
05:21It has skills.
05:22It has a it has you know a bunch of different things in it and we want to leverage that for planning for code generation and code review and the whole point is that we want to verify
05:34the quality early and often and
05:38[snorts]
05:38ideally it's the same standards that we are asking the agents to check against across each of those phases.
05:46That's that's really the whole point.
05:51Oh, these are my skills.
05:52I have so many.
05:54Um, and yeah, so I think uh I'm going to go ahead and just jump into what we need to do.
06:02Um, so first let's go to the Kodto quality workshop signal pay.
06:14If y'all can see that.
06:17Oh, I can't increase the uh let's see.
06:30Okay, so this is the repo that we're going to.
06:32It is Kodo Quality Workshop Signal Pay.
06:35This is a payments app.
06:36And what I want you to do is fork this repo and let me know when you have done that.
06:42And then what you're going to do is clone it.
06:44[snorts]
06:45If you want to work through this on your own time, all of the instructions are there in the readme.
06:50And I'm going to be leaning heavily on my coding agents to actually help me implement this, you know, like at a meta level.
07:02So, code quality workshop-signal pay.
07:06And in this repo, this is where we're just going to walk through the process of planning for a coding change.
07:16We are going to read the repo rules to determine, okay, which rules are relevant for this particular coding change.
07:22We're going to use skills and then make sure that we're writing proper tests using some TDD and BDD skills, test driven, behavior driven development.
07:32Then we're going to ask our coding agent to implement um the code change and then run local gates.
07:39We have llinters, we have um you know static analysis that we're running because we want some deterministic checks and we want AI as well like a good combination.
07:49And then we'll open a pull request and have Codto uh review our PR in uh directly in GitHub.
07:56And after that we'll actually review what the coding uh the issues were that Codto found and we're going to resolve that automatically with an agent skill.
08:06[snorts]
08:06And what are we building?
08:07This is a self-contained fast API payments API and um there's no front end or anything.
08:14So we can just uh go ahead and focus on it'll be like a it'll be like an API that you can view in the UI.
08:23[snorts]
08:27Okay, some things to look at with the repo structure.
08:32We have an agents MD File.
08:34We have a hidden plan directory hidden um and rules, skills, docs.
08:41We have the actual API itself.
08:43We have tests.
08:44And we're using SER Grep as well for static analysis.
08:48And we have our CI workflow like verification um process.
08:53[snorts]
08:54So, first things first, we want to run this app locally.
08:58So, I'm we're going to use the instructions make setup once you have cloned this repo.
09:04And like I said, you can walk through this on your own and just copy and paste the uh instructions.
09:10Let's see.
09:11Where do I have Okay, once we've done that, it should be make run, I believe.
09:29All right, now it's running and locally it should be available.
09:38And this is the payments API that we have.
09:42Nothing crazy, very simple.
09:44Um, not really the point of this, but uh this is what you should see when the app is running locally.
09:56Okay.
09:57There's also some other endpoints available if you want to take a look at that.
10:01[snorts]
10:01Now, you can uh choose your lane if you want to just do this hands-on or if you just want to observe.
10:08Here are the quality gate checkpoints that we'll be walking through.
10:12And uh there's a whole setup gate process, right?
10:15We have these different commands that we're running and we have a verification command that is uh pretty important.
10:22So, you have this uh working already in your local repo.
10:26And if you need help, ask your agent for um changing anything locally to make sure that this can run very smoothly.
10:33The next step we're doing is we are going to use Kodto.
10:37And uh what's great about Kodto here is that we're not just going to run this in a pull request.
10:43We want to set up our account with Kodto so it can actually ingest the skills that are in this repo already and to ingest the rules that are in this repo.
10:52And this is going to be critical for us to be able to then apply that as context to the code review phase as well, not just with your your coding agents or when you're planning.
11:03Um, and like I said, you don't have to do this, but we can um we'll get to that part in a second.
11:10[snorts]
11:10So, some standards gates that are important to this are the rules themselves.
11:15We have different payments API focused rules that are contained in this repo and it is the agent's job to be able to determine when I have to implement this code change that you're requesting for me.
11:27Uh I need to determine which of these rules are going to be relevant for this so I can adhere to them.
11:33They're all here in this um in this repo as well as the skills for implementation.
11:40Um and then we have the planning and test-driven development gate.
11:43There is a highlevel plan in a planning template in this repo and then we have an execution focus plan and this is very detailed templates because from what I have found is the more detailed that you are
11:58uh relatively speaking uh the the more detailed that you are or consistent and structured with your planning uh phase and the document that your agent has to go back to and read
12:09then it's going to adhere a bit better and also correct itself.
12:13itself when it realizes it's missing some uh critical points of your uh process and you want to be able to identify what is the expected behavior what are the skills that I'm going to use for implementation
12:28all of that is here as a part of the planning template and then we have the local verification gates I use rough pyite bandit pi test sp and uh conventional commits
12:41deterministic evidence that the changes made locally preserve the local contract.
12:48I am a huge believer in combining deterministic tools with uh AI.
12:54[snorts]
12:54And then the last gate is the pull request review and the remediation gate.
12:58When we open a PR, Kodos findings are inspected, fixes and deferrals are documented and then we can rerun our verification command after we fix those um coding changes.
13:11And that is going to be a process of completing the code review loop um on your own and as opposed to just stopping right there at that local uh passing that local run.
13:22[snorts]
13:23Here are the prerequisites.
13:25We need git, GitHub account, uh GitHub CLI and Python, uh UV, Node.js JS and npm and any coding agent that you use, preferably co codeex or cloud code and then we will get our Kodto account.
13:42[snorts]
13:42Okay, so here are the setup uh instructions.
13:44I'll let you all do that.
13:46And we're going to sign into the Kodto app.
13:50This is app.codto.ai.
13:54And that's it.
13:55Um, if you can sign in here, you'll be able to get set up to connect the GitHub application to the forked repository.
14:03So, app.codto.ai.
14:06Um, use Who here has a Codto account already?
14:10Does anyone Okay, great.
14:12Um,
14:12[snorts]
14:13so yeah, sign in and sign in using your GitHub email.
14:16I think that's the easiest way.
14:18and uh and you will be able to uh you'll land let's see here at the integrations page or the homepage and you'll need to add a connection which will be the GitHub
14:32cloud uh connection and it will take you through the installation guide.
14:36I recommend not connecting like all of your repositories just the forked repo.
14:41It should be listed under there but let me know if it's not.
14:46[snorts]
14:46So, if I click on GitHub and the installation guide, connect your organization.
14:52Um, I'm already connected, but it should be listed right here for you to add.
14:59Um, this is just an example like we're using Kodo as an example of whether you use Kodo or not, you'll want skills and rules to live somewhere.
15:09Ideally, if you're working on a team, it's going to be outside of your local repo, unless you have a good use case for containing them in that repo and it's like hidden or something.
15:19If not, there are many there are a few context management layers that are out there.
15:24I think unblocked is one of them.
15:26And that's it's just a separate place for you to manage that context at at all times that you should be able to pull down wherever you are.
15:35Does that make sense?
15:39and us using Codto is just an example.
15:44Okay, when you've signed into the Kodto portal, you've connected GitHub to your fork, then you'll need to generate an API key from the Kodto portal and you can do that down here
15:56by clicking on API keys on your profile and create an API key and name it.
16:06You'll probably want to save this API key somewhere um locally so that you can then add it to codto.codto uh slashconfig.json.
16:21So generate the API key first and then you'll want to save it save that through um this uh JSON structure.
16:30And again, if you have access to the repo, you can have your coding agent do all of this for you.
16:35[snorts]
16:37We also have Kodo skills.
16:40You can use np npx skills add-toskills.
16:48If you don't want to use it, the skills are already local in in the uh repo.
16:55[snorts]
16:55Okay, this is the task that we're going to work on a payment workflow safety process.
17:02The task is to add one small payment workflow.
17:05It's going to be either a refund workflow or a capture retry handling workflow.
17:10You can choose which one you want, but this is an intentionally small change, right?
17:15with um some potential risks kind of built into um the code or or it's a sensitive aspect of this uh this payments API
17:24[snorts]
17:24and uh payment mutations are where a lot of AI generated code can look like plausibly correct but can introduce some issues.
17:33So I think it's a perfect example for us to walk through.
17:37[snorts]
17:38Um and yeah, Veru's going to use this all of this text really.
17:44We're going to have our coding agent use all of this to do either one of these tasks.
17:49Refund workflow or capture retry handling.
17:54[snorts]
17:56And here is where you can copy paste the agent prompts to get started.
18:00First, it's like help me configure this repo for the workshop.
18:03I have a Kodto API key from the portal if you're using that.
18:07And then here are the requirements for getting set up locally.
18:10Um I am going to since I already have this part I'm just going to copy and paste install or verify Kodto skills read agents.mmd and rules and select the relevant repo local rule ids for this hands-on task
18:29and I'm doing that in uh codeex.
18:32I wish I could.
18:34Let's see.
18:36[snorts]
18:38So, I'm in my uh I'm in the project and I'm just copy and pasting, getting my agent really ready to start working on this project, giving it that context that it needs.
18:55and raise your hand if you have any questions of where we're at at this point or if you want me to slow down or go back.
19:08Right here,
19:10installer verify codto skills.
19:13We're reading the agents.mmd file and the rules and then we are asking it to take a look at the repo rules for
19:21[snorts]
19:21the hands-on tasks that we're about to implement and explain why each of these rules will be relevant for the upcoming task and then I have optionally compare those rules with kodto get rules if those are those skills are available if you want to download the skills
19:42Okay,
19:42[snorts]
19:43let's see what is it saying.
19:44I'll ground this in the repo's own instructions first and verify the local codto setup and dependency health before selecting the rules.
19:53Okay, it found uh it's aware that we're inside the workshop project and it is has also identified that I have the Kodto skills.
20:09[snorts]
20:11Oh, the repo local workshop skills are not installed yet.
20:14So, I'm going to run the repos installer which will only copies committed local skill folders into the agent skills directory.
20:22Okay, great.
20:23So, it's going to process the local skills that we have.
20:26[snorts]
20:28And I'm going to let that run.
20:30I'll go back up to where we are at.
20:34It's the setup prompt.
20:44Let's see if it's made progress.
20:47After we are clear with the agents and they're good to go, then we're going to switch into plan mode.
20:53And so for this planning uh mode, we're saying use the local the repo local workshop planning skill to turn the hands-on task in the readme into a highle implementation
21:05plan
21:06[snorts]
21:06and a build session execution plan.
21:10And then we wanted to read agents, read the rules and uh use any relevant skills to determine what are the constraints.
21:19And this plan must include an implementation skill handoff.
21:23And that names the first uh uh that names the first implementation skill and the test-driven development prompt that it's going to be chained to and run next.
21:33Do not write the code yet.
21:35I'm a big fan of plan mode, but if you don't want to use plan mode, it still has this prompt to uh to
21:41[snorts]
21:42make sure that you're not going to implement anything.
21:46It is taking a while.
21:49Well, that's because I have all the verification stuff set up.
21:54All right,
21:56[snorts]
21:58let's see.
22:09[clears throat]
22:15All right.
22:16Okay.
22:16It's taking a while to do this.
22:28[snorts]
22:29I'm probably going to do the capture retry.
22:32Uh, I've What about you?
22:35Well, I was thinking the refund, but where do you tell it that you're doing that?
22:39Uh, it should prompt you and ask you.
22:41Okay.
22:42Um, and if not, then you
22:45[snorts]
22:46can make that clear as a part of the prompt.
22:51And what point are you at now?
22:53[snorts]
22:54Okay.
22:56Awesome.
22:58Okay.
22:58Finally, it's done.
22:59verified the repo instructions.
23:02It ran the install skills um instructions or uh command.
23:09All of the local skills are now pulled down.
23:12And here are the selected repo local rules for this payment workflow task.
23:21So these are all the rules that are going to be considered as a part of this uh process.
23:28[snorts]
23:29you know, maybe I'll do the refund uh workflow with you as well.
23:33I'm going to copy and paste the planning prompt and switch into plan mode.
23:37[snorts]
23:41Um, it should ask me, but I will say I will work on the refund.
23:49What was it?
23:51What is it called?
23:52The refund.
23:54[snorts]
23:57Uh, here we go.
24:00The refund workflow plan.
24:06[snorts]
24:20[snorts]
24:25Okay, let's see.
24:26So I'm going to use the repo local workshop plan from task skill and the payment item potency skill and then ground the plan in agents.mmd readme and the rules.
24:37I'll only read and inspect files.
24:39No code or planned files will be written in this particular mode.
24:48So
24:48[snorts]
24:48now codeex is assessing the current behavior of the app.
24:54We obviously we don't have this implemented um but it's determining okay what needs to be done what would take what would be the shape of the code change that needs to take place
25:05this is uh I should probably dig into the templates actually right now as codto is working so that you can see what the planning templates look like.
25:16So here are the planning templates.
25:20I truly believe that this is a really important artifact to be able to take with you if you're not already using a structured template.
25:30Um, I think that a plan should answer what behavior are we changing, which rules apply, what skills would guide the agent, which tests prove the expected behavior, and what kind of quality gates should pass before we review.
25:45[snorts]
25:46And it starts with a template for a highle plan and then a build session execution plan.
25:52And for this workshop, this is what we would expect it to look somewhat like.
26:02And here are the templates.
26:04We have a summary, the problem that we're solving, any particular constraints that we need to worry about, and then we have skill routing.
26:14This can be helpful if you need to plan ahead of all the skills that you would expect your coding agent during the implementation to start kicking off one by one,
26:23especially if you want to use some type of loops um or like goals in codeex or cloud code to work on your behalf without you stepping in.
26:33And
26:33[snorts]
26:33we have a scope behavior scenarios and verification gates, failure and recovery rules and a commit plan.
26:42This allows for a very structured process for uh your coding agents to work on very specific tasks, organize the code in a nice way that is presentable, make those commits in an organized way using conventional commits,
26:57and then get to a point where you can actually submit a pull request and everything is organized the same way that one would working on a team back when we didn't have AI.
27:08[snorts]
27:13and then the build execution template.
27:17What I've noticed is super helpful is having a documentation of the starting state.
27:23What branch are we in?
27:24Um what is the current local verification process?
27:28Are there any commands that are expected to run to determine that uh this code uh passes or fails?
27:34um any of the expectations with llinters and uh and then of course this is the implementation skill handoff and execution steps.
27:44Um I want the I want to know that the agent knows every single step in this process of making the code change and I want it to determine what the test plan is
27:55um as well.
27:57And then it's definition of done or completion notes.
28:00You need to make sure you've run all of the commands uh for this uh for this branch.
28:05You need to make sure that you've identified what was deferred.
28:08If you've changed your mind at some point, everything is documented in this plan.
28:13[snorts]
28:15And now we've got one, the refund workflow workshop plan.
28:21We've got our summary.
28:22We have the repo rules that are associated, the highle implementation plan, and the build session execution plan.
28:33And also there's some assumptions here.
28:35Um,
28:37[snorts]
28:38who here has arrived at this at this point?
28:42Okay, cool.
28:43So, I'm going to go ahead and implement this.
28:49[snorts]
28:52And while this is happening,
28:55[snorts]
29:01we can go to some of the slides.
29:07So what I was saying about operationalizing context, planning, codegen, and code review, this is what I found to be incredibly helpful for consistency across the SDLC.
29:19I use a combination of different agent skills and it depends on the repos that I'm working in.
29:25It depends on the programming languages and frameworks that I'm utilizing.
29:29But at a high level and so you'll see some Python skills here at the top like Python, Pyantic, Pyrite or Python typing uh contracts.
29:37Um a lot of these skills are available uh all over the internet or you can have your coding agent build it themselves.
29:43And there are some ones that I've gotten from from some popular um developers out there.
29:48Um, and those are one's from Aaron Francis are the audit skills.
29:55There's audit boundaries, drift, audit names, audit abstractions, dead code, and auditing errors.
30:02I found that these are incredibly powerful for really simplifying and improving the quality of code during the implementation um phase.
30:10I don't know if he has these publicly available, but if you can um look these up or find them or have your coding agent create them yourselves, this is definitely a step forward in like the right direction for quality.
30:24[snorts]
30:25Um as I said, enforcing quality early and often.
30:29So one of the values of using skills and the codto get rules here.
30:34This is about pulling down those rules from your context management layer and then determining um which uh which rules are going to be relevant for the task
30:44[snorts]
30:45and then I believe in then running local code review workflows.
30:49I don't care what you know tool that you use.
30:52If it's uh codeex, if it's cloud code, their built-in code review, if you're using Kodo, whatever it is, always run a local code review because there are always these small changes that you could probably make earlier before another developer has to see your code.
31:07And it's worth it to be able to address that sooner rather than later.
31:12We're trying to mitigate um issues um popping up.
31:16And then we're running automated code review in CI or git provider which is by default.
31:21[snorts]
31:22And when I did this um I walked through this entire workflow for an open source tool that I'm building.
31:28And the proof was in the amount of lines that I changed in a large PR.
31:33It was intentionally large because I want to test wanted to test the the quality of this actual workflow itself.
31:39And only one bug was uncovered in over,900 lines of code um using uh Kodto for that like automated code review and it was one reliability bug which made a lot of sense.
31:52Um but this was a really good exercise for me to see that this there was value in this process.
32:06[snorts]
32:08Yeah.
32:09So I I I wish I had like a graph I could um show you of how I have like higher level uh I have like planning templates that I'll use for like no matter what repo I'm in here is the plan structure that I expect and then there's overall code quality like at the highest level like engineering
32:29standards um that are specific or that are that can apply to any programming language and then like below that it's okay all the repos that use Python.
32:40Here are all the skills for that.
32:42And like you you kind of have to decide what would be most relevant depending on your use case.
32:50But the more general a skill is then the more access it you need to have it other many repos need to have to it.
33:01Is that does that make sense?
33:02Um take a look at this uh for example.
33:06Uh, let me make sure I'm scoping this just to my uh repos.
33:17Oh, okay.
33:18Okay.
33:19Well, take a look at this.
33:20We're in the Kodto portal and this is the rules section.
33:25So across multiple repos there are rules that were inferred and ingested and saved here in the portal and they are divided by there there's like a a rule name
33:38in general and like all the details that are generated here based off of the context from your repository and then they're divided by categories.
33:46There's accessibility, correctness, maintainability, architecture, security.
33:51Um, and then they're also uh organized by level of severity.
33:56Not every uh not every rule should be treated with the same level of severity.
34:01There are some rules that are strong enough to where you want to block a merge.
34:05There are others that are just like it's good practice, but this isn't something that it would be detrimental if it were to get through.
34:12[snorts]
34:12Um and yeah so this is these are some of the ways in which uh th these rules can be categorized and I believe they can also be scoped to specific repos if you want to get granular
34:28or if you want kod to infer that uh itself or whichever tool that you use.
34:33You can actually design that process yourself to scope those rules so that they rules or skills so that they only apply to very specific repositories.
34:46Can you can you What was that?
34:53I don't think so.
34:55I think Well, the only thing you can do is like filter on that in this dashboard.
35:00Um, but that's a really good question.
35:03I'll probably bring that back to the product team to to verify.
35:06Um, but what brought up that idea?
35:21the repositories,
35:23right?
35:24They're they're divided right now by org level and by um repo level um naturally or natively
35:34[snorts]
35:36that those were good questions.
35:37Yes.
35:38Yeah.
35:53I guess you know how
36:00[snorts]
36:14[snorts]
36:14I'm just curious.
36:19It can be.
36:20And so great question.
36:21How do rules become rules?
36:23Like how does COD determine um what is a rule or what isn't?
36:27Because it seems very automated.
36:29So you can it depends on how you've organized things already.
36:34If you have coding rules that are literally in a rules uh you know directory, it's going to be very easy for COD to be like okay every single one of the rules that I see here I'm just going to copy and paste like most of those contents
36:47um in and uh ingest it here into the portal.
36:50But there are some rules that will show up as uh suggestions and this is from specifically from reading or analyzing the actual codebase.
37:04Um so you might not have explicitly written these rules into your code but these are the conventions that Codto has noticed based on analyzing the entire codebase.
37:14It will not automatically create a rule for it, but it will come up as a suggestion for you to be able to review.
37:21And this is what will be generated.
37:23You can edit all of this.
37:25There are examples of good code examples and bad code um just to make it very clear to AI agents uh what good and bad is.
37:34And then you can define the category and the the warning levels.
37:38And then once you're happy with the way that this looks, you can then accept it uh and activate that rule and it'll automatically be applied.
37:48Yes.
37:49Keep going.
37:50on the other side.
37:52How does that kind of
38:07um for Kodto it's or
38:10well I saw like I'm going
38:23[snorts]
38:27Um I think for for this it's just pulling in the local uh repo rules that are most of them if not all of them are associated with the different paths to take in this workshop.
38:40Um but in general like the codto get rule skill is talking to the back end of the codto platform and
38:47[snorts]
38:48it's sort of like this agentic uh rag process of determining um what which ones are actually irrelevant and it filters it down
39:00yeah
39:07[snorts]
39:07that was a really good question Uh, okay.
39:10So, the plan was implemented and I don't know why it created a draft.
39:14I like real PRs, but fine, whatever.
39:18Um, copy the link.
39:25All right.
39:26Add refund workflow with quality gates.
39:28Did you get there?
39:29Uh, okay.
39:31[snorts]
39:31Awesome.
39:32And here we've got the summary.
39:36I'm just going to go ahead and make this a real PR because what I want is for COTO to review this without merging the pull request.
39:47[snorts]
39:48If COTO is connected to your repo, this is what you should see automatically pop up.
39:55It's going to start the analysis process.
39:59[snorts]
39:59And of course, I'm not looking at all of the code because we don't have all day to do that.
40:03Um but the you understand the point of uh implementing these processes.
40:09So in a few minutes um COD will be done analyzing this uh code and we'll bring up any potential issues and we're going to take a look at what those are.
40:18Um as a result of any potential coding issues that come up, we're going to run an agent skill to automatically resolve it.
40:29[snorts]
40:30Are there any other questions?
40:31Those were really good.
40:32Yes.
40:33I have a question and a question that follows from it is something I've seen with skills is they can be like super general like right try code and like this is what
40:42looks like and I have arguments with engineers where I'm like this is super general it should be in the agent's weights by now and like I think you should wait until something pops up but to settle these arguments it would be really useful to have data like this skill
40:54in the metric shows that it's actually useful does does Kodto offer anything like that or how do you know these skills are actually valuable versus just like Um, COD is just ingesting the skills.
41:05It's not evaluating uh the quality of them, but there's a tool that does do that.
41:10Tessle, have you heard of Tessle before?
41:13TeSSL is a um company that is all about skills eval.
41:18And you can determine it will tell you are there any security concerns with this skill or is this actually as effective as you expect it to be.
41:26That's a tool that I would recommend for like integrating that to uh to determine to determine that.
41:33Thank you.
41:34Yeah.
41:41Okay.
41:42Um, a PR summary was generated automatically.
41:46We've got a description and we've got a diagram of the changes and it's showing the legend API inmemory store in a function.
41:56We've got the API client and the fast API app and the different uh let's see the endpoint of the payment refund endpoint that's pointing to four different uh
42:10[snorts]
42:11fields.
42:17[snorts]
42:22Um, so one of the things that I would recommend, but I I realized I just skipped over was the verification prompt.
42:29So we can after doing a, you know, a a process of creating the the making the code change and actually doing a local review, we run a verification prompt to ensure that everything passes.
42:43But Codeex already did that automatically on its own um, just because of the way that the repo is is actually set up.
42:50But this is would be a best practice for walking through that process
42:55[snorts]
42:59and any day now codto would be great.
43:04[laughter]
43:05Well, we can take a look at a PR that I've already created.
43:10And this was a PR where I didn't run any of the processes that we were just talking about.
43:18But I do have the context uh pulled in through the Kodto portal and this is something that uh highly consider you looking into when it comes to that context management layer that I was talking about
43:32here in the code review.
43:34Codto identified that there were rules in the codto portal that are relevant for this actual code review.
43:41We're not talking about planning or implementation anymore.
43:44We're talking about code review specifically.
43:46And then it pulled in also the relevant skills um for this uh this code change.
43:54And here
43:55[snorts]
43:55there were some action required uh results that came up.
43:59There's a rule inside of this um inside of my org where Python functions must all have a dock string attached to it.
44:09And so this was shown as a rule violation and an issue around maintainability.
44:14The only reason why it's action required is because that was determined by our engineering org that this is of high as high of a severity because you can customize the level of severity of your rules.
44:25It doesn't have to be something that is blockw worthy.
44:28It just needs to be something that is addressed.
44:30That is up to you.
44:32[snorts]
44:33And here is the evidence that's associated with it.
44:35Require dock strings for all Python functions.
44:38Well, I would say methods.
44:39Yes.
44:48Yes.
45:03Yeah.
45:03[snorts]
45:03Um so this is being determined right during the code review phase of okay which rules are going to be relevant.
45:12The part that can be deterministic is that you can decide based on the level of severity of the rule this uh code this uh branch or PR will be will be blocked.
45:23And so that's something you can configure in the Kodto portal is in the case of any action required or high severity issue uh is identified we block the merge until you have fixed it and code will run again and verify that you have.
45:39in terms of making sure that
45:46[snorts]
45:46uh there's a potential that it won't determine that their relevant rule will actually Yes.
45:52Um and I I'm really glad that you brought that up because it is something that we have talked about with the product team.
45:59Um and the more uh enterprise customers that we speak to the especially in highly regulated industries, they really want determinism in exactly that fashion.
46:10Um so that's something I'm hoping we can push for for making a real thing because I think it's extremely important.
46:16I agree with you.
46:23Sorry.
46:24Yes.
46:25after fixes doer verification also because changed.
46:34Yeah.
46:39You mean this?
46:41Yes.
46:42Um after I've you've resolved issues.
46:46Yes.
46:46I think it's best practice too just in case.
46:49Um you you never you never know.
46:52So
46:52is it done automatically?
46:55um how many
47:00you're I think that's more on your your agent might be the one that will determine like oh I'm going to run verification or not and you can prompt your agent to do that anyway
47:12um regardless um but yeah so it depends on how your your agent kind of responds
47:22[snorts]
47:22and finally Here are the results.
47:26Oh, okay.
47:27Five skills were invoked.
47:29Uh, all right.
47:31So, we've got one bug, three rule violations, and two skill insights.
47:37There
47:37[snorts]
47:37is an issue with the refund item potency stored in memory.
47:43So, refund payment uses the process local uh dictionary to store and replay fund results, which is not durable persistent storage.
47:52and we have the piece of code that is related to that and we have the evidence as well, the rules and the following files.
48:02Um, the important part here that you all should definitely play around with is copy and pasting this agent prompt that is meant to fix the suggestions that uh pop up.
48:12Um, you can do that manually for each of the issues that present themselves or you can run the Kodto PR resolver to automatically resolve.
48:22Did you already run that that skill?
48:36I wanted everything to be code or be rechecked because
48:43[snorts]
48:44it can happen something else.
48:47Absolutely.
48:48It's important if we have any kind of automation to include verification section again.
48:56Yep.
48:56I think that's best practice and that's um that's what I would do or I guess pregenai era um just to be careful and that's what you want your agents to do as well.
49:08So that was a really uh good point.
49:11[snorts]
49:12Um after seeing all of the results in this uh PR, then that's when I ran the COD PR resolver skill and I ran it against the pull request number four
49:23and I saw that it's pulled down all of the issues that COD has already identified from critical to high.
49:32And where did it go?
49:36Oh, okay.
49:37So, it pulled down all of the results and now I can decide, do I want to review each issue individually or do I want to autofix all um for the sake of this uh workshop,
49:50I'm just going to autofix all.
49:52But, uh, this is something if you want to be a bit more hands-on and make decisions with your agent to actually plan how you're going to make the fix, you can do that yourself with the review each issue.
50:04Uh, um, option
50:07[snorts]
50:08autofix all.
50:16And now let's see how it reacts.
50:19So I'm going to autofix all six codto issues.
50:22And what should happen is okay.
50:27And you can see here it says then it's going to run verification and it's going to make the uh commits and it won't be the commit will be blocked if it's not structured in the correct way automatically.
50:40Um, but this is that point where it'll do that after and it'll take that next step further by actually posting back to the PR and uh with a summary of all the changes that were fixed.
50:54And from there, that's how the PR will be updated and all of the issues that were resolved will be uh crossed out.
51:12I I mean I've done I've used work trees for for that.
51:17Is that something that you've used?
51:20Yeah.
51:27Um yeah, that's not something that Kodto really handles.
51:31Like that's something I I found the coding agents kind of handle themselves.
51:36That's a good point though, a good question.
51:40Any other uh questions before I wrap this up?
51:47Yes.
51:51[snorts]
51:56Yes, you can run a code review.
51:58And I I didn't run that for this because time was flying.
52:01But um you mean for COD or for your coding your coding agent?
52:07So we are coming out with a COD review local workflow and it will talk directly to the Kodto backend.
52:15So all of the agentic capabilities that you would have in a pull request you would also have locally and be able to address.
52:28What was that?
52:31[snorts]
52:40How does it review all of the code?
52:42It clones the repo.
52:45Yeah, that's the process that we do on the back end.
52:49[snorts]
52:50It's like a temporary cloning just so it has an understanding of Yeah.
52:54the the broader scope of the changes beyond the code diff.
52:58[snorts]
52:58Really good question.
53:06Okay, I think we are about just about out of time.
53:09So,
53:09[snorts]
53:10it wasn't able to go through um everything in detail, but like I said, uh if nothing else that you take from this, take this repo and ask your AI some tips on what kind of verification
53:24steps that you would want to implement or you would want to practice implementing locally.
53:29Um, and I think that that would be a really good major step to just put add automatically add more structure into your workflows and consider any tools.
53:39There are some uh open-source tools that have similar context management layers to what Kodto does.
53:45I think um uh there's there's one of them out there, but if you look it up, you should try practicing having rules and skills and other um engineering standards, whatever it is, stored elsewhere
53:58that your um across so that you can pull down that information um when relevant to your the stages of your workflow.
54:08Oh yes, it's called packmind, I think.
54:12Packmind.com.
54:14[snorts]
54:15So definitely look into this if you want an open- source solution for um uh governing your AI code um through different rules and things like that consistently across um
54:29[snorts]
54:29your teams and your repos.
54:39[snorts]
54:41Okay.
54:41I wish the internet was faster, but it's not.
54:44So, I I I don't I can't uh this never happens, but here we are.
54:51Uh let me know if y'all have any questions.
54:54Um I'm I'm here, I guess, for the next couple minutes.
54:59Thank you.
55:01[applause]
55:07Let's
55:08Yes.
55:15I did.
55:22I'm at a point now where I don't clear like I let the the the context like compact and I haven't really had issues like I would have maybe three four months ago
55:32with the I guess the advancements that they have made.
55:36I just don't experience much degradation.
55:39Okay.
55:39like comp.
55:41Yeah, I do.
55:42Um, but if I really feel like it, I'll just create a new a new session and maybe copy and paste some of the stuff from the previous session into it.
55:54[snorts]
55:55Yes, we have a a massive booth here.
55:58So,
55:58yeah, check us out.
56:07Um, Yes.
56:10Yes.
56:10Do you have any local options for security reasons?
56:16Yes, we have like onrem deployments and pretty much all deployment options that you would want um because we do work with like banks and you know things like that and
56:27um companies like some teams at Nvidia there's like no other option but to do that.
56:33So yeah we do that integration.
56:39Thank you.
56:42[snorts]
56:47Let's see where did Oh, there it is.
56:52And here is the fix summary.
56:54[snorts]
57:00Yeah, it did.
57:01I'm like not even I don't even know what time it actually uh it is right now.
57:07And how many things were resolved?
57:10All of them.
57:11All issues were resolved.
57:15Awesome.
57:17Okay.
57:18Thank you all.
57:19Uh if you have any questions, I'll be out there.
57:22[snorts]
57:43[music]