WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=-AKPRSYccJc

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/-AKPRSYccJc.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:13.120 --> 00:00:20.080
Um, so my name is Nana Andukquay and I am uh developer relations lead at Kodto.

s2
00:00:20.080 --> 00:00:26.800
We're we used to be just like an AI code uh code review platform but now we're kind of expanding beyond that.

s3
00:00:26.800 --> 00:00:31.679
that's all about code quality and governance across the software development life cycle.

s4
00:00:31.679 --> 00:00:40.640
And I've spent a lot of time um thinking about and working on like what does a structured AI coding uh process look like.

s5
00:00:40.640 --> 00:00:42.800
There are many different ways that we're all doing it.

s6
00:00:42.800 --> 00:00:46.480
I would love to hear from y'all especially towards the end of this.

s7
00:00:46.480 --> 00:00:51.120
But I'm just going to have what I want is to walk through what that can look like.

s8
00:00:51.120 --> 00:00:54.079
There's a repo I have that everyone can fork.

s9
00:00:54.079 --> 00:01:04.640
And if you do nothing else from this, ask your AI to analyze and grab some tips from the structure of this the repo and try to see what you can apply to your own workflow.

s10
00:01:04.640 --> 00:01:06.640
All right.

s11
00:01:06.720 --> 00:01:07.119
All right.

s12
00:01:07.119 --> 00:01:12.159
So, we're going to walk through just a little bit of the beginning, who I am, Nana.

s13
00:01:12.159 --> 00:01:16.720
Um, definitely passionate about uh AI for developers.

s14
00:01:16.720 --> 00:01:21.680
This is my thing as a Devril and as a former uh full-time software engineer.

s15
00:01:21.680 --> 00:01:23.119
And I live in Boston.

s16
00:01:23.119 --> 00:01:25.119
And this is my cat, Pippen.

s17
00:01:25.119 --> 00:01:31.439
He's named after Lord of the Rings Hobbit, not the basketball player, which people ask me a lot about.

s18
00:01:31.439 --> 00:01:36.000
And you know, that's what this is what I'm here for, implementing structured AI coding workflow.

s19
00:01:36.000 --> 00:01:38.079
Um we're going to be using a COD.

s20
00:01:38.079 --> 00:01:40.400
Um you'll be able to sign into the platform.

s21
00:01:40.400 --> 00:01:50.000
If you don't want to use it for um some of the earlier stages of the SDLC we'll walk through, you can just use it for uh PR code reviews.

s22
00:01:50.799 --> 00:02:03.040
And uh first things first uh if you want to you you'll want to have access to the slides and I think the best way to do that or the easiest way to do that is to go to github.commyame

s23
00:02:04.399 --> 00:02:18.879
and then it should be the repo should be at the top of the repository or um those are the slides um that will link to my GitHub or my um my Google slides.

s24
00:02:18.879 --> 00:02:24.640
So hopefully everybody can I'll give everyone a minute to do that.

s25
00:02:25.920 --> 00:02:27.920
So yeah, slides are here.

s26
00:02:27.920 --> 00:02:31.520
The repository should be at the top of that.

s27
00:02:31.520 --> 00:02:37.840
And somebody anybody raise their hand if they don't see it or can't get access.

s28
00:02:43.680 --> 00:02:48.160
Thumbs up if I'm good to move on or or I'll wait.

s29
00:02:48.160 --> 00:02:50.160
Okay.

s30
00:02:54.560 --> 00:02:59.120
Um, so in general, where and how code quality can break down?

s31
00:02:59.120 --> 00:03:02.959
Uh, who who here is, you know, using AI for coding?

s32
00:03:02.959 --> 00:03:05.440
I'm assuming everybody.

s33
00:03:05.440 --> 00:03:12.319
We have do we have Claude code fans here and Codeex?

s34
00:03:12.959 --> 00:03:13.440
Wow.

s35
00:03:13.440 --> 00:03:15.040
Wait, less hands for Codeex.

s36
00:03:15.040 --> 00:03:15.280
Okay.

s37
00:03:15.280 --> 00:03:17.760
I thought they like blew up in the past couple months.

s38
00:03:17.760 --> 00:03:20.480
I've been using codecs for a lot of stuff.

s39
00:03:20.480 --> 00:03:26.480
But um where can code quality break down in the agentic development life cycle?

s40
00:03:26.480 --> 00:03:28.319
Uh everywhere.

s41
00:03:28.319 --> 00:03:32.239
Planning and design, development, code review, testing, deployment.

s42
00:03:32.239 --> 00:03:40.239
There are so many potential failure modes for um for using AI for coding just to ship high quality software.

s43
00:03:40.239 --> 00:03:56.799
And that is where we come into this process where we kind of have to overdo it with steering AI coding agents to get what we need out of it just so that we can get behind or stand behind the quality of the code and be like, okay, you know what, I can put my name on that.

s44
00:03:56.799 --> 00:03:57.840
This is good to go.

s45
00:03:57.840 --> 00:04:03.662
We can move this forward through the pipeline or I'm I feel safe to have another developer review my code at this point.

s46
00:04:03.662 --> 00:04:03.920
[snorts]

s47
00:04:03.920 --> 00:04:08.239
And so I really believe that we should be building qualitydriven systems.

s48
00:04:08.239 --> 00:04:23.440
I think that there needs to be a verification um aspect to agentic engineering and it really starts with number one defining what code quality means to you uh and your team if you're working on a team of developers

s49
00:04:23.440 --> 00:04:27.440
that needs to be defined and then that needs to be that needs to live somewhere.

s50
00:04:27.440 --> 00:04:40.880
it needs to be codified so that our AI, no matter what interface you're using, can extract that information and do its best to adhere to the code standards that you have in place.

s51
00:04:40.880 --> 00:04:55.360
And then once you've decided where those standards live, whether they're locally in your specific individual repos or if they're in a context management layer, which we'll get into, then you can decide and pull that into the different interfaces.

s52
00:04:55.360 --> 00:05:04.240
If that's your CLI, if that's your IDE, um your your CI workflows, it should be uh distributable.

s53
00:05:06.320 --> 00:05:10.160
And uh we do not have to go through all of that.

s54
00:05:10.160 --> 00:05:14.080
Um I already explained that we might just be able to Okay.

s55
00:05:14.080 --> 00:05:15.440
Well, here's what we're going to do.

s56
00:05:15.440 --> 00:05:19.680
We are going to operationalize our context.

s57
00:05:19.680 --> 00:05:21.600
Um I have a repo that has rules.

s58
00:05:21.600 --> 00:05:22.400
It has skills.

s59
00:05:22.400 --> 00:05:34.320
It has a it has you know a bunch of different things in it and we want to leverage that for planning for code generation and code review and the whole point is that we want to verify

s60
00:05:34.320 --> 00:05:38.638
the quality early and often and

s61
00:05:38.638 --> 00:05:38.639
[snorts]

s62
00:05:38.639 --> 00:05:46.240
ideally it's the same standards that we are asking the agents to check against across each of those phases.

s63
00:05:46.240 --> 00:05:49.520
That's that's really the whole point.

s64
00:05:51.360 --> 00:05:52.639
Oh, these are my skills.

s65
00:05:52.639 --> 00:05:54.240
I have so many.

s66
00:05:54.240 --> 00:06:02.320
Um, and yeah, so I think uh I'm going to go ahead and just jump into what we need to do.

s67
00:06:02.320 --> 00:06:14.000
Um, so first let's go to the Kodto quality workshop signal pay.

s68
00:06:14.080 --> 00:06:17.039
If y'all can see that.

s69
00:06:17.039 --> 00:06:24.800
Oh, I can't increase the uh let's see.

s70
00:06:30.400 --> 00:06:32.960
Okay, so this is the repo that we're going to.

s71
00:06:32.960 --> 00:06:35.520
It is Kodo Quality Workshop Signal Pay.

s72
00:06:35.520 --> 00:06:36.960
This is a payments app.

s73
00:06:36.960 --> 00:06:42.240
And what I want you to do is fork this repo and let me know when you have done that.

s74
00:06:42.240 --> 00:06:44.794
And then what you're going to do is clone it.

s75
00:06:44.794 --> 00:06:45.120
[snorts]

s76
00:06:45.120 --> 00:06:50.560
If you want to work through this on your own time, all of the instructions are there in the readme.

s77
00:06:50.560 --> 00:06:59.120
And I'm going to be leaning heavily on my coding agents to actually help me implement this, you know, like at a meta level.

s78
00:07:02.240 --> 00:07:06.800
So, code quality workshop-signal pay.

s79
00:07:06.960 --> 00:07:16.160
And in this repo, this is where we're just going to walk through the process of planning for a coding change.

s80
00:07:16.160 --> 00:07:22.560
We are going to read the repo rules to determine, okay, which rules are relevant for this particular coding change.

s81
00:07:22.560 --> 00:07:32.080
We're going to use skills and then make sure that we're writing proper tests using some TDD and BDD skills, test driven, behavior driven development.

s82
00:07:32.080 --> 00:07:39.120
Then we're going to ask our coding agent to implement um the code change and then run local gates.

s83
00:07:39.120 --> 00:07:49.120
We have llinters, we have um you know static analysis that we're running because we want some deterministic checks and we want AI as well like a good combination.

s84
00:07:49.120 --> 00:07:56.560
And then we'll open a pull request and have Codto uh review our PR in uh directly in GitHub.

s85
00:07:56.560 --> 00:08:06.015
And after that we'll actually review what the coding uh the issues were that Codto found and we're going to resolve that automatically with an agent skill.

s86
00:08:06.015 --> 00:08:06.240
[snorts]

s87
00:08:06.240 --> 00:08:07.680
And what are we building?

s88
00:08:07.680 --> 00:08:14.240
This is a self-contained fast API payments API and um there's no front end or anything.

s89
00:08:14.240 --> 00:08:23.599
So we can just uh go ahead and focus on it'll be like a it'll be like an API that you can view in the UI.

s90
00:08:23.700 --> 00:08:25.700
[snorts]

s91
00:08:27.840 --> 00:08:32.320
Okay, some things to look at with the repo structure.

s92
00:08:32.320 --> 00:08:34.880
We have an agents MD File.

s93
00:08:34.880 --> 00:08:41.519
We have a hidden plan directory hidden um and rules, skills, docs.

s94
00:08:41.519 --> 00:08:43.360
We have the actual API itself.

s95
00:08:43.360 --> 00:08:44.159
We have tests.

s96
00:08:44.159 --> 00:08:48.080
And we're using SER Grep as well for static analysis.

s97
00:08:48.080 --> 00:08:53.176
And we have our CI workflow like verification um process.

s98
00:08:53.176 --> 00:08:54.560
[snorts]

s99
00:08:54.560 --> 00:08:58.399
So, first things first, we want to run this app locally.

s100
00:08:58.399 --> 00:09:04.720
So, I'm we're going to use the instructions make setup once you have cloned this repo.

s101
00:09:04.720 --> 00:09:10.720
And like I said, you can walk through this on your own and just copy and paste the uh instructions.

s102
00:09:10.720 --> 00:09:11.279
Let's see.

s103
00:09:11.279 --> 00:09:25.920
Where do I have Okay, once we've done that, it should be make run, I believe.

s104
00:09:29.360 --> 00:09:38.560
All right, now it's running and locally it should be available.

s105
00:09:38.560 --> 00:09:42.240
And this is the payments API that we have.

s106
00:09:42.240 --> 00:09:44.000
Nothing crazy, very simple.

s107
00:09:44.000 --> 00:09:51.279
Um, not really the point of this, but uh this is what you should see when the app is running locally.

s108
00:09:56.640 --> 00:09:57.040
Okay.

s109
00:09:57.040 --> 00:10:01.296
There's also some other endpoints available if you want to take a look at that.

s110
00:10:01.296 --> 00:10:01.360
[snorts]

s111
00:10:01.360 --> 00:10:08.399
Now, you can uh choose your lane if you want to just do this hands-on or if you just want to observe.

s112
00:10:08.399 --> 00:10:12.480
Here are the quality gate checkpoints that we'll be walking through.

s113
00:10:12.480 --> 00:10:15.680
And uh there's a whole setup gate process, right?

s114
00:10:15.680 --> 00:10:22.000
We have these different commands that we're running and we have a verification command that is uh pretty important.

s115
00:10:22.000 --> 00:10:26.320
So, you have this uh working already in your local repo.

s116
00:10:26.320 --> 00:10:33.680
And if you need help, ask your agent for um changing anything locally to make sure that this can run very smoothly.

s117
00:10:33.680 --> 00:10:37.200
The next step we're doing is we are going to use Kodto.

s118
00:10:37.200 --> 00:10:43.600
And uh what's great about Kodto here is that we're not just going to run this in a pull request.

s119
00:10:43.600 --> 00:10:52.800
We want to set up our account with Kodto so it can actually ingest the skills that are in this repo already and to ingest the rules that are in this repo.

s120
00:10:52.800 --> 00:11:03.519
And this is going to be critical for us to be able to then apply that as context to the code review phase as well, not just with your your coding agents or when you're planning.

s121
00:11:03.519 --> 00:11:10.727
Um, and like I said, you don't have to do this, but we can um we'll get to that part in a second.

s122
00:11:10.727 --> 00:11:10.800
[snorts]

s123
00:11:10.800 --> 00:11:15.360
So, some standards gates that are important to this are the rules themselves.

s124
00:11:15.360 --> 00:11:27.360
We have different payments API focused rules that are contained in this repo and it is the agent's job to be able to determine when I have to implement this code change that you're requesting for me.

s125
00:11:27.360 --> 00:11:33.360
Uh I need to determine which of these rules are going to be relevant for this so I can adhere to them.

s126
00:11:33.360 --> 00:11:40.480
They're all here in this um in this repo as well as the skills for implementation.

s127
00:11:40.480 --> 00:11:43.760
Um and then we have the planning and test-driven development gate.

s128
00:11:43.760 --> 00:11:58.240
There is a highlevel plan in a planning template in this repo and then we have an execution focus plan and this is very detailed templates because from what I have found is the more detailed that you are

s129
00:11:58.240 --> 00:12:09.920
uh relatively speaking uh the the more detailed that you are or consistent and structured with your planning uh phase and the document that your agent has to go back to and read

s130
00:12:09.920 --> 00:12:13.440
then it's going to adhere a bit better and also correct itself.

s131
00:12:13.440 --> 00:12:28.079
itself when it realizes it's missing some uh critical points of your uh process and you want to be able to identify what is the expected behavior what are the skills that I'm going to use for implementation

s132
00:12:28.079 --> 00:12:41.600
all of that is here as a part of the planning template and then we have the local verification gates I use rough pyite bandit pi test sp and uh conventional commits

s133
00:12:41.600 --> 00:12:48.079
deterministic evidence that the changes made locally preserve the local contract.

s134
00:12:48.079 --> 00:12:54.218
I am a huge believer in combining deterministic tools with uh AI.

s135
00:12:54.218 --> 00:12:54.560
[snorts]

s136
00:12:54.560 --> 00:12:58.720
And then the last gate is the pull request review and the remediation gate.

s137
00:12:58.720 --> 00:13:11.440
When we open a PR, Kodos findings are inspected, fixes and deferrals are documented and then we can rerun our verification command after we fix those um coding changes.

s138
00:13:11.440 --> 00:13:22.384
And that is going to be a process of completing the code review loop um on your own and as opposed to just stopping right there at that local uh passing that local run.

s139
00:13:22.384 --> 00:13:23.600
[snorts]

s140
00:13:23.600 --> 00:13:25.120
Here are the prerequisites.

s141
00:13:25.120 --> 00:13:41.680
We need git, GitHub account, uh GitHub CLI and Python, uh UV, Node.js JS and npm and any coding agent that you use, preferably co codeex or cloud code and then we will get our Kodto account.

s142
00:13:42.034 --> 00:13:42.079
[snorts]

s143
00:13:42.079 --> 00:13:44.720
Okay, so here are the setup uh instructions.

s144
00:13:44.720 --> 00:13:46.000
I'll let you all do that.

s145
00:13:46.000 --> 00:13:50.560
And we're going to sign into the Kodto app.

s146
00:13:50.560 --> 00:13:54.079
This is app.codto.ai.

s147
00:13:54.639 --> 00:13:55.680
And that's it.

s148
00:13:55.680 --> 00:14:03.839
Um, if you can sign in here, you'll be able to get set up to connect the GitHub application to the forked repository.

s149
00:14:03.839 --> 00:14:06.160
So, app.codto.ai.

s150
00:14:06.399 --> 00:14:10.399
Um, use Who here has a Codto account already?

s151
00:14:10.399 --> 00:14:12.720
Does anyone Okay, great.

s152
00:14:12.720 --> 00:14:12.819
Um,

s153
00:14:12.819 --> 00:14:13.279
[snorts]

s154
00:14:13.279 --> 00:14:16.959
so yeah, sign in and sign in using your GitHub email.

s155
00:14:16.959 --> 00:14:18.959
I think that's the easiest way.

s156
00:14:18.959 --> 00:14:32.000
and uh and you will be able to uh you'll land let's see here at the integrations page or the homepage and you'll need to add a connection which will be the GitHub

s157
00:14:32.000 --> 00:14:36.880
cloud uh connection and it will take you through the installation guide.

s158
00:14:36.880 --> 00:14:41.920
I recommend not connecting like all of your repositories just the forked repo.

s159
00:14:41.920 --> 00:14:46.225
It should be listed under there but let me know if it's not.

s160
00:14:46.225 --> 00:14:46.320
[snorts]

s161
00:14:46.320 --> 00:14:52.079
So, if I click on GitHub and the installation guide, connect your organization.

s162
00:14:52.079 --> 00:14:59.360
Um, I'm already connected, but it should be listed right here for you to add.

s163
00:14:59.360 --> 00:15:09.839
Um, this is just an example like we're using Kodo as an example of whether you use Kodo or not, you'll want skills and rules to live somewhere.

s164
00:15:09.839 --> 00:15:19.600
Ideally, if you're working on a team, it's going to be outside of your local repo, unless you have a good use case for containing them in that repo and it's like hidden or something.

s165
00:15:19.600 --> 00:15:24.240
If not, there are many there are a few context management layers that are out there.

s166
00:15:24.240 --> 00:15:26.480
I think unblocked is one of them.

s167
00:15:26.480 --> 00:15:35.120
And that's it's just a separate place for you to manage that context at at all times that you should be able to pull down wherever you are.

s168
00:15:35.120 --> 00:15:37.680
Does that make sense?

s169
00:15:39.839 --> 00:15:44.000
and us using Codto is just an example.

s170
00:15:44.720 --> 00:15:56.720
Okay, when you've signed into the Kodto portal, you've connected GitHub to your fork, then you'll need to generate an API key from the Kodto portal and you can do that down here

s171
00:15:56.720 --> 00:16:05.199
by clicking on API keys on your profile and create an API key and name it.

s172
00:16:06.639 --> 00:16:20.959
You'll probably want to save this API key somewhere um locally so that you can then add it to codto.codto uh slashconfig.json.

s173
00:16:21.199 --> 00:16:30.560
So generate the API key first and then you'll want to save it save that through um this uh JSON structure.

s174
00:16:30.560 --> 00:16:35.611
And again, if you have access to the repo, you can have your coding agent do all of this for you.

s175
00:16:35.611 --> 00:16:37.611
[snorts]

s176
00:16:37.839 --> 00:16:40.079
We also have Kodo skills.

s177
00:16:40.079 --> 00:16:44.320
You can use np npx skills add-toskills.

s178
00:16:48.160 --> 00:16:54.720
If you don't want to use it, the skills are already local in in the uh repo.

s179
00:16:55.261 --> 00:16:55.680
[snorts]

s180
00:16:55.680 --> 00:17:02.480
Okay, this is the task that we're going to work on a payment workflow safety process.

s181
00:17:02.480 --> 00:17:05.280
The task is to add one small payment workflow.

s182
00:17:05.280 --> 00:17:10.400
It's going to be either a refund workflow or a capture retry handling workflow.

s183
00:17:10.400 --> 00:17:15.280
You can choose which one you want, but this is an intentionally small change, right?

s184
00:17:15.280 --> 00:17:24.736
with um some potential risks kind of built into um the code or or it's a sensitive aspect of this uh this payments API

s185
00:17:24.736 --> 00:17:24.880
[snorts]

s186
00:17:24.880 --> 00:17:33.679
and uh payment mutations are where a lot of AI generated code can look like plausibly correct but can introduce some issues.

s187
00:17:33.679 --> 00:17:37.182
So I think it's a perfect example for us to walk through.

s188
00:17:37.182 --> 00:17:38.080
[snorts]

s189
00:17:38.080 --> 00:17:44.559
Um and yeah, Veru's going to use this all of this text really.

s190
00:17:44.559 --> 00:17:49.039
We're going to have our coding agent use all of this to do either one of these tasks.

s191
00:17:49.039 --> 00:17:52.960
Refund workflow or capture retry handling.

s192
00:17:54.867 --> 00:17:56.640
[snorts]

s193
00:17:56.640 --> 00:18:00.720
And here is where you can copy paste the agent prompts to get started.

s194
00:18:00.720 --> 00:18:03.520
First, it's like help me configure this repo for the workshop.

s195
00:18:03.520 --> 00:18:07.120
I have a Kodto API key from the portal if you're using that.

s196
00:18:07.120 --> 00:18:10.640
And then here are the requirements for getting set up locally.

s197
00:18:10.640 --> 00:18:29.200
Um I am going to since I already have this part I'm just going to copy and paste install or verify Kodto skills read agents.mmd and rules and select the relevant repo local rule ids for this hands-on task

s198
00:18:29.200 --> 00:18:32.960
and I'm doing that in uh codeex.

s199
00:18:32.960 --> 00:18:34.880
I wish I could.

s200
00:18:34.880 --> 00:18:36.787
Let's see.

s201
00:18:36.787 --> 00:18:38.400
[snorts]

s202
00:18:38.400 --> 00:18:51.039
So, I'm in my uh I'm in the project and I'm just copy and pasting, getting my agent really ready to start working on this project, giving it that context that it needs.

s203
00:18:55.600 --> 00:19:04.240
and raise your hand if you have any questions of where we're at at this point or if you want me to slow down or go back.

s204
00:19:08.640 --> 00:19:10.880
Right here,

s205
00:19:10.880 --> 00:19:13.039
installer verify codto skills.

s206
00:19:13.039 --> 00:19:21.328
We're reading the agents.mmd file and the rules and then we are asking it to take a look at the repo rules for

s207
00:19:21.328 --> 00:19:21.600
[snorts]

s208
00:19:21.600 --> 00:19:40.240
the hands-on tasks that we're about to implement and explain why each of these rules will be relevant for the upcoming task and then I have optionally compare those rules with kodto get rules if those are those skills are available if you want to download the skills

s209
00:19:42.720 --> 00:19:42.943
Okay,

s210
00:19:42.943 --> 00:19:43.039
[snorts]

s211
00:19:43.039 --> 00:19:44.559
let's see what is it saying.

s212
00:19:44.559 --> 00:19:53.840
I'll ground this in the repo's own instructions first and verify the local codto setup and dependency health before selecting the rules.

s213
00:19:53.840 --> 00:20:09.143
Okay, it found uh it's aware that we're inside the workshop project and it is has also identified that I have the Kodto skills.

s214
00:20:09.143 --> 00:20:11.143
[snorts]

s215
00:20:11.679 --> 00:20:14.799
Oh, the repo local workshop skills are not installed yet.

s216
00:20:14.799 --> 00:20:22.720
So, I'm going to run the repos installer which will only copies committed local skill folders into the agent skills directory.

s217
00:20:22.720 --> 00:20:23.360
Okay, great.

s218
00:20:23.360 --> 00:20:26.829
So, it's going to process the local skills that we have.

s219
00:20:26.829 --> 00:20:28.080
[snorts]

s220
00:20:28.080 --> 00:20:30.480
And I'm going to let that run.

s221
00:20:30.480 --> 00:20:34.080
I'll go back up to where we are at.

s222
00:20:34.080 --> 00:20:36.960
It's the setup prompt.

s223
00:20:44.880 --> 00:20:47.440
Let's see if it's made progress.

s224
00:20:47.440 --> 00:20:53.760
After we are clear with the agents and they're good to go, then we're going to switch into plan mode.

s225
00:20:53.760 --> 00:21:05.440
And so for this planning uh mode, we're saying use the local the repo local workshop planning skill to turn the hands-on task in the readme into a highle implementation

s226
00:21:05.440 --> 00:21:06.129
plan

s227
00:21:06.129 --> 00:21:06.400
[snorts]

s228
00:21:06.400 --> 00:21:10.080
and a build session execution plan.

s229
00:21:10.080 --> 00:21:19.039
And then we wanted to read agents, read the rules and uh use any relevant skills to determine what are the constraints.

s230
00:21:19.039 --> 00:21:23.120
And this plan must include an implementation skill handoff.

s231
00:21:23.120 --> 00:21:33.520
And that names the first uh uh that names the first implementation skill and the test-driven development prompt that it's going to be chained to and run next.

s232
00:21:33.520 --> 00:21:35.039
Do not write the code yet.

s233
00:21:35.039 --> 00:21:41.499
I'm a big fan of plan mode, but if you don't want to use plan mode, it still has this prompt to uh to

s234
00:21:41.499 --> 00:21:42.799
[snorts]

s235
00:21:42.799 --> 00:21:46.240
make sure that you're not going to implement anything.

s236
00:21:46.559 --> 00:21:49.600
It is taking a while.

s237
00:21:49.760 --> 00:21:54.080
Well, that's because I have all the verification stuff set up.

s238
00:21:54.960 --> 00:21:56.565
All right,

s239
00:21:56.565 --> 00:21:58.565
[snorts]

s240
00:21:58.960 --> 00:22:01.280
let's see.

s241
00:22:09.665 --> 00:22:11.665
[clears throat]

s242
00:22:15.360 --> 00:22:16.080
All right.

s243
00:22:16.080 --> 00:22:16.480
Okay.

s244
00:22:16.480 --> 00:22:22.799
It's taking a while to do this.

s245
00:22:28.660 --> 00:22:29.120
[snorts]

s246
00:22:29.120 --> 00:22:32.240
I'm probably going to do the capture retry.

s247
00:22:32.240 --> 00:22:35.600
Uh, I've What about you?

s248
00:22:35.600 --> 00:22:39.120
Well, I was thinking the refund, but where do you tell it that you're doing that?

s249
00:22:39.120 --> 00:22:41.840
Uh, it should prompt you and ask you.

s250
00:22:41.840 --> 00:22:42.799
Okay.

s251
00:22:42.799 --> 00:22:45.700
Um, and if not, then you

s252
00:22:45.700 --> 00:22:46.000
[snorts]

s253
00:22:46.000 --> 00:22:50.720
can make that clear as a part of the prompt.

s254
00:22:51.440 --> 00:22:53.550
And what point are you at now?

s255
00:22:53.550 --> 00:22:54.960
[snorts]

s256
00:22:54.960 --> 00:22:56.159
Okay.

s257
00:22:56.159 --> 00:22:58.000
Awesome.

s258
00:22:58.000 --> 00:22:58.400
Okay.

s259
00:22:58.400 --> 00:22:59.760
Finally, it's done.

s260
00:22:59.760 --> 00:23:02.559
verified the repo instructions.

s261
00:23:02.559 --> 00:23:09.440
It ran the install skills um instructions or uh command.

s262
00:23:09.440 --> 00:23:12.640
All of the local skills are now pulled down.

s263
00:23:12.640 --> 00:23:18.960
And here are the selected repo local rules for this payment workflow task.

s264
00:23:21.280 --> 00:23:28.159
So these are all the rules that are going to be considered as a part of this uh process.

s265
00:23:28.921 --> 00:23:29.280
[snorts]

s266
00:23:29.280 --> 00:23:33.200
you know, maybe I'll do the refund uh workflow with you as well.

s267
00:23:33.200 --> 00:23:37.436
I'm going to copy and paste the planning prompt and switch into plan mode.

s268
00:23:37.436 --> 00:23:39.436
[snorts]

s269
00:23:41.039 --> 00:23:49.919
Um, it should ask me, but I will say I will work on the refund.

s270
00:23:49.919 --> 00:23:51.760
What was it?

s271
00:23:51.760 --> 00:23:52.480
What is it called?

s272
00:23:52.480 --> 00:23:54.466
The refund.

s273
00:23:54.466 --> 00:23:56.466
[snorts]

s274
00:23:57.360 --> 00:24:00.559
Uh, here we go.

s275
00:24:00.559 --> 00:24:06.256
The refund workflow plan.

s276
00:24:06.256 --> 00:24:08.256
[snorts]

s277
00:24:20.666 --> 00:24:22.666
[snorts]

s278
00:24:25.200 --> 00:24:26.640
Okay, let's see.

s279
00:24:26.640 --> 00:24:37.760
So I'm going to use the repo local workshop plan from task skill and the payment item potency skill and then ground the plan in agents.mmd readme and the rules.

s280
00:24:37.760 --> 00:24:39.440
I'll only read and inspect files.

s281
00:24:39.440 --> 00:24:44.799
No code or planned files will be written in this particular mode.

s282
00:24:48.720 --> 00:24:48.832
So

s283
00:24:48.832 --> 00:24:48.960
[snorts]

s284
00:24:48.960 --> 00:24:54.080
now codeex is assessing the current behavior of the app.

s285
00:24:54.080 --> 00:25:05.200
We obviously we don't have this implemented um but it's determining okay what needs to be done what would take what would be the shape of the code change that needs to take place

s286
00:25:05.200 --> 00:25:15.600
this is uh I should probably dig into the templates actually right now as codto is working so that you can see what the planning templates look like.

s287
00:25:16.640 --> 00:25:20.400
So here are the planning templates.

s288
00:25:20.400 --> 00:25:30.080
I truly believe that this is a really important artifact to be able to take with you if you're not already using a structured template.

s289
00:25:30.080 --> 00:25:45.817
Um, I think that a plan should answer what behavior are we changing, which rules apply, what skills would guide the agent, which tests prove the expected behavior, and what kind of quality gates should pass before we review.

s290
00:25:45.817 --> 00:25:46.799
[snorts]

s291
00:25:46.799 --> 00:25:52.720
And it starts with a template for a highle plan and then a build session execution plan.

s292
00:25:52.720 --> 00:25:58.960
And for this workshop, this is what we would expect it to look somewhat like.

s293
00:26:02.159 --> 00:26:04.880
And here are the templates.

s294
00:26:04.880 --> 00:26:14.159
We have a summary, the problem that we're solving, any particular constraints that we need to worry about, and then we have skill routing.

s295
00:26:14.159 --> 00:26:23.679
This can be helpful if you need to plan ahead of all the skills that you would expect your coding agent during the implementation to start kicking off one by one,

s296
00:26:23.679 --> 00:26:33.440
especially if you want to use some type of loops um or like goals in codeex or cloud code to work on your behalf without you stepping in.

s297
00:26:33.440 --> 00:26:33.633
And

s298
00:26:33.633 --> 00:26:33.679
[snorts]

s299
00:26:33.679 --> 00:26:42.400
we have a scope behavior scenarios and verification gates, failure and recovery rules and a commit plan.

s300
00:26:42.400 --> 00:26:57.120
This allows for a very structured process for uh your coding agents to work on very specific tasks, organize the code in a nice way that is presentable, make those commits in an organized way using conventional commits,

s301
00:26:57.120 --> 00:27:08.348
and then get to a point where you can actually submit a pull request and everything is organized the same way that one would working on a team back when we didn't have AI.

s302
00:27:08.348 --> 00:27:10.348
[snorts]

s303
00:27:13.039 --> 00:27:17.279
and then the build execution template.

s304
00:27:17.600 --> 00:27:23.200
What I've noticed is super helpful is having a documentation of the starting state.

s305
00:27:23.200 --> 00:27:24.720
What branch are we in?

s306
00:27:24.720 --> 00:27:28.159
Um what is the current local verification process?

s307
00:27:28.159 --> 00:27:34.799
Are there any commands that are expected to run to determine that uh this code uh passes or fails?

s308
00:27:34.799 --> 00:27:44.880
um any of the expectations with llinters and uh and then of course this is the implementation skill handoff and execution steps.

s309
00:27:44.880 --> 00:27:55.760
Um I want the I want to know that the agent knows every single step in this process of making the code change and I want it to determine what the test plan is

s310
00:27:55.760 --> 00:27:57.679
um as well.

s311
00:27:57.679 --> 00:28:00.480
And then it's definition of done or completion notes.

s312
00:28:00.480 --> 00:28:05.520
You need to make sure you've run all of the commands uh for this uh for this branch.

s313
00:28:05.520 --> 00:28:08.640
You need to make sure that you've identified what was deferred.

s314
00:28:08.640 --> 00:28:13.849
If you've changed your mind at some point, everything is documented in this plan.

s315
00:28:13.849 --> 00:28:15.760
[snorts]

s316
00:28:15.760 --> 00:28:21.120
And now we've got one, the refund workflow workshop plan.

s317
00:28:21.120 --> 00:28:22.640
We've got our summary.

s318
00:28:22.640 --> 00:28:31.279
We have the repo rules that are associated, the highle implementation plan, and the build session execution plan.

s319
00:28:33.200 --> 00:28:35.600
And also there's some assumptions here.

s320
00:28:35.600 --> 00:28:37.429
Um,

s321
00:28:37.429 --> 00:28:38.080
[snorts]

s322
00:28:38.080 --> 00:28:42.159
who here has arrived at this at this point?

s323
00:28:42.799 --> 00:28:43.919
Okay, cool.

s324
00:28:43.919 --> 00:28:49.120
So, I'm going to go ahead and implement this.

s325
00:28:49.874 --> 00:28:51.874
[snorts]

s326
00:28:52.080 --> 00:28:55.769
And while this is happening,

s327
00:28:55.769 --> 00:28:57.769
[snorts]

s328
00:29:01.200 --> 00:29:07.039
we can go to some of the slides.

s329
00:29:07.919 --> 00:29:19.039
So what I was saying about operationalizing context, planning, codegen, and code review, this is what I found to be incredibly helpful for consistency across the SDLC.

s330
00:29:19.039 --> 00:29:25.360
I use a combination of different agent skills and it depends on the repos that I'm working in.

s331
00:29:25.360 --> 00:29:29.360
It depends on the programming languages and frameworks that I'm utilizing.

s332
00:29:29.360 --> 00:29:37.520
But at a high level and so you'll see some Python skills here at the top like Python, Pyantic, Pyrite or Python typing uh contracts.

s333
00:29:37.520 --> 00:29:43.760
Um a lot of these skills are available uh all over the internet or you can have your coding agent build it themselves.

s334
00:29:43.760 --> 00:29:48.559
And there are some ones that I've gotten from from some popular um developers out there.

s335
00:29:48.559 --> 00:29:55.120
Um, and those are one's from Aaron Francis are the audit skills.

s336
00:29:55.120 --> 00:30:02.080
There's audit boundaries, drift, audit names, audit abstractions, dead code, and auditing errors.

s337
00:30:02.080 --> 00:30:10.960
I found that these are incredibly powerful for really simplifying and improving the quality of code during the implementation um phase.

s338
00:30:10.960 --> 00:30:24.195
I don't know if he has these publicly available, but if you can um look these up or find them or have your coding agent create them yourselves, this is definitely a step forward in like the right direction for quality.

s339
00:30:24.195 --> 00:30:25.760
[snorts]

s340
00:30:25.760 --> 00:30:29.520
Um as I said, enforcing quality early and often.

s341
00:30:29.520 --> 00:30:34.159
So one of the values of using skills and the codto get rules here.

s342
00:30:34.159 --> 00:30:44.501
This is about pulling down those rules from your context management layer and then determining um which uh which rules are going to be relevant for the task

s343
00:30:44.501 --> 00:30:45.120
[snorts]

s344
00:30:45.120 --> 00:30:49.279
and then I believe in then running local code review workflows.

s345
00:30:49.279 --> 00:30:52.240
I don't care what you know tool that you use.

s346
00:30:52.240 --> 00:31:07.919
If it's uh codeex, if it's cloud code, their built-in code review, if you're using Kodo, whatever it is, always run a local code review because there are always these small changes that you could probably make earlier before another developer has to see your code.

s347
00:31:07.919 --> 00:31:12.880
And it's worth it to be able to address that sooner rather than later.

s348
00:31:12.880 --> 00:31:16.240
We're trying to mitigate um issues um popping up.

s349
00:31:16.240 --> 00:31:21.836
And then we're running automated code review in CI or git provider which is by default.

s350
00:31:21.836 --> 00:31:22.240
[snorts]

s351
00:31:22.240 --> 00:31:28.159
And when I did this um I walked through this entire workflow for an open source tool that I'm building.

s352
00:31:28.159 --> 00:31:33.039
And the proof was in the amount of lines that I changed in a large PR.

s353
00:31:33.039 --> 00:31:39.600
It was intentionally large because I want to test wanted to test the the quality of this actual workflow itself.

s354
00:31:39.600 --> 00:31:52.240
And only one bug was uncovered in over,900 lines of code um using uh Kodto for that like automated code review and it was one reliability bug which made a lot of sense.

s355
00:31:52.240 --> 00:31:59.600
Um but this was a really good exercise for me to see that this there was value in this process.

s356
00:32:06.377 --> 00:32:08.377
[snorts]

s357
00:32:08.720 --> 00:32:09.120
Yeah.

s358
00:32:09.120 --> 00:32:29.919
So I I I wish I had like a graph I could um show you of how I have like higher level uh I have like planning templates that I'll use for like no matter what repo I'm in here is the plan structure that I expect and then there's overall code quality like at the highest level like engineering

s359
00:32:29.919 --> 00:32:40.399
standards um that are specific or that are that can apply to any programming language and then like below that it's okay all the repos that use Python.

s360
00:32:40.399 --> 00:32:42.640
Here are all the skills for that.

s361
00:32:42.640 --> 00:32:50.240
And like you you kind of have to decide what would be most relevant depending on your use case.

s362
00:32:50.240 --> 00:33:01.120
But the more general a skill is then the more access it you need to have it other many repos need to have to it.

s363
00:33:01.120 --> 00:33:02.559
Is that does that make sense?

s364
00:33:02.559 --> 00:33:06.960
Um take a look at this uh for example.

s365
00:33:06.960 --> 00:33:17.120
Uh, let me make sure I'm scoping this just to my uh repos.

s366
00:33:17.120 --> 00:33:18.720
Oh, okay.

s367
00:33:18.720 --> 00:33:19.039
Okay.

s368
00:33:19.039 --> 00:33:20.080
Well, take a look at this.

s369
00:33:20.080 --> 00:33:25.120
We're in the Kodto portal and this is the rules section.

s370
00:33:25.120 --> 00:33:38.399
So across multiple repos there are rules that were inferred and ingested and saved here in the portal and they are divided by there there's like a a rule name

s371
00:33:38.399 --> 00:33:46.640
in general and like all the details that are generated here based off of the context from your repository and then they're divided by categories.

s372
00:33:46.640 --> 00:33:51.519
There's accessibility, correctness, maintainability, architecture, security.

s373
00:33:51.519 --> 00:33:56.720
Um, and then they're also uh organized by level of severity.

s374
00:33:56.720 --> 00:34:01.919
Not every uh not every rule should be treated with the same level of severity.

s375
00:34:01.919 --> 00:34:05.279
There are some rules that are strong enough to where you want to block a merge.

s376
00:34:05.279 --> 00:34:12.793
There are others that are just like it's good practice, but this isn't something that it would be detrimental if it were to get through.

s377
00:34:12.793 --> 00:34:12.800
[snorts]

s378
00:34:12.800 --> 00:34:28.800
Um and yeah so this is these are some of the ways in which uh th these rules can be categorized and I believe they can also be scoped to specific repos if you want to get granular

s379
00:34:28.800 --> 00:34:33.599
or if you want kod to infer that uh itself or whichever tool that you use.

s380
00:34:33.599 --> 00:34:44.320
You can actually design that process yourself to scope those rules so that they rules or skills so that they only apply to very specific repositories.

s381
00:34:46.000 --> 00:34:49.440
Can you can you What was that?

s382
00:34:53.359 --> 00:34:55.599
I don't think so.

s383
00:34:55.599 --> 00:35:00.960
I think Well, the only thing you can do is like filter on that in this dashboard.

s384
00:35:00.960 --> 00:35:03.200
Um, but that's a really good question.

s385
00:35:03.200 --> 00:35:06.880
I'll probably bring that back to the product team to to verify.

s386
00:35:06.880 --> 00:35:10.320
Um, but what brought up that idea?

s387
00:35:21.359 --> 00:35:23.599
the repositories,

s388
00:35:23.680 --> 00:35:24.400
right?

s389
00:35:24.400 --> 00:35:33.760
They're they're divided right now by org level and by um repo level um naturally or natively

s390
00:35:34.014 --> 00:35:36.014
[snorts]

s391
00:35:36.320 --> 00:35:37.599
that those were good questions.

s392
00:35:37.599 --> 00:35:38.240
Yes.

s393
00:35:38.240 --> 00:35:40.240
Yeah.

s394
00:35:53.520 --> 00:35:58.800
I guess you know how

s395
00:36:00.214 --> 00:36:02.214
[snorts]

s396
00:36:14.625 --> 00:36:14.960
[snorts]

s397
00:36:14.960 --> 00:36:17.280
I'm just curious.

s398
00:36:19.839 --> 00:36:20.560
It can be.

s399
00:36:20.560 --> 00:36:21.920
And so great question.

s400
00:36:21.920 --> 00:36:23.520
How do rules become rules?

s401
00:36:23.520 --> 00:36:27.440
Like how does COD determine um what is a rule or what isn't?

s402
00:36:27.440 --> 00:36:29.280
Because it seems very automated.

s403
00:36:29.280 --> 00:36:34.320
So you can it depends on how you've organized things already.

s404
00:36:34.320 --> 00:36:47.680
If you have coding rules that are literally in a rules uh you know directory, it's going to be very easy for COD to be like okay every single one of the rules that I see here I'm just going to copy and paste like most of those contents

s405
00:36:47.680 --> 00:36:50.960
um in and uh ingest it here into the portal.

s406
00:36:50.960 --> 00:37:04.079
But there are some rules that will show up as uh suggestions and this is from specifically from reading or analyzing the actual codebase.

s407
00:37:04.079 --> 00:37:14.720
Um so you might not have explicitly written these rules into your code but these are the conventions that Codto has noticed based on analyzing the entire codebase.

s408
00:37:14.720 --> 00:37:21.520
It will not automatically create a rule for it, but it will come up as a suggestion for you to be able to review.

s409
00:37:21.520 --> 00:37:23.440
And this is what will be generated.

s410
00:37:23.440 --> 00:37:25.440
You can edit all of this.

s411
00:37:25.440 --> 00:37:34.640
There are examples of good code examples and bad code um just to make it very clear to AI agents uh what good and bad is.

s412
00:37:34.640 --> 00:37:38.720
And then you can define the category and the the warning levels.

s413
00:37:38.720 --> 00:37:48.079
And then once you're happy with the way that this looks, you can then accept it uh and activate that rule and it'll automatically be applied.

s414
00:37:48.720 --> 00:37:49.040
Yes.

s415
00:37:49.040 --> 00:37:50.240
Keep going.

s416
00:37:50.240 --> 00:37:52.720
on the other side.

s417
00:37:52.720 --> 00:38:05.200
How does that kind of

s418
00:38:07.280 --> 00:38:10.000
um for Kodto it's or

s419
00:38:10.000 --> 00:38:20.720
well I saw like I'm going

s420
00:38:23.006 --> 00:38:25.006
[snorts]

s421
00:38:27.839 --> 00:38:40.560
Um I think for for this it's just pulling in the local uh repo rules that are most of them if not all of them are associated with the different paths to take in this workshop.

s422
00:38:40.560 --> 00:38:47.896
Um but in general like the codto get rule skill is talking to the back end of the codto platform and

s423
00:38:47.896 --> 00:38:48.079
[snorts]

s424
00:38:48.079 --> 00:38:59.119
it's sort of like this agentic uh rag process of determining um what which ones are actually irrelevant and it filters it down

s425
00:39:00.480 --> 00:39:02.480
yeah

s426
00:39:07.547 --> 00:39:07.839
[snorts]

s427
00:39:07.839 --> 00:39:10.720
that was a really good question Uh, okay.

s428
00:39:10.720 --> 00:39:14.800
So, the plan was implemented and I don't know why it created a draft.

s429
00:39:14.800 --> 00:39:18.480
I like real PRs, but fine, whatever.

s430
00:39:18.480 --> 00:39:22.160
Um, copy the link.

s431
00:39:25.280 --> 00:39:26.000
All right.

s432
00:39:26.000 --> 00:39:28.079
Add refund workflow with quality gates.

s433
00:39:28.079 --> 00:39:29.280
Did you get there?

s434
00:39:29.280 --> 00:39:31.127
Uh, okay.

s435
00:39:31.127 --> 00:39:31.680
[snorts]

s436
00:39:31.680 --> 00:39:32.800
Awesome.

s437
00:39:32.800 --> 00:39:36.640
And here we've got the summary.

s438
00:39:36.640 --> 00:39:47.502
I'm just going to go ahead and make this a real PR because what I want is for COTO to review this without merging the pull request.

s439
00:39:47.502 --> 00:39:48.079
[snorts]

s440
00:39:48.079 --> 00:39:55.520
If COTO is connected to your repo, this is what you should see automatically pop up.

s441
00:39:55.520 --> 00:39:59.292
It's going to start the analysis process.

s442
00:39:59.292 --> 00:39:59.440
[snorts]

s443
00:39:59.440 --> 00:40:03.920
And of course, I'm not looking at all of the code because we don't have all day to do that.

s444
00:40:03.920 --> 00:40:09.280
Um but the you understand the point of uh implementing these processes.

s445
00:40:09.280 --> 00:40:18.480
So in a few minutes um COD will be done analyzing this uh code and we'll bring up any potential issues and we're going to take a look at what those are.

s446
00:40:18.480 --> 00:40:27.440
Um as a result of any potential coding issues that come up, we're going to run an agent skill to automatically resolve it.

s447
00:40:29.422 --> 00:40:30.160
[snorts]

s448
00:40:30.160 --> 00:40:31.680
Are there any other questions?

s449
00:40:31.680 --> 00:40:32.800
Those were really good.

s450
00:40:32.800 --> 00:40:33.359
Yes.

s451
00:40:33.359 --> 00:40:42.320
I have a question and a question that follows from it is something I've seen with skills is they can be like super general like right try code and like this is what

s452
00:40:42.320 --> 00:40:54.560
looks like and I have arguments with engineers where I'm like this is super general it should be in the agent's weights by now and like I think you should wait until something pops up but to settle these arguments it would be really useful to have data like this skill

s453
00:40:54.560 --> 00:41:05.839
in the metric shows that it's actually useful does does Kodto offer anything like that or how do you know these skills are actually valuable versus just like Um, COD is just ingesting the skills.

s454
00:41:05.839 --> 00:41:10.560
It's not evaluating uh the quality of them, but there's a tool that does do that.

s455
00:41:10.560 --> 00:41:13.280
Tessle, have you heard of Tessle before?

s456
00:41:13.280 --> 00:41:18.800
TeSSL is a um company that is all about skills eval.

s457
00:41:18.800 --> 00:41:26.160
And you can determine it will tell you are there any security concerns with this skill or is this actually as effective as you expect it to be.

s458
00:41:26.160 --> 00:41:33.520
That's a tool that I would recommend for like integrating that to uh to determine to determine that.

s459
00:41:33.520 --> 00:41:34.000
Thank you.

s460
00:41:34.000 --> 00:41:36.000
Yeah.

s461
00:41:41.599 --> 00:41:42.400
Okay.

s462
00:41:42.400 --> 00:41:46.720
Um, a PR summary was generated automatically.

s463
00:41:46.720 --> 00:41:56.800
We've got a description and we've got a diagram of the changes and it's showing the legend API inmemory store in a function.

s464
00:41:56.800 --> 00:42:10.949
We've got the API client and the fast API app and the different uh let's see the endpoint of the payment refund endpoint that's pointing to four different uh

s465
00:42:10.949 --> 00:42:11.440
[snorts]

s466
00:42:11.440 --> 00:42:13.440
fields.

s467
00:42:17.499 --> 00:42:19.499
[snorts]

s468
00:42:22.079 --> 00:42:29.520
Um, so one of the things that I would recommend, but I I realized I just skipped over was the verification prompt.

s469
00:42:29.520 --> 00:42:43.920
So we can after doing a, you know, a a process of creating the the making the code change and actually doing a local review, we run a verification prompt to ensure that everything passes.

s470
00:42:43.920 --> 00:42:50.960
But Codeex already did that automatically on its own um, just because of the way that the repo is is actually set up.

s471
00:42:50.960 --> 00:42:55.489
But this is would be a best practice for walking through that process

s472
00:42:55.489 --> 00:42:57.489
[snorts]

s473
00:42:59.280 --> 00:43:03.760
and any day now codto would be great.

s474
00:43:04.004 --> 00:43:05.440
[laughter]

s475
00:43:05.440 --> 00:43:10.240
Well, we can take a look at a PR that I've already created.

s476
00:43:10.240 --> 00:43:18.480
And this was a PR where I didn't run any of the processes that we were just talking about.

s477
00:43:18.480 --> 00:43:32.800
But I do have the context uh pulled in through the Kodto portal and this is something that uh highly consider you looking into when it comes to that context management layer that I was talking about

s478
00:43:32.800 --> 00:43:34.480
here in the code review.

s479
00:43:34.480 --> 00:43:41.280
Codto identified that there were rules in the codto portal that are relevant for this actual code review.

s480
00:43:41.280 --> 00:43:44.160
We're not talking about planning or implementation anymore.

s481
00:43:44.160 --> 00:43:46.319
We're talking about code review specifically.

s482
00:43:46.319 --> 00:43:53.760
And then it pulled in also the relevant skills um for this uh this code change.

s483
00:43:54.800 --> 00:43:55.095
And here

s484
00:43:55.095 --> 00:43:55.440
[snorts]

s485
00:43:55.440 --> 00:43:59.520
there were some action required uh results that came up.

s486
00:43:59.520 --> 00:44:09.119
There's a rule inside of this um inside of my org where Python functions must all have a dock string attached to it.

s487
00:44:09.119 --> 00:44:14.319
And so this was shown as a rule violation and an issue around maintainability.

s488
00:44:14.319 --> 00:44:25.520
The only reason why it's action required is because that was determined by our engineering org that this is of high as high of a severity because you can customize the level of severity of your rules.

s489
00:44:25.520 --> 00:44:28.160
It doesn't have to be something that is blockw worthy.

s490
00:44:28.160 --> 00:44:30.800
It just needs to be something that is addressed.

s491
00:44:30.800 --> 00:44:32.430
That is up to you.

s492
00:44:32.430 --> 00:44:33.040
[snorts]

s493
00:44:33.040 --> 00:44:35.520
And here is the evidence that's associated with it.

s494
00:44:35.520 --> 00:44:38.319
Require dock strings for all Python functions.

s495
00:44:38.319 --> 00:44:39.520
Well, I would say methods.

s496
00:44:39.520 --> 00:44:41.520
Yes.

s497
00:44:48.079 --> 00:44:50.079
Yes.

s498
00:45:03.119 --> 00:45:03.216
Yeah.

s499
00:45:03.216 --> 00:45:03.520
[snorts]

s500
00:45:03.520 --> 00:45:12.160
Um so this is being determined right during the code review phase of okay which rules are going to be relevant.

s501
00:45:12.160 --> 00:45:23.599
The part that can be deterministic is that you can decide based on the level of severity of the rule this uh code this uh branch or PR will be will be blocked.

s502
00:45:23.599 --> 00:45:39.119
And so that's something you can configure in the Kodto portal is in the case of any action required or high severity issue uh is identified we block the merge until you have fixed it and code will run again and verify that you have.

s503
00:45:39.119 --> 00:45:42.000
in terms of making sure that

s504
00:45:46.446 --> 00:45:46.560
[snorts]

s505
00:45:46.560 --> 00:45:52.560
uh there's a potential that it won't determine that their relevant rule will actually Yes.

s506
00:45:52.560 --> 00:45:59.359
Um and I I'm really glad that you brought that up because it is something that we have talked about with the product team.

s507
00:45:59.359 --> 00:46:10.240
Um and the more uh enterprise customers that we speak to the especially in highly regulated industries, they really want determinism in exactly that fashion.

s508
00:46:10.240 --> 00:46:16.480
Um so that's something I'm hoping we can push for for making a real thing because I think it's extremely important.

s509
00:46:16.480 --> 00:46:19.119
I agree with you.

s510
00:46:23.760 --> 00:46:24.880
Sorry.

s511
00:46:24.880 --> 00:46:25.920
Yes.

s512
00:46:25.920 --> 00:46:34.000
after fixes doer verification also because changed.

s513
00:46:34.000 --> 00:46:36.000
Yeah.

s514
00:46:39.599 --> 00:46:41.040
You mean this?

s515
00:46:41.040 --> 00:46:42.000
Yes.

s516
00:46:42.000 --> 00:46:46.240
Um after I've you've resolved issues.

s517
00:46:46.240 --> 00:46:46.640
Yes.

s518
00:46:46.640 --> 00:46:49.760
I think it's best practice too just in case.

s519
00:46:49.760 --> 00:46:52.079
Um you you never you never know.

s520
00:46:52.079 --> 00:46:52.720
So

s521
00:46:52.720 --> 00:46:55.119
is it done automatically?

s522
00:46:55.119 --> 00:46:58.960
um how many

s523
00:47:00.880 --> 00:47:12.960
you're I think that's more on your your agent might be the one that will determine like oh I'm going to run verification or not and you can prompt your agent to do that anyway

s524
00:47:12.960 --> 00:47:21.040
um regardless um but yeah so it depends on how your your agent kind of responds

s525
00:47:22.732 --> 00:47:22.960
[snorts]

s526
00:47:22.960 --> 00:47:26.640
and finally Here are the results.

s527
00:47:26.640 --> 00:47:27.440
Oh, okay.

s528
00:47:27.440 --> 00:47:29.920
Five skills were invoked.

s529
00:47:29.920 --> 00:47:31.280
Uh, all right.

s530
00:47:31.280 --> 00:47:37.119
So, we've got one bug, three rule violations, and two skill insights.

s531
00:47:37.119 --> 00:47:37.142
There

s532
00:47:37.142 --> 00:47:37.359
[snorts]

s533
00:47:37.359 --> 00:47:43.599
is an issue with the refund item potency stored in memory.

s534
00:47:43.599 --> 00:47:52.880
So, refund payment uses the process local uh dictionary to store and replay fund results, which is not durable persistent storage.

s535
00:47:52.880 --> 00:48:02.000
and we have the piece of code that is related to that and we have the evidence as well, the rules and the following files.

s536
00:48:02.000 --> 00:48:12.000
Um, the important part here that you all should definitely play around with is copy and pasting this agent prompt that is meant to fix the suggestions that uh pop up.

s537
00:48:12.000 --> 00:48:22.640
Um, you can do that manually for each of the issues that present themselves or you can run the Kodto PR resolver to automatically resolve.

s538
00:48:22.640 --> 00:48:26.160
Did you already run that that skill?

s539
00:48:36.160 --> 00:48:43.953
I wanted everything to be code or be rechecked because

s540
00:48:43.953 --> 00:48:44.319
[snorts]

s541
00:48:44.319 --> 00:48:47.040
it can happen something else.

s542
00:48:47.040 --> 00:48:48.559
Absolutely.

s543
00:48:48.559 --> 00:48:56.400
It's important if we have any kind of automation to include verification section again.

s544
00:48:56.400 --> 00:48:56.720
Yep.

s545
00:48:56.720 --> 00:49:08.240
I think that's best practice and that's um that's what I would do or I guess pregenai era um just to be careful and that's what you want your agents to do as well.

s546
00:49:08.720 --> 00:49:11.463
So that was a really uh good point.

s547
00:49:11.463 --> 00:49:12.079
[snorts]

s548
00:49:12.079 --> 00:49:23.520
Um after seeing all of the results in this uh PR, then that's when I ran the COD PR resolver skill and I ran it against the pull request number four

s549
00:49:23.520 --> 00:49:32.800
and I saw that it's pulled down all of the issues that COD has already identified from critical to high.

s550
00:49:32.800 --> 00:49:35.760
And where did it go?

s551
00:49:36.960 --> 00:49:37.760
Oh, okay.

s552
00:49:37.760 --> 00:49:50.240
So, it pulled down all of the results and now I can decide, do I want to review each issue individually or do I want to autofix all um for the sake of this uh workshop,

s553
00:49:50.240 --> 00:49:52.400
I'm just going to autofix all.

s554
00:49:52.400 --> 00:50:04.559
But, uh, this is something if you want to be a bit more hands-on and make decisions with your agent to actually plan how you're going to make the fix, you can do that yourself with the review each issue.

s555
00:50:04.559 --> 00:50:07.139
Uh, um, option

s556
00:50:07.139 --> 00:50:08.720
[snorts]

s557
00:50:08.720 --> 00:50:12.160
autofix all.

s558
00:50:16.000 --> 00:50:19.520
And now let's see how it reacts.

s559
00:50:19.520 --> 00:50:22.880
So I'm going to autofix all six codto issues.

s560
00:50:22.880 --> 00:50:27.440
And what should happen is okay.

s561
00:50:27.440 --> 00:50:40.079
And you can see here it says then it's going to run verification and it's going to make the uh commits and it won't be the commit will be blocked if it's not structured in the correct way automatically.

s562
00:50:40.079 --> 00:50:54.000
Um, but this is that point where it'll do that after and it'll take that next step further by actually posting back to the PR and uh with a summary of all the changes that were fixed.

s563
00:50:54.000 --> 00:51:04.240
And from there, that's how the PR will be updated and all of the issues that were resolved will be uh crossed out.

s564
00:51:12.160 --> 00:51:17.839
I I mean I've done I've used work trees for for that.

s565
00:51:17.839 --> 00:51:20.079
Is that something that you've used?

s566
00:51:20.079 --> 00:51:22.079
Yeah.

s567
00:51:27.440 --> 00:51:31.280
Um yeah, that's not something that Kodto really handles.

s568
00:51:31.280 --> 00:51:36.480
Like that's something I I found the coding agents kind of handle themselves.

s569
00:51:36.480 --> 00:51:40.000
That's a good point though, a good question.

s570
00:51:40.800 --> 00:51:46.720
Any other uh questions before I wrap this up?

s571
00:51:47.280 --> 00:51:49.280
Yes.

s572
00:51:51.940 --> 00:51:53.940
[snorts]

s573
00:51:56.319 --> 00:51:58.400
Yes, you can run a code review.

s574
00:51:58.400 --> 00:52:01.359
And I I didn't run that for this because time was flying.

s575
00:52:01.359 --> 00:52:07.280
But um you mean for COD or for your coding your coding agent?

s576
00:52:07.280 --> 00:52:15.040
So we are coming out with a COD review local workflow and it will talk directly to the Kodto backend.

s577
00:52:15.040 --> 00:52:23.280
So all of the agentic capabilities that you would have in a pull request you would also have locally and be able to address.

s578
00:52:28.160 --> 00:52:30.559
What was that?

s579
00:52:31.241 --> 00:52:33.241
[snorts]

s580
00:52:40.480 --> 00:52:42.720
How does it review all of the code?

s581
00:52:42.720 --> 00:52:45.680
It clones the repo.

s582
00:52:45.680 --> 00:52:49.581
Yeah, that's the process that we do on the back end.

s583
00:52:49.581 --> 00:52:50.160
[snorts]

s584
00:52:50.160 --> 00:52:54.720
It's like a temporary cloning just so it has an understanding of Yeah.

s585
00:52:54.720 --> 00:52:58.751
the the broader scope of the changes beyond the code diff.

s586
00:52:58.751 --> 00:52:58.880
[snorts]

s587
00:52:58.880 --> 00:53:01.440
Really good question.

s588
00:53:06.400 --> 00:53:09.839
Okay, I think we are about just about out of time.

s589
00:53:09.839 --> 00:53:09.886
So,

s590
00:53:09.886 --> 00:53:10.079
[snorts]

s591
00:53:10.079 --> 00:53:24.559
it wasn't able to go through um everything in detail, but like I said, uh if nothing else that you take from this, take this repo and ask your AI some tips on what kind of verification

s592
00:53:24.559 --> 00:53:29.200
steps that you would want to implement or you would want to practice implementing locally.

s593
00:53:29.200 --> 00:53:39.200
Um, and I think that that would be a really good major step to just put add automatically add more structure into your workflows and consider any tools.

s594
00:53:39.200 --> 00:53:45.839
There are some uh open-source tools that have similar context management layers to what Kodto does.

s595
00:53:45.839 --> 00:53:58.720
I think um uh there's there's one of them out there, but if you look it up, you should try practicing having rules and skills and other um engineering standards, whatever it is, stored elsewhere

s596
00:53:58.720 --> 00:54:07.839
that your um across so that you can pull down that information um when relevant to your the stages of your workflow.

s597
00:54:08.640 --> 00:54:12.400
Oh yes, it's called packmind, I think.

s598
00:54:12.400 --> 00:54:14.077
Packmind.com.

s599
00:54:14.077 --> 00:54:15.119
[snorts]

s600
00:54:15.119 --> 00:54:29.142
So definitely look into this if you want an open- source solution for um uh governing your AI code um through different rules and things like that consistently across um

s601
00:54:29.142 --> 00:54:29.200
[snorts]

s602
00:54:29.200 --> 00:54:33.280
your teams and your repos.

s603
00:54:39.622 --> 00:54:41.359
[snorts]

s604
00:54:41.359 --> 00:54:41.760
Okay.

s605
00:54:41.760 --> 00:54:44.000
I wish the internet was faster, but it's not.

s606
00:54:44.000 --> 00:54:51.440
So, I I I don't I can't uh this never happens, but here we are.

s607
00:54:51.839 --> 00:54:54.880
Uh let me know if y'all have any questions.

s608
00:54:54.880 --> 00:54:59.599
Um I'm I'm here, I guess, for the next couple minutes.

s609
00:54:59.680 --> 00:55:01.237
Thank you.

s610
00:55:01.237 --> 00:55:03.237
[applause]

s611
00:55:07.680 --> 00:55:08.880
Let's

s612
00:55:08.880 --> 00:55:10.880
Yes.

s613
00:55:15.920 --> 00:55:18.240
I did.

s614
00:55:22.559 --> 00:55:32.880
I'm at a point now where I don't clear like I let the the the context like compact and I haven't really had issues like I would have maybe three four months ago

s615
00:55:32.880 --> 00:55:36.640
with the I guess the advancements that they have made.

s616
00:55:36.640 --> 00:55:39.280
I just don't experience much degradation.

s617
00:55:39.280 --> 00:55:39.680
Okay.

s618
00:55:39.680 --> 00:55:41.599
like comp.

s619
00:55:41.599 --> 00:55:42.800
Yeah, I do.

s620
00:55:42.800 --> 00:55:52.640
Um, but if I really feel like it, I'll just create a new a new session and maybe copy and paste some of the stuff from the previous session into it.

s621
00:55:54.948 --> 00:55:55.200
[snorts]

s622
00:55:55.200 --> 00:55:58.000
Yes, we have a a massive booth here.

s623
00:55:58.000 --> 00:55:58.799
So,

s624
00:55:58.799 --> 00:56:01.520
yeah, check us out.

s625
00:56:07.680 --> 00:56:10.079
Um, Yes.

s626
00:56:10.079 --> 00:56:10.640
Yes.

s627
00:56:10.640 --> 00:56:15.920
Do you have any local options for security reasons?

s628
00:56:16.960 --> 00:56:27.920
Yes, we have like onrem deployments and pretty much all deployment options that you would want um because we do work with like banks and you know things like that and

s629
00:56:27.920 --> 00:56:33.440
um companies like some teams at Nvidia there's like no other option but to do that.

s630
00:56:33.440 --> 00:56:36.720
So yeah we do that integration.

s631
00:56:39.599 --> 00:56:41.920
Thank you.

s632
00:56:42.764 --> 00:56:44.764
[snorts]

s633
00:56:47.040 --> 00:56:52.000
Let's see where did Oh, there it is.

s634
00:56:52.240 --> 00:56:54.554
And here is the fix summary.

s635
00:56:54.554 --> 00:56:56.554
[snorts]

s636
00:57:00.799 --> 00:57:01.599
Yeah, it did.

s637
00:57:01.599 --> 00:57:07.599
I'm like not even I don't even know what time it actually uh it is right now.

s638
00:57:07.599 --> 00:57:10.079
And how many things were resolved?

s639
00:57:10.079 --> 00:57:11.760
All of them.

s640
00:57:11.760 --> 00:57:15.200
All issues were resolved.

s641
00:57:15.599 --> 00:57:17.119
Awesome.

s642
00:57:17.119 --> 00:57:18.319
Okay.

s643
00:57:18.319 --> 00:57:19.280
Thank you all.

s644
00:57:19.280 --> 00:57:22.064
Uh if you have any questions, I'll be out there.

s645
00:57:22.064 --> 00:57:24.064
[snorts]

s646
00:57:43.024 --> 00:57:45.024
[music]