Agents, codebases, and teams — Aditya Khandelwal, Amazon AGI Lab

AI Engineer · 16 min · 243 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Today, I'm going to be talking about agents code bases and teams.
00:15Essentially, how do you get your team to actually ship together with agents?
00:20And like, I think for the longest time, like the one thing that's bugged me is like there's so much content about, you know, how do you set up your own code base to like work well with agents, you know, like what skills do you add?
00:31You know, this skill's better, that setup's better.
00:33But it all seems to break the moment you like actually try to use it with your team in your actual production setup.
00:39And like for individual repos, it kind of makes sense.
00:42But the moment you actually try to use it with your own like team setup, it tends to break.
00:47And like I think over the past few months, I've figured out how to make it work with a team of folks.
00:53So like I was leading a team of like 10 people over the last few months, and I think we found like a good solution, and I want to share that with you guys.
01:00But before we get into that, I just want to recap like, you know, what's been the journey that we've been on.
01:04Right?
01:05So we started, you know, code coding agents took off, and a few people got really, really good leverage, right?
01:11And I think all of us were kind of asking like, is this AGI?
01:15You know, like did we achieve it?
01:17Then, you know, companies took that and said, "Well, if one person can do so well, let's just get everyone and let's mandate it."
01:24And like token max, right?
01:27And that was clearly a a galaxy brain moment, right?
01:30And then the inevitable happened, you know, AI slop shipped, and there's a bunch of sev twos.
01:36I'm not going to name which companies, but you know, essentially you saw people kind of retracting.
01:40They said, "I don't think I don't think this is the best option here."
01:45And eventually model prices you know, climbed.
01:48We saw, you know, people figured out that tokens have to be paid for.
01:52Like you just can't token max your way through life.
01:54And budgets got bolted on.
01:56And you know, essentially money is being lit on fire, and the money has to come from somewhere.
02:01So, given this journey, I want to actually like this is the enterprise journey, right?
02:07And what does that do for a single developer?
02:09And I think this is an important framing cuz it really talks about people as a part of a team, right?
02:15And I want to look at it from like two axes.
02:17So, there is the fear axis where people lie on the spectrum, right?
02:22They're coming It's like Is it coming from my job?
02:24Like am I going to be out of a job?
02:25Or is it like a really handy tool?
02:27And they're really, you know, they're not that fearful.
02:29Versus like the confidence they have in how much they're executing it.
02:32So, they can either use it a lot or they can use it, you know, not that much because they don't really know how to use it that well.
02:39Now, when we started, like people said, "Oh, you know, what is this?
02:43Is this the end?
02:44Like am I needed?"
02:45And fear was pretty high.
02:46Utilization was pretty low cuz people didn't really know how to use it.
02:50And then when a few people got outsized leverage, you saw, you know, early adopters basically people saw them and people said, "Okay, well, it looks like I'm still kind of needed if I figure out how to use this thing.
03:01So, let me actually try using it, right?"
03:04And then we saw, you know, mandates and token maxing, and people kind of got a little skeptical.
03:10Like confidence stayed the same, but people tried to use it a lot more, right?
03:14And then we realized, uh you know, there's a bunch of slop shipping, there's self-tours, and I, you know, it's like I'm not really that scared cuz, you know, it just slip it just ships slop.
03:23Like I'm going to still be needed.
03:25And they don't even know how to use it that well cuz like now the confidence is greater, right?
03:29And so you got to figure out how to get people from all the way wherever they are on the spectrum to where they're not fearful and they're actually using it a whole lot more.
03:39And this is the framing that I want to like I want everyone to keep in mind as they're actually trying to get a team to adopt good like AI usage and good AI patterns, right?
03:48And so, the The is, you know, what does it take?
03:51Step one, create a Cloud MD.
03:54Step two, add some skills.
03:57Is that it?
03:58Do we like solve it?
04:00You know, I think we all know like you guys are here because like clearly life's not that simple and like stuff's messy, right?
04:08And I think a few people might ask, you know, why doesn't this work?
04:11Like isn't that what like everyone does?
04:13And I want to like just talk about a few things you might see that actually indicate that yeah, like this isn't working.
04:22So, the first thing is if you're babysitting your agents, it's not the right setup, right?
04:27And you got to realize that.
04:29If you're seeing people in your team babysitting their agents, something's wrong.
04:34Is, you know, one of the things that I heard a lot was is, you know, insert whatever latest model there is being really dumb today.
04:41The model didn't change, right?
04:42The hardness may have changed underneath.
04:44But if it's really like that's that's acceptable to like small changes in the hardness, clearly your own code base isn't set up well.
04:54It's silently burning context and money.
04:56Like you don't realize it, you know, you go you blow through like 500k context, you might go to like 750k million and hit auto compact even though you're not doing like a really complicated task.
05:06Clearly something's wrong.
05:08You know, if you have long ass sessions needing constant intervention, there's still something that's wrong.
05:15If you're getting a constant slot factory, obviously know like things are things are not good.
05:20And if you're asking if you find yourself asking, you know, how are these other companies shipping so fast?
05:25Like how are model companies releasing models at like a month and a half two-month cadence?
05:30Clearly they they have something which we don't, right?
05:34And so, I guess everyone's kind of thinking so, you know, how do we solve this correctly?
05:40And so, I think the first thing to realize is we need to frame it correctly, right?
05:45It isn't really an IC's job.
05:47It's a job for leadership.
05:48It's a job for the company, right?
05:51Making engineers work well with their agents is truly the most impactful thing you could do as an organization.
05:58Because that's going to enable your engineers to ship faster and with confidence and avoid, you know, a lot of incidents.
06:06If we live in this figure it out for yourself paradigm, like people are going to get outsized productivity, some people aren't, and like the people who are generating like 10 PRs a day
06:15are going to like look like, you know, gods compared to people who are shipping like one to two.
06:20And the one to two PR people are actually going to get left with the review burden.
06:24And that's actually a really, really bad thing.
06:26Cuz now not only can they not ship, they're going to actually see bad code and then curse the agents and hence not be able to get on to the let's ship 10 PRs, right?
06:36And so it's really important to do this.
06:39If it's a problem facing the team, there's a few things you can do, right?
06:43The most impactful things that you can do to set up your code base to like make it work well require team buy-in.
06:49You can't just like if you want to change the way your code base is organized, you can't do that as an IC, right?
06:55And if you're treated as a leadership problem, then you can do things like this.
07:01So, the other thing this needs is harness engineering, right?
07:05Per code base.
07:06And I think there's a lot of content on this, so I just want to talk about a few principles, but I don't want to like make this talk about that because there's a lot of smart people.
07:13You're at AI engineer, right?
07:15Like this this conference is all about people telling you how to you best set up, you know, your code base to make things function well.
07:21So, I don't want to talk too much about this, but there's a few key principles here.
07:25Smart prompt injection is one of them.
07:27You want to treat your entire code base as one way to that so that you're able to like smartly prompt inject the model with just the right context at just the right time.
07:36Without you needing to do it.
07:38And that's the framing.
07:39You want to be able to say, "Okay, I've set it off on this task.
07:43It has like a map of how to find the things it needs at the time it needs it."
07:48If it's looking at some code and that code has, let's say some documentation, it the documentation needs to live in the comments.
07:54So, if it ever grabbed into that code, it reads the comment, goes to that file, finds all the information about it.
08:01That's just one example.
08:02The second is close the loop, right?
08:04You got to make a self-healing system cuz slop is inevitable.
08:08There is going to be some slop that's going to seep in.
08:11But, you need to have a pipeline and a way to close the loop to remove the slop, to detect it, and to be able to like self-heal the system.
08:19And then you need to iterate continuously.
08:21And I can't emphasize this enough.
08:22Like, you can't assume that you do this for a month and you're done.
08:26Like, things are going to change constantly underneath.
08:28So, you need to keep this as one of the things that you have to do as an organization.
08:32And the third most important thing is treat it like a human problem, guys.
08:36Like, this isn't It's not, you know, oh, it's this tool.
08:39Like, people will figure it out.
08:40Let's just mandate our way through life.
08:41Like, that's just not going to work.
08:44So, treat it like a human problem.
08:45Fear is real.
08:47Human emotions are real.
08:48We should recognize it.
08:50So, you know, enough gyan or as like it's more like the Hindi way to say like, you know, I'm enough prof like I'm kind of giving you sermons.
08:58But, like, how do you really do this, right?
09:00Like, these are all principles.
09:02What's the real playbook?
09:03So, here's what we did and here's like I'm not going to over like emphasize on this is the exact way to do it, but this is like roughly how we did it and you can take from it what you choose.
09:14The first thing is do the basics, right?
09:16You got to do them right.
09:17Progressive disclosure, I can't emphasize this enough, is really, really powerful, right?
09:22Find your best ICs and find what how they are making the code base work for them.
09:27Take those practices and pass them or guide.
09:31People can't live in their own practices.
09:32And this is really hard for engineers to do it.
09:34It's basically accepting that my setup is imperfect.
09:37And like engineers don't like to hear that.
09:39But you got to figure out a way to find those best practices and ship them across.
09:44Make sure that that's kind of a shared setup.
09:47The second thing we did was there's one high value skill that we invested in.
09:51In our case it was this thing called ship it.
09:53What it did was the moment you're done with your code, it takes care of everything from code done to PR ready for review.
10:00Which means you got to open a PR, figure out your opinions, handle the handle all the comments, you know, handle all the PR descriptions, the merge comments, everything, right?
10:10It handles CI failures.
10:11It basically runs through these loops.
10:14And what this meant was often the skill was running for over an hour.
10:17And that scared people, but once you actually figure out once they saw the value, they get invested, right?
10:23Because it's one skill which tells them, "Okay, this AI thing can actually work for me.
10:28I don't need to constantly baby sit it.
10:30I can trust it."
10:32The third thing and really important is to close the loop, right?
10:34So we wired issues and boards into the repo.
10:36Like we added CICD.
10:38We added gigantic reviews.
10:39We have like a code gardener that actually goes back and looks through a whole bunch of like every night it'll run and look at the code and check if something like not organized correctly.
10:48What does correct organization mean will depend on your code base.
10:53Get people invested and I can't emphasize this enough.
10:56You have to win over the skeptics.
10:58It's really easy to say like the skeptic is just someone who's scared.
11:01It's really hard to get them to buy in, but if you can get them to buy in, you know you're doing something right.
11:07You have to get them to be able to edit and play with the shared setup cuz that's the true way you know that they're actually invested, right?
11:15And this is where you got to it ensure you're iterating constantly.
11:19If people are like and this is the hardest thing for engineers again cuz you're basically saying I'm never going to like get to perfection in my setup.
11:27But you got to be okay with that.
11:29You have to do it and you You to treat it like, you know, X% of your IC time is probably going to be spent on it trading on this thing, which is not going to lead to like meaningful PRs
11:37like up front, but it's useful and it's worth it.
11:42And, you know, I don't want to say this is perfect, right?
11:44Like we faced a ton of issues while doing this.
11:47And I'm just going to walk you through some of them.
11:49But it's an iteration loop, so you got to treat it like a piece of feedback.
11:52So, what are the problems we hit, right?
11:54There are too many issues.
11:55Like when we started like we blew up to like 4 or 500 issues uh without like, you know, just I think within like a couple of weeks, which is a crazy number for like a repo.
12:05And then, you know, there are so many like different agents all trying to create issues cuz they've not been wired correctly.
12:11There's a lack of agreement.
12:12People, you know, as soon as people saw like, "Oh, this isn't working perfectly or the way I expected it."
12:17There is super easy for them to say, "You know what?
12:19I'm just going to go back to babysitting my agent."
12:21You don't want that.
12:22You want to actually take their feedback and put it back into the skill and improve the skill.
12:28Agents are taking too long.
12:29This is actually one of those expectation setting things.
12:32It's good if agents take too long.
12:33That means you can actually go off and do other things and you have confidence that they're doing the right thing.
12:38At the end of the day like the moment we hit this reasoning paradigm, the longer the agent like thought, the better its output.
12:44You can treat like it's It's almost like a similar mindset for your entire code base and for your skills.
12:51There's going to be merge hell and like we just have to deal with it.
12:54We have to figure out a way to deal with this.
12:57There is going to be slop when you're going to write experiments.
13:00Treat it like it's own thing, right?
13:02What we said was okay, people are generating this code, but it's not relevant.
13:06It's not going to be shipped.
13:07It's just an It's a prototype.
13:09So, treat it like one.
13:10Get it to opt out of all the rigorous other standards you've got across your code base.
13:15And realize people vary on the spectrum, right?
13:18And depending on the day, depending on like what they're going through, they're going to vary on the spectrum.
13:22You have to be able to talk to them and figure out, "Hey, okay, why why are you facing this?
13:27Like is it if the model changed, the hardness changed, again you need to go revisit something.
13:31Figure that out.
13:33And I think the biggest like the easiest way to say this is instead of saying like the model is so dumb, like we have to ask how can I make it smarter or how can I edit like and not This is where I have like crossed out the my.
13:46It's not a personal setup.
13:47It's the shared setup that you have to invest in.
13:51And it's a mindset, right?
13:53You have to go full send.
13:55And like I want to end you with I want to end with this.
13:57Like, I learned skiing like a couple years back and the first like the hardest thing for me was you actually have to commit to it.
14:05If you're pizza braking, you're going to crash.
14:07Like, no matter what.
14:08You have to commit to the speed in order to actually get and feel like, okay, that's how I can turn and that's how I can truly ski.
14:15And so, I'm going to leave you with this.
14:17Like, just be okay with be okay with failing.
14:21You have to, you know, go full send and be okay with falling.
14:24It's fine.
14:25The point is to be able to recover from that and that will allow you to truly feel the AGI.
14:32Yeah, well, that's that's me and I'm happy to take any questions.
14:40Yep.
14:45Uh so, the I'm going to repeat the question for the recording.
14:47Uh strategies that you found best for progressive disclosure.
14:51So, I think a couple things, right?
14:52Uh the first thing is even in your skill.md files, don't overload it.
14:58Like, we've kind of set a hard limit for like 100 lines in your skill.md cuz your skill is really a folder.
15:03So, that's step one.
15:05Make sure and I think I spoke about this during the talk, but when you have like some code that requires like a runbook, make sure the runbook is reflected in the comments
15:16so that if a code if somehow the code like the agent figures its way into like grapping into the code base and find that file, it knows I need to go look at this for like all the description of how this is relevant.
15:27Right?
15:28You have to organize and I think this is why I talk about harness engineering because your entire code base can be set up to encourage progressive disclosure.
15:36Don't overload your cloud MD or your agent's MD file into like one big thing.
15:40You want to make sure that like it's a thin index that can point through the right files and that's what the agent gets in its like first prompt cuz that's what gets loaded when it starts to work.
15:50So these are like some really powerful strategies that and the way you know this is working is when you give it a prompt.
15:56When you give it the first prompt, see what it's doing.
15:59Is it grabbing or does it know where to go?
16:02But how much context is it burning when it like immediately?
16:04So is it like I think like 20, 25K tokens get taken anyway, but like how much more is getting added?
16:10If you're coming to like 40K, 50K, like something's wrong.
16:13That's not really progressive disclosure.
16:15So you have to figure out these boundaries and then like based on this it's an iteration cycle.
16:31All right.
16:31Well, if there there aren't any other questions, feel free to find me.
16:34Happy to talk about harness engineering in general or like anything else, but yeah, thank you for listening.
16:54[music]