Building an Autonomous Engineering Org - Angie Jones, Agentic AI Foundation

AI Engineer · 17 min · 175 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00So, I've spent the last couple of years transforming Block's 3,500-person engineering org into an autonomous one.
00:08And this is a challenge that most tech companies are actively trying to solve or will be in the very near future.
00:14So, today I'll share our path to getting there.
00:18Our agentic coding journey started very early.
00:20We were building Goose, our internal coding agent, even before the LLM supported tool calling.
00:27We worked with Anthropic as design partners for the initial release of MCP, and Goose became the reference implementation for the MCP client.
00:35So, internally, our most curious engineers were some of the very first in the industry to use these types of coding agents.
00:44After a couple of months, about 90% of our engineers were now using tools like Goose and Claude Code regularly to generate code.
00:53So, on paper, it looked like we were all in on AI.
00:57But our CEO was convinced that engineering wasn't using AI at all.
01:00Like, they couldn't be, right?
01:02As far as he was concerned, because we weren't shipping any faster.
01:07Well, I had the numbers, both the metrics and the token bills.
01:10So, I knew that engineering was, in fact, using AI, but he was right.
01:15Features certainly weren't making it to our customers any faster.
01:18So, I began to dig into this a bit.
01:22I think of AI enablements in three phases: experimentation, adoption, and impact.
01:28I'd say that we surpassed the experimentation phase as 90% of our engineering org was using AI, but they still were kind of using it inside of their IDE, you know, asking questions or writing boilerplate code.
01:43And I knew that if we really wanted to see impactful outcomes, then we needed a way to integrate AI into how we built and ship.
01:51I spent the first half of 2025 leading AI enablement for our entire company.
01:58So, 12,000 employees across every function, marketing, design, finance, legal.
02:04Now, our CTO tasked me with building an agentic engineering org.
02:09Okay, sure.
02:11But, what does that actually mean?
02:13There's no playbook for any of this stuff, right?
02:15And I know because I went looking at your blogs hoping that you all had it all figured out.
02:22But, I only saw a bunch of posts saying how you all were making it up as you went along.
02:26So, I did the same, right?
02:29In the simplest of terms, I defined an agentic engineering org as one where engineers leverage AI agents as their primary means of producing engineering outcomes.
02:40So, that meant that our engineers needed to treat agents as core members of their engineering workflow.
02:47Not just using AI to help them write code here and there, but actually working with agents.
02:53Decomposing problems, delegating work, reviewing and verifying what was done.
02:59And we wanted them directing the work as their default way of operating.
03:03But, of course, the vast majority of our engineers were not here.
03:07So, to see where we needed to go, I came up with the maturity model.
03:12This model measures the engineer's relationship with AI agents.
03:16So, how they think and delegate and orchestrate.
03:20And while I had some form of this model in Q3 of last year, Steve Yegge's Gastown article helped me reorganize it into a better model.
03:30So, stage zero is where the engineer doesn't use AI tools in their workflow at all.
03:35Stage one, they use AI to auto complete, but they've never used it in agent mode, maybe.
03:43Uh stage two is where engineers are chatting with agents, but not using it to produce any PRs.
03:49Stage three, engineers are delegating tasks to agents and reviewing the output.
03:54Those at stage four are running multiple agents in parallel.
03:57And then stage five is that final boss where engineers are delegating complete tasks to agents and the agent is able to produce shippable results without the human necessarily needing to guide it.
04:09So, I'd say by the end of the first half, the bulk of our engineers were between stages one and two.
04:15And I needed to get them to stage five.
04:18Now, I wasn't quite sure how to get 3,500 engineers to that level, especially when one, this is all highly experimental, right?
04:27There's no playbook, again, for any of this.
04:30Um two, things were moving so fast that what might be a best practice this week could be outdated when a new tool or a new model drops next week.
04:40And honestly, this was leading to AI fatigue.
04:43And then three, people were already feeling turned off by the top-down pressure from leadership to essentially AI or die.
04:52So, I thought about the 1990 rule where in digital communities about 1% create, 9% interact, and 90% passively consume.
05:02And this maps almost perfectly to how engineers adopt AI.
05:06So, you'll have a small percentage that'll deep dive and start creating agentic patterns and discovering useful techniques for working with agents.
05:15You'll have some that might tweak the agent's MD file here or there.
05:20But most people aren't going to spend the extra cycles to figure this stuff out.
05:25So, I realized that if my AI strategy depends on every individual leveling themselves up, I'm never going to see that broad impact.
05:35So, I leaned into this model.
05:37And I used it to my advantage.
05:39Instead of focusing on all 3,500 engineers, I decided to focus on forming the 1%, the power users, from our most critical teams.
05:50So, I started an AI champions program where I handpicked about 50 engineers across the company who could pioneer what a genetic engineering looks like for their team.
06:00Now, this was not a call for volunteers, all right?
06:03I needed to be strategic about the selection.
06:07I needed engineers who were willing to dedicate at least 30% of their time to investing in AI enablement.
06:14I needed those who wouldn't be discouraged by the non-deterministic nature of AI and give up on it all when it didn't work out of the box, which it often did not, right?
06:24So, I spent the week talking to tech leads and managers to find the 50 engineers who could represent our most critical repositories.
06:34Now, remember at this point, most of our engineers were between stages one and two where they're chatting with AI but not really using it to produce PRs.
06:43So, I wanted to get the AI champions to stage three.
06:47This was about June 2025.
06:49So, the models were decent enough to write a feature for you, right?
06:53But chances were pretty high that the code wouldn't conform to your team's conventions and standards.
06:59So, developers didn't trust the agents enough to delegate work to it yet.
07:03So, the first area that we focused on was making our repos AI ready.
07:09My theory here was if I can get engineers to embed AI directly into their repos, then not only would the agents perform better but the entire team would benefit, not just the 1%.
07:22And the repos are a central point of reference for every engineer that's contributing code.
07:29So, to make their repos AI friendly, they added assets like contacts and rules files so that agents could reliably understand and navigate the code base as well as contribute to it.
07:41Now, when building the AI champions, I made sure that we have folks from every corner of Block engineering: Square, Cash App, Afterpay, Tidal, across front end, back end, mobile, data, infra, right?
07:55Um we covered repos of all shapes and sizes, the nasty big legacy massive mono repos, the smaller services, mobile apps, right?
08:05And that mix let us pressure test patterns across very different engineering realities so that we could quickly see what actually scales.
08:15As expected, mono repos came with their own challenges.
08:18But my JVM devs were already strong in inheritance patterns, so we set shared contexts and rules at the root, then layered more specific ones at the service levels.
08:31We also learned fast that what worked for web didn't necessarily work for mobile.
08:36So Android and iOS even needed different approaches at times.
08:41So instead of forcing a one-size-fits-all solution, each champion figured out what worked for their repo, and then teams with similar shapes and sizes naturally converged on the same tools and patterns.
08:55And honestly, engineers loved this, right?
08:58We let them choose what made sense for their repo instead of pushing a top-down mandate on them.
09:05So we ended up with a standard set of components that would make a repo AI friendly, but again, customized for each team's needs.
09:13Some context files like agents.md or claude.md to provide the agent with repo guidance, rules files to provide the agent with some guardrails, repeatable workflows like slash commands and later agent skills,
09:28um an enabled AI code reviewer, preferably with some instructions on what matters and what it should review.
09:35And then AI attribution on PRs.
09:39So at this point, we've done a lot of great work and technically, we do have the agents writing PRs, but I had a couple of concerns, right?
09:48Not many outside of the AI champions had gotten to this level where they were delegating work to agents.
09:54And the idea is to have the work of the 1% lift everyone up.
09:58So, we weren't there yet.
10:01And then even with the champions, they were delegating, but they were still kind of babysitting the process.
10:07So, I wanted to explore if we could delegate work to the agents in a more hands-off approach and make it easier for others outside of the champions program to do so as well.
10:19So, there's three common places where engineers receive requirements for work.
10:23Issue trackers like Jira or Linear, GitHub issues, and informally in Slack.
10:29I wanted to be able to delegate work to the agents from any of these places.
10:33And the champions implemented all three.
10:36So, true story.
10:38We were in Slack one day, an engineer saw a bug with the product and wrote, you know, like, "Hey, I saw this bug.
10:44Anybody else seen it?"
10:45So, engineer two responds to that and goes, "Mhm, no, I hadn't seen that one."
10:51Engineer three @goose right there in Slack and says, "Hey goose, have you seen this bug before?
10:58Like, can you can you go and check if this is a bug?"
11:01So, goose goes to the repo, it like pulls the files down, and says, "Yeah, this is a bug.
11:06Here it is right here.
11:07But, here's also three options of how you might implement a solution, right?"
11:14Right there in Slack, code snippets.
11:15And so, engineer one goes, "I like option one."
11:18Engineer two goes, "Yeah, yeah, me too."
11:20So, engineer three says, "Hey goose, go implement option one."
11:24Goose does so, returns with a link to the PR, right?
11:28So, the entire cycle from discussion, diagnosis, issue creation, alignment, and the fix took like 5 minutes all right there in Slack.
11:37Very cool party trick, by the way.
11:39Um engineers were also able to assign linear and Jira tickets, as well as GitHub issues to an agent, and have the agent implement the work end to end.
11:49So, now the agents have become a part of the sprint.
11:53This blew everyone's minds.
11:54The first time we did this, the team ran out of work and had to pull in more tickets like twice, right?
12:01So, of course, engineering managers, product teams, they love this flow.
12:06And the beauty about these flows is that all engineers didn't necessarily have to learn a new skill.
12:12The champions had already laid the foundation so that agents were able to work well within these repos, and they worked well all of this worked well because it made delegation feel native to how people
12:24already work.
12:27So, now I felt comfortable claiming that we were truly delegating work to the agents.
12:32At this point, it had been 3 months since launching the champions program, and we were seeing great results.
12:39AI-authored code was up by 69% reported time savings increased 37%, and automated PRs increased 21 times.
12:49We were ready to pursue stage four, multi-agent parallelism.
12:54And with our setup, honestly, moving to stage four was almost free.
12:58We were pretty good at delegating work to the agents now.
13:02But, this stage introduced several new challenges that we needed to solve.
13:07Engineers are now tripling, quadrupling the number of PRs that they're producing, but the PRs are stuck waiting for code reviews, right?
13:15So, people struggled to stay on top of code reviews even before we added AI to the mix, so you know that it's bad now.
13:23And I'll be honest, this isn't a totally solved problem, but I'll share how we did stop the bleeding a bit.
13:30We absolutely had to get the bots to help us review PRs.
13:34So, earlier in the process, this was optional.
13:37Mostly because the AI code reviewers sucked so badly, and we were just pissing the engineers off by having them use them.
13:44But now, with the repo readiness work that the champions had done, coupled with better models and tools, shout out to Codex, uh we were getting much better results here.
13:55So, we enabled Codex on all the repos, and we also created an auto-fix loop where if Codex identified issues, another agent will automatically fix those issues and commit them to the PR.
14:07So, now, at least people would no longer complain about not wanting to review sloppy PRs from bots, right?
14:13By time they got to them, they were in pretty good shape.
14:18Another issue, specifically when running multiple agents in parallel, was that the agents were bumping into each other as they were trying to work.
14:25And more importantly, our machines couldn't handle the load anymore.
14:30Like, engineers' laptops were running out of memory.
14:33CPU was choking, you know?
14:34So, we invested in dedicated cloud workspaces where each agent ran in its own isolated environment.
14:41And this allowed us to easily run them in parallel and from anywhere.
14:47So, at this point, the engineers are cooking.
14:50They got four or five agents running at any given time, and that number is growing.
14:55So, a small group of engineers, including many of the AI champions, started building our own orchestrator to coordinate all of these agents, Builder Bot.
15:07And we needed Builder Bot to get us to an autonomous engineering org.
15:13We realized that if we're trying to build anything close to an autonomous engineering org, the agents need to understand where everything lives and what depends on what.
15:23So, we built a company world modeled based on the entirety of our 25,000 repo code base, right?
15:33And this was a machine-readable view of every single service and how they all connect.
15:40And this allowed orchestrators and the agents that they delegated to to pull context in as needed and understand the landscape as they implement.
15:50And this enabled multiple agents to explore different parts of the system in parallel, each building their own understanding and then come back together and have the orchestrator put all of this together,
16:01right?
16:01Put in a plan that spans across multiple code bases, which was especially useful for us as we were building offerings that spanned multiple products.
16:12Now, with this, we had successfully reached stage five, where engineers are delegating complete tasks to agents.
16:19The agents are able to produce shippable results without human hand-holding.
16:23In fact, this wasn't even limited to engineers anymore.
16:27Anyone at the company could act Build-A-Bot in Slack and have it fix a bug or implement a new feature.
16:36They didn't even need GitHub.
16:38This felt like a dream.
16:43Until it became a nightmare.
16:47You know, of course, all layoffs are tough, but this one felt different.
16:52I had so many questions.
16:54Was this my fault?
16:56Did enabling employees to do the most incredible work of their careers ultimately result in their dismissal?
17:06Just the day before I had felt so proud.
17:09I was in awe of the way we were working, feeling quite accomplished that I successfully built an autonomous engineering But to what end?
17:21I guess I'll leave you all with a few questions.
17:25What are we doing?
17:27Where are we heading?
17:30You know, we sure that it's where we want to end up?