WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=exiwa9QbQXI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/exiwa9QbQXI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.639 --> 00:00:18.720 Hey folks, I'm Davis Palmy, a member of technical staff here at factory. s3 00:00:18.720 --> 00:00:34.239 Uh before this I co-founded Lummetric which was an agent platform for investment teams that we took through Y Combinator and was acquired by factory and prior to that I was a engineering lead for Slalom's innovation lab. s4 00:00:34.239 --> 00:00:41.680 So I've seen AI deployments both across large enterprises and more AI native orgs. s5 00:00:41.680 --> 00:00:55.199 With that in mind, today I'm going to be talking to you about the software factory and how you can move towards an autonomous system that translates input signals into production code. s6 00:00:55.199 --> 00:01:02.320 So, first we've moved through three distinct eras of AI engineering. s7 00:01:02.320 --> 00:01:11.760 First, we had tab autocomplete, next token prediction, and the engineer was still firmly in the driver's seat. s8 00:01:11.760 --> 00:01:19.520 Then AI could generate entire files and the software engineer became a bit of a discerning copy pastaster. s9 00:01:19.520 --> 00:01:23.680 Um, but they still heavily had control over the output of the AI. s10 00:01:23.680 --> 00:01:27.360 And then you had agents that were more tightly coupled with the code. s11 00:01:27.360 --> 00:01:32.880 They could gather their own context, call tools, and debug their own output. s12 00:01:32.880 --> 00:01:39.840 Now, autocomplete used to feel magical, and now at best it's it's a quaint thing we remember. s13 00:01:39.840 --> 00:01:52.240 You want to look at the speed at which AI engineering is climbing the abstraction ladder from tokens to files to context now to entire systems. s14 00:01:52.240 --> 00:01:56.079 Every shift you have here is an adjustment. s15 00:01:56.079 --> 00:02:04.320 But each move up the abstraction ladder has made engineers more valuable, not less. s16 00:02:04.320 --> 00:02:07.040 And so now we're at a transformational point. s17 00:02:07.040 --> 00:02:17.360 Engineers are going from typing code to governing agents and eventually you'll organize those agents into a system. s18 00:02:17.360 --> 00:02:19.680 That's your software factory. s19 00:02:19.680 --> 00:02:32.000 So with this the engineer's role shifts to keeping guard rails tight, catching code drift and setting the highle priorities and direction. s20 00:02:32.000 --> 00:02:43.200 But meanwhile, the system is ingesting input signals like bug reports or user feedback and outputting production code. s21 00:02:43.599 --> 00:02:52.720 So if you have AI systems that are producing more code than humans, there are some big questions that everyone is rightly asking. s22 00:02:52.720 --> 00:02:57.760 First, how can I prevent massive cost overruns? s23 00:02:57.760 --> 00:03:10.800 We've all seen the headlines about Uber's CTO blowing through the years, AI budget by April, or Microsoft clawing back clawed code licenses to try and cut down on spend. s24 00:03:10.800 --> 00:03:14.800 At Factory, you're not locked into a single model provider. s25 00:03:14.800 --> 00:03:21.360 And this means you unlock the entire spectrum of cost performance with LLMs. s26 00:03:21.360 --> 00:03:29.920 You want the maximum uh cost efficiency and the minimum effectiveness necessary for each task. s27 00:03:29.920 --> 00:03:39.760 As our CEO likes to say, uh if your kid needs a algebra tutor, you can definitely find someone cheaper than Albert Einstein. s28 00:03:40.000 --> 00:03:47.120 Second point, how can I ensure the best access to whatever model is top right now? s29 00:03:47.120 --> 00:03:52.400 Well, we think that having to back the right horse is a false choice. s30 00:03:52.400 --> 00:03:57.040 Um, especially with the increasing effectiveness of open source models. s31 00:03:57.040 --> 00:04:05.360 We're strong proponents of being able to choose the right tool for the right job or even now have it dynamically routed. s32 00:04:05.439 --> 00:04:08.480 Number three, what happens to software engineers? s33 00:04:08.480 --> 00:04:12.000 What does the future of software engineering look like? s34 00:04:12.000 --> 00:04:19.919 Well, the role is obviously evolving and the line between product and engineering is blurring. s35 00:04:19.919 --> 00:04:32.000 Engineers will now build and maintain the system that's responsible for building your production code and they'll set the highle direction for the software factory. s36 00:04:32.240 --> 00:04:35.680 And lastly, how is this going to change the organization? s37 00:04:35.680 --> 00:04:45.040 Well, team boundaries are also starting to blur and context needs to flow seamlessly across these boundaries. s38 00:04:45.040 --> 00:04:47.360 Information can't be siloed. s39 00:04:47.360 --> 00:04:52.400 That was true for humans, but it's especially true for AI agents. s40 00:04:52.400 --> 00:05:00.080 Everyone across all your teams will have a shared responsibility for the pillars of the software factory. s41 00:05:00.240 --> 00:05:04.080 Now, coding was the easy part. s42 00:05:04.080 --> 00:05:06.160 This actually was never the bottleneck. s43 00:05:06.160 --> 00:05:10.080 Your context is laid out in natural language. s44 00:05:10.080 --> 00:05:17.680 I bet if you look at your real engineering metrics, what's the median time to review a PR? s45 00:05:17.680 --> 00:05:24.160 How long does it take you to reproduce user bug reports or go through and debug code? s46 00:05:24.160 --> 00:05:30.240 Who keeps your docs current with every PR and who keeps it current with every strategy meeting? s47 00:05:30.240 --> 00:05:32.240 Who maintains your test suites? s48 00:05:32.240 --> 00:05:38.240 oftentimes the testing teams and the engineering teams and the product teams are each on their own continent. s49 00:05:38.240 --> 00:05:49.919 If you pull your dora metric, the time from code written to time in production, I would bet dwarfs the time actually making those engineering changes. s50 00:05:50.000 --> 00:05:54.720 The real need you have today is cohesion across all of these surfaces. s51 00:05:54.720 --> 00:06:05.600 Right now, docs will go stale, dead code lingers, and there are so many layers of communications between your different teams. s52 00:06:05.600 --> 00:06:12.960 Again, this is hard for human engineers, but it is especially more pronounced for agents. s53 00:06:12.960 --> 00:06:24.160 The necessary undertaking to build your factory is to make sure the agents can bridge all of these services and understand the nuance within your organization. s54 00:06:24.240 --> 00:06:35.520 So this brings us to the software factory which is the system of agents that will take you from input signal to production deployment. s55 00:06:35.520 --> 00:06:38.560 Your agent's not just coding anymore. s56 00:06:38.560 --> 00:06:46.479 It's taking on customer support, product engineering, deployment ops, and many other roles. s57 00:06:46.479 --> 00:07:03.440 Across these pillars, your agent should be triaging incidents, making plans, creating and updating docs, actually executing the code, reviewing it, testing it, rolling it out, and then monitoring that release. s58 00:07:03.440 --> 00:07:10.560 Now, that's obviously a ton of work, and you don't want to jump into that all at once. s59 00:07:10.560 --> 00:07:19.680 For one, it's too heavy of an undertaking, but for two, your teams will not have time to build trust in this system. s60 00:07:19.680 --> 00:07:25.520 You want to start with something narrow and verifiable like incident triage. s61 00:07:25.520 --> 00:07:31.759 Let your agent read a sentry alert, pull the traces in other context, and then post in Slack. s62 00:07:31.759 --> 00:07:37.039 But keep it so your engineers can review this and decide what to do. s63 00:07:37.039 --> 00:07:46.560 And as more correct diagnoses roll in and in from the agent, your team can start to build trust and an understanding of this pillar. s64 00:07:46.560 --> 00:07:55.280 So each step in the software factory needs careful ownership and monitoring before you even think about letting the the pipeline loop. s65 00:07:55.280 --> 00:08:06.960 But once you can tie the output of this system back to the input signal, that's when you unlock organizational self-improvement. s66 00:08:07.840 --> 00:08:14.800 Now we feel pretty strongly that your software factory should have a few key principles. s67 00:08:14.800 --> 00:08:18.720 First, it should be model agnostic. s68 00:08:18.720 --> 00:08:25.919 Obviously, different models excel at different tasks and with very different cost efficiencies. s69 00:08:25.919 --> 00:08:35.919 We found, for instance, that you can achieve the same performance on code review with GPT 5.2 as with the latest Opus models, but at half the price. s70 00:08:35.919 --> 00:08:44.640 And now if you look at the open- source alternatives that can slash your cost down to 10 to 30x cheaper. s71 00:08:44.640 --> 00:08:51.600 So at factory again you have the entire interplay of speed to task efficacy to cost. s72 00:08:51.600 --> 00:08:59.440 Not being locked into a single lab means you unlock the entire PTO frontier of LLMs. s73 00:08:59.440 --> 00:09:05.040 Now number two your deployment model should be sovereign. s74 00:09:05.040 --> 00:09:13.839 You shouldn't have to compromise on your security architecture or your policies to try and fit your AI deployment model in. s75 00:09:13.839 --> 00:09:17.600 Now at factory you can choose your workspace and your deployment model. s76 00:09:17.600 --> 00:09:25.120 We support everything from fully managed to bring your own machine to entirely airgapped. s77 00:09:25.120 --> 00:09:30.720 And number three, your software factory needs to be integrated across the SDLC. s78 00:09:30.720 --> 00:09:34.640 These are no longer separate uncoupled concerns. s79 00:09:34.640 --> 00:09:39.839 Docs changing should inform your code and vice versa. s80 00:09:39.839 --> 00:09:50.800 Planning goals should inform what you test for, not however many communication layers there are between strategy, product, engineering, and your testing teams. s81 00:09:50.800 --> 00:09:54.560 Each one of these pillars supports the software factory. s82 00:09:54.560 --> 00:10:00.800 And weakness in one means a weakness in the entire system. s83 00:10:00.959 --> 00:10:08.160 So tactically in an AI native or what does this look like? s84 00:10:08.160 --> 00:10:19.600 Well, we typically see it as the fusion and the incremental automation of many sub teams under the umbrella of one software factory. s85 00:10:19.600 --> 00:10:31.839 Some really common patterns we see are triaging incoming signals like sentry alerts, having the agent make diagnosis before the human can even answer the page. s86 00:10:31.839 --> 00:10:38.640 Proactively testing security, scanning for secrets or vulnerabilities. s87 00:10:38.800 --> 00:10:46.399 Creating, maintaining your tests, monitoring the ones that are becoming brittle or unneeded. s88 00:10:46.399 --> 00:10:48.560 Again, keeping docs in sync with code. s89 00:10:48.560 --> 00:11:05.519 keeping docs in sync with your different meeting channels and then the classic reviewing code, commenting on PRs, rollouts, monitoring releases and with the agent doing all of this, the human engineer's role shifts to responding s90 00:11:05.519 --> 00:11:13.120 to exercising judgment and correcting the AI in such a way that it can learn from these mistakes. s91 00:11:13.120 --> 00:11:17.360 Now everything is transitive across sub teams. s92 00:11:17.360 --> 00:11:23.200 An improvement to one group means an improvement to the entire system. s93 00:11:23.200 --> 00:11:28.880 So why should your enterprise care about a software factory? s94 00:11:28.880 --> 00:11:37.600 Well, the larger and the more fragmented you are, the more you feel the linear pain of managing teams. s95 00:11:37.600 --> 00:11:45.120 In tech terms, your software factory should take this pain from O of N to O of one conservatively. s96 00:11:45.120 --> 00:11:59.600 You can define your guardrails, your agents, your docs, testing policies, really governance just one time and then standardize it across the organization. s97 00:11:59.760 --> 00:12:08.480 And this standard should make sure that everyone is pulled to the highest bar, not the lowest common denominator. s98 00:12:08.480 --> 00:12:20.560 Now also if agents can move across these surfaces then they can hold that quality bar across teams without you having to linearly scale your time to manage each one. s99 00:12:20.560 --> 00:12:32.399 And if agents are moving across all these services then you have one place where context can be unified and you can have executive visibility into how the entire SDLC s100 00:12:32.399 --> 00:12:34.720 is performing. s101 00:12:35.279 --> 00:12:44.800 Now, just like model agnosticism removes lock in, your agents should also be surface agnostic. s102 00:12:44.800 --> 00:12:47.920 They need to be highly available to your teams. s103 00:12:47.920 --> 00:12:59.040 Regardless of what tools they use, your teams should not have to compromise on their processes or their tooling to try and fit your AI provider in. s104 00:12:59.040 --> 00:13:07.600 At Factory, we have agents available everywhere from remote machines to the CLI to Slack, any tool your teams use. s105 00:13:07.600 --> 00:13:16.880 And even though humans and agents may use different services, everything should share the same underlying harness and context. s106 00:13:17.680 --> 00:13:28.800 So the undertaking of building the factory will expose tribal knowledge, manual processes, and outdated info. s107 00:13:28.800 --> 00:13:35.839 Unblocking these is not only going to help your agents, but it's also going to help your engineering teams. s108 00:13:35.839 --> 00:13:43.440 At factory, we have the concept of agent readiness and we evaluate this based on eight pillars. s109 00:13:43.440 --> 00:13:45.440 The first is validation. s110 00:13:45.440 --> 00:13:49.920 Do you have guard rails like llinters and formatterers in place? s111 00:13:49.920 --> 00:13:52.399 Then your build system. s112 00:13:52.399 --> 00:13:56.639 Are your build commands and CI well documented? s113 00:13:56.720 --> 00:13:57.839 Feedback loops. s114 00:13:57.839 --> 00:14:06.240 You have unit and integration tests tight and you have docs like readmes, agents.mmd files. s115 00:14:06.480 --> 00:14:12.240 Are your dev environments cleanly reproducible such that an agent can use them? s116 00:14:12.240 --> 00:14:17.920 Is your code modular with clear boundaries and rules against sprawl? s117 00:14:17.920 --> 00:14:19.839 Do you have observability? s118 00:14:19.839 --> 00:14:24.560 How long does it take you to find out why something went wrong? s119 00:14:24.560 --> 00:14:32.079 and are you proactively security scanning so that you don't have to worry about leaked secrets or vulnerabilities. s120 00:14:32.079 --> 00:14:44.959 Basically, the key takeaway here is that if your agents and your human engineers are going to walk the same roads, paving them well is doubly useful. s121 00:14:45.440 --> 00:14:48.959 Now, you can't govern what you can't see. s122 00:14:48.959 --> 00:14:52.320 And that's true for agents and human engineers. s123 00:14:52.320 --> 00:15:03.040 You can start with this agent readiness assessment, but no matter what, your engineers must monitor agent performance and control behavior. s124 00:15:03.040 --> 00:15:06.399 At factory, every action is auditable. s125 00:15:06.399 --> 00:15:11.440 It operates under your standards with role-based access control and least privilege. s126 00:15:11.440 --> 00:15:20.480 For example, your teams that are building this software factory need to first build trust and understanding. s127 00:15:20.480 --> 00:15:24.000 That's why you should roll this out incrementally. s128 00:15:24.000 --> 00:15:31.120 You want to automate the individual pillars first before you turn this system into a loop. s129 00:15:31.600 --> 00:15:37.440 And again, precise execution does not prevent poor design. s130 00:15:37.440 --> 00:15:39.279 It never has. s131 00:15:39.279 --> 00:15:46.560 You still heavily need humans in the loop for architecture, for strategy, and for setting direction. s132 00:15:46.560 --> 00:15:58.399 Engineers will not be replaced, but judgment and prioritization now beat mechanical implementation skill. s133 00:15:58.399 --> 00:16:00.560 Agents won't be able to handle everything. s134 00:16:00.560 --> 00:16:04.720 There will be guardrails to tighten, code that drifts. s135 00:16:04.720 --> 00:16:09.040 Humans need to stand as validation gates in this factory. s136 00:16:09.040 --> 00:16:17.040 But just like we don't have to feed punch cards into the machine anymore, code does not have to be written by hand. s137 00:16:17.040 --> 00:16:24.720 Your team should be able to set the strategy and their risk parameters and then let agents execute. s138 00:16:24.800 --> 00:16:31.839 And knowing if your agents are executing well comes down to measuring the right things. s139 00:16:31.839 --> 00:16:37.040 Basically, you want to measure outcomes, not tokens. s140 00:16:37.040 --> 00:16:41.839 Lines of code and tokens generated are gameable metrics. s141 00:16:41.839 --> 00:16:45.680 These are not actually associated with outcomes. s142 00:16:45.680 --> 00:16:51.920 Token leaderboards at big companies just encourage proflegate spending. s143 00:16:51.920 --> 00:16:53.920 It's goodart's law in real time. s144 00:16:53.920 --> 00:17:03.600 If you decide to measure token usage, engineers are going to optimize for token usage and then it's no longer a good metric. s145 00:17:03.759 --> 00:17:06.559 It's not about who spends the most on tokens. s146 00:17:06.559 --> 00:17:12.559 It's about who can get the highest leverage outcomes from that spend. s147 00:17:12.559 --> 00:17:25.439 Some metrics that we like at Factory are signal to production time, human intervention count, median time to repair, shelf life of code, and cost per PR. s148 00:17:25.439 --> 00:17:30.000 These are real tangible outcomes. s149 00:17:30.960 --> 00:17:37.039 So agent readiness is really a precursor to effective engineering. s150 00:17:37.039 --> 00:17:52.480 Whether it's an agent or a human that ends up in that system, your agents need the exact same careful governance as your engineers, tooling, access levels, systems for review. s151 00:17:52.480 --> 00:17:56.400 Don't just turn them loose in your codebase. s152 00:17:56.880 --> 00:17:59.440 And don't token max. s153 00:17:59.440 --> 00:18:02.240 Measure what actually matters. s154 00:18:02.240 --> 00:18:06.640 Your end users don't care about how much you're spending on AI. s155 00:18:06.640 --> 00:18:12.400 They care about feeling the dramatic improvements that will come from your software factory. s156 00:18:12.400 --> 00:18:14.559 Thank you. s157 00:18:30.237 --> 00:18:32.237 [music]