WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=HvboD89DyQ8 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/HvboD89DyQ8.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.560 --> 00:00:15.880 Well, thank you for joining the final day of the conference here. s2 00:00:15.880 --> 00:00:18.480 I know it's been a long week. s3 00:00:18.480 --> 00:00:24.480 But I'm excited to talk about a topic here that is very near and dear to me. s4 00:00:24.480 --> 00:00:26.280 Software factories. s5 00:00:26.280 --> 00:00:31.360 Um You this is probably not a new concept to any of you. s6 00:00:31.360 --> 00:00:34.840 I'm guessing if you're all here, you're very familiar with software factories. s7 00:00:34.840 --> 00:00:39.560 You know, Ramp blogged about their inspects several months ago. s8 00:00:39.560 --> 00:00:40.840 I think maybe December last year. s9 00:00:40.840 --> 00:00:42.080 I forget exactly when. s10 00:00:42.080 --> 00:00:42.840 It's been a little while. s11 00:00:42.840 --> 00:00:53.800 And that really kind of like excited the entire industry of thinking like, oh, we can actually build software very differently now that AI is extremely capable of writing code. s12 00:00:53.800 --> 00:00:57.240 Um a lot of other companies have followed suit. s13 00:00:57.240 --> 00:01:00.440 Um and there's kind of like a standard set up here. s14 00:01:00.440 --> 00:01:03.560 Uh you have a sandbox that you put your code into. s15 00:01:03.560 --> 00:01:07.480 Um you put a an AI agent in there. s16 00:01:07.480 --> 00:01:11.320 Maybe a Claude code section or an open code. s17 00:01:11.320 --> 00:01:15.160 Um and with a prompt, it generates a PR and you merge it. s18 00:01:15.160 --> 00:01:18.920 Um You might be building this already [clears throat] yourself. s19 00:01:18.920 --> 00:01:23.880 We have taken a slightly different approach at WorkOS. s20 00:01:23.880 --> 00:01:34.960 Um and the motivation for what we are building is that we see a lot of the kind of success metrics um really focusing on output, right? s21 00:01:34.960 --> 00:01:47.440 Like you'll see posts on X and blogs talking about the percentage of PRs or the number of pull requests or how much code uh is being generated by AI s22 00:01:47.440 --> 00:01:49.000 moving into production. s23 00:01:49.000 --> 00:01:59.080 Um and this is just really output metrics that um you know, this can actually disguise how well these systems are working. s24 00:01:59.080 --> 00:02:04.760 Percentage of PRs or number of PRs may be lying in overall increase in PRs. s25 00:02:04.760 --> 00:02:11.120 It's very hard to distinguish whether the output is actually driving outcomes. s26 00:02:11.120 --> 00:02:16.520 And so that's kind of where we focus our efforts when we're thinking about a software factory. s27 00:02:16.520 --> 00:02:25.280 It's like if we're going to go put engineering time into building this level of automations and there's a bit of work that needs to go into this. s28 00:02:25.280 --> 00:02:27.760 Full-time teams dedicated to this. s29 00:02:27.760 --> 00:02:36.480 How do we want to think about the success of this and whether this is actually driving value for our organization, our engineering organization? s30 00:02:36.480 --> 00:02:44.160 And so we measure instead of just code output, we're looking at outcome metrics. s31 00:02:44.160 --> 00:02:50.040 And that is really about like are we accelerating our ability to deliver features? s32 00:02:50.040 --> 00:02:57.360 We think that the dream of the software factory is that it gives each of our engineers a small engineering team for themselves. s33 00:02:57.360 --> 00:03:09.920 And so the consequence of that really needs to be that we're building more and we're shipping more and that as we embark on complex features, those can get built a lot more a lot more quickly. s34 00:03:11.239 --> 00:03:12.800 So what does this actually look like? s35 00:03:12.800 --> 00:03:21.040 Well, we kind of started with the same thing you've seen in other other factories. s36 00:03:21.040 --> 00:03:24.120 We start with a sandbox we built on top of Cloudflare. s37 00:03:24.120 --> 00:03:27.320 We put an open code model router in there. s38 00:03:27.320 --> 00:03:29.280 We're feeding it prompts. s39 00:03:29.280 --> 00:03:36.480 Um, and we kind of very quickly kind of ran into this situation where this wasn't driving more outcomes. s40 00:03:36.480 --> 00:03:43.120 This was not an incremental increase or an exponential increase over engineers just driving cloud code on their laptops. s41 00:03:43.120 --> 00:03:47.000 It was actually pretty indistinguishable for us. s42 00:03:47.040 --> 00:03:55.840 And so we really kind of started to think about, well, how can we embed our engineering processes into the factory itself? s43 00:03:55.840 --> 00:03:59.320 It's not sufficient that the factory is producing code. s44 00:03:59.320 --> 00:04:08.000 We actually want it to take a lot of the other work that our engineers do to produce products and automate that as well. s45 00:04:08.000 --> 00:04:10.800 So, we kind of split this into two parts. s46 00:04:10.800 --> 00:04:13.840 One is we have the system we call TARS. s47 00:04:13.840 --> 00:04:19.640 This is a way for users to interact with the coding agent. s48 00:04:19.640 --> 00:04:22.640 And it is embedded into the tools that we use. s49 00:04:22.640 --> 00:04:27.440 So, not just Slack, but also Linear and GitHub. s50 00:04:27.440 --> 00:04:38.680 We subscribe to webhooks through TARS so that TARS can actually track the progress of projects in addition to generating the code outputs. s51 00:04:38.720 --> 00:04:42.120 And then we built a separate system called Horizon. s52 00:04:42.120 --> 00:04:44.400 And this is the infrastructure orchestration layer. s53 00:04:44.400 --> 00:04:52.640 This is a lot more similar to Inspects and Minions and and some of the other systems that other companies have built. s54 00:04:52.640 --> 00:04:56.200 But, it sits in front of an MCP gateway. s55 00:04:56.200 --> 00:05:04.200 And I'm going to talk a little bit more about why that MCP gateway has been really transformational for us. s56 00:05:07.280 --> 00:05:21.840 So, by feeding our webhooks, activities that are happening in other uh source code systems and project tracking systems, we're starting to get to that level of autonomy where our s57 00:05:21.840 --> 00:05:25.720 factory is performing product engineering work, not just code work. s58 00:05:25.720 --> 00:05:27.560 So, what's an example of this? s59 00:05:27.560 --> 00:05:31.360 In Linear, we can define tickets that have dependencies. s60 00:05:31.360 --> 00:05:33.080 So, one ticket blocks another ticket. s61 00:05:33.080 --> 00:05:38.800 Pretty common for, you know, breaking down uh large units of work into smaller ones. s62 00:05:38.800 --> 00:05:46.840 Because uh TARS is getting webhooks on ticket completion, it can automatically pick up the next ticket in a cycle. s63 00:05:46.840 --> 00:05:58.720 And so, we can do through our planning process construct like a map of how this plan we think this plan is going to get executed and TARS can start to execute on that autonomously. s64 00:05:58.720 --> 00:06:08.800 The other thing that we have is in between those steps when a ticket is completed, we can ask TARS "Can you reevaluate the linear project and let me know if it's missing now tickets?" s65 00:06:08.800 --> 00:06:13.560 Because as you do work, you're learning about where the gaps in your plan. s66 00:06:13.560 --> 00:06:20.160 And we want to continuously keep our our plan fresh by using the agent itself. s67 00:06:20.160 --> 00:06:30.040 Because the agent is determining that there may be missing pieces to what we initially planned for the project and the goal that we have in mind. s68 00:06:31.560 --> 00:06:34.640 So, we run this product engineering culture at WorkOS. s69 00:06:34.640 --> 00:06:38.720 This is where engineers are responsible for a lot of the product functions. s70 00:06:38.720 --> 00:06:41.440 We don't have product managers on teams today. s71 00:06:41.440 --> 00:06:48.600 Um and one of the rituals as part of this product engineering process is we create a hilltop document. s72 00:06:48.600 --> 00:07:00.520 It's a PRD and it is intended to both define what is the purpose of this project, but also incorporate um what are customers talking about? s73 00:07:00.520 --> 00:07:03.400 Like where are we seeing the need for this unit of work? s74 00:07:03.400 --> 00:07:05.760 Uh it looks at competitive analysis. s75 00:07:05.760 --> 00:07:10.320 Like are there similar products out in the market today that we can draw inspiration from? s76 00:07:10.320 --> 00:07:13.880 We start to bring design early design screens into this. s77 00:07:13.880 --> 00:07:17.240 Um and we outline like what are the major milestones. s78 00:07:17.240 --> 00:07:28.680 And so, by kind of consolidating and canonicalizing this information into a document, this is something our product engineers have been doing uh throughout the entire uh company. s79 00:07:28.680 --> 00:07:34.960 Um now we can ask give this resource to an agent and an agent can break this into units of work. s80 00:07:34.960 --> 00:07:43.960 Uh and so, this has been a really uh powerful way for us to take existing processes and encode them into the factory themselves. s81 00:07:43.960 --> 00:07:46.200 Like I mentioned that it's listening for web hooks. s82 00:07:46.200 --> 00:08:00.520 I'll show an example of what this looks like, but it it can see that the hilltop document through linear tickets has been reviewed and approved and by virtue of it being approved can start work on this project automatically. s83 00:08:00.520 --> 00:08:07.120 So we don't need a human to be shepherding this agent through every single step of the life cycle. s84 00:08:09.920 --> 00:08:13.320 So we created an agent specifically for this part of the process. s85 00:08:13.320 --> 00:08:14.960 We call it the PM. s86 00:08:14.960 --> 00:08:21.800 And it is doing the first draft of the hilltop based on the brief specification that we give it. s87 00:08:21.800 --> 00:08:31.760 It's adding context to that in addition to reading the human reviews and then it picks up the implementation and breaking that into tickets. s88 00:08:31.760 --> 00:08:36.320 And there's an opportunity for humans to stay in the loop for each part of this process. s89 00:08:36.320 --> 00:08:44.039 I mean it's very common for a human to intervene or comment on a ticket that's generated by AI, give it further guidance or refinements. s90 00:08:44.039 --> 00:08:52.280 Um but it's really about that like cold start problem where we can kind of get over the hump of creating all these different resources. s91 00:08:52.280 --> 00:08:55.040 Okay, so what does this actually look like? s92 00:08:55.040 --> 00:08:59.360 So here's a project I kicked off last month two months ago. s93 00:08:59.360 --> 00:09:03.720 Um we're adding a new API to products I lead called Vaults. s94 00:09:03.720 --> 00:09:16.000 And with a command here in um in Slack I can kick off this project with a simple description, usually a few sentences of what I'm trying to accomplish. s95 00:09:16.000 --> 00:09:18.920 Tarsin goes in and creates all these resources for me. s96 00:09:18.920 --> 00:09:21.600 I don't need to go and create the project in linear. s97 00:09:21.600 --> 00:09:25.160 I don't need to create a draft of the notion document. s98 00:09:25.160 --> 00:09:29.440 Um all of the decision logs and the open questions. s99 00:09:29.440 --> 00:09:31.960 Um Tars is creating a first pass at that. s100 00:09:31.960 --> 00:09:44.520 And so this then gives me an easy framework for me to step into as the lead product engineer and start giving more specification, rounding out areas that aren't well defined s101 00:09:44.520 --> 00:09:52.720 with knowledge that I had know about what we're trying to accomplish with the project, and then hand it back to Tars uh for execution. s102 00:09:53.200 --> 00:09:57.160 So, here's an example of uh the project. s103 00:09:57.160 --> 00:10:04.240 Um and it's like I said, it sets up these first milestones around our product engineering process. s104 00:10:04.240 --> 00:10:07.240 So, we haven't done the hilltop review yet here. s105 00:10:07.240 --> 00:10:17.120 When that's done, we'll uh we'll we'll mark this ticket as complete and Tars will pick that up and move on to the next stage of project implementation. s106 00:10:17.800 --> 00:10:29.640 So, what we found is like a lot of teams are now doing more project brief and hilltop work because our agent can seed all that information. s107 00:10:29.640 --> 00:10:32.080 It's that blank page problem with writing. s108 00:10:32.080 --> 00:10:37.120 If you can come into a document that already has information, and you know, these agents aren't perfect. s109 00:10:37.120 --> 00:10:45.360 There are times where it grossly overestimates what we're trying to accomplish with this project, and we have to cut out a lot of the scope that it comes up with. s110 00:10:45.360 --> 00:10:45.920 But, that's fine. s111 00:10:45.920 --> 00:10:54.880 That's a lot simpler for an engineer to add input into into rather than them spending time to set up all these primitives themselves. s112 00:10:54.880 --> 00:11:03.960 And then we can continue to drive this work, you know, through Slack, through Linear, um and ultimately uh results in PRs in GitHub. s113 00:11:03.960 --> 00:11:12.320 The other thing that this lets us do is we don't necessarily always need to use our coding agent to implement uh pieces of work. s114 00:11:12.320 --> 00:11:13.400 Uh Devin is very popular. s115 00:11:13.400 --> 00:11:15.720 We have a lot of folks that really like using Devin. s116 00:11:15.720 --> 00:11:24.480 They can point Devin to these tickets and the same documentation to give Devin the context it needs to go and implement different parts of the work. s117 00:11:24.480 --> 00:11:29.120 Uh same with the local cloud code if you're just using Opus in a local harness. s118 00:11:29.120 --> 00:11:41.240 Um it can grab through MCP all of these uh all of these pieces of documentation and use that as context for its work. s119 00:11:41.520 --> 00:11:45.440 And then, of course, we have people that collaborate in these project channels as well. s120 00:11:45.440 --> 00:11:53.400 You'll notice our security team will come in and look take a look at new projects, weigh in on the security implications, and so forth. s121 00:11:59.720 --> 00:12:07.520 So, I alluded to we we kind of early on in the the development of our factory is created our own MCP gateway. s122 00:12:07.520 --> 00:12:10.000 Um, we call it our context engine. s123 00:12:10.000 --> 00:12:21.960 Um, this connects into all of our internal systems, but it also builds system prompts and context around what how to navigate these tools and when to use these tools. s124 00:12:21.960 --> 00:12:27.000 So, it is connected into Snowflake, which is our main data lake. s125 00:12:27.000 --> 00:12:35.480 We have semantic tables that we've built in Snowflake that describe product utilization or customer conversations. s126 00:12:35.480 --> 00:12:46.560 And then, we can provide an agent through our MCP server in the tool description list of here are the tables, this is the content they contain. s127 00:12:46.560 --> 00:12:52.160 If you're trying to answer questions about this type of content, write a query for these tables. s128 00:12:52.160 --> 00:13:02.839 And so, it gives us a little bit of orientation to an agent to understand how to use how we use our tools, how we've organized linear in Snowflake, and s129 00:13:02.839 --> 00:13:06.720 give some guidance on where to find information. s130 00:13:06.720 --> 00:13:15.000 What has been really surprising about this is we actually now use this MCP server across a lot of other different internal tools. s131 00:13:15.000 --> 00:13:21.480 We open it up for people to query directly from Slack to do data or customer analysis. s132 00:13:21.480 --> 00:13:28.600 And so, while we initially kind of envisioned that this would just be a way to connect our agent or coding agent to all of our systems. s133 00:13:28.600 --> 00:13:37.880 Uh this has been an incredible piece of leverage for a lot of our internal teams, and we're building a lot of other tools on top of this MCP gateway. s134 00:13:37.880 --> 00:13:49.520 So, if you're just getting started with thinking about a software factory, it's well worth your time to invest in an internal MCP gateway server that both connects all your tools, s135 00:13:49.520 --> 00:14:02.160 has descriptions that can tell the agent how to use those tools and how you specifically organize the information in them, and I think you'll find that that ends up being useful in a lot of other use cases. s136 00:14:06.600 --> 00:14:11.880 Like I mentioned, we try to automate other parts of our software development. s137 00:14:11.880 --> 00:14:17.200 Um we have like bug bug requests that come in through Slack. s138 00:14:17.200 --> 00:14:25.800 Uh so, TARS through webhooks can listen to those and take a first pass at triaging and implementing opening PRs to fix bugs. s139 00:14:25.800 --> 00:14:30.400 Uh we have a lot of our customers uh in shared channels in Slack, as well. s140 00:14:30.400 --> 00:14:42.560 Uh we found TARS to be pretty useful to uh triage support requests that we're getting through them, because it can look at the code, it can often understand like where the customer might be running into s141 00:14:42.560 --> 00:14:46.400 problems with our product uh from the code perspective. s142 00:14:46.400 --> 00:14:51.960 Um I showed you an example of how we kind of organize major features around this. s143 00:14:51.960 --> 00:14:59.080 Uh and then the last place I think is really exciting, we're all trying to get to the self-improving software, or the self-driving software. s144 00:14:59.080 --> 00:15:05.040 So, we're using uh TARS to actually build out our own uh sandbox infrastructure. s145 00:15:05.040 --> 00:15:11.960 So, we want to move off of some of the sandboxes as a service and own that infrastructure layer ourselves. s146 00:15:11.960 --> 00:15:25.520 Um Some of the reasons we have for that is we want really deep control over the session information, and uh be able to move workloads around different parts of our infrastructure. s147 00:15:25.520 --> 00:15:37.960 Uh so what we're building out next is our memory layer and we want this to be an evergreen context uh about what every person in the company is working on, what team do they sit in, s148 00:15:37.960 --> 00:15:45.360 what products are they responsible for, and then at an organization level, what is the semantics about Work OS and how we do our work. s149 00:15:45.360 --> 00:15:55.080 And this we want to be able to plug into both our software factory, but lift that context out of our factory and into our other AI tools as well. s150 00:15:57.800 --> 00:16:08.320 So, I think this is kind of our our major takeaway of where we've seen a lot of folks invest effort just around outputs and where we think we can actually get that s151 00:16:08.320 --> 00:16:11.840 human exponential value out of our factory. s152 00:16:11.840 --> 00:16:15.640 Can we be actually delivering values to our customers? s153 00:16:15.640 --> 00:16:22.680 We think a lot about shipping and customer impact and so we want to encode that into their our software factory itself. s154 00:16:22.680 --> 00:16:28.040 Um certainly worried about, you know, introducing instability into our products. s155 00:16:28.040 --> 00:16:36.400 So, all right, can we measure uh defect rate and other um you know, time to recovery uh metrics? s156 00:16:36.400 --> 00:16:46.640 Um and then a lot of anecdotal, like we want to see our engineers using TARS as a sign that this is giving them value and accelerating their work. s157 00:16:46.640 --> 00:16:54.040 We want to see that they're moving off electing to move out of their local harness and into a cloud sandbox. s158 00:16:58.120 --> 00:17:04.720 And then the real goal is we can use this information because we own the infrastructure, we can see what's happening in the infrastructure. s159 00:17:04.720 --> 00:17:08.079 We use this information to self-improve the factory. s160 00:17:08.079 --> 00:17:13.880 We want it to learn, we want it to get better about how it writes code. s161 00:17:13.880 --> 00:17:18.959 We want to see where um our engineers can improve their skills. s162 00:17:18.959 --> 00:17:21.839 There's so much moving quickly in AI. s163 00:17:21.839 --> 00:17:28.560 It's kind of like a constant race to keep up with the latest um you know, tips and techniques. s164 00:17:28.560 --> 00:17:34.880 We can actually point agents at sessions and see where are the gaps and how people are using our factory. s165 00:17:34.880 --> 00:17:40.200 Where is there a skill that we should be building because the agent made a mistake. s166 00:17:40.200 --> 00:17:41.880 What skills are no longer relevant? s167 00:17:41.880 --> 00:17:48.280 Maybe the code has changed to a degree where a skill that we wrote six months ago is now obsolete. s168 00:17:48.280 --> 00:17:50.760 So, that continues verification. s169 00:17:50.760 --> 00:17:58.480 Semi-online is where we kind of see a big advantage to actually owning all the infrastructure ourselves. s170 00:18:02.520 --> 00:18:07.200 So, say it for the third time, it's worth repeating. s171 00:18:07.200 --> 00:18:09.120 Sandboxes are great. s172 00:18:09.120 --> 00:18:12.200 It's cool to run an agent and have it open a PR. s173 00:18:12.200 --> 00:18:18.640 We really think about our software engineering practices and processes and we want to encode those in automation. s174 00:18:21.120 --> 00:18:21.720 I'm Ryan. s175 00:18:21.720 --> 00:18:23.640 I'm one of the engineers here at WorkOS. s176 00:18:23.640 --> 00:18:26.080 We have a booth right opposite the speaking area. s177 00:18:26.080 --> 00:18:29.120 If you are building software factories, I would love to talk to you. s178 00:18:29.120 --> 00:18:31.600 I would really love to hear how you're handling authorization. s179 00:18:31.600 --> 00:18:38.200 This is something I don't talk about because we haven't figured it out yet for ourselves, but maybe you all you all have some insights. s180 00:18:38.200 --> 00:18:41.880 So, please come by the booth, talk to me, would love to chat. s181 00:18:41.880 --> 00:18:44.080 Thank you. s182 00:18:55.783 --> 00:18:57.783 [music]