What It Actually Takes to Build a Software Factory — Tereza Tížková, Factory

AI Engineer · 22 min · 186 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:36Uh hi everyone.
00:38Hey what's up?
00:39I'm Teresa.
00:40Uh and today I will be talking about software factory.
00:44uh everyone is talking about software factory but only few people are actually building one and even fewer people know what it actually takes to build one and how to define it.
00:54So I will be talking about what is a software factory, should I build your own or outsource it, what works in production already, what are the main challenges and how much is this all going to cost?
01:08Uh my name is Theresa.
01:09I work at a company called factory.com and we've been building uh this concept of the software factory for quite a long time but finally uh the technology is catching up and it's possible to build this in production
01:23for enterprises like EY or Adobe uh I would define the software factory as the whole loop the whole life cycle of developing software with autonomy which doesn't mean just coding and generating code
01:36by that I mean collecting all the signals reacting to user feedback to logs, prioritizing what's important, then orchestrating it all, executing, validating, doing really good uh testing in production and then iterating on all this uh while also continuously
01:54improving in the process and gaining new knowledge and new skills.
01:58So this is just our dashboard how our software factory looks if you if you look at it and it's catching the whole cycle there.
02:06uh this is just some history and by history I mean 2023 in AI.
02:12So the software factory wasn't really possible before even though we always had this idea from the beginning of uh CH GPT lounge and others we had this idea of the continuous loop there was autog
02:24baby agi already with the concepts of iterating on the software but it didn't work before because the LMS were hallucinating there was problem with context length uh problem with the reasoning quality
02:37uh different problems like this or we were missing good environments where the agents can actually work in the isolation.
02:44So all this kind of led to why the software factory is the paradigm that's starting to be popular now and actually useful.
02:55I like to define things by what they are not.
02:58So I want to say software factory is not just coding agent and it's not even a swarm of coding agents even thousands of agents because generating code writing code that's the easy part
03:10compared to all the others.
03:11uh engineers don't even spend most of the time uh just writing code.
03:15The challenges are the rest of it.
03:17Uh so this is what I want to enhance and it's also not just some consultancy or just some abstract strategy that someone will try to sell you and to your organization.
03:28Uh I think the approaches are of course different but we believe you should really rebuild your organization from ground up to be ready to become a software factory like you can't
03:40invite a consultancy and just throw something in the middle of your organization.
03:44You should really be mindful and rebuild it from scratch.
03:50uh this is the approach I like uh showing to building the software factory and I think it's similar to building the real team of humans because there is a lot of agents
04:01as I said they will be testing validating iterating and it all can end up as a big chaos uh so you need to be mindful and I would say the three important things is being agnostic
04:13in the software factory so being independent of LLM choices uh of how you already work as the organization then you need to be autonomous.
04:22So you need to give the agents the right trusts and all this permissions and governance and you need to trust it to run really for a long time because the predictions are that agents will run even a year or more years without the human in the loop.
04:35Of course that sounds ambitious but for example our missions already our long runninging sessions already build uh things and run for weeks.
04:43So it's not that crazy.
04:45And third thing always improving.
04:47So same as human organization you need to onboard any new people in your team by giving them good uh code base understanding good structure documentation and also let them improve in the process and gain new knowledge and share it share it within the team.
05:04So these are the three points and now let's talk about the agnostic one.
05:09Uh I think the important thing to point out if you are a builder and building different primitives of the software factory you need to be ready for how everyone is already working
05:19you need to work across environments like Slack, GitHub and connected to everything and you need to also allow people to already bring subscriptions they are using because there is every
05:31every time there is like new technology, new frontier models, uh new benchmarks and it's so difficult to catch up but also to predict what we'll be winning.
05:40So it's best to be agnostic for everyone.
05:44Uh then there's the topic of being agnostic to models.
05:47Uh this is a topic of a lot of discussions recently and this chart is from this chart is from CEO of Coinbase uh who tweeted about how they started
05:58uh started saving money in their organ or organization on AI but without reducing the token spend.
06:04So the black line is how much they spent on tokens.
06:07They continue growing and token maxing but they stopped spending so much money.
06:13Uh how they did it uh different tricks like different default default models for people.
06:18So stop pushing people to use only the frontier as a default.
06:22Then caching of course uh to stop preferring the stuff every time.
06:27And then no limits on spending but just needing to see the results if you spend a lot.
06:32And the last thing that's really important topic is routing.
06:36So smart routing between LLMs and just a token optimizing between uh the LMS instead of just spending so much tokens.
06:46Uh we built uh a thing in factory called uh automatic model routing and there's been a lot of questions around it but what it does is it automatically routes between different models
06:59in your process.
07:00The important thing is it decides first what model is most optimal and then uh it can switch to different model if it's failing the task or in case of any troubles but it usually doesn't happen
07:12and important thing is it doesn't just help uh spend money.
07:18It also helps with reliability or with speed because open source models are often faster and uh if one LM provider fails you can just switch to another one automatically.
07:29So it really does more things than just saving the money.
07:33This uh is our benchmark which is very conservative.
07:36I prefer conservative benchmarks but I think the direction is clear.
07:40You can save for example 25% but even more probably.
07:46Uh how the routing works uh there are four parts.
07:49First you just assign the task and you don't really need to do anything but as an organization you can for example give different permissions and different default models to different people
08:00which is very useful like marketing or sales or engineers can both uh have different default models and then you have the classification which is the important part.
08:09This is the magic of the routing.
08:11You need to really uh look at the structure of the prompt of the uh code base of how difficult the task is, what tools are being used, all these factors and make a classification
08:22of difficulty of the task.
08:24Uh after that you basically do a threshold of what is enough to accomplish the task and you choose the cheapest model above the threshold.
08:34So cheapest model that is predicted to accomplish your task and then you go.
08:41Okay.
08:41So when we launched this we got a lot of questions like does it actually work?
08:45What if uh the model mis routes?
08:48Uh is it slower?
08:49Is it more expensive actually?
08:51What if the model can't accomplish the task or what if it needs to upgrade?
08:55And how do you handle caching?
08:57So I think these are all valid questions and that's why it's so difficult to build a good router like you need to basically classify it very well and the challenge is to not not to need to switch too often but even if you're switching in the middle of the task to difficult to more difficult model
09:15you still overall are faster probably because it's still worth uh this is just some overview of caching uh lot of questions we got on this was also So how do we handle caching?
09:29Do we do discounts for users?
09:31Uh because LLM LLM labs of course save a lot of money by caching and skipping all the context prefill all the time.
09:39Uh open models can do this as well.
09:41I think people sometimes forget this and you can just host open models as well uh on dedicated compute and you can take the same advantage of the caching.
09:51Uh so I want to enhance uh the final price for users.
09:55is just a pricing decision.
09:57It's not a technical challenge because everyone can do caching.
10:00It's just what price you pass on the users and what deals you make with the API providers.
10:07So this is uh being agnostic and now the autonomy which is the core of the software factory.
10:15Uh everyone is talking about loops.
10:17Uh loops I think have been here the whole time already in different context.
10:23They are just now leveling up and moving to the context of agents.
10:27You probably heard about the Ralph loop and specifying the tasks and splitting to subtasks for agents.
10:34Uh this is a known concept and uh the question is not the loop itself but the question is how how you define what it means to be done in the loop.
10:43This is one example from my colleague uh he he made a loop uh of agents building 3D printing of our logo.
10:52uh I think here you can see what is the challenge of the loops because uh before in programming loops had clear criteria of what it means to be done while now the criteria become open-ended
11:03because a lot of the tasks are very nondeterministic it's basically open world you can do even real stuff like printing something and it's really difficult to define how do you verify
11:14that the loop was done that the task was accomplished so that's the difficult part uh yeah they just need to be verifiable This I call the scary chart.
11:23It basically shows that the tasks and how long they run autonomously with agents have been increasing.
11:30But still it's not that reliable.
11:32Even if you can run for a very long time, it doesn't mean that it will be reliable in production and you still probably need to need to iterate on it and uh provide feedback.
11:43So it's still not solved and there are problems like cheating.
11:46For example, if you write the what it means to be done in the wrong way, the agent can try to pass your tests but not really like not really verify what you need to do and accomplish but instead
11:58try to solve just passing your test.
12:00So cheating by that uh we have something called factory missions and this is like long longunning sessions of the agents.
12:09We just call it missions because we send the agent to the mission and uh the missions work in the loop and iterating on task until it's done and they can do it even for weeks
12:19and uh the main agent is orchestrator and then it's uh assigns work to workers agents and then validators who review the task.
12:28So this is one example uh real mission from our customers that run 16 hours and just just to see how important is the validation as well.
12:38it takes even 40% of the whole process.
12:41Uh to summarize, the orchestrator just decides and writes the conditions what it needs to be done.
12:47Then it gives it to the agents called workers.
12:50The agents work on it.
12:51And what is interesting or worth noting is that they work in a sequence.
12:55So they don't work in a swarm or parallel.
12:58The agents work in sequence and everyone accomplishes something and passes it to the next one.
13:03uh we actually found that if you do this you end up with more fresh context and kind of fresh head.
13:09Same with when with humans have like other colleagues verifying their code.
13:14So similar thing we just let the agents pass to the next one.
13:18But still every worker of the sequence can have parallel agents doing smaller tasks.
13:23So I don't know researching on the web or building files during that.
13:27So they are in sequence and everyone has sub agents as well.
13:30and validators they review the output and provide feedback and send it back to the beginning.
13:38Uh important thing is uh the validators judge code that they didn't write.
13:44Uh there's something from the orchestrator agent called validation contract uh that's written before any code is done.
13:51And first type is scrutiny validator.
13:53Second is user testing validator.
13:55The scrutiny is the one who really verifies how the codebase looks.
14:00the liners types tests it's really rigorous check of the code but the second I think is really interesting is user testing validator and that's the one who is really in the arena trying the things so it doesn't care how it was made it just goes
14:16it works in its virtual computer and it clicks on the stuff and really checks if everything works I've seen one engineer migrating uh code bases with droid with our agent
14:28and uh it really needed the agent in the end to click through the stuff because with other products it just created created the product but it didn't work and was wasn't interactive.
14:39It was just a dummy result.
14:41So our agent that really clicked on the stuff uh allowed to check that it's actually working and not just looking good in the code.
14:49I think this is one cool thing that was also allowed by the progress in computer use and the persistent environments virtual machines for agents which also wasn't here before or wasn't that great but now it's really great so it's allowing
15:04to go in the arena as an agent.
15:07Uh this is just one example of real mission from our users.
15:10I I like to just learn what users are building and get feedback and this is just uh how it can look in your dashboard.
15:18And this is how I visualize the missions.
15:21What is interesting is that it's just a loop with smaller loops because every of the agents are running in loops as well, but overall it's like one big loop.
15:32Okay, maybe this is a bit weird, but I just like that it's all loops with smaller loops.
15:39Uh, okay.
15:40And the third part, always improving, always learning.
15:44Uh there is this elephant in the room context with agents.
15:48A lot of questions which are valid are like how do you navigate the context blo because you work on these long missions in the software factory and how do you actually
15:57keep the context clean and that's very valid because enterprises especially they use hundreds tools they use Figma, notion, Gmail, drive, slack like hundreds tools on average.
16:09So all these have specification in the code.
16:12all these have schema and parameters and log descriptions.
16:15So, uh it can cause agent to really bloat with that and pick wrong tools if there are two tools that sound similar or to actually lose context because they fill up the the context window and need to compress.
16:28So, this is really dangerous.
16:31Uh for this we have something within the software factory called uh deferred context engine and we build it such that we just progressively disclose what's in the context and what tools to use.
16:43So we basically have a surprise tools uh that help later only if they are actually needed.
16:50So it first just has short list of the tools and uh only a short descriptions and when it's needed actually in the code uh they can call the tool and fully load it.
17:00So important thing is nothing is actually removed it's just hidden and not reachable until needed.
17:06So it's just for later and important thing is it actually saves a lot of tokens uh at scale once you use more and more tools.
17:15uh the more actually you save it really scales and you can save 50% of tokens or more.
17:22Okay.
17:22Uh another thing which is tricky and was surprising for me for the first time is that when adopting AI you either succeed big or you can fail big the same way it's a bit of power law.
17:34So if your codebase is not ready, if you don't have structured codebase and all important things, then adopting or turning into software factory can actually make you end up worse and make your code degrading.
17:48And uh there is a big and growing gap in productivity between those who just adopted AI versus those who actually thought about it a bit more.
17:58Uh there is another data from Stanford about yeah without structured uh code base and being ready and documenting well the AI can make your code worse and I think if you are engineers you probably
18:10can agree that sometimes AI really makes a mess and this can really compound more and more and it's difficult to go back.
18:19uh we have something as a part of the software factory uh we have something called agent readiness and it's a framework I don't like the word framework but uh take it as a hygiene check of your codebase and things you should do because there is actually nice correlation between
18:35how your codebase is looking and all the details there and how good you're going to adopt the AI and end up productive instead of uh with a messy code base.
18:44So there are things like how how reproducible is your developer environment or if you have written good tests if you have all things well documented what's the style of your code all the tests and llinters
18:57uh everything that you would do also to keep codebased screen.
19:01So these factors actually show to be very useful and we have our bigger customers actually go through this agent readiness framework and do all checks and then they can follow up with recommended actions
19:14uh how to fix that.
19:17Uh I think one more thing about the context and the continuous learning in the software factory is the most of the time when working with AI or at least from my experience is that you keep repeating to agents how how do you want the things done and you feel like the agent doesn't really get you
19:35and I think this is again nice parallel with human organizations and teams because when you join a new company there are a lot of rules that are not really codified anywhere like you just learn and observe how things are done and there are a lot of things between the lines
19:51that you can't really learn anywhere else than just observing.
19:55I think this is big challenge for agents as well and one thing we launched for that is plugins which are like packaged reusable skills and context and the things behind the scenes that you can really codify
20:08or auto which is automatically updating our documentations and uh yeah and um reviewing and documenting what you have.
20:19So the question is what will happen to us humans if we build software factory like this?
20:25Uh will humans just lose jobs and go to permanent underclass?
20:30I think they will not.
20:31And uh I want to be positive.
20:34So I'm thinking of this parallel of humans just moving up uh levels up to the uh to the cooler tasks than before.
20:43I think we actually had before the same same experience of us as humans abstracting some levels up and we actually started ourselves as human computers.
20:54We actually were the computers doing all the most detailed stuff at the beginning.
20:59Then we had programming languages codifying some of it, abstracting and then coding agents.
21:04So outsourcing some of it but uh still being in the loop very closely uh and very monitoring everything.
21:10And now we are moving kind of up to software factories where we manage these agents as I said orchestrator agents and workers and validators and all these teams structured teams of agents that work really continuously
21:24in the loop and we just monitor and decide what to build.
21:29Uh so I want to really end this positively.
21:32We should be as humans deciding what to build in the software not how to build it.
21:38uh because that's up for the agents and by this uh continuous cycle of autonomous software I think we can actually achieve that.
21:48Uh this is one warning chart or if someone thinks AI will take the cool cool stuff from us.
21:55I believe actually it will take the annoying stuff and in enterprises and organizations you already spend a lot on alignments on the meetings as I mentioned basically the thing behind
22:06between the lines that you really need you need to get context from everyone share your status sync on the things on meetings so all these things could be actually outsourced to your software factory and you could be just talking about the cool stuff
22:20okay so this is it thank you so much go touch some grass and let your agents uh built for you.
22:26Thank you.