WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=vGCJ7diEtrw

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/vGCJ7diEtrw.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:36.880 --> 00:00:38.559
Uh hi everyone.

s2
00:00:38.559 --> 00:00:39.200
Hey what's up?

s3
00:00:39.200 --> 00:00:40.640
I'm Teresa.

s4
00:00:40.640 --> 00:00:44.399
Uh and today I will be talking about software factory.

s5
00:00:44.399 --> 00:00:54.800
uh everyone is talking about software factory but only few people are actually building one and even fewer people know what it actually takes to build one and how to define it.

s6
00:00:54.800 --> 00:01:08.640
So I will be talking about what is a software factory, should I build your own or outsource it, what works in production already, what are the main challenges and how much is this all going to cost?

s7
00:01:08.640 --> 00:01:09.920
Uh my name is Theresa.

s8
00:01:09.920 --> 00:01:23.600
I work at a company called factory.com and we've been building uh this concept of the software factory for quite a long time but finally uh the technology is catching up and it's possible to build this in production

s9
00:01:23.600 --> 00:01:36.960
for enterprises like EY or Adobe uh I would define the software factory as the whole loop the whole life cycle of developing software with autonomy which doesn't mean just coding and generating code

s10
00:01:36.960 --> 00:01:54.000
by that I mean collecting all the signals reacting to user feedback to logs, prioritizing what's important, then orchestrating it all, executing, validating, doing really good uh testing in production and then iterating on all this uh while also continuously

s11
00:01:54.000 --> 00:01:58.640
improving in the process and gaining new knowledge and new skills.

s12
00:01:58.640 --> 00:02:06.960
So this is just our dashboard how our software factory looks if you if you look at it and it's catching the whole cycle there.

s13
00:02:06.960 --> 00:02:12.400
uh this is just some history and by history I mean 2023 in AI.

s14
00:02:12.400 --> 00:02:24.480
So the software factory wasn't really possible before even though we always had this idea from the beginning of uh CH GPT lounge and others we had this idea of the continuous loop there was autog

s15
00:02:24.480 --> 00:02:37.599
baby agi already with the concepts of iterating on the software but it didn't work before because the LMS were hallucinating there was problem with context length uh problem with the reasoning quality

s16
00:02:37.599 --> 00:02:44.640
uh different problems like this or we were missing good environments where the agents can actually work in the isolation.

s17
00:02:44.640 --> 00:02:54.959
So all this kind of led to why the software factory is the paradigm that's starting to be popular now and actually useful.

s18
00:02:55.200 --> 00:02:58.160
I like to define things by what they are not.

s19
00:02:58.160 --> 00:03:10.000
So I want to say software factory is not just coding agent and it's not even a swarm of coding agents even thousands of agents because generating code writing code that's the easy part

s20
00:03:10.000 --> 00:03:11.440
compared to all the others.

s21
00:03:11.440 --> 00:03:15.519
uh engineers don't even spend most of the time uh just writing code.

s22
00:03:15.519 --> 00:03:17.440
The challenges are the rest of it.

s23
00:03:17.440 --> 00:03:28.879
Uh so this is what I want to enhance and it's also not just some consultancy or just some abstract strategy that someone will try to sell you and to your organization.

s24
00:03:28.879 --> 00:03:40.080
Uh I think the approaches are of course different but we believe you should really rebuild your organization from ground up to be ready to become a software factory like you can't

s25
00:03:40.080 --> 00:03:44.879
invite a consultancy and just throw something in the middle of your organization.

s26
00:03:44.879 --> 00:03:49.599
You should really be mindful and rebuild it from scratch.

s27
00:03:50.080 --> 00:04:01.360
uh this is the approach I like uh showing to building the software factory and I think it's similar to building the real team of humans because there is a lot of agents

s28
00:04:01.360 --> 00:04:13.040
as I said they will be testing validating iterating and it all can end up as a big chaos uh so you need to be mindful and I would say the three important things is being agnostic

s29
00:04:13.040 --> 00:04:22.240
in the software factory so being independent of LLM choices uh of how you already work as the organization then you need to be autonomous.

s30
00:04:22.240 --> 00:04:35.600
So you need to give the agents the right trusts and all this permissions and governance and you need to trust it to run really for a long time because the predictions are that agents will run even a year or more years without the human in the loop.

s31
00:04:35.600 --> 00:04:43.280
Of course that sounds ambitious but for example our missions already our long runninging sessions already build uh things and run for weeks.

s32
00:04:43.280 --> 00:04:45.040
So it's not that crazy.

s33
00:04:45.040 --> 00:04:47.199
And third thing always improving.

s34
00:04:47.199 --> 00:05:04.160
So same as human organization you need to onboard any new people in your team by giving them good uh code base understanding good structure documentation and also let them improve in the process and gain new knowledge and share it share it within the team.

s35
00:05:04.160 --> 00:05:09.520
So these are the three points and now let's talk about the agnostic one.

s36
00:05:09.520 --> 00:05:19.280
Uh I think the important thing to point out if you are a builder and building different primitives of the software factory you need to be ready for how everyone is already working

s37
00:05:19.280 --> 00:05:31.199
you need to work across environments like Slack, GitHub and connected to everything and you need to also allow people to already bring subscriptions they are using because there is every

s38
00:05:31.199 --> 00:05:40.080
every time there is like new technology, new frontier models, uh new benchmarks and it's so difficult to catch up but also to predict what we'll be winning.

s39
00:05:40.080 --> 00:05:44.000
So it's best to be agnostic for everyone.

s40
00:05:44.000 --> 00:05:47.680
Uh then there's the topic of being agnostic to models.

s41
00:05:47.680 --> 00:05:58.639
Uh this is a topic of a lot of discussions recently and this chart is from this chart is from CEO of Coinbase uh who tweeted about how they started

s42
00:05:58.639 --> 00:06:04.960
uh started saving money in their organ or organization on AI but without reducing the token spend.

s43
00:06:04.960 --> 00:06:07.919
So the black line is how much they spent on tokens.

s44
00:06:07.919 --> 00:06:13.199
They continue growing and token maxing but they stopped spending so much money.

s45
00:06:13.199 --> 00:06:18.720
Uh how they did it uh different tricks like different default default models for people.

s46
00:06:18.720 --> 00:06:22.479
So stop pushing people to use only the frontier as a default.

s47
00:06:22.479 --> 00:06:27.840
Then caching of course uh to stop preferring the stuff every time.

s48
00:06:27.840 --> 00:06:32.560
And then no limits on spending but just needing to see the results if you spend a lot.

s49
00:06:32.560 --> 00:06:36.639
And the last thing that's really important topic is routing.

s50
00:06:36.639 --> 00:06:46.479
So smart routing between LLMs and just a token optimizing between uh the LMS instead of just spending so much tokens.

s51
00:06:46.479 --> 00:06:59.360
Uh we built uh a thing in factory called uh automatic model routing and there's been a lot of questions around it but what it does is it automatically routes between different models

s52
00:06:59.360 --> 00:07:00.800
in your process.

s53
00:07:00.800 --> 00:07:12.960
The important thing is it decides first what model is most optimal and then uh it can switch to different model if it's failing the task or in case of any troubles but it usually doesn't happen

s54
00:07:12.960 --> 00:07:18.080
and important thing is it doesn't just help uh spend money.

s55
00:07:18.080 --> 00:07:29.039
It also helps with reliability or with speed because open source models are often faster and uh if one LM provider fails you can just switch to another one automatically.

s56
00:07:29.039 --> 00:07:33.120
So it really does more things than just saving the money.

s57
00:07:33.120 --> 00:07:36.240
This uh is our benchmark which is very conservative.

s58
00:07:36.240 --> 00:07:40.479
I prefer conservative benchmarks but I think the direction is clear.

s59
00:07:40.479 --> 00:07:46.479
You can save for example 25% but even more probably.

s60
00:07:46.479 --> 00:07:49.919
Uh how the routing works uh there are four parts.

s61
00:07:49.919 --> 00:08:00.400
First you just assign the task and you don't really need to do anything but as an organization you can for example give different permissions and different default models to different people

s62
00:08:00.400 --> 00:08:09.759
which is very useful like marketing or sales or engineers can both uh have different default models and then you have the classification which is the important part.

s63
00:08:09.759 --> 00:08:11.199
This is the magic of the routing.

s64
00:08:11.199 --> 00:08:22.240
You need to really uh look at the structure of the prompt of the uh code base of how difficult the task is, what tools are being used, all these factors and make a classification

s65
00:08:22.240 --> 00:08:24.639
of difficulty of the task.

s66
00:08:24.639 --> 00:08:34.399
Uh after that you basically do a threshold of what is enough to accomplish the task and you choose the cheapest model above the threshold.

s67
00:08:34.399 --> 00:08:40.959
So cheapest model that is predicted to accomplish your task and then you go.

s68
00:08:41.039 --> 00:08:41.440
Okay.

s69
00:08:41.440 --> 00:08:45.920
So when we launched this we got a lot of questions like does it actually work?

s70
00:08:45.920 --> 00:08:48.560
What if uh the model mis routes?

s71
00:08:48.560 --> 00:08:49.760
Uh is it slower?

s72
00:08:49.760 --> 00:08:51.120
Is it more expensive actually?

s73
00:08:51.120 --> 00:08:55.519
What if the model can't accomplish the task or what if it needs to upgrade?

s74
00:08:55.519 --> 00:08:57.600
And how do you handle caching?

s75
00:08:57.600 --> 00:09:15.360
So I think these are all valid questions and that's why it's so difficult to build a good router like you need to basically classify it very well and the challenge is to not not to need to switch too often but even if you're switching in the middle of the task to difficult to more difficult model

s76
00:09:15.360 --> 00:09:29.279
you still overall are faster probably because it's still worth uh this is just some overview of caching uh lot of questions we got on this was also So how do we handle caching?

s77
00:09:29.279 --> 00:09:31.200
Do we do discounts for users?

s78
00:09:31.200 --> 00:09:39.680
Uh because LLM LLM labs of course save a lot of money by caching and skipping all the context prefill all the time.

s79
00:09:39.680 --> 00:09:41.760
Uh open models can do this as well.

s80
00:09:41.760 --> 00:09:51.760
I think people sometimes forget this and you can just host open models as well uh on dedicated compute and you can take the same advantage of the caching.

s81
00:09:51.760 --> 00:09:55.760
Uh so I want to enhance uh the final price for users.

s82
00:09:55.760 --> 00:09:57.440
is just a pricing decision.

s83
00:09:57.440 --> 00:10:00.800
It's not a technical challenge because everyone can do caching.

s84
00:10:00.800 --> 00:10:07.680
It's just what price you pass on the users and what deals you make with the API providers.

s85
00:10:07.920 --> 00:10:15.040
So this is uh being agnostic and now the autonomy which is the core of the software factory.

s86
00:10:15.040 --> 00:10:17.839
Uh everyone is talking about loops.

s87
00:10:17.839 --> 00:10:23.200
Uh loops I think have been here the whole time already in different context.

s88
00:10:23.200 --> 00:10:27.440
They are just now leveling up and moving to the context of agents.

s89
00:10:27.440 --> 00:10:34.000
You probably heard about the Ralph loop and specifying the tasks and splitting to subtasks for agents.

s90
00:10:34.000 --> 00:10:43.920
Uh this is a known concept and uh the question is not the loop itself but the question is how how you define what it means to be done in the loop.

s91
00:10:43.920 --> 00:10:52.320
This is one example from my colleague uh he he made a loop uh of agents building 3D printing of our logo.

s92
00:10:52.320 --> 00:11:03.360
uh I think here you can see what is the challenge of the loops because uh before in programming loops had clear criteria of what it means to be done while now the criteria become open-ended

s93
00:11:03.360 --> 00:11:14.079
because a lot of the tasks are very nondeterministic it's basically open world you can do even real stuff like printing something and it's really difficult to define how do you verify

s94
00:11:14.079 --> 00:11:23.920
that the loop was done that the task was accomplished so that's the difficult part uh yeah they just need to be verifiable This I call the scary chart.

s95
00:11:23.920 --> 00:11:30.880
It basically shows that the tasks and how long they run autonomously with agents have been increasing.

s96
00:11:30.880 --> 00:11:32.560
But still it's not that reliable.

s97
00:11:32.560 --> 00:11:43.200
Even if you can run for a very long time, it doesn't mean that it will be reliable in production and you still probably need to need to iterate on it and uh provide feedback.

s98
00:11:43.200 --> 00:11:46.399
So it's still not solved and there are problems like cheating.

s99
00:11:46.399 --> 00:11:58.399
For example, if you write the what it means to be done in the wrong way, the agent can try to pass your tests but not really like not really verify what you need to do and accomplish but instead

s100
00:11:58.399 --> 00:12:00.720
try to solve just passing your test.

s101
00:12:00.720 --> 00:12:09.120
So cheating by that uh we have something called factory missions and this is like long longunning sessions of the agents.

s102
00:12:09.120 --> 00:12:19.279
We just call it missions because we send the agent to the mission and uh the missions work in the loop and iterating on task until it's done and they can do it even for weeks

s103
00:12:19.279 --> 00:12:28.079
and uh the main agent is orchestrator and then it's uh assigns work to workers agents and then validators who review the task.

s104
00:12:28.079 --> 00:12:38.160
So this is one example uh real mission from our customers that run 16 hours and just just to see how important is the validation as well.

s105
00:12:38.160 --> 00:12:41.519
it takes even 40% of the whole process.

s106
00:12:41.519 --> 00:12:47.120
Uh to summarize, the orchestrator just decides and writes the conditions what it needs to be done.

s107
00:12:47.120 --> 00:12:50.079
Then it gives it to the agents called workers.

s108
00:12:50.079 --> 00:12:51.600
The agents work on it.

s109
00:12:51.600 --> 00:12:55.360
And what is interesting or worth noting is that they work in a sequence.

s110
00:12:55.360 --> 00:12:58.240
So they don't work in a swarm or parallel.

s111
00:12:58.240 --> 00:13:03.920
The agents work in sequence and everyone accomplishes something and passes it to the next one.

s112
00:13:03.920 --> 00:13:09.600
uh we actually found that if you do this you end up with more fresh context and kind of fresh head.

s113
00:13:09.600 --> 00:13:14.079
Same with when with humans have like other colleagues verifying their code.

s114
00:13:14.079 --> 00:13:18.320
So similar thing we just let the agents pass to the next one.

s115
00:13:18.320 --> 00:13:23.200
But still every worker of the sequence can have parallel agents doing smaller tasks.

s116
00:13:23.200 --> 00:13:27.680
So I don't know researching on the web or building files during that.

s117
00:13:27.680 --> 00:13:30.959
So they are in sequence and everyone has sub agents as well.

s118
00:13:30.959 --> 00:13:38.480
and validators they review the output and provide feedback and send it back to the beginning.

s119
00:13:38.720 --> 00:13:44.320
Uh important thing is uh the validators judge code that they didn't write.

s120
00:13:44.320 --> 00:13:51.279
Uh there's something from the orchestrator agent called validation contract uh that's written before any code is done.

s121
00:13:51.279 --> 00:13:53.600
And first type is scrutiny validator.

s122
00:13:53.600 --> 00:13:55.839
Second is user testing validator.

s123
00:13:55.839 --> 00:14:00.480
The scrutiny is the one who really verifies how the codebase looks.

s124
00:14:00.480 --> 00:14:16.240
the liners types tests it's really rigorous check of the code but the second I think is really interesting is user testing validator and that's the one who is really in the arena trying the things so it doesn't care how it was made it just goes

s125
00:14:16.240 --> 00:14:28.079
it works in its virtual computer and it clicks on the stuff and really checks if everything works I've seen one engineer migrating uh code bases with droid with our agent

s126
00:14:28.079 --> 00:14:39.440
and uh it really needed the agent in the end to click through the stuff because with other products it just created created the product but it didn't work and was wasn't interactive.

s127
00:14:39.440 --> 00:14:41.680
It was just a dummy result.

s128
00:14:41.680 --> 00:14:49.760
So our agent that really clicked on the stuff uh allowed to check that it's actually working and not just looking good in the code.

s129
00:14:49.760 --> 00:15:04.480
I think this is one cool thing that was also allowed by the progress in computer use and the persistent environments virtual machines for agents which also wasn't here before or wasn't that great but now it's really great so it's allowing

s130
00:15:04.480 --> 00:15:07.440
to go in the arena as an agent.

s131
00:15:07.440 --> 00:15:10.959
Uh this is just one example of real mission from our users.

s132
00:15:10.959 --> 00:15:18.639
I I like to just learn what users are building and get feedback and this is just uh how it can look in your dashboard.

s133
00:15:18.639 --> 00:15:21.760
And this is how I visualize the missions.

s134
00:15:21.760 --> 00:15:32.240
What is interesting is that it's just a loop with smaller loops because every of the agents are running in loops as well, but overall it's like one big loop.

s135
00:15:32.240 --> 00:15:39.040
Okay, maybe this is a bit weird, but I just like that it's all loops with smaller loops.

s136
00:15:39.040 --> 00:15:40.079
Uh, okay.

s137
00:15:40.079 --> 00:15:44.720
And the third part, always improving, always learning.

s138
00:15:44.720 --> 00:15:48.480
Uh there is this elephant in the room context with agents.

s139
00:15:48.480 --> 00:15:57.759
A lot of questions which are valid are like how do you navigate the context blo because you work on these long missions in the software factory and how do you actually

s140
00:15:57.759 --> 00:16:09.680
keep the context clean and that's very valid because enterprises especially they use hundreds tools they use Figma, notion, Gmail, drive, slack like hundreds tools on average.

s141
00:16:09.680 --> 00:16:12.240
So all these have specification in the code.

s142
00:16:12.240 --> 00:16:15.920
all these have schema and parameters and log descriptions.

s143
00:16:15.920 --> 00:16:28.639
So, uh it can cause agent to really bloat with that and pick wrong tools if there are two tools that sound similar or to actually lose context because they fill up the the context window and need to compress.

s144
00:16:28.639 --> 00:16:31.040
So, this is really dangerous.

s145
00:16:31.040 --> 00:16:43.519
Uh for this we have something within the software factory called uh deferred context engine and we build it such that we just progressively disclose what's in the context and what tools to use.

s146
00:16:43.519 --> 00:16:50.240
So we basically have a surprise tools uh that help later only if they are actually needed.

s147
00:16:50.240 --> 00:17:00.720
So it first just has short list of the tools and uh only a short descriptions and when it's needed actually in the code uh they can call the tool and fully load it.

s148
00:17:00.720 --> 00:17:06.799
So important thing is nothing is actually removed it's just hidden and not reachable until needed.

s149
00:17:06.799 --> 00:17:15.039
So it's just for later and important thing is it actually saves a lot of tokens uh at scale once you use more and more tools.

s150
00:17:15.039 --> 00:17:21.760
uh the more actually you save it really scales and you can save 50% of tokens or more.

s151
00:17:22.000 --> 00:17:22.480
Okay.

s152
00:17:22.480 --> 00:17:34.559
Uh another thing which is tricky and was surprising for me for the first time is that when adopting AI you either succeed big or you can fail big the same way it's a bit of power law.

s153
00:17:34.559 --> 00:17:48.160
So if your codebase is not ready, if you don't have structured codebase and all important things, then adopting or turning into software factory can actually make you end up worse and make your code degrading.

s154
00:17:48.160 --> 00:17:58.320
And uh there is a big and growing gap in productivity between those who just adopted AI versus those who actually thought about it a bit more.

s155
00:17:58.320 --> 00:18:10.160
Uh there is another data from Stanford about yeah without structured uh code base and being ready and documenting well the AI can make your code worse and I think if you are engineers you probably

s156
00:18:10.160 --> 00:18:18.880
can agree that sometimes AI really makes a mess and this can really compound more and more and it's difficult to go back.

s157
00:18:19.039 --> 00:18:35.679
uh we have something as a part of the software factory uh we have something called agent readiness and it's a framework I don't like the word framework but uh take it as a hygiene check of your codebase and things you should do because there is actually nice correlation between

s158
00:18:35.679 --> 00:18:44.799
how your codebase is looking and all the details there and how good you're going to adopt the AI and end up productive instead of uh with a messy code base.

s159
00:18:44.799 --> 00:18:57.840
So there are things like how how reproducible is your developer environment or if you have written good tests if you have all things well documented what's the style of your code all the tests and llinters

s160
00:18:57.840 --> 00:19:01.760
uh everything that you would do also to keep codebased screen.

s161
00:19:01.760 --> 00:19:14.960
So these factors actually show to be very useful and we have our bigger customers actually go through this agent readiness framework and do all checks and then they can follow up with recommended actions

s162
00:19:14.960 --> 00:19:17.280
uh how to fix that.

s163
00:19:17.280 --> 00:19:35.760
Uh I think one more thing about the context and the continuous learning in the software factory is the most of the time when working with AI or at least from my experience is that you keep repeating to agents how how do you want the things done and you feel like the agent doesn't really get you

s164
00:19:35.760 --> 00:19:51.520
and I think this is again nice parallel with human organizations and teams because when you join a new company there are a lot of rules that are not really codified anywhere like you just learn and observe how things are done and there are a lot of things between the lines

s165
00:19:51.520 --> 00:19:55.840
that you can't really learn anywhere else than just observing.

s166
00:19:55.840 --> 00:20:08.320
I think this is big challenge for agents as well and one thing we launched for that is plugins which are like packaged reusable skills and context and the things behind the scenes that you can really codify

s167
00:20:08.320 --> 00:20:19.600
or auto which is automatically updating our documentations and uh yeah and um reviewing and documenting what you have.

s168
00:20:19.679 --> 00:20:25.039
So the question is what will happen to us humans if we build software factory like this?

s169
00:20:25.039 --> 00:20:30.159
Uh will humans just lose jobs and go to permanent underclass?

s170
00:20:30.159 --> 00:20:31.679
I think they will not.

s171
00:20:31.679 --> 00:20:34.000
And uh I want to be positive.

s172
00:20:34.000 --> 00:20:43.120
So I'm thinking of this parallel of humans just moving up uh levels up to the uh to the cooler tasks than before.

s173
00:20:43.120 --> 00:20:54.240
I think we actually had before the same same experience of us as humans abstracting some levels up and we actually started ourselves as human computers.

s174
00:20:54.240 --> 00:20:59.200
We actually were the computers doing all the most detailed stuff at the beginning.

s175
00:20:59.200 --> 00:21:04.320
Then we had programming languages codifying some of it, abstracting and then coding agents.

s176
00:21:04.320 --> 00:21:10.640
So outsourcing some of it but uh still being in the loop very closely uh and very monitoring everything.

s177
00:21:10.640 --> 00:21:24.799
And now we are moving kind of up to software factories where we manage these agents as I said orchestrator agents and workers and validators and all these teams structured teams of agents that work really continuously

s178
00:21:24.799 --> 00:21:29.679
in the loop and we just monitor and decide what to build.

s179
00:21:29.679 --> 00:21:32.400
Uh so I want to really end this positively.

s180
00:21:32.400 --> 00:21:38.400
We should be as humans deciding what to build in the software not how to build it.

s181
00:21:38.400 --> 00:21:48.240
uh because that's up for the agents and by this uh continuous cycle of autonomous software I think we can actually achieve that.

s182
00:21:48.240 --> 00:21:55.200
Uh this is one warning chart or if someone thinks AI will take the cool cool stuff from us.

s183
00:21:55.200 --> 00:22:06.400
I believe actually it will take the annoying stuff and in enterprises and organizations you already spend a lot on alignments on the meetings as I mentioned basically the thing behind

s184
00:22:06.400 --> 00:22:20.880
between the lines that you really need you need to get context from everyone share your status sync on the things on meetings so all these things could be actually outsourced to your software factory and you could be just talking about the cool stuff

s185
00:22:20.880 --> 00:22:26.720
okay so this is it thank you so much go touch some grass and let your agents uh built for you.

s186
00:22:26.720 --> 00:22:28.960
Thank you.