WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=7vn4WpqNpck

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/7vn4WpqNpck.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.320 --> 00:00:15.880
So, it's not just my AI pipeline that's on fire, but also my PowerPoint.

s3
00:00:15.880 --> 00:00:17.680
So, it's 2025.

s4
00:00:17.680 --> 00:00:20.800
Uh we're scaling as a business and things are going poorly.

s5
00:00:20.800 --> 00:00:27.600
We're adding too many customers, we're not getting the throughput we need, and we need to improve our underlying technology.

s6
00:00:27.600 --> 00:00:30.200
And there's three main issues that we're facing.

s7
00:00:30.200 --> 00:00:33.040
The first one is that we're too slow to meet customer demand.

s8
00:00:33.040 --> 00:00:37.080
The second one is that this AI pipeline that we've built is too complicated to update.

s9
00:00:37.080 --> 00:00:42.880
And the third one is because it's a legacy code base, or actually more than 10 repos, nobody actually wants to touch the code.

s10
00:00:42.880 --> 00:00:45.400
It's not a fun experience.

s11
00:00:45.600 --> 00:00:49.360
So, we made this decision to refactor over the course of 6 months.

s12
00:00:49.360 --> 00:00:54.760
And the real question for this talk today was, "Is this the right move to do?"

s13
00:00:54.760 --> 00:00:57.720
So, I'll spend this time answering this question, but let's start off with the use case.

s14
00:00:57.720 --> 00:01:04.199
So, the company I work at, WiseDocs, processes complex medical claims, which are PDFs that are more than 10,000 pages in size.

s15
00:01:04.199 --> 00:01:06.960
Some of these files are bigger than video files.

s16
00:01:06.960 --> 00:01:12.840
So, it's a pretty complex application, and because of this, it's actually non-trivial to scale the different parts.

s17
00:01:12.840 --> 00:01:17.160
So, we're going to talk about the pipeline today, which has a number of ML models.

s18
00:01:17.160 --> 00:01:18.960
So, I'll divide this talk into a number of chapters.

s19
00:01:18.960 --> 00:01:23.360
We'll start off with the first one, which is the concept of tech debt.

s20
00:01:23.400 --> 00:01:28.640
So, I think we all have this feeling universally, if we've been developers for a while, that we all write bad code.

s21
00:01:28.640 --> 00:01:31.440
The question is, do we do this intentionally or not?

s22
00:01:31.440 --> 00:01:35.560
If I look back to some of the earliest code I used to write, uh it was bad.

s23
00:01:35.560 --> 00:01:36.960
This was more than 15 years ago.

s24
00:01:36.960 --> 00:01:46.960
I tried to print an image of this character from a video game, and I didn't understand that you can't uh system.out.println in Java to render something on the screen.

s25
00:01:46.960 --> 00:01:55.320
So, hopefully I've come uh further from that point in time, but there's these moments where we all know that we've written bad code before.

s26
00:01:55.440 --> 00:02:01.680
Now, if we think about technical debt as financial debt, it compounds in mysterious and sometimes unexpected ways.

s27
00:02:01.680 --> 00:02:04.760
But you should think about it in a rigorous format as well.

s28
00:02:04.760 --> 00:02:12.880
For us to achieve some kind of ROI by taking on technical debt, such as building a feature or getting new customers, we want to make sure that the ROI makes sense.

s29
00:02:12.880 --> 00:02:19.880
If we introduce additional complexity into our code base, we can very quickly outrun the ROI we've generated.

s30
00:02:21.400 --> 00:02:26.880
Now with AI engineering, you've probably seen a number of different stories that have come out to showcase the progress that's been made.

s31
00:02:26.880 --> 00:02:37.560
These are two case studies from Anthropic, one from Spotify and the other from Stripe, talking about the immense progress that they've made both in shipping velocity and also the ability to refactor code.

s32
00:02:37.560 --> 00:02:43.400
So at this point in time, writing code or making changes is something that teams are doing faster and faster.

s33
00:02:43.400 --> 00:02:45.880
Now I'll pause here.

s34
00:02:46.160 --> 00:02:49.560
Who here thinks that products have gotten better in the past 20 years?

s35
00:02:49.560 --> 00:02:50.720
Technical products.

s36
00:02:50.720 --> 00:02:52.840
Also raise your hand.

s37
00:02:52.840 --> 00:02:54.000
I hope everybody, right?

s38
00:02:54.000 --> 00:02:55.640
Phones are pretty cool.

s39
00:02:55.640 --> 00:02:58.360
How about 5 years?

s40
00:02:58.360 --> 00:03:00.200
How about the past year?

s41
00:03:00.200 --> 00:03:01.160
Okay.

s42
00:03:01.160 --> 00:03:06.880
So the challenge is that we're going faster and faster through the technology life cycle, but we've lost something.

s43
00:03:06.880 --> 00:03:13.080
The the product focused on customers in some way has degraded, the maintainability of the code and the reliability has degraded.

s44
00:03:13.080 --> 00:03:16.280
You can see some of the uptimes here from two leading companies.

s45
00:03:16.280 --> 00:03:24.280
I blurred out their names for for it it doesn't actually matter who they are, but we are below a three-nine or even four-nine reliability.

s46
00:03:24.280 --> 00:03:30.640
So even though we're shipping faster and faster, the code quality and the product quality has not necessarily gone up.

s47
00:03:31.200 --> 00:03:32.920
So let's talk about the refactor that we did.

s48
00:03:32.920 --> 00:03:38.360
So we started this refactor with actual code implementation in April and did some pre-work earlier.

s49
00:03:38.360 --> 00:03:46.880
So I'll go through five different tasks that we did and share some of the the findings that we had before and after, especially with as new models have come out.

s50
00:03:46.880 --> 00:03:50.760
So we spent around 2 months evaluating orchestrators for our AI pipeline.

s51
00:03:50.760 --> 00:03:56.160
We looked at five open-source projects and we wanted to benchmark and see how effective they were for our use case.

s52
00:03:56.160 --> 00:04:04.960
And we started this off before deep research came out as part of Google and OpenAI, so that web search capability to do a comprehensive analysis was still not there.

s53
00:04:04.960 --> 00:04:12.440
Now, after we actually gathered these requirements, we built out proof of concepts with a team of three to make sure that we actually got the right results.

s54
00:04:12.880 --> 00:04:17.320
Now, I'm pretty confident we could do this 90% faster now with the tooling that we have.

s55
00:04:17.320 --> 00:04:26.240
Before we would manually go through, use a little bit of AI, but put everything into a Confluence doc, and we'd evaluate across 17 different criteria that we came up with.

s56
00:04:26.240 --> 00:04:35.440
Nowadays, we could build a much more agentic workflow to do that, starting off with deep research, uh making sure that we match that against the problem statements that we have,

s57
00:04:35.440 --> 00:04:41.480
creating sub-agents for each of these criteria and uh products, and then finally building POCs and evaluating.

s58
00:04:41.480 --> 00:04:46.320
So, things have changed in the past year and a half where we could actually go much, much faster.

s59
00:04:46.320 --> 00:04:56.200
But, we still have to maintain that same set of quality because it's very easy to undergo AI psychosis, where you look at a deep research report that's 20 pages long and you say, "Wow, this looks good."

s60
00:04:56.200 --> 00:05:01.120
And then those features don't actually exist in the product, and you've set yourself back.

s61
00:05:01.680 --> 00:05:07.320
Now, after we've done the initial uh orchestration research and model serving research, we wanted to actually commit in code.

s62
00:05:07.320 --> 00:05:11.160
This is just a example of what happened when we were experimenting.

s63
00:05:11.160 --> 00:05:20.240
I was doing some initial research with Temporal and committed some activities and workflow code to make sure that we can actually replicate what we have in the legacy code base.

s64
00:05:20.720 --> 00:05:28.120
So then, I did what we wanted to do over a number of iterations and at the time gave it to O3 to actually try to implement this code.

s65
00:05:28.120 --> 00:05:30.080
And it did it much faster than I would be.

s66
00:05:30.080 --> 00:05:36.919
This refactor took uh 3 hours of back and forth chatting with Incursr, but it made 10 major mistakes.

s67
00:05:36.919 --> 00:05:44.360
So, at the time when we were going through this refactor, agentic coding was getting better and better, but it still hadn't reached the point of where it is now.

s68
00:05:44.360 --> 00:05:51.480
And it was still a very manual process where you had to intervene and actually guide the model and manually edit or delete code.

s69
00:05:52.160 --> 00:05:55.640
Now, I reran these benchmarks on some some modern models.

s70
00:05:55.640 --> 00:05:59.960
So, we have Sonnet 4.6 and Opus 4.8. And things were much faster.

s71
00:05:59.960 --> 00:06:04.080
Sonnet 4.6 with one additional iteration was able to solve the task.

s72
00:06:04.080 --> 00:06:07.200
And with Opus, it was basically able to one-shot this problem.

s73
00:06:07.200 --> 00:06:10.400
So, models are getting significantly better along with harnesses.

s74
00:06:10.400 --> 00:06:15.600
And the interesting part here as well is that the way that the models interacted has changed substantially as well.

s75
00:06:15.600 --> 00:06:20.000
Before with O3, there weren't substantial tool calls on certain categories.

s76
00:06:20.000 --> 00:06:31.680
And then as we moved into Sonnet 4.6 and Opus, we see now that in modern harnesses, we get sub-agents, we get some of those plan calls, we get different shell commands, and we get different verifications.

s77
00:06:31.680 --> 00:06:36.919
And overall, this process, even though the model execution was a little bit more expensive, it was a lot less manual.

s78
00:06:36.919 --> 00:06:39.160
So, we could actually accomplish a lot more.

s79
00:06:39.160 --> 00:06:48.280
So, if I was rebuilding the same task that I had for for this refactor, it would take around 1/5 of the time to accomplish, which is pretty good progress.

s80
00:06:48.800 --> 00:06:54.480
So, I think all of us realize the scenario that models are substantially better now than they were before.

s81
00:06:55.560 --> 00:07:00.120
Now, this is really important because it shapes the way we think about the software development life cycle.

s82
00:07:00.120 --> 00:07:03.240
We think about 2025 and the types of work that we were doing.

s83
00:07:03.240 --> 00:07:04.760
We were making some small changes.

s84
00:07:04.760 --> 00:07:07.680
We would give specific code snippets to models.

s85
00:07:07.680 --> 00:07:12.360
We were just starting to get into this agentic framework of the type of work we can do.

s86
00:07:12.360 --> 00:07:19.560
And now if we provide a well-constructed spectrum model, it could generally execute it at a very very high capability level.

s87
00:07:19.560 --> 00:07:25.160
And we can see this both in anecdotal experiences as well as some of the thought leadership that has been coming out of the big labs.

s88
00:07:25.160 --> 00:07:28.520
The this image is one from Anthropic.

s89
00:07:29.919 --> 00:07:31.080
Let me ask the group a question.

s90
00:07:31.080 --> 00:07:38.680
Who here has kicked off an agent and realized that either the prompt, the plan, or the requirements were incomplete or missing?

s91
00:07:38.680 --> 00:07:39.560
A lot of people, yeah?

s92
00:07:39.560 --> 00:07:40.880
It's it's very frustrating, right?

s93
00:07:40.880 --> 00:07:42.960
You you're like, "Okay, I'm ready to go.

s94
00:07:42.960 --> 00:07:47.000
It's 11:00 p.m. or 5:00 p.m. I'm going to set off an agent and then come back."

s95
00:07:47.000 --> 00:07:50.840
And then you realize there is a there is a critical flaw.

s96
00:07:50.920 --> 00:07:57.240
Now, the reason I bring this up is it's very important to have a good mental model and understanding how accurate models can be in accomplishing tasks.

s97
00:07:57.240 --> 00:08:00.160
Who here has seen this meter graph before?

s98
00:08:00.160 --> 00:08:01.360
I think a decent number of people.

s99
00:08:01.360 --> 00:08:07.880
So, this is pretty common on actually mapping how much time models can complete tasks of certain categories for.

s100
00:08:07.880 --> 00:08:11.960
So, the idea being that as models get better and better, they can do longer-running tasks.

s101
00:08:11.960 --> 00:08:19.080
Now, typically this graph is shared with the 50% accuracy rate, but I think it's much better to actually look at the 80% accuracy rate or higher.

s102
00:08:19.080 --> 00:08:27.640
And you can see there you can still see a similar exponential trend, but we're no longer claiming that models can accomplish tasks that would take a human 18+ hours.

s103
00:08:27.640 --> 00:08:34.880
Now, I actually think it's much better to measure the accuracy at 90% or 99% because this is where the mental model is most efficient.

s104
00:08:34.880 --> 00:08:36.800
You construct a plan, you create a spec,

s105
00:08:36.800 --> 00:08:37.120
[snorts]

s106
00:08:37.120 --> 00:08:40.880
you hand it off to an agent, and you're pretty sure that it'll get things done, right?

s107
00:08:40.880 --> 00:08:48.320
You don't want to be creating a plan or a spec and then have a 50/50 chance of coming back and knowing that you wasted compute and your attention span.

s108
00:08:48.320 --> 00:08:58.640
Now, if you're kicking off a process that is going to take an hour and it has a 50% chance of completing, there's a very high chance you just wasted that hour and you could have been doing something different.

s109
00:08:58.760 --> 00:09:05.040
Now, if you think about uh broader evaluation, so Meter does have some more information about their frontier models.

s110
00:09:05.040 --> 00:09:21.520
So, this is one for for Metis' preview that they did roughly a month ago, and you can see here that generally the success uh rate starts to decline significantly at that 4-hour mark, but even before then at the 15-second mark or even before the 15-minute mark, there are certain tasks that Metis, uh in in all its glory,

s111
00:09:21.520 --> 00:09:25.160
cannot complete uh effectively and consistently.

s112
00:09:25.160 --> 00:09:33.560
So, we're making rapid progress in in the AI model space, but we're still not there where you can just kick off an agent and have something be completed reliably.

s113
00:09:33.560 --> 00:09:42.640
So, again, this is really important for your software engineering teams and for you as an IC to understand what is your mental model and how are you going to contribute to that.

s114
00:09:43.200 --> 00:09:52.520
I think what's really important is I think you've been hearing this throughout this conference is that there are a number of different frameworks and primitives that you need to have implemented in order to have good agentic development.

s115
00:09:52.520 --> 00:09:54.840
And this is no different from from what we found.

s116
00:09:54.840 --> 00:10:07.440
As we were continuing to mature as an organization and going through our refactor, these are the things that made sure that we can implement the solutions effectively and not waste our time just running in loops with models.

s117
00:10:08.280 --> 00:10:09.200
So let's go to chapter three.

s118
00:10:09.200 --> 00:10:13.200
Let's talk about the refactor itself and some of the productivity gains that we saw.

s119
00:10:13.200 --> 00:10:21.240
So the core idea is that we had these 10 repositories, we put them into a mono repo, and we wanted to build additional features on top of it.

s120
00:10:21.960 --> 00:10:23.760
So this is this is the result.

s121
00:10:23.760 --> 00:10:29.920
The previous repos had been around for more than six years, and you can see the progress that was being made it is pretty slow.

s122
00:10:29.920 --> 00:10:32.240
Part of it was because of the tech debt that was taken on.

s123
00:10:32.240 --> 00:10:35.960
Other parts were because we didn't have AI coding tools.

s124
00:10:35.960 --> 00:10:43.920
And you can see that within the first six months of the rebuild when we got to parity that we had before, that steepness of that curve is immense.

s125
00:10:43.920 --> 00:10:46.360
And it didn't slow down after we we kept shipping.

s126
00:10:46.360 --> 00:10:53.120
So after that that dotted line in the middle there, we kept adding new and new features into the repository.

s127
00:10:53.120 --> 00:11:01.120
And we shipped a lot faster both in terms of the amount of code even though that's not a great metric, but also the commit rate that we had among developers.

s128
00:11:01.120 --> 00:11:05.680
And we actually saw that a lot more developers actually joined in into the contributions.

s129
00:11:05.680 --> 00:11:12.920
So this is a log graph on the commits that we had from the repository initially, and then we slowly on-boarded more and more people.

s130
00:11:12.920 --> 00:11:22.600
And we had fewer commits because it's much easier to commit code when you're just refactoring and replicating something, but we still kept up that velocity as we were adding product features towards the end.

s131
00:11:22.600 --> 00:11:30.800
And now almost every developer within the company is committing to this new mono repo even though it might not be their area of expertise, but they might need to make changes to schemas,

s132
00:11:30.800 --> 00:11:34.880
API calls, and other parts of of the stack.

s133
00:11:35.400 --> 00:11:37.520
So, let's go into chapter four.

s134
00:11:37.520 --> 00:11:39.640
Can a modern LLM zero-shot this problem?

s135
00:11:39.640 --> 00:11:44.320
Can I say, "Hey, amazing LLM, go refactor this code base."

s136
00:11:44.320 --> 00:11:54.280
So, I ran this experiment with GPT 5.5 extra high, and I gave it this goal, giving some of the names of the repositories with the underlying models and other components,

s137
00:11:54.280 --> 00:11:59.120
and it completed its goal in in 10 minutes and 22 seconds.

s138
00:11:59.120 --> 00:12:02.320
And it only wrote 2,000 lines of code, which was a little bit fishy.

s139
00:12:02.320 --> 00:12:04.200
So, I I dug deeper.

s140
00:12:04.200 --> 00:12:09.040
Um and it actually just implemented a bunch of scaffolding and didn't implement the models.

s141
00:12:09.040 --> 00:12:13.760
So, you can see here, I did not add a ray serve deployment or bootstrap command yet, right?

s142
00:12:13.760 --> 00:12:20.080
So, we're still not there where models can self-validate and just one-shot these kinds of problems, but we're getting close.

s143
00:12:20.080 --> 00:12:29.680
I think in in 6 months, we'll get to the point that we can complete pretty substantial refactors, as we saw in the in the Stripe example, consistently across the board.

s144
00:12:30.280 --> 00:12:31.320
So, get to the core question.

s145
00:12:31.320 --> 00:12:32.840
Was this refactor worthwhile?

s146
00:12:32.840 --> 00:12:39.800
Should we have waited a year to do this refactor as models and harnesses continue to get better, or did it make sense to do it at the time?

s147
00:12:39.800 --> 00:12:43.440
Now, I'll say the the other side of the argument, right?

s148
00:12:43.440 --> 00:12:45.080
Things are getting substantially better.

s149
00:12:45.080 --> 00:12:46.080
Models are getting better.

s150
00:12:46.080 --> 00:12:47.240
They can call tools better.

s151
00:12:47.240 --> 00:12:55.040
We have a lot more infrastructure like sandboxes and monitoring frameworks in order for us to actually understand what's happening under the hood with these models.

s152
00:12:55.040 --> 00:13:02.960
So, taking on technical debt and refactoring later is getting exponentially easier as the days go by.

s153
00:13:03.280 --> 00:13:13.400
Now, the problem is that a lot of times when when you build a lot of code and you do this kind of development in an AI AI-native world, it starts looking like some of the legacy code we've we've seen in the past.

s154
00:13:13.400 --> 00:13:14.880
There's a lot of code written.

s155
00:13:14.880 --> 00:13:20.080
It's written with low performance or quality, and the broader problem is people don't actually understand what's happening there.

s156
00:13:20.080 --> 00:13:26.400
So, if you have some issues within the code base or you want to adjust based on customer requirements, it's actually much harder to do so.

s157
00:13:26.400 --> 00:13:34.000
So, you do have to make sure that there are appropriate guardrails, whether or not you do a full refactor or only a partial one.

s158
00:13:34.080 --> 00:13:35.640
So, if you ask me, was it worthwhile?

s159
00:13:35.640 --> 00:13:37.320
I'd say yes.

s160
00:13:37.320 --> 00:13:44.440
Uh we had built out the patterns that we had earlier with the number of different repos in in order to match customer requirements and demands.

s161
00:13:44.440 --> 00:13:48.720
Uh it took an amount of time, but we ultimately achieved the goals of the business.

s162
00:13:48.720 --> 00:13:52.120
Then we came back and refactored and we were able to accelerate.

s163
00:13:52.120 --> 00:13:56.120
Uh we were able to actually reduce the amount of time the pipeline took.

s164
00:13:56.120 --> 00:13:57.640
We were able to reduce the costs.

s165
00:13:57.640 --> 00:14:03.000
We could support larger files, and now we can ship features that would take multiple months in under a week.

s166
00:14:03.000 --> 00:14:09.440
So, the monorepo refactor, the cleanup was worthwhile, and we have some of the productivity metrics we saw there.

s167
00:14:09.440 --> 00:14:13.840
The other part is that beyond just shipping velocity, developers actually want to work in this codebase.

s168
00:14:13.840 --> 00:14:17.120
So, everybody comes along and says, "Hey, can I work in this codebase?

s169
00:14:17.120 --> 00:14:19.480
It's much cleaner compared to the other ones.

s170
00:14:19.480 --> 00:14:22.040
Uh can we actually contribute in a way that makes sense?"

s171
00:14:22.040 --> 00:14:27.760
And a lot of the patterns we have adopted here have spread to other repos within the company.

s172
00:14:28.560 --> 00:14:33.560
Now, whether or not you refactor, uh the AI delivery system is is a layered approach.

s173
00:14:33.560 --> 00:14:40.280
Uh you can isolate different parts of your codebase to avoid a full refactor, but there's so many components that you need to keep in mind.

s174
00:14:40.280 --> 00:14:43.240
And hopefully throughout this conference, you've heard more details about this.

s175
00:14:43.240 --> 00:14:51.839
But I I really encourage everybody to think about uh the the business value of delivering a a big refactor and the tradeoffs of doing it now versus in the future.

s176
00:14:51.839 --> 00:14:58.839
So, models will continue to get better, uh but sometimes it's good to pause, build a monorepo, and forge ahead.

s177
00:14:58.839 --> 00:14:59.720
So, thank you everybody.

s178
00:14:59.720 --> 00:15:01.945
Uh happy to take any questions.

s179
00:15:01.945 --> 00:15:03.945
[applause]

s180
00:15:15.800 --> 00:15:20.240
Yeah, so the question was before we had multiple repos and did we move into monorepo?

s181
00:15:20.240 --> 00:15:21.800
Yes, uh we did that.

s182
00:15:21.800 --> 00:15:25.560
One of the things we found now was that models are much better at navigating multiple repos.

s183
00:15:25.560 --> 00:15:30.720
So, if you put it into a higher-level folder, right, they could navigate the the file directory.

s184
00:15:30.720 --> 00:15:36.520
But, for doing that end-to-end testing and verification and deployment, it's still much harder to do with multiple repos.

s185
00:15:36.520 --> 00:15:44.360
Uh and if you're building a sandbox environment to run sort of a full AI factory, it's it also takes more time to clone repos and get everything set up.

s186
00:15:44.360 --> 00:15:46.200
So.

s187
00:15:46.200 --> 00:15:48.200
Yep.

s188
00:15:59.120 --> 00:16:07.000
Yeah, so the question was uh when we define certain features and requirements that we go back and check them and make changes uh as well as sort of the guardrails framework.

s189
00:16:07.000 --> 00:16:07.560
Uh we did.

s190
00:16:07.560 --> 00:16:13.880
I think we got uh 15 out of 17 requirements right when we were going ahead with the refactor.

s191
00:16:13.880 --> 00:16:20.600
Um and some of the processes that we added for the actual AI engineering verification, that evolved over time.

s192
00:16:20.600 --> 00:16:30.440
So, for example, when we started, plan mode was just barely coming into Cloud Code and didn't exist in Cursor, but we adopted it as part of our development life cycle.

s193
00:16:30.600 --> 00:16:32.600
Yep.

s194
00:16:36.280 --> 00:16:40.760
So, our PR reviews were all all human PR reviews during that refactor.

s195
00:16:40.760 --> 00:16:44.800
Uh we did some local checks where we ran skills to say, "Hey, review this code.

s196
00:16:44.800 --> 00:16:45.720
Make sure that it's good."

s197
00:16:45.720 --> 00:16:49.160
And they're continuing to get more autonomous as time goes on.

s198
00:16:49.160 --> 00:16:57.240
But, at that point, PRs were really good way for us to build context uh for that repo as we only had a few developers working on it, and we wanted to make sure people understood

s199
00:16:57.240 --> 00:17:00.080
what had gone into the refactor.

s200
00:17:00.680 --> 00:17:02.680
Yep.

s201
00:17:07.520 --> 00:17:17.480
Um in terms of factors, I think that the complexity of the task you can give to a model uh is going to be different, and many more companies will have more scaffolding

s202
00:17:17.480 --> 00:17:19.319
in terms of actually doing a refactor.

s203
00:17:19.319 --> 00:17:32.800
So, for example, when I showed the the life cycle of doing the research, the POC work, um validating the code quality, checking hidden assumptions, like you you thought an open-source library had this feature, but it was actually in

s204
00:17:32.800 --> 00:17:34.040
uh a beta, for example.

s205
00:17:34.040 --> 00:17:44.400
I think that is going to be much much faster on top of sort of the standard refactoring of hey, here's a file, rewrite it to to match this uh set of requirements.

s206
00:17:47.200 --> 00:17:47.800
All right.

s207
00:17:47.800 --> 00:17:48.080
Great.

s208
00:17:48.080 --> 00:17:48.680
Thank everybody.

s209
00:17:48.680 --> 00:17:51.320
Have a great rest of the conference.

s210
00:18:04.037 --> 00:18:06.037
[music]