I Spent Two Years Building the Median — Hariprasad Sudharshan

AI Engineer · 36 min · 378 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01If I tell you the New York City tonight suddenly stops working, you tell me what could be possible reasons for no flights leaving JFK.
00:11The subway suddenly stops at the next station and stays there.
00:16None of your tap pay is working.
00:19The elevators you get in, they don't know which floor they are.
00:25The phones are not ringing.
00:27The Wall Street and the entire world comes to a standstill.
00:33What could be the reason?
00:37Software.
00:39Unless otherwise, there is a software issue that affects the entire globe.
00:45There is nothing that can stop this modern world.
00:49This is how much the city of New York and our entire globe is actually driven by unseen software.
00:58Not the AI kind of software that we mostly talk about.
01:01It's the traditional, deterministic, rule-based, reliable software that has brought the modern world to us.
01:12It is handcrafted for over 70 years by humans.
01:17I'm Hari.
01:19I'm the product person who has built software for almost like two decades now.
01:25And I have this one genuine theory.
01:29If at all there is a new industry in the human history, it must be software industry.
01:35Because you take navigation, agriculture, construction, and even astronomy, we had this in different forms and shapes earlier.
01:45Now that it may be more sophisticated, but they're nothing new to us.
01:50Today from getting to work to watching this video, from nuclear reactor to all the way to space station that drives the 125 trillion dollar economy the one common factor is software.
02:05So, it would be catastrophic if for some reason software stops, just like the Y2K.
02:12And our modern world was a lot to this kind of a software.
02:19The deterministic kind, where you trust it with your pacemaker and when you fly on your 747.
02:27So, if you look into the product here on the left side, you have the engineers.
02:36The engineers are hired and they are the source of judgment for every product out there with their unique taste.
02:47And I'll go to an extent and call their belief systems and value systems are reflective in the software that they shipped.
02:56The tool chain owned deterministic rules all the way.
03:02The software that runs the world and the tool chains next to it in a cross feedback loop where the assemblers built C, C built Unix, Unix built everything and then
03:14we built Kafka, Zookeeper, all the way up to Kubernetes that runs the world today.
03:20The only judgment in this picture is human judgment.
03:25And we all know how varied we are.
03:29And that variance is where every product edge originated.
03:36The block here is a high-level representation of the entire software industry and the world economy.
03:43Who has the IP?
03:44Where is the continuous loop?
03:46And how that loop where every brick on the tool chain helps us scale new heights.
03:54Whether the software is deterministic or probabilistic, that's a question.
03:59And hold this picture.
04:01Let's watch one card change.
04:04Which is, yes, in the age of AI, the engineers today are assisted or driven or sometimes they claim it's completely autonomous by models that I would call rented.
04:19The good thing is that it creates a level playing field with everyone having equal access to the bleeding edge models and harnesses.
04:28Teams who cannot afford earlier to the talent pools of Google and Meta who handcrafted this modern software can now afford intelligence through APIs where it claims to have solved
04:44the software problem.
04:45Let's see whether it really solved or what's happening.
04:48If you actually pay attention and look under the hood everything around the model, the hardness, the agents, the tools, the MCP are still the same old deterministic software.
05:02People who are building this would definitely know that the these are not AI products.
05:09These are the same old deterministic software built through maybe the models.
05:15And what did not change and stay put is what's the outcome?
05:22At the end of the world day, the world requires software that does the same thing reliably every time.
05:31So, the final artifact is a deterministic software that drives the world economy as we know it.
05:41If that is the case, then what is the role of AI here?
05:45Its role is to one to accelerate the development of deterministic software at the lowest cost possible.
05:57And write deterministic software that never existed before because of budget not an interesting problem, hard problems.
06:07So, those are the two use cases I see going to play out in the next 5 to 10 years.
06:13So, the bottom line is we will be using a probabilistic gear to crank out a reliable deterministic software by augmenting or somebody will go to an extent say replacing
06:31human judgment.
06:33Is it possible?
06:34Where are we today?
06:36We will see in this talk.
06:41Who knows this person?
06:43I'm a big fanboy of him.
06:45He's a superhuman and his friends.
06:51Okay, now I see you will connect dots because you know the couple of men's there and he's part of the elite sub-10 club.
07:03Two of them are relatively well known, but all three are superhumans.
07:07Aren't they?
07:11So, in 2015 nine souls competed and the difference between them is only 27 hundredths of a second.
07:24And Gatlin lost the gold by 100th of a second.
07:26Can you believe it?
07:27It's not comprehensible to human mind.
07:30The scientific measurements gave that, okay, that is how Usain Bolt won over Gatlin that night.
07:38On another night the order could be completely different.
07:41I still run those 100 meters and I take 10 13 seconds, not 10.
07:46How about you?
07:49That's the difference between us and these extraordinary athletes competing at the edge of human capability.
07:57But as a fan boy, I will argue my favorites like I just did earlier.
08:03But I also speak to professionals in this game.
08:07And in their opinion, the gap is so tiny that the previous best or who holds the world record does not matter when they get onto the field.
08:18On a given day, anybody can beat anyone.
08:23I'm going to apply the same yardstick on models to drive a couple of points.
08:28Six labs the recent study came out from Stanford a few dozen Elo points here and there.
08:39And the leaderboard is changing with every release cadence.
08:42And the popular question this week which model are you using and how smarter did you feel by using that model?
08:49Isn't that the great dinner table argument these days?
08:54But I can tell you one thing, it is not necessarily a good boardroom argument.
09:00Why?
09:00Because often the emotional side of us, like the fan boy moment, peaks into meeting rooms.
09:05After all, we're all humans.
09:07I'm not saying don't compare models.
09:10All I'm saying is don't make a small moving frontier gap into your product roadmap as an advantage.
09:20Let's dive a little bit deeper here because there are some numbers to unpack.
09:25Anthropic 1503 Mistral 1460 87 points here and there.
09:33It does not mean Anthropic is 87 points smarter than Mistral.
09:39Actually, the scores, the way we should read them, is relative rating derived from model comparison.
09:45It actually tells us how these models rank relative to one another in that evaluation environment.
09:54It's not the absolute intelligence scale.
09:58But here is the elephant in the room.
10:00In all these conversations across multiple teams, I have seen multiple organizations, while we think we are comparing models, what I've observed is many times, they're not even comparing models.
10:13We mistakenly compare the harness or the product for models.
10:18The Claude code versus Cursor or Codex isn't a model comparison.
10:23It's closer to comparing Jamaica versus USA.
10:26You're comparing an athlete and a training system around the athlete.
10:31So, the model is the athlete or the engine.
10:33It provides the underlying capability.
10:36The harness determines how this engine is used.
10:41The tools, context, instructions, orchestration, memory, workflows, the MCP tool callings, that's the harness around it.
10:51The product determines why anyone should care because that's the end product.
10:56Today, you and I can rent the engine and very much all the harnesses that come with engine.
11:04Some come tightly coupled, some are loosely coupled.
11:09So, if everybody has the same source, like we saw in the previous picture, where is the differentiation?
11:18The one that used to come, which made these different companies thrive in different areas, the source or the first principles of that is from the diversity and the variance of their engineer minds.
11:32Given the model is rented, increasingly, what we are seeing is that once the capability is available to everyone, then what are we staring at?
11:45The model and the hardness do not care who is typing and from which part of the world.
11:52Or whether you are a E6, E7, or a E1 engineer.
11:59Find at the same problem, same tools.
12:03It gives the next person what it gave you.
12:07And I want to call this answer as a median.
12:10And this median keeps changing with every release cadence, as we saw in the diagram there.
12:15The previous problems solved get into these LLMs, and then incrementally, that's what these releases come one after another.
12:24Just like software cadence.
12:27I'm super simplifying here, but in a nutshell, that's what keeps the median moving up front, which is a good thing.
12:35So, at the same time, it's also a kind of a trap.
12:40Because it is good enough to ship, but it is hard to tell apart that what the next person is going to be shipped.
12:51Because the next person or the competitor could have also shipped something similar or exactly the same.
12:59This could not have happened before 2023.
13:03The median product was expensive.
13:05You needed a team, time, stack.
13:11Today, that solved problem is free.
13:18And the problem that is free is free for everyone.
13:23And you go out to the customer like how everybody goes out.
13:27And then this problem of so many products which are similar flooding the market then the distribution becomes the moat, not your product.
13:38So, what is that moat that you can create in this new environment?
13:43I'm going to quickly go through this GitHub's own number to reiterate the point that this model unharness are producing enormous amount of new software.
13:57And to me these are really good software.
14:03But the outcomes are similar.
14:06This is not crowd of copycats.
14:08Definitely not.
14:09It's a crowd of people who have arrived at the same answer because the source or the cookie cutter is the same.
14:17And the word that our industry choose for what they built is AI slop.
14:24The software AI slop.
14:26I want to take that word away from you and give you a new meaning to it.
14:30Let me be fair first.
14:32If you have been developing for a long time and you stumbled upon this AI assisted development it felt like magic, isn't it?
14:42Because we were taking a black box, pointing something to it.
14:45You would have definitely felt at some point in time, "Oh, it did really well."
14:49Two weeks ago I shipped an iOS app.
14:51I've been a developer, but I've never dreamt of developing an app because remember we had lanes, embedded developer, app developer.
15:00AI today is making everyone a developer.
15:02To me that's a good thing and I mean it.
15:05The word slop is negative.
15:07And they use it without any empirical analysis.
15:10Here is why I feel that those 100,000 of repos and proliferation of AI assisted development is bringing new people into the field who has never developed.
15:23The success metrics for these new products cannot be looked at from the lens of those experts.
15:30I would like to define the success of those million AI apps as long as they meet the success of their creators.
15:38Because who defines what a successful product looks like?
15:41For us, it could be something that brings a revenue.
15:44For someone else, it could be something that they can show it next to their colleagues that hey, see I've built this wonderful website.
15:51That's a success, too.
15:53And I've seen millions of people actually done that with lovable and Replit, which is cool.
15:59They are not AI snobs.
16:05What do you get when you point a good problem at a coding agent?
16:12Most times than not, today it's useful and the finished artifact may be similar because, you know, the same cutter went into the same dough.
16:23But most software developed this route is falling into what I call the median.
16:29I am not telling you to stop developing or shipping it.
16:32It's perfectly fine for internal consumption.
16:36The one caveat here is that stop counting this as a differentiating factor.
16:46But this median is lacking one thing.
16:48We need to also know the positives as well as the disadvantages.
16:53It's not a new problem.
16:55Back in '90s, everybody was trying to solve the search problem and most of them had the basic crawler and an index.
17:05And whoever is going to solve that problem really well or who has that novelty element is going to rise up because that's a differentiating factor.
17:16So, today I say the median of most software that you can think of building is actually free.
17:23But, if you want to compete in the open market, you need that element of novelty.
17:32So, some credibility before I go further because I've spoke almost for a long time.
17:38I did not start building when the models got better.
17:40I have built more than a dozens of applications in public and contributed to Lama index another repos.
17:47The last 3 years have been quite exciting and that brought me to this topic and tiring at the same time.
17:53I want to show you this because the project I'm going to show you are not hobbyist first attempts.
18:01I tried to solve a real-world problem and I ended up with a median.
18:05I never expected that.
18:10So, I want to coin few terms here.
18:12The build first reflex.
18:14When we all got excited with this AI and things, we stopped doing something which is back in the days when building was costly and difficult, I used to search
18:25more than I built.
18:27Today, it's flipped.
18:29You point a problem at a coding agent is more easier than going and searching and reading through those verbose whatever comes out and making a decision.
18:39We don't want to make decisions anymore.
18:41And this is getting the new normal in our craft.
18:45So, what do we do?
18:47Let's ask the coding agent to build a rack system.
18:52And I call this the build first reflex.
18:55Open a problem, open an agent.
18:59Yeah, you get the scaffoldings in the first 2 minutes and I'm sure if you've been used to this kind of development, you would do a demo in 2 hours, working demo.
19:09Where you give it to someone with rough edges, it works.
19:13And you could actually launch within 4 weeks.
19:16A real good product with support and everything.
19:19But only then you find out that, okay, I have built the 400th or 4,000th rag system.
19:24I took rag as an example here.
19:28So, I pointed the same problem and this is what we got.
19:32And I asked, "Hey, why did you build the nth rag system when more mature, feature-rich, open-source systems are available with thousands of stars and maturity?
19:43We could have just leveraged it.
19:44We should not have built it.
19:463 years ago, we wouldn't have built it."
19:50So, it accepted that, "Hey, okay, I'm the culprit here.
19:55You said, 'Can you build?'
19:57I went ahead and built.
19:58If you had told me this, then I would have done something different."
20:02The median rag is what actually built, which is a solved problem.
20:08But if you're building it for the first time, it's going to be exciting for you because you're going to feel that I gave a PDF, I pointing it to a back-end LLM,
20:17and I'm asking the questions.
20:19The chunking and all those factors, which I understand or do not understand, it's going to solve it for me, the vector DB, and I got a solution.
20:28Small wins produce a lot of dopamine, and I fall for the same things as you are.
20:36And this is the verbose response.
20:38I defined that on the previous slide.
20:42Now, I'll stop talking about those trivial problems and show one of my applications.
20:48This one project is one of the representatives of the first 2 years, where I developed a lot of apps.
20:58Like you all know, my build-first reflex kicks in.
21:03I did this a lot of times to coin this term.
21:07I built it by sacrificing going out to play.
21:12I skipped many socials and movie nights.
21:17And I was building like crazy.
21:18It was like playing a game of chess where you are addicted to it.
21:23And I had every AI platform you could imagine.
21:27Max plan.
21:28Sometimes free.
21:32After all that effort, I finally soft launched.
21:36I'm a bit of a perfectionist.
21:39So, until then I was bombastic because I could see my idea become something which is real in no time.
21:49Where as a product manager, it would have taken me a big team, architects.
21:55I was able to do it all alone.
21:59I was bombastic until I found out that I haven't calibrated to reality yet.
22:06The truth is my circle told me that they have seen similar calculators elsewhere.
22:15And before anyone thinks I stole from them or they from me, the surprising thing is that nobody copied from anybody.
22:24It's the same thing.
22:24It's a copycat work.
22:26When I analyzed some of those open and closed calculators, I found it same sometimes.
22:35Not even similar.
22:37And how could people across could arrive at similar solution?
22:43Even if you look at the LLM routing, everywhere is converging.
22:48Like those models there, even solutions are converging.
22:52And competing for a tiny space.
22:56And it's like having same developers in your team and your competitors also have the same developers.
23:05So, you can build intuition only through building and I'm sure you're able to those builders would agree with me.
23:15We now have something to recalibrate and find a successful path.
23:20So, after 2 years where I spent almost all time I was tired cuz I spent multiple Any exciting idea I had, I went ahead and built with a build first reflex.
23:31And almost every time I ended up with a crowded lane where there are already two or three people or I would be the first and I would take a month off and come back there would be 60
23:41similar products.
23:43And whoever has a better distribution is obviously going to uh win that battle.
23:51So, I would like to coin another new term.
23:54Okay, the build first then what?
23:57In the AI software development life cycle, a novelty budget.
24:03So, if you can get at least one unit of true novelty like how that page rank was for Google, you can cross this median at least for a good amount of time.
24:13You pick a problem and you want to be the best solution for the problem in the world, then you have a good chance of beating the crowd.
24:22For the first 2 years I spent, you know, my days and nights on architecture, agent design, prompt chains, skills, and somebody would share, "Hey, this is my Claude new workflow.
24:33That's my Claude new workflow."
24:35I did all those things.
24:37And most of the times than not, trust me, you would end up with similar outputs if not same.
24:45So, I had to rework my calculator that I showed you and this time I spent almost all of the time on how to differentiate and almost none on flaunting my Claude code or code skills which I used to do earlier.
25:00Because that felt like magic.
25:02The median from model isn't opinionated.
25:05It is converged best practices.
25:07So I picked a lane and I treated it like a budget.
25:13After 2 years of burning several million tokens on many max plans, I feel everything else is free now.
25:21And the highest return of investment is a novelty budget.
25:25So before the bill starts, name the one thing that is novel.
25:31If you don't have one, then you can take the median because there are a lot of people who have built the solution for the problem that you have and I'm sure of it.
25:40You tell me a problem for which there are no solutions.
25:44I'll go and find it for you.
25:46And then put this novelty plan as part of your road map and your project planning.
25:52I do this now.
25:55And review it the way you actually review spending like a project.
26:03Honestly, if you cannot name the novelty in your project, then you should concede that it's a learning exercise.
26:14That's fine.
26:19Uh the failures, the learnings, and the mechanics to beat medians for 2 years, I came up with this four rules.
26:26And these are not side project rules.
26:29I want you to run them as your road map for the next quarter speeches.
26:35Because your products will definitely have a twin and sometimes it might be open source.
26:40Why take all the effort in building it?
26:43Leverage it, fork it.
26:45So one of the things is be highly opinionated.
26:49The AI assistants most of the times are going to give you a probable answers.
26:53Be it anything.
26:56And those things are going to be same for almost everybody.
26:59It might be similar, but that's in the spectrum of already solved.
27:05In the world of medians, an opinion is the cheapest novelty you could own.
27:13And go for it because the model cannot supply.
27:16Two.
27:18The prompt to production is a median path.
27:22And there's absolutely nothing wrong with it.
27:24If you feel that it's an internal consumption you want to build and your team has to use it, go for it.
27:31I'm all for it.
27:34And then the third.
27:36This is the most fascinating things.
27:37When AI says done, then it means that to me, okay, AI has given its median product.
27:44I have to start working from here.
27:47That's the signal from any of your coding agents because your work begins there.
27:53And remember this.
27:57These coding agents so far has given the last great design for free.
28:05It's not going to invent the next Kubernetes or Zookeeper or Kafka or whatever those building blocks are.
28:12It's going to do a scaffolding and it does a really good job in multiple places, but it's it's it's going to be the last great design for free.
28:21And what's your job?
28:23If you want to thrive, you need to find the next one.
28:29And here are some methods I found useful.
28:32I've shared this method with multiple leaders across organizations and I've got good feedback on them.
28:40If you are leading and don't build now with this AI, this is exactly for you.
28:47You'll be seeing a bunch of demos.
28:48I'm sure you're bombarded with them every week.
28:51And it looks like a big deal because I've seen the instincts of leadership were not calibrated because back when they were building their intuition the deliverable that they are seeing
29:07took them a quarter or more time and a team.
29:10Today a new engineer shows up and claims to have built the same that would have taken a team and a quarter uh to build it.
29:22So, it's one of the hardest time to be on deciding where to put your eggs on.
29:29Especially when you're not immersed in this tech.
29:32Because nobody can recalibrate by reading or watching YouTube or LinkedIn posts.
29:37You recalibrate by building.
29:40You build intuition by building.
29:42So, the playbook is simple.
29:43Three questions.
29:44It's not going to solve your problems, but these are good handy questions to ask.
29:49One.
29:50Who has already done this?
29:52This question forces the search because the world of exciting AI, we need to tame the build first reflex powered by AI.
30:03Two.
30:05What do we have that the next team or our competitors or another person in the team don't have?
30:15This forces novelty as a first-class citizen into the minds of the team members.
30:22Because a better prompt, somebody would say, it's just going to throw you a median problem median solution.
30:30And third, where will this application live?
30:36In the age of exciting AI, forcing outcome-based thinking when everyone is high on building, it's a way to show them the demo is the expensive way to find out
30:50if you have no distribution mode.
30:53And I applied all this rigorously in the last 1 year.
30:57And you may be asking, what happened to the one project that I was talking to you?
31:03[snorts]
31:04I asked these questions.
31:06Okay, if I have to be providing that value, which is above median, then what stands between an open model and a token generating system?
31:19And what cost does an enterprise have to pay for it?
31:22Because open weights are at rage, but open weights are not open inferences.
31:31And I found that the hard way because there are 10 layers that sit between an open weight and a system that generates these tokens.
31:41We take the labs for granted for providing us there is ML ops, there is SRE on a different scale altogether that the world has not seen.
31:54So, precision KV cache, uh batching, execution parallelism, and there are different slice and dices of them.
32:01So, some of them you have to hire experts.
32:04And I don't know where these experts come from because this is not a tech or a skill that the industry already has.
32:14And some of them, layer nine, is utilization.
32:17It is structural.
32:18You cannot buy it.
32:23So, unless like I said, you take one problem and want it to be the best at solving that particular problem, you're not going to cross the median just by using these AI tools.
32:36And I searched and I found that none of the 244 calculator that I was showing you earlier did this.
32:44And I changed the entire application into kind of what the identical weights cost to run inside your enterprise versus running it in the Neo cloud and taking tokens.
33:03That is what this calculator gives you.
33:06Because if you have open models and want to go and build it, then what's the difference?
33:14And received great feedback from a lot of enterprises who want to actually go that path because when people say open models, they're excited.
33:24For what?
33:28Organizations considering deploying open weight models, the weights are free.
33:34But you may want to check.
33:35That's the QR code.
33:37So, scan it, put it into your workload, and see which way it goes for you.
33:43It's open source and free.
33:49So, here is where it all lands and ends.
33:54With the power of the LLMs on the harness, the last 3 years building has got exponentially easier.
34:04But the changes and the dynamics of software development still hasn't changed at the end.
34:14The prompt-based median products are useful.
34:17I ship them, too.
34:18They're not slops.
34:20But they have little durability when they go to open competition.
34:23And they may survive if you have a huge distribution mode.
34:27So, the one attitude if you want to try thrive in this field is again I'm I may be repeating this, but take that one problem and pursue it with the intent of
34:40being the best at it in the entire world.
34:44Yeah, we may not be able to crack it, but that intent will definitely put you above median.
34:54The only question left is can your AI-assisted products thrive?
34:59I spent 2 years on the median, but that was education.
35:03It was worth every token I burnt it.
35:06I asked two questions at the start.
35:08What happens to the software?
35:11To me, the software that is running this 100 trillion-dollar economy, nothing happens to it.
35:17The world still runs on deterministic software, at least for the foreseeable future, and it might always be.
35:24The age of AI means that we can write it more faster, but it is not going to take out the deterministic software.
35:38So, I would like to wrap up by saying that spend everything you have on one thing, which is the novelty.
35:53Squeeze every vein, every ounce of your brain on novelty, because novelty is scarce, and the median is free.
36:00It's up to you to choose.
36:03Thank you.