Stop Renting Your Cognitive Infrastructure - Thiyagarajan Maruthavanan, Kalmantic Labs

AI Engineer · 7 min · 90 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:00One of the largest retailers in the country spent close to $200 million on inference with Anthropic and decided that things got way out of hand and built their own infrastructure.
2. 00:09I'm pretty sure most of you have read the news from Uber CTO on how they had planned a budget of their tokens for an entire year and it got over in month four.
3. 00:19I'm also confident that half of you in this room have come to a very similar conclusion that as time goes by the cost of intelligence really built.
4. 00:27Using inference feels like, you know, it's one of the most inexpensive thing.
5. 00:30But then this is very different from using a phone where you get a bill once every month and then you have like a specific set of amount that you can actually anchor your mind to.
6. 00:40But in case of using these rented intelligence platform, they are like prepaid.
7. 00:44You load credits.
8. 00:46It's almost as if you're loading credits inside a casino.
9. 00:49You put some and then you pull it and then you are so addicted to it then you end up doing more and more of it and by some time you realize that you've blown past the threshold that you had mentally kept in mind.
10. 01:00And I had this experience myself.
11. 01:03I built an app called Ultrazone or I experienced the inference cost ballooning here.
12. 01:07Suno.com has anybody heard about Suno.com?
13. 01:10Yeah, Suno.com is is this application that allows a user to turn a text prompt into music.
14. 01:16What I was interested in is is doing the reverse which is given a particular song, what prompt could have actually generated it.
15. 01:23This is something that I wanted.
16. 01:25So I built this.
17. 01:26I was having a lot of fun using this application, shared it with a few friends and spread wide around.
18. 01:32I had hundreds of thousands of users, but then the cost ballooned way more than what I had anticipated.
19. 01:38Hundreds of thousands of dollars I had to spend on inference.
20. 01:41Now, this happens for many reason.
21. 01:43There are many talks that are there at AIE itself where people talk about how you need to manage your context better.
22. 01:50And many people forget about doing compression of their input token.
23. 01:54And when there are agent loops, then there are many of these calls that are happening which are very, very wasteful.
24. 02:00The inference endpoint that is consuming this is is completely unaware of the shape of the workload and which is why this happens.
25. 02:06And I have this other issue that had happened.
26. 02:083 weeks ago, my key got stolen.
27. 02:11Someone in China got hold of it and then was sucking my endpoint dry.
28. 02:15I could see the cost rise up from 7,000 to 7,500 dollars to 8,000 and going and so forth.
29. 02:22Thanks to my co-founder who heads the research and technology, we were able to arrest it at 10,000 dollars.
30. 02:28Otherwise, it could have been 100,000 dollars.
31. 02:30Now, many people suggest that the alternative to rented intelligence platform is is to use token factory.
32. 02:35Token factory is is basically saying that why are you paying money to Anthropic and OpenAI?
33. 02:39Instead, go open source, have these open source models that are already deployed somewhere in the cloud, and then they are provisioned as tokens per second.
34. 02:48There are neo clouds and then there are inference endpoint providers who actually do this.
35. 02:52In fact, there is also an argument saying that, you know, you can build this token factory locally.
36. 02:56There are AI Twitter influencers who actually talk about building inference in your garage, in your basement.
37. 03:03Buy GPU cards, rig them up together, and then you could actually run a local token factory.
38. 03:08In fact, I was in inspired by that a little bit.
39. 03:11I bought my own DGX box and then I first moved Ulta Sono from Anthropic to DGX box.
40. 03:19It worked well.
41. 03:19I ran into this one issue of memory being the bottleneck.
42. 03:23And then, it was good enough that I started building my next applications.
43. 03:26I started having agents.
44. 03:28I have some agents that I need for running my research lab.
45. 03:30So, these agents started shaping up inside the DGX box, you know, two, six, eight, 12.
46. 03:36And it worked all right.
47. 03:37The issue though is is that, you know, it may not be reliable for enterprise, which is what I exactly faced.
48. 03:43Three enterprises reached out to me to replicate the same setup for them.
49. 03:47But for enterprises, renting and leasing don't cut it.
50. 03:50Bill is a problem, but then there are secondary set of problems that makes it extremely ineffective approach.
51. 03:57The enterprises that I reached out to me, one was a fund, another a hospital, and the third a tax practice.
52. 04:03And each of them had different wall that they had hit.
53. 04:06The fund, it was an investment fund, they were running an investment analyst on an email client architecture, and they didn't want somebody else to dictate as to what the rate limit that they could consume.
54. 04:16So, control become a big issue for them to actually go with token factories.
55. 04:20Hospital had a different issue.
56. 04:22They used the use case, it worked well, but later when they went through an audit, a third-party vendor dependency was redlined, and then they couldn't go forward.
57. 04:32The tax practice was a completely different issue.
58. 04:35In a tax practice, what is happening is this, when an intelligent generates a recommendation, you want to be able to recreate it.
59. 04:42And when you don't have access to the in-depth of the model, you will not be able to do this, and that became the issue.
60. 04:48So, that brings me to the most important point of this presentation.
61. 04:52Where do you sit?
62. 04:53When do you stop renting infrastructure?
63. 04:55If you're a startup, if you're a founder who is doing pre-product market fit work, so you're still figuring out that the use case that you have, if there is demand for it, you can get by by renting.
64. 05:05But if you're post-product market fit, you cannot afford to And if you're an enterprise who's already budgeted a project, which means you're telling that you are assuming that this particular use case has product market fit, then again, you cannot ignore to build your own infrastructure.
65. 05:21Which is what I realized, and I said that this this situation is this like, if you're going to a new city, you may initially start with saying that I don't want to buy a house.
66. 05:31Let me actually rent and see.
67. 05:32Sometimes you might even Airbnb.
68. 05:34You experience the environment, you experience the city, the neighborhood, but then eventually you have to buy the house.
69. 05:40You cannot raise a family in an Airbnb.
70. 05:42As I went through this experience, I decided I came to the conclusion that I need to build my own inference infrastructure for the apps, the agents, and the scaling of the apps that I'm building.
71. 05:53And I call this as just infra.
72. 05:55And while I went through this exercise, I realized that there is optimization to be done at multiple layers.
73. 06:00Even at the renting and the lease layer, you can do optimization around input cost to token management, and then, you know, context management, and so on and so forth.
74. 06:09Some of those experiences that I've had in the last couple of months combined it into an open-source project and published it as just token max.
75. 06:17If you have used headroom from Netflix, then this is an alternative to it.
76. 06:21We benchmarked against headroom and on many parameters, just token maxes is far superior.
77. 06:27If this is a thing that is of interest to you, give it a try, maybe a GitHub star if you like it.
78. 06:31I also wrote the book called peak inference infraeconomics of AI inference when you have to think about building your own inference infrastructure.
79. 06:40The AI market is is very different compared to the rest of the technology market that used to exist because here the rules of the game change every three to six months, which means it becomes a very noisy marketplace.
80. 06:51You talk to someone like Jensen, he would say token factory is the future.
81. 06:55You hear someone like a Satya Nadella, he will say unmetered intelligence is the future, it is going to be local.
82. 07:00And then when you hear new clouds and inference endpoint providers, they'll say, "Hey, inference endpoint providers are the ones that are going to capture the value in the marketplace."
83. 07:08Now, my experience walking from application to agents to scaling them and then building my own inference infrastructure taught me that if you want to learn, you can rent, but if you want to earn, then you have to own.
84. 07:21And if there was the one sentence that you had to take away from this entire presentation, it is that.
85. 07:26Rent to learn, own to earn.
86. 07:29But then, you have to come to your own answers.
87. 07:32Thank you.
88. 07:32And if any of these topics are of interest to you, then I'm happy to talk to you about renting, about just token max, about how to build your own inference infrastructure.
89. 07:43I'm here at the AI Engineer's conference for the next 3 days.
90. 07:46Hit me up on Twitter at mtraj or through my site mtraj.com.