WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Bck7ABCZRZI NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Bck7ABCZRZI.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.000 --> 00:00:09.880 One of the largest retailers in the country spent close to $200 million on inference with Anthropic and decided that things got way out of hand and built their own infrastructure. s2 00:00:09.880 --> 00:00:19.320 I'm pretty sure most of you have read the news from Uber CTO on how they had planned a budget of their tokens for an entire year and it got over in month four. s3 00:00:19.320 --> 00:00:27.040 I'm also confident that half of you in this room have come to a very similar conclusion that as time goes by the cost of intelligence really built. s4 00:00:27.040 --> 00:00:30.720 Using inference feels like, you know, it's one of the most inexpensive thing. s5 00:00:30.720 --> 00:00:40.720 But then this is very different from using a phone where you get a bill once every month and then you have like a specific set of amount that you can actually anchor your mind to. s6 00:00:40.720 --> 00:00:44.920 But in case of using these rented intelligence platform, they are like prepaid. s7 00:00:44.920 --> 00:00:46.360 You load credits. s8 00:00:46.360 --> 00:00:49.080 It's almost as if you're loading credits inside a casino. s9 00:00:49.080 --> 00:01:00.040 You put some and then you pull it and then you are so addicted to it then you end up doing more and more of it and by some time you realize that you've blown past the threshold that you had mentally kept in mind. s10 00:01:00.040 --> 00:01:03.200 And I had this experience myself. s11 00:01:03.200 --> 00:01:07.840 I built an app called Ultrazone or I experienced the inference cost ballooning here. s12 00:01:07.840 --> 00:01:10.720 Suno.com has anybody heard about Suno.com? s13 00:01:10.720 --> 00:01:16.440 Yeah, Suno.com is is this application that allows a user to turn a text prompt into music. s14 00:01:16.440 --> 00:01:23.320 What I was interested in is is doing the reverse which is given a particular song, what prompt could have actually generated it. s15 00:01:23.320 --> 00:01:25.080 This is something that I wanted. s16 00:01:25.080 --> 00:01:26.160 So I built this. s17 00:01:26.160 --> 00:01:32.040 I was having a lot of fun using this application, shared it with a few friends and spread wide around. s18 00:01:32.040 --> 00:01:38.160 I had hundreds of thousands of users, but then the cost ballooned way more than what I had anticipated. s19 00:01:38.160 --> 00:01:41.480 Hundreds of thousands of dollars I had to spend on inference. s20 00:01:41.480 --> 00:01:43.320 Now, this happens for many reason. s21 00:01:43.320 --> 00:01:50.080 There are many talks that are there at AIE itself where people talk about how you need to manage your context better. s22 00:01:50.080 --> 00:01:54.600 And many people forget about doing compression of their input token. s23 00:01:54.600 --> 00:02:00.600 And when there are agent loops, then there are many of these calls that are happening which are very, very wasteful. s24 00:02:00.600 --> 00:02:06.600 The inference endpoint that is consuming this is is completely unaware of the shape of the workload and which is why this happens. s25 00:02:06.600 --> 00:02:08.840 And I have this other issue that had happened. s26 00:02:08.840 --> 00:02:11.240 3 weeks ago, my key got stolen. s27 00:02:11.240 --> 00:02:15.640 Someone in China got hold of it and then was sucking my endpoint dry. s28 00:02:15.640 --> 00:02:22.920 I could see the cost rise up from 7,000 to 7,500 dollars to 8,000 and going and so forth. s29 00:02:22.920 --> 00:02:28.080 Thanks to my co-founder who heads the research and technology, we were able to arrest it at 10,000 dollars. s30 00:02:28.080 --> 00:02:30.440 Otherwise, it could have been 100,000 dollars. s31 00:02:30.440 --> 00:02:35.080 Now, many people suggest that the alternative to rented intelligence platform is is to use token factory. s32 00:02:35.080 --> 00:02:39.720 Token factory is is basically saying that why are you paying money to Anthropic and OpenAI? s33 00:02:39.720 --> 00:02:48.360 Instead, go open source, have these open source models that are already deployed somewhere in the cloud, and then they are provisioned as tokens per second. s34 00:02:48.360 --> 00:02:52.360 There are neo clouds and then there are inference endpoint providers who actually do this. s35 00:02:52.360 --> 00:02:56.560 In fact, there is also an argument saying that, you know, you can build this token factory locally. s36 00:02:56.560 --> 00:03:03.400 There are AI Twitter influencers who actually talk about building inference in your garage, in your basement. s37 00:03:03.400 --> 00:03:08.840 Buy GPU cards, rig them up together, and then you could actually run a local token factory. s38 00:03:08.840 --> 00:03:11.600 In fact, I was in inspired by that a little bit. s39 00:03:11.600 --> 00:03:19.040 I bought my own DGX box and then I first moved Ulta Sono from Anthropic to DGX box. s40 00:03:19.040 --> 00:03:19.840 It worked well. s41 00:03:19.840 --> 00:03:23.040 I ran into this one issue of memory being the bottleneck. s42 00:03:23.040 --> 00:03:26.720 And then, it was good enough that I started building my next applications. s43 00:03:26.720 --> 00:03:28.360 I started having agents. s44 00:03:28.360 --> 00:03:30.800 I have some agents that I need for running my research lab. s45 00:03:30.800 --> 00:03:36.120 So, these agents started shaping up inside the DGX box, you know, two, six, eight, 12. s46 00:03:36.120 --> 00:03:37.239 And it worked all right. s47 00:03:37.239 --> 00:03:43.120 The issue though is is that, you know, it may not be reliable for enterprise, which is what I exactly faced. s48 00:03:43.120 --> 00:03:47.000 Three enterprises reached out to me to replicate the same setup for them. s49 00:03:47.000 --> 00:03:50.200 But for enterprises, renting and leasing don't cut it. s50 00:03:50.200 --> 00:03:57.120 Bill is a problem, but then there are secondary set of problems that makes it extremely ineffective approach. s51 00:03:57.120 --> 00:04:03.000 The enterprises that I reached out to me, one was a fund, another a hospital, and the third a tax practice. s52 00:04:03.000 --> 00:04:06.800 And each of them had different wall that they had hit. s53 00:04:06.800 --> 00:04:16.560 The fund, it was an investment fund, they were running an investment analyst on an email client architecture, and they didn't want somebody else to dictate as to what the rate limit that they could consume. s54 00:04:16.560 --> 00:04:20.880 So, control become a big issue for them to actually go with token factories. s55 00:04:20.880 --> 00:04:22.680 Hospital had a different issue. s56 00:04:22.680 --> 00:04:32.680 They used the use case, it worked well, but later when they went through an audit, a third-party vendor dependency was redlined, and then they couldn't go forward. s57 00:04:32.680 --> 00:04:35.440 The tax practice was a completely different issue. s58 00:04:35.440 --> 00:04:42.160 In a tax practice, what is happening is this, when an intelligent generates a recommendation, you want to be able to recreate it. s59 00:04:42.160 --> 00:04:48.720 And when you don't have access to the in-depth of the model, you will not be able to do this, and that became the issue. s60 00:04:48.720 --> 00:04:52.040 So, that brings me to the most important point of this presentation. s61 00:04:52.040 --> 00:04:53.200 Where do you sit? s62 00:04:53.200 --> 00:04:55.400 When do you stop renting infrastructure? s63 00:04:55.400 --> 00:05:05.400 If you're a startup, if you're a founder who is doing pre-product market fit work, so you're still figuring out that the use case that you have, if there is demand for it, you can get by by renting. s64 00:05:05.400 --> 00:05:21.040 But if you're post-product market fit, you cannot afford to And if you're an enterprise who's already budgeted a project, which means you're telling that you are assuming that this particular use case has product market fit, then again, you cannot ignore to build your own infrastructure. s65 00:05:21.040 --> 00:05:31.200 Which is what I realized, and I said that this this situation is this like, if you're going to a new city, you may initially start with saying that I don't want to buy a house. s66 00:05:31.200 --> 00:05:32.360 Let me actually rent and see. s67 00:05:32.360 --> 00:05:34.200 Sometimes you might even Airbnb. s68 00:05:34.200 --> 00:05:40.080 You experience the environment, you experience the city, the neighborhood, but then eventually you have to buy the house. s69 00:05:40.080 --> 00:05:42.960 You cannot raise a family in an Airbnb. s70 00:05:42.960 --> 00:05:53.480 As I went through this experience, I decided I came to the conclusion that I need to build my own inference infrastructure for the apps, the agents, and the scaling of the apps that I'm building. s71 00:05:53.480 --> 00:05:55.400 And I call this as just infra. s72 00:05:55.400 --> 00:06:00.880 And while I went through this exercise, I realized that there is optimization to be done at multiple layers. s73 00:06:00.880 --> 00:06:09.360 Even at the renting and the lease layer, you can do optimization around input cost to token management, and then, you know, context management, and so on and so forth. s74 00:06:09.360 --> 00:06:17.160 Some of those experiences that I've had in the last couple of months combined it into an open-source project and published it as just token max. s75 00:06:17.160 --> 00:06:21.600 If you have used headroom from Netflix, then this is an alternative to it. s76 00:06:21.600 --> 00:06:27.080 We benchmarked against headroom and on many parameters, just token maxes is far superior. s77 00:06:27.080 --> 00:06:31.720 If this is a thing that is of interest to you, give it a try, maybe a GitHub star if you like it. s78 00:06:31.720 --> 00:06:40.680 I also wrote the book called peak inference infraeconomics of AI inference when you have to think about building your own inference infrastructure. s79 00:06:40.680 --> 00:06:51.840 The AI market is is very different compared to the rest of the technology market that used to exist because here the rules of the game change every three to six months, which means it becomes a very noisy marketplace. s80 00:06:51.840 --> 00:06:55.480 You talk to someone like Jensen, he would say token factory is the future. s81 00:06:55.480 --> 00:07:00.680 You hear someone like a Satya Nadella, he will say unmetered intelligence is the future, it is going to be local. s82 00:07:00.680 --> 00:07:08.800 And then when you hear new clouds and inference endpoint providers, they'll say, "Hey, inference endpoint providers are the ones that are going to capture the value in the marketplace." s83 00:07:08.800 --> 00:07:21.960 Now, my experience walking from application to agents to scaling them and then building my own inference infrastructure taught me that if you want to learn, you can rent, but if you want to earn, then you have to own. s84 00:07:21.960 --> 00:07:26.800 And if there was the one sentence that you had to take away from this entire presentation, it is that. s85 00:07:26.800 --> 00:07:29.120 Rent to learn, own to earn. s86 00:07:29.120 --> 00:07:32.120 But then, you have to come to your own answers. s87 00:07:32.120 --> 00:07:32.840 Thank you. s88 00:07:32.840 --> 00:07:43.960 And if any of these topics are of interest to you, then I'm happy to talk to you about renting, about just token max, about how to build your own inference infrastructure. s89 00:07:43.960 --> 00:07:46.760 I'm here at the AI Engineer's conference for the next 3 days. s90 00:07:46.760 --> 00:07:52.080 Hit me up on Twitter at mtraj or through my site mtraj.com.