Why AI Startups Will Disappear Without Compute Markets — Evan Conrad, San Francisco Compute

AI Engineer · 17 min · 213 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12So, this is a talk about compute markets.
00:15Um, but it's kind of more of a talk about why to make a compute market.
00:19Um, if you don't know me, I'm Evan.
00:22Um, we run a company called San Francisco Compute.
00:24Um, we make a physically settled order, physically settled forwards market.
00:29Um, so if you were here for the last talk, last talk was basically why all the operators of GPU clusters um don't really work very well.
00:37Um, when we say physically settled, we mean we're the operator.
00:42We're both the operator of the hardware in the same way that a cloud is the operator of the hardware and then we're also the market at the same time.
00:49It's kind of like a grid for compute.
00:51So the general gist of this is that in order to run and operate a supercomputer um we think problems start right at the design of the supercomputer um right in the design of the bill of materials
01:05um it's control of the backplane fabric as well it's also control of um bare metal orchestration it's also services on top VMs and so on um there's a lot of different parts of the stack that you have to do effectively
01:18otherwise um the folks who have a data data center um and would like to make more money on that data center by becoming the GPU cloud um end up
01:30failing for the customer.
01:31So that's what we do.
01:33We offer what's called supercomputing as a service.
01:35We go to folks who want to become a cloud.
01:37That's like folks who want to become what Nvidia would call an NCP um or venture firms that want to have clusters for their portfolio um and want to be able to reallocate that capacity when a company like Instinct is taking off.
01:50um or AI companies who would like the residual value on the compute because they've realized that's really important to their financials um but don't want to operate a supercomputer.
01:59So that's physical settlement and then on top of that we run a live order book for capacity that you can buy reserved either right now or in the future.
02:08Um this um together is not at all what we thought we were going to be doing.
02:14Originally, we were this company called Junlark um that was trying to make kind of like Sunno or UDIO, but prior to Sunno or UDIO.
02:22Um at the time, um what you needed to do to realistically start a lab was you needed to raise a big deposit, a huge one, which was $10 million at the time.
02:33Um we thought that was a crazy amount of money.
02:35Um and so we thought, no, that's stupid.
02:36We're going to raise 500k and obviously what we're going to do is we're going to rent from somebody else.
02:40Um obviously today um it's not $10 million, it's a billion dollars.
02:46And um
02:48[snorts]
02:48nonetheless, um we had raised our 500k.
02:51We buy a big cluster and then we're underwater on our cluster every month.
02:55Um and if we didn't subleasase our cluster, we would have gone bankrupt.
02:58Um so if you've heard of GPU brokers, we were the first GPU broker because we were forced into being a GPU broker.
03:04So the first year of the company was us just trying to sell our cluster, otherwise we would go bankrupt.
03:08Um, so I would wander around with these like actual um pieces of paper at Nurups which both say H100 and A100 at the same time because that's how I made uh I made the uh poster so quickly.
03:21Um, and so this talk is not really about um necessarily like how a compute market works.
03:28It's how a compute market um gets to exist.
03:32Uh we like to think about it like there's this pamphlet and on the pamphlet on the front side it's like man starting a Neolab sucks.
03:38There's all these different like financial and economic problems and on the back of the p back of the pamphlet it says compute markets.
03:45It's it's not super clear how one gets to the other until you open the pamphlet and then like oh my holy there's like so much stuff in this pamphlet and it keeps going and there's so many folds.
03:53It's just a very difficult and challenging problem because you need to solve two like miracle problems to get working.
03:59You have to both make a market which is uh a uniquely difficult problem because it has a cold start problem.
04:05You also have to build a cloud and then you have to not become a cloud.
04:08Um which is a whole separate challenge.
04:10But these are the problems that the labs are facing and it's about to be the problems that everyone is about to facing.
04:16So if you work at a startup these are your problems now.
04:18Um right now everybody is sold out of compute.
04:21Um this is um probably not going to change for a while.
04:26Um, so that means that you're going to have to buy a new cluster, a net new one, and there's a lead time.
04:30You have no way to pay for that cluster to come online now.
04:33You're going to pay for it to come online later.
04:35There's nothing in the market that can realistically get that cluster on online for you now.
04:39Um, meanwhile, you're going to pay your entire bank balance as a deposit.
04:43Um, the reason you're paying your entire bank balance as a deposit is because the lender has no idea how to render under underwrite AGI.
04:50Um, how in the world are you supposed to do that?
04:52You've got two person startups in San Francisco who or maybe New York um who say I'm going to build super intelligence.
05:00No lender knows how to underwrite super intelligence.
05:03Um so they have no idea what the asset is worth.
05:05Um meanwhile your cluster is very likely to be delayed.
05:09That's because the lender needed to underwrite against the credit risk of the customer and so they needed to know who is the customer first before you deploy the cluster.
05:16So right when the customer signs is the riskiest point in the entire operation.
05:21um you wanted it to be the case that you built the cluster first and then you sell it.
05:25This also helps reduce all sorts of different types of problems um where clusters break.
05:29Meanwhile, most of the people who are setting up clusters are people who have the actual asset that makes the thing cheap.
05:35Like you've got the data center space already.
05:38What that doesn't get is supercomputing expertise.
05:41um as we saw in the last talk if you've um watched that one.
05:45Um meanwhile, you're going to run into the exact same problem that Junark did, which is you're going to either way overby or you're going to weigh underby.
05:52Um that's because AI doesn't have margins that look like SAS companies.
05:56And so you don't have very much margin for error.
05:58And so the chances that you either underby and somebody else wins because they did it correctly or you over buy and then you just become stability AI or you blow up.
06:07you um end up like character AI um are really high.
06:11So these are all the same problem.
06:12They're all just different types of risk.
06:14They're technical risk, their financial risk, their credit risk.
06:16Um they're all one big bundled risk problem.
06:19Uh which is the problem that we end up solving and it's also the problem that we think compute markets broadly the entire industry solves.
06:24Um so we'll walk through each one of these and walk how that gets to a compute market because if you don't understand these, it's kind of like unclear why you would want a compute market.
06:32Um, and it almost seem like this problem that somebody thinks, "Oh, wow.
06:36Compute markets are cool, so I'm going to go start one."
06:38We think they should be derived from the actual problems that people face, which is first, everyone is sold out.
06:44Why are they sold out?
06:45Because the lender couldn't finance the cluster unless you had a long-term offtake agreement.
06:49In CPUs, uh, this wasn't the case because CPU buyers are SAS companies.
06:54SAS companies have high margins.
06:56And because they have high margins, um you can build a product that rewards you for having um a great um that rewards you with high margins um for your um your CPUs.
07:08Um think about it like this way.
07:11If you're a SAS company and your underlying cloud doubles their prices, you don't go bankrupt.
07:15You hate your cloud, you're mad, but you don't go bankrupt.
07:18If you're an AI company and your underlying cloud doubles their prices, you go bankrupt unless you go to some other underlying cloud.
07:24And what that means is that you're actually price sensitive.
07:27And because you're actually price sensitive, um, and there's nothing keeping you at any particular cloud.
07:32So you can't sell on demand because your customers will immediately leave to whoever is cheaper.
07:37And so um, that means that in order to finance GPU clusters, you have to have long-term take-pay contracts, hopefully with someone who has investment grade credit.
07:47However, if you don't have liquidity on those, if you're selling reserved and your customer can't get out, then that also means that some other customer who wants the capacity right now because they're like instinct and their product is taking off can't pay for anything.
07:58They have to finance a new cluster which has a lead time because the physical world just exists and it's going to take a while.
08:03Sometimes you even have to build the damn data center.
08:06Um, and then because you're financing a new cluster, you're going to pay a deposit because almost nobody in AI has any reasonable credit.
08:13And because this is a brand new technology, all these companies are new.
08:17Meanwhile, even as you um start to scale up that ladder, you're still going to run into credit issues all the way through.
08:23It's the number one bottleneck of AI at the moment.
08:25It's not power.
08:25You hit credit way before you hit AI or way before you hit um um uh power.
08:31So, um what that means is every inference company, um uh every Neol, they're bottlenecked by how much cash do they have in the bank account.
08:39And so they go back to the venture capitalist who says, "Oh, I'm going to fund at the model layer, the application layer, the inference layer because I don't want to fund all of that capex stuff."
08:47Um, and so the venture capitalist then funds all the capex stuff by giving the um the new neolab or new inference company um the check that they then send to the cloud um which is the deposit for the cluster and that's how you underwrite the customer.
09:04So um the clusters then are delayed because um the c the cloud couldn't actually deploy the cluster until they had the customer in hand.
09:15And so that was the riskiest point.
09:17So therefore um what should have happened was the cloud had bought the cluster ahead of time and then waited until it was landed in the data center and then sold it.
09:27And that means you have way less likelihood of delays.
09:31Meanwhile, the cluster is very likely to break when you get the cluster as the customer because you don't have liquidity.
09:37And so, if the failure rate is higher than what you expect and you don't have the ability to transition a node in, then you don't have anywhere you can pull nodes from, which means that if you need new nodes to make up for the failed ones, you have to RMA them, which means you have to get a
09:50new node.
09:51And then that gets us back to several steps before where you have uh the lead time problem again.
09:57And so, everything breaks for everybody all the time.
10:00Once you have gotten through all of that as a new AI lab, um your problems are you you you know you nobody was sold like everyone was sold out.
10:08Um you then had to wait a lead time, you paid this big deposit.
10:11Um the thing was delayed and then it broke and finally you got through all of that.
10:15Um and the likelihood that you predict your demand correctly is very low.
10:19And so you're either going to be way overprovisioned or way underprovisioned and you have no way out.
10:23Um there's no way to sell your capacity to somebody else and there's no way to buy more capacity if you need it.
10:27So, computer markets um to solve these problems, you need to solve a bunch of different problems all at once.
10:33Um you need to underwrite the value of the GPU cluster rather than underwriting the value or underwriting the credit risk of the customer.
10:40Um this is how it could work.
10:42Um it's not guaranteed to work like this.
10:45In other industries, what happened is everything just consolidated.
10:49You got one big creditworthy customer that does the whole damn industry.
10:53That's what the path of AI is currently on track to do.
10:57It's instead of going to be um a liquid market where anybody can participate, it's going to look like TSMC.
11:03You're going to get one buyer um or a few buyers that are the clouds.
11:07You're going to get a few model labs that are the AI labs.
11:10And then you won't get other startups.
11:12Um this is what we think is going to happen.
11:14And this is why 3 years ago when we were doing our AI lab, we had a chance.
11:19We could have said, "No, we're not gonna do our new um we're not gonna like, you know, stick it out for several years just trying to keep our company alive.
11:27Um we're just going to go do a new AI lab."
11:29But we saw this problem and we said, "Oh if we don't do anything about this, then we will just hit this 3 years from now."
11:37And here we are 3 years from now hitting this problem and where everyone's going to hit this problem.
11:42So um as if you watched the last talk um it's very hard to do solve this problem unless you are operating both the market and the clusters at the same time.
11:53Um so this is Dylan Page.
11:54Um Dylan is a co-chair of the CNCF infrastructure group.
11:57That's the organization that makes Kubernetes formerly worked at um Lambda doing infrastructure there.
12:01Um and the reason and he joined San Francisco compute because um GPU compute isn't sufficiently standardized reliable for the market operator to be divorced from the physical infrastructure.
12:11It doesn't work if you aren't the one who's buying the thing.
12:14Sorry, it doesn't work if the person you're buying from isn't operating it.
12:17To give an example of why this doesn't work is I'm sure some of you are flying in from overseas or sorry um out of town um and you have a hotel room.
12:26Um wouldn't it be great if you could, you know, sell that hotel room to somebody else.
12:31I go to you and I want to buy that hotel room.
12:34Um and I say, "Can I get room service?
12:37Can I get my room cleaned?
12:38Um I've got a problem with my room.
12:40who do I call?
12:41Um, if you don't operate the hotel, you can't fix any of those problems.
12:46That's why brokering clusters doesn't work.
12:48You have to operate the cluster like the hotel.
12:51You have to run the whole damn thing.
12:53Otherwise, you can't actually make a market.
12:55It doesn't work.
12:55You can't do physical settlement.
12:57Um, so why does nobody do physical settlement?
13:00Um, the problem is there's a very strong incentive to never do physical settlement.
13:04Um, that's because if you build out physical settlement for compute, what that literally means is you're building out the entire operating capacity of a GPU cloud and if you are building out the entire operating capacity of a GPU cloud,
13:17you should just be a GPU cloud because it looks to be the case that you will make an ungodly amount of money.
13:23Um, you will sign billion dollar deal after billion dollar deal.
13:27And so, it.
13:29Let's not make a GPU or let's not make a compute market.
13:31let's instead just become a GPU cloud, which is what basically everybody does because the numbers are astronomical and it's a very reasonable thing to do.
13:38Um, the reason why we ended up building physical settlement was because we knew that if we didn't start this, um, we would then not be able to start an AI lab, which was the real damn thing that we wanted to do in the first place.
13:48We wanted to make a lab.
13:49We didn't get to make a lab unless this exists.
13:52So, when we started, it was 10 million um, as the deposit that you had to pay um, to start a new lab.
13:59This is like how you get over the credit risk problem.
14:02Um then the cluster sizes got bigger.
14:04The sort of minimum ante into the space got bigger.
14:07Then it was 100 million.
14:09Now it's a billion.
14:10We don't think it can continue.
14:12We don't think there is a world in which you can get a $10 billion round right out the bat before you raise uh before you have revenue or traction or your product or anything.
14:20Um and what that means is that if you don't solve this problem, we are currently on track to not have any startups in AI.
14:28Um, now one of the things that you could think could happen um is maybe there can be new inference companies and new model labs.
14:35And the way that they solve the credit problem is that instead they just ship the risk to the application layer.
14:41They say look I will um I I need to get over my credit um problem.
14:47So what I'm going to do is I'm going to only sign customers who are investment grade um credit, you know, Fortune 500.
14:53Um, I'm going to only sign customers on five-year contracts for tokens.
14:58I'm going to only sign customers for tokens that require a big deposit upfront.
15:03Um, that is the end of everything.
15:07Um, if you have inference layer getting bottlenecked by credit, what happens is you can't get tokens unless you have credit or you have a big deposit.
15:18And the problem of that is that tokens are used by the whole damn economy.
15:22There is no rule that says that startups have to exist.
15:26Um if every startup is suddenly bottlenecked by credit um in order to access intelligence, what happens is existing firms just ship everything.
15:35They themselves get to make the product um because they've got credit and then they make the revenue from that product and then it makes their credit better and that's a flywheel that makes consolidation happen throughout the entire economy.
15:48If you don't fix this bottleneck, there aren't any startups.
15:51This is why we started doing this and because at one point um uh or what we think is going to happen is you're going to end up in like 3 to 5 years you're going to look out and say I want to start a new startup
16:03and to get there you need to buy so many tokens in order to compete with the existing partner or like existing players.
16:10So you need to buy a hundred million dollars of tokens.
16:14Um we think this is what you are actually going to have to do in order to start a new company because other people are spending hundreds of millions of dollars of tokens.
16:20If you just draw the line out, this is what happens.
16:23So, you need a big giant deposit because you're two people who have no idea what they're doing and so nobody trusts you.
16:29So, you're going to go and raise $100 million from your venture capitalist.
16:33Um, meanwhile, nobody wants to work with you because you're a twoerson startup.
16:37Then, you're going to be asked to pay for um that hundred or you're going to be asked to pay for a 5-year contract on that 100 million um if you're not paying it all as a deposit.
16:46And then you just don't get new model labs.
16:47You don't get new startups.
16:49um you get a consolidation in the industry.
16:51So we at one point um uh were this little lab um and what we did is we lived in San Francisco um and we knew all the other people who were also just two little folks um who had started a startup and wanted to be able to participate in the most important technology of our time.
17:09Um so I want these people to exist.
17:13I want this person to exist and I want this person to exist.
17:16I want this startup to exist and this startup to exist and this startup to exist.
17:20This one and this one and this one and this one and this one and this one and this one and this one.
17:24If you get these companies going, maybe they'll become big folks that build your cancer cures or they build something that nobody thought of um or something that wasn't just sort of AI slopped out.
17:36This is kind of the problem that compute markets solve.
17:39And if you don't solve it, we don't think there's going to be any startups.
17:43[applause]