Why AI Startups Will Disappear Without Compute Markets — Evan Conrad, San Francisco Compute
https://www.youtube.com/watch?v=Wk2bUUGFC_o
AI Engineer · 17 min · 213 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] So, this is a talk about compute markets.
[00:15] Um, but it's kind of more of a talk about why to make a compute market.
[00:19] Um, if you don't know me, I'm Evan.
[00:22] Um, we run a company called San Francisco Compute.
[00:24] Um, we make a physically settled order, physically settled forwards market.
[00:29] Um, so if you were here for the last talk, last talk was basically why all the operators of GPU clusters um don't really work very well.
[00:37] Um, when we say physically settled, we mean we're the operator.
[00:42] We're both the operator of the hardware in the same way that a cloud is the operator of the hardware and then we're also the market at the same time.
[00:49] It's kind of like a grid for compute.
[00:51] So the general gist of this is that in order to run and operate a supercomputer um we think problems start right at the design of the supercomputer um right in the design of the bill of materials
[01:05] um it's control of the backplane fabric as well it's also control of um bare metal orchestration it's also services on top VMs and so on um there's a lot of different parts of the stack that you have to do effectively
[01:18] otherwise um the folks who have a data data center um and would like to make more money on that data center by becoming the GPU cloud um end up
[01:30] failing for the customer.
[01:31] So that's what we do.
[01:33] We offer what's called supercomputing as a service.
[01:35] We go to folks who want to become a cloud.
[01:37] That's like folks who want to become what Nvidia would call an NCP um or venture firms that want to have clusters for their portfolio um and want to be able to reallocate that capacity when a company like Instinct is taking off.
[01:50] um or AI companies who would like the residual value on the compute because they've realized that's really important to their financials um but don't want to operate a supercomputer.
[01:59] So that's physical settlement and then on top of that we run a live order book for capacity that you can buy reserved either right now or in the future.
[02:08] Um this um together is not at all what we thought we were going to be doing.
[02:14] Originally, we were this company called Junlark um that was trying to make kind of like Sunno or UDIO, but prior to Sunno or UDIO.
[02:22] Um at the time, um what you needed to do to realistically start a lab was you needed to raise a big deposit, a huge one, which was $10 million at the time.
[02:33] Um we thought that was a crazy amount of money.
[02:35] Um and so we thought, no, that's stupid.
[02:36] We're going to raise 500k and obviously what we're going to do is we're going to rent from somebody else.
[02:40] Um obviously today um it's not $10 million, it's a billion dollars.
[02:46] And um
[02:48] [snorts]
[02:48] nonetheless, um we had raised our 500k.
[02:51] We buy a big cluster and then we're underwater on our cluster every month.
[02:55] Um and if we didn't subleasase our cluster, we would have gone bankrupt.
[02:58] Um so if you've heard of GPU brokers, we were the first GPU broker because we were forced into being a GPU broker.
[03:04] So the first year of the company was us just trying to sell our cluster, otherwise we would go bankrupt.
[03:08] Um, so I would wander around with these like actual um pieces of paper at Nurups which both say H100 and A100 at the same time because that's how I made uh I made the uh poster so quickly.
[03:21] Um, and so this talk is not really about um necessarily like how a compute market works.
[03:28] It's how a compute market um gets to exist.
[03:32] Uh we like to think about it like there's this pamphlet and on the pamphlet on the front side it's like man starting a Neolab sucks.
[03:38] There's all these different like financial and economic problems and on the back of the p back of the pamphlet it says compute markets.
[03:45] It's it's not super clear how one gets to the other until you open the pamphlet and then like oh my holy there's like so much stuff in this pamphlet and it keeps going and there's so many folds.
[03:53] It's just a very difficult and challenging problem because you need to solve two like miracle problems to get working.
[03:59] You have to both make a market which is uh a uniquely difficult problem because it has a cold start problem.
[04:05] You also have to build a cloud and then you have to not become a cloud.
[04:08] Um which is a whole separate challenge.
[04:10] But these are the problems that the labs are facing and it's about to be the problems that everyone is about to facing.
[04:16] So if you work at a startup these are your problems now.
[04:18] Um right now everybody is sold out of compute.
[04:21] Um this is um probably not going to change for a while.
[04:26] Um, so that means that you're going to have to buy a new cluster, a net new one, and there's a lead time.
[04:30] You have no way to pay for that cluster to come online now.
[04:33] You're going to pay for it to come online later.
[04:35] There's nothing in the market that can realistically get that cluster on online for you now.
[04:39] Um, meanwhile, you're going to pay your entire bank balance as a deposit.
[04:43] Um, the reason you're paying your entire bank balance as a deposit is because the lender has no idea how to render under underwrite AGI.
[04:50] Um, how in the world are you supposed to do that?
[04:52] You've got two person startups in San Francisco who or maybe New York um who say I'm going to build super intelligence.
[05:00] No lender knows how to underwrite super intelligence.
[05:03] Um so they have no idea what the asset is worth.
[05:05] Um meanwhile your cluster is very likely to be delayed.
[05:09] That's because the lender needed to underwrite against the credit risk of the customer and so they needed to know who is the customer first before you deploy the cluster.
[05:16] So right when the customer signs is the riskiest point in the entire operation.
[05:21] um you wanted it to be the case that you built the cluster first and then you sell it.
[05:25] This also helps reduce all sorts of different types of problems um where clusters break.
[05:29] Meanwhile, most of the people who are setting up clusters are people who have the actual asset that makes the thing cheap.
[05:35] Like you've got the data center space already.
[05:38] What that doesn't get is supercomputing expertise.
[05:41] um as we saw in the last talk if you've um watched that one.
[05:45] Um meanwhile, you're going to run into the exact same problem that Junark did, which is you're going to either way overby or you're going to weigh underby.
[05:52] Um that's because AI doesn't have margins that look like SAS companies.
[05:56] And so you don't have very much margin for error.
[05:58] And so the chances that you either underby and somebody else wins because they did it correctly or you over buy and then you just become stability AI or you blow up.
[06:07] you um end up like character AI um are really high.
[06:11] So these are all the same problem.
[06:12] They're all just different types of risk.
[06:14] They're technical risk, their financial risk, their credit risk.
[06:16] Um they're all one big bundled risk problem.
[06:19] Uh which is the problem that we end up solving and it's also the problem that we think compute markets broadly the entire industry solves.
[06:24] Um so we'll walk through each one of these and walk how that gets to a compute market because if you don't understand these, it's kind of like unclear why you would want a compute market.
[06:32] Um, and it almost seem like this problem that somebody thinks, "Oh, wow.
[06:36] Compute markets are cool, so I'm going to go start one."
[06:38] We think they should be derived from the actual problems that people face, which is first, everyone is sold out.
[06:44] Why are they sold out?
[06:45] Because the lender couldn't finance the cluster unless you had a long-term offtake agreement.
[06:49] In CPUs, uh, this wasn't the case because CPU buyers are SAS companies.
[06:54] SAS companies have high margins.
[06:56] And because they have high margins, um you can build a product that rewards you for having um a great um that rewards you with high margins um for your um your CPUs.
[07:08] Um think about it like this way.
[07:11] If you're a SAS company and your underlying cloud doubles their prices, you don't go bankrupt.
[07:15] You hate your cloud, you're mad, but you don't go bankrupt.
[07:18] If you're an AI company and your underlying cloud doubles their prices, you go bankrupt unless you go to some other underlying cloud.
[07:24] And what that means is that you're actually price sensitive.
[07:27] And because you're actually price sensitive, um, and there's nothing keeping you at any particular cloud.
[07:32] So you can't sell on demand because your customers will immediately leave to whoever is cheaper.
[07:37] And so um, that means that in order to finance GPU clusters, you have to have long-term take-pay contracts, hopefully with someone who has investment grade credit.
[07:47] However, if you don't have liquidity on those, if you're selling reserved and your customer can't get out, then that also means that some other customer who wants the capacity right now because they're like instinct and their product is taking off can't pay for anything.
[07:58] They have to finance a new cluster which has a lead time because the physical world just exists and it's going to take a while.
[08:03] Sometimes you even have to build the damn data center.
[08:06] Um, and then because you're financing a new cluster, you're going to pay a deposit because almost nobody in AI has any reasonable credit.
[08:13] And because this is a brand new technology, all these companies are new.
[08:17] Meanwhile, even as you um start to scale up that ladder, you're still going to run into credit issues all the way through.
[08:23] It's the number one bottleneck of AI at the moment.
[08:25] It's not power.
[08:25] You hit credit way before you hit AI or way before you hit um um uh power.
[08:31] So, um what that means is every inference company, um uh every Neol, they're bottlenecked by how much cash do they have in the bank account.
[08:39] And so they go back to the venture capitalist who says, "Oh, I'm going to fund at the model layer, the application layer, the inference layer because I don't want to fund all of that capex stuff."
[08:47] Um, and so the venture capitalist then funds all the capex stuff by giving the um the new neolab or new inference company um the check that they then send to the cloud um which is the deposit for the cluster and that's how you underwrite the customer.
[09:04] So um the clusters then are delayed because um the c the cloud couldn't actually deploy the cluster until they had the customer in hand.
[09:15] And so that was the riskiest point.
[09:17] So therefore um what should have happened was the cloud had bought the cluster ahead of time and then waited until it was landed in the data center and then sold it.
[09:27] And that means you have way less likelihood of delays.
[09:31] Meanwhile, the cluster is very likely to break when you get the cluster as the customer because you don't have liquidity.
[09:37] And so, if the failure rate is higher than what you expect and you don't have the ability to transition a node in, then you don't have anywhere you can pull nodes from, which means that if you need new nodes to make up for the failed ones, you have to RMA them, which means you have to get a
[09:50] new node.
[09:51] And then that gets us back to several steps before where you have uh the lead time problem again.
[09:57] And so, everything breaks for everybody all the time.
[10:00] Once you have gotten through all of that as a new AI lab, um your problems are you you you know you nobody was sold like everyone was sold out.
[10:08] Um you then had to wait a lead time, you paid this big deposit.
[10:11] Um the thing was delayed and then it broke and finally you got through all of that.
[10:15] Um and the likelihood that you predict your demand correctly is very low.
[10:19] And so you're either going to be way overprovisioned or way underprovisioned and you have no way out.
[10:23] Um there's no way to sell your capacity to somebody else and there's no way to buy more capacity if you need it.
[10:27] So, computer markets um to solve these problems, you need to solve a bunch of different problems all at once.
[10:33] Um you need to underwrite the value of the GPU cluster rather than underwriting the value or underwriting the credit risk of the customer.
[10:40] Um this is how it could work.
[10:42] Um it's not guaranteed to work like this.
[10:45] In other industries, what happened is everything just consolidated.
[10:49] You got one big creditworthy customer that does the whole damn industry.
[10:53] That's what the path of AI is currently on track to do.
[10:57] It's instead of going to be um a liquid market where anybody can participate, it's going to look like TSMC.
[11:03] You're going to get one buyer um or a few buyers that are the clouds.
[11:07] You're going to get a few model labs that are the AI labs.
[11:10] And then you won't get other startups.
[11:12] Um this is what we think is going to happen.
[11:14] And this is why 3 years ago when we were doing our AI lab, we had a chance.
[11:19] We could have said, "No, we're not gonna do our new um we're not gonna like, you know, stick it out for several years just trying to keep our company alive.
[11:27] Um we're just going to go do a new AI lab."
[11:29] But we saw this problem and we said, "Oh if we don't do anything about this, then we will just hit this 3 years from now."
[11:37] And here we are 3 years from now hitting this problem and where everyone's going to hit this problem.
[11:42] So um as if you watched the last talk um it's very hard to do solve this problem unless you are operating both the market and the clusters at the same time.
[11:53] Um so this is Dylan Page.
[11:54] Um Dylan is a co-chair of the CNCF infrastructure group.
[11:57] That's the organization that makes Kubernetes formerly worked at um Lambda doing infrastructure there.
[12:01] Um and the reason and he joined San Francisco compute because um GPU compute isn't sufficiently standardized reliable for the market operator to be divorced from the physical infrastructure.
[12:11] It doesn't work if you aren't the one who's buying the thing.
[12:14] Sorry, it doesn't work if the person you're buying from isn't operating it.
[12:17] To give an example of why this doesn't work is I'm sure some of you are flying in from overseas or sorry um out of town um and you have a hotel room.
[12:26] Um wouldn't it be great if you could, you know, sell that hotel room to somebody else.
[12:31] I go to you and I want to buy that hotel room.
[12:34] Um and I say, "Can I get room service?
[12:37] Can I get my room cleaned?
[12:38] Um I've got a problem with my room.
[12:40] who do I call?
[12:41] Um, if you don't operate the hotel, you can't fix any of those problems.
[12:46] That's why brokering clusters doesn't work.
[12:48] You have to operate the cluster like the hotel.
[12:51] You have to run the whole damn thing.
[12:53] Otherwise, you can't actually make a market.
[12:55] It doesn't work.
[12:55] You can't do physical settlement.
[12:57] Um, so why does nobody do physical settlement?
[13:00] Um, the problem is there's a very strong incentive to never do physical settlement.
[13:04] Um, that's because if you build out physical settlement for compute, what that literally means is you're building out the entire operating capacity of a GPU cloud and if you are building out the entire operating capacity of a GPU cloud,
[13:17] you should just be a GPU cloud because it looks to be the case that you will make an ungodly amount of money.
[13:23] Um, you will sign billion dollar deal after billion dollar deal.
[13:27] And so, it.
[13:29] Let's not make a GPU or let's not make a compute market.
[13:31] let's instead just become a GPU cloud, which is what basically everybody does because the numbers are astronomical and it's a very reasonable thing to do.
[13:38] Um, the reason why we ended up building physical settlement was because we knew that if we didn't start this, um, we would then not be able to start an AI lab, which was the real damn thing that we wanted to do in the first place.
[13:48] We wanted to make a lab.
[13:49] We didn't get to make a lab unless this exists.
[13:52] So, when we started, it was 10 million um, as the deposit that you had to pay um, to start a new lab.
[13:59] This is like how you get over the credit risk problem.
[14:02] Um then the cluster sizes got bigger.
[14:04] The sort of minimum ante into the space got bigger.
[14:07] Then it was 100 million.
[14:09] Now it's a billion.
[14:10] We don't think it can continue.
[14:12] We don't think there is a world in which you can get a $10 billion round right out the bat before you raise uh before you have revenue or traction or your product or anything.
[14:20] Um and what that means is that if you don't solve this problem, we are currently on track to not have any startups in AI.
[14:28] Um, now one of the things that you could think could happen um is maybe there can be new inference companies and new model labs.
[14:35] And the way that they solve the credit problem is that instead they just ship the risk to the application layer.
[14:41] They say look I will um I I need to get over my credit um problem.
[14:47] So what I'm going to do is I'm going to only sign customers who are investment grade um credit, you know, Fortune 500.
[14:53] Um, I'm going to only sign customers on five-year contracts for tokens.
[14:58] I'm going to only sign customers for tokens that require a big deposit upfront.
[15:03] Um, that is the end of everything.
[15:07] Um, if you have inference layer getting bottlenecked by credit, what happens is you can't get tokens unless you have credit or you have a big deposit.
[15:18] And the problem of that is that tokens are used by the whole damn economy.
[15:22] There is no rule that says that startups have to exist.
[15:26] Um if every startup is suddenly bottlenecked by credit um in order to access intelligence, what happens is existing firms just ship everything.
[15:35] They themselves get to make the product um because they've got credit and then they make the revenue from that product and then it makes their credit better and that's a flywheel that makes consolidation happen throughout the entire economy.
[15:48] If you don't fix this bottleneck, there aren't any startups.
[15:51] This is why we started doing this and because at one point um uh or what we think is going to happen is you're going to end up in like 3 to 5 years you're going to look out and say I want to start a new startup
[16:03] and to get there you need to buy so many tokens in order to compete with the existing partner or like existing players.
[16:10] So you need to buy a hundred million dollars of tokens.
[16:14] Um we think this is what you are actually going to have to do in order to start a new company because other people are spending hundreds of millions of dollars of tokens.
[16:20] If you just draw the line out, this is what happens.
[16:23] So, you need a big giant deposit because you're two people who have no idea what they're doing and so nobody trusts you.
[16:29] So, you're going to go and raise $100 million from your venture capitalist.
[16:33] Um, meanwhile, nobody wants to work with you because you're a twoerson startup.
[16:37] Then, you're going to be asked to pay for um that hundred or you're going to be asked to pay for a 5-year contract on that 100 million um if you're not paying it all as a deposit.
[16:46] And then you just don't get new model labs.
[16:47] You don't get new startups.
[16:49] um you get a consolidation in the industry.
[16:51] So we at one point um uh were this little lab um and what we did is we lived in San Francisco um and we knew all the other people who were also just two little folks um who had started a startup and wanted to be able to participate in the most important technology of our time.
[17:09] Um so I want these people to exist.
[17:13] I want this person to exist and I want this person to exist.
[17:16] I want this startup to exist and this startup to exist and this startup to exist.
[17:20] This one and this one and this one and this one and this one and this one and this one and this one.
[17:24] If you get these companies going, maybe they'll become big folks that build your cancer cures or they build something that nobody thought of um or something that wasn't just sort of AI slopped out.
[17:36] This is kind of the problem that compute markets solve.
[17:39] And if you don't solve it, we don't think there's going to be any startups.
[17:43] [applause]