Why Compute Still Trades in Group Chats — Affaan Mustafa & Alejandro Soumah Fernandez, Jump Trading

AI Engineer · 20 min · 215 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Um hey everyone.
00:13My name is Afaan and I'm I'm a lead for compute trading at Jump Trading, which is a proprietary trading firm, alongside my co-lead over here, Alejandro.
00:23So, today we're going to be looking over some of the market structure as it applies to compute trading, what compute trading even is, um what we can see it becoming.
00:34And I think we'll start with some macro, look at micro structure, and then talk a little bit about where we see things in policy.
00:44So, the title of the talk is the dark pools of compute, but some of you guys are in finance, so you might be confused as to why I'm using this terminology.
00:52Uh what do I mean by dark pool here?
00:54Essentially, what I'm referring to is an unlit market.
00:57Um right now, most compute trades in blocks, physical compute here, and these are private quotes.
01:02We have a lot of brokers and these are long bilateral contracts.
01:06Now, at the end of the day, these are really commercial agreements.
01:09So, we're using the term trading here to kind of like look at how it's transacted between parties.
01:14These commercial agreements tend to be some mixture of an MSA, an SLA, and an order form, cluster specific uh with hardware specific language in each kind of um in each order.
01:28And but when we're looking at pricing and looking at some of these other kind of uh metrics, a lot of the time, most of the talk is around the skew.
01:35So, what the chip itself is, the base board.
01:37So, H100, H200, B300, uh etc. And so, um a lot of the negotiation here happens kind of outside the public view.
01:47There's no, you know, like exchange for physically traded compute that's widely, you know, adopted and you know, used for everyone goes there to like take a look and trade
01:57uh compute.
01:58These are done in chats, right?
02:00They're out of the public view.
02:02Um there's public marketplaces, there's price references, there's a lot of um you know, on-demand marketplaces.
02:07However, for these actual tenured, so you know, on a given term for a fixed amount of capacity uh with a certain total contract value, these trade over Slack messages,
02:18X messages, Telegram group chats.
02:21It's a bunch of OTC kind of channels that are all over in these kind of I want to call them like neo neo chat.
02:26Um and some of you might be familiar with commodities and how a lot of you know, there's chatting happening on Ice Chat, Bloomberg Chat, etc. This is kind of where where the where the kind of markets here lie.
02:39And so I want to look at a little bit here on the macro market structure and a little bit on the players here as well.
02:45And so this was a recent kind of a16z, some of you guys might have seen this.
02:50So I'm I'm putting this out there because you know, you can go on a16z's actual X and take a look at this graph yourself.
02:56What we're looking at is that the cost of a given capability has fallen sharply.
03:01The bill of course still depends on the model usage, capacity, and other factors such as speed is a big thing right now with the new code ChatGPT models.
03:10Um cheaper tokens unlock more usage and a team can do more work for the same budget.
03:16So what we're seeing is that a lot of teams are simply just using way more tokens.
03:20So we have this latent demand that hasn't really been hit yet and people were just going to use more with the same budget that they were using before.
03:27One model's cost more and they're just expending more tokens.
03:31Um so applications that were too expensive become possible.
03:34Um however, as we're kind of talking about compute here, the capacity underneath still has to be built and paid for um even as the cost of a particular task comes down.
03:43And so the important distinction is between the price of the intelligence, the number of tasks people want to run, and the amount of hardware needed to serve them.
03:53And so this is where we get to kind of the Jevons paradox and a little of some of the stuff that we've been looking at lately where we had
04:00a very interesting thing where, you know, intuitively you'd assume that compute and inference kind of like rises together, correlates very, you know, very closely as you can see.
04:11You can see there's a little bit of a decoupling that happened and token prices declined further but compute prices have remained robust.
04:17And so like what what can be an explanation for this?
04:19Now, there's a lot of like potential explanations.
04:23One here is the whole Jevons paradox is that even though token prices and inference is getting extremely efficient to serve on these new baseboards and and and and GPUs,
04:32people are just consuming a lot more.
04:34And so the demand for compute rises and people just consume a lot more tokens even if the cost per cost per million for input and for caching and, you know, for output becomes a lot more efficient.
04:47And so making the task cheaper, encouraging the use that way resource saving per task.
04:52For inference clouds, that's going to be more applications, serving more tokens, bigger market share.
04:58And for app layer individuals, that means you guys can serve to a lot more kind of customers.
05:02So anyone who's building utilizing the token infrastructure.
05:08And so what really goes into, you know, the macro structure.
05:11Now like this is very, you know, bare bones here but you've got your chips which we've been talking about.
05:17You have power, you have buildings, grid connections and then of course you need a lot of capital to get these build outs going and then that's kind of been the whole talk here.
05:26And so um chips, powers, buildings, money, they all have to come together before, you know, you can even get to an hour of like actual usable GPU per hour compute.
05:35And so the IEA April update projects that the global global data center electricity is going to rise from 485 terawatt hours in 2025 to 915 2030.
05:45And so a lot of the talk right now is around build-outs, about getting the power grid set up.
05:51Will we have enough power grid here?
05:52Is there other energy sources that we can tap into?
05:56Where are we, you know, able to get the land to actually be able to do this?
05:59And so, colocation sites and data center operators, and people who are doing the financing the build-outs or the lenders themselves, are all kind of looking into this.
06:06And that type of everything that's upstream, essentially, of the compute is going to then flow down into the compute uh costs and also usage.
06:16And so, what does this like actual macro map look like underneath a quote chip and system vendors supply the GPU servers and networking, sites and operators supply the power,
06:27space assembly and management right here, GPUs memory servers and networking, um capacity providers, these are hyperscalers and neo clouds and any other operator that's like operating similar to how a neo cloud would operate, reselling bare metal capacity,
06:40and then the users.
06:40So, who's the actual end user?
06:42The startups, the inference clouds, uh labs, neo labs, and also anyone on the app layer and enterprises for anyone doing open weight stuff as well.
06:52And um these boxes, like I've kind of boxed them out here, but, you know, there's people can be all of the above.
07:00A chip and system vendor, like someone could do a full end-to-end build-out and also be the end user of everything.
07:06And so, there's a little bit of um variation that that that occurs here.
07:17And so, as I was kind of mentioning about the inference clouds before, they might rent a cluster and they'll serve tokens off of it and they'll use that revenue to the pay off the cluster.
07:27Um and so, that might be a use case for the actual inference cloud themselves.
07:32However, for hyperscalers, frontier labs, and larger neo clouds, they can make substantial commitments because they have the ability to get into the large equity, they have the ability to take on more debt.
07:41And uh they have the ability to do long horizon kind of deals.
07:46So, 10-year deal opening I contracted it out, etc. So, of that nature.
07:50However, anyone in the middle market isn't actually able to do this.
07:54Who's underwriting a Series B startup with enough capacity to get in a 200 million plus contract over 3 years for compute?
08:02How many equity rounds will they have to raise, right?
08:05Will they have to use their entire equity round to fund just the actual cluster and then they don't have any remaining capital left to use for actual costs?
08:14And so, we're looking at a little bit of the financing chain here and how that plays in.
08:18So, lenders are financing the buildout against contracted revenue even in many cases for inference clouds.
08:23And so, they want to finance the buildout.
08:25They want confidence that their loan will get repaid.
08:27I mean, like how are you going to underwrite a startup like a private company that doesn't have like much else going for them right now.
08:32Like what are the, you know, they're not publicly traded.
08:34There's not that much cash flows, balance sheet, income statement.
08:36They might even be pre-profit, pre-revenue, etc. And so, what they're looking at is that can the person get contracted revenue over a certain scale.
08:45And so, Coreweave's public filings give large-scale example infrastructure financing supported by take or pay customer contracts.
08:51And so, this diagram kind of shows the smaller illustrative example here.
08:56And the mismatch is that hardware financing they want very long terms, but customers are only able to afford or don't really want to like get into a long-term financing over 36 months.
09:05They have to put up on up to 15 to 25% is what Coreweave has quoted that they've seen their median prepayment to be, 15 to 25% up front on a 100 million TCV cluster.
09:17That's 15 to 25 million dollars that a startup has to put up front.
09:21That might be their entire round.
09:25And so, this is kind of a look at what I'm you know, referencing here.
09:29It's like you're raising 40 million, you need a 100 million contract over 3 years.
09:32You put down 30% prepayment.
09:34This is an illustrative example.
09:35You've got 10 10 million left to run the business, um and so you might have this kind of step function as you can see here.
09:41So, that's your contracted capacity and what you can actually in the demand is like, you know, uh below that.
09:47And so, now every time that you have to get into a new cluster, it's like tied to your equity round or whatever else it may be, but you might have to cap out and turn away customers.
09:54So, what are your options?
09:55You either turn away customers until you can get into your next, um you know, equity round so you can put down another 30% prepayment so you can get next little bit of capacity going, or do you use on demand to close the area under the gap and bridge the shortfall?
10:10That's like an That's like an option now.
10:12Now you have to look into the economics.
10:13What does that look like?
10:14What's the difference between, you know, maybe instead what they do is they go to a lender to finance the prepayment component, but if the yield on the prepayment component that they're getting financed
10:22is less than, um them just buying it out on demand at that time if they're able to get into that amount of capacity, then does the economics make sense there?
10:36And so, here are some of the operating models that we've seen.
10:39It's like you can build and operate the full stack.
10:42Hardware, site, your team is going to arrange power, permit, the assembly, the management.
10:47You can also own servers in a colocation facility with more or less the assembly and operation handled by a partner.
10:52Um on the other end, an off-take contract buys defined capacity and the provider is going to own and operate the asset, and they might not give you complete root access to the bare metal or there might be kind of like levels to how much access you get depending on if it's tenant, multi-tenant,
11:06and etc. type of like configurations that can also affect them the pricing.
11:10Um And so, some businesses are going to value the physical access, some businesses are going to value, um you know, confidential compute, uh you know, they might value certain topology changes or the ability to reconfigure the system.
11:23And so, if that is something for your business and they might actually want to have more and more control and build and operate the stack and the economics for them might make sense given the opportunity cost they would have to give up um if they were to just take rental off take contracts.
11:36So they might be willing to pay more there.
11:44And so here's how I ownership and rental is a very like crude graph.
11:48How ownership and rental can compare as utilization arises.
11:52And so what we're looking at is that usually you know an off take contract or these off take contracts are getting to the cost of actually almost building out and financing your own cluster.
12:02And it's not because I mean if you're spending a hundred million over a three year period then maybe if the economics make sense it would have made more sense to simply buy out the compute operator yourself outside of the time cost and then off the operational cost that might exist.
12:15So there's a there's a equilibrium that exists that that has to be followed where you know the ownership cost outweigh the you know rental cost and the rental cost
12:28should sit below the ownership cost.
12:30But if you're utilizing at a such a rate and you know in such variable kind of increments it might be best to just own the stack out right.
12:42And so we're seeing this new dynamic that compute the compute procurement affects how much runway a growing company actually has.
12:50And so this also gives investors an interest in computer procurement.
12:53So we're kind of seeing this.
12:54Access and payment terms affect how much runway their port codes are going to have from a VC perspective.
13:00So helping a company buy a suitable capacity can improve the underlying investment and they're very connected.
13:05Right?
13:05So they're able to procure and they're able to actually find compute capacity for their port codes and and and fund that.
13:12And so that also brings another interesting aspect to the macro structure into the micro structure is that who helps place the capacity and how does this offer get compared and what risks these contracts can can transfer.
13:26So we have a lot of different players around this GPU hour that are helping kind of shape this this market here.
13:32You've got your price discovery individuals.
13:34So, people with indexes, data, the benchmarks that we're all looking at right now and referencing, which would be used to, you know, in the future launch the futures based on the benchmark rates,
13:46you know, talked about in in the associated contracts per spec per pending approval.
13:53And then there's the physical matching venue auctions aggregators.
13:56So, anyone who can match capacity.
13:58We're looking at essentially decentralized marketplaces for on-demand compute.
14:03You're looking at liquid pools of compute that offer like liquid pool services and parted out themselves and have their own standardized contract.
14:11You're looking at bare metal resellers.
14:12You're looking at neo clouds themselves doing larger capacity sales.
14:16And then you're looking at individual brokers and individual operators here as well on the bare metal reselling.
14:21You have desk and brokers.
14:23They're looking at the bilateral quotes and execution.
14:25They might know a lot of people.
14:27They might be able to move fast.
14:28They might be able to see quotes at very similar specs in one place and know that there's a buyer that is extremely inelastic to price right now.
14:35It needs the compute immediately and they can get the RFS in time so they can charge a decent spread on it.
14:40And then what they do is they're going to have the source negotiating and then they're going to be carrying and placing the risk.
14:45They're going to have to operate that cluster for the person and it has to be worth that spread.
14:49And you know, you have looked in financial markets, the exchanges that are coming that are based on the GPU hour, the futures and derivatives.
14:56People are going to be using it to take a look at how to maybe hedge out and look into more financial structures.
15:01So, hedge out risk.
15:02So, hedge out, you know, my balance sheet risk.
15:04Maybe get into more interesting agreements.
15:07We're looking we're seeing kind of agreements such as prepayment payment facilities are coming online.
15:13We're seeing people utilize futures and are interested in utilizing futures and other kind of um um options to be able to There's demand for this to be able to hedge out the risk that they currently have on the books.
15:25We have buy side individuals looking for flex contracts, flexible contracts like hey, is there ability that if I lock myself into a 3 to 5 year contract that I'll be able to you know, at least upgrade when a new chip that comes out or maybe I'll be able to roll it over or maybe let me sell it
15:38back to you.
15:40And then there's a credit financing.
15:41So equipment lenders, leasers, specialist credit providers.
15:46And so the way we're seeing it right now is that every deal is bespoke.
15:50There's different spec sheets, there's formats.
15:51The bill of materials may not match the actual like delivery or there might be subject to change if a part is missing or part is out of stock and you know, there's like very long lead times
16:02that some of these OEM and ODM kind of providers have.
16:06And the same term can mean different things when you're negotiating very quickly.
16:09If someone's talking about memory, are we talking about you know, VRAM or we talking about the CPU memory, talking about additional storage here?
16:16Like what did you exactly mean there?
16:18It affects quite a bit what type of loads can be run on you know, the actual end users needs and also with pricing.
16:25And delivery dates.
16:27So there's a lot of back and forth before anyone can really compare and this is where the concept of like non-fungibility kind of comes into place and how to partition that out, what parts can be fungible, what parts hold the most of the weight such as the skew etc.
16:39And what parts are still like traded as if it was a product, e-commerce etc. where you know, it's only for that particular cluster that this price applies.
16:48And so prices can update daily or faster.
16:50Quotes still a good starting point but it can get chopped rather than it is changes what's on offer.
16:54There's all these kind of latent factors, hidden factors as well why we're calling it a dark market.
16:59These are some public H100 references snapshotted today.
17:04I updated them.
17:04So take a look at the what the public so those are the references for if you wanted to go on their websites to see.
17:09This is just an example here of like what a public on demand or spot pricing looks like for an H100 today.
17:18You can see kind of various various prices here.
17:20And so, people might use this or they might use their own kind of internal what they've seen to actually like assume or use it to charge what they want to charge on their actual cluster specification,
17:31taking to account every other kind of factor that goes into it.
17:36And so, why are these new structures kind of needed and what are these microstructures interesting?
17:41It's mainly to solve for these mismatches, right?
17:43Bridging capacity, more flexible commitments, resale rights, and protection against price moves.
17:48Um you know, you're selling a fixed price service from an inference cloud point of view, you'd be selling a fixed price service while the GPU bill floats and that creates price exposure.
17:57And so, locking capacity for years can solve a part of that, but it'll create idle hour idle hour exposure elsewhere as well, unless they're always using it.
18:04Reselling spare capacity only helps if someone actually can buy the hours.
18:08A financial hedge can address a reference price, but can't deliver the cluster right now in terms of cash settlement or guarantee a utilization.
18:15So, the useful structure has to start with a particular risk that the business has and needs to manage.
18:24And I'll I'll go pretty fast on this policy section because we're running out of time.
18:29Everything that a fan has said sits inside of a structure that is regulated and is affected by policy.
18:40In various ways.
18:42One way is export controls, one of the examples.
18:44In January, the US Commerce Department moved qualifying H200 class license reviews for China and Macau to case-by-case review with conditions.
18:56Furthermore, open weight models connect policy back to the US economics we started with.
19:02The US AI action plan explicitly encourages open source and open weight AI and recommends improving the financial market for compute so that startups and academics can access large-scale computing
19:14power.
19:17So, I'm going to tie this out right here, but where's this opportunity lie?
19:20We have this growing middle market.
19:22We need balance sheets that can bridge buyer and supplier needs.
19:25There needs to be some level of standardization, a standard capacity contract, common specs, common terms, and definitions.
19:31And so, growing companies with needs and commitment do not fit the contracts the hyperscalers and frontier labs can sign.
19:37Those companies have real workloads.
19:39We want them to grow.
19:40They still face this financing problem, a timing problem, a specification problem.
19:43A supplier might want the long-term commitment for the build-out while the buyer wants to expand in smaller blocks as demand becomes clearer as it's so volatile, front of curve.
19:50The buyer needs a way to avoid paying for capacity can't use yet.
19:53And so, participants with strong balance sheets that are uncorrelated with the market and the ability to assess those risks with a strong credit profile can help bridge the gap between what
20:02buyers need and what suppliers offers.
20:04That requires more than quoting the actual GPU hour.
20:07It means understanding the delivery, workload, all of that.
20:09And so, there's very few kind of players in the market, and you know, a very well-suited player can be a firm that's uncorrelated with market, with strong balance sheet, strong credit rating, an incredible
20:19amounts of hardware talent that sits inside for high-performance computing.
20:24And so, just making a CTA here, let's build this together.
20:27Thank you guys so much for attending our talk.
20:32[applause]