WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=eZ8WWZzoaR0 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/eZ8WWZzoaR0.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.559 --> 00:00:21.119 Please welcome to the stage the professor emmeritus at Stanford University, John Sterhout. s3 00:00:37.840 --> 00:00:38.480 Good morning. s4 00:00:38.480 --> 00:00:48.719 It's really great to be here to talk about the network side of AI applications and in particular to make the case that latency matters and it's probably going to be mattering more in the future. s5 00:00:48.719 --> 00:00:51.360 But I just want to say this is a talk is unusual for me. s6 00:00:51.360 --> 00:00:55.520 I've never before given a talk where there are fog generators in the auditorium. s7 00:00:55.520 --> 00:01:00.079 Just a really San Francisco experience, I guess. s8 00:01:01.600 --> 00:01:07.600 So, it's it's well known that AI workloads depend on really great networking performance in order to achieve their own performance. s9 00:01:07.600 --> 00:01:15.200 And of course, that's because the workloads are so large that they have to be distributed across machines and then you have to communicate between the machines. s10 00:01:15.200 --> 00:01:19.600 But what I want to talk about today is that it seems that those workloads are changing. s11 00:01:19.600 --> 00:01:22.880 And so I hope to do three things over the next 15 or 20 minutes. s12 00:01:22.880 --> 00:01:34.240 First to convince you that in fact the workloads are changing and that whereas the workloads used to be completely dominated by large transfers where throughput is the key metric that matters s13 00:01:34.240 --> 00:01:40.640 that we're seeing more and more smaller transfers where the latency is crucial. s14 00:01:40.640 --> 00:01:47.840 The second thing I hope to do is to convince you that legacy protocols like TCP and RDMMA are poorly suited to this environment. s15 00:01:47.840 --> 00:01:56.000 They weren't designed for this environment and unfortunately they suffer from very high tail latency when you mix small messages with large ones. s16 00:01:56.000 --> 00:01:58.960 I'll talk a little bit about why that's the case. s17 00:01:58.960 --> 00:02:14.400 Then third, I'd like to introduce Homo which is a new protocol we've developed at Stanford that actually was designed in a clean slate redesign to handle data center workloads like these and in fact it does quite well on those workloads and can reduce tail latency by an order of magnitude or more. s18 00:02:14.400 --> 00:02:16.959 So I'll tell you a little bit about Homa. s19 00:02:16.959 --> 00:02:18.720 So let's dive in. s20 00:02:18.720 --> 00:02:20.560 First, workloads. s21 00:02:20.560 --> 00:02:27.599 Historically, AI workloads have consisted of enormous transfers between machines, and that's all that really mattered. s22 00:02:27.599 --> 00:02:32.480 Gigabytes of data for things like weight gradients and and so on. s23 00:02:32.480 --> 00:02:39.519 In these workloads, what you really care about is throughput, how many gigabits per second you can pump through the pipes. s24 00:02:39.519 --> 00:02:48.879 And these are relatively easy workloads for networks because if it takes a while to set up the connection and start the transfer, it doesn't matter. s25 00:02:48.879 --> 00:02:53.760 The transfers go on for so long that all that really matters is the throughput. s26 00:02:53.760 --> 00:02:57.120 And so in these environments, TCP and RDMA perform pretty well. s27 00:02:57.120 --> 00:03:08.080 Uh, by the way, when I say RDMA, what I really mean is Rocky RDMA over converged Ethernet, which is the underlying transport that's used by RDMA for most purposes today. s28 00:03:08.080 --> 00:03:16.400 So anyhow, the old workloads, big transfers, throughput matters, uh the legacy protocols work pretty well. s29 00:03:16.480 --> 00:03:19.200 However, it appears that the workloads are changing. s30 00:03:19.200 --> 00:03:25.120 They're becoming more granular with smaller chunks of computation and smaller exchanges of data. s31 00:03:25.120 --> 00:03:30.640 And this seems to be particularly true in the world of inference and also in agentic workloads. s32 00:03:30.640 --> 00:03:34.239 Not so much for training workloads are still massive transfers. s33 00:03:34.239 --> 00:03:48.480 And so what's happening is that more and more there are small message exchanges typically for things like metadata and coordination such as checking to see if a particular entry is present in a KV cache that's distributed or doing barrier synchronization at the end of s34 00:03:48.480 --> 00:03:50.000 periods of compute. s35 00:03:50.000 --> 00:03:54.239 And for these workloads what really matters is latency. s36 00:03:54.239 --> 00:04:03.120 That is what's the roundtrip time to send some small piece of data across the network do a little bit of computation and get a small result back again. s37 00:04:03.120 --> 00:04:07.439 And in fact, it isn't just just latency or average latency that matters. s38 00:04:07.439 --> 00:04:09.760 What really matters is tail latency. s39 00:04:09.760 --> 00:04:15.920 That is, you'd like to know that if we send a whole lot of small messages, all of them will complete quickly. s40 00:04:15.920 --> 00:04:20.400 So, for example, we typically measure things like 99th percentile latency. s41 00:04:20.400 --> 00:04:24.560 And if we have high tail latency, that can limit the overall throughput of the system. s42 00:04:24.560 --> 00:04:26.639 So, here's an example. s43 00:04:26.639 --> 00:04:37.360 Suppose a common thing is to take a workload and split it up across several nodes which do intensive computation using their GPUs for some period of time and then once they've all finished their computation s44 00:04:37.360 --> 00:04:49.759 you do some small exchange between the nodes exchange data meta data and then it'll go on to the next round of computation and while that exchange is happening that synchronization is happening the GPUs s45 00:04:49.759 --> 00:04:56.960 are sitting idle so if even one of those exchanges takes a long time. s46 00:04:56.960 --> 00:04:58.960 It turns out the whole process stalls. s47 00:04:58.960 --> 00:05:04.639 You need all of those exchanges to complete before you can go on to the next phase of computation. s48 00:05:04.639 --> 00:05:11.440 Now, if the computation phase is say 5 seconds and it takes a few milliseconds for the exchange, you know, not a not a problem. s49 00:05:11.440 --> 00:05:13.600 And that's historically what it's been. s50 00:05:13.600 --> 00:05:24.880 But now with agentic workloads where you're trying to pump out tokens relatively rapidly at a regular rate, the periods of computation are getting down into sort of the millisecond time scale. s51 00:05:24.880 --> 00:05:36.160 And if it also takes milliseconds to do that synchronization, then you're wasting a significant fraction of your GPU resources waiting for the the synchronization to occur. s52 00:05:36.160 --> 00:05:36.800 So I'm curious. s53 00:05:36.800 --> 00:05:39.199 I'd like to just do a a quick audience poll here. s54 00:05:39.199 --> 00:05:47.360 Is there anybody here where you have reason to believe that the latency of small messages is impacting the overall throughput of your applications? s55 00:05:47.360 --> 00:05:49.199 If so, can you just raise your hand? s56 00:05:49.199 --> 00:05:51.600 See, is there anybody out there today? s57 00:05:51.600 --> 00:05:53.600 Actually, more hands than I expected. s58 00:05:53.600 --> 00:05:56.000 So quite a few people out there are raising their hands. s59 00:05:56.000 --> 00:06:01.280 I think this problem is likely to get worse as the trends continue. s60 00:06:01.280 --> 00:06:02.639 So what's going on? s61 00:06:02.639 --> 00:06:05.199 Why is tail latency bad? s62 00:06:05.199 --> 00:06:10.400 Well, typically the cause is congestion resulting from incast. s63 00:06:10.400 --> 00:06:18.560 So incast is when several nodes all decide simultaneously to transfer data to some destination node. s64 00:06:18.560 --> 00:06:23.520 And if they all send large messages, well, the links are the same everywhere in the network. s65 00:06:23.520 --> 00:06:28.560 So three nodes can transfer three times as fast as one node can possibly receive. s66 00:06:28.560 --> 00:06:40.479 And so what happens is that packets accumulate at the last hop going to that destination in the top of rack switch at its egress port for the destination node. s67 00:06:40.479 --> 00:06:49.759 Then if some other node decides it wants to send a short message to that same destination, the short message gets stuck behind the long ones in the queue there. s68 00:06:49.759 --> 00:07:03.599 And actually that causes delay and in the worst case so many packets arrive that the switch runs out of buffer space that it has to drop packets and then there are you timeouts and retransmissions that make everything even worse. s69 00:07:03.599 --> 00:07:10.000 So somehow we need some way to reduce the congestion in those cues. s70 00:07:10.000 --> 00:07:17.840 Somehow we have to get the sending nodes to stop sending so fast so the cues don't just build up without limit. s71 00:07:18.000 --> 00:07:29.520 So the way this is done historically virtually all network protocols before HOMA including TCP and RDMA congestion control is the responsibility of the sender. s72 00:07:29.520 --> 00:07:34.800 So senders somehow have to figure out that congestion is happening and they have to slow down their rate of transmission. s73 00:07:34.800 --> 00:07:37.039 Now you you might wonder why are senders doing it? s74 00:07:37.039 --> 00:07:40.479 because the congestion is way over at the other end of the data center network. s75 00:07:40.479 --> 00:07:43.280 How does the the sender find out? s76 00:07:43.280 --> 00:07:49.120 Well, in the in the old really old days, the way they would find out is the cues would overflow and packets would get dropped. s77 00:07:49.120 --> 00:07:57.440 The sender would detect the packets got lost because it wouldn't get acknowledgements back and it would assume that means there's congestion and then slow down its rate of transfer. s78 00:07:57.440 --> 00:07:58.560 That's really expensive. s79 00:07:58.560 --> 00:08:04.319 So today there are better techniques that mostly involve the switches providing information. s80 00:08:04.319 --> 00:08:16.879 So a top of rack switch when it sees that the Q length for an ingress port has reached some threshold starting to fill long before the queue overflows it starts marking all of the packets that pass through s81 00:08:16.879 --> 00:08:20.479 with what's called early congestion notification ECN marking. s82 00:08:20.479 --> 00:08:26.240 And so when those packets pass through to the receiver, the receiver sees the marking in the packets. s83 00:08:26.240 --> 00:08:34.719 And then when it communicates back to the sender next, for example, to send an acknowledgement, then it includes that marking that goes back to the sender. s84 00:08:34.719 --> 00:08:38.479 And now the sender sees that the sender realizes, oh, there's congestion someplace. s85 00:08:38.479 --> 00:08:41.760 I've got to slow down my rate of transmission. s86 00:08:41.760 --> 00:08:43.120 So that's the basic idea. s87 00:08:43.120 --> 00:08:45.440 Unfortunately, getting this right is really hard. s88 00:08:45.440 --> 00:08:45.920 Really hard. s89 00:08:45.920 --> 00:08:52.399 It's very hard for the congesture to figure out exactly how to set its rates because it gets one bit of information. s90 00:08:52.399 --> 00:08:57.360 There's congestion someplace and there are multiple senders all sending to the same destination. s91 00:08:57.360 --> 00:09:00.240 They're all trying to make adjustments simultaneously. s92 00:09:00.240 --> 00:09:04.640 How much do you cut back and how do I know when I can ramp up again? s93 00:09:04.640 --> 00:09:09.440 And even worse, it's really hard to do this in a way that's stable because there's control lag. s94 00:09:09.440 --> 00:09:14.320 That is, it takes time before the sender finds out that there's congestion. s95 00:09:14.320 --> 00:09:24.480 And in fact, using this process, it typically takes several round trips for the sender to gradually adjust its rate to get just the right rate to match the available bandwidth. s96 00:09:24.480 --> 00:09:31.040 But by the time you do that in a network that things have changed, new transmissions have started or old ones have finished. s97 00:09:31.040 --> 00:09:33.279 And so these systems tend to never stabilize. s98 00:09:33.279 --> 00:09:37.600 They're constantly oscillating between sending too much and and sending too little. s99 00:09:37.600 --> 00:09:39.920 Now, this problem's been around for a long time. s100 00:09:39.920 --> 00:09:42.720 It's been known in the research community for more than 20 years now. s101 00:09:42.720 --> 00:09:45.360 There have been tons of papers published on it. s102 00:09:45.360 --> 00:09:47.040 There have been some improvements made. s103 00:09:47.040 --> 00:09:50.959 That's undeniable, but we're still a long ways from anything that works well. s104 00:09:50.959 --> 00:09:55.120 And the problem is is with the fundamental nature of it doing the the congestion control on the center side. s105 00:09:55.120 --> 00:09:57.360 It just doesn't work very well. s106 00:09:57.360 --> 00:09:59.279 So you end up with a lot of Q buildup. s107 00:09:59.279 --> 00:10:07.440 And in fact, you can see the only way to find out that there's congestion is if there's cues and so by that point, we're already experiencing delays. s108 00:10:07.440 --> 00:10:09.360 So that's a problem. s109 00:10:09.360 --> 00:10:18.720 There's one other problem with TCP and RDMA also is that their their basic data model is a bite stream just a stream of bytes with no differentiation in it. s110 00:10:18.720 --> 00:10:30.959 So if you send a series of messages say through a TCP socket they get serialized into that stream and on this slide I've you know I've shown the messages appear like they have different colors in the stream. s111 00:10:30.959 --> 00:10:32.720 Well there are no colors in real life. s112 00:10:32.720 --> 00:10:35.600 TCP has no idea where the message boundaries are. s113 00:10:35.600 --> 00:10:37.760 And that also makes life hard. s114 00:10:37.760 --> 00:10:40.320 For example, you don't know how much more data is coming. s115 00:10:40.320 --> 00:10:44.000 If you knew how big the message was, you know how much more is coming. s116 00:10:44.000 --> 00:10:47.279 And you can't prioritize short messages, which we'd really like to do. s117 00:10:47.279 --> 00:10:49.600 Get the short messages through faster. s118 00:10:49.600 --> 00:11:05.760 And you can end up with what's called head of line blocking where somebody sends a series of messages to the same destination and they send two really large ones and then a small one after that that gets stuck behind them in that stream and so it gets delayed and again you have tail latency issues. s119 00:11:05.760 --> 00:11:11.760 So all in all, TCP and RDMMA are just not well suited to this environment. s120 00:11:11.760 --> 00:11:13.279 So what do we do? s121 00:11:13.279 --> 00:11:22.079 Well, what I'd like to do next is tell you about a new protocol called HOMA that we've developed at Stanford, which was based on a completely clean slate redesign for network transport. s122 00:11:22.079 --> 00:11:28.079 If you could start from scratch and rethink how you do transport for data centers, how would you do it? s123 00:11:28.079 --> 00:11:34.320 And it turns out in home of virtually every major design decision is different from TCP and RDMA. s124 00:11:34.320 --> 00:11:41.360 TCP for all the amazing things it's done is just not a good match to today's data centers nor RDMA. s125 00:11:41.360 --> 00:11:51.440 So what Homa does particularly well is to manage the combination of large and small messages and to make sure that the messages have short messages have really low latency. s126 00:11:51.440 --> 00:12:03.680 So this started off as a PhD dissertation for one of my students, Benam Montazeri, and then the results were so great that I decided to make it my personal project to see if we could get it out of the lab and into production. s127 00:12:03.680 --> 00:12:07.360 Uh, as you may know, I'm I'm not like most professors and that I love to code. s128 00:12:07.360 --> 00:12:09.839 And so I turned this into my my own programming project. s129 00:12:09.839 --> 00:12:12.399 I have created a kernel module for Linux. s130 00:12:12.399 --> 00:12:15.600 I'm currently working through the process of getting that upstreamed into the kernel. s131 00:12:15.600 --> 00:12:18.079 It's available on GitHub for download. s132 00:12:18.079 --> 00:12:21.600 So let me tell you just a little bit about how Homa works. s133 00:12:21.600 --> 00:12:23.680 I want to mention three things. s134 00:12:23.680 --> 00:12:27.363 First, it's messagebased, not streambased. s135 00:12:27.363 --> 00:12:33.519 [clears throat] In fact, the fundamental unit at Homa is a remote procedure call which consists of two things. s136 00:12:33.519 --> 00:12:42.320 A request message sent from a client to a server and then a response message returned back from the server to the client. s137 00:12:42.320 --> 00:12:46.399 So the key thing here is that Homa knows about message lengths. s138 00:12:46.399 --> 00:12:49.680 They're buried in the transport all the way down to the bottom. s139 00:12:49.680 --> 00:12:51.920 And this has a bunch of advantages. s140 00:12:51.920 --> 00:12:54.720 First, it allows us to predict the future. s141 00:12:54.720 --> 00:13:00.800 As soon as a receiver gets the first packet of a message, it knows exactly how much more data the sender wants to send. s142 00:13:00.800 --> 00:13:05.600 And that's so has so much more information for doing congestion control. s143 00:13:05.600 --> 00:13:08.480 Second, Home Prioritizes shorter messages. s144 00:13:08.480 --> 00:13:15.839 It uses SRPT, shortest remaining processing time first to try and prioritize shorter messages. s145 00:13:15.839 --> 00:13:22.480 And third, because messages are all independent, they're not serialized into a stream. s146 00:13:22.480 --> 00:13:23.600 Every message is independent. s147 00:13:23.600 --> 00:13:29.360 Shorter messages can bypass long ones so they don't get cued behind long messages. s148 00:13:30.000 --> 00:13:35.360 The second thing about Homa that's different is that it controls congestion from the receiver. s149 00:13:35.360 --> 00:13:42.320 Now, when you think about it, this makes sense because the congestion happens primarily at that last down link to the receiver. s150 00:13:42.320 --> 00:13:44.959 And so, the receiver has way more information. s151 00:13:44.959 --> 00:13:50.079 In fact, with Homer, as soon as it gets the first packet of a message, it knows exactly how much more is coming. s152 00:13:50.079 --> 00:14:00.399 So, it has essentially complete information about congestion and it can therefore respond to congestion much more quickly and much more precisely. s153 00:14:00.800 --> 00:14:13.040 The way things work with HOMA is that when a sender has a message to send, it breaks it up into packets, but it only transmits the first few packets, those are called unscheduled packets, to the receiver. s154 00:14:13.040 --> 00:14:19.279 Packets after that are called scheduled packets and they only get transmitted when the receiver asks for them. s155 00:14:19.279 --> 00:14:22.480 So the receiver will send grant packets back. s156 00:14:22.480 --> 00:14:29.839 It'll paste them out and send those back to the sender over time telling the sender it's now time for you to send me the next chunk of data. s157 00:14:29.839 --> 00:14:32.320 And the receiver can delay those grants. s158 00:14:32.320 --> 00:14:42.480 So for example, if the receiver has 10 messages that are incoming, there's no point in sending grants to all 10 of them because then you'll just get congestion in the in the top of wrap cues. s159 00:14:42.480 --> 00:14:51.600 So it can use the grants to reduce congestion and then it can also use the grants to give preference to its most favorite messages which would be the shorter ones. s160 00:14:51.600 --> 00:14:56.800 So it's a way of of implementing SRPT by favoring short messages. s161 00:14:57.519 --> 00:15:02.880 The third aspect of Homa is that it takes advantage of the priority cues in modern switches. s162 00:15:02.880 --> 00:15:19.760 So modern data center switches have more than one Q at each egress port typically eight and they can be used in a priority mechanism where packets get transmitted preferentially from the highest priority Q. So I've shown only two cues on the slide here but typically there's more than that s163 00:15:19.760 --> 00:15:31.199 you can specify in packets using the various fields of the packet you can specify which queue it should go into and so Homa dynamically makes those choices in a way to give priority to shorter messages. s164 00:15:31.199 --> 00:15:38.160 So if we go back to the incast example from a few slides ago, all of those long messages will pile up in the lowest priority queue. s165 00:15:38.160 --> 00:15:42.000 But if there's a short message coming, it will use a higher priority queue. s166 00:15:42.000 --> 00:15:52.560 And so it will immediately bypass all of the cued packets from the short from the uh the longer messages and get through to the destination more quickly. s167 00:15:52.639 --> 00:15:54.959 So how much of a difference does this make? s168 00:15:54.959 --> 00:15:56.000 Here's a this slide. s169 00:15:56.000 --> 00:16:02.000 I've got one sample benchmark that I use as part of my tuning and evaluation of Homa. s170 00:16:02.000 --> 00:16:10.240 It consists of a workload of a bunch of machines on a network that are exchanging messages back and forth of different sizes ranging from very small to very large. s171 00:16:10.240 --> 00:16:16.720 And on this graph you can see on the x-axis is the message length from about 50 bytes up to a megabyte. s172 00:16:16.720 --> 00:16:20.720 The yaxis shows you the roundtrip time for messages of that length. s173 00:16:20.720 --> 00:16:24.480 So this request this uses request and response messages that are the same length. s174 00:16:24.480 --> 00:16:29.839 You can see TCP in green, Homa in blue, and the Y-axis is is roundtrip time. s175 00:16:29.839 --> 00:16:31.920 So lower is better. s176 00:16:31.920 --> 00:16:34.800 And for each protocol, I've got two curves. s177 00:16:34.800 --> 00:16:36.639 One curve is the P50 curve. s178 00:16:36.639 --> 00:16:40.320 That's the median latency for messages of this length. s179 00:16:40.320 --> 00:16:46.800 And then P99 is the 99th percentile, i.e. tail latency for messages of this length. s180 00:16:46.800 --> 00:16:48.240 So I want to point out two things. s181 00:16:48.240 --> 00:16:52.560 First, the P99 for short messages is dramatically better for HOM. s182 00:16:52.560 --> 00:16:55.519 So with TCP it's more than a millisecond tail latency. s183 00:16:55.519 --> 00:17:00.000 Home is less than 100 microsconds about 13 times faster. s184 00:17:00.079 --> 00:17:09.360 Second interestingly you might think that because Homa favors shorter messages that long messages suffer and get worse performance. s185 00:17:09.360 --> 00:17:11.679 It turns out that's actually not the case. s186 00:17:11.679 --> 00:17:16.640 Even on the longest messages, Hom is almost a factor of two better than TCP. s187 00:17:16.640 --> 00:17:26.880 I don't have time to explain that today, but it has to do with the fact that HOMA uses run to completion approaches which are much more effective than fair than the fair scheduling used by TCP. s188 00:17:27.919 --> 00:17:34.160 So, just to wrap up, the role of short messages in AI appears to be increasing. s189 00:17:34.160 --> 00:17:37.360 I think I think it's likely that it's going to continue to increase. s190 00:17:37.360 --> 00:17:39.840 We'll see over the next year or two if that happens. s191 00:17:39.840 --> 00:17:41.520 And I just want to pose a question to you. s192 00:17:41.520 --> 00:17:52.000 You know, as you're running your applications and measuring performance and seeing what the bottlenecks are, ask yourself, is high latency for short messages affecting your throughput? s193 00:17:52.000 --> 00:17:55.840 If the answer is yes, then just know there is a solution available. s194 00:17:55.840 --> 00:17:57.360 You should give HOMA a try. s195 00:17:57.360 --> 00:18:00.720 You can probably reduce your tail latency by an order of magnitude or more. s196 00:18:00.720 --> 00:18:04.400 And by the way, this is Home is basically my life mission right now. s197 00:18:04.400 --> 00:18:06.240 I'm sort of semi-retired from Stanford. s198 00:18:06.240 --> 00:18:10.480 The reason I did that is so I can spend 100% of my time hacking on Homa. s199 00:18:10.480 --> 00:18:14.720 So I'd be delighted to work with you and help you if you decide you want to experiment with Homa. s200 00:18:14.720 --> 00:18:21.760 If you need help getting started, answer questions, bug fixes, whatever, you know, I'd be happy to work with you to try and make you successful with it. s201 00:18:21.760 --> 00:18:24.160 So if that is interesting, feel free to contact me. s202 00:18:24.160 --> 00:18:28.480 My email is on the slide or you can Google me too and find me over the internet. s203 00:18:28.480 --> 00:18:33.360 So thanks very much for listening and hope to hear from some of you. s204 00:18:45.302 --> 00:18:47.302 [music]