AI Networking is CRAZY!! (but is it fast enough?)

0:00 We've got a problem. The network is slowing down AI and it's getting mad at us,

0:04 I think. But seriously, AI models like chat, GPT have to be trained.

0:08 They need some knowledge before they can start helping us or take over the

0:11 world, whichever comes first. But how they're trained, this is crazy.

0:14 And also where our problem lies. So picture this,

0:17 thousands of insanely powerful and expensive GPU servers.

0:21 These things are like $350,000 a pop and they're clustered together,

0:24 crunching through hundreds of terabytes of data. Time is money here.

0:27 And also our enemy, we're tweaking everything to make the training go faster.

0:31 We need that GPT five. And even though we're rocking the most powerful GPUs,

0:34 money can buy, it doesn't matter.

0:36 The network can and sometimes is the bottleneck.

0:40 It's like driving a Lamborghini and rush hour traffic.

0:42 A training process relies on these roads or network connections.

0:46 And not only are we transferring just insane amounts of data from our storage

0:49 servers to our GPU servers, this is Mike. Massive amounts of data.

0:53 But the GPUs are also communicating constantly with each other,

0:56 working together to complete jobs.

0:58 But if you have any stragglers or traffic jams, that slows down everyone.

1:02 That job completion time, that lag even has a name. It's called tail latency.

1:06 The network can make or break. AI training made a claim that last year,

1:10 33% of AI ML elapsed time was spent waiting on the network.

1:15 So we've got to do something because at this point AI is taking over the world

1:18 Skynet. It's inevitable. And all we can do is really just help it along,

1:22 score some points. Maybe we'll make nice pets or a battery. I don't know.

1:26 Put me somewhere nice and give me a juicy steak. Now, before we dive in,

1:28 you got to know it's not the network's fault, it's AI's fault.

1:31 It's their big beefy GPU servers. Did it hear me say that?

1:34 If you look inside a data center, the networks are amazing.

1:37 I mean most of the time, Netflix, Facebook, Twitter,

1:40 all these things are still up. And they work really fast. Thank you.

1:42 Data centers. But ai, that's tricky.

1:46 AI came in and did some weird stuff.

1:48 Not only does it generate just an insane amount of network traffic,

1:51 like ridiculous amounts,

1:53 it's also kind of weird traffic stuff we're not used to. And honestly,

1:56 we didn't build our networks for this. Now it's not all doom and gloom.

1:59 We actually have built new types of networks to deal with AI traffic.

2:03 It's a little thing called InfiniBand, and that's a whole conversation.

2:06 But honestly, I'm not happy with it. I don't think it's our solution.

2:09 So in this video, I want to break that down.

2:10 Why is AI traffic weird and what are we doing about it?

2:13 And by I mean companies like Juniper, not me,

2:15 they are the sponsor of this video and assistant to the regional AI overlord.

2:19 They're scoring some serious points and they're helping us win the war. Oh,

2:22 you didn't hear about that war? Oh, not that war.

2:24 The AI networking wars and they're intense and finna band versus ethernet.

2:29 Scared to even talk about it, but we're going to talk about it. So part one,

2:31 here we go. Let's talk about the infestation. Why do I say infestation? Well,

2:34 because AI is kind of taking over data centers and it's becoming kind of a toxic

2:38 work environment. And it's not just networking power's taken a hit.

2:41 These data centers were not built to handle the amount of power these GPU

2:44 servers need. They're hungry.

2:45 One GPU server can consume the power allotted for an entire rack of servers.

2:50 Cooling is an issue too.

2:51 Before we could just simply air cool all of our stuff with fans,

2:54 but now we're having to look at liquid cooling. Are you serious?

2:57 Like liquid cooling in a data center? But why is that a problem?

2:59 Why are data centers kind of like what do we do? Well,

3:01 it's because data centers were meant for CPU workloads.

3:04 That's how we built them. But then something big happened,

3:06 the industry figured out that the same GPUs that give you amazing graphics and

3:10 heat your mom's basement up to face melting temperatures are also kind of

3:14 amazing at AI stuff. Things like learning and inference workloads. Yeah,

3:18 it's like a hundred times faster at that than a CPU. So the industry said,

3:21 we'll take 20,000. They loaded up GPU servers inside data centers like crazy.

3:25 Everyone's trying to have the best AI and it takes a lot of GPUs to do that.

3:29 And look at these servers. Some of these have eight GPUs.

3:31 No wonder we're having power and cooling issues.

3:33 And then of course we have the networking. Now again,

3:36 the networking in most data centers is amazingly fast,

3:38 but AI is kind of a strange beast and it all has to do with AI's training,

3:42 how it learns. And just like college, it's super expensive. You see,

3:44 for an AI model to mature or grow up, there are three steps. First,

3:48 we gather a ton of data and in the case of things like chat, GBT,

3:51 we're talking hundreds of terabytes of data, Reddit, post diary entries,

3:55 normal stuff like that.

3:56 I'm very hungry.

3:58 And then we take all of that data and we spoonfeed it to the GPU servers.

4:03 We're saying here,

4:03 kid learn and it's so much and we're giving it to him so fast.

4:06 These links are incredibly fast. And then after it's done eating,

4:09 I think that's where my analogy ends. It's trained up, it's good.

4:13 I know everything.

4:17 It's an inference mode. It's ready to help us.

4:19 It's ready to help you with that next soup recipe. I dunno what you're doing.

4:22 Now, the first big problem with the training process is just the sheer amount of

4:25 traffic we're dealing with here. And it's not just that AI networks,

4:28 they can't deal with latency or a loss.

4:31 So essentially what we need is infinite bandwidth with no latency or loss

4:34 something. Our typical ethernet networks running T-C-B-I-P really couldn't do.

4:38 So in came a new player called Infinite Band.

4:40 Infinite Band is meant for high performance computing or HPC.

4:44 And honestly it's just brilliant marketing, infinite band,

4:47 infinite bandwidth genius. And it offered almost just that.

4:50 Right now we're seeing 400 gigabit ethernet within InfiniBand and InfiniBand.

4:55 It's different. It was purpose built for HPC environments.

4:58 Things like AI training and if you're familiar with networking,

5:01 it doesn't use T-C-P-I-P, it has its own stack, its own transport protocol,

5:05 which does away with a lot of the overhead that TCP IP introduces.

5:09 So less time is spent processing data packets.

5:11 It's designed to be a lossless fabric,

5:13 which means it's not going to be dropping packets or retransmitting packets.

5:16 And the biggest thing it does and what makes it really powerful is a little

5:20 thing called RDMA,

5:21 remote direct memory access is kind of like that meme with the kid that's

5:25 skipping all the stairs.

5:26 That's what it's doing for our networks and how it transfers data.

5:29 It obliterates so much latency, it's kind of awesome. And honestly,

5:32 latency was a really big issue, which is why RDMA became a thing. Now,

5:35 where is a latency in a typical network with servers?

5:38 It normally looks like this.

5:39 Let's say we have our storage server with all your Reddit posts and diary

5:42 entries and YouTube videos, all the stuff. And it's transferring that stuff,

5:46 that data to A GPU server first,

5:48 that GPU server will request data from that server using the TCP IP stack.

5:52 Keeping in mind this is like a regular network, not infin band.

5:54 Then the CPU of that storage server will process that data,

5:57 retrieving it from memory,

5:58 and then that retrieved data will travel through the network stack of the

6:01 storage server's os, which involves multiple layers of processing.

6:04 You can kind of see that latency stacking up a little bit.

6:06 The data is then sent over the network. So think ethernet cables and switches.

6:10 And this again is using T-C-P-I-P and all the overhead that it might introduce.

6:14 And then it arrives at the GPU server and it kind of does the same thing in

6:17 reverse.

6:17 It goes to the server's network stack arrives in memory and the CP will process

6:21 that data and it's transferred to the GPU's memory.

6:23 Now this is how most systems work.

6:24 Like it's how your computer transfers data and in most cases it does work fine.

6:28 But in the case of AI with a zero tolerance for latency and the massive amount

6:32 of data, it becomes so painful and noticeable latency was a huge problem.

6:37 So here comes RDMA and RDMA is so cool.

6:40 It's all about removing roadblocks and removing latency.

6:43 And here's what it looks like. It's pure magic.

6:45 The GPU server will first initiate an RDMA operation,

6:48 which will allow it to skip a few stairs and by a few,

6:50 I mean a whole stinking lot directly accessing the system memory of the stored

6:54 server bypassing the CPU and the OS kernels of both servers. Think about that.

6:59 This is a massive deal because the data's not going through the usual network

7:02 stack in its processing,

7:03 which involves multiple layers of copying and contact switching.

7:05 The data is simply transferred directly from the memory of the stored server to

7:10 the memory of the GPU server. It's just like, Hey, there you go. That simple.

7:13 And it's made even faster because of the underlying network InfiniBand.

7:16 They're probably thinking, Chuck, I don't see a problem.

7:18 It sounds like we already solved everything. And I would say for some people,

7:21 if you've got the money and the know-how and the staff and you don't mind

7:24 overhauling a bit of your network and dealing with a bit of headaches.

7:27 And that brings in some problems I have within InfiniBand because while it does

7:30 sound great, it has some major things for me that I don't like. Number one,

7:33 it's different.

7:34 And there's nothing wrong with different except in the case where it has

7:36 different routers, different switches, different connections, and nicks,

7:40 it has its own stack. It's basically a different type of networking,

7:43 which could be fun to learn honestly.

7:44 But not a lot of network engineers know it, especially compared to ethernet.

7:47 I mean as a network engineer myself and any other network engineer,

7:51 you cut your teeth on ethernet it. That's what everybody runs.

7:54 So as someone who's trying to spin up AI networking in their data center,

7:57 it's going to be kind of hard to find talent staff to manage that for you

8:01 because not everyone knows Infin band.

8:03 And it can be complex to set up with compatibility being a huge issue because I

8:07 mean, right now most people are running ethernet and if they make the switch to

8:10 InfiniBand, they're switching from ethernet to InfiniBand,

8:13 but they're still going to have ethernet running somewhere and they're going to

8:16 have some compatibility. And there are things that can make that work,

8:19 but it's just more complexity. And in the world of networking or anything in it,

8:23 complexity is there, but you want to avoid it.

8:25 You want to focus on the simple things that work and be complex.

8:28 Only when you have to my other big beef with it, it'll often in most cases,

8:31 be more expensive than ethernet switches and routers. And the third,

8:35 I think this is kind of a big one,

8:36 the community is not as big and ethernet based. Networking.

8:39 If you have a problem, you can pretty much go to any form out there, Twitter,

8:42 anything and get some help in Finn Band. It's going to be harder,

8:44 not impossible, but definitely harder. So yes, InfiniBand is pretty awesome,

8:48 but it's not the only option. I told you there's a war. Ethernet isn't dead.

8:53 It didn't take this lying down. It saw what Infin band did and said,

8:55 you know what? Hold my coffee. And it did some pretty cool stuff.

8:59 First speed Infin Band boasts a lot of speed. Ethernet's got speed too.

9:03 400 gigabit ethernet, we've got it.

9:05 And actually 800 gigabit ethernet is on the horizon.

9:08 Actually 800 gigabit ethernet is here.

9:10 Juniper announced their 800 gig ethernet boxes on January 29th. So we got that.

9:14 And 1.6 terabits per second is on the horizon now,

9:18 which is just insane to think about. I just installed 10 gigabit.

9:21 Now the big thing with Infinite band was RDMA.

9:23 That memory to memory access is killer bypassing CPU stuff and all that Ethernet

9:28 was like, you know what? We can do it too.

9:30 R-O-C-E-R-D-M-A over converged ethernet. It's a thing. Look it up.

9:34 Or I'll just tell you about it right now actually.

9:36 That's pretty much what it is. It's RDMA over ethernet. It had a V one,

9:40 now we're at V two V one could do layers one and two in the TPIP stack and then

9:44 V two can go over layer three. So that sounds awesome, right?

9:47 But it does kind of bring about a problem or at least a question.

9:50 We know that AI networking requires a lossless fabric.

9:54 It has no patience for latency. We can't be losing packets left and right.

9:57 And especially with running RDMA, you got to have a lossless fabric.

10:00 Ethernet inherently is not lossless, it's lossy. Let's break that down.

10:03 In a typical ethernet network with T-C-P-I-P when things get hairy or whether

10:07 it's congestion like a traffic jam,

10:09 too much traffic going across one connection,

10:12 how TCP will handle that is by dropping packets and then retransmitting them.

10:15 This is the overhead we're talking about with T CCP IP. That causes latency.

10:19 And even with R-O-C-E-R-D-M-A over converged ethernet,

10:21 which will bypass the higher layers of T CCP ip,

10:23 it still relies on the physical and data link layer to get across and without

10:27 anything there to help it, there's going to be congestion.

10:30 And because it's sitting there in layers one and two,

10:32 it's not relying on TCP to handle any kind of retransmission or error handling.

10:35 So there's nothing stopping it from getting so congested like rushed hour

10:38 traffic where you're just like, not even stop and go, just stop.

10:41 So what has ethernet done to help with that? Two amazing things. First,

10:45 ECN or explicit congestion notification.

10:47 This is kind of magic and it's a very clever way to do this between two devices

10:51 that support it, which will be anything we deploy in AI networking.

10:54 They can essentially let each other know when things are getting too congested.

10:57 Getting back to our analogy, like a road or a traffic jam. If one ends like,

11:01 Hey, I see a lot of cars coming my way, it's going to be a traffic jam.

11:04 I know it. It'll start marking cars, put a sticker on. It says, Hey,

11:07 congestion's coming, sticker congestion's coming are marking packets,

11:10 congestion's coming.

11:12 So the other end will know about it and then the other end will go, oh, okay,

11:15 well I'm just going to slow it down a bit.

11:16 I'm not going to let so many cars through.

11:18 I'm going to slow it down to avoid that congestion where packets would be

11:22 dropped. And we don't have a lossless fabric. That alone is pretty cool.

11:25 But then we also have this PFC or priority flow control.

11:29 PFC can pause certain types of traffic from flowing while allowing others.

11:33 So in the scenario of AI training,

11:35 it'll prioritize any training data traffic being sent over and pause other flows

11:40 to avoid that congestion.

11:41 So PFC and ECN will work together to create a smooth road for your

11:45 Lamborghini just to speed down. So at this point,

11:48 ethernet and a FEA band are neck and neck, RDMA, both have it a lossless fabric.

11:52 They both do that. Now you just got to cast your vote ethernet or InfiniBand.

11:57 Actually, I would love to hear in the comments below what you think.

11:59 You heard my argument,

12:00 maybe do a bit of research and let me know if I'm missing something.

12:03 But like Juniper, I am team ethernet.

12:05 I'm rooting for it to win because next to ip,

12:07 ethernet is the world's most widely adopted network technology.

12:10 It dominates every type of networking including data center networks.

12:13 Everyone already uses it. Network engineers know and love it.

12:16 And there's really no need to jump to a more proprietary protocol. Now,

12:19 I'm not saying infin band is proprietary because it's not,

12:22 but it doesn't feel as open as ethernet, not as much competition,

12:24 which does affect price. And we haven't even talked about the future.

12:27 Who's going to innovate faster? Ethernet or InfiniBand?

12:30 My boat would be for ethernet. It can innovate at the speed of the industry.

12:34 Now, to close this out, I want to play a little game.

12:35 What would my data center look like? My AI data center, if I had to build one,

12:39 what would be my diagram? Boom. Here it is. First, it's a cloths fabric or CLOS.

12:43 No, I'm blocking kind of a fat trees apology. I know I'm throwing out words.

12:47 I got a video about data center networking. If you want to check it out.

12:49 For my spine and leaf switches, I'm rocking Juniper. 800 gig ethernet.

12:53 I can go with either the QFX series or the PTX and look at these guys. I mean,

12:56 they're insane. And then to make sure my network is always smarter than I am,

12:59 I'll be running some software defined networking.

13:01 Juniper has this thing called ra.

13:03 It'll run the show and make your network awesome.

13:05 It'll make sure that things like PFC and ECN are configured correctly and tuned

13:10 based on the current needs of the network so you don't have to worry about it.

13:12 And of course, they'll have the full suite of DCB or data center bridging,

13:15 which is kind of a suite of protocols that include PFC and ECN,

13:19 and some other stuff. I'll list right here. So what team are you?

13:21 Team Ethernet or Team InfiniBand? Let me know below. And again,

13:24 shout out to Juniper for sponsoring this video and making videos like this

13:27 possible. I love getting to talk about networking and data centers and ai.

13:30 So let me know if you want to see more of that. And of course,

13:32 if you want to learn more about what Juniper's doing with AI and other crazy

13:35 switches and routers that costs more than your entire neighborhood,

13:38 check out the link below. I'll catch you guys next time.