# The FORBIDDEN Motherboard!! Hacking More Lanes into AM5: LR-Link Broadcom PCIe Expansion

## Метаданные

- **Канал:** Level1Techs
- **YouTube:** https://www.youtube.com/watch?v=7MEkVig19I4
- **Дата:** 30.04.2026
- **Длительность:** 25:29
- **Просмотры:** 53,586

## Описание

Does it exist? Not yet. Wendell has to do some wendelling to make it work!

0:00 Intro
0:29 Backstory
1:35 Comparisons
5:06 Chipsets and Lanes
10:19 What Things Currently Look Like
11:16 Good News Bad News
14:53 Performance with Current Set Up
20:38 So What Does the Ideal Motherboard Look Like?


You can find us...
Twitter - https://twitter.com/level1techs
Twitch - https://twitch.tv/teampgp
Patreon - https://www.patreon.com/level1

For all our social links, websites, and more, check out our link tree!
https://linktr.ee/level1techs

Thank you for watching!

-------------------------------------------------------------------------------------------------------------
*IMPORTANT* Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.

## Содержание

### [0:00](https://www.youtube.com/watch?v=7MEkVig19I4) Intro

All right, today we're going to build our fantasy AM5 motherboard that I've always wanted since the dawn of time that's going to have more PCIe lanes than is physically possible and yet it's still going to work. Let me put it to you another way. AM5 with 32 lanes of PCIe Gen 5. We need our friends at LRLink to help us out. But before I get to that, there's some explanation that is needed.

### [0:29](https://www.youtube.com/watch?v=7MEkVig19I4&t=29s) Backstory

Okay, look. It's hard to get components right now. That's true for workstation and everything else, but some people have a business to run. Some of you, a lot of you in the audience have small and mediumsiz businesses that you have to support. And I think gone are the days that you could really splurge on super high-end hardware. Not only that, AMD has AMD Epic server CPUs in the AM5 socket. You can also use just regular desktop AM5 CPUs as a server CPU. I've got this is a 650 chipset from ASRock rack and this will make a perfectly reasonable file server or workg groupoup server. I mean you can run 50 maybe 100 people off of this thing. 128 192 gigs of RAM depending on what your workload is and what you're doing. It's perfectly reasonable. But the part about it that's not reasonable is PCIe connectivity. But you have to understand PCIe connectivity is just lanes and lanes are fungeible. They can be spent on all kinds of different things, GPUs, storage, different kinds of storage, networking, you name it. This

### [1:35](https://www.youtube.com/watch?v=7MEkVig19I4&t=95s) Comparisons

is a pretty typical very low-end but still server class motherboard. We've got one physical X16 slot and X8 slot and then a B1 slot. And so there's not really a lot of PCIe lanes here. And you can see from the CPU socket, the CPU socket is not very large. So, it's not really a true server class socket. It's a desktop class socket, but there are CPUs that are available. We've only got two memory channels, four memory slots, but how you spend the PCIe lanes matters a fair bit. Over here, there are four Oculink connectors, and this gives you some more PCIe lanes. What do motherboard designers do when you've only got, you know, 20 or 24 PCIe lanes out of the CPU? How do you spend them? Well, 16 of them usually goes to a PCIe slot or two and like maybe four of them would go to an M. 2 and the rest go to the chipset. And the chipset itself takes PCIe lanes and connects to the CPU, but also has PCIe lanes that it manages itself. Typically slower PCIe lanes. So, you know, you might have a Gen 5 connection from the CPU to the chipset, and then the chipset might have some Gen 4 PCIe lanes for slower peripherals, or it might have a bunch of, you know, X1 peripherals. That's what we got going on here. This X1 slot doesn't connect to the CPU, even though it's closest to the CPU. It connects to the chipset. The only PCIe slot here that connects to the CPU is the X16 slot. So, this is not really a fabulous motherboard. If we look at this desktop motherboard from MSI, we have the primary x16 slot, which really is 16 PCIe lanes. PCIe gen 5 direct to the CPU. We also have the primary M. 2, which is five PCIe gen 5, four lanes directly to the CPU. And we've also got some more M. 2 that goes to the chipset. So, there is a high-speed PCIe connection from the CPU to the chipset. But then the chipset has USB and the interface that the network card uses, which is usually also PCIe, although not necessarily. The CPU actually does have a couple of USB resources built right into it, which is sort of a fun interesting thing that AMD did. AMD has the X870E chipset, which is actually two chips that are daisy chained together. You would think that there would be, you know, you want that tree to be as short and fat as possible, the tree of peripherals so that each chipset connects directly to the CPU. No, that's not how it works. Part of the reason for that is PCIe5 is such a high-speed signal. It's hard to keep the signals clean. It's more convenient to just daisy chain the connections together from one chipset to another. Because of the length of the board, the copper wires in a PCB like this are a little problematic. But PCIe again, like I say, is fungeible. So it's like this is a PCIe slot like you're used to. It's 16 lanes. Depending on other resources that are available for timing, you could break that out into 16 one lane PCIe slots. Commonly on some motherboards, you might see an X16 slot and then another slot down here. And in the manual, it would say it's X8 uh/z8 or X8 X8. So, you can take eight of your lanes from the primary slot and optionally use them down here. But what if you don't want to give up the PCIe lanes? What if you want 16 lanes up here and 16 lanes down there? It's a math problem. AM5 doesn't physically have that many PCIe lanes. But the clue is in the chipset because the chipset's doing

### [5:06](https://www.youtube.com/watch?v=7MEkVig19I4&t=306s) Chipsets and Lanes

something. Aren't there other chipsets? Yes. Enter our adapter from LRL. Now, this is an LRLink PCIe 5 um x16. It's got 16 Gen 5 lanes, but it has 32 Gen 5 lanes out the other side. Now, this is a special controller from LRING, but it's this is uh primarily designed to drive a bunch of in NVMe. So, you could have two, four, six, eight, four lane Gen 5 NVME connected to this and it will give you, you know, theoretically like if you were using these really nice like the Kioxia CM5, these are 15 gigabytes per second. you're going to overrun this the amount of bandwidth that you get from this slot. But the chipset that's in here is Broadcom and it's reprogrammable and I can reprogram it to do different fun interesting things. This connector is called MCIO as we've covered in the past. I can take that MCIO connector and I can go to a board like this. This board has two PCIe slots that are wired for eight lanes each. I could also take this to another board where each slot has 16 lanes. And I could have two 16 lane connections from this card. Or I could have two of these carrier boards that each have two slots. Each one of these with eight lanes. So 16 lanes on this board in two slots plus another one connected to the other set of connectors on the end of this card. And I've got four eight lane PCIe slots. There are actually motherboards from the AM4 era and from the Intel era when the number of PCIe lanes was even less where there were chipsets on the motherboards to do this to you know provide even more PCIe lanes. But they got expensive because servers needed more PCIe lanes first and PCIe lanes is still a problem in bigger servers. One of the pieces of magic in modern AI servers from Nvidia that we saw at Computex is that the modern ConnectX nick has a built-in PCIe switching fabric. And that switching fabric is actually PCIe Gen 6. All this is PCIe Gen 5. The CPUs, the slow thing CPU is not keeping up. There's not CPUs that are PCIe Gen 6. Well, I mean, there are, they're just not out yet. But what Jensen did with, you know, modern connect X9 is he moved the networking and the PCIe fabric onto one carrier. So, we saw super micro servers where you had all the connections at the back for the networking, one network port per GPU, and that's all on one carrier card. And then that carrier card has lots of PCIe lanes that connect back to the CPU. But the reason that you want that is because the GPUs can talk to each other. That also works great with this. So, if I've got two R9700s in a system that's only AM5, those GPUs can talk to each other twice as fast with this as they can through the CPU. And that is worth it to some people. And that will also let you run four of these R9700 GPUs, 128 GB of VRAM on an AM5 system with just one of these Broadcom chipsets. uh that's going to run the GPUs with eight lanes instead of 16 lanes. But for AI applications, the GPUs mostly need to talk to each other, not the system. But if you have one GPU that's receiving a bunch of stuff, it can receive a bunch of data at 16 lanes worth of speed and then the information could be copied to the other GPUs without ever leaving this PCIe bridge. So you can still get a ton of performance. There are other companies that are doing fun things with how you spend your lanes. Uh Icy do for example. Icy do has this kind of a thing which is eight lanes. It's made for PCIe bifurcation mode but it'll give you two M. 2 on this PCIe carrier card with a hot swap on the end. And these PCIe lanes don't necessarily have to be CPU PCIe lanes. They could be chipset PCIe lanes. It doesn't really matter. I could also go from MCIO to this with eight lanes here. This gets a little complicated because then I have to reprogram that bridge and tell it that this is going to be two X4 devices in an X8 slot. So, it's like as if this cable is going to, you know, two M. 2 or two U. 2. And I have other cables for that here. This is an MCIO header on this side that's eight PCIe lanes wide. And on this side, I've got two U. 2 two connectors that are four PCI Express lanes wide. And depending on the cabling and the red driver and a bunch of settings that you have on your card, this can maintain PCIe Gen 5 signal integrity. It can be a little tough because PCIe Gen 5 is unforgiving and complex and sensitive to lots of interference. There's foil around all these wires to block hopefully RF interference. So, even though if you look at a diagram of our B650 motherboard here, even though it's only got one X16 physical slot, I can go into a PCIe controller like this and then not elegantly, but I can break it out into a different case and then have a bunch more room for GPUs.

### [10:19](https://www.youtube.com/watch?v=7MEkVig19I4&t=619s) What Things Currently Look Like

And this is what it looks like in practice. I mean, yeah, it's a little bit mess of wires. I'll give you that. But this is a fun way to connect four GPUs even to an AM5 system. And it looks a little bit like a mining rig, but understand that it's not a mining rig at all because most mining rigs would connect to a GPU with one PCIe lane. The work to be done would live entirely on one GPU. Here, it's really important that the peripheral be able to communicate with the system at full speed as well as the peripheral being able to communicate with other peripherals. See, there's very little benefit for running multiple GPUs if the GPUs can't talk to each other at full speed. That's what the chipset gives us. It bypasses the CPU because, well, even at scale, even with $10,000 CPUs, the CPUs are too slow. PCIe Gen 6 is needed for full bandwidth. So, we're doing the same kind of thing, just at a smaller scale. Now, to an extent, I

### [11:16](https://www.youtube.com/watch?v=7MEkVig19I4&t=676s) Good News Bad News

think AMD could solve these kinds of problems via the chipset. Remember, their chipset is four PCIe lanes wide. So, it's going to be a maximum of eight in PCIe Gen 4 mode or 15 16 gigabytes per second in PCIe Gen 5 mode. I think it'd be better for the chip design to have eight lanes, an eight lane chipset. And then in the eight lane chipset, you could go to four two four lane PCIe Gen 5 M. 2 or another PCIe Gen 5 expansion slot and something for slower peripherals. Maybe we're going to add a 25 gig nick and that would be perfectly fine at PCIe Gen 4. It would be amazing to have an 8 lane PCIe Gen 4 slot on some kind of a board like this. I think in 2026 when we've got this these massive parts shortages, it makes sense to have an AM5 motherboard that has some kind of a chipset on it to give you four eight lane slots and maybe a couple of fourlane slots. That would be ideal because then you can pack GPUs on or high-speed networking or some combination of GPUs and high-speed networking. And the bottleneck there becomes a little bit the CPU, but now we have dual 3D vcache. The other big component that's been missing up until now was having an AM5 CPU fast enough to keep all of this fed. Ah, but we have the 9950X 3D2 dual edition. So the bad news, these are really in demand in server context still. Even though Nvidia has basically cornered the market and where previously we were using as media controllers or Broadcom controllers on the motherboard, they've, you know, sort of slid into adjacent markets from that. I mean, there's still a huge market for all of that stuff, but it's less important because of what Nvidia is doing. And also there are other competitors that have sprung up that are offering PCIe chipsets like Fison and Texas and you know Texas Instruments even. It's like over $1,000 for one of these cards. But if you want to host eight GPUs on a Thread Ripper board that doesn't physically have eight slots, it can only physically have seven slots. This can get it done. LRLink has several uh PCIe bridges or PCIe rettimer red driver cards that are appropriate for this kind of thing. If you are building a server that has a lot of GPUs and you're facing PCIe AER errors, a card from LR link and cabling will help alleviate that because it will either completely redrive and rettime the signal or it'll go through a PCIe bridge and it's literally signal in the bridge does PCIe bridge stuff and then the signal goes back out the other side because this is addressable on the bridge. It also solves some compatibility issues from different peripherals. Uh have some proprietary FPGAAS in another different work related scenario and this solves lots of compatibility issues. There are certain ASUS motherboards where the slots that are closer to the CPU don't have any red drivers or rettimers and the slots that are farther away from the CPU have red drivers and rettimers. And sometimes you got to fiddle with the drive strength and the retraining stuff in the BIOS. And you could just get one of these and the problem goes away. And even though it's like $1,200, there's a downtime price of admission reliability aspect to it. And it's like, yeah, it's worth the $1,200 to just buy the card and be done with it. Sometimes arranging the PCBs in a physical fashion becomes a little bit of a challenge, but yeah. Now, what kind

### [14:53](https://www.youtube.com/watch?v=7MEkVig19I4&t=893s) Performance with Current Set Up

of a performance difference might it make? I'm glad you asked. This is our Orus X870E, and it's got really good spacing for two GPUs. I'm running two GPUs off of the CPU, eight lanes each, that is connected to our R9700s. So, I got a pair of them 64 gigs of VRAM in this system. and our 9850X3D 2 dual edition. Also, uh right now it's only 64 gigs of memory because memory is okay with Quinn 3. 6 this is 107 tokens per second for inferencing for training and anything like that. With this connection, each GPU only has eight lanes and the CPU is responsible for managing that. In this configuration, 107 tokens per second is pretty nice. But what about 125 tokens per second? It's about a 10% performance uplift for inferencing. And if you're doing training or uh fine-tuning a model, something like that, it can be upwards of 15% faster. How the LR link connection. Instead of having two GPUs like this, we have it set up like our other machine where there's one card in the primary x16 slot and then our LRL link card fans that out into two more x16 slots. Through this card, the two GPUs can talk to each other at the full PCIe gen 5x6 lanes bandwidth. And when you're doing training or anything like that, you're going to get a little bit of a speed bump. Now, is that much of a speed bump worth $1,000? It depends on what you're doing. For some people, it it definitely would be. For some people, this setup is reasonable. A reasonable way to get you to 64 gigs of VRAM and a state-of-the-art model like Quinn 3. 6 35B that is in the Q8 uh running at 117 well about 100 tokens per second. And with a deeper context, we're still at over 80 tokens per second in this setup. Of course, with the deeper context, this is also faster. Now, if you're running even higherend GPUs like RTX 6000s and you're doing fine-tuning and training, the performance benefit is even more pronounced because those GPUs have more memory. They're 600 watt cards as opposed to these. They become a lot more difficult to juggle in other aspects like power and physical layout and dealing with the heat that much power generates. But for inferencing and training, you can get upwards of 20% better performance. For just pure inferencing, if you're not doing, you know, a combination task, the performance uplift from just inferencing is not as much. It's also on the order of about 10%. But 10% difference in, you know, tokens per second for token generation, it's pretty good. Prompt processing, you know, in that 10 to 20% neighborhood also not bad. Now, just to be clear, if you're looking at LR Link's website, they have red driver, rettimer, and then they also have this. Now, this is meant for I mean, look at this contraption PCIe card on one side and PCIe slots on the other. And these are the eight lane slots. We could also go to 16 lane slots, so like two cables cuz each cable is eight lanes. But this is the four MCIO 8i NVME expansion card. This normally comes configured for NVME. And what this looks like is you have two NVME headers per connector. So you can connect eight of these U. 2 to this controller. That is what this normally comes for. It is very off label and very me and very I'm reprogramming this thing in order to be able to do the PCIe slot thing unless you just want eight four lane PCIe slots. So don't do what I did with the graphics cards because it's not really meant to do that. But if you do want to do something like that, you're better off using your motherboard bifurcation and the other card that LRL link offers which does retiming and redriving. This one from their website, it's a better choice and is less expensive because it doesn't involve the crazy expensive Broadcom chipset that can lift and shift PCIe lanes however you want. Just for demonstration, I'm using the ARC B50. It's convenient because it doesn't require external power, but it also validates that I'm getting power through my little add-on board. So, yay. And so in this configuration, it's in the shipping config, which is four groups of four lanes per port group. So you've got 32 lanes divided by four for eight NVME. And our GPU still works fine because again PCIe is kind of funible. The only tricky part is configuring how many lanes you want where. You got to be able to reprogram the PCIe bridge. Not for the faint of heart. Um, but you can see that we've got both of our U. 2 NVME in device manager plus our Intel Arc Pro B50 all connected to our plucky little AM5 platform. And to just be clear, to really drive the point home, the motherboard slot is configured for X16. We are not using motherboard bifurcation. If you're going to use motherboard bifurcation, you can, but you should use not the Broadcom adapter, but the passive one, because then the CPU controls how that 16 lanes is divvied up. You could split it up into four groups of four lanes. Well, we have eight groups of four lanes because it's a PCIe bridge. There's a chip on here that's doing active work figuring out how to sort and route PCIe lanes as was the case in the days of your all the way back on like X299 like the really high-end like the first X299 motherboards out the gate in some X99 motherboards. Same thing. It's pretty bananas. Okay, so my crazy

### [20:38](https://www.youtube.com/watch?v=7MEkVig19I4&t=1238s) So What Does the Ideal Motherboard Look Like?

fantasy motherboard probably doesn't look like a mess of cables. How do we go from something like this to something that can look a little bit more like this? And I think we can put our eye toward AM6. What does AM6 look like? I'm glad you asked. I've got a diagram. I think it makes a lot of sense to move back to a two chipset solution like X870e, but not X870E. I want to use just two PCIe Gen 6 lanes for all the relatively low-speed IO at the back of the motherboard. That'll include Thunderbolt 5, USB, 10 GB Ethernet. I think we can get an interface for, you know, real tech's inexpensive 10 gigabit Ethernet, 5 Gbit, you know, two 10 Gbit interfaces, whatever we need, everything that we need at the rear IO connected to the CPU with just two PCI Express 6 lanes. That gives us as much bandwidth as four PCI Express Gen 5 lanes, which is great. That means all the stuff at the rear of your computer has about 15 gigabytes per second of total aggregate bandwidth to the CPU. Fantastic. I'll take it. to keep costs down. I guess probably maybe we could have 16 lanes for the GPU and we could keep doing the same old where it's eight lanes for two GPUs or 16 lanes for one GPU directly into the CPU. Now, physical fulls size ATX motherboards have got room for seven slots. I'm going to eliminate two of the slots because I think everybody is going to have at least a double width GPU. It definitely doesn't make sense to try to shove a bunch of slots on here. And this is also not the end all beall. So, two of our slots are spoken for. We've got three other possible physical slots. Where do those come from? Those, I think, come from the chipset and those cap out at PCIe Gen 4. Maybe we have one slot that's eight lanes or up to eight lanes and two other slots that are four lanes. The rest of our lanes from the chipset can go to M. 2 at PCIe Gen 5 speeds. It would be nice if we had eight PCIe Gen 6 lanes between the CPU and the chipset. So, that' be 8 + 16 plus two. Now, got a choice here. We can have two more PCI Express Gen 6 lanes that we could do something creative with, or we could have four Gen 6 lanes for a dedicated M. 2. I'm kind of on the fence here. I could go back and forth between having our M. 2s have to hang off of the chipset as opposed to being directly connected to the CPU, but it is really convenient and nice from a latency perspective, having the storage connected directly to the CPU. two lanes of Gen 6, 5 for existing Gen 5 SSDs. Don't know that I like that trade-off. I would really like to squeeze in four lanes direct to the CPU, but that's 4 + 8 + 2 + 16 lanes total on our AM5 CPU. That gives us a total of 30 lanes. That might sound crazy, but there was a Chinese AM4 motherboard that actually had 32 lanes out of the AM4 sockets. There was even a dual socket version of that where there were 16 lanes between two AM4 sockets and then each socket provided 16 lanes to the platform. That was high. That was a long time ago. We get to let highs be high. So AM6 having 30 or 32 PCIe lanes would be great. And motherboard makers don't have to go with this layout. Like we could have an inexpensive four PCIe gen six lane chipset and then that chipset maybe has fewer or slower M. 2 or fewer or slower PCIe lanes. But I think the top tier skew being eight PCIe lanes from the CPU assuagages any bandwidth concerns that we would have for the lifetime of that AM6 socket. But hey, that's just my fantasy wish list. I do have this thing and if you want to have me build something or test something, uh, we have a community for that in the level one forums. you should definitely come and check that out and maybe we can put something together in the AM5 side of things and just sort of do a test pilot to see how that goes cuz remember PCIe lanes are fungeible and what other channel are you going to hear that from see like you spend your PCIe lanes however you want storage U. 2 E1 E3 I would love to see E1 and E3 come to the desktop for something a little higherend than M. 2 too. Especially when you're shelling out as much money as these X870E motherboards cost in the first place. Like if you're going to spend that kind of money on your workstation, like you're going to probably get some more expensive peripherals, right? Especially in this day and age, although I think we're in for a reset pretty soon. So, that's about it for this one on thoughts. My sort of fantasy AM5 motherboard that hopefully will translate into AM6 reality cuz I don't think anybody's going to build AM5. I'm just level one. If you have any questions or missed anything you want to see me do a test build, you can find me in the level one forum. I'm signing out and I'll see you there. Woo! Go community

---
*Источник: https://ekstraktznaniy.ru/video/49725*