How to extend PCI express to setup 25 boards?

Question

I need to populate a PC with 25 PCI express boards, mostly I/O boards of different kinds. One solution is to have a distributed architecture with several PC in a private LAN and a master clock to synchronize all I/O boards timestamps.

For some reasons, I don't want a LAN, so I would like to either find a big motherboard with 25 PCI express connectors, either extend the PCI express in some way.

I/O boards throughput are : 20 boards at 30Mbps x 32 lines + 5 boards at 60Mbps x 32 lines. Use cases requires full throughput on all lines.

If my understanding is correct, PCI express goes up to 32x at 4 GB/s/1x = 32Gbps/1x with PPCIe 5.0. So I could use 1x for each of my 25 boards, and even 2x for the 5 most demanding ones.

So to summarize, how could I plug 25 PCIe 1x boards in the same PC?

Answer 1

This is going to be tough to achieve without custom hardware. With the right BIOS support, you can bifurcate/breakout PCIe slots (see the excellent cards from c-payne for example), but you are going to really struggle to reach a number that high, because of power. Your best hope are crypto mining motherboards, but I have not seen any that support PCIe gen 5 yet, and the most I have seen in terms of PCIe slots on a single board is 19 - see the B250 mining from ASUS. The next highest is the ASRock H110 BTC Pro with 13.

The ASUS board comes with triple-ATX12V power, and references separate capacitors for each of its 19 PCIe slots to deliver stable power. From the manual:

In order to fully support 19 graphic cards mining, we recommend that you use 3 power supply units (PSU) that are designed for mining with sufficient 12V power plugs and provide a minimum power of 3750W in total (21250W + 11350W are recommended).

Now, you don't mention the power requirements of the boards you are intending to use, but even if they are significantly less than GPUs, it's still going to be something you have to seriously consider and plan for.

Then there are the PCIe lanes (and generation). ASUS doesn't list the spec on their board, but I would assume the max is PCIe 3.0 and the rest PCIe 2.0 (this is the case on the ASRock board, which does list the specs). There are newer ASRock boards, but they focus on larger slots rather than more x1 slots even though they are limited to x1 speeds.

So, that about does it for low cost options - the miners are more about looking for the cheapest (stable) way to get as many GPU/mining cards working with the least power/space possible rather than guaranteeing bandwidth per slot and at a certain point it just becomes easier (especially for space and power management) to go with another motherboard rather than cramming everything into one (they are not concerned with syncing).

There are server motherboards with enough PCIe lanes to do what you want (see this recent supermicro board), but they are a lot more expensive, and not generally looking to accommodate 25 PCIe cards. So, not only are you looking at an expensive board to start with, you are also going to need additional cards to break out the slots, which are not cheap, especially for PCIe 4/5 (a good quality PCIe 4.0 compatible riser card for a GPU will run you >$100 generally speaking).

Overall, if you have to get everything into one system, I don't think you are going to get what you want with off-the-shelf hardware, and will need to talk to a vendor about something more exotic to get what you need here, unfortunately. Two systems, with a high speed interconnect (not ethernet) might be a far easier prospect, but I'd have to do a lot more research to figure out the options there.

Answer 2

I don't know what environmental, power, space and cost constraints you have, but onestopsystems have a range of professional looking rack mount PCIe expansion chassis's. Some of the expansion chassis's have online prices, and for others it looks like you have you to request a quote.

I haven't actually used such a chassis, it was just the first example I found.