Increasing the capabilities of a machine for deep learning purposes

Question

The department of the university where I am currently pursuing a Ph.D. has a machine with the following specifications:

Feature	Value
OS	Windows 10 Enterprise LTSC
Processor	Intel® Xeon Gold 5120 CPU @ 2.20GHz 2.19GHz (x2 processors)
RAM Memory	96 GB (94.7 GB for real use)
Memory	2.5 TB (2TB HDD + 500GB SSD)
GPU 1	Nvidia Quadro P5000 with 16 GB
GPU 2	Nvidia Quadro RTX 8000 with 48 GB

The problem is that the deep learning tasks to be performed by both the principal researchers and other Ph.D. students has been increasing in recent months, thus considering the possibility of increasing the resources of the current machine. In this sense, in an attempt to find out what would be the best option, I have reviewed several resources such as some of the posts published by Tim Dettmers (Post 1, Post 2). However, I have a few questions such as the following:

• Given our situation, would it be better to continue using the shared workstation approach that we are currently using or move to a cluster of GPUs?
• Our actual single Nvidia Quadro RTX 8000 offers us 48 GB of memory, being very valuable for training and loading heavy models. In this sense, I am afraid of moving to the state-of-the-art RTX 3080 or 3090 cards, which individually contain much less memory (10GB and 24 GB, respectively). In case of building a cluster with these latter type of cards, is the final available memory the result of adding the memories of each of the cards? Or is not as simple as that? Sorry if this is a very basic question.
• Would you consider as an option to include a second Nvidia Quadro RTX 8000 replacing the P5000? Or would it be better to sell both GPUs and get, for example, 2 RTX 3090? From what I have been able to read in the posts I have quoted above, it seems that the second option would be the most suitable one...

Any recommendations are welcome. Thank you very much in advance for your help.

Answer 1

1. You can continue to use that workstation, probably just upgrade it

2. With Quadro cards, you can add memory by adding more cards and an NVlink bridge, but you cannot do the same with GeForce cards

3. Second Quadro RTX 8000 would be best in my opinion, but do add an NVlink bridge and if you need display, also a GT 1030, because when NVlink is used, you cannot use the display outputs of the quadro's.

Answer 2

Increasing the capabilities of a machine for deep learning purposes

check out https://developer.nvidia.com/cuda-gpus and the compute capability number reported to compare between all the NVIDIA cards.

• if you must continue to use the existing computer hardware (the tower and motherboard) and can only buy add in PCIe cards then you need to evaluate the existing power supply and cabling and physical space in the tower to determine how many gpgpu or graphics cards can fit. As well as if the cards can be bridged (nvlink'd) with a cable for added performance.
• if your options are open then I would suggest looking into rack server / data center type hardware that supports gpgu and choose accordingly from the nvidia compute capability link above; i think this is the category of system that will offer the most performance vs cobbling together a tower, psu, motherboard, and a graphics card. With a rack server (such as from Dell or HP) you are typically looking at getting at least 128gb ram and they can go up to 768gb no problem (you can neglect this aspect if u know you don't need that much RAM)
• The best performance, if you know your code would benefit from inter-gpu communication (i.e. the nvlink) is to get an SXM2 type gpu which is chip on motherboard opposed to a PCIe type graphics/gpgpu card. For example a dell c4140.
• for your existing system though, a the nvidia RTX A4xxx and Geforce RTX 3000's are currently reported at 8.6. The quadro line tops out at 7.5.