I'm looking for a server to run large number of parallel processes for statistical calculations (Ubuntu + Python). Currently we use 2x16 cores and 512Gb RAM (disk is of secondary importance):
- Supermicro SuperWorkstation 7048GR-TR Barebone System - Intel C612 Express Chipset - Socket R3 (LGA2011-3) - 2 x Processor Support
- 2 x INTEL HASWELL 16C E5-2698V3 2.3G 40M 9.6GT/s QPI
- 8 x 32GB DDR4-2133
We paid 14K+ for this config 3 years ago (bought from Supermicro resellers). Any cheaper way to replicate this or significantly improve in the same budget?
Some more detail on our calculations (EDIT)
We can parallelize our calculations to 3000 separate processes. Runtime is 6 hours on current setup, would like to make this smaller so I can experiment with system parameters at least 2x in a working day.
The calculation is mostly in LAPACK (through Python/numpy). Prefer one big machine to several small ones, until a reasonable limit on price efficiency. Can spend 15K or 30K (if performance still doubles for the extra money).
The code: Python with numpy, vectorized. Very few matrix multiplications happen, most of the operations are element-wise addition, multiplication.
Code is already written (100K lines), we don't have the capacity to re-write. Also we change it every day and are glad for a flexible language as Python. This means GPU / resource sharing / multiple machines / AWS is not a possibility.
Any hint is welcome on what to look at, I'm newbie in hardware.