Deep Learning workstation advice

Question

I am using Keras with Theano backend. Until now I have used AWS EC2 GPU instances for train my keras network. As my experiments become larger, my network trains became longer and my bills became bigger. I calculated that good workstation would be a better investment than renting AWS EC2 GPU instances on the cloud. I am copying components of NVIDIA DIGITS DevBox, except there are new GPUs on the market, so I have couple of question about them.

Is it better, for deep learning, to buy four GTX 1080 or dual Titan X Pascal (specs)? Four GTX 1080s give me 10240 CUDA cores with boost speed @1733 Mhz, 32GB of GDDR5X VRAM and 36 TFLOPs of FP32 Compute. For the same price dual TITAN X Pascal will give me 7168 CUDA cores boost clock @ 1530 Mhz, 24 VRAM GDDR5X, and 22 TFLOPs of FP32 Compute.

My plan of using multiple GPUs is train same model with different parameters on different GPUs, to see which combination of parameters is most effective. I need the GPU solution that is most efficient for this type of usage.

Answer 1

As long as electrical bills are not a problem and you're going to get more than one GPU anyway, you should get the four GTX 1080s.

The reason this is the better choice is because it provides better flops per dollar amount than any other option (at least on the Nvidia side; a real pity that Theano only seems to see OpenCL work on CPU for the future; AMD options could really go to town if the work could be given to them in an OpenCL format).

Theano uses FP32 calculations, and it can support multiple GPUs (still somewhat experimentally).

One final thing to note: the GTX 1080 uses DDR5 for its vRAM, not the DDR5X as earlier speculated. AFAIK the same should be true for the Pascal Titans.

An edge case where I might be wrong about this would be if you were going to be usually running fewer than 3 parameter sets simultaneously, because then the two more powerful Titans would be a better choice. I think that will ultimately end up being very unlikely, however, so I'm not going to let it influence my answer. If OP knows otherwise he should definitely speak up.