I haven't personally used CUDA for this, though I am planning to. From my research I concluded, that you for sure want to have 'Computing Capability' at least equal to 3.5, as some libraries requre it already. The (tough to find) list of computing capability is under this link. From this list one can conclude, that having GTX 980 or Titan (both 5.2 score) is as good as you can get, but note, that if you are buing a graphics card only for this, Nvidia has an answer for professional and academic use named Tesla - it's just a computing box, far from graphics card (it has no display ports even!), costs from 3k $ for K20, to 5k $ for K80 model, and is a behemoth:
A quick comparision (CC stands for Compute Capability):
- Tesla K20: for desktops, peek for float: 3.52 Tflops, 5 GB, 2496 CUDA cores, 2.9 k $, CC: 3.5
- Tesla K40: for desktops, peek for float: 4.29 Tflops, 12 GB, 2880 CUDA cores, 3.1k $, CC: 3.5
- Tesla K80: for servers, peek for float: 8 Tflops, 24 GB, 4992 CUDA Cores, 5k $, CC: 3.7
and customer grade, most popular and new graphics cards:
- GTX960: for desktops, peek for float: 2.3 Tflops, 2 or 4 GB, 1024 CUDA cores, 200$, CC: 5.2
- GTX980: for desktops, peek for float: 4.61 Tflops, 4 GB, 2048 CUDA cores, 500$, CC: 5.2
- Titan X: for desktops, peek for float: 6.1 Tflops, 12 GB, 3072 CUDA cores, 1000$, CC: 5.2
Also see comparision on wccftech.
To conclude: the commercial grade GPU seem to be more cost efficient, but only when we compare the specs. There can be other trade-offs, I am not awere of. Thus, I cannot confidently say "go with customer grade", but I can tell you what I would (will) do - I will buy GTX 960 or 970, because I plan on gaming and I'm quite cost limited, and this cards will do just fine for CUDA learning. If you buy for an institution, do not plan gaming, the calculations will go 24/7, consider the academic grade Teslas.
Also, if you'll be interested in boosting your 'conventional' integer based processing power on a high-end computation server, you may want to look up Xeon phi.
[EDIT] Please note, that switching from CPU based floating point arithmetic to GPU enchanced, is a change in quality, almost one order of magnitute, and will be very pronounced and noticible, but switching from ex. Tesla K20 to Tesla K40 will be just a change in quantity (K80 is just two K40 bundled together), so if you go for speed to price ratio, go with cheapest GPU acceleration, that'll work for you.