1

Problem

I'm working with huge brain imaging raster data with four dimensions (x,y,z,t). The dimension sizes are respectively ~1000, ~600, ~20, ~3000 with data stored as uint16, which makes me ~100GB files. To store the data, I have a RAID of 8 8TB hard drives (64TB). Now comes the problem of accessing the data : sometimes I have to take a single (x,y) image, sometimes I have to take all the t's for one xy pixel. Since I can not load all the data in the RAM (I have 64GB), I use memory mapping which means I load the data only when I need it. But it makes the slicing along the z axis is very, very long.

Question

How can I access and process or visualize my data quickly ? It would be faster with a RAID of SSD's, but much more expansive. Is there a way to use only one 1TB SSD as the cache for the RAID ?

Improve this question
3
  • Do you know what RAID level you are currently using? There might be a better configuration for your set-up that would give better read performance, but I am not an expert in RAID.
    – Cfinley
    Commented Oct 25, 2018 at 5:06
  • What is your budget for the SSD? PCI SSDs are going to be much better than the standard SATA SSD, but can be extremely expensive.
    – Cfinley
    Commented Oct 25, 2018 at 5:09
  • @Cfinley I think it is RAID 0, but it might be RAID 1, I have to check. I did not build it myself. For the SSDs, we could spend around 1000€, but it is for several computers...
    – Hugo Trentesaux
    Commented Oct 25, 2018 at 11:10

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.