Picking VM server hardware for development

Question

I am an experienced programmer with very little hardware experience, and completely overwhelmed with the unlimited number of choices available. Since technology changes so rapidly, it's very difficult to Google up a guide that is both timely and relevant for what I'd like to do.

The best blog I've seen so far is this one, but it's already a few years old now.

I currently have several KVM/VirtualBox VMs on my Debian laptop. It is extremely slow, and I can't run more than 1 VM at a time (or 2 with the second having a very small memory/processor footprint).

I'd like to replace my laptop with a tower-based server for running these VM images, so I can remote in from any device on my LAN. My line of thinking is that I can get a lot more power into a larger enclosure for less money than an equivalent laptop would cost, with the possibility of upgrading down the line.

Here's the basic idea of what I'd like:

• Spend less than or equal to $1,500 US, tax included
• Install a hypervisor as the "main" OS (suggestions welcome as to which; was thinking of vSphere or KVM)
• Be able to run at least 2-3 Windows/Linux VMs simultaneously for development and testing
• Upgrade-able processor/memory would be nice, if possible
• Good performance for compiling large applications (noting that parallel computing is used in GCC/msbuild compilation) - developing both Web and desktop software
• Lots of RAM, which is used by compilation processes and VMs

So it should be kind of like a "home lab", but mainly used for professional development work.

I found some pre-built Lenovo servers on Amazon, and almost bought one, but I am second guessing myself in terms of what exactly to get.

Where to start? Build or buy? How to get started picking the right components (e.g. motherboard, drive, processor)? I know my way around the inside of a computer... I can replace parts, etc., so building a server is not out of the question-- but unfortunately I don't know how to pick "the best" parts. My last few laptop picks have been flops.

EDIT:

Here's an example of a sample workload that I might want to have. As a personal interest, I might want to compile AOSP (android) in a Linux image. That eats up a lot of RAM and a lot of CPU. While that's happening, I'm working on professional windows development in a 2nd VM w/Visual Studio. It's not graphic intensive, but I'd be compiling regularly, which, employs parallelism. There would also a web server running in a third VM that would be serving REST requests for testing purposes, related to the development work.

EDIT 2:

I'm overwhemed with the massive amount of choice when it comes to hardware. I'd like some advice from experts on (a) whether it's better to build or buy pre-built, and (b) how to get started picking something that is a good quality build, and upgradeable in terms of RAM (and possibly CPU). What are good sources for getting this type of information? There is so much information out there, it's hard to tell what's reliable and what's misleading. I don't care if it's a pre-built server, or if it's DIY, so long as it is good quality.

Answer 1

If you are worried about building a machine, don't build it. You may lose out a tiny bit on hardware, but in return you usually get limited manufacturer support. The trade off between more hardware and more support is a valid reason to buy pre-built (and configured) hardware.

The next question you need to answer for yourself, is whether you are going to actually upgrade your hardware at some point or are you just going to plan on doing it, but never get around to it. Everyone wants more hardware to throw at stuff, but if you are not actually going to utilize it to it's fullest potential at some point, there is no sense in buying a machine that can have it's capacity doubled. Instead, focus on maximizing the hardware now. It sounds like this is going to be an experiment in visualization for you. It may be worth while to simply build a machine that works for now and see if visualization will work as you are hoping. If it does, consider this experiment a success and plan a truly beefy machine in a few years time to replace this proof of concept.

---

For my recommendations, I've made a few assumptions that may or may not fit your situation.

• Your goal is to have server versions of the operating systems be visualized. Thus, you won't be running full desktop GUIs (frequently)
• The goal is to have at least the minimum amount of RAM each operating system requires.
• You will run 3 VMs
• You will upgrade this machine to utilize more of it's capabilties at some point

---

The server you found on Amazon, doesn't appear to work for your requirements, as is. There are a few things that will trip you up in this case.

First, there are only 8 GB of RAM included. Personally, I wouldn't consider that "lots of RAM".

Second, it doesn't come with hard drives. That means you have to decide what kind of storage space you want. On top of that, the drive bays don't appear to be included and the reviews say there isn't a place to mount regular drives. Configuring the storage, once you've gotten it, is an exercise left to you. Do you want one giant array? Do you want mirrored RAID? Do you want individual drives? You have to decide an configure the system.

Finally, the CPU is 6 cores. This means, that with 3 VMs, each of your VM can have only 2 cores. Depending on the type of development you do, this may or may not be enough.

On the plus side, this server appears to support dual power supplies. Only one is included, but if you wanted you could add a second one.

---

Now, can we make this server work? I think you can. Especially, since the price on Amazon is about $400 cheaper than I can find elsewhere. Let's use that to upgrade the specs.

This machine can handle a lot more hardware. The specs sheet from Lenovo shows what it can do.

• CPU: Up to two 10-core Intel Xeons.
• Ram: Up to 192 GB
• Drives: Up to 12 3.5" drives / 16 2.5" drives (requires the second power supply)

So, assuming we can't get all of that for $1,500, let's find something reasonable.

• RAM: Add at least another 16GB for a total of at least 24GB for an additional $110 (per 16 GB).
• Drives: If you want speed, get a few SSDs. If you want storage, get a few spinning drives. If you want a combination of speed and storage, mix it up. This is going to cost you a few hundred though. Adding in drive bays for the 3.5" drives at $15 each, adds an additional $15-$60 (assuming you are only filling 4 bays).

Right now, the total cost is roughly $1,200-$1,300. For this, you get a system that has manufacturer support (of some kind), states in it's spec sheets that it is compatiable with a variety of operating systems (including VMware ESXi and Citrix XenServer), and has been upgraded above basic levels. It is also $200-$300 under budget (excluding taxes and shipping). It will require you to add a bit of hardware yourself though.

If you are willing to go over your budget by a couple hundred, it may be useful to look at filling that second CPU slot. The system already comes with one Intel Xeon E5-2420 v2 2.2GHz 6C/12T. A second one will cost you about $650. Before you do this, though, consider how you are going to use this machine. Are you going to use it as an experiment and see if this works? If so, adding the second CPU at this point may be overkill.

Answer 2

It would cost you more to build one from scratch than just buying a used one

My recommendation is based on experience,

In last 6 months i have used 4U servers from 3 different vendors, Dell, HP, IBM

Models

Dell PowerEdge R900(more choices in HDD)

HP DL580 G5(most favorite)

IBM System X3850 M2(least favorite)

Dell PowerEdge R900

Pros

1. 4 Quad Cores
2. 128GB or 256GB RAM
3. You can buy used from ebay starting $300
4. Maximum 16 or 10TB Storage Choice of 5 of 3.5 or 8 x 2.5 inch drives formats.

Cons

1. Noise level : Like a jet engine
2. Heavy, difficult to lift and move with just one person

HP DL 580 G5

Pros

1. 4 Quad Cores
2. 128GB or 256GB RAM
3. You can buy used from ebay starting $200 for 128GB
4. Maximum 32TB or 16TB storage 8 x 2.5 inch drives or if you add another SAS RAID Card you can go up 16 x 2.5 Drives. Maximum storage 2TB per drive

Cons

1. Noise level : Loud but less louder than R900
2. Heavy, but lighter than R900 and it has hooks on both side for easy lifting so one person can lift it

IBM System X3850

Pros

1. 4 Quad Cores
2. 128GB or 256GB RAM
3. You can buy used from ebay starting $200 for 128GB

Cons

1. Noise level : quieter compared to both DL580 and R900

2. Similar in weight as DL580 it has hooks on both side for easy lifting so one person can lift it

3. Maximum 8TB Storage : 4 x 2.5 inch drives, yes only 4

My Personal Comments

I am divided between DL580 and R900. i like both of these and i have both of them at home. IBM System X3850 was Dead on arrival, machine powered on but it never recognized physical drives so i had to return it. It was quieter out of the three.

How i use them

I use them to Run Virtual Machines. I power them on when i need them. I have WOL Enabled with port forwarding for WOL so i power them on from my phone whether i am home or not. They sit in my office where i do not sit so i never hear the noise.

Storage Needs

Comparing only Dl580 and R900

I use SATA to store my data coz it's cheap and high capacity

R900 comes with either 3.5 or 2.5 inch drives. So if you storage need is 10TB or less then i would recommend going with 3.5 drives as you can get those cheaper like $50/each. and if money is not a concern then you can go with 2.5 inch drives and you can go to 8x 2TB SATA = 16TB on both R900 and DL580. one 2TB 2.5 STA would cost you around $100

Now if you have a need for even higher storage let's say 32TB Then go with Dl580 G5 you can add a second RAID card for like $15 and then just add 8 more 2.5 SATA drives so total storage 16 x 2TB = 32TB

Summary

• All the above 3 machines have similar CPUs and Similar memory and storage is the key deciding factor among st others mentioned above
• if you want 32TB of storage get DL580 add a second RAID card you would have to purchase 16 x 2 TB 2.5 SATA Drives that would cost you $1600 just for hard Drives + Server Cost

• If you want 16TB of space you can go with either R900 or DL580, my blind recommendation DL580, nicer and quieter compared to R900, it would cost you $800 just for Hard Drives + Server Cost

• if you are okay with 10TB of storage of less, go with R900 but make sure to get the one that takes 3.5 drives, then you need 5 x 3.5 2TB SATA Drives that's like 5 x $60 = $300 + Server Cost. Cheapest option with 10TB Capacity.

Answer 3

If you're OK with the noise and the potentially large power bill, you could always go for an off-lease rack-mount server.

For example, I currently own two high-grade but outdated (from the enterprise perspective) servers. One is an Apple Xserve from 2008, which I upgraded to dual quad-core 3GHz Xeon CPUs, 32GB of RAM and 6TB of storage - total cost including server and upgrades was $450. The second is a Dell PowerEdge R900 that I got for $350 - a deal I couldn't pass up - which came with quad quad-core Xeons (16 cores), 128 Gigabytes of RAM, and eight 146GB 15K SAS disks. While I got a good deal on this one, servers of this level of power do appear on eBay in the $400-550 range often.

The major issues you'll have with rack mount servers are noise and power consumption. The Xserve is the server I keep running all the time - its noise output is tolerable especially if it's in another room, and its power consumption is roughly the same as a mid-range desktop (250 watts idle, 375 watts under full 8-core load). The PowerEdge unfortunately is a bit too noisy, and even worse, its power consumption would put a hefty premium on my power bill (760 watts idle, 1000 watts under full 16-core load.) It only comes on when I actually plan to use its impressive power. :-)

There are of course other options out there. Dell has smaller 1U servers (PowerEdge 1950 line for example) which are even cheaper, in the sub-$200 range. If you're looking for a traditional desktop formfactor, Dell Precision workstations from 2009-2010 tend to go for decent prices and include many of the same advantages of the PowerEdge servers - ECC RAM, SAS disk support, high maximum RAM, etc. If you're willing to splurge, I've seen 12-core 64GB RAM machines with SAS drives going for $600-700. Of course, I'm not specifically plugging Dell here - HP, IBM, et al all have similar hardware going for roughly equivalent prices.

The upshot is that if you're serious about a home virtualization lab solution, older server gear is an awesome way to get into it without spending a huge amount of money. Hardware of this vintage is well past its prime in the enterprise and datacenter market, but it's just right for home users who want some powerful hardware for cheap. You of course forfeit modern things like ultra-low CPU power consumption and newer instruction sets (AES-NI etc), but again, for the cost, can't go wrong.

As for virtualization software - My Xserve is currently running VMware ESXi. Apple stopped supporting this machine years ago, but it runs ESXi on the bare metal - and does so beautifully. I have four VMs running constantly, with no appreciable speed issues. I haven't looked into Xen yet, but I will say that one of the biggest drawbacks to the free ESXi server is that VMware does not give you access to vCenter Server, their management engine, for free - thus you miss out on some of the advanced features of ESXi as you are stuck with the older, less capable vSphere Client.

Answer 4

I'd rather build than buy. It really depends on your needs though.

If you're worried about noise, a home system actually makes a decent VM box.

There's a few things I go for.

1. More cores. Get at least a quad core. For serious work a 'mainstream' core. I like a minimum of a quad core box - a core i7 is a decent workhorse machine, but both my recommendations are xeons. I typically run single core VMs

2. At least 16 gigs of ram - the 'right' amount is enough ram for the host + enough ram for each VM. 8gb is a bit of a minimum I guess

3. Enough storage. I favour SSDs for booting and speed reliant things - the samsung 850 pro 256gb is my preferred mainline SSD but the evo would work if you have the budget and want a 1tb SSD.

For reliable storage, I prefer HGST - they have a nas line of deskstars I'd suggest taking a look at, though I have a pile of older, very trusty desktar models

4. No OS. Most of the good VM servers are free,

If you're going to use this as a development box and want to run linux, fedora is a good bet. Excellent support for KVM and virt manager.

If you're buying as opposed to building, a dell precison might be a good option. Old workplace had them and they were tanky, reliable beasts, and they fall within your price range.

The 5000 series might be a good starting point - possibly the base model. I'd take a look at parts prices before adding any options since it may make sense to buy parts elsewhere to upgrade the ram and HDD. These things can go up to 8 cores if I remember correctly and are basically server grade parts set up as a desktop. I'm more familiar with the 7000 series, but those are for when you need two processors and is somewhat out of budget. Nonetheless, very solid machines, you're covered by the warranty, and if you're willing to spend a little extra, trivial to order what you need and know that it works

Upgrades from the base model? Up the ram to 16gb, drop the OS (most of the good VM hosts are free anyway). Get a 3TB HGST deskstar drive. I'd be tempted to slot in an SSD for the OS (I would suggest a 256gb Samsung 850 pro) , but these things take a while to boot due to the raid controller.

If you're building

Might take a bit of tweaking to get it in your budget but a X10SDV-8C+-LN2F might be a good start Its ~1000usd but that gets you 8 cores. Load it up with 32gb (4x8, or 2x16) of ram at least of the cheapest standard DDR4 ram you can find. In theory, you can go up to 128 gb with Rdimms. Don't bother with ECC in this case, not really worth it.

Its mini itx (so you can use a smaller case).

Its somewhat limited in terms of video (VGA...) but also has IPMI (for out of band management, which is cool) and is designed for being a VM host, but at reasonable power consumption. Most of the xeon-d models are interesting, but this is the 'cheapest' one I could find with a fan.

USB ports are somewhat limited as well, and you have one pciex16, but that's more than you would on a laptop.