UnRAID Server Build

[ddrueding]

After watching some videos on YouTube demonstrating VMs with direct exposed hardware while wanting a NAS with 10GbE and SSD file caching, I decided to start a build similar to that in the videos (links at bottom of post).

UnRAID will do a bunch of things, the things I'm interested in at the moment are the following:

3x direct-connected Windows 10 VMs
NAS with 10GbE and SSD cache
PLEX/DropBox/TeamSpeak servers in a Docker containers
Firewall?

So for hardware, I'm scavenging some server stuff that I should be selling and is pretty solid overkill for this project.

ASUS Z9PE-D8 WS
2x Intel Xeon CPU E5-2687W (16 cores+Hyperthreading @ 3.10GHz turbo to 3.8GHz)
8x Kingston KVR16LR11D4/16 (128GB ECC DDR3 1.365v @ 1600MHz)
Seasonic SS-600H2U
Silverstone Tec 3U RM316

-I/O-
2x nVidia Quadro K1200
1x nVidia Quadro K420
+Something cheap and low-power for server interface

-Drives-
2x Kingston 480GB SSD
2x Samsung 480GB SSD
6x HGST 8TB 7200RPM

-Networking-
Onboard 2x GbE
1x Intel X540T2 (2x 10GbE)
1x Intel X520-DA2 (2x SFP+)

At the very least the CPUs will be water cooled, possibly the GPUs as well (all card slots may end up full).
I'll probably pull the X540T2 as the system won't boot with it in, this seems to be a common issue with this card in Linux.
I'm having a hard time getting the hotswap bays to play with the onboard SATA ports; I'm using SFF8087 to SATA cables "backwards" and not sure that will work. None of the drives are being detected.

PSU is a poor choice for this chassis as they are different form factors, but I couldn't find an appropriately powerful (600W may not actually be enough) maximum efficiency PSU of the right size.

Current NIC plan is to use one of the SFP+ to direct-connect to my workstation and the other into the switch that runs everything else.
I'd like to use the two onboard NICs as part of a firewall. Haven't looked into that yet, may end up learning a lot about Linux iptables and such.

[video=youtube_share;LuJYMCbIbPk]https://youtu.be/LuJYMCbIbPk[/video]
[video=youtube_share;OhVlcw1Wpns]https://youtu.be/OhVlcw1Wpns[/video]

8:30PM, time to leave work. More to come.

 

[Handruin]

I'm certainly interested to see where this goes. I've not tried using UnRAID for anything yet but it looks to be based on KVM under the covers. I don't fully follow the use of a firewall in this config can you explain a little more. Are you connecting this entire setup right onto the public network?

 

[ddrueding]

This is for my house. The goal is to fit as much of my computing infrastructure into this machine as possible to simplify the hardware stack and reduce power consumption. Putting the firewall in the same machine eliminates another electronic device that consumes power. Rather than connect this whole thing to the internet, my plan was to build a VM that directly exposed to the two Gigabit NICs. One would be WAN and one would be LAN. This would, as much as possible, separate the security of the firewall from the rest of the machine. The LAN port on the firewall would physically have to be wired into the switch and through that provide internet to the rest of the box.

 

[Handruin]

This is for my house. The goal is to fit as much of my computing infrastructure into this machine as possible to simplify the hardware stack and reduce power consumption. Putting the firewall in the same machine eliminates another electronic device that consumes power. Rather than connect this whole thing to the internet, my plan was to build a VM that directly exposed to the two Gigabit NICs. One would be WAN and one would be LAN. This would, as much as possible, separate the security of the firewall from the rest of the machine. The LAN port on the firewall would physically have to be wired into the switch and through that provide internet to the rest of the box.

If you're looking to firewall that setup you might want to see if you can leverage pfsense as a VM. A get wanting to simplify the hardware stack but I feel you're trading it for complexity in software stack and possibly increased risk due to an unconventional setup. I still think it's worth keeping the router/firewall outside of this environment if you can manage but if you want to learn and experiment it's a neat way to do it. I think Mercutio did something similar a while back under ESXi but did it with virtual switches rather than connecting the LAN side back into the same machine. You may be able to do this in UnRAID also but I don't know it's abilities. A network config like that would be possible under ESXi and your inside VM communication would be 10Gb+ via virtual switches. If you move your firewall into this VM setup what will you do for Wifi and access points? You may even be able to estimate if you're actually saving any power by putting it into this config.

 

[ddrueding]

Currently I'm running a dedicated firewall without WiFi and running a pair of Ubiquiti AC Lite access points in the house. The firewall is fast (not willing to compromise my 150Mbps+ connection), but power hungry. Can't remember the name, but we've discussed it before and it is a 1U chassis with a cooling fan.

I'm sure it is possible to connect a VM firewall directly to the internal virtual switch, but that seems much easier to get wrong. My current thought is to use a software package I know (Smoothwall) in a stock-ish installation. I have some experience with pfsense as well, so we'll see which one plays better. Right now I need to get the hardware bugs sorted first.

 

[ddrueding]

This is probably the right chassis/MB/power supply for this kind of project:

[video=youtube_share;DhZJ66l82r8]https://youtu.be/DhZJ66l82r8[/video]

 

[ddrueding]

For those concerned about the performance of their virtualized workstations. The reason I'm not putting my own workstation in the same machine is that unRAID won't support SLI, and I think I'll want that when the Oculus arrives.

[video=youtube_share;FvcxPufSRNo]https://youtu.be/FvcxPufSRNo[/video]

 

[CougTek]

This is probably the right chassis/MB/power supply for this kind of project:

I haven't seen the second video you linked, but for the one with the SuperMicro 4027GR-TR enclosure, it's obvios that this guy ddoesn't know much about SuperMicro's lineup. All the models with LGA2011 finishing with a 7 (like 4027[/7]) are for the first and second generation of Xeon E5. The models for the third generation of Xeon E5 (those who use DDR4 memory) finish with an 8. I guess he'll be quite disappointed when he'll try to install DDR4 modules into his new 4027GR-TR rendering server. He'll also be disapointed when he'll figure out that no, you can't put 18-core Xeon into this motherboard, but 12-core Xeon E5-2697v2 maximum.

The model he should have gotten, since he wanted 10G ports, was the 4028GR-TRT. I guess he'll learn quickly, especially considering the sticker price of his mistake.

 

[ddrueding]

Sounds about right. One of the things I like about that channel is that all the errors make it in; there is no hand-waving "what mistake?" editing shenanigans going on. Looking forward to the next video...

 

[CougTek]

Started to watch the second video...

It's the first time I hear someone calling a network card a "NIC". I've heard and used "N.I.C.", but never NIC.

 

[Handruin]

Started to watch the second video...

It's the first time I hear someone calling a network card a "NIC". I've heard and used "N.I.C.", but never NIC.

I've always pronounced them as "NIC" (nick) as does anyone I've ever worked with.

 

[CougTek]

Sounds about right. One of the things I like about that channel is that all the errors make it in; there is no hand-waving "what mistake?" editing shenanigans going on. Looking forward to the next video...

Speaking about mistakes, can I mention his outrageous lack of E.S.D. procedure? I mean, he handles a motherboard like it's a plastic board and throws it on foam inside a cartoon box. Damn! I would never buy stuff from this guy.

 

[CougTek]

I've always pronounced them as "NIC" (nick) as does anyone I've ever worked with.

Oh well. My english-speaking sample is quite limited, so I'll trust you guys on this. There must be a ton of other things I don't pronounce like you guys anyway, not being a native english speaker.

 

[Handruin]

Speaking about mistakes, can I mention his outrageous lack of E.S.D. procedure? I mean, he handles a motherboard like it's a plastic board and throws it on foam inside a cartoon box. Damn! I would never buy stuff from this guy.

If his lack of ESD precautions bother you...you will not believe some of the other videos he's made such as the ones cutting apart Titan X GPUs or how he cleaned out tons of drywall dust from his servers. Linus and his team have some fun projects and I enjoy their channel as entertainment much like I enjoy Top Gear for entertainment value over and above anything serious about reviewing cars. Linus and crew do put together some bizarre projects like their whole-room water cooling. Not all are meant to be practical but I admire his creative outlet. Plus, he gets a shit ton of vendor-supplied demo parts so it's likely he never even paid for that super-expensive SuperMicro chassis.

 

[Howell]

I've always pronounced them as "NIC" (nick) as does anyone I've ever worked with.

For me, in the old days it was a network adapter much like the video adapter or scsi adapter. Post 10base2 though it's been a NIC.

 

[Handruin]

For me, in the old days it was a network adapter much like the video adapter or scsi adapter. Post 10base2 though it's been a NIC.

I also use network adapter and likely fluctuate between calling it a NIC or network adapter. Lately I've also been calling them SFP+ adapters for the 10Gb cards.

 

[Howell]

Haha, I likely depends whether we are looking at the inside or outside.

 

[Stereodude]

I've always pronounced them as "NIC" (nick) as does anyone I've ever worked with.

Same here. "nick"

 

[Stereodude]

Speaking about mistakes, can I mention his outrageous lack of E.S.D. procedure? I mean, he handles a motherboard like it's a plastic board and throws it on foam inside a cartoon box. Damn! I would never buy stuff from this guy.

Do you actually wear a ground strap and work on an ESD mat? I don't.

I do keep the system plugged, but switch off at the power supply. I tried to keep one hand on the case as much as possible so that I'm ground to it, and I touch the case to get grounded before touching things if I get up and come back...

 

[blakerwry]

I don't use an ESD strap either. I do the same procedure of keeping the machine plugged into AC ground and grounding myself to the case before touching anything. If the PSU doesn't have an off switch, there's usually a power strip that does. I was taught at Uni that you can pass components between two people by making contact with your off-hands and then passing the component. Once the component is passed, you can stop holding hands.

 

[Handruin]

Do you actually wear a ground strap and work on an ESD mat? I don't.

I do keep the system plugged, but switch off at the power supply. I tried to keep one hand on the case as much as possible so that I'm ground to it, and I touch the case to get grounded before touching things if I get up and come back...

I don't use an ESD strap either. I do the same procedure of keeping the machine plugged into AC ground and grounding myself to the case before touching anything. If the PSU doesn't have an off switch, there's usually a power strip that does. I was taught at Uni that you can pass components between two people by making contact with your off-hands and then passing the component. Once the component is passed, you can stop holding hands.

This is how I've always managed ESD when building/working on components for a system.

 

[CougTek]

Do you actually wear a ground strap and work on an ESD mat? I don't.

Nope

I do keep the system plugged, but switch off at the power supply. I tried to keep one hand on the case as much as possible so that I'm ground to it, and I touch the case to get grounded before touching things if I get up and come back...

That's what I do too. I also avoid touching the connectors when handling cards or RAM sticks. I generally hold them by the PCB side. When you manipulate a lot of electronic hardware, caring about E.S.D. in the way you handle stuff does make a difference in the amount of defective hardware you end up with. Of course, there's always one lunatic claiming that he never cared about this and he's never killed any equipment that way, much in the way that some people tell that one of their grandpa lived up to 90 years old despite being a heavy drinking/smoker so that cigarettes killing you is bullshit, but I hope you all know better.

 

[snowhiker]

Started to watch the second video...

It's the first time I hear someone calling a network card a "NIC". I've heard and used "N.I.C.", but never NIC.

Funny, cause it's the opposite for me. I've never heard it spelled out N.I.C, but have always pronounced it and heard it as NICK.

 

[Mercutio]

From my recollection, they've been "nicks" since the early 1990s.

What format are you using for your virtualized desktop?

Am I wrong or does Unraid use yet another weird VM format? It doesn't look like it's set up to handle import of VHDs or OVF files.

 

[Handruin]

From my recollection, they've been "nicks" since the early 1990s.

What format are you using for your virtualized desktop?

Am I wrong or does Unraid use yet another weird VM format? It doesn't look like it's set up to handle import of VHDs or OVF files.

I'm unclear what format the VMs use but I thought it was based off Xen. I have no experience with the Unraid product.

 

[Mercutio]

I only mention it because of the annoyance at having to convert 100GB+ files from one VM file type to another. The only mention I could find is that it uses "RAW" files, which I suppose in this context is probably dd images or something.

 

[ddrueding]

The video above says that unRAID uses the KVM tech from RedHat. I don't know what VM format that requires.

 

[Handruin]

Interesting. I wonder if they're leveraging Open Stack and generalizing on that? When I did some quick searches I saw Unraid turn up as being Xen but maybe that's a recent change or it supports both hypervisors? I agree with you Merc that the conversion for large VMs is a PITA.

 

[Howell]

http://www.linux-kvm.org/page/FAQ#What_virtual_disk_formats_can_KVM_use.3F

Haven't heard QEMU in a while.

 

[Mercutio]

Here's an interesting question for folks:

Does anyone know what if any format I could get away with for building a semi-universal collection of Guest Windows OS desktops that I could use without running afoul of Product Activation? I've found that if I virtualize on Hyper-V, I definitely have to re-activate the licenses, so that's probably out. I wouldn't care except that I have some Windows 8 retail keys I'll never, ever use and making five or six Windows 10 VMs seems like a more productive application than anything else I could do with them.

 

[Howell]

What necessitates reactivation under hyper-V? Once you get them to windows 10 they should reactivate fine so I guess I'm not seeing the problem.

 

[Mercutio]

I'm not sure but I've observed it to be the case when I move the guest files from one Hyper-V host (e.g. my laptop to a full-blown Server 2012 install) to another. Oddly, I haven't seen Windows Server guests do it, just desktop Windows.

Ideally, I'm trying to find some combination of settings to make a hypervisor-agnostic setup that I can move around if I need to. I have Xen, Virtualbox, ESXi and Hyper-V hosts scattered around my customer sites, so there's at least a thought that I could need any of the above for one reason or other.

 

[Howell]

I've not used Hyper-V and it has been awhile since I've used VMWare workstation. But I don't think I've had any trouble moving VMDK disks from workstation to ESXi, or even back over to Virtualbox. I've not had trouble moving virtualbox guests either. I'm only working with Workstation versions. I don't have to do it very often so I'm not 100%.

 

[Howell]

Are you exporting or just copying?

https://blogs.msdn.microsoft.com/vi...to-activate-windows-when-i-move-a-vm-hyper-v/

 

[ddrueding]

I don't have any information for you I'm afraid. So far I've kept things in the VMWare world and haven't had to worry about it.

 

[Mercutio]

I've tried both ways. I normally just copy them over, but it shouldn't matter since SIDs et al aren't terribly relevant and are relatively easy to fix after the fact in any case.

My thinking is that Microsoft is fingerprinting the guest OS installs somehow, or that switching from an up-level Host (Windows 10) to a down-level one (Server 2012 that isn't R2) might be the cause, but it's still a matter of concern since there's still at least a technical tangible value to each OS activation.

 

[Howell]

Yeah, I would expect that when whatever the MS equivalent of the .vmx changes. But I would not expect the guest container description to change with an export.