I am trying to decide on two bays of NAS. Should I alloctate a spare "hot" drive? It's not clear if the drive is spinning always or if it it powered, but the NAS only spins it up when there is a need to resilver the array.
Home NAS
- Thread starter LiamC
- Start date
ddrueding
Fixture
Interestingly the weather here now is just about what the weather was in Monterey, California in the '80s. Only another few years and it'll get there.
I'm trying to access different arrays. For some really stupidly reason, the QNAP have segregated the OS and FS so the QTS Hero is ONLY ZFS and the QTS is only EXt4. The only way to change is to flash the firmwares each time, which is not feasible. So now I have two QNAPs locally and two QNAP external drives, and yet another NAS in a different region-zone and not all compatible! It's a cluster....
Synolology Allows EXT4 and brfts volumes in the same system. But that ship has sailed due to lack of compatibility with normal hard drives.
Synolology Allows EXT4 and brfts volumes in the same system. But that ship has sailed due to lack of compatibility with normal hard drives.
Mercutio
Fatwah on Western Digital
I really do think your frustration at existing NAS products is a compelling argument for building some kind of appliance device. There are a number of ITX boards with integrated Celeron CPUs, 6x SATA, 4x2.5Gbps LAN and an M.2 port that are very affordable. Combine with something like a Silverstone DS380 and you have a pretty good low power build with tons of flexibility from FreeNAS or TrueNAS.
Mercutio
Fatwah on Western Digital
Do you have a datacenter to host such a system in another region or do you have a second residence somewhere besides California?
ddrueding
Fixture
Putting it in storage isn't useless, but one of the advantages to having it powered up is that when failures do happen you can fix the system and get the redundancy back. In storage failures are less likely, but what is more likely is that enough failures will happen to overcome the redundancy and it will lose data.
Just my 2c
Just my 2c
ddrueding
Fixture
All good. I'd consider it sitting there good for a year or two, but at that point it should be spun up, a consistency check run, and any drives that failed replaced and array rebuilt. That may be paranoid, but I don't like unknowns, and with it offline it feels like Schrödinger's hardware.
Mercutio
Fatwah on Western Digital
MaxDigitalData is offering what appear to be rebadged 7200rpm Seagate Drives with a three year warranty for EXTREMELY reasonable prices. 14TB for $129? 16TB for $169? I'm assuming they're pulled working datacenter drives, but the specs on Amazon say they're SATA and don't mention whether they're renewed or not.
Mercutio
Fatwah on Western Digital
I probably will soon.
Interesting that they sell their own labeled drives but also their labeled drives but marked as "Renewed" for even a little less money. Some searches hint that MDD wipes the smart data on them before rebranding.
I'm not sure what I would trust these with. I do need to pull the trigger on refreshing my 6TB HDDs sooner than later.
I'm not sure what I would trust these with. I do need to pull the trigger on refreshing my 6TB HDDs sooner than later.
Chewy509
Wotty wot wot.
In case anyone missed it, Drobo has ceased sales...
arstechnica.com
Drobo, having stopped sales and support, reportedly files Chapter 7 bankruptcy
Company posted in January that it had "transitioned to a self-service model."
arstechnica.com
ddrueding
Fixture
48TB of nearly silent storage with a 10Gb link and low power consumption for $3500 is very tempting for me at the moment. Would be nice to be able to have my local NAS in my office instead of buried in a distant closet.
The 10W Celerion Jasper Lakes 11th gen CPU is quite underpowered, but it's a good trend. The older QNAP M.2 SSD NAS has only four NVMes and 2.5GbE. I would not be using those cheap DRAMless QLC SSDs, but that's me. Why not just put SSDs inside your computer case? You'd get more bandwidth even with SATA III drives on a RAID controller and mnay times more with various M.2s on the board or a PCIe card.
Last edited:
ddrueding
Fixture
I do lots of experimentation and general shenanigans with both my desktop and my home server. I want a thing to hold my stuff that I don't mess with. Just an appliance that I don't have to worry about. I have an online backup (AWS), so I'm not really worried about data loss, just downtime. I think SSDs are reliable enough that I'll swing for a 12-drive RAID-0 so that the CPU doesn't have to do much work. I don't think the performance of the SSDs is that big a factor if they only have a single 10Gb port to connect to. We'll see; I have too many things on my shopping list at the moment.
Mercutio
Fatwah on Western Digital
That system is kind of weird though. It doesn't support being an iSCSI target, so you'd be doing SMB or maybe NFS. That's definitely less than ideal for running VMs, and it's not like the local CPU could manage such a thing. All SSD media storage is cool from a bragging rights perspective though.
My primary storage server/VM host is running on a pair of 16 Core Haswell Xeon CPUs, so I feel the power draw pain, but whatever miracle Lenovo worked with cooling an RD550, at least the damned thing barely makes any noise.
My primary storage server/VM host is running on a pair of 16 Core Haswell Xeon CPUs, so I feel the power draw pain, but whatever miracle Lenovo worked with cooling an RD550, at least the damned thing barely makes any noise.
That Asustor is a neat machine even if it's not ideally something I'd want. Having "only" a single 10Gb port isn't a big deal either. I think many people (not you all) underestimate how much data it's actually capable of moving and usually isn't the bottleneck.
I would also love to reduce the power draw of my old NAS which is a dual socket E5-2670 (~2012). That's great to hear the RD550 isn't super loud given its 1U size. I would like a 12-bay 3.5" rack mount and had looked at Lenovo but they're pricey. Did you buy yours new?
I would also love to reduce the power draw of my old NAS which is a dual socket E5-2670 (~2012). That's great to hear the RD550 isn't super loud given its 1U size. I would like a 12-bay 3.5" rack mount and had looked at Lenovo but they're pricey. Did you buy yours new?
ddrueding
Fixture
Fortunately I don't need it as primary storage for VMs as the server has a 2TB M.2 in it and that's overkill for my Factorio docker and sandbox. Backing up over SMB is fine. After doing so much work to quiet my office down, I can't have a spinning disk in here. At the moment my old Synology box is in a utility closet only accessible over WiFi. 6E, but still painful.
Mercutio
Fatwah on Western Digital
One of my customers closed their office during the pandemic and I just brought it home. Their setup is a just VMs in my datacenter now. Which are also sitting on an RD550.
I added a second PSU and switched the single CPU for a pair of bigger boys (I think I went E5 2670 v3 to now having 2698s, which cost me IIRC $100) and quadrupled the RAM. Only down side is that it doesn't have U.2 bays. Much like having an older Thinkpad, the parts are readily available and right now at least, they're dirt cheap. I have copies of customer VMs on hand and my home lab needs are met.
It's not silent, but I can't hear it with the ceiling fan in the back bedroom turned on, except when it first powers on. Then it makes a noise that can wake the dead for about 10 seconds.
Mercutio
Fatwah on Western Digital
My servers in production are Broadwell or Haswell Xeons, either Dell R630s or Lenovo RD550s. They have crap tons of RAM and most of them are SMP, ~2.4GHz/core systems that are perfectly adequate for the load. They're old, old enough to be E-waste for corporate systems, but they're also perfectly fine except for storage needs.
These machines don't have U.2 and many of them don't have any open PCI slots. The Intel S4500 drives I use were new in 2018 and are some of the last high endurance 2.5" SATA drives made (note: I have cold spares; I'm not going to run out of those drives anytime soon). I can't remember if they're MLC or TLC off the top of my head, but they're good for .95DWPD. If there are any spinning drives in those same machines, they're also no bigger than 4TB; Seagate makes 5TB 2.5 drives, but they're SMR and screw that for anything in production. Anything bigger than 4TB in 2.5" SATA or for that matter nVMe is probably on crummy NAND.
The combination of drive density (only 8 2.5" drives per chassis) and capacity is becoming an issue. I don't really want to buy a new server just to get some U.2/U.3/EDSFF, but it's looking like I either have buy something new enough to support better drives or I have to homebrew something. Preferably something 1U, because if I use more than that I get kicked up to the next tier of pricing.
I'm tempted to put together a modest system with a 10GbE NIC, an extra 8 or 12 of those 4TB Intel S4500s and HOPEFULLY find room inside a 1U chassis for a 4x U.2HBA so I can have some larger drives as well. I guess they'd probably just sit loose inside the chassis. Whatever.. Set the whole thing up as an iSCSI target.
What I do know is that nobody is selling a generic chassis with better-than-SAS drive connectivity, and in the land of affordable NAS, the trend is to use either nonstandard drives (thanks, Synology) or shitty consumer SSDs. I'm assuming I can find a small motherboard with 10GbE integrated and 2x 16xPCIe slots. That's probably a tall order, but I'll bet it's still cheaper than trying to deal with a new, entry level 1U system from a tier-1 OEM.
These machines don't have U.2 and many of them don't have any open PCI slots. The Intel S4500 drives I use were new in 2018 and are some of the last high endurance 2.5" SATA drives made (note: I have cold spares; I'm not going to run out of those drives anytime soon). I can't remember if they're MLC or TLC off the top of my head, but they're good for .95DWPD. If there are any spinning drives in those same machines, they're also no bigger than 4TB; Seagate makes 5TB 2.5 drives, but they're SMR and screw that for anything in production. Anything bigger than 4TB in 2.5" SATA or for that matter nVMe is probably on crummy NAND.
The combination of drive density (only 8 2.5" drives per chassis) and capacity is becoming an issue. I don't really want to buy a new server just to get some U.2/U.3/EDSFF, but it's looking like I either have buy something new enough to support better drives or I have to homebrew something. Preferably something 1U, because if I use more than that I get kicked up to the next tier of pricing.
I'm tempted to put together a modest system with a 10GbE NIC, an extra 8 or 12 of those 4TB Intel S4500s and HOPEFULLY find room inside a 1U chassis for a 4x U.2HBA so I can have some larger drives as well. I guess they'd probably just sit loose inside the chassis. Whatever.. Set the whole thing up as an iSCSI target.
What I do know is that nobody is selling a generic chassis with better-than-SAS drive connectivity, and in the land of affordable NAS, the trend is to use either nonstandard drives (thanks, Synology) or shitty consumer SSDs. I'm assuming I can find a small motherboard with 10GbE integrated and 2x 16xPCIe slots. That's probably a tall order, but I'll bet it's still cheaper than trying to deal with a new, entry level 1U system from a tier-1 OEM.
ddrueding
Fixture
Some sort of homebrew SAN sounds like the right play. Those cheap SSDs in large enough numbers with enough redundancy might be the best option?
Mercutio
Fatwah on Western Digital
I spend a lot of time worrying about write endurance, probably more than anything else, which makes cheap SSDs unappealing. I'm tempted to just buy something like a barebones Lenovo SR630 with either 10x 2.5" SAS/U.2 combo or EDSFF bays. I'm a little daunted at what I might be getting into for big, enterprise SSDs though. Samsung PM1743s have that magical 1DWPD rating and they're supposed to ship in capacities up to 16GB. I'd at least be able to scale up from where I'm at now using those.
I'm a de-facto tiny cloud provider as it is, but my reasoning is that the costs to build out whatever NAS or fileserver setup will be lower than hundreds of dollars in monthly fees for a slow elastic storage pool on AWS or Azure.
I'm a de-facto tiny cloud provider as it is, but my reasoning is that the costs to build out whatever NAS or fileserver setup will be lower than hundreds of dollars in monthly fees for a slow elastic storage pool on AWS or Azure.
Mercutio
Fatwah on Western Digital
There aren't many options for a roll-your-own 1U hot swap storage chassis. Newegg has a discontinued Athena Power case with 12 2.5" bays, but those bays only supports an older Mini-SAS connector spec that doesn't include the high-bandwidth interface I'm looking for, and I don't see an option from Athena Power to buy an upgraded backplane.
I can get an Adaptec HBA-1200-16i which would more than cover a full spread of internal U.2/U.3 drives, which is definitely the way the wind is blowing for high capacity nVMe (why not m.2? In a lot of cases m.2 products aren't being made in high capacity high endurance configs; the m.2 variants of those drive will top out at 4 or 8TB while the bigger guys are on u.2/3). Those drives aren't that BIG, so I suppose that an old 1U chassis could just have a bunch of internal drives sitting loose jand a Kaiju of cabling nightmare. And also thermal problems, probably. Is that worth a shot?
If dd's Asusstor-weirdo-thing didn't have a joke of a CPU and had 10GbE, it would be worthy of consideration. I don't think 2.5Gb networking is acceptable for direct access to nVMe though. It's still m.2-only as well.
I can get an Adaptec HBA-1200-16i which would more than cover a full spread of internal U.2/U.3 drives, which is definitely the way the wind is blowing for high capacity nVMe (why not m.2? In a lot of cases m.2 products aren't being made in high capacity high endurance configs; the m.2 variants of those drive will top out at 4 or 8TB while the bigger guys are on u.2/3). Those drives aren't that BIG, so I suppose that an old 1U chassis could just have a bunch of internal drives sitting loose jand a Kaiju of cabling nightmare. And also thermal problems, probably. Is that worth a shot?
If dd's Asusstor-weirdo-thing didn't have a joke of a CPU and had 10GbE, it would be worthy of consideration. I don't think 2.5Gb networking is acceptable for direct access to nVMe though. It's still m.2-only as well.
Finding even an enterprise 1U server with 12 x 2.5" (in the front) is rare. I couldn't find any at Dell, Lenovo, Gigabyte, or Supermicro. Some have 10 in the front and two in the rear. That Athena Power case is neat having 12.
Mercutio
Fatwah on Western Digital
If nothing else, I feel like I've identified a product that needs to exist.
I've been looking at the TrueNAS Mini R (2U rack mount) which starts at $1900 for bare bones (no HDDs) and I like the idea of the appliance given it's low power consumption and relatively quiet in a 2U - 12 x 3.5" bay size. It's a Supermicro MB with an Intel Atom C3758 CPU that's a few years older (2017) but it has 10Gb and IPMI which I both want.
I've been trying to part together a similar build for the same or less money but it's proving more challenging that I expected. The closest options are a used Supermicro chassis and motherboard but from ebay/used. Are there other 2U - 12x3.5" chassis I should be looking at?
My basic requirements are:
Low power consumption and heat generation (~150W-200W)
32-64GB ECC RAM
IPMI remote management
10Gb (preferably SFP+)
12 bay 3.5", 2U rack mount
No vendor lock-in
SAS backplane
single or non-redundant PSU is fine but dual PSU is also fine
I've been trying to part together a similar build for the same or less money but it's proving more challenging that I expected. The closest options are a used Supermicro chassis and motherboard but from ebay/used. Are there other 2U - 12x3.5" chassis I should be looking at?
My basic requirements are:
Low power consumption and heat generation (~150W-200W)
32-64GB ECC RAM
IPMI remote management
10Gb (preferably SFP+)
12 bay 3.5", 2U rack mount
No vendor lock-in
SAS backplane
single or non-redundant PSU is fine but dual PSU is also fine
Mercutio
Fatwah on Western Digital
I'm finding that U.2/U.3 support on a backplane is the most exotic part of my wishlist. An Adaptec HBA1200 is affordable and can support tri-mode (SAS/u.2/u.3) operation, but I'm really only seeing u.2/u.3 backplanes on Supermicro and the usual Tier 1 OEMs.
Supermicro makes a 20 bay 1U backplane but it doesn't seem to be compatible with any of its 1U servers (at least, I don't see one with a 20 bay option as I'm looking at their site), and 18 of those bays are nVME only.
HP makes the Proliant DL380, which exists in a 12x3.5" configuration, which looks like it retails for around $1600 in relatively bare versions. You'd probably have to fight to find one with a 10GbE module instead of 4x1GbE but those systems seem pretty easy to find off-lease. Dell R740xd is going to be pretty similar, although it can be tricked out with both mid-bay and rear-bay nVME as well, for a total of 8x2.5" drives inside.
I just don't wanna mess with 2U if I can help it.
Supermicro makes a 20 bay 1U backplane but it doesn't seem to be compatible with any of its 1U servers (at least, I don't see one with a 20 bay option as I'm looking at their site), and 18 of those bays are nVME only.
HP makes the Proliant DL380, which exists in a 12x3.5" configuration, which looks like it retails for around $1600 in relatively bare versions. You'd probably have to fight to find one with a 10GbE module instead of 4x1GbE but those systems seem pretty easy to find off-lease. Dell R740xd is going to be pretty similar, although it can be tricked out with both mid-bay and rear-bay nVME as well, for a total of 8x2.5" drives inside.
I just don't wanna mess with 2U if I can help it.
Mercutio
Fatwah on Western Digital
Looks like $2000 for an R740xd with 24C/48T, 32GB RAM and a 2x 10Gb NIC. The TDP on both CPUs pushes your requirements but there's probably a reduced power option in the firmware; 85W looks about as low as Xeon Scale stuff seems to get and these are sold with two CPUs. I'm pretty sure you can access iDRAC via whatever IPMI client you have and this system is using a baseline HBA rather than a RAID card setup. Because of how all this stuff works, wait until Xmastime and maybe the price will have dropped another $300 as the next things start coming in for refurbishers.
Thanks for that link, I wasn't familiar with that site. Have you used them for server purchases or know if they're a reliable source?
Coincidentally I'm very familiar with the R740xd platform. The clustered appliance product I developed with my former company is based on the R740 so I know them inside and out. It's been very reliable overall with loads of them in the field and idrac is fine for my needs.
Is there an issue with the 2U size in general or are you avoiding them to conserve rack space?
Coincidentally I'm very familiar with the R740xd platform. The clustered appliance product I developed with my former company is based on the R740 so I know them inside and out. It's been very reliable overall with loads of them in the field and idrac is fine for my needs.
Is there an issue with the 2U size in general or are you avoiding them to conserve rack space?
Mercutio
Fatwah on Western Digital
The issue with 2U for me is that it will bump me up to the point in my datacenter billing where I need to pay for a full rack every month rather than half a rack. As you might imagine, I'd rather not go there.
I bought Dell hardware from Savemyserver a couple times, but it's been a few years. They seem easy to deal with, but honestly, I just saw that they have a bunch of stuff on Ebay yesterday. I'm glad they're still out there. They don't handle Lenovo stuff, so they haven't been on my radar as much.
A Netgear ReadyNAS 3112 is kind of an interesting device. 1U, 12x3.5" drives. Down sides are that it doesn't do hot swap, they're SATA only, has no better network connection than GbE and run whatever weird OS Netgear put on it rather than something familiar like FreeNAS. They have Atom CPUs and modest RAM, but they do support acting as an iSCSI target. Throughput after protocol overhead is probably somewhere around 90MB/sec, but they're cheap and I'm guessing they don't make much noise. Not quite what I'm looking for, but I'd forgotten that Netgear made NAS units at all and I was surprised that they had anything that could catch my attention at all.
I bought Dell hardware from Savemyserver a couple times, but it's been a few years. They seem easy to deal with, but honestly, I just saw that they have a bunch of stuff on Ebay yesterday. I'm glad they're still out there. They don't handle Lenovo stuff, so they haven't been on my radar as much.
A Netgear ReadyNAS 3112 is kind of an interesting device. 1U, 12x3.5" drives. Down sides are that it doesn't do hot swap, they're SATA only, has no better network connection than GbE and run whatever weird OS Netgear put on it rather than something familiar like FreeNAS. They have Atom CPUs and modest RAM, but they do support acting as an iSCSI target. Throughput after protocol overhead is probably somewhere around 90MB/sec, but they're cheap and I'm guessing they don't make much noise. Not quite what I'm looking for, but I'd forgotten that Netgear made NAS units at all and I was surprised that they had anything that could catch my attention at all.