Building a Storage Server Thread

[Mercutio]

OK, I know I'm not the only person here who has put together a workgroup storage server.

How about some suggestions to do firingsquad one better?

The goal of this process will be suggestions that can be integrated into a single article.

It might also be a good idea to impose a budget constraint, say, $1500 or $2000, and to propose applications and software for your TB-class server (e.g. "I see a need for a video server and so I'd set things up this way...")

 

[timwhit]

Any server that will store large amounts of data should run at least RAID1. RAID5 is more economical. Most Server OS's can handle either RAID level on a software level.

 

[Adcadet]

remind me again why they went with such a powerful CPU....would not a P2-400 been sufficient?

 

[Adcadet]

oops, wrong thread.

 

[Mercutio]

They wanted something they could use as a workstation. I can theorize that they might've wanted to realize the performance improvement of the 200Mhz-native bus used by high-end P4s, but they didn't make any other choices that would lead to a fast disk subsystem (e.g. WD *JB drives, RAID0, RAID1 on an Escalade...)

Translation: We want an XP machine with a bunch of hard disks.

I agree that RAID1 or RAID5 is a practical necessity. That much storage with no redundancy at all is a travesty.

I'd like to put forth that there is no better bang for the buck than Samsung 160GB drives. $110 apiece. 6 of 'em = 960GB. Plus they're quiet. Plus they're cool.

We're pretty much going to need some kind of exotic solution for all the IDE channels. 3ware 7506-8 if we're looking for high-end solutions (One -8 is $100 cheaper than 2 -4s), or some bastardized collection of onboard and promise controllers if not. Firingsquad went with the latter option.

 

[sechs]

Is buying a cheap, off-the-rack server, like a Dell, stuffing a lot RAID controllers and disks into it, and installing your favourite *nix just too simple?

 

[blakerwry]

if you're buying many drives (ie, not getting special on rebates) then yes, the Samsungs are good deals price/capacity wise.


Personally, i would get a hardware card (you'll need the extra channels anyway) and either a linux or winNT based OS (depending on your particular knowledge).


You'll also want gigabit ethernet or dual 100baseT. Most linux distros dont support Transmit load balancing yet (has to be a 2.4.22 or later kernel I believe), so if going dual NIC then win2k3 would probably be better.

CPU doesn't matter too much, but has to be atleast a pentiumII on an intel motherboard, Athlon on a newer mobo, or intel/SiS for a p4.

I'd suggest staying away from NForce because of possible linux incompatibility... onboard graphics and sound of course.

160GB dirves are good sized and are in the sweet spot as far as price, however using 200GB+ drives would mean less drives total... hopefully equating to lower chance of failure, as well as lower power usage and heat... unfortunately other drive manufacturers are charging $0.83 per Gb while samsung charges $0.68, making the samsung damn attractive when you have 1000+GB of capacity.

The 3WARE Escalade 7506-8 is a mere $410.. allowing for a total capacity of 1,120GB in RAID 5 with 8 drives.

I would choose a case like the antec 3700AMB that has a removable HDD mount(capable of 5 drives alone) with a big 120mm fan sitting right infront of it... the case allows for 11 drives total which should be fine.

Just these parts(8 drives, controller, mobo, CPU, and case) is going to put you at $1500 (assuming $60 each for CPU and mobo)

Then you'll need to think about the PSU(would an Antec True Power 480 work? or would a beefier PC power 500 watt be better?) and any extra networking equip like a gigabit NIC or dual 100baseT.

 

[CougTek]

The main problem I see for a cheap storage server is the lack of faster PCI slots in budget motherboards. It doesn't take a lot of modern hard drives working simultaneously to reach the bandwidth cap of the old 32bit PCI bus. So a 3Ware Escalade into such a slot is partially wasted.

Fortunately, in some recent chipsets, the LAN is separated from the PCI bus. Sure, a low traffic server probably won't be slowed down too often by its PCI bus, but still. I'm eager to see the old PCI standard be phased out of modern platforms.

So we start the config with 6 x Samsung SP1614C (660$)?

And add a 3Ware Escalade 8506-8 (530$)?

We're already close to the 1500$ mark.

 

[blakerwry]

1) That's not enough disks.. keep going to get to 1TB
2) why do you want S-ATA?


I dont think using regular PCI is that much of an issue considering that gigabit ethernet over cat6 is probably going to top out at about 40MB/sec... dual 100baseT at ~18MB/sec.

 

[CougTek]

2) why do you want S-ATA?

To avoid cable clutter and reduced air flow.

4x or 6x PATA cables (to have one drive per channel) is going to create quite a mess inside a normal enclosure.

Regarding the chassis, would this do?

 

[blakerwry]

looks like it'd work.. but maybe not as much airflow over the disks w/ their sideways orientation...

http://images10.newegg.com/productimage/11-186-009-03.JPG

 

[blakerwry]

http://images10.newegg.com/productimage/11-129-122-05.JPG

 

[blakerwry]

Is buying a cheap, off-the-rack server, like a Dell, stuffing a lot RAID controllers and disks into it, and installing your favourite *nix just too simple?

no

 

[blakerwry]

4x or 6x PATA cables (to have one drive per channel) is going to create quite a mess inside a normal enclosure.

you can always 1) fold the ribbon cables to reduce their obtrusiveness 2) use rounded cables instead 3) lay the cables flat against the side of the case and tape them in place.




regarding the Dell, I dont think it would be bad to suggest using a Dell or other manuf. bare bones server as a base... but I would want to be able to also spec a similar config using standard components.

 

[Bozo]

Serious thought should be given to SATA. In 3-4 years you may be able to upgrade this server for more capacity. Are large PATA drives still going to be available?

Bozo :mrgrn:

 

[Fushigi]

The main problem I see for a cheap storage server is the lack of faster PCI slots in budget motherboards. It doesn't take a lot of modern hard drives working simultaneously to reach the bandwidth cap of the old 32bit PCI bus. So a 3Ware Escalade into such a slot is partially wasted.

I think this is a valid point and needs some further discussion. Are talking a storage server that can dish out lots of sustained video streams, something serving AutoCAD files, something that needs a lot of capacity but not much bandwidth, or what? Really, what you build will depend on it's intended usage.

For my usage, I'd be interested in:
- Capability to grow to several TB of space
- Win2K or Linux
- Single or dual 100MB Etherenet is sufficient
- Remote management
- RAID5
- Connect an external optical jukebox (or jukebox chain)?
- Maybe A/V in/out and possibly TiVO-type support. (getting OT so these are probably best left to another box that just uses the storage server)
- Perhaps some backup ability. High-capacity 5400RPM removeable drives would be good. So dedicate at least 1 case slot for a drive tray or have USB2/FW support.

Obviously my desired use is a media server for the house. Using hard disk space, optical jukebox(es), etc. I want something I can dump all of my media into. That means cataloging and storing my DVD & CD collection or converting them to be stored on hard drives, capturing my non-digital media (VHS/SVHS, LD, LP, cassette), and supporting at least 2 or 3 simultaneous output streams over Ethernet. Playback of DVDs must include DVD menus & other options like all languages & subtitle tracks.

If I leave DVDs & CDs in a jukebox, I'll still need a couple of TB of disk for everything else.

Regarding the Dell barebones server, it'd be a decent starting point. Just make sure the PSU can handle the additional drives.

OK, as I ramble on I'm beginning to think a simple NAS device is a better idea than an actual storage 'server'. What about it? Maybe build a budget NAS instead? Then, in my case anyway, I'd just add a 2U or microATX machine to my home theater and access everything from the NAS.

 

[Howell]

We need to decide on who the audience is.

IMO rounded cables are inappropriate for production equipment, since they are not supported by the spec.

On the other hand, they would be fine for enthusiast use.

 

[Howell]

Maybe build a budget NAS instead? Then, in my case anyway, I'd just add a 2U or microATX machine to my home theater and access everything from the NAS.

I think a you should not try to put all that equipment into your HTPC. The more equipment the more cooling the more cooling the more fans. The more fans the higher the ambient temperature and noise of the listening area. Not an enjoyable HT experience in my opinion.

 

[Fushigi]

Maybe build a budget NAS instead? Then, in my case anyway, I'd just add a 2U or microATX machine to my home theater and access everything from the NAS.

I think a you should not try to put all that equipment into your HTPC. The more equipment the more cooling the more cooling the more fans. The more fans the higher the ambient temperature and noise of the listening area. Not an enjoyable HT experience in my opinion.

The storage server/NAS goes in the basement on it's own UPS. Day-to-day management, if any, goes through TSE/VNC/whatever. Media loading would be copied over the network from my main PC in the den or maybe through the HTPC. It can be as loud as it needs to be; it's competing with the freezer, dehumidifier, and sump pump.

The HTPC would be: decent but not cutting-edge CPU, AIW-type graphics card or just a card with A/V out (audio out being optical or coaxial digital for carrying 5.1 to the amp), 1GB RAM or so, DVDRW, Ethernet, and a single quiet smallish HD.

 

[Buck]

We can always get:

Addtronics W8500 ATX Case
5.25” bays: 5 exposed
3.5” bays: 1 exposed 5 hidden

Room for several 80mm and 92mm fans.
$259.00 from Addtronics.

I would recommend redundant power supplies, RAID 5, and SATA if possible, if not for airflow, then at least for future flexibility. Although I prefer Samsung drives, for capacity reasons, and enclosure real estate, I would go with 6 Seagate ST3200822AS drives; 5 in RAID 5 and one spare. Then we could use a simple smaller capacity SATA, like the Samsung SP1213C or SP0812C for the boot/system/swap drive. If we expect the server to run applications that require many hits to the swap file, then a separate drive for the swap file would also be on me list. Let’s see, the W8500 case has 10 bays for us to use, 6 or 7 are taken with our hard drives, that leaves one bay for a DVD writer which is nice for making data portable, and then a tape backup system for the appropriate data redundancy. Oh yeah, and a 1.44 floppy drive for making our NT repair disks if that is the OS we go with.

One gigabyte of memory is nice which we can plug into a Tyan S2707G2N-533 motherboard. This will support PCI-X and actually has 3 independent PCI channels. Plus the board comes with onboard video and 2-Gigabit Ethernet ports, and will support 4GB of DDR memory. The board only takes one CPU with either 533 or 400 MHz FSB. This particular model number has no onboard SCSI, which is fine for us.

After this, add the 3ware SATA controller and plug in a P4 cpu, and we’re good to go.

 

[Bozo]

Actually there is another item to keep in mind if you plan to upgrade the box in a couple of years. The ATX standerd is going away, to be relaced by the BTX standerd. If I read the description correctly, you won't be able to reuses your case. So it looks like anything built now is going to be a throw-a-way in a couple of years.

Bozo :mrgrn:

 

[Buck]

<covers ears and yells> Don't listen to Bozo!</>

 

[Jan Kivar]

The main problem I see for a cheap storage server is the lack of faster PCI slots in budget motherboards. It doesn't take a lot of modern hard drives working simultaneously to reach the bandwidth cap of the old 32bit PCI bus. So a 3Ware Escalade into such a slot is partially wasted.

I think this is a valid point and needs some further discussion. Are talking a storage server that can dish out lots of sustained video streams, something serving AutoCAD files, something that needs a lot of capacity but not much bandwidth, or what? Really, what you build will depend on it's intended usage.

Using 100baseT You'll limit the max. transfer rate to 10 MB/s (to the clients, that is). So 32 bit PCI is not that much of a limitation to the 3Ware, as You can't get the data through the Ethernet port any faster.

But with Gbit Ethernet You will get into trouble fast. Not because of the added speed, but with the added packet count. I don't know if CSA (which is in some PIV boards) would ease anything for the PCI.

Cheers,

Jan

 

[CityK]

Regarding the chassis, would this do?

That looks like another variation from Chemming. I like the idea of the rotated HDD cage, but I don't like the air flow that one would have, nor do I like the fact that the cage leaves no room at the back (i.e. its positioned against the cases right panel, thereby taking away the option of routing and tucking the cables neatly along the right side of the case, where they would be out of the way and permit maximize air flow through the case).

 

[Mercutio]

CSA is its own bus to RAM, IIRC. Using GBoC/CSA on the i875 would mean extra PCI bandwidth for disk controllers.

I'd still like to suggest something in a rackmount case, simply because of cable length limitations with IDE and with tower cases. I don't have any one in particular in mind, but it's something to consider.

 

[timwhit]

I'd still like to suggest something in a rackmount case, simply because of cable length limitations with IDE and with tower cases. I don't have any one in particular in mind, but it's something to consider.

Is there any point in using a rackmount case if you don't have a rack?

Isn't all this information going to eventually make it into an article?

For the article how about suggestions on a regular tower case and a rackmount case, depending on the users needs.

I have purchased a couple rackmount cases from here, and have been very satisfied. This case can accomodate up to 24 hot swappable drives either IDE, SATA, or SCA; plus, it all fits in 4U. I don't know how much it costs, but that would be one sweet storage server.

 

[CougTek]

The rackmount enclosures I've seen at NewEgg (the SuperMicro one's) cost 220$-250$. For a 1500$ system, or even a 2000$ one, it is dangerously expensive.

So despite the PCI bottleneck for the Escalade, an i-875P-based motherboard with gagabit ethernet seems the way to go for a budget server since at least it leaves all the PCI bus for the HBA without stealing some bandwidth for the NIC.

Buck, nice motherboard, but Pricewatch tells me 325$ for that motherboard. Ouch! Since it includes graphics, then it makes it more acceptable, but we'll have to do the math and see if it doesn't blow up our budget. No bandwidth problem with it though. Let's say we use it, along with the Escalade 8500-8 and 6 Samsung drives, plus a 250$ enclosure (including a high quality PSU), a mere 512MB of RAM and a 2.6GHz Pentium 4 (similar price as the slower 2.26GHz model) :

• Pentium 4 2.6Ghz 533MHz FSB retail........................................160$
  Tyan S2707G2N-533 (with 2 Gbe ports)...................................325$
  512MB PC2700 ECC reg...........................................................100$
  3Ware Escalade 8500-8...........................................................530$
  Samsung SP1614C 160GB (x6).................................................660$
  Super-mega-I'm-drooling-everywhere ATX rackmount tower...250$

Sub-total : 2025U$

Prices are approximate (Pricewath showed me the RAM was available for as low as ~95$).

No tape backup, no optical drive (practicle), no operating system, no input device (how can we have a server without a Logitech MX700 mouse???), no monitor (only useful if no other systems are near). I guess we can omit the monitor and the input devices, if we asume the system will be the kind that accumulates dust in a corner where no one goes and everything is configured via network, but this isn't applicable to all servers.

 

[Buck]

Shouldn't this be a Terabyte storage server? Six Samsung SP1614C gets us 960GB of gross storage. Make a RAID 5 array and you just dropped to 800GB. Although not at the top of my list for drives, the WD2000JD would make more sense from a space-usage point of view. Plus, wouldn't we want a spare drive for our array if this is going to be a true storage server? Thus we would be hit with seven WD2000JD drives for the array alone. That is roughly a $1267.00 punch to our fragile budget. Plus we need a boot/system/swap drive. We can switch to WD2500JDs, but that would only drop the storage part of our array down to $1210.00 (according to pricewatch).

 

[Buck]

Oh yeah, other peripherals aren't necessary, as this would be a remotely operated machine if used as a true server, in my opinion. Anything handy around the shop can be used during installation.

 

[CougTek]

Then scratch the 660$ of Samsung drives and replace it by 1210$ of Western Digital WD2500JD. Add another 100$ for a boot drive (or as low as ~60$ for a PATA WD400JB). Sub-total now is roughly 2675$. Still no OS. I do believe at least an optical drive should be included (a 20$ CD-ROM would be fine).

I think so far we are aiming for a different market segment then Merc originally proposed (1500$-2000$). Although I don't think it's possible to have a TeraByte server with a non-free OS for 2000$.

 

[Buck]

Then scratch the 660$ of Samsung drives and replace it by 1210$ of Western Digital WD2500JD. Add another 100$ for a boot drive (or as low as ~60$ for a PATA WD400JB). Sub-total now is roughly 2675$. Still no OS. I do believe at least an optical drive should be included (a 20$ CD-ROM would be fine).

I think so far we are aiming for a different market segment then Merc originally proposed (1500$-2000$). Although I don't think it's possible to have a TeraByte server with a non-free OS for 2000$.

I agree.

 

[CougTek]

I think so far we are aiming for a different market segment then Merc originally proposed (1500$-2000$).

Ah Tabarnack! I hate it when I make such mistakes. I meant "than". Sacrament.

 

[Buck]

Pentium 4 2.6Ghz 533MHz FSB retail........................................160$
Tyan S2707G2N-533 (with 2 Gbe ports)...............................325$
512MB PC2700 ECC reg...........................................................100$
3Ware Escalade 8500-8...........................................................530$
Samsung SP0411N 40GB (boot/system/swap............................59$
Samsung SP1614C 160GB (x7).................................................770$
TEAC DVDRW+-........................................................................145$
SuperMicro SC742T-550 (CSE-742T-550)..........................350$

Roughly: US$2,500.00

Although this is all moot, until we decide what this server is for. Firingsquad did give their machine dual roles, namely storage server and workstation. In addition, they required a minimum 1TB of storage, with a quick upgrade path to another TB of data. How will a RAID5 scale in this type of situation?

We should have a poll on the type of server/system we want to collaborate on before we obviously write an article (as others have already noted). What are the strict roles we expect? Is this going to be a mulitpurpose machine, or not?

Storage Server:
Office files
Database files
A/V files
CAD files

Would this storage be used to simply backup data, or will the data be modified on the server?

There are many more points to consider, but I'm tired of thinking.

 

[Mercutio]

re: rackmounts without a rack - Think of it as a really big desktop case; there's plenty of places to put one without bolting it in a cabinet, particularly since the large ones are basically cube-shaped.

For rackmount cases, the one I've used most recently is the Antec 4U22. It only has 7 3.5" drive bays though, but comes with a pretty decent Antec PSU. It's like $225 or so.

Horror! You say?
It also has 6 5.25" bays and various companies make drive cages like this (uh, those are SATA units, 4 drives in 3 5.25" bays, but rest assured that there are PATA models and models that fit 3 drives in 2 bays out in the world. You just don't want to pay 3ware for 'em). Every once in awhile I see 'em on compgeeks.com for $50. If the drives are low heat, and the Samsung drives are, they work just fine.

I have a PC running 8 hard drives on a 400W Enermax PSU. 400W probably shouldn't be pushed past that, but hey, it's enough for our needs. We need more in that one case, we can add a bigger or second PSU, too.

3ware cards are pretty easy to find on ebay, still. That's where I got both of mine. If I was doing this again for myself, $150 BIN for a 7506-8 would be a dream come true.

Absent the 3ware card, I don't see a need for high-end Tyan Serverworks boards. I'd probably hunt up an Asus or Tyan or SuperMicro workstation board with a couple 64-bit PCI slots, one for GBoC and one for the eventual 3ware card, or maybe go with a vanilla i875 such as the 8i875 from Gigabyte, just to get gigabit and CSA.

 

[Howell]

Could someone remind me what CSA is again?.

 

[Mercutio]

Communications Streaming Architecture. Another excuse to not put better PCI slots on a desktop board, just like AGP.

Because gigabit ethernet can saturate 32bit 33Mhz PCI so easily, Intel gave it its own bus and path to RAM. There's no "CSA slot", it's just a bus-mastered transport between the memory controller and an onboard gigabit NIC.

 

[Mercutio]

Serious thought should be given to SATA. In 3-4 years you may be able to upgrade this server for more capacity. Are large PATA drives still going to be available?

In 3 - 4 years I'd fully expect single drives in the 1TB range to be mainstream products, just like 120GB drives are now. In early 1998 I put together a fileserver with a then-mind-bending 36GB of disk space. By 2001 everyone and their brother had a PC with 40GB.

I also think that SATA isn't a terribly mature technology. In 3-4 years we'll be looking at 10TB storage servers and SATA as the connection interface, but for right now, it's hard to justify the benefit of SATA when enthusiasts are still putting a premium on a drive just for being on a different interface and PSUs don't include the #$%ing dinky-ass SATA power connector (so we'd be paying a premium for SATA, and paying again for the stupid converter thing that most SATA drives need for power).

 

[Mercutio]

Serious thought should be given to SATA. In 3-4 years you may be able to upgrade this server for more capacity. Are large PATA drives still going to be available?

In 3 - 4 years I'd fully expect single drives in the 1TB range to be mainstream products, just like 120GB drives are now. In early 1998 I put together a fileserver with a then-mind-bending 36GB of disk space. By 2001 everyone and their brother had a PC with 40GB.

I also think that SATA isn't a terribly mature technology. In 3-4 years we'll be looking at 10TB storage servers and SATA as the connection interface, but for right now, it's hard to justify the benefit of SATA when enthusiasts are still putting a premium on a drive just for being on a different interface and PSUs don't include the #$%ing dinky-ass SATA power connector (so we'd be paying a premium for SATA, and paying again for the stupid converter thing that most SATA drives need for power). Right now we also hear about weird issues with SATA BIOSes and incompatibilities with other hardware - the same issues that were common to early IDE-RAID cards.

I don't think SATA is ready yet.

 

[CougTek]

3ware cards are pretty easy to find on ebay, still. That's where I got both of mine. If I was doing this again for myself, $150 BIN for a 7506-8 would be a dream come true.

But we can't tell our readers to buy their stuff on eBay for their server or many of them will write something like "They used second-hand parts, pardon me while I point and laugh". Some advices that are sound for individual recommendations are simply inapropriate for serious articles. Besides, while eBay USA might have a hefty supply of 3Ware RAID controller, eBay rest-of-the-world doesn't. Given the international nature of our membership, eBay can't be considered a reliable source for purchasing non-mainstream components, especially for server use.

Absent the 3ware card, I don't see a need for high-end Tyan Serverworks boards. I'd probably hunt up an Asus or Tyan or SuperMicro workstation board with a couple 64-bit PCI slots, one for GBoC and one for the eventual 3ware card, or maybe go with a vanilla i875 such as the 8i875 from Gigabyte, just to get gigabit and CSA.

Seen any i875P-based motherboard with 64bit/66MHz or better PCI slots? Significantly more affordable than the Tyan linked above? A vanilla i875P mainboard with a cheap graphic card such as an ATI Radeon 7000 would maybe save around 100$-125$ compared to the Tyan. Plus a few other ten of $$ considering the RAM won't need to be ECC registered type.

 

[Mercutio]

I was thinking of something along the lines of an A7M266D or a Supermicro P3TDLE.