Large RAID arrays

[Bozo]

Generally when you set ou a system with RAID 5, you loose the capacity of one hard drive.
If you have 12-750GB hard drives for an 8TB array, does that rule still apply? Or do you loose more than the capacity of one drive?

I would imagine it could take a day or more to initialize an 8TB RAID array???

Bozo :joker:

 

[P5-133XL]

With raid-5 you will lose exactly one drive to overhead regardless of the number of drives or the size of the array. With a 3 drive setup, you will get two drives of useful space and with an 18 drive setup, you will get 17 drives of useful space. It is not a generality, but rather a characteristic of raid-5.

 

[ddrueding]

I'll second Mark. My array started with 4 750GB drives (2.25TB) and is now 6 750GB drives (3.75TB). The rebuild that does the online storage expansion took 3+ days to add the 6th drive.

 

[Bozo]

Thanks!
Would anyone know of a reasonally priced enclosure for 12+ SATA drives?
Promise has the VTrak http://www.newegg.com/Product/Product.aspx?Item=N82E16811777006. If I am reading the spec correctly, this is a NAS device that is connected to a server via CAT5 gigabit network cards???
I have an e-mail into Dell for their NAS.
Other suggestions are welcome.

Thanks,
Bozo :joker:

 

[ddrueding]

Are you after a full NAS solution? Or something to connect directly to an existing computer? There are quite a few cases that support that many drives, if you want local access.

 

[mubs]

Never mind.

 

[ddrueding]

Thanks!
Would anyone know of a reasonally priced enclosure for 12+ SATA drives?
Promise has the VTrak http://www.newegg.com/Product/Product.aspx?Item=N82E16811777006. If I am reading the spec correctly, this is a NAS device that is connected to a server via CAT5 gigabit network cards???
I have an e-mail into Dell for their NAS.
Other suggestions are welcome.

Thanks,
Bozo :joker:

It looks to me like it has a pair of fiber channel ports for data access, and an RJ45 for management.

 

[Bozo]

Storage systems this large are new to me. So are NASs and SANs.
We would like to attach this to an existing XP computer. But everything I have read indicates we would need Server 2003.
If I understand what I read, some of these connect via a gigabyte NIC. Some have a RAID card that is installed in a computer and connects to the enclosure with iSCSI cables??? Even if the enclosure is filled with SATA drives???
Actually, I need to grill the person that wants this to be sure he knows want he wants and what the intended purpose is.
Google has definately become my friend

Bozo :joker:

 

[Stereodude]

I would imagine it could take a day or more to initialize an 8TB RAID array???

Bozo :joker:

It look about a day to initialize my 2TB RAID-5 array (5x500GB), so I'd expect a day minimum.

 

[ddrueding]

The most important thing for OS support is that it can handle GPT partitions. IIRC, this includes XP x64, 2003 Server, and Vista. If it is an external array, it could, in theory, use any type of connector imaginable regardless of the disk interface type; there is a RAID controller between the drives and the rear interface. iSCSI is a way to send SCSI packets over Ethernet, so "iSCSI cables" are CAT-5

 

[Pradeep]

I would assign at least one hot spare if you are running 12+ drives in a single array. Especially if the controller is farking slow to rebuild.

 

[ddrueding]

Especially if the controller is farking slow to rebuild.

Especially especially if you won't be able to get to the array to replace the drive immediately.

Especially especially especially if you won't have active fault monitoring on the array.

 

[Bozo]

I have switched gears some what. I still need 8TB but now the engineer ask if it can be in a case complete with an OS. Basically a 8TB server.

I am looking at theis case: http://www.newegg.com/Product/Product.aspx?Item=N82E16811112061

I am wondering how would you supply power to all the hard drives? Is there a wire loom available?
The plan is to also install a Supermicro 5-into-3 hard drive enclosure in the front of the case. This would house two hard drives in RAID 1 controlled by the motherboard onboard RAID. This enclosure would also hold the hot spare for the 8TB array. They array would be controled with a 3Ware RAID controller.
Any and all sugestions would be most welcome.

Bozo :joker:

 

[Stereodude]

I am looking at theis case: http://www.newegg.com/Product/Product.aspx?Item=N82E16811112061

Any and all sugestions would be most welcome.

Bozo :joker:

Find another case? I have an older version of this one with my server in it at home. There is a ton of space in the bottom of it for HDs, but there's one big problem with the case. It holds the motherboard upside down. That might not be a problem with SATA, but it's a big problem with PATA when the RAID card is so far from the drives in the bottom and the cables come off the "top" side of the card. I ended up having to install most of the HDs upside down the machine to keep from twisting all the pata cables.

 

[Bozo]

I don't think it will be a problem with SATA, but I will keep it in mind.

My concern now is the power supply. Will an 850W power supply fit insde and can I get enough connectors for all the hard drives?

Bozo :joker:

 

[Stereodude]

My concern now is the power supply. Will an 850W power supply fit insde and can I get enough connectors for all the hard drives?

Bozo :joker:

Just grab a handful of splitters.

 

[ddrueding]

I'm going to try and talk you out of that case as well. I have one sitting by my front door as we speak. It used to be my main case (I think it is still in my sig). Cooling for the drives is awful; the 120mm intake fan is completely choked off and the 2 optional 80mm fans in the rear need to be in jet engine mode to keep the drives under 50C.

What you want is something with a lot of 5.25" bays along the front and some 5-in-3 hotswap enclosures. I typically use Coolermaster Stackers, but for my latest I used a (smaller) Antec Nine Hundred.

Don't worry about the SATA power connectors, each of the hotswap enclosures only needs a few molex plugs. I'm using a Seasonic M12 750W and it handles 11 drives (5 of them 10k rpm) just fine.

 

[ddrueding]

What I'm looking for is a hot-swap enclosure that supports multi-lane plugs directly (multi-lane is a single cable that carries 4 SATA channels) to clean up the cabling. My 3Ware card currently has multi-lane connectors on it, and the break-out cables make a mess.

 

[Bozo]

I believe 3Ware has hot swappable enclosures. http://www.3ware.com/products/ata.asp

I'll reconsider the case. Thanks!

Bozo :joker:

 

[ddrueding]

I believe 3Ware has hot swappable enclosures. http://www.3ware.com/products/ata.asp

True. But they still only use regular SATA plugs on the back. If I'm going to be stuck with regular SATA plugs, I'll use the Supermicro enclosures (higher drive density and swappable fans for reasonably quiet operation).

 

[Bozo]

True. But they still only use regular SATA plugs on the back.

That sucks.

I found this case: http://www.newegg.com/Product/Product.aspx?Item=N82E16811196011

With three of the Supermicro 5-into-3 enclosures, I should have more than enough space.

Anyone have any experiance with an AntecTPQ-850 power supply?

Thanks for everones help.

Bozo :joker:

 

[blakerwry]

I'd highly suggest a direct attached storage product like the Dell PowerVault MD 1000 with a matching Dell PERC5/E PCIe controller.

You can spec it out with up to 15 SATA or SAS drives (12 for raid 5 storate, 2 for raid 1 OS, 1 for global hot spare).

Dell handles the compatibility, the warranty, etc. You just install the RAID card and manage the drives as if they were local to PC.

You'll pay a bit of a premium for a Dell system, but if this is for a business I'd go this route vs a custom build. Spec'd out with 15 750GB SATA disks, the raid controller, and a 1500VA UPS they're about $9000. (If you contact a Dell sales rep via telephone you may be able to get 20% knocked off that price.)

 

[ddrueding]

That case looks fine. It also features an inverted motherboard tray, but depending on your motherboard that should be fine.

If this machine is going to be near humans, I recommend replacing the 92mm fans on the Supermicro enclosures with quieter ones. If they are low-rpm fans, you might have to move a jumper on the back of the enclosure to disable the "fan fail" alarm; this is not difficult. With the 3 92mm fans in front, I wouldn't even bother with an exhaust fan in back; it will be a positive pressure case anyway.

What are the other system specs? What RAID card will you be using.

 

[ddrueding]

I'd highly suggest a direct attached storage product like the Dell PowerVault MD 1000 with a matching Dell PERC5/E PCIe controller.

You can spec it out with up to 15 SATA or SAS drives (12 for raid 5 storate, 2 for raid 1 OS, 1 for global hot spare).

Dell handles the compatibility, the warranty, etc. You just install the RAID card and manage the drives as if they were local to PC.

You'll pay a bit of a premium for a Dell system, but if this is for a business I'd go this route vs a custom build. Spec'd out with 15 750GB SATA disks, the raid controller, and a 1500VA UPS they're about $9000. (If you contact a Dell sales rep via telephone you may be able to get 20% knocked off that price.)

That is a nice (and expensive!) solution Blake. Perhaps I did something wrong, but I configured an MD1000 with 750GB drives and the controller and it came to $15k.

 

[Bozo]

Blake; I already got a quote from Dell, $17,280.29. According to Dell, the RAID card will only work in a Dell server, which we don't have.
ddrueding; I have used the Supermicro enclosures before. I usually replace the fans with temperature controlled Antec units.
The parts I am looking at are:
Intel DG965WHMKR 'board
Intel C2D 6320
4GB Kingston 800MHz RAM
14 WD 750GB hard drives
2 WD 80GB hard drives. RAID 1 boot drives from the motherboard.
3Ware 9550SXU-12 RAID controller for the 8TB storage.
Antec TPQ-850 Power supply
3 Supermicro 5-in-3 enclosures

This box is for working with and storing videos. Don't ask, I'm just the hardware guy.

Bozo :joker:

 

[ddrueding]

Looks like you have things well under control. I noticed a significant RADI5 write speed improvement when I added the battery backup unit and set the card to "performance" mode. I would expect to see speeds around 130MB/s with that setup.

 

[blakerwry]

Ah, I see it was actually 12k - 3k = 9k for buying through Dell small business.

For reference, the enclosure is scsi (sas to be exact). It'll work with any external SAS controller. Dell's PERC5 controllers are PCIe, and should work (but may not be qualified with) any PCIe motherboard.


It sounds like you have a handle on things bozo... My reason for recommending the Dell is that it's a pre-tested box that has great warranty support (NBD). And as the saying goes ... nobody ever got fired for buying dell...

You doing any UPS battery backup or does this system require high availability?

 

[Bozo]

The system needs to ba available 24/7. It is the repository for quality videos of our process taken in real time.
The system will be on a UPS and I'm considering a battery on the RAID card.

Bozo :joker:

 

[blakerwry]

In that case, I'd recommend both.

-Blake

 

[GIANT]

There's so much patently bad advice and wrong information in this thread that I don't know where to begin.

 

[Bozo]

There's so much patently bad advice and wrong information in this thread that I don't know where to begin.

I'm all ears....

Bozo :joker:

 

[[Edit]]

I'm all ears....

I could do a Spock joke here, but...


I don't know your precise needs, expectations, or situation there at the "factory." But, from what I gathered by skimming through the talk in this thread (the last time I was here) is that it looks like a SAN -- or possibly NAS -- should have been planned and built starting a couple of years ago. And, in those 1, 2, or even 3 years, y'all should have been adding the proper type of storage as needed.

I don't want to get into particulars, but, much of the stuff mentioned above would be a poor investment or simply a disaster.

There are a lot of questions to be asked about your needs (or perceived needs).

The system needs to ba available 24/7. It is the repository for quality videos of our process taken in real time. The system will be on a UPS and I'm considering a battery on the RAID card.

A NAS device or a SAN (and its devices) needs to be left on 24/7. So, you need redundant power circuits (208-volt preferably, unless we are talking about 3-phase power) and redundant power supplies to everything possible. If there are any single powered devices, those need to go onto automatic failover power switches (which are powered by redundant power circuits).

The best solution for you could be either a SAN or NAS or even both!

A SAN fabric can be based on:

* InfiniBand (way fast and sophisticated, and likely way too expensive for you; IB could kill Fibre-Channel in the next 10 years)

* Fibre-Channel (copper and optical -- chances are you only need 1Gb F-C; 2Gb, 4Gb, and faster F-C is likely not needed and is quite expensive)

* iSCSI (block storage over Ethernet media using TCP/IP; UTP copper Ethernet media unless there's an actual need for fiber optics; any Ethernet switch or hub will work with iSCSI, but you really need to have quality Gb Ethernet switches if you want to get any reasonable ROI and performance)

* AoE (basically iSCSI without the TCP/IP overhead)





There are many many questions to ask. I guess I could start off with one for now:

The system needs to ba available 24/7. It is the repository for quality videos of our process taken in real time. The system will be on a UPS and I'm considering a battery on the RAID card.

So, I take it that nobody has a need to *edit* video on this storage device? Correct?

This is a compressed streaming video source, or maybe low-frame-rate video source (or both)?

 

[blakerwry]

If only one person in the organization needs to have direct access to the videos what is the advantage of using SAN technology?

The obvious downside is the specialized equipment, infrastructure, and complexity that would be necessary for such a setup. I should stress complexity, as most of the cost of such a project can come from the time troubleshooting or replacing equipment that doesn't work as initially intended.

I generally agree that networked storage can make a lot of sense for organizations. Especially if you're looking at a large repository. Once the capability is there, people will often find the application(s) to use it.

However, in my opinion, direct attached storage offers simplicity and familiarity along with the lost cost to provide what the op was asking for (8TB for one user)

 

[Bozo]

Thanks for everyones input.
I ended up using a Supermicro 3U case CSE-932T and a Supermicro PDSBA+ motherboard. Everything is up and running.
The operating system and programs are on a 3Ware 8006-2LP in RAID 1.
The storage area is on a 3Ware 9650SE-16ML.
I installed Server 2003 plus server packs. Now it wants to convert the storage array to GPT disk. What is that? Is this the only way to have all 8TB as one large partition?

Thanks,
Bozo :joker:

 

[ddrueding]

Yup. GPT is GUID partition table (IIRC). It works well with larger arrays. Mine uses it and it works really well. How are you liking the 9650SE-16ML? I love mine, it is a beast.

 

[Bozo]

Yup. GPT is GUID partition table (IIRC). It works well with larger arrays. Mine uses it and it works really well. How are you liking the 9650SE-16ML? I love mine, it is a beast.

The 9650SE-16ML is great. It initialized the array ( 11-750GB hard drives ) in a shade under 10 hours including the hot spare (drive number 12).
I might try to run ATTO test on it if I get time.

Bozo :joker:

 

[ddrueding]

My array is about the same size. It maxed out at around 350MB/sec.

 

[Bozo]

I just finished running ATTO
Writes = 425+ MB/sec
Reads = 745+ MB/sec

In the 3Ware management program, I set up the I/O for faster performance.

Bozo :joker:

 

[ddrueding]

Wow. I have mine set to max performance as well. Perhaps I should be looking at a firmware update?

 

[Bozo]

I have the following on the 3Ware 9650SE
BIOS: BE9X 3.08.00.002
Firm: FE9X 3.08.02.007
Driver: 3.02.90

I also have "enable advanced performance" checked in Device Manager under Disk Drives, Policies tab.

Bozo :joker: