HP SAS Expander

[MaxBurn]

Has anyone here played with one of these? I wasn't even aware that they existed until I stumbled across this thread. The possibilities are really something to think about.

It only needs the PCIE slot for power so if you could find a backplane cheap enough that would work, guys in the thread are doing things with atom boards because they are so low dollar now too.

http://www.hardforum.com/showthread.php?t=1484614

The HP SAS Expander: It has a lot of ports.

When the Norco RPC-4020 server case first arrived in the summer of 2008, it was a pivotal device that brought the idea of "a lot of storage for little cost" within reach for many people. The HP SAS Expander is another game changing device- combine it with a low-cost RAID or HBA adapter and you've got 32 ports of harddisk connectivity for only a few hundred dollars.

Overview: The HP SAS Expander is based on the next-gen PMC Sierra PM8005 SAS-2 chip which features 36 x 6Gbps ports and 6G/3G multiplexing, SAS 2.0 zoning, self-configuration, table-to-table routing, and an integrated MIPS processor for SES and enclosure management support. Full specs here: http://www.pmc-sierra.com/products/details/pm8005/

FAQ

Q: What's so special about this card?
A: The PM8005 chip has proven to be more compatible in more HBA/Drive configurations than previous generation expanders like the LSI-based Chenbro CK series. While HP intended this card be used in their HP Proliant servers, the card is driverless and essentially "universal" and can be used with non-HP branded RAID and non-RAID HBA cards. What's unique is the ability to connect 32 drives at a relatively low cost, when combined with a low cost 4 or 8 port RAID or non-RAID HBA.

Q: Where do I purchase this card and why do I want the green card and NOT the yellow card?
A: There are multiple hardware versions of this card, available from Amazon, ebay, etc. The yellow PCB version is the oldest hardware revision, has a smaller heatsink, and CANNOT be flash upgraded from its v.0.20. The green PCB cards are the newer hardware revision. I've bought green PCB cards from ebay and they came with firmware v1.00. I've bought a card from Amazon and it came with v1.52. Shipping firmware will depend how old your source's inventory is. To work correctly at 3Gbps with newer SATA2 drives you need firmware v1.52 firmware or higher. Click thumbnail to see a side-by-side of both cards. DO NOT buy the yellow PCB card, it is useless and will not work.

Q: How do I install the HP SAS Expander and what are the power requirements?
A: The card needs an x4 PCIe slot on a motherboard and draws 11 watts of power. The card doesn't require software drivers, it is invisible to the operating system and motherboard. A common dilemma is people wishing to use this card in an empty chassis like a Norco RPC-4220 as a "drive cage". Some people have resorted to using an old motherboard, or buying a low-cost Intel Atom based motherboard to serve as an ON/OFF switch for the chassis and to power the card - this solution is still far cheaper than buying a prefab expander chassis. Unfortunately this card has no external 4-pin Molex power connector like the Chenbro expanders, but it's a fair tradeoff given Chenbro expanders aren't really available anymore, and the HP expander uses a newer SAS2/SATA3 expander chip (support 6Gbps operation), whereas the Chenbro expanders are SAS/SATA2 (3Gbps).

Q: How do I connect harddisks to the HP SAS Expander?
A: There are 9 ports total which can connect four harddisks each. The external SFF-8088 port is Port #1. The internal SFF-8087 are Port #2 through Port #9. I recommend populating drives beginning with internal Port #2. To connect the expander to your RAID/HBA, use Port #1 with an SFF-8088 cable *if* your RAID/HBA has an external connector, otherwise use Port #9 with an SFF-8087 cable.

NOTE: Even though the printed circuit board says "to controller" on Ports #8 and #9, this is trivial. The SAS expander acts much like an ethernet switch- you can use any port for input or output, it makes no difference, there's nothing unique about any one port from the others. HP suggests using the last two ports only because its the end of the range.

Q: How many cables do I need to connect the HP SAS expander to my RAID or HBA card?
A: You only need 1, at 2.4Gbps it's enough bandwidth for all your drives, however 2 cables provide a 4.8Gbps interlink. In benchmark tests I didn't see any performance difference between using 1 and 2 interlink cables. That may change however when the expander is combined with SAS2 RAID cards and/or SATA3 harddisks. When a SAS2 RAID or non-RAID HBA drives this expander, the interlink speeds are double (4.8Gbps for single cable, 9.6Gbps for two cables).

Q: How many harddisks can you attach to the HP SAS expander?
A: Up to 32. I've stress various configurations including a RAID6 array of 32 x 1Tb drives and had no issues.

Q: Which RAID and non-RAID HBA cards have been tested with the HP SAS Expander?
A: In general if your card is SAS compliant and vendor specs state that it supports expanders, it *should* work, but there are exceptions. The following cards were tested by myself unless stated otherwise.

RAID HBA's
Areca 1680 series: YES (must set SES2 to Disabled in BIOS)
3ware 9690SA Series (verified by SeanG): YES
Adaptec 5085 (2 x SFF-8088 connectors): YES
Adaptec 5805 (2 x SFF-8087 connectors): NO (hopefully in future Adaptec firmware update)
HP P212, P410, P410i, P411, P411i, P712: YES
Adaptec 2045 (1 x SFF-8087): Untested but *should* work
Adaptec 3085 (2 x SFF-8088): Untested but *should* work
LSI 9280-8e (2 x SFF-8088): Currently being tested

Non-RAID HBA's
Areca ARC-1300-4X: YES
Supermicro AOC-SASLP-MV8: YES
Adaptec 1045 (1 x SFF-8088): Untested but *should* work
Adaptec 1405 (1 x SFF-8087): Untested but *should* work

Q: Why do I need firmware 1.52 or higher on the HP SAS Expander?
A: The most notable change in firmware 1.52 brought upgraded SAS/SATA speeds from 3Gbps/1.5Gbps to 6Gbps/3Gbps respectively. Most newer harddisks ship defaulted to 3Gbps, and for compatibility and performance reasons firmware 1.52 or higher is recommended.

Q: How do I update the firmware on the HP SAS Expander?
A: A flash update requires a newer HP branded RAID card- P212, P410, P411, or P712. If you have a green PCB card with lower than firmware v1.52, you can post in this thread and someone may be able to flash it for you.

 

[Handruin]

That's pretty neat. So basically when you have one of these, you have a coouple options.

One option is to power the card by a basic motherboard with PCIe x4 in some rack/case, you then connect the HP SAS expander to a RAID controller via port #1 in another computer?

Another option is to have it all in one case and connect internally to port 9.

They claim that a single 2.4Gb cable provides enough bandwidth...but for 32 drives? That gives you a little over 300MB/s bandwidth which doesn't seem like enough.

Looks like Newegg has the recommended Areca ARC-1300 for about $160.

What kind of cable makes the connection? In the case of a non-raid ARC-1300, it just becomes a large JBOD, right? I'm guessing software RAID is the option like Merc prefers.

In the case of a RAID card, won't the bottleneck be the one or two cables that connect the expander to the raid card? It is certainly a nice way to get a large volume of ports. Any idea how much the expander card goes for?

 

[MaxBurn]

They are standard 8087 cables. I think all the cabling concern is directed at the use senario, what are you going to do with it and those 8087 cables have four lanes each inside.

Link cable:
http://www.wiredzone.com/itemdesc.asp/ic/10019503/model/CBL-SFF8087-06M
Breakout:
http://www.wiredzone.com/itemdesc.asp?ic=32019992


In the original thread someone pointed out one of these that might be able to power the card.
http://www.orbitmicro.com/global/pe-2sd1-r10-p-9375.html
Basically I think any back plane with a PCIE slot could be used.

 

[Mercutio]

The Expander card is $250, minimum, which is still a bargain. You've got to have another SAS controller, figure that'll be $150. You need a chassis ($250), power supply (probably at least $100) and enough drives to make something like this actually interesting.

Figure $2500 to put something like this together.

Still, I'm probably the only person here with more than 60 1TB+ drives and if I had known about that sort of configuration, I probably would have put them in two chassis rather than four.

 

[Handruin]

So it sounds like the progression would be as follows (please correct me if I'm wrong):

4 Drives > (4 to 1 on an 8087 cable) >
expander port (ports 2 through 9) > (1-2 link cable(s)) > raid controller

 

[MaxBurn]

On the pricing the guys on the forum there are getting the port expander through some gray channels for $100 or thereabouts. Similar to how the Perc6 costs a lot more than you can get them for. Actually on the threads very last page there is a guy offering them from some vendor, sounds like he will even update the firmware for you.

Supermicro AOC-SASLP-MV8 $103.99
http://www.wiredzone.com/itemdesc.asp?ic=10018973

What case are you talking about Merc? I may be interested as my current case tops out at 8 3.5" plus whatever you can do with 6 5.25".

Handruin: You have it but were thinking backwards from where I was initially thinking starting at the HBA.

What I am not clear on is you might be able to do TWO breakout cards off of one SASLP-MV8 (or other HBA) for a total of 64 drives. Granted thruput wouldn't be great as each stack of drives goes to one 8087 cable. Don't see a reason why you couldn't have two HBA's though.

 

[Mercutio]

What case are you talking about Merc? I may be interested as my current case tops out at 8 3.5" plus whatever you can do with 6 5.25".

The case they're talking about in your link has 20 3.5" drives and a SATA/SAS backplane. They're about $250 pretty much everyplace.

I have four of them already.