3ware 7506-8: The tale of a quirky RAID-5 implementation

[Stereodude]

So back in July of 2003 I bought a 3ware 7506-8 for a storage server. I put 6 200GB 7200RPM drives on it in RAID-5 and 2 60GB 5400RPM drives on it in RAID-1. Over time the RAID-1 array turned into 2 320GB 7200 RPM drives, the RAID-5 array turned into 5 500GB 7200RPM drives, and the card moved through a few different motherboard / processor setups (a few AMD, a few Intel). At times and with certain applications the RAID arrays seemed very slow. Other times they seemed to perform reasonably well. The RAID-5 array was especially slow in some applications like HDTVtoMPEG2, Perfectdisk, and ATTO to the point of being unusable. 3ware had benchmarks that showed the card could push reads of 180+MB/sec and writes of 70+MB/sec in IOmeter so I couldn't understand the poor performance I was seeing.

Last month I finally took the 3ware 7506-8 out of service and replaced it with a Dell Perc 5/i. I got curious though why the 3ware card delivered such lackluster performance in some apps, yet seemed ok in others. I went back and looked at all the reviews that were still online of the card, and went back and revisited 3ware's benchmarks. I talked to a friend who has the same card and similar experiences to mine. The point of discrepancy I could find was that I never used the card in a 64-bit PCI slot. It was always in a 32-bit slot. The reviews where a 64-bit slot were used got good results. The one where a 32-bit slot was used got bad RAID-5 results. My friend and I also got bad RAID-5 results with 32-bit slots.

So, I convinced myself that the 64-bit PCI slot was the key to my mystery, so I bought a motherboard off ebay with 2 64-bit PCI slots. It arrived today, after a clean install of XP SP2 later I discovered that the 64-bit PCI slot was not the key after all.

32-bit PCI slot:

64-bit PCI slot:

The reads are better, but they weren't really bad to start with. The writes are no better. I did a quick file copy test and calculated ~43MB/sec. Much better than the ATTO number show.

3ware had fairly impressive (especially for 2003) IOmeter results, so I decided to also test using IOmeter. Here's what I found using IOmeter to mimic the sort of testing ATTO does.

32-bit PCI slot:

64-bit PCI slot:

Oddly enough, the 64-bit PCI slot does allow for faster writes. This is somewhat surprising since a 32-bit PCI slot has ample bandwidth to accommodate ~70MB/sec.

Long story short, the best I can tell the card has a problem with applications that bypass the standard Windows file writing routines and try to access the drive directly. I don't fully understand why it behaves the way it does, but it does.

 

[Stereodude]

What? No one wants to discuss a 6 year old PATA RAID controller? :???:

You guys really let me down. :rotfl:

 

[Bozo]

I had the same problem with a different 3Ware card. After poking around on the 3Ware site, I found a registry hack and a firmware update.
Seemed to help some.
Also, my 8006 and 8506 I am using now really improved when I install the 'beta' drivers. Been using them for over a year with no problems.

 

[Stereodude]

I have the latest firmware for the card (has been installed on it for years) and have tried a few different versions of the drivers (including the most recent beta drivers) all with similar results.

Do you happen to remember the registry hack?

 

[Stereodude]

I think I found it. link

Is that what you remember?

 

[MaxBurn]

I wasn't ever impressed with the 7500-8 and something older from 6xxx series under RAID5 either but they did carry me flawlessly through several drive failures over its lifetime. Was used on several motherboards, some 64bit and some not and I didn't test or even pick up on the difference between the two slots. From what little I read RAID5 performance has ramped up quite a bit on newer model cards.

 

[Stereodude]

I wasn't ever impressed with the 7500-8 and something older from 6xxx series under RAID5 either but they did carry me flawlessly through several drive failures over its lifetime.

The 7500 and later the 7506 (7500 but with 66MHz PCI support) were the first PATA RAID cards to have a hardware XOR engine for (supposedly) fast RAID-5 writes. My 7506-8 saved my bacon a few times with drive failures as well. I likewise have to admit that I was never impressed with it though.

 

[Stereodude]

From what I was able to gather from my digging online. Prior to Win2k SP3 the SCSI driver in Windows didn't honor the FUA (Force Unit Access) command, or the various cache flushing commands. That was added with SP3 to Win2k and it seems the 3ware card doesn't have a good way to deal with them. It honors the commands but I'm not really entirely sure why or how it does so. As I understand it the underlying IDE/PATA drives use write back cache and have no concept of such SCSI commands (there are no IDE equivalents of them) so the controller flushes its cache, or refuses to read from it's cache, but the drives could still be doing it.

The only way to honor the commands that I can think of would be to have the 3ware card attempt compliance with a delay of a somewhat arbitrary length of time allowing the drives to empty their write cache on their own and become idle. To ensure that this happens the delay is probably far longer than is necessary for nearly all circumstances so when an app like ATTO (which uses the FUA command extensively to prevent write back caching from affecting the results) the card effectively is idle most of the time while these delays stack up.

Ultimately though, if I can get the performance I want in the application I'll be using I'd rather leave the 3ware drivers alone instead of potentially risking my data.

 

[Bozo]

I think I found it. link

Is that what you remember?

Yep, those look like the ones.

 

[Stereodude]

After thinking about this even more I bet I can calculate the length of the delay the 3ware card inserts after each FUA from ATTO.

 

[Stereodude]

By comparing the ATTO results with FUA enabled and FUA ignored and doing some math it would appear the 3ware card inserts a 15-16ms delay after it dumps it's own cache to disk every time it's hit with a FUA.

Here's what ATTO does with FUA ignored.

 

[Stereodude]

Yep, those look like the ones.

Did you ever have any problems with data corruption or the "Delayed Write Failed" problem described here?

Microsoft can't seem to make up their mind. In this KB page they say, "Support for the FUA bit is optional, and only some SCSI and Fiber Channel (FC) devices, typically individual drives (as opposed to RAID arrays), implement this functionality." But, apparently it can cause also problems when disabled.

 

[Bozo]

Did you ever have any problems with data corruption or the "Delayed Write Failed" problem described here?.

I can't say for sure. It's been a while.

 

[P5-133XL]

Did you ever have any problems with data corruption or the "Delayed Write Failed" problem described here?

Microsoft can't seem to make up their mind. In this KB page they say, "Support for the FUA bit is optional, and only some SCSI and Fiber Channel (FC) devices, typically individual drives (as opposed to RAID arrays), implement this functionality." But, apparently it can cause also problems when disabled.

Yes, there was a time that I had frequent problems with that. It kept breaking my raid set. I ended up abandoning my 3ware controller in favor of Microsoft's software raid.

 

[Stereodude]

Here's what ATTO does with FUA ignored

FWIW, ATTO showed no real improvements when SRB flush commands and/or SCSI flush commands were ignored.

I'm working on finishing up some longer IOmeter runs to see if the different settings for FUA and ignoring the flush commands have any impact on IOmeter performance.

 

[Stereodude]

So there are the 3 long IOmeter runs compared. Overall I'd say the tests don't show any change or improvement.

No tweaks (default):

FUA ignored:

FUA & flush cache commands ignored:

 

[Mercutio]

I can't believe you're doing this much work for a six year old RAID controller.

 

[Stereodude]

I can't believe you're doing this much work for a six year old RAID controller.

Just call me curious. :rofl:

After six years this the first real chance I've had to do any extensive testing on it. When it's holding your data your not likely to start experimenting putting the data and the array at risk. And, in the same vein, you can't run IOmeter on a disk that has partitions on it.

But you can sleep easy because all the testing is done. I'm comfortable with the results and what I found. Based on my testing, I'm about to put it back into service in a new roll since I think it will do the job adequately. It delivers the same 46-47MB/sec writing performance copying large (>1GB) files to the array in Windows explorer regardless of the 3ware registry tweaks. :thumbleft:

 

[Bozo]

I like poking around with old stuff. Right now I have an Adaptec 3000S SCSI controller with four 18GB Cheetahs attached. Now to find drivers. :grin:

 

[blakerwry]

How does the Perc5 compare with the 3ware in your applications?

I always had acceptable performance out of my 7500-4LP in RAID5 (linux file server), but a 7500-2 RAID1 (Win2k3) that I installed really sucks. I attributed it to some limitation of the 2 channel cards.

I wouldn't recommend the 7xxx or 8xxx series cards, but the 95xx look interesting as well as the later PERCs. I have a hard time recommending the Adaptec after decades of terrible performance and high price even though it looks like they may have a decent product right now.

 

[Stereodude]

How does the Perc5 compare with the 3ware in your applications?

The Perc 5/i is a monster. The 3ware is kinda ho-hum. Even after seeing the Perc 5/i's performance, had I known about the registry tweaks for the 3ware 7506-8 it would have been more than adequate in my main server. I wouldn't have felt the need to upgrade for speed. It was just over half full when I took it out of commission a few weeks ago.

Oh well... The 3ware will work fine in my backup server (even without registry tweaks). The gigabit ethernet connecting the two servers is the speed bottleneck, not the 3ware card.

I always had acceptable performance out of my 7500-4LP in RAID5 (linux file server), but a 7500-2 RAID1 (Win2k3) that I installed really sucks. I attributed it to some limitation of the 2 channel cards.

I'd guess it was the whole FUA thing like my 8 channel card in Windows. The Linux drivers probably either don't support FUA, or the writes in Linux don't use issue FUA's.

I wouldn't recommend the 7xxx or 8xxx series cards, but the 95xx look interesting as well as the later PERCs. I have a hard time recommending the Adaptec after decades of terrible performance and high price even though it looks like they may have a decent product right now.

The Perc is a LSI, and really the 3ware is a LSI now too (LSI bought 3ware), so I'm not sure where Adaptec comes into the discussion. :erm:

 

[blakerwry]

The Perc is a LSI, and really the 3ware is a LSI now too (LSI bought 3ware), so I'm not sure where Adaptec comes into the discussion. :erm:

We are already comparing two popular manufacturers - why not throw in another?

 

[Howell]

I like poking around with old stuff. Right now I have an Adaptec 3000S SCSI controller with four 18GB Cheetahs attached. Now to find drivers. :grin:

Would you like an 8G Seagate x15?

 

[Bozo]

Would you like an 8G Seagate x15?

I'll pass thank you. Right now I have 7 Adaptec 3000s SCSI cards and about 20, 18GB x15's sitting in my cube. I've been thinking about a giant RAID 0 array, just for fun.