Further research leads me to believe that:
Software RAID (as on most motherboards) is done in a device driver loaded immediately post boot. Almost by definition, in the PC world, that would have to be in the first 640K of memory.
The same goes for bootable add-in cards that load their own BIOS.
Trying to load load two of them at the same time would cause not enough free low (<640KB) memory to be available to any PC operating system. Hence the issue with XP.
So what I would need is a hardware RAID card.
Question. Say I have six drives attached to a hardware RAID card, can I assign four drives to a RAID array and and have two as standalone - bootable drives? Or would I be better off just attaching them to the on-board SATA ports?
Actually, the issue is a little more subtle than you realise. (You're mostly there though).
With a traditional IBM based x86 BIOS, you only have approx 160KB for BIOS ROM space, as the first 1MB memory map is:
0x00000 - 0x9FFFFF - 640KB RAM.
0xA0000 - 0xAFFFFF - 64KB VGA Video framebuffer.
0xB0000 - 0xBFFFFF - 64KB Text mode framebuffers.
0xC0000 - 0xc7FFF - 32KB VGA option ROM. (approx - VESA cards typically use 64KB).
0xC8000 - 0xDFFFF - 160KB Option ROM space.
0xEFFFF - 0xFFFFF - 64KB BIOS ROM.
So you're got between 160KB and 128KB of ROM space for *any* add-in cards, and this space can also to be used by VGA cards, or the main BIOS (eg legacy USB support, etc), or any network cards, or any audio cards (even on motherboard audio will use this space). So on a modern BIOS based motherboard (UEFI based boards are a lot different), you may only have 64KB or less for any add-on card ROMs. IIRC blocks are allocated in either 8KB or 16KB blocks.
Stick in a PCI IDE RAID card, and it's used an additional 48KB of this space, and try to add in something else, say PXE boot support on your LAN adapter and you just run out of BIOS ROM space for any other cards. (With BIOS ROMs in the legacy x86 sense, it was all in KBs, hence why UEFI was a god-send for card manufacturers, pity it took sooo long to get into the x86 area). Also, for onboard drive controllers, the mode (AHCI, IDE, RAID) will also effect the amount of free ROM space you have as well (IDE having the smallest footprint (typically), and RAID having the largest due to having to support the RAID levels during the boot sequence).
Now, a software RAID or hardware RAID card will make zero difference in this regard, the code/ROM that accesses the card during boot (hooking int 13h) will have to be somewhere in that 160KB of space. What a hardware RAID card will give you is:
1. true multi-OS support, as most of the "driver" code is actually ROM firmware and runs independently of the systems CPUs (most of the time).
2. accelerated parity for RAID 3/4/5/6.
3. And more device options, eg SAS, more ports, options for port expanders, etc.
What I would do:
1. disable as many onboard items or features/functions as possible to free up the option ROM space. See if this helps with the current hardware.
2. Buy a drive controller that supports the number and types of drives you need. (even go as far as disabling the onboard ICH drive controller or setting it to AHCI mode if possible).
3. Have everything on the one drive controller if possible... like Merc mentions, you can use port expanders to get more drives on a SAS card. (I have seen up to 32 devices on a single card, but the theoretical limit is 65356 devices).
Also hardware RAID cards are mixed bag in what they support, but all decent ones do support JBOD with a single disk in the array so that will get around your issue. Also if you are not booting from the RAID array, why not use the OS provided RAID levels? (eg Dynamic Disks on Windows or LVM on Linux or ZFS on BSD/Solaris). That way, you don't need RAID support on the controller (allowing you to use what you have), or attempt to simplify your setup... (or ESXi the issue here)?
Now with UEFI systems (and UEFI compatible cards), IIRC the option ROM limit "per card" is 1MB, and IIRC 32-64MB is reserved for option ROMs. (but be aware the option ROMs in UEFI have to a do a lot more than with x86 legacy BIOSes).
PS. with the drive controllers, be careful with ESXi, it doesn't support everything out there (but I'm sure you were aware of this).