Hardware or Software RAID?

About 4-5 years ago when I first made a start on learning and using Linux one of the questions was towards RAID, given you have more than one way to skin a cat so to speak. Which way to skin it?

I was told by a manager (and he was saying this with 100% solidity)"hardware RAID IS the best RAID". - I have yet to see this proven.

Loose Background

Years ago hardware RAID used to be the better option as CPU's were considerably slower so whilst software RAID is constantly running will consume a fair amount of CPU resources (thus additional overhead) combined with the lack of well designed software RAID (or for example firmware RAID on older motherboards) meant you would be better of paying for a dedicated card to handle this as it also has things like BBU + cache so it is able to reorganise write operations prior to flushing to disk at same time keeping writes ready to be flushed even if power is temporarily out to maintain a consistent state.

Questions arised and can be asked such as:

What if the hardware RAID card fails?

If software RAID is improved can we spend less money on HW?

Can rebuilds be done faster through software than hardware RAID?

Perhaps we should integrate LVM/VFS layer together?

Should software RAID be done user space or kernel space?

Is it possible to have software reorganize I/Os like hardware?

What happens to the state of the array if the cache after 72 hours is gone?

etc...

Linux mdadm is quite alot better, you also can use BTRFS or ZFS. I've played around removing drives and rebuilding etc using mdadm. I no longer bother now as I just use ZFS for all my storage needs.

In short Software RAID is now at a stage that it is faster than hardware RAID, provides end-to-end checksumming (so no data corruption), organizing writes to convert random writes into sequential writes (whilst providing dynamic block allocation) and can be very efficient in terms of it's resource usage.

See Tomasz's comments throughout - http://markmail.org/message/6t6d7tp4yrneorzr#query:+page:1+mid:duk2cb3a6nzoai7a+state:results

Test that compares software and hardware RAID by Robert - http://milek.blogspot.co.uk/2006/08/hw-raid-vs-zfs-software-raid-part-ii.html

and as referenced also from "Unix and Linux System Administration Handbook fourth edition"