For All You RAID Junkies

[THE JEW (RaVeN)]

A very comprehensive guide to RAID controllers:

http://www.tweakers.net/reviews/557/1


This one blows the one GamePC had on RAID 5 outta the water.

[THE JEW (RaVeN)]

RAID Setup Guide For Promise Based Controllers (ie. P4P800-E or P4C800-E)

http://www.techsupportforum.com/compute ... 723-1.html

[THE JEW (RaVeN)]

Very basic introduction to RAID:

http://www.aselabs.com/articles.php?id=200

[THE JEW (RaVeN)]

Another overview of RAID:

http://anandtech.com/storage/showdoc.aspx?i=3087

[THE JEW (RaVeN)]

Why FreeBSD has a very good file server option:

http://blogs.sun.com/bonwick/entry/raid_z

FreeBSD + zfs + RAIDZ

[THE JEW (RaVeN)]

Hetero vs homogeneous RAID setups:

http://www.tomshardware.com/2008/03/13/ ... rays_work/

Conclusion: mixing hard drive brands/models results in 5-25% performance loss.

[THE JEW (RaVeN)]

ZFS file server overviews:

http://breden.org.uk/2008/03/02/a-home- ... using-zfs/
http://breden.org.uk/2008/03/02/home-fi ... -hardware/
http://blogs.sun.com/timf/entry/tales_a ... ing_zfs_as


Why ZFS over Ext3:

http://scottstuff.net/blog/articles/200 ... ver-part-2


Build a ZFS file server:

http://blogs.sun.com/PlasticPixel/entry ... i_terabyte


RAIDZ:

http://weblog.etherized.com/?p=126
http://breden.org.uk/2008/09/01/home-fi ... expansion/


ZFS vs. Linux Raid vs. Linux LVM vs. Linux LVM + Raid

http://www.unixconsult.org/zfs_vs_lvm.html


OpenSolaris ZFS vs. Linux ext3 RAID5

http://www.prestonlee.com/archives/121


ZFS with a web GUI:

http://home.comcast.net/~robhensel/NAS/ ... 10_NAS.htm




BTRFS:

http://btrfs.wiki.kernel.org/index.php/Main_Page

The Linux GPL alternative to ZFS. Too bad it's only in beta.


FreeNAS installation:

http://developer.novell.com/wiki/index. ... ll_FreeNAS


FreeNAS packages:

http://192.168.201.143/system_packages_edit.php

[THE JEW (RaVeN)]

Toaster file servers using RAID:

http://www.tomshardware.com/reviews/toa ... ,2136.html

Has a good list of NAS OS software.

[THE JEW (RaVeN)]

Its days are numbered?

http://hardware.slashdot.org/story/09/0 ... art_pos=11


I disagree with the article, but whatever.

[[Fallen]Thumperup]

me too I love raiding things....although I have had a bad experience with Raid 5 like this article where my raid controler flaked out causing errors on the entire raid 0_o bad times entire raid dead and cant recover.

[THE JEW (RaVeN)]

Non-proprietary (ie. software RAID) for teh win.

[THE JEW (RaVeN)]

Open Solaris ZFS guides:

http://www.h-online.com/open/OpenSolari ... res/112212

http://blogs.sun.com/PlasticPixel/entry ... i_terabyte

http://blogs.sun.com/ralkire/entry/home_nas_server

http://swik.net/zfs/del.icio.us%2Ftag%2 ... aris/czm29





Solaris NAS:

http://seanmcgrath.wordpress.com/2007/0 ... -with-zfs/


Minimal Open Solaris Install:

http://opensolaris.org/jive/thread.jspa?threadID=107921


EON Storage:

http://sites.google.com/site/eonstorage/faq

http://eonstorage.blogspot.com/

http://forums.smallnetbuilder.com/showthread.php?t=1817

[THE JEW (RaVeN)]

A write-up on the Dell / LSI Perc and what to expect from it:

http://www.overclockers.com/dell-perc-5 ... enchmarks/

An excellent card for one's NAS.

[THE JEW (RaVeN)]

ZFS and Marvel hptrr tuning guide to get the Supermicro AOC-SAT2-MV8 to work:

http://www.endeavoursofanengineer.com/blog/tag/freebsd/

The new server for ieee.ucsd.edu arrived on Saturday. I have been working since Thursday to put together a configuration script to easily format all the hard drives and write the necessary config files to get the array up and running. Unfortunately, this sort of thing is never as easy as it seems. ZFS works very well in OpenSolaris, but I’m used to Ubuntu Linux. FreeBSD isn’t that much of a stretch, but Solaris is just a bit more different than I can handle at the moment. The plan I had walking in to this is to create a ZFS RAIDZ array across all the hard drives (10 in nanjing in 2 groups of 5 and 6 in the IEEE server) and have FreeBSD boot off of that. However, I ran into trouble from the get-go.

FreeBSD provides a memstick image for installing off of a flash drive. Unfortunately, the memstick install of FreeBSD wasn’t powerful enough, so I installed it straight to my flash drive and then copied the contents of the installation iso to /dist. Then I booted the flash drive in the server. FreeBSD only picked up half of the array. It turns out that somewhere along the line some nitwit included a binary blob driver for some obscure High Point raid controller that has a rather overzealous detection routine. The SuperMicro AOC-SAT2-MV8 cards that were in the server happen to use a generic version of the same chipset, so the driver grabbed onto the cards before the generic ATA driver could pick them up. Since the hptrr driver can’t communicate with the cards, none of the drives are detected. There are two possible fixes: either recompile the kernel to remove the hptrr driver, or use the 8.0-CURRENT development release that has a fix in it. So I downloaded 8.0-CURRENT and installed it to my flash drive. Unfortunately, I also needed to edit /boot/loader.conf to add hw.hptrr.attach_generic=”0″, but didn’t know it at the time. So I booted it up again and it still couldn’t see the drives. After some more googling and looking at the source of the actual patch in 8.0-CURRENT, I edited loader.conf and rebooted. Finally, the drives were detected properly.

Next, I started working on a script to write all the necessary partitions to the disks. It’s not a terribly complicated process, it just requires some bookkeeping and quite a bit of testing. The script creates three partitions on each drive – one for the GPT loader, one for swap, and the balance for the ZFS array. I found a tutorial on the FreeBSD wiki that covers the whole process and I copied most of the commands out of it. The only thing I didn’t like was the fact that there wasn’t any redundancy on the swap partitions, so I incorporated information from another FreeBSD ZFS wiki article that explained how to mirror the swap partitions. After many restarts and quite a bit of head banging, I finally got the script working. It could initialize the array and reinit any particular disk, just in case one fails and needs to be replaced or the array size needs to be upgraded by replacing disks one at a time. Unfortunately, I couldn’t get hot plug support to work properly on the controller card. It worked on the motherboard’s internal controller, but I just couldn’t get the Supermicro card to pick up a drive after insertion. I moved on from that since it’s really not something I can fix without buying a different controller card or investing an inordinate amount of time. The next problem is the operating system. Installing this involved little more than copying over code from the original tutorial. So I ran the script and installed the OS.

It failed to boot, complaining “zfs: I/O error, all block copies not available.” Apparently it can’t boot when it can’t see the whole array, since it can’t see the other half of the array though the Supermicro controller before the kernel’s drivers are loaded. So now I need to figure out how to get the dumb thing to boot without being able to see those drives. I didn’t want to split the pool, because that would be a nightmare. So I finally settled on a separate boot partition. I converted the first mirror from freebsd-swap to freebsd-ufs. Then I changed my script to mount this new partition to /zroot/boot, which would end up becoming /boot once the OS was installed properly. So I went ahead and ran my script again to installed everything, followed by a reboot.

It failed to boot, complaining that it couldn’t find the loader. Apparently freebsd always looks in /boot even if everything is sitting on the root, so you have to write a boot.config file to tell it where to look. Unfortunately, it wasn’t that simple. It managed to start the loader with a boot.config, but it couldn’t load the kernel because it was still looking for stuff in the /boot directory. Sheesh. So I changed my script (again) so that it mounted the boot partition to /zroot/mnt/boot and then created /zroot/mnt/boot/boot and then mount_nullfs’d that new directory to /zroot/boot. That finally worked and allowed the system to boot the kernel. I had to reboot a couple of more times to add a few lines to config files and correct a couple of bugs in my fstab generator, but I finally managed to get a boot prompt. The only problem I have now is hardware-related – sustained ZFS operations will cause the server to immediately shut down. I originally thought it was a bad raid controller board (it’s used, bought it off ebay from israel; in hindsight probably not the best idea), but flip-flopping several cards has not made the problem go away in nanjing. However, ZFS works beautifully on the IEEE server. I am quite impressed. Now, the trick is to wait for virtualbox and all of its dependencies to compile, one of the downsides to the freebsd ports system. It does make for a good stress test, though – the compiler regularly pegs both CPUs to 100% and performs a heck of a lot of disk i/o to the zpool. So far, the only reboots have been ones that I initiated for configuration purposes.

The issues with nanjing, however, have been extremely disappointing. I am still trying various permutations on hardware and I am still getting completely repeatable sudden shutdowns, definitely indicating a hardware problem of some sort. Nanjing and shanghai are both supermicro servers with 15 SATA hard drive bays containing 10 250GB hard drives, one 8GB CF card in a CF to SATA converter, 8GB of RAM, 2x AMD Opteron 2352 quad-core processors and three redundant power supplies. They are both pretty nice servers, but nanjing is more or less down for the count until I figure out what’s wrong with it. It’s really quite annoying since it shuts down with no messages of any sort, so the only thing that can be done is guess and try different hardware configurations.

The next step for the IEEE sever is getting virtualbox up and running. So far, I managed to get it compiled an installed. I also need to import all the data from the old server hard drives. Hopefully that will be pretty straightforward.

[THE JEW (RaVeN)]

RAID Recovery for Windows:
http://www.runtime.org/raid-recovery-windows.htm

RAID Reconstructor:
http://www.runtime.org/raid.htm


No matter what card or chipset use, it will recreate the RAID so you can save your data if possible.