Ubuntu software raid5 and raid6 with mdadm

Oh the joys of mdadm

So I have time and time again either seen or heard about someone losing all their data.  When asked, did you have a backup of your stuff.  The answer is then a depressing no.  Well I have way to many important photos, important documents, music, etc… that I am certainly not willing to just lose because of a hard drive failure.  Roughly  two years ago I decided I couldn’t take the stress of the possibility of me losing my wives pictures or my pictures.  Pictures are the number one reason I wanted a secure backup.  I looked into hardware raid and software raid.  If I were to go hardware RAID the cards that I would be choosing from are in the $400 range.  I simple could not afford that on top of the price of hard drives.  So I researched a lot into software raid within ubuntu using mdadm.   The cost would only be for the hard drives and also my time of learning how to get it up and running.   I thought of ways to sell this idea to my wife because it was going to cost around $300 to get three 1TB hard drives.  Doesn’t seem like much but at the time we were broke college students and I didn’t really see her saying ok to that price.   Well I told her the idea of how I wanted to have a raid setup so we wouldn’t just lose everything from a stupid hard drive failure.  Of course I married a frikin sweet chica because she was all for it.  I was super excited because I was going to be buying not one, not two, but three 1TB hard drives.  At that time the was a lot of space, even though I knew I would be losing one drive to parity bits.  The price to pay for not being stressed out of losing data.

The sweet part about software raid with mdadm is that it is transferable.  I have changed out my motherboard and boom in one command line I have my raid up with all new components.  With mdadm I have the ability to expand existing RAID-5 arrays or even convert a RAID-5 to a RAID-6.

So to setup my RAID-5 system I bought 3 samsung 1TB hard drives and installed them.

I then needed to prepare the disks.

As root, issue an “fdisk -l” to show a list of available disks.

I had three, 1TB hard drives…  sdb1, sdc1,sdd1,  I wanted to use these for my RAID-5.  I had a drive sda1 as my main operating system drive.  The disks were unformatted so I then formatted them ext3 set the RAID flag in gParted.  gParted is “Gnome Partition Editor” and is extra piece of software to install to easily change partition information visually.  To install gParted simple type in:

sudo apt-get install gparted

In gParted you have to select each disk separately in the upper right hand corner.  With each drive I formatted it to ext3 and then right clicked on each new volume and selected the “Manage Flags” and checked the “RAID” flag.

Next had to create the md Device.

I installed mdadm “Mirrored Device Admin” by typing in:

sudo apt-get install mdadm

To create the RAID volume I typed:

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1

The “–verbose” is useful for debugging any issues because it displays more information on the screen as the RAID is being created.  The “–level=5” decides that it create a RAID-5 volume.  The “–raid-devices=3” tells it that there are three devices to the raid array.  Then the /dev/sdb1 /dev/sdc1 /dev/sdd1  are the three disks I wanted to use in the RAID.

On a side note, the man pages for mdadm is very useful just type in:

man mdadm

When I initiated the command for the raid to be created I was able to monitor the building of the array by typing:

sudo watch cat /proc/mdstat

Before I created a file system on the raid, I had to create my mdadm.conf file first.  The mdm daemon will need the information in this file on boot to re-assemble my array to be ready for mounting.  I created the file by typing:

sudo echo "DEVICE partitions" > /etc/mdadm/mdadm.conf

sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf

After that file was created, I then needed to create the filesystem and mount point.  To create the file system I formatted the new RAID to ext3 by typing:

sudo mke2fs -j /dev/mdo

This might take some time.  I then edited my fstab to make the RAID-5 volume automatically mount on boot.  I had my mount in /mnt/raid.  I added this line to my /etc/fstab file:

/dev/mdo /var/media auto defaults 0 3

I then mounted the array with is at /dev/md0.

So after all that and the time to let it re-sync, I had a RAID-5.

Later I changed all my drives over to 2TB drives.  The drives I went to are the Western Digital Green drives.  The model of the drive I got were the WD20EARS and they have the 4KB Physical Sectors.  I had to prepare the drives to make sure that each partition is aligned to begin on the 64th sector / 4KB boundary.  You can check the alignment of your drive with :

sudo fdisk -l -u

If any of the drives listed that you’re planning to use with a RAID-5/RAID-6 have a start lised at 63, you need to change it to 64.  Here is how to change it:

Run fdisk - "sudo fdisk -u /dev/sda"
Delete existing partitions - "d"
Create new partitions - "n"
Tell it primary partition - "p"
Tell it partition 1 = "1"
Tell it to align at sector 64 - "64"
Change the partition type (if you want the drive as part of your mdadm raid array) - "t"
Select Linux raid autodetect fs - "fd"
Write & Quit - "w"

Repeat that procedure with all your disks that you are going to be using in your raid.

I had three 2 TB drives and I just bought another.  I think I have an sickness.  Well I had plenty of space with the raid 5 and I am way to paranoid of losing data.  With the 4th drive I just bought I wanted to convert my RAID-5 into a RAID-6.  I had to update to my MDADM software in order for me to convert it with only one drive.  In order to convert I had to add the extra drive as a spare to the raid:

mdadm --add /dev/md0 /dev/sde1

Then I used the “–grow” command to use that 4th drive and convert the raid into a RAID-6.

mdadm --grow /dev/md0 --level=6 --raid-devices=4 --backup-file=/root/md0.backup

I can not stress how important it is to have the “–backup-file” in the command.

If –backup-file was used to grow the number of raid-devices in a RAID5, and the system crashed during the critical section, then the same –backup-file must be presented to –assemble to allow possibly corrupted data to be restored.

Pretty much that saved my butt.  I was in the middle of converting the raid and I went out that evening.  When I came back I found out the power went out.  I was sweating bullets!!!  I wasn’t sure if I should try the grow command or assemble.  I went with assemble.

sudo mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 --backup-file=/root/md0.backup

mdadm: restoring critical section
mdadm: /dev/md0 has been started with 3 drives (out of 4).

I was more than relieved when I saw that and mounted my degraded raid and saw all my data perfectly fine.  It was rebuilding after that and after a long wait I had a RAID-6 array.  It was a beautiful sight.  Now I can have up to any 2 of my drives fail and I still have my data.

I just bought  new drive again. (I know it is a sickness)  Well this time I just growing the space of the raid and keeping it a raid 6.  So I add the drive to the array as a spare:

mdadm --add /dev/md0 /dev/sdf1

Then I give the grow command.

mdadm --grow /dev/md0 --raid-devices=5

After that was done, which took about 2 full days, the filesystem then needed to be expanded to fill up the new space.  We check the filesystem then resize it.

fsck.ext3 /dev/md0

resize2fs /dev/md0

Then boom… More space.

Hopefully it wasn’t too confusing and I hope it was informational.

Disclaimer:  RAID is no perfect replacement for having good backups!  Has been said one needs at least have an offsite backup as well.