Migrating from single disk to 3-disk RAID 5 (on Gentoo)

When following this guide a disk crash before the final step might still result in data loss, so if your data *is* important, back it up!

As my 1TB harddrive was filling up I was considering the ways I could expand storage. I hate usb drives, just for the simple fact that they're slow. I also don't like external storage at all, unless it's network storage. However, network storage would be slow for me too unless I at least spend some money to buy a decent gigabit switch instead of my 10/100 switch.

The option I liked most turned out to be to add another internal drive. I was already using LVM, so extending the volume group to span another disk wouldn't be hard, but that had it's serious downsides too. I'm not someone who does regular backups, I actually just don't do them at all unless something is really broken or I really feel I need to. Last time either of those happened is longer than a year ago I think, so a lot has changed since then. Now there's a lot of stuff I just don't need to backup, for the simple fact that it's someone else's service. I only use IMAP mail accounts and that's about the most important thing I could lose...

Because I don't do regular backups, I hope my harddisks will stay alive. Starting with a little bit of data and a new harddisk I can survive a near-instant crash. However when I would extend my 1TB lvm volume group with another drive, the risk of losing all data would immediately become twice as high. That's something I'm not willing to risk, because there's ways in which I simply don't have to risk it.

The solution I came up with was to buy 2 extra drives and create a RAID 5 array on top of them. A RAID 5 array requires at least 3 block devices, best to use 3 different disks for that, and will allow recovery of all data if at most one drive fails. It's possible to start with 3 drives and later expand to more while keeping all data on the disks. Another advantage is that data will be spread across disks, so reads and writes will also be spread across disks, so they will both be faster (the processing overhead for RAID 5 is very low).

When I made up my mind I had to figure out how to transition from my single disk to the RAID array, without having to use yet another disk to temporarily store my current data. A while ago I found some pages that pointed out that it's possible to create a 'degraded' RAID array. A degraded array means that on drive fail you might lose data, but when I still have all data on the old disk that won't be an issue. After having all the data on the degraded array, the old drive can be added to the array to fix the array integrity again.

So to summarize all of the above this is what I had to do:
- Add the drives to the system
- Create a degraded RAID 5 array on the 2 new drives
- Copy all data to the degraded RAID array
- Clean the old disk and add it to the array

Of course in reality I also had to compile a kernel with linux software raid support, because it was not there yet.

And here's what I typed with some additional explanation:

# paludis -i mdadm

Install mdadm, the linux software raid utility

# fdisk /dev/sda
# fdisk /dev/sdc

Partition the disks, in my case it just meant creating a full size partition with type fd (linux raid autodetect). Yes, my new disks are sda and sdc, I might have switched cables somewhere...

# mdadm --create -f /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdc1 missing

Create the degraded RAID 5 array with a missing drive.

# mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Create a usable config file. Don't know when I would need it, but hey, it's there when I need it...

# pvcreate /dev/md0

Initialize a physical volume for lvm.

# vgcreate vg /dev/md0

Create the volume group 'vg' on the RAID array. Keep in mind that it needs to be different from the volume group name you was already using, if you were already using one...

# lvcreate -L30G -nroot vg
# lvcreate -L1200G -nhome vg

Create logical volumes for the root and home partition (that's all I use).

# mkfs.ext4 -L root /dev/vg/root
# mkfs.ext4 -L home /dev/vg/home

Create a filesystem, why not use ext4 immediately? Especially nice with biggish files.

At this point it's time to copy over all files, everyone has their own opinions about how this should be done, I just cped them over from the live install, leaving out /dev, /proc, /sys and creating them afterwards, copied /dev/null and /dev/console (copying all of /dev isn't really feasible, and those are all you need in gentoo) and I was done copying.

Now there's some additional stuff to worry about. I'm using gentoo, so I have to worry about being able to start from my RAID array too. Because I wanted to use the full disks for my RAID array I'm going to have to boot from another device, like a usb flash drive. So I started by copying my kernel with RAID and lvm support to my usb drive, created the entry in my syslinux config and created an initrd with

genkernel --lvm --mdadm initrd

. This created a nice initrd, with the helpful message that to use the lvm stuff I also had to pass dolvm on my kernel commandline. Now it was time to see if it all worked, and... fail... My RAID array was not detected, so all I could use was my old disk. After a while I found that for mdadm support you also need to pass 'domdadm' on the kernel commandline, and that solved all of my issues with booting, now let's hope that gets documented somewhere too.

I was now able to boot using my new lvm logical volumes on my new RAID 5 array. I actually started copying my home volume when I was able to boot, because that would take quite some time. All in all there was just one thing left to do, and that was adding the old drive to the RAID array (after changing the partition layout of course):

# mdadm /dev/md0 --add /dev/sdb1

Add sdb1 to the raid array.

Immediately the raid array will be resynced. This will take quite a while, because the RAID array itself has no knowledge about the data on it, so the entire disks are being synced.

While this all happens and after this all happens you can

$ cat /proc/mdstat

to see the status of the RAID array.

Now you should have a working RAID 5 array, just make sure you don't delete your data because a RAID array won't protect you from being stupid.

For a more detailed guide on just setting up the RAID array with a missing drive see Linux Software RAID 101 -- Part 3: creating an array with a missing drive, the guide I used to create my RAID array (you will notice all the mdadm commands are directly copied from that page).

Issues with syslinux

A small followup to my post about a syslinux bootable usb stick, I'm writing down the current issues I have ran into.
1. syslinux is not your average bootloader
2. syslinux on a fat32 usb drive doesn't like to get recognized as USB-FDD

Point 1 is easy to understand. Syslinux is made to be useful in the special cases, not to be useful in about any case. Want to boot to a different kernel? Edit the config file. I think grub is far more userfriendly than syslinux will ever be, so I'm definately doing that grub bootable usb drive soon.

Point 2 is something different... I don't know if grub will work when the system tries to boot from the usb device as if it's a floppy drive. I don't know what's going wrong for syslinux either, but I would not be amazed if either the system or syslinux decides that it can't read beyound 2.88M (or 1.44, but I guess it would at least support large floppy disks). If it's the system, then grub probably won't make any difference at all... So now you might ask, why would you want to boot with usb floppy drive emulation? Well, I don't want to, but it seems some machines have no other option depending on the kind of usb device you use...

Expect a grub multiboot usb drive howto soon.

Kernel Mode-Setting on Intel hardware

Today I was going to try kernel modesetting, or kms in short, on my machines with intel graphic chipsets. First I was going to try it on was my 'old' laptop with an intel 915 chipset. I configured a 2.6.29-rc1 kernel and tried it, but as soon as it wanted to change mode my screen would turn black and only show something as soon as X was started again. One time it did seem to have working consoles again after X was started, but that didn't seem reproducible.

I decided to move to the next victim, my MSI Wind with intel 945 chipset. I was hoping a newer chipset would work better, so I took the same route again and rebooted, and it worked! The only thing that didn't work was X, but I knew I had to get something near version 2.6 of the intel Xorg driver to have that working with kms again.

So while I installed new drivers I went looking for the differences between the kernel configurations of both machines. I started at the framebuffer section, because I knew I had the uvesafb driver compiled in on my 'old' laptop and I had intelfb compiled in on both laptops, so my first guess was to disable the uvesafb driver and try what happens. Immediate success, I had native 1680x1050 on the console on my 'old' laptop, and X would even start. I didn't have recent intel Xorg drivers however, so switching between X and consoles was still as slows as it had always been. But having the consoles at native resolution is already so much of an improvement, that you shouldn't really care.

Meanwhile, my MSI Wind had finished installing newer intel Xorg drivers and I was ready to start X, success again. Switching between X and consoles is simply amazingly fast. I have however noticed a small issue here, somehow gdm thinks the vertical resolution is about twice what it really is, so I only see the top half of the screen and thus no login box. Once I'm logged in the issue seems to persist and my cursor is invisible, but my terminal keyboard shortcut works and the terminal is completely visible, so I can live with that for now.

Next machine is my Asus EEEpc 701/4G surf, which features an intel 915 chipset again. I compiled a 2.6.29-rc1 kernel again, using the config I made last week when I created my bootable usb stick. Copied the kernel to my usb stick, booted and... failure. Same kind of problem as on my 'old' laptop, screen turns black. Because I use the gentoo minimal image I never get to a X server either, so it stays black. Back to the kernel config, what went wrong? Somehow console framebuffer support was turned off, and I also had some other options turned off like low level display driver support. I don't think the latter really matters, but the first was an issue. After that it booted fine, except for the fact that I now used the in-kernel squashfs support which was at version 4.0 already, while the patch I previously used to patch my kernel sources was at version 3.4. Seems like the disk-format changed in between, so I can't mount the image, but I can get the fancy native resolution console.

Everything seems to work now, except for one thing, my 'old' laptop refuses to start X after updating to one of the 2.6 beta intel graphic drivers. Seems to get a hard lock, so I'll have to dig into that. I might have some newer packages installed on my msi wind, so I'll start by comparing versions and then I'll see. So far I'm happy.

Other issues I noticed are that on all the machines I seem to get some output from the drm driver that something is going wrong, including occasional call traces where something apparently went terribly wrong, but almost everything works. I also noticed that after a while the screen blanks, even though I think it shouldn't. Even worse, pressing a key won't turn it back on again, but switching to console, pressing a key does. After that I can switch back to X again and it works.

So now I only have one thing left to do, and that's check if it also works on the MacBook me and my girlfriend use together, but I guess I'll have to do some more work on my bootable usb stick before that's going to work.

Creating a multiboot USB-stick using syslinux

Since I have an eeepc-701, I need a way to boot it from external media so I can mess with it. I have a Corsair Flash Survivor (16GB), so I'm going to use that for rescue-goodness. I'll give step by step instructions on how to create a similar boot disk. In these instructions I'll assume that /dev/sdc is the usb drive, /dev/sdc1 is the partition we're using for the filesystem. Further mine will be mounted at /media/Corsair and I'll also assume you can find the appropiate packages if you haven't gotten them yet (dosfstools and syslinux in gentoo). Last but not least most of these commands will need root permissions, so prepend them with sudo if that's your thing, or become root and do it all as root.

Format the usb drive as fat 32
Of course there are other options than fat 32 to create a bootable usb drive, but syslinux needs a FAT filesystem to work.

# mkdosfs -F 32 -n Corsair /dev/sdc1

Mount the usb drive
Unplugging and replugging will work for me as I use Gnome with automount support, should probably work for most people using a larger desktop environment. If not:

# mkdir -p /media/Corsair
# mount /dev/sdc1 /media/Corsair

Create a syslinux directory
Syslinux needs somewhere to store its files, I prefer doing that in a directory so I don't get a cluttered view every time I look at the files on my usb drive.

$ mkdir /media/Corsair/syslinux

Unmount and install syslinux
Unmount in whatever way suits you, just don't unplug yet. The -d option tells syslinux to use our directory 'syslinux' to store its files, just make sure the directory is there before running syslinux.

# umount /media/Corsair
# syslinux -d syslinux /dev/sdc1

Mount again
Remount the device to put more on it than just syslinux.

# mount /dev/sdc1 /media/Corsair

There's only two things left to do now:
1. Move images to the usb drive
2. Create the syslinux.cfg file

Move images to the usb drive
I create two directories to start with: gentoo and gparted. Inside the gentoo directory I put the kernel, initrd and squashfs from the minimal install cd and renamed them to include the release number in the files. At first I named my kernel just 'kernel-2008.0', but that seems to confuse syslinux so I renamed it to kernel-2008.0.img in the end. Better extensions suggestions for the kernel are welcome. I also copied all the .msg files from the gentoo cd to the syslinux directory I created on the root of the filesystem, so I can press F1-F6 and see irrelevant gentoo messages, however, changing these is easy and will be done when I have more time. It also seems gentoo needs a file called 'livecd' on the root, else it won't find the media as the fs media. It's cluttering the drive, but removing the file will just cause gentoo not to find a valid root fs. Last but not least I compiled a kernel that has support for most of the hardware in my eeepc 701/4G surf, and named that 'kernel-eee' straight on the root of the USB drive. I needed at least 2.6.28 so I could have support for the onboard network card, whatever way I'm installing it I'm gonna want network support. The kernel is monolithic (no modules and no module loading support) and works fine with the gentoo 2008.0 squashfs.

Inside the gparted directory I again put the kernel, initrd and squashfs files. I just took them from the livecd, left names as they were.

Create the syslinux.cfg file
I took most of this file straight out of the isolinux.cfg that came on the gentoo cd and customized where necessary.

default gentoo
timeout 150
prompt 1
display boot.msg
F1 kernels.msg
F2 F2.msg
F3 F3.msg
F4 F4.msg
F5 F5.msg
F6 F6.msg
F7 F7.msg

label eee
  kernel /kernel-eee
  append root=/dev/ram0 init=/linuxrc  dokeymap looptype=squashfs loop=/gentoo/minimal-2008.0.squashfs  cdroot initrd=/gentoo/initrd-2008.0.igz

label gentoo
  kernel /gentoo/kernel-2008.0.img
  append root=/dev/ram0 init=/linuxrc  dokeymap looptype=squashfs loop=/gentoo/minimal-2008.0.squashfs  cdroot initrd=/gentoo/initrd-2008.0.igz vga=791

label gentoo-nofb
  kernel /gentoo/kernel-2008.0.img
  append root=/dev/ram0 init=/linuxrc  dokeymap looptype=squashfs loop=/gentoo/minimal-2008.0.squashfs  cdroot initrd=/gentoo/initrd-2008.0.igz

label gparted
  kernel /gparted/vmlinuz
  append initrd=/gparted/initrd1.img boot=live union=aufs noswap vga=791 ip=frommedia nolocales live-media-path=gparted

label memtest86
  kernel memtest86

Syslinux uses relative paths by default, so if you use no '/' syslinux will look for the files inside the dir syslinux is. The solution is simple, use absolute paths for the kernel and initrd and it's all OK. Most important change made to this file is the append lines. For gentoo, the loop parameter needs to point to the squashs image, so I changed that to /gentoo/minimal-2008.0.squashfs (name of the image in my case). I added a line for my eeepc kernel, which just boots into a gentoo minimal cd.

Next was gparted. Gparted supports a 'live-media-path=' parameter, this needs to be used when the gparted squashfs image is not directly in the root of the filesystem on the usb drive. Other than that gparted just looks for any file with name *.{squashfs,ext2,ext3,jffs2} so the name doesn't really matter as long as the extension hasn't changed.

I also left the memtest line in place, added the memtest image to the root of the drive and my rescue-drive is mostly done.

It should also be possible to do the same thing using grub, except for the F1-F6 messages and the default message (or can grub do that too?). I'm definately gonna add grub to it, probably replacing syslinux with grub. Using grub might also allow for use of other filesystems on the drive, which might end up being a big plus too. Come back again to read about that, once I've tried it myself.

Configuring a linux kernel for another architecture than host

On my 64 bit gentoo installation I wanted to compile a kernel for 32 bit machines (specifically my eee pc 701), but somehow menuconfig was as smart to notice I was running a 64 bit machine and so showed me stuff to build a 64 bit kernel. After a quick search I found the solution:
make ARCH=i386 (oldconfig|menuconfig|)

GDM in Xnest

I wanted to change my GDM theme and watch it without logging out, so the logical thing for me was to run it inside an Xnest. I had no clue how to do that though, but after some searching I found the following:

$ gdmflexiserver --xnest

Voila, you never need to know more...

Using ext4 on Gentoo (and probably other distro's)

Keeping it short:
when creating ext4 filesystems with e2fsprogs' (1.41.1 works great for me) mkfs.ext4 I couldn't mount them. The error I found in dmesg was

[ 5994.790520] EXT4-fs: loop0: not marked OK to use with test code.

After a bit of googling I found that to use ext4 filesystems with the current in-kernel ext4dev driver you need to tell the filesystem it can be used by an experimental driver. The option for mkfs.ext4 to do this is -E test_fs. After this you can mount your ext4 filesystems without any issues (not without specifying ext4dev as filesystem with -t though).