Magic comments

So you're writing some code, and there seems to be an error somewhere. As a real programmer, you're going to use printf or echo to debug your code. I happen to be writing a lot of PHP lately, so I also debug with var_dump. Not that it really matters, because after all it boils down to eliminating lines with errors, and as long as you use a programming language that supports // for single line comments and /* ... */ for multiline comments you can use magic comments.

Honestly, there's no magic involved, just a bit of logic. Let's say there's some code consisting of 3 blocks of lines:

[block 1]

[block 2]

[block 3]

If I want to comment one block, I could simply put /*, */ around it like this:

/*
[block 1]
*/
[block 2]

[block 3]

However, soon enough I figure I needed to comment block 2 for a while. Because I'm lazy, I try to type no more than needed, so it becomes this:

//*
[block 1]
/*/
[block 2]
*/
[block 3]

I'm almost happy now, but it seems I need to switch back to having block 1 commented again instead of block 2... Of course I can remove the added characters, or I can just add a few more:

/*/*
[block 1]
/*/
[block 2]
/*/
//*/
[block 3]

Now if I decide that I need to comment block 2 again, I only need to edit one character:

//*
[block 1]
/*/
[block 2]
/*/
//*/
[block 3]

And it's actually possible to chain these by using a /*/ between 2 blocks. If the block in front was commented, the next block won't be commented. If the block in front however was not commented, this block will be commented. And there you go, boolean logic with comments, it's almost magic.

Creating a bootable USB flash drive formatted as NTFS

Next in the series of bootable flash drives: NTFS support.

I don't really care what filesystem my devices are, as long as I can read and write them. So FAT32 would be a decent choice, right? No. FAT32 has a file size limit of only 4G and I tend to store larger stuff on my Corsair Flash Survivor. I came up with two different solutions:
1. Create two separate partitions, one with stuff GRUB needs access to, one with large stuff like the filesystem images and my data,
2. Create a NTFS partition and boot from that.

Of course I first came up with solution 2, but I couldn't find much information about grub and ntfs, except for total lack of support for ntfs. I was thinking about solution 1 and then decided it sucks if you're forced to use two partitions while you don't really want to, so I went on looking for booting from ntfs drives.

After a while I came upon grub4dos. I can't say I really love it (without spending any time investigating it seems it's more of a hack than an enhancement to grub), but it's grub and it boots from ntfs formatted drives. Once I had that figured out, it was time to start doing some work.

First, partition the flash drive so there's a large ntfs partition (in my case the whole 16G). In my case there was already a partition there, if there's not you can skip deletion of the partition.

# fdisk /dev/sdx

Command (m for help): d
Selected partition 1

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-125, default 1): [return]
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-125, default 125): [return]
Using default value 125

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 7
Changed system type of partition 1 to 7 (HPFS/NTFS)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Now it's time to create a filesystem on the flash drive, since we've chosen ntfs we'll need ntfsprogs, your linux distribution probably has it in it's repositories. Enter the following command to format the drive:

# mkfs.ntfs -L label -Q /dev/sdxY
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
mkntfs completed successfully. Have a nice day.

This shouldn't have been hard, if I actually needed to explain those steps to you at all, so let's continue with the serious part. The next few steps need a grub4dos archive. You can find one at their downloads page, I took grub4dos-0.4.4-2009-03-07.zip. Extract this and open a command line in the directory where the extracted files are. Now enter the following command:

# ./bootlace.com /dev/sdx

Disk geometry calculated according to the partition table:

        Sectors per track = 63, Number of heads = 255

Success.

If it says Success or something similar, then grub4dos is on your flash drive. You can actually boot from it, however for grub4dos to be useful it needs some files. Copy grldr to your flash drive (of course you need to mount it for that). It should be in the root of the drive. Now create a menu.lst in the root of the drive too (yes, this differs from grub behaviour). Edit the menu.lst file so it resembles a usable grub menu.lst and you're done.

The following block is my menu.lst, if there's any interest in the files and layout I used then just leave a comment and I'll be sure to answer any questions about it.

default 0
timeout 60
splashimage=/boot/splash.xpm.gz

title Ubuntu Jaunty Alternate Install AMD64
kernel /ubuntu/jaunty/vmlinuz
initrd /ubuntu/jaunty/initrd.gz

title Gentoo Minimal X86
kernel /gentoo/x86/gentoo root=/dev/ram0 init=/linuxrc looptype=squashfs loop=/gentoo/x86/image.squashfs cdroot
initrd /gentoo/x86/gentoo.igz

title Gentoo Minimal AMD64
kernel /gentoo/amd64/gentoo root=/dev/ram0 init=/linuxrc looptype=squashfs loop=/gentoo/amd64/image.squashfs cdroot
initrd /gentoo/amd64/gentoo.igz

title GParted
kernel /gparted/vmlinuz boot=live union=aufs noswap vga=791 ip=frommedia nolocales live-media-path=gparted
initrd /gparted/initrd1.img

title Memtest 86+
kernel /memtest86/memtest86-3.5

title Boot first harddisk
rootnoverify (hd1)
chainloader +1

title Reboot
reboot

title Shut down
halt

How not to fix an issue in an ebuild

As seen on http://bugs.gentoo.org/show_bug.cgi?id=224781 Comment #2 From Christian Hoffmann 2008-06-08 13:59:44:

cherokee ships a patched version of lighty's spawn-fcgi. We could try to make
spawn-fcgi an independent package again and patch in the additional features of
cherokee based on a USE flag. Or we could introduce a "spawn-fcgi" USE flag for
both lighty and cherokee and die in the ebuild if the other web server was
installed with USE=spawn-fcgi as well. Any ideas?

Yes dude, you totally understood it. I bet mister Hoffman likes to tell portage (or whatever package manager he uses) to continue after every second ebuild because there's a check halfway in the ebuild instead of a real solution...

What if the check fails? Well, then package install fails of course. And what then? Well if you were just rebuilding world then you can fix the issue and start over again, or continue and know that you still have to fix it...

Playing videos on a second display using mplayer

I decided I didn't need my home theatre pc anymore, because I don't record that much anymore (and that's the thing I liked most about it). I do however still have the need to play stuff on my tv and watch it from my bed, so I started by using mplayer to play videos to my already connected tv. I'm not covering the display setup here, just the needed information. My tv is display :0.1, I have multiple sound cards (my desktop speakers feature an internal soundcard), and the one my tv is plugged into is hw:1.0 (alsa). Putting this together with stuff lyign around on the web you get this:

DISPLAY=:0.1 mplayer -vo xv -ao alsa:device=hw=1.0 -fs /path/to/file

Using the above I have audio and video from my tv, so when I'm done watching I can turn the tv off and go to sleep... Now all I need is a remote!

Fallback kernels using GRUB (or safe rebooting of remote machines)

If you can't reach a certain computer (like a server that's colocated), but need to reboot it anyway, it's better to do it in a safe way. GRUB allows the use of the "fallback" command to specify a kernel to boot if it can't boot the kernel specified by default, but what if it can boot the default kernel but the kernel dies at some point? Well GRUB will be happy, it did boot a kernel.

The solution is in the "savedefault" option. When using "default saved" in the GRUB config file, GRUB will boot the kernel that was saved as default. Very easy to understand. But how can we use this to our advantage? Well let's assume we have a new kernel and an old kernel, named new and old, the config will probably look something like this:

default 0
fallback 1
timeout 5

title New
kernel /new root=/dev/sda1

title Old
kernel /old root=/dev/sda1

There's two things that need to be changed here. First the kernel needs to know what to do when it panics. Because there's one reasonable thing to do, it's implemented by means of a kernel commandline argument. Append "panic=5" to the kernel command line in the GRUB config and the system will reboot five seconds after the kernel panics.

The second change is that we need to tell GRUB that it needs to boot the second kernel if the first fails. We can't easily tell GRUB to boot the second only if the first has failed, but we can reliably tell GRUB to make the second default whenever the first boots. The changes will result in the following config (menu.lst):

default saved
fallback 1
timeout 5

title New
kernel /new root=/dev/sda1 panic=5
savedefault 1

title Old
kernel /old root=/dev/sda1

This config will instruct GRUB to set Old as the default kernel whenever New gets booted, and the "default saved" line will tell GRUB to boot the kernel saved as default. This can be chained to have multiple fallback kernels, but that's all up to you.

There's one issue left, if you want to reboot the system but the new kernel did work out, GRUB will still (correctly) assume it has to boot the old kernel. However there's a linux userspace utity that will allow you to change the default:

# grub-set-default 0

This will reset the default to 0. Of course if the system boots you can edit the menu.lst file by hand, setting the default to 0, but if you want to use it again you had better reset the default beforehand.

Creating a multiboot USB flash drive using GRUB

So I am currently booting my system from a USB flash drive, and probably will be for quite some time. One thing that annoyed me about syslinux is that it's really nothing more than a very simple boot loader, but sometimes you want more.

I've always used GRUB. GRUB has support for splash images so it looks nice. I also think the editing features of GRUB are great. Mistyped the kernel name after updating? Edit it in the menu. I also really like GRUB's default feature, which allows for some nice tricks, but that's for another post.

All in all, I wanted to get booting using GRUB. I've always installed GRUB 'by hand'. I've noticed the existance of grub-install, but the first time I ever read about it, I read about some issue with it. It was probably not important, but since I knew GRUB's setup commands by hand I didn't need it anyway. Turns out that if you run linux, GRUB will just map your USB flash drive as another harddrive (or at least it did for me with two different drives).

To install GRUB, it expects some files to be present. It also expects a filesystem to be present too if you install the same way I do. Because I like my USB flash drives easily accessible on multiple platforms, I format them as FAT32. I must say I only used them in linux so far, so if you're just like me you might as well format them ext2. In typable commands:

# mkdosfs -F 32 -n [somename] /dev/sdxy

Or:

# mke2fs -L [somename] /dev/sdxy

Of course sdxy refers to the partition on the flash drive you want to use, in my case it's /dev/sde1. You don't have to use labels (-n and -L options respectively), but I prefer to label my filesystems.

I also prefer to have my filesystems clean, so I try to keep all this bootable stuff contained in a folder. Mount your flash drive, and create a folder boot:

# mount /dev/sdxy /mnt
# mkdir /mnt/boot

I will be using the folder boot to store my kernels, initramfs's and the GRUB folder with GRUB files. GRUB needs just three (3) files to get installed: stage1, stage2 and the appropiate stage1_5. Easiest way to get these is from an existing install, if that's not possible for you, ask google for another solution (i.e.: "compile grub"). When using a FAT32 formatted flash drive you need to copy fat_stage1_5, when using ext2 or ext3 you should use e2fs_stage1_5.

# mkdir /mnt/boot/grub
# cp /boot/grub/stage{1,2} /mnt/boot/grub
# cp /boot/grub/fat_stage1_5 /mnt/boot/grub

GRUB now has all the files it needs, so let's install it:

# grub
grub> device (hd7) /dev/sdx

grub> root (hd7,z)
 Filesystem type is fat, partition type 0xb

grub> setup (hd7)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/fat_stage1_5" exists... yes
 Running "embed /boot/grub/fat_stage1_5 (hd7)"...  16 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd7) (hd7)1+16 p (hd7,0)/boot/grub/stage2
/boot/grub/menu.lst"... succeeded
Done.

grub> quit

this maps (hd7) to /dev/sdx, the root line needs to be changed to match z = y - 1, in other words if you're using partition 1, enter a 0 for z, increment both equally for other partitions. You should now be able to boot from the USB flash drive, but we're not done yet. I like to have a config too, and I included a nice menu background, so I copied splash.xpm.gz over to /mnt/boot/grub. As for the config, here's mine for a gentoo 2008.0 minimal livecd squashfs image on my flash drive:

default 0
timeout 30
splashimage=/boot/grub/splash.xpm.gz

title Gentoo Minimal LiveCD 2008.0
kernel /boot/kernel-2008.0.img root=/dev/ram0 init=/linuxrc dokeymap looptype=squashfs loop=/minimal-2008.0.squashfs cdroot vga=791
initrd /boot/initrd-2008.0.igz

Name the file menu.lst and GRUB should be able to find it. If you now boot from the flash drive, you should see the menu. And as another bonus, booting entries seems to be way faster than it is with syslinux!

Migrating from single disk to 3-disk RAID 5 (on Gentoo)

When following this guide a disk crash before the final step might still result in data loss, so if your data *is* important, back it up!

As my 1TB harddrive was filling up I was considering the ways I could expand storage. I hate usb drives, just for the simple fact that they're slow. I also don't like external storage at all, unless it's network storage. However, network storage would be slow for me too unless I at least spend some money to buy a decent gigabit switch instead of my 10/100 switch.

The option I liked most turned out to be to add another internal drive. I was already using LVM, so extending the volume group to span another disk wouldn't be hard, but that had it's serious downsides too. I'm not someone who does regular backups, I actually just don't do them at all unless something is really broken or I really feel I need to. Last time either of those happened is longer than a year ago I think, so a lot has changed since then. Now there's a lot of stuff I just don't need to backup, for the simple fact that it's someone else's service. I only use IMAP mail accounts and that's about the most important thing I could lose...

Because I don't do regular backups, I hope my harddisks will stay alive. Starting with a little bit of data and a new harddisk I can survive a near-instant crash. However when I would extend my 1TB lvm volume group with another drive, the risk of losing all data would immediately become twice as high. That's something I'm not willing to risk, because there's ways in which I simply don't have to risk it.

The solution I came up with was to buy 2 extra drives and create a RAID 5 array on top of them. A RAID 5 array requires at least 3 block devices, best to use 3 different disks for that, and will allow recovery of all data if at most one drive fails. It's possible to start with 3 drives and later expand to more while keeping all data on the disks. Another advantage is that data will be spread across disks, so reads and writes will also be spread across disks, so they will both be faster (the processing overhead for RAID 5 is very low).

When I made up my mind I had to figure out how to transition from my single disk to the RAID array, without having to use yet another disk to temporarily store my current data. A while ago I found some pages that pointed out that it's possible to create a 'degraded' RAID array. A degraded array means that on drive fail you might lose data, but when I still have all data on the old disk that won't be an issue. After having all the data on the degraded array, the old drive can be added to the array to fix the array integrity again.

So to summarize all of the above this is what I had to do:
- Add the drives to the system
- Create a degraded RAID 5 array on the 2 new drives
- Copy all data to the degraded RAID array
- Clean the old disk and add it to the array

Of course in reality I also had to compile a kernel with linux software raid support, because it was not there yet.

And here's what I typed with some additional explanation:

# paludis -i mdadm

Install mdadm, the linux software raid utility

# fdisk /dev/sda
# fdisk /dev/sdc

Partition the disks, in my case it just meant creating a full size partition with type fd (linux raid autodetect). Yes, my new disks are sda and sdc, I might have switched cables somewhere...

# mdadm --create -f /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdc1 missing

Create the degraded RAID 5 array with a missing drive.

# mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Create a usable config file. Don't know when I would need it, but hey, it's there when I need it...

# pvcreate /dev/md0

Initialize a physical volume for lvm.

# vgcreate vg /dev/md0

Create the volume group 'vg' on the RAID array. Keep in mind that it needs to be different from the volume group name you was already using, if you were already using one...

# lvcreate -L30G -nroot vg
# lvcreate -L1200G -nhome vg

Create logical volumes for the root and home partition (that's all I use).

# mkfs.ext4 -L root /dev/vg/root
# mkfs.ext4 -L home /dev/vg/home

Create a filesystem, why not use ext4 immediately? Especially nice with biggish files.

At this point it's time to copy over all files, everyone has their own opinions about how this should be done, I just cped them over from the live install, leaving out /dev, /proc, /sys and creating them afterwards, copied /dev/null and /dev/console (copying all of /dev isn't really feasible, and those are all you need in gentoo) and I was done copying.

Now there's some additional stuff to worry about. I'm using gentoo, so I have to worry about being able to start from my RAID array too. Because I wanted to use the full disks for my RAID array I'm going to have to boot from another device, like a usb flash drive. So I started by copying my kernel with RAID and lvm support to my usb drive, created the entry in my syslinux config and created an initrd with

genkernel --lvm --mdadm initrd

. This created a nice initrd, with the helpful message that to use the lvm stuff I also had to pass dolvm on my kernel commandline. Now it was time to see if it all worked, and... fail... My RAID array was not detected, so all I could use was my old disk. After a while I found that for mdadm support you also need to pass 'domdadm' on the kernel commandline, and that solved all of my issues with booting, now let's hope that gets documented somewhere too.

I was now able to boot using my new lvm logical volumes on my new RAID 5 array. I actually started copying my home volume when I was able to boot, because that would take quite some time. All in all there was just one thing left to do, and that was adding the old drive to the RAID array (after changing the partition layout of course):

# mdadm /dev/md0 --add /dev/sdb1

Add sdb1 to the raid array.

Immediately the raid array will be resynced. This will take quite a while, because the RAID array itself has no knowledge about the data on it, so the entire disks are being synced.

While this all happens and after this all happens you can

$ cat /proc/mdstat

to see the status of the RAID array.

Now you should have a working RAID 5 array, just make sure you don't delete your data because a RAID array won't protect you from being stupid.

For a more detailed guide on just setting up the RAID array with a missing drive see Linux Software RAID 101 -- Part 3: creating an array with a missing drive, the guide I used to create my RAID array (you will notice all the mdadm commands are directly copied from that page).