Monthly Archive for February, 2010

I was not happy with the partitioning of one of the cluster infrastructure servers. It had a software RAID for /boot, one for swap and the rest was a big software RAID for /. I should have used LVM for / for easy resizing, but I forgot and so I had to do it the hard way. I wanted to resize /dev/md2 which was used for / and then use LVM for the rest.

First I had to resize the filesystem. Online shrinking is not supported for resize2fs (at least I was not able to do it) and so I had to boot the CentOS 5.4 rescue system.

After dropping to the shell of the rescue system (without mounting the filesystems) I copied a mdadm.conf from a similar system to /etc so that I would be able to start the RAIDs:

• mdadm -A /dev/md0
• mdadm -A /dev/md1
• mdadm -A /dev/md2

Only starting /dev/md2 would have be enough, but I wanted to make sure that everything is working as it is supposed to. Then, before running resize2fs, I had to do a filesystem check:

• e2fsck -f /dev/md2 -C 0

Next step was to actually shrink the filesystem and make it smaller than the desired final size:

• resize2fs /dev/md2 30G

Then I shrunk the RAID to about 40GB:

• mdadm --grow /dev/md2 -z 40000000

and after that I had to resize the filesystem again to use the 40GB:

• resize2fs /dev/md2

At this point I mounted the filesystem to see if it actually worked and it looked good (and smaller). Now came the hard part; to use the remaining space I had to re-partition the disk. I started fdisk and deleted the corresponding partitions and created at the same start point smaller partitions (42GB). This was the part were I was really worried about losing all my data which was fortunately backed up (of course). After I created the smaller partitions I tried to start /dev/md2 and it failed, saying that it could not find any RAID partitions.
I then tried to create the RAID again, hoping all data would be still available. I first created the RAID with only one device:

• mdadm --create /dev/md2 -n 2 -l 1 /dev/sdb3 missing

This seemed to work and after mounting the new RAID I saw that all my files were still there. So the next step was to add the second device to the RAID with:

• mdadm --manage -a /dev/md2 /dev/sda3

At this point the RAID started to re-sync and 20 minutes later I was able to grow the RAID to the new partition size:

• mdadm --grow /dev/md2 -z max

Again I had to wait and before doing the final filesystem resize another filesystem check was necessary:

• e2fsck -f /dev/md2 -C 0
• resize2fs /dev/md2

And after only two hours I finally had what I wanted. I rebooted the system and it came up with the smaller / partition. I used the remaining space to create a new RAID (/dev/md3) which will probably be used with LVM if I ever need more space on this server in the future.

Without having a backup I would have not done all the steps because I was not always sure it would actually work.