With my Linux SW RAID screaming “grow me!” for quite a while now, I finally brought myself to replace the old 2TB disks with new 6TB ones (RAID 5 with 4 disks). While such a disk-upgrade has to be performed regularly, the frequency is so low that it is hard to remember the details when you finally get to do it again. Unfortunately the “official” method (replace & resync disk-by-disk and then grow the md and the filesystem) as suggested in the Linux RAID Wiki has a few drawbacks:

• you have no backup in case of failures during the 4 RAID rebuilds
• you continue to operate on the old filesystem, in my case where the RAID has been full for quite a while you will inherit quite a bit of unnecessary fragmentation – and you cannot switch nor re-tune the filesystem which could make sense for a significantly bigger RAID

Luckily Adrian reminded me of mdadm’s missing parameter, so I could perform this alternate RAID upgrade which I’ll detail below (should come in handy for my next upgrade).

1. After running tests on all new disks (a full write and a long S.M.A.R.T. self test) I copied all data from the RAID to one of the new disks. As the server was still in operation I opted for the tar|tar approach with a final rsync to complete the replication (for details see this stackexchange thread) to copy the data. Note that if you don’t have a spare port you will have to fail one of your existing drives to use it’s port for the new disk instead.
2. Now remount the filesystem so that the server uses the new copy (will require stopping and restarting services that were using the relevant mount point).
3. Replace all old drives and replace them with the remaining new disks, create a new RAID but do not include the drive on which you have copied the contents of the RAID – note that you will have to replace the blue variables with the real identifiers – as you can see /dev/sd4 the one where the currently mounted copy of the data resides:
   mdadm --create /dev/mdx --chunk=256 --level=5 --raid-devices=4 /dev/sd1 /dev/sd2 /dev/sd3 missing
   I went with full drives instead of partitions and also for a smaller than default chunk size (512k) as I wanted to use XFS and the current implementation doesn’t support chunk sizes bigger than 256k (there seems to be a dispute on the usefulness of the new 512k default in mdadm).
4. Update the DEVICE information in mdadm.conf if necessary, remove the old ARRAY entry and append one for the new drive with this command:
   mdadm --detail --scan >> /etc/mdadm/mdadm.conf
5. Now create the filesystem on the RAID device, to match my chunksize I used this command to create it:
   mkfs -t xfs -d su=256k -d sw=3 /dev/mdx
6. Mount the new RAID to a temporary location and repeat the replication as outlined in the first step.
7. Remount to the new RAID device and add the now unmounted disk to the new array with:
   mdadm --manage /dev/mdx --add /dev/sd4

The migration is complete with the final RAID sync that starts automatically after adding the drive. Except for the remounts, the system can stay operational during the complete procedure (thanks to SATA hot-plugging) and as the old disks stay untouched you always have a backup available.