Archive for the 'grid' Category

Cluster Installation Finished

Published on April 2, 2010 in grid. 0 Comments Tags: , , .

The hardware of our cluster is finally installed and ready. All 180 compute nodes (almost) are ready, Infiniband is working and the lustre is mounted.

First Infiniband benchmarks gave us results of about 23 GBit/s which is the expected bandwidth with our QDR network.

As a mirror admin I am bit frustrated that i cannot use the big filesystem which is mounted on every compute node for my mirror server:

172.31.100.222@o2ib,172.30.100.222@tcp:172.31.100.221@o2ib,172.30.100.221@tcp:/lprod
                       29T  819M   28T   1% /lustre/ws1

Now I still need to install the frontend servers. One is used for the users to log in and submit jobs and the other will contain the grid software as this cluster wil be part of the bwGRiD.

28th Open Grid Forum

Published on March 14, 2010 in adrian and grid. 0 Comments Tags: .

Starting tomorrow (2010-03-15), I will be at the 28th Open Grid Forum (OGF28) in Munich for four days.

80 Nodes Up And Running

Published on February 23, 2010 in grid. 0 Comments Tags: , .

80 compute nodes from our cluster are up and running. We are now waiting for more switches and the filesystem servers to finally get the complete cluster (with all compute nodes) operational. To get the remaining nodes operational all I have to do is to add their MAC address to a file and with the magic of some scripts everything else is configured automatically. Unfortunately it all depends on the missing ethernet switches which should arrive any day now.

RAID 1 Shrinking

Published on February 11, 2010 in adrian and grid. 0 Comments Tags: , .

I was not happy with the partitioning of one of the cluster infrastructure servers. It had a software RAID for /boot, one for swap and the rest was a big software RAID for /. I should have used LVM for / for easy resizing, but I forgot and so I had to do it the hard way. I wanted to resize /dev/md2 which was used for / and then use LVM for the rest.

First I had to resize the filesystem. Online shrinking is not supported for resize2fs (at least I was not able to do it) and so I had to boot the CentOS 5.4 rescue system.

After dropping to the shell of the rescue system (without mounting the filesystems) I copied a mdadm.conf from a similar system to /etc so that I would be able to start the RAIDs:

  • mdadm -A /dev/md0
  • mdadm -A /dev/md1
  • mdadm -A /dev/md2

Only starting /dev/md2 would have be enough, but I wanted to make sure that everything is working as it is supposed to. Then, before running resize2fs, I had to do a filesystem check:

  • e2fsck -f /dev/md2 -C 0

Next step was to actually shrink the filesystem and make it smaller than the desired final size:

  • resize2fs /dev/md2 30G

Then I shrunk the RAID to about 40GB:

  • mdadm --grow /dev/md2 -z 40000000

and after that I had to resize the filesystem again to use the 40GB:

  • resize2fs /dev/md2

At this point I mounted the filesystem to see if it actually worked and it looked good (and smaller). Now came the hard part; to use the remaining space I had to re-partition the disk. I started fdisk and deleted the corresponding partitions and created at the same start point smaller partitions (42GB). This was the part were I was really worried about losing all my data which was fortunately backed up (of course). After I created the smaller partitions I tried to start /dev/md2 and it failed, saying that it could not find any RAID partitions.
I then tried to create the RAID again, hoping all data would be still available. I first created the RAID with only one device:

  • mdadm --create /dev/md2 -n 2 -l 1 /dev/sdb3 missing

This seemed to work and after mounting the new RAID I saw that all my files were still there. So the next step was to add the second device to the RAID with:

  • mdadm --manage -a /dev/md2 /dev/sda3

At this point the RAID started to re-sync and 20 minutes later I was able to grow the RAID to the new partition size:

  • mdadm --grow /dev/md2 -z max

Again I had to wait and before doing the final filesystem resize another filesystem check was necessary:

  • e2fsck -f /dev/md2 -C 0
  • resize2fs /dev/md2

And after only two hours I finally had what I wanted. I rebooted the system and it came up with the smaller / partition. I used the remaining space to create a new RAID (/dev/md3) which will probably be used with LVM if I ever need more space on this server in the future.

Without having a backup I would have not done all the steps because I was not always sure it would actually work.

Cluster Installation: First Nodes Up

Published on January 13, 2010 in grid. 0 Comments Tags: , , .

Since Monday I am at the High Performance Computing Center Stuttgart (HLRS) and I have started the initial installation of our cluster.The people from the HLRS have offered to support us with the initial installation, which we gladly accepted because they know how to do clusters.

On Monday I installed the three infrastructure servers which are used to control the 180 nodes of the cluster. The cluster is running Scientific Linux and my first task was to get it on those three infrastructure servers.

Those servers have two 500GB disks and they were supposed to be running as software RAID. After the seventh failed attempt to configure the partitions as RAID1 with the Scientific Linux installer we used a Debian install DVD to partition the disks and after the successful configuration of the partitions as RAID1 we installed Scientific Linux on all three systems. Not knowing how to use anaconda to configure a RAID1 (like we wanted to) was a bit embarrassing, but with all the Fedora and CentOS installation I have done I have never configured a software RAID1 from the installer; either the system had only one disk, a hardware RAID controller or I configured the RAID manually after the installation. But at the end of the day all three system were installed and configured for their tasks.

Today (Tuesday) we used the installation to boot the first two nodes of the cluster. All the nodes are running disk-less and are booting over TFTP/NFS from a single read-only image.

Back In School

Published on December 10, 2009 in adrian, grid and mirror server. 0 Comments

Not really back in school, but it has been now more than one week that I started my new job at my old university in Esslingen at the beginning of December 2009. After only 11 months at my previous workplace (Matrix Vision) I am now working for the faculty of Information Technology.

I will be responsible for the setup and installation of the new cluster of the university. The cluster will be part of the bwGRiD and it will have around 1500 cores and is currently being installed. It is partly water-cooled and a few days ago the racks were delivered and installed. The cluster is from NEC and we are expecting the servers to be delivered in the next few days. The cluster will be running Scientific Linux.

I am now in the same building as my mirror server. This might be a good thing, because now I am much closer to the hardware and can act faster if something unexpected happens… It might also be a bad thing, because now I am much closer and can experiment with things I would not do if I was not in the same building.