Dynamic DNS

For the last ten years I wanted to set up my own dynamic DNS service but was never motivated enough. Recently enough motivation was provided and using the scripts from http://www.fischglas.de/software/dyn/ made it really easy to set up a dynamic DNS service using bind. Following changes were necessary to the named.conf file:

zone "dyn.domain" in {         type master;         file "db.dyn.domain";         allow-update {                 key host.domain.;         }; };

Whenever the IP address of my host changes I am loading a URL with my hostname and password encoded. The script behind the URL checks if my hostname and password is correct and updates the zone file using nsupdate with a TTL of 120 seconds.

The script uses a simple configuration file (/etc/dyn/dyn.cfg) with the following content:

dns.key.name:host.domain. dns.key:yyeofEWfgvdfgdfgerX== authfile:/etc/dyn/secrets dns.host:host.domain debug:0

bcache Follow-Up

After using bcache for about three weeks it still works without any problems. I am serving around 700GB per day from the bcache device and looking at the munin results cache hits are averaging at about 12000 and cache misses are averaging at around 700. So, only looking at the statistics, it still seems to work very effectively for our setup.

RPM Fusion’s MirrorManager moved

After running RPM Fusion’s MirrorManager instance for many years on Fedora I moved it to a CentOS 6.4 VM. This was necessary because the MirrorManager installation was really ancient and still running from a modified git checkout I did many years ago. I expected that the biggest obstacle in this upgrade and move would be the database upgrade of MirrorManager as its schema has changed over the years. But I was fortunate and MirrorManager included all the necessary scripts to update the database (thanks Matt). Even from the ancient version I was running.

RPM Fusion’s MirrorManager instance uses postgresql to store its data and so I dumped the data on the one system to import it into the database on the new system. MirrorManager stores information about the files as pickled python data in the database and those columns were not possible to be imported due to problems with the character encoding. As this is data that is provided by the master mirror I just emptied those columns and after the first run MirrorManager recreated those informations.

Moving the MirrorManager instance to a VM means that, if you are running a RPM Fusion mirror, the crawler which checks if your mirror is up to date will now connect from another IP address (129.143.116.115) to your mirror. The data collected by MirrorManager’s crawler is then used to create http://mirrors.rpmfusion.org/mm/publiclist/ and the mirrorlist used by yum (http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-updates-released-19&arch=x86_64). There are currently four systems serving as mirrors.rpmfusion.org

Looking at yesterday’s statistics (http://mirrors.rpmfusion.org/statistics/?date=2013-08-20) it seems there were about 400000 accesses per day to our mirrorlist servers.

bcache on Fedora 19

After having upgraded our mirror server from Fedora 17 to Fedora 19 two weeks ago I was curious to try out bcache. Knowing how important filesystem caching for a file server like ours is we always tried to have as much memory as “possible”. The current system has 128GB of memory and at least 90% are used as filesystem cache. So bcache sounds like a very good idea to provide another layer of caching for all the IOs we are doing. By chance I had an external RAID available with 12 x 1TB hard disc drives which I configured as a RAID6 and 4 x 128GB SSDs configured as a RAID10.

After modprobing the bcache kernel module and installing the necessary bcache-tools I created the bcache backing device and caching device like it is described here. I then created the filesystem like I did it with our previous RAIDs. For RAID6 with 12 hard disc drive and a RAID chunk size of 512KB I used mkfs.ext4 -b 4096 -E stride=128,stripe-width=1280 /dev/bcache0. Although I am unsure how useful these options are when using bcache.

So far it worked pretty flawlessly. To know what to expect from /dev/bcache0 I benchmarked it using bonnie++. I got 670MB/s for writing and 550MB/s for reading. Again, I am unsure how to interpret these values as bcache tries to detect sequential IO and bypasses the cache device for sequential IO larger than 4MB.

Anyway. I started copying my fedora and fedora-archive mirror to the bcache device and we are now serving those two mirrors (only about 4.1TB) from our bcache device.

I have created a munin plugin to monitor the usage of the bcache device and there are many cache hits (right now more than 25K) and some cache misses (about 1K). So it seems that it does what is supposed to do and the number of IOs directly hitting the hard disc drives is much lower than it would be:

I also increased the cutoff for sequential IO which should bypass the cache from 4MB to 64MB.

The user-space tools (bcache-tools) are not yet available in Fedora (as far as I can tell) but I found http://terjeros.fedorapeople.org/bcache-tools/ which I updated to the latest git: http://lisas.de/~adrian/bcache-tools/

Update: as requested the munin plugin: bcache

Remove Old Kernels

Mainly using Fedora, I am accustomed that old kernel images are automatically uninstalled after a certain number of kernel images have been installed using yum. The default is to have three kernel images installed and so far this has always worked.

I am also maintaining a large number of Ubuntu VMs and every now and then we have the problem that the filesystem is full, because too many kernel images are installed. I have searched for some time but there seems to be no automatic kernel image removal in apt-get. There is one command which is often recommended which is something like:

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/(.*)-([^0-9]+)/1/")"'/d; s/^[^ ]*[^ ]* ([^ ]*).*/1/;/[0-9]/!d' | xargs sudo apt-get -y purge^[1]^

This works, but only if you are already running the latest kernel and therefore I have adapted it a little for our needs. Instead of removing all kernel images except the running kernel image I remove all kernel images except the running and the newest kernel image. No real big difference but important for our setup where we do not reboot all VMs with every kernel image update.

Running the script gives me following output
# remove-old-kernels

linux-image-3.2.0-23-generic linux-image-3.2.0-36-generic linux-image-3.2.0-37-generic linux-image-3.2.0-38-generic linux-image-3.2.0-39-generic linux-image-3.2.0-40-generic linux-image-3.2.0-43-generic linux-image-3.2.0-45-generic linux-image-3.2.0-48-generic linux-image-3.2.0-49-generic

The output of the script can then be easily used to remove the unnecessary kernel images with apt-get purge.

The script can be downloaded here: remove-old-kernels

And before anybody complains: I know it is not really the most elegant solution and I should have not written it using bash.

Process Migration coming to Fedora 19 (probably)

With the recent approved review of the package crtools in Fedora I have made a feature proposal for checkpoint/restore.

To test checkpoint/restore on Fedora you need to run the current development version of Fedora and install crtools using yum (yum install crtools). Until it is decided if it actually will be a Fedora 19 feature and the necessary changes in the Fedora kernel packages have been implemented it is necessary to install a kernel which is not in the repository. I have built a kernel in Fedora’s buildsystem which enables the following config options: CHECKPOINT_RESTORE, NAMESPACES, EXPERT.

A kernel with these changes enabled is available from koji as a scratch build: http://koji.fedoraproject.org/koji/taskinfo?taskID=4899525

After installing this kernel I am able to migrate a process from one Fedora system to another. For my test case I am migrating a UDP ping pong (udpp.c) program from one system to another while communicating with a third system.

udpp

udpp is running in server mode on 129.143.116.10 and on 134.108.34.90 udpp is started in client mode. After a short time I am migrating, with the help of crtools, the udpp client to 85.214.67.247. The following is part of the output on the udpp server:

-->

Received ping packet from 134.108.34.90:38374
Data: This is ping packet 6

Sending pong packet 6
<–
–>

Received ping packet from 134.108.34.90:38374
Data: This is ping packet 7

Sending pong packet 7
<–
–>

Received ping packet from 85.214.67.247:38374
Data: This is ping packet 8

Sending pong packet 8
<–
–>

Received ping packet from 85.214.67.247:38374
Data: This is ping packet 9

Sending pong packet 9
<–

So with only little changes to the kernel configuration it is possible to migrate a process by checkpointing and restoring a process with the help of crtools.