Category Archives: free software

Linux Plumbers Conference 2016

It is a bit late but I still wanted to share my presentations from this year’s Linux Plumbers Conference:

On my way back home I had to stay one night in Albuquerque and it looks like the hotel needs to upgrade its TV system. It is still running Fedora 10 which is EOL since 2009-12-18:

Still Fedora 10

Influence which PID will be the next

To restore a checkpointed process with CRIU the process ID (PID) has to be the same it was during checkpointing. CRIU uses /proc/sys/kernel/ns_last_pid to set the PID to one lower as the process to be restored just before fork()-ing into the new process.

The same interface (/proc/sys/kernel/ns_last_pid) can also be used from the command-line to influence which PID the kernel will use for the next process.

# cat /proc/sys/kernel/ns_last_pid
1626
# echo -n 9999 > /proc/sys/kernel/ns_last_pid
# cat /proc/sys/kernel/ns_last_pid
10000

Writing ‘9999’ (without a ‘new line’) to /proc/sys/kernel/ns_last_pid tells the kernel, that the next PID should be ‘10000’. This only works if between after writing to /proc/sys/kernel/ns_last_pid and forking the new process no other process has been created. So it is not possible to guarantee which PID the new process will get but it can be influenced.

There is also a posting which describes how to do the same with C: How to set PID using ns_last_pid

Combining pre-copy and post-copy migration

In my last post about CRIU in May 2016 I mentioned lazy memory transfer to decrease process downtime during migration. Since May 2016 Mike Rapoport’s patches for remote lazy process migration have been merged into CRIU‘s criu-dev branch as well as my patches to combine pre-copy and post-copy migration.

Using pre-copy (criu pre-dump) it has “always” been possible to dump the memory of a process using soft-dirty-tracking. criu pre-dump can be run multiple times and each time only the changed memory pages will be written to the checkpoint directory.

Depending on the processes to be migrated and how fast they are changing their memory, this can still lead to a situation where the final dump can be rather large which can mean a longer downtime during migration than desired. This is why we started to work on post-copy migration (also know as lazy migration). There are, however, situations where post-copy migration can also increase the process downtime during migration instead of decreasing it.

The latest changes regarding post-copy migration in the criu-dev branch offer the possibility to combine pre-copy and post-copy migration. The memory pages of the process are pre-dumped using soft-dirty-tracking and transferred to the destination while the process on the source machine keeps on running. Once the process is actually migrated to the destination system everything besides the memory pages is transferred to the destination system. Excluding the memory pages (as the remaining memory pages will be migrated lazily) usually only a few hundred kilobytes have to be transferred which reduces the process downtime during migration significantly.

Using criu with pre-copy and post-copy could look like this:

Source system:

# criu pre-dump -D /tmp/cp/1 -t PID
# rsync -a /tmp/cp destination:/tmp
# criu dump -D /tmp/cp/2 -t PID --port 27 --lazy-pages \
  --prev-images-dir ../1/ --track-mem

The first criu command dumps the memory of the process PID and resets the soft-dirty memory tracking. The initial dump is then transferred using rsync to the destination system. During that time the process PID keeps on running. The last criu command starts the lazy page mode which dumps everything besides memory pages which can be transferred lazily and waits for connections over the network on port 27. Only pages which have changed since the last pre-dump are considered for the lazy restore. At this point the process is no longer running and the process downtime starts.

Destination system:

# rsync -a source:/tmp/cp /tmp/
# criu lazy-pages --page-server --address source --port 27 \
  -D /tmp/cp/2 &
# criu restore --lazy-pages -D /tmp/cp/2

Once criu is waiting on port 27 on the source system the remaining checkpoint images can be transferred from the source system to the destination system (using rsync in this case). Now criu can be started in lazy-pages mode connecting to the page server on port 27 on the source system. This is the part we usually call the UFFD daemon. The last step is the actual restore (criu restore).

The following diagrams try to visualize what happens during the last step: criu restore.

step1

It all starts with criu restore (on the right). criu does its magic to restore the process and copies the memory pages from criu pre-dump to the process and marks lazy pages as being handled by userfaultfd. Once everything is restored criu jumps into the restored process and the restored process continues to run where it was when checkpointed. Once the process accesses a userfaultfd marked memory address the process will be paused until a memory page (hopefully the correct one) is copied to that address.

step2

The part that we call the UFFD daemon or criu lazy-pages listens on the userfault file descriptor for a message and as soon as a valid UFFD request arrives it requests that page from the source system via TCP where criu is still running in page-server mode. If the page-server finds that memory page it transfers the actual page back to the destination system to the UFFD daemon which injects the page into the kernel using the same userfault file descriptor it previously got the page request from. Now that the page which initially triggered the page-fault or in our case userfault is at its place the restored process continues to run until another missing page is accessed and the whole procedure starts again.

To be able to remove the UFFD daemon and the page-server at some point we currently push all unused pages into the restored process if there are no further userfaultfd requests for 5 seconds.

The whole procedure still has a lot of possibilities for optimization but now that we finally can combine pre-copy and post-copy memory migration we are a lot closer to decreasing process downtime during migration.

The next steps are to get support for pre-copy and post-copy into p.haul (Process Hauler) and into different container runtimes which already support migration via criu.

My other recently posted criu related articles:

HS100 – Wi-Fi Smart Plug

For my recently installed PXACB I was looking for a way to remotely power it on and off. I found the Wi-Fi Smart Plug “HS100” and a blog post that it can be controlled from the command-line.

The referenced script uses captured results from wireshark and just re-transmits these messages from a shell script. In one of the comments someone points out that this is XOR’d JSON and how it can be decoded. Instead of a shell script I re-implemented it in Python and I am now always using XOR to encode and decode the JSON messages without needing to include the encoded commands in my script. This makes it easier to read the script and to extend the script.

The protocol used is JSON which is XOR’d and then transmitted to the device. Same goes for the answers. The JSON string is XOR’d with the previous character of the JSON string and the value of the first XOR operation is 0xAB. Additionally each message is prefixed with ‘\x00\x00\x00\x23’.

The message to turn on the power looks like this:

{
 "system": {
  "set_relay_state": {
   "state": 1
  }
 }
}

To find more about which commands the device understands I used the information I got from: Why not root your Christmas gift?

I downloaded the firmware for the US model of my smart plug and used binwalk to analyze the content of the firmware. The firmware contains busybox based ramdisk which includes the smart plug relevant programs /usr/bin/shd and /usr/bin/shdTester and it seems at least following commands exist:

  • system
  • reset
  • get_sysinfo
  • set_test_mode
  • set_dev_alias
  • set_relay_state
  • check_new_config
  • download_firmware
  • get_download_state
  • flash_firmware
  • set_mac_addr
  • set_device_id
  • set_hw_id
  • test_check_uboot
  • get_dev_icon
  • set_dev_icon
  • set_led_off
  • set_dev_location

With the knowledge from the original shell script implementation and the results from binwalk I wrote the following script: https://lisas.de/~adrian/hs100.py

Using this script I can power the device behind the smart plug easily on and off:

$ ./hs100.py -H p-pxcab.example.com off
$ ./hs100.py -H p-pxcab.example.com state
Power OFF
$ ./hs100.py -H p-pxcab.example.com on
$ ./hs100.py -H p-pxcab.example.com state
Power ON

The only annoying thing about the smart plug is, that it tries to communicate with some cloud systems so that it could be controlled from anywhere. After starting the smart plug it makes a name lookup for devs.tplinkcloud.com and connects to port 50443. I can connect to that system with openssl s_client -connect devs.tplinkcloud.com:50443 but what the smart plug actually sends to that system I do not know. If I do not block the smart plug in the firewall I see a NTP request after that and then the communication seems to stop. Right now the smart plug is blocked and does no NTP requests but it still tries to reach devs.tplinkcloud.com:50443 once a minute.

PXCAB

A long time ago (2007 or 2008) I was developing firmware for Cell processor based systems. Most of the Slimline Open Firmware (SLOF) has been released and is also available in Fedora as firmware for QEMU: SLOF.

One of the systems we have been developing firmware for was a PCI Express card called PXCAB. The processor on this PCI Express card was not the original Cell processor but the newer PowerXCell 8i which has a much better double precision floating point performance. A few weeks ago I was able to get one of those PCI Express cards in a 1U chassis:

PXCAB

This chassis was designed to hold two PXCABs: one running in root complex mode and the other in endpoint mode. That way one card was the host system and the other the PCI express connected device. This single card is now running in root complex mode.

I can boot a kernel either via TFTP or from the flash. As writing the flash takes some time I am booting it right now via TFTP. Compiling the latest kernel from git for PPC64 is thanks to the available cross compiler (gcc-powerpc64-linux-gnu.x86_64) no problem: make CROSS_COMPILE=powerpc64-linux-gnu- ARCH=powerpc.

The more difficult part was to compile user space tools but fortunately I was able to compile it natively on a PPC64 system. With this minimal busybox based system I can boot the system and chroot into a Fedora 24 NFS mount.

I was trying to populate a directory with a minimal PPC64 based Fedora 24 system with following command:

dnf --setopt arch=ppc64 --installroot $PWD/ppc64 install dnf --releasever 24

Unfortunately that does not work as there currently seems to be no way to tell dnf to install the packages for another architecture. I was able to download a few RPMs and directly install them with rpm using the option --ignorearch. In the end I also installed the data for the chroot on my PPC64 system as that was faster and easier.

Now I can boot the PXCAB via TFTP into the busybox based ramdisk and from there I can chroot in to the NFS mounted Fedora 24 system.

The system has one CPU with two threads and 4GB of RAM. In addition to the actual RAM there is also 256MB of memory which can be accessed as a block device using the axonram driver. My busybox based ramdisk is copied to that ramdisk and thus freeing some more actual RAM:

# df -h
Filesystem         Size    Used Available Use% Mounted on
/dev/axonram0    247.9M   15.6M    219.5M   7% /

System information from the firmware:

SYSTEM INFORMATION
 Processor  = PowerXCell DD1.0 @ 2800 MHz
 I/O Bridge = Cell BE companion chip DD3.0
 Timebase   = 14318 kHz (external)
 Config     = SMP disabled
 SMP Size   = 1 (2 threads)
 Boot-Date  = 2016-07-21 19:37
 Memory     = 4096MB (CPU0: 4096MB)

Lazy Process Migration

Process Migration

Using CRIU it is possible to checkpoint/save/dump the state of a process into a set of files which can then be used to restore/restart the process at a later point in time. If the files from the checkpoint operation are transferred from one system to another and then used to restore the process, this is probably the simplest form of process migration.

Source system:

  • criu dump -D /checkpoint/destination -t PID
  • rsync -a /checkpoint/destination destination.system:/checkpoint/destination

Destination system:

  • criu restore -D /checkpoint/destination

For large processes the migration duration can be rather long. For a process using 24GB this can lead to migration duration longer than 280 seconds. The limiting factor in most cases is the interconnect between the systems involved in the process migration.

Optimization: Pre-Copy

One existing solution to decrease process downtime during migration is pre-copy. In one or multiple runs the memory of the process is copied from the source to the destination system. With every run only memory pages which have change since the last run have to be transferred. This can lead to situations where the process downtime during migration can be dramatically decreased.

This depends on the type of application which is migrated and especially how often/fast the memory content is changed. In extreme cases it was possible to decrease process downtime during migration for a 24GB process from 280 seconds to 8 seconds with the help of pre-copy.

This approach is basically the same if migrating single processes (or process groups) or virtual machines.

It Always Depends On…

Unfortunately pre-copy optimization can also lead to situations where the so called optimized case with pre-copy can require more time than the unoptimized case:

In the example above a process has been migrated during three stages of its lifetime and there are situations (state: Calculation) where pre-copy has enormous advantages (14 seconds with pre-copy and 51 seconds without pre-copy) but there are also situations (state: Initialization) where the pre-copy optimization increases the process downtime during migration (40 seconds with pre-copy and 27 seconds without pre-copy). It depends on the memory change rate.

Optimization: Post-Copy

Another approach to reduce the process downtime during migration is post-copy. The required memory pages are not dumped and transferred before restoring the process but on demand. Each time a missing memory page is accessed the migrated process is halted until the required memory pages has been transferred from the source system to the destination system:

Thanks to userfaultfd this approach (or optimization) can be now integrated into CRIU. With the help of userfaultfd it is possible to mark memory pages to be handled by userfaultfd. If such a memory page is accessed, the process is halted until the requested page is provided. The listener for the userfaultfd requests is running in user-space and listening on a file descriptor. The same approach has already been implemented for QEMU.

Enough Theory

With all the background information on why and how the initial code to restore processes with userfaultfd support has been merged into the CRIU development branch: criu-dev. This initial implementation of lazy-pages support does not yet support lazy process migration between two hosts, but with the upstream merged patches it is at least possible to checkpoint a process and to restore the process using userfaultfd. A lazy restore consists of two parts. The usual ‘criu restore‘ part and an additional, what we call uffd daemon, ‘criu lazy-pages‘ part. To better demonstrate the advantages of a lazy restore there are patches to enhance crit (CRiu Image Tool) to remove pages which can be restored with userfaultfd from a checkpoint directory. Using a test case which allocates about 200MB of memory (and which writes one byte in each page over and over) requires after being dumped about 200MB. Using the mentioned crit enhancement make-lazy reduces the size of the checkpoint down to 116KB:

$ crit make-lazy /tmp/checkpoint/ /tmp/lazy-checkpoint
$ du -hs /tmp/checkpoint/ /tmp/lazy-checkpoint
     201M       /tmp/checkpoint
     116K       /tmp/lazy-checkpoint

With this the data which actually has to be transferred during process downtime is drastically reduced and the required memory pages are inserted in the restored process on demand using userfaultfd. Restoring the checkpointed process using lazy-restore would look something like this:

First the uffd daemon:

$ criu lazy-pages -D /tmp/checkpoint \
--address /tmp/userfault.socket

And then the actual restore:

$ criu restore -D /tmp/lazy-checkpoint \
--lazy-pages --address /tmp/userfault.socket

The socket specified with --address is used to exchange information about the restored process required by the uffd daemon. Once criu restore has done all its magic to restore the process except restoring the lazy memory pages, the process to be restored is actually started and runs until the first userfaultfd handled memory page is accessed. At that point the process hangs and the uffd daemon gets a message to provide the required memory pages. Once the uffd daemon provides the requested memory page, the restored process continues to run until the next page is requested. As potentially not all memory pages are requested, as they might not get accessed for some time, the uffd daemon starts to transfer unrequested memory pages into the restored process so that the uffd daemon can shut down after a certain time.

Checkpoint and almost Restart in Open MPI

Now that checkpoint/restart with CRIU is possible since Fedora 19 I started adding CRIU support to Open MPI. With my commit 30772 it is now possible to checkpoint a process running under Open MPI. The restart functionality is not yet implemented but should be soon available. I have a test case (orte-test) which prints its PID and sleeps one second in a loop which I start under orterun like this:

/path/to/orterun --mca ft_cr_enabled 1 --mca opal_cr_use_thread 1 --mca oob tcp --mca crs_criu_verbose 30 --np 1 orte-test

The options have following meaning:

  • –mca ft_cr_enabled 1
    • ft stands for fault tolerance
    • cr stands for checkpoint/restart
    • this option is to enable the checkpoint/restart functionality
  • –mca opal_cr_use_thread 1: use an additional thread to control checkpoint/restart operations
  • –mca oob tcp: use TCP instead of unix domain sockets (the socket code needs some additional changes for C/R to work)
  • –mca crs_criu_verbose 30: print all CRIU debug messages
  • –np 1: spawn one test case

The output of the test case looks like this:


[dcbz:12563] crs:criu: open()
[dcbz:12563] crs:criu: open: priority = 10
[dcbz:12563] crs:criu: open: verbosity = 30
[dcbz:12563] crs:criu: open: log_file = criu.log
[dcbz:12563] crs:criu: open: log_level = 0
[dcbz:12563] crs:criu: open: tcp_established = 1
[dcbz:12563] crs:criu: open: shell_job = 1
[dcbz:12563] crs:criu: open: ext_unix_sk = 1
[dcbz:12563] crs:criu: open: leave_running = 1
[dcbz:12563] crs:criu: component_query()
[dcbz:12563] crs:criu: module_init()
[dcbz:12563] crs:criu: opal_crs_criu_prelaunch
[dcbz:12565] crs:criu: open()
[dcbz:12565] crs:criu: open: priority = 10
[dcbz:12565] crs:criu: open: verbosity = 30
[dcbz:12565] crs:criu: open: log_file = criu.log
[dcbz:12565] crs:criu: open: log_level = 0
[dcbz:12565] crs:criu: open: tcp_established = 1
[dcbz:12565] crs:criu: open: shell_job = 1
[dcbz:12565] crs:criu: open: ext_unix_sk = 1
[dcbz:12565] crs:criu: open: leave_running = 1
[dcbz:12565] crs:criu: component_query()
[dcbz:12565] crs:criu: module_init()
[dcbz:12565] crs:criu: opal_crs_criu_reg_thread
Process 12565
Process 12565
Process 12565

To start the checkpoint operation the Open MPI tool orte-checkpoint is used:

/path/to/orte-checkpoint -V 10 `pidof orterun`

which outputs the following:


[dcbz:12570] orte_checkpoint: Checkpointing...
[dcbz:12570] PID 12563
[dcbz:12570] Connected to Mpirun [[56676,0],0]
[dcbz:12570] orte_checkpoint: notify_hnp: Contact Head Node Process PID 12563
[dcbz:12570] orte_checkpoint: notify_hnp: Requested a checkpoint of jobid [INVALID]
[dcbz:12570] orte_checkpoint: hnp_receiver: Receive a command message.
[dcbz:12570] orte_checkpoint: hnp_receiver: Status Update.
[dcbz:12570] [ 0.00 / 0.08] Requested - ...
[dcbz:12570] orte_checkpoint: hnp_receiver: Receive a command message.
[dcbz:12570] orte_checkpoint: hnp_receiver: Status Update.
[dcbz:12570] [ 0.00 / 0.08] Pending - ...
[dcbz:12570] orte_checkpoint: hnp_receiver: Receive a command message.
[dcbz:12570] orte_checkpoint: hnp_receiver: Status Update.
[dcbz:12570] [ 0.00 / 0.08] Running - ...
[dcbz:12570] orte_checkpoint: hnp_receiver: Receive a command message.
[dcbz:12570] orte_checkpoint: hnp_receiver: Status Update.
[dcbz:12570] [ 0.06 / 0.14] Locally Finished - ...
[dcbz:12570] orte_checkpoint: hnp_receiver: Receive a command message.
[dcbz:12570] orte_checkpoint: hnp_receiver: Status Update.
[dcbz:12570] [ 0.00 / 0.14] Checkpoint Established - ompi_global_snapshot_12563.ckpt
[dcbz:12570] orte_checkpoint: hnp_receiver: Receive a command message.
[dcbz:12570] orte_checkpoint: hnp_receiver: Status Update.
[dcbz:12570] [ 0.00 / 0.14] Continuing/Recovered - ompi_global_snapshot_12563.ckpt
Snapshot Ref.: 0 ompi_global_snapshot_12563.ckpt

orte-checkpoint tries to connect to the previously started orterun process and requests that a checkpoint should be taken. orterun outputs the following after receiving the checkpoint request:


[dcbz:12565] crs:criu: checkpoint(12565, ---)
[dcbz:12565] crs:criu: criu_init_opts() returned 0
[dcbz:12565] crs:criu: opening snapshot directory /home/adrian/ompi_global_snapshot_12563.ckpt/0/opal_snapshot_0.ckpt
[dcbz:12563] 12563: Checkpoint established for process [56676,0].
[dcbz:12563] 12563: Successfully restarted process [56676,0].
Process 12565

At this point the checkpoint has been written to disk and the process continues (printing its PID).

For a complete checkpoint/restart functionality I still have to implement the restart functionality in Open MPI and I also have to take care of the unix domain sockets (shutting them down for the checkpointing).

This requires the latest criu package (criu-1.1-4) which includes headers to build Open MPI against CRIU as well as the CRIU service.

Dynamic DNS

For the last ten years I wanted to set up my own dynamic DNS service but was never motivated enough. Recently enough motivation was provided and using the scripts from http://www.fischglas.de/software/dyn/ made it really easy to set up a dynamic DNS service using bind. Following changes were necessary to the named.conf file:

zone "dyn.domain" in {
        type master;
        file "db.dyn.domain";
        allow-update {
                key host.domain.;
        };
};

Whenever the IP address of my host changes I am loading a URL with my hostname and password encoded. The script behind the URL checks if my hostname and password is correct and updates the zone file using nsupdate with a TTL of 120 seconds.

The script uses a simple configuration file (/etc/dyn/dyn.cfg) with the following content:

dns.key.name:host.domain.
dns.key:yyeofEWfgvdfgdfgerX==
authfile:/etc/dyn/secrets
dns.host:host.domain
debug:0

Remove Old Kernels

Mainly using Fedora, I am accustomed that old kernel images are automatically uninstalled after a certain number of kernel images have been installed using yum. The default is to have three kernel images installed and so far this has always worked.

I am also maintaining a large number of Ubuntu VMs and every now and then we have the problem that the filesystem is full, because too many kernel images are installed. I have searched for some time but there seems to be no automatic kernel image removal in apt-get. There is one command which is often recommended which is something like:

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;
s/^[^ ]*[^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge
[1]

This works, but only if you are already running the latest kernel and therefore I have adapted it a little for our needs. Instead of removing all kernel images except the running kernel image I remove all kernel images except the running and the newest kernel image. No real big difference but important for our setup where we do not reboot all VMs with every kernel image update.

Running the script gives me following output
# remove-old-kernels

linux-image-3.2.0-23-generic linux-image-3.2.0-36-generic linux-image-3.2.0-37-generic linux-image-3.2.0-38-generic linux-image-3.2.0-39-generic linux-image-3.2.0-40-generic linux-image-3.2.0-43-generic linux-image-3.2.0-45-generic linux-image-3.2.0-48-generic linux-image-3.2.0-49-generic

The output of the script can then be easily used to remove the unnecessary kernel images with apt-get purge.

The script can be downloaded here: remove-old-kernels

And before anybody complains: I know it is not really the most elegant solution and I should have not written it using bash.

A New Home

After having received my Raspberry Pi in November, I am finally using it. I have connected it to my television using raspbmc.Using XBMC Remote I can control it without the need for a mouse, keyboard or lirc based remote control and so far it works pretty good. Following are a few pictures with the new case I bought a few days ago:

Pi

Pi

Pi