This month’s come together will be held on 2018-01-03, 8pm at our regular pub (Trödler).
After a few initial discussions with Christian I started with pre-copy migration. Container migration in LXD is based on CRIU, just as in runc and CRIU’s pre-copy migration support is based on dirty page tracking support of Linux: SOFT-DIRTY PTEs.
As LXD uses LXC for the actual container checkpointing and restoring I was curious if there was already pre-copy migration support in LXC. After figuring out the right command-line parameters it almost worked thanks to the great checkpoint and restore support implemented by Tycho some time ago.
Now that I knew that it works in LXC I focused on getting pre-copy migration support into LXD. LXD supports container live migration using the move command: lxc move <container> <remote>:<container>
This move command, however, did not use any optimization yet. It basically did:
The downtime for the container in this scenario is between step 2 and step
5 and depends on the used memory of the processes inside the container. The goal of pre-copy migration is to dump the memory of the container and transfer it to the remote destination while the container keeps on running and doing a final dump with only the memory pages that changed since the last pre-dump (more about process migration optimization theories).
Back to LXD: At the end of the day I had a very rough (and very hardcoded) first pre-copy migration implementation ready and I kept working on it until it was ready to be submitted upstream. The pull request has already been merged upstream and now LXD supports pre-copy migration.
As not all architecture/kernel/criu combinations support pre-copy migration it has to be turned on manually right now, but we already discussed adding pre-copy support detection to LXC. To tell LXD to use pre-copy migration, the parameter ‘migration.incremental.memory’ needs to be set to ‘true’. Once that is done and if LXD is instructed to migrate a container the following will happen:
So instead of doing a single checkpoint and transferring it, there are now multiple pre-copy checkpoints and the container keeps on running during those transfers. The container is only suspended during the last delta checkpoint and the transfer of the last delta checkpoint. In many cases this reduces the container downtime during migration, but there is the possibility that pre-copy migration also increases the container downtime during migration. This depends (as always) on the workload.
To control how many pre-copy iterations LXD does there are two additional variables:
The first variable (iterations) is used to tell LXD how many pre-copy iterations it should do before doing the final dump and the second variable (goal) is used to tell LXD the percentage of pre-copied memory pages that should not change between pre-copy iterations before doing the final dump.
So LXD, in the default configuration, does either 10 pre-copy iterations before doing the final migration or the final migration is triggered when at least 70% of the memory pages have been transferred by the last pre-copy iteration.
Now that this pull request is merged and if pre-copy migration is enabled a lxc move <container> <remote>:<container> should live migrate the container with a reduced downtime.
I want to thank Christian for the collaboration on getting CRIU’s pre-copy support into LXD, Tycho for his work preparing LXC and LXD to support migration so nicely and the developers of p.haul for the ideas how to implement pre-copy container migration. Next step: lazy migration.
For almost two years Mike Rapoport and I have been working on lazy process migration. Lazy process migration (or post-copy migration) is a technique to decrease the process or container downtime during the live migration. I described the basic functionality in the following previous articles:
Those articles are not 100% correct anymore as we changed some of the parameters during the last two years, but the concepts stayed the same.
Mike and I started about two years ago to work on it and the latest CRIU release (3.5) includes the possibility to use lazy migration. Now that the post-copy migration feature has been merged from the criu-dev branch to the master branch it is part of the normal CRIU releases.
With CRIU’s 3.5 release lazy migration can be used on any kernel which supports userfaultfd. I already updated the CRIU packages in Fedora to 3.5 so that lazy process migration can be used just by installing the latest CRIU packages with dnf (still in the testing repository right now).
More information about container live migration in our upcoming Open Source Summit Europe talk: Container Migration Around The World.
Another interesting change about CRIU is that it started as x86_64 only and now it is also available on aarch64, ppc64le and s390x. The support to run on s390x has just been added with the previous 3.4 release and starting with Fedora 27 the necessary kernel configuration options are also active on s390x in addition to the other supported architectures.
As I’m currently switching phones I had to revisit the issue of how to get sshd running on a pristine LineageOS install. I decided to collect the steps here as the how-to formerly available on the CM wiki has vanished together with CM itself. Note that some steps are not incredibly detailed and you really should be aware of the security implications before going ahead with this.
While LineageOS includes all necessary software, the configuration of sshd must be completed manually:
adb push ~/.ssh/id_rsa.pub /data/ssh/authorized_keys
Subsystem sftp internal-sftp
sed 's#/system/etc/ssh#/data/ssh#' /system/bin/start-ssh
chmod 755 /data/local/userinit.d/99sshd
chmod 600 /data/ssh/authorized_keys
chown shell /data/ssh/authorized_keys
chmod 644 /data/ssh/sshd_config
Now you should be able to run sshd manually by executing
If so you can log on as user shell to the device using your ssh key. See my previous post to find out how you can make sure sshd is started whenever the device is booted.
Added IPv6 reverse record for 2001:7c0:700::10 – rhlx01.hs-esslingen.de
Added AAAA record for rhlx01.hs-esslingen.de – 2001:7c0:700::10
Enabled IPv6 address 2001:7c0:700::10
It is a bit late but I still wanted to share my presentations from this year’s Linux Plumbers Conference:
On my way back home I had to stay one night in Albuquerque and it looks like the hotel needs to upgrade its TV system. It is still running Fedora 10 which is EOL since 2009-12-18: