After many years the whole RPM Fusion repository has grown to over
320GB. There have been occasional requests to move the unsupported
releases to an archive, just like Fedora handles its mirror setup, but
until last week this did not happen.

As of now we have moved all unsupported releases (EL-5, Fedora 8 – 25)
to our archive (http://archive.rpmfusion.org/) and clients are now
being redirected to the new archive system. The archive consists of
260GB which means we can reduce the size mirrors need to carry by more
than 75%.

From a first look at the archive logs the amount of data requested by
all clients for the archived releases is only about 30GB per day. Those
30GB are downloaded by over 350000 HTTP requests and over 98% of those
requests are downloading the repository metdata only (repomd.xml,
*filelist*, *primary*, *comps*).

I will be giving two talks about OpenHPC in the next weeks. The first
talk will be at DevConf.cz 2018: OpenHPC
Introduction

The other talk will be at the CentOS Dojo in
Brussels
.

I hope I will be able to demonstrate my two node HPC system based on
Raspberry
Pis

and it definitely will be about OpenHPC’s building blocks:

OpenHPC Building
Blocks

And the results:

OpenHPC Building BlocksOpenHPC Building
BlocksOpenHPC Building
BlocksOpenHPC Building
Blocks

Come to one of my talks and you will able to build your OpenHPC engineer
from the available building blocks.

After having worked on optimizing live container migration based on
runc (pre-copy migration and
post-copy migration)
I tried to optimize container migration in
LXD.

After a few initial discussions with
Christian I started with pre-copy
migration. Container migration in LXD is based on
CRIU, just as in runc and CRIU’s pre-copy
migration support is based on dirty page tracking support of Linux:
SOFT-DIRTY
PTEs
.

As LXD uses LXC for the actual
container checkpointing and restoring I was curious if there was already
pre-copy migration support in LXC. After figuring out the right
command-line parameters it almost worked thanks to the great checkpoint
and restore support implemented by Tycho some time
ago.

Now that I knew that it works in LXC I focused on getting pre-copy
migration support into LXD. LXD supports container live migration using
the move command: lxc move <container> <remote>:<container>
This move command, however, did not use any optimization yet. It
basically did:

  1. Initial sync of the filesystem
  2. Checkpoint container using CRIU
  3. Transfer container checkpoint
  4. Final sync of the filesystem
  5. Restart container on the remote system

The downtime for the container in this scenario is between step 2 and
step
5 and depends on the used memory of the processes inside the container.
The goal of pre-copy migration is to dump the memory of the container
and transfer it to the remote destination while the container keeps on
running and doing a final dump with only the memory pages that changed
since the last pre-dump (more about process migration optimization
theories
).

Back to LXD: At the end of the day I had a very rough (and very
hardcoded) first pre-copy migration implementation ready and I kept
working on it until it was ready to be submitted upstream. The pull
request
has already been merged
upstream and now LXD supports pre-copy migration.

As not all architecture/kernel/criu combinations support pre-copy
migration it has to be turned on manually right now, but we already
discussed adding pre-copy support detection to LXC. To tell LXD to use
pre-copy migration, the parameter ‘migration.incremental.memory’ needs
to be set to ‘true’. Once that is done and if LXD is instructed to
migrate a container the following will happen:

  • Initial sync of the filesystem
  • Start pre-copy checkpointing loop using CRIU
    • Check if maximum number pre-copy iterations has been reached
    • Check if threshold of unchanged memory pages has been reached
    • Transfer container checkpoint
    • Continue pre-copy checkpointing loop if neither of those
      conditions is true
  • Final container delta checkpoint using CRIU
  • Transfer final delta checkpoint
  • Final sync of the filesystem
  • Restart container on the remote system

So instead of doing a single checkpoint and transferring it, there are
now multiple pre-copy checkpoints and the container keeps on running
during those transfers. The container is only suspended during the last
delta checkpoint and the transfer of the last delta checkpoint. In many
cases this reduces the container downtime during migration, but there is
the possibility that pre-copy migration also increases the container
downtime during migration. This depends (as always) on the workload.

To control how many pre-copy iterations LXD does there are two
additional variables:

  1. migration.incremental.memory.iterations (defaults to 10)
  2. migration.incremental.memory.goal (defaults to 70%)

The first variable (iterations) is used to tell LXD how many pre-copy
iterations it should do before doing the final dump and the second
variable (goal) is used to tell LXD the percentage of pre-copied
memory pages that should not change between pre-copy iterations before
doing the final dump.

So LXD, in the default configuration, does either 10 pre-copy iterations
before doing the final migration or the final migration is triggered
when at least 70% of the memory pages have been transferred by the last
pre-copy iteration.

Now that this pull request is merged and if pre-copy migration is
enabled a lxc move <container> <remote>:<container> should live
migrate the container with a reduced downtime.

I want to thank Christian for the collaboration on getting CRIU’s
pre-copy support into LXD, Tycho for his work preparing LXC and LXD to
support migration so nicely and the developers of
p.haul for the ideas how
to implement pre-copy container migration. Next step: lazy
migration
.

For almost two years Mike Rapoport and I have been working on lazy
process migration. Lazy process migration (or post-copy migration) is a
technique to decrease the process or container downtime during the live
migration. I described the basic functionality in the following previous
articles:

Those articles are not 100% correct anymore as we changed some of the
parameters during the last two years, but the concepts stayed the same.

Mike and I started about two years ago to work on it and the latest
CRIU release
(3.5)
includes the possibility to
use lazy migration. Now that the post-copy migration feature has been
merged from the criu-dev branch to the master branch it is part of
the normal CRIU releases.

With CRIU’s 3.5 release lazy migration can be used on any kernel which
supports
userfaultfd.
I already updated the CRIU packages in Fedora to 3.5 so that lazy
process migration can be used just by installing the latest CRIU
packages with dnf (still in the testing repository right now).

More information about container live migration in our upcoming Open
Source Summit Europe talk: Container Migration Around The
World
.

My pull request to
support lazy migration in
runC was also recently
merged, so that it is now possible to migrate containers using pre-copy
migration and post-copy migration. It can also be combined.

Another interesting change about CRIU is that it started as x86_64
only and now it is also available on aarch64, ppc64le and s390x.
The support to run on s390x has just been added with the previous 3.4
release and starting with Fedora 27 the necessary kernel configuration
options are also active on s390x in addition to the other supported
architectures.

As I’m currently switching phones I had to revisit the issue of how to get sshd running on a pristine LineageOS install. I decided to collect the steps here as the how-to formerly available on the CM wiki has vanished together with CM itself. Note that some  steps are not incredibly detailed and you really should be aware of the security implications before going ahead with this.

Prerequisites

Configuring SSHD

While LineageOS includes all necessary software, the configuration of sshd must be completed manually:

  1. Connect the device via USB
  2. Run adb with root privileges:
    adb root
  3. Upload your public ssh key to the device:
    adb push ~/.ssh/id_rsa.pub /data/ssh/authorized_keys
  4. Now, open a root shell and switch to bash to get vim to behave nicely on the device through adb:
    adb shell
    bash
  5. Use cat or vim to make the following fragment the contents of /data/ssh/sshd_config:

    AuthorizedKeysFile /data/ssh/authorized_keys
    ChallengeResponseAuthentication no
    PasswordAuthentication no
    PermitRootLogin no
    Subsystem sftp internal-sftp
    pidfile /data/ssh/sshd.pid

  6. Place a modified version of the start script in the userinit.d directory:
    mkdir /data/local/userinit.d
    sed 's#/system/etc/ssh#/data/ssh#' /system/bin/start-ssh
          > /data/local/userinit.d/99sshd
  7. Now correct the file privileges:
    chmod 755 /data/local/userinit.d/99sshd
    chmod 600 /data/ssh/authorized_keys
    chown shell /data/ssh/authorized_keys
    chmod 644 /data/ssh/sshd_config

Running SSHD

Now you should be able to run sshd manually by executing
/data/local/userinit.d/99sshd
If so you can log on as user shell to the device using your ssh key. See my previous post to find out how you can make sure sshd is started whenever the device is booted.