{"id":7,"date":"2016-10-14T15:19:57","date_gmt":"2017-03-30T16:16:25","guid":{"rendered":"https:\/\/lisas.de\/~adrian\/?p=1253"},"modified":"2018-10-10T15:41:01","modified_gmt":"2018-10-10T13:41:01","slug":"combining-pre-copy-and-post-copy-migration","status":"publish","type":"post","link":"https:\/\/lisas.de\/luges\/index.php\/2016\/10\/14\/combining-pre-copy-and-post-copy-migration\/","title":{"rendered":"Combining pre-copy and post-copy migration"},"content":{"rendered":"

In my last post<\/a> about CRIU<\/em> in May 2016 I mentioned lazy memory transfer to decrease process downtime during migration. Since May 2016 Mike Rapoport’s patches for remote lazy process migration have been merged into CRIU<\/em>‘s criu-dev<\/a> branch as well as my patches to combine pre-copy<\/em> and post-copy<\/em> migration.<\/p>\n

Using pre-copy (criu pre-dump<\/em>) it has “always” been possible to dump the memory of a process using soft-dirty-tracking<\/a>. criu pre-dump<\/em> can be run multiple times and each time only the changed memory pages will be written to the checkpoint directory.<\/p>\n

Depending on the processes to be migrated and how fast they are changing their memory, this can still lead to a situation where the final dump can be rather large which can mean a longer downtime during migration than desired. This is why we started to work on post-copy<\/em> migration (also know as lazy<\/em> migration). There are, however, situations where post-copy<\/em> migration can also increase the process downtime during migration instead of decreasing it.<\/p>\n

The latest changes regarding post-copy<\/em> migration in the criu-dev<\/a> branch offer the possibility to combine pre-copy<\/em> and post-copy<\/em> migration. The memory pages of the process are pre-dumped using soft-dirty-tracking<\/a> and transferred to the destination while the process on the source machine keeps on running. Once the process is actually migrated to the destination system everything besides the memory pages is transferred to the destination system. Excluding the memory pages (as the remaining memory pages will be migrated lazily<\/em>) usually only a few hundred kilobytes have to be transferred which reduces the process downtime during migration significantly.<\/p>\n

Using criu<\/em> with pre-copy<\/em> and post-copy<\/em> could look like this:<\/p>\n

Source system:<\/p>\n

# criu pre-dump -D \/tmp\/cp\/1 -t PID\n# rsync -a \/tmp\/cp destination:\/tmp\n# criu dump -D \/tmp\/cp\/2 -t PID --port 27 --lazy-pages \n  --prev-images-dir ..\/1\/ --track-mem<\/pre>\n
The first criu<\/em> command dumps the memory of the process PID<\/em> and resets the soft-dirty memory tracking. The initial dump is then transferred using rsync<\/em> to the destination system. During that time the process PID<\/em> keeps on running. The last criu<\/em> command starts the lazy page<\/em> mode which dumps everything besides memory pages which can be transferred lazily and waits for connections over the network on port 27. Only pages which have changed since the last pre-dump<\/em> are considered for the lazy restore. At this point the process is no longer running and the process downtime starts.<\/p>\n
Destination system:<\/p>\n
# rsync -a source:\/tmp\/cp \/tmp\/\n# criu lazy-pages --page-server --address source --port 27 \n  -D \/tmp\/cp\/2 &\n# criu restore --lazy-pages -D \/tmp\/cp\/2<\/pre>\n
Once criu<\/em> is waiting on port 27 on the source system the remaining checkpoint images can be transferred from the source system to the destination system (using rsync<\/em> in this case). Now criu<\/em> can be started in lazy-pages<\/em> mode connecting to the page server on port 27 on the source system. This is the part we usually call the UFFD daemon. The last step is the actual restore (criu restore<\/em>).<\/p>\n
The following diagrams try to visualize what happens during the last step: criu restore<\/em>.<\/p>\n
\"step1\"<\/a><\/p>\n
It all starts with criu restore<\/em> (on the right). criu<\/em> does its magic to restore the process and copies the memory pages from criu pre-dump<\/em> to the process and marks lazy pages<\/em> as being handled by userfaultfd<\/em>. Once everything is restored criu<\/em> jumps into the restored process and the restored process continues to run where it was when checkpointed. Once the process accesses a userfaultfd<\/em> marked memory address the process will be paused until a memory page (hopefully the correct one) is copied to that address.<\/p>\n
\"step2\"<\/a><\/p>\n
The part that we call the UFFD daemon<\/em> or criu lazy-pages<\/em> listens on the userfault file descriptor for a message and as soon as a valid UFFD<\/em> request arrives it requests that page from the source system via TCP where criu<\/em> is still running in page-server<\/em> mode. If the page-server<\/em> finds that memory page it transfers the actual page back to the destination system to the UFFD daemon<\/em> which injects the page into the kernel using the same userfault file descriptor it previously got the page request from. Now that the page which initially triggered the page-fault or in our case userfault<\/em> is at its place the restored process continues to run until another missing page is accessed and the whole procedure starts again.<\/p>\n
To be able to remove the UFFD daemon<\/em> and the page-server<\/em> at some point we currently push all unused pages into the restored process if there are no further userfaultfd<\/em> requests for 5 seconds.<\/p>\n
The whole procedure still has a lot of possibilities for optimization but now that we finally can combine pre-copy<\/em> and post-copy<\/em> memory migration we are a lot closer to decreasing process downtime during migration.<\/p>\n
The next steps are to get support for pre-copy<\/em> and post-copy<\/em> into p.haul<\/a> (Process Hauler) and into different container runtimes which already support migration via criu<\/em>.<\/p>\n
My other recently posted criu<\/em> related articles:<\/p>\n