Our mirror server has been generating download maps for almost 10 years (since August 2009). This is done by going through all our download log files (HTTP, FTP, RSYNC) and using GeoIP and the Matplotlib Basemap Toolkit to draw maps from where our mirror server is being accessed.

I have taken the output from almost ten years and created the following animations. The first animation shows clients accessing all mirrored content:

As the mirror server is running Fedora it is updated once a year which might result in an updated version of Basemap once a year. The update usually happens in December or January which sometimes can be seen in the animation when the output changes. Updating to Fedora 27 (December 2017) resulted in a Basemap version which started to draw different results and the last update to Fedora 29 (December 2018) can also be seen as switching to Python 3 removed most of the clients from the map (only visible in the last second of the animation). It seems some of the calculations are giving different results in Python 3.

In addition to the map showing the accesses for all mirrored data, there is also an animation for clients accessing files from our Fedora mirror:

The interesting thing about only looking at clients accessing Fedora files is that it can be seen that most accesses are actually from Europe. This seems to indicate that Fedora's mirroring system partially succeeds in directing clients to close by mirrors. Looking at the location of clients accessing our EPEL mirror it seems to work even better. This is probably related to the much larger number of existing EPEL mirrors:

Another interesting effect of upgrading once a year can be seen around 6:42 in the EPEL animation. After upgrading to Fedora 25 the generated maps where upside down for a few days until I was able to fix it.

I have updated the scripts which are using the mirrored project status information in our database to display even more information about what is going on on our mirror server. In addition to the overall traffic of the last 14 days, 12 months and all the years since we started to collect this data, the overall traffic is now broken down to transferred HTTP, FTP, RSYNC and other data (blue=other, red=http, green=rsync, yellow=ftp). The most traffic is generated by HTTP, followed by RSYNC and last (but not surprising) is FTP.

In addition to breakdown by traffic type I added an overview of the mirror size (in bytes and number files) at the bottom of the status page of each mirrored project. Looking at the status page of our apache mirror it is now possible to see the growth of the mirror since 2005. It started with 7GB in 2005 and has now reached almost 50GB at the end of 2012.

Adding the new functionality to the PHP scripts I had to change code I have written many years ago and unfortunately I must confess that this is embarrassingly bad code and it already hurts looking at it. Adding new functionality to it was even worse, but despite my urge to rewrite it I just added the new functionality which makes the code now even more unreadable.

Fedora. Nobody expected anything else (of course).

The first one and a half days since the release of Fedora 9 we are maxing out our bandwidth again. Today we already pushed more than 5.5TB and it looks like we will get close to transmitting 7TB on one day. This is much more than during the last Ubuntu release.

With the help of munin I can again provide a nice bandwidth graph:

The small dent, just after the start of the release, is due to the fact that I had to restart apache because of our cache drive. We are using a fast hard disk to reduce the load on our main RAID as cache, but it seems that it somehow cannot handle over a thousand simultaneous accesses and that is why I disabled that cache drive (which should have improved the situation and not worsened it). I can also prove that the Fedora release is the reason for all the traffic:

We always thought our mirror server is connected with 2 GBit/s (two times an e1000 card using bonding mode=6), but the current Ubuntu release proved that somewhere along the way to the Internet there must still be something that limits us to 1 GBit/s. The following diagram shows this pretty clearly:

Now we only need to find out where and if it is something that we can fix or if we need help from our provider.

Maybe we can fix it before the release of Fedora 9 so that we finally can transmit more than 1 GBit/s.