Over the years we have added more and more information about the setup of our mirror server, the changes, the status and information about some statistics. The one thing always missing, however, was the information about which mirrored project is responsible for which amount of traffic.
We had webalizer running for a few years but it never felt right. The information was just not in a format which gave us all the information we wanted.
But now we finally have a traffic breakdown by mirrored project. And all thanks to Alex and his godlike Python skills (compared to mine).
One of the strange things about apache is that it writes the amount of data requested to the logfile and not the data actually transmitted. We are now collecting both values but the difference is huge. Yesterday there were 563.28GB actually transmitted but 561.03TB were requested. To get both values from apache’s logfile I changed following line:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
to:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" %I %O \"%{User-Agent}i\"" combined
The current implementation reads our main apache logfiles, the one from the kernel.org mirror and the logfiles from vsftpd.
And Fedora is always the winner (during the last three days the tool is now running).
Yesterday’s result
The last few days there are so many connections to our mirror server from China that I started to block certain subnets. There are usually around 10 clients connecting via HTTP and each is opening over 50 connections to our server. They are downloading mainly ISO images and other big files. I can see that each client is starting to download lots of different things. From Fedora 3 to Fedora 7 ISO images, Ubuntu ISO images, openSUSE ISO images and other old and large files.
I started to block individual IP addresses but there are just too many so that I started to block whole subnets. I am using the following command to get an overview about which clients are opening many connections at once:
lynx -dump -width=2000 http://localhost/server-status | awk -F\ '{ print $11} ' | sort -n | uniq -c | sort -n.
The output looks something like this:
21 122.48.129.75
23 210.21.106.229
24 218.17.228.216
26 220.175.101.252
27 222.67.18.227
30 222.27.89.136
39 123.116.101.186
52 121.231.17.153
63
63 ::1
With the following command I am calculating the netmask which will be
blocked:
$ whois 121.231.17.153 | grep inetn | sed -e "s, - ,:,g" | awk ' { print "netmask "$2 }' | sh
121.224.0.0/12
And then I am using a simple iptables rule to drop any traffic coming from
that network:
iptables -A INETIN -s 121.224.0.0/12 -j DROP
Currently this is the only idea I have how to get rid of those ~500 connections which seem to be some kind of abuse.
Yesterday I was trying to formulate a SQL statement and it took me over three hours. It was really painful.
For our mirror server we are storing the amount of transmitted bytes for each day. My goal was to get all the records from the database from the last twelve months for each day. That is easy. But I wanted to get the data from last twelve months starting at the beginning of a month. So it was not just “today” – “12 months”; it was from “start of the current month” – “12 months” until “today”. And as I am doing probably not more than one SQL statement in six months it was really painful to get a statement which does exactly what I want. But I managed it and here it is in all its glory:
SELECT * from transmitted where date > (select extract(epoch from date_trunc('month', now()-interval'12 month')::date)) order by date asc;
The result of this statement can be seen on our transmitted data overview page.
For our mirror data filesystems I have now enabled the dir_index feature.
I have read about it existence on a mailing list and the tune2fs(8)
man page also sounds like it might be a good idea:
dir_index
Use hashed b-trees to speed up lookups in large directories.
With the following commands I have enabled the feature:
# tune2fs /dev/sdc1 -O dir_index
tune2fs 1.40.2 (12-Jul-2007)
# e2fsck -C 0 -f -D /dev/sdc1
e2fsck 1.40.2 (12-Jul-2007)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdc1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdc1: 568142/178782208 files (5.4% non-contiguous), 305699485/357562713 blocks
This takes a few hours on our filesystems (one is >3TB the other >1TB). If
it really increases performance I will probably never know but it hopefully
does not decrease performance.
I had to reboot my notebook (T43p) after 40 days. So for over a month I was
suspending and rebooting my notebook without any problems multiple times a
day. It works really perfect. The only problems are the
applications. For example firefox or evolution need to be restarted at least
once a week because they somehow are not able to run over a longer period
without problems. Evolution usually starts to refuse to open local mailboxes
because there are not enough filehandles available anymore.
The main reason for the reboot was that the gnome-panel consumed lots of
memory and as I was running a mixture between Fedora 8 and rawhide and as I had
already updated a few gnome packages i thought I could restart the gnome-panel
with killall gnome-panel. Unfortunately it refused to reappear. It
actually was restarted but something was not quite right. It consumed 100% of
the CPU but did not appear. So I decided that I could update even more
packages. I finally switched to a rawhide kernel, updated my (maybe a month
old) firefox 3 package and I also installed upstart.
I was exited about upstart, but it just works and looks exactly the same.
Firefox, however, lost all of my bookmarks and refused to create any new
bookmarks or import a backup of my old bookmarks. It took about thirty minutes
to fix firefox. The problem was that firefox somehow did not like my version of sqlite.
I had about 200 places.sqlite-corrupt files in my
.mozilla/firefox directory but after a yum upgrade sqlite*
everything was back to normal.
Eight bottles of Springbank. Cask strength. 2000. Cask #662.
