Raymii.org

Table of Contents

• The issue
• Workaround

Recently a few of my Ubuntu 16.04 machines had issues and I was troubleshooting them, noticing atop logs missing. atop is a very handy tool which can be setup to record system state every X minutes, and we set it up to run every 5 minutes. You can then at a later moment see what the server was doing, even sorting by disk, memory, cpu or network usage.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

Atop logs in /var/log/atop by date:

root@server:~# ls -la /var/log/atop/
total 29916
drwxr-xr-x  2 root root      4096 Sep 18 00:00 .
drwxrwxr-x 10 root syslog    4096 Sep 18 06:25 ..
-rw-r--r--  1 root root   3051661 Aug 22 00:00 atop_20170821
-rw-r--r--  1 root root   1553161 Aug 25 00:00 atop_20170824
-rw-r--r--  1 root root   1593178 Aug 26 00:00 atop_20170825

Using the commandline:

atop -r /var/log/atop/atop_20170821 # -b time_to_start

will open up atop with that log. Using t and T you can go back and forward in time.

The issue

Checking the system log this message showed up on the dates that the log was missing:

Sep 18 00:00:01 server kernel: [82125.706464] traps: atop[517] trap divide error ip:4073c2 sp:7ffebe903810 error:0 in atop[400000+26000]

There is a debian bug report where they state this bug is solved in the 2.x release (debian experimental). Ubuntu 16.04 sadly still ships 1.x, and due to policy will never update to 2.x. The bug is solved in December 2016, but the ubuntu changelog still shows that the last update was in January 2016:

atop (2.2.6-4) unstable; urgency=medium

  * remove versioned dependency on initscripts. See also #852314.
    Thanks to Stephen Kitt (Closes: 852239)

 -- Marc Haber <mh+debian-packages@zugschlus.de>  Wed, 25 Jan 2017 20:45:37 +0100

You can check the version you're running by opening atop and pressing V:

Version: 1.26 - 2010/11/17 13:42:37

Workaround

The atop version in Ubuntu 16.10 or 17.04 is the 2.x release, which you can download on the ubuntu packages site:

wget http://mirrors.kernel.org/ubuntu/pool/universe/a/atop/atop_2.2.6-4_amd64.deb
dpkg -i atop_2.2.6-4_amd64.deb
service atop reload

Afterwards you will notice that the log format is not compatible along versions:

-rw-r--r--  1 root root   1004276 Sep 18 14:54 atop_20170918
-rw-r--r--  1 root root     18101 Sep 18 09:44 atop_20170918_pre_2_0

If you try to open a previous format logfile that will fail:

root@server:~# atop -r /var/log/atop/atop_20170917 -b 23:00
raw file /var/log/atop/atop_20170917 has incompatible format
(created by version 1.26 - current version 2.2)
trying to activate atop-1.26....
activation of atop-1.26 failed!

If you want to replay those logs, replay them on a machine with an older version of atop.

Do note that using packages from another major version is not recommended. This method of installing a .deb file is also quick and dirty, a better way is to use apt pinning.

Tags: articles , atop , monitoring , ubuntu