18-09-2017 | Remy van Elst
Recently a few of my Ubuntu 16.04 machines had issues and I was troubleshooting them, noticing
atop logs missing. atop is a very handy tool which can be setup to record system state every X minutes, and we set it up to run every 5 minutes. You can then at a later moment see what the server was doing, even sorting by disk, memory, cpu or network usage.
Atop logs in
/var/log/atop by date:
root@server:~# ls -la /var/log/atop/ total 29916 drwxr-xr-x 2 root root 4096 Sep 18 00:00 . drwxrwxr-x 10 root syslog 4096 Sep 18 06:25 .. -rw-r--r-- 1 root root 3051661 Aug 22 00:00 atop_20170821 -rw-r--r-- 1 root root 1553161 Aug 25 00:00 atop_20170824 -rw-r--r-- 1 root root 1593178 Aug 26 00:00 atop_20170825
Using the commandline:
atop -r /var/log/atop/atop_20170821 # -b time_to_start
will open up atop with that log. Using
T you can go back and forward in time.
Checking the system log this message showed up on the dates that the log was missing:
Sep 18 00:00:01 server kernel: [82125.706464] traps: atop trap divide error ip:4073c2 sp:7ffebe903810 error:0 in atop[400000+26000]
There is a debian bug report where they state this bug is solved in the 2.x release (debian experimental). Ubuntu 16.04 sadly still ships 1.x, and due to policy will never update to 2.x. The bug is solved in December 2016, but the ubuntu changelog still shows that the last update was in January 2016:
atop (2.2.6-4) unstable; urgency=medium * remove versioned dependency on initscripts. See also #852314. Thanks to Stephen Kitt (Closes: 852239) -- Marc Haber <firstname.lastname@example.org> Wed, 25 Jan 2017 20:45:37 +0100
You can check the version you're running by opening
atop and pressing
Version: 1.26 - 2010/11/17 13:42:37
The atop version in Ubuntu 16.10 or 17.04 is the 2.x release, which you can download on the ubuntu packages site:
wget http://mirrors.kernel.org/ubuntu/pool/universe/a/atop/atop_2.2.6-4_amd64.deb dpkg -i atop_2.2.6-4_amd64.deb service atop reload
Afterwards you will notice that the log format is not compatible along versions:
-rw-r--r-- 1 root root 1004276 Sep 18 14:54 atop_20170918 -rw-r--r-- 1 root root 18101 Sep 18 09:44 atop_20170918_pre_2_0
If you try to open a previous format logfile that will fail:
root@server:~# atop -r /var/log/atop/atop_20170917 -b 23:00 raw file /var/log/atop/atop_20170917 has incompatible format (created by version 1.26 - current version 2.2) trying to activate atop-1.26.... activation of atop-1.26 failed!
If you want to replay those logs, replay them on a machine with an older version of atop.
Do note that using packages from another major version is not recommended. This method of installing a .deb file is also quick and dirty, a better way is to use apt pinning.