Skip to main content

Raymii.org Logo (IEC resistor symbol)logo

Quis custodiet ipsos custodes?
Home | About | All pages | RSS Feed | Gopher

Set up a Collectd server with web frontend

Published: 09-04-2013 | Author: Remy van Elst | Text only version of this article


Table of Contents


This tutorial shows you how to set up a collectd server. It also shows you howto set up the collectd-web frontend, an interactive gui for collectd and has itall firewalled. What is collectd? collectd gathers statistics about the systemit is running on and stores this information. Those statistics can then be usedto find current performance bottlenecks (i.e. performance analysis) and predictfuture system load (i.e. capacity planning). Or if you just want pretty graphsof your private server and are fed up with some homegrown solution you're at theright place, too ;). A collectd server is able to receive data from collectdclients.

If you like this article, consider sponsoring me by trying out a Digital OceanVPS. With this link you'll get $100 credit for 60 days). (referral link)

The collectd client tutorial can be found here.

We will first set up Collectd on the server, and configure it to listen on thenetwork. Then we will enable the web GUI, and use NGINX to reverse proxy it.Collectd client configuration is not handled by this tutorial. This tutorialis tested on Debian 6, Debian 7, Ubuntu 10.04 and Ubuntu 12.04. However, theconfiguration works on any other distro.

Installing Collectd

First install all the required packages:

sudo apt-get install collectd librrds-perl libconfig-general-perl libhtml-parser-perl libregexp-common-perl liburi-perl libjson-perl restartd python nginx

The collectd web interface has a few perl dependencies. Restartd will be used tomake sure the webinterface is running, and NGINX will be used to proxy thewebinterface to the outside.

Configuring Collectd as a network server

Open your favorite editor and edit the /etc/collectd/collectd.conf file. Readit, then remove it all and make sure it looks like the below config file:

## /etc/collectd/collectd.conf Hostname $HOSTNAME$FQDNLookup false## This can be higher if you have a more powerfull boxInterval 30## This can be higher if you have a more powerfull boxReadThreads 1LoadPlugin syslog<Plugin syslog>        LogLevel info</Plugin>LoadPlugin cpuLoadPlugin dfLoadPlugin diskLoadPlugin entropyLoadPlugin interfaceLoadPlugin irqLoadPlugin loadLoadPlugin memoryLoadPlugin processesLoadPlugin rrdtoolLoadPlugin swapLoadPlugin usersLoadPlugin network## Server config<Plugin "network">  # Can also be "*" "25826" to listen on 0.0.0.0  Listen "$EXTERNAL_IPV4$" "25826"  Listen "$EXTERNAL_IPV6$" "25826"  ReportStats true  SecurityLevel None </Plugin>## Extra Plugins## remove to disableLoadPlugin nginxLoadPlugin iptablesLoadPlugin uptimeLoadPlugin dnsLoadPlugin ping<Plugin rrdtool>        DataDir "/var/lib/collectd/rrd"</Plugin>Include "/etc/collectd/filters.conf"Include "/etc/collectd/plugins.conf"Include "/etc/collectd/thresholds.conf"

The configuration file is relatively simple. Make sure to replace $VARIABLE$ bythe correct on for your server. You load plugins via "LoadPlugin $name". Thenetwork part is important, this defines the server. Collectd supports both IPv4and IPv6, I have a few IPv6 IP's in the listen part, and "0.0.0.0" as IPv4address. Make sure the file has a blank newline at the end. If it has not,collectd will fail to start/run correctly.

Configure collectd plugins

Now create the following file: /etc/collectd/plugins.conf, it doesn't exist bydefault. This will house the plugin config. Add the following content to it, butmake sure it matches your LoadPlugin settings above. If you don't have the pingplugin, you also don't need the config for it.

## /etc/collectd/plugins.conf ## Static Plugins (every host has them)<Plugin swap>       ReportByDevice false</Plugin>## Dynamic Plugins (loaded by Ansible based on options)<Plugin nginx>       URL "http://127.0.0.1/nginx_status"</Plugin><Plugin ntpd>       Host "localhost"       Port 123       ReverseLookups false</Plugin><Plugin ping>       Host "google.com"</Plugin><Plugin sensors>       SensorConfigFile "/etc/sensors3.conf"       Sensor "it8712-isa-0290/temperature-temp1"       Sensor "it8712-isa-0290/fanspeed-fan3"       Sensor "it8712-isa-0290/voltage-in8"       IgnoreSelected false</Plugin><Plugin write_graphite>       <Carbon>               Host "$GRAPHITE_HOST$"               Port "2003"               Prefix "collectd"               Postfix "collectd"               StoreRates false               AlwaysAppendDS false               EscapeCharacter "_"       </Carbon></Plugin>

Collectd server is now set up in "server" mode. To test it we restart theservice:

/etc/init.d/collectd restart

And then check the /var/lib/collectd/rrd/ folder and you should see some filesand folders (rrd libraries). If not then your collectd is setup wrong, seesyslog for more info.

Set up the web frontend

Clone the git repository to your home directory, or any other folder, butremember the path:

git://github.com/RaymiiOrg/collectd-web.git

Move into the folder and start the app:

cd collectd-webpython runserver.py

You should now be able to go to "localhost:8888" on the host and see the webinterface. If you need to test it from the outside, use an ssh tunnel:

ssh -t -t -L 8888:localhost:8888 USER@SERVER.COM

You can now view the app in your local web browser via http://localhost:8888

Set up restartd

We will use restartd to make sure the app works even after reboot or when itcrashes. This could also be done with a nice init script, but this works theeasiest way.

Edit /etc/restartd.conf and make sure it has the following in it:

collectd-web ".*runserver.py" "su $USER -l -c 'pushd /home/$USER/collectd-web/ &amp;&amp; /usr/bin/python /home/$USER/collectd-web/runserver.py' >> /var/log/$USER-collectd-server.log" "/bin/echo 'collectd-server running' >> /var/log/$USER-collectd-server.log"

Make sure you change $USER to the username which has the application, and checkif the paths are correct. If so then restart restartd:

/etc/init.d/restartd restart

(Funny isn't that? Restarting restartd?)

Use NGINX as a reverse proxy

If you want to make your collectd publicly available the you should follow thispart. If you don't want that and you find the SSH port forwarding tunnel worksfor you, then use that.

Add the following to your NGINX configuration to set it up as reverse proxy forthe collectd server:

It should be in a server {} block.

location /collectd {    rewrite ^/collectd(/.*)$ $1 break;    proxy_pass  http://127.0.0.1:8888/;    proxy_set_header Host $host;    proxy_set_header X-Real-IP $remote_addr;    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;    proxy_set_header X-Forwarded-Proto https;    proxy_redirect    off;}

Restart NGINX:

/etc/init.d/nginx configtest/etc/init.d/nginx restart

Now you can reach the collectd at http://your-server/collectd.

Configuring the firewall

You should only allow hosts you set up to connect and send data to collectd.Collectd supports authentication and singing, but I've had performance issueswith that on <512MB VPS servers, so that's why I firewall. The followingiptables and ip6tables rules should be added for all the hosts, so changethe IP address every time:

/sbin/iptables -A INPUT -p udp -s 1.2.3.4 --dport 25826 -j ACCEPT/sbin/ip6tables -A INPUT -p udp -s fe80::feda:6cc1 --dport 25826 -j ACCEPT

If you have set up all the collectd client IP addresses in iptables, close thegate:

/sbin/iptables -A INPUT -p -udp --dport 25826 -j REJECT --reject-with udp-reset/sbin/ip6tables -A INPUT -p -udp --dport 25826 -j REJECT --reject-with udp-reset

Now you have a fully working collectd server set up.

Tags: collectd, collectd-web, monitoring, munin, nginx, restartd, rrd, rrdtool, statistics, tutorials