Raymii.org

Table of Contents

• Source Code and Downloads
• Building
• Usage
• rsyslog and logrotate config
• Docker

For almost 5 years I've been using the mqtt-data-logger python script to log all MQTT communication. This script works, but is a bit limited in how it stores logs (files per time it's started) and it often can't keep up with the message rate (tens of thousands per second) of my mqtt brokers. So I've written a new tool in C++ to log all MQTT messages to syslog and optionally to the terminal. Then, via syslog, you can send it to where ever you like, be it a file or an ELK (Logstash/Elasticsearch/Kibana) stack. It even compiles without syslog support, if you only want to log to a file or the terminal. There's a docker image and a docker compose file, so you can easily deploy it wherever you have an MQTT broker running.

Kibana showing the tool logging via rsyslog to logstash, the test.mosquitto.org broker

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below. It means the world to me if you show your appreciation and you'll help pay the server costs:

GitHub Sponsorship

PCBWay referral link (You get $5, I get $20 after you've placed an order)

Digital Ocea referral link ($200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!)

Every message is logged on a separate thread in a thread pool, which is what makes is fast and able to process thousands of messages per second as long as your system has enough resources. Using Pipe Viewer to measure lines per second from a test broker:

$ ./remys_fast_mqtt_logger -b test.mosquitto.org:1883 | pv --line-mode --rate --average-rate >/dev/null
[4.79k/s] (3.35k/s)

This is similar to mosquitto_sub:

$ mosquitto_sub -h test.mosquitto.org -p 1883 -t "#"  | pv --line-mode --rate --average-rate >/dev/null
[3.39k/s] (3.37k/s)

Connection failures are logged and handled with an automatic reconnect.

It is cross-platform, meaning that if you don't have syslog.h, it will also compile and run, but it only prints to the terminal.

Source Code and Downloads

The source code can be found on github.

A ready made docker image is here.

This project is licensed under the GNU AGPLv3 License. See the LICENSE file for more details.

Building

The program is written using C++ 17 and uses CMake as the build system.

Install dependencies

The following libraries are required to build and run the application:

1. Paho MQTT C++ library.

2. syslog.h

• On Ubuntu/Debian:

  apt install libpaho-mqttpp-dev libpaho-mqtt-dev build-essential git

Clone or download the repository:

• Clone the git repository or download the source code:

  git clone https://github.com/RaymiiOrg/remys_fast_mqtt_logger
  

Compiling:

Use CMake:

cd remys_fast_mqtt_logger
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release  ..

Usage

The logger can be configured with command line options or $ENVIRONMENT variables. The latter don't end up in your shell history and are used in the docker-compose file.

Command line arguments

• -b BROKER, --broker=BROKER

  • The MQTT broker URL (including the port) to connect to.
  • Example: test.mosquitto.org:1883.

• -t TOPIC, --topic=TOPIC

  • The MQTT topic to subscribe to.
  • Default is # (subscribe to all topics).

• -u USERNAME, --username=USERNAME

  • Username for authenticating with the MQTT broker.

• -p PASSWORD, --password=PASSWORD

  • Password for authenticating with the MQTT broker.

• -f FACILITY, --facility=FACILITY

  • The syslog facility to log to. Must prefix with LOG_.
  • Default is LOG_LOCAL6.

• -s, --no-log-to-stderr

  • Disable logging to STDERR
  • Default is disabled, meaning messages will be logged to both stderr and syslog unless specified.

Environment Variables

You can configure the application using the following environment variables:

• BROKER: MQTT broker URL (e.g., mqtt.example.com:1883).
• USERNAME: MQTT broker username.
• PASSWORD: MQTT broker password.
• TOPIC: MQTT topic to subscribe to (default: #).
• FACILITY: Syslog facility (default: LOG_LOCAL6).
• NO_LOG_TO_STDERR: Disable logging to STDERR (default: on)

Example Commands

Basic usage:

./remys_fast_mqtt_logger -b "mqtt.example.com:1883" \
-t "home/livingroom/temperature" \
-u "user" \
-p "password"

Viewing Logs in Syslog

To view logs for LOG_LOCAL6 in journalctl:

journalctl SYSLOG_FACILITY=22

Or to view logs directly in log file:

tail -f /var/log/syslog

rsyslog and logrotate config

You can save logs to /var/log/mqtt_msgs.log if you have configured rsyslog accordingly:

vim /etc/rsyslog.d/30-local6.conf

Add:

local6.*    /var/log/mqtt_msgs.log

Then:

touch /var/log/mqtt_msgs.log
chmod 644 /var/log/mqtt_msgs.log
chown syslog:adm /var/log/mqtt_msgs.log

Configure logrotate to not delete these logs:

vim /etc/logrotate.d/remys_fast_mqtt_logger

Contents:

/var/log/mqtt_msgs.log {
    missingok
    notifempty
    size 100M
    rotate 9999
    compress
    delaycompress
    create 0644 root root
    sharedscripts
    postrotate
        systemctl reload rsyslog > /dev/null 2>&1 || true
    endscript
}

Docker

To build and run the application with docker, use the following commands inside the git checkout.

Build and run the Docker image:

docker build -t mqtt-logger .
docker run --env BROKER=test.mosquitto.org mqtt_logger 

Or, use my image and run the Docker container:

docker run --name remys_fast_mqtt_logger --env BROKER=test.mosquitto.org raymii/remys_fast_mqtt_logger:latest 

There is also a docker-compose.yml file included which you can edit and use:

docker compose up -d
docker compose logs -f


[+] Running 1/1
 Container mqtt_logger-remys-fast-mqtt-logger-1  Created0.0s
Attaching to remys-fast-mqtt-logger-1
remys-fast-mqtt-logger-1  | topic='/116484256345/0/current', qos='0', retained='true', msg='1.87'
remys-fast-mqtt-logger-1  | topic='/116484256345/0/efficiency', qos='0', retained='true', msg='95.015'
remys-fast-mqtt-logger-1  | topic='/116484256345/0/frequency', qos='0', retained='true', msg='49.96'
remys-fast-mqtt-logger-1  | topic='/116484256345/0/power', qos='0', retained='true', msg='442.2'
remys-fast-mqtt-logger-1  | topic='/116484256345/0/powerfactor', qos='0', retained='true', msg='0.999'
remys-fast-mqtt-logger-1  | topic='/116484256345/0/temperature', qos='0', retained='true', msg='28.6'
remys-fast-mqtt-logger-1  | topic='/116484256345/0/voltage', qos='0', retained='true', msg='236.3'
remys-fast-mqtt-logger-1  | topic='/116484256345/0/yieldday', qos='0', retained='true', msg='5547'
remys-fast-mqtt-logger-1  | topic='/116484256345/0/reactivepower', qos='0', retained='true', msg='0.3'
remys-fast-mqtt-logger-1  | topic='SHRDZM/483FDA46C2EE/483FDA46C2EE/sensor', qos='0', retained='false', msg='{
remys-fast-mqtt-logger-1  | "lasterror":"cipherkey not set!",
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/1

Tags: apt , c++ , cpp , debian , development , docker , docker-compose , elasticsearch , kibana , linux , logstash , mqtt , paho , rsyslog , software , syslog