I'm a Linux/Unix sysadmin with experience in high availability, scaling and clustering, security, (Open)SSL and linux system administration. I've worked as a sysadmin (in a devops role) for certificate authorities, hospitals, managed service providers, datacenters, cloud providers, development shops, construction companies and large ISP's. I currently work for an Openstack provider. I like to design, build and manage large, complex and high available systems. I'm a team player that likes to work with configuration management tools and version control systems. If I do something more than three times I automate it. Documentation, monitoring and backups are things I do first, not when the time allows it later. Last but not least I have an interest in legacy systems like the PDP-11, PDP-8 and operating systems like OpenVMS, HP-UX and old UNIX systems.
This is my personal website, please do note that these articles do not reflect opinions or policies of any of my (previous) employers, only my personal one.

Latest Items

Small OpenVMS titbits

22-04-2018 | Remy van Elst

Here are some small titbits I found out this week on the DECUServe OpenVMS system. Not enough to write a blogpost on their own, but collected together.


Ansible - add apt_key inline

19-04-2018 | Remy van Elst

Using the apt_key module one can add an APT key with ansible. You can get the key from a remote server or from a file, or just a key ID. I got the request to do some stuff on a machine which was quite rescricted (so no HKP protocol) and I was asked not to place to many files on the machine. The apt_key was needed but it could not be a file, so using a YAML Literal Block Scalar I was able to add the key inline in the playbook. Not the best way to do it, but one of the many ways Ansible allows it.


OpenVMS 7.3 install log with simh VAX on Ubuntu 16.04

16-04-2018 | Remy van Elst

Using a guide I was able to install OpenVMS 7.3 for VAX on simh on Ubuntu 16.04. This is a copy-paste of my terminal for future reference. This is not one of my usual articles, a guide with comprehensive information an background. Just a log of my terminal.


File versioning and deleting on OpenVMS with DELETE and PURGE

15-04-2018 | Remy van Elst

I'm now a few weeks into my OpenVMS adventure and my home folder on the [DECUS](http://decus.org) system is quite cluttered with files. More specifically, with different versions of files, since OpenVMS by default has file versioning built in. This means that when you edit a file, or copy a file over an existing file, the old file is not overwritten but a new file with a new version is written. The old file still is there. This is one of the best things in my humble opinion so far on OpenVMS, but it does require maintenance to not have the disk get filled up fast. This article goes into the PURGE and DELETE commands which help you deal with file versioning and removal.


Synergy, no mouse cursor on Ubuntu 17.10

11-04-2018 | Remy van Elst

Synergy is an application to control remote screens with your local mouse and keyboard over the network, cross platform. You could use Linux as your main OS and have a seperate box with Windows next to it, and use your Linux mouse to control windows. On Ubuntu 17.10 the mouse cursor is not visible, but does work. This snippet provides a fix. Hint, wayland is in the way.


FreeIPA DNS workaround for DNS zone [...]. already exists in DNS and is handled by server(s):

10-04-2018 | Remy van Elst

Recently I ran into an issue with FreeIPA when trying to add an existing DNS zone. The zone already exists on the internet so, logically, FreeIPA wouldn't allow me to hijack this domain locally. My usecase is special, so I wanted to forcefully add this zone as a forward zone.


Backspace and delete key behaviour on OpenVMS

09-04-2018 | Remy van Elst

While working on the DECUServe OpenVMS system I found out quickly that pressing BACKSPACE moves the cursor on the shell to the beginning of the line instead of deleting the character to the left of the cursor. This made me very aware of my typing, since when I made an error I had to retype the entire line (the terminal is in insert mode it seems). After reading through some documentation it seems that is default behaviour but there are terminal options to change it.


SSH public key authentication on OpenVMS

05-04-2018 | Remy van Elst

My OpenVMS adventure continues, after my rabbit hole of folder removal, this time I actually get public key authentication working with OpenSSH so that I don't have to type my password to login.


Delete a directory on OpenVMS

03-04-2018 | Remy van Elst

My OpenVMS adventure continues, in this small item I talk about the removal of folders on OpenVMS. As you might expect, different than on linux. This rabbit hole got started when I made a typo in the creation of a folder, which I created in the process of SSH public key authentication. Pubkey auth still doens't work but my OpenVMS knowledge increased.


Mail on OpenVMS

01-04-2018 | Remy van Elst

Last week I registered myself with the DECUServe OpenVMS system. I found out how to navigate the filesystem and create files and folders, it was awesome. This week I learned how to use the OpenVMS MAIL program to read and reply to an email I got from George Cornelius, another user on the DECUServe system.


My first OpenVMS

31-03-2018 | Remy van Elst

Last week I registered myself with the DECUServe OpenVMS system. I also registered with HP as an OpenVMS hobbyist and got OpenVMS 7.3 for VAX. This small blog item describes my first steps with the hosted DECUS OpenVMS system. I'm excited since I now know how to create folders, navigate the filesystem and edit files. Oh and I had a nice chat with another OpenVMS user via the PHONE program.


OpenStack nova get-password, set-password and post encrypted password to metadata service

25-03-2018 | Remy van Elst

When you create images for an OpenStack Cloud you want to use 'cloud' features. Fancy term for automatic resizing of your instance disk, adding an SSH key, (re)setting passwords and executing scripts on first boot to configure your instance further. OpenStack provides the metadata service for instances, which supplies information for the instance, like its public IP, SSH public key that was provided and vendor or user provided data like scripts or information. The OpenStack metadata service allows an instance to post data to an endpoint wich can be retreived with the 'nova get-password' command. It is meant to be an encrypted password (with the public SSH key) but it can be any plain text as well and it doesn't have to be the root password. In this guide I'll go over the scripts I use inside linux images to post a password to the metadata service and the 'nova' commands such as 'set-password' and 'get-password'. That includes decrypting a password with an SSH key that is password-protected (Horizon and nova don't support that) and the 'nova set-password' command, which sets the root password inside an instance when it has the 'qemu-guest-agent' installed and running.


Essential Monitoring checks

20-03-2018 | Remy van Elst

In this article I'll provide a list of checks I consider essential for monitoring and why they are usefull. It's on different levels, ranging from your application (health checks), to operating system (disk usage, load) and hardware (iDrac, disks, power). Use it as a starting point when setting up your monitoring.


My Yubikey broke, but I had a backup. So should you with your 2FA

18-03-2018 | Remy van Elst

Today my trusty old first generation Yubikey didn't light up when I plugged it in. No problem for me, I had a backup key. But most people don't, so here's an important tip when you use two factor authentication like a Yubikey, Nitrokey or Google Authenticator (HOTP). TL;DR: Have a second hardware token stored away safely and backup your QR codes (print/screenshot) somewhere secure. Swap the hardware tokens often to make sure they both work with all services. Just as with regular data, make backups and test restores.


haproxy: restrict specific URLs to specific IP addresses

04-03-2018 | Remy van Elst

This snippet shows you how to use haproxy to restrict certain URLs to certain IP addresses. For example, to make sure your admin interface can only be accessed from your company IP address. It also includes an example to prompt for a password if the visitor is from a different network.


Dell PowerEdge firmware upgrades via iDrac

26-01-2018 | Remy van Elst

The recent spectre and meltdown vulnerabilities require BIOS and firmware updates. Dell provides binaries for Windows and Linux, but just for Red Hat and SUSE. Some firmware updates can be run on Ubuntu or Debian, but some fail with the error that RPM could not be found. Which is correct since it's not Red Hat. In this small article I'll show you how to upgrade the firmware via the iDrac, which I recently discovered.


ncdu - for troubleshooting diskspace and inode issues

29-10-2017 | Remy van Elst

In my box of sysadmin tools there are multiple gems I use for troubleshooting servers. Since I work at a cloud provider sometimes I have to fix servers that are not mine. One of those tools is `ncdu`. It's a very usefull tool when a server has a full disk, both full of used space or full of used inodes. This article covers ncdu and shows the process of finding the culprit when you're out of disk space or inodes.


Adding IPv6 to a keepalived and haproxy cluster

24-09-2017 | Remy van Elst

At work I regularly build high-available clusters for customers, where the setup is distributed over multiple datacenters with failover software. If one component fails, the service doesn't experience issues or downtime due to the failure. Recently I was tasked with expanding a cluster setup to be also reachable via IPv6. This article goes over the settings and configuration required for haproxy and keepalived for IPv6. The internal cluster will only be IPv4, the loadbalancer terminates HTTP and HTTPS connections.


atop is broken on Ubuntu 16.04 (version 1.26): trap divide error

18-09-2017 | Remy van Elst

Recently a few of my Ubuntu 16.04 machines had issues and I was troubleshooting them, noticing `atop` logs missing. atop is a very handy tool which can be setup to record system state every X minutes, and we set it up to run every 5 minutes. You can then at a later moment see what the server was doing, even sorting by disk, memory, cpu or network usage. This post discusses the error and a quick fix.


Backup OpenStack object store or S3 with rclone

17-08-2017 | Remy van Elst

This is a guide that shows you how to make backups of an object storage service like OpenStack swift or S3. Most object store services save data on multiple servers, but deleting a file also deletes it from all servers. Tools like rsync or scp are not compatible most of the time with these services, unless there is a proxy that translates the object store protocol to something like SFTP. rclone is an rsync-like, command line tool that syncs files and directories from cloud storage services like OpenStack swift, Amazon S3, Google cloud/drive, dropbox and more. By having a local backup of the contents of your cloud object store you can restore from accidental deletion or easily migrate between cloud providers. Syncing between cloud providers is also possible. It can also help to lower the RTO (recovery time objective) and backups are just always a good thing to have and test.


Openstack Horizon, remove the loading modal with uBlock Origin

25-05-2017 | Remy van Elst

The OpenStack dashboard, Horizon, is a great piece of software to manage your OpenStack resources via the web. However, it has, in my opinion, a very big usability issue. The loading dialog that appears after you click a link. It blocks the entire page and all other links. So, whenever I click, I have to wait three to five seconds before I can do anything else. Clicked the wrong menu item? Sucks to be you, here have some loading. Clicked a link and quickly want to open something in a new tab while the page is still loading? Nope, not today. It's not that browsers have had a function to show that a page is loading, no, of course, the loading indication that has been there forever is not good enough. Let's re-invent the wheel and significantly impact the user experience. With two rules in uBlock Origin this loading modal is removed and you can work normally again in Horizon


All Items