Migrating personal webapps and services
Published: 05-05-2016 | Author: Remy van Elst | Text only version of this article
Table of Contents
Recently I've migrated some of my personal servers and services to new machinesand newer operating system versions. I prefer to migrate instead of upgradingthe OS for a number of reasons. I'll also talk about the migration process andsome stuff to remember when migrating web applications and services.
As many of you might also do, I run a number of servers with applications for mypersonal use and a number of websites for friends and family. My own applicationlandscape consists of:
- 7 Wordpress websites (family and friends)
- 3 Drupal websites (family and friends)
- OwnCloud for Contacts and Calendar sync
- Tiny Tiny RSS
- My SSL tester
- Certificate Expiry Monitor
- Seven static websites with email hosting for family.
This was all spread over three servers at different VPS providers, namelyLinode, Digital Ocean and CloudVPS. The three servers were running Ubuntu 12.04and CentOS 6, with Apache, PHP and MySQL. Plus exim and dovecot for email. Quitea few servers and setups to maintain.
Reasons for migrating
There are multiple reasons for migrating and consolidating all the services.
The first one is that the server with the most websites on it, including theSSL Decoder had a broken OpenSSL version. That was my own fault as I wasplaying with compiling prereleases and not using a (docker) container or otherconfinement, thus overwriting system OpenSSL and libssl libraries. In generalthat's not a big problem, but all kinds of little things broke because of this.
The second reason is that I got a nice message from Andrew, the CommunityManager at Digital Ocean regarding Cipherli.st. He said the site wasused inside of Digital Ocean and they like it. I got some credits appliedto my account, so that means free servers for a few months, which is awesome.Thanks to Andrew and DO for that. If you like this article, considersponsoring me by trying out a Digital Ocean VPS. With this link you'll get a $5VPS for 2 months free (as in, you get $10 credit). (referral link)
The third reason is that the servers were running older versions of Operatingsystems, like Ubuntu 12.04 LTS and CentOS 6. Both of those still receive supportand updates, but lack new technology like Docker, LXD or systemd. I'm planningon using both Docker and LXD to test the SSL Decoder.
The SSL Decoder now includes a Dockerfile which sets up Apache, PHP and thelatest OpenSSL (1.1.0). That means I can't screw up the system OpenSSL anymore.In the future the SSL Decoder will probably run in a container for production aswell. I've set up a new VPS with Ubuntu 16.04 on Digital Ocean, whichbecause it's KVM and not OpenVZ or Xen PV allows me to run Docker withoutproblems.
Plus, in my humble opinion this is just regular maintenance. Like you do regularmaintenance on your car or bicycle, you should also do that for your servers.
In the next few paragraphs I'll talk about the migration itself and what to dobefore and after migrating. The process itself is fairly simple, not much morethan a few database dumps and
rsync copy's. As always, it's the small thingssurrounding the process that make it tricky.
Before the migration
There are a few things to do before you migrate (web)services. The mostimportant thing is to make a list of all the things you're going to migrate.This includes:
- Websites (URL, logins, IP)
- Email accounts
- DNS zones
- Current monitoring
Note down all these things and make sure you test beforehand. That way you'llknow what the result should be in a working setup.
The other important thing to do is to lower the DNS TTL (Time-to-live) of thedomain's DNS zones. If it's a TTL of 1 week, the migration will take a lotlonger. Lower it to 1 hour (nobody (public DNS) honours anything lower (the 5minutes)). Don't forget that you have to await the current TTL before the newone is active. If it's now a week, then you'll have to wait a week before the 1hour TTL is active.
The TTL is used to point people to the IP when they visit an URL. A high TTLlets DNS servers cache the IP longer, so people will be sent to the wrong IP. Ifyou want you can raise the TTL after the migration.
Depending on the service you're migrating you also want to send the users aprior notice, maybe a few. I notified the relatives and users that the migrationwas upcoming three times and afterwards also emailed them the migrationcompleted successfully.
With your list, lowered DNS TTL and notified users, continue on to the actualmigration.
The migration itself consists out of two things. Setting up the server andtransferring the data.
In my case the setup of the server was super easy. It is set up with Ansible andthe playbooks used to set up Apache, PHP and MySQL on Ubuntu 12.04 and CentOS 6also worked without issues on Ubuntu 16.04. Within 15 minutes the whole setupwas done, just by adding a new host to my
ansible_hosts file and updating the
webserver hostgroup with this extra host.
The dovecot and exim playbook required a few
if statements in theconfiguration templates, but nothing worldshocking. I tested the playbooks firston a disposable Ubuntu 16.04 VPS.
If you have a more manual setup then this step consists of installing andconfiguring all the required software manually and hopefully having some sort ofchecklist.
The data transfer part was just doing a
mysqldump, recreating those databaseson the new server with the database users and importing the data. If you don'twant to recreate the users you can
mysql database, whichcontains the
users table. The import of that is quite funny:
mysql mysql < mysql
mysql is the command, the second is the database name and the thirdis the filename to import, which I cleverly named
The other data, websites, scripts, mailboxes and more were transferred with
ssh. Check the permissions afterwards if you decide to changeusernames. I didn't do that so it all went without issues or
After the migration
When all the data is transferred and the software has been set up properly it istime to test the services. I did that using my
/etc/hosts file, pointing thedomains one by one to my new servers IP. I had my list of sites and services totest, which I did one by one.
One of the things I forgot to migrate was the certificate for the SMTP and IMAPservice. I thought Ansible would set that up but appearantly it didn't. Iupdated the playbooks, so now it does set it up correctly. If I hadn't tested,my users would complain about a warning, and we put in all this effort to makesure that doesn't happen.
When all the tests were finished I changed the DNS for all the domains to pointto the new IP. Leaving the TTL to one hour, so in case of a failure the fallbackwouldn't take long.
After a few days of no complaints I shut off the old servers. My monitoring wentoff, but not for the services I migrated. Just the host checks, as expected.
I keep a backup of the machines locally just to be sure. The servers werecancelled at the respective providers two weeks after the migration completed.
Stuff to remember when migrating webapps that email.
My webapplication sends email, so there are a few extra things that need to beset up and checked. I've listed them below for my own reference:
- Reverse DNS for IPv4 Reverse DNS for IPv6 SMTP mailname must match hostname
- and reverse name SPF records must include new IP DKIM keys need to be updated
A good service to test email sending in Mail Tester. If you forget themailsettings your emails will go right into the SPAM folder of many users.
With a little bit of preparation and thought, a migration like this is easy todo and relatively painless. By keeping your servers up to date you are not onlymore secure, you will also have less hassle in the future.
I've seen many last-minute migrations gone wrong. For example, a big newsecurity vulnerability affects your services. And particularly those importantmachines were the migration keeps getting deferred for a boatload of (stupid)reasons. And then you're forced to migrate them on an inconvinient time, nothaving a good prepared plan or test scheme and getting delays on your otherprojects as well.
Better to just do regular maintenance in your own pace.
I'd like to thank Digital Ocean again for the generous credit. It was thething that pushed me over the edge to plan and do this consolidation andmigration.Tags: blog, dns, migrate, migration, php, ubuntu, website