Skip to main content

Raymii.org Raymii.org Logo

Quis custodiet ipsos custodes?
Home | About | All pages | Cluster Status | RSS Feed

Migrating personal webapps and services

Published: 05-05-2016 | Author: Remy van Elst | Text only version of this article


❗ This post is over seven years old. It may no longer be up to date. Opinions may have changed.

ubuntu

Recently I've migrated some of my personal servers and services to new machines and newer operating system versions. I prefer to migrate instead of upgrading the OS for a number of reasons. I'll also talk about the migration process and some stuff to remember when migrating web applications and services.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.

As many of you might also do, I run a number of servers with applications for my personal use and a number of websites for friends and family. My own application landscape consists of:

This was all spread over three servers at different VPS providers, namely Linode, Digital Ocean and CloudVPS. The three servers were running Ubuntu 12.04 and CentOS 6, with Apache, PHP and MySQL. Plus exim and dovecot for email. Quite a few servers and setups to maintain.

Reasons for migrating

There are multiple reasons for migrating and consolidating all the services.

The first one is that the server with the most websites on it, including the SSL Decoder had a broken OpenSSL version. That was my own fault as I was playing with compiling prereleases and not using a (docker) container or other confinement, thus overwriting system OpenSSL and libssl libraries. In general that's not a big problem, but all kinds of little things broke because of this.

The second reason is that I got a nice message from Andrew, the Community Manager at Digital Ocean regarding Cipherli.st. He said the site was used inside of Digital Ocean and they like it. I got some credits applied to my account, so that means free servers for a few months, which is awesome. Thanks to Andrew and DO for that. If you like this article, consider sponsoring me by trying out a Digital Ocean VPS. With this link you'll get a $5 VPS for 2 months free (as in, you get $10 credit). (referral link)

The third reason is that the servers were running older versions of Operating systems, like Ubuntu 12.04 LTS and CentOS 6. Both of those still receive support and updates, but lack new technology like Docker, LXD or systemd. I'm planning on using both Docker and LXD to test the SSL Decoder.

The SSL Decoder now includes a Dockerfile which sets up Apache, PHP and the latest OpenSSL (1.1.0). That means I can't screw up the system OpenSSL anymore. In the future the SSL Decoder will probably run in a container for production as well. I've set up a new VPS with Ubuntu 16.04 on Digital Ocean, which because it's KVM and not OpenVZ or Xen PV allows me to run Docker without problems.

Plus, in my humble opinion this is just regular maintenance. Like you do regular maintenance on your car or bicycle, you should also do that for your servers.

In the next few paragraphs I'll talk about the migration itself and what to do before and after migrating. The process itself is fairly simple, not much more than a few database dumps and rsync copy's. As always, it's the small things surrounding the process that make it tricky.

Before the migration

There are a few things to do before you migrate (web)services. The most important thing is to make a list of all the things you're going to migrate. This includes:

  • Websites (URL, logins, IP)
  • Databases
  • Email accounts
  • DNS zones
  • Dependencies
  • Current monitoring

Note down all these things and make sure you test beforehand. That way you'll know what the result should be in a working setup.

The other important thing to do is to lower the DNS TTL (Time-to-live) of the domain's DNS zones. If it's a TTL of 1 week, the migration will take a lot longer. Lower it to 1 hour (nobody (public DNS) honours anything lower (the 5 minutes)). Don't forget that you have to await the current TTL before the new one is active. If it's now a week, then you'll have to wait a week before the 1 hour TTL is active.

The TTL is used to point people to the IP when they visit an URL. A high TTL lets DNS servers cache the IP longer, so people will be sent to the wrong IP. If you want you can raise the TTL after the migration.

Depending on the service you're migrating you also want to send the users a prior notice, maybe a few. I notified the relatives and users that the migration was upcoming three times and afterwards also emailed them the migration completed successfully.

With your list, lowered DNS TTL and notified users, continue on to the actual migration.

Migration process

The migration itself consists out of two things. Setting up the server and transferring the data.

In my case the setup of the server was super easy. It is set up with Ansible and the playbooks used to set up Apache, PHP and MySQL on Ubuntu 12.04 and CentOS 6 also worked without issues on Ubuntu 16.04. Within 15 minutes the whole setup was done, just by adding a new host to my ansible_hosts file and updating the webserver hostgroup with this extra host.

The dovecot and exim playbook required a few if statements in the configuration templates, but nothing worldshocking. I tested the playbooks first on a disposable Ubuntu 16.04 VPS.

If you have a more manual setup then this step consists of installing and configuring all the required software manually and hopefully having some sort of checklist.

The data transfer part was just doing a mysqldump, recreating those databases on the new server with the database users and importing the data. If you don't want to recreate the users you can mysqldump the mysql database, which contains the users table. The import of that is quite funny:

mysql mysql < mysql

The first mysql is the command, the second is the database name and the third is the filename to import, which I cleverly named mysql.

The other data, websites, scripts, mailboxes and more were transferred with rsync over ssh. Check the permissions afterwards if you decide to change usernames. I didn't do that so it all went without issues or chown-ing.

After the migration

When all the data is transferred and the software has been set up properly it is time to test the services. I did that using my /etc/hosts file, pointing the domains one by one to my new servers IP. I had my list of sites and services to test, which I did one by one.

One of the things I forgot to migrate was the certificate for the SMTP and IMAP service. I thought Ansible would set that up but appearantly it didn't. I updated the playbooks, so now it does set it up correctly. If I hadn't tested, my users would complain about a warning, and we put in all this effort to make sure that doesn't happen.

When all the tests were finished I changed the DNS for all the domains to point to the new IP. Leaving the TTL to one hour, so in case of a failure the fallback wouldn't take long.

After a few days of no complaints I shut off the old servers. My monitoring went off, but not for the services I migrated. Just the host checks, as expected.

I keep a backup of the machines locally just to be sure. The servers were cancelled at the respective providers two weeks after the migration completed.

Stuff to remember when migrating webapps that email.

My webapplication sends email, so there are a few extra things that need to be set up and checked. I've listed them below for my own reference:

  • Reverse DNS for IPv4 Reverse DNS for IPv6 SMTP mailname must match hostname
  • and reverse name SPF records must include new IP DKIM keys need to be updated

A good service to test email sending in Mail Tester. If you forget the mailsettings your emails will go right into the SPAM folder of many users.

Conclusion

With a little bit of preparation and thought, a migration like this is easy to do and relatively painless. By keeping your servers up to date you are not only more secure, you will also have less hassle in the future.

I've seen many last-minute migrations gone wrong. For example, a big new security vulnerability affects your services. And particularly those important machines were the migration keeps getting deferred for a boatload of (stupid) reasons. And then you're forced to migrate them on an inconvinient time, not having a good prepared plan or test scheme and getting delays on your other projects as well.

Better to just do regular maintenance in your own pace.

I'd like to thank Digital Ocean again for the generous credit. It was the thing that pushed me over the edge to plan and do this consolidation and migration.

Tags: blog , dns , migrate , migration , php , ubuntu , website