Skip to main content

Raymii.org Logo (IEC resistor symbol)logo

Quis custodiet ipsos custodes?
Home | About | All pages | RSS Feed | Gopher

OpenStack: Quick and automatic instance snapshot backup and restore (and before an apt upgrade) with nova backup

Published: 20-12-2016 | Author: Remy van Elst | Text only version of this article


Table of Contents


openStack Logo

This is a guide that shows you how to create OpenStack instance snapshotsautomatically, quick and easy. This allows you to create a full backup of theentire instance. This guide has a script that makes creating snapshots from anOpenStack VM automatic via cron. The script uses the nova backup function,therefore it also has retention and rotation of the backups. It also features anoption to create a snapshot before every apt action, upgrade/install/remove.This way, you can easily restore from the snapshot when something goes wrongafter an upgrade. Snapshots are very usefull to restore the entire instance toan earlier state. Do note that this is not the same as a file based backup, youcan't select a few files to restore, it's all or nothing.

OpenStack Compute (Nova) has a feature that allows you to create a snapshot of a(running) instance, namely nova image-create. This uses the underlyinghypervisor to create an instant snapshot of the virtual machine. In the case ofKVM/qemu this is a qemu-snapshot. This image is then stored in the Glance imagestore, allowing you to create a new VM based on the image, thus restoring thebackup or creating a clone. If you use floating IP's, it's a matter of assigningthe floating IP to the new VM and you're all set.

If you like this article, consider sponsoring me by trying out a Digital OceanVPS. With this link you'll get $100 credit for 60 days). (referral link)

Snapshot before an apt install/upgrade

Snapshots are a fast and easy way to clone or restore an instance to an earlierpoint. Let's say you're doing an upgrade of your application and it goeshorribly wrong. Rollback fails and it's the middle of the night. Do you want togo and restore all the files and databases from the backup of 18 hours earlier,manually? Or do you want to revert the entire thing to a full snapshot frombefore the upgrade? If I could choose, I would know.

Do note that I still recommend you create regular backups. They allow for a morefine-grained way to restore files/folders/databases and have things likecompression and incremental/differential support, thus backing up only thethings that have changed since the last backup. This snapshot method is just acopy of the enitre disk, taking up more space.

Nova Backup

The compute part of OpenStack, nova, has multiple options to create images(snapshots) of an instance. One is with the nova image-create command. Thiscreates an instant snapshot and uploads that to Glance, the image storagecomponent.

The second option is nova backup. This is almost the same as image-create,with the addition of rotation of the backup snapshots. There are two extraparameters, backup_type and rotation. The documentation is a bit sparseon this, but the api and the source code provide more detail.

The backups are normal glance images. Based on the backup_type a set number ofimages is saved. Once there are more images of that type then specified inrotation, the oldest is deleted. If we create a backup every day withbackup_type set to daily and rotation set to 7, on the 8'th day, theoldest image will be automatically removed (on creation of the new backup).

There is no scheduling in OpenStack, you need to create these backups yourself,from cron for example. The script allows you to specify a backup_type and arotation. The example sets up the following scheme:

The snapshots after an apt upgrade, if configured, have backup_type set toapt and rotation set to 7. If no options are provided, the backup_typeis set to manual and the rotation to 7. If just a backup_type is provided,the rotation defaults to 7.

We do need to schedule this in cron on the VM itself. You could also adapt thescript to run on a management VM to backup all the VM's in an account.

Below is a screenshot of an example daily snapshot setup. We see 7 snapshotscreated with rotation set to 7:

The 8'th snapshot is created:

Only after the 8'th snapshot has succeeded, the 1st snapshot is removedautomatically:

If for whatever reason the new snapshot fails, the old one isn't deleted.

In OpenStack the operator can configure where the Glance images are stored. Inthe case of my prefered OpenStack provider (CloudVPS, non-affiliate link)this is their object store (swift). I happen to know that that object store isreplicated over three data centers, making the snapshot storage extremelyredundant.

The script works on both CentOS and Ubuntu, the apt-upgrade snapshot part onlyon Ubuntu. Yum has no such pre-hooks as far as I know of. On CentOS you'rebetter of just executing it manually before a yum update.

OpenStack command line tools and dependencies

The script runs on the server you want to snapshot itself. It's required to havethe command line tools installed and to have a credentials file. First installthe dependencies:

# Ubuntuapt-get install dmidecode wget python-pip# CentOSyum install dmidecode wget python-pip

Recent Ubuntu releases have the OpenStack command line tools packaged:

apt-get install python-keystoneclient python-glanceclient python-novaclient

On older versions and CentOS you can use pip, the Python Pypi package tool:

pip install python-keystoneclient python-glanceclient python-novaclient

You also need a credentials file (computerc). Below you'll find an examplefile, fill it in with the credentials your OpenStack provider sent you.

nano /root/.openstack_snapshotrcexport OS_AUTH_URL="https://identity.stack.cloudvps.com/v2.0"export OS_TENANT_NAME="PROJECT_UUID"export OS_TENANT_ID="PROJECT_UUID"export OS_USERNAME="USERNAME"export OS_PASSWORD="PASSWORD"

Test if the file is correct by sourcing it in the shell and issueing a command:

source /root/.openstack_snapshotrcnova credentials

Example output:

+------------------+-------------------------+| User Credentials | Value                   |+------------------+-------------------------+| id               | f3[...]11               || name             | image-test-1            || roles            | [{"name": "compute"}]   || roles_links      | []                      || username         | image-test-1            |+------------------+-------------------------+

Install the script

The script is on my Github and can be downloaded with wget:

wget -O "/usr/local/bin/create-snapshot.sh" "https://github.com/RaymiiOrg/openstack-nova-snapshot/blob/master/create_snapshot.sh"

Set the executable permission on the file:

chmod +x /usr/local/bin/create-snapshot.sh

The script is simple and accepts two parameters, namely the backup_type androtation. They are not required, by default it will give the manual backuptype and a rotation of 7. For your daily backup with 7 snapshots saved you canuse the following:

/usr/local/bin/create-snapshot.sh daily 7

For your apt snapshots you should give another type:

 /usr/local/bin/create-snapshot.sh apt 7

The auto-removal is based on the backup_type field, so make sure you give thecorrect one. Otherwise you might just delete other backups.

If the credentials file is set up you should now be able to create a snapshot:

/usr/local/bin/create-snapshot.sh 

Output:

INFO: Start OpenStack snapshot creation.SUCCESS: Backup image created and pending upload.

The state of the server will change:

Do note that while the image is being uploaded you cannot create a new snapshotor do other management actions like reboot or shut down, unless you do that viathe VM itself (console).

The image will be queued first and then start uploading:

Backup snapshot schedule

Now that the snapshots are working we can use cron to schedule them As discussedearlier, we have a schedule for a daily, weekly, monthly and yearly backup. Youcould just limit it to a daily schedule if you want to save space. Or for animportant machine, create a snapshot every two hours during office hours, changeit to suit your requirement.

Place the following in a file in /etc/cron.d/

# Daily snapshot1 1 * * * root /bin/bash /usr/local/bin/create-snapshot.sh daily 7# Weekly snapshot1 1 * * 0 root /bin/bash /usr/local/bin/create-snapshot.sh weekly 4# Montlhy snapshot1 1 1 * * root /bin/bash /usr/local/bin/create-snapshot.sh monthly 12# Yearly snapshots1 1 1 1 * root /bin/bash /usr/local/bin/create-snapshot.sh yearly 2

This enables the schedule as described above. If you for example want to have asnapshot every two hours between office hours you can use the following cronsyntax:

# Snapshot past every 2nd hour from 6 through 181 6-18/2 * * * root /bin/bash /usr/local/bin/create-snapshot.sh officehours 12

Apt hook

apt, the package manager, has a hook named pre-invoke (and post-invoke).This hook is called before before (or after) apt calls dpkg. via.

We can use this hook to create the snapshot before an apt action, say an apt-get upgrade, apt-get install or apt-get remove. Create the following file:

nano /etc/apt/apt.conf.d/00openstacksnapshot

Insert the following:

DPkg::Pre-Invoke {"/bin/bash /usr/local/bin/create-snapshot.sh apt 5";};

This will create a snapshot before the actual action is done, allowing you torevert to a snapshot after a (failed) upgrade. For example when the kernel isnot working or other issues.

Snapshot before an apt install/upgrade

Try it out by installing a package. You should now see the output as in theabove screenshot added to the apt command output.

Restore

The most important thing of a backup is of course that you can restore it whenneeded. We can do this via the command line client with the nova boot command.This creates a new instance based on the snapshot. Do note that you need tosource the credentials file in your shell:

source .computercnova boot --image "SNAPSHOT_NAME" --poll --flavor "FLAVOR_NAME" --availability-zone NL1 --nic net-id=NETWORK_ID --key "SSH_KEY" "VM_NAME"

The syntax is specific to CloudVPS but you get the gist. If you have a floatingIP you can deassign it from the old server and assign it to the new one andyou're all set.

If you don't have a floating IP or other reasons to keep the same VM, you canuse the nova rebuild command. This will reinstall the VM with the snapshotimage, OVERWRITING ALL DATA ON THE INSTANCE!:

nova rebuild --poll "INSTANCE_UUID" "SNAPSHOT_IMAGE_UUID"

Afterwards your snapshot image should be deployed on the instance.

Remember to test this often. I test my backups and restores at least once amonth.

Tags: backup, cloud, debian, glance, keystone, objectstore, openstack, python, swift, tutorials, ubuntu