Raymii.org

Table of Contents

• nova get-password
  • Manual decryption
• nova set-password
• Post encrypted passwords to the metadata service
  • Running the script at first boot

When you create images for an OpenStack Cloud you want to use 'cloud' features. Fancy term for automatic resizing of your instance disk, adding an SSH key, (re)setting passwords and executing scripts on first boot to configure your instance further. OpenStack provides the metadata service for instances, which supplies information for the instance, like its public IP, SSH public key that was provided and vendor or user provided data like scripts or information. The OpenStack metadata service allows an instance to post data to an endpoint wich can be retreived with the nova get-password command. It is meant to be an encrypted password (with the public SSH key) but it can be any plain text as well and it doesn't have to be the root password. In this guide I'll go over the scripts I use inside linux images to post a password to the metadata service and the nova commands such as set-password and get-password. That includes decrypting a password with an SSH key that is password-protected (Horizon and nova don't support that) and the nova set-password command, which sets the root password inside an instance when it has the qemu-guest-agent installed and running.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!

The first section of the guide goes over the nova get-password, nova set- password and the manual decryption of the password.

The second part of the article shows the script I use to set the password inside cloud instances.

nova get-password

The nova command line client supports the get-password command. This command only works when a password has been posted to the metadata service (by the instance).

$ nova help get-password
usage: nova get-password <server> [<private-key>]

Get the admin password for a server.

Positional arguments:
  <server>       Name or ID of server.
  <private-key>  Private key (used locally to decrypt password) (Optional).
                 When specified, the command displays the clear (decrypted) VM
                 password. When not specified, the ciphered VM password is
                 displayed.

Using it on a server that has a password set in the metadata service:

$ nova get-password $instance_uuid

Output:

bPLUcppp+MXbiO4kFMsIZ1zj4VbS/1kfafWTvqPBKr3B5Rk9z0eLotwVNDGCJkXYGmDmKca0q4hWdAt03N7QZ58DbgF24cvumgQIHddT5D14M98n85oiI3yPd0DCrEhvZQb5jwmYIWBBhqDCuyl+savwFNmVEn2cxdPleCZqvIN7BHs2HRVY7esHDcPiY1+Xdq5SMa92lcsJyn8LJRUUUuq0o0k75+4TTtEWEtClXiMyURIRDgDOpVDXBl2kgeU22a/e1rRKKOKg6CasVu7g0V9fmmLC/zXRmgTqfMwZlm0wDdq0X4jRUYzRNpxLarj6AXGYoSQT6moKx9ttcNL6JQ==

To decrypt it, supply the SSH private key with the command. It's a client side decryption:

$ nova get-password $instance_uuid ~/.ssh/cloud_nopass.priv

Output:

example_password

If your private key has a password, decrypting only works if you have ssh- agent running:

$ nova get-password $instance_uuid ~/.ssh/cloud_pass.priv

Output:

Enter pass phrase for ~/.ssh/cloud_pass.priv:
example_password

This way you can even use this feature if your SSH private key is stored on a hardware token like a NitroKey HSM (smartcard), OpenPGP token like the NitroKey Start or a YubiKey.

Using the Horizon dashboard you can also decrypt the password. This is also a client side operation and doesn't work if your private key is password protected.

Manual decryption

Using the openssl command line tools we can decrypt the encrypted password. It's base64 encoded so there is an extra step to decode the base64.

$ echo " bPLUcppp+MXbiO4kFMsIZ1zj4VbS/1kfafWTvqPBKr3B5Rk9z0eLotwVNDGCJkXYGmDmKca0q4hWdAt03N7QZ58DbgF24cvumgQIHddT5D14M98n85oiI3yPd0DCrEhvZQb5jwmYIWBBhqDCuyl+savwFNmVEn2cxdPleCZqvIN7BHs2HRVY7esHDcPiY1+Xdq5SMa92lcsJyn8LJRUUUuq0o0k75+4TTtEWEtClXiMyURIRDgDOpVDXBl2kgeU22a/e1rRKKOKg6CasVu7g0V9fmmLC/zXRmgTqfMwZlm0wDdq0X4jRUYzRNpxLarj6AXGYoSQT6moKx9ttcNL6JQ==" \
| openssl base64 -d \
| openssl rsautl -decrypt -inkey ~/.ssh/cloud_pass.priv -keyform PEM

Output:

Enter pass phrase for ~/.ssh/cloud_pass.priv:
example_password

nova set-password

The nova set-password command allows you to change the password in a server. But, only if the qemu-guest-agent is installed.

$ nova help set-password
usage: nova set-password <server>

Change the admin password for a server.

Positional arguments:
  <server>  Name or ID of server.

This can only be done by the owner of the instance or an OpenStack admin. It then allows you to login to the console for example.

By default there is no command line output if successfull.

The set-password command fails when a password is already set. Looking at the source code of the metadata api there is no command to remove a stored password. There is a nova API command but that is admin-only by default.

$ nova set-password $instance_uuid

Output:

New password:
Again:
ERROR (Conflict): Failed to set admin password on $instance_uuid because error setting admin password (HTTP 409) (Request-ID: req-f3f6feed-4171-45d1-ab16-28a6875688d8)

Following the nova source code we can see that in the case of Libvirt (KVM/QEMU) it calls virDomainSetUserPassword.

So it technically would be possible to (re)set the password again but currently it's not supported.

Post encrypted passwords to the metadata service

The metadata service allows posting a password. Technically this can be any sort of (unencrypted) data that will be made visible in Horizon or the nova get- password command. It is recommended to encrypt this password with the users provided public key.

Why? Because any code running inside the guest can access the password via the metadata service. Any web application can request the API URL and get the data:

$ curl http://169.254.169.254/openstack/latest/password

Output:

 bPLUcppp+MXbiO4kFMsIZ1zj4VbS/1kfafWTvqPBKr3B5Rk9z0eLotwVNDGCJkXYGmDmKca0q4hWdAt03N7QZ58DbgF24cvumgQIHddT5D14M98n85oiI3yPd0DCrEhvZQb5jwmYIWBBhqDCuyl+savwFNmVEn2cxdPleCZqvIN7BHs2HRVY7esHDcPiY1+Xdq5SMa92lcsJyn8LJRUUUuq0o0k75+4TTtEWEtClXiMyURIRDgDOpVDXBl2kgeU22a/e1rRKKOKg6CasVu7g0V9fmmLC/zXRmgTqfMwZlm0wDdq0X4jRUYzRNpxLarj6AXGYoSQT6moKx9ttcNL6JQ==

The password is also stored on the config_drive but that requires root privileges to mount. If I post unencrypted data to that endpoint:

DATA="this_is_an_example_raymii.org"
curl -s -X POST http://169.254.169.254/openstack/latest/password -d $DATA

Horizon (and the API) happily show that:

So that's why you want to encrypt it. What we're doing in the script is the following:

• Check if password was already set, and if so, exit
• Get public SSH key from metadata service.
• Convert the public key to a format usable with OpenSSL
• Generate random root password
• Set root password (and any other user like admin in the case of DirectAdmin)
• Encrypt the root password with the SSL keyfile
• Post the encrypted password to the metadata service.

There is almost no error handling, and a weary catch all in the curl (|| true), since this script is meant to run via /etc/rc.local (or via cloud- init) and a non exit 0 in rc.local can result in bootup failure.

#!/bin/bash
# Copyright (C) 2018 Remy van Elst.
# Author: Remy van Elst for https://www.cloudvps.com
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
#

# This script sets the root password
# to a random password generated on the instance
# and posts that if possible to the openstack
# metadata service for nova get-password.

logger "[CLOUDVPS] Started set root password and post to metadata service"

if [[ -f "/var/lib/cloud/instance/rootpassword-random" ]]; then
    # script already ran on this instance. 
    # /var/lib/cloud/instance/ is a symlink to /var/lib/cloud/instances/$instance_uuid
    # if user creates an image and deploys image, this must run again, that file will not exist
    exit 0
fi

# Centos 6 doens't support ssh-keygen's pcks8 option.
# it has OpenSSH < 5.6
if [[ -f "/etc/redhat-release" ]]; then
    NAME="$(awk '{ print $1 }' /etc/redhat-release)"
    VERSION="$(grep -Eo "[0-9]\.[0-9]" /etc/redhat-release | cut -d . -f 1)"
fi

# Two tmp files for the SSH and SSL pubkey
SSH_KEYFILE=$(mktemp)
SSL_KEYFILE=$(mktemp)

# get the ssh public key from the metadata server.
curl -s -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > $SSH_KEYFILE
if [[ $? != 0 ]]; then
  logger "[CLOUDVPS] Instance public SSH key not found on metadata service. Unable to set password"
  exit 0
fi

# generate a random password
# our images have haveged installed so should have enough entropy at boot.
RANDOM_PASSWORD="$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -c 30)"
if [[ -z ${RANDOM_PASSWORD} ]]; then
    logger "[CLOUDVPS] unable to generate random password."
    exit 0
fi

# set the root password to this random password
# add any other password changes like admin for DirectAdmin.
echo "root:${RANDOM_PASSWORD}" | chpasswd
if [[ -s "$SSH_KEYFILE" ]]; then
    # convert the ssh pubkey to an SSL keyfile so that we can use it to encrypt with OpenSSL
    if [[ ${VERSION} -eq 6 && ${NAME} == "CentOS" ]]; then
        logger "centos 6 doesnt support PKCS8 in ssh-keygen so we have a workaround."
        # our centos6 image has this file.
        # otherwise, https://gist.github.com/RaymiiOrg/7cd92ec7b9711fd4b9f6a1178c1bf04f
        python2 /etc/cloudvps/sshpub-to-rsa.py $SSH_KEYFILE > $SSL_KEYFILE
    else
        ssh-keygen -e -f $SSH_KEYFILE -m PKCS8 > $SSL_KEYFILE
    fi
    ENCRYPTED=$(echo "$RANDOM_PASSWORD" | openssl rsautl -encrypt -pubin -inkey $SSL_KEYFILE -keyform PEM | openssl base64 -e -A)
    # post encrypted blob to metadata service. Must return true otherwise instance might fail to boot.
    curl -s -X POST http://169.254.169.254/openstack/2013-04-04/password -d $ENCRYPTED 2>&1 > /dev/null || true
fi
# housekeeping
rm -rf $SSH_KEYFILE $SSL_KEYFILE

touch /var/lib/cloud/instance/rootpassword-random

exit 0

Running the script at first boot

This script can be ran at first boot of an instance using the good old trusty /etc/rc.local file. If you use a newer distribution with systemd, you can create a unit file to run this at boot.

You can also leverage cloud-init by creating a config file:

vim /etc/cloud/cloud.cfg.d/00-cloudvps-rootpasswd.cfg

Add the contents:

runcmd:
- /bin/bash /etc/cloudvps/rootpassword-to-metadata.sh

In my case the script is injected in the image on that location. You could also add a curl command if you use cloud-config (you do not control the cloud- init config but use user-data).

Tags: cloud , debian , encryption , metadata , nova , openstack , password , security , ssh , tutorials