Encrypt and decrypt files to public keys via the OpenSSL Command Line

10-11-2015 | Remy van Elst


Table of Contents


This small tutorial will show you how to use the openssl command line to encrypt and decrypt a file using a public key. We will first generate a random key, encrypt that random key against the public key of the other person and use that random key to encrypt the actual file with using symmetric encryption.

If you like this website and want to support it AND get $10 Digital Ocean credit (2 months free), use this link to order: https://www.digitalocean.com/?refcode=7435ae6b8212 (referral link).

Because of how the RSA algorithm works it is not possible to encrypt large files. If you create a key of n bits, then the file you want to encrypt must not larger than (n minus 11) bits. The most effective use of RSA crypto is to encrypt a random generated password, then encrypt the file with the password using symmetric crypto. If the file is larger then the key size the encryption command will fail:

RSA operation error:  020:error:0406D06E:rsa routines:RSA_padding_add_PKCS1_type_2:data too large for key size:.\crypto\rsa\rsa_pk1.c:151:

We generate a random file and use that as the key to encrypt the large file with symmetric crypto. That random file acts as the password so to say. We encrypt the large file with the small password file as password. Then we send the encrypted file and the encrypted key to the other party and then can decrypt the key with their public key, the use that key to decrypt the large file.

The following commands are relevant when you work with RSA keys:

  • openssl genrsa: Generates an RSA private keys.
  • openssl rsa: Manage RSA private keys (includes generating a public key from it).
  • openssl rsautl: Encrypt and decrypt files with RSA keys.

The key is just a string of random bytes. We use a base64 encoded string of 128 bytes, which is 175 characters. Since 175 characters is 1400 bits, even a small RSA key will be able to encrypt it.

Get the public key

Let the other party send you a certificate or their public key. If they send to a certificate you can extract the public key using this command:

openssl rsa -in certificate.pem -out publickey.pem -outform PEM -pubout

Generate the random password file

Use the following command to generate the random key:

openssl rand -base64 128 -out key.bin

Do this every time you encrypt a file. Use a new key every time!

Encrypt the file with the random key

Use the following command to encrypt the large file with the random key:

openssl enc -aes-256-cbc -salt -in largefile.pdf -out largefile.pdf.enc -pass file:./bin.key

The file size doesn't grows that much:

  $ ls -larth
  -rw-r--r-- 1 user group  40M Nov  9 21:14 Linux-Voice-Issue-020.pdf
  -rw-r--r-- 1 user group  40M Nov  9 22:03 Linux-Voice-Issue-020.pdf.enc

It's encrypted however:

$ file Linux-Voice-Issue-020.pdf
Linux-Voice-Issue-020.pdf: PDF document, version 1.4

$ file Linux-Voice-Issue-020.pdf.enc 
Linux-Voice-Issue-020.pdf.enc: data

Encrypt the random key with the public keyfile

Use the following command to encrypt the random keyfile with the other persons public key:

openssl rsautl -encrypt -inkey publickey.pem -pubin -in key.bin -out key.bin.enc

You can safely send the key.bin.enc and the largefile.pdf.enc to the other party.

You might want to sign the two files with your public key as well.

Decrypt the random key with our private key file

If you want to decrypt a file encrypted with this setup, use the following command with your privte key (beloning to the pubkey the random key was crypted to) to decrypt the random key:

openssl rsautl -decrypt -inkey privatekey.pem -in key.bin.enc -out key.bin

This will result in the decrypted random key we encrypted the file in.

Decrypt the large file with the random key

Once you have the random key, you can decrypt the encrypted file with the decrypted key:

openssl enc -d -aes-256-cbc -in largefile.pdf.enc -out largefile.pdf -pass file:./bin.key

This will result in the decrypted large file.


Tags: ca, certificate, decrypt, encrypt, openssl, pki, ssl, tls,