Skip to main content

Raymii.org Logo (IEC resistor symbol)logo

Quis custodiet ipsos custodes?
Home | About | All pages | RSS Feed | Gopher

Use the Nitrokey HSM or SmartCard-HSM with mod_nss and Apache

Published: 21-06-2016 | Author: Remy van Elst | Text only version of this article


Table of Contents


This is a guide on using the Nitrokey HSM with mod_nss and the Apachewebserver. The HSM allows you to store the private key for a SSL certificateinside the HSM (instead of on the filesystem), so that it can never leave thedevice and thus never be stolen.

The guide covers the installation and configuration of mod_nss, coupling theHSM to NSS, generating the keys and configuring Apache, and last but not leastwe also do some benchmarks on Apache with the HSM and different key sizes.

If you like this article, consider sponsoring me by trying out a Digital OceanVPS. With this link you'll get $100 credit for 60 days). (referral link)

Introduction

The Nitrokey HSM in a sealed package

The Nitrokey HSM is an open hardware and open software device. It is a USBversion of the SmartCard-HSM. Both the SmartCard-HSM as the NitrokeyHSM have sources available and are fully supported by the OpenSCproject.

If you are new to the NitroKey HSM/SmartCard HSM, please also read my gettingstarted article. It explains what the HSM is, how to set it up and how touse it with OpenSSH for example.

I have multiple articles on this nice device, so make sure to read theothers as well.

The SmartCard-HSM

In this entire tutorial I will be using slot 1 and ID 1 of the Nitrokey HSM. TheUser PIN is 648219 as in all the examples. It is very important to read mytutorial to get started with the Nitrokey HSM first as that explains a lotof concepts and usage. The guide was tested on both Arch Linux and Ubuntu 16.04,both with the available latest version of Apache 2.4.

The default mod_ssl of Apache has no support for the PKCS#11 protocol. PKCS#11is the protocol HSM's use to communicate their crypto operations. Instead of thesoftware using the private keyfiles directly themselves, they ask the HSM thingslike "hey, here's some data, sign/encrypt that please with key X", which thenthe HSM does that and returns the result of the requested action.

mod_ssldoes support the SSLCryptoDevice, but the documentation statesthat it only enables use of a cryptographic hardware accelerator board tooffload some of the SSL processing overhead. The Nitrokey HSM uses theOpenSC projects code, and they do have a openssl engine wrapper butI could not find any documentation and small tests resulted in nothing.

mod_nss however does support PKCS#11. The website of mod_nss tells usthat it is an SSL provider derived from the mod_ssl module for the Apache webserver that uses the Network Security Services (NSS) libraries. For the mostpart there is a 1-1 mapping between the capabilities of mod_nss and mod_ssl.

The management of mod_nss is different from the usual with mod_ssl. You needto use the nss tools like certutil and modutil for certificate management,no more simple files.

This is a good guide on certutil and here is more documentation oncertutil. The man page also helps. Here is a list with copy pasteactions for the NSS suite.

mod_nss installation

In Ubuntu you can install the mod_nss Apache module with the followingcommand:

apt-get install libapache2-mod-nss apache2 libpkcs11-helper1 libengine-pkcs11-openssl libnss3-tools opensc-pkcs11

You enable the module with the following command:

a2enmod nss

Disable mod_ssl with the folllowing command:

a2dismod ssl

On other Linux distro's you might need to manually change config files. On myArch installation I had to add the following to /etc/http/conf/http.conf:

Include conf/extra/nss.conf

I also had to install mod_nss from the AUR:

pacaur -Sy mod_nss

NSS and Certutil config

NSS requires a certificate database. We also need to tell NSS to load thepkcs11 module.

Create the database folder:

mkdir -p /etc/nss/db/

Create a new NSS database:

certutil -N -d /etc/nss/db/

It will ask you for a password. Enter a secure one, or, when testing, just presRETURN twice. Output:

Enter a password which will be used to encrypt your keys.The password should be at least 8 characters long,and should contain at least one non-alphabetic character.Enter new password: Re-enter password: 

Add the HSM module to NSS:

modutil -add pkcs11 -libfile /usr/lib/opensc-pkcs11.so -dbdir /etc/nss/db/

On Ubuntu the file is located here:

/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

Output:

WARNING: Performing this operation while the browser is running could causecorruption of your security databases. If the browser is currently running,you should exit browser before continuing this operation. Type 'q <enter>' to abort, or <enter> to continue: Module "pkcs11" added to database.

Enable the pkcs11 module:

modutil -enable pkcs11 -dbdir /etc/nss/db/

Output:

WARNING: Performing this operation while the browser is running could causecorruption of your security databases. If the browser is currently running,you should exit browser before continuing this operation. Type 'q <enter>' to abort, or <enter> to continue: Slot "Virtual hotplug slot" enabled.Slot "Nitrokey Nitrokey HSM (010000000000000000000000) 00 00" enabled.

HSM Key generation and NSS usage

Generate a 2048 bit RSA key in the HSM:

pkcs11-tool --module opensc-pkcs11.so --login --pin 648219 --keypairgen --key-type rsa:2048 --id 1 --label "httpd hsm"

Generate a new certificate with that key. This one is self signed. See mytutorial on the Nitrokey if you want to generate a CSR and what to put inhsm.conf:

OPENSSL_CONF=./hsm.conf openssl req -engine pkcs11 -keyform engine -new -key 1:1 -nodes -days 3560 -x509 -sha256 -out "rsahsm.tst.raymii.org.pem" -subj "/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=IT Dept/CN=rsahsm.tst.raymii.org"

Do note that you can also generate a CSR and submit that to your certificateprovider.

Convert the PEM certificate into DER format, since DER is what the HSM uses:

openssl x509 -in rsahsm.tst.raymii.org.pem -out rsahsm.tst.raymii.org.der -outform der

Load the DER certificate into the HSM together with the key:

pkcs11-tool --module opensc-pkcs11.so --login --pin 648219 --write-object rsahsm.tst.raymii.org.der --type cert --id 1 --label 'httpdcert'

Output:

Using slot 1 with a present token (0x1)Created certificate:Certificate Object, type = X.509 cert  label:      httpdcert  ID:         01

Find out the correct names for the HSM. The tooling is very picky about naming,it took me quite a while to figure it out:

modutil -dbdir /etc/nss/db/ -list pkcs11

We're looking for the Token Name. Example Output:

-----------------------------------------------------------Name: pkcs11Library file: /usr/lib/opensc-pkcs11.soManufacturer: OpenSC (www.opensc-project.org) Description: Smart card PKCS#11 API          PKCS #11 Version 2.20Library Version: 0.0Cipher Enable Flags: NoneDefault Mechanism Flags: NoneSlot: Nitrokey Nitrokey HSM (010000000000000000000000) 00 00Slot Mechanism Flags: NoneManufacturer: OpenSC (www.opensc-project.org) Type: HardwareVersion Number: 0.0Firmware Version: 0.0Status: EnabledToken Name: SmartCard-HSM (UserPIN)         Token Manufacturer: www.CardContact.de              Token Model: PKCS#15 emulatedToken Serial Number: DENK0100186     Token Version: 24.13Token Firmware Version: 2.0Access: NOT Write ProtectedLogin Type: Login requiredUser Pin: Initialized

In this case the Token name is: SmartCard-HSM (UserPIN). Now we want to knowthe exact certificate name. Find that using the following command:

certutil -d /etc/nss/db -h 'SmartCard-HSM (UserPIN)' -L 

(-L is list all certificates. -h indicats the specific token we want touse.)

Certificate Nickname    Trust Attributes                        SSL,S/MIME,JAR/XPIEnter Password or Pin for "SmartCard-HSM (UserPIN)": 648219SmartCard-HSM (UserPIN):httpdcert                          u,u,u

In this case the name of the certificate is: SmartCard-HSM(UserPIN):httpdcert.

View the PEM certificate:

certutil -d /etc/nss/db -h 'SmartCard-HSM (UserPIN)' -L -n "SmartCard-HSM (UserPIN):httpdcert" -a

(-n names the specific certificate we want to list/show. -a enables ASCIIoutput.)

Output:

Enter Password or Pin for "SmartCard-HSM (UserPIN)": 648219-----BEGIN CERTIFICATE-----MIIDcjCCAloCCQCxS7T+z2D3ADANBgkqhkiG9w0BAQsFADB7MQswCQYDVQQGEwJOTDEVMBMGA1UECAwMWnVpZCBIb2xsYW5kMRIwEAYDVQQHDAlSb3R0ZXJkYW0xGjAYBgNVBAoMEVNwYXJrbGluZyBOZXR3b3JrMRAwDgYDVQQLDAdJVCBEZXB0MRMwEQYD[...]-----END CERTIFICATE-----

(-n names a specific certificate.)

If you omit the -a option you will get the regular text output:

certutil -d /etc/nss/db -h 'SmartCard-HSM (UserPIN)' -L -n "SmartCard-HSM (UserPIN):httpdcert" 

Output:

Enter Password or Pin for "SmartCard-HSM (UserPIN)": 648219Certificate:    Data:        Version: 1 (0x0)        Serial Number:            00:b1:4b:b4:fe:cf:60:f7:00        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption        Issuer: "CN=rsahsm.tst.raymii.org,OU=IT Dept,O=Sparkling Network,L=Rotterdam,ST=            Zuid Holland,C=NL"        Validity:            Not Before: Mon Jun 20 17:13:55 2016            Not After : Fri Mar 20 17:13:55 2026        Subject: "CN=rsahsm.tst.raymii.org,OU=IT Dept,O=Sparkling Network,L=Rotterdam,ST=Zuid Holland,C=NL"[...]

Check the certificate chain with certutil:

certutil -d /etc/nss/db -h SmartCard-HSM -O -n "SmartCard-HSM (UserPIN):httpdcert"

(-O prints the certificate chain.)

Example Output:

Enter Password or Pin for "SmartCard-HSM (UserPIN)": 648219"SmartCard-HSM (UserPIN):httpdcert" [CN=rsahsm.tst.raymii.org,OU=IT Dept,O=Sparkling Network,L=Rotterdam,ST=Zuid Holland,C=NL]

If you use a self-signed certificate it will be just one line. If you have anofficial certificate it will show the full chain:

"COMODORSAAddTrustCA" [CN=COMODO RSA Certification Authority,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB]  "COMODORSADomainValidationSecureServerCA" [CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB]    "SmartCard-HSM (UserPIN):webrsa2048" [CN=rsa2048hsm.tst.raymii.org,OU=PositiveSSL,OU=Domain Control Validated]

If it doesn't show the full chain you might need to import root certificates inthe NSS database.

Import Root CA Chains

To make sure the correct certificate chain is sent we need to import rootcertificates in our NSS database. If you use a self signed certificate then youwon't need to do this, but if you use an actual signed certificate by a CA youneed to do this.

The curl project has a good Root CA Chain. Download it:

mkdir sslcd sslwget https://curl.haxx.se/ca/cacert.pem

We need to remove all the comments from the file and just keep the chains. Usethe following command to do so:

sed -n '/-----BEGIN/,/-----END/p' cacert.pem > chains.pem

Now split the file into all seperate files:

csplit -f cert- chains.pem '/-----BEGIN CERTIFICATE-----/' '{*}'

That creates a lot of files named cert-###.

Import the seperate chains with certutil:

for file in ./cert-*; do   echo $file;   certutil -d /etc/nss/db -A -n "${file}" -t "CT,," -a -i ./${file};  sleep 1 done

The -t "CT,," flags make sure the certificates are trusted. (+ T - Trusted CAto issue client certificates, C - Trusted CA to issue server certificates). The-A flag adds an existing certificate to the DB. The sleep is added to makesure everything is imported and the HSM is not overloaded.

If you just need to import one (PEM) certificate use the following command, thisexample is for the Addtrust External Root CA:

certutil -d /etc/nss/db -A -n 'ADDTUST' -t "CT,," -a -i addtrust.pem 

Apache2 and mod_nss config

Place the PIN for the slot in this file:

cat /etc/nss/db/pin.txtSmartCard-HSM (UserPIN):648219

The format is: tokenname:pin.

Also make sure to give the correct permissions to the NSS db:

chown -R http:root /etc/nss/db/

If http is not your apache user, change it to the correct username. Otherwiseyour apache error log will fill up with errors like:

[:error] [pid 4873:tid 139841160738688] NSS_Initialize failed. Certificate database: /etc/nss/db.[:error] [pid 4873:tid 139841160738688] SSL Library Error: -8038 SEC_ERROR_NOT_INITIALIZED

Make sure you have disabled mod_ssl in the Apache configuration and haveenabled mod_nss. This is my nss.conf file:

# grep -v -e "#" -e '^$' /etc/httpd/conf/extra/nss.confLoadModule nss_module modules/libmodnss.soListen 8443AddType application/x-x509-ca-cert .crtAddType application/x-pkcs7-crl    .crlNSSPassPhraseDialog file:/etc/nss/db/pin.txtNSSPassPhraseHelper /usr/bin/nss_pcacheNSSEnforceValidCerts offNSSSessionCacheTimeout 100NSSSession3CacheTimeout 86400NSSRandomSeed startup builtinNSSRenegotiation offNSSRequireSafeNegotiation off<VirtualHost _default_:8443>  DocumentRoot "/etc/httpd/htdocs"  ServerName rsahsm.tst.raymii.org:8443  ErrorLog /etc/httpd/logs/error_log  TransferLog /etc/httpd/logs/access_log  LogLevel info  NSSEngine on  NSSCipherSuite ALL  NSSProtocol TLSv1.0,TLSv1.1,TLSv1.2  NSSNickname "SmartCard-HSM (UserPIN):httpdcert"  NSSCertificateDatabase /etc/nss/db  <Files ~ "\.(cgi|shtml|phtml|php3?)$">      NSSOptions +StdEnvVars  </Files>  <Directory "/var/www/cgi-bin">      NSSOptions +StdEnvVars  </Directory></VirtualHost>  

On Arch I had to comment out NSSSessionCacheSize and replaceNSSPassPhraseHelper /usr/libexec/nss_pcache with NSSPassPhraseHelper/usr/bin/nss_pcache. For the testing certificate I also had to addNSSEnforceValidCerts off. The rest is just the example config.

Make sure the certificate name is correct here:

NSSNickname "SmartCard-HSM (UserPIN):httpdcert"

A manual test shows that it works:

curl -kI https://127.0.0.1:8443

Output:

HTTP/1.1 200 OKDate: Mon, 20 Jun 2016 19:06:03 GMTServer: Apache/2.4.20 (Unix) mod_nss/2.4.20 NSS/3.23 Basic ECCLast-Modified: Mon, 20 Jun 2016 19:05:33 GMTETag: "f-535ba6383fe65"Accept-Ranges: bytesContent-Length: 15Content-Type: text/html

OpenSSL also agrees:

echo | openssl s_client  -servername raymii.org -connect 127.0.0.1:8443

Output:

CONNECTED(00000003)depth=0 C = NL, ST = Zuid Holland, L = Rotterdam, O = Sparkling Network, OU = IT Dept, CN = raymii.orgverify error:num=18:self signed certificateverify return:1depth=0 C = NL, ST = Zuid Holland, L = Rotterdam, O = Sparkling Network, OU = IT Dept, CN = raymii.orgverify return:1---Certificate chain 0 s:/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=IT Dept/CN=raymii.org   i:/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=IT Dept/CN=raymii.org---

And below is a screenshot of a signed certificate from Comodo (viaXolphin), with the private key on the HSM:

Just to show that it will work with all major browsers. No trust issues orexceptions here.

Here's the command line to show that the chain is also included:

echo | openssl s_client  -servername raymii.org -connect rsa2048hsm.tst.raymii.org:8443

Output:

CONNECTED(00000003)depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authorityverify return:1depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CAverify return:1depth=0 OU = Domain Control Validated, OU = PositiveSSL, CN = rsa2048hsm.tst.raymii.orgverify return:1---Certificate chain 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=rsa2048hsm.tst.raymii.org   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority---

If you like this article, consider sponsoring me by trying out a Digital OceanVPS. With this link you'll get a $5 VPS for 2 months free (as in, you get $10credit). (referral link)

Benchmarking 2048 bit RSA

The Nitrokey HSM is not a fast HSM. I've worked with HSM's that are capable ofmultiple hundreds of signs per second, and this HSM is not one that can do that.But, as we know, that's not the intended target for the device. It's meant forsafe and secure key storage, for example to encrypt files or protect SSH orS/MIME keys. I did a few siege benchmarks, and as you can see it is quiteslow.

The page that was being loaded is a plain text file containing only the textJeej it works!.

A siege test with 5 concurrent users, 30 seconds:

siege -c5 -d5 -t30S https://127.0.0.1:8443

Output:

** SIEGE 4.0.1** Preparing 5 concurrent users for battle.The server is now under siege...HTTP/1.1 200     2.95 secs:      15 bytes ==> GET  /HTTP/1.1 200     5.73 secs:      15 bytes ==> GET  /[...]Lifting the server siege...Transactions:             35 hitsAvailability:             100.00 %Elapsed time:             29.39 secsData transferred:         0.00 MBResponse time:            1.95 secsTransaction rate:         1.19 trans/secThroughput:               0.00 MB/secConcurrency:              2.32Successful transactions:  35Failed transactions:      0Longest transaction:      4.76Shortest transaction:     0.74

That's quite reasonable.

A siege test with 10 concurrent users, 30 seconds:

siege -c10 -d5 -t30S https://127.0.0.1:8443

Output:

Transactions:             39 hitsAvailability:             100.00 %Elapsed time:             29.54 secsData transferred:         0.00 MBResponse time:            4.64 secsTransaction rate:         1.32 trans/secThroughput:               0.00 MB/secConcurrency:              6.13Successful transactions:  39Failed transactions:      0Longest transaction:      17.85Shortest transaction:     0.75

It starts taking longer.

Comparing the above siege to a test against raymii.org, with all the assetsand such loaded:

siege -c10 -d5 -t30S https://raymii.org/s/

Output:

Transactions:             258 hitsAvailability:             100.00 %Elapsed time:             29.15 secsData transferred:         4.42 MBResponse time:            0.73 secsTransaction rate:         8.85 trans/secThroughput:               0.15 MB/secConcurrency:              6.45Successful transactions:  258Failed transactions:      0Longest transaction:      2.25Shortest transaction:     0.18

At 20 connections failures start to occur:

siege -c20 -d5 -t30S https://127.0.0.1:8443

Output:

[...][error] Failed to make an SSL connection: 5HTTP/1.1 200     3.48 secs:      15 bytes ==> GET  /[error] Failed to make an SSL connection: 5Transactions:             26 hitsAvailability:             72.22 %Elapsed time:             29.65 secsData transferred:         0.00 MBResponse time:            8.85 secsTransaction rate:         0.88 trans/secThroughput:               0.00 MB/secConcurrency:              7.76Successful transactions:  26Failed transactions:      10Longest transaction:      19.97Shortest transaction:     0.00

And at 60 concurrent connections in benchmark mode most fail:

  siege -c60 -b -t30S https://127.0.0.1:8443

Output:

Transactions:             17 hitsAvailability:             48.57 %Elapsed time:             29.04 secsData transferred:         0.00 MBResponse time:            11.28 secsTransaction rate:         0.59 trans/secThroughput:               0.00 MB/secConcurrency:              6.60Successful transactions:  17Failed transactions:      18Longest transaction:      20.67Shortest transaction:     0.00

Comparing it to raymii.org:

  siege -c60 -b -t30S https://raymii.org/s/

Output:

Transactions:             539 hitsAvailability:             100.00 %Elapsed time:             29.80 secsData transferred:         7.46 MBResponse time:            3.12 secsTransaction rate:         18.09 trans/secThroughput:               0.25 MB/secConcurrency:              56.36Successful transactions:  539Failed transactions:      0Longest transaction:      4.87Shortest transaction:     0.27

As you can see, the HSM and mod_nss isn't that fast, but it is very secure.The 1 second request overhead comes from the fact that every HTTP requestrequires HSM access.

Benchmarking EC keys

Let's try it with an EC key to see if that makes any difference. Generate an ECkey in the HSM, using a different ID:

pkcs11-tool --module opensc-pkcs11.so --login -pin 648219 --keypairgen --key-type EC:prime256v1 --id 3 --label "web ecc"

Output:

Using slot 1 with a present token (0x1)Logging in to "SmartCard-HSM (UserPIN)".Key pair generated:Private Key Object; EC  label:      web ecc  ID:         03  Usage:      sign, derivePublic Key Object; EC  EC_POINT 256 bits  EC_POINT:   044104a898bc5e554f28adc3a89df89dd074a1b169e220f1c8050498925aa056f22d69fb63fed3a7cb287b7e2db259762142ca503a3a31b8c53d75944eeb49751ffa1f  EC_PARAMS:  06082a8648ce3d030107  label:      web ecc  ID:         03  Usage:      verify

Generate a self signed certificate from it. See above to find out what to put inthe hsm.conf file. Do note that id 3 is used:

OPENSSL_CONF=./hsm.conf openssl req -engine pkcs11 -keyform engine -new -key 1:3 -nodes -days 3560 -x509 -sha256 -out "echsm.tst.raymii.org.ec.pem" -subj "/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=IT Dept/CN=echsm.tst.raymii.org"

Convert the PEM file into DER, since the HSM needs a DER file:

openssl x509 -in echsm.tst.raymii.org.pem -out echsm.tst.raymii.org.der -outform der

Load the DER certificate into the HSM together with the key:

pkcs11-tool --module opensc-pkcs11.so --login --pin 648219 --write-object echsm.tst.raymii.org.der --type cert --id 3 --label 'webec'

I gave the certificate object a different label. If we don't do that thencertutil can't differentiate between them.

There should be two certificates now:

certutil -d /etc/nss/db -h 'SmartCard-HSM (UserPIN)' -L

Output:

Certificate Nickname      Trust Attributes                          SSL,S/MIME,JAR/XPIEnter Password or Pin for "SmartCard-HSM (UserPIN)": 648219SmartCard-HSM (UserPIN):httpdcert         u,u,uSmartCard-HSM (UserPIN):webec             u,u,u

In the Apache configuration, change the name of the certificate from:

NSSNickname "SmartCard-HSM (UserPIN):httpdcert"

To:

NSSNickname "SmartCard-HSM (UserPIN):webec"

Also change the Ciphersuite to allow ECC ciphers:

NSSCipherSuite +aes_128_sha_256,+aes_256_sha_256,+ecdhe_ecdsa_aes_128_gcm_sha_256,+ecdhe_ecdsa_aes_128_sha,+ecdhe_ecdsa_aes_256_gcm_sha_384,+ecdhe_ecdsa_aes_256_sha,+ecdhe_rsa_aes_128_gcm_sha_256,+ecdhe_rsa_aes_128_sha,+ecdhe_rsa_aes_256_gcm_sha_384,+ecdhe_rsa_aes_256_sha,+rsa_aes_128_gcm_sha_256,+rsa_aes_128_sha,+rsa_aes_256_gcm_sha_384,+rsa_aes_256_sha

Restart the webserver and you should be good to go. Except for, in my case, whenconnecting it fails and this is in the error_log:

[:info] [pid 25561:tid 140498883753856] Using nickname SmartCard-HSM (UserPIN):webec.[:info] [pid 25561:tid 140498711348992] Connection to child 64 established (server echsm.tst.raymii.org:8443, client 127.0.0.1)[:info] [pid 25561:tid 140498711348992] SSL input filter read failed.[:error] [pid 25561:tid 140498711348992] SSL Library Error: -12192 Peer reports failure of signature verification or key exchange

Playing around with different settings, ciphersuites and protocols didn't help.Time for a bugreport.

Benchmarking a 1024 bit RSA key

A product folder mentions that the HSM should be a bit faster with smallerkeys. Now do note that it's not recommended to use a 1024 bit key in production.But, this is just a benchmark test.

Generate the key:

pkcs11-tool --module opensc-pkcs11.so --login --pin 648219 --keypairgen --key-type rsa:1024 --id 4 --label "httpd hsm"

Output:

Using slot 1 with a present token (0x1)Please enter User PIN: Key pair generated:Private Key Object; RSA   label:      httpd hsm  ID:         04  Usage:      decrypt, sign, unwrapPublic Key Object; RSA 1024 bits  label:      httpd hsm  ID:         04  Usage:      encrypt, verify, wrap

Generate a certificate. See above to find out what to put in the hsm.conffile. Do note that id 4 is used:

OPENSSL_CONF=./hsm.conf openssl req -engine pkcs11 -keyform engine -new -key 1:4 -nodes -days 3560 -x509 -sha256 -out "rsa1024hsm.tst.raymii.org.pem" -subj "/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=IT Dept/CN=rsa1024hsm.tst.raymii.org"

Convert the certificate to DER:

openssl x509 -in rsa1024hsm.tst.raymii.org.pem -out rsa1024hsm.tst.raymii.org.der -outform der

Load the DER certificate into the HSM together with the key:

pkcs11-tool --module opensc-pkcs11.so --login --pin 648219 --write-object rsa1024hsm.tst.raymii.org.der --type cert --id 4 --label 'webrsa1024'

I gave the certificate object a different label. If we don't do that thencertutil can't differentiate between them.

There should be three certificates now:

certutil -d /etc/nss/db -h 'SmartCard-HSM (UserPIN)' -L

Output:

Certificate Nickname    Trust Attributes                        SSL,S/MIME,JAR/XPIEnter Password or Pin for "SmartCard-HSM (UserPIN)":SmartCard-HSM (UserPIN):httpdcert        u,u,uSmartCard-HSM (UserPIN):webrsa1024       u,u,uSmartCard-HSM (UserPIN):webec            u,u,u

In the Apache configuration, change the name of the certificate from:

NSSNickname "SmartCard-HSM (UserPIN):httpdcert"

To:

NSSNickname "SmartCard-HSM (UserPIN):webrsa1024"

Restart the server and you're good to go. No errors this time. Lets see how thebenchmarks compare.

5 concurrent users:

siege -c5 -d5 -t30S https://127.0.0.1:8443

Output:

Transactions:             51 hitsAvailability:             100.00 %Elapsed time:             29.74 secsData transferred:         0.00 MBResponse time:            0.54 secsTransaction rate:         1.71 trans/secThroughput:               0.00 MB/secConcurrency:              0.93Successful transactions:  51Failed transactions:      0Longest transaction:      4.24Shortest transaction:     0.23

Response time is a lot faster.

10 concurrent users:

siege -c10 -d5 -t30S https://127.0.0.1:8443

Output:

Transactions:             103 hitsAvailability:             100.00 %Elapsed time:             29.91 secsData transferred:         0.00 MBResponse time:            0.60 secsTransaction rate:         3.44 trans/secThroughput:               0.00 MB/secConcurrency:              2.05Successful transactions:  103Failed transactions:      0Longest transaction:      2.83Shortest transaction:     0.23

Faster, and holds up pretty well.

20 concurrent users:

siege -c20 -d5 -t30S https://127.0.0.1:8443

Output:

Transactions:             124 hitsAvailability:             100.00 %Elapsed time:             29.03 secsData transferred:         0.00 MBResponse time:            2.47 secsTransaction rate:         4.27 trans/secThroughput:               0.00 MB/secConcurrency:              10.54Successful transactions:  124Failed transactions:      0Longest transaction:      7.53Shortest transaction:     0.23

No failed connections as we saw with the 2048 bit key, but the response time ishigher.

60 benchmark mode:

siege -c60 -b -t30S https://127.0.0.1:8443

Output:

Transactions:             75 hitsAvailability:             74.26 %Elapsed time:             29.54 secsData transferred:         0.00 MBResponse time:            6.13 secsTransaction rate:         2.54 trans/secThroughput:               0.00 MB/secConcurrency:              15.57Successful transactions:  75Failed transactions:      26Longest transaction:      20.54Shortest transaction:     0.25

And as expected that results in failed connections. But, with a 2048 bit key wehad 48% Availability, now we have 74%.

As we can see the HSM is faster with a smaller keysize, which is not a strangething. I'm sad that the EC keys didn't work, since EC is a lot less CPUintensive.

Clustering

If you have multiple HSM's you can use a tool like haproxy in TCP mode to loadbalance connections. Using the HSM's DKEK backup and restore functionality youcan have the same private key on multiple devices. I only have one HSM, but myguess is that with more HSM's the speed increase will be noticable.

Tags: apache, articles, cryptoki, hsm, mod_nss, nitrokey, nitrokey-hsm, openssl, pkcs11, safenet, smartcard, smartcard-hsm