Secure Distribution of SSL Private Keys with NGINX

NGINX | April 02, 2019

Owen GarrettSr. Director, Product Management

This blog post describes several methods for securely distributing the SSL private keys that NGINX uses when hosting SSL‑encrypted websites. It explains:

The standard approach for configuring SSL with NGINX, and the potential security limitations
How to encrypt the keys using passwords that are stored separately from the NGINX configuration
How to distribute the encryption passwords securely, avoiding disk storage, and then revoke them when needed

For many deployments, the standard approach is sufficient. The two more sophisticated approaches discussed in this post block other ways an attacker can obtain SSL private keys. We’ll also look at a couple more techniques in follow‑up posts:

Using third‑party secret stores such as HashiCorp Vault to securely distribute passwords
Automating the provisioning of certificates from Vault to NGINX Plus’s key‑value store, so that private key material is never stored on disk

The approaches presented in this post apply to users who need to manage their own keys and create their own secure key‑distribution strategy. They are not necessary for users who are running NGINX in environments that already integrate with a secret store, such as Kubernetes.

This post applies to both NGINX Open Source and NGINX Plus. For ease of reading, we’ll refer to NGINX throughout.

Editor – This post is the first in a series about securing SSL private keys in NGINX. See also the other posts in the series:

Protecting SSL Private Keys in NGINX with HashiCorp Vault – Discusses the use of HashiCorp Vault or a hardware security module (HSM) to automate and scale a key‑protection system
Using the NGINX Plus Key-Value Store to Secure Ephemeral SSL Keys from HashiCorp Vault – Explains how to bypass on‑disk storage of SSL keys by generating ephemeral SSL keys from HashiCorp Vault and storing them in the NGINX Plus in‑memory key‑value store

Why Protect the SSL Private Key?

SSL/TLS is used to authenticate, encrypt, and verify the integrity of network transactions. Websites authenticate themselves using a public certificate signed by a Certificate Authority (CA), and demonstrate they own the certificate by performing calculations using the corresponding private key (which must be kept secret).

If the private key is compromised (disclosed to another entity), there are two main risks.

Risk 1: Impersonation. An attacker who has the private key can intercept network traffic and then mount a man-in-the-middle (MITM) attack. This attack captures and decrypts all traffic, perhaps also modifying it, without clients or the website being aware.
Risk 2: Decryption. An attacker who has the private key and has recorded the network traffic is then able to decrypt the network traffic offline. Note that this attack cannot be used against connections that use a Perfect Forward Secrecy (PFS) cipher.

If the private key is compromised, your only recourse is to contact the CA and request that your certificate be revoked; you must then rely on clients to check and honor the revocation status.

In addition, it is good practice to use certificates with short expiry times (for example, Let’s Encrypt certificates expire after 90 days). Shortly before a certificate expires, you need to generate a new private key and obtain a new certificate from the CA. This reduces your exposure in the event the private key is compromised.

The NGINX Security Boundary

Which people and processes can access SSL private keys in NGINX?

First of all, any user who gains root access to the server running NGINX is able to read and use all resources that NGINX itself uses. For example, there are known methods to extract the SSL private key from the memory of a running process.

Therefore, no matter how the private key is stored and distributed, it’s not possible to protect the private key from an attacker with root privileges on the host server.

Next, any user who can modify and commit NGINX configuration can use that power in many ways – to open proxy access to internal services, to bypass authentication measures, etc. He or she can modify NGINX configuration to obtain root access (or equivalent) to the server, although tools like SELinux and AppArmor help mitigate against that possibility.

Therefore, it is generally not possible to protect the private key from an attacker who can modify and commit NGINX configuration.

Fortunately, any competent organization has sound security processes to make it difficult for an attacker to gain root privileges or to modify NGINX configuration.

However, there are two other ways that a less privileged attacker might obtain access to the private key:

A user might have a legitimate reason to need to view the NGINX configuration, or might obtain access to a configuration database or backup. NGINX private keys are typically stored in the configuration.
A user might obtain access to the filesystem of the NGINX server, perhaps through a hypervisor or system backup. Any data stored on the filesystem, including the private key material, is potentially accessible.

The processes described in this document seal these two attack methods.

Standard NGINX Configuration

We begin by reviewing what a typical NGINX configuration with SSL/TLS looks like:

The SSL public certificate (a.dev0.crt) and private key (a.dev0.key) are stored in the filesystem, at /etc/nginx/ssl/. The private key is only read by the NGINX master process, which typically runs as root, so you can set the strictest possible access permissions on it:

The private key must be available at all times; the NGINX master process reads it whenever the NGINX software starts, configuration is reloaded, or a syntax check is performed (nginx -t).

For more information on configuring SSL/TLS, see the NGINX Plus Admin Guide.

Security Implications of the Standard Configuration

As noted above, the SSL private key can be read by an attacker who gains root access to the running container, virtual machine, or server that is running the NGINX software.

Encrypting SSL Private Keys

NGINX supports encrypted private keys, using secure algorithms such as AES256:

When you then start NGINX, or reload or test NGINX configuration, NGINX requests the decryption password interactively:

Using an SSL Password File

Entering passwords interactively is inconvenient and difficult to automate, but you can configure NGINX to use a list of passwords stored in a separate file named by the ssl_password_file directive. When NGINX needs to read a private key, it attempts to decrypt the key using each of the passwords in the file in turn. If none of the passwords is valid, NGINX refuses to start.

The ssl_password_file must be distributed separately from the configuration, and be readable only by the root user. You can regard it as an authorization token that is placed on trusted servers. NGINX can only decrypt the private keys when it is running on a server with the authorization token.

Security Implications of Encrypted Keys in a Separate File

This method reduces the attack surface by making the NGINX configuration alone useless to an attacker. The attacker must also obtain the contents of the ssl_password_file.

If an attacker does gain root access to the filesystem where the ssl_password_file is stored (for example, from a backup or through the host system), he or she can read the file and use the passwords to decrypt SSL private keys.

You can reduce this risk by storing the ssl_password_file on a RAM disk or tmpfs. This storage is generally less accessible to an external attacker (for example, it’s cleared when the server is restarted) and can be excluded from system backups. You need to ensure that the password file is initialized on system boot.

Distributing SSL Password Lists More Securely

The process below describes a more secure way to distribute lists of SSL passwords, from a central distribution point.

Whenever NGINX needs to decrypt an SSL key, it queries the central distribution point and uses the passwords without ever storing them on the local disk. To authenticate itself with the central password server, the NGINX instance uses a token which you can revoke at any time to cut off access to the passwords.

Creating a Central Password Distribution Point

Begin by creating a password distribution point (PDP). For this simple implementation, we’re using an HTTPS service to deliver the password list, authenticated by username and password:

You can then enable or revoke access by adding or removing authentication tokens at the PDP as needed. You can implement the password distribution server using a web server such as NGINX, and use whatever kind of authentication tokens is appropriate.

Next, we need to set up NGINX to retrieve the passwords from the PDP. We start by creating a shell script called connector.sh with the following contents:

The script needs to run as a background process, invoked as follows:

The connector attaches to the specified local path (/var/run/nginx/ssl_passwords), and you use the ssl_password_file directive to configure NGINX to access that path:

Test the connector by reading from the connector path:

Verify that NGINX can read the password and decrypt the SSL keys:

You can use the central PDP approach to securely distribute any resource that NGINX normally reads from disk, for example individual private keys or other sensitive data.

Security Implications of a PDP

This solution has several benefits compared to storing SSL passwords on disk:

The SSL passwords are never stored on the server’s filesystem, so an attacker who has access to the filesystem cannot access them directly.
Passwords are distributed from a central access point, making monitoring and auditing easier to perform.
Individual servers’ access can be controlled centrally. For example, once a server is decommissioned, you revoke its access token.

Note that a user who has access to the filesystem can potentially extract the credentials used to access the PDP. It is important to revoke these credentials when they are no longer needed.

Summary

There are many ways to protect SSL private keys from disclosure, with increasing levels of security and complexity:

For the large majority of organizations, it is sufficient to restrict access to the environments running NGINX so that unauthorized users cannot gain root access and cannot look at NGINX configuration.
For some environments, it might not be possible to fully restrict access to NGINX configuration, so an SSL password file can be used.
In limited cases, organizations might wish to ensure that keys and passwords are never stored on disk. The password distribution point process illustrates a proof of concept for this solution.

The other posts in this series describe additional steps you can take to secure SSL keys:

Protecting SSL Private Keys in NGINX with HashiCorp Vault – Shows how to set up HashiCorp Vault as the PDP, providing scalable, secure distribution of secrets with fine‑grained access control. Also discusses use of a hardware security module (HSM) to store private keys remotely, exposing an API that can be used on‑demand to perform a key operation.
Using the NGINX Plus Key-Value Store to Secure Ephemeral SSL Keys from HashiCorp Vault – Shows how to bypass on‑disk storage of SSL keys by generating ephemeral SSL keys from HashiCorp Vault and storing them in the NGINX Plus in‑memory key‑value store.

Try NGINX Plus for yourself – start your free 30-day trial today or contact us to discuss your use cases.