NGINX Tutorial: How to Securely Manage Secrets in Containers

NGINX | March 14, 2023

Robert HaynesTechnical Marketing Manager

This post is one of four tutorials that help you put into practice concepts from Microservices March 2023: Start Delivering Microservices:

How to Deploy and Configure Microservices
How to Securely Manage Secrets in Containers (this post)
How to Use GitHub Actions to Automate Microservices Canary Releases
How to Use OpenTelemetry Tracing to Understand Your Microservices

Many of your microservices need secrets to operate securely. Examples of secrets include the private key for an SSL/TLS certificate, an API key to authenticate to another service, or an SSH key for remote login. Proper secrets management requires strictly limiting the contexts where secrets are used to only the places they need to be and preventing secrets from being accessed except when needed. But this practice is often skipped in the rush of application development. The result? Improper secrets management is a common cause of information leakage and exploits.

Tutorial Overview

In this tutorial, we show how to safely distribute and use a JSON Web Token (JWT) which a client container uses to access a service. In the four challenges in this tutorial, you experiment with four different methods for managing secrets, to learn not only how to manage secrets correctly in your containers but also about methods that are inadequate:

Hardcode secrets in your app
Pass secrets as environment variables
Use local secrets
Use a secrets manager

Although this tutorial uses a JWT as a sample secret, the techniques apply to anything for containers that you need to keep secret, such as database credentials, SSL private keys, and other API keys.

The tutorial leverages two main software components:

API server – A container running NGINX Open Source and some basic NGINX JavaScript code that extracts a claim from the JWT and returns a value from one of the claims or, if no claim is present, an error message
API client – A container running very simple Python code that simply makes a GET request to the API server

Watch this video for a demo of the tutorial in action.

The easiest way to do this tutorial is to register for Microservices March and use the browser‑based lab that’s provided. This post provides instructions for running the tutorial in your own environment.

Prerequisites and Set Up

Prerequisites

To complete the tutorial in your own environment, you need:

A Linux/Unix‑compatible environment
Basic familiarity with the Linux command line
A text editor like nano or vim
Docker (including Docker Compose and Docker Engine Swarm)
curl (already installed on most systems)
git (already installed on most systems)

Notes:

The tutorial makes use of a test server listening on port 80. If you’re already using port 80, use the ‑p flag to set a different value for the test server when you start it with the docker run command. Then include the :<port_number> suffix on localhost in the curl commands.
Throughout the tutorial the prompt on the Linux command line is omitted, to make it easier to cut and paste the commands into your terminal. The tilde (~) represents your home directory.

Set Up

In this section you clone the tutorial repo, start the authentication server, and send test requests with and without a token.

Clone the Tutorial Repo

In your home directory, create the microservices-march directory and clone the GitHub repository into it. (You can also use a different directory name and adapt the instructions accordingly.) The repo includes config files and separate versions of the API client application that use different methods to obtain secrets.
Display the secret. It’s a signed JWT, commonly used to authenticate API clients to servers.

While there are a few ways to use this token for authentication, in this tutorial the API client app passes it to the authentication server using the OAuth 2.0 Bearer Token Authorization framework. That involves prefixing the JWT with Authorization: Bearer as in this example:

Build and Start the Authentication Server

Change to the authentication server directory:
Build the Docker image for the authentication server (note the final period):
Start the authentication server and confirm that it’s running (the output is spread over multiple lines for legibility):

Test the Authentication Server

Verify that the authentication server rejects a request that doesn’t include the JWT, returning 401 Authorization Required:
Provide the JWT using the Authorization header. The 200 OK return code indicates the API client app authenticated successfully.

Challenge 1: Hardcode Secrets in Your App (Not!)

Before you begin this challenge, let’s be clear: hardcoding secrets into your app is a terrible idea! You’ll see how anyone with access to the container image can easily find and extract hardcoded credentials.

In this challenge, you copy the code for the API client app into the build directory, build and run the app, and extract the secret.

Copy the API Client App

The app_versions subdirectory of the apiclient directory contains different versions of the simple API client app for the four challenges, each slightly more secure than the previous one (see Tutorial Overview for more information).

Change to the API client directory:
Copy the app for this challenge – the one with a hardcoded secret – to the working directory:
Take a look at the app:The code simply makes a request to a local host and prints out either a success message or failure code.The request adds the Authorization header on this line:Do you notice anything else? Perhaps a hardcoded JWT? We will get to that in a minute. First let’s build and run the app.

Build and Run the API Client App

We’re using the docker compose command along with a Docker Compose YAML file – this makes it a little easier to understand what’s going on.

(Notice that in Step 2 of the previous section you renamed the Python file for the API client app that’s specific to Challenge 1 (very_bad_hard_code.py) to app.py. You’ll also do this in the other three challenges. Using app.py each time simplifies logistics because you don’t need to change the Dockerfile. It does mean that you need to include the ‑build argument on the docker compose command to force a rebuild of the container each time.)

The docker compose command builds the container, starts the application, makes a single API request, and then shuts down the container, while displaying the results of the API call on the console.

The 200 Success code on the second-to-last line of the output indicates that authentication succeeded. The apiKey1 value is further confirmation, because it shows the auth server was able to decode the claim of that name in the JWT:

So hardcoded credentials worked correctly for our API client app – not surprising. But is it secure? Maybe so, since the container runs this script just once before it exits and doesn’t have a shell?

In fact – no, not secure at all.

Retrieve the Secret from the Container Image

Hardcoding credentials leaves them open to inspection by anyone who can access the container image, because extracting the filesystem of a container is a trivial exercise.

Create the extract directory and change to it:
List basic information about the container images. The --format flag makes the output more readable (and the output is spread across two lines here for the same reason):
Extract the most recent apiclient image as a .tar file. For <container_ID>, substitute the value from the CONTAINER ID field in the output above (11b73106fdf8 in this tutorial):It takes a few seconds to create the api.tar archive, which includes the container’s entire file system. One approach to finding secrets is to extract the whole archive and parse it, but as it turns out there is a shortcut for finding what’s likely to be interesting – displaying the container’s history with the docker history command. (This shortcut is especially handy because it also works for containers that you find on Docker Hub or another container registry and thus might not have the Dockerfile, but only the container image).
Display the history of the container: The lines of output are in reverse chronological order. They show that the working directory was set to /usr/app/src, then the file of Python code for the app was copied in and run. It doesn’t take a great detective to deduce that the core codebase of this container is in /usr/app/src/app.py, and as such that’s a likely location for credentials.
Armed with that knowledge, extract just that file:
Display the file’s contents and, just like that, we have gained access to the “secure” JWT:

Challenge 2: Pass Secrets as Environment Variables (Again, No!)

If you completed Unit 1 of Microservices March 2023 (Apply the Twelve‑Factor App to Microservices Architectures), you’re familiar with using environment variables to pass configuration data to containers. If you missed it, never fear – it’s available on demand after you register.

In this challenge, you pass secrets as environment variables. Like the method from Challenge 1, we don’t recommend this one! It’s not as bad as hardcoding secrets, but as you’ll see it has some weaknesses.

There are four ways to pass environment variables to a container:

Use the ENV statement in a Dockerfile to do variable substitution (set the variable for all images built). For example:
Use the ‑e flag on the docker run command. For example:
Use the environment key in a Docker Compose YAML file.
Use a .env file containing the variables.

In this challenge, you use an environment variable to set the JWT and examine the container to see if the JWT is exposed.

Pass an Environment Variable

Change back to the API client directory:
Copy the app for this challenge – the one that uses environment variables – to the working directory, overwriting the app.py file from Challenge 1:
Take a look at the app. In the relevant lines of output, the secret (JWT) is read as an environment variable in the local container:
As explained above, there’s a choice of ways to get the environment variable into the container. For consistency, we’re sticking with Docker Compose. Display the contents of the Docker Compose YAML file, which uses the environment key to set the JWT environment variable:
Run the app without setting the environment variable. The 401 Unauthorized code on the second-to-last line of the output confirms that authentication failed because the API client app didn’t pass the JWT:
For simplicity, set the environment variable locally. It’s fine to do that at this point in the tutorial, since it’s not the security issue of concern right now:
Run the container again. Now the test succeeds, with the same message as in Challenge 1:

So at least now the base image doesn’t contain the secret and we can pass it at run time, which is safer. But there is still a problem.

Examine the Container

Display information about the container images to get the container ID for the API client app (the output is spread across two lines for legibility):
Inspect the container for the API client app. For <container_ID>, substitute the value from the CONTAINER ID field in the output above (here 6b20c75830df).The docker inspect command lets you inspect all launched containers, whether they are currently running or not. And that’s the problem – even though the container is not running, the output exposes the JWT in the Env array, insecurely saved in the container config.

Challenge 3: Use Local Secrets

By now you’ve learned that hardcoding secrets and using environment variables is not as safe as you (or your security team) need it to be.

To improve security, you can try using local Docker secrets to store sensitive information. Again, this isn’t the gold‑standard method, but it’s good to understand how it works. Even if you don’t use Docker in production, the important takeaway is how you can make it difficult to extract the secret from a container.

In Docker, secrets are exposed to a container via the file system mount /run/secrets/ where there’s a separate file containing the value of each secret.

In this challenge you pass a locally stored secret to the container using Docker Compose, then verify that the secret isn’t visible in the container when this method is used.

Pass a Locally Stored Secret to the Container

As you might expect by now, you start by changing to the apiclient directory:
Copy the app for this challenge – the one that uses secrets from within a container – to the working directory, overwriting the app.py file from Challenge 2:
Take a look at the Python code, which reads the JWT value from the /run/secrets/jot file. (And yes, we should probably be checking that the file only has one line. Maybe in Microservices March 2024?)OK, so how are we going to create this secret? The answer is in the docker-compose.secrets.yml file.
Take a look at the Docker Compose file, where the secret file is defined in the secrets section and then referenced by the apiclient service:

Verify the Secret Isn’t Visible in the Container

Run the app. Because we’ve made the JWT accessible within the container, authentication succeeds with the now‑familiar message:
Display information about the container images, noting the container ID for the API client app (for sample output, see Step 1 in Examine the Container from Challenge 2):
Inspect the container for the API client app. For <container_ID>, substitute the value from the CONTAINER ID field in the output from the previous step. Unlike the output in Step 2 of Examine the Container, there is no JWT= line at the start of the Env section: So far, so good, but our secret is in the container filesystem at /run/secrets/jot. Maybe we can extract it from there using the same method as in Retrieve the Secret from the Container Image from Challenge 1.
Change to the extract directory (which you created during Challenge 1) and export the container into a tar archive:
Look for the secrets file in the tar file:Uh oh, the file with the JWT in it is visible. Didn’t we say embedding secrets in the container was “secure”? Are things just as bad as in Challenge 1?
Let’s see – extract the secrets file from the tar file and look at its contents:Good news! There’s no output from the cat command, meaning the run/secrets/jot file in the container filesystem is empty – no secret to see in there! Even if there is a secrets artifact in our container, Docker is smart enough to not store any sensitive data in the container.

That said, even though this container configuration is secure, it has one shortcoming. It depends on the existence of a file called token1.jwt in the local filesystem when you run the container. If you rename the file, an attempt to restart the container fails. (You can try this yourself by renaming [not deleting!] token1.jwt and running the docker compose command from Step 1 again.)

So we are halfway there: the container uses secrets in a way that protects them from easy compromise, but the secret is still unprotected on the host. You don’t want secrets stored unencrypted in a plain text file. It’s time to bring in a secrets management tool.

Challenge 4: Use a Secrets Manager

A secrets manager helps you manage, retrieve, and rotate secrets throughout their lifecycles. There are a lot of secrets managers to choose from and they all fulfill similar a similar purpose:

Store secrets securely
Control access
Distribute them at run time
Enable secret rotation

Your options for secrets management include:

Cloud providers all have a secrets service (for example AWS Secrets Manager, Google Cloud Platform’s Secret Manager, and Microsoft Azure’s Key Vault)
Kubernetes has the Secret object
Hashicorp Vault is a popular cross‑platform secrets manager
OpenShift has a secrets management service
Docker Swarm has a secrets service

For simplicity, this challenge uses Docker Swarm, but the principles are the same for many secrets managers.

In this challenge, you create a secret in Docker, copy over the secret and API client code, deploy the container, see if you can extract the secret, and rotate the secret.

Configure a Docker Secret

As is tradition by now, change to the apiclient directory:
Initialize Docker Swarm:
Create a secret and store it in token1.jwt:
Display information about the secret. Notice that the secret value (the JWT) is not itself displayed:

Use a Docker Secret

Using the Docker secret in the API client application code is exactly the same as using a locally created secret – you can read it from the /run/secrets/ filesystem. All you need to do is change the secret qualifier in your Docker Compose YAML file.

Take a look at the Docker Compose YAML file. Notice the value true in the external field, indicating we are using a Docker Swarm secret:So, we can expect this Compose file to work with our existing API client application code. Well, almost. While Docker Swarm (or any other container orchestration platform) brings a lot of extra value, it does bring some additional complexity.Since docker compose does not work with external secrets, we’re going to have to use some Docker Swarm commands, docker stack deploy in particular. Docker Stack hides the console output, so we have to write the output to a log and then inspect the log.To make things easier, we also use a continuous while True loop to keep the container running.
Copy the app for this challenge – the one that uses a secrets manager – to the working directory, overwriting the app.py file from Challenge 3. Displaying the contents of app.py, we see that the code is nearly identical to the code for Challenge 3. The only difference is the addition of the while True loop:

Deploy the Container and Check the Logs

Build the container (in previous challenges Docker Compose took care of this):
Deploy the container:
List the running containers, noting the container ID for secretstack_apiclient (as before, the output is spread across multiple lines for readability).
Display the Docker log file; for <container_ID>, substitute the value from the CONTAINER ID field in the output from the previous step (here, 20d0c83a8b86). The log file shows a series of success messages, because we added the while True loop to the application code. Press Ctrl+c to exit the command.

Try to Access the Secret

We know that no sensitive environment variables are set (but you can always check with the docker inspect command as in Step 2 of Examine the Container in Challenge 2).

From Challenge 3 we also know that /run/secrets/jot file is empty, but you can check:

Success! You can’t get the secret from the container, nor read it directly from the Docker secret.

Rotate the Secret

Of course, with the right privileges we can create a service and configure it to read the secret into the log or set it as an environment variable. In addition, you might have noticed that communication between our API client and server is unencrypted (plain text).

So leakage of secrets is still possible with almost any secrets management system. One way to limit the possibility of resulting damage is to rotate (replace) secrets regularly.

With Docker Swarm, you can only delete and then re‑create secrets (Kubernetes allows dynamic update of secrets). You also can’t delete secrets attached to running services.

List the running services:
Delete the secretstack_apiclient service.
Delete the secret and re‑create it with a new token:
Re‑create the service:
Look up the container ID for apiclient (for sample output, see Step 3 in Deploy the Container and Check the Logs):
Display the Docker log file, which shows a series of success messages. For <container_ID>, substitute the value from the CONTAINER ID field in the output from the previous step. Press Ctrl+c to exit the command.

See the change from apiKey1 to apiKey2? You’ve rotated the secret.

In this tutorial, the API server still accepts both JWTs, but in a production environment you can deprecate older JWTs by requiring certain values for claims in the JWT or checking the expiration dates of JWTs.

Note also that if you’re using a secrets system that allows your secret to be updated, your code needs to reread the secret frequently so as to pick up new secret values.

Clean Up

To clean up the objects you created in this tutorial:

Delete the secretstack_apiclient service.
Delete the secret.
Leave the swarm (assuming you created a swarm just for this tutorial).
Kill the running apiserver container.
Delete unwanted containers by listing and then deleting them.
Delete any unwanted container images by listing and deleting them.

Next Steps

You can use this blog to implement the tutorial in your own environment or try it out in our browser‑based lab (register here). To learn more on the topic of exposing Kubernetes services, follow along with the other activities in Unit 2: Microservices Secrets Management 101.

To learn more about production‑grade JWT authentication with NGINX Plus, check out our documentation and read Authenticating API Clients with JWT and NGINX Plus on our blog.

Featured Blog Posts

Current trends in cloud-native technologies, platform engineering, and AI

Full lifecycle management of app and API delivery with F5 NGINX One

Simplifying Modern Application Delivery: Nine Months of F5 NGINX One

Tags: Security, F5 NGINX, Tech

About the Author