Enabling Self-Service DNS and Certificate Management in Kubernetes

Jason Schmidt

The ultimate goal of application development is, of course, to expose apps on the Internet. For a developer, Kubernetes simplifies this process to a degree by providing the Ingress controller as the mechanism for routing requests to the application. But not everything is as self‑service as you probably would like: you still need a record in the Domain Name System (DNS) to map the domain name for the app to the Ingress controller’s IP address and a TLS certificate to secure connections using HTTPS. In most organizations, you don’t own DNS or TLS yourself and so have to coordinate with the operational group (or groups!) that do.

Things aren’t necessarily any easier for operators. In most organizations the need to update DNS records is rare enough that procedures – both business rules and the actual technical steps – tend to be sparse or non‑existent. This means that when you need to add a DNS record you first need to find the documentation, ask a colleague, or (in a worst case) figure it out. You also need to ensure you’re in compliance with any corporate security rules and make sure that the ingress is tagged properly for the firewalls.

Fortunately, there is a way to make life easier for both developers and operators. In this post, we show how operators can configure a Kubernetes deployment to enable self‑service for developers to update DNS records and generate TLS certificates in a Kubernetes environment. By building out the infrastructure ahead of time, you can assure that all necessary business and technical requirements are being satisfied.

Overview and Prerequisites

With the solution in place, all a developer needs to do to expose an application to the Internet is create an Ingress controller following a supplied template that includes a fully qualified domain name (FQDN) within a domain managed by the Kubernetes installation. Kubernetes uses the template to allocate an IP address for the Ingress controller, create the DNS A record to map the FQDN to the IP address, and generate TLS certificates for the FQDN and add them to the Ingress controller. Cleanup is just as easy: when the Ingress is removed, the DNS records are cleaned up.

The solution leverages the following technologies (we provide installation and configuration instructions below):

• Kubernetes.
• ExternalDNS.
• cert-manager.
• Let’s Encrypt.
• NGINX Ingress Controller from F5 NGINX, based on either NGINX Open Source or NGINX Plus. The solution doesn’t work with the NGINX Ingress Controller maintained by the Kubernetes community. For more details on the differences between the two projects, see our blog.

Before configuring the solution, you need:

• A Kubernetes cloud installation with an egress (LoadBalancer) object. The solution uses Linode, but other cloud providers also work.
• A domain name hosted with Cloudflare, which we chose because it’s one of the supported DNS providers for cert-manager and supports ExternalDNS (in beta as of the time of writing). We strongly recommend that the domain not be used for production or any other critical purpose.
• Access to the Cloudflare API, which is included in the free tier.
• Helm for installing and deploying Kubernetes.
• kubectl as the command‑line interface for Kubernetes.
• Optionally, K9s, a well‑constructed tangible user interface (TUI) that provides a more structured way to interact with Kubernetes.

We also assume you have a basic understanding of Kubernetes (how to apply a manifest, use a Helm chart, and issue kubectl commands to view output and troubleshoot). Understanding the basic concepts of Let’s Encrypt is helpful but not required; for an overview, see our blog. You also don’t need to know how cert-manager works, but if you’re interested how it (and certificates in general) work with NGINX Ingress Controller, see my recent post, Automating Certificate Management in a Kubernetes Environment.

We have tested the solution on both macOS and Linux. We haven’t tested on Windows Subsystem for Linux version 2 (WSL2), but don’t foresee any issues.

Note: The solution is intended as a sample proof of concept, and not for production use. In particular, it does not incorporate all best practices for operation and security. For information on those topics, see the cert-manager and ExternalDNS documentation.

Deploying the Solution

Follow the steps in these sections to deploy the solution:

• Download Software
• Deploy NGINX Ingress Controller
• Deploy cert-manager
• Deploy ExternalDNS
• Deploy the Sample Application
• Validate the Solution

Download Software

1. Download your Cloudflare API Token.
2. Clone the NGINX Ingress Controller repository:[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
3. Verify that you can connect to the Kubernetes cluster.[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop

Deploy NGINX Ingress Controller

1. Using Helm, deploy NGINX Ingress Controller. Note that we are adding three non‑standard configuration options:[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
   • controller.enableCustomResources – Instructs Helm to install the custom resource definitions (CRDs) used to create the NGINX VirtualServer and VirtualServerRoute custom resources.
   • controller.enableCertManager – Configures NGINX Ingress Controller to communicate with cert-manager components.
   • controller.enableExternalDNS – Configures the Ingress Controller to communicate with ExternalDNS components.
2. Verify that NGINX Ingress Controller is running and note the value in the EXTERNAL-IP field – it’s the IP address for NGINX Ingress Controller (here, www.xxx.yyy.zzz). The output is spread across two lines for legibility.[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop

Deploy cert-manager

In the solution, cert-manager uses the DNS-01 challenge type when obtaining a TLS certificate, which requires the Cloudflare API token be provided during creation of the ClusterIssuer resource. In the solution, the API token is provided as a Kubernetes Secret.

1. Using Helm, deploy cert-manager:[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
2. Deploy the Cloudflare API token as a Kubernetes Secret, substituting it for <your-API-token>:[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
3. Create a ClusterIssuer object, specifying Cloudflare-api-token-secret (defined in the previous step) as the place to retrieve the token. If you wish, you can replace example-issuer in the metadata.name field (and example-issuer-account-key in the spec.acme.privateKeySecretRef.name field) with a different name.[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
4. Verify that the ClusterIssuer is deployed and ready (the value in the READY field is True).[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop

Deploy ExternalDNS

Like cert-manager, the ExternalDNS project requires a Cloudflare API Token to manage DNS. The same token can be used for both projects, but that is not required.

1. Create the ExternalDNS CRDs for NGINX Ingress Controller to enable integration between the projects.[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
2. Create the External DNS service (external-dns). Because the manifest is rather long, here we break it into two parts. The first part configures accounts, roles, and permissions:[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` propThe second part of the manifest creates the ExternalDNS deployment:[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
   • Creates a ServiceAccount object called external-dns to manage all write and update operations for managing DNS.
   • Creates a ClusterRole object (also called external-dns) that defines the required permissions.
   • Binds the ClusterRole to the ServiceAccount.
   • Creates a domain filter, which limits the scope of possible damage done by ExternalDNS as it manages domains. For example, you might specify the domain names of staging environments to prevent changes to production environments. In this example, we set domain-filter to example.com.
   • Sets the CF_API_TOKEN environment variable to your Cloudflare API Token. For <your-API-token>, substitute either the actual token or a Secret containing the token. In the latter case, you also need to project the Secret into the container using an environment variable.
   • Sets the FREE_TIER environment variable to "true" (appropriate unless you have a paid Cloudflare subscription).

Deploy the Sample Application

Use the standard NGINX Ingress Controller sample application called Cafe for testing purposes.

1. Deploy the Cafe application.[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
2. Deploy NGINX Ingress Controller for the Cafe application. Note the following settings:Note that the time it takes for this step to complete is highly dependent on the DNS provider, as Kubernetes is interacting with the provider’s DNS API.[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
   • kind: VirtualServer – We are using the NGINX VirtualServer custom resource, not the standard Kubernetes Ingress resource.
   • spec.host – Replace cafe.example.com with the name of the host you are deploying. The host must be within the domain being managed with ExternalDNS.
   • spec.tls.cert-manager.cluster-issuer – If you’ve been using the values specified in this post, this is example-issuer. If necessary, substitute the name you chose in Step 3 of Deploy cert‑manager.
   • spec.externalDNS.enable – The value true tells ExternalDNS to create a DNS A record.

Validate the Solution

1. Verify the DNS A record – in particular that in the ANSWER SECTION block the FQDN (here, cafe.example.com) is mapped to the correct IP address (www.xxx.yyy.zzz). [@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
2. Check that the certificate is valid (the value in the READY field is True).[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop
3. Verify that you can reach the application.[@portabletext/react] Unknown block type "codeBlock", specify a component for it in the `components.types` prop

What Happens When a Developer Deploys NGINX Ingress Controller

A lot happens under the covers once the solution is in place. The diagram shows what happens when a developer deploys the NGINX Ingress Controller with an NGINX VirtualServer custom resource. Note that some operational details are omitted.

1. Developer deploys a VirtualServer resource using the NGINX CRD
2. Kubernetes creates the VirtualServer using NGINX Ingress Controller
3. NGINX Ingress Controller calls ExternalDNS to create a DNS A record
4. ExternalDNS creates the A record in DNS
5. NGINX Ingress Controller calls cert-manager to request a TLS certificate
6. cert-manager adds a DNS record for use during the DNS-01 challenge
7. cert-manager contacts Let’s Encrypt to complete the challenge
8. Let’s Encrypt validates the challenge against DNS
9. Let’s Encrypt issues the TLS certificate
10. cert-manager provides the TLS certificate to NGINX Ingress Controller
11. NGINX Ingress Controller routes TLS‑secured external requests to the application pods

Troubleshooting

Given the complexity of Kubernetes along with the components we are using, it is difficult to provide a comprehensive troubleshooting guide. That said, there are some basic suggestions to help you determine the problem.

• Use the kubectl get and kubectl describe commands to validate the configuration of deployed objects.
• Use the kubectl logs <component> command to view log files for the various deployed components.
• Use K9s to inspect the installation; the software highlights problems in yellow or red (depending on severity) and provides an interface to access logs and details about objects.

If you are still having issues, please find us on the NGINXCommunity Slack and ask for help! We have a vibrant community and are always happy to work through issues.

To try the NGINX Ingress Controller based on NGINX Plus, start your 30-day free trial today or contact us to discuss your use cases.

About the Author

Jason Schmidt

More blogs by Jason Schmidt