If you are using dynamic modules, you might see an error like this during the upgrade to a new NGINX or NGINX Plus release:
The most likely reason is that you haven’t upgraded the specified dynamic module (.so file):
- If you are running NGINX Open Source, dynamic modules must be compiled against the version you are upgrading to.
- If you are running NGINX Plus, dynamic modules must be compiled against the NGINX Open Source version that corresponds to the NGINX Plus release you are upgrading to.
For the sake of brevity we’ll refer to NGINX Plus only from now on.
Don’t Panic!
Rest assured that this error message does not indicate that your NGINX Plus server is down. NGINX Plus upgrades are seamless, so the old version continues running when an upgrade fails. However, the upgrade is incomplete and your system is in an inconsistent state:
- NGINX Plus processes in memory are running the old version
- All of the files on disk relating to NGINX Plus correspond to the new version
- The dynamic module mentioned in the error message is compiled against the old version
Do not manually restart nginx, or reboot the system. Doing so will cause downtime because new NGINX Plus processes cannot start while there are dynamic modules out of sync with the new NGINX Plus release.
Note that the error message relates only to the first incompatible module found during the upgrade process, so it is important to find all load_module directives in your configuration and ensure that every module is compiled against the appropriate NGINX version. Correct versions of all NGINX‑authored and certified third‑party dynamic modules are provided for each NGINX Plus release<, so you need to recompile only non‑certified modules.This article describes how to safely complete the upgrade process.
What Went Wrong?
NGINX 1.11.5 and NGINX Plus R11 introduced the ability to compile dynamic modules against NGINX Open Source and load them into NGINX Plus. This binary compatibility requires the module and NGINX Plus to share the same base open source version. Upgrading NGINX Plus without first installing dynamic modules built against the corresponding NGINX Open Source version fails because of this version mismatch.
If your dynamic module was supplied by a third‑party vendor then you need to contact the vendor for a new version, matched to the NGINX version for the NGINX Plus release you intend to upgrade to. If your dynamic module was compiled from source (and you have access to the sources) then continue reading.
The Road to Recovery
Step 0: Prepare the Build Environment
We strongly recommend that you compile dynamic modules on a separate system, which we refer to here as the “build environment”. Doing so minimizes the risk and complexity of the system where you are running NGINX Plus with the dynamic module (we refer to this as the “production environment”). The build environment must have the same operating system and version as the production environment; in addition, the following components must be installed:
- UnZip utility
- Compiler and make utility
- Perl‑compatible regular expressions library (development files)
- Zlib compression libraries (development files)
To ensure your build environment has these prerequisites installed, run the following command.
- For Ubuntu/Debian:
- For CentOS/RHEL/Oracle Linux:
Step 1: Obtain NGINX Open Source
- Working in the production environment, run the following command to identify the NGINX Open Source version that corresponds to the running NGINX Plus release. It’s highlighted in orange in this output: NGINX 1.11.10, which corresponds to NGINX Plus R12.
- Working in the build environment, download the sources for the appropriate NGINX Open Source version.
Step 2: Obtain the Dynamic Module Sources
Copy the source code for the dynamic module to the build directory of your choice. Here we copy a sample NGINX “hello world” module from its GitHub repository.
Step 3: Compile the Dynamic Module
- Compile the dynamic module by first running the NGINX
configurescript with the--with-compatargument to make the dynamic module binary‑compatible with NGINX Plus. Then runmakemodulesto compile just the module. - Verify that the build process has created the dynamic module as a .so file in the objs subdirectory.
- Create a copy of the module file with the NGINX Open Source version in the filename. This makes it simpler to manage multiple versions of the dynamic module in the production environment.
Step 4: Copy the Dynamic Module to the Production Environment
Because we created the dynamic module with the NGINX Open Source version in the filename, we can safely copy it to the production environment without affecting operation.
Step 5: Swap Over the Dynamic Module
Working in the production environment, swap in the new .so file to replace the current one. This is safe to do at this point because dynamic modules are loaded into memory only when nginx restarts or the configuration is reloaded.
Step 6: Complete the Upgrade of NGINX Plus
- Run the following command to test that the module version is correct and that NGINX Plus is ready to complete the upgrade process.
- Complete the upgrade process so that NGINX Plus is using the new release and dynamic modules.
Preventing the Upgrade Failure
The next time you upgrade NGINX Plus, you can save time and avoid errors by following these instructions before the upgrade so that updated dynamic modules are already available when you start the upgrade process.
About the Author
Related Blog Posts
Secure Your API Gateway with NGINX App Protect WAF
As monoliths move to microservices, applications are developed faster than ever. Speed is necessary to stay competitive and APIs sit at the front of these rapid modernization efforts. But the popularity of APIs for application modernization has significant implications for app security.
How Do I Choose? API Gateway vs. Ingress Controller vs. Service Mesh
When you need an API gateway in Kubernetes, how do you choose among API gateway vs. Ingress controller vs. service mesh? We guide you through the decision, with sample scenarios for north-south and east-west API traffic, plus use cases where an API gateway is the right tool.
Deploying NGINX as an API Gateway, Part 2: Protecting Backend Services
In the second post in our API gateway series, Liam shows you how to batten down the hatches on your API services. You can use rate limiting, access restrictions, request size limits, and request body validation to frustrate illegitimate or overly burdensome requests.
New Joomla Exploit CVE-2015-8562
Read about the new zero day exploit in Joomla and see the NGINX configuration for how to apply a fix in NGINX or NGINX Plus.
Why Do I See “Welcome to nginx!” on My Favorite Website?
The ‘Welcome to NGINX!’ page is presented when NGINX web server software is installed on a computer but has not finished configuring