Tracing and Troubleshooting NGINX, OpenResty, and Their Backends

Yichun Zhang, Founder & CEO at OpenResty Inc.

Got some online nginx processes eating too much memory, CPU time, or disk I/O resources? Got some very slow requests that occur randomly online, but cannot be reproduced offline? Seen some mysterious NGINX error log messages but have no clues about the actual causes? Bothered by some random nginx process crashes and core dumps in production?

In his session at NGINX Conf 2018, Yichun investigates various real‑world performance and other behavioral issues in the context of NGINX and OpenResty, and explains how to use dynamic tracing and other advanced postmortem debugging technologies to quickly pinpoint the causes online without disrupting production services.

Yichun describes and compares the advantages and disadvantages of various open source debugging frameworks like GDB, SystemTap, and BCC/eBPF, with a brief discussion of the unique challenges of tracing Docker containers in production. He then introduces the new OpenResty Trace platform which unifies these debugging technologies and improves their usability, scalability, and extensibility to a new level, especially in the context of distributed online systems like a microservices mesh or traffic gateway clusters as in a CDN network. He includes examples of writing custom tracing tools using the universal debugging languages provided by OpenResty Trace.

Yichun also uses real examples to demonstrate the use of advanced tracing tools atop OpenResty Trace for the NGINX core and the OpenResty core with LuaJIT. He covers tracing other traditional backend services running behind NGINX or OpenResty via FastCGI, uWSGI, or NGINX Unit, like PHP, Python, Perl, and Ruby. He concludes with examples of tracing data services like PostgreSQL, Memcached, and Redis accessed by NGINX, OpenResty, or their backend applications.

Next Steps