App Tiers Affected:
Organizations all around the world are transitioning into fully supporting the HTTP/2 protocol to deliver their applications. As we know, new protocols come with new functionality and new implementations, thus creating a vast, fresh attack surface for both researchers and adversaries. Researchers have already started studying this protocol and have even reported vulnerabilities.
As part of our continuous research at F5 Labs, we are threat modeling new web technologies, exploring the actual implementations, and evaluating their potential weak points. During our research of the HTTP/2 protocol we discovered several vulnerabilities, which we discuss in this article. Following our responsible disclosure process, affected vendors have already addressed those vulnerabilities, and the majority of them have patched their products.
Attack Vector 1: SETTINGS Frame Abuse Could Cause Denial of Service
In order to gain a better understanding of the protocol, we went through RFC 7540 and came across an explicit warning for implementations regarding a possible abuse of the HTTP/2 SETTINGS frame.1 The RFC warning discusses the potential for attackers to cause a server to waste unnecessary processing resources. Attackers can accomplish this by sending a SETTINGS frame consisting of invalid settings parameters, or by changing the same settings multiple times within a single SETTINGS frame.
The HTTP/2 SETTINGS frame is one of many different frame types supported by the HTTP/2 protocol. It is sent by both the client and the server throughout a connection in order to specify their supported settings, such as the maximum concurrent streams, the maximum size of frames, and more.
Although vulnerabilities related to the HTTP/2 protocol have been reported in the past, surprisingly, we haven’t found any related research referencing this warning. So, we decided to check whether and how vendors addressed this warning. Our tested scenario consisted of the following steps, each one designed to stretch the server’s limits:
- Send a SETTINGS frame with the maximum accepted size (as advertised via the SETTINGS_MAX_FRAME_SIZE setting)
- Fill this large SETTINGS frame with repeating settings parameters with changing values
- Do this over multiple HTTP/2 connections to the server
Surprisingly, we found that none of the tested HTTP/2 implementations—including the leading web servers such as NGINX, Apache and IIS—took this warning into consideration.
In all the tested implementations, the attack caused the server’s CPU utilization to reach 100%. In a portion of them, the attack led to a full denial of service. Besides fully knocking out the CPU, it even caused a few servers to slowly consume all of the machine memory. For some of the vendors, bad connections never timed out, even though no actual request content was sent to the server. This allowed the attacker to keep the attack going over the same connections.
The following table summarizes the vulnerability impact for each of the tested HTTP/2 implementations:
|Tested Webserver||Test result||Mitigation applied following disclosure|
|Apache httpd||Stops responding to new requests. The connection never timed out.||Patched and allocated CVE-2018-11763|
|NGINX||Consumes 100% of the server CPU resources and makes NGINX respond slower to incoming connections. The connection never timed out.||Patched and allocated CVE-2018-16844|
|Jetty||Stops responding to new requests. The connection never timed out.||Patched and allocated CVE-2018-12545|
|Microsoft IIS||Consumes 100% of the server CPU resources until timeout is reached.||Patched and published ADV190005|
|Python hyper-h2||Stops responding to new requests and slowly consumes all the available memory. The connection never timed out.||Reported via GitHub issue; answer received was that it could be mitigated by adjusting the server settings.|
|golang||Consumes 100% of the server CPU resources and makes the server respond slower to incoming connections. The connection never timed out.||Patched, but no CVE was allocated|
|Node.js||Consumes 100% of the server CPU resources and makes the server respond slower to incoming connections. The connection never timed out.||No response received|
Attack Vector 2: Denial of Service Via Slow POST (Slowloris)
Some web server implementations maintain a pool of worker processes/threads and assign a free worker from the pool for each incoming request. After the request is done, the worker process is freed back to the pool.
In 2009, a security researcher named Robert Hanson, also known as “RSnake,” discovered a highly effective denial of service attack. It allowed an attacker to take down a server using a minimum amount of resources—a single laptop could literally take down a server. Since then, the attack has evolved to many variants such as Slow POST and Slow Read. The vulnerability was eventually patched in the affected web servers and is mitigated today by all the anti-DDoS providers.
The idea behind this attack was to open many connections to the web server from a single attacking machine, and then on every connection start slowly transmitting the request without ever completing it. This, in turn, prevented the allocated worker process from being freed back to the pool. As the attacker opened multiple concurrent connections, all the worker processes of the server became busy and the server could not handle new incoming requests.
This attack was found to be highly effective against Apache web servers. Since Apache version 2.2.15, Apache has a module named “mod_reqtimeout” that is enabled by default and is designed to stop Slowloris variants.
In order to test whether Apache’s “mod_reqtimeout” detects similar attacks over the HTTP/2 protocol, we translated the Slow POST variant of the attack into HTTP/2. We did this by opening an HTTP/2 connection to the server and initiating the maximum number of streams allowed by the server, as advertised by the server via the SETTINGS_MAX_CONCURRENT_STREAMS setting.
In order to initiate the request streams, the following HEADERS frame was sent to the server on each one of the streams:
“:method”: “POST”, “:path”: “/testing”, “:scheme”: “https”, “:authority”: “apacheserver.com”, “content-length”: “999999999”
This HEADERS frame tells the server to prepare to receive a POST request with a large request body. In HTTP/2 terms, sending a HEADERS frame on a closed stream changes the state of the stream to “open,” and this is the point at which Apache allocates a worker process from its pool during the HTTP/2 connection. Once the stream was opened and the worker process allocated, we started slowly transmitting the request body, one byte every 20 seconds, which forced the worker process to stay busy. It turned out that although Apache’s HTTP/2 implementation was previously found to be vulnerable to a similar Slow Read attack and was subsequently patched, the simpler variant of the attack slipped under the radar of “mod_reqtimeout” and caused the server to stop responding to new requests. The issue was patched by Apache and CVE-2018-17189 was allocated.
The Jetty web server was found to be vulnerable to this attack, as well. The vulnerability was patched, but no CVE was allocated.
In order to mitigate the attacks mentioned in this article, it is recommended that organizations install the patch or follow vendor instructions. These vulnerabilities can also be mitigated by using an advanced full proxy web application firewall that supports HTTP/2 protocol inspections.