Top Risks
April 29, 2019

Denial of Service Vulnerabilities Discovered in HTTP/2

article
7 min. read
By Gal Goldshtein

Organizations all around the world are transitioning into fully supporting the HTTP/2 protocol to deliver their applications. As we know, new protocols come with new functionality and new implementations, thus creating a vast, fresh attack surface for both researchers and adversaries. Researchers have already started studying this protocol and have even reported vulnerabilities.

As part of our continuous research at F5 Labs, we are threat modeling new web technologies, exploring the actual implementations, and evaluating their potential weak points. During our research of the HTTP/2 protocol we discovered several vulnerabilities, which we discuss in this article. Following our responsible disclosure process, affected vendors have already addressed those vulnerabilities, and the majority of them have patched their products.

Attack Vector 1: SETTINGS Frame Abuse Could Cause Denial of Service

In order to gain a better understanding of the protocol, we went through RFC 7540 and came across an explicit warning for implementations regarding a possible abuse of the HTTP/2 SETTINGS frame.1 The RFC warning discusses the potential for attackers to cause a server to waste unnecessary processing resources. Attackers can accomplish this by sending a SETTINGS frame consisting of invalid settings parameters, or by changing the same settings multiple times within a single SETTINGS frame.

Figure 1: HTTP/2 RFC warning of possible SETTINGS frame abuse (RFC 7540, page 71)

The HTTP/2 SETTINGS frame is one of many different frame types supported by the HTTP/2 protocol. It is sent by both the client and the server throughout a connection in order to specify their supported settings, such as the maximum concurrent streams, the maximum size of frames, and more.

Figure 2: HTTP/2 SETTINGS frame format

Although vulnerabilities related to the HTTP/2 protocol have been reported in the past, surprisingly, we haven’t found any related research referencing this warning. So, we decided to check whether and how vendors addressed this warning. Our tested scenario consisted of the following steps, each one designed to stretch the server’s limits:

  • Send a SETTINGS frame with the maximum accepted size (as advertised via the SETTINGS_MAX_FRAME_SIZE setting)
  • Fill this large SETTINGS frame with repeating settings parameters with changing values
  • Do this over multiple HTTP/2 connections to the server

Surprisingly, we found that none of the tested HTTP/2 implementations—including the leading web servers such as NGINX, Apache and IIS—took this warning into consideration.

In all the tested implementations, the attack caused the server’s CPU utilization to reach 100%. In a portion of them, the attack led to a full denial of service. Besides fully knocking out the CPU, it even caused a few servers to slowly consume all of the machine memory. For some of the vendors, bad connections never timed out, even though no actual request content was sent to the server. This allowed the attacker to keep the attack going over the same connections.

Figure 3: Jetty server fully occupied four CPU cores with 396% CPU usage overall

The following table summarizes the vulnerability impact for each of the tested HTTP/2 implementations:
 

Tested Webserver Test result Mitigation applied following disclosure
Apache httpd Stops responding to new requests. The connection never timed out. Patched and allocated CVE-2018-11763
NGINX Consumes 100% of the server CPU resources and makes NGINX respond slower to incoming connections. The connection never timed out. Patched and allocated CVE-2018-16844
Jetty Stops responding to new requests. The connection never timed out. Patched and allocated CVE-2018-12545
Microsoft IIS Consumes 100% of the server CPU resources until timeout is reached. Patched and published ADV190005
Python hyper-h2 Stops responding to new requests and slowly consumes all the available memory. The connection never timed out. Reported via GitHub issue; answer received was that it could be mitigated by adjusting the server settings.
golang Consumes 100% of the server CPU resources and makes the server respond slower to incoming connections. The connection never timed out. Patched, but no CVE was allocated
Node.js Consumes 100% of the server CPU resources and makes the server respond slower to incoming connections. The connection never timed out. No response received

Attack Vector 2: Denial of Service Via Slow POST (Slowloris)

Some web server implementations maintain a pool of worker processes/threads and assign a free worker from the pool for each incoming request. After the request is done, the worker process is freed back to the pool.

In 2009, a security researcher named Robert Hanson, also known as “RSnake,” discovered a highly effective denial of service attack. It allowed an attacker to take down a server using a minimum amount of resources—a single laptop could literally take down a server. Since then, the attack has evolved to many variants such as Slow POST and Slow Read. The vulnerability was eventually patched in the affected web servers and is mitigated today by all the anti-DDoS providers.

The idea behind this attack was to open many connections to the web server from a single attacking machine, and then on every connection start slowly transmitting the request without ever completing it. This, in turn, prevented the allocated worker process from being freed back to the pool. As the attacker opened multiple concurrent connections, all the worker processes of the server became busy and the server could not handle new incoming requests.

This attack was found to be highly effective against Apache web servers. Since Apache version 2.2.15, Apache has a module named “mod_reqtimeout” that is enabled by default and is designed to stop Slowloris variants.

In order to test whether Apache’s “mod_reqtimeout” detects similar attacks over the HTTP/2 protocol, we translated the Slow POST variant of the attack into HTTP/2. We did this by opening an HTTP/2 connection to the server and initiating the maximum number of streams allowed by the server, as advertised by the server via the SETTINGS_MAX_CONCURRENT_STREAMS setting.

In order to initiate the request streams, the following HEADERS frame was sent to the server on each one of the streams:

“:method”: “POST”,
“:path”: “/testing”,
“:scheme”: “https”,
“:authority”: “apacheserver.com”,
“content-length”: “999999999”

This HEADERS frame tells the server to prepare to receive a POST request with a large request body. In HTTP/2 terms, sending a HEADERS frame on a closed stream changes the state of the stream to “open,” and this is the point at which Apache allocates a worker process from its pool during the HTTP/2 connection. Once the stream was opened and the worker process allocated, we started slowly transmitting the request body, one byte every 20 seconds, which forced the worker process to stay busy. It turned out that although Apache’s HTTP/2 implementation was previously found to be vulnerable to a similar Slow Read attack and was subsequently patched, the simpler variant of the attack slipped under the radar of “mod_reqtimeout” and caused the server to stop responding to new requests. The issue was patched by Apache and CVE-2018-17189 was allocated.

Figure 4: Error shown in Apache log, indicating that the slow POST attack succeeded
Figure 5: Apache server stopped responding to requests while under attack with Slowloris over HTTP/2

The Jetty web server was found to be vulnerable to this attack, as well. The vulnerability was patched, but no CVE was allocated.

Mitigation

In order to mitigate the attacks mentioned in this article, it is recommended that organizations install the patch or follow vendor instructions. These vulnerabilities can also be mitigated by using an advanced full proxy web application firewall that supports HTTP/2 protocol inspections.

Preventative
Technical
  • Install patches promptly or follow vendor instructions.
  • Use a web application firewall that supports HTTP/2 protocol inspections.

Need-to-Know

Expertly picked stories on threat intelligence

Hundreds of apps will be attacked by the time you read this.

So, we get to work. We obsess over effective attack methods. We monitor the growth of IoT and its evolving threats. We dive deep into the latest crypto-mining campaigns. We analyze banking Trojan targets. We dissect exploits. We hunt for the latest malware. And then our team of experts share it all with you. For more than 20 years, F5 has been leading the app delivery space. With our experience, we are passionate about educating the security community-providing the intel you need to stay informed so your apps can stay safe.

Every

9 hrs

a critical vulnerability—with the potential for remote code execution—is released.