Application Protection Report 2019, Episode 5: API Breaches and the Visibility Problem

APIs and the New-Old Problem of Visibility

So far in our 2019 Application Protection Research Series,¹ we have explored reconnaissance campaigns directed against PHP, looked at causes for known breaches, and mapped specific attack techniques to different industries, business models, and architecture. We’ve looked into the details of the two most successful attack types, injection and access attacks, unpacked why they work, and explained how you can avoid being a victim of these types of attacks.

In this episode we investigate a component of web architecture whose importance is growing dramatically, and explore the implications that the use and configuration of this component has on protecting (or failing to protect) apps. Application Programming Interfaces (APIs) are processes and structures through which applications (or sub-application modules or functions) can connect to one another and communicate autonomously. While APIs have been around for a long time, the growth of serverless architectures, containers, and DevOps have made them absolutely critical for contemporary web operations. You might say that web applications have shifted from being blankets to being quilts, and APIs are the stitching that holds the quilts together.

APIs, then, are hard to categorize. They serve a combined access control and data translation role, coordinating disparate and distributed functions behind the scenes to present the user with a unified application service. From a security standpoint, this is what really matters: the user experience is that of one app. In other words, APIs raise the same fundamental issue of visibility that we discussed in the third episode.

Last year we released a follow-up to our 2018 Application Protection Report that focused on APIs. We highlighted two patterns of API use that were implicated in API breaches: 1) large platforms with numerous third-party integrations, and 2) mobile apps. Most of these attacks tended to follow the same form: access or injection attacks against the API authentication surface followed by exploitation of excessive permissions. In other words, APIs tend to be compromised in ways similar to breaches of other web applications, but because they are both increasingly important and hidden from view, they arguably represent a bigger risk to the business than other assets.²

As a result of this increased risk, we are devoting an entire episode to protecting APIs (and the applications they stitch together) to help the security industry keep pace with the change in the rest of the tech industry.

API Usage

APIs do not serve in an exclusively structural role to make applications work; they can transform an organization’s business model by directly generating revenue. In 2015, the Harvard Business Review reported that 90% of Expedia’s revenue came through APIs. Ebay and Salesforce also generated much of their revenue this way, reporting 60% and 50% API-driven revenue, respectively.³ As a result of the growth of APIs, organizations are recognizing new opportunities to generate traffic and revenue, often using existing components of their environments with minimal modification.

F5 Networks’ 2019 State of Application Services (SOAS) report found that of 2000 technology professionals who responded to the survey, 42% had already deployed API gateways, with 27% planning to deploy gateways in 2019.⁴ The API information portal and community forum programmableweb.com listed 12,000 APIs in their directory in 2015, and in early 2019 listed nearly 22,000, illustrating the rapid growth in their use.⁵

What Is an API?

In the simplest possible terms, an API is a user interface for other apps instead of users. (Technically, it is a standard to create a user interface for other apps). An API gateway is a piece of lightweight software, running on an application server, that creates a connection point through which other app services or mobile apps can push or pull data.⁶ Ideally, API gateways should have some form of access control, and they typically support a range of authentication, authorization, and federation services such as OAuth, JWT, OpenID Connect, and SAML. The API gateway authenticates the requesting device and validates that its request or payload is structured correctly for the software that sits behind the API. This allows other applications to use the output of the original service in a different way, in a different app, or in a different environment, without having to recreate the original service from scratch. This is what makes APIs such a great way to scale and embed functionality into other apps.

Google Earth provides an illustrative example of how APIs work. Most of us have run Google Earth, either through a desktop client or through a browser, to find satellite imagery of our own home, think briefly about how that’s kind of creepy, then go fwooosh around the globe. That’s the normal graphical user interface (GUI). But many of us have also used the mobile app, which pulls data from Google Earth servers through an API. In fact, many more apps use the Google Earth Engine API to achieve more specific goals, such as visualizing and measuring change on the surface of the Earth, than Google Earth itself does.⁷ One of my favorite Google Earth API tools is the Streetview Zombie Apocalypse simulator, which allows you to see how your neighborhood would fare if the dead walked.⁸ Google has even released an Earth Engine API dedicated exclusively to the user interface, enabling other app developers to create new UIs without having to reinvent the wheel on the backend.⁹

Types of APIs

APIs differ in their architectural styles, which in turn affects the format of the machine-machine interactions and how those APIs are deployed. API architecture is a complex topic that exceeds the scope of this article, but it is important to know a few terms and concepts:

REST APIs

REST stands for REpresentational State Transfer. REST is an architectural style (as opposed to a strict standard or protocol) that defines how computers interact with one another on the Internet. REST APIs are transactional in nature, receiving requests and providing responses on demand. They use standard HTTP methods (that is, GET, POST, PUT, PATCH, DELETE), require a specific location to which they can connect (that is, a URI).¹⁰ They also need to define data model standards to ensure that data are structured correctly. REST APIs make up the bulk of currently deployed APIs, having replaced most of the earlier generation of Remote Procedure Call (RPC) architectures, such as SOAP and XML-RPC.¹¹

The fact that REST APIs use the standard HTTP methods is a big part of why they are so common. This standard allows them to be lightweight and straightforward to create and deploy, which means that they can be used across a wide variety of devices, even those with minimal processing power. For instance, REST APIs can allow an Internet of Things (IoT) device like a networked video security camera, which does not have a lot of computer power onboard, to work in a remote setting. A user connects to the camera through an app on the phone to see what happened overnight (or to check on cats/kids). On the backend, the mobile app for the security camera would be sending an API pull request to the camera, which would return a still photo across the web (we’ll talk about video data across APIs below).

RESTful APIs can be described in a format called OpenAPI 2.0 (formerly Swagger) that defines authentication modes, servers used, methods accepted, and other expected behavior. This is especially handy when implementing API security services.¹²

Real-time APIs

In contrast to the transactional model of REST APIs, real-time APIs offer a continuous feed of data. This can be advantageous in specific scenarios that are time-sensitive, such as for financial data, web chatting, and, in the example of our IoT webcam, sending out a stream of video instead of still photographs. Video information is time-sensitive because it needs to be reassembled from individual packets in the order in which they were taken before it can be viewed at the destination, and so streaming video is a great use case for real-time APIs.

Others

There are other kinds of APIs, such as native APIs embedded in systems for specific data output, or others that are only accessible through software development kits (SDKs). These are less common, and the differences are not relevant from a security standpoint, so we’ll leave it at that for now.

What Makes APIs a Good Target?

There is a combination of factors that make APIs such rich targets. One of the biggest issues is overly broad permissions. Because they are not intended for human use, APIs are often set up to access any data within the application environment. Permissions are usually set up for the user making the original request, and these permissions are, in turn, passed to the API. That is all well and good, until an attack bypasses the user authentication process, going directly to the downstream app. Because the API has unrestricted access, attacks through the API provide attackers with visibility into everything.

Like basic web requests, API calls incorporate URIs, methods, headers, and other parameters. All of these can be abused in an attack. In fact, most typical web attacks, such as injection, credential brute force, parameter tampering, and session snooping work surprisingly well. To attackers, APIs are an easy target for all their old, dirty tricks.

Another issue is visibility. As an industry, we haven’t often maintained a lot of situational awareness of APIs. They are supposed to run behind the scenes; this is great until they are compromised behind the scenes, and all of our valuables get stolen behind the scenes. As we noted in our API follow-up to last year’s report, APIs often connect to ports other than 80/443, they are frequently buried in deep paths somewhere on web servers, and the details of their architecture are often known only by development teams. All of this adds up to the reality that security teams may be unaware that connections with that potential impact are even possible in their environment.

What Kinds of APIs Get Hacked, and Why?

Last year, we found that API compromises tended to fall into two patterns of API use that correspond to common breaches. We have also identified a third emerging pattern, which we’ll unpack below. Here’s what the landscape for successful API breaches looks like:

Large Platforms with Lots of Third-Party Integrations

Organizations with high traffic sites offering a wide range of services often feature a large number of third-party integrations. These integrations rely on APIs to collect data from third-parties and serve it up to the user in a seamless fashion. The growing decentralization of infrastructure, represented by multi-cloud environments, third-party functions and content, and serverless and containerized architectures, means that APIs are not a luxury. Today they are absolutely essential for modern, high-volume platforms. Some of these platforms have hundreds of APIs, all of which need to be managed and monitored. These kinds of organizations and business models have tended to figure prominently in the API breach notifications we’ve seen.

Mobile Apps

As we noted briefly above and elsewhere, most mobile apps rely on APIs to pull data from servers, which allows the apps to use fewer resources on the devices themselves. Because of some of the inherent challenges with securing mobile applications, there is a vibrant community of attackers who decompile and reverse engineer mobile applications looking for vulnerabilities or opportunities, such as hardcoded credentials or weak access control. The API is often a focal point for these efforts.

The Misconfigured Big App

Both of those patterns—large apps with many third-party integrations and mobile apps—were common in breaches we reviewed last year, but there is an additional emerging trend that deserves mention: the misconfigured API. In many of the new cases we saw in the second half of 2018 and the first half of 2019, breaches occurred because stakeholders in organizations were not aware of either the existence of an API, or the impact of an insecure one, and so put no authentication (or weak authentication) in front of it. As silly as it sounds, it is hardly surprising, since it once again points to the fundamental challenge of visibility, both from the standpoint of information systems and large organizations.

The Threat Actor Model Is a Poor Fit for APIs

In examining the data on API breaches over the past year, we have found that understanding the threat actors and their motivations is particularly instructive for understanding the threat in this case, but more because the CHEW (criminal, hacktivist, espionage, warfare) model is a poor fit rather than a good fit. To illustrate what we mean, let’s examine the model.

Criminals

Criminals want to attack APIs for relatively obvious reasons. APIs are an obscure but startlingly direct path to valuable data, like payment card information, that criminals can resell on the market, or use to buy commodities that they then resell as a form of money laundering. No surprises there.

Hacktivists

Here is where the model starts to fail in interesting ways. There are people exposing gaping holes in API security, but not to make a political point about power imbalances or perceived injustice. In actuality, the intentional but non-malicious exposures of API flaws that we see are from security researchers using recognized but obscure attack chains as a vehicle for self-promotion. The result is a loss of credibility and brand trust for the victims, but no direct financial or legal costs, for the most part. We will explore this in greater detail a little later.

Espionage

Similar to the poor fit in the hacktivist category above, the most prominent espionage scenario was not espionage in the sense of either corporate espionage against a competitor, or nation-state espionage against a rival state. The Cambridge Analytica/Facebook scandal was instead an intentional exploitation of contractual loopholes for the purposes of data mining the public. Nevertheless, it exploited the functionality of an API in a way that it was not (ostensibly) intended to be used.

One of the questions this raises is whether the Cambridge Analytica scandal was an actual abuse of functionality, or an intentional exploitation of a business opportunity—right up until the point that the scandal broke. We don’t know, but it is worth noting here that, once again, the standard categories of threat actors don’t apply. It appears that the poor fit of our actor model says more about the nature of API security than it does about the model.

Warfare

We have not yet seen evidence of API attacks in a warfare context, but the most obvious scenario would be to flood an API with traffic as a form of distributed denial of service (DDoS) attack against the application itself. In a warfare scenario, compromising availability or integrity can be as tactically advantageous as compromising confidentiality, depending on the context. The potential for an API to become a medium for an obscure and difficult-to-diagnose DDoS attack is rich, and we will not be surprised to see it deployed that way soon.

Public Records Review

In terms of public breaches, all of the 2018 breaches up through November were covered in the ancillary Application Protection Report on APIs. We won’t bother to reprint them here, but all of the breaches we identified fit into the first two patterns listed above–huge platforms with loads of APIs, or mobile apps that depend heavily on a few APIs to function.

Breaches

In terms of new events, every single breach we’ve identified between November 2018 and the present were attributable to misconfigured access controls. In other words, system owners did not realize that their API was wide open. However, as we noted above, the security researcher looking for a headline to put their name on was the principal threat actor in these scenarios, so the good news is that data were not leaked to the black market. You might say that these system owners got lucky, as did the researchers.

Another piece of evidence from the same time period that illustrated the prevalence and importance of APIs—as well as their weak security—was a study by North Carolina State University (NCSU) that found that more than 100,000 code repositories on GitHub had API tokens and cryptographic keys—the literal tools for API access control—stored in plaintext and visible in plain sight.¹³ These findings illustrate a trend we’ve seen for a long time: developers using workarounds or insecure practices during development, when it doesn’t matter, and then failing to mitigate those issues when the project goes live. Storing API authentication information in plaintext on GitHub is the latest incarnation of this old issue.

Incidents (Not Breaches)

The list of API incidents that did not lead to a breach had the same characteristics as the confirmed breaches. For the most part, these incidents were caused by security researchers finding routine bugs that had escaped detection precisely because they were in APIs.

API Vulnerability Data

The sensor network that our partner Lorkya maintains found only 0.1% of attack traffic was definitively looking for API vulnerabilities. However, this is probably better attributed to the limitations of the sensor network than any trends about API attacks. Loryka’s sensors primarily detect wide-ranging probes and reconnaissance campaigns where attackers are looking for something to target, but API attacks will, by nature, be targeted and bespoke. There were still a few interesting findings worth mentioning.

Stalker_Portal

There was a burst of traffic from a single IP address that lasted for a few days targeting a specific path on any system it could find. The path that the traffic was targeting was www.example.com/stalker_portal/server/api/file.php. In just a few days this traffic came to constitute more than one third of the total traffic targeting APIs, and we don’t know what it is. It could be a misconfigured vulnerability scanner that was supposed to be scanning an internal environment, but we don’t really know, and we can’t draw any big conclusions from it.

PHP API Attacks

One interesting finding was that, of the traffic looking for APIs, 86.7% of it was scanning for sites with the PHP API. While notable, this is not actually surprising; we found in Episode One that 81% of the malicious web traffic Loryka detected on port 80 was targeting PHP. In other words, our API intelligence just cements PHP’s position as one of the great targets of the Internet.

Crafty Syntax Live Help

Loryka’s data also revealed that 1.8% of the API attack data was targeting paths containing a CSS file for an open source customer service product known as Crafty Syntax Live Help.¹⁴ An exploit was published in 2006 regarding a remote file inclusion vulnerability in version 2.9.9 of this product, specifically regarding a text-based database support API.¹⁵ It appears that someone was looking for an opportunity to exploit a very old vulnerability on any target they could find.

Generic and Other API Attacks

About twelve percent of the API-targeted traffic we saw did not fit into any broader trend, or was malformed. With the limited detail we get from Loryka’s sensors, it’s not possible for us to form a complete picture of this traffic. It is still worth exploring some specific traffic in detail, however:

• /api/.git/HEAD–This path probably represents an attempt to scrub HTTP header information from any system set up with a GIT API.¹⁶
• /zabbix/api_jsonrpc.php–Zabbix is a network monitoring appliance with an RPC API that encodes traffic in a JSON format. This is probably an attacker scanning for servers with Zabbix APIs that have no access control.
• /include/api/xmlrpc–In all likelihood this is looking for any RPC-style APIs that might not require authentication.
• http://clientapi.ipip.net/echo.php?info=20180211134656–This path, which we saw exactly ten times in the dataset with different 14-digit values at the end, was interesting because it targeted an actual domain instead of any domain that had something at the target path. The target, IPIP.NET, offers IP geolocation and research services. We don’t know what the payload values were intended to do, or whether this was an attack or a misconfiguration.
• /api/log_browser/validate–This is another path whose intended target and function we do not understand, but we can reasonably guess that it was targeting the API of network traffic logging, server logging, or similar appliance, possibly for the purposes of network reconnaissance.

Known API Exploits with Plausible Linkages to Attack Patterns

We also found known exploits that line up with specific attack patterns, as noted by Loryka, that are worth mentioning:

• /wsdl/–Web Services Description Language (WSDL) is an XML-based description language that is often used for Remote Procedure Call-type SOAP APIs. In May 2019, a researcher published an exploit for ReadyAPI that allows attackers to upload WSDL files with malicious content to the API, such that when a victim requests information from the API, the malicious code runs on the victim’s computer.¹⁷ This attack path was probably targeting this vulnerability.
• StarOffice.ComponentContext–This attack path was targeting a remote code execution vulnerability in the Apache OpenOffice UNO API that StarOffice uses along with other open source office suites like Libre Office and OpenOffice.¹⁸
• /v1/agent/self OR /v1/agent/service/register–These two paths are targeting a remote command execution vulnerability in the Hashicorp Consul service mesh API. The exploit for this vulnerability was published in early 2019.¹⁹
• \x41–We saw this path show up exactly 8000 times in our dataset, which corresponds to a known denial of service vulnerability in a network utility suite called Megaping, which is caused by a buffer overflow.²⁰
• pages.aspx?pageref=[SQL]–This path is an injection attack against the Grapixel New Media 2 tool’s ASP.NET API.²¹ Attacks like this illustrate how APIs are susceptible to many of the same kinds of vulnerabilities, and hence attacks, as other forms of software. Unvalidated user input like this is a risk no matter where it lands in a system.
• /api/hosts–Similar to the above exploit in an ASP.NET API, this is an attempt to target the REST API for the Unitrends Enterprise Backup (UEB) appliance with unvalidated user input, which results in remote code execution and root privilege.²²
• miner_getfile–Nanopool is a cryptocurrency mining and pooling platform that offers APIs for remote crypto miner management. The API has a remote code execution vulnerability that the attacker was presumably trying to exploit.²³
• /api.txt–This path probably represents attempts to use the network appliance Shodan’s API to force memcached servers to flood targets with a distributed denial of service (DDoS) attack.²⁴

OK, I Get It. APIs Are Risky. What Do I Do?

Inventory

To start, as always, you’ll need an inventory of your APIs. For some organizations, this is not as easy as it sounds. The large platforms that we mentioned earlier often depend on APIs to pull specific functions into their application, such as payment information processing, or linking to social media. The growing complexity of information systems, along with the accelerating rate of change, means that obtaining this inventory is not a trivial task. Maintaining the inventory over time can be even more difficult, because it is usually harder for people with other jobs to budget time for ongoing, routine maintenance tasks than it is to get together for a single discovery session. Nevertheless, it is absolutely essential to keep the inventory up to date, since APIs are quickly becoming the new nexus of business logic and application architecture.

As of June 2019, we know of approximately 22,000 published APIs. The majority of those are estimated to be private APIs, which means that access to them is restricted to internal developers or specific partners, as opposed to the public.²⁵ Understanding which ones are which within your organization, and how each one contributes to your business operations is important for adding context to your inventory. This information allows you begin the process of assessing the risks that APIs might bring to your environment. The reason this is such a big deal is that all of the breaches we’ve seen so far have been attributable not to sophisticated zero-day exploits by genius hackers, but instead to failures to apply basic security principles to new operational needs.

There are two specific things we recommend for every API inventory, reflecting the role of APIs in contemporary systems: 1) Conduct perimeter scans. Perimeter scans are valuable particularly because they replicate the hacker’s eye view.²⁶ 2) Perform in-depth discovery interviews with development and operations teams. This may save time and get you valuable information to supplement the results of internal scans. Find out what the de juris and the de facto states are, and prepare risk assessments for both.

API Authentication

By design, APIs execute commands. They should be considered root/admin interfaces and require strong authentication. F5’s State of Application Services 2019 found that 25% of the organizations surveyed didn’t deploy authentication for APIs at all.²⁷ 38% reported that they did ”some of the time,” and 37% said ”most of the time.” Two things stand out about this to us. The first is that a full quarter of F5’s customer base is not controlling access to a component that offers a direct path right to the heart of some of their most valuable capital. The other is that even the leaders in this issue are still not controlling access all of the time. This is a prime example of risk models failing to keep up with a changing landscape.

There are different forms of API authentication; practitioners must consider the pros and cons of each type based on risk.²⁸ Generally speaking, OAuth 2.0 is considered the best option for most REST APIs, but it really depends on the nature and design of the API, its intended uses, and the business model and threat model of the organization.²⁹ (Note also that OAuth 2.0 authentication traffic must be encrypted to be effective.)³⁰ There are also emerging frameworks for service-level authentication in modern architectures such as the SPIFFE framework, which offer a potential step forward in terms of managing authentication in complex environments.³¹ The real point here is that authentication of APIs is not optional given the trends and risk we’ve outlined above.

Furthermore, as in all authentication, credentials must be stored in a secure way. Whether your credentials take the form of user/password combinations (for either machines or human users) or API keys (which are simplified authentication strings that have specific uses), it is critical to treat them as sensitive information, because they really are the keys to the kingdom.³²

API Authorization

The other half of API access control is authorization, by which we essentially mean the permissions associated with any credential set. As with all credentials, API creds must be treated using the principle of least privilege. Role-based access control is the best way to do this for API accounts of all types. At a minimum, this should take the form of limiting the HTTP methods that specific roles can implement (DELETE being an obvious one to limit, but not the only one–again, let your own environment and business logic dictate this in a risk-based way). To go one better, define specific sequences of actions that correspond to the specific API use case, and limit the API to that sequence instead of simply specifying a list of permitted actions.

At no time should APIs be allowed to pass unsanitized or unvalidated input to the application. That is a sure recipe for an injection attack.

Log the API Connection

After implementing authentication and authorization controls, the next level of maturity is to implement monitoring on the API. As we mentioned in Episode 4, brute force attacks can easily go unnoticed; all the more so with the advent of clever, distributed, low-and-slow access attacks. As a result, you should log API connections regardless of their outcome and behavior, and review those logs, whether through humans, bots, or cloud services. Furthermore, given the number of APIs and connections across them, it’s best to also monitor the assets that the APIs serve to ensure their integrity and confidentiality.

Encrypt the API Connection

At the risk of sounding repetitive, APIs are like user interfaces for machine users. Their traffic still moves across the web like any other traffic. We increasingly encrypt all user traffic on the web—API traffic should be treated the same way. Encrypt those connections and validate the certificates like you would for any other service. That also takes care of the requirement to encrypt OAuth 2.0 authentication traffic, as noted above.

Look at API Security Tools

Consider looking at a proxy or a firewall that is “API aware” to inspect, validate, and throttle API requests. Some API security services can analyze the originating client and attempt to determine if a request is legitimate or malicious. They can also ensure that API requests stay where they’re supposed to stay, and do not escalate their privileges or exceed their reach into the app or data. A tool like this makes all of the aforementioned mitigation tasks easier. Many API security tools ingest OpenAPI/Swagger files, which will auto-configure the enforcement of the expected behavior.

Test Your API

APIs must be scoped in for scans, vulnerability assessments, and penetration tests. The point here is that the prevalence of APIs is matched only by their obscurity. Many organizations don’t even have a single person who is aware of all of the org’s APIs, and almost none of the orgs we’ve talked to have authentication on all of them. Testing is a part of any decent security program, and it applies here as well.

Given the difficulty of maintaining situational awareness, it is also a good idea to place a bug bounty on API vulnerabilities. Security researchers are constantly looking for things to report about, and Shodan has made it easy for researchers (and attackers) to discover your systems in detail. It is much, much cheaper to pay them quickly and quietly, patch your issues, and release a statement when applicable, than for them to go public—which could lead to bad press at best and a compromise at worst.

Wrap-Up

APIs are not new, but they are newly relevant for the way the Internet is growing and applications are evolving. As such, they do not so much introduce new risks as they reintroduce existing risks in forms that are more likely to be exploited, more impactful, and harder to recognize. At the same time, they are an unavoidable component in contemporary architectures, which means that avoiding or ignoring their issues is not an option for security teams. The Open Web Application Security Project (OWASP) has released one of their excellent Top 10 lists specifically for APIs. It is a good place to begin to explore these issues in greater detail.

Each chapter of the 2019 research series has featured well-understood risks, many of which have been around for decades, but that have taken on new forms as a result of changes in the tools and techniques that we use to provide web services. While this is in keeping with our long-running assertion that successful security operations are mostly about the thorough implementation of the basics across space, time, and diverse systems, this is not to say that we think that the changes are bad. When we talk about increased architectural complexity, this should be understood in light of the fact that applications also feature less complexity in the code itself. Other than for operating systems and similar heavy pieces of software, the days of millions of lines of code in a single piece of software are probably coming to an end. Instead, that complexity is re-appearing in architecture and infrastructure, as we assemble many small parts to create a complete whole. There are many advantages to these trends, but they also introduce added layers of abstraction in operations, which, in turn, raises issues for visibility and cascading failure modes. This is why the old issues are coming back in new forms. As security practitioners, we must not lament these changes, but adapt our practices to them with evidence, perspective, and a focus on enabling operations rather than hindering them.