As I write this, it’s been six days since the Apache Software Foundation announced the first Log4Shell vulnerability (CVE-2021-44228). As a researcher who is focused on long term trends, I haven’t played any response role, which means I’ve been in a position to observe both the unfolding of the event itself and the discussion about it. That discussion has been voluminous, which isn’t surprising, but it has also been, at least as far as I’ve seen, remarkably good.1, 2, 3, 4 What’s been notable about most of what’s been published is the spirit of collaboration and constructiveness running through it, whether it is addressing indicators of compromise and mitigation or broader strategic considerations.
As someone who often laments the needless competition and reluctance to share intelligence in the operational side of security, I have been heartened by this collective response. This comparatively small silver lining makes it seem that the impetus for competition over collaboration in security comes more from business objectives and less from operational goals. In short (and without wishing to overstate the differences between security staff and business stakeholders), this makes me believe that infosec operators would collaborate more constructively and often if they were given the chance.
It is in this spirit of collaboration, and with all the respect in the world for the incident responders who have had to deal with Log4Shell, that we want to use this event to reflect a bit on the present and immediate future of security as both an industry and a body of knowledge. In my mind, this event can help us sharpen our understanding of two aspects of security: software supply chain risk and Zero Trust.
Seeking Truth Along the Software Supply Chain
All of the various analyses of this vulnerability and its prevalence agree that cleaning this problem up will take some time, due to the combination of the many different ways that threat actors are weaponizing it and the difficulties in ensuring that all environments are completely clean. Bob Rudis, Chief Security Data Scientist at Rapid7, pointed out that “even organizations that have remediated deployed applications might miss some virtual machine or container images that get spun up regularly or infrequently.”5
However, I also think it is vital to conceptually distinguish between the future impact of this specific vulnerability, and the future risk of other but similar vulnerabilities, which is what we’re really talking about when we talk about the software supply chain in general. This vulnerability is already bad and will continue to be bad; that doesn’t inherently mean that we are powerless in the face of our own architectural complexities. It’s true that the growing reticulation of software dependencies has made risk cascade in ways we are not yet habituated to predict (or even inventory—more on that later). It is not necessarily true that all is lost. In the middle is the scope for our agency to mitigate this risk.
Earlier this week, as the incident was raging, I revisited a recently published and exceptionally topical report from our peers at Veracode in collaboration with the Cyentia Institute: the State of Software Security Vol. 11: Open Source Edition.6 The findings about our ability to control supply chain risk are mixed, but they help quantify a problem and show that we can mitigate it, at least in theory. To cherry pick a few significant findings (page numbers in parentheses):
- On the one hand, 79% of libraries are never updated. (13)
- On the other hand, vulnerable libraries are updated faster than non-vulnerable libraries.(15)
- The most frequent explanation for long update times was lack of developer resources. (23)
- In all code scans, no circular update dependency hell scenarios were found… (31)
- …although 13.9% of scanned applications had update chains that terminated in a flawed version. (31)
- Deep transitive dependencies, such as update chains with more than two steps, always terminated in an unflawed version. In other words, as long as the developers who use those libraries just keep chasing down updates, they will eventually succeed. (32)
Veracode characterizes software dependency risk as largely a problem of information. In both the key insights and the conclusions to the report, they emphasize that “developers who have the information they need” remediate third-party vulnerabilities significantly faster than those who don’t (4, 33). Based on the findings above, however, there is a logistical element here as well, and logistical problems are primarily management problems. The mechanisms for controlling software supply chain risk are understood, and often already in place, but they are starved for resources. This gives me the impression that while supply chain risk is increasing, controlling this risk is not a lost cause—it’s just a task that we haven’t properly scoped and prioritized yet. We don’t need to give up on third-party software. To quote Rudis again, “Use [libraries], but verify, and support!”