Log4Shell: Rebooting (The Same Old) Security Principles In its Wake

As I write this, it’s been six days since the Apache Software Foundation announced the first Log4Shell vulnerability (CVE-2021-44228). As a researcher who is focused on long term trends, I haven’t played any response role, which means I’ve been in a position to observe both the unfolding of the event itself and the discussion about it. That discussion has been voluminous, which isn’t surprising, but it has also been, at least as far as I’ve seen, remarkably good.^{1, 2, 3, 4} What’s been notable about most of what’s been published is the spirit of collaboration and constructiveness running through it, whether it is addressing indicators of compromise and mitigation or broader strategic considerations.

As someone who often laments the needless competition and reluctance to share intelligence in the operational side of security, I have been heartened by this collective response. This comparatively small silver lining makes it seem that the impetus for competition over collaboration in security comes more from business objectives and less from operational goals. In short (and without wishing to overstate the differences between security staff and business stakeholders), this makes me believe that infosec operators would collaborate more constructively and often if they were given the chance.

It is in this spirit of collaboration, and with all the respect in the world for the incident responders who have had to deal with Log4Shell, that we want to use this event to reflect a bit on the present and immediate future of security as both an industry and a body of knowledge. In my mind, this event can help us sharpen our understanding of two aspects of security: software supply chain risk and Zero Trust.

Seeking Truth Along the Software Supply Chain

All of the various analyses of this vulnerability and its prevalence agree that cleaning this problem up will take some time, due to the combination of the many different ways that threat actors are weaponizing it and the difficulties in ensuring that all environments are completely clean. Bob Rudis, Chief Security Data Scientist at Rapid7, pointed out that “even organizations that have remediated deployed applications might miss some virtual machine or container images that get spun up regularly or infrequently.”⁵

However, I also think it is vital to conceptually distinguish between the future impact of this specific vulnerability, and the future risk of other but similar vulnerabilities, which is what we’re really talking about when we talk about the software supply chain in general. This vulnerability is already bad and will continue to be bad; that doesn’t inherently mean that we are powerless in the face of our own architectural complexities. It’s true that the growing reticulation of software dependencies has made risk cascade in ways we are not yet habituated to predict (or even inventory—more on that later). It is not necessarily true that all is lost. In the middle is the scope for our agency to mitigate this risk.

Earlier this week, as the incident was raging, I revisited a recently published and exceptionally topical report from our peers at Veracode in collaboration with the Cyentia Institute: the State of Software Security Vol. 11: Open Source Edition.⁶ The findings about our ability to control supply chain risk are mixed, but they help quantify a problem and show that we can mitigate it, at least in theory. To cherry pick a few significant findings (page numbers in parentheses):

• On the one hand, 79% of libraries are never updated. (13)
• On the other hand, vulnerable libraries are updated faster than non-vulnerable libraries.(15)
• The most frequent explanation for long update times was lack of developer resources. (23)
• In all code scans, no circular update dependency hell scenarios were found… (31)
• …although 13.9% of scanned applications had update chains that terminated in a flawed version. (31)
• Deep transitive dependencies, such as update chains with more than two steps, always terminated in an unflawed version. In other words, as long as the developers who use those libraries just keep chasing down updates, they will eventually succeed. (32)

Veracode characterizes software dependency risk as largely a problem of information. In both the key insights and the conclusions to the report, they emphasize that “developers who have the information they need” remediate third-party vulnerabilities significantly faster than those who don’t (4, 33). Based on the findings above, however, there is a logistical element here as well, and logistical problems are primarily management problems. The mechanisms for controlling software supply chain risk are understood, and often already in place, but they are starved for resources. This gives me the impression that while supply chain risk is increasing, controlling this risk is not a lost cause—it’s just a task that we haven’t properly scoped and prioritized yet. We don’t need to give up on third-party software. To quote Rudis again, “Use [libraries], but verify, and support!”

So, not all is lost. However, that’s not to say that vulnerability management is sufficient on its own. Many attacks take advantage of the aggregate behavior of disparate components as they interact. This is how three low-risk vulnerabilities become one high, and why low-severity vulnerabilities need to be addressed. More troubling than unpatched versions of log4j, therefore, are unknown versions of log4j—those organizations who only found out that they even had log4j running in their environments after the event started. Both Bob Rudis and Fernando Montenegro, among others, have emphasized the ongoing importance of an up-to-date inventory for security operations,⁷ which leads us to our next point: what this means for Zero Trust.

Zero Trust, like Absolute Zero, Is Not Something to be Achieved

Judging by the number of Zero Trust webinars to which I was invited in 2021, this was the year that this principle really gained mainstream traction. It’s ironic, then, that Log4Shell would happen this year, because I can’t help but see it as a collective, industry-wide failed audit of Zero Trust principles. Following our parent organization’s conceptualization of Zero Trust around four pillars, identity, network, endpoints, and applications,⁸ it looks as though we’ve all missed the mark not merely on the applications pillar but also on that of networking. Leaving aside the presence of the vulnerability in various applications, Log4j should never have been able to communicate with outside servers.

Again, I don’t mention this to criticize, but to make sure that we learn as much as we can from this event. Among all of the vendor discussion about Zero Trust it can be easy to start to think of it as a project that one finishes, as in, “we implemented Zero Trust in 2021.” Even if that were possible, if Zero Trust was a thing to be accomplished, the mutability of modern environments means that the accomplishment would be unraveling constantly. This is why we always come back to inventory. “Here, teams should beware the challenge of striving for a ‘perfect’ inventory: It's inevitable that environments change, so inventory information should be dynamic.”⁹ We can go further and apply this to both networks and to applications.

Viewed as a circular, multilevel process, Zero Trust starts to look like a modernized implementation of the older security principles of defense in depth and least privilege—a decentralized vision of security for decentralized systems. Given that third party integrations at all levels are not going away, Zero Trust will probably appear increasingly like the only viable approach for security, but only if we recognize that it is a principle that we always approach but never quite reach.