F5 is a big, international company. We have over a dozen major offices worldwide and thousands of employees. But even more, as a technology company, we have a huge Internet-visible presence with over 33,000 production IT assets enterprise-wide. A footprint that broad can be a real challenge to manage when a huge new vulnerability is announced and then followed by a zero-day exploit in the wild.
Many compliance regimes spell out requirements to patch vulnerable systems in a reasonable amount of time, for example, within 30 days. But in my opinion, compliance sets a minimum bar. I feel if we consistently practice good security and good hygiene, then we should be compliant.
Some organizations are still in the process of learning this lesson. In July 2018, F5 released its first annual Application Protection Report. For that report, we commissioned Ponemon to survey of 3,135 IT security practitioners about their application security processes. One of those key processes was how often they scan for vulnerabilities in their applications. As you can see in Figure 1, the frequency varied widely—from weekly, to no particular schedule, to none.
At F5, we dedicate a lot of time to identifying and validating vulnerabilities. We use a variety of vulnerability scanning tools at a regular, frequent tempo to give us an up-to-date picture of our risk footprint. On top of that, we pay attention to user reporting, information we get from various threat intelligence sources, and warnings from critical vendors like Microsoft that provide us more detailed information about known vulnerabilities and the potential implications they might have on our infrastructure. We may also confirm vulnerabilities by using additional scanning tools, checking asset configurations against applicable industry standards and best practices, interviewing stakeholders, attempting to reproduce behavior in a non-production environment, and checking log files for additional information.
Currently, all vulnerabilities are scored using the Common Vulnerability Scoring System (CVSS) (none, low, medium, critical, high). However, for us, a CVSS score is just a starting point for prioritization. We patch on a risk basis, which means we classify a vulnerability as high if it’s still a root-level breaching vulnerability with no exploit in the wild. Those we close within 30 days.
We consider a vulnerability to be critical if we see an exploit in the wild. We close those holes within seven days. When other mitigation tools are available, such as firewall rules or intrusion prevention signatures, we extend that timeframe but still try to patch as quickly as possible. Overall, we try to patch as near to real-time as possible.
Patching is a team effort, so everyone needs to collaborate. Of course the information security team is involved with the scanning, prioritizing, and governanace of the process. However, IT Operations does the actual patching because they own those systems. Most importantly, our Business Solutions Partners are heavily involved in the conversation, as they determine the actual business risks and the timing of a patch.
Yes, we occasionally delay patching for medium- and low-risk items if there is a valid case. A good example is making changes to financial systems during year-end close. This requires the approval of a business line VP as well as myself, the CISO. We have a well-defined governance structure to manage and monitor our remediation process.
Like for any critical business process, we track key metrics such as risk level, patch count, and patch completion times and percentages. The context of those metrics is as telling as the metrics themselves. For instance, having a hundred servers unpatched for a “high” vulnerability can mean different things depending on whether those servers are on the perimeter or are on an internal network behind firewall rules.
It’s important to note that patching is just one part of a full spectrum of vulnernability management tools. We leverage many tools to assist with inventory gathering, risk scoring, threat analysis, visualization, and asset tracking. All of these combine to give us a big picture of the “on-ground truth” of the risk of an unpatched vulnerability for a particular device.
For anyone looking to revamp their patching processes, gathering this kind of data is a good place to start. You need to step back and get a clear idea of the full extent of the potential problem. With that in hand, you can do ballpark estimates on the amount of work that needs to be done and have a better idea what needs to be reported to upper management.
We also incorporate careful selection of approved applications to ensure they are hardened enough to resist known attacks and that they support our suite of security tools like multi-factor authentication and logging. In addition, we ensure all our critical applications have appropriate network access control and availability protections. Even with all these things in place, we still always strive to patch as much as we can as fast as we can, as should you.
Patching isn’t the most exciting part of cyber security. In fact, it’s probably one of the most mundane. But it’s like brushing your teeth—a boring but necessary component of good hygiene that will keep you out of trouble.
