If I Had to Do It Over Again, Part 2

We wrote an article recently asking security leaders to talk about their past failures and the lessons they wanted to pass on to others. We called it If I Had to Do It Over Again, and our readers really liked it. A number of folks approached me wanting to tell their stories as well, so we’re doing a Part 2. Without any more preamble, here are their contributions, in their own words.

It’s Never Fire and Forget

One of my biggest wins was also one of my biggest failures. After years of battling the business (including Dev, QA, and DevOps teams for each primary web property, as well as their GMs), we finally got a WAF deployed! We got to a state of maturity where we would see an attack coming and could tweak a config to block it. Everyone felt proud and confident that the control was working.

Until one day we were dealing with a compromised site that was supposed to have been protected by the new WAF. It turned out, the business had deployed a new set of virtual servers and forgot to apply the WAF policy to them. The DevOps team had administrative rights in Puppet that controlled whether the WAF config was applied, and they regularly turned it off in testing.

Long story short, I had to tell the board about yet another incident. After that, we wrote a script that would fire off an alert anytime it didn’t detect the WAF policy enabled on a production server. Oh—and we forced the entire DevOps team to give us their SSH keys and removed their admin access to Puppet.

For me, it was a warning that control management is never point-in-time. There’s a shelf life to controls that are purchased but never fully implemented. On top of that, there’s the false sense of comfort in thinking that something is working when it isn’t. Never trust, always verify.

Everything is a Project, Even Physical Security

If I were to do it all over again, I would delegate the physical access control systems budget and planning early on, as I had done with our technical assessment and countermeasures. We were quite assertive in addressing the network and systems technical areas, which was fairly natural for the IT staff. However, we considered the physical elements of door locks as very elementary—something that could be done quickly and eventually.

Experience in that endeavor revealed to us that a keycard system with electro-magnetic control bars, card readers, and centralized control was a fairly sophisticated project. It also required longer than anticipated set-up time since we had to find the right partner to do the survey and installation. And finally, we needed to plan the various layers and schedules for the workforce key card system to ensure everyone had minimum necessary privileges. In the end, we did go through with an even pace to ensure we had convergence to enable ease of management and ROI. CISOs should apply respect evenly to the estimates of implementation to all controls from the start to avoid unnecessary concerns about urgency and cost overruns.

People Power

I wish I’d known a lot earlier that influencing people and leveraging human response would be more than half of the solution to dealing with human adversaries backed by great technology of their own. Supplementing computer science with study in cultural anthropology, behavioral economics, and cognitive psychology has been critical to being effective in information security. Now I look for solutions that acknowledge the humans in the loop—aided, not replaced by, technology.

Check Your Audit Logging

I've found that you need to make sure you understand what audit capabilities are in place for your systems. Ensuring systems are appropriately logging who accessed which records in what system is a critical first step. Next, I found I needed processes in place to regularly audit those logs. If there is no internal audit function, partner to build one as soon as possible, or hire an IT auditor within your department. Capturing a lot of logs to a SIEM does not make you more secure. You need to review those logs for security and access violations. Are authorized users granted too much access? How would you know?

Part 2 of this article happened because more folks stepped forward with lessons learned that they wanted to share. If we get more, we can run Part 3. So readers, show me what you got.

Contact us at f5labs@f5.com!