We wrote an article recently asking security leaders to talk about their past failures and the lessons they wanted to pass on to others. We called it If I Had to Do It Over Again, and our readers really liked it. A number of folks approached me wanting to tell their stories as well, so we’re doing a Part 2. Without any more preamble, here are their contributions, in their own words.
It’s Never Fire and Forget
One of my biggest wins was also one of my biggest failures. After years of battling the business (including Dev, QA, and DevOps teams for each primary web property, as well as their GMs), we finally got a WAF deployed! We got to a state of maturity where we would see an attack coming and could tweak a config to block it. Everyone felt proud and confident that the control was working.
Until one day we were dealing with a compromised site that was supposed to have been protected by the new WAF. It turned out, the business had deployed a new set of virtual servers and forgot to apply the WAF policy to them. The DevOps team had administrative rights in Puppet that controlled whether the WAF config was applied, and they regularly turned it off in testing.
For me, it was a warning that control management is never point-in-time. There’s a shelf life to controls that are purchased but never fully implemented.
Long story short, I had to tell the board about yet another incident. After that, we wrote a script that would fire off an alert anytime it didn’t detect the WAF policy enabled on a production server. Oh—and we forced the entire DevOps team to give us their SSH keys and removed their admin access to Puppet.
For me, it was a warning that control management is never point-in-time. There’s a shelf life to controls that are purchased but never fully implemented. On top of that, there’s the false sense of comfort in thinking that something is working when it isn’t. Never trust, always verify.
Everything is a Project, Even Physical Security
If I were to do it all over again, I would delegate the physical access control systems budget and planning early on, as I had done with our technical assessment and countermeasures. We were quite assertive in addressing the network and systems technical areas, which was fairly natural for the IT staff. However, we considered the physical elements of door locks as very elementary—something that could be done quickly and eventually.