A Sociotechnical Metrics Framework for Network and Security Operations Centers

Back in January 2019, I published an article on Information Security Buzz titled, So, You Wanna Be A Security Star? which explored today’s lack of available security talent, the education required, a SOC analyst’s typical day, real job fatigue, and ways to boost morale of information security workers. We also dug into a 2015 study, A Human Capital Model for Mitigating Security Analyst Burnout, which took an anthropological approach to explore the burnout phenomenon. They were able to train and then place researchers within different Security Operations Centers to better understand, beyond interviews, what is driving the exhaustion. They looked at morale, automation, operational efficiency, management metrics and of course, how this leads to analyst burnout. The study identified four factors that impact the creation and preservation of efficient security analysts: Skills, Empowerment, Creativity and Growth.

As we were preparing to publish, I contacted one of the authors of the study Alex Bardas, currently an Assistant Professor of Computer Science at the University of Kansas. I wanted to thank him for the research and give him a chance to review how we represented the study.

During that conversation, Alex mentioned that he was working on a new grant proposal to the National Science Foundation (NSF) for another research project. This proposal was focused on developing a new metrics framework for security operation centers (SOCs) that measures and validates SOC performance against enterprise network security. It is a collaboration with Professor Bradley Fidler of the Stevens Institute of Technology, who studies the long-term evolution of network architectures from a social and institutional perspective. Alex asked if F5 would be interested in collaborating on this two-year project and the F5 SOC enthusiastically agreed.

Network Operations Centers and Security Operations Centers (NOCs/SOCs) are central components of modern enterprise networks. Organizations deploy NOCs/SOCs to manage their network operations, defend against cyber threats, and maintain regulatory compliance. Traditionally, these organizations are provided with an abstract view of network security through the interface of NOC/SOC metrics, and the NOC/SOC, in turn, interfaces with the network through monitoring software. By isolating a narrow subset of “performance” measurements, most typically a closed ticket count, these metrics misrepresent both the effectiveness of the NOC/SOC and the security posture of the network itself. These metrics tend to incentivize unproductive behavior in a NOC/SOC, conceal potentially fundamental security vulnerabilities in the network itself and trigger destabilizing “right-sizing” processes in the controlling organization.

Alex and team want to develop a new metrics framework that will harmonize NOC/SOC performance against enterprise network security. They are building metrics that serve as proxies for factors such as strategic and long-term planning and provide on-the-ground NOC/SOC operators with ways to input local knowledge into higher-up decisions. In the end, they want to:

1. Be transformative in the ability to communicate real-world security effectiveness,

2. Fit within preexisting NOC/SOC operations and management practices,

3. Serve as the basis for a new generation of enterprise network security tools

(which in turn will ultimately…)

4. Fix the vicious cycle between NOC/SOC practices and management decision-making.

By treating networks, security components, and operations staff as part of an interdependent system, the metrics will be able to account for factors such as outstanding security vulnerabilities, strategic and long-term planning, and constituency interests, and will provide on-the-ground SOC analysts with ways to input local knowledge into higher-up decisions. This could have the potential to ignite a major shift in the security landscape by providing a powerful new framework for real-world security assessments.

Thus far their research team has embedded an academic researcher in a separate security operations center and is analyzing the evolution of the interfaces between NOC/SOC staff, network monitoring software, and enterprise network architecture. They are still in the early stages of the project and our involvement has been more consultative than hands-on. We look forward to potentially testing some of the hypothesis elements from the study and providing feedback to the team when appropriate.

Looking forward, we’ll plan to post periodic updates to the research project including milestones, early analysis and hopefully some results that could help you in your NOC/SOC operations.