Lessons learned from the Microsoft SOC Part 2b: Career paths and readiness

The “Lessons learned from the Microsoft SOC” blog series is designed to share our approach and experience with security operations center (SOC) operations, so you can use what we learned to improve your SOC. The learnings in the series come primarily from Microsoft’s corporate IT security operation team, one of several specialized teams in the Microsoft Cyber Defense Operations Center (CDOC). We’ve also included lessons our Detection and Response Team (DART) have learned helping our customers respond to major incidents and insights from the other internal SOC teams.

Today, we wrap up our discussion on people—our most valuable resource in the SOC. In the first part of our discussion, Part 2a: Organizing people, we covered how to set up people in the security operations center (SOC) for success. Today, we talk about our investments into readiness programs and career paths for our SOC analysts as well as recruiting for success. We’ll close the series with discussions about the technology that enables our people to accomplish their mission.

Something new every day

When an analyst walks into our SOC for a shift, they never know what to expect. They must be ready for anything as they face off with intelligent, adaptable, and well-funded adversaries who are intent on evading our defenses. For each problem, they must apply their unique knowledge and experience, the accumulated learnings from our SOC, and the expertise of their SOC teammates.

Our investments into readiness programs, career paths, and recruitment strategies are designed so our SOC analysts are prepared to succeed in their duties, increase mastery of their discipline, and grow as individuals. This ensures that our SOC staff brings their best to every shift, every time.

You may have to adapt some of these practices to the unique needs of your security operations team to be successful. We’re fortunate to have dedicated security operations teams, dedicated facilities, and experienced peers to learn from already on staff, but understand not all security organizations have these resources available.

Analyst roles and career paths

Empowering humans means investing in them. A SOC analyst is a high stress job and we know our success is built upon actively engaged people applying their experience and problem solving creativity. The longer our analysts do this work the better they get, so it’s important to nurture a long-running, sustainable workforce. This starts by clearly defining a career path. Our tier model not only organizes the work of the SOC, but also guides our analysts in building their knowledge and skills and shapes their careers with increasing levels of skills and different challenges.

Because we strive to empower and attract smart people with a continuous learning mindset, we’re motivated to promote from within. An analyst’s career path typically progresses from Tier 1 to Tier 2 to Tier 3 or to incident response, program management, security product engineering, or leadership tracks. There are exceptions, but this tends to be the norm.

• Tier 1—Analysts acquire and refine core skills including attacker mindset and techniques, using detection and investigation tools, working with internal teams and processes, and calmly applying a thoughtful approach in a high pressure situation. This is similar to martial arts where beginners acquire basic competencies (marked by a progression of colored belts) until they have achieved their black belt and move to the next stage of skills. Similarly, transition from Tier 1 to Tier 2 is a key turning point in the career of an analyst.
• Tier 2—Analysts continue to hone their skills as they move from executing well-defined playbooks for (mostly) predictable incidents at Tier 1 to investigating advanced incidents with greater unpredictability. Tier 2 analysts investigate attack operations conducted by organized groups with specialized skills and a specific targeted goal. Analysts investigating these incidents continue growing skills while learning from Tier 2 peer analysts and the incidents themselves. Over time, senior Tier 2 analysts often shadow different Tier 3 teams as they try out potential career paths and/or prepare for the next stage of their career.
• Tier 3—At this level, the analyst career paths typically start to diverge more into deeper specialties. Analysts can choose to pursue mastery of a particular skill or increasing competency/mastery across multiple skills. Tier 3 is increasingly requiring more data analytic skillsets on the team. This is because proactive hunting, investigation of advanced attacks, and automation development frequently require navigating many datasets with massive amounts of information.

Careful balancing

Defining a clear career path is important, but like all disciplines dealing with people, we must carefully balance and manage some nuances along the way.

• Balancing short and long term goals—As our analysts learn new skills and progress through their career, they learn to balance goals, such as ensuring alerts and cases are handled as top priority while simultaneously developing creative solutions that can reduce toil and increase efficiency over the long term.
• Balancing empowerment and guidance—Managers and senior personnel need to strike this careful balance as they mentor analysts in their career. This is particularly important for key transition points like when an analyst first begins onboarding a new role. Much like we see in many marital arts films when the talented but “not fully trained” student has an overabundance of confidence and tries to take on more than they can handle, we see a similar dynamic as analysts begin shadowing Tier 3 roles. In this situation, we have to be careful not to discourage this creative impulse (offering a feedback channel for ideas) while coaching and guiding analysts to complete their learning from seasoned professionals and focusing on the journey ahead.

Recruiting for success

Recruiting people and developing their skills is one of the most critical aspects of the SOC’s success. The biggest challenges in this space are the scarcity of people with the right skillsets, the speed at which skillsets must evolve, the potential for analyst burnout, and the need to blend diverse skills and perspectives to address both the human and technical aspects of attacks.

Much has been written about the scarcity of cybersecurity skills. We recommend reading a relevant blog on this topic that offers different ways of addressing the scarcity of talent in security. Additionally, you may want to watch a recent RSA Conference Keynote from Ann Johnson (Corporate Vice President of Cybersecurity Solutions Group at Microsoft), which addresses many related topics including the mental health and burnout risks our industry faces.

The evolving skillset challenge is particularly acute for our SOC because classic SOCs tend to be network centric, but our detection and investigation have evolved to rely primarily on device, identity, and application specific tooling. While we still have and use advanced network security tools, we’ve seen the utility of these network tools diminish significantly over the years to supporting investigation and advanced hunting. As of the writing of this blog, it’s been over two years since the last primary detection of an attack on our corporate environment came in from a network tool. We expect this trend to continue and have oriented our analyst readiness accordingly.

When it comes to recruiting and building skilled analysts, we’ve found that we require a combination of diverse perspectives and some common traits. As with any role, success requires having a diverse team with different backgrounds, mindsets, and skillsets to bring more perspective to the problems at hand and surface better solutions faster. We’ve also found certain personality traits tend to make analysts more successful in a fast-paced high-pressure work environment of a SOC.

Its critical to note that the following observations are general trends and not absolute rules. The primary factor of success in hiring an individual into a role is most heavily reliant upon that particular person and how well they fit that role. With that said, we tend to look for people with a kind of “grace under pressure” as we find it’s easier to train technical and security skills to people with a growth mindset and calm demeanor under pressure than it is to do the reverse.

For example, we have found that people with military experience are often a good fit because they have experience focusing on the mission despite the strong distractions in ambiguous situations with active hostile adversaries.

We’ve also had success with recruiting and investing into people early in their careers who are eager to learn and have few preconceptions. We’ve had good results with integrating seasoned professionals, but there are simply not enough available for the needs of the marketplace today.

An interesting aspect of the SOC attracting mission-oriented personalities is that when we have a major incident off hours, we more often get too many people volunteering to help versus not enough—a good “problem” to have!

Building skills and job readiness

Because of the high complexity required to be an effective SOC analyst, it’s difficult to educate new analysts in the ways of the SOC through formal training alone. We’ve tried different training approaches to build skills over the years and have found the apprenticeship model to be most effective at rapidly and consistently building skills. For new analysts we take an “I do, we do, you do” approach that progresses from observation to hands on with supervision of a seasoned analyst to independent investigation with support from peers and mentors.

This is similar to other industries with a need to transfer rich context and nuance during real world practice, such as an internship or a residency during a medical career.

The readiness process focuses on building understanding and competency in three domains:

1. Technical tools/capabilities.
2. Our organization (mission and assets being protected).
3. Attackers (motivations, tools, techniques, habits, etc.).

These competencies map well to established doctrine on human conflict. Sun Tzu’s advice to “know thyself” and “know thy enemy” map well to the second and third domains. Our SOC processes also map well to thinking from Colonel John Boyd’s OODA ‘loop’ on real-time human conflict: observe, orient, decide, act.

Beyond the competencies, we also need to train our analysts to be big picture thinkers and maintain an end-to-end view of the attack. It’s not enough to focus on a single threat, but to also “look left and right.” We need our analysts to think about how else the attacker might be trying to gain access and what else they may be after. For example, a password spray may be a potential entry to a multi-stage attack. An attacker may be using a distributed denial-of-service (DDoS) attack to provide a smokescreen to distract from their real objective.

We supplement this apprenticeship model with structured, formal training on topics, such as new products or features and SOC procedures. We also encourage attendance at conferences and work hard to ensure our staffing model supports these and other learning opportunities, so they aren’t empty promises.

This approach has been successful allowing us to train new Tier 1 analysts in approximately 10–12 weeks and we’re continuously looking for ways to improve our readiness processes. In addition, our staffing approach has been critical at mitigating burnout risk.

Learn more

For a visual depiction of our SOC philosophy, download our Minutes matter poster. Also, read previous posts in the “Lessons learned from the Microsoft SOC” series, including Part 1: Organization and Part 2a: Organizing people as well as see our full CISO series to learn more.

For more discussion on some of these topics, see John and Kristina’s session (starting at 1:05:48) at Microsoft’s recent Virtual Security Summit.

Stayed tuned for the next segment in “Lessons learned from the Microsoft SOC” where we discuss the technology that enables our people to accomplish their mission.

READ MORE HERE