DevOps threat matrix

The use of DevOps practices, which enable organizations to deliver software more quickly and efficiently, has been on the rise. This agile approach minimizes the time-to-market of new features and bug fixes. More and more companies are implementing DevOps services, each with its own infrastructure and emphasis on distinctive features.

As the use of these environments increases, so does attackers’ interest. Several high-profile attacks that originated from a DevOps environment show the significant impact of these attacks if successful. Examples of such attacks from the past few years include the following:

• The attack on SolarWinds’s Orion software. Attackers managed to attack the DevOps environment and injected malware into an upgrade of Orion that may have spread to over 17,000 customers.
• Codecov breach. Attackers successfully attacked the company’s docker images and stole credentials and private data of thousands of customers.
• Dependency confusion attacks. For example, malicious PyPi packages inside the registry, npm packages, PHP packages and others. These attacks are widely used and can serve as an entry point for attackers to sensitive networks and environments.

These incidents demonstrate the importance of securing code before it is deployed. Defenses are shifting left to the earliest stages of the application development. In addition to the development phase, we must make sure DevOps environments are secured as well.

At Microsoft, we have conducted extensive research into the techniques that malicious adversaries may use to attack DevOps environments. We categorized these techniques into their related tactics and mapped these into a threat matrix. This mapping aims to help defenders to better understand the landscape and possible attacker actions, so defenders are better equipped to defend against each technique and protect DevOps environments.

DevOps threat matrix

Our goal for developing the threat matrix for DevOps is to build a comprehensive knowledgebase that defenders can use to keep track of and build defenses against relevant attack techniques. Using the MITRE ATT&CK framework as a base, we collected techniques and attack vectors associated with DevOps environments and created a matrix dedicated to DevOps attack methods.

It is worth noting that the tactics in this matrix must be looked at from the DevOps perspective. For example, execution techniques in a Virtual Machine running Windows or Linux OS are different from the Execution in a DevOps pipeline. In the Linux case, execution means running code in the OS. When we talk about DevOps environments, it means running code in the pipeline or DevOps resources. In addition to using this threat matrix to categorize attacks and corresponding defense methods, defenders can work alongside red teams to continuously test assumptions and find new potential attack techniques.

1. Initial access

The initial access tactic refers to techniques an attacker may use for gaining access to the DevOps resources – repositories, pipelines, and dependencies. The following techniques may be a precondition for the next steps:

SCM authentication – Access by having an authentication method to the organization’s source code management. It may be a personal access token (PAT), an SSH key, or any other allowed authentication credential. An example of a method an attacker can use to achieve this technique is using a phishing attack against an organization.

CI/CD service authentication – Similar to SCM authentication, an attacker can leverage authentication to the CI/CD service in order to attack the organization’s DevOps.

Organization’s public repositories – Access to the organization’s public repositories that are configured with CI/CD capabilities. Depending on the organization’s configuration, these repositories may have the ability to trigger a pipeline run after a pull request (PR) is created.

Endpoint compromise – Using an existing compromise, an attacker can leverage the compromised developer’s workstation, thus gaining access to the organization’s SCM, registry, or any other resource the developer has access to.

Configured webhooks – When an organization has a webhook configured, an attacker could use it as an initial access method into the organization’s network by using the SCM itself to trigger requests into that network. This could grant the attacker access to services that are not meant to be publicly exposed, or that are running old and vulnerable software versions inside the private network.

2. Execution

The execution tactic refers to techniques that may be used by a malicious adversary to gain execution access on pipeline resources – the pipeline itself or the deployment resources. Some of the techniques in this section contain explanations about the different ways to perform them, or what we call sub-techniques:

Poisoned pipeline execution (PPE) – Refers to a technique where an attacker can inject code to an organization’s repository, resulting in code execution in the repository’s CI/CD system. There are different sub-techniques to achieve code execution:

1. Direct PPE (d-PPE) – Cases where the attacker can directly modify the configuration file inside the repository. Since the pipeline is triggered by a new PR and run according to the configuration file – the attacker can inject malicious commands to the configuration file, and these commands are executed in the pipeline.
2. Indirect PPE (i-PPE) – Cases where the attacker cannot directly change the configuration files, or that these changes are not taken into account when triggered. In these cases, the attacker can infect scripts used by the pipeline in order to run code, for example, make-files, test scripts, build scripts, etc.
3. Public PPE – Cases where the pipeline is triggered by an open-source project. In these cases, the attacker can use d-PPE or i-PPE on the public repository in order to infect the pipeline.

Dependency tampering – Refers to a technique where an attacker can execute code in the DevOps environment or production environment by injecting malicious code into a repository’s dependencies. Thus, when the dependency is downloaded, the malicious code is executed. Some sub-techniques that can be used to achieve code execution include:

1. Public dependency confusion – A technique where the adversary publishes public malicious packages with the same name as private packages. In this case, because package search in package-control mechanisms typically looks in public registries first, the malicious package is downloaded.
2. Public package hijack (“repo-jacking”) – Hijacking a public package by taking control of the maintainer account, for example, by exploiting the GitHub user rename feature.
3. Typosquatting – Publishing malicious packages with similar names to known public packages. In this way, an attacker can confuse users to download the malicious package instead of the desired one.

DevOps resources compromise – Pipelines are, at the core, a set of compute resources executing the CI/CD agents, alongside other software. An attacker can target these resources by exploiting a vulnerability in the OS, the agent’s code, other software installed in the VMs, or other devices in the network to gain access to the pipeline.

Control of common registry – An attacker can gain control of a registry used by the organization, resulting in malicious images or packages executed by the pipeline VMs or production VMs.

3. Persistence

The persistency tactic consists of different techniques that an attacker may use for maintaining access to a victim environment:

Changes in repository – Adversaries can use the automatic tokens from inside the pipeline to access and push code to the repository (assuming the automatic token has enough permissions to do so). They can achieve persistency this way using several sub-techniques:

1. Change/add scripts in code – we can change some of the initialization scripts/add new scripts, so they download a backdoor/starter for the attacker, so each time the pipeline is executing these scripts, the attacker’s code will be executed too.
2. Change the pipeline configuration – we can add new steps in the pipeline to download an attacker-controlled script to the pipeline before continuing with the build process.
3. Change the configuration for dependencies locations – to use attacker-controlled packages.

Inject in Artifacts – some CI environments have the functionality for creating artifacts to be shared between different pipeline executions. For example, in GitHub we can store artifacts and download them using a GitHub action from the pipeline configuration.

Modify images in registry – In cases where the pipelines have permissions to access the image registry (for example, for writing back images to the registry after build is done) the attacker could modify and plant malicious images in the registry, which would continue to be executed by the user’s containers.

Create service credentials – A malicious adversary can leverage the access they already have on the environment and create new credentials for use in case the initial access method is lost. This could be done by creating an access token to the SCM, to the application itself, to the cloud resources, and more.

4. Privilege escalation

The privilege escalation techniques are used by an attacker to elevate the privileges in the victim’s environment, gaining higher privileges for already compromised resources:

Secrets in private repositories – Leveraging an already gained initial access method, an attacker could scan private repositories for hidden secrets. The chances of finding hidden secrets in a private repo are higher than in a public repository, as, from the developer’s point of view, this is inaccessible from outside the organization.

Commit/push to protected branches – The pipeline has access to the repository that may be configured with permissive access, which could allow to push code directly to protected branches, allowing an adversary to inject code directly into the important branches without team intervention.

Certificates and identities from metadata services – Once an attacker is running on cloud-hosted pipelines, the attacker could access the metadata services from inside the pipeline and extract certificates (requires high privileges) and identities from these services.

5. Credential access

Credential access techniques are used by an attacker to steal credentials:

User credentials – In cases where the customer requires access to external services from the CI pipeline (for example, an external database), these credentials reside inside the pipeline (can be set by CI secrets, environment variables, etc.) and could be accessible to the adversary.

Service credentials – There are cases where the attacker can find service credentials, such as service-principal-names (SPN), shared-access-signature (SAS) tokens, and more, which could allow access to other services directly from the pipeline.

6. Lateral movement

The lateral movement tactic refers to techniques used by attackers to move through different resources. In CI/CD environments, this may refer to moving to deployment resources, to build artifacts and registries, or to new targets.

Compromise build artifacts – As in other supply chain attacks, once the attacker has control of the CI pipelines, they can interfere with the build artifacts. This way, malicious code could be injected into the building materials before building is done, hence injecting the malicious functionality into the build artifacts.

Registry injection – If the pipeline is configured with a registry for the build artifacts, the attacker could infect the registry with malicious images, which later would be downloaded and executed by containers using this registry.

Spread to deployment resources – If the pipeline is configured with access to deployment resources, then the attacker has the same access to these resources, allowing the attacker to spread. This could result in code execution, data exfiltration and more, depending on the permissions granted to the pipelines.

7. Defense evasion

Defense evasion techniques could be used by attackers to bypass defenses used in a DevOps environment and allow attacks to continue under the radar:

Service logs manipulation – Service logs enable defenders to detect attacks in their environment. An attacker running inside an environment (for example, in the build pipelines) could change the logs to prevent defenders from observing the attack. This is similar to an attacker changing the history logs on a Linux machine, preventing any observer from seeing the commands executed by the attacker.

Compilation manipulation – if an attacker wishes to leave no traces in the SCM service, the attacker may change the compilation process in order to inject the malicious code. This may be done in several ways:

1. Changing the code on the fly – Changing the code right before the build process begins, without changing it in the repository and leaving traces in it.
2. Tampered compiler – Changing the compiler in the build environment to introduce the malicious code without leaving any traces before that process begins.

Reconfigure branch protections – Branch protection tools allow an organization to configure steps before a PR/commit is approved into a branch. Once an attacker has admin permissions, they may change these configurations and introduce code into the branch without any user intervention.

8. Impact

The impact tactic refers to the techniques an attacker could use for exploiting access to the CI/CD resources for malicious purposes, and not as another step in the attack, as these techniques could be noisy and easy to detect:

DDoS – An adversary could use the compute resources they gained access to in order to execute distributed denial of services (DDoS) attacks on external targets.

Cryptocurrency mining – The compute resources could be used for crypto mining controlled by an adversary.

Local DoS – Once the attacker is running on the CI pipelines, the attacker can perform a denial service attack from  said pipelines to customers by shutting down agents, rebooting, or by overloading the VMs.

Resource deletion – An attacker with access to resources (cloud resources, repositories, etc.) could permanently delete the resources to achieve denial of services.

9. Exfiltration

The exfiltration tactic refers to different techniques that could be used by an attacker to exfiltrate sensitive data from victim environment:

Clone private repositories – Once attackers have access to CI pipelines, they also gain access to the private repositories (for example, the GITHUB_TOKEN can be used in GitHub), and therefore could clone and access the code, thus gaining access to private IP.

Pipeline logs – An adversary could access the pipeline execution logs, view the access history, the build steps, etc. These logs may contain sensitive information about the build, the deployment, and in some cases even credentials to services, to user accounts and more.

Exfiltrate data from production resources – In cases where the pipelines can access the production resources, the attackers will have access to these resources as well. Therefore, they can abuse this access for exfiltrating production data.

Summary

The threat matrix we’ve presented here is meant to help defenders gain better understanding of the attack surface associated with DevOps environments. Better understanding of these threats will help Microsoft and other security vendors build stronger defenses and implement better coverage against these threats.

Organizations can use this matrix to map the weak spots in their own infrastructure and harden their environments, including the following steps:

• Hardening the repositories – permissions, protections, etc
• Hardening the pipeline and triggers, to ensure external entities cannot trigger unwanted pipeline executions
• Verifying dependencies
• Review access to secrets from pipeline execution and limit accordingly

The final step would be building a better organization culture for DevOps security and keeping the environment secure.

Learn more tangible ways to implement DevOps threat response tactics and operationalizing DevOps security.

READ MORE HERE