How to investigate service provider trust chains in the cloud

In a recent Microsoft blog post, we documented technical guidance for organizations to protect themselves from the latest NOBELIUM activity that was found to target technology service providers, which are privileged in their downstream customer tenants, as a method to gain access to their downstream customers and other organizations within the trust chain.

Microsoft Detection and Response Team (DART) has been assisting multiple organizations around the world in investigating the impact of NOBELIUM’s activities. While we have already engaged directly with affected customers to assist with incident response related to NOBELIUM’s recent activity, our goal with this blog is to help you answer the common and fundamental questions: How do I determine if I am a victim? If I am a victim, what did the threat actor do? How can I regain control over my environment and make it more difficult for this threat actor to regain access to our environments?

This blog outlines steps incident responders can take to investigate potential abuse of these delegated admin permissions, independent of the threat actor. In this blog, we’ll cover:

• Identifying trust chains in Microsoft 365 and Microsoft Azure.
• Investigating trust chains.
• Mitigating malicious activity.
• Recommendations: detect and protect.

Identifying trust chains in Microsoft 365 and Microsoft Azure

Several types of trust chains exist in Microsoft 365 and Microsoft Azure, which include delegated administration privileges (DAP), Azure admin-on-behalf-of (AOBO), Microsoft Azure Active Directory (Azure AD) business-to-business (B2B), multi-tenant Azure AD applications, as well as guest users. Many of these trust chains can grant a high level of access to Azure resources and Microsoft 365, requiring close monitoring.

Delegated administration privileges

DAP is a method by which your service providers can administer a Microsoft 365 environment without needing to maintain local identities. DAP can be beneficial for both the service provider and end customer because it allows a service provider to administer a downstream tenant using their own identities and security policies. More information about delegated administration privileges and other admin-on-behalf-of scenarios are available in the following resources:

Service providers with DAP can be identified in the Microsoft 365 admin center by navigating to Settings then to Partner relationships. In the Partner relationships pane, you can view a list of all service providers that have established a billing relationship with the tenant and whether the service provider has any roles assigned (refer to Figure 1).

Figure 1. Identifying DAP as a downstream customer.

While end customers cannot see a list of all users in the service provider’s tenant that can make administrative changes to the end customer tenant, they can view logins by a service provider (refer to Figure 2) by viewing the Azure Active Directory sign-in logs and filtering for a Cross tenant access type of Service provider. The results can be exported by clicking Download and leveraged to further target your triage across Azure and Microsoft 365.

Figure 2: Sign-ins by service providers.

Azure AOBO

Azure AOBO is similar in nature to DAP, albeit the access is scoped to Azure Resource Manager (ARM) role assignments on individual Azure subscriptions and resources, as well as Azure Key Vault access policies. Azure AOBO brings similar management benefits as DAP does.

Note: To fully assess the AOBO permissions in your subscriptions, ensure you have granted access to the Global Administrator who will be assessing service provider access to all subscriptions in each tenant. Read our documentation for details on how to elevate to user access administrator on the tenant root group.

The Azure AOBO access is added at subscription creation time and can be seen under Access control (IAM) on a given Azure subscription (refer to Figure 3).

Figure 3: Foreign Principal with Owner role on subscription.

If you have multiple subscriptions, consider running the following command to identify subscriptions where service providers might have access to resources:

Get-AzSubscription | % { Set-AzContext -Subscription $_; Get-AzRoleAssignment -Scope "/subscriptions/$($_.Id)" | Where-Object {$_.DisplayName -like "Foreign Principal for * in Role 'TenantAdmins' (*)"} | Select DisplayName, Scope | Format-Table}

It is also possible to grant CSPs direct access to Key Vaults. The following PowerShell command can be used to identify Key Vaults with access policies that allow access via AOBO:

Get-AzKeyVault | % { $vault = Get-AzKeyVault -VaultName $_.VaultName; if ($vault.AccessPolicies | Where-Object {$_.DisplayName -like "Foreign Principal for '*' in role 'TenantAdmins' (*)"}) { $vault |select VaultName,ResourceId | Format-Table}}

The Azure Red Team tool Stormspotter can also be used in addition to the above commands for large environments.

The information gathered from the previous steps will be used to scope log review during triage.

Azure AD B2B

Azure AD B2B accounts (guests) can be used to administer Azure and Microsoft 365 resources. This method of administrative access leverages an individual existing identity in another tenant and is not typically recommended by Microsoft due to the limitations of control over the identity. Investigators should be mindful of the many ways in which guests can be granted access to resources in Microsoft 365, which may include Exchange Online roles and SharePoint online roles. The guidance for this type of identity should be considered non-exhaustive and focused on Azure AD and Azure specifically. For more information, read our documentation about Azure AD B2B best practices.

Azure subscriptions

In order to fully assess the B2B permissions in your subscriptions, ensure you have granted access to users who will be assessing service provider access to all subscriptions in each tenant by following the following guidance: Elevate access to manage all Azure subscriptions and management groups.

Azure AD B2B identities granted Azure roles appear in the Access control blade in the Azure Portal with (Guest) next to them (see Figure 4).

Figure 4: Guest user with Owner role on subscription.

Azure AD B2B identities can be systematically identified with the following command, which will produce a list of identities and resources that can be used to target initial triage.

Get-AzSubscription | % { Set-AzContext -Subscription $_; Get-AzRoleAssignment -Scope "/subscriptions/$($_.Id)" | Where-Object {$_.SignInName -like "*#EXT#@*"} | Select DisplayName, SignInName, Scope | Format-Table}.

Microsoft 365 (Azure AD)

Azure AD B2B identities that have been granted roles in Azure AD can be viewed in the assignments blade of Azure AD Privileged Identity Management blade. Filtering for “#EXT#” will allow you to view all guest users assigned to administrative roles (see Figure 5).

Figure 5: Filtering for guest users.

The following PowerShell can also be used to identify guest accounts with administrative roles. This identity information will be used to help target triage.

Get-AzureADDirectoryRole | Get-AzureADDirectoryRoleMember | Where-Object {$_.UserPrincipalName -like "*#EXT#@*"}.

Investigating trust chains

In Microsoft 365 and Microsoft Azure, there are multiple points of observability where activity via trust chains can be seen, including the Azure AD Audit log, Azure Activity log, Intune audit log, and the unified audit log. Using the data collected in the “identification” phase, a targeted review of logs can be performed to identify trust chain abuse. Each log should be reviewed for activity sourced from trust chains, specifically with a focus on activity that facilitates persistence, data collection, and reconnaissance.

Figure 6: Indicators of tenant compromise.

Azure AD

Adversaries will often establish persistence using various methods including the creation of new service principals, addition of new secrets on to existing application registrations, service principals, creation of new privileged users, and the takeover of existing privileged accounts. You can identify modifications made to Azure AD via trust chains by reviewing the Azure AD Audit log and filtering for the users identified as having recent sign-ins during the “identification” phase. Some specific activities of interest:

• Password resets.
• Modification of service principals.
• Addition of users to privileged roles.
• Changes to multifactor authentication (MFA).
• Creation of new users.

Unified audit log

The unified audit log can be used to identify activity performed via trust chains in SharePoint Online, Exchange Online, Azure AD, and other Microsoft 365 products.

Keep in mind that the unified audit log ingests data from across Azure AD and Office 365 and retains this data for at least 90 days, making it an incredibly valuable source of centralized information, typically with longer retention than the source (for example, Azure AD only retains data for up to 30 days). If E5 licenses are applied, this data will be retained for 1 year, with a maximum configurable retention period of 10 years using Advanced Audit.

The Search-UnifiedAuditLog cmdlet can be used to search for actions performed by the users identified during the “identification” phase. Alternatively, the logs can be searched using a GUI in the Microsoft 365 Defender portal.

Azure activity

Access by a malicious actor to Azure resources enables them to exfiltrate data and move laterally to other environments that are connected to the targeted Azure environment. Actors with access to the subscription can deploy new resources, access existing resources via virtual machine extensions, or simply exfiltrate data and keys directly from the Azure subscription. Access and manipulation of Azure resources can be audited by reviewing the Azure Activity logs that are present in each subscription. Refer to our blog post, Investigating Azure Activity with Microsoft Sentinel, for information about using Microsoft Sentinel queries to identify areas of interest.

Microsoft Endpoint Manager

It may be possible for a malicious actor to access Microsoft Endpoint Manager via various trust chains and as Microsoft Endpoint Manager manages the configuration of devices, it is another important audit log to review. The Microsoft Endpoint Manager audit log can be accessed under the Tenant Administration blade of the Microsoft Endpoint Manager portal. In the audit log, the initiator, “Partner,” can be used to filter for actions initiated by Partners. Actions taken by guest users, identified as having privileges during the “identification” phase, will need to be searched for by User Principal Name. These log events should be reviewed to ensure no malicious activity occurred via the identified trust chains.

Figure 7: Actions initiated by Partners.

Mitigating malicious activity

If during the investigation, malicious activity is discovered and confirmed or unneeded and overly permissive trust chains are discovered, decisive action should be taken to block or minimize access. Depending on the type of trust chain, different steps may need to be taken to block access. It is not recommended to fully delete the artifacts until the conclusion of any ongoing investigation; deleting certain artifacts may delay or make completing an investigation more difficult. Customers should talk with their service provider to understand what protections they have in place, and in the event of potential malicious activity, notify their service provider to obtain their assistance with activity validation.

Delegated administrative privileges

DAP should be removed if it is not required for the active, day-to-day administration of the tenant by the service provider. In some cases, permissions are required to facilitate administration by the service provider. In these instances, Microsoft will be introducing granular delegated admin privileges (GDAP), which will allow partners to control more granular and time-bound access to their customers’ workloads.

We recommend service providers leverage named accounts in the customer tenant to reduce blast radius and risk. In the event there is evidence of compromise stemming from a service provider relationship, it is recommended to remove the delegated admin privileges from the relationship at least until the conclusion of the investigation.

To remove delegated admin privileges, navigate to Settings then to Partner relationships in the Microsoft 365 admin center. From the Partner relationships pane, click on the relationship and then select Remove roles in the details pane. Taking this action will prevent the service provider from accessing the tenant as a Global Administrator or Helpdesk Administrator. Removing this access will not change or alter the billing relationship or licenses currently purchased through the service provider.

Azure AOBO

Azure AOBO access should be removed if it is not required for the active, day-to-day administration of the Azure subscription. If the service provider requires access to the Azure subscriptions, least privilege should be applied by adding the Foreign Principal with the proper roles and permissions. If there is evidence of compromise stemming from a service provider, the foreign group principal should be removed from every Azure Subscription.

Permissions granted via AOBO can be monitored by leveraging Azure Policy. You can deploy an Azure Policy at the Tenant Root Group that will throw non-compliance if a foreign principal is assigned permissions to resources in Azure. While the Azure Policy cannot block the creation of subscriptions with foreign principals, it simplifies reporting on the existence of them and allows the automation of their removal or prevention of creation if desired.

Azure AOBO permissions can be removed by navigating to the Access control (IAM) blade on the impacted subscription, selecting the foreign principal for the service provider, and then pressing Remove.

Figure 8: Removing Azure AOBO permissions for the foreign principal. 

The foreign principal can be added back with more specific permissions if required, following the best practice of least privilege.

Recommendations

Detect

The centralized availability of logging is critical for responding to and investigating potential incidents and is the top blocker to DART investigations of this type. If an organization is monitoring their cloud environment for privileged access and administrative changes, then malicious activities involving delegated admin privilege abuse should be discoverable and alerted.

Cloud activity logs should be ingested into a security information and event manager (SIEM) and retained for analysis. This should include:

• Office 365 unified audit log.
• Azure AD admin audit logs and sign-in logs.
• Microsoft Endpoint Manager audit log.
• Azure Activity logs and specific data plane logs, such as Azure Key Vault and Storage Azure Policy, can be leveraged to enforce a consistent logging standard.

As incident responders, DART are at their most effective when there is data available which is rich in both quantity and quality. One log type of interest is sign-in logs; identity events can tell us a lot about an actor’s activity. Patterns can often be identified in these logs to give us confidence in our analysis of threat actor activity. These patterns can be something as simple as an IP address matching, or as complex as a UserAgent string, time of day, and application ID match.

With that said, the most critical logging is that of administrative activity. Any usage of or actions performed by administrative accounts are of great interest and should be monitored and deconflicted. In enterprise environments, most changes are usually made during approved change windows, and changes outside of this should be assessed for their validity and integrity.

Logs on their own are useful, but alerting is critical to surfacing unusual or malicious activity in a timely manner. The Microsoft 365 Defender portal has some useful alerting built-in to identify suspicious activity. Some examples of these are:

• Elevation of Exchange admin privilege.
• eDiscovery search started or exported.
• Creation of forwarding or redirect rule.

Custom alerts can also be created to alert for other types of activity. Another excellent tool for alerting is Microsoft Defender for Cloud Apps (formerly Microsoft Cloud App Security). This tool can ingest data from Azure AD, Office 365, Azure, Defender for Endpoint, Defender for Identity, along with many third-party services. A policy engine can be used to create alert policies based on built in templates or custom definitions. Some examples of the templated policies are:

• Administrative activity from a non-corporate IP address.
• Unusual administrative activity (by user).
• Unusual addition of credentials to an OAuth application.
• Suspicious OAuth application file download activities.
• Multiple virtual machine creation activities.

Protect

We recommend customers engage in a dialogue with their service providers on a regular basis to understand security controls that are in place for access to their tenant. Access to resources by the service provider should be closely monitored, and if unused for a period, removed following a strong least privilege process.

Review the Microsoft Security Best Practices and Azure Security Benchmark for guidance on improving security posture in combination with Microsoft Secure Score in the Microsoft 365 Security Center and Secure Score in Microsoft Defender for Cloud.

Some specific examples for protecting administrative access includes using just-in-time administrative solutions such as Privileged Identity Management, including regular reviews of administrators to ensure their access is still required. MFA is also critical, and not just the enablement of MFA, but also ensuring that all administrators have registered MFA methods. DART has seen threat actors find an account which is enabled for MFA but has never been registered, and this allows the threat actor to register their own MFA details, elevating their level of trust in the environment.

Learn more

To learn more about Microsoft Security solutions, visit our website. Bookmark the Security blog to keep up with our expert coverage on security matters. Also, follow us at @MSFTSecurity for the latest news and updates on cybersecurity.

READ MORE HERE