Introducing kernel sanitizers on Microsoft platforms

As part of Microsoft’s commitment to continuously raise security baselines, we have been introducing innovations to the foundation of the chip-to-cloud security outlined in the Windows 11 Security Book. Strong foundational security enables us to build defenses from the ground up and develop secure-by-design products that are hardened against current and future threats. 

These innovations enable Microsoft to further improve the security embedded into Windows and other Microsoft products before they are delivered to our customers. For example, in the past few years, we have been architecting and developing support for kernel sanitizers—powerful detection features that can uncover bugs in kernel-mode components—on different Microsoft platforms. With reach and precision that exceed the capabilities of other similar features, kernel sanitizers enable Microsoft engineering teams to identify and fix vulnerabilities earlier in the software development cycle than ever before possible.

Various teams at Microsoft use these kernel sanitizers for fuzzing, stress-testing, and other development tasks. Kernel sanitizers have shown that they have the potential to eliminate whole classes of memory bugs, and we continue to expand implementations of these features from Windows to other platforms, including Xbox and Hyper-V. This work leads to lasting improvements in software quality and security across Microsoft products and services, and ultimately contributes to better and more secure user experiences for customers.

In this blog post, we share technical details of the work by Microsoft Offensive Research & Security Engineering (MORSE) on kernel sanitizers, the impact they have on Windows and other platforms, and the opportunities they provide to continuously advance built-in security.

User-mode AddressSanitizer and why we took security even further

AddressSanitizer (ASAN) is a compiler and runtime technology initially developed by Google that detects several classes of memory bugs in C/C++ programs, including critical security bugs like buffer overflows. The support for ASAN on Windows user-mode applications was introduced in 2019, and it was extended in 2021 to cover Xbox user-mode applications.

Within Microsoft, ASAN has been leveraged to identify and fix bugs in user-mode software components and is now used routinely on user-mode components during development.

However, user mode is only one layer of Windows. The Windows operating system is a modern and complex piece of software that involves several components operating in different privilege domains and interacting with each other:

While ASAN is effective in catching bugs in Windows user-mode components, we needed a similar feature to equally detect bugs in the other layers of the operating system. We began with the Windows kernel attack surface.

Introducing the Kernel AddressSanitizer

The Kernel AddressSanitizer (KASAN) is a variation of the user-mode ASAN that has been architected to work specifically for the Windows kernel and its drivers.

Implementation details

Let’s dive into the technical details of the implementation, focusing on the use case where KASAN is enabled on a driver. It should be noted that the same principle applies when KASAN is enabled on the Windows kernel itself.

Tracking logic: The shadow

KASAN works by first tracking the validity of each byte of memory in the kernel virtual address space using a new 16TB virtual memory region called the shadow, which acts similarly to a large bitmap that indicates whether each byte of the kernel virtual address space is valid or not.

The shadow is of size 1/8 of the entirety of the kernel virtual address space size, and linearly backs all of it:

Each set of 8 bytes in the kernel address space is backed by one byte in the shadow. As the KASAN shadow is a region of kernel memory, it resides within the kernel virtual address space and therefore implicitly backs itself too:

Initially, the shadow is just a large 16TB read-only region that maps into a single 4KB physical page full of zeroes. Therefore, even though 16TB of virtual memory is reserved, only a single page of physical memory is used:

Later, at run time, when the kernel performs memory allocations, for example via ExAllocatePool2(), it makes the shadow that backs these allocations writable. It does this by dynamically allocating new physical pages and remapping portions of the shadow to these pages, while keeping the other portions untouched and still pointing to the initial read-only zero page:

This process of splitting the shadow takes place during each backend memory allocation. Overall, the memory consumption of the shadow increases as portions of it are progressively made writable to back the kernel memory allocations.

The KASAN runtime then proceeds to dynamically initialize the shadow values of each memory allocation, depending on its current state. During a memory allocation once the shadow has been made writable for the allocated buffer, the KASAN runtime marks the allocated buffer as valid in the shadow, and the paddings below and above it as invalid:

Later, when this buffer gets freed, the KASAN runtime marks it as entirely invalid in the shadow:

By updating the shadow contents this way, the KASAN runtime maintains a consistent view of which bytes of memory are valid and which bytes are invalid. From there, the ASAN instrumentation, which we describe below, then gets used as part of the verification logic to enforce validity checks on memory accesses using the information provided by the shadow.

Verification logic: The ASAN instrumentation

As part of enabling KASAN on a target kernel-mode component such as a kernel driver, the component must be recompiled with a specific set of compiler flags that cause the compiler to insert the ASAN instrumentation into the compiled binary directly.

The compiler chooses one of two verification methods:

1. Function calls to __asan_*(): Before each access that a program makes to memory, the compiler inserts a call to one of the __asan_{load,store}{#n}(Address) functions, chosen depending on the specifics of the access, and passes as argument the memory address about to be accessed. For example, if the access is a write of two bytes, then __asan_store2() is chosen and is therefore called by the compiled program before the access is performed. We won’t discuss the details of the __asan_*() functions as public documentation is available, but overall, these functions calculate the shadow address of the memory about to be accessed and verify whether the shadow says that #n bytes starting from the address given as argument are valid. If not, these functions are expected to halt program execution.
2. Verification bytecodes: To improve performance, in certain cases instead of inserting a function call, the compiler directly inlines a bytecode that implements the same logic—the  bytecode reads the value of a global variable called __asan_shadow_memory_dynamic_address (which must exist in the program), calculates from it the shadow address of the memory about to be accessed, and verifies whether the shadow says that the memory is valid. If not, the bytecode calls an __asan_report_*() function that is expected to halt program execution.

While this ASAN instrumentation was initially developed for user-mode components, it is reused as-is in KASAN, such that:

• The NTOS kernel has a KASAN runtime that exports the __asan_*() functions. These functions can be imported by drivers compiled with KASAN, and are also used when NTOS itself is compiled with KASAN.
• Within NTOS, an __asan_shadow_memory_dynamic_address global variable is declared, and is initialized and used when NTOS is compiled with KASAN, but is not exported to drivers. For drivers, a different mechanism based on KasanLib is used and is described below.
• The halting of program execution is implemented in the form of a kernel bug check: when an access is made to memory bytes marked as invalid in the shadow, KASAN triggers a KASAN_ILLEGAL_ACCESS (0x1F2) bug check. The parameters of this bug check include useful information for debugging, such as the type of memory being accessed (heap, stack, and so forth), the number of bytes being accessed, whether the access is a read or a write, along with additional metadata.

The ASAN instrumentation therefore constitutes the KASAN verification logic: it verifies whether each memory byte that the component accesses at run time is marked as valid in the KASAN shadow, and triggers a bug check if not, to report any illegal memory access.

Challenges with the instrumentation

There were several challenges in getting the ASAN instrumentation to work in KASAN, especially in cases where the compiler inserts a bytecode instead of a function call.

Using the correct calculation

The expected calculation to get the shadow address of a regular address is the following:

Shadow(Address) = ShadowBaseAddress + OffsetWithinAddressSpace(Address) / 8

In the context of the user-mode AddressSanitizer, the user-mode address space starts at address 0x0. Therefore, any user-mode address is equal to the offset of that address within the user-mode address space:

OffsetWithinUserAddressSpace(UserAddress) = UserAddress – 0 = UserAddress

For this reason, the verification bytecodes inserted as part of the ASAN instrumentation use the following formula, where __asan_shadow_memory_dynamic_address contains the base address of the shadow:

Shadow(Address) = __asan_shadow_memory_dynamic_address + (Address / 8)

The following is an example of a generated bytecode assembly:

mov rcx, cs:__asan_shadow_memory_dynamic_address
shr rax, 3
add rcx, rax

Here the bytecode calculates the shadow address of RAX by reading __asan_shadow_memory_dynamic_address and adding to it RAX right-shifted by 3 (meaning divided by 8). This implements the aforementioned formula to get the shadow of an address.

In the context of the kernel AddressSanitizer however, the kernel-mode address space starts at address 0xFFFF800000000000:

OffsetWithinKernelAddressSpace(KernelAddress) = KernelAddress – 0xFFFF800000000000 != KernelAddress

Therefore, reusing the simplified formula as-is in KASAN would not be correct: it would result in the bytecodes using the wrong shadow addresses when verifying memory accesses. We solved that issue in KASAN by initializing __asan_shadow_memory_dynamic_address to a different value that is not exactly the KASAN shadow base address:

__asan_shadow_memory_dynamic_address = KasanShadowBaseAddress – (0xFFFF800000000000 / 8)

Developing the formula using this value gives the following:

Shadow(KernelAddress) = __asan_shadow_memory_dynamic_address + (KernelAddress / 8) = KasanShadowBaseAddress – (0xFFFF800000000000 / 8) + (KernelAddress / 8) = KasanShadowBaseAddress + (KernelAddress - 0xFFFF800000000000) / 8 = KasanShadowBaseAddress + OffsetWithinKernelAddressSpace(KernelAddress) / 8

The formula therefore falls back to the expected calculation with KASAN: the bytecodes take the base address of the KASAN shadow, add to it the offset of the address within the kernel address space divided by 8, and this results in the correct shadow address for the given kernel address.

Using this trick, we avoided the need to make a compiler change to modify the bytecode generation for KASAN.

Dealing with non-tracked memory

When we described how the KASAN shadow is mapped, we did not explain why we were using a splitting mechanism with a zeroed page. The reason is simple: the verification bytecodes always want to read the shadow of the buffers they verify, and they do not have any knowledge about whether a buffer has a shadow that backs it or not. There must therefore always be a shadow mapped for every byte of kernel virtual address memory, and we achieve that thanks to the splitting mechanism, which guarantees that a shadow always exists while minimizing the memory consumption of the non-tracked regions by having their shadow point to a single zeroed physical page.

The fact that the physical page used in the splitting mechanism is full of zeroes causes KASAN to always consider non-tracked memory as valid.

Dealing with user-mode pointers

The Windows kernel and its drivers are allowed to directly access user-mode memory, for example to fetch the user-mode arguments passed to a syscall. This creates an issue with the verification bytecodes, because they need to get the shadow of an address that is outside of the kernel address space and that therefore does not have a shadow.

To deal with this case, we pass a compiler flag as part of KASAN that instructs the compiler to never use bytecodes and always prefer function calls to __asan_*(), except when the compiler is certain that the accesses are to stack memory.

This means in practice that in order to verify the accesses to local variables on the stack, the compiler generates verification bytecodes, but for any other access the compiler uses function calls to __asan_*(). Given that these __asan_*() functions are implemented in the KASAN runtime, we have full control over their verification logic and can make sure to exclude user-mode pointers from the verification via a simple if condition.

Using this trick, we again avoided the need to make a compiler change to have the instrumentation deal with user-mode pointers.

Telling the kernel to export KASAN support

By default, the kernel does not create the KASAN shadow, and does not export the KASAN runtime. In other words, it does not make KASAN available to drivers by default. For this to be done, the user must explicitly set the following registry key:

HKLM\System\CurrentControlSet\Control\Session Manager\Kernel\KasanEnabled

The bootloader reads this key at boot time and decides based on its value whether or not to instruct the kernel to make KASAN support available to drivers.

With this established, the following sections contain details of how the kernel loads drivers compiled with KASAN.

Loading kernel drivers with KASAN

For drivers compiled with KASAN, a small code-only library called KasanLib is linked into the final driver binary and does two things:

1. It declares an __asan_shadow_memory_dynamic_address global variable that remains local to the driver itself and is not exported to the kernel namespace. The verification bytecodes described earlier that get inserted into the driver use this global variable as part of their calculation of the KASAN shadow.
2. It publishes a section called “KASAN” in the resulting PE binary of the driver. This section contains information and metadata, the format of which may change in the future and is not relevant to discuss here.

Upon loading a driver, the kernel verifies whether the driver has a “KASAN” section, and can take two paths:

1. If the driver has a “KASAN” section and the KasanEnabled registry key is not set, then the kernel will refuse to load the driver. This is to prevent the system from malfunctioning; there is, after all, no way the driver is going to work, since it will try to use a shadow that wasn’t created by the kernel and call a runtime that the kernel does not export.
2. If the driver has a “KASAN” section and the KasanEnabled registry key is set, then the kernel parses this section in order to initialize KASAN on the driver. Part of this initialization includes setting the value described earlier in the driver’s __asan_shadow_memory_dynamic_address global variable. After this initialization is complete, the “KASAN” section is no longer used and is discarded from memory to save up kernel memory.

From then on, the driver can start executing.

How it all falls together: example of a buggy driver

We have now exposed all the ingredients required for KASAN to work on drivers: how the shadow is created, how the instrumentation operates, how the kernel exports KASAN support, and how KASAN gets initialized on drivers when they are loaded. To give an example of how it all falls together, let’s consider a hypothetical driver that we compiled with KASAN.

We have set the KasanEnabled registry key in our system, and the kernel has therefore created a KASAN shadow and is exporting the KASAN runtime. We proceed to load the driver in the system. The kernel sees that the PE of the driver has a “KASAN” section, parses it, initializes KASAN on the driver, and discards the section. The driver finally starts executing.

Let’s assume that the driver contains this buggy code:

PCHAR buffer;
buffer = ExAllocatePool2(POOL_FLAG_NON_PAGED, 18, WHATEVER_TAG);
buffer[18] = 'a';

Here, a heap buffer of size 18 bytes is allocated. During this allocation, the KASAN runtime initialized two redzones below and above the buffer, as described earlier. Then an ‘a’ is written into the 19^th byte of the buffer. This is, of course, an out-of-bounds write access, which is incorrect and can cause a serious security risk.

Given that our driver was compiled with KASAN, it was subject to the ASAN instrumentation, meaning that the actual compiled code looks like the following:

PCHAR buffer;
buffer = ExAllocatePool2(POOL_FLAG_NON_PAGED, 18, WHATEVER_TAG);
__asan_store1(&buffer[18]);
buffer[18] = 'a';

Here the compiler inserted a function call to __asan_store1() and did not choose a verification bytecode because it couldn’t conclude that “buffer” was a pointer to stack memory (which it is not).

__asan_store1() is part of the KASAN runtime that the kernel exported and that the driver imported. This function looks at the shadow of &buffer[18], sees that it is marked as invalid (because the byte at this address is part of the right redzone of the buffer), and proceeds to issue a KASAN_ILLEGAL_ACCESS bug check to halt system execution.

As the owners of the system, we can then collect the crash dump and investigate what was the memory safety bug that KASAN detected using the actionable information provided alongside the KASAN bug check.

Without KASAN, this bug would not be easily observed. With KASAN, however, it is immediately detected before the bug triggers and turns into a real security risk. As such, KASAN is able to detect whole classes of memory bugs that could otherwise remain undiscovered.

Granularity, and memory regions covered

As can be deduced from the details we provided thus far, KASAN operates at the byte granularity. KASAN is currently able to detect illegal memory accesses on several types of memory regions:

• The global variables
• The kernel stacks
• The pool allocators (ExAllocatePool*())
• The lookaside list allocators (ExAllocateFromLookasideListEx(), etc.)
• The IO/contiguous allocators (MmMapIoSpaceEx(), MmAllocateContiguousNodeMemory(), etc.)

Internally, the support for these regions is implemented using a KASAN API that is also exported by the NTOS kernel. Microsoft will continue to improve this API to expand its implementation to other scenarios.

Thanks to the ability to detect bugs at the byte granularity and the large number of memory regions covered, KASAN exceeds the capabilities of existing bug-detection technologies such as the Special Pool, which typically operate at a coarser granularity and do not cover the kernel stacks and other regions.

Performance cost

Naturally, the KASAN shadow consumes memory, and the validity checks inserted by the ASAN instrumentation consume CPU time and increase the size of the compiled binaries.

Some effort has gone into micro-optimizing KASAN by limiting the number of instructions that the KASAN runtime emits, by making the KASAN shadow NUMA-aware, by compressing the KASAN metadata in order to reduce the binary sizes, and so forth.

Overall, KASAN currently introduces a ~2x slowdown, which we measured using widely available benchmarking tools. As such, KASAN cannot be seen as a production feature since its performance cost is not negligible. This cost, however, is acceptable for debug, development, stress-testing, or security-related setups.

It should be noted that this cost is higher than that of existing technologies, such as the Special Pool, and that KASAN does not have a performance impact when not explicitly enabled. In other words, KASAN does not affect the performance of Windows 11 by default.

Immediate impact

Microsoft generates special builds of Windows, called MegaAsan builds, that produce fully bootable Windows disks that have KASAN enabled on the Windows kernel and on more than 95% of all kernel drivers shipped by Microsoft in Windows 11.

By using these builds in testing, fuzzing, but also in simple desktop setups, MORSE has been able to identify and fix more than 35 memory safety bugs in various drivers and in the Windows kernel, that were not previously detectable by existing technologies.

We also implemented KASAN support for the Xbox kernels and the drivers they load, and similarly generate builds of Xbox systems with KASAN enabled. As such, KASAN also contributes to the quality and security of the Xbox product line.

Extending ASAN to the other ring0 domains

We have so far discussed KASAN on Windows kernel drivers and on the Windows kernel:

Having KASAN is a considerable step forward, because it provides precise detection of memory errors on large and critical parts of the system in a way that wasn’t achievable before. Following up on our work on KASAN, we developed similar detection capabilities on the remaining parts of the system.

Introducing SKASAN…

The Secure kernel is a different kernel, completely separated from the Windows kernel, that executes in a more privileged domain and is in charge of a number of security operations in the system. It is part of virtualization-based security on Windows.

We developed the Secure Kernel AddressSanitizer (SKASAN), which covers the Secure kernel and a few of the modules it loads dynamically.

SKASAN has a number of similarities with KASAN. For example, the SKASAN support for Secure Kernel modules is implemented using a “SKASAN” section, comparable to the “KASAN” section used in Windows kernel drivers. Overall, SKASAN works similarly to KASAN, but simply applied to the Secure kernel domain.

…and HASAN

Finally, Hyper-V is the Microsoft hypervisor that plays a central role on Windows and in Azure, and it too could benefit from the capabilities that ASAN provides; we therefore developed the Hyper-V AddressSanitizer (HASAN) which is yet another ASAN implementation but tied to the Hyper-V kernel.

Same, but different… but still same

KASAN, SKASAN, and HASAN are built on the same logic, which is having a shadow and a compiler instrumentation, and overall have similar costs in terms of memory consumption and slowdown.

Some inherent differences do exist, however. First, the Windows kernel, Secure kernel, and Hyper-V kernel have different allocators, and the *ASAN support for them differs accordingly. Second, the memory layout of these kernels is not the same, and this leads to drastic implementation differences; for instance, HASAN actually uses two different shadows concatenated together.

We leave the rest of the technical differences as a reverse engineering exercise to interested readers.

The final picture, and results

As of November 2022, we have developed and stabilized KASAN, SKASAN, and HASAN. Combined together, these deliver precise detection of memory errors on all the kernel-mode components that execute on Windows 11:

We produce internal MegaAsan builds with all of these *ASAN implementations enabled, and internal teams are using them in a number of fuzzing and stress-testing scenarios. As a result, we have been able to identify and fix dozens of memory bugs of various severity:

Finally, as part of our *ASAN work we have also applied numerous improvements and cleanups to various areas, such as the Windows and Hyper-V kernels, but also to the Microsoft Visual C++ (MSVC) compiler to improve the *ASAN experience on Microsoft platforms.

Overall, these *ASAN features have the potential to eliminate whole classes of memory bugs and, going forward, will significantly contribute to ensuring the quality and security of the Microsoft products.

This concludes our first blog post on kernel sanitizers. Beyond *ASAN, we have implemented several other sanitizers that specialize in uncovering other classes of bugs. We will communicate about them in future posts.

READ MORE HERE