KVM: x86/mmu: Zap _all_ roots when unmapping gfn range in TDP MMU

In the Linux kernel, the following vulnerability has been resolved:

KVM: x86/mmu: Zap _all_ roots when unmapping gfn range in TDP MMU

Zap both valid and invalid roots when zapping/unmapping a gfn range, as

KVM must ensure it holds no references to the freed page after returning

from the unmap operation. Most notably, the TDP MMU doesn't zap invalid

roots in mmu_notifier callbacks. This leads to use-after-free and other

issues if the mmu_notifier runs to completion while an invalid root

zapper yields as KVM fails to honor the requirement that there must be

_no_ references to the page after the mmu_notifier returns.

The bug is most easily reproduced by hacking KVM to cause a collision

between set_nx_huge_pages() and kvm_mmu_notifier_release(), but the bug

exists between kvm_mmu_notifier_invalidate_range_start() and memslot

updates as well. Invalidating a root ensures pages aren't accessible by

the guest, and KVM won't read or write page data itself, but KVM will

trigger e.g. kvm_set_pfn_dirty() when zapping SPTEs, and thus completing

a zap of an invalid root _after_ the mmu_notifier returns is fatal.

WARNING: CPU: 24 PID: 1496 at arch/x86/kvm/../../../virt/kvm/kvm_main.c:173 [kvm]

RIP: 0010:kvm_is_zone_device_pfn+0x96/0xa0 [kvm]

Call Trace:

<TASK>

kvm_set_pfn_dirty+0xa8/0xe0 [kvm]

__handle_changed_spte+0x2ab/0x5e0 [kvm]

__handle_changed_spte+0x2ab/0x5e0 [kvm]

__handle_changed_spte+0x2ab/0x5e0 [kvm]

zap_gfn_range+0x1f3/0x310 [kvm]

kvm_tdp_mmu_zap_invalidated_roots+0x50/0x90 [kvm]

kvm_mmu_zap_all_fast+0x177/0x1a0 [kvm]

set_nx_huge_pages+0xb4/0x190 [kvm]

param_attr_store+0x70/0x100

module_attr_store+0x19/0x30

kernfs_fop_write_iter+0x119/0x1b0

new_sync_write+0x11c/0x1b0

vfs_write+0x1cc/0x270

ksys_write+0x5f/0xe0

do_syscall_64+0x38/0xc0

entry_SYSCALL_64_after_hwframe+0x44/0xae

</TASK>