• Sean Christopherson's avatar
    KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast() · ed69a6cb
    Sean Christopherson authored
    Acquire the per-VM slots_lock when zapping all shadow pages as part of
    toggling nx_huge_pages.  The fast zap algorithm relies on exclusivity
    (via slots_lock) to identify obsolete vs. valid shadow pages, because it
    uses a single bit for its generation number. Holding slots_lock also
    obviates the need to acquire a read lock on the VM's srcu.
    
    Failing to take slots_lock when toggling nx_huge_pages allows multiple
    instances of kvm_mmu_zap_all_fast() to run concurrently, as the other
    user, KVM_SET_USER_MEMORY_REGION, does not take the global kvm_lock.
    (kvm_mmu_zap_all_fast() does take kvm->mmu_lock, but it can be
    temporarily dropped by kvm_zap_obsolete_pages(), so it is not enough
    to enforce exclusivity).
    
    Concurrent fast zap instances causes obsolete shadow pages to be
    incorrectly identified as valid due to the single bit generation number
    wrapping, which results in stale shadow pages being left in KVM's MMU
    and leads to all sorts of undesirable behavior.
    The bug is easily confirmed by running with CONFIG_PROVE_LOCKING and
    toggling nx_huge_pages via its module param.
    
    Note, until commit 4ae5acbc4936 ("KVM: x86/mmu: Take slots_lock when
    using kvm_mmu_zap_all_fast()", 2019-11-13) the fast zap algorithm used
    an ulong-sized generation instead of relying on exclusivity for
    correctness, but all callers except the recently added set_nx_huge_pages()
    needed to hold slots_lock anyways.  Therefore, this patch does not have
    to be backported to stable kernels.
    
    Given that toggling nx_huge_pages is by no means a fast path, force it
    to conform to the current approach instead of reintroducing the previous
    generation count.
    
    Fixes: b8e8c830 ("kvm: mmu: ITLB_MULTIHIT mitigation", but NOT FOR STABLE)
    Signed-off-by: default avatarSean Christopherson <sean.j.christopherson@intel.com>
    Signed-off-by: default avatarPaolo Bonzini <pbonzini@redhat.com>
    ed69a6cb
mmu.c 171 KB