From: Andi Kleen <ak@muc.de>

Make sure we never access anything in kernel mapping while
doing the prefetch workaround checks on x86-64.

Originally suggested by Jamie Lockier.