Commit c1a88e91 authored by Mark Rutland's avatar Mark Rutland Committed by Catalin Marinas

arm64: kasan: avoid TLB conflicts

The page table modification performed during the KASAN init risks the
allocation of conflicting TLB entries, as it swaps a set of valid global
entries for another without suitable TLB maintenance.

The presence of conflicting TLB entries can result in the delivery of
synchronous TLB conflict aborts, or may result in the use of erroneous
data being returned in response to a TLB lookup. This can affect
explicit data accesses from software as well as translations performed
asynchronously (e.g. as part of page table walks or speculative I-cache
fetches), and can therefore result in a wide variety of problems.

To avoid this, use cpu_replace_ttbr1 to swap the page tables. This
ensures that when the new tables are installed there are no stale
entries from the old tables which may conflict. As all updates are made
to the tables while they are not active, the updates themselves are
safe.

At the same time, add the missing barrier to ensure that the tmp_pg_dir
entries updated via memcpy are visible to the page table walkers at the
point the tmp_pg_dir is installed. All other page table updates made as
part of KASAN initialisation have the requisite barriers due to the use
of the standard page table accessors.
Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
Reviewed-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Tested-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
Tested-by: default avatarJeremy Linton <jeremy.linton@arm.com>
Cc: Laura Abbott <labbott@fedoraproject.org>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
parent 50e1881d
...@@ -16,6 +16,7 @@ ...@@ -16,6 +16,7 @@
#include <linux/memblock.h> #include <linux/memblock.h>
#include <linux/start_kernel.h> #include <linux/start_kernel.h>
#include <asm/mmu_context.h>
#include <asm/page.h> #include <asm/page.h>
#include <asm/pgalloc.h> #include <asm/pgalloc.h>
#include <asm/pgtable.h> #include <asm/pgtable.h>
...@@ -108,15 +109,6 @@ static void __init clear_pgds(unsigned long start, ...@@ -108,15 +109,6 @@ static void __init clear_pgds(unsigned long start,
set_pgd(pgd_offset_k(start), __pgd(0)); set_pgd(pgd_offset_k(start), __pgd(0));
} }
static void __init cpu_set_ttbr1(unsigned long ttbr1)
{
asm(
" msr ttbr1_el1, %0\n"
" isb"
:
: "r" (ttbr1));
}
void __init kasan_init(void) void __init kasan_init(void)
{ {
struct memblock_region *reg; struct memblock_region *reg;
...@@ -130,8 +122,8 @@ void __init kasan_init(void) ...@@ -130,8 +122,8 @@ void __init kasan_init(void)
* setup will be finished. * setup will be finished.
*/ */
memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir));
cpu_set_ttbr1(__pa(tmp_pg_dir)); dsb(ishst);
flush_tlb_all(); cpu_replace_ttbr1(tmp_pg_dir);
clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END);
...@@ -165,8 +157,7 @@ void __init kasan_init(void) ...@@ -165,8 +157,7 @@ void __init kasan_init(void)
pfn_pte(virt_to_pfn(kasan_zero_page), PAGE_KERNEL_RO)); pfn_pte(virt_to_pfn(kasan_zero_page), PAGE_KERNEL_RO));
memset(kasan_zero_page, 0, PAGE_SIZE); memset(kasan_zero_page, 0, PAGE_SIZE);
cpu_set_ttbr1(__pa(swapper_pg_dir)); cpu_replace_ttbr1(swapper_pg_dir);
flush_tlb_all();
/* At this point kasan is fully initialized. Enable error messages */ /* At this point kasan is fully initialized. Enable error messages */
init_task.kasan_depth = 0; init_task.kasan_depth = 0;
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment