• Mark Rutland's avatar
    arm64: Support PREEMPT_DYNAMIC · 1b2d3451
    Mark Rutland authored
    This patch enables support for PREEMPT_DYNAMIC on arm64, allowing the
    preemption model to be chosen at boot time.
    
    Specifically, this patch selects HAVE_PREEMPT_DYNAMIC_KEY, so that each
    preemption function is an out-of-line call with an early return
    depending upon a static key. This leaves almost all the codegen up to
    the compiler, and side-steps a number of pain points with static calls
    (e.g. interaction with CFI schemes). This should have no worse overhead
    than using non-inline static calls, as those use out-of-line trampolines
    with early returns.
    
    For example, the dynamic_cond_resched() wrapper looks as follows when
    enabled. When disabled, the first `B` is replaced with a `NOP`,
    resulting in an early return.
    
    | <dynamic_cond_resched>:
    |        bti     c
    |        b       <dynamic_cond_resched+0x10>     // or `nop`
    |        mov     w0, #0x0
    |        ret
    |        mrs     x0, sp_el0
    |        ldr     x0, [x0, #8]
    |        cbnz    x0, <dynamic_cond_resched+0x8>
    |        paciasp
    |        stp     x29, x30, [sp, #-16]!
    |        mov     x29, sp
    |        bl      <preempt_schedule_common>
    |        mov     w0, #0x1
    |        ldp     x29, x30, [sp], #16
    |        autiasp
    |        ret
    
    ... compared to the regular form of the function:
    
    | <__cond_resched>:
    |        bti     c
    |        mrs     x0, sp_el0
    |        ldr     x1, [x0, #8]
    |        cbz     x1, <__cond_resched+0x18>
    |        mov     w0, #0x0
    |        ret
    |        paciasp
    |        stp     x29, x30, [sp, #-16]!
    |        mov     x29, sp
    |        bl      <preempt_schedule_common>
    |        mov     w0, #0x1
    |        ldp     x29, x30, [sp], #16
    |        autiasp
    |        ret
    
    Since arm64 does not yet use the generic entry code, we must define our
    own `sk_dynamic_irqentry_exit_cond_resched`, which will be
    enabled/disabled by the common code in kernel/sched/core.c. All other
    preemption functions and associated static keys are defined there.
    Signed-off-by: default avatarMark Rutland <mark.rutland@arm.com>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Acked-by: default avatarArd Biesheuvel <ardb@kernel.org>
    Acked-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
    Acked-by: default avatarFrederic Weisbecker <frederic@kernel.org>
    Link: https://lore.kernel.org/r/20220214165216.2231574-8-mark.rutland@arm.com
    1b2d3451
Kconfig 68.5 KB