• Daniel Borkmann's avatar
    bpf, x64: implement retpoline for tail call · a493a87f
    Daniel Borkmann authored
    Implement a retpoline [0] for the BPF tail call JIT'ing that converts
    the indirect jump via jmp %rax that is used to make the long jump into
    another JITed BPF image. Since this is subject to speculative execution,
    we need to control the transient instruction sequence here as well
    when CONFIG_RETPOLINE is set, and direct it into a pause + lfence loop.
    The latter aligns also with what gcc / clang emits (e.g. [1]).
    
    JIT dump after patch:
    
      # bpftool p d x i 1
       0: (18) r2 = map[id:1]
       2: (b7) r3 = 0
       3: (85) call bpf_tail_call#12
       4: (b7) r0 = 2
       5: (95) exit
    
    With CONFIG_RETPOLINE:
    
      # bpftool p d j i 1
      [...]
      33:	cmp    %edx,0x24(%rsi)
      36:	jbe    0x0000000000000072  |*
      38:	mov    0x24(%rbp),%eax
      3e:	cmp    $0x20,%eax
      41:	ja     0x0000000000000072  |
      43:	add    $0x1,%eax
      46:	mov    %eax,0x24(%rbp)
      4c:	mov    0x90(%rsi,%rdx,8),%rax
      54:	test   %rax,%rax
      57:	je     0x0000000000000072  |
      59:	mov    0x28(%rax),%rax
      5d:	add    $0x25,%rax
      61:	callq  0x000000000000006d  |+
      66:	pause                      |
      68:	lfence                     |
      6b:	jmp    0x0000000000000066  |
      6d:	mov    %rax,(%rsp)         |
      71:	retq                       |
      72:	mov    $0x2,%eax
      [...]
    
      * relative fall-through jumps in error case
      + retpoline for indirect jump
    
    Without CONFIG_RETPOLINE:
    
      # bpftool p d j i 1
      [...]
      33:	cmp    %edx,0x24(%rsi)
      36:	jbe    0x0000000000000063  |*
      38:	mov    0x24(%rbp),%eax
      3e:	cmp    $0x20,%eax
      41:	ja     0x0000000000000063  |
      43:	add    $0x1,%eax
      46:	mov    %eax,0x24(%rbp)
      4c:	mov    0x90(%rsi,%rdx,8),%rax
      54:	test   %rax,%rax
      57:	je     0x0000000000000063  |
      59:	mov    0x28(%rax),%rax
      5d:	add    $0x25,%rax
      61:	jmpq   *%rax               |-
      63:	mov    $0x2,%eax
      [...]
    
      * relative fall-through jumps in error case
      - plain indirect jump as before
    
      [0] https://support.google.com/faqs/answer/7625886
      [1] https://github.com/gcc-mirror/gcc/commit/a31e654fa107be968b802786d747e962c2fcdb2bSigned-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
    Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    a493a87f
bpf_jit_comp.c 32.1 KB