• Alexei Starovoitov's avatar
    bpf: x64: add JIT support for multi-function programs · 1c2a088a
    Alexei Starovoitov authored
    Typical JIT does several passes over bpf instructions to
    compute total size and relative offsets of jumps and calls.
    With multitple bpf functions calling each other all relative calls
    will have invalid offsets intially therefore we need to additional
    last pass over the program to emit calls with correct offsets.
    For example in case of three bpf functions:
    main:
      call foo
      call bpf_map_lookup
      exit
    foo:
      call bar
      exit
    bar:
      exit
    
    We will call bpf_int_jit_compile() indepedently for main(), foo() and bar()
    x64 JIT typically does 4-5 passes to converge.
    After these initial passes the image for these 3 functions
    will be good except call targets, since start addresses of
    foo() and bar() are unknown when we were JITing main()
    (note that call bpf_map_lookup will be resolved properly
    during initial passes).
    Once start addresses of 3 functions are known we patch
    call_insn->imm to point to right functions and call
    bpf_int_jit_compile() again which needs only one pass.
    Additional safety checks are done to make sure this
    last pass doesn't produce image that is larger or smaller
    than previous pass.
    
    When constant blinding is on it's applied to all functions
    at the first pass, since doing it once again at the last
    pass can change size of the JITed code.
    
    Tested on x64 and arm64 hw with JIT on/off, blinding on/off.
    x64 jits bpf-to-bpf calls correctly while arm64 falls back to interpreter.
    All other JITs that support normal BPF_CALL will behave the same way
    since bpf-to-bpf call is equivalent to bpf-to-kernel call from
    JITs point of view.
    Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
    Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
    1c2a088a
verifier.c 158 KB