1. 02 May, 2017 9 commits
    • Daniel Borkmann's avatar
      bpf, arm64: fix jit branch offset related to ldimm64 · ddc665a4
      Daniel Borkmann authored
      When the instruction right before the branch destination is
      a 64 bit load immediate, we currently calculate the wrong
      jump offset in the ctx->offset[] array as we only account
      one instruction slot for the 64 bit load immediate although
      it uses two BPF instructions. Fix it up by setting the offset
      into the right slot after we incremented the index.
      
      Before (ldimm64 test 1):
      
        [...]
        00000020:  52800007  mov w7, #0x0 // #0
        00000024:  d2800060  mov x0, #0x3 // #3
        00000028:  d2800041  mov x1, #0x2 // #2
        0000002c:  eb01001f  cmp x0, x1
        00000030:  54ffff82  b.cs 0x00000020
        00000034:  d29fffe7  mov x7, #0xffff // #65535
        00000038:  f2bfffe7  movk x7, #0xffff, lsl #16
        0000003c:  f2dfffe7  movk x7, #0xffff, lsl #32
        00000040:  f2ffffe7  movk x7, #0xffff, lsl #48
        00000044:  d29dddc7  mov x7, #0xeeee // #61166
        00000048:  f2bdddc7  movk x7, #0xeeee, lsl #16
        0000004c:  f2ddddc7  movk x7, #0xeeee, lsl #32
        00000050:  f2fdddc7  movk x7, #0xeeee, lsl #48
        [...]
      
      After (ldimm64 test 1):
      
        [...]
        00000020:  52800007  mov w7, #0x0 // #0
        00000024:  d2800060  mov x0, #0x3 // #3
        00000028:  d2800041  mov x1, #0x2 // #2
        0000002c:  eb01001f  cmp x0, x1
        00000030:  540000a2  b.cs 0x00000044
        00000034:  d29fffe7  mov x7, #0xffff // #65535
        00000038:  f2bfffe7  movk x7, #0xffff, lsl #16
        0000003c:  f2dfffe7  movk x7, #0xffff, lsl #32
        00000040:  f2ffffe7  movk x7, #0xffff, lsl #48
        00000044:  d29dddc7  mov x7, #0xeeee // #61166
        00000048:  f2bdddc7  movk x7, #0xeeee, lsl #16
        0000004c:  f2ddddc7  movk x7, #0xeeee, lsl #32
        00000050:  f2fdddc7  movk x7, #0xeeee, lsl #48
        [...]
      
      Also, add a couple of test cases to make sure JITs pass
      this test. Tested on Cavium ThunderX ARMv8. The added
      test cases all pass after the fix.
      
      Fixes: 8eee539d ("arm64: bpf: fix out-of-bounds read in bpf2a64_offset()")
      Reported-by: default avatarDavid S. Miller <davem@davemloft.net>
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Cc: Xi Wang <xi.wang@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      ddc665a4
    • Daniel Borkmann's avatar
      bpf, arm64: implement jiting of BPF_XADD · 85f68fe8
      Daniel Borkmann authored
      This work adds BPF_XADD for BPF_W/BPF_DW to the arm64 JIT and therefore
      completes JITing of all BPF instructions, meaning we can thus also remove
      the 'notyet' label and do not need to fall back to the interpreter when
      BPF_XADD is used in a program!
      
      This now also brings arm64 JIT in line with x86_64, s390x, ppc64, sparc64,
      where all current eBPF features are supported.
      
      BPF_W example from test_bpf:
      
        .u.insns_int = {
          BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
          BPF_ST_MEM(BPF_W, R10, -40, 0x10),
          BPF_STX_XADD(BPF_W, R10, R0, -40),
          BPF_LDX_MEM(BPF_W, R0, R10, -40),
          BPF_EXIT_INSN(),
        },
      
        [...]
        00000020:  52800247  mov w7, #0x12 // #18
        00000024:  928004eb  mov x11, #0xffffffffffffffd8 // #-40
        00000028:  d280020a  mov x10, #0x10 // #16
        0000002c:  b82b6b2a  str w10, [x25,x11]
        // start of xadd mapping:
        00000030:  928004ea  mov x10, #0xffffffffffffffd8 // #-40
        00000034:  8b19014a  add x10, x10, x25
        00000038:  f9800151  prfm pstl1strm, [x10]
        0000003c:  885f7d4b  ldxr w11, [x10]
        00000040:  0b07016b  add w11, w11, w7
        00000044:  880b7d4b  stxr w11, w11, [x10]
        00000048:  35ffffab  cbnz w11, 0x0000003c
        // end of xadd mapping:
        [...]
      
      BPF_DW example from test_bpf:
      
        .u.insns_int = {
          BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
          BPF_ST_MEM(BPF_DW, R10, -40, 0x10),
          BPF_STX_XADD(BPF_DW, R10, R0, -40),
          BPF_LDX_MEM(BPF_DW, R0, R10, -40),
          BPF_EXIT_INSN(),
        },
      
        [...]
        00000020:  52800247  mov w7,  #0x12 // #18
        00000024:  928004eb  mov x11, #0xffffffffffffffd8 // #-40
        00000028:  d280020a  mov x10, #0x10 // #16
        0000002c:  f82b6b2a  str x10, [x25,x11]
        // start of xadd mapping:
        00000030:  928004ea  mov x10, #0xffffffffffffffd8 // #-40
        00000034:  8b19014a  add x10, x10, x25
        00000038:  f9800151  prfm pstl1strm, [x10]
        0000003c:  c85f7d4b  ldxr x11, [x10]
        00000040:  8b07016b  add x11, x11, x7
        00000044:  c80b7d4b  stxr w11, x11, [x10]
        00000048:  35ffffab  cbnz w11, 0x0000003c
        // end of xadd mapping:
        [...]
      
      Tested on Cavium ThunderX ARMv8, test suite results after the patch:
      
        No JIT:   [ 3751.855362] test_bpf: Summary: 311 PASSED, 0 FAILED, [0/303 JIT'ed]
        With JIT: [ 3573.759527] test_bpf: Summary: 311 PASSED, 0 FAILED, [303/303 JIT'ed]
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      85f68fe8
    • David S. Miller's avatar
      Merge branch 'bpf-test-prog-fixes' · a481649e
      David S. Miller authored
      I say:
      
      ====================
      Fix some bpf program testing framework bugs
      
      This series fixes two issue:
      
      1) Accidental user pointer dereference in bpf_test_finish()
      
      2) The packet data given to the test programs is not aligned correctly
      
      The first issue is fixed simply because we have a kernel side copy
      of the datastructure in question already.  And the second bug is
      a simple matter of applying NET_IP_ALIGN where needed.
      ====================
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      a481649e
    • David Miller's avatar
      bpf: Align packet data properly in program testing framework. · 586f8525
      David Miller authored
      Make sure we apply NET_IP_ALIGN when reserving headroom for SKB
      and XDP test runs, just like a real driver would.
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      586f8525
    • David Miller's avatar
      bpf: Do not dereference user pointer in bpf_test_finish(). · 78e52272
      David Miller authored
      Instead, pass the kattr in which has a kernel side copy of this
      data structure from userspace already.
      
      Fix based upon a suggestion from Alexei Starovoitov.
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      78e52272
    • David S. Miller's avatar
      selftests: bpf: Use bpf_endian.h in test_xdp.c · 4e9c3a66
      David S. Miller authored
      This fixes the testcase on big-endian.
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      4e9c3a66
    • Jakub Kicinski's avatar
      xdp: fix parameter kdoc for extack · b5d60989
      Jakub Kicinski authored
      Fix kdoc parameter spelling from extact to extack.
      Signed-off-by: default avatarJakub Kicinski <jakub.kicinski@netronome.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      b5d60989
    • Daniel Borkmann's avatar
      bpf, samples: fix build warning in cookie_uid_helper_example · eb6211d3
      Daniel Borkmann authored
      Fix the following warnings triggered by 51570a5a ("A Sample of
      using socket cookie and uid for traffic monitoring"):
      
        In file included from /home/foo/net-next/samples/bpf/cookie_uid_helper_example.c:54:0:
        /home/foo/net-next/samples/bpf/cookie_uid_helper_example.c: In function 'prog_load':
        /home/foo/net-next/samples/bpf/cookie_uid_helper_example.c:119:27: warning: overflow in implicit constant conversion [-Woverflow]
           -32 + offsetof(struct stats, uid)),
                                 ^
        /home/foo/net-next/samples/bpf/libbpf.h:135:12: note: in definition of macro 'BPF_STX_MEM'
         .off   = OFF,     \
                  ^
        /home/foo/net-next/samples/bpf/cookie_uid_helper_example.c:121:27: warning: overflow in implicit constant conversion [-Woverflow]
           -32 + offsetof(struct stats, packets), 1),
                                 ^
        /home/foo/net-next/samples/bpf/libbpf.h:155:12: note: in definition of macro 'BPF_ST_MEM'
         .off   = OFF,     \
                  ^
        /home/foo/net-next/samples/bpf/cookie_uid_helper_example.c:129:27: warning: overflow in implicit constant conversion [-Woverflow]
           -32 + offsetof(struct stats, bytes)),
                                 ^
        /home/foo/net-next/samples/bpf/libbpf.h:135:12: note: in definition of macro 'BPF_STX_MEM'
         .off   = OFF,     \
                  ^
        HOSTLD  /home/foo/net-next/samples/bpf/per_socket_stats_example
      
      Fixes: 51570a5a ("A Sample of using socket cookie and uid for traffic monitoring")
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      eb6211d3
    • David S. Miller's avatar
      sparc64: Fix BPF JIT wrt. branches and ldimm64 instructions. · e3bf4c61
      David S. Miller authored
      Like other JITs, sparc64 maintains an array of instruction offsets but
      stores the entries off by one.  This is done because jumps to the
      exit block are indexed to one past the last BPF instruction.
      
      So if we size the array by the program length, we need to record
      the previous instruction in order to stay within the array bounds.
      
      This is explained in ARM JIT commit 8eee539d ("arm64: bpf: fix
      out-of-bounds read in bpf2a64_offset()").
      
      But this scheme requires a little bit of careful handling when
      the instruction before the branch destination is a 64-bit load
      immediate.  It takes up 2 BPF instruction slots.
      
      Therefore, we have to fill in the array entry for the second
      half of the 64-bit load immediate instruction rather than for
      the one for the beginning of that instruction.
      
      Fixes: 7a12b503 ("sparc64: Add eBPF JIT.")
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      e3bf4c61
  2. 01 May, 2017 31 commits