1. 19 Oct, 2018 12 commits
    • Alexei Starovoitov's avatar
      Merge branch 'improve_perf_barriers' · 2929ad29
      Alexei Starovoitov authored
      Daniel Borkmann says:
      
      ====================
      This set first adds smp_* barrier variants to tools infrastructure
      and updates perf and libbpf to make use of them. For details, please
      see individual patches, thanks!
      
      Arnaldo, if there are no objections, could this be routed via bpf-next
      with Acked-by's due to later dependencies in libbpf? Alternatively,
      I could also get the 2nd patch out during merge window, but perhaps
      it's okay to do in one go as there shouldn't be much conflict in perf
      itself.
      
      Thanks!
      
      v1 -> v2:
        - add common helper and switch to acquire/release variants
          when possible, thanks Peter!
      ====================
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      2929ad29
    • Daniel Borkmann's avatar
      bpf, libbpf: use correct barriers in perf ring buffer walk · a64af0ef
      Daniel Borkmann authored
      Given libbpf is a generic library and not restricted to x86-64 only,
      the compiler barrier in bpf_perf_event_read_simple() after fetching
      the head needs to be replaced with smp_rmb() at minimum. Also, writing
      out the tail we should use WRITE_ONCE() to avoid store tearing.
      
      Now that we have the logic in place in ring_buffer_read_head() and
      ring_buffer_write_tail() helper also used by perf tool which would
      select the correct and best variant for a given architecture (e.g.
      x86-64 can avoid CPU barriers entirely), make use of these in order
      to fix bpf_perf_event_read_simple().
      
      Fixes: d0cabbb0 ("tools: bpf: move the event reading loop to libbpf")
      Fixes: 39111695 ("samples: bpf: add bpf_perf_event_output example")
      Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      a64af0ef
    • Daniel Borkmann's avatar
      tools, perf: add and use optimized ring_buffer_{read_head, write_tail} helpers · 09d62154
      Daniel Borkmann authored
      Currently, on x86-64, perf uses LFENCE and MFENCE (rmb() and mb(),
      respectively) when processing events from the perf ring buffer which
      is unnecessarily expensive as we can do more lightweight in particular
      given this is critical fast-path in perf.
      
      According to Peter rmb()/mb() were added back then via a94d342b
      ("tools/perf: Add required memory barriers") at a time where kernel
      still supported chips that needed it, but nowadays support for these
      has been ditched completely, therefore we can fix them up as well.
      
      While for x86-64, replacing rmb() and mb() with smp_*() variants would
      result in just a compiler barrier for the former and LOCK + ADD for
      the latter (__sync_synchronize() uses slower MFENCE by the way), Peter
      suggested we can use smp_{load_acquire,store_release}() instead for
      architectures where its implementation doesn't resolve in slower smp_mb().
      Thus, e.g. in x86-64 we would be able to avoid CPU barrier entirely due
      to TSO. For architectures where the latter needs to use smp_mb() e.g.
      on arm, we stick to cheaper smp_rmb() variant for fetching the head.
      
      This work adds helpers ring_buffer_read_head() and ring_buffer_write_tail()
      for tools infrastructure that either switches to smp_load_acquire() for
      architectures where it is cheaper or uses READ_ONCE() + smp_rmb() barrier
      for those where it's not in order to fetch the data_head from the perf
      control page, and it uses smp_store_release() to write the data_tail.
      Latter is smp_mb() + WRITE_ONCE() combination or a cheaper variant if
      architecture allows for it. Those that rely on smp_rmb() and smp_mb() can
      further improve performance in a follow up step by implementing the two
      under tools/arch/*/include/asm/barrier.h such that they don't have to
      fallback to rmb() and mb() in tools/include/asm/barrier.h.
      
      Switch perf to use ring_buffer_read_head() and ring_buffer_write_tail()
      so it can make use of the optimizations. Later, we convert libbpf as
      well to use the same helpers.
      
      Side note [0]: the topic has been raised of whether one could simply use
      the C11 gcc builtins [1] for the smp_load_acquire() and smp_store_release()
      instead:
      
        __atomic_load_n(ptr, __ATOMIC_ACQUIRE);
        __atomic_store_n(ptr, val, __ATOMIC_RELEASE);
      
      Kernel and (presumably) tooling shipped along with the kernel has a
      minimum requirement of being able to build with gcc-4.6 and the latter
      does not have C11 builtins. While generally the C11 memory models don't
      align with the kernel's, the C11 load-acquire and store-release alone
      /could/ suffice, however. Issue is that this is implementation dependent
      on how the load-acquire and store-release is done by the compiler and
      the mapping of supported compilers must align to be compatible with the
      kernel's implementation, and thus needs to be verified/tracked on a
      case by case basis whether they match (unless an architecture uses them
      also from kernel side). The implementations for smp_load_acquire() and
      smp_store_release() in this patch have been adapted from the kernel side
      ones to have a concrete and compatible mapping in place.
      
        [0] http://patchwork.ozlabs.org/patch/985422/
        [1] https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.htmlSigned-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
      Cc: Will Deacon <will.deacon@arm.com>
      Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      09d62154
    • Anders Roxell's avatar
      selftests/bpf: add missing executables to .gitignore · 78de3546
      Anders Roxell authored
      Fixes: 371e4fcc ("selftests/bpf: cgroup local storage-based network counters")
      Fixes: 370920c4 ("selftests/bpf: Test libbpf_{prog,attach}_type_by_name")
      Signed-off-by: default avatarAnders Roxell <anders.roxell@linaro.org>
      Acked-by: default avatarYonghong Song <yhs@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      78de3546
    • Alexei Starovoitov's avatar
      Merge branch 'queue_stack_maps' · 43ed375f
      Alexei Starovoitov authored
      Mauricio Vasquez says:
      
      ====================
      In some applications this is needed have a pool of free elements, for
      example the list of free L4 ports in a SNAT.  None of the current maps allow
      to do it as it is not possible to get any element without having they key
      it is associated to, even if it were possible, the lack of locking mecanishms in
      eBPF would do it almost impossible to be implemented without data races.
      
      This patchset implements two new kind of eBPF maps: queue and stack.
      Those maps provide to eBPF programs the peek, push and pop operations, and for
      userspace applications a new bpf_map_lookup_and_delete_elem() is added.
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      
      v2 -> v3:
       - Remove "almost dead code" in syscall.c
       - Remove unnecessary copy_from_user in bpf_map_lookup_and_delete_elem
       - Rebase
      
      v1 -> v2:
       - Put ARG_PTR_TO_UNINIT_MAP_VALUE logic into a separated patch
       - Fix missing __this_cpu_dec & preempt_enable calls in kernel/bpf/syscall.c
      
      RFC v4 -> v1:
       - Remove roundup to power of 2 in memory allocation
       - Remove count and use a free slot to check if queue/stack is empty
       - Use if + assigment for wrapping indexes
       - Fix some minor style issues
       - Squash two patches together
      
      RFC v3 -> RFC v4:
       - Revert renaming of kernel/bpf/stackmap.c
       - Remove restriction on value size
       - Remove len arguments from peek/pop helpers
       - Add new ARG_PTR_TO_UNINIT_MAP_VALUE
      
      RFC v2 -> RFC v3:
       - Return elements by value instead that by reference
       - Implement queue/stack base on array and head + tail indexes
       - Rename stack trace related files to avoid confusion and conflicts
      
      RFC v1 -> RFC v2:
       - Create two separate maps instead of single one + flags
       - Implement bpf_map_lookup_and_delete syscall
       - Support peek operation
       - Define replacement policy through flags in the update() method
       - Add eBPF side tests
      ====================
      Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      43ed375f
    • Mauricio Vasquez B's avatar
      selftests/bpf: add test cases for queue and stack maps · 43b987d2
      Mauricio Vasquez B authored
      test_maps:
      Tests that queue/stack maps are behaving correctly even in corner cases
      
      test_progs:
      Tests new ebpf helpers
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      43b987d2
    • Mauricio Vasquez B's avatar
      Sync uapi/bpf.h to tools/include · da4e1b15
      Mauricio Vasquez B authored
      Sync both files.
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      da4e1b15
    • Mauricio Vasquez B's avatar
      bpf: add MAP_LOOKUP_AND_DELETE_ELEM syscall · bd513cd0
      Mauricio Vasquez B authored
      The previous patch implemented a bpf queue/stack maps that
      provided the peek/pop/push functions.  There is not a direct
      relationship between those functions and the current maps
      syscalls, hence a new MAP_LOOKUP_AND_DELETE_ELEM syscall is added,
      this is mapped to the pop operation in the queue/stack maps
      and it is still to implement in other kind of maps.
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      bd513cd0
    • Mauricio Vasquez B's avatar
      bpf: add queue and stack maps · f1a2e44a
      Mauricio Vasquez B authored
      Queue/stack maps implement a FIFO/LIFO data storage for ebpf programs.
      These maps support peek, pop and push operations that are exposed to eBPF
      programs through the new bpf_map[peek/pop/push] helpers.  Those operations
      are exposed to userspace applications through the already existing
      syscalls in the following way:
      
      BPF_MAP_LOOKUP_ELEM            -> peek
      BPF_MAP_LOOKUP_AND_DELETE_ELEM -> pop
      BPF_MAP_UPDATE_ELEM            -> push
      
      Queue/stack maps are implemented using a buffer, tail and head indexes,
      hence BPF_F_NO_PREALLOC is not supported.
      
      As opposite to other maps, queue and stack do not use RCU for protecting
      maps values, the bpf_map[peek/pop] have a ARG_PTR_TO_UNINIT_MAP_VALUE
      argument that is a pointer to a memory zone where to save the value of a
      map.  Basically the same as ARG_PTR_TO_UNINIT_MEM, but the size has not
      be passed as an extra argument.
      
      Our main motivation for implementing queue/stack maps was to keep track
      of a pool of elements, like network ports in a SNAT, however we forsee
      other use cases, like for exampling saving last N kernel events in a map
      and then analysing from userspace.
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      f1a2e44a
    • Mauricio Vasquez B's avatar
      bpf/verifier: add ARG_PTR_TO_UNINIT_MAP_VALUE · 2ea864c5
      Mauricio Vasquez B authored
      ARG_PTR_TO_UNINIT_MAP_VALUE argument is a pointer to a memory zone
      used to save the value of a map.  Basically the same as
      ARG_PTR_TO_UNINIT_MEM, but the size has not be passed as an extra
      argument.
      
      This will be used in the following patch that implements some new
      helpers that receive a pointer to be filled with a map value.
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      2ea864c5
    • Mauricio Vasquez B's avatar
      bpf/syscall: allow key to be null in map functions · c9d29f46
      Mauricio Vasquez B authored
      This commit adds the required logic to allow key being NULL
      in case the key_size of the map is 0.
      
      A new __bpf_copy_key function helper only copies the key from
      userpsace when key_size != 0, otherwise it enforces that key must be
      null.
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      c9d29f46
    • Mauricio Vasquez B's avatar
      bpf: rename stack trace map operations · 14499160
      Mauricio Vasquez B authored
      In the following patches queue and stack maps (FIFO and LIFO
      datastructures) will be implemented.  In order to avoid confusion and
      a possible name clash rename stack_map_ops to stack_trace_map_ops
      Signed-off-by: default avatarMauricio Vasquez B <mauricio.vasquez@polito.it>
      Acked-by: default avatarSong Liu <songliubraving@fb.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      14499160
  2. 18 Oct, 2018 3 commits
  3. 17 Oct, 2018 5 commits
  4. 16 Oct, 2018 20 commits