Commit a64af0ef authored by Daniel Borkmann's avatar Daniel Borkmann Committed by Alexei Starovoitov

bpf, libbpf: use correct barriers in perf ring buffer walk

Given libbpf is a generic library and not restricted to x86-64 only,
the compiler barrier in bpf_perf_event_read_simple() after fetching
the head needs to be replaced with smp_rmb() at minimum. Also, writing
out the tail we should use WRITE_ONCE() to avoid store tearing.

Now that we have the logic in place in ring_buffer_read_head() and
ring_buffer_write_tail() helper also used by perf tool which would
select the correct and best variant for a given architecture (e.g.
x86-64 can avoid CPU barriers entirely), make use of these in order
to fix bpf_perf_event_read_simple().

Fixes: d0cabbb0 ("tools: bpf: move the event reading loop to libbpf")
Fixes: 39111695 ("samples: bpf: add bpf_perf_event_output example")
Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
parent 09d62154
...@@ -27,6 +27,7 @@ ...@@ -27,6 +27,7 @@
#include <linux/list.h> #include <linux/list.h>
#include <linux/limits.h> #include <linux/limits.h>
#include <linux/perf_event.h> #include <linux/perf_event.h>
#include <linux/ring_buffer.h>
#include <sys/stat.h> #include <sys/stat.h>
#include <sys/types.h> #include <sys/types.h>
#include <sys/vfs.h> #include <sys/vfs.h>
...@@ -2418,13 +2419,12 @@ bpf_perf_event_read_simple(void *mem, unsigned long size, ...@@ -2418,13 +2419,12 @@ bpf_perf_event_read_simple(void *mem, unsigned long size,
unsigned long page_size, void **buf, size_t *buf_len, unsigned long page_size, void **buf, size_t *buf_len,
bpf_perf_event_print_t fn, void *priv) bpf_perf_event_print_t fn, void *priv)
{ {
volatile struct perf_event_mmap_page *header = mem; struct perf_event_mmap_page *header = mem;
__u64 data_head = ring_buffer_read_head(header);
__u64 data_tail = header->data_tail; __u64 data_tail = header->data_tail;
__u64 data_head = header->data_head;
int ret = LIBBPF_PERF_EVENT_ERROR; int ret = LIBBPF_PERF_EVENT_ERROR;
void *base, *begin, *end; void *base, *begin, *end;
asm volatile("" ::: "memory"); /* in real code it should be smp_rmb() */
if (data_head == data_tail) if (data_head == data_tail)
return LIBBPF_PERF_EVENT_CONT; return LIBBPF_PERF_EVENT_CONT;
...@@ -2467,8 +2467,6 @@ bpf_perf_event_read_simple(void *mem, unsigned long size, ...@@ -2467,8 +2467,6 @@ bpf_perf_event_read_simple(void *mem, unsigned long size,
data_tail += ehdr->size; data_tail += ehdr->size;
} }
__sync_synchronize(); /* smp_mb() */ ring_buffer_write_tail(header, data_tail);
header->data_tail = data_tail;
return ret; return ret;
} }
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment