• Alexei Starovoitov's avatar
    bpf: fix hashmap extra_elems logic · 8c290e60
    Alexei Starovoitov authored
    In both kmalloc and prealloc mode the bpf_map_update_elem() is using
    per-cpu extra_elems to do atomic update when the map is full.
    There are two issues with it. The logic can be misused, since it allows
    max_entries+num_cpus elements to be present in the map. And alloc_extra_elems()
    at map creation time can fail percpu alloc for large map values with a warn:
    WARNING: CPU: 3 PID: 2752 at ../mm/percpu.c:892 pcpu_alloc+0x119/0xa60
    illegal size (32824) or align (8) for percpu allocation
    
    The fixes for both of these issues are different for kmalloc and prealloc modes.
    For prealloc mode allocate extra num_possible_cpus elements and store
    their pointers into extra_elems array instead of actual elements.
    Hence we can use these hidden(spare) elements not only when the map is full
    but during bpf_map_update_elem() that replaces existing element too.
    That also improves performance, since pcpu_freelist_pop/push is avoided.
    Unfortunately this approach cannot be used for kmalloc mode which needs
    to kfree elements after rcu grace period. Therefore switch it back to normal
    kmalloc even when full and old element exists like it was prior to
    commit 6c905981 ("bpf: pre-allocate hash map elements").
    
    Add tests to check for over max_entries and large map values.
    Reported-by: default avatarDave Jones <davej@codemonkey.org.uk>
    Fixes: 6c905981 ("bpf: pre-allocate hash map elements")
    Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
    Acked-by: default avatarMartin KaFai Lau <kafai@fb.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    8c290e60
hashtab.c 29.6 KB