• Jesper Dangaard Brouer's avatar
    net: bulk free infrastructure for NAPI context, use napi_consume_skb · 795bb1c0
    Jesper Dangaard Brouer authored
    Discovered that network stack were hitting the kmem_cache/SLUB
    slowpath when freeing SKBs.  Doing bulk free with kmem_cache_free_bulk
    can speedup this slowpath.
    
    NAPI context is a bit special, lets take advantage of that for bulk
    free'ing SKBs.
    
    In NAPI context we are running in softirq, which gives us certain
    protection.  A softirq can run on several CPUs at once.  BUT the
    important part is a softirq will never preempt another softirq running
    on the same CPU.  This gives us the opportunity to access per-cpu
    variables in softirq context.
    
    Extend napi_alloc_cache (before only contained page_frag_cache) to be
    a struct with a small array based stack for holding SKBs.  Introduce a
    SKB defer and flush API for accessing this.
    
    Introduce napi_consume_skb() as replacement for e.g. dev_consume_skb_any()
    when running in NAPI context.  A small trick to handle/detect if we
    are called from netpoll is to see if budget is 0.  In that case, we
    need to invoke dev_consume_skb_irq().
    
    Joint work with Alexander Duyck.
    Signed-off-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
    Signed-off-by: default avatarAlexander Duyck <alexander.h.duyck@redhat.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    795bb1c0
skbuff.c 113 KB