1. 13 Dec, 2018 13 commits
  2. 04 Dec, 2018 23 commits
  3. 03 Dec, 2018 4 commits
    • David S. Miller's avatar
      Merge branch 'udp-msg_zerocopy' · 6e360f73
      David S. Miller authored
      Willem de Bruijn says:
      
      ====================
      udp msg_zerocopy
      
      Enable MSG_ZEROCOPY for udp sockets
      
      Patch 1/3 is the main patch, a rework of RFC patch
        http://patchwork.ozlabs.org/patch/899630/
        more details in the patch commit message
      
      Patch 2/3 is an optimization to remove a branch from the UDP hot path
        and refcount_inc/refcount_dec_and_test pair when zerocopy is used.
        This used to be included in the first patch in v2.
      
      Patch 3/3 runs the already existing udp zerocopy tests
        as part of kselftest
      
      See also recent Linux Plumbers presentation
        https://linuxplumbersconf.org/event/2/contributions/106/attachments/104/128/willemdebruijn-lpc2018-udpgso-presentation-20181113.pdf
      
      Changes:
        v1 -> v2
          - Fixup reverse christmas tree violation
        v2 -> v3
          - Split refcount avoidance optimization into separate patch
            - Fix refcount leak on error in fragmented case
              (thanks to Paolo Abeni for pointing this one out!)
            - Fix refcount inc on zero
        v3 -> v4
          - Move skb_zcopy_set below the only kfree_skb that might cause
            a premature uarg destroy before skb_zerocopy_put_abort
            - Move the entire skb_shinfo assignment block, to keep that
      	cacheline access in one place
      ====================
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      6e360f73
    • Willem de Bruijn's avatar
      selftests: extend zerocopy tests to udp · db63e489
      Willem de Bruijn authored
      Both msg_zerocopy and udpgso_bench have udp zerocopy variants.
      Exercise these as part of the standard kselftest run.
      
      With udp, msg_zerocopy has no control channel. Ensure that the
      receiver exits after the sender by accounting for the initial
      delay in starting them (in msg_zerocopy.sh).
      Signed-off-by: default avatarWillem de Bruijn <willemb@google.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      db63e489
    • Willem de Bruijn's avatar
      udp: elide zerocopy operation in hot path · 52900d22
      Willem de Bruijn authored
      With MSG_ZEROCOPY, each skb holds a reference to a struct ubuf_info.
      Release of its last reference triggers a completion notification.
      
      The TCP stack in tcp_sendmsg_locked holds an extra ref independent of
      the skbs, because it can build, send and free skbs within its loop,
      possibly reaching refcount zero and freeing the ubuf_info too soon.
      
      The UDP stack currently also takes this extra ref, but does not need
      it as all skbs are sent after return from __ip(6)_append_data.
      
      Avoid the extra refcount_inc and refcount_dec_and_test, and generally
      the sock_zerocopy_put in the common path, by passing the initial
      reference to the first skb.
      
      This approach is taken instead of initializing the refcount to 0, as
      that would generate error "refcount_t: increment on 0" on the
      next skb_zcopy_set.
      
      Changes
        v3 -> v4
          - Move skb_zcopy_set below the only kfree_skb that might cause
            a premature uarg destroy before skb_zerocopy_put_abort
            - Move the entire skb_shinfo assignment block, to keep that
              cacheline access in one place
      Signed-off-by: default avatarWillem de Bruijn <willemb@google.com>
      Acked-by: default avatarPaolo Abeni <pabeni@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      52900d22
    • Willem de Bruijn's avatar
      udp: msg_zerocopy · b5947e5d
      Willem de Bruijn authored
      Extend zerocopy to udp sockets. Allow setting sockopt SO_ZEROCOPY and
      interpret flag MSG_ZEROCOPY.
      
      This patch was previously part of the zerocopy RFC patchsets. Zerocopy
      is not effective at small MTU. With segmentation offload building
      larger datagrams, the benefit of page flipping outweights the cost of
      generating a completion notification.
      
      tools/testing/selftests/net/msg_zerocopy.sh after applying follow-on
      test patch and making skb_orphan_frags_rx same as skb_orphan_frags:
      
          ipv4 udp -t 1
          tx=191312 (11938 MB) txc=0 zc=n
          rx=191312 (11938 MB)
          ipv4 udp -z -t 1
          tx=304507 (19002 MB) txc=304507 zc=y
          rx=304507 (19002 MB)
          ok
          ipv6 udp -t 1
          tx=174485 (10888 MB) txc=0 zc=n
          rx=174485 (10888 MB)
          ipv6 udp -z -t 1
          tx=294801 (18396 MB) txc=294801 zc=y
          rx=294801 (18396 MB)
          ok
      
      Changes
        v1 -> v2
          - Fixup reverse christmas tree violation
        v2 -> v3
          - Split refcount avoidance optimization into separate patch
            - Fix refcount leak on error in fragmented case
              (thanks to Paolo Abeni for pointing this one out!)
            - Fix refcount inc on zero
            - Test sock_flag SOCK_ZEROCOPY directly in __ip_append_data.
              This is needed since commit 5cf4a853 ("tcp: really ignore
      	MSG_ZEROCOPY if no SO_ZEROCOPY") did the same for tcp.
      Signed-off-by: default avatarWillem de Bruijn <willemb@google.com>
      Acked-by: default avatarPaolo Abeni <pabeni@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      b5947e5d