1. 19 Apr, 2018 3 commits
  2. 18 Apr, 2018 22 commits
    • David S. Miller's avatar
      Merge branch 'ipv6-Separate-data-structures-for-FIB-and-data-path' · 0565de29
      David S. Miller authored
      David Ahern says:
      
      ====================
      net/ipv6: Separate data structures for FIB and data path
      
      IPv6 uses the same data struct for both control plane (FIB entries) and
      data path (dst entries). This struct has elements needed for both paths
      adding memory overhead and complexity (taking a dst hold in most places
      but an additional reference on rt6i_ref in a few). Furthermore, because
      of the dst_alloc tie, all FIB entries are allocated with GFP_ATOMIC.
      
      This patch set separates FIB entries from dst entries, better aligning
      IPv6 code with IPv4, simplifying the reference counting and allowing
      FIB entries added by userspace (not autoconf) to use GFP_KERNEL. It is
      first step to a number of performance and scalability changes.
      
      The end result of this patch set:
        - FIB entries (fib6_info):
              /* size: 208, cachelines: 4, members: 25 */
              /* sum members: 207, holes: 1, sum holes: 1 */
      
        - dst entries (rt6_info)
             /* size: 240, cachelines: 4, members: 11 */
      
      Versus the the single rt6_info struct today for both paths:
            /* size: 320, cachelines: 5, members: 28 */
      
      This amounts to a 35% reduction in memory use for FIB entries and a
      25% reduction for dst entries.
      
      With respect to locking FIB entries use RCU and a single atomic
      counter with fib6_info_hold and fib6_info_release helpers to manage
      the reference counting. dst entries use only the traditional dst
      refcounts with dst_hold and dst_release.
      
      FIB entries for host routes are referenced by inet6_ifaddr and
      ifacaddr6. In both cases, additional holds are taken -- similar to
      what is done for devices.
      
      This set is the first of many changes to improve the scalability of the
      IPv6 code. Follow on changes include:
      - consolidating duplicate fib6_info references like IPv4 does with
        duplicate fib_info
      
      - moving fib6_info into a slab cache to avoid allocation roundups to
        power of 2 (the 208 size becomes a 256 actual allocation)
      
      - Allow FIB lookups without generating a dst (e.g., most rt6_lookup
        users just want to verify the egress device). Means moving dst
        allocation to the other side of fib6_rule_lookup which again aligns
        with IPv4 behavior
      
      - using separate standalone nexthop objects which have performance
        benefits beyond fib_info consolidation
      
      At this point I am not seeing any refcount leaks or underflows, no
      oops or bug_ons, or warnings from kasan, so I think it is ready for
      others to beat up on it finding errors in code paths I have missed.
      
      v2 changes
      - rebased to top of tree
      - improved commit message on patch 7
      
      v1 changes
      - rebased to top of tree
      - fix memory leak of metrics as noted by Ido
      - MTU fixes based on pmtu tests (thanks Stefano Brivio for writing)
      
      RFC v2 changes
      - improved commit messages
      - move common metrics code from dst.c to net/ipv4/metrics.c (comment
        from DaveM)
      - address comments from Wei Wang and Martin KaFai Lau (let me know if
        I missed something)
      - fixes detected by kernel test robots
        + added fib6_metric_set to change metric on a FIB entry which could
          be pointing to read-only dst_default_metrics
        + 0day testing found a problem with an intermediate patch; added
          dst_hold_safe on rt->from. Code is removed 3 patches later
      - allow cacheinfo to handle NULL dst; means only expires is pushed to
        userspace
      ====================
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      0565de29
    • David Ahern's avatar
      net/ipv6: Remove unused code and variables for rt6_info · 77634cc6
      David Ahern authored
      Drop unneeded elements from rt6_info struct and rearrange layout to
      something more relevant for the data path.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      77634cc6
    • David Ahern's avatar
      net/ipv6: Flip FIB entries to fib6_info · 8d1c802b
      David Ahern authored
      Convert all code paths referencing a FIB entry from
      rt6_info to fib6_info.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      8d1c802b
    • David Ahern's avatar
      net/ipv6: separate handling of FIB entries from dst based routes · 93531c67
      David Ahern authored
      Last step before flipping the data type for FIB entries:
      - use fib6_info_alloc to create FIB entries in ip6_route_info_create
        and addrconf_dst_alloc
      - use fib6_info_release in place of dst_release, ip6_rt_put and
        rt6_release
      - remove the dst_hold before calling __ip6_ins_rt or ip6_del_rt
      - when purging routes, drop per-cpu routes
      - replace inc and dec of rt6i_ref with fib6_info_hold and fib6_info_release
      - use rt->from since it points to the FIB entry
      - drop references to exception bucket, fib6_metrics and per-cpu from
        dst entries (those are relevant for fib entries only)
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      93531c67
    • David Ahern's avatar
      net/ipv6: introduce fib6_info struct and helpers · a64efe14
      David Ahern authored
      Add fib6_info struct and alloc, destroy, hold and release helpers.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      a64efe14
    • David Ahern's avatar
      net/ipv6: Cleanup exception and cache route handling · 23fb93a4
      David Ahern authored
      IPv6 FIB will only contain FIB entries with exception routes added to
      the FIB entry. Once this transformation is complete, FIB lookups will
      return a fib6_info with the lookup functions still returning a dst
      based rt6_info. The current code uses rt6_info for both paths and
      overloads the rt6_info variable usually called 'rt'.
      
      This patch introduces a new 'f6i' variable name for the result of the FIB
      lookup and keeps 'rt' as the dst based return variable. 'f6i' becomes a
      fib6_info in a later patch which is why it is introduced as f6i now;
      avoids the additional churn in the later patch.
      
      In addition, remove RTF_CACHE and dst checks from fib6 add and delete
      since they can not happen now and will never happen after the data
      type flip.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      23fb93a4
    • David Ahern's avatar
      net/ipv6: Add gfp_flags to route add functions · acb54e3c
      David Ahern authored
      Most FIB entries can be added using memory allocated with GFP_KERNEL.
      Add gfp_flags to ip6_route_add and addrconf_dst_alloc. Code paths that
      can be reached from the packet path (e.g., ndisc and autoconfig) or
      atomic notifiers use GFP_ATOMIC; paths from user context (adding
      addresses and routes) use GFP_KERNEL.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      acb54e3c
    • David Ahern's avatar
      net/ipv6: Create a neigh_lookup for FIB entries · f8a1b43b
      David Ahern authored
      The router discovery code has a FIB entry and wants to validate the
      gateway has a neighbor entry. Refactor the existing dst_neigh_lookup
      for IPv6 and create a new function that takes the gateway and device
      and returns a neighbor entry. Use the new function in
      ndisc_router_discovery to validate the gateway.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      f8a1b43b
    • David Ahern's avatar
      net/ipv6: Move dst flags to booleans in fib entries · 3b6761d1
      David Ahern authored
      Continuing to wean FIB paths off of dst_entry, use a bool to hold
      requests for certain dst settings. Add a helper to convert the
      flags to DST flags when a FIB entry is converted to a dst_entry.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      3b6761d1
    • David Ahern's avatar
      net/ipv6: Add rt6_info create function for ip6_pol_route_lookup · dec9b0e2
      David Ahern authored
      ip6_pol_route_lookup is the lookup function for ip6_route_lookup and
      rt6_lookup. At the moment it returns either a reference to a FIB entry
      or a cached exception. To move FIB entries to a separate struct, this
      lookup function needs to convert FIB entries to an rt6_info that is
      returned to the caller.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      dec9b0e2
    • David Ahern's avatar
      net/ipv6: Add fib6_null_entry · 421842ed
      David Ahern authored
      ip6_null_entry will stay a dst based return for lookups that fail to
      match an entry.
      
      Add a new fib6_null_entry which constitutes the root node and leafs
      for fibs. Replace existing references to ip6_null_entry with the
      new fib6_null_entry when dealing with FIBs.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      421842ed
    • David Ahern's avatar
      net/ipv6: move expires into rt6_info · 14895687
      David Ahern authored
      Add expires to rt6_info for FIB entries, and add fib6 helpers to
      manage it. Data path use of dst.expires remains.
      
      The transition is fairly straightforward: when working with fib entries,
      rt->dst.expires is just rt->expires, rt6_clean_expires is replaced with
      fib6_clean_expires, rt6_set_expires becomes fib6_set_expires, and
      rt6_check_expired becomes fib6_check_expired, where the fib6 versions
      are added by this patch.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      14895687
    • David Ahern's avatar
      net/ipv6: move metrics from dst to rt6_info · d4ead6b3
      David Ahern authored
      Similar to IPv4, add fib metrics to the fib struct, which at the moment
      is rt6_info. Will be moved to fib6_info in a later patch. Copy metrics
      into dst by reference using refcount.
      
      To make the transition:
      - add dst_metrics to rt6_info. Default to dst_default_metrics if no
        metrics are passed during route add. No need for a separate pmtu
        entry; it can reference the MTU slot in fib6_metrics
      
      - ip6_convert_metrics allocates memory in the FIB entry and uses
        ip_metrics_convert to copy from netlink attribute to metrics entry
      
      - the convert metrics call is done in ip6_route_info_create simplifying
        the route add path
        + fib6_commit_metrics and fib6_copy_metrics and the temporary
          mx6_config are no longer needed
      
      - add fib6_metric_set helper to change the value of a metric in the
        fib entry since dst_metric_set can no longer be used
      
      - cow_metrics for IPv6 can drop to dst_cow_metrics_generic
      
      - rt6_dst_from_metrics_check is no longer needed
      
      - rt6_fill_node needs the FIB entry and dst as separate arguments to
        keep compatibility with existing output. Current dst address is
        renamed to dest.
        (to be consistent with IPv4 rt6_fill_node really should be split
        into 2 functions similar to fib_dump_info and rt_fill_info)
      
      - rt6_fill_node no longer needs the temporary metrics variable
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      d4ead6b3
    • David Ahern's avatar
      net/ipv6: Defer initialization of dst to data path · 6edb3c96
      David Ahern authored
      Defer setting dst input, output and error until fib entry is copied.
      
      The reject path from ip6_route_info_create is moved to a new function
      ip6_rt_init_dst_reject with a helper doing the conversion from fib6_type
      to dst error.
      
      The remainder of the new ip6_rt_init_dst is an amalgamtion of dst code
      from addrconf_dst_alloc and the non-reject path of ip6_route_info_create.
      The dst output function is always ip6_output and the input function is
      either ip6_input (local routes), ip6_mc_input (multicast routes) or
      ip6_forward (anything else).
      
      A couple of places using dst.error are updated to look at rt6i_flags.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      6edb3c96
    • David Ahern's avatar
      net/ipv6: Move nexthop data to fib6_nh · 5e670d84
      David Ahern authored
      Introduce fib6_nh structure and move nexthop related data from
      rt6_info and rt6_info.dst to fib6_nh. References to dev, gateway or
      lwtstate from a FIB lookup perspective are converted to use fib6_nh;
      datapath references to dst version are left as is.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      5e670d84
    • David Ahern's avatar
      net/ipv6: Save route type in rt6_info · e8478e80
      David Ahern authored
      The RTN_ type for IPv6 FIB entries is currently embedded in rt6i_flags
      and dst.error. Since dst is going to be removed, it can no longer be
      relied on for FIB dumps so save the route type as fib6_type.
      
      fc_type is set in current users based on the algorithm in rt6_fill_node:
        - rt6i_flags contains RTF_LOCAL: fc_type = RTN_LOCAL
        - rt6i_flags contains RTF_ANYCAST: fc_type = RTN_ANYCAST
        - else fc_type = RTN_UNICAST
      
      Similarly, fib6_type is set in the rt6_info templates based on the
      RTF_REJECT section of rt6_fill_node converting dst.error to RTN type.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      e8478e80
    • David Ahern's avatar
      net/ipv6: Move support functions up in route.c · ae90d867
      David Ahern authored
      Code move only.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      ae90d867
    • David Ahern's avatar
      net/ipv6: Pass net namespace to route functions · afb1d4b5
      David Ahern authored
      Pass network namespace reference into route add, delete and get
      functions.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      afb1d4b5
    • David Ahern's avatar
      net/ipv6: Pass net to fib6_update_sernum · 7aef6859
      David Ahern authored
      Pass net namespace to fib6_update_sernum. It can not be marked const
      as fib6_new_sernum will change ipv6.fib6_sernum.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      7aef6859
    • David Ahern's avatar
      vrf: Move fib6_table into net_vrf · 43b059a3
      David Ahern authored
      A later patch removes rt6i_table from rt6_info. Save the ipv6
      table for a VRF in net_vrf. fib tables can not be deleted so
      no reference counting or locking is required.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      43b059a3
    • David Ahern's avatar
      net: Handle null dst in rtnl_put_cacheinfo · 3940746d
      David Ahern authored
      Need to keep expires time for IPv6 routes in a dump of FIB entries.
      Update rtnl_put_cacheinfo to allow dst to be NULL in which case
      rta_cacheinfo will only contain non-dst data.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      3940746d
    • David Ahern's avatar
      net: Move fib_convert_metrics to metrics file · a919525a
      David Ahern authored
      Move logic of fib_convert_metrics into ip_metrics_convert. This allows
      the code that converts netlink attributes into metrics struct to be
      re-used in a later patch by IPv6.
      
      This is mostly a code move with the following changes to variable names:
        - fi->fib_net becomes net
        - fc_mx and fc_mx_len are passed as inputs pulled from fib_config
        - metrics array is passed as an input from fi->fib_metrics->metrics
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      a919525a
  3. 17 Apr, 2018 15 commits
    • Lorenzo Bianconi's avatar
      ipv6: send netlink notifications for manually configured addresses · a2d481b3
      Lorenzo Bianconi authored
      Send a netlink notification when userspace adds a manually configured
      address if DAD is enabled and optimistic flag isn't set.
      Moreover send RTM_DELADDR notifications for tentative addresses.
      
      Some userspace applications (e.g. NetworkManager) are interested in
      addr netlink events albeit the address is still in tentative state,
      however events are not sent if DAD process is not completed.
      If the address is added and immediately removed userspace listeners
      are not notified. This behaviour can be easily reproduced by using
      veth interfaces:
      
      $ ip -b - <<EOF
      > link add dev vm1 type veth peer name vm2
      > link set dev vm1 up
      > link set dev vm2 up
      > addr add 2001:db8:a:b:1:2:3:4/64 dev vm1
      > addr del 2001:db8:a:b:1:2:3:4/64 dev vm1
      EOF
      
      This patch reverts the behaviour introduced by the commit f784ad3d
      ("ipv6: do not send RTM_DELADDR for tentative addresses")
      Suggested-by: default avatarThomas Haller <thaller@redhat.com>
      Signed-off-by: default avatarLorenzo Bianconi <lorenzo.bianconi@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      a2d481b3
    • Ganesh Goudar's avatar
      cxgb4vf: display pause settings · a64dcddc
      Ganesh Goudar authored
      Add support to display pause settings
      Signed-off-by: default avatarGanesh Goudar <ganeshgr@chelsio.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      a64dcddc
    • Hangbin Liu's avatar
      vxlan: add ttl inherit support · 72f6d71e
      Hangbin Liu authored
      Like tos inherit, ttl inherit should also means inherit the inner protocol's
      ttl values, which actually not implemented in vxlan yet.
      
      But we could not treat ttl == 0 as "use the inner TTL", because that would be
      used also when the "ttl" option is not specified and that would be a behavior
      change, and breaking real use cases.
      
      So add a different attribute IFLA_VXLAN_TTL_INHERIT when "ttl inherit" is
      specified with ip cmd.
      Reported-by: default avatarJianlin Shi <jishi@redhat.com>
      Suggested-by: default avatarJiri Benc <jbenc@redhat.com>
      Signed-off-by: default avatarHangbin Liu <liuhangbin@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      72f6d71e
    • Samuel Mendoza-Jonas's avatar
      net/ncsi: Refactor MAC, VLAN filters · 062b3e1b
      Samuel Mendoza-Jonas authored
      The NCSI driver defines a generic ncsi_channel_filter struct that can be
      used to store arbitrarily formatted filters, and several generic methods
      of accessing data stored in such a filter.
      However in both the driver and as defined in the NCSI specification
      there are only two actual filters: VLAN ID filters and MAC address
      filters. The splitting of the MAC filter into unicast, multicast, and
      mixed is also technically not necessary as these are stored in the same
      location in hardware.
      
      To save complexity, particularly in the set up and accessing of these
      generic filters, remove them in favour of two specific structs. These
      can be acted on directly and do not need several generic helper
      functions to use.
      
      This also fixes a memory error found by KASAN on ARM32 (which is not
      upstream yet), where response handlers accessing a filter's data field
      could write past allocated memory.
      
      [  114.926512] ==================================================================
      [  114.933861] BUG: KASAN: slab-out-of-bounds in ncsi_configure_channel+0x4b8/0xc58
      [  114.941304] Read of size 2 at addr 94888558 by task kworker/0:2/546
      [  114.947593]
      [  114.949146] CPU: 0 PID: 546 Comm: kworker/0:2 Not tainted 4.16.0-rc6-00119-ge156398bfcad #13
      ...
      [  115.170233] The buggy address belongs to the object at 94888540
      [  115.170233]  which belongs to the cache kmalloc-32 of size 32
      [  115.181917] The buggy address is located 24 bytes inside of
      [  115.181917]  32-byte region [94888540, 94888560)
      [  115.192115] The buggy address belongs to the page:
      [  115.196943] page:9eeac100 count:1 mapcount:0 mapping:94888000 index:0x94888fc1
      [  115.204200] flags: 0x100(slab)
      [  115.207330] raw: 00000100 94888000 94888fc1 0000003f 00000001 9eea2014 9eecaa74 96c003e0
      [  115.215444] page dumped because: kasan: bad access detected
      [  115.221036]
      [  115.222544] Memory state around the buggy address:
      [  115.227384]  94888400: fb fb fb fb fc fc fc fc 04 fc fc fc fc fc fc fc
      [  115.233959]  94888480: 00 00 00 fc fc fc fc fc 00 04 fc fc fc fc fc fc
      [  115.240529] >94888500: 00 00 04 fc fc fc fc fc 00 00 04 fc fc fc fc fc
      [  115.247077]                                             ^
      [  115.252523]  94888580: 00 04 fc fc fc fc fc fc 06 fc fc fc fc fc fc fc
      [  115.259093]  94888600: 00 00 06 fc fc fc fc fc 00 00 04 fc fc fc fc fc
      [  115.265639] ==================================================================
      Reported-by: default avatarJoel Stanley <joel@jms.id.au>
      Signed-off-by: default avatarSamuel Mendoza-Jonas <sam@mendozajonas.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      062b3e1b
    • Eric Biggers's avatar
      KEYS: DNS: limit the length of option strings · c210f7b4
      Eric Biggers authored
      Adding a dns_resolver key whose payload contains a very long option name
      resulted in that string being printed in full.  This hit the WARN_ONCE()
      in set_precision() during the printk(), because printk() only supports a
      precision of up to 32767 bytes:
      
          precision 1000000 too large
          WARNING: CPU: 0 PID: 752 at lib/vsprintf.c:2189 vsnprintf+0x4bc/0x5b0
      
      Fix it by limiting option strings (combined name + value) to a much more
      reasonable 128 bytes.  The exact limit is arbitrary, but currently the
      only recognized option is formatted as "dnserror=%lu" which fits well
      within this limit.
      
      Also ratelimit the printks.
      
      Reproducer:
      
          perl -e 'print "#", "A" x 1000000, "\x00"' | keyctl padd dns_resolver desc @s
      
      This bug was found using syzkaller.
      Reported-by: default avatarMark Rutland <mark.rutland@arm.com>
      Fixes: 4a2d7892 ("DNS: If the DNS server returns an error, allow that to be cached [ver #2]")
      Signed-off-by: default avatarEric Biggers <ebiggers@google.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      c210f7b4
    • Davide Caratti's avatar
    • Stephen Suryaputra's avatar
      ipv6: Count interface receive statistics on the ingress netdev · bdb7cc64
      Stephen Suryaputra authored
      The statistics such as InHdrErrors should be counted on the ingress
      netdev rather than on the dev from the dst, which is the egress.
      Signed-off-by: default avatarStephen Suryaputra <ssuryaextr@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      bdb7cc64
    • David Ahern's avatar
      net/ipv6: Make __inet6_bind static · 032234d8
      David Ahern authored
      BPF core gets access to __inet6_bind via ipv6_bpf_stub_impl, so it is
      not invoked directly outside of af_inet6.c. Make it static and move
      inet6_bind after to avoid forward declaration.
      Signed-off-by: default avatarDavid Ahern <dsahern@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      032234d8
    • David S. Miller's avatar
      Merge branch 'XDP-redirect-memory-return-API' · 684009d4
      David S. Miller authored
      Jesper Dangaard Brouer says:
      
      ====================
      XDP redirect memory return API
      
      Submitted against net-next, as it contains NIC driver changes.
      
      This patchset works towards supporting different XDP RX-ring memory
      allocators.  As this will be needed by the AF_XDP zero-copy mode.
      
      The patchset uses mlx5 as the sample driver, which gets implemented
      XDP_REDIRECT RX-mode, but not ndo_xdp_xmit (as this API is subject to
      change thought the patchset).
      
      A new struct xdp_frame is introduced (modeled after cpumap xdp_pkt).
      And both ndo_xdp_xmit and the new xdp_return_frame end-up using this.
      
      Support for a driver supplied allocator is implemented, and a
      refurbished version of page_pool is the first return allocator type
      introduced.  This will be a integration point for AF_XDP zero-copy.
      
      The mlx5 driver evolve into using the page_pool, and see a performance
      increase (with ndo_xdp_xmit out ixgbe driver) from 6Mpps to 12Mpps.
      
      The patchset stop at 16 patches (one over limit), but more API changes
      are planned.  Specifically extending ndo_xdp_xmit and xdp_return_frame
      APIs to support bulking.  As this will address some known limits.
      
      V2: Updated according to Tariq's feedback
      V3: Updated based on feedback from Jason Wang and Alex Duyck
      V4: Updated based on feedback from Tariq and Jason
      V5: Fix SPDX license, add Tariq's reviews, improve patch desc for perf test
      V6: Updated based on feedback from Eric Dumazet and Alex Duyck
      V7: Adapt to i40e that got XDP_REDIRECT support in-between
      V8:
       Updated based on feedback kbuild test robot, and adjust for mlx5 changes
       page_pool only compiled into kernel when drivers Kconfig 'select' feature
      V9:
       Remove some inline statements, let compiler decide what to inline
       Fix return value in virtio_net driver
       Adjust for mlx5 changes in-between submissions
      V10:
       Minor adjust for mlx5 requested by Tariq
       Resubmit against net-next
      V11: avoid leaking info stored in frame data on page reuse
      ====================
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      684009d4
    • Jesper Dangaard Brouer's avatar
      xdp: avoid leaking info stored in frame data on page reuse · 6dfb970d
      Jesper Dangaard Brouer authored
      The bpf infrastructure and verifier goes to great length to avoid
      bpf progs leaking kernel (pointer) info.
      
      For queueing an xdp_buff via XDP_REDIRECT, xdp_frame info stores
      kernel info (incl pointers) in top part of frame data (xdp->data_hard_start).
      Checks are in place to assure enough headroom is available for this.
      
      This info is not cleared, and if the frame is reused, then a
      malicious user could use bpf_xdp_adjust_head helper to move
      xdp->data into this area.  Thus, making this area readable.
      
      This is not super critical as XDP progs requires root or
      CAP_SYS_ADMIN, which are privileged enough for such info.  An
      effort (is underway) towards moving networking bpf hooks to the
      lesser privileged mode CAP_NET_ADMIN, where leaking such info
      should be avoided.  Thus, this patch to clear the info when
      needed.
      Signed-off-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      6dfb970d
    • Jesper Dangaard Brouer's avatar
      xdp: transition into using xdp_frame for ndo_xdp_xmit · 44fa2dbd
      Jesper Dangaard Brouer authored
      Changing API ndo_xdp_xmit to take a struct xdp_frame instead of struct
      xdp_buff.  This brings xdp_return_frame and ndp_xdp_xmit in sync.
      
      This builds towards changing the API further to become a bulk API,
      because xdp_buff is not a queue-able object while xdp_frame is.
      
      V4: Adjust for commit 59655a5b ("tuntap: XDP_TX can use native XDP")
      V7: Adjust for commit d9314c47 ("i40e: add support for XDP_REDIRECT")
      Signed-off-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      44fa2dbd
    • Jesper Dangaard Brouer's avatar
      xdp: transition into using xdp_frame for return API · 03993094
      Jesper Dangaard Brouer authored
      Changing API xdp_return_frame() to take struct xdp_frame as argument,
      seems like a natural choice. But there are some subtle performance
      details here that needs extra care, which is a deliberate choice.
      
      When de-referencing xdp_frame on a remote CPU during DMA-TX
      completion, result in the cache-line is change to "Shared"
      state. Later when the page is reused for RX, then this xdp_frame
      cache-line is written, which change the state to "Modified".
      
      This situation already happens (naturally) for, virtio_net, tun and
      cpumap as the xdp_frame pointer is the queued object.  In tun and
      cpumap, the ptr_ring is used for efficiently transferring cache-lines
      (with pointers) between CPUs. Thus, the only option is to
      de-referencing xdp_frame.
      
      It is only the ixgbe driver that had an optimization, in which it can
      avoid doing the de-reference of xdp_frame.  The driver already have
      TX-ring queue, which (in case of remote DMA-TX completion) have to be
      transferred between CPUs anyhow.  In this data area, we stored a
      struct xdp_mem_info and a data pointer, which allowed us to avoid
      de-referencing xdp_frame.
      
      To compensate for this, a prefetchw is used for telling the cache
      coherency protocol about our access pattern.  My benchmarks show that
      this prefetchw is enough to compensate the ixgbe driver.
      
      V7: Adjust for commit d9314c47 ("i40e: add support for XDP_REDIRECT")
      V8: Adjust for commit bd658dda ("net/mlx5e: Separate dma base address
      and offset in dma_sync call")
      Signed-off-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      03993094
    • Jesper Dangaard Brouer's avatar
      mlx5: use page_pool for xdp_return_frame call · 60bbf7ee
      Jesper Dangaard Brouer authored
      This patch shows how it is possible to have both the driver local page
      cache, which uses elevated refcnt for "catching"/avoiding SKB
      put_page returns the page through the page allocator.  And at the
      same time, have pages getting returned to the page_pool from
      ndp_xdp_xmit DMA completion.
      
      The performance improvement for XDP_REDIRECT in this patch is really
      good.  Especially considering that (currently) the xdp_return_frame
      API and page_pool_put_page() does per frame operations of both
      rhashtable ID-lookup and locked return into (page_pool) ptr_ring.
      (It is the plan to remove these per frame operation in a followup
      patchset).
      
      The benchmark performed was RX on mlx5 and XDP_REDIRECT out ixgbe,
      with xdp_redirect_map (using devmap) . And the target/maximum
      capability of ixgbe is 13Mpps (on this HW setup).
      
      Before this patch for mlx5, XDP redirected frames were returned via
      the page allocator.  The single flow performance was 6Mpps, and if I
      started two flows the collective performance drop to 4Mpps, because we
      hit the page allocator lock (further negative scaling occurs).
      
      Two test scenarios need to be covered, for xdp_return_frame API, which
      is DMA-TX completion running on same-CPU or cross-CPU free/return.
      Results were same-CPU=10Mpps, and cross-CPU=12Mpps.  This is very
      close to our 13Mpps max target.
      
      The reason max target isn't reached in cross-CPU test, is likely due
      to RX-ring DMA unmap/map overhead (which doesn't occur in ixgbe to
      ixgbe testing).  It is also planned to remove this unnecessary DMA
      unmap in a later patchset
      
      V2: Adjustments requested by Tariq
       - Changed page_pool_create return codes not return NULL, only
         ERR_PTR, as this simplifies err handling in drivers.
       - Save a branch in mlx5e_page_release
       - Correct page_pool size calc for MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ
      
      V5: Updated patch desc
      
      V8: Adjust for b0cedc84 ("net/mlx5e: Remove rq_headroom field from params")
      V9:
       - Adjust for 121e8927 ("net/mlx5e: Refactor RQ XDP_TX indication")
       - Adjust for 73281b78 ("net/mlx5e: Derive Striding RQ size from MTU")
       - Correct handling if page_pool_create fail for MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ
      
      V10: Req from Tariq
       - Change pool_size calc for MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ
      Signed-off-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Reviewed-by: default avatarTariq Toukan <tariqt@mellanox.com>
      Acked-by: default avatarSaeed Mahameed <saeedm@mellanox.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      60bbf7ee
    • Jesper Dangaard Brouer's avatar
      xdp: allow page_pool as an allocator type in xdp_return_frame · 57d0a1c1
      Jesper Dangaard Brouer authored
      New allocator type MEM_TYPE_PAGE_POOL for page_pool usage.
      
      The registered allocator page_pool pointer is not available directly
      from xdp_rxq_info, but it could be (if needed).  For now, the driver
      should keep separate track of the page_pool pointer, which it should
      use for RX-ring page allocation.
      
      As suggested by Saeed, to maintain a symmetric API it is the drivers
      responsibility to allocate/create and free/destroy the page_pool.
      Thus, after the driver have called xdp_rxq_info_unreg(), it is drivers
      responsibility to free the page_pool, but with a RCU free call.  This
      is done easily via the page_pool helper page_pool_destroy() (which
      avoids touching any driver code during the RCU callback, which could
      happen after the driver have been unloaded).
      
      V8: address issues found by kbuild test robot
       - Address sparse should be static warnings
       - Allow xdp.o to be compiled without page_pool.o
      
      V9: Remove inline from .c file, compiler knows best
      Signed-off-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      57d0a1c1
    • Jesper Dangaard Brouer's avatar
      page_pool: refurbish version of page_pool code · ff7d6b27
      Jesper Dangaard Brouer authored
      Need a fast page recycle mechanism for ndo_xdp_xmit API for returning
      pages on DMA-TX completion time, which have good cross CPU
      performance, given DMA-TX completion time can happen on a remote CPU.
      
      Refurbish my page_pool code, that was presented[1] at MM-summit 2016.
      Adapted page_pool code to not depend the page allocator and
      integration into struct page.  The DMA mapping feature is kept,
      even-though it will not be activated/used in this patchset.
      
      [1] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
      
      V2: Adjustments requested by Tariq
       - Changed page_pool_create return codes, don't return NULL, only
         ERR_PTR, as this simplifies err handling in drivers.
      
      V4: many small improvements and cleanups
      - Add DOC comment section, that can be used by kernel-doc
      - Improve fallback mode, to work better with refcnt based recycling
        e.g. remove a WARN as pointed out by Tariq
        e.g. quicker fallback if ptr_ring is empty.
      
      V5: Fixed SPDX license as pointed out by Alexei
      
      V6: Adjustments requested by Eric Dumazet
       - Adjust ____cacheline_aligned_in_smp usage/placement
       - Move rcu_head in struct page_pool
       - Free pages quicker on destroy, minimize resources delayed an RCU period
       - Remove code for forward/backward compat ABI interface
      
      V8: Issues found by kbuild test robot
       - Address sparse should be static warnings
       - Only compile+link when a driver use/select page_pool,
         mlx5 selects CONFIG_PAGE_POOL, although its first used in two patches
      Signed-off-by: default avatarJesper Dangaard Brouer <brouer@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      ff7d6b27