1. 21 Apr, 2010 2 commits
  2. 20 Apr, 2010 4 commits
  3. 19 Apr, 2010 13 commits
  4. 18 Apr, 2010 2 commits
    • Eric Dumazet's avatar
      net: Introduce skb_orphan_try() · fc6055a5
      Eric Dumazet authored
      Transmitted skb might be attached to a socket and a destructor, for
      memory accounting purposes.
      
      Traditionally, this destructor is called at tx completion time, when skb
      is freed.
      
      When tx completion is performed by another cpu than the sender, this
      forces some cache lines to change ownership. XPS was an attempt to give
      tx completion to initial cpu.
      
      David idea is to call destructor right before giving skb to device (call
      to ndo_start_xmit()). Because device queues are usually small, orphaning
      skb before tx completion is not a big deal. Some drivers already do
      this, we could do it in upper level.
      
      There is one known exception to this early orphaning, called tx
      timestamping. It needs to keep a reference to socket until device can
      give a hardware or software timestamp.
      
      This patch adds a skb_orphan_try() helper, to centralize all exceptions
      to early orphaning in one spot, and use it in dev_hard_start_xmit().
      
      "tbench 16" results on a Nehalem machine (2 X5570  @ 2.93GHz)
      before: Throughput 4428.9 MB/sec 16 procs
      after: Throughput 4448.14 MB/sec 16 procs
      
      UDP should get even better results, its destructor being more complex,
      since SOCK_USE_WRITE_QUEUE is not set (four atomic ops instead of one)
      Signed-off-by: default avatarEric Dumazet <eric.dumazet@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      fc6055a5
    • Eric Dumazet's avatar
      net: remove time limit in process_backlog() · 9958da05
      Eric Dumazet authored
      - There is no point to enforce a time limit in process_backlog(), since
      other napi instances dont follow same rule. We can exit after only one
      packet processed...
      The normal quota of 64 packets per napi instance should be the norm, and
      net_rx_action() already has its own time limit.
      Note : /proc/net/core/dev_weight can be used to tune this 64 default
      value.
      
      - Use DEFINE_PER_CPU_ALIGNED for softnet_data definition.
      Signed-off-by: default avatarEric Dumazet <eric.dumazet@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      9958da05
  5. 17 Apr, 2010 1 commit
  6. 16 Apr, 2010 4 commits
    • Tom Herbert's avatar
      rfs: Receive Flow Steering · fec5e652
      Tom Herbert authored
      This patch implements receive flow steering (RFS).  RFS steers
      received packets for layer 3 and 4 processing to the CPU where
      the application for the corresponding flow is running.  RFS is an
      extension of Receive Packet Steering (RPS).
      
      The basic idea of RFS is that when an application calls recvmsg
      (or sendmsg) the application's running CPU is stored in a hash
      table that is indexed by the connection's rxhash which is stored in
      the socket structure.  The rxhash is passed in skb's received on
      the connection from netif_receive_skb.  For each received packet,
      the associated rxhash is used to look up the CPU in the hash table,
      if a valid CPU is set then the packet is steered to that CPU using
      the RPS mechanisms.
      
      The convolution of the simple approach is that it would potentially
      allow OOO packets.  If threads are thrashing around CPUs or multiple
      threads are trying to read from the same sockets, a quickly changing
      CPU value in the hash table could cause rampant OOO packets--
      we consider this a non-starter.
      
      To avoid OOO packets, this solution implements two types of hash
      tables: rps_sock_flow_table and rps_dev_flow_table.
      
      rps_sock_table is a global hash table.  Each entry is just a CPU
      number and it is populated in recvmsg and sendmsg as described above.
      This table contains the "desired" CPUs for flows.
      
      rps_dev_flow_table is specific to each device queue.  Each entry
      contains a CPU and a tail queue counter.  The CPU is the "current"
      CPU for a matching flow.  The tail queue counter holds the value
      of a tail queue counter for the associated CPU's backlog queue at
      the time of last enqueue for a flow matching the entry.
      
      Each backlog queue has a queue head counter which is incremented
      on dequeue, and so a queue tail counter is computed as queue head
      count + queue length.  When a packet is enqueued on a backlog queue,
      the current value of the queue tail counter is saved in the hash
      entry of the rps_dev_flow_table.
      
      And now the trick: when selecting the CPU for RPS (get_rps_cpu)
      the rps_sock_flow table and the rps_dev_flow table for the RX queue
      are consulted.  When the desired CPU for the flow (found in the
      rps_sock_flow table) does not match the current CPU (found in the
      rps_dev_flow table), the current CPU is changed to the desired CPU
      if one of the following is true:
      
      - The current CPU is unset (equal to RPS_NO_CPU)
      - Current CPU is offline
      - The current CPU's queue head counter >= queue tail counter in the
      rps_dev_flow table.  This checks if the queue tail has advanced
      beyond the last packet that was enqueued using this table entry.
      This guarantees that all packets queued using this entry have been
      dequeued, thus preserving in order delivery.
      
      Making each queue have its own rps_dev_flow table has two advantages:
      1) the tail queue counters will be written on each receive, so
      keeping the table local to interrupting CPU s good for locality.  2)
      this allows lockless access to the table-- the CPU number and queue
      tail counter need to be accessed together under mutual exclusion
      from netif_receive_skb, we assume that this is only called from
      device napi_poll which is non-reentrant.
      
      This patch implements RFS for TCP and connected UDP sockets.
      It should be usable for other flow oriented protocols.
      
      There are two configuration parameters for RFS.  The
      "rps_flow_entries" kernel init parameter sets the number of
      entries in the rps_sock_flow_table, the per rxqueue sysfs entry
      "rps_flow_cnt" contains the number of entries in the rps_dev_flow
      table for the rxqueue.  Both are rounded to power of two.
      
      The obvious benefit of RFS (over just RPS) is that it achieves
      CPU locality between the receive processing for a flow and the
      applications processing; this can result in increased performance
      (higher pps, lower latency).
      
      The benefits of RFS are dependent on cache hierarchy, application
      load, and other factors.  On simple benchmarks, we don't necessarily
      see improvement and sometimes see degradation.  However, for more
      complex benchmarks and for applications where cache pressure is
      much higher this technique seems to perform very well.
      
      Below are some benchmark results which show the potential benfit of
      this patch.  The netperf test has 500 instances of netperf TCP_RR
      test with 1 byte req. and resp.  The RPC test is an request/response
      test similar in structure to netperf RR test ith 100 threads on
      each host, but does more work in userspace that netperf.
      
      e1000e on 8 core Intel
         No RFS or RPS		104K tps at 30% CPU
         No RFS (best RPS config):    290K tps at 63% CPU
         RFS				303K tps at 61% CPU
      
      RPC test	tps	CPU%	50/90/99% usec latency	Latency StdDev
        No RFS/RPS	103K	48%	757/900/3185		4472.35
        RPS only:	174K	73%	415/993/2468		491.66
        RFS		223K	73%	379/651/1382		315.61
      Signed-off-by: default avatarTom Herbert <therbert@google.com>
      Signed-off-by: default avatarEric Dumazet <eric.dumazet@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      fec5e652
    • Shan Wei's avatar
      ipv6: fix the comment of ip6_xmit() · b5d43998
      Shan Wei authored
      ip6_xmit() is used by upper transport protocol.
      Signed-off-by: default avatarShan Wei <shanwei@cn.fujitsu.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      b5d43998
    • Shan Wei's avatar
      net: replace ipfragok with skb->local_df · 4e15ed4d
      Shan Wei authored
      As Herbert Xu said: we should be able to simply replace ipfragok
      with skb->local_df. commit f88037(sctp: Drop ipfargok in sctp_xmit function)
      has droped ipfragok and set local_df value properly.
      
      The patch kills the ipfragok parameter of .queue_xmit().
      Signed-off-by: default avatarShan Wei <shanwei@cn.fujitsu.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      4e15ed4d
    • Shan Wei's avatar
      ipv6: cancel to setting local_df in ip6_xmit() · 0eecb784
      Shan Wei authored
      commit f88037(sctp: Drop ipfargok in sctp_xmit function)
      has droped ipfragok and set local_df value properly.
      
      So the change of commit 77e2f1(ipv6: Fix ip6_xmit to
      send fragments if ipfragok is true) is not needed.
      So the patch remove them.
      Signed-off-by: default avatarShan Wei <shanwei@cn.fujitsu.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      0eecb784
  7. 15 Apr, 2010 10 commits
  8. 14 Apr, 2010 4 commits