1. 09 Feb, 2017 3 commits
  2. 07 Feb, 2017 7 commits
    • David S. Miller's avatar
      Merge branch 'bridge-improve-cache-utilization' · 152bff37
      David S. Miller authored
      Nikolay Aleksandrov says:
      
      ====================
      bridge: improve cache utilization
      
      This is the first set which begins to deal with the bad bridge cache
      access patterns. The first patch rearranges the bridge and port structs
      a little so the frequently (and closely) accessed members are in the same
      cache line. The second patch then moves the garbage collection to a
      workqueue trying to improve system responsiveness under load (many fdbs)
      and more importantly removes the need to check if the matched entry is
      expired in __br_fdb_get which was a major source of false-sharing.
      The third patch is a preparation for the final one which
      If properly configured, i.e. ports bound to CPUs (thus updating "updated"
      locally) then the bridge's HitM goes from 100% to 0%, but even without
      binding we get a win because previously every lookup that iterated over
      the hash chain caused false-sharing due to the first cache line being
      used for both mac/vid and used/updated fields.
      
      Some results from tests I've run:
      (note that these were run in good conditions for the baseline, everything
       ran on a single NUMA node and there were only 3 fdbs)
      
      1. baseline
      100% Load HitM on the fdbs (between everyone who has done lookups and hit
                                  one of the 3 hash chains of the communicating
                                  src/dst fdbs)
      Overall 5.06% Load HitM for the bridge, first place in the list
      
      2. patched & ports bound to CPUs
      0% Local load HitM, bridge is not even in the c2c report list
      Also there's 3% consistent improvement in netperf tests.
      ====================
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      152bff37
    • Nikolay Aleksandrov's avatar
      bridge: fdb: write to used and updated at most once per jiffy · 83a718d6
      Nikolay Aleksandrov authored
      Writing once per jiffy is enough to limit the bridge's false sharing.
      After this change the bridge doesn't show up in the local load HitM stats.
      Suggested-by: default avatarDavid S. Miller <davem@davemloft.net>
      Signed-off-by: default avatarNikolay Aleksandrov <nikolay@cumulusnetworks.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      83a718d6
    • Nikolay Aleksandrov's avatar
      bridge: move write-heavy fdb members in their own cache line · 1214628c
      Nikolay Aleksandrov authored
      Fdb's used and updated fields are written to on every packet forward and
      packet receive respectively. Thus if we are receiving packets from a
      particular fdb, they'll cause false-sharing with everyone who has looked
      it up (even if it didn't match, since mac/vid share cache line!). The
      "used" field is even worse since it is updated on every packet forward
      to that fdb, thus the standard config where X ports use a single gateway
      results in 100% fdb false-sharing. Note that this patch does not prevent
      the last scenario, but it makes it better for other bridge participants
      which are not using that fdb (and are only doing lookups over it).
      The point is with this move we make sure that only communicating parties
      get the false-sharing, in a later patch we'll show how to avoid that too.
      Signed-off-by: default avatarNikolay Aleksandrov <nikolay@cumulusnetworks.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      1214628c
    • Nikolay Aleksandrov's avatar
      bridge: move to workqueue gc · f7cdee8a
      Nikolay Aleksandrov authored
      Move the fdb garbage collector to a workqueue which fires at least 10
      milliseconds apart and cleans chain by chain allowing for other tasks
      to run in the meantime. When having thousands of fdbs the system is much
      more responsive. Most importantly remove the need to check if the
      matched entry has expired in __br_fdb_get that causes false-sharing and
      is completely unnecessary if we cleanup entries, at worst we'll get 10ms
      of traffic for that entry before it gets deleted.
      Signed-off-by: default avatarNikolay Aleksandrov <nikolay@cumulusnetworks.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      f7cdee8a
    • Nikolay Aleksandrov's avatar
      bridge: modify bridge and port to have often accessed fields in one cache line · 1f90c7f3
      Nikolay Aleksandrov authored
      Move around net_bridge so the vlan fields are in the beginning since
      they're checked on every packet even if vlan filtering is disabled.
      For the port move flags & vlan group to the beginning, so they're in the
      same cache line with the port's state (both flags and state are checked
      on each packet).
      Signed-off-by: default avatarNikolay Aleksandrov <nikolay@cumulusnetworks.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      1f90c7f3
    • William Tu's avatar
      bpf: enable verifier to add 0 to packet ptr · 63dfef75
      William Tu authored
      The patch fixes the case when adding a zero value to the packet
      pointer.  The zero value could come from src_reg equals type
      BPF_K or CONST_IMM.  The patch fixes both, otherwise the verifer
      reports the following error:
        [...]
          R0=imm0,min_value=0,max_value=0
          R1=pkt(id=0,off=0,r=4)
          R2=pkt_end R3=fp-12
          R4=imm4,min_value=4,max_value=4
          R5=pkt(id=0,off=4,r=4)
        269: (bf) r2 = r0     // r2 becomes imm0
        270: (77) r2 >>= 3
        271: (bf) r4 = r1     // r4 becomes pkt ptr
        272: (0f) r4 += r2    // r4 += 0
        addition of negative constant to packet pointer is not allowed
      Signed-off-by: default avatarWilliam Tu <u9012063@gmail.com>
      Signed-off-by: default avatarMihai Budiu <mbudiu@vmware.com>
      Cc: Daniel Borkmann <daniel@iogearbox.net>
      Cc: Alexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      63dfef75
    • Josef Bacik's avatar
      bpf: test for AND edge cases · 29200c19
      Josef Bacik authored
      These two tests are based on the work done for f23cc643.  The first test is
      just a basic one to make sure we don't allow AND'ing negative values, even if it
      would result in a valid index for the array.  The second is a cleaned up version
      of the original testcase provided by Jann Horn that resulted in the commit.
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      29200c19
  3. 06 Feb, 2017 30 commits