1. 26 Nov, 2016 28 commits
  2. 25 Nov, 2016 8 commits
    • David S. Miller's avatar
      Merge branch 'cgroup-bpf' · ca89fa77
      David S. Miller authored
      Daniel Mack says:
      
      ====================
      Add eBPF hooks for cgroups
      
      This is v9 of the patch set to allow eBPF programs for network
      filtering and accounting to be attached to cgroups, so that they apply
      to all sockets of all tasks placed in that cgroup. The logic also
      allows to be extendeded for other cgroup based eBPF logic.
      
      Again, only minor details are updated in this version.
      
      Changes from v8:
      
      * Move the egress hooks into ip_finish_output() and ip6_finish_output()
        so they run after the netfilter hooks. For IPv4 multicast, add a new
        ip_mc_finish_output() callback that is invoked on success by
        netfilter, and call the eBPF program from there.
      
      Changes from v7:
      
      * Replace the static inline function cgroup_bpf_run_filter() with
        two specific macros for ingress and egress.  This addresses David
        Miller's concern regarding skb->sk vs. sk in the egress path.
        Thanks a lot to Daniel Borkmann and Alexei Starovoitov for the
        suggestions.
      
      Changes from v6:
      
      * Rebased to 4.9-rc2
      
      * Add EXPORT_SYMBOL(__cgroup_bpf_run_filter). The kbuild test robot
        now succeeds in building this version of the patch set.
      
      * Switch from bpf_prog_run_save_cb() to bpf_prog_run_clear_cb() to not
        tamper with the contents of skb->cb[]. Pointed out by Daniel
        Borkmann.
      
      * Use sk_to_full_sk() in the egress path, as suggested by Daniel
        Borkmann.
      
      * Renamed BPF_PROG_TYPE_CGROUP_SOCKET to BPF_PROG_TYPE_CGROUP_SKB, as
        requested by David Ahern.
      
      * Added Alexei's Acked-by tags.
      
      Changes from v5:
      
      * The eBPF programs now operate on L3 rather than on L2 of the packets,
        and the egress hooks were moved from __dev_queue_xmit() to
        ip*_output().
      
      * For BPF_PROG_TYPE_CGROUP_SOCKET, disallow direct access to the skb
        through BPF_LD_[ABS|IND] instructions, but hook up the
        bpf_skb_load_bytes() access helper instead. Thanks to Daniel Borkmann
        for the help.
      
      Changes from v4:
      
      * Plug an skb leak when dropping packets due to eBPF verdicts in
        __dev_queue_xmit(). Spotted by Daniel Borkmann.
      
      * Check for sk_fullsock(sk) in __cgroup_bpf_run_filter() so we don't
        operate on timewait or request sockets. Suggested by Daniel Borkmann.
      
      * Add missing @parent parameter in kerneldoc of __cgroup_bpf_update().
        Spotted by Rami Rosen.
      
      * Include linux/jump_label.h from bpf-cgroup.h to fix a kbuild error.
      
      Changes from v3:
      
      * Dropped the _FILTER suffix from BPF_PROG_TYPE_CGROUP_SOCKET_FILTER,
        renamed BPF_ATTACH_TYPE_CGROUP_INET_{E,IN}GRESS to
        BPF_CGROUP_INET_{IN,E}GRESS and alias BPF_MAX_ATTACH_TYPE to
        __BPF_MAX_ATTACH_TYPE, as suggested by Daniel Borkmann.
      
      * Dropped the attach_flags member from the anonymous struct for BPF
        attach operations in union bpf_attr. They can be added later on via
        CHECK_ATTR. Requested by Daniel Borkmann and Alexei.
      
      * Release old_prog at the end of __cgroup_bpf_update rather that at
        the beginning to fix a race gap between program updates and their
        users. Spotted by Daniel Borkmann.
      
      * Plugged an skb leak when dropping packets on the egress path.
        Spotted by Daniel Borkmann.
      
      * Add cgroups@vger.kernel.org to the loop, as suggested by Rami Rosen.
      
      * Some minor coding style adoptions not worth mentioning in particular.
      
      Changes from v2:
      
      * Fixed the RCU locking details Tejun pointed out.
      
      * Assert bpf_attr.flags == 0 in BPF_PROG_DETACH syscall handler.
      
      Changes from v1:
      
      * Moved all bpf specific cgroup code into its own file, and stub
        out related functions for !CONFIG_CGROUP_BPF as static inline nops.
        This way, the call sites are not cluttered with #ifdef guards while
        the feature remains compile-time configurable.
      
      * Implemented the new scheme proposed by Tejun. Per cgroup, store one
        set of pointers that are pinned to the cgroup, and one for the
        programs that are effective. When a program is attached or detached,
        the change is propagated to all the cgroup's descendants. If a
        subcgroup has its own pinned program, skip the whole subbranch in
        order to allow delegation models.
      
      * The hookup for egress packets is now done from __dev_queue_xmit().
      
      * A static key is now used in both the ingress and egress fast paths
        to keep performance penalties close to zero if the feature is
        not in use.
      
      * Overall cleanup to make the accessors use the program arrays.
        This should make it much easier to add new program types, which
        will then automatically follow the pinned vs. effective logic.
      
      * Fixed locking issues, as pointed out by Eric Dumazet and Alexei
        Starovoitov. Changes to the program array are now done with
        xchg() and are protected by cgroup_mutex.
      
      * eBPF programs are now expected to return 1 to let the packet pass,
        not >= 0. Pointed out by Alexei.
      
      * Operation is now limited to INET sockets, so local AF_UNIX sockets
        are not affected. The enum members are renamed accordingly. In case
        other socket families should be supported, this can be extended in
        the future.
      
      * The sample program learned to support both ingress and egress, and
        can now optionally make the eBPF program drop packets by making it
        return 0.
      ====================
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      ca89fa77
    • Daniel Mack's avatar
      samples: bpf: add userspace example for attaching eBPF programs to cgroups · d8c5b17f
      Daniel Mack authored
      Add a simple userpace program to demonstrate the new API to attach eBPF
      programs to cgroups. This is what it does:
      
       * Create arraymap in kernel with 4 byte keys and 8 byte values
      
       * Load eBPF program
      
         The eBPF program accesses the map passed in to store two pieces of
         information. The number of invocations of the program, which maps
         to the number of packets received, is stored to key 0. Key 1 is
         incremented on each iteration by the number of bytes stored in
         the skb.
      
       * Detach any eBPF program previously attached to the cgroup
      
       * Attach the new program to the cgroup using BPF_PROG_ATTACH
      
       * Once a second, read map[0] and map[1] to see how many bytes and
         packets were seen on any socket of tasks in the given cgroup.
      
      The program takes a cgroup path as 1st argument, and either "ingress"
      or "egress" as 2nd. Optionally, "drop" can be passed as 3rd argument,
      which will make the generated eBPF program return 0 instead of 1, so
      the kernel will drop the packet.
      
      libbpf gained two new wrappers for the new syscall commands.
      Signed-off-by: default avatarDaniel Mack <daniel@zonque.org>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      d8c5b17f
    • Daniel Mack's avatar
      net: ipv4, ipv6: run cgroup eBPF egress programs · 33b48679
      Daniel Mack authored
      If the cgroup associated with the receiving socket has an eBPF
      programs installed, run them from ip_output(), ip6_output() and
      ip_mc_output(). From mentioned functions we have two socket contexts
      as per 7026b1dd ("netfilter: Pass socket pointer down through
      okfn()."). We explicitly need to use sk instead of skb->sk here,
      since otherwise the same program would run multiple times on egress
      when encap devices are involved, which is not desired in our case.
      
      eBPF programs used in this context are expected to either return 1 to
      let the packet pass, or != 1 to drop them. The programs have access to
      the skb through bpf_skb_load_bytes(), and the payload starts at the
      network headers (L3).
      
      Note that cgroup_bpf_run_filter() is stubbed out as static inline nop
      for !CONFIG_CGROUP_BPF, and is otherwise guarded by a static key if
      the feature is unused.
      Signed-off-by: default avatarDaniel Mack <daniel@zonque.org>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      33b48679
    • Daniel Mack's avatar
      net: filter: run cgroup eBPF ingress programs · c11cd3a6
      Daniel Mack authored
      If the cgroup associated with the receiving socket has an eBPF
      programs installed, run them from sk_filter_trim_cap().
      
      eBPF programs used in this context are expected to either return 1 to
      let the packet pass, or != 1 to drop them. The programs have access to
      the skb through bpf_skb_load_bytes(), and the payload starts at the
      network headers (L3).
      
      Note that cgroup_bpf_run_filter() is stubbed out as static inline nop
      for !CONFIG_CGROUP_BPF, and is otherwise guarded by a static key if
      the feature is unused.
      Signed-off-by: default avatarDaniel Mack <daniel@zonque.org>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      c11cd3a6
    • Daniel Mack's avatar
      bpf: add BPF_PROG_ATTACH and BPF_PROG_DETACH commands · f4324551
      Daniel Mack authored
      Extend the bpf(2) syscall by two new commands, BPF_PROG_ATTACH and
      BPF_PROG_DETACH which allow attaching and detaching eBPF programs
      to a target.
      
      On the API level, the target could be anything that has an fd in
      userspace, hence the name of the field in union bpf_attr is called
      'target_fd'.
      
      When called with BPF_ATTACH_TYPE_CGROUP_INET_{E,IN}GRESS, the target is
      expected to be a valid file descriptor of a cgroup v2 directory which
      has the bpf controller enabled. These are the only use-cases
      implemented by this patch at this point, but more can be added.
      
      If a program of the given type already exists in the given cgroup,
      the program is swapped automically, so userspace does not have to drop
      an existing program first before installing a new one, which would
      otherwise leave a gap in which no program is attached.
      
      For more information on the propagation logic to subcgroups, please
      refer to the bpf cgroup controller implementation.
      
      The API is guarded by CAP_NET_ADMIN.
      Signed-off-by: default avatarDaniel Mack <daniel@zonque.org>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      f4324551
    • Daniel Mack's avatar
      cgroup: add support for eBPF programs · 30070984
      Daniel Mack authored
      This patch adds two sets of eBPF program pointers to struct cgroup.
      One for such that are directly pinned to a cgroup, and one for such
      that are effective for it.
      
      To illustrate the logic behind that, assume the following example
      cgroup hierarchy.
      
        A - B - C
              \ D - E
      
      If only B has a program attached, it will be effective for B, C, D
      and E. If D then attaches a program itself, that will be effective for
      both D and E, and the program in B will only affect B and C. Only one
      program of a given type is effective for a cgroup.
      
      Attaching and detaching programs will be done through the bpf(2)
      syscall. For now, ingress and egress inet socket filtering are the
      only supported use-cases.
      Signed-off-by: default avatarDaniel Mack <daniel@zonque.org>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      30070984
    • Daniel Mack's avatar
      bpf: add new prog type for cgroup socket filtering · 0e33661d
      Daniel Mack authored
      This program type is similar to BPF_PROG_TYPE_SOCKET_FILTER, except that
      it does not allow BPF_LD_[ABS|IND] instructions and hooks up the
      bpf_skb_load_bytes() helper.
      
      Programs of this type will be attached to cgroups for network filtering
      and accounting.
      Signed-off-by: default avatarDaniel Mack <daniel@zonque.org>
      Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      0e33661d
    • Colin Ian King's avatar
      cxgb4: fix memory leak on txq_info · 619228d8
      Colin Ian King authored
      Currently if txq_info->uldtxq cannot be allocated then
      txq_info->txq is being kfree'd (which is redundant because it
      is NULL) instead of txq_info. Fix this by instead kfree'ing
      txq_info.
      Signed-off-by: default avatarColin Ian King <colin.king@canonical.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      619228d8
  3. 24 Nov, 2016 4 commits
    • Jason Wang's avatar
      tuntap: remove unnecessary sk_receive_queue length check during xmit · 436acceb
      Jason Wang authored
      After commit 1576d986 ("tun: switch to use skb array for tx"),
      sk_receive_queue was not used any more. So remove the uncessary
      sk_receive_queue length check during xmit.
      Signed-off-by: default avatarJason Wang <jasowang@redhat.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      436acceb
    • Alexei Starovoitov's avatar
      samples/bpf: fix bpf loader · db6a71dd
      Alexei Starovoitov authored
      llvm can emit relocations into sections other than program code
      (like debug info sections). Ignore them during parsing of elf file
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      db6a71dd
    • Alexei Starovoitov's avatar
      samples/bpf: fix sockex2 example · d2b024d3
      Alexei Starovoitov authored
      since llvm commit "Do not expand UNDEF SDNode during insn selection lowering"
      llvm will generate code that uses uninitialized registers for cases
      where C code is actually uses uninitialized data.
      So this sockex2 example is technically broken.
      Fix it by initializing on the stack variable fully.
      Also increase verifier buffer limit, since verifier output
      may not fit in 64k for this sockex2 code depending on llvm version.
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      d2b024d3
    • Eric Dumazet's avatar
      mlx4: reorganize struct mlx4_en_tx_ring · e3f42f84
      Eric Dumazet authored
      Goal is to reorganize this critical structure to increase performance.
      
      ndo_start_xmit() should only dirty one cache line, and access as few
      cache lines as possible.
      
      Add sp_ (Slow Path) prefix to fields that are not used in fast path,
      to make clear what is going on.
      
      After this patch pahole reports something much better, as all
      ndo_start_xmit() needed fields are packed into two cache lines instead
      of seven or eight
      
      struct mlx4_en_tx_ring {
      	u32                        last_nr_txbb;         /*     0   0x4 */
      	u32                        cons;                 /*   0x4   0x4 */
      	long unsigned int          wake_queue;           /*   0x8   0x8 */
      	struct netdev_queue *      tx_queue;             /*  0x10   0x8 */
      	u32                        (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /*  0x18   0x8 */
      	struct mlx4_en_rx_ring *   recycle_ring;         /*  0x20   0x8 */
      
      	/* XXX 24 bytes hole, try to pack */
      
      	/* --- cacheline 1 boundary (64 bytes) --- */
      	u32                        prod;                 /*  0x40   0x4 */
      	unsigned int               tx_dropped;           /*  0x44   0x4 */
      	long unsigned int          bytes;                /*  0x48   0x8 */
      	long unsigned int          packets;              /*  0x50   0x8 */
      	long unsigned int          tx_csum;              /*  0x58   0x8 */
      	long unsigned int          tso_packets;          /*  0x60   0x8 */
      	long unsigned int          xmit_more;            /*  0x68   0x8 */
      	struct mlx4_bf             bf;                   /*  0x70  0x18 */
      	/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
      	__be32                     doorbell_qpn;         /*  0x88   0x4 */
      	__be32                     mr_key;               /*  0x8c   0x4 */
      	u32                        size;                 /*  0x90   0x4 */
      	u32                        size_mask;            /*  0x94   0x4 */
      	u32                        full_size;            /*  0x98   0x4 */
      	u32                        buf_size;             /*  0x9c   0x4 */
      	void *                     buf;                  /*  0xa0   0x8 */
      	struct mlx4_en_tx_info *   tx_info;              /*  0xa8   0x8 */
      	int                        qpn;                  /*  0xb0   0x4 */
      	u8                         queue_index;          /*  0xb4   0x1 */
      	bool                       bf_enabled;           /*  0xb5   0x1 */
      	bool                       bf_alloced;           /*  0xb6   0x1 */
      	u8                         hwtstamp_tx_type;     /*  0xb7   0x1 */
      	u8 *                       bounce_buf;           /*  0xb8   0x8 */
      	/* --- cacheline 3 boundary (192 bytes) --- */
      	long unsigned int          queue_stopped;        /*  0xc0   0x8 */
      	struct mlx4_hwq_resources  sp_wqres;             /*  0xc8  0x58 */
      	/* --- cacheline 4 boundary (256 bytes) was 32 bytes ago --- */
      	struct mlx4_qp             sp_qp;                /* 0x120  0x30 */
      	/* --- cacheline 5 boundary (320 bytes) was 16 bytes ago --- */
      	struct mlx4_qp_context     sp_context;           /* 0x150  0xf8 */
      	/* --- cacheline 9 boundary (576 bytes) was 8 bytes ago --- */
      	cpumask_t                  sp_affinity_mask;     /* 0x248  0x20 */
      	enum mlx4_qp_state         sp_qp_state;          /* 0x268   0x4 */
      	u16                        sp_stride;            /* 0x26c   0x2 */
      	u16                        sp_cqn;               /* 0x26e   0x2 */
      
      	/* size: 640, cachelines: 10, members: 36 */
      	/* sum members: 600, holes: 1, sum holes: 24 */
      	/* padding: 16 */
      };
      
      Instead of this silly placement :
      
      struct mlx4_en_tx_ring {
      	u32                        last_nr_txbb;         /*     0   0x4 */
      	u32                        cons;                 /*   0x4   0x4 */
      	long unsigned int          wake_queue;           /*   0x8   0x8 */
      
      	/* XXX 48 bytes hole, try to pack */
      
      	/* --- cacheline 1 boundary (64 bytes) --- */
      	u32                        prod;                 /*  0x40   0x4 */
      
      	/* XXX 4 bytes hole, try to pack */
      
      	long unsigned int          bytes;                /*  0x48   0x8 */
      	long unsigned int          packets;              /*  0x50   0x8 */
      	long unsigned int          tx_csum;              /*  0x58   0x8 */
      	long unsigned int          tso_packets;          /*  0x60   0x8 */
      	long unsigned int          xmit_more;            /*  0x68   0x8 */
      	unsigned int               tx_dropped;           /*  0x70   0x4 */
      
      	/* XXX 4 bytes hole, try to pack */
      
      	struct mlx4_bf             bf;                   /*  0x78  0x18 */
      	/* --- cacheline 2 boundary (128 bytes) was 16 bytes ago --- */
      	long unsigned int          queue_stopped;        /*  0x90   0x8 */
      	cpumask_t                  affinity_mask;        /*  0x98  0x10 */
      	struct mlx4_qp             qp;                   /*  0xa8  0x30 */
      	/* --- cacheline 3 boundary (192 bytes) was 24 bytes ago --- */
      	struct mlx4_hwq_resources  wqres;                /*  0xd8  0x58 */
      	/* --- cacheline 4 boundary (256 bytes) was 48 bytes ago --- */
      	u32                        size;                 /* 0x130   0x4 */
      	u32                        size_mask;            /* 0x134   0x4 */
      	u16                        stride;               /* 0x138   0x2 */
      
      	/* XXX 2 bytes hole, try to pack */
      
      	u32                        full_size;            /* 0x13c   0x4 */
      	/* --- cacheline 5 boundary (320 bytes) --- */
      	u16                        cqn;                  /* 0x140   0x2 */
      
      	/* XXX 2 bytes hole, try to pack */
      
      	u32                        buf_size;             /* 0x144   0x4 */
      	__be32                     doorbell_qpn;         /* 0x148   0x4 */
      	__be32                     mr_key;               /* 0x14c   0x4 */
      	void *                     buf;                  /* 0x150   0x8 */
      	struct mlx4_en_tx_info *   tx_info;              /* 0x158   0x8 */
      	struct mlx4_en_rx_ring *   recycle_ring;         /* 0x160   0x8 */
      	u32                        (*free_tx_desc)(struct mlx4_en_priv *, struct mlx4_en_tx_ring *, int, u8, u64, int); /* 0x168   0x8 */
      	u8 *                       bounce_buf;           /* 0x170   0x8 */
      	struct mlx4_qp_context     context;              /* 0x178  0xf8 */
      	/* --- cacheline 9 boundary (576 bytes) was 48 bytes ago --- */
      	int                        qpn;                  /* 0x270   0x4 */
      	enum mlx4_qp_state         qp_state;             /* 0x274   0x4 */
      	u8                         queue_index;          /* 0x278   0x1 */
      	bool                       bf_enabled;           /* 0x279   0x1 */
      	bool                       bf_alloced;           /* 0x27a   0x1 */
      
      	/* XXX 5 bytes hole, try to pack */
      
      	/* --- cacheline 10 boundary (640 bytes) --- */
      	struct netdev_queue *      tx_queue;             /* 0x280   0x8 */
      	int                        hwtstamp_tx_type;     /* 0x288   0x4 */
      
      	/* size: 704, cachelines: 11, members: 36 */
      	/* sum members: 587, holes: 6, sum holes: 65 */
      	/* padding: 52 */
      };
      Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
      Reviewed-by: default avatarTariq Toukan <tariqt@mellanox.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      e3f42f84