• Tonghao Zhang's avatar
    net: sched: use queue_mapping to pick tx queue · 2f1e85b1
    Tonghao Zhang authored
    This patch fixes issue:
    * If we install tc filters with act_skbedit in clsact hook.
      It doesn't work, because netdev_core_pick_tx() overwrites
      queue_mapping.
    
      $ tc filter ... action skbedit queue_mapping 1
    
    And this patch is useful:
    * We can use FQ + EDT to implement efficient policies. Tx queues
      are picked by xps, ndo_select_queue of netdev driver, or skb hash
      in netdev_core_pick_tx(). In fact, the netdev driver, and skb
      hash are _not_ under control. xps uses the CPUs map to select Tx
      queues, but we can't figure out which task_struct of pod/containter
      running on this cpu in most case. We can use clsact filters to classify
      one pod/container traffic to one Tx queue. Why ?
    
      In containter networking environment, there are two kinds of pod/
      containter/net-namespace. One kind (e.g. P1, P2), the high throughput
      is key in these applications. But avoid running out of network resource,
      the outbound traffic of these pods is limited, using or sharing one
      dedicated Tx queues assigned HTB/TBF/FQ Qdisc. Other kind of pods
      (e.g. Pn), the low latency of data access is key. And the traffic is not
      limited. Pods use or share other dedicated Tx queues assigned FIFO Qdisc.
      This choice provides two benefits. First, contention on the HTB/FQ Qdisc
      lock is significantly reduced since fewer CPUs contend for the same queue.
      More importantly, Qdisc contention can be eliminated completely if each
      CPU has its own FIFO Qdisc for the second kind of pods.
    
      There must be a mechanism in place to support classifying traffic based on
      pods/container to different Tx queues. Note that clsact is outside of Qdisc
      while Qdisc can run a classifier to select a sub-queue under the lock.
    
      In general recording the decision in the skb seems a little heavy handed.
      This patch introduces a per-CPU variable, suggested by Eric.
    
      The xmit.skip_txqueue flag is firstly cleared in __dev_queue_xmit().
      - Tx Qdisc may install that skbedit actions, then xmit.skip_txqueue flag
        is set in qdisc->enqueue() though tx queue has been selected in
        netdev_tx_queue_mapping() or netdev_core_pick_tx(). That flag is cleared
        firstly in __dev_queue_xmit(), is useful:
      - Avoid picking Tx queue with netdev_tx_queue_mapping() in next netdev
        in such case: eth0 macvlan - eth0.3 vlan - eth0 ixgbe-phy:
        For example, eth0, macvlan in pod, which root Qdisc install skbedit
        queue_mapping, send packets to eth0.3, vlan in host. In __dev_queue_xmit() of
        eth0.3, clear the flag, does not select tx queue according to skb->queue_mapping
        because there is no filters in clsact or tx Qdisc of this netdev.
        Same action taked in eth0, ixgbe in Host.
      - Avoid picking Tx queue for next packet. If we set xmit.skip_txqueue
        in tx Qdisc (qdisc->enqueue()), the proper way to clear it is clearing it
        in __dev_queue_xmit when processing next packets.
    
      For performance reasons, use the static key. If user does not config the NET_EGRESS,
      the patch will not be compiled.
    
      +----+      +----+      +----+
      | P1 |      | P2 |      | Pn |
      +----+      +----+      +----+
        |           |           |
        +-----------+-----------+
                    |
                    | clsact/skbedit
                    |      MQ
                    v
        +-----------+-----------+
        | q0        | q1        | qn
        v           v           v
      HTB/FQ      HTB/FQ  ...  FIFO
    
    Cc: Jamal Hadi Salim <jhs@mojatatu.com>
    Cc: Cong Wang <xiyou.wangcong@gmail.com>
    Cc: Jiri Pirko <jiri@resnulli.us>
    Cc: "David S. Miller" <davem@davemloft.net>
    Cc: Jakub Kicinski <kuba@kernel.org>
    Cc: Jonathan Lemon <jonathan.lemon@gmail.com>
    Cc: Eric Dumazet <edumazet@google.com>
    Cc: Alexander Lobakin <alobakin@pm.me>
    Cc: Paolo Abeni <pabeni@redhat.com>
    Cc: Talal Ahmad <talalahmad@google.com>
    Cc: Kevin Hao <haokexin@gmail.com>
    Cc: Ilias Apalodimas <ilias.apalodimas@linaro.org>
    Cc: Kees Cook <keescook@chromium.org>
    Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>
    Cc: Antoine Tenart <atenart@kernel.org>
    Cc: Wei Wang <weiwan@google.com>
    Cc: Arnd Bergmann <arnd@arndb.de>
    Suggested-by: default avatarEric Dumazet <edumazet@google.com>
    Signed-off-by: default avatarTonghao Zhang <xiangxia.m.yue@gmail.com>
    Acked-by: default avatarJamal Hadi Salim <jhs@mojatatu.com>
    Signed-off-by: default avatarPaolo Abeni <pabeni@redhat.com>
    2f1e85b1
act_skbedit.c 11.7 KB