• Tejun Heo's avatar
    sched_ext: Implement DSQ iterator · 650ba21b
    Tejun Heo authored
    DSQs are very opaque in the consumption path. The BPF scheduler has no way
    of knowing which tasks are being considered and which is picked. This patch
    adds BPF DSQ iterator.
    
    - Allows iterating tasks queued on a DSQ in the dispatch order or reverse
      from anywhere using bpf_for_each(scx_dsq) or calling the iterator kfuncs
      directly.
    
    - Has ordering guarantee where only tasks which were already queued when the
      iteration started are visible and consumable during the iteration.
    
    v5: - Add a comment to the naked list_empty(&dsq->list) test in
          consume_dispatch_q() to explain the reasoning behind the lockless test
          and by extension why nldsq_next_task() isn't used there.
    
        - scx_qmap changes separated into its own patch.
    
    v4: - bpf_iter_scx_dsq_new() declaration in common.bpf.h was using the wrong
          type for the last argument (bool rev instead of u64 flags). Fix it.
    
    v3: - Alexei pointed out that the iterator is too big to allocate on stack.
          Added a prep patch to reduce the size of the cursor. Now
          bpf_iter_scx_dsq is 48 bytes and bpf_iter_scx_dsq_kern is 40 bytes on
          64bit.
    
        - u32_before() comparison factored out.
    
    v2: - scx_bpf_consume_task() is separated out into a separate patch.
    
        - DSQ seq and iter flags don't need to be u64. Use u32.
    Signed-off-by: default avatarTejun Heo <tj@kernel.org>
    Reviewed-by: default avatarDavid Vernet <dvernet@meta.com>
    Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Cc: bpf@vger.kernel.org
    650ba21b
ext.c 181 KB