• Daniel Borkmann's avatar
    bpf: direct packet write and access for helpers for clsact progs · 36bbef52
    Daniel Borkmann authored
    This work implements direct packet access for helpers and direct packet
    write in a similar fashion as already available for XDP types via commits
    4acf6c0b ("bpf: enable direct packet data write for xdp progs") and
    6841de8b ("bpf: allow helpers access the packet directly"), and as a
    complementary feature to the already available direct packet read for tc
    (cls/act) programs.
    
    For enabling this, we need to introduce two helpers, bpf_skb_pull_data()
    and bpf_csum_update(). The first is generally needed for both, read and
    write, because they would otherwise only be limited to the current linear
    skb head. Usually, when the data_end test fails, programs just bail out,
    or, in the direct read case, use bpf_skb_load_bytes() as an alternative
    to overcome this limitation. If such data sits in non-linear parts, we
    can just pull them in once with the new helper, retest and eventually
    access them.
    
    At the same time, this also makes sure the skb is uncloned, which is, of
    course, a necessary condition for direct write. As this needs to be an
    invariant for the write part only, the verifier detects writes and adds
    a prologue that is calling bpf_skb_pull_data() to effectively unclone the
    skb from the very beginning in case it is indeed cloned. The heuristic
    makes use of a similar trick that was done in 233577a2 ("net: filter:
    constify detection of pkt_type_offset"). This comes at zero cost for other
    programs that do not use the direct write feature. Should a program use
    this feature only sparsely and has read access for the most parts with,
    for example, drop return codes, then such write action can be delegated
    to a tail called program for mitigating this cost of potential uncloning
    to a late point in time where it would have been paid similarly with the
    bpf_skb_store_bytes() as well. Advantage of direct write is that the
    writes are inlined whereas the helper cannot make any length assumptions
    and thus needs to generate a call to memcpy() also for small sizes, as well
    as cost of helper call itself with sanity checks are avoided. Plus, when
    direct read is already used, we don't need to cache or perform rechecks
    on the data boundaries (due to verifier invalidating previous checks for
    helpers that change skb->data), so more complex programs using rewrites
    can benefit from switching to direct read plus write.
    
    For direct packet access to helpers, we save the otherwise needed copy into
    a temp struct sitting on stack memory when use-case allows. Both facilities
    are enabled via may_access_direct_pkt_data() in verifier. For now, we limit
    this to map helpers and csum_diff, and can successively enable other helpers
    where we find it makes sense. Helpers that definitely cannot be allowed for
    this are those part of bpf_helper_changes_skb_data() since they can change
    underlying data, and those that write into memory as this could happen for
    packet typed args when still cloned. bpf_csum_update() helper accommodates
    for the fact that we need to fixup checksum_complete when using direct write
    instead of bpf_skb_store_bytes(), meaning the programs can use available
    helpers like bpf_csum_diff(), and implement csum_add(), csum_sub(),
    csum_block_add(), csum_block_sub() equivalents in eBPF together with the
    new helper. A usage example will be provided for iproute2's examples/bpf/
    directory.
    Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
    Acked-by: default avatarAlexei Starovoitov <ast@kernel.org>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    36bbef52
verifier.c 79.9 KB