- 17 Jun, 2021 6 commits
-
-
Andrii Nakryiko authored
migrate_reuseport.c selftest relies on having TCP_FASTOPEN_CONNECT defined in system-wide netinet/tcp.h. Selftests can use up-to-date uapi/linux/tcp.h, but that one doesn't have SOL_TCP. So instead of switching everything to uapi header, add #define for TCP_FASTOPEN_CONNECT to fix the build. Fixes: c9d0bdef ("bpf: Test BPF_SK_REUSEPORT_SELECT_OR_MIGRATE.") Signed-off-by:
Andrii Nakryiko <andrii@kernel.org> Signed-off-by:
Daniel Borkmann <daniel@iogearbox.net> Acked-by:
Kuniyuki Iwashima <kuniyu@amazon.co.jp> Link: https://lore.kernel.org/bpf/20210617041446.425283-1-andrii@kernel.org
-
Daniel Borkmann authored
syzbot reported a shift-out-of-bounds that KUBSAN observed in the interpreter: [...] UBSAN: shift-out-of-bounds in kernel/bpf/core.c:1420:2 shift exponent 255 is too large for 64-bit type 'long long unsigned int' CPU: 1 PID: 11097 Comm: syz-executor.4 Not tainted 5.12.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:79 [inline] dump_stack+0x141/0x1d7 lib/dump_stack.c:120 ubsan_epilogue+0xb/0x5a lib/ubsan.c:148 __ubsan_handle_shift_out_of_bounds.cold+0xb1/0x181 lib/ubsan.c:327 ___bpf_prog_run.cold+0x19/0x56c kernel/bpf/core.c:1420 __bpf_prog_run32+0x8f/0xd0 kernel/bpf/core.c:1735 bpf_dispatcher_nop_func include/linux/bpf.h:644 [inline] bpf_prog_run_pin_on_cpu include/linux/filter.h:624 [inline] bpf_prog_run_clear_cb include/linux/filter.h:755 [inline] run_filter+0x1a1/0x470 net/packet/af_packet.c:2031 packet_rcv+0x313/0x13e0 net/packet/af_packet.c:2104 dev_queue_xmit_nit+0x7c2/0xa90 net/core/dev.c:2387 xmit_one net/core/dev.c:3588 [inline] dev_hard_start_xmit+0xad/0x920 net/core/dev.c:3609 __dev_queue_xmit+0x2121/0x2e00 net/core/dev.c:4182 __bpf_tx_skb net/core/filter.c:2116 [inline] __bpf_redirect_no_mac net/core/filter.c:2141 [inline] __bpf_redirect+0x548/0xc80 net/core/filter.c:2164 ____bpf_clone_redirect net/core/filter.c:2448 [inline] bpf_clone_redirect+0x2ae/0x420 net/core/filter.c:2420 ___bpf_prog_run+0x34e1/0x77d0 kernel/bpf/core.c:1523 __bpf_prog_run512+0x99/0xe0 kernel/bpf/core.c:1737 bpf_dispatcher_nop_func include/linux/bpf.h:644 [inline] bpf_test_run+0x3ed/0xc50 net/bpf/test_run.c:50 bpf_prog_test_run_skb+0xabc/0x1c50 net/bpf/test_run.c:582 bpf_prog_test_run kernel/bpf/syscall.c:3127 [inline] __do_sys_bpf+0x1ea9/0x4f00 kernel/bpf/syscall.c:4406 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 entry_SYSCALL_64_after_hwframe+0x44/0xae [...] Generally speaking, KUBSAN reports from the kernel should be fixed. However, in case of BPF, this particular report caused concerns since the large shift is not wrong from BPF point of view, just undefined. In the verifier, K-based shifts that are >= {64,32} (depending on the bitwidth of the instruction) are already rejected. The register-based cases were not given their content might not be known at verification time. Ideas such as verifier instruction rewrite with an additional AND instruction for the source register were brought up, but regularly rejected due to the additional runtime overhead they incur. As Edward Cree rightly put it: Shifts by more than insn bitness are legal in the BPF ISA; they are implementation-defined behaviour [of the underlying architecture], rather than UB, and have been made legal for performance reasons. Each of the JIT backends compiles the BPF shift operations to machine instructions which produce implementation-defined results in such a case; the resulting contents of the register may be arbitrary but program behaviour as a whole remains defined. Guard checks in the fast path (i.e. affecting JITted code) will thus not be accepted. The case of division by zero is not truly analogous here, as division instructions on many of the JIT-targeted architectures will raise a machine exception / fault on division by zero, whereas (to the best of my knowledge) none will do so on an out-of-bounds shift. Given the KUBSAN report only affects the BPF interpreter, but not JITs, one solution is to add the ANDs with 63 or 31 into ___bpf_prog_run(). That would make the shifts defined, and thus shuts up KUBSAN, and the compiler would optimize out the AND on any CPU that interprets the shift amounts modulo the width anyway (e.g., confirmed from disassembly that on x86-64 and arm64 the generated interpreter code is the same before and after this fix). The BPF interpreter is slow path, and most likely compiled out anyway as distros select BPF_JIT_ALWAYS_ON to avoid speculative execution of BPF instructions by the interpreter. Given the main argument was to avoid sacrificing performance, the fact that the AND is optimized away from compiler for mainstream archs helps as well as a solution moving forward. Also add a comment on LSH/RSH/ARSH translation for JIT authors to provide guidance when they see the ___bpf_prog_run() interpreter code and use it as a model for a new JIT backend. Reported-by: syzbot+bed360704c521841c85d@syzkaller.appspotmail.com Reported-by:Kurt Manucredo <fuzzybritches0@gmail.com> Signed-off-by:
Eric Biggers <ebiggers@kernel.org> Co-developed-by:
Alexei Starovoitov <ast@kernel.org> Acked-by:
-
Lorenz Bauer authored
bpf2go is the Go equivalent of libbpf skeleton. The convention is that the compiled BPF is checked into the repository to facilitate distributing BPF as part of Go packages. To make this portable, bpf2go by default generates both bpfel and bpfeb variants of the C. Using bpf_tracing.h is inherently non-portable since the fields of struct pt_regs differ between platforms, so CO-RE can't help us here. The only way of working around this is to compile for each target platform independently. bpf2go can't do this by default since there are too many platforms. Define the various PT_... macros when no target can be determined and turn them into compilation failures. This works because bpf2go always compiles for bpf targets, so the compiler fallback doesn't kick in. Conditionally define __BPF_MISSING_TARGET so that we can inject a more appropriate error message at build time. The user can then choose which platform to target explicitly. Signed-off-by:
Lorenz Bauer <lmb@cloudflare.com> Signed-off-by:
-
Wang Hai authored
xdp_sample_pkts usage() is missing the introduction of the "-S" option, this patch adds it. Fixes: d50ecc46 ("samples/bpf: Attach XDP programs in driver mode by default") Signed-off-by:
Wang Hai <wanghai38@huawei.com> Signed-off-by:
Jesper Dangaard Brouer <brouer@redhat.com> Link: https://lore.kernel.org/bpf/20210615135724.29528-1-wanghai38@huawei.com
-
Wang Hai authored
xdp_fwd usage() is missing the introduction of the "-S" and "-F" options, this patch adds it. Fixes: d50ecc46 ("samples/bpf: Attach XDP programs in driver mode by default") Signed-off-by:
-
Shuyi Cheng authored
Fix s/sleeable/sleepable/ typo in a comment. Signed-off-by:
Shuyi Cheng <chengshuyi@linux.alibaba.com> Signed-off-by:
-
- 16 Jun, 2021 1 commit
-
-
Daniel Xu authored
Somehow test_progs.h was being included by the existing rule: /test_progs* This is bad because: 1) test_progs.h is a checked in file 2) grep-like tools like ripgrep[0] respect gitignore and test_progs.h was being hidden from searches [0]: https://github.com/BurntSushi/ripgrep Fixes: 74b5a596 ("selftests/bpf: Replace test_progs and test_maps w/ general rule") Signed-off-by:Daniel Xu <dxu@dxuuu.xyz> Signed-off-by:
-
- 15 Jun, 2021 14 commits
-
-
Daniel Borkmann authored
Kuniyuki Iwashima says: ==================== The SO_REUSEPORT option allows sockets to listen on the same port and to accept connections evenly. However, there is a defect in the current implementation [1]. When a SYN packet is received, the connection is tied to a listening socket. Accordingly, when the listener is closed, in-flight requests during the three-way handshake and child sockets in the accept queue are dropped even if other listeners on the same port could accept such connections. This situation can happen when various server management tools restart server (such as nginx) processes. For instance, when we change nginx configurations and restart it, it spins up new workers that respect the new configuration and closes all listeners on the old workers, resulting in the in-flight ACK of 3WHS is responded by RST. To avoid such a situation, users have to know deeply how the kernel handles SYN packets and implement connection draining by eBPF [2]: 1. Stop routing SYN packets to the listener by eBPF. 2. Wait for all timers to expire to complete requests 3. Accept connections until EAGAIN, then close the listener. or 1. Start counting SYN packets and accept syscalls using the eBPF map. 2. Stop routing SYN packets. 3. Accept connections up to the count, then close the listener. In either way, we cannot close a listener immediately. However, ideally, the application need not drain the not yet accepted sockets because 3WHS and tying a connection to a listener are just the kernel behaviour. The root cause is within the kernel, so the issue should be addressed in kernel space and should not be visible to user space. This patchset fixes it so that users need not take care of kernel implementation and connection draining. With this patchset, the kernel redistributes requests and connections from a listener to the others in the same reuseport group at/after close or shutdown syscalls. Although some software does connection draining, there are still merits in migration. For some security reasons, such as replacing TLS certificates, we may want to apply new settings as soon as possible and/or we may not be able to wait for connection draining. The sockets in the accept queue have not started application sessions yet. So, if we do not drain such sockets, they can be handled by the newer listeners and could have a longer lifetime. It is difficult to drain all connections in every case, but we can decrease such aborted connections by migration. In that sense, migration is always better than draining. Moreover, auto-migration simplifies user space logic and also works well in a case where we cannot modify and build a server program to implement the workaround. Note that the source and destination listeners MUST have the same settings at the socket API level; otherwise, applications may face inconsistency and cause errors. In such a case, we have to use the eBPF program to select a specific listener or to cancel migration. Special thanks to Martin KaFai Lau for bouncing ideas and exchanging code snippets along the way. Link: [1] The SO_REUSEPORT socket option https://lwn.net/Articles/542629/ [2] Re: [PATCH 1/1] net: Add SO_REUSEPORT_LISTEN_OFF socket option as drain mode https://lore.kernel.org/netdev/1458828813.10868.65.camel@edumazet-glaptop3.roam.corp.google.com/ Changelog: v8: * Make reuse const in reuseport_sock_index() * Don't use __reuseport_add_sock() in reuseport_alloc() * Change the arg of the second memcpy() in reuseport_grow() * Fix coding style to use goto in reuseport_alloc() * Keep sk_refcnt uninitialized in inet_reqsk_clone() * Initialize ireq_opt and ipv6_opt separately in reqsk_migrate_reset() [ This series does not include a stats patch suggested by Yuchung Cheng not to drop Acked-by/Reviewed-by tags and save reviewer's time. I will post the patch as a follow up after this series is merged. ] v7: https://lore.kernel.org/bpf/20210521182104.18273-1-kuniyu@amazon.co.jp/ * Prevent attaching/detaching a bpf prog via shutdowned socket * Fix typo in commit messages * Split selftest into subtests v6: https://lore.kernel.org/bpf/20210517002258.75019-1-kuniyu@amazon.co.jp/ * Change description in ip-sysctl.rst * Test IPPROTO_TCP before reading tfo_listener * Move reqsk_clone() to inet_connection_sock.c and rename to inet_reqsk_clone() * Pass req->rsk_listener to inet_csk_reqsk_queue_drop() and reqsk_queue_removed() in the migration path of receiving ACK * s/ARG_PTR_TO_SOCKET/PTR_TO_SOCKET/ in sk_reuseport_is_valid_access() * In selftest, use atomic ops to increment global vars, drop ACK by XDP, enable force fastopen, use "skel->bss" instead of "skel->data" v5: https://lore.kernel.org/bpf/20210510034433.52818-1-kuniyu@amazon.co.jp/ * Move initializtion of sk_node from 6th to 5th patch * Initialize sk_refcnt in reqsk_clone() * Modify some definitions in reqsk_timer_handler() * Validate in which path/state migration happens in selftest v4: https://lore.kernel.org/bpf/20210427034623.46528-1-kuniyu@amazon.co.jp/ * Make some functions and variables 'static' in selftest * Remove 'scalability' from the cover letter v3: https://lore.kernel.org/bpf/20210420154140.80034-1-kuniyu@amazon.co.jp/ * Add sysctl back for reuseport_grow() * Add helper functions to manage socks[] * Separate migration related logic into functions: reuseport_resurrect(), reuseport_stop_listen_sock(), reuseport_migrate_sock() * Clone request_sock to be migrated * Migrate request one by one * Pass child socket to eBPF prog v2: https://lore.kernel.org/netdev/20201207132456.65472-1-kuniyu@amazon.co.jp/ * Do not save closed sockets in socks[] * Revert 607904c3 * Extract inet_csk_reqsk_queue_migrate() into a single patch * Change the spin_lock order to avoid lockdep warning * Add static to __reuseport_select_sock * Use refcount_inc_not_zero() in reuseport_select_migrated_sock() * Set the default attach type in bpf_prog_load_check_attach() * Define new proto of BPF_FUNC_get_socket_cookie * Fix test to be compiled successfully * Update commit messages v1: https://lore.kernel.org/netdev/20201201144418.35045-1-kuniyu@amazon.co.jp/ * Remove the sysctl option * Enable migration if eBPF progam is not attached * Add expected_attach_type to check if eBPF program can migrate sockets * Add a field to tell migration type to eBPF program * Support BPF_FUNC_get_socket_cookie to get the cookie of sk * Allocate an empty skb if skb is NULL * Pass req_to_sk(req)->sk_hash because listener's hash is zero * Update commit messages and coverletter RFC: https://lore.kernel.org/netdev/20201117094023.3685-1-kuniyu@amazon.co.jp/ ==================== Signed-off-by:
-
Kuniyuki Iwashima authored
This patch adds a test for BPF_SK_REUSEPORT_SELECT_OR_MIGRATE and removes 'static' from settimeo() in network_helpers.c. Signed-off-by:
Eric Dumazet <edumazet@google.com> Acked-by:
Martin KaFai Lau <kafai@fb.com> Link: https://lore.kernel.org/bpf/20210612123224.12525-12-kuniyu@amazon.co.jp
-
Kuniyuki Iwashima authored
This commit introduces a new section (sk_reuseport/migrate) and sets expected_attach_type to two each section in BPF_PROG_TYPE_SK_REUSEPORT program. Signed-off-by:
-
Kuniyuki Iwashima authored
This patch introduces a new bpf_attach_type for BPF_PROG_TYPE_SK_REUSEPORT to check if the attached eBPF program is capable of migrating sockets. When the eBPF program is attached, we run it for socket migration if the expected_attach_type is BPF_SK_REUSEPORT_SELECT_OR_MIGRATE or net.ipv4.tcp_migrate_req is enabled. Currently, the expected_attach_type is not enforced for the BPF_PROG_TYPE_SK_REUSEPORT type of program. Thus, this commit follows the earlier idea in the commit aac3fc32 ("bpf: Post-hooks for sys_bind") to fix up the zero expected_attach_type in bpf_prog_load_fixup_attach_type(). Moreover, this patch adds a new field (migrating_sk) to sk_reuseport_md to select a new listener based on the child socket. migrating_sk varies depending on if it is migrating a request in the accept queue or during 3WHS. - accept_queue : sock (ESTABLISHED/SYN_RECV) - 3WHS : request_sock (NEW_SYN_RECV) In the eBPF program, we can select a new listener by BPF_FUNC_sk_select_reuseport(). Also, we can cancel migration by returning SK_DROP. This feature is useful when listeners have different settings at the socket API level or when we want to free resources as soon as possible. - SK_PASS with selected_sk, select it as a new listener - SK_PASS with selected_sk NULL, fallbacks to the random selection - SK_DROP, cancel the migration. There is a noteworthy point. We select a listening socket in three places, but we do not have struct skb at closing a listener or retransmitting a SYN+ACK. On the other hand, some helper functions do not expect skb is NULL (e.g. skb_header_pointer() in BPF_FUNC_skb_load_bytes(), skb_tail_pointer() in BPF_FUNC_skb_load_bytes_relative()). So we allocate an empty skb temporarily before running the eBPF program. Suggested-by:
-
Kuniyuki Iwashima authored
We will call sock_reuseport.prog for socket migration in the next commit, so the eBPF program has to know which listener is closing to select a new listener. We can currently get a unique ID of each listener in the userspace by calling bpf_map_lookup_elem() for BPF_MAP_TYPE_REUSEPORT_SOCKARRAY map. This patch makes the pointer of sk available in sk_reuseport_md so that we can get the ID by BPF_FUNC_get_socket_cookie() in the eBPF program. Suggested-by:
-
Kuniyuki Iwashima authored
This patch also changes the code to call reuseport_migrate_sock() and inet_reqsk_clone(), but unlike the other cases, we do not call inet_reqsk_clone() right after reuseport_migrate_sock(). Currently, in the receive path for TCP_NEW_SYN_RECV sockets, its listener has three kinds of refcnt: (A) for listener itself (B) carried by reuqest_sock (C) sock_hold() in tcp_v[46]_rcv() While processing the req, (A) may disappear by close(listener). Also, (B) can disappear by accept(listener) once we put the req into the accept queue. So, we have to hold another refcnt (C) for the listener to prevent use-after-free. For socket migration, we call reuseport_migrate_sock() to select a listener with (A) and to increment the new listener's refcnt in tcp_v[46]_rcv(). This refcnt corresponds to (C) and is cleaned up later in tcp_v[46]_rcv(). Thus we have to take another refcnt (B) for the newly cloned request_sock. In inet_csk_complete_hashdance(), we hold the count (B), clone the req, and try to put the new req into the accept queue. By migrating req after winning the "own_req" race, we can avoid such a worst situation: CPU 1 looks up req1 CPU 2 looks up req1, unhashes it, then CPU 1 loses the race CPU 3 looks up req2, unhashes it, then CPU 2 loses the race ... Signed-off-by:
-
Kuniyuki Iwashima authored
As with the preceding patch, this patch changes reqsk_timer_handler() to call reuseport_migrate_sock() and inet_reqsk_clone() to migrate in-flight requests at retransmitting SYN+ACKs. If we can select a new listener and clone the request, we resume setting the SYN+ACK timer for the new req. If we can set the timer, we call inet_ehash_insert() to unhash the old req and put the new req into ehash. The noteworthy point here is that by unhashing the old req, another CPU processing it may lose the "own_req" race in tcp_v[46]_syn_recv_sock() and drop the final ACK packet. However, the new timer will recover this situation. Signed-off-by:
-
Kuniyuki Iwashima authored
When we call close() or shutdown() for listening sockets, each child socket in the accept queue are freed at inet_csk_listen_stop(). If we can get a new listener by reuseport_migrate_sock() and clone the request by inet_reqsk_clone(), we try to add it into the new listener's accept queue by inet_csk_reqsk_queue_add(). If it fails, we have to call __reqsk_free() to call sock_put() for its listener and free the cloned request. After putting the full socket into ehash, tcp_v[46]_syn_recv_sock() sets NULL to ireq_opt/pktopts in struct inet_request_sock, but ipv6_opt can be non-NULL. So, we have to set NULL to ipv6_opt of the old request to avoid double free. Note that we do not update req->rsk_listener and instead clone the req to migrate because another path may reference the original request. If we protected it by RCU, we would need to add rcu_read_lock() in many places. Suggested-by:
-
Kuniyuki Iwashima authored
reuseport_migrate_sock() does the same check done in reuseport_listen_stop_sock(). If the reuseport group is capable of migration, reuseport_migrate_sock() selects a new listener by the child socket hash and increments the listener's sk_refcnt beforehand. Thus, if we fail in the migration, we have to decrement it later. We will support migration by eBPF in the later commits. Signed-off-by:
-
Kuniyuki Iwashima authored
When we close a listening socket, to migrate its connections to another listener in the same reuseport group, we have to handle two kinds of child sockets. One is that a listening socket has a reference to, and the other is not. The former is the TCP_ESTABLISHED/TCP_SYN_RECV sockets, and they are in the accept queue of their listening socket. So we can pop them out and push them into another listener's queue at close() or shutdown() syscalls. On the other hand, the latter, the TCP_NEW_SYN_RECV socket is during the three-way handshake and not in the accept queue. Thus, we cannot access such sockets at close() or shutdown() syscalls. Accordingly, we have to migrate immature sockets after their listening socket has been closed. Currently, if their listening socket has been closed, TCP_NEW_SYN_RECV sockets are freed at receiving the final ACK or retransmitting SYN+ACKs. At that time, if we could select a new listener from the same reuseport group, no connection would be aborted. However, we cannot do that because reuseport_detach_sock() sets NULL to sk_reuseport_cb and forbids access to the reuseport group from closed sockets. This patch allows TCP_CLOSE sockets to remain in the reuseport group and access it while any child socket references them. The point is that reuseport_detach_sock() was called twice from inet_unhash() and sk_destruct(). This patch replaces the first reuseport_detach_sock() with reuseport_stop_listen_sock(), which checks if the reuseport group is capable of migration. If capable, it decrements num_socks, moves the socket backwards in socks[] and increments num_closed_socks. When all connections are migrated, sk_destruct() calls reuseport_detach_sock() to remove the socket from socks[], decrement num_closed_socks, and set NULL to sk_reuseport_cb. By this change, closed or shutdowned sockets can keep sk_reuseport_cb. Consequently, calling listen() after shutdown() can cause EADDRINUSE or EBUSY in inet_csk_bind_conflict() or reuseport_add_sock() which expects such sockets not to have the reuseport group. Therefore, this patch also loosens such validation rules so that a socket can listen again if it has a reuseport group with num_closed_socks more than 0. When such sockets listen again, we handle them in reuseport_resurrect(). If there is an existing reuseport group (reuseport_add_sock() path), we move the socket from the old group to the new one and free the old one if necessary. If there is no existing group (reuseport_alloc() path), we allocate a new reuseport group, detach sk from the old one, and free it if necessary, not to break the current shutdown behaviour: - we cannot carry over the eBPF prog of shutdowned sockets - we cannot attach/detach an eBPF prog to/from listening sockets via shutdowned sockets Note that when the number of sockets gets over U16_MAX, we try to detach a closed socket randomly to make room for the new listening socket in reuseport_grow(). Signed-off-by: -
Kuniyuki Iwashima authored
As noted in the following commit, a closed listener has to hold the reference to the reuseport group for socket migration. This patch adds a field (num_closed_socks) to struct sock_reuseport to manage closed sockets within the same reuseport group. Moreover, this and the following commits introduce some helper functions to split socks[] into two sections and keep TCP_LISTEN and TCP_CLOSE sockets in each section. Like a double-ended queue, we will place TCP_LISTEN sockets from the front and TCP_CLOSE sockets from the end. TCP_LISTEN----------> <-------TCP_CLOSE +---+---+ --- +---+ --- +---+ --- +---+ | 0 | 1 | ... | i | ... | j | ... | k | +---+---+ --- +---+ --- +---+ --- +---+ i = num_socks - 1 j = max_socks - num_closed_socks k = max_socks - 1 This patch also extends reuseport_add_sock() and reuseport_grow() to support num_closed_socks. Signed-off-by:
-
Kuniyuki Iwashima authored
This commit adds a new sysctl option: net.ipv4.tcp_migrate_req. If this option is enabled or eBPF program is attached, we will be able to migrate child sockets from a listener to another in the same reuseport group after close() or shutdown() syscalls. Signed-off-by:
Benjamin Herrenschmidt <benh@amazon.com> Reviewed-by:
-
Kumar Kartikeya Dwivedi authored
This got lost during the refactoring across versions. We always use NLM_F_EXCL when creating some TC object, so reflect what the function says and set the flag. Fixes: 715c5ce4 ("libbpf: Add low level TC-BPF management API") Signed-off-by:
Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
Coverity complained about this being unreachable code. It is right because we already enforce flags to be unset, so a check validating the flag value is redundant. Fixes: 715c5ce4 ("libbpf: Add low level TC-BPF management API") Signed-off-by:
-
- 11 Jun, 2021 2 commits
-
-
Zhihao Cheng authored
Fix to return a negative error code from the error handling case instead of 0, as done elsewhere in this function. Fixes: 668da745 ("tools: bpftool: add support for quotations ...") Reported-by:
Hulk Robot <hulkci@huawei.com> Signed-off-by:
Zhihao Cheng <chengzhihao1@huawei.com> Signed-off-by:
Quentin Monnet <quentin@isovalent.com> Link: https://lore.kernel.org/bpf/20210609115916.2186872-1-chengzhihao1@huawei.com
-
Wang Hai authored
There is no need for special treatment of the 'ret == 0' case. This patch simplifies the return expression. Signed-off-by:
Yonghong Song <yhs@fb.com> Link: https://lore.kernel.org/bpf/20210609115651.3392580-1-wanghai38@huawei.com
-
- 08 Jun, 2021 3 commits
-
-
Joe Stringer authored
Previously, if rst2man caught errors, then these would be ignored and the output file would be written anyway. This would allow developers to introduce regressions in the docs comments in the BPF headers. Additionally, even if you instruct rst2man to fail out, it will still write out to the destination target file, so if you ran the tests twice in a row it would always pass. Use a temporary file for the initial run to ensure that if rst2man fails out under "--strict" mode, subsequent runs will not automatically pass. Tested via ./tools/testing/selftests/bpf/test_doc_build.sh Signed-off-by:
Joe Stringer <joe@cilium.io> Signed-off-by:
-
Michal Suchanek authored
The printed value is ptrdiff_t and is formatted wiht %ld. This works on 64bit but produces a warning on 32bit. Fix the format specifier to %td. Fixes: 67234743 ("libbpf: Generate loader program out of BPF ELF file.") Signed-off-by:
Michal Suchanek <msuchanek@suse.de> Signed-off-by:
-
Jean-Philippe Brucker authored
When the bootstrap and final bpftool have different architectures, we need to build two distinct disasm.o objects. Add a recipe for the bootstrap disasm.o. After commit d510296d ("bpftool: Use syscall/loader program in "prog load" and "gen skeleton" command.") cross-building bpftool didn't work anymore, because the bootstrap bpftool was linked using objects from different architectures: $ make O=/tmp/bpftool ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -C tools/bpf/bpftool/ V=1 [...] aarch64-linux-gnu-gcc ... -c -MMD -o /tmp/bpftool/disasm.o /home/z/src/linux/kernel/bpf/disasm.c gcc ... -c -MMD -o /tmp/bpftool//bootstrap/main.o main.c gcc ... -o /tmp/bpftool//bootstrap/bpftool /tmp/bpftool//bootstrap/main.o ... /tmp/bpftool/disasm.o /usr/bin/ld: /tmp/bpftool/disasm.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: /tmp/bpftool/disasm.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: /tmp/bpftool/disasm.o: Relocations in generic ELF (EM: 183) /usr/bin/ld: /tmp/bpftool/disasm.o: error adding symbols: file in wrong format collect2: error: ld returned 1 exit status [...] The final bpftool was built for e.g. arm64, while the bootstrap bpftool, executed on the host, was built for x86. The problem here was that disasm.o linked into the bootstrap bpftool was arm64 rather than x86. With the fix we build two disasm.o, one for the target bpftool in arm64, and one for the bootstrap bpftool in x86. Fixes: d510296d ("bpftool: Use syscall/loader program in "prog load" and "gen skeleton" command.") Signed-off-by:
Jean-Philippe Brucker <jean-philippe@linaro.org> Signed-off-by:
-
- 03 Jun, 2021 4 commits
-
-
Andrii Nakryiko authored
When xdp_redirect_multi test binary was added recently, it wasn't added to .gitignore. Fix that. Fixes: d2329247 ("selftests/bpf: Add xdp_redirect_multi test") Signed-off-by:
-
Andrii Nakryiko authored
Light skeleton code assumes skel_internal.h header to be installed system-wide by libbpf package. Make sure it is actually installed. Fixes: 67234743 ("libbpf: Generate loader program out of BPF ELF file.") Signed-off-by:
-
Andrii Nakryiko authored
As we gradually get more headers that have to be installed, it's quite annoying to copy/paste long $(call) commands. So extract that logic and do a simple $(foreach) over the list of headers. Signed-off-by:
-
Andrii Nakryiko authored
Official libbpf 0.4 release doesn't include three APIs that were tentatively put into 0.4 section. Fix libbpf.map and move these three APIs: - bpf_map__initial_value; - bpf_map_lookup_and_delete_elem_flags; - bpf_object__gen_loader. Signed-off-by:
-
- 01 Jun, 2021 1 commit
-
-
Harishankar Vishwanathan authored
This patch introduces a new algorithm for multiplication of tristate numbers (tnums) that is provably sound. It is faster and more precise when compared to the existing method. Like the existing method, this new algorithm follows the long multiplication algorithm. The idea is to generate partial products by multiplying each bit in the multiplier (tnum a) with the multiplicand (tnum b), and adding the partial products after appropriately bit-shifting them. The new algorithm, however, uses just a single loop over the bits of the multiplier (tnum a) and accumulates only the uncertain components of the multiplicand (tnum b) into a mask-only tnum. The following paper explains the algorithm in more detail: https://arxiv.org/abs/2105.05398. A natural way to construct the tnum product is by performing a tnum addition on all the partial products. This algorithm presents another method of doing this: decompose each partial product into two tnums, consisting of the values and the masks separately. The mask-sum is accumulated within the loop in acc_m. The value-sum tnum is generated using a.value * b.value. The tnum constructed by tnum addition of the value-sum and the mask-sum contains all possible summations of concrete values drawn from the partial product tnums pairwise. We prove this result in the paper. Our evaluations show that the new algorithm is overall more precise (producing tnums with less uncertain components) than the existing method. As an illustrative example, consider the input tnums A and B. The numbers in the parenthesis correspond to (value;mask). A = 000000x1 (1;2) B = 0010011x (38;1) A * B (existing) = xxxxxxxx (0;255) A * B (new) = 0x1xxxxx (32;95) Importantly, we present a proof of soundness of the new algorithm in the aforementioned paper. Additionally, we show that this new algorithm is empirically faster than the existing method. Co-developed-by:
Matan Shachnai <m.shachnai@rutgers.edu> Co-developed-by:
Srinivas Narayana <srinivas.narayana@rutgers.edu> Co-developed-by:
Santosh Nagarakatte <santosh.nagarakatte@rutgers.edu> Signed-off-by:
Harishankar Vishwanathan <harishankar.vishwanathan@rutgers.edu> Signed-off-by:
Edward Cree <ecree.xilinx@gmail.com> Link: https://arxiv.org/abs/2105.05398 Link: https://lore.kernel.org/bpf/20210531020157.7386-1-harishankar.vishwanathan@rutgers.edu
-
- 28 May, 2021 2 commits
-
-
Hangbin Liu authored
As Colin pointed out, the first drops assignment after declaration will be overwritten by the second drops assignment before using, which makes it useless. Since the drops variable will be used only once. Just remove it and use "cnt - sent" in trace_xdp_devmap_xmit(). Fixes: cb261b59 ("bpf: Run devmap xdp_prog on flush instead of bulk enqueue") Reported-by:
Colin Ian King <colin.king@canonical.com> Signed-off-by:
Hangbin Liu <liuhangbin@gmail.com> Signed-off-by:
John Fastabend <john.fastabend@gmail.com> Link: https://lore.kernel.org/bpf/20210528024356.24333-1-liuhangbin@gmail.com
-
Yonghong Song authored
LLVM upstream commit https://reviews.llvm.org/D102712 made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced. Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools. Signed-off-by:
-
- 26 May, 2021 7 commits
-
-
Florent Revest authored
These macros are convenient wrappers around the bpf_seq_printf and bpf_snprintf helpers. They are currently provided by bpf_tracing.h which targets low level tracing primitives. bpf_helpers.h is a better fit. The __bpf_narg and __bpf_apply are needed in both files and provided twice. __bpf_empty isn't used anywhere and is removed from bpf_tracing.h Reported-by:
Florent Revest <revest@chromium.org> Signed-off-by:
-
Daniel Borkmann authored
Hangbin Liu says: ==================== This patchset is a new implementation for XDP multicast support based on my previous 2 maps implementation[1]. The reason is that Daniel thinks the exclude map implementation is missing proper bond support in XDP context. And there is a plan to add native XDP bonding support. Adding a exclude map in the helper also increases the complexity of verifier and has drawbacks on performance. The new implementation just add two new flags BPF_F_BROADCAST and BPF_F_EXCLUDE_INGRESS to extend xdp_redirect_map for broadcast support. With BPF_F_BROADCAST the packet will be broadcasted to all the interfaces in the map. with BPF_F_EXCLUDE_INGRESS the ingress interface will be excluded when do broadcasting. The patchv11 link is here [2]. [1] https://lore.kernel.org/bpf/20210223125809.1376577-1-liuhangbin@gmail.com [2] https://lore.kernel.org/bpf/20210513070447.1878448-1-liuhangbin@gmail.com v12: As Daniel pointed out: a) defined as const u64 for flag_mask and action_mask in __bpf_xdp_redirect_map() b) remove BPF_F_ACTION_MASK in uapi header c) remove EXPORT_SYMBOL_GPL for xdpf_clone() v11: a) Use unlikely() when checking if this is for broadcast redirecting. b) Fix a tracepoint NULL pointer issue Jesper found c) Remove BPF_F_REDIR_MASK and just use OR flags to make the reader more clear about what's flags we are using d) Add the performace number with multi veth interfaces in patch 01 description. e) remove some sleeps to reduce the testing time in patch04. Re-struct the test and make clear what flags we are testing. v10: use READ/WRITE_ONCE when read/write map instead of xchg() v9: Update patch 01 commit description v8: use hlist_for_each_entry_rcu() when looping the devmap hash ojbs v7: No need to free xdpf in dev_map_enqueue_clone() if xdpf_clone failed. v6: Fix a skb leak in the error path for generic XDP v5: Just walk the map directly to get interfaces as get_next_key() of devmap hash may restart looping from the first key if the device get removed. After update the performace has improved 10% compired with v4. v4: Fix flags never cleared issue in patch 02. Update selftest to cover this. v3: Rebase the code based on latest bpf-next v2: fix flag renaming issue in patch 02 ==================== Signed-off-by:
-
Hangbin Liu authored
Add a bpf selftest for new helper xdp_redirect_map_multi(). In this test there are 3 forward groups and 1 exclude group. The test will redirect each interface's packets to all the interfaces in the forward group, and exclude the interface in exclude map. Two maps (DEVMAP, DEVMAP_HASH) and two xdp modes (generic, drive) will be tested. XDP egress program will also be tested by setting pkt src MAC to egress interface's MAC address. For more test details, you can find it in the test script. Here is the test result. ]# time ./test_xdp_redirect_multi.sh Pass: xdpgeneric arp(F_BROADCAST) ns1-1 Pass: xdpgeneric arp(F_BROADCAST) ns1-2 Pass: xdpgeneric arp(F_BROADCAST) ns1-3 Pass: xdpgeneric IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-1 Pass: xdpgeneric IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-2 Pass: xdpgeneric IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-3 Pass: xdpgeneric IPv6 (no flags) ns1-1 Pass: xdpgeneric IPv6 (no flags) ns1-2 Pass: xdpdrv arp(F_BROADCAST) ns1-1 Pass: xdpdrv arp(F_BROADCAST) ns1-2 Pass: xdpdrv arp(F_BROADCAST) ns1-3 Pass: xdpdrv IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-1 Pass: xdpdrv IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-2 Pass: xdpdrv IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-3 Pass: xdpdrv IPv6 (no flags) ns1-1 Pass: xdpdrv IPv6 (no flags) ns1-2 Pass: xdpegress mac ns1-2 Pass: xdpegress mac ns1-3 Summary: PASS 18, FAIL 0 real 1m18.321s user 0m0.123s sys 0m0.350s Signed-off-by:
Toke Høiland-Jørgensen <toke@redhat.com> Link: https://lore.kernel.org/bpf/20210519090747.1655268-5-liuhangbin@gmail.com
-
Hangbin Liu authored
This is a sample for xdp redirect broadcast. In the sample we could forward all packets between given interfaces. There is also an option -X that could enable 2nd xdp_prog on egress interface. Signed-off-by:
-
Hangbin Liu authored
This patch adds two flags BPF_F_BROADCAST and BPF_F_EXCLUDE_INGRESS to extend xdp_redirect_map for broadcast support. With BPF_F_BROADCAST the packet will be broadcasted to all the interfaces in the map. with BPF_F_EXCLUDE_INGRESS the ingress interface will be excluded when do broadcasting. When getting the devices in dev hash map via dev_map_hash_get_next_key(), there is a possibility that we fall back to the first key when a device was removed. This will duplicate packets on some interfaces. So just walk the whole buckets to avoid this issue. For dev array map, we also walk the whole map to find valid interfaces. Function bpf_clear_redirect_map() was removed in commit ee75aef2 ("bpf, xdp: Restructure redirect actions"). Add it back as we need to use ri->map again. With test topology: +-------------------+ +-------------------+ | Host A (i40e 10G) | ---------- | eno1(i40e 10G) | +-------------------+ | | | Host B | +-------------------+ | | | Host C (i40e 10G) | ---------- | eno2(i40e 10G) | +-------------------+ | | | +------+ | | veth0 -- | Peer | | | veth1 -- | | | | veth2 -- | NS | | | +------+ | +-------------------+ On Host A: # pktgen/pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -s 64 On Host B(Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz, 128G Memory): Use xdp_redirect_map and xdp_redirect_map_multi in samples/bpf for testing. All the veth peers in the NS have a XDP_DROP program loaded. The forward_map max_entries in xdp_redirect_map_multi is modify to 4. Testing the performance impact on the regular xdp_redirect path with and without patch (to check impact of additional check for broadcast mode): 5.12 rc4 | redirect_map i40e->i40e | 2.0M | 9.7M 5.12 rc4 | redirect_map i40e->veth | 1.7M | 11.8M 5.12 rc4 + patch | redirect_map i40e->i40e | 2.0M | 9.6M 5.12 rc4 + patch | redirect_map i40e->veth | 1.7M | 11.7M Testing the performance when cloning packets with the redirect_map_multi test, using a redirect map size of 4, filled with 1-3 devices: 5.12 rc4 + patch | redirect_map multi i40e->veth (x1) | 1.7M | 11.4M 5.12 rc4 + patch | redirect_map multi i40e->veth (x2) | 1.1M | 4.3M 5.12 rc4 + patch | redirect_map multi i40e->veth (x3) | 0.8M | 2.6M Signed-off-by:
-
Jesper Dangaard Brouer authored
This changes the devmap XDP program support to run the program when the bulk queue is flushed instead of before the frame is enqueued. This has a couple of benefits: - It "sorts" the packets by destination devmap entry, and then runs the same BPF program on all the packets in sequence. This ensures that we keep the XDP program and destination device properties hot in I-cache. - It makes the multicast implementation simpler because it can just enqueue packets using bq_enqueue() without having to deal with the devmap program at all. The drawback is that if the devmap program drops the packet, the enqueue step is redundant. However, arguably this is mostly visible in a micro-benchmark, and with more mixed traffic the I-cache benefit should win out. The performance impact of just this patch is as follows: Using 2 10Gb i40e NIC, redirecting one to another, or into a veth interface, which do XDP_DROP on veth peer. With xdp_redirect_map in sample/bpf, send pkts via pktgen cmd: ./pktgen_sample03_burst_single_flow.sh -i eno1 -d $dst_ip -m $dst_mac -t 10 -s 64 There are about +/- 0.1M deviation for native testing, the performance improved for the base-case, but some drop back with xdp devmap prog attached. Version | Test | Generic | Native | Native + 2nd xdp_prog 5.12 rc4 | xdp_redirect_map i40e->i40e | 1.9M | 9.6M | 8.4M 5.12 rc4 | xdp_redirect_map i40e->veth | 1.7M | 11.7M | 9.8M 5.12 rc4 + patch | xdp_redirect_map i40e->i40e | 1.9M | 9.8M | 8.0M 5.12 rc4 + patch | xdp_redirect_map i40e->veth | 1.7M | 12.0M | 9.4M When bq_xmit_all() is called from bq_enqueue(), another packet will always be enqueued immediately after, so clearing dev_rx, xdp_prog and flush_node in bq_xmit_all() is redundant. Move the clear to __dev_flush(), and only check them once in bq_enqueue() since they are all modified together. This change also has the side effect of extending the lifetime of the RCU-protected xdp_prog that lives inside the devmap entries: Instead of just living for the duration of the XDP program invocation, the reference now lives all the way until the bq is flushed. This is safe because the bq flush happens at the end of the NAPI poll loop, so everything happens between a local_bh_disable()/local_bh_enable() pair. However, this is by no means obvious from looking at the call sites; in particular, some drivers have an additional rcu_read_lock() around only the XDP program invocation, which only confuses matters further. Cleaning this up will be done in a separate patch series. Signed-off-by:
-
Alexei Starovoitov authored
Andrii Nakryiko says: ==================== Implement error reporting changes discussed in "Libbpf: the road to v1.0" ([0]) document. Libbpf gets a new API, libbpf_set_strict_mode() which accepts a set of flags that turn on a set of libbpf 1.0 changes, that might be potentially breaking. It's possible to opt-in into all current and future 1.0 features by specifying LIBBPF_STRICT_ALL flag. When some of the 1.0 "features" are requested, libbpf APIs might behave differently. In this patch set a first set of changes are implemented, all related to the way libbpf returns errors. See individual patches for details. Patch #1 adds a no-op libbpf_set_strict_mode() functionality to enable updating selftests. Patch #2 gets rid of all the bad code patterns that will break in libbpf 1.0 (exact -1 comparison for low-level APIs, direct IS_ERR() macro usage to check pointer-returning APIs for error, etc). These changes make selftest work in both legacy and 1.0 libbpf modes. Selftests also opt-in into 100% libbpf 1.0 mode to automatically gain all the subsequent changes, which will come in follow up patches. Patch #3 streamlines error reporting for low-level APIs wrapping bpf() syscall. Patch #4 streamlines errors for all the rest APIs. Patch #5 ensures that BPF skeletons propagate errors properly as well, as currently on error some APIs will return NULL with no way of checking exact error code. [0] https://docs.google.com/document/d/1UyjTZuPFWiPFyKk1tV5an11_iaRuec6U-ZESZ54nNTY v1->v2: - move libbpf_set_strict_mode() implementation to patch #1, where it belongs (Alexei); - add acks, slight rewording of commit messages. ==================== Signed-off-by:
-
