- 30 Jul, 2024 1 commit
-
-
Zhu Jun authored
The format specifier of "unsigned int" in printf() should be "%u", not "%d". Signed-off-by:
Zhu Jun <zhujun2@cmss.chinamobile.com> Signed-off-by:
Andrii Nakryiko <andrii@kernel.org> Acked-by:
Quentin Monnet <qmo@kernel.org> Link: https://lore.kernel.org/bpf/20240724111120.11625-1-zhujun2@cmss.chinamobile.com
-
- 29 Jul, 2024 39 commits
-
-
David Vernet authored
In prog_tests/test_global_maps_resize.c, we test various use cases for resizing global maps. Commit 7244100e0389 ("libbpf: Don't take direct pointers into BTF data from st_ops") updated libbpf to not store pointers to volatile BTF data, which for some users, was causing a UAF when resizing a datasec array. Let's ensure we have coverage for resizing datasec arrays with struct_ops progs by also including a struct_ops map and struct_ops prog in the test_global_map_resize skeleton. The map is automatically loaded, so we don't need to do anything other than add it to the BPF prog being tested to get the coverage. Signed-off-by:David Vernet <void@manifault.com> Signed-off-by:
-
Alexis Lothoré (eBPF Foundation) authored
test_xdp_veth.sh tests that XDP return codes work as expected, by bringing up multiple veth pairs isolated in different namespaces, attaching specific xdp programs to each interface, and ensuring that the whole chain allows to ping one end interface from the first one. The test runs well but is currently not integrated in test_progs, which prevents it from being run automatically in the CI infrastructure. Rewrite it as a C test relying on libbpf to allow running it in the CI infrastructure. The new code brings up the same network infrastructure and reuses the same eBPF programs as test_xdp_veth.sh, for which skeletons are already generated by the bpf tests makefile. Signed-off-by:
Alexis Lothoré (eBPF Foundation) <alexis.lothore@bootlin.com> Signed-off-by:
Daniel Borkmann <daniel@iogearbox.net> Acked-by:
Stanislav Fomichev <sdf@fomichev.me> Link: https://lore.kernel.org/bpf/20240721-convert_test_xdp_veth-v4-2-23bdba21b2f9@bootlin.comSigned-off-by:
-
Alexis Lothoré (eBPF Foundation) authored
xdp_redirect_map.c is a bpf program used by test_xdp_veth.sh, which is not handled by the generic test runner (test_progs). To allow converting this test to test_progs, the corresponding program must be updated to allow handling it through skeletons generated by bpftool and libbpf. Update programs section names to allow to manipulate those with libbpf. Signed-off-by:
-
David Vernet authored
In struct bpf_struct_ops, we have take a pointer to a BTF type name, and a struct btf_type. This was presumably done for convenience, but can actually result in subtle and confusing bugs given that BTF data can be invalidated before a program is loaded. For example, in sched_ext, we may sometimes resize a data section after a skeleton has been opened, but before the struct_ops scheduler map has been loaded. This may cause the BTF data to be realloc'd, which can then cause a UAF when loading the program because the struct_ops map has pointers directly into the BTF data. We're already storing the BTF type_id in struct bpf_struct_ops. Because type_id is stable, we can therefore just update the places where we were looking at those pointers to instead do the lookups we need from the type_id. Fixes: 590a0088 ("bpf: libbpf: Add STRUCT_OPS support") Signed-off-by:
-
Andrii Nakryiko authored
Tony Ambardar says: ==================== selftests/bpf: Improve libc portability / musl support (part 1) Hello all, This series includes the bulk of libc-related compile fixes accumulated to support systems using musl, with smaller numbers to follow. These patches are simple and straightforward, and the series has been tested with the kernel-patches/bpf CI and locally using mips64el-gcc/musl-libc and QEMU with an OpenWrt rootfs. The patches address a few general categories of libc portability issues: - missing, redundant or incorrect include headers - disabled GNU header extensions (i.e. missing #define _GNU_SOURCE) - issues with types and casting Feedback and suggestions for improvement are welcome! Thanks, Tony ==================== Link: https://lore.kernel.org/r/cover.1721713597.git.tony.ambardar@gmail.comSigned-off-by:
-
Tony Ambardar authored
Remove a redundant include of '<asm/types.h>', whose needed definitions are already included (via '<linux/types.h>') in cg_storage_multi_egress_only.c, cg_storage_multi_isolated.c, and cg_storage_multi_shared.c. This avoids redefinition errors seen compiling for mips64el/musl-libc like: In file included from progs/cg_storage_multi_egress_only.c:13: In file included from progs/cg_storage_multi.h:6: In file included from /usr/mips64el-linux-gnuabi64/include/asm/types.h:23: /usr/include/asm-generic/int-l64.h:29:25: error: typedef redefinition with different types ('long' vs 'long long') 29 | typedef __signed__ long __s64; | ^ /usr/include/asm-generic/int-ll64.h:30:44: note: previous definition is here 30 | __extension__ typedef __signed__ long long __s64; | ^ Fixes: 9e5bd1f7 ("selftests/bpf: Test CGROUP_STORAGE map can't be used by multiple progs") Signed-off-by:Tony Ambardar <tony.ambardar@gmail.com> Signed-off-by:
-
Tony Ambardar authored
Remove a redundant include of '<linux/in6.h>', whose needed definitions are already provided by 'test_progs.h'. This avoids errors seen compiling for mips64el/musl-libc: In file included from .../arpa/inet.h:9, from ./test_progs.h:17, from prog_tests/crypto_sanity.c:10: .../netinet/in.h:23:8: error: redefinition of 'struct in6_addr' 23 | struct in6_addr { | ^~~~~~~~ In file included from crypto_sanity.c:7: .../linux/in6.h:33:8: note: originally defined here 33 | struct in6_addr { | ^~~~~~~~ .../netinet/in.h:34:8: error: redefinition of 'struct sockaddr_in6' 34 | struct sockaddr_in6 { | ^~~~~~~~~~~~ .../linux/in6.h:50:8: note: originally defined here 50 | struct sockaddr_in6 { | ^~~~~~~~~~~~ .../netinet/in.h:42:8: error: redefinition of 'struct ipv6_mreq' 42 | struct ipv6_mreq { | ^~~~~~~~~ .../linux/in6.h:60:8: note: originally defined here 60 | struct ipv6_mreq { | ^~~~~~~~~ Fixes: 91541ab1 ("selftests: bpf: crypto skcipher algo selftests") Signed-off-by:Vadim Fedorenko <vadim.fedorenko@linux.dev> Link: https://lore.kernel.org/bpf/911293968f424ad7b462d8805aeb3baee8f4985b.1721713597.git.tony.ambardar@gmail.com
-
Tony Ambardar authored
Remove a redundant include of '<linux/in6.h>', whose needed definitions are already provided by 'test_progs.h'. This avoids errors seen compiling for mips64el/musl-libc: In file included from .../arpa/inet.h:9, from ./test_progs.h:17, from prog_tests/decap_sanity.c:9: .../netinet/in.h:23:8: error: redefinition of 'struct in6_addr' 23 | struct in6_addr { | ^~~~~~~~ In file included from decap_sanity.c:7: .../linux/in6.h:33:8: note: originally defined here 33 | struct in6_addr { | ^~~~~~~~ .../netinet/in.h:34:8: error: redefinition of 'struct sockaddr_in6' 34 | struct sockaddr_in6 { | ^~~~~~~~~~~~ .../linux/in6.h:50:8: note: originally defined here 50 | struct sockaddr_in6 { | ^~~~~~~~~~~~ .../netinet/in.h:42:8: error: redefinition of 'struct ipv6_mreq' 42 | struct ipv6_mreq { | ^~~~~~~~~ .../linux/in6.h:60:8: note: originally defined here 60 | struct ipv6_mreq { | ^~~~~~~~~ Fixes: 70a00e2f ("selftests/bpf: Test bpf_skb_adjust_room on CHECKSUM_PARTIAL") Signed-off-by: -
Tony Ambardar authored
Remove a redundant include of '<linux/icmp.h>' which is already provided in 'lwt_helpers.h'. This avoids errors seen compiling for mips64el/musl-libc: In file included from .../arpa/inet.h:9, from lwt_redirect.c:51: .../netinet/in.h:23:8: error: redefinition of 'struct in6_addr' 23 | struct in6_addr { | ^~~~~~~~ In file included from .../linux/icmp.h:24, from lwt_redirect.c:50: .../linux/in6.h:33:8: note: originally defined here 33 | struct in6_addr { | ^~~~~~~~ .../netinet/in.h:34:8: error: redefinition of 'struct sockaddr_in6' 34 | struct sockaddr_in6 { | ^~~~~~~~~~~~ .../linux/in6.h:50:8: note: originally defined here 50 | struct sockaddr_in6 { | ^~~~~~~~~~~~ .../netinet/in.h:42:8: error: redefinition of 'struct ipv6_mreq' 42 | struct ipv6_mreq { | ^~~~~~~~~ .../linux/in6.h:60:8: note: originally defined here 60 | struct ipv6_mreq { | ^~~~~~~~~ Fixes: 43a7c3ef ("selftests/bpf: Add lwt_xmit tests for BPF_REDIRECT") Signed-off-by: -
Tony Ambardar authored
The type 'loff_t' is a GNU extension and not exposed by the musl 'fcntl.h' header unless _GNU_SOURCE is defined. Add this definition to fix errors seen compiling for mips64el/musl-libc: In file included from tools/testing/selftests/bpf/prog_tests/core_reloc.c:4: ./bpf_testmod/bpf_testmod.h:10:9: error: unknown type name 'loff_t' 10 | loff_t off; | ^~~~~~ ./bpf_testmod/bpf_testmod.h:16:9: error: unknown type name 'loff_t' 16 | loff_t off; | ^~~~~~ Fixes: 6bcd39d3 ("selftests/bpf: Add CO-RE relocs selftest relying on kernel module BTF") Signed-off-by: -
Tony Ambardar authored
The GNU version of 'struct tcp_info' in 'netinet/tcp.h' is not exposed by musl headers unless _GNU_SOURCE is defined. Add this definition to fix errors seen compiling for mips64el/musl-libc: tcp_rtt.c: In function 'wait_for_ack': tcp_rtt.c:24:25: error: storage size of 'info' isn't known 24 | struct tcp_info info; | ^~~~ tcp_rtt.c:24:25: error: unused variable 'info' [-Werror=unused-variable] cc1: all warnings being treated as errors Fixes: 1f4f80fe ("selftests/bpf: test_progs: convert test_tcp_rtt") Signed-off-by: -
Tony Ambardar authored
The GNU version of 'struct tcphdr' has members 'doff', 'source' and 'dest', which are not exposed by musl libc headers unless _GNU_SOURCE is defined. Add this definition to fix errors seen compiling for mips64el/musl-libc: flow_dissector.c:118:30: error: 'struct tcphdr' has no member named 'doff' 118 | .tcp.doff = 5, | ^~~~ flow_dissector.c:119:30: error: 'struct tcphdr' has no member named 'source' 119 | .tcp.source = 80, | ^~~~~~ flow_dissector.c:120:30: error: 'struct tcphdr' has no member named 'dest' 120 | .tcp.dest = 8080, | ^~~~ Fixes: ae173a91 ("selftests/bpf: support BPF_FLOW_DISSECTOR_F_PARSE_1ST_FRAG") Signed-off-by: -
Tony Ambardar authored
The GNU version of 'struct tcphdr' with member 'doff' is not exposed by musl headers unless _GNU_SOURCE is defined. Add this definition to fix errors seen compiling for mips64el/musl-libc: In file included from kfree_skb.c:2: kfree_skb.c: In function 'on_sample': kfree_skb.c:45:30: error: 'struct tcphdr' has no member named 'doff' 45 | if (CHECK(pkt_v6->tcp.doff != 5, "check_tcp", | ^ Fixes: 580d656d ("selftests/bpf: Add kfree_skb raw_tp test") Signed-off-by: -
Tony Ambardar authored
The GNU version of 'struct tcphdr', with members 'doff' and 'urg_ptr', is not exposed by musl headers unless _GNU_SOURCE is defined. Add this definition to fix errors seen compiling for mips64el/musl-libc: parse_tcp_hdr_opt.c:18:21: error: 'struct tcphdr' has no member named 'urg_ptr' 18 | .pk6_v6.tcp.urg_ptr = 123, | ^~~~~~~ parse_tcp_hdr_opt.c:19:21: error: 'struct tcphdr' has no member named 'doff' 19 | .pk6_v6.tcp.doff = 9, /* 16 bytes of options */ | ^~~~ Fixes: cfa7b011 ("selftests/bpf: tests for using dynptrs to parse skb and xdp buffers") Signed-off-by: -
Tony Ambardar authored
Update ns_current_pid_tgid.c to use '#include <fcntl.h>' and avoid compile error against mips64el/musl libc: In file included from .../prog_tests/ns_current_pid_tgid.c:14: .../include/sys/fcntl.h:1:2: error: #warning redirecting incorrect #include <sys/fcntl.h> to <fcntl.h> [-Werror=cpp] 1 | #warning redirecting incorrect #include <sys/fcntl.h> to <fcntl.h> | ^~~~~~~ cc1: all warnings being treated as errors Fixes: 09c02d55 ("bpf, selftests: Fold test_current_pid_tgid_new_ns into test_progs.") Signed-off-by: -
Tony Ambardar authored
Explicitly include '<linux/build_bug.h>' to fix errors seen compiling with gcc targeting mips64el/musl-libc: user_ringbuf.c: In function 'test_user_ringbuf_loop': user_ringbuf.c:426:9: error: implicit declaration of function 'BUILD_BUG_ON' [-Werror=implicit-function-declaration] 426 | BUILD_BUG_ON(total_samples <= c_max_entries); | ^~~~~~~~~~~~ cc1: all warnings being treated as errors Fixes: e5a9df51 ("selftests/bpf: Add selftests validating the user ringbuf") Signed-off-by: -
Tony Ambardar authored
Include <limits.h> in 'bench.h' to provide a UINT_MAX definition and avoid multiple compile errors against mips64el/musl-libc like: benchs/bench_local_storage.c: In function 'parse_arg': benchs/bench_local_storage.c:40:38: error: 'UINT_MAX' undeclared (first use in this function) 40 | if (ret < 1 || ret > UINT_MAX) { | ^~~~~~~~ benchs/bench_local_storage.c:11:1: note: 'UINT_MAX' is defined in header '<limits.h>'; did you forget to '#include <limits.h>'? 10 | #include <test_btf.h> +++ |+#include <limits.h> 11 | seen with bench_local_storage.c, bench_local_storage_rcu_tasks_trace.c, and bench_bpf_hashmap_lookup.c. Fixes: 73087489 ("selftests/bpf: Add benchmark for local_storage get") Signed-off-by: -
Tony Ambardar authored
Add a "bpf_util.h" include to avoid the following error seen compiling for mips64el with musl libc: bench.c: In function 'find_benchmark': bench.c:590:25: error: implicit declaration of function 'ARRAY_SIZE' [-Werror=implicit-function-declaration] 590 | for (i = 0; i < ARRAY_SIZE(benchs); i++) { | ^~~~~~~~~~ cc1: all warnings being treated as errors Fixes: 8e7c2a02 ("selftests/bpf: Add benchmark runner infrastructure") Signed-off-by: -
Tony Ambardar authored
The addition of general support for unprivileged tests in test_loader.c breaks building test_verifier on non-glibc (e.g. musl) systems, due to the inclusion of glibc extension '<error.h>' in 'unpriv_helpers.c'. However, the header is actually not needed, so remove it to restore building. Similarly for sk_lookup.c and flow_dissector.c, error.h is not necessary and causes problems, so drop them. Fixes: 1d56ade0 ("selftests/bpf: Unprivileged tests for test_loader.c") Fixes: 0ab5539f ("selftests/bpf: Tests for BPF_SK_LOOKUP attach point") Fixes: 0905beec ("selftests/bpf: run flow dissector tests in skb-less mode") Signed-off-by:
-
Tony Ambardar authored
Existing code calls getsockname() with a 'struct sockaddr_in6 *' argument where a 'struct sockaddr *' argument is declared, yielding compile errors when building for mips64el/musl-libc: bpf_iter_setsockopt.c: In function 'get_local_port': bpf_iter_setsockopt.c:98:30: error: passing argument 2 of 'getsockname' from incompatible pointer type [-Werror=incompatible-pointer-types] 98 | if (!getsockname(fd, &addr, &addrlen)) | ^~~~~ | | | struct sockaddr_in6 * In file included from .../netinet/in.h:10, from .../arpa/inet.h:9, from ./test_progs.h:17, from bpf_iter_setsockopt.c:5: .../sys/socket.h:391:23: note: expected 'struct sockaddr * restrict' but argument is of type 'struct sockaddr_in6 *' 391 | int getsockname (int, struct sockaddr *__restrict, socklen_t *__restrict); | ^ cc1: all warnings being treated as errors This compiled under glibc only because the argument is declared to be a "funky" transparent union which includes both types above. Explicitly cast the argument to allow compiling for both musl and glibc. Fixes: eed92afd ("bpf: selftest: Test batching and bpf_(get|set)sockopt in bpf tcp iter") Signed-off-by:Geliang Tang <geliang@kernel.org> Link: https://lore.kernel.org/bpf/f41def0f17b27a23b1709080e4e3f37f4cc11ca9.1721713597.git.tony.ambardar@gmail.com
-
Tony Ambardar authored
Cast 'rlim_t' argument to match expected type of printf() format and avoid compile errors seen building for mips64el/musl-libc: In file included from map_tests/sk_storage_map.c:20: map_tests/sk_storage_map.c: In function 'test_sk_storage_map_stress_free': map_tests/sk_storage_map.c:414:56: error: format '%lu' expects argument of type 'long unsigned int', but argument 2 has type 'rlim_t' {aka 'long long unsigned int'} [-Werror=format=] 414 | CHECK(err, "setrlimit(RLIMIT_NOFILE)", "rlim_new:%lu errno:%d", | ^~~~~~~~~~~~~~~~~~~~~~~ 415 | rlim_new.rlim_cur, errno); | ~~~~~~~~~~~~~~~~~ | | | rlim_t {aka long long unsigned int} ./test_maps.h:12:24: note: in definition of macro 'CHECK' 12 | printf(format); \ | ^~~~~~ map_tests/sk_storage_map.c:414:68: note: format string is defined here 414 | CHECK(err, "setrlimit(RLIMIT_NOFILE)", "rlim_new:%lu errno:%d", | ~~^ | | | long unsigned int | %llu cc1: all warnings being treated as errors Fixes: 51a0e301 ("bpf: Add BPF_MAP_TYPE_SK_STORAGE test to test_maps") Signed-off-by: -
Tony Ambardar authored
Use pid_t rather than __pid_t when allocating memory for 'worker_pids' in 'struct test_env', as this is its declared type and also avoids compile errors seen building against musl libc on mipsel64: test_progs.c:1738:49: error: '__pid_t' undeclared (first use in this function); did you mean 'pid_t'? 1738 | env.worker_pids = calloc(sizeof(__pid_t), env.workers); | ^~~~~~~ | pid_t test_progs.c:1738:49: note: each undeclared identifier is reported only once for each function it appears in Fixes: 91b2c0af ("selftests/bpf: Add parallelism to test_progs") Signed-off-by: -
Ihor Solodrai authored
%.test.o should depend on %.test.d order-only to avoid unnecessary recompilations due to compiler dumping .d and .o files in random order. Signed-off-by:
Ihor Solodrai <ihor.solodrai@pm.me> Signed-off-by:
-
Alexei Starovoitov authored
Eduard Zingerman says: ==================== no_caller_saved_registers attribute for helper calls This patch-set seeks to allow using no_caller_saved_registers gcc/clang attribute with some BPF helper functions (and kfuncs in the future). As documented in [1], this attribute means that function scratches only some of the caller saved registers defined by ABI. For BPF the set of such registers could be defined as follows: - R0 is scratched only if function is non-void; - R1-R5 are scratched only if corresponding parameter type is defined in the function prototype. The goal of the patch-set is to implement no_caller_saved_registers (nocsr for short) in a backwards compatible manner: - for kernels that support the feature, gain some performance boost from better register allocation; - for kernels that don't support the feature, allow programs execution with minor performance losses. To achieve this, use a scheme suggested by Alexei Starovoitov: - for nocsr calls clang allocates registers as-if relevant r0-r5 registers are not scratched by the call; - as a post-processing step, clang visits each nocsr call and adds spill/fill for every live r0-r5; - stack offsets used for spills/fills are allocated as lowest stack offsets in whole function and are not used for any other purpose; - when kernel loads a program, it looks for such patterns (nocsr function surrounded by spills/fills) and checks if spill/fill stack offsets are used exclusively in nocsr patterns; - if so, and if current JIT inlines the call to the nocsr function (e.g. a helper call), kernel removes unnecessary spill/fill pairs; - when old kernel loads a program, presence of spill/fill pairs keeps BPF program valid, albeit slightly less efficient. Corresponding clang/llvm changes are available in [2]. The patch-set uses bpf_get_smp_processor_id() function as a canary, making it the first helper with nocsr attribute. For example, consider the following program: #define __no_csr __attribute__((no_caller_saved_registers)) #define SEC(name) __attribute__((section(name), used)) #define bpf_printk(fmt, ...) bpf_trace_printk((fmt), sizeof(fmt), __VA_ARGS__) typedef unsigned int __u32; static long (* const bpf_trace_printk)(const char *fmt, __u32 fmt_size, ...) = (void *) 6; static __u32 (*const bpf_get_smp_processor_id)(void) __no_csr = (void *)8; SEC("raw_tp") int test(void *ctx) { __u32 task = bpf_get_smp_processor_id(); bpf_printk("ctx=%p, smp=%d", ctx, task); return 0; } char _license[] SEC("license") = "GPL"; Compiled (using [2]) as follows: $ clang --target=bpf -O2 -g -c -o nocsr.bpf.o nocsr.bpf.c $ llvm-objdump --no-show-raw-insn -Sd nocsr.bpf.o ... 3rd parameter for printk call removable spill/fill pair .--- 0: r3 = r1 | ; | __u32 task = bpf_get_smp_processor_id(); | | 1: *(u64 *)(r10 - 0x8) = r3 <----------| | 2: call 0x8 | | 3: r3 = *(u64 *)(r10 - 0x8) <----------' ; | bpf_printk("ctx=%p, smp=%d", ctx, task); | 4: r1 = 0x0 ll | 6: r2 = 0xf | 7: r4 = r0 '--> 8: call 0x6 ; return 0; 9: r0 = 0x0 10: exit Here is how the program looks after verifier processing: # bpftool prog load ./nocsr.bpf.o /sys/fs/bpf/nocsr-test # bpftool prog dump xlated pinned /sys/fs/bpf/nocsr-test int test(void * ctx): 0: (bf) r3 = r1 <--- 3rd printk parameter ; __u32 task = bpf_get_smp_processor_id(); 1: (b4) w0 = 197324 <--. inlined helper call, 2: (bf) r0 = &(void __percpu *)(r0) <--- spill/fill 3: (61) r0 = *(u32 *)(r0 +0) <--' pair removed ; bpf_printk("ctx=%p, smp=%d", ctx, task); 4: (18) r1 = map[id:5][0]+0 6: (b7) r2 = 15 7: (bf) r4 = r0 8: (85) call bpf_trace_printk#-125920 ; return 0; 9: (b7) r0 = 0 10: (95) exit [1] https://clang.llvm.org/docs/AttributeReference.html#no-caller-saved-registers [2] https://github.com/eddyz87/llvm-project/tree/bpf-no-caller-saved-registers Change list: - v3 -> v4: - When nocsr spills/fills are removed in the subprogram, allow these spills/fills to reside in [-MAX_BPF_STACK-48..MAX_BPF_STACK) range (suggested by Alexei); - Dropped patches with special handling for bpf_probe_read_kernel() (requested by Alexei); - Reset aux .nocsr_pattern and .nocsr_spills_num fields in check_nocsr_stack_contract() (requested by Andrii). Andrii, I have not added an additional flag to struct bpf_subprog_info, it currently does not have holes and I really don't like adding a bool field there just as an alternative indicator that nocsr is disabled. Indicator at the moment: - nocsr_stack_off >= S16_MIN means that nocsr rewrite is enabled; - nocsr_stack_off == S16_MIN means that nocsr rewrite is disabled. - v2 -> v3: - As suggested by Andrii, 'nocsr_stack_off' is no longer checked at rewrite time, instead mark_nocsr_patterns() now does two passes over BPF program: - on a first pass it computes the lowest stack spill offset for the subprogram; - on a second pass this offset is used to recognize nocsr pattern. - As suggested by Alexei, a new mechanic is added to work around a situation mentioned by Andrii, when more helper functions are marked as nocsr at compile time than current kernel supports: - all {spill*,helper call,fill*} patterns are now marked as insn_aux_data[*].nocsr_pattern, thus relaxing failure condition for check_nocsr_stack_contract(); - spill/fill pairs are not removed for patterns where helper can't be inlined; - see mark_nocsr_pattern_for_call() for details an example. - As suggested by Alexei, subprogram stack depth is now adjusted if all spill/fill pairs could be removed. This adjustment has to take place before optimize_bpf_loop(), hence the rewrite is moved from do_misc_fixups() to remove_nocsr_spills_fills() (again). - As suggested by Andrii, special measures are taken to work around bpf_probe_read_kernel() access to BPF stack, see patches 11, 12. Patch #11 is very simplistic, a more comprehensive solution would be to change the type of the third parameter of the bpf_probe_read_kernel() from ARG_ANYTHING to something else and not only check nocsr contract, but also propagate stack slot liveness information. However, such change would require update in struct bpf_call_arg_meta processing, which currently implies that every memory parameter is followed by a size parameter. I can work on these changes, please comment. - Stylistic changes suggested by Andrii. - Added acks from Andrii. - Dropped RFC tag. - v1 -> v2: - assume that functions inlined by either jit or verifier conform to no_caller_saved_registers contract (Andrii, Puranjay); - allow nocsr rewrite for bpf_get_smp_processor_id() on arm64 and riscv64 architectures (Puranjay); - __arch_{x86_64,arm64,riscv64} macro for test_loader; - moved remove_nocsr_spills_fills() inside do_misc_fixups() (Andrii); - moved nocsr pattern detection from check_cfg() to a separate pass (Andrii); - various stylistic/correctness changes according to Andrii's comments. Revisions: - v1 https://lore.kernel.org/bpf/20240629094733.3863850-1-eddyz87@gmail.com/ - v2 https://lore.kernel.org/bpf/20240704102402.1644916-1-eddyz87@gmail.com/ - v3 https://lore.kernel.org/bpf/20240715230201.3901423-1-eddyz87@gmail.com/ ==================== Link: https://lore.kernel.org/r/20240722233844.1406874-1-eddyz87@gmail.comSigned-off-by:Alexei Starovoitov <ast@kernel.org> Signed-off-by:
-
Eduard Zingerman authored
Tests for no_caller_saved_registers processing logic (see verifier.c:match_and_mark_nocsr_pattern()): - a canary positive test case; - a canary test case for arm64 and riscv64; - various tests with broken patterns; - tests with read/write fixed/varying stack access that violate nocsr stack access contract; - tests with multiple subprograms; - tests using nocsr in combination with may_goto/bpf_loop, as all of these features affect stack depth; - tests for nocsr stack spills below max stack depth. Signed-off-by:
Eduard Zingerman <eddyz87@gmail.com> Link: https://lore.kernel.org/r/20240722233844.1406874-11-eddyz87@gmail.comSigned-off-by:
-
Eduard Zingerman authored
Add annotations __arch_x86_64, __arch_arm64, __arch_riscv64 to specify on which architecture the test case should be tested. Several __arch_* annotations could be specified at once. When test case is not run on current arch it is marked as skipped. For example, the following would be tested only on arm64 and riscv64: SEC("raw_tp") __arch_arm64 __arch_riscv64 __xlated("1: *(u64 *)(r10 - 16) = r1") __xlated("2: call") __xlated("3: r1 = *(u64 *)(r10 - 16);") __success __naked void canary_arm64_riscv64(void) { asm volatile ( "r1 = 1;" "*(u64 *)(r10 - 16) = r1;" "call %[bpf_get_smp_processor_id];" "r1 = *(u64 *)(r10 - 16);" "exit;" : : __imm(bpf_get_smp_processor_id) : __clobber_all); } On x86 it would be skipped: #467/2 verifier_nocsr/canary_arm64_riscv64:SKIP Acked-by: -
Eduard Zingerman authored
Add a macro __xlated("...") for use with test_loader tests. When such annotations are present for the test case: - bpf_prog_get_info_by_fd() is used to get BPF program after all rewrites are applied by verifier. - the program is disassembled and patterns specified in __xlated are searched for in the disassembly text. __xlated matching follows the same mechanics as __msg: each subsequent pattern is matched from the point where previous pattern ended. This allows to write tests like below, where the goal is to verify the behavior of one of the of the transformations applied by verifier: SEC("raw_tp") __xlated("1: w0 = ") __xlated("2: r0 = &(void __percpu *)(r0)") __xlated("3: r0 = *(u32 *)(r0 +0)") __xlated("4: exit") __success __naked void simple(void) { asm volatile ( "call %[bpf_get_smp_processor_id];" "exit;" : : __imm(bpf_get_smp_processor_id) : __clobber_all); } Acked-by: -
Eduard Zingerman authored
Non-functional change: use a separate data structure to represented expected messages in test_loader. This would allow to use the same functionality for expected set of disassembled instructions in the follow-up commit. Acked-by:
-
Eduard Zingerman authored
The call stack for validate_case() function looks as follows: - test_loader__run_subtests() - process_subtest() - run_subtest() - prepare_case(), which does 'tester->next_match_pos = 0'; - validate_case(), which increments tester->next_match_pos. Hence, each subtest is run with next_match_pos freshly set to zero. Meaning that there is no need to persist this variable in the struct test_loader, use local variable instead. Acked-by: -
Eduard Zingerman authored
Adjust disasm_helpers.c:disasm_insn() to account for the following part of the verifier.c:jit_subprogs: for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) { /* ... */ if (!bpf_pseudo_call(insn)) continue; insn->off = env->insn_aux_data[i].call_imm; subprog = find_subprog(env, i + insn->off + 1); insn->imm = subprog; } Where verifier moves offset of the subprogram to the insn->off field. Signed-off-by: -
Yonghong Song authored
Add a few reg_bounds selftests to test 32/16/8-bit ldsx and subreg comparison. Without the previous patch, all added tests will fail. Acked-by:
Yonghong Song <yonghong.song@linux.dev> Link: https://lore.kernel.org/r/20240723162940.2732171-1-yonghong.song@linux.devSigned-off-by:
-
Eduard Zingerman authored
struct bpf_insn *disasm_insn(struct bpf_insn *insn, char *buf, size_t buf_sz); Disassembles instruction 'insn' to a text buffer 'buf'. Removes insn->code hex prefix added by kernel disassembly routine. Returns a pointer to the next instruction (increments insn by either 1 or 2). Acked-by:
-
Yonghong Song authored
With latest llvm19, the selftest iters/iter_arr_with_actual_elem_count failed with -mcpu=v4. The following are the details: 0: R1=ctx() R10=fp0 ; int iter_arr_with_actual_elem_count(const void *ctx) @ iters.c:1420 0: (b4) w7 = 0 ; R7_w=0 ; int i, n = loop_data.n, sum = 0; @ iters.c:1422 1: (18) r1 = 0xffffc90000191478 ; R1_w=map_value(map=iters.bss,ks=4,vs=1280,off=1144) 3: (61) r6 = *(u32 *)(r1 +128) ; R1_w=map_value(map=iters.bss,ks=4,vs=1280,off=1144) R6_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) ; if (n > ARRAY_SIZE(loop_data.data)) @ iters.c:1424 4: (26) if w6 > 0x20 goto pc+27 ; R6_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=32,var_off=(0x0; 0x3f)) 5: (bf) r8 = r10 ; R8_w=fp0 R10=fp0 6: (07) r8 += -8 ; R8_w=fp-8 ; bpf_for(i, 0, n) { @ iters.c:1427 7: (bf) r1 = r8 ; R1_w=fp-8 R8_w=fp-8 8: (b4) w2 = 0 ; R2_w=0 9: (bc) w3 = w6 ; R3_w=scalar(id=1,smin=smin32=0,smax=umax=smax32=umax32=32,var_off=(0x0; 0x3f)) R6_w=scalar(id=1,smin=smin32=0,smax=umax=smax32=umax32=32,var_off=(0x0; 0x3f)) 10: (85) call bpf_iter_num_new#45179 ; R0=scalar() fp-8=iter_num(ref_id=2,state=active,depth=0) refs=2 11: (bf) r1 = r8 ; R1=fp-8 R8=fp-8 refs=2 12: (85) call bpf_iter_num_next#45181 13: R0=rdonly_mem(id=3,ref_obj_id=2,sz=4) R6=scalar(id=1,smin=smin32=0,smax=umax=smax32=umax32=32,var_off=(0x0; 0x3f)) R7=0 R8=fp-8 R10=fp0 fp-8=iter_num(ref_id=2,state=active,depth=1) refs=2 ; bpf_for(i, 0, n) { @ iters.c:1427 13: (15) if r0 == 0x0 goto pc+2 ; R0=rdonly_mem(id=3,ref_obj_id=2,sz=4) refs=2 14: (81) r1 = *(s32 *)(r0 +0) ; R0=rdonly_mem(id=3,ref_obj_id=2,sz=4) R1_w=scalar(smin=0xffffffff80000000,smax=0x7fffffff) refs=2 15: (ae) if w1 < w6 goto pc+4 20: R0=rdonly_mem(id=3,ref_obj_id=2,sz=4) R1=scalar(smin=0xffffffff80000000,smax=smax32=umax32=31,umax=0xffffffff0000001f,smin32=0,var_off=(0x0; 0xffffffff0000001f)) R6=scalar(id=1,smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=32,var_off=(0x0; 0x3f)) R7=0 R8=fp-8 R10=fp0 fp-8=iter_num(ref_id=2,state=active,depth=1) refs=2 ; sum += loop_data.data[i]; @ iters.c:1429 20: (67) r1 <<= 2 ; R1_w=scalar(smax=0x7ffffffc0000007c,umax=0xfffffffc0000007c,smin32=0,smax32=umax32=124,var_off=(0x0; 0xfffffffc0000007c)) refs=2 21: (18) r2 = 0xffffc90000191478 ; R2_w=map_value(map=iters.bss,ks=4,vs=1280,off=1144) refs=2 23: (0f) r2 += r1 math between map_value pointer and register with unbounded min value is not allowed The source code: int iter_arr_with_actual_elem_count(const void *ctx) { int i, n = loop_data.n, sum = 0; if (n > ARRAY_SIZE(loop_data.data)) return 0; bpf_for(i, 0, n) { /* no rechecking of i against ARRAY_SIZE(loop_data.n) */ sum += loop_data.data[i]; } return sum; } The insn #14 is a sign-extenstion load which is related to 'int i'. The insn #15 did a subreg comparision. Note that smin=0xffffffff80000000 and this caused later insn #23 failed verification due to unbounded min value. Actually insn #15 R1 smin range can be better. Before insn #15, we have R1_w=scalar(smin=0xffffffff80000000,smax=0x7fffffff) With the above range, we know for R1, upper 32bit can only be 0xffffffff or 0. Otherwise, the value range for R1 could be beyond [smin=0xffffffff80000000,smax=0x7fffffff]. After insn #15, for the true patch, we know smin32=0 and smax32=32. With the upper 32bit 0xffffffff, then the corresponding value is [0xffffffff00000000, 0xffffffff00000020]. The range is obviously beyond the original range [smin=0xffffffff80000000,smax=0x7fffffff] and the range is not possible. So the upper 32bit must be 0, which implies smin = smin32 and smax = smax32. This patch fixed the issue by adding additional register deduction after 32-bit compare insn. If the signed 32-bit register range is non-negative then 64-bit smin is in range of [S32_MIN, S32_MAX], then the actual 64-bit smin/smax should be the same as 32-bit smin32/smax32. With this patch, iters/iter_arr_with_actual_elem_count succeeded with better register range: from 15 to 20: R0=rdonly_mem(id=7,ref_obj_id=2,sz=4) R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=31,var_off=(0x0; 0x1f)) R6=scalar(id=1,smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=32,var_off=(0x0; 0x3f)) R7=scalar(id=9,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R8=scalar(id=9,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-8=iter_num(ref_id=2,state=active,depth=3) refs=2 Acked-by:Shung-Hsi Yu <shung-hsi.yu@suse.com> Signed-off-by:
-
Eduard Zingerman authored
The function bpf_get_smp_processor_id() is processed in a different way, depending on the arch: - on x86 verifier replaces call to bpf_get_smp_processor_id() with a sequence of instructions that modify only r0; - on riscv64 jit replaces call to bpf_get_smp_processor_id() with a sequence of instructions that modify only r0; - on arm64 jit replaces call to bpf_get_smp_processor_id() with a sequence of instructions that modify only r0 and tmp registers. These rewrites satisfy attribute no_caller_saved_registers contract. Allow rewrite of no_caller_saved_registers patterns for bpf_get_smp_processor_id() in order to use this function as a canary for no_caller_saved_registers tests. Signed-off-by:
-
Yonghong Song authored
The following tests are added to verifier_ldsx.c: - sign extension of data/data_end/data_meta for tcx programs. The actual checking is in bpf_skb_is_valid_access() which is called by sk_filter, cg_skb, lwt, tc(tcx) and sk_skb. - sign extension of data/data_end/data_meta for xdp programs. - sign extension of data/data_end for flow_dissector programs. All newly-added tests have verification failure with message "invalid bpf_context access". Without previous patch, all these tests succeeded verification. Acked-by: -
Eduard Zingerman authored
GCC and LLVM define a no_caller_saved_registers function attribute. This attribute means that function scratches only some of the caller saved registers defined by ABI. For BPF the set of such registers could be defined as follows: - R0 is scratched only if function is non-void; - R1-R5 are scratched only if corresponding parameter type is defined in the function prototype. This commit introduces flag bpf_func_prot->allow_nocsr. If this flag is set for some helper function, verifier assumes that it follows no_caller_saved_registers calling convention. The contract between kernel and clang allows to simultaneously use such functions and maintain backwards compatibility with old kernels that don't understand no_caller_saved_registers calls (nocsr for short): - clang generates a simple pattern for nocsr calls, e.g.: r1 = 1; r2 = 2; *(u64 *)(r10 - 8) = r1; *(u64 *)(r10 - 16) = r2; call %[to_be_inlined] r2 = *(u64 *)(r10 - 16); r1 = *(u64 *)(r10 - 8); r0 = r1; r0 += r2; exit; - kernel removes unnecessary spills and fills, if called function is inlined by verifier or current JIT (with assumption that patch inserted by verifier or JIT honors nocsr contract, e.g. does not scratch r3-r5 for the example above), e.g. the code above would be transformed to: r1 = 1; r2 = 2; call %[to_be_inlined] r0 = r1; r0 += r2; exit; Technically, the transformation is split into the following phases: - function mark_nocsr_patterns(), called from bpf_check() searches and marks potential patterns in instruction auxiliary data; - upon stack read or write access, function check_nocsr_stack_contract() is used to verify if stack offsets, presumably reserved for nocsr patterns, are used only from those patterns; - function remove_nocsr_spills_fills(), called from bpf_check(), applies the rewrite for valid patterns. See comment in mark_nocsr_pattern_for_call() for more details. Suggested-by:Alexei Starovoitov <alexei.starovoitov@gmail.com> Signed-off-by:
-
Yonghong Song authored
syzbot reported a kernel crash due to commit 1f1e864b ("bpf: Handle sign-extenstin ctx member accesses"). The reason is due to sign-extension of 32-bit load for packet data/data_end/data_meta uapi field. The original code looks like: r2 = *(s32 *)(r1 + 76) /* load __sk_buff->data */ r3 = *(u32 *)(r1 + 80) /* load __sk_buff->data_end */ r0 = r2 r0 += 8 if r3 > r0 goto +1 ... Note that __sk_buff->data load has 32-bit sign extension. After verification and convert_ctx_accesses(), the final asm code looks like: r2 = *(u64 *)(r1 +208) r2 = (s32)r2 r3 = *(u64 *)(r1 +80) r0 = r2 r0 += 8 if r3 > r0 goto pc+1 ... Note that 'r2 = (s32)r2' may make the kernel __sk_buff->data address invalid which may cause runtime failure. Currently, in C code, typically we have void *data = (void *)(long)skb->data; void *data_end = (void *)(long)skb->data_end; ... and it will generate r2 = *(u64 *)(r1 +208) r3 = *(u64 *)(r1 +80) r0 = r2 r0 += 8 if r3 > r0 goto pc+1 If we allow sign-extension, void *data = (void *)(long)(int)skb->data; void *data_end = (void *)(long)skb->data_end; ... the generated code looks like r2 = *(u64 *)(r1 +208) r2 <<= 32 r2 s>>= 32 r3 = *(u64 *)(r1 +80) r0 = r2 r0 += 8 if r3 > r0 goto pc+1 and this will cause verification failure since "r2 <<= 32" is not allowed as "r2" is a packet pointer. To fix this issue for case r2 = *(s32 *)(r1 + 76) /* load __sk_buff->data */ this patch added additional checking in is_valid_access() callback function for packet data/data_end/data_meta access. If those accesses are with sign-extenstion, the verification will fail. [1] https://lore.kernel.org/bpf/000000000000c90eee061d236d37@google.com/ Reported-by: syzbot+ad9ec60c8eaf69e6f99c@syzkaller.appspotmail.com Fixes: 1f1e864b ("bpf: Handle sign-extenstin ctx member accesses") Acked-by:
-
Eduard Zingerman authored
Extract the part of check_helper_call() as a utility function allowing to query 'struct bpf_func_proto' for a specific helper function id. Acked-by:
-
Tony Ambardar authored
Actually use previously defined LDFLAGS during build and add support for LDLIBS to link extra standalone libraries e.g. 'argp' which is not provided by musl libc. Fixes: 585bf464 ("tools: runqslower: Add EXTRA_CFLAGS and EXTRA_LDFLAGS support") Signed-off-by:
Ilya Leoshkevich <iii@linux.ibm.com> Link: https://lore.kernel.org/bpf/20240723003045.2273499-1-tony.ambardar@gmail.com
-
