- 18 Mar, 2022 22 commits
-
-
Jiri Olsa authored
Adding bpf_program__attach_kprobe_multi_opts function for attaching kprobe program to multiple functions. struct bpf_link * bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog, const char *pattern, const struct bpf_kprobe_multi_opts *opts); User can specify functions to attach with 'pattern' argument that allows wildcards (*?' supported) or provide symbols or addresses directly through opts argument. These 3 options are mutually exclusive. When using symbols or addresses, user can also provide cookie value for each symbol/address that can be retrieved later in bpf program with bpf_get_attach_cookie helper. struct bpf_kprobe_multi_opts { size_t sz; const char **syms; const unsigned long *addrs; const __u64 *cookies; size_t cnt; bool retprobe; size_t :0; }; Symbols, addresses and cookies are provided through opts object (syms/addrs/cookies) as array pointers with specified count (cnt). Each cookie value is paired with provided function address or symbol with the same array index. The program can be also attached as return probe if 'retprobe' is set. For quick usage with NULL opts argument, like: bpf_program__attach_kprobe_multi_opts(prog, "ksys_*", NULL) the 'prog' will be attached as kprobe to 'ksys_*' functions. Also adding new program sections for automatic attachment: kprobe.multi/<symbol_pattern> kretprobe.multi/<symbol_pattern> The symbol_pattern is used as 'pattern' argument in bpf_program__attach_kprobe_multi_opts function. Signed-off-by:Jiri Olsa <jolsa@kernel.org> Signed-off-by:
Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/20220316122419.933957-10-jolsa@kernel.org
-
Jiri Olsa authored
Adding new kprobe_multi struct to bpf_link_create_opts object to pass multiple kprobe data to link_create attr uapi. Signed-off-by:
-
Jiri Olsa authored
Move the kallsyms parsing in internal libbpf_kallsyms_parse function, so it can be used from other places. It will be used in following changes. Signed-off-by:
Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20220316122419.933957-8-jolsa@kernel.org
-
Jiri Olsa authored
Adding support to call bpf_get_attach_cookie helper from kprobe programs attached with kprobe multi link. The cookie is provided by array of u64 values, where each value is paired with provided function address or symbol with the same array index. When cookie array is provided it's sorted together with addresses (check bpf_kprobe_multi_cookie_swap). This way we can find cookie based on the address in bpf_get_attach_cookie helper. Suggested-by:
-
Jiri Olsa authored
Adding support to inline it on x86, because it's single load instruction. Signed-off-by:
-
Jiri Olsa authored
Adding support to call bpf_get_func_ip helper from kprobe programs attached by multi kprobe link. Signed-off-by:
-
Jiri Olsa authored
Adding new link type BPF_LINK_TYPE_KPROBE_MULTI that attaches kprobe program through fprobe API. The fprobe API allows to attach probe on multiple functions at once very fast, because it works on top of ftrace. On the other hand this limits the probe point to the function entry or return. The kprobe program gets the same pt_regs input ctx as when it's attached through the perf API. Adding new attach type BPF_TRACE_KPROBE_MULTI that allows attachment kprobe to multiple function with new link. User provides array of addresses or symbols with count to attach the kprobe program to. The new link_create uapi interface looks like: struct { __u32 flags; __u32 cnt; __aligned_u64 syms; __aligned_u64 addrs; } kprobe_multi; The flags field allows single BPF_TRACE_KPROBE_MULTI bit to create return multi kprobe. Signed-off-by:Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by:
-
Jiri Olsa authored
When kallsyms_lookup_name is called with empty string, it will do futile search for it through all the symbols. Skipping the search for empty string. Signed-off-by:
-
Jiri Olsa authored
Adding support to have priv pointer in swap callback function. Following the initial change on cmp callback functions [1] and adding SWAP_WRAPPER macro to identify sort call of sort_r. Signed-off-by:
-
Alexei Starovoitov authored
Masami Hiramatsu says: ==================== Hi, Here is the 12th version of fprobe. This version fixes a possible gcc-11 issue which was reported as kretprobes on arm issue, and also I updated the fprobe document. The previous version (v11) is here[1]; [1] https://lore.kernel.org/all/164701432038.268462.3329725152949938527.stgit@devnote2/T/#u This series introduces the fprobe, the function entry/exit probe with multiple probe point support for x86, arm64 and powerpc64le. This also introduces the rethook for hooking function return as same as the kretprobe does. This abstraction will help us to generalize the fgraph tracer, because we can just switch to it from the rethook in fprobe, depending on the kernel configuration. The patch [1/12] is from Jiri's series[2]. [2] https://lore.kernel.org/all/20220104080943.113249-1-jolsa@kernel.org/T/#u And the patch [9/10] adds the FPROBE_FL_KPROBE_SHARED flag for the case if user wants to share the same code (or share a same resource) on the fprobe and the kprobes. I forcibly updated my kprobes/fprobe branch, you can pull this series from: https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git kprobes/fprobe Thank you, --- Jiri Olsa (1): ftrace: Add ftrace_set_filter_ips function ==================== Signed-off-by:
-
Masami Hiramatsu authored
Add a KUnit based selftest for fprobe interface. Signed-off-by:
Steven Rostedt (Google) <rostedt@goodmis.org> Tested-by:
-
Masami Hiramatsu authored
Add a documentation of fprobe for the user who needs this interface. Signed-off-by:
-
Masami Hiramatsu authored
Introduce FPROBE_FL_KPROBE_SHARED flag for sharing fprobe callback with kprobes safely from the viewpoint of recursion. Since the recursion safety of the fprobe (and ftrace) is a bit different from the kprobes, this may cause an issue if user wants to run the same code from the fprobe and the kprobes. The kprobes has per-cpu 'current_kprobe' variable which protects the kprobe handler from recursion in any case. On the other hand, the fprobe uses only ftrace_test_recursion_trylock(), which will allow interrupt context calls another (or same) fprobe during the fprobe user handler is running. This is not a matter in cases if the common callback shared among the kprobes and the fprobe has its own recursion detection, or it can handle the recursion in the different contexts (normal/interrupt/NMI.) But if it relies on the 'current_kprobe' recursion lock, it has to check kprobe_running() and use kprobe_busy_*() APIs. Fprobe has FPROBE_FL_KPROBE_SHARED flag to do this. If your common callback code will be shared with kprobes, please set FPROBE_FL_KPROBE_SHARED *before* registering the fprobe, like; fprobe.flags = FPROBE_FL_KPROBE_SHARED; register_fprobe(&fprobe, "func*", NULL); This will protect your common callback from the nested call. Signed-off-by:
-
Masami Hiramatsu authored
Add a sample program for the fprobe. The sample_fprobe puts a fprobe on kernel_clone() by default. This dump stack and some called address info at the function entry and exit. The sample_fprobe.ko gets 2 parameters. - symbol: you can specify the comma separated symbols or wildcard symbol pattern (in this case you can not use comma) - stackdump: a bool value to enable or disable stack dump in the fprobe handler. Signed-off-by:
-
Masami Hiramatsu authored
Add exit_handler to fprobe. fprobe + rethook allows us to hook the kernel function return. The rethook will be enabled only if the fprobe::exit_handler is set. Signed-off-by:
-
Masami Hiramatsu authored
Add rethook arm implementation. Most of the code has been copied from kretprobes on arm. Since the arm's ftrace implementation is a bit special, this needs a special care using from fprobe. Signed-off-by:
-
Masami Hiramatsu authored
Add rethook powerpc64 implementation. Most of the code has been copied from kretprobes on powerpc64. Signed-off-by:
-
Masami Hiramatsu authored
Add rethook arm64 implementation. Most of the code has been copied from kretprobes on arm64. Signed-off-by:
-
Masami Hiramatsu authored
Add rethook for x86 implementation. Most of the code has been copied from kretprobes on x86. Signed-off-by:
-
Masami Hiramatsu authored
Add a return hook framework which hooks the function return. Most of the logic came from the kretprobe, but this is independent from kretprobe. Note that this is expected to be used with other function entry hooking feature, like ftrace, fprobe, adn kprobes. Eventually this will replace the kretprobe (e.g. kprobe + rethook = kretprobe), but at this moment, this is just an additional hook. Signed-off-by:
-
Masami Hiramatsu authored
The fprobe is a wrapper API for ftrace function tracer. Unlike kprobes, this probes only supports the function entry, but this can probe multiple functions by one fprobe. The usage is similar, user will set their callback to fprobe::entry_handler and call register_fprobe*() with probed functions. There are 3 registration interfaces, - register_fprobe() takes filtering patterns of the functin names. - register_fprobe_ips() takes an array of ftrace-location addresses. - register_fprobe_syms() takes an array of function names. The registered fprobes can be unregistered with unregister_fprobe(). e.g. struct fprobe fp = { .entry_handler = user_handler }; const char *targets[] = { "func1", "func2", "func3"}; ... ret = register_fprobe_syms(&fp, targets, ARRAY_SIZE(targets)); ... unregister_fprobe(&fp); Signed-off-by: -
Jiri Olsa authored
Adding ftrace_set_filter_ips function to be able to set filter on multiple ip addresses at once. With the kprobe multi attach interface we have cases where we need to initialize ftrace_ops object with thousands of functions, so having single function diving into ftrace_hash_move_and_update_ops with ftrace_lock is faster. The functions ips are passed as unsigned long array with count. Signed-off-by:
-
- 17 Mar, 2022 4 commits
-
-
Kaixi Fan authored
In namespace at_ns0, the IP address of tnl dev is 10.1.1.100 which is the overlay IP, and the ip address of veth0 is 172.16.1.100 which is the vtep IP. When doing 'ping 10.1.1.100' from root namespace, the remote_ip should be 172.16.1.100. Fixes: 933a741e ("selftests/bpf: bpf tunnel test.") Signed-off-by:
Kaixi Fan <fankaixi.li@bytedance.com> Signed-off-by:
Daniel Borkmann <daniel@iogearbox.net> Acked-by:
John Fastabend <john.fastabend@gmail.com> Link: https://lore.kernel.org/bpf/20220313164116.5889-1-fankaixi.li@bytedance.comSigned-off-by:
-
Lorenzo Bianconi authored
Allow increasing the MTU over page boundaries on veth devices if the attached xdp program declares to support xdp fragments. Signed-off-by:
Lorenzo Bianconi <lorenzo@kernel.org> Signed-off-by:
Toke Høiland-Jørgensen <toke@redhat.com> Acked-by:
-
Lorenzo Bianconi authored
Introduce veth_convert_skb_to_xdp_buff routine in order to convert a non-linear skb into a xdp buffer. If the received skb is cloned or shared, veth_convert_skb_to_xdp_buff will copy it in a new skb composed by order-0 pages for the linear and the fragmented area. Moreover veth_convert_skb_to_xdp_buff guarantees we have enough headroom for xdp. This is a preliminary patch to allow attaching xdp programs with frags support on veth devices. Signed-off-by:
-
Lorenzo Bianconi authored
Even if this is a theoretical issue since it is not possible to perform XDP_REDIRECT on a non-linear xdp_frame, veth driver does not account paged area in ndo_xdp_xmit function pointer. Introduce xdp_get_frame_len utility routine to get the xdp_frame full length and account total frame size running XDP_REDIRECT of a non-linear xdp frame into a veth device. Signed-off-by:
-
- 16 Mar, 2022 7 commits
-
-
Hou Tao authored
When bpf_jit_harden is toggled between 0 and 2, subprog jit may fail due to inconsistent twice read values of bpf_jit_harden during jit. So add a test to ensure the problem is fixed. Signed-off-by:
Hou Tao <houtao1@huawei.com> Signed-off-by:
-
Hou Tao authored
It is the bpf_jit_harden counterpart to commit 60b58afc ("bpf: fix net.core.bpf_jit_enable race"). bpf_jit_harden will be tested twice for each subprog if there are subprogs in bpf program and constant blinding may increase the length of program, so when running "./test_progs -t subprogs" and toggling bpf_jit_harden between 0 and 2, jit_subprogs may fail because constant blinding increases the length of subprog instructions during extra passs. So cache the value of bpf_jit_blinding_enabled() during program allocation, and use the cached value during constant blinding, subprog JITing and args tracking of tail call. Signed-off-by:
-
Hou Tao authored
Extra pass for subprog jit may fail (e.g. due to bpf_jit_harden race), but bpf_func is not cleared for the subprog and jit_subprogs will succeed. The running of the bpf program may lead to oops because the memory for the jited subprog image has already been freed. So fall back to interpreter mode by clearing bpf_func/jited/jited_len when extra pass fails. Signed-off-by:
-
Alexei Starovoitov authored
Martin KaFai Lau says: ==================== After upgrading to the newer libcap (>= 2.60), the libcap commit aca076443591 ("Make cap_t operations thread safe.") added a "__u8 mutex;" to the "struct _cap_struct". It caused a few byte shift that breaks the assumption made in the "struct libcap" definition in test_verifier.c. This set is to remove the libcap dependency from the bpf selftests. v2: - Define CAP_PERFMON and CAP_BPF when the older <linux/capability.h> does not have them. (Andrii) ==================== Signed-off-by: -
Martin KaFai Lau authored
This patch removes the libcap usage from test_progs. bind_perm.c is the only user. cap_*_effective() helpers added in the earlier patch are directly used instead. No other selftest binary is using libcap, so '-lcap' is also removed from the Makefile. Signed-off-by:
Martin KaFai Lau <kafai@fb.com> Signed-off-by:
Stanislav Fomichev <sdf@google.com> Acked-by:
-
Martin KaFai Lau authored
This patch removes the libcap usage from test_verifier. The cap_*_effective() helpers added in the earlier patch are used instead. Signed-off-by:
-
Martin KaFai Lau authored
After upgrading to the newer libcap (>= 2.60), the libcap commit aca076443591 ("Make cap_t operations thread safe.") added a "__u8 mutex;" to the "struct _cap_struct". It caused a few byte shift that breaks the assumption made in the "struct libcap" definition in test_verifier.c. The bpf selftest usage only needs to enable and disable the effective caps of the running task. It is easier to directly syscall the capget and capset instead. It can also remove the libcap library dependency. The cap_helpers.{c,h} is added. One __u64 is used for all CAP_* bits instead of two __u32. Signed-off-by:
-
- 15 Mar, 2022 7 commits
-
-
Daniel Xu authored
The top-level (bpftool.8) man page was missing docs for a few subcommands and their respective sub-sub-commands. This commit brings the top level man page up to date. Note that I've kept the ordering of the subcommands the same as in `bpftool help`. Signed-off-by:
Daniel Xu <dxu@dxuuu.xyz> Signed-off-by:
-
Dmitrii Dolgov authored
Commit 82e6b1ee ("bpf: Allow to specify user-provided bpf_cookie for BPF perf links") introduced the concept of user specified bpf_cookie, which could be accessed by BPF programs using bpf_get_attach_cookie(). For troubleshooting purposes it is convenient to expose bpf_cookie via bpftool as well, so there is no need to meddle with the target BPF program itself. Implemented using the pid iterator BPF program to actually fetch bpf_cookies, which allows constraining code changes only to bpftool. $ bpftool link 1: type 7 prog 5 bpf_cookie 123 pids bootstrap(81) Signed-off-by:
Dmitrii Dolgov <9erthalion6@gmail.com> Signed-off-by:
Yonghong Song <yhs@fb.com> Acked-by:
Quentin Monnet <quentin@isovalent.com> Link: https://lore.kernel.org/bpf/20220309163112.24141-1-9erthalion6@gmail.com
-
Guo Zhengkui authored
Clean up the array_size.cocci warnings under tools/testing/selftests/bpf/: Use `ARRAY_SIZE(arr)` instead of forms like `sizeof(arr)/sizeof(arr[0])`. tools/testing/selftests/bpf/test_cgroup_storage.c uses ARRAY_SIZE() defined in tools/include/linux/kernel.h (sys/sysinfo.h -> linux/kernel.h), while others use ARRAY_SIZE() in bpf_util.h. Signed-off-by:
Guo Zhengkui <guozhengkui@vivo.com> Signed-off-by:
-
Niklas Söderlund authored
When running xdpsock for a fix duration of time before terminating using --duration=<n>, there is a race condition that may cause xdpsock to terminate immediately. When running for a fixed duration of time the check to determine when to terminate execution is in is_benchmark_done() and is being executed in the context of the poller thread, if (opt_duration > 0) { unsigned long dt = (get_nsecs() - start_time); if (dt >= opt_duration) benchmark_done = true; } However start_time is only set after the poller thread have been created. This leaves a small window when the poller thread is starting and calls is_benchmark_done() for the first time that start_time is not yet set. In that case start_time have its initial value of 0 and the duration check fails as it do not correlate correctly for the applications start time and immediately sets benchmark_done which in turn terminates the xdpsock application. Fix this by setting start_time before creating the poller thread. Fixes: d3f11b01 ("samples/bpf: xdpsock: Add duration option to specify how long to run") Signed-off-by:Niklas Söderlund <niklas.soderlund@corigine.com> Signed-off-by:
Simon Horman <simon.horman@corigine.com> Signed-off-by:
-
Wang Yufen authored
If tcp_bpf_sendmsg is running during a tear down operation, psock may be freed. tcp_bpf_sendmsg() tcp_bpf_send_verdict() sk_msg_return() tcp_bpf_sendmsg_redir() unlikely(!psock)) sk_msg_free() The mem of msg has been uncharged in tcp_bpf_send_verdict() by sk_msg_return(), and would be uncharged by sk_msg_free() again. When psock is null, we can simply returning an error code, this would then trigger the sk_msg_free_nocharge in the error path of __SK_REDIRECT and would have the side effect of throwing an error up to user space. This would be a slight change in behavior from user side but would look the same as an error if the redirect on the socket threw an error. This issue can cause the following info: WARNING: CPU: 0 PID: 2136 at net/ipv4/af_inet.c:155 inet_sock_destruct+0x13c/0x260 Call Trace: <TASK> __sk_destruct+0x24/0x1f0 sk_psock_destroy+0x19b/0x1c0 process_one_work+0x1b3/0x3c0 worker_thread+0x30/0x350 ? process_one_work+0x3c0/0x3c0 kthread+0xe6/0x110 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Fixes: 604326b4 ("bpf, sockmap: convert to generic sk_msg interface") Signed-off-by:Wang Yufen <wangyufen@huawei.com> Signed-off-by:
-
Wang Yufen authored
In tcp_bpf_send_verdict(), if msg has more data after tcp_bpf_sendmsg_redir(): tcp_bpf_send_verdict() tosend = msg->sg.size //msg->sg.size = 22220 case __SK_REDIRECT: sk_msg_return() //uncharged msg->sg.size(22220) sk->sk_forward_alloc tcp_bpf_sendmsg_redir() //after tcp_bpf_sendmsg_redir, msg->sg.size=11000 goto more_data; tosend = msg->sg.size //msg->sg.size = 11000 case __SK_REDIRECT: sk_msg_return() //uncharged msg->sg.size(11000) to sk->sk_forward_alloc The msg->sg.size(11000) has been uncharged twice, to fix we can charge the remaining msg->sg.size before goto more data. This issue can cause the following info: WARNING: CPU: 0 PID: 9860 at net/core/stream.c:208 sk_stream_kill_queues+0xd4/0x1a0 Call Trace: <TASK> inet_csk_destroy_sock+0x55/0x110 __tcp_close+0x279/0x470 tcp_close+0x1f/0x60 inet_release+0x3f/0x80 __sock_release+0x3d/0xb0 sock_close+0x11/0x20 __fput+0x92/0x250 task_work_run+0x6a/0xa0 do_exit+0x33b/0xb60 do_group_exit+0x2f/0xa0 get_signal+0xb6/0x950 arch_do_signal_or_restart+0xac/0x2a0 ? vfs_write+0x237/0x290 exit_to_user_mode_prepare+0xa9/0x200 syscall_exit_to_user_mode+0x12/0x30 do_syscall_64+0x46/0x80 entry_SYSCALL_64_after_hwframe+0x44/0xae </TASK> WARNING: CPU: 0 PID: 2136 at net/ipv4/af_inet.c:155 inet_sock_destruct+0x13c/0x260 Call Trace: <TASK> __sk_destruct+0x24/0x1f0 sk_psock_destroy+0x19b/0x1c0 process_one_work+0x1b3/0x3c0 worker_thread+0x30/0x350 ? process_one_work+0x3c0/0x3c0 kthread+0xe6/0x110 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Fixes: 604326b4 ("bpf, sockmap: convert to generic sk_msg interface") Signed-off-by:
-
Wang Yufen authored
If tcp_bpf_sendmsg() is running while sk msg is full. When sk_msg_alloc() returns -ENOMEM error, tcp_bpf_sendmsg() goes to wait_for_memory. If partial memory has been alloced by sk_msg_alloc(), that is, msg_tx->sg.size is greater than osize after sk_msg_alloc(), memleak occurs. To fix we use sk_msg_trim() to release the allocated memory, then goto wait for memory. Other call paths of sk_msg_alloc() have the similar issue, such as tls_sw_sendmsg(), so handle sk_msg_trim logic inside sk_msg_alloc(), as Cong Wang suggested. This issue can cause the following info: WARNING: CPU: 3 PID: 7950 at net/core/stream.c:208 sk_stream_kill_queues+0xd4/0x1a0 Call Trace: <TASK> inet_csk_destroy_sock+0x55/0x110 __tcp_close+0x279/0x470 tcp_close+0x1f/0x60 inet_release+0x3f/0x80 __sock_release+0x3d/0xb0 sock_close+0x11/0x20 __fput+0x92/0x250 task_work_run+0x6a/0xa0 do_exit+0x33b/0xb60 do_group_exit+0x2f/0xa0 get_signal+0xb6/0x950 arch_do_signal_or_restart+0xac/0x2a0 exit_to_user_mode_prepare+0xa9/0x200 syscall_exit_to_user_mode+0x12/0x30 do_syscall_64+0x46/0x80 entry_SYSCALL_64_after_hwframe+0x44/0xae </TASK> WARNING: CPU: 3 PID: 2094 at net/ipv4/af_inet.c:155 inet_sock_destruct+0x13c/0x260 Call Trace: <TASK> __sk_destruct+0x24/0x1f0 sk_psock_destroy+0x19b/0x1c0 process_one_work+0x1b3/0x3c0 kthread+0xe6/0x110 ret_from_fork+0x22/0x30 </TASK> Fixes: 604326b4 ("bpf, sockmap: convert to generic sk_msg interface") Signed-off-by:
-
