- 27 Sep, 2022 4 commits
-
-
Jiri Olsa authored
Martynas reported bpf_get_func_ip returning +4 address when CONFIG_X86_KERNEL_IBT option is enabled. When CONFIG_X86_KERNEL_IBT is enabled we'll have endbr instruction at the function entry, which screws return value of bpf_get_func_ip() helper that should return the function address. There's short term workaround for kprobe_multi bpf program made by Alexei [1], but we need this fixup also for bpf_get_attach_cookie, that returns cookie based on the entry_ip value. Moving the fixup in the fprobe handler, so both bpf_get_func_ip and bpf_get_attach_cookie get expected function address when CONFIG_X86_KERNEL_IBT option is enabled. Also renaming kprobe_multi_link_handler entry_ip argument to fentry_ip so it's clearer this is an ftrace __fentry__ ip. [1] commit 7f0059b5 ("selftests/bpf: Fix kprobe_multi test.") Cc: Peter Zijlstra <peterz@infradead.org> Reported-by:
Martynas Pumputis <m@lambda.lt> Acked-by:
Andrii Nakryiko <andrii@kernel.org> Signed-off-by:
Jiri Olsa <jolsa@kernel.org> Link: https://lore.kernel.org/r/20220926153340.1621984-5-jolsa@kernel.orgSigned-off-by:
Alexei Starovoitov <ast@kernel.org>
-
Jiri Olsa authored
Using function address given at the generation time as the trampoline ip argument. This way we get directly the function address that we need, so we don't need to: - read the ip from the stack - subtract X86_PATCH_SIZE - subtract ENDBR_INSN_SIZE if CONFIG_X86_KERNEL_IBT is enabled which is not even implemented yet ;-) Signed-off-by: -
Jiri Olsa authored
Keeping the resolved 'addr' in kallsyms_callback, instead of taking ftrace_location value, because we depend on symbol address in the cookie related code. With CONFIG_X86_KERNEL_IBT option the ftrace_location value differs from symbol address, which screwes the symbol address cookies matching. There are 2 users of this function: - bpf_kprobe_multi_link_attach for which this fix is for - get_ftrace_locations which is used by register_fprobe_syms this function needs to get symbols resolved to addresses, but does not need 'ftrace location addresses' at this point there's another ftrace location translation in the path done by ftrace_set_filter_ips call: register_fprobe_syms addrs = get_ftrace_locations register_fprobe_ips(addrs) ... ftrace_set_filter_ips ... __ftrace_match_addr ip = ftrace_location(ip); ... Reviewed-by:Masami Hiramatsu (Google) <mhiramat@kernel.org> Signed-off-by:
-
Jiri Olsa authored
Adding KPROBE_FLAG_ON_FUNC_ENTRY kprobe flag to indicate that attach address is on function entry. This is used in following changes in get_func_ip helper to return correct function address. Acked-by:
-
- 26 Sep, 2022 3 commits
-
-
Liu Jian authored
In sk_psock_backlog function, for ingress direction skb, if no new data packet arrives after the skb is cached, the cached skb does not have a chance to be added to the receive queue of psock. As a result, the cached skb cannot be received by the upper-layer application. Fix this by reschedule the psock work to dispose the cached skb in sk_msg_recvmsg function. Fixes: 604326b4 ("bpf, sockmap: convert to generic sk_msg interface") Signed-off-by:
Liu Jian <liujian56@huawei.com> Signed-off-by:
Daniel Borkmann <daniel@iogearbox.net> Acked-by:
John Fastabend <john.fastabend@gmail.com> Link: https://lore.kernel.org/bpf/20220907071311.60534-1-liujian56@huawei.com
-
Liu Jian authored
Add one test for wait redirect sock's send memory test for sockmap. Signed-off-by:
-
Liu Jian authored
Fixes the below NULL pointer dereference: [...] [ 14.471200] Call Trace: [ 14.471562] <TASK> [ 14.471882] lock_acquire+0x245/0x2e0 [ 14.472416] ? remove_wait_queue+0x12/0x50 [ 14.473014] ? _raw_spin_lock_irqsave+0x17/0x50 [ 14.473681] _raw_spin_lock_irqsave+0x3d/0x50 [ 14.474318] ? remove_wait_queue+0x12/0x50 [ 14.474907] remove_wait_queue+0x12/0x50 [ 14.475480] sk_stream_wait_memory+0x20d/0x340 [ 14.476127] ? do_wait_intr_irq+0x80/0x80 [ 14.476704] do_tcp_sendpages+0x287/0x600 [ 14.477283] tcp_bpf_push+0xab/0x260 [ 14.477817] tcp_bpf_sendmsg_redir+0x297/0x500 [ 14.478461] ? __local_bh_enable_ip+0x77/0xe0 [ 14.479096] tcp_bpf_send_verdict+0x105/0x470 [ 14.479729] tcp_bpf_sendmsg+0x318/0x4f0 [ 14.480311] sock_sendmsg+0x2d/0x40 [ 14.480822] ____sys_sendmsg+0x1b4/0x1c0 [ 14.481390] ? copy_msghdr_from_user+0x62/0x80 [ 14.482048] ___sys_sendmsg+0x78/0xb0 [ 14.482580] ? vmf_insert_pfn_prot+0x91/0x150 [ 14.483215] ? __do_fault+0x2a/0x1a0 [ 14.483738] ? do_fault+0x15e/0x5d0 [ 14.484246] ? __handle_mm_fault+0x56b/0x1040 [ 14.484874] ? lock_is_held_type+0xdf/0x130 [ 14.485474] ? find_held_lock+0x2d/0x90 [ 14.486046] ? __sys_sendmsg+0x41/0x70 [ 14.486587] __sys_sendmsg+0x41/0x70 [ 14.487105] ? intel_pmu_drain_pebs_core+0x350/0x350 [ 14.487822] do_syscall_64+0x34/0x80 [ 14.488345] entry_SYSCALL_64_after_hwframe+0x63/0xcd [...] The test scenario has the following flow: thread1 thread2 ----------- --------------- tcp_bpf_sendmsg tcp_bpf_send_verdict tcp_bpf_sendmsg_redir sock_close tcp_bpf_push_locked __sock_release tcp_bpf_push //inet_release do_tcp_sendpages sock->ops->release sk_stream_wait_memory // tcp_close sk_wait_event sk->sk_prot->close release_sock(__sk); *** lock_sock(sk); __tcp_close sock_orphan(sk) sk->sk_wq = NULL release_sock **** lock_sock(__sk); remove_wait_queue(sk_sleep(sk), &wait); sk_sleep(sk) //NULL pointer dereference &rcu_dereference_raw(sk->sk_wq)->wait While waiting for memory in thread1, the socket is released with its wait queue because thread2 has closed it. This caused by tcp_bpf_send_verdict didn't increase the f_count of psock->sk_redir->sk_socket->file in thread1. We should check if SOCK_DEAD flag is set on wakeup in sk_stream_wait_memory before accessing the wait queue. Suggested-by:Jakub Sitnicki <jakub@cloudflare.com> Signed-off-by:
-
- 24 Sep, 2022 6 commits
-
-
Alexei Starovoitov authored
Andrii Nakryiko says: ==================== A small patch set adding few usability improvements and features making veristat a more convenient tool to be used for work on BPF verifier: - patch #2 speeds up and makes stats parsing from BPF verifier log more robust; - patch #3 makes veristat less strict about input object files; veristat will ignore non-BPF ELF files; - patch #4 adds progress log, by default, so that user doing mass-verification is aware that veristat is not stuck; - patch #5 allows to tune requested BPF verifier log level, which makes veristat a simplest way to get BPF verifier log, especially successfully verified ones. v1->v2: - don't emit progress in non-table mode, as it breaks CSV output. ==================== Signed-off-by: -
Andrii Nakryiko authored
Add -l (--log-level) flag to override default BPF verifier log lever. This only matters in verbose mode, which is the mode in which veristat emits verifier log for each processed BPF program. This is important because for successfully verified BPF programs log_level 1 is empty, as BPF verifier truncates all the successfully verified paths. So -l2 is the only way to actually get BPF verifier log in practice. It looks sometihng like this: [vmuser@archvm bpf]$ sudo ./veristat xdp_tx.bpf.o -vl2 Processing 'xdp_tx.bpf.o'... PROCESSING xdp_tx.bpf.o/xdp_tx, DURATION US: 19, VERDICT: success, VERIFIER LOG: func#0 @0 0: R1=ctx(off=0,imm=0) R10=fp0 ; return XDP_TX; 0: (b4) w0 = 3 ; R0_w=3 1: (95) exit verification time 19 usec stack depth 0 processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0 File Program Verdict Duration (us) Total insns Total states Peak states ------------ ------- ------- ------------- ----------- ------------ ----------- xdp_tx.bpf.o xdp_tx success 19 2 0 0 ------------ ------- ------- ------------- ----------- ------------ ----------- Done. Processed 1 files, 0 programs. Skipped 1 files, 0 programs. Signed-off-by:
-
Andrii Nakryiko authored
Emit "Processing <filepath>..." for each BPF object file to be processed, to show progress. But also add -q (--quiet) flag to silence such messages. Doing something more clever (like overwriting same output line) is to cumbersome and easily breakable if there is any other console output (e.g., errors from libbpf). Signed-off-by:
-
Andrii Nakryiko authored
Make veristat ignore non-BPF object files. This allows simpler mass-verification (e.g., `sudo ./veristat *.bpf.o` in selftests/bpf directory). Note that `sudo ./veristat *.o` would also work, but with selftests's multiple copies of BPF object files (.bpf.o and .bpf.linked{1,2,3}.o) it's 4x slower. Also, given some of BPF object files could be incomplete in the sense that they are meant to be statically linked into final BPF object file (like linked_maps, linked_funcs, linked_vars), note such instances in stderr, but proceed anyways. This seems like a better trade off between completely silently ignoring BPF object file and aborting mass-verification altogether. Signed-off-by: -
Andrii Nakryiko authored
Make sure veristat doesn't spend ridiculous amount of time parsing verifier stats from verifier log, especially for very large logs or truncated logs (e.g., when verifier returns -ENOSPC due to too small buffer). For this, parse lines from the end of the log and make sure we parse only up to 100 last lines, where stats should be, if at all. Suggested-by:
-
Andrii Nakryiko authored
Add sign-file to .gitignore to avoid accidentally checking it in. Signed-off-by:
-
- 23 Sep, 2022 3 commits
-
-
Andrii Nakryiko authored
When attach_prog_fd field was removed in libbpf 1.0 and replaced with `long: 0` placeholder, it actually shifted all the subsequent fields by 8 byte. This is due to `long: 0` promising to adjust next field's offset to long-aligned offset. But in this case we were already long-aligned as pin_root_path is a pointer. So `long: 0` had no effect, and thus didn't feel the gap created by removed attach_prog_fd. Non-zero bitfield should have been used instead. I validated using pahole. Originally kconfig field was at offset 40. With `long: 0` it's at offset 32, which is wrong. With this change it's back at offset 40. While technically libbpf 1.0 is allowed to break backwards compatibility and applications should have been recompiled against libbpf 1.0 headers, but given how trivial it is to preserve memory layout, let's fix this. Reported-by:
Grant Seltzer Richman <grantseltzer@gmail.com> Fixes: 146bf811 ("libbpf: remove most other deprecated high-level APIs") Signed-off-by:
Martin KaFai Lau <martin.lau@kernel.org>
-
Wang Yufen authored
Move snprintf and len check to common helper pathname_concat() to make the code simpler. Signed-off-by:
Wang Yufen <wangyufen@huawei.com> Signed-off-by:
-
Yosry Ahmed authored
The cgroup_hierarchical_stats selftest is complicated. It has to be, because it tests an entire workflow of recording, aggregating, and dumping cgroup stats. However, some of the complexity is unnecessary. The test now enables the memory controller in a cgroup hierarchy, invokes reclaim, measure reclaim time, THEN uses that reclaim time to test the stats collection and aggregation. We don't need to use such a complicated stat, as the context in which the stat is collected is orthogonal. Simplify the test by using a simple stat instead of reclaim time, the total number of times a process has ever entered a cgroup. This makes the test simpler and removes the dependency on the memory controller and the memory reclaim interface. Signed-off-by:
Yosry Ahmed <yosryahmed@google.com> Signed-off-by:
KP Singh <kpsingh@kernel.org> Link: https://lore.kernel.org/bpf/20220919175330.890793-1-yosryahmed@google.com
-
- 22 Sep, 2022 24 commits
-
-
Martin KaFai Lau authored
Hou Tao says: ==================== From: Hou Tao <houtao1@huawei.com> Hi, It is just a tiny patch set aims to fix the resource leaks in test_maps after test case succeeds or is skipped. And these leaks are spotted by using address sanitizer and checking the content of /proc/$pid/fd. Please see indiviual patch for more details. Change Log: v2: * Add the missing header file unistd.h for close() (From kernel-patches/bpf) The reason Why I miss that is that -Werror is removed from Makefile when enabling clang address sanitizer. v1: * https://lore.kernel.org/bpf/20220921025855.115463-1-houtao@huaweicloud.com/T/ ==================== Signed-off-by:
-
Hou Tao authored
Free the created fd or allocated bpf_object after test case succeeds, else there will be resource leaks. Spotted by using address sanitizer and checking the content of /proc/$pid/fd directory. Signed-off-by:
Hou Tao <houtao1@huawei.com> Link: https://lore.kernel.org/r/20220921070035.2016413-3-houtao@huaweicloud.comSigned-off-by:
-
Hou Tao authored
Destroy the created skeleton when CONFIG_PREEMPT is off, else will be resource leak. Fixes: 73b97bc7 ("selftests/bpf: Test concurrent updates on bpf_task_storage_busy") Signed-off-by:
-
Yauheni Kaliuta authored
Added urandom_read shared lib is missing from the list of installed files what makes urandom_read test after `make install` or `make gen_tar` broken. Add the library to TEST_GEN_FILES. The names in the list do not contain $(OUTPUT) since it's added by lib.mk code. Fixes: 00a0fa2d ("selftests/bpf: Add urandom_read shared lib and USDTs") Signed-off-by:
Yauheni Kaliuta <ykaliuta@redhat.com> Signed-off-by:
-
Alexei Starovoitov authored
Andrii Nakryiko says: ==================== Add three more critical features to veristat tool, which make it sufficient for a practical work on BPF verifier: - CSV output, which allows easier programmatic post-processing of stats; - building upon CSV output, veristat now supports comparison mode, in which two previously captured CSV outputs from veristat are compared with each other in a convenient form; - flexible allow/deny filtering using globs for BPF object files and programs, allowing to narrow down target BPF programs to be verified. See individual patches for more details and examples. v1->v2: - split out double-free fix into patch #1 (Yonghong); - fixed typo in verbose flag (Quentin); - baseline and comparison stats were reversed in output table, fixed that. ==================== Signed-off-by: -
Andrii Nakryiko authored
Add -f (--filter) argument which accepts glob-based filters for narrowing down what BPF object files and programs within them should be processed by veristat. This filtering applies both to comparison and main (verification) mode. Filter can be of two forms: - file (object) filter: 'strobemeta*'; in this case all the programs within matching files are implicitly allowed (or denied, depending if it's positive or negative rule, see below); - file and prog filter: 'strobemeta*/*unroll*' will further filter programs within matching files to only allow those program names that match '*unroll*' glob. As mentioned, filters can be positive (allowlisting) and negative (denylisting). Negative filters should start with '!': '!strobemeta*' will deny any filename which basename starts with "strobemeta". Further, one extra special syntax is supported to allow more convenient use in practice. Instead of specifying rule on the command line, veristat allows to specify file that contains rules, both positive and negative, one line per one filter. This is achieved with -f @<filepath> use, where <filepath> points to a text file containing rules (negative and positive rules can be mixed). For convenience empty lines and lines starting with '#' are ignored. This feature is useful to have some pre-canned list of object files and program names that are tested repeatedly, allowing to check in a list of rules and quickly specify them on the command line. As a demonstration (and a short cut for nearest future), create a small list of "interesting" BPF object files from selftests/bpf and commit it as veristat.cfg. It currently includes 73 programs, most of which are the most complex and largest BPF programs in selftests, as judged by total verified instruction count and verifier states total. If there is overlap between positive or negative filters, negative filter takes precedence (denylisting is stronger than allowlisting). If no allow filter is specified, veristat implicitly assumes '*/*' rule. If no deny rule is specified, veristat (logically) assumes no negative filters. Also note that -f (just like -e and -s) can be specified multiple times and their effect is cumulative. Signed-off-by: -
Andrii Nakryiko authored
Add ability to compare and contrast two veristat runs, previously recorded with veristat using CSV output format. When veristat is called with -C (--compare) flag, veristat expects exactly two input files specified, both should be in CSV format. Expectation is that it's output from previous veristat runs, but as long as column names and formats match, it should just work. First CSV file is designated as a "baseline" provided, and the second one is comparison (experiment) data set. Establishing baseline matters later when calculating difference percentages, see below. Veristat parses these two CSV files and "reconstructs" verifier stats (it could be just a subset of all possible stats). File and program names are mandatory as they are used as joining key (these two "stats" are designated as "key stats" in the code). Veristat currently enforces that the set of stats recorded in both CSV has to exactly match, down to exact order. This is just a simplifying condition which can be lifted with a bit of additional pre-processing to reorded stat specs internally, which I didn't bother doing, yet. For all the non-key stats, veristat will output three columns: one for baseline data, one for comparison data, and one with an absolute and relative percentage difference. If either baseline or comparison values are missing (that is, respective CSV file doesn't have a row with *exactly* matching file and program name), those values are assumed to be empty or zero. In such case relative percentages are forced to +100% or -100% output, for consistency with a typical case. Veristat's -e (--emit) and -s (--sort) specs still apply, so even if CSV contains lots of stats, user can request to compare only a subset of them (and specify desired column order as well). Similarly, both CSV and human-readable table output is honored. Note that input is currently always expected to be CSV. Here's an example shell session, recording data for biosnoop tool on two different kernels and comparing them afterwards, outputting data in table format. # on slightly older production kernel $ sudo ./veristat biosnoop_bpf.o File Program Verdict Duration (us) Total insns Total states Peak states -------------- ------------------------ ------- ------------- ----------- ------------ ----------- biosnoop_bpf.o blk_account_io_merge_bio success 37 24 1 1 biosnoop_bpf.o blk_account_io_start failure 0 0 0 0 biosnoop_bpf.o block_rq_complete success 76 104 6 6 biosnoop_bpf.o block_rq_insert success 83 85 7 7 biosnoop_bpf.o block_rq_issue success 79 85 7 7 -------------- ------------------------ ------- ------------- ----------- ------------ ----------- Done. Processed 1 object files, 5 programs. $ sudo ./veristat ~/local/tmp/fbcode-bpf-objs/biosnoop_bpf.o -o csv > baseline.csv $ cat baseline.csv file_name,prog_name,verdict,duration,total_insns,total_states,peak_states biosnoop_bpf.o,blk_account_io_merge_bio,success,36,24,1,1 biosnoop_bpf.o,blk_account_io_start,failure,0,0,0,0 biosnoop_bpf.o,block_rq_complete,success,82,104,6,6 biosnoop_bpf.o,block_rq_insert,success,78,85,7,7 biosnoop_bpf.o,block_rq_issue,success,74,85,7,7 # on latest bpf-next kernel $ sudo ./veristat biosnoop_bpf.o File Program Verdict Duration (us) Total insns Total states Peak states -------------- ------------------------ ------- ------------- ----------- ------------ ----------- biosnoop_bpf.o blk_account_io_merge_bio success 31 24 1 1 biosnoop_bpf.o blk_account_io_start failure 0 0 0 0 biosnoop_bpf.o block_rq_complete success 76 104 6 6 biosnoop_bpf.o block_rq_insert success 83 91 7 7 biosnoop_bpf.o block_rq_issue success 74 91 7 7 -------------- ------------------------ ------- ------------- ----------- ------------ ----------- Done. Processed 1 object files, 5 programs. $ sudo ./veristat biosnoop_bpf.o -o csv > comparison.csv $ cat comparison.csv file_name,prog_name,verdict,duration,total_insns,total_states,peak_states biosnoop_bpf.o,blk_account_io_merge_bio,success,71,24,1,1 biosnoop_bpf.o,blk_account_io_start,failure,0,0,0,0 biosnoop_bpf.o,block_rq_complete,success,82,104,6,6 biosnoop_bpf.o,block_rq_insert,success,83,91,7,7 biosnoop_bpf.o,block_rq_issue,success,87,91,7,7 # now let's compare with human-readable output (note that no sudo needed) # we also ignore verification duration in this case to shortned output $ ./veristat -C baseline.csv comparison.csv -e file,prog,verdict,insns File Program Verdict (A) Verdict (B) Verdict (DIFF) Total insns (A) Total insns (B) Total insns (DIFF) -------------- ------------------------ ----------- ----------- -------------- --------------- --------------- ------------------ biosnoop_bpf.o blk_account_io_merge_bio success success MATCH 24 24 +0 (+0.00%) biosnoop_bpf.o blk_account_io_start failure failure MATCH 0 0 +0 (+100.00%) biosnoop_bpf.o block_rq_complete success success MATCH 104 104 +0 (+0.00%) biosnoop_bpf.o block_rq_insert success success MATCH 91 85 -6 (-6.59%) biosnoop_bpf.o block_rq_issue success success MATCH 91 85 -6 (-6.59%) -------------- ------------------------ ----------- ----------- -------------- --------------- --------------- ------------------ While not particularly exciting example (it turned out to be kind of hard to quickly find a nice example with significant difference just because of kernel version bump), it should demonstrate main features. Signed-off-by:
-
Andrii Nakryiko authored
Teach veristat to output results as CSV table for easier programmatic processing. Change what was --output/-o argument to now be --emit/-e. And then use --output-format/-o <fmt> to specify output format. Currently "table" and "csv" is supported, table being default. For CSV output mode veristat is using spec identifiers as column names. E.g., instead of "Total states" veristat uses "total_states" as a CSV header name. Internally veristat recognizes three formats, one of them (RESFMT_TABLE_CALCLEN) is a special format instructing veristat to calculate column widths for table output. This felt a bit cleaner and more uniform than either creating separate functions just for this. Signed-off-by:
-
Andrii Nakryiko authored
bpf_object__close(obj) is called twice for BPF object files with single BPF program in it. This causes crash. Fix this by not calling bpf_object__close() unnecessarily. Fixes: c8bc5e05 ("selftests/bpf: Add veristat tool for mass-verifying BPF object files") Signed-off-by:
-
Alexei Starovoitov authored
Lorenzo Bianconi says: ==================== Introduce bpf_ct_set_nat_info kfunc helper in order to set source and destination nat addresses/ports in a new allocated ct entry not inserted in the connection tracking table yet. Introduce support for per-parameter trusted args. Changes since v2: - use int instead of a pointer for port in bpf_ct_set_nat_info signature - modify KF_TRUSTED_ARGS definition in order to referenced pointer constraint just for PTR_TO_BTF_ID - drop patch 2/4 Changes since v1: - enable CONFIG_NF_NAT in tools/testing/selftests/bpf/config Kumar Kartikeya Dwivedi (1): bpf: Tweak definition of KF_TRUSTED_ARGS ==================== Signed-off-by: -
Lorenzo Bianconi authored
Introduce self-tests for bpf_ct_set_nat_info kfunc used to set the source or destination nat addresses/ports. Signed-off-by:
Lorenzo Bianconi <lorenzo@kernel.org> Link: https://lore.kernel.org/r/803e33294e247744d466943105879414344d3235.1663778601.git.lorenzo@kernel.orgSigned-off-by:
-
Lorenzo Bianconi authored
Introduce bpf_ct_set_nat_info kfunc helper in order to set source and destination nat addresses/ports in a new allocated ct entry not inserted in the connection tracking table yet. Signed-off-by:
-
Kumar Kartikeya Dwivedi authored
Instead of forcing all arguments to be referenced pointers with non-zero reg->ref_obj_id, tweak the definition of KF_TRUSTED_ARGS to mean that only PTR_TO_BTF_ID (and socket types translated to PTR_TO_BTF_ID) have that constraint, and require their offset to be set to 0. The rest of pointer types are also accomodated in this definition of trusted pointers, but with more relaxed rules regarding offsets. The inherent meaning of setting this flag is that all kfunc pointer arguments have a guranteed lifetime, and kernel object pointers (PTR_TO_BTF_ID, PTR_TO_CTX) are passed in their unmodified form (with offset 0). In general, this is not true for PTR_TO_BTF_ID as it can be obtained using pointer walks. Signed-off-by:
Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by:
-
Hou Tao authored
For a non-preallocated hash map on RT kernel, regular spinlock instead of raw spinlock is used for bucket lock. The reason is that on RT kernel memory allocation is forbidden under atomic context and regular spinlock is sleepable under RT. Now hash map has been fully converted to use bpf_map_alloc, and there will be no synchronous memory allocation for non-preallocated hash map, so it is safe to always use raw spinlock for bucket lock on RT. So removing the usage of htab_use_raw_lock() and updating the comments accordingly. Signed-off-by:
-
Jiri Olsa authored
We got report from sysbot [1] about warnings that were caused by bpf program attached to contention_begin raw tracepoint triggering the same tracepoint by using bpf_trace_printk helper that takes trace_printk_lock lock. Call Trace: <TASK> ? trace_event_raw_event_bpf_trace_printk+0x5f/0x90 bpf_trace_printk+0x2b/0xe0 bpf_prog_a9aec6167c091eef_prog+0x1f/0x24 bpf_trace_run2+0x26/0x90 native_queued_spin_lock_slowpath+0x1c6/0x2b0 _raw_spin_lock_irqsave+0x44/0x50 bpf_trace_printk+0x3f/0xe0 bpf_prog_a9aec6167c091eef_prog+0x1f/0x24 bpf_trace_run2+0x26/0x90 native_queued_spin_lock_slowpath+0x1c6/0x2b0 _raw_spin_lock_irqsave+0x44/0x50 bpf_trace_printk+0x3f/0xe0 bpf_prog_a9aec6167c091eef_prog+0x1f/0x24 bpf_trace_run2+0x26/0x90 native_queued_spin_lock_slowpath+0x1c6/0x2b0 _raw_spin_lock_irqsave+0x44/0x50 bpf_trace_printk+0x3f/0xe0 bpf_prog_a9aec6167c091eef_prog+0x1f/0x24 bpf_trace_run2+0x26/0x90 native_queued_spin_lock_slowpath+0x1c6/0x2b0 _raw_spin_lock_irqsave+0x44/0x50 __unfreeze_partials+0x5b/0x160 ... The can be reproduced by attaching bpf program as raw tracepoint on contention_begin tracepoint. The bpf prog calls bpf_trace_printk helper. Then by running perf bench the spin lock code is forced to take slow path and call contention_begin tracepoint. Fixing this by skipping execution of the bpf program if it's already running, Using bpf prog 'active' field, which is being currently used by trampoline programs for the same reason. Moving bpf_prog_inc_misses_counter to syscall.c because trampoline.c is compiled in just for CONFIG_BPF_JIT option. Reviewed-by:
Stanislav Fomichev <sdf@google.com> Reported-by: syzbot+2251879aa068ad9c960d@syzkaller.appspotmail.com [1] https://lore.kernel.org/bpf/YxhFe3EwqchC%2FfYf@krava/T/#tSigned-off-by:
-
Alexei Starovoitov authored
Roberto Sassu says: ==================== One of the desirable features in security is the ability to restrict import of data to a given system based on data authenticity. If data import can be restricted, it would be possible to enforce a system-wide policy based on the signing keys the system owner trusts. This feature is widely used in the kernel. For example, if the restriction is enabled, kernel modules can be plugged in only if they are signed with a key whose public part is in the primary or secondary keyring. For eBPF, it can be useful as well. For example, it might be useful to authenticate data an eBPF program makes security decisions on. After a discussion in the eBPF mailing list, it was decided that the stated goal should be accomplished by introducing four new kfuncs: bpf_lookup_user_key() and bpf_lookup_system_key(), for retrieving a keyring with keys trusted for signature verification, respectively from its serial and from a pre-determined ID; bpf_key_put(), to release the reference obtained with the former two kfuncs, bpf_verify_pkcs7_signature(), for verifying PKCS#7 signatures. Other than the key serial, bpf_lookup_user_key() also accepts key lookup flags, that influence the behavior of the lookup. bpf_lookup_system_key() accepts pre-determined IDs defined in include/linux/verification.h. bpf_key_put() accepts the new bpf_key structure, introduced to tell whether the other structure member, a key pointer, is valid or not. The reason is that verify_pkcs7_signature() also accepts invalid pointers, set with the pre-determined ID, to select a system-defined keyring. key_put() must be called only for valid key pointers. Since the two key lookup functions allocate memory and one increments a key reference count, they must be used in conjunction with bpf_key_put(). The latter must be called only if the lookup functions returned a non-NULL pointer. The verifier denies the execution of eBPF programs that don't respect this rule. The two key lookup functions should be used in alternative, depending on the use case. While bpf_lookup_user_key() provides great flexibility, it seems suboptimal in terms of security guarantees, as even if the eBPF program is assumed to be trusted, the serial used to obtain the key pointer might come from untrusted user space not choosing one that the system administrator approves to enforce a mandatory policy. bpf_lookup_system_key() instead provides much stronger guarantees, especially if the pre-determined ID is not passed by user space but is hardcoded in the eBPF program, and that program is signed. In this case, bpf_verify_pkcs7_signature() will always perform signature verification with a key that the system administrator approves, i.e. the primary, secondary or platform keyring. Nevertheless, key permission checks need to be done accurately. Since bpf_lookup_user_key() cannot determine how a key will be used by other kfuncs, it has to defer the permission check to the actual kfunc using the key. It does it by calling lookup_user_key() with KEY_DEFER_PERM_CHECK as needed permission. Later, bpf_verify_pkcs7_signature(), if called, completes the permission check by calling key_validate(). It does not need to call key_task_permission() with permission KEY_NEED_SEARCH, as it is already done elsewhere by the key subsystem. Future kfuncs using the bpf_key structure need to implement the proper checks as well. Finally, the last kfunc, bpf_verify_pkcs7_signature(), accepts the data and signature to verify as eBPF dynamic pointers, to minimize the number of kfunc parameters, and the keyring with keys for signature verification as a bpf_key structure, returned by one of the two key lookup functions. bpf_lookup_user_key() and bpf_verify_pkcs7_signature() can be called only from sleepable programs, because of memory allocation and crypto operations. For example, the lsm.s/bpf attach point is suitable, fexit/array_map_update_elem is not. The correctness of implementation of the new kfuncs and of their usage is checked with the introduced tests. The patch set includes a patch from another author (dependency) for sake of completeness. It is organized as follows. Patch 1 from KP Singh allows kfuncs to be used by LSM programs. Patch 2 exports the bpf_dynptr definition through BTF. Patch 3 splits is_dynptr_reg_valid_init() and introduces is_dynptr_type_expected(), to know more precisely the cause of a negative result of a dynamic pointer check. Patch 4 allows dynamic pointers to be used as kfunc parameters. Patch 5 exports bpf_dynptr_get_size(), to obtain the real size of data carried by a dynamic pointer. Patch 6 makes available for new eBPF kfuncs and programs some key-related definitions. Patch 7 introduces the bpf_lookup_*_key() and bpf_key_put() kfuncs. Patch 8 introduces the bpf_verify_pkcs7_signature() kfunc. Patch 9 changes the testing kernel configuration to compile everything as built-in. Finally, patches 10-13 introduce the tests. Changelog v17: - Remove unnecessary typedefs in test_verify_pkcs7_sig.c (suggested by KP) - Add patch to export bpf_dynptr through BTF (reported by KP) - Rename u{8,16,32,64} variables to __u{8,16,32,64} in the tests, for consistency with other eBPF programs (suggested by Yonghong) v16: - Remove comments in include/linux/key.h for KEY_LOOKUP_* - Change kmalloc() flag from GFP_ATOMIC to GFP_KERNEL in bpf_lookup_user_key(), as the kfunc needs anyway to be sleepable (suggested by Kumar) - Test passing a dynamic pointer with NULL data to bpf_verify_pkcs7_signature() (suggested by Kumar) v15: - Add kfunc_dynptr_param test to deny list for s390x v14: - Explain that is_dynptr_type_expected() will be useful also for BTF (suggested by Joanne) - Rename KEY_LOOKUP_FLAGS_ALL to KEY_LOOKUP_ALL (suggested by Jarkko) - Swap declaration of spi and dynptr_type in is_dynptr_type_expected() (suggested by Joanne) - Reimplement kfunc dynptr tests with a regular eBPF program instead of executing them with test_verifier (suggested by Joanne) - Make key lookup flags as enum so that they are automatically exported through BTF (suggested by Alexei) v13: - Split is_dynptr_reg_valid_init() and introduce is_dynptr_type_expected() to see if the dynamic pointer type passed as argument to a kfunc is supported (suggested by Kumar) - Add forward declaration of struct key in include/linux/bpf.h (suggested by Song) - Declare mask for key lookup flags, remove key_lookup_flags_check() (suggested by Jarkko and KP) - Allow only certain dynamic pointer types (currently, local) to be passed as argument to kfuncs (suggested by Kumar) - For each dynamic pointer parameter in kfunc, additionally check if the passed pointer is to the stack (suggested by Kumar) - Split the validity/initialization and dynamic pointer type check also in the verifier, and adjust the expected error message in the test (a test for an unexpected dynptr type passed to a helper cannot be added due to missing suitable helpers, but this case has been tested manually) - Add verifier tests to check the dynamic pointers passed as argument to kfuncs (suggested by Kumar) v12: - Put lookup_key and verify_pkcs7_sig tests in deny list for s390x (JIT does not support calling kernel function) v11: - Move stringify_struct() macro to include/linux/btf.h (suggested by Daniel) - Change kernel configuration options in tools/testing/selftests/bpf/config* from =m to =y v10: - Introduce key_lookup_flags_check() and system_keyring_id_check() inline functions to check parameters (suggested by KP) - Fix descriptions and comment of key-related kfuncs (suggested by KP) - Register kfunc set only once (suggested by Alexei) - Move needed kernel options to the architecture-independent configuration for testing v9: - Drop patch to introduce KF_SLEEPABLE kfunc flag (already merged) - Rename valid_ptr member of bpf_key to has_ref (suggested by Daniel) - Check dynamic pointers in kfunc definition with bpf_dynptr_kern struct definition instead of string, to detect structure renames (suggested by Daniel) - Explicitly say that we permit initialized dynamic pointers in kfunc definition (suggested by Daniel) - Remove noinline __weak from kfuncs definition (reported by Daniel) - Simplify key lookup flags check in bpf_lookup_user_key() (suggested by Daniel) - Explain the reason for deferring key permission check (suggested by Daniel) - Allocate memory with GFP_ATOMIC in bpf_lookup_system_key(), and remove KF_SLEEPABLE kfunc flag from kfunc declaration (suggested by Daniel) - Define only one kfunc set and remove the loop for registration (suggested by Alexei) v8: - Define the new bpf_key structure to carry the key pointer and whether that pointer is valid or not (suggested by Daniel) - Drop patch to mark a kfunc parameter with the __maybe_null suffix - Improve documentation of kfuncs - Introduce bpf_lookup_system_key() to obtain a key pointer suitable for verify_pkcs7_signature() (suggested by Daniel) - Use the new kfunc registration API - Drop patch to test the __maybe_null suffix - Add tests for bpf_lookup_system_key() v7: - Add support for using dynamic and NULL pointers in kfunc (suggested by Alexei) - Add new kfunc-related tests v6: - Switch back to key lookup helpers + signature verification (until v5), and defer permission check from bpf_lookup_user_key() to bpf_verify_pkcs7_signature() - Add additional key lookup test to illustrate the usage of the KEY_LOOKUP_CREATE flag and validate the flags (suggested by Daniel) - Make description of flags of bpf_lookup_user_key() more user-friendly (suggested by Daniel) - Fix validation of flags parameter in bpf_lookup_user_key() (reported by Daniel) - Rename bpf_verify_pkcs7_signature() keyring-related parameters to user_keyring and system_keyring to make their purpose more clear - Accept keyring-related parameters of bpf_verify_pkcs7_signature() as alternatives (suggested by KP) - Replace unsigned long type with u64 in helper declaration (suggested by Daniel) - Extend the bpf_verify_pkcs7_signature() test by calling the helper without data, by ensuring that the helper enforces the keyring-related parameters as alternatives, by ensuring that the helper rejects inaccessible and expired keyrings, and by checking all system keyrings - Move bpf_lookup_user_key() and bpf_key_put() usage tests to ref_tracking.c (suggested by John) - Call bpf_lookup_user_key() and bpf_key_put() only in sleepable programs v5: - Move KEY_LOOKUP_ to include/linux/key.h for validation of bpf_verify_pkcs7_signature() parameter - Remove bpf_lookup_user_key() and bpf_key_put() helpers, and the corresponding tests - Replace struct key parameter of bpf_verify_pkcs7_signature() with the keyring serial and lookup flags - Call lookup_user_key() and key_put() in bpf_verify_pkcs7_signature() code, to ensure that the retrieved key is used according to the permission requested at lookup time - Clarified keyring precedence in the description of bpf_verify_pkcs7_signature() (suggested by John) - Remove newline in the second argument of ASSERT_ - Fix helper prototype regular expression in bpf_doc.py v4: - Remove bpf_request_key_by_id(), don't return an invalid pointer that other helpers can use - Pass the keyring ID (without ULONG_MAX, suggested by Alexei) to bpf_verify_pkcs7_signature() - Introduce bpf_lookup_user_key() and bpf_key_put() helpers (suggested by Alexei) - Add lookup_key_norelease test, to ensure that the verifier blocks eBPF programs which don't decrement the key reference count - Parse raw PKCS#7 signature instead of module-style signature in the verify_pkcs7_signature test (suggested by Alexei) - Parse kernel module in user space and pass raw PKCS#7 signature to the eBPF program for signature verification v3: - Rename bpf_verify_signature() back to bpf_verify_pkcs7_signature() to avoid managing different parameters for each signature verification function in one helper (suggested by Daniel) - Use dynamic pointers and export bpf_dynptr_get_size() (suggested by Alexei) - Introduce bpf_request_key_by_id() to give more flexibility to the caller of bpf_verify_pkcs7_signature() to retrieve the appropriate keyring (suggested by Alexei) - Fix test by reordering the gcc command line, always compile sign-file - Improve helper support check mechanism in the test v2: - Rename bpf_verify_pkcs7_signature() to a more generic bpf_verify_signature() and pass the signature type (suggested by KP) - Move the helper and prototype declaration under #ifdef so that user space can probe for support for the helper (suggested by Daniel) - Describe better the keyring types (suggested by Daniel) - Include linux/bpf.h instead of vmlinux.h to avoid implicit or redeclaration - Make the test selfcontained (suggested by Alexei) v1: - Don't define new map flag but introduce simple wrapper of verify_pkcs7_signature() (suggested by Alexei and KP) ==================== Signed-off-by: -
Roberto Sassu authored
Add tests to ensure that only supported dynamic pointer types are accepted, that the passed argument is actually a dynamic pointer, that the passed argument is a pointer to the stack, and that bpf_verify_pkcs7_signature() correctly handles dynamic pointers with data set to NULL. The tests are currently in the deny list for s390x (JIT does not support calling kernel function). Signed-off-by:
Roberto Sassu <roberto.sassu@huawei.com> Acked-by:
-
Roberto Sassu authored
Perform several tests to ensure the correct implementation of the bpf_verify_pkcs7_signature() kfunc. Do the tests with data signed with a generated testing key (by using sign-file from scripts/) and with the tcp_bic.ko kernel module if it is found in the system. The test does not fail if tcp_bic.ko is not found. First, perform an unsuccessful signature verification without data. Second, perform a successful signature verification with the session keyring and a new one created for testing. Then, ensure that permission and validation checks are done properly on the keyring provided to bpf_verify_pkcs7_signature(), despite those checks were deferred at the time the keyring was retrieved with bpf_lookup_user_key(). The tests expect to encounter an error if the Search permission is removed from the keyring, or the keyring is expired. Finally, perform a successful and unsuccessful signature verification with the keyrings with pre-determined IDs (the last test fails because the key is not in the platform keyring). The test is currently in the deny list for s390x (JIT does not support calling kernel function). Signed-off-by:
-
Roberto Sassu authored
Add a test to ensure that bpf_lookup_user_key() creates a referenced special keyring when the KEY_LOOKUP_CREATE flag is passed to this function. Ensure that the kfunc rejects invalid flags. Ensure that a keyring can be obtained from bpf_lookup_system_key() when one of the pre-determined keyring IDs is provided. The test is currently blacklisted for s390x (JIT does not support calling kernel function). Signed-off-by:
-
Roberto Sassu authored
Add verifier tests for bpf_lookup_*_key() and bpf_key_put(), to ensure that acquired key references stored in the bpf_key structure are released, that a non-NULL bpf_key pointer is passed to bpf_key_put(), and that key references are not leaked. Also, slightly modify test_verifier.c, to find the BTF ID of the attach point for the LSM program type (currently, it is done only for TRACING). Signed-off-by:
-
Roberto Sassu authored
Since the eBPF CI does not support kernel modules, change the kernel config to compile everything as built-in. Signed-off-by:
Daniel Müller <deso@posteo.net> Acked-by:
-
Roberto Sassu authored
Add the bpf_verify_pkcs7_signature() kfunc, to give eBPF security modules the ability to check the validity of a signature against supplied data, by using user-provided or system-provided keys as trust anchor. The new kfunc makes it possible to enforce mandatory policies, as eBPF programs might be allowed to make security decisions only based on data sources the system administrator approves. The caller should provide the data to be verified and the signature as eBPF dynamic pointers (to minimize the number of parameters) and a bpf_key structure containing a reference to the keyring with keys trusted for signature verification, obtained from bpf_lookup_user_key() or bpf_lookup_system_key(). For bpf_key structures obtained from the former lookup function, bpf_verify_pkcs7_signature() completes the permission check deferred by that function by calling key_validate(). key_task_permission() is already called by the PKCS#7 code. Signed-off-by:
Song Liu <song@kernel.org> Link: https://lore.kernel.org/r/20220920075951.929132-9-roberto.sassu@huaweicloud.comSigned-off-by:
-
Roberto Sassu authored
Add the bpf_lookup_user_key(), bpf_lookup_system_key() and bpf_key_put() kfuncs, to respectively search a key with a given key handle serial number and flags, obtain a key from a pre-determined ID defined in include/linux/verification.h, and cleanup. Introduce system_keyring_id_check() to validate the keyring ID parameter of bpf_lookup_system_key(). Signed-off-by:
-
Roberto Sassu authored
In preparation for the patch that introduces the bpf_lookup_user_key() eBPF kfunc, move KEY_LOOKUP_ definitions to include/linux/key.h, to be able to validate the kfunc parameters. Add them to enum key_lookup_flag, so that all the current ones and the ones defined in the future are automatically exported through BTF and available to eBPF programs. Also, add KEY_LOOKUP_ALL to the enum, with the logical OR of currently defined flags as value, to facilitate checking whether a variable contains only those flags. Signed-off-by:
Jarkko Sakkinen <jarkko@kernel.org> Link: https://lore.kernel.org/r/20220920075951.929132-7-roberto.sassu@huaweicloud.comSigned-off-by:
-
