- 22 May, 2020 14 commits
-
-
Björn Töpel authored
In order to reduce the number of function calls, the struct xsk_buff_pool definition is moved to xsk_buff_pool.h. The functions xp_get_dma(), xp_dma_sync_for_cpu(), xp_dma_sync_for_device(), xp_validate_desc() and various helper functions are explicitly inlined. Further, move xp_get_handle() and xp_release() to xsk.c, to allow for the compiler to perform inlining. rfc->v1: Make sure xp_validate_desc() is inlined for Tx perf. (Maxim) Signed-off-by:
Björn Töpel <bjorn.topel@intel.com> Signed-off-by:
Alexei Starovoitov <ast@kernel.org> Link: https://lore.kernel.org/bpf/20200520192103.355233-15-bjorn.topel@gmail.com
-
Björn Töpel authored
The xdp_return_{frame,frame_rx_napi,buff} function are never used, except in xdp_convert_zc_to_xdp_frame(), by the MEM_TYPE_XSK_BUFF_POOL memory type. To simplify and reduce code, change so that xdp_convert_zc_to_xdp_frame() calls xsk_buff_free() directly since the type is know, and remove MEM_TYPE_XSK_BUFF_POOL from the switch statement in __xdp_return() function. Suggested-by:Maxim Mikityanskiy <maximmi@mellanox.com> Signed-off-by:
-
Björn Töpel authored
There are no users of MEM_TYPE_ZERO_COPY. Remove all corresponding code, including the "handle" member of struct xdp_buff. rfc->v1: Fixed spelling in commit message. (Björn) Signed-off-by:
-
Björn Töpel authored
Use the new MEM_TYPE_XSK_BUFF_POOL API in lieu of MEM_TYPE_ZERO_COPY in mlx5e. It allows to drop a lot of code from the driver (which is now common in AF_XDP core and was related to XSK RX frame allocation, DMA mapping, etc.) and slightly improve performance (RX +0.8 Mpps, TX +0.4 Mpps). rfc->v1: Put back the sanity check for XSK params, use XSK API to get the total headroom size. (Maxim) v1->v2: Fix DMA address handling, set XDP metadata to invalid. (Maxim) v2->v3: Handle frame_sz, use xsk_buff_xdp_get_frame_dma, use xsk_buff API for DMA sync on TX, add performance numbers. (Maxim) v3->v4: Remove unused variable num_xsk_frames. (Jakub) Signed-off-by: -
Björn Töpel authored
Remove MEM_TYPE_ZERO_COPY in favor of the new MEM_TYPE_XSK_BUFF_POOL APIs. v1->v2: Fixed xdp_buff data_end update. (Björn) Signed-off-by:
-
Björn Töpel authored
Remove MEM_TYPE_ZERO_COPY in favor of the new MEM_TYPE_XSK_BUFF_POOL APIs. v4->v5: Fixed "warning: Excess function parameter 'alloc' description in 'ice_alloc_rx_bufs_zc'" and "warning: Excess function parameter 'xdp' description in 'ice_construct_skb_zc'". (Jakub) Signed-off-by:Maciej Fijalkowski <maciej.fijalkowski@intel.com> Signed-off-by:
-
Björn Töpel authored
Remove MEM_TYPE_ZERO_COPY in favor of the new MEM_TYPE_XSK_BUFF_POOL APIs. The AF_XDP zero-copy rx_bi ring is now simply a struct xdp_buff pointer. v4->v5: Fixed "warning: Excess function parameter 'bi' description in 'i40e_construct_skb_zc'". (Jakub) Signed-off-by: -
Björn Töpel authored
Continuing the path to support MEM_TYPE_XSK_BUFF_POOL, the AF_XDP zero-copy/sk_buff rx_bi rings are now separate. Functions to properly allocate the different rings are added as well. v3->v4: Made i40e_fd_handle_status() static. (kbuild test robot) v4->v5: Fix kdoc for i40e_clean_programming_status(). (Jakub) Signed-off-by:
-
Björn Töpel authored
As a first step to migrate i40e to the new MEM_TYPE_XSK_BUFF_POOL APIs, code that accesses the rx_bi (SW/shadow ring) is refactored to use an accessor function. Signed-off-by:
-
Björn Töpel authored
In order to simplify AF_XDP zero-copy enablement for NIC driver developers, a new AF_XDP buffer allocation API is added. The implementation is based on a single core (single producer/consumer) buffer pool for the AF_XDP UMEM. A buffer is allocated using the xsk_buff_alloc() function, and returned using xsk_buff_free(). If a buffer is disassociated with the pool, e.g. when a buffer is passed to an AF_XDP socket, a buffer is said to be released. Currently, the release function is only used by the AF_XDP internals and not visible to the driver. Drivers using this API should register the XDP memory model with the new MEM_TYPE_XSK_BUFF_POOL type. The API is defined in net/xdp_sock_drv.h. The buffer type is struct xdp_buff, and follows the lifetime of regular xdp_buffs, i.e. the lifetime of an xdp_buff is restricted to a NAPI context. In other words, the API is not replacing xdp_frames. In addition to introducing the API and implementations, the AF_XDP core is migrated to use the new APIs. rfc->v1: Fixed build errors/warnings for m68k and riscv. (kbuild test robot) Added headroom/chunk size getter. (Maxim/Björn) v1->v2: Swapped SoBs. (Maxim) v2->v3: Initialize struct xdp_buff member frame_sz. (Björn) Add API to query the DMA address of a frame. (Maxim) Do DMA sync for CPU till the end of the frame to handle possible growth (frame_sz). (Maxim) Signed-off-by: -
Björn Töpel authored
Move the XSK_NEXT_PG_CONTIG_{MASK,SHIFT}, and XDP_UMEM_USES_NEED_WAKEUP defines from xdp_sock.h to the AF_XDP internal xsk.h file. Also, start using the BIT{,_ULL} macro instead of explicit shifts. Signed-off-by: -
Magnus Karlsson authored
Move the AF_XDP zero-copy driver interface to its own include file called xdp_sock_drv.h. This, hopefully, will make it more clear for NIC driver implementors to know what functions to use for zero-copy support. v4->v5: Fix -Wmissing-prototypes by include header file. (Jakub) Signed-off-by:
Magnus Karlsson <magnus.karlsson@intel.com> Signed-off-by:
-
Björn Töpel authored
The XSKMAP is partly implemented by net/xdp/xsk.c. Move xskmap.c from kernel/bpf/ to net/xdp/, which is the logical place for AF_XDP related code. Also, move AF_XDP struct definitions, and function declarations only used by AF_XDP internals into net/xdp/xsk.h. Signed-off-by:
-
Björn Töpel authored
Calculating the "data_hard_end" for an XDP buffer coming from AF_XDP zero-copy mode, the return value of xsk_umem_xdp_frame_sz() is added to "data_hard_start". Currently, the chunk size of the UMEM is returned by xsk_umem_xdp_frame_sz(). This is not correct, if the fixed UMEM headroom is non-zero. Fix this by returning the chunk_size without the UMEM headroom. Fixes: 2a637c5b ("xdp: For Intel AF_XDP drivers add XDP frame_sz") Signed-off-by:
-
- 19 May, 2020 14 commits
-
-
Andrii Nakryiko authored
b9f4c01f ("selftest/bpf: Make bpf_iter selftest compilable against old vmlinux.h") missed the fact that bpf_iter_test_kern{3,4}.c are not just including bpf_iter_test_kern_common.h and need similar bpf_iter_meta re-definition explicitly. Fixes: b9f4c01f ("selftest/bpf: Make bpf_iter selftest compilable against old vmlinux.h") Signed-off-by:
Andrii Nakryiko <andriin@fb.com> Signed-off-by:
-
Andrii Nakryiko authored
It's good to be able to compile bpf_iter selftest even on systems that don't have the very latest vmlinux.h, e.g., for libbpf tests against older kernels in Travis CI. To that extent, re-define bpf_iter_meta and corresponding bpf_iter context structs in each selftest. To avoid type clashes with vmlinux.h, rename vmlinux.h's definitions to get them out of the way. Signed-off-by:
Yonghong Song <yhs@fb.com> Acked-by:
Jesper Dangaard Brouer <brouer@redhat.com> Link: https://lore.kernel.org/bpf/20200518234516.3915052-1-andriin@fb.com
-
Alexei Starovoitov authored
Sync tools/include/uapi/linux/bpf.h from include/uapi. Signed-off-by: -
Alexei Starovoitov authored
Daniel Borkmann says: ==================== Trivial patch to add get{peer,sock}name cgroup attach types to the BPF sock_addr programs in order to enable rewriting sockaddr structs from both calls along with libbpf and bpftool support as well as selftests. Thanks! v1 -> v2: - use __u16 for ports in start_server_with_port() signature and in expected_{local,peer} ports in the test case (Andrey) - Added both Andrii's and Andrey's ACKs ==================== Signed-off-by: -
Daniel Borkmann authored
Extend the existing connect_force_port test to assert get{peer,sock}name programs as well. The workflow for e.g. IPv4 is as follows: i) server binds to concrete port, ii) client calls getsockname() on server fd which exposes 1.2.3.4:60000 to client, iii) client connects to service address 1.2.3.4:60000 binds to concrete local address (127.0.0.1:22222) and remaps service address to a concrete backend address (127.0.0.1:60123), iv) client then calls getsockname() on its own fd to verify local address (127.0.0.1:22222) and getpeername() on its own fd which then publishes service address (1.2.3.4:60000) instead of actual backend. Same workflow is done for IPv6 just with different address/port tuples. # ./test_progs -t connect_force_port #14 connect_force_port:OK Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by:Daniel Borkmann <daniel@iogearbox.net> Signed-off-by:
Andrey Ignatov <rdna@fb.com> Link: https://lore.kernel.org/bpf/3343da6ad08df81af715a95d61a84fb4a960f2bf.1589841594.git.daniel@iogearbox.net
-
Daniel Borkmann authored
Make bpftool aware and add the new get{peer,sock}name attach types to its cli, documentation and bash completion to allow attachment/detachment of sock_addr programs there. Signed-off-by: -
Daniel Borkmann authored
Trivial patch to add the new get{peer,sock}name attach types to the section definitions in order to hook them up to sock_addr cgroup program type. Signed-off-by: -
Daniel Borkmann authored
As stated in 983695fa ("bpf: fix unconnected udp hooks"), the objective for the existing cgroup connect/sendmsg/recvmsg/bind BPF hooks is to be transparent to applications. In Cilium we make use of these hooks [0] in order to enable E-W load balancing for existing Kubernetes service types for all Cilium managed nodes in the cluster. Those backends can be local or remote. The main advantage of this approach is that it operates as close as possible to the socket, and therefore allows to avoid packet-based NAT given in connect/sendmsg/recvmsg hooks we only need to xlate sock addresses. This also allows to expose NodePort services on loopback addresses in the host namespace, for example. As another advantage, this also efficiently blocks bind requests for applications in the host namespace for exposed ports. However, one missing item is that we also need to perform reverse xlation for inet{,6}_getname() hooks such that we can return the service IP/port tuple back to the application instead of the remote peer address. The vast majority of applications does not bother about getpeername(), but in a few occasions we've seen breakage when validating the peer's address since it returns unexpectedly the backend tuple instead of the service one. Therefore, this trivial patch allows to customise and adds a getpeername() as well as getsockname() BPF cgroup hook for both IPv4 and IPv6 in order to address this situation. Simple example: # ./cilium/cilium service list ID Frontend Service Type Backend 1 1.2.3.4:80 ClusterIP 1 => 10.0.0.10:80 Before; curl's verbose output example, no getpeername() reverse xlation: # curl --verbose 1.2.3.4 * Rebuilt URL to: 1.2.3.4/ * Trying 1.2.3.4... * TCP_NODELAY set * Connected to 1.2.3.4 (10.0.0.10) port 80 (#0) > GET / HTTP/1.1 > Host: 1.2.3.4 > User-Agent: curl/7.58.0 > Accept: */* [...] After; with getpeername() reverse xlation: # curl --verbose 1.2.3.4 * Rebuilt URL to: 1.2.3.4/ * Trying 1.2.3.4... * TCP_NODELAY set * Connected to 1.2.3.4 (1.2.3.4) port 80 (#0) > GET / HTTP/1.1 > Host: 1.2.3.4 > User-Agent: curl/7.58.0 > Accept: */* [...] Originally, I had both under a BPF_CGROUP_INET{4,6}_GETNAME type and exposed peer to the context similar as in inet{,6}_getname() fashion, but API-wise this is suboptimal as it always enforces programs having to test for ctx->peer which can easily be missed, hence BPF_CGROUP_INET{4,6}_GET{PEER,SOCK}NAME split. Similarly, the checked return code is on tnum_range(1, 1), but if a use case comes up in future, it can easily be changed to return an error code instead. Helper and ctx member access is the same as with connect/sendmsg/etc hooks. [0] https://github.com/cilium/cilium/blob/master/bpf/bpf_sock.cSigned-off-by:
-
Jesper Dangaard Brouer authored
Commit bc56c919 ("bpf: Add xdp.frame_sz in bpf_prog_test_run_xdp().") recently changed bpf_prog_test_run_xdp() to use larger frames for XDP in order to test tail growing frames (via bpf_xdp_adjust_tail) and to have memory backing frame better resemble drivers. The commit contains a bug, as it tries to copy the max data size from userspace, instead of the size provided by userspace. This cause XDP unit tests to fail sporadically with EFAULT, an unfortunate behavior. The fix is to only copy the size specified by userspace. Fixes: bc56c919 ("bpf: Add xdp.frame_sz in bpf_prog_test_run_xdp().") Signed-off-by:
-
Daniel T. Lee authored
Because the previous two commit replaced the bpf_load implementation of the user program with libbpf, the corresponding kernel program's MAP definition can be replaced with new BTF-defined map syntax. This commit only updates the samples which uses libbpf API for loading bpf program not with bpf_load. Signed-off-by:
Daniel T. Lee <danieltimlee@gmail.com> Signed-off-by:
-
Daniel T. Lee authored
This commit adds tracex7 test file (testfile.img) to .gitignore which comes from test_override_return.sh. Signed-off-by:
-
Daniel T. Lee authored
BPF tail call uses the BPF_MAP_TYPE_PROG_ARRAY type map for calling into other BPF programs and this PROG_ARRAY should be filled prior to use. Currently, samples with the PROG_ARRAY type MAP fill this program array with bpf_load. For bpf_load to fill this map, kernel BPF program must specify the section with specific format of <prog_type>/<array_idx> (e.g. SEC("socket/0")) But by using libbpf instead of bpf_load, user program can specify which programs should be added to PROG_ARRAY. The advantage of this approach is that you can selectively add only the programs you want, rather than adding all of them to PROG_ARRAY, and it's much more intuitive than the traditional approach. This commit refactors user programs with the PROG_ARRAY type MAP with libbpf instead of using bpf_load. Signed-off-by: -
Daniel T. Lee authored
Currently, the kprobe BPF program attachment method for bpf_load is quite old. The implementation of bpf_load "directly" controls and manages(create, delete) the kprobe events of DEBUGFS. On the other hand, using using the libbpf automatically manages the kprobe event. (under bpf_link interface) By calling bpf_program__attach(_kprobe) in libbpf, the corresponding kprobe is created and the BPF program will be attached to this kprobe. To remove this, by simply invoking bpf_link__destroy will clean up the event. This commit refactors kprobe tracing programs (tracex{1~7}_user.c) with libbpf using bpf_link interface and bpf_program__attach. tracex2_kern.c, which tracks system calls (sys_*), has been modified to append prefix depending on architecture. Signed-off-by: -
Daniel T. Lee authored
Current method of checking pointer error is not user friendly. Especially the __must_check define makes this less intuitive. Since, libbpf has an API libbpf_get_error() which checks pointer error, this commit refactors existing pointer error check logic with libbpf. Signed-off-by:
-
- 16 May, 2020 10 commits
-
-
John Fastabend authored
Until now we have only had minimal ktls+sockmap testing when being used with helpers and different sendmsg/sendpage patterns. Add a pass with ktls here. To run just ktls tests, $ ./test_sockmap --whitelist="ktls" Signed-off-by:
John Fastabend <john.fastabend@gmail.com> Signed-off-by:
Jakub Sitnicki <jakub@cloudflare.com> Link: https://lore.kernel.org/bpf/158939736278.15176.5435314315563203761.stgit@john-Precision-5820-Tower
-
John Fastabend authored
This adds a blacklist to test_sockmap. For example, now we can run all apply and cork tests except those with timeouts by doing, $ ./test_sockmap --whitelist "apply,cork" --blacklist "hang" Signed-off-by:
-
John Fastabend authored
Allow running specific tests with a comma deliminated whitelist. For example to run all apply and cork tests. $ ./test_sockmap --whitelist="cork,apply" Signed-off-by:
-
John Fastabend authored
Pass options from command line args into individual tests which allows us to use verbose option from command line with selftests. Now when verbose option is set individual subtest details will be printed. Also we can consolidate cgroup bring up and tear down. Additionally just setting verbose is very noisy so introduce verbose=1 and verbose=2. Really verbose=2 is only useful when developing tests or debugging some specific issue. For example now we get output like this with --verbose, #20/17 sockhash:txmsg test pull-data:OK [TEST 160]: (512, 1, 3, sendpage, pop (1,3),): msg_loop_rx: iov_count 1 iov_buf 1 cnt 512 err 0 [TEST 161]: (100, 1, 5, sendpage, pop (1,3),): msg_loop_rx: iov_count 1 iov_buf 3 cnt 100 err 0 [TEST 162]: (2, 1024, 256, sendpage, pop (4096,8192),): msg_loop_rx: iov_count 1 iov_buf 255 cnt 2 err 0 [TEST 163]: (512, 1, 3, sendpage, redir,pop (1,3),): msg_loop_rx: iov_count 1 iov_buf 1 cnt 512 err 0 [TEST 164]: (100, 1, 5, sendpage, redir,pop (1,3),): msg_loop_rx: iov_count 1 iov_buf 3 cnt 100 err 0 [TEST 165]: (512, 1, 3, sendpage, cork 512,pop (1,3),): msg_loop_rx: iov_count 1 iov_buf 1 cnt 512 err 0 [TEST 166]: (100, 1, 5, sendpage, cork 512,pop (1,3),): msg_loop_rx: iov_count 1 iov_buf 3 cnt 100 err 0 [TEST 167]: (512, 1, 3, sendpage, redir,cork 4,pop (1,3),): msg_loop_rx: iov_count 1 iov_buf 1 cnt 512 err 0 [TEST 168]: (100, 1, 5, sendpage, redir,cork 4,pop (1,3),): msg_loop_rx: iov_count 1 iov_buf 3 cnt 100 err 0 Signed-off-by:
-
John Fastabend authored
At the moment test_sockmap runs all 800+ tests ungrouped which is not ideal because it makes it hard to see what is failing but also more importantly its hard to confirm all cases are tested. Additionally, after inspecting we noticed the runtime is bloated because we run many duplicate tests. Worse some of these tests are known error cases that wait for the recvmsg handler to timeout which creats long delays. Also we noted some tests were not clearing their options and as a result the following tests would run with extra and incorrect options. Fix this by reorganizing test code so its clear what tests are running and when. Then it becomes easy to remove duplication and run tests with only the set of send/recv patterns that are relavent. To accomplish this break test_sockmap into subtests and remove unnecessary duplication. The output is more readable now and the runtime reduced. Now default output prints subtests like this, $ ./test_sockmap # 1/ 6 sockmap:txmsg test passthrough:OK ... #22/ 1 sockhash:txmsg test push/pop data:OK Pass: 22 Fail: 0 Signed-off-by:
-
John Fastabend authored
The recv thread in test_sockmap waits to receive all bytes from sender but in the case we use pop data it may wait for more bytes then actually being sent. This stalls the test harness for multiple seconds. Because this happens in multiple tests it slows time to run the selftest. Fix by doing a better job of accounting for total bytes when pop helpers are used. Signed-off-by:
-
John Fastabend authored
Its helpful to know the error value if an error occurs. Signed-off-by:
-
John Fastabend authored
Running test_sockmap with arguments to specify a test pattern requires including a cgroup argument. Instead of requiring this if the option is not provided create one This is not used by selftest runs but I use it when I want to test a specific test. Most useful when developing new code and/or tests. Signed-off-by:
-
John Fastabend authored
The prints in the test_sockmap programs were only useful when we didn't have enough control over test infrastructure to know from user program what was being pushed into kernel side. Now that we have or will shortly have better test controls lets remove the printers. This means we can remove half the programs and cleanup bpf side. Signed-off-by:
-
John Fastabend authored
Moves test_sockmap_kern.h into progs directory but does not change code at all. Signed-off-by:
-
- 15 May, 2020 2 commits
-
-
Stanislav Fomichev authored
There is a much higher chance we can see the regressions if the test is part of test_progs. Signed-off-by:
Stanislav Fomichev <sdf@google.com> Signed-off-by:
-
Stanislav Fomichev authored
Commit 294f2fc6 ("bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()") changed the way verifier logs some of its state, adjust the test_align accordingly. Where possible, I tried to not copy-paste the entire log line and resorted to dropping the last closing brace instead. Fixes: 294f2fc6 ("bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()") Signed-off-by:
-
