1. 08 Mar, 2022 3 commits
  2. 06 Mar, 2022 5 commits
    • Alexei Starovoitov's avatar
      Merge branch 'bpf: add __percpu tagging in vmlinux BTF' · c344b9fc
      Alexei Starovoitov authored
      Hao Luo says:
      
      ====================
      
      This patchset is very much similar to Yonghong's patchset on adding
      __user tagging [1], where a "user" btf_type_tag was introduced to
      describe __user memory pointers. Similar approach can be applied on
      __percpu pointers. The __percpu attribute in kernel is used to identify
      pointers that point to memory allocated in percpu region. Normally,
      accessing __percpu memory requires using special functions like
      per_cpu_ptr() etc. Directly accessing __percpu pointer is meaningless.
      
      Currently vmlinux BTF does not have a way to differentiate a __percpu
      pointer from a regular pointer. So BPF programs are allowed to load
      __percpu memory directly, which is an incorrect behavior.
      
      With the previous work that encodes __user information in BTF, a nice
      framework has been set up to allow us to encode __percpu information in
      BTF and let the verifier to reject programs that try to directly access
      percpu pointer. Previously, there is a PTR_TO_PERCPU_BTF_ID reg type which
      is used to represent those percpu static variables in the kernel. Pahole
      is able to collect variables that are stored in ".data..percpu" section
      in the kernel image and emit BTF information for those variables. The
      bpf_per_cpu_ptr() and bpf_this_cpu_ptr() helper functions were added to
      access these variables. Now with __percpu information, we can tag those
      __percpu fields in a struct (such as cgroup->rstat_cpu) and allow the
      pair of bpf percpu helpers to access them as well.
      
      In addition to adding __percpu tagging, this patchset also fixes a
      harmless bug in the previous patch that introduced __user. Patch 01/04
      is for that. Patch 02/04 adds the new attribute "percpu". Patch 03/04
      adds MEM_PERCPU tag for PTR_TO_BTF_ID and replaces PTR_TO_PERCPU_BTF_ID
      with (BTF_ID | MEM_PERCPU). Patch 04/04 refactors the btf_tag test a bit
      and adds tests for percpu tag.
      
      Like [1], the minimal requirements for btf_type_tag is
      clang (>= clang14) and pahole (>= 1.23).
      
      [1] https://lore.kernel.org/bpf/20211220015110.3rqxk5qwub3pa2gh@ast-mbp.dhcp.thefacebook.com/t/
      ====================
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      c344b9fc
    • Hao Luo's avatar
      selftests/bpf: Add a test for btf_type_tag "percpu" · 50c6b8a9
      Hao Luo authored
      Add test for percpu btf_type_tag. Similar to the "user" tag, we test
      the following cases:
      
       1. __percpu struct field.
       2. __percpu as function parameter.
       3. per_cpu_ptr() accepts dynamically allocated __percpu memory.
      
      Because the test for "user" and the test for "percpu" are very similar,
      a little bit of refactoring has been done in btf_tag.c. Basically, both
      tests share the same function for loading vmlinux and module btf.
      
      Example output from log:
      
       > ./test_progs -v -t btf_tag
      
       libbpf: prog 'test_percpu1': BPF program load failed: Permission denied
       libbpf: prog 'test_percpu1': -- BEGIN PROG LOAD LOG --
       ...
       ; g = arg->a;
       1: (61) r1 = *(u32 *)(r1 +0)
       R1 is ptr_bpf_testmod_btf_type_tag_1 access percpu memory: off=0
       ...
       test_btf_type_tag_mod_percpu:PASS:btf_type_tag_percpu 0 nsec
       #26/6 btf_tag/btf_type_tag_percpu_mod1:OK
      
       libbpf: prog 'test_percpu2': BPF program load failed: Permission denied
       libbpf: prog 'test_percpu2': -- BEGIN PROG LOAD LOG --
       ...
       ; g = arg->p->a;
       2: (61) r1 = *(u32 *)(r1 +0)
       R1 is ptr_bpf_testmod_btf_type_tag_1 access percpu memory: off=0
       ...
       test_btf_type_tag_mod_percpu:PASS:btf_type_tag_percpu 0 nsec
       #26/7 btf_tag/btf_type_tag_percpu_mod2:OK
      
       libbpf: prog 'test_percpu_load': BPF program load failed: Permission denied
       libbpf: prog 'test_percpu_load': -- BEGIN PROG LOAD LOG --
       ...
       ; g = (__u64)cgrp->rstat_cpu->updated_children;
       2: (79) r1 = *(u64 *)(r1 +48)
       R1 is ptr_cgroup_rstat_cpu access percpu memory: off=48
       ...
       test_btf_type_tag_vmlinux_percpu:PASS:btf_type_tag_percpu_load 0 nsec
       #26/8 btf_tag/btf_type_tag_percpu_vmlinux_load:OK
      
       load_btfs:PASS:could not load vmlinux BTF 0 nsec
       test_btf_type_tag_vmlinux_percpu:PASS:btf_type_tag_percpu 0 nsec
       test_btf_type_tag_vmlinux_percpu:PASS:btf_type_tag_percpu_helper 0 nsec
       #26/9 btf_tag/btf_type_tag_percpu_vmlinux_helper:OK
      Signed-off-by: default avatarHao Luo <haoluo@google.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarYonghong Song <yhs@fb.com>
      Link: https://lore.kernel.org/bpf/20220304191657.981240-5-haoluo@google.com
      50c6b8a9
    • Hao Luo's avatar
      bpf: Reject programs that try to load __percpu memory. · 5844101a
      Hao Luo authored
      With the introduction of the btf_type_tag "percpu", we can add a
      MEM_PERCPU to identify those pointers that point to percpu memory.
      The ability of differetiating percpu pointers from regular memory
      pointers have two benefits:
      
       1. It forbids unexpected use of percpu pointers, such as direct loads.
          In kernel, there are special functions used for accessing percpu
          memory. Directly loading percpu memory is meaningless. We already
          have BPF helpers like bpf_per_cpu_ptr() and bpf_this_cpu_ptr() that
          wrap the kernel percpu functions. So we can now convert percpu
          pointers into regular pointers in a safe way.
      
       2. Previously, bpf_per_cpu_ptr() and bpf_this_cpu_ptr() only work on
          PTR_TO_PERCPU_BTF_ID, a special reg_type which describes static
          percpu variables in kernel (we rely on pahole to encode them into
          vmlinux BTF). Now, since we can identify __percpu tagged pointers,
          we can also identify dynamically allocated percpu memory as well.
          It means we can use bpf_xxx_cpu_ptr() on dynamic percpu memory.
          This would be very convenient when accessing fields like
          "cgroup->rstat_cpu".
      Signed-off-by: default avatarHao Luo <haoluo@google.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarYonghong Song <yhs@fb.com>
      Link: https://lore.kernel.org/bpf/20220304191657.981240-4-haoluo@google.com
      5844101a
    • Hao Luo's avatar
      compiler_types: Define __percpu as __attribute__((btf_type_tag("percpu"))) · 9216c916
      Hao Luo authored
      This is similar to commit 7472d5a6 ("compiler_types: define __user as
      __attribute__((btf_type_tag("user")))"), where a type tag "user" was
      introduced to identify the pointers that point to user memory. With that
      change, the newest compile toolchain can encode __user information into
      vmlinux BTF, which can be used by the BPF verifier to enforce safe
      program behaviors.
      
      Similarly, we have __percpu attribute, which is mainly used to indicate
      memory is allocated in percpu region. The __percpu pointers in kernel
      are supposed to be used together with functions like per_cpu_ptr() and
      this_cpu_ptr(), which perform necessary calculation on the pointer's
      base address. Without the btf_type_tag introduced in this patch,
      __percpu pointers will be treated as regular memory pointers in vmlinux
      BTF and BPF programs are allowed to directly dereference them, generating
      incorrect behaviors. Now with "percpu" btf_type_tag, the BPF verifier is
      able to differentiate __percpu pointers from regular pointers and forbids
      unexpected behaviors like direct load.
      
      The following is an example similar to the one given in commit
      7472d5a6:
      
        [$ ~] cat test.c
        #define __percpu __attribute__((btf_type_tag("percpu")))
        int foo(int __percpu *arg) {
        	return *arg;
        }
        [$ ~] clang -O2 -g -c test.c
        [$ ~] pahole -JV test.o
        ...
        File test.o:
        [1] INT int size=4 nr_bits=32 encoding=SIGNED
        [2] TYPE_TAG percpu type_id=1
        [3] PTR (anon) type_id=2
        [4] FUNC_PROTO (anon) return=1 args=(3 arg)
        [5] FUNC foo type_id=4
        [$ ~]
      
      for the function argument "int __percpu *arg", its type is described as
      	PTR -> TYPE_TAG(percpu) -> INT
      The kernel can use this information for bpf verification or other
      use cases.
      
      Like commit 7472d5a6, this feature requires clang (>= clang14) and
      pahole (>= 1.23).
      Signed-off-by: default avatarHao Luo <haoluo@google.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarYonghong Song <yhs@fb.com>
      Link: https://lore.kernel.org/bpf/20220304191657.981240-3-haoluo@google.com
      9216c916
    • Hao Luo's avatar
      bpf: Fix checking PTR_TO_BTF_ID in check_mem_access · bff61f6f
      Hao Luo authored
      With the introduction of MEM_USER in
      
       commit c6f1bfe8 ("bpf: reject program if a __user tagged memory accessed in kernel way")
      
      PTR_TO_BTF_ID can be combined with a MEM_USER tag. Therefore, most
      likely, when we compare reg_type against PTR_TO_BTF_ID, we want to use
      the reg's base_type. Previously the check in check_mem_access() wants
      to say: if the reg is BTF_ID but not NULL, the execution flow falls
      into the 'then' branch. But now a reg of (BTF_ID | MEM_USER), which
      should go into the 'then' branch, goes into the 'else'.
      
      The end results before and after this patch are the same: regs tagged
      with MEM_USER get rejected, but not in a way we intended. So fix the
      condition, the error message now is correct.
      
      Before (log from commit 696c3901):
      
        $ ./test_progs -v -n 22/3
        ...
        libbpf: prog 'test_user1': BPF program load failed: Permission denied
        libbpf: prog 'test_user1': -- BEGIN PROG LOAD LOG --
        R1 type=ctx expected=fp
        0: R1=ctx(id=0,off=0,imm=0) R10=fp0
        ; int BPF_PROG(test_user1, struct bpf_testmod_btf_type_tag_1 *arg)
        0: (79) r1 = *(u64 *)(r1 +0)
        func 'bpf_testmod_test_btf_type_tag_user_1' arg0 has btf_id 136561 type STRUCT 'bpf_testmod_btf_type_tag_1'
        1: R1_w=user_ptr_bpf_testmod_btf_type_tag_1(id=0,off=0,imm=0)
        ; g = arg->a;
        1: (61) r1 = *(u32 *)(r1 +0)
        R1 invalid mem access 'user_ptr_'
      
      Now:
      
        libbpf: prog 'test_user1': BPF program load failed: Permission denied
        libbpf: prog 'test_user1': -- BEGIN PROG LOAD LOG --
        R1 type=ctx expected=fp
        0: R1=ctx(id=0,off=0,imm=0) R10=fp0
        ; int BPF_PROG(test_user1, struct bpf_testmod_btf_type_tag_1 *arg)
        0: (79) r1 = *(u64 *)(r1 +0)
        func 'bpf_testmod_test_btf_type_tag_user_1' arg0 has btf_id 104036 type STRUCT 'bpf_testmod_btf_type_tag_1'
        1: R1_w=user_ptr_bpf_testmod_btf_type_tag_1(id=0,ref_obj_id=0,off=0,imm=0)
        ; g = arg->a;
        1: (61) r1 = *(u32 *)(r1 +0)
        R1 is ptr_bpf_testmod_btf_type_tag_1 access user memory: off=0
      
      Note the error message for the reason of rejection.
      
      Fixes: c6f1bfe8 ("bpf: reject program if a __user tagged memory accessed in kernel way")
      Signed-off-by: default avatarHao Luo <haoluo@google.com>
      Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
      Acked-by: default avatarYonghong Song <yhs@fb.com>
      Link: https://lore.kernel.org/bpf/20220304191657.981240-2-haoluo@google.com
      bff61f6f
  3. 05 Mar, 2022 32 commits