• Manu Bretelle's avatar
    selftests/bpf: Add --json-summary option to test_progs · 2be7aa76
    Manu Bretelle authored
    Currently, test_progs outputs all stdout/stderr as it runs, and when it
    is done, prints a summary.
    
    It is non-trivial for tooling to parse that output and extract meaningful
    information from it.
    
    This change adds a new option, `--json-summary`/`-J` that let the caller
    specify a file where `test_progs{,-no_alu32}` can write a summary of the
    run in a json format that can later be parsed by tooling.
    
    Currently, it creates a summary section with successes/skipped/failures
    followed by a list of failed tests and subtests.
    
    A test contains the following fields:
    - name: the name of the test
    - number: the number of the test
    - message: the log message that was printed by the test.
    - failed: A boolean indicating whether the test failed or not. Currently
    we only output failed tests, but in the future, successful tests could
    be added.
    - subtests: A list of subtests associated with this test.
    
    A subtest contains the following fields:
    - name: same as above
    - number: sanme as above
    - message: the log message that was printed by the subtest.
    - failed: same as above but for the subtest
    
    An example run and json content below:
    ```
    $ sudo ./test_progs -a $(grep -v '^#' ./DENYLIST.aarch64 | awk '{print
    $1","}' | tr -d '\n') -j -J /tmp/test_progs.json
    $ jq < /tmp/test_progs.json | head -n 30
    {
      "success": 29,
      "success_subtest": 23,
      "skipped": 3,
      "failed": 28,
      "results": [
        {
          "name": "bpf_cookie",
          "number": 10,
          "message": "test_bpf_cookie:PASS:skel_open 0 nsec\n",
          "failed": true,
          "subtests": [
            {
              "name": "multi_kprobe_link_api",
              "number": 2,
              "message": "kprobe_multi_link_api_subtest:PASS:load_kallsyms 0 nsec\nlibbpf: extern 'bpf_testmod_fentry_test1' (strong): not resolved\nlibbpf: failed to load object 'kprobe_multi'\nlibbpf: failed to load BPF skeleton 'kprobe_multi': -3\nkprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3\n",
              "failed": true
            },
            {
              "name": "multi_kprobe_attach_api",
              "number": 3,
              "message": "libbpf: extern 'bpf_testmod_fentry_test1' (strong): not resolved\nlibbpf: failed to load object 'kprobe_multi'\nlibbpf: failed to load BPF skeleton 'kprobe_multi': -3\nkprobe_multi_attach_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3\n",
              "failed": true
            },
            {
              "name": "lsm",
              "number": 8,
              "message": "lsm_subtest:PASS:lsm.link_create 0 nsec\nlsm_subtest:FAIL:stack_mprotect unexpected stack_mprotect: actual 0 != expected -1\n",
              "failed": true
            }
    ```
    
    The file can then be used to print a summary of the test run and list of
    failing tests/subtests:
    
    ```
    $ jq -r < /tmp/test_progs.json '"Success: \(.success)/\(.success_subtest), Skipped: \(.skipped), Failed: \(.failed)"'
    
    Success: 29/23, Skipped: 3, Failed: 28
    $ jq -r < /tmp/test_progs.json '.results | map([
        if .failed then "#\(.number) \(.name)" else empty end,
        (
            . as {name: $tname, number: $tnum} | .subtests | map(
                if .failed then "#\($tnum)/\(.number) \($tname)/\(.name)" else empty end
            )
        )
    ]) | flatten | .[]' | head -n 20
     #10 bpf_cookie
     #10/2 bpf_cookie/multi_kprobe_link_api
     #10/3 bpf_cookie/multi_kprobe_attach_api
     #10/8 bpf_cookie/lsm
     #15 bpf_mod_race
     #15/1 bpf_mod_race/ksym (used_btfs UAF)
     #15/2 bpf_mod_race/kfunc (kfunc_btf_tab UAF)
     #36 cgroup_hierarchical_stats
     #61 deny_namespace
     #61/1 deny_namespace/unpriv_userns_create_no_bpf
     #73 fexit_stress
     #83 get_func_ip_test
     #99 kfunc_dynptr_param
     #99/1 kfunc_dynptr_param/dynptr_data_null
     #99/4 kfunc_dynptr_param/dynptr_data_null
     #100 kprobe_multi_bench_attach
     #100/1 kprobe_multi_bench_attach/kernel
     #100/2 kprobe_multi_bench_attach/modules
     #101 kprobe_multi_test
     #101/1 kprobe_multi_test/skel_api
    ```
    Signed-off-by: default avatarManu Bretelle <chantr4@gmail.com>
    Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
    Link: https://lore.kernel.org/bpf/20230317163256.3809328-1-chantr4@gmail.com
    2be7aa76
test_progs.c 41.3 KB