• Andrii Nakryiko's avatar
    selftests/bpf: add comparison mode to veristat · 394169b0
    Andrii Nakryiko authored
    Add ability to compare and contrast two veristat runs, previously
    recorded with veristat using CSV output format.
    
    When veristat is called with -C (--compare) flag, veristat expects
    exactly two input files specified, both should be in CSV format.
    Expectation is that it's output from previous veristat runs, but as long
    as column names and formats match, it should just work. First CSV file
    is designated as a "baseline" provided, and the second one is
    comparison (experiment) data set. Establishing baseline matters later
    when calculating difference percentages, see below.
    
    Veristat parses these two CSV files and "reconstructs" verifier stats
    (it could be just a subset of all possible stats). File and program
    names are mandatory as they are used as joining key (these two "stats"
    are designated as "key stats" in the code).
    
    Veristat currently enforces that the set of stats recorded in both CSV
    has to exactly match, down to exact order. This is just a simplifying
    condition which can be lifted with a bit of additional pre-processing to
    reorded stat specs internally, which I didn't bother doing, yet.
    
    For all the non-key stats, veristat will output three columns: one for
    baseline data, one for comparison data, and one with an absolute and
    relative percentage difference. If either baseline or comparison values
    are missing (that is, respective CSV file doesn't have a row with
    *exactly* matching file and program name), those values are assumed to
    be empty or zero. In such case relative percentages are forced to +100%
    or -100% output, for consistency with a typical case.
    
    Veristat's -e (--emit) and -s (--sort) specs still apply, so even if CSV
    contains lots of stats, user can request to compare only a subset of
    them (and specify desired column order as well). Similarly, both CSV and
    human-readable table output is honored. Note that input is currently
    always expected to be CSV.
    
    Here's an example shell session, recording data for biosnoop tool on two
    different kernels and comparing them afterwards, outputting data in table
    format.
    
      # on slightly older production kernel
      $ sudo ./veristat biosnoop_bpf.o
      File            Program                   Verdict  Duration (us)  Total insns  Total states  Peak states
      --------------  ------------------------  -------  -------------  -----------  ------------  -----------
      biosnoop_bpf.o  blk_account_io_merge_bio  success             37           24             1            1
      biosnoop_bpf.o  blk_account_io_start      failure              0            0             0            0
      biosnoop_bpf.o  block_rq_complete         success             76          104             6            6
      biosnoop_bpf.o  block_rq_insert           success             83           85             7            7
      biosnoop_bpf.o  block_rq_issue            success             79           85             7            7
      --------------  ------------------------  -------  -------------  -----------  ------------  -----------
      Done. Processed 1 object files, 5 programs.
      $ sudo ./veristat ~/local/tmp/fbcode-bpf-objs/biosnoop_bpf.o -o csv > baseline.csv
      $ cat baseline.csv
      file_name,prog_name,verdict,duration,total_insns,total_states,peak_states
      biosnoop_bpf.o,blk_account_io_merge_bio,success,36,24,1,1
      biosnoop_bpf.o,blk_account_io_start,failure,0,0,0,0
      biosnoop_bpf.o,block_rq_complete,success,82,104,6,6
      biosnoop_bpf.o,block_rq_insert,success,78,85,7,7
      biosnoop_bpf.o,block_rq_issue,success,74,85,7,7
    
      # on latest bpf-next kernel
      $ sudo ./veristat biosnoop_bpf.o
      File            Program                   Verdict  Duration (us)  Total insns  Total states  Peak states
      --------------  ------------------------  -------  -------------  -----------  ------------  -----------
      biosnoop_bpf.o  blk_account_io_merge_bio  success             31           24             1            1
      biosnoop_bpf.o  blk_account_io_start      failure              0            0             0            0
      biosnoop_bpf.o  block_rq_complete         success             76          104             6            6
      biosnoop_bpf.o  block_rq_insert           success             83           91             7            7
      biosnoop_bpf.o  block_rq_issue            success             74           91             7            7
      --------------  ------------------------  -------  -------------  -----------  ------------  -----------
      Done. Processed 1 object files, 5 programs.
      $ sudo ./veristat biosnoop_bpf.o -o csv > comparison.csv
      $ cat comparison.csv
      file_name,prog_name,verdict,duration,total_insns,total_states,peak_states
      biosnoop_bpf.o,blk_account_io_merge_bio,success,71,24,1,1
      biosnoop_bpf.o,blk_account_io_start,failure,0,0,0,0
      biosnoop_bpf.o,block_rq_complete,success,82,104,6,6
      biosnoop_bpf.o,block_rq_insert,success,83,91,7,7
      biosnoop_bpf.o,block_rq_issue,success,87,91,7,7
    
      # now let's compare with human-readable output (note that no sudo needed)
      # we also ignore verification duration in this case to shortned output
      $ ./veristat -C baseline.csv comparison.csv -e file,prog,verdict,insns
      File            Program                   Verdict (A)  Verdict (B)  Verdict (DIFF)  Total insns (A)  Total insns (B)  Total insns (DIFF)
      --------------  ------------------------  -----------  -----------  --------------  ---------------  ---------------  ------------------
      biosnoop_bpf.o  blk_account_io_merge_bio  success      success      MATCH                        24               24         +0 (+0.00%)
      biosnoop_bpf.o  blk_account_io_start      failure      failure      MATCH                         0                0       +0 (+100.00%)
      biosnoop_bpf.o  block_rq_complete         success      success      MATCH                       104              104         +0 (+0.00%)
      biosnoop_bpf.o  block_rq_insert           success      success      MATCH                        91               85         -6 (-6.59%)
      biosnoop_bpf.o  block_rq_issue            success      success      MATCH                        91               85         -6 (-6.59%)
      --------------  ------------------------  -----------  -----------  --------------  ---------------  ---------------  ------------------
    
    While not particularly exciting example (it turned out to be kind of hard to
    quickly find a nice example with significant difference just because of kernel
    version bump), it should demonstrate main features.
    Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
    Link: https://lore.kernel.org/r/20220921164254.3630690-4-andrii@kernel.orgSigned-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
    394169b0
veristat.c 24.9 KB