• Kirill Smelkov's avatar
    py.bench: Rework output to match Go benchmarking format · 51f252d4
    Kirill Smelkov authored
    Rework py.bench output to match output of Go benchmarking[1] so that go
    tools like benchstat & friends could be used to analyze and compare the
    timings.
    
    Before patch:
    
    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.14, pytest-3.3.2.dev2+g88f2cc9b.d20171206, py-1.5.2, pluggy-0.6.0
    rootdir: /home/kirr/src/wendelin/wendelin.core, inifile:
    collected 11 items
    
    pymod: bigfile/tests/bench_0virtmem.py
    bench_file_mmapread_hole	0.21  (0.39 0.22 0.21)
    bench_file_read_hole    	0.24  (0.24 0.24 0.24)
    bench_file_readbig_hole 	0.30  (0.30 0.31 0.31)
    bench_bigf_read_hole    	0.44  (0.44 0.45 0.44)
    bench_file_mmapwrite0   	0.13  (0.13 0.13 0.13)
    bench_file_write55      	0.08  (0.08 0.08 0.08)
    bench_bigf_writeff      	0.47  (0.47 0.48 0.47)
    bench_file_mmapread     	0.22  (0.22 0.22 0.22)
    bench_file_read         	0.25  (0.25 0.25 0.26)
    bench_file_readbig      	0.31  (0.31 0.31 0.31)
    bench_bigf_read         	0.44  (0.45 0.44 0.44)
    
    ========================== 11 passed in 12.92 seconds ==========================
    
    After patch:
    
    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.14, pytest-3.3.2.dev2+g88f2cc9b.d20171206, py-1.5.2, pluggy-0.6.0
    rootdir: /home/kirr/src/wendelin/wendelin.core, inifile:
    collected 11 items
    
    pymod: bigfile/tests/bench_0virtmem.py
    Benchmarkfile_mmapread_hole	1	385839.939 µs/op
    Benchmarkfile_mmapread_hole	1	219214.916 µs/op
    Benchmarkfile_mmapread_hole	1	210209.131 µs/op
    Benchmarkfile_read_hole    	1	238974.094 µs/op
    Benchmarkfile_read_hole    	1	237294.197 µs/op
    Benchmarkfile_read_hole    	1	238043.070 µs/op
    Benchmarkfile_readbig_hole 	1	301330.090 µs/op
    Benchmarkfile_readbig_hole 	1	301767.111 µs/op
    Benchmarkfile_readbig_hole 	1	301135.063 µs/op
    Benchmarkbigf_read_hole    	1	434718.132 µs/op
    Benchmarkbigf_read_hole    	1	435019.970 µs/op
    Benchmarkbigf_read_hole    	1	434729.099 µs/op
    Benchmarkfile_mmapwrite0   	1	126471.996 µs/op
    Benchmarkfile_mmapwrite0   	1	125886.917 µs/op
    Benchmarkfile_mmapwrite0   	1	125730.038 µs/op
    Benchmarkfile_write55      	1	86760.044 µs/op
    Benchmarkfile_write55      	1	87507.010 µs/op
    Benchmarkfile_write55      	1	87735.891 µs/op
    Benchmarkbigf_writeff      	1	448369.980 µs/op
    Benchmarkbigf_writeff      	1	448238.850 µs/op
    Benchmarkbigf_writeff      	1	447322.845 µs/op
    Benchmarkfile_mmapread     	1	207049.131 µs/op
    Benchmarkfile_mmapread     	1	207813.978 µs/op
    Benchmarkfile_mmapread     	1	210857.868 µs/op
    Benchmarkfile_read         	1	238364.935 µs/op
    Benchmarkfile_read         	1	236908.913 µs/op
    Benchmarkfile_read         	1	238602.161 µs/op
    Benchmarkfile_readbig      	1	303429.842 µs/op
    Benchmarkfile_readbig      	1	302191.973 µs/op
    Benchmarkfile_readbig      	1	304115.057 µs/op
    Benchmarkbigf_read         	1	435079.098 µs/op
    Benchmarkbigf_read         	1	434193.850 µs/op
    Benchmarkbigf_read         	1	434813.976 µs/op
    
    ========================== 11 passed in 12.54 seconds ==========================
    
    benchstat of new output:
    
    	name                µs/op
    	file_mmapread_hole   272k ±42%
    	file_read_hole       238k ± 0%
    	file_readbig_hole    301k ± 0%
    	bigf_read_hole       435k ± 0%
    	file_mmapwrite0      126k ± 0%
    	file_write55        87.3k ± 1%
    	bigf_writeff         448k ± 0%
    	file_mmapread        209k ± 1%
    	file_read            238k ± 0%
    	file_readbig         303k ± 0%
    	bigf_read            435k ± 0%
    
    --------
    
    Not only the output format is reworked here, but also b fixture is added
    to BenchPlugin so that functions which accept it as arg, can run the
    benchmark b.N times, can stop/restart timer etc - similar to
    https://golang.org/pkg/testing/#B . If a bench_* func does not accept b,
    B is still created, but b.N is assumed to be always 1.
    
    The first benchmark which actually uses b will come in the next patch.
    
    [1] https://github.com/golang/proposal/blob/master/design/14313-benchmark-format.md
    51f252d4
py.bench 8.93 KB