1. 12 Dec, 2022 18 commits
    • Daniel Latypov's avatar
      kunit: tool: make parser preserve whitespace when printing test log · c2bb92bc
      Daniel Latypov authored
      Currently, kunit_parser.py is stripping all leading whitespace to make
      parsing easier. But this means we can't accurately show kernel output
      for failing tests or when the kernel crashes.
      
      Embarassingly, this affects even KUnit's own output, e.g.
      [13:40:46] Expected 2 + 1 == 2, but
      [13:40:46] 2 + 1 == 3 (0x3)
      [13:40:46] not ok 1 example_simple_test
      [13:40:46] [FAILED] example_simple_test
      
      After this change, here's what the output in context would look like
      [13:40:46] =================== example (4 subtests) ===================
      [13:40:46] # example_simple_test: initializing
      [13:40:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
      [13:40:46] Expected 2 + 1 == 2, but
      [13:40:46]     2 + 1 == 3 (0x3)
      [13:40:46] [FAILED] example_simple_test
      [13:40:46] [SKIPPED] example_skip_test
      [13:40:46] [SKIPPED] example_mark_skipped_test
      [13:40:46] [PASSED] example_all_expect_macros_test
      [13:40:46]     # example: initializing suite
      [13:40:46] # example: pass:1 fail:1 skip:2 total:4
      [13:40:46] # Totals: pass:1 fail:1 skip:2 total:4
      [13:40:46] ===================== [FAILED] example =====================
      
      This example shows one minor cosmetic defect this approach has.
      The test counts lines prevent us from dedenting the suite-level output.
      But at the same time, any form of non-KUnit output would do the same
      unless it happened to be indented as well.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      c2bb92bc
    • David Gow's avatar
      Documentation: kunit: Fix "How Do I Use This" / "Next Steps" sections · a81fe7ec
      David Gow authored
      The "How Do I Use This" section of index.rst and "Next Steps" section of
      start.rst were just copies of the table of contents, and therefore
      weren't really useful either when looking a sphinx generated output
      (which already had the TOC visible) or when reading the source (where
      it's just a list of files that ls could give you).
      
      Instead, provide a small number of concrete next steps, and a bit more
      description about what the pages contain.
      
      This also removes the broken reference to 'tips.rst', which was
      previously removed.
      
      Fixed git am whitespace complaints during commit:
      Shuah Khan <skhan@linuxfoundation.org>
      
      Fixes: 4399c737a97d ("Documentation: kunit: Remove redundant 'tips.rst' page")
      Signed-off-by: default avatarDavid Gow <davidgow@google.com>
      Reviewed-by: default avatarSadiya Kazi <sadiyakazi@google.com>
      Reviewed-by: default avatarBagas Sanjaya <bagasdotme@gmail.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      a81fe7ec
    • Daniel Latypov's avatar
      kunit: tool: don't include KTAP headers and the like in the test log · 5937e0c0
      Daniel Latypov authored
      We print the "test log" on failure.
      This is meant to be all the kernel output that happened during the test.
      
      But we also include the special KTAP lines in it, which are often
      redundant.
      
      E.g. we include the "not ok" line in the log, right before we print
      that the test case failed...
      [13:51:48] Expected 2 + 1 == 2, but
      [13:51:48] 2 + 1 == 3 (0x3)
      [13:51:48] not ok 1 example_simple_test
      [13:51:48] [FAILED] example_simple_test
      
      More full example after this patch:
      [13:51:48] =================== example (4 subtests) ===================
      [13:51:48] # example_simple_test: initializing
      [13:51:48] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
      [13:51:48] Expected 2 + 1 == 2, but
      [13:51:48] 2 + 1 == 3 (0x3)
      [13:51:48] [FAILED] example_simple_test
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      5937e0c0
    • Rae Moar's avatar
      kunit: improve KTAP compliance of KUnit test output · 6c738b52
      Rae Moar authored
      Change KUnit test output to better comply with KTAP v1 specifications
      found here: https://kernel.org/doc/html/latest/dev-tools/ktap.html.
      1) Use "KTAP version 1" instead of "TAP version 14" as test output header
      2) Remove '-' between test number and test name on test result lines
      2) Add KTAP version lines to each subtest header as well
      
      Note that the new KUnit output still includes the “# Subtest” line now
      located after the KTAP version line. This does not completely match the
      KTAP v1 spec but since it is classified as a diagnostic line, it is not
      expected to be disruptive or break any existing parsers. This
      “# Subtest” line comes from the TAP 14 spec
      (https://testanything.org/tap-version-14-specification.html) and it is
      used to define the test name before the results.
      
      Original output:
      
       TAP version 14
       1..1
         # Subtest: kunit-test-suite
         1..3
         ok 1 - kunit_test_1
         ok 2 - kunit_test_2
         ok 3 - kunit_test_3
       # kunit-test-suite: pass:3 fail:0 skip:0 total:3
       # Totals: pass:3 fail:0 skip:0 total:3
       ok 1 - kunit-test-suite
      
      New output:
      
       KTAP version 1
       1..1
         KTAP version 1
         # Subtest: kunit-test-suite
         1..3
         ok 1 kunit_test_1
         ok 2 kunit_test_2
         ok 3 kunit_test_3
       # kunit-test-suite: pass:3 fail:0 skip:0 total:3
       # Totals: pass:3 fail:0 skip:0 total:3
       ok 1 kunit-test-suite
      Signed-off-by: default avatarRae Moar <rmoar@google.com>
      Reviewed-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Tested-by: default avatarAnders Roxell <anders.roxell@linaro.org>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      6c738b52
    • Rae Moar's avatar
      kunit: tool: parse KTAP compliant test output · 434498a6
      Rae Moar authored
      Change the KUnit parser to be able to parse test output that complies with
      the KTAP version 1 specification format found here:
      https://kernel.org/doc/html/latest/dev-tools/ktap.html. Ensure the parser
      is able to parse tests with the original KUnit test output format as
      well.
      
      KUnit parser now accepts any of the following test output formats:
      
      Original KUnit test output format:
      
       TAP version 14
       1..1
         # Subtest: kunit-test-suite
         1..3
         ok 1 - kunit_test_1
         ok 2 - kunit_test_2
         ok 3 - kunit_test_3
       # kunit-test-suite: pass:3 fail:0 skip:0 total:3
       # Totals: pass:3 fail:0 skip:0 total:3
       ok 1 - kunit-test-suite
      
      KTAP version 1 test output format:
      
       KTAP version 1
       1..1
         KTAP version 1
         1..3
         ok 1 kunit_test_1
         ok 2 kunit_test_2
         ok 3 kunit_test_3
       ok 1 kunit-test-suite
      
      New KUnit test output format (changes made in the next patch of
      this series):
      
       KTAP version 1
       1..1
         KTAP version 1
         # Subtest: kunit-test-suite
         1..3
         ok 1 kunit_test_1
         ok 2 kunit_test_2
         ok 3 kunit_test_3
       # kunit-test-suite: pass:3 fail:0 skip:0 total:3
       # Totals: pass:3 fail:0 skip:0 total:3
       ok 1 kunit-test-suite
      Signed-off-by: default avatarRae Moar <rmoar@google.com>
      Reviewed-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      434498a6
    • David Gow's avatar
      mm: slub: test: Use the kunit_get_current_test() function · 909c6475
      David Gow authored
      Use the newly-added function kunit_get_current_test() instead of
      accessing current->kunit_test directly. This function uses a static key
      to return more quickly when KUnit is enabled, but no tests are actively
      running. There should therefore be a negligible performance impact to
      enabling the slub KUnit tests.
      
      Other than the performance improvement, this should be a no-op.
      
      Cc: Oliver Glitta <glittao@gmail.com>
      Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>
      Cc: Christoph Lameter <cl@linux.com>
      Cc: Vlastimil Babka <vbabka@suse.cz>
      Cc: David Rientjes <rientjes@google.com>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarDavid Gow <davidgow@google.com>
      Acked-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Acked-by: default avatarHyeonggon Yoo <42.hyeyoo@gmail.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      909c6475
    • David Gow's avatar
      kunit: Use the static key when retrieving the current test · 91e93592
      David Gow authored
      In order to detect if a KUnit test is running, and to access its
      context, the 'kunit_test' member of the current task_struct is used.
      Usually, this is accessed directly or via the kunit_fail_current_task()
      function.
      
      In order to speed up the case where no test is running, add a wrapper,
      kunit_get_current_test(), which uses the static key to fail early.
      Equally, Speed up kunit_fail_current_test() by using the static key.
      
      This should make it convenient for code to call this
      unconditionally in fakes or error paths, without worrying that this will
      slow the code down significantly.
      
      If CONFIG_KUNIT=n (or m), this compiles away to nothing. If
      CONFIG_KUNIT=y, it will compile down to a NOP (on most architectures) if
      no KUnit test is currently running.
      
      Note that kunit_get_current_test() does not work if KUnit is built as a
      module. This mirrors the existing restriction on kunit_fail_current_test().
      
      Note that the definition of kunit_fail_current_test() still wraps an
      empty, inline function if KUnit is not built-in. This is to ensure that
      the printf format string __attribute__ will still work.
      
      Also update the documentation to suggest users use the new
      kunit_get_current_test() function, update the example, and to describe
      the behaviour when KUnit is disabled better.
      
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Sadiya Kazi <sadiyakazi@google.com>
      Signed-off-by: default avatarDavid Gow <davidgow@google.com>
      Reviewed-by: default avatarDaniel Latypov <dlatypov@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      91e93592
    • David Gow's avatar
      kunit: Provide a static key to check if KUnit is actively running tests · 908d0c17
      David Gow authored
      KUnit does a few expensive things when enabled. This hasn't been a
      problem because KUnit was only enabled on test kernels, but with a few
      people enabling (but not _using_) KUnit on production systems, we need a
      runtime way of handling this.
      
      Provide a 'kunit_running' static key (defaulting to false), which allows
      us to hide any KUnit code behind a static branch. This should reduce the
      performance impact (on other code) of having KUnit enabled to a single
      NOP when no tests are running.
      
      Note that, while it looks unintuitive, tests always run entirely within
      __kunit_test_suites_init(), so it's safe to decrement the static key at
      the end of this function, rather than in __kunit_test_suites_exit(),
      which is only there to clean up results in debugfs.
      Signed-off-by: default avatarDavid Gow <davidgow@google.com>
      Reviewed-by: default avatarDaniel Latypov <dlatypov@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      908d0c17
    • Daniel Latypov's avatar
      kunit: tool: make --json do nothing if --raw_ouput is set · 309e22ef
      Daniel Latypov authored
      When --raw_output is set (to any value), we don't actually parse the
      test results. So asking to print the test results as json doesn't make
      sense.
      
      We internally create a fake test with one passing subtest, so --json
      would actually print out something misleading.
      
      This patch:
      * Rewords the flag descriptions so hopefully this is more obvious.
      * Also updates --raw_output's description to note the default behavior
        is to print out only "KUnit" results (actually any KTAP results)
      * also renames and refactors some related logic for clarity (e.g.
        test_result => test, it's a kunit_parser.Test object).
      
      Notably, this patch does not make it an error to specify --json and
      --raw_output together. This is an edge case, but I know of at least one
      wrapper around kunit.py that always sets --json. You'd never be able to
      use --raw_output with that wrapper.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      309e22ef
    • Daniel Latypov's avatar
      kunit: tool: tweak error message when no KTAP found · 0a7d5c30
      Daniel Latypov authored
      We currently tell people we "couldn't find any KTAP output" with no
      indication as to what this might mean.
      
      After this patch, we get:
      
      $ ./tools/testing/kunit/kunit.py parse /dev/null
      ============================================================
      [ERROR] Test: <missing>: Could not find any KTAP output. Did any KUnit tests run?
      ============================================================
      Testing complete. Ran 0 tests: errors: 1
      
      Note: we could try and generate a more verbose message like
      > Please check .kunit/test.log to see the raw kernel output.
      or the like, but we'd need to know what the build dir was to know where
      test.log actually lives.
      
      This patch tries to make a more minimal improvement.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      0a7d5c30
    • Daniel Latypov's avatar
      kunit: remove KUNIT_INIT_MEM_ASSERTION macro · 34c68f43
      Daniel Latypov authored
      Commit 870f63b7cd78 ("kunit: eliminate KUNIT_INIT_*_ASSERT_STRUCT
      macros") removed all the other macros of this type.
      
      But it raced with commit b8a926be ("kunit: Introduce
      KUNIT_EXPECT_MEMEQ and KUNIT_EXPECT_MEMNEQ macros"), which added another
      instance.
      
      Remove KUNIT_INIT_MEM_ASSERTION and just use the generic
      KUNIT_INIT_ASSERT macro instead.
      Rename the `size` arg to avoid conflicts by appending a "_" (like we did
      in the previous commit).
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      34c68f43
    • David Gow's avatar
      Documentation: kunit: Remove redundant 'tips.rst' page · a5b9abaa
      David Gow authored
      The contents of 'tips.rst' was mostly included in 'usage.rst' way back in
      commit 95357439 ("Documentation: KUnit: Rework writing page to focus on writing tests"),
      but the tips page remained behind as well.
      
      The parent patches in this series fill in the gaps, so now 'tips.rst' is
      redundant.
      Therefore, delete 'tips.rst'.
      
      While I regret breaking any links to 'tips' which might exist
      externally, it's confusing to have two subtly different versions of the
      same content around.
      Signed-off-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarSadiya Kazi <sadiyakazi@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      a5b9abaa
    • Daniel Latypov's avatar
      Documentation: KUnit: reword description of assertions · ec0a42a1
      Daniel Latypov authored
      The existing wording implies that kunit_kmalloc_array() is "the method
      under test". We're actually testing the sort() function in that example.
      This is because the example was changed in commit 95357439
      ("Documentation: KUnit: Rework writing page to focus on writing tests"),
      but the wording was not.
      
      Also add a `note` telling people they can use the KUNIT_ASSERT_EQ()
      macros from any function. Some users might be coming from a framework
      like gUnit where that'll compile but silently do the wrong thing.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarSadiya Kazi <sadiyakazi@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      ec0a42a1
    • Daniel Latypov's avatar
      Documentation: KUnit: make usage.rst a superset of tips.rst, remove duplication · 65c48a48
      Daniel Latypov authored
      usage.rst had most of the content of the tips.rst page copied over.
      But it's missing https://www.kernel.org/doc/html/v6.0/dev-tools/kunit/tips.html#customizing-error-messages
      Copy it over so we can retire tips.rst w/o losing content.
      
      And in that process, it also gained a duplicate section about how
      KUNIT_ASSERT_*() exit the test case early. Remove that.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarSadiya Kazi <sadiyakazi@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      65c48a48
    • Daniel Latypov's avatar
      kunit: eliminate KUNIT_INIT_*_ASSERT_STRUCT macros · 697365c0
      Daniel Latypov authored
      These macros exist because passing an initializer list to other macros
      is hard.
      
      The goal of these macros is to generate a line like
        struct $ASSERT_TYPE __assertion = $APPROPRIATE_INITIALIZER;
      e.g.
        struct kunit_unary_assertion __assertion = {
      	  .condition = "foo()",
      	  .expected_true = true
        };
      
      But the challenge is you can't pass `{.condition=..., .expect_true=...}`
      as a macro argument, since the comma means you're actually passing two
      arguments, `{.condition=...` and `.expect_true=....}`.
      So we'd made custom macros for each different initializer-list shape.
      
      But we can work around this with the following generic macro
        #define KUNIT_INIT_ASSERT(initializers...) { initializers }
      
      Note: this has the downside that we have to rename some macros arguments
      to not conflict with the struct field names (e.g. `expected_true`).
      It's a bit gross, but probably worth reducing the # of macros.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      697365c0
    • Daniel Latypov's avatar
      kunit: tool: remove redundant file.close() call in unit test · 101e32a0
      Daniel Latypov authored
      We're using a `with` block above, so the file object is already closed.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      101e32a0
    • Daniel Latypov's avatar
      kunit: tool: unit tests all check parser errors, standardize formatting a bit · 05d9d2c3
      Daniel Latypov authored
      Let's verify that the parser isn't reporting any errors for valid
      inputs.
      
      This change also
      * does result.status checking on one line
      * makes sure we consistently do it outside of the `with` block
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      05d9d2c3
    • Daniel Latypov's avatar
      kunit: tool: make TestCounts a dataclass · f473dd94
      Daniel Latypov authored
      Since we're using Python 3.7+, we can use dataclasses to tersen the
      code.
      
      It also lets us create pre-populated TestCounts() objects and compare
      them in our unit test. (Before, you could only create empty ones).
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      f473dd94
  2. 31 Oct, 2022 2 commits
    • Daniel Latypov's avatar
      kunit: tool: print summary of failed tests if a few failed out of a lot · f19dd011
      Daniel Latypov authored
      E.g. all the hw_breakpoint tests are failing right now.
      So if I run `kunit.py run --altests --arch=x86_64`, then I see
      > Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7
      
      Seeing which 9 tests failed out of the hundreds is annoying.
      If my terminal doesn't have scrollback support, I have to resort to
      looking at `.kunit/test.log` for the `not ok` lines.
      
      Teach kunit.py to print a summarized list of failures if the # of tests
      reachs an arbitrary threshold (>=100 tests).
      
      To try and keep the output from being too long/noisy, this new logic
      a) just reports "parent_test failed" if every child test failed
      b) won't print anything if there are >10 failures (also arbitrary).
      
      With this patch, we get an extra line of output showing:
      > Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7
      > Failures: hw_breakpoint
      
      This also works with parameterized tests, e.g. if I add a fake failure
      > Failures: kcsan.test_atomic_builtins_missing_barrier.threads=6
      
      Note: we didn't have enough tests for this to be a problem before.
      But with commit 980ac3ad ("kunit: tool: rename all_test_uml.config,
      use it for --alltests"), --alltests works and thus running >100 tests
      will probably become more common.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      f19dd011
    • Daniel Latypov's avatar
      kunit: tool: make unit test not print parsed testdata to stdout · 3ffdcf7e
      Daniel Latypov authored
      Currently, if you run
      $ ./tools/testing/kunit/kunit_tool_test.py
      you'll see a lot of output from the parser as we feed it testdata.
      
      This makes the output hard to read and fairly confusing, esp. since our
      testdata includes example failures, which get printed out in red.
      
      Silence that output so real failures are easier to see.
      Signed-off-by: default avatarDaniel Latypov <dlatypov@google.com>
      Reviewed-by: default avatarDavid Gow <davidgow@google.com>
      Signed-off-by: default avatarShuah Khan <skhan@linuxfoundation.org>
      3ffdcf7e
  3. 27 Oct, 2022 6 commits
  4. 23 Oct, 2022 9 commits
  5. 22 Oct, 2022 5 commits