Skip to content

L
linux
Commit ffb39098 authored by Linus Torvalds's avatar Linus Torvalds
Browse files
Options

Merge tag 'linux-kselftest-kunit-6.1-rc1' of... 

Merge tag 'linux-kselftest-kunit-6.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest

Pull KUnit updates from Shuah Khan:
 "Several documentation fixes, UML related cleanups, and a feature to
  enable/disable KUnit tests

  This includes the change to rename all_test_uml.config, and use it for
  '--alltests'. Note: if anyone was using all_tests_uml.config, this
  change breaks them.

  This change simplifies the usage and eliminates the need to type:

     --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config

  A simple workaround to create a symlink to the new name can solve the
  problem for anyone using all_tests_uml.config.

  all_tests_uml.config should work across ~all architectures"

* tag 'linux-kselftest-kunit-6.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest:
  Documentation: Kunit: Use full path to .kunitconfig
  kunit: tool: rename all_test_uml.config, use it for --alltests
  kunit: tool: remove UML specific options from all_tests_uml.config
  lib: stackinit: update reference to kunit-tool
  lib: overflow: update reference to kunit-tool
  Documentation: KUnit: update links in the index page
  Documentation: KUnit: add intro to the getting-started page
  Documentation: KUnit: Reword start guide for selecting tests
  Documentation: KUnit: add note about mrproper in start.rst
  Documentation: KUnit: avoid repeating "kunit.py run" in start.rst
  Documentation: KUnit: remove duplicated docs for kunit_tool
  Documentation: Kunit: Add ref for other kinds of tests
  Documentation: KUnit: Fix non-uml anchor
  Documentation: Kunit: Fix inconsistent titles
  Documentation: kunit: fix trivial typo
  kunit: no longer call module_info(test, "Y") for kunit modules
  kunit: add kunit.enable to enable/disable KUnit test
  kunit: tool: make --raw_output=kunit (aka --raw_output) preserve leading spaces
parents dd42d9c3 4e370573
Hide whitespace changes
Inline Side-by-side
Showing with 220 additions and 412 deletions
Documentation/admin-guide/kernel-parameters.txt
View file @ ffb39098
......@@ -2432,6 +2432,12 @@
0: force disabled
1: force enabled
kunit.enable= [KUNIT] Enable executing KUnit tests. Requires
CONFIG_KUNIT to be set to be fully enabled. The
default value can be overridden via
KUNIT_DEFAULT_ENABLED.
Default is 1 (enabled)
kvm.ignore_msrs=[KVM] Ignore guest accesses to unhandled MSRs.
Default is 0 (don't ignore, but inject #GP)
......
Documentation/dev-tools/kunit/architecture.rst
View file @ ffb39098
......@@ -6,8 +6,8 @@ KUnit Architecture
The KUnit architecture can be divided into two parts:
- Kernel testing library
- kunit_tool (Command line test harness)
- `In-Kernel Testing Framework`_
- `kunit_tool (Command Line Test Harness)`_
In-Kernel Testing Framework
===========================
......
Documentation/dev-tools/kunit/faq.rst
View file @ ffb39098
......@@ -31,13 +31,16 @@ For the most part, the KUnit core framework (what we use to write the tests)
can compile to any architecture. It compiles like just another part of the
kernel and runs when the kernel boots, or when built as a module, when the
module is loaded. However, there is infrastructure, like the KUnit Wrapper
(``tools/testing/kunit/kunit.py``) that does not support other architectures.
(``tools/testing/kunit/kunit.py``) that might not support some architectures
(see :ref:`kunit-on-qemu`).
In short, yes, you can run KUnit on other architectures, but it might require
more work than using KUnit on UML.
For more information, see :ref:`kunit-on-non-uml`.
.. _kinds-of-tests:
What is the difference between a unit test and other kinds of tests?
====================================================================
Most existing tests for the Linux kernel would be categorized as an integration
......@@ -95,8 +98,7 @@ things to try.
seeing. When tests are built-in, they will execute when the kernel boots, and
modules will automatically execute associated tests when loaded. Test results
can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and
can be parsed with ``kunit.py parse``. For more details, see "KUnit on
non-UML architectures" in Documentation/dev-tools/kunit/usage.rst.
can be parsed with ``kunit.py parse``. For more details, see :ref:`kunit-on-qemu`.
If none of the above tricks help, you are always welcome to email any issues to
kunit-dev@googlegroups.com.
Documentation/dev-tools/kunit/index.rst
View file @ ffb39098
......@@ -13,7 +13,6 @@ KUnit - Linux Kernel Unit Testing
run_wrapper
run_manual
usage
kunit-tool
api/index
style
faq
......@@ -29,10 +28,10 @@ KUnit (Kernel unit testing framework) provides a common framework for
unit tests within the Linux kernel. Using KUnit, you can define groups
of test cases called test suites. The tests either run on kernel boot
if built-in, or load as a module. KUnit automatically flags and reports
failed test cases in the kernel log. The test results appear in `TAP
(Test Anything Protocol) format <https://testanything.org/>`_. It is inspired by
JUnit, Python’s unittest.mock, and GoogleTest/GoogleMock (C++ unit testing
framework).
failed test cases in the kernel log. The test results appear in
:doc:`KTAP (Kernel - Test Anything Protocol) format</dev-tools/ktap>`.
It is inspired by JUnit, Python’s unittest.mock, and GoogleTest/GoogleMock
(C++ unit testing framework).
KUnit tests are part of the kernel, written in the C (programming)
language, and test parts of the Kernel implementation (example: a C
......@@ -46,8 +45,9 @@ internal system functionality. KUnit runs in kernel space and is not
restricted to things exposed to user-space.
In addition, KUnit has kunit_tool, a script (``tools/testing/kunit/kunit.py``)
that configures the Linux kernel, runs KUnit tests under QEMU or UML (`User Mode
Linux <http://user-mode-linux.sourceforge.net/>`_), parses the test results and
that configures the Linux kernel, runs KUnit tests under QEMU or UML
(:doc:`User Mode Linux </virt/uml/user_mode_linux_howto_v2>`),
parses the test results and
displays them in a user friendly manner.
Features
......@@ -95,6 +95,8 @@ Unit Testing Advantages
- Improves code quality.
- Encourages writing testable code.
Read also :ref:`kinds-of-tests`.
How do I use it?
================
......@@ -107,7 +109,5 @@ How do I use it?
examples.
* Documentation/dev-tools/kunit/api/index.rst - KUnit APIs
used for testing.
* Documentation/dev-tools/kunit/kunit-tool.rst - kunit_tool helper
script.
* Documentation/dev-tools/kunit/faq.rst - KUnit common questions and
answers.
Documentation/dev-tools/kunit/kunit-tool.rst deleted 100644 → 0
View file @ dd42d9c3
.. SPDX-License-Identifier: GPL-2.0
=================
kunit_tool How-To
=================
What is kunit_tool?
===================
kunit_tool is a script (``tools/testing/kunit/kunit.py``) that aids in building
the Linux kernel as UML (`User Mode Linux
<http://user-mode-linux.sourceforge.net/>`_), running KUnit tests, parsing
the test results and displaying them in a user friendly manner.
kunit_tool addresses the problem of being able to run tests without needing a
virtual machine or actual hardware with User Mode Linux. User Mode Linux is a
Linux architecture, like ARM or x86; however, unlike other architectures it
compiles the kernel as a standalone Linux executable that can be run like any
other program directly inside of a host operating system. To be clear, it does
not require any virtualization support: it is just a regular program.
What is a .kunitconfig?
=======================
It's just a defconfig that kunit_tool looks for in the build directory
(``.kunit`` by default). kunit_tool uses it to generate a .config as you might
expect. In addition, it verifies that the generated .config contains the CONFIG
options in the .kunitconfig; the reason it does this is so that it is easy to
be sure that a CONFIG that enables a test actually ends up in the .config.
It's also possible to pass a separate .kunitconfig fragment to kunit_tool,
which is useful if you have several different groups of tests you wish
to run independently, or if you want to use pre-defined test configs for
certain subsystems.
Getting Started with kunit_tool
===============================
If a kunitconfig is present at the root directory, all you have to do is:
.. code-block:: bash
./tools/testing/kunit/kunit.py run
However, you most likely want to use it with the following options:
.. code-block:: bash
./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all`
- ``--timeout`` sets a maximum amount of time to allow tests to run.
- ``--jobs`` sets the number of threads to use to build the kernel.
.. note::
This command will work even without a .kunitconfig file: if no
.kunitconfig is present, a default one will be used instead.
If you wish to use a different .kunitconfig file (such as one provided for
testing a particular subsystem), you can pass it as an option.
.. code-block:: bash
./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig
For a list of all the flags supported by kunit_tool, you can run:
.. code-block:: bash
./tools/testing/kunit/kunit.py run --help
Configuring, Building, and Running Tests
========================================
It's also possible to run just parts of the KUnit build process independently,
which is useful if you want to make manual changes to part of the process.
A .config can be generated from a .kunitconfig by using the ``config`` argument
when running kunit_tool:
.. code-block:: bash
./tools/testing/kunit/kunit.py config
Similarly, if you just want to build a KUnit kernel from the current .config,
you can use the ``build`` argument:
.. code-block:: bash
./tools/testing/kunit/kunit.py build
And, if you already have a built UML kernel with built-in KUnit tests, you can
run the kernel and display the test results with the ``exec`` argument:
.. code-block:: bash
./tools/testing/kunit/kunit.py exec
The ``run`` command which is discussed above is equivalent to running all three
of these in sequence.
All of these commands accept a number of optional command-line arguments. The
``--help`` flag will give a complete list of these, or keep reading this page
for a guide to some of the more useful ones.
Parsing Test Results
====================
KUnit tests output their results in TAP (Test Anything Protocol) format.
kunit_tool will, when running tests, parse this output and print a summary
which is much more pleasant to read. If you wish to look at the raw test
results in TAP format, you can pass the ``--raw_output`` argument.
.. code-block:: bash
./tools/testing/kunit/kunit.py run --raw_output
The raw output from test runs may contain other, non-KUnit kernel log
lines. You can see just KUnit output with ``--raw_output=kunit``:
.. code-block:: bash
./tools/testing/kunit/kunit.py run --raw_output=kunit
If you have KUnit results in their raw TAP format, you can parse them and print
the human-readable summary with the ``parse`` command for kunit_tool. This
accepts a filename for an argument, or will read from standard input.
.. code-block:: bash
# Reading from a file
./tools/testing/kunit/kunit.py parse /var/log/dmesg
# Reading from stdin
dmesg | ./tools/testing/kunit/kunit.py parse
This is very useful if you wish to run tests in a configuration not supported
by kunit_tool (such as on real hardware, or an unsupported architecture).
Filtering Tests
===============
It's possible to run only a subset of the tests built into a kernel by passing
a filter to the ``exec`` or ``run`` commands. For example, if you only wanted
to run KUnit resource tests, you could use:
.. code-block:: bash
./tools/testing/kunit/kunit.py run 'kunit-resource*'
This uses the standard glob format for wildcards.
Running Tests on QEMU
=====================
kunit_tool supports running tests on QEMU as well as via UML (as mentioned
elsewhere). The default way of running tests on QEMU requires two flags:
``--arch``
Selects a collection of configs (Kconfig as well as QEMU configs
options, etc) that allow KUnit tests to be run on the specified
architecture in a minimal way; this is usually not much slower than
using UML. The architecture argument is the same as the name of the
option passed to the ``ARCH`` variable used by Kbuild. Not all
architectures are currently supported by this flag, but can be handled
by the ``--qemu_config`` discussed later. If ``um`` is passed (or this
this flag is ignored) the tests will run via UML. Non-UML architectures,
e.g. i386, x86_64, arm, um, etc. Non-UML run on QEMU.
``--cross_compile``
Specifies the use of a toolchain by Kbuild. The argument passed here is
the same passed to the ``CROSS_COMPILE`` variable used by Kbuild. As a
reminder this will be the prefix for the toolchain binaries such as gcc
for example ``sparc64-linux-gnu-`` if you have the sparc toolchain
installed on your system, or
``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux-``
if you have downloaded the microblaze toolchain from the 0-day website
to a directory in your home directory called ``toolchains``.
In many cases it is likely that you may want to run an architecture which is
not supported by the ``--arch`` flag, or you may want to just run KUnit tests
on QEMU using a non-default configuration. For this use case, you can write
your own QemuConfig. These QemuConfigs are written in Python. They must have an
import line ``from ..qemu_config import QemuArchParams`` at the top of the file
and the file must contain a variable called ``QEMU_ARCH`` that has an instance
of ``QemuArchParams`` assigned to it. An example can be seen in
``tools/testing/kunit/qemu_configs/x86_64.py``.
Once you have a QemuConfig you can pass it into kunit_tool using the
``--qemu_config`` flag; when used this flag replaces the ``--arch`` flag. If we
were to do this with the ``x86_64.py`` example from above, the invocation would
look something like this:
.. code-block:: bash
./tools/testing/kunit/kunit.py run \
--timeout=60 \
--jobs=12 \
--qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py
Other Useful Options
====================
kunit_tool has a number of other command-line arguments which can be useful
when adapting it to fit your environment or needs.
Some of the more useful ones are:
``--help``
Lists all of the available options. Note that different commands
(``config``, ``build``, ``run``, etc) will have different supported
options. Place ``--help`` before the command to list common options,
and after the command for options specific to that command.
``--build_dir``
Specifies the build directory that kunit_tool will use. This is where
the .kunitconfig file is located, as well as where the .config and
compiled kernel will be placed. Defaults to ``.kunit``.
``--make_options``
Specifies additional options to pass to ``make`` when compiling a
kernel (with the ``build`` or ``run`` commands). For example, to enable
compiler warnings, you can pass ``--make_options W=1``.
``--alltests``
Builds a UML kernel with all config options enabled using ``make
allyesconfig``. This allows you to run as many tests as is possible,
but is very slow and prone to breakage as new options are added or
modified. In most cases, enabling all tests which have satisfied
dependencies by adding ``CONFIG_KUNIT_ALL_TESTS=1`` to your
.kunitconfig is preferable.
There are several other options (and new ones are often added), so do check
``--help`` if you're looking for something not mentioned here.
Documentation/dev-tools/kunit/run_wrapper.rst
View file @ ffb39098
.. SPDX-License-Identifier: GPL-2.0
=========================
Run Tests with kunit_tool
=========================
=============================
Running tests with kunit_tool
=============================
We can either run KUnit tests using kunit_tool or can run tests
manually, and then use kunit_tool to parse the results. To run tests
......@@ -22,7 +22,7 @@ We should see the following:
.. code-block::
Generating .config...
Configuring KUnit Kernel ...
Building KUnit kernel...
Starting KUnit kernel...
......@@ -30,7 +30,7 @@ We may want to use the following options:
.. code-block::
./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all
./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all`
- ``--timeout`` sets a maximum amount of time for tests to run.
- ``--jobs`` sets the number of threads to build the kernel.
......@@ -58,8 +58,8 @@ To view kunit_tool flags (optional command-line arguments), run:
./tools/testing/kunit/kunit.py run --help
Create a ``.kunitconfig`` File
===============================
Creating a ``.kunitconfig`` file
================================
If we want to run a specific set of tests (rather than those listed
in the KUnit ``defconfig``), we can provide Kconfig options in the
......@@ -98,8 +98,8 @@ have not included the options dependencies.
The build dir needs to be set for ``make menuconfig`` to
work, therefore by default use ``make O=.kunit menuconfig``.
Configure, Build, and Run Tests
===============================
Configuring, building, and running tests
========================================
If we want to make manual changes to the KUnit build process, we
can run part of the KUnit build process independently.
......@@ -125,11 +125,11 @@ argument:
./tools/testing/kunit/kunit.py exec
The ``run`` command discussed in section: **Run Tests with kunit_tool**,
The ``run`` command discussed in section: **Running tests with kunit_tool**,
is equivalent to running the above three commands in sequence.
Parse Test Results
==================
Parsing test results
====================
KUnit tests output displays results in TAP (Test Anything Protocol)
format. When running tests, kunit_tool parses this output and prints
......@@ -152,8 +152,8 @@ standard input.
# Reading from stdin
dmesg | ./tools/testing/kunit/kunit.py parse
Run Selected Test Suites
========================
Filtering tests
===============
By passing a bash style glob filter to the ``exec`` or ``run``
commands, we can run a subset of the tests built into a kernel . For
......@@ -165,8 +165,10 @@ example: if we only want to run KUnit resource tests, use:
This uses the standard glob format with wildcard characters.
Run Tests on qemu
=================
.. _kunit-on-qemu:
Running tests on QEMU
=====================
kunit_tool supports running tests on qemu as well as
via UML. To run tests on qemu, by default it requires two flags:
......@@ -229,8 +231,8 @@ as
--jobs=12 \
--qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py
Command-Line Arguments
======================
Running command-line arguments
==============================
kunit_tool has a number of other command-line arguments which can
be useful for our test environment. Below are the most commonly used
......
Documentation/dev-tools/kunit/start.rst
View file @ ffb39098
......@@ -4,6 +4,10 @@
Getting Started
===============
This page contains an overview of the kunit_tool and KUnit framework,
teaching how to run existing tests and then how to write a simple test case,
and covers common problems users face when using KUnit for the first time.
Installing Dependencies
=======================
KUnit has the same dependencies as the Linux kernel. As long as you can
......@@ -19,30 +23,53 @@ can run kunit_tool:
./tools/testing/kunit/kunit.py run
For more information on this wrapper, see:
Documentation/dev-tools/kunit/run_wrapper.rst.
.. note ::
You may see the following error:
"The source tree is not clean, please run 'make ARCH=um mrproper'"
Creating a ``.kunitconfig``
---------------------------
This happens because internally kunit.py specifies ``.kunit``
(default option) as the build directory in the command ``make O=output/dir``
through the argument ``--build_dir``. Hence, before starting an
out-of-tree build, the source tree must be clean.
By default, kunit_tool runs a selection of tests. However, you can specify which
unit tests to run by creating a ``.kunitconfig`` file with kernel config options
that enable only a specific set of tests and their dependencies.
The ``.kunitconfig`` file contains a list of kconfig options which are required
to run the desired targets. The ``.kunitconfig`` also contains any other test
specific config options, such as test dependencies. For example: the
``FAT_FS`` tests - ``FAT_KUNIT_TEST``, depends on
``FAT_FS``. ``FAT_FS`` can be enabled by selecting either ``MSDOS_FS``
or ``VFAT_FS``. To run ``FAT_KUNIT_TEST``, the ``.kunitconfig`` has:
There is also the same caveat mentioned in the "Build directory for
the kernel" section of the :doc:`admin-guide </admin-guide/README>`,
that is, its use, it must be used for all invocations of ``make``.
The good news is that it can indeed be solved by running
``make ARCH=um mrproper``, just be aware that this will delete the
current configuration and all generated files.
.. code-block:: none
If everything worked correctly, you should see the following:
CONFIG_KUNIT=y
CONFIG_MSDOS_FS=y
CONFIG_FAT_KUNIT_TEST=y
.. code-block::
1. A good starting point for the ``.kunitconfig`` is the KUnit default config.
You can generate it by running:
Configuring KUnit Kernel ...
Building KUnit Kernel ...
Starting KUnit Kernel ...
The tests will pass or fail.
.. note ::
Because it is building a lot of sources for the first time,
the ``Building KUnit Kernel`` step may take a while.
For detailed information on this wrapper, see:
Documentation/dev-tools/kunit/run_wrapper.rst.
Selecting which tests to run
----------------------------
By default, kunit_tool runs all tests reachable with minimal configuration,
that is, using default values for most of the kconfig options. However,
you can select which tests to run by:
- `Customizing Kconfig`_ used to compile the kernel, or
- `Filtering tests by name`_ to select specifically which compiled tests to run.
Customizing Kconfig
~~~~~~~~~~~~~~~~~~~
A good starting point for the ``.kunitconfig`` is the KUnit default config.
If you didn't run ``kunit.py run`` yet, you can generate it by running:
.. code-block:: bash
......@@ -54,48 +81,69 @@ or ``VFAT_FS``. To run ``FAT_KUNIT_TEST``, the ``.kunitconfig`` has:
``.kunitconfig`` lives in the ``--build_dir`` used by kunit.py, which is
``.kunit`` by default.
.. note ::
Before running the tests, kunit_tool ensures that all config options
set in ``.kunitconfig`` are set in the kernel ``.config``. It will warn
you if you have not included dependencies for the options used.
There are many ways to customize the configurations:
a. Edit ``.kunit/.kunitconfig``. The file should contain the list of kconfig
options required to run the desired tests, including their dependencies.
You may want to remove CONFIG_KUNIT_ALL_TESTS from the ``.kunitconfig`` as
it will enable a number of additional tests that you may not want.
If you need to run on an architecture other than UML see :ref:`kunit-on-qemu`.
2. You can then add any other Kconfig options, for example:
b. Enable additional kconfig options on top of ``.kunit/.kunitconfig``.
For example, to include the kernel's linked-list test you can run::
.. code-block:: none
./tools/testing/kunit/kunit.py run \
--kconfig_add CONFIG_LIST_KUNIT_TEST=y
CONFIG_LIST_KUNIT_TEST=y
c. Provide the path of one or more .kunitconfig files from the tree.
For example, to run only ``FAT_FS`` and ``EXT4`` tests you can run::
Before running the tests, kunit_tool ensures that all config options
set in ``.kunitconfig`` are set in the kernel ``.config``. It will warn
you if you have not included dependencies for the options used.
./tools/testing/kunit/kunit.py run \
--kunitconfig ./fs/fat/.kunitconfig \
--kunitconfig ./fs/ext4/.kunitconfig
.. note ::
If you change the ``.kunitconfig``, kunit.py will trigger a rebuild of the
d. If you change the ``.kunitconfig``, kunit.py will trigger a rebuild of the
``.config`` file. But you can edit the ``.config`` file directly or with
tools like ``make menuconfig O=.kunit``. As long as its a superset of
``.kunitconfig``, kunit.py won't overwrite your changes.
Running Tests (KUnit Wrapper)
-----------------------------
1. To make sure that everything is set up correctly, invoke the Python
wrapper from your kernel repository:
.. code-block:: bash
.. note ::
./tools/testing/kunit/kunit.py run
To save a .kunitconfig after finding a satisfactory configuration::
If everything worked correctly, you should see the following:
make savedefconfig O=.kunit
cp .kunit/defconfig .kunit/.kunitconfig
.. code-block::
Filtering tests by name
~~~~~~~~~~~~~~~~~~~~~~~
If you want to be more specific than Kconfig can provide, it is also possible
to select which tests to execute at boot-time by passing a glob filter
(read instructions regarding the pattern in the manpage :manpage:`glob(7)`).
If there is a ``"."`` (period) in the filter, it will be interpreted as a
separator between the name of the test suite and the test case,
otherwise, it will be interpreted as the name of the test suite.
For example, let's assume we are using the default config:
Generating .config ...
Building KUnit Kernel ...
Starting KUnit Kernel ...
a. inform the name of a test suite, like ``"kunit_executor_test"``,
to run every test case it contains::
The tests will pass or fail.
./tools/testing/kunit/kunit.py run "kunit_executor_test"
.. note ::
Because it is building a lot of sources for the first time, the
``Building KUnit kernel`` may take a while.
b. inform the name of a test case prefixed by its test suite,
like ``"example.example_simple_test"``, to run specifically that test case::
./tools/testing/kunit/kunit.py run "example.example_simple_test"
c. use wildcard characters (``*?[``) to run any test case that matches the pattern,
like ``"*.*64*"`` to run test cases containing ``"64"`` in the name inside
any test suite::
./tools/testing/kunit/kunit.py run "*.*64*"
Running Tests without the KUnit Wrapper
=======================================
......@@ -217,7 +265,7 @@ Now we are ready to write the test cases.
obj-$(CONFIG_MISC_EXAMPLE_TEST) += example_test.o
4. Add the following lines to ``.kunitconfig``:
4. Add the following lines to ``.kunit/.kunitconfig``:
.. code-block:: none
......@@ -254,7 +302,5 @@ Next Steps
examples.
* Documentation/dev-tools/kunit/api/index.rst - KUnit APIs
used for testing.
* Documentation/dev-tools/kunit/kunit-tool.rst - kunit_tool helper
script.
* Documentation/dev-tools/kunit/faq.rst - KUnit common questions and
answers.
Documentation/dev-tools/kunit/usage.rst
View file @ ffb39098
......@@ -165,6 +165,8 @@ built as a module).
For more information, see Documentation/dev-tools/kunit/api/test.rst.
.. _kunit-on-non-uml:
Writing Tests For Other Architectures
-------------------------------------
......@@ -544,8 +546,6 @@ By reusing the same ``cases`` array from above, we can write the test as a
{}
};
.. _kunit-on-non-uml:
Exiting Early on Failed Expectations
------------------------------------
......
include/kunit/test.h
View file @ ffb39098
......@@ -228,6 +228,8 @@ static inline void kunit_set_failure(struct kunit *test)
WRITE_ONCE(test->status, KUNIT_FAILURE);
}
bool kunit_enabled(void);
void kunit_init_test(struct kunit *test, const char *name, char *log);
int kunit_run_tests(struct kunit_suite *suite);
......@@ -251,7 +253,6 @@ static inline int kunit_run_all_tests(void)
#endif /* IS_BUILTIN(CONFIG_KUNIT) */
#define __kunit_test_suites(unique_array, ...) \
MODULE_INFO(test, "Y"); \
static struct kunit_suite *unique_array[] \
__aligned(sizeof(struct kunit_suite *)) \
__used __section(".kunit_test_suites") = { __VA_ARGS__ }
......
lib/kunit/Kconfig
View file @ ffb39098
......@@ -59,4 +59,15 @@ config KUNIT_ALL_TESTS
If unsure, say N.
config KUNIT_DEFAULT_ENABLED
bool "Default value of kunit.enable"
default y
help
Sets the default value of kunit.enable. If set to N then KUnit
tests will not execute unless kunit.enable=1 is passed to the
kernel command line.
In most cases this should be left as Y. Only if additional opt-in
behavior is needed should this be set to N.
endif # KUNIT
lib/kunit/executor.c
View file @ ffb39098
......@@ -190,6 +190,10 @@ int kunit_run_all_tests(void)
{
struct suite_set suite_set = {__kunit_suites_start, __kunit_suites_end};
int err = 0;
if (!kunit_enabled()) {
pr_info("kunit: disabled\n");
goto out;
}
if (filter_glob_param) {
suite_set = kunit_filter_suites(&suite_set, filter_glob_param, &err);
......
lib/kunit/test.c
View file @ ffb39098
......@@ -54,6 +54,17 @@ void __kunit_fail_current_test(const char *file, int line, const char *fmt, ...)
EXPORT_SYMBOL_GPL(__kunit_fail_current_test);
#endif
/*
* Enable KUnit tests to run.
*/
#ifdef CONFIG_KUNIT_DEFAULT_ENABLED
static bool enable_param = true;
#else
static bool enable_param;
#endif
module_param_named(enable, enable_param, bool, 0);
MODULE_PARM_DESC(enable, "Enable KUnit tests");
/*
* KUnit statistic mode:
* 0 - disabled
......@@ -586,10 +597,20 @@ static void kunit_init_suite(struct kunit_suite *suite)
suite->suite_init_err = 0;
}
bool kunit_enabled(void)
{
return enable_param;
}
int __kunit_test_suites_init(struct kunit_suite * const * const suites, int num_suites)
{
unsigned int i;
if (!kunit_enabled() && num_suites > 0) {
pr_info("kunit: disabled\n");
return 0;
}
for (i = 0; i < num_suites; i++) {
kunit_init_suite(suites[i]);
kunit_run_tests(suites[i]);
......@@ -607,6 +628,9 @@ void __kunit_test_suites_exit(struct kunit_suite **suites, int num_suites)
{
unsigned int i;
if (!kunit_enabled())
return;
for (i = 0; i < num_suites; i++)
kunit_exit_suite(suites[i]);
......
lib/overflow_kunit.c
View file @ ffb39098
// SPDX-License-Identifier: GPL-2.0 OR MIT
/*
* Test cases for arithmetic overflow checks. See:
* https://www.kernel.org/doc/html/latest/dev-tools/kunit/kunit-tool.html#configuring-building-and-running-tests
* "Running tests with kunit_tool" at Documentation/dev-tools/kunit/start.rst
* ./tools/testing/kunit/kunit.py run overflow [--raw_output]
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
......
lib/stackinit_kunit.c
View file @ ffb39098
......@@ -3,7 +3,7 @@
* Test cases for compiler-based stack variable zeroing via
* -ftrivial-auto-var-init={zero,pattern} or CONFIG_GCC_PLUGIN_STRUCTLEAK*.
* For example, see:
* https://www.kernel.org/doc/html/latest/dev-tools/kunit/kunit-tool.html#configuring-building-and-running-tests
* "Running tests with kunit_tool" at Documentation/dev-tools/kunit/start.rst
* ./tools/testing/kunit/kunit.py run stackinit [--raw_output] \
* --make_option LLVM=1 \
* --kconfig_add CONFIG_INIT_STACK_ALL_ZERO=y
......
tools/testing/kunit/configs/all_tests_uml.config → tools/testing/kunit/configs/all_tests.config
View file @ ffb39098
......@@ -16,8 +16,6 @@ CONFIG_EXT4_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_VIRTIO_UML=y
CONFIG_UML_PCI_OVER_VIRTIO=y
CONFIG_PCI=y
CONFIG_USB4=y
......
tools/testing/kunit/configs/broken_on_uml.config deleted 100644 → 0
View file @ dd42d9c3
# These are currently broken on UML and prevent allyesconfig from building
# CONFIG_STATIC_LINK is not set
# CONFIG_UML_NET_VECTOR is not set
# CONFIG_UML_NET_VDE is not set
# CONFIG_UML_NET_PCAP is not set
# CONFIG_NET_PTP_CLASSIFY is not set
# CONFIG_IP_VS is not set
# CONFIG_BRIDGE_EBT_BROUTE is not set
# CONFIG_BRIDGE_EBT_T_FILTER is not set
# CONFIG_BRIDGE_EBT_T_NAT is not set
# CONFIG_MTD_NAND_CADENCE is not set
# CONFIG_MTD_NAND_NANDSIM is not set
# CONFIG_BLK_DEV_NULL_BLK is not set
# CONFIG_BLK_DEV_RAM is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_NET_VENDOR_XILINX is not set
# CONFIG_NULL_TTY is not set
# CONFIG_PTP_1588_CLOCK is not set
# CONFIG_PINCTRL_EQUILIBRIUM is not set
# CONFIG_DMABUF_SELFTESTS is not set
# CONFIG_COMEDI is not set
# CONFIG_XIL_AXIS_FIFO is not set
# CONFIG_EXFAT_FS is not set
# CONFIG_STM_DUMMY is not set
# CONFIG_FSI_MASTER_ASPEED is not set
# CONFIG_JFS_FS is not set
# CONFIG_UBIFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_CRYPTO_DEV_SAFEXCEL is not set
# CONFIG_CRYPTO_DEV_AMLOGIC_GXL is not set
# CONFIG_KCOV is not set
# CONFIG_LKDTM is not set
# CONFIG_REED_SOLOMON_TEST is not set
# CONFIG_TEST_RHASHTABLE is not set
# CONFIG_TEST_MEMINIT is not set
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
# CONFIG_DEBUG_INFO_BTF is not set
# CONFIG_PTP_1588_CLOCK_INES is not set
# CONFIG_QCOM_CPR is not set
# CONFIG_RESET_BRCMSTB_RESCAL is not set
# CONFIG_RESET_INTEL_GW is not set
# CONFIG_ADI_AXI_ADC is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_PAGE_POISONING is not set
tools/testing/kunit/kunit.py
View file @ ffb39098
......@@ -44,7 +44,6 @@ class KunitConfigRequest:
@dataclass
class KunitBuildRequest(KunitConfigRequest):
jobs: int
alltests: bool
@dataclass
class KunitParseRequest:
......@@ -55,7 +54,6 @@ class KunitParseRequest:
class KunitExecRequest(KunitParseRequest):
build_dir: str
timeout: int
alltests: bool
filter_glob: str
kernel_args: Optional[List[str]]
run_isolated: Optional[str]
......@@ -90,8 +88,7 @@ def build_tests(linux: kunit_kernel.LinuxSourceTree,
stdout.print_with_timestamp('Building KUnit Kernel ...')
build_start = time.time()
success = linux.build_kernel(request.alltests,
request.jobs,
success = linux.build_kernel(request.jobs,
request.build_dir,
request.make_options)
build_end = time.time()
......@@ -118,7 +115,7 @@ def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest)
args.extend(request.kernel_args)
output = linux.run_kernel(args=args,
timeout=None if request.alltests else request.timeout,
timeout=request.timeout,
filter_glob=request.filter_glob,
build_dir=request.build_dir)
lines = kunit_parser.extract_tap_lines(output)
......@@ -165,7 +162,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -
test_start = time.time()
run_result = linux.run_kernel(
args=request.kernel_args,
timeout=None if request.alltests else request.timeout,
timeout=request.timeout,
filter_glob=filter_glob,
build_dir=request.build_dir)
......@@ -206,7 +203,7 @@ def parse_tests(request: KunitParseRequest, metadata: kunit_json.Metadata, input
if request.raw_output == 'all':
pass
elif request.raw_output == 'kunit':
output = kunit_parser.extract_tap_lines(output)
output = kunit_parser.extract_tap_lines(output, lstrip=False)
for line in output:
print(line.rstrip())
......@@ -288,7 +285,7 @@ def add_common_opts(parser) -> None:
help='X=Y make option, can be repeated.',
action='append', metavar='X=Y')
parser.add_argument('--alltests',
help='Run all KUnit tests through allyesconfig',
help='Run all KUnit tests via tools/testing/kunit/configs/all_tests.config',
action='store_true')
parser.add_argument('--kunitconfig',
help='Path to Kconfig fragment that enables KUnit tests.'
......@@ -381,8 +378,14 @@ def tree_from_args(cli_args: argparse.Namespace) -> kunit_kernel.LinuxSourceTree
for arg in cli_args.qemu_args:
qemu_args.extend(shlex.split(arg))
kunitconfigs = cli_args.kunitconfig if cli_args.kunitconfig else []
if cli_args.alltests:
# Prepend so user-specified options take prio if we ever allow
# --kunitconfig options to have differing options.
kunitconfigs = [kunit_kernel.ALL_TESTS_CONFIG_PATH] + kunitconfigs
return kunit_kernel.LinuxSourceTree(cli_args.build_dir,
kunitconfig_paths=cli_args.kunitconfig,
kunitconfig_paths=kunitconfigs,
kconfig_add=cli_args.kconfig_add,
arch=cli_args.arch,
cross_compile=cli_args.cross_compile,
......@@ -441,7 +444,6 @@ def main(argv):
request = KunitRequest(build_dir=cli_args.build_dir,
make_options=cli_args.make_options,
jobs=cli_args.jobs,
alltests=cli_args.alltests,
raw_output=cli_args.raw_output,
json=cli_args.json,
timeout=cli_args.timeout,
......@@ -469,8 +471,7 @@ def main(argv):
linux = tree_from_args(cli_args)
request = KunitBuildRequest(build_dir=cli_args.build_dir,
make_options=cli_args.make_options,
jobs=cli_args.jobs,
alltests=cli_args.alltests)
jobs=cli_args.jobs)
result = config_and_build_tests(linux, request)
stdout.print_with_timestamp((
'Elapsed time: %.3fs\n') % (
......@@ -483,7 +484,6 @@ def main(argv):
build_dir=cli_args.build_dir,
json=cli_args.json,
timeout=cli_args.timeout,
alltests=cli_args.alltests,
filter_glob=cli_args.filter_glob,
kernel_args=cli_args.kernel_args,
run_isolated=cli_args.run_isolated)
......
tools/testing/kunit/kunit_kernel.py
View file @ ffb39098
......@@ -25,7 +25,7 @@ KCONFIG_PATH = '.config'
KUNITCONFIG_PATH = '.kunitconfig'
OLD_KUNITCONFIG_PATH = 'last_used_kunitconfig'
DEFAULT_KUNITCONFIG_PATH = 'tools/testing/kunit/configs/default.config'
BROKEN_ALLCONFIG_PATH = 'tools/testing/kunit/configs/broken_on_uml.config'
ALL_TESTS_CONFIG_PATH = 'tools/testing/kunit/configs/all_tests.config'
UML_KCONFIG_PATH = 'tools/testing/kunit/configs/arch_uml.config'
OUTFILE_PATH = 'test.log'
ABS_TOOL_PATH = os.path.abspath(os.path.dirname(__file__))
......@@ -57,9 +57,6 @@ class LinuxSourceTreeOperations:
def make_arch_config(self, base_kunitconfig: kunit_config.Kconfig) -> kunit_config.Kconfig:
return base_kunitconfig
def make_allyesconfig(self, build_dir: str, make_options) -> None:
raise ConfigError('Only the "um" arch is supported for alltests')
def make_olddefconfig(self, build_dir: str, make_options) -> None:
command = ['make', 'ARCH=' + self._linux_arch, 'O=' + build_dir, 'olddefconfig']
if self._cross_compile:
......@@ -144,26 +141,6 @@ class LinuxSourceTreeOperationsUml(LinuxSourceTreeOperations):
kconfig.merge_in_entries(base_kunitconfig)
return kconfig
def make_allyesconfig(self, build_dir: str, make_options) -> None:
stdout.print_with_timestamp(
'Enabling all CONFIGs for UML...')
command = ['make', 'ARCH=um', 'O=' + build_dir, 'allyesconfig']
if make_options:
command.extend(make_options)
process = subprocess.Popen(
command,
stdout=subprocess.DEVNULL,
stderr=subprocess.STDOUT)
process.wait()
stdout.print_with_timestamp(
'Disabling broken configs to run KUnit tests...')
with open(get_kconfig_path(build_dir), 'a') as config:
with open(BROKEN_ALLCONFIG_PATH, 'r') as disable:
config.write(disable.read())
stdout.print_with_timestamp(
'Starting Kernel with all configs takes a few minutes...')
def start(self, params: List[str], build_dir: str) -> subprocess.Popen:
"""Runs the Linux UML binary. Must be named 'linux'."""
linux_bin = os.path.join(build_dir, 'linux')
......@@ -343,10 +320,8 @@ class LinuxSourceTree:
os.remove(kconfig_path)
return self.build_config(build_dir, make_options)
def build_kernel(self, alltests, jobs, build_dir: str, make_options) -> bool:
def build_kernel(self, jobs, build_dir: str, make_options) -> bool:
try:
if alltests:
self._ops.make_allyesconfig(build_dir, make_options)
self._ops.make_olddefconfig(build_dir, make_options)
self._ops.make(jobs, build_dir, make_options)
except (ConfigError, BuildError) as e:
......@@ -359,6 +334,7 @@ class LinuxSourceTree:
args = []
if filter_glob:
args.append('kunit.filter_glob='+filter_glob)
args.append('kunit.enable=1')
process = self._ops.start(args, build_dir)
assert process.stdout is not None # tell mypy it's set
......
tools/testing/kunit/kunit_parser.py
View file @ ffb39098
......@@ -218,7 +218,7 @@ TAP_START = re.compile(r'TAP version ([0-9]+)$')
KTAP_END = re.compile('(List of all partitions:|'
'Kernel panic - not syncing: VFS:|reboot: System halted)')
def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream:
def extract_tap_lines(kernel_output: Iterable[str], lstrip=True) -> LineStream:
"""Extracts KTAP lines from the kernel output."""
def isolate_ktap_output(kernel_output: Iterable[str]) \
-> Iterator[Tuple[int, str]]:
......@@ -244,9 +244,11 @@ def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream:
# stop extracting KTAP lines
break
elif started:
# remove prefix and any indention and yield
# line with line number
line = line[prefix_len:].lstrip()
# remove the prefix and optionally any leading
# whitespace. Our parsing logic relies on this.
line = line[prefix_len:]
if lstrip:
line = line.lstrip()
yield line_num, line
return LineStream(lines=isolate_ktap_output(kernel_output))
......
tools/testing/kunit/kunit_tool_test.py
View file @ ffb39098
......@@ -549,7 +549,7 @@ class KUnitMainTest(unittest.TestCase):
def test_build_passes_args_pass(self):
kunit.main(['build'])
self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
self.linux_source_mock.build_kernel.assert_called_once_with(False, kunit.get_default_jobs(), '.kunit', None)
self.linux_source_mock.build_kernel.assert_called_once_with(kunit.get_default_jobs(), '.kunit', None)
self.assertEqual(self.linux_source_mock.run_kernel.call_count, 0)
def test_exec_passes_args_pass(self):
......@@ -664,7 +664,7 @@ class KUnitMainTest(unittest.TestCase):
build_dir = '.kunit'
jobs = kunit.get_default_jobs()
kunit.main(['build', '--build_dir', build_dir])
self.linux_source_mock.build_kernel.assert_called_once_with(False, jobs, build_dir, None)
self.linux_source_mock.build_kernel.assert_called_once_with(jobs, build_dir, None)
def test_exec_builddir(self):
build_dir = '.kunit'
......@@ -695,6 +695,18 @@ class KUnitMainTest(unittest.TestCase):
qemu_config_path=None,
extra_qemu_args=[])
def test_config_alltests(self):
kunit.main(['config', '--kunitconfig=mykunitconfig', '--alltests'])
# Just verify that we parsed and initialized it correctly here.
self.mock_linux_init.assert_called_once_with('.kunit',
kunitconfig_paths=[kunit_kernel.ALL_TESTS_CONFIG_PATH, 'mykunitconfig'],
kconfig_add=None,
arch='um',
cross_compile=None,
qemu_config_path=None,
extra_qemu_args=[])
@mock.patch.object(kunit_kernel, 'LinuxSourceTree')
def test_run_multiple_kunitconfig(self, mock_linux_init):
mock_linux_init.return_value = self.linux_source_mock
......@@ -712,7 +724,7 @@ class KUnitMainTest(unittest.TestCase):
kunit.main(['run', '--kconfig_add=CONFIG_KASAN=y', '--kconfig_add=CONFIG_KCSAN=y'])
# Just verify that we parsed and initialized it correctly here.
self.mock_linux_init.assert_called_once_with('.kunit',
kunitconfig_paths=None,
kunitconfig_paths=[],
kconfig_add=['CONFIG_KASAN=y', 'CONFIG_KCSAN=y'],
arch='um',
cross_compile=None,
......@@ -723,7 +735,7 @@ class KUnitMainTest(unittest.TestCase):
kunit.main(['run', '--arch=x86_64', '--qemu_args', '-m 2048'])
# Just verify that we parsed and initialized it correctly here.
self.mock_linux_init.assert_called_once_with('.kunit',
kunitconfig_paths=None,
kunitconfig_paths=[],
kconfig_add=None,
arch='x86_64',
cross_compile=None,
......@@ -742,7 +754,7 @@ class KUnitMainTest(unittest.TestCase):
self.linux_source_mock.run_kernel.return_value = ['TAP version 14', 'init: random output'] + want
got = kunit._list_tests(self.linux_source_mock,
kunit.KunitExecRequest(None, None, '.kunit', 300, False, 'suite*', None, 'suite'))
kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'suite'))
self.assertEqual(got, want)
# Should respect the user's filter glob when listing tests.
......@@ -757,7 +769,7 @@ class KUnitMainTest(unittest.TestCase):
# Should respect the user's filter glob when listing tests.
mock_tests.assert_called_once_with(mock.ANY,
kunit.KunitExecRequest(None, None, '.kunit', 300, False, 'suite*.test*', None, 'suite'))
kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, 'suite'))
self.linux_source_mock.run_kernel.assert_has_calls([
mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', timeout=300),
mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', timeout=300),
......@@ -770,7 +782,7 @@ class KUnitMainTest(unittest.TestCase):
# Should respect the user's filter glob when listing tests.
mock_tests.assert_called_once_with(mock.ANY,
kunit.KunitExecRequest(None, None, '.kunit', 300, False, 'suite*', None, 'test'))
kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'test'))
self.linux_source_mock.run_kernel.assert_has_calls([
mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', timeout=300),
mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', timeout=300),
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7