Change common RUN_TESTS to print messages in user friendly TAP13 format.
This change add TAP13 header at the start of RUN_TESTS target run, and
prints the resulting pass/fail messages with test number information in
the TAP 13 format for each test in the run tests list.

This change covers test scripts as well as test programs. Test programs
have an option to use ksft_ API, however test scripts won't be able to.
With this change, test scripts can print TAP13 format output without any
changes to individual scripts.

Test programs can provide TAP13 format output as needed as some tests
already do. Tests that haven't been converted will benefit from this
change. Tests that are converted benefit from the test counts for all
the tests in each test directory.

Running firmware tests:
make --silent -C tools/testing/selftests/firmware/ run_tests

Before the change:

modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
./fw_filesystem.sh: /sys/devices/virtual/misc/test_firmware not present
You must have the following enabled in your kernel:
CONFIG_TEST_FIRMWARE=y
selftests:  fw_filesystem.sh [FAIL]
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
selftests:  fw_fallback.sh [FAIL]

After the change:

TAP version 13
selftests: fw_filesystem.sh
========================================
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
./fw_filesystem.sh: /sys/devices/virtual/misc/test_firmware not present
You must have the following enabled in your kernel:
CONFIG_TEST_FIRMWARE=y
not ok 1..1 selftests:  fw_filesystem.sh [FAIL]
selftests: fw_fallback.sh
========================================
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
not ok 1..2 selftests:  fw_fallback.sh [FAIL]
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Change lib.mk RUN_TESTS to take test list as an argument. This will
allow it to be called from individual test makefiles to run additional
tests that aren't suitable for a default kselftest run. As an example,
timers test includes destructive tests that aren't included in the
common run_tests target.

Change times/Makefile to use RUN_TESTS call with destructive test list
as an argument instead of using its own RUN_TESTS target.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Suppress "cd" output from run_tests while running tests to declutter the
test results.

Running efivarfs test:
make --silent -C tools/testing/selftests/efivarfs/ run_tests

Before the change:

skip all tests: must be run as root
selftests: efivarfs.sh [PASS]
/lkml/linux-kselftest/tools/testing/selftests/efivarfs

After the change:

skip all tests: must be run as root
selftests: efivarfs.sh [PASS]
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

When a test is skipped, instead of using a special exit code of 4, treat
it as pass condition and use exit code of 0. It makes sense to treat skip
as pass since the test couldn't be run as opposed to a failed test.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

clocksource_list array is defined as char [10][30] so
to initialise it we only have to iterate 10 times.
Signed-off-by: Benjamin Gaignard <benjamin.gaignard@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Remove rtctest_setdate from run_destructive_tests target. Leave it in
TEST_GEN_PROGS_EXTENDED to be included in the install targets.
Suggested-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Reviewed-by: Benjamin Gaignard <benjamin.gaignard@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

When a test exits with skip exit code of 4, "make run_destructive_tests"
halts testing. Fix run_destructive_tests target to handle error exit codes.

Cc: stable <stable@vger.kernel.org> [4.13+]
Reported-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Change default arguments for leap-a-day to always set the time
each iteration (rather then waiting for midnight UTC), and to
only run 10 interations (rather then infinite).

If one wants to wait for midnight UTC, they can use the new -w
flag, and we add a note to the argument help that -i -1 will
run infinitely.

Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Miroslav Lichvar <mlichvar@redhat.com>
Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Prarit Bhargava <prarit@redhat.com>
Cc: Stephen Boyd <stephen.boyd@linaro.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: linux-kselftest@vger.kernel.org
Signed-off-by: John Stultz <john.stultz@linaro.org>
Cc: stable <stable@vger.kernel.org> [4.13+]
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

There is no need to keep timers tests in sync with external timers
repo. Drop support for !KTEST to support for building and running
timers tests without kselftest framework.

Reference: https://lkml.org/lkml/2017/8/10/952Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Acked-by: John Stultz <john.stultz@linaro.org>

The rtc-generic and opal-rtc are failing to run this test as they do not
support all the features. Let's treat the error returns and skip to the
following test.

Theoretically the test_DATE should be also adjusted, but as it's enabled
on demand I think it makes sense to fail in such case.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

For only one online cpu case, 'make run_tests' try to offline the cpu0 that will
always fail since the host can't offline this unique online cpu.

this patch will skip the test to avoid this failure.
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Previously, 'make run_tests -C cpu-hotplug' always PASS since cpu-on-off-test.sh
always exits 0 even though the test got some unexpected errors like below:
root@debian9:/home/lizhijian/chroot/linux/tools/testing/selftests/cpu-hotplug# make run_tests
pid 878's current affinity mask: 1
pid 878's new affinity mask: 1
CPU online/offline summary:
Cpus in online state: 0
        Cpus in offline state: 0
Limited scope test: one hotplug cpu
        (leaves cpu in the original state):
        online to offline to online: cpu 0
./cpu-on-off-test.sh: line 83: /sys/devices/system/cpu/cpu0/online: Permission denied
offline_cpu_expect_success 0: unexpected fail
./cpu-on-off-test.sh: line 78: /sys/devices/system/cpu/cpu0/online: Permission denied
online_cpu_expect_success 0: unexpected fail
selftests: cpu-on-off-test.sh [PASS]

after this patch, the test will exit with failure once it occurs some unexpected behaviors
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Convert test to use ksft TAP13 framework to print user friendly
test output which is consistent across kselftest suite.
Acked-by: Darren Hart (VMware) <dvhart@infradead.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Convert errx() and err() usage to appropriate TAP13 ksft API.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

The stack size should be 16 bytes aligned in arm64 system. The similar
patch has been merged already.

> <commit id: 1f78dda2>
> selftests: memfd_test: Revised STACK_SIZE to make it 16-byte aligned
>
>    There is a mandate of 16-byte aligned stack on AArch64 [1], so the
>    STACK_SIZE here should also be 16-byte aligned, otherwise we would
>    get an error when calling clone().
>
>    [1] http://lxr.free-electrons.com/source/arch/arm64/kernel/process.c#L265
>
>    Signed-off-by: Chunyan Zhang <zhang.chunyan@linaro.org>
>    Acked-by: Arnd Bergmann <arnd@arndb.de>
>    Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Signed-off-by: Orson Zhai <orson.zhai@linaro.org>
Reviewed-by: Sumit Semwal <sumit.semwal@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Executing selftests is fragile as if someone forgot to set a secript
as executable the test will fail, and you won't know for sure if the
failure was caused by the lack of proper permissions or something else.

Setting scripts as executable is required, this also enable folks to
execute selftests as independent units.
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Some tests track errors in addition to test failures. Add ksft_error
counter, ksft_get_error_cnt(), and ksft_test_result_error() API to
get the counter value and print error message.

Update ksft_print_cnts(), and ksft_test_num() to include error counter.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Convert the test to use TAP13 ksft framework for test output. Converting
error paths using err() and errx() will be done in another patch to make
it easier for review and change management.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

do_tests() runs sgidnonroot test without fork_wait(). As a result the
last test "Non-root +ia, sgidroot => i test" is left out. Fix it.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Add the usr/include subdirectory of the top-level tree to the include
path to fix build when cross compiling for ARM.
testptp.c: In function 'main':
testptp.c:289:15: error: 'struct ptp_clock_caps' has no member named 'cross_timestamping'
           caps.cross_timestamping);
Signed-off-by: Grygorii Strashko <grygorii.strashko@ti.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Convert to use TAP13 ksft framework to output results.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Add .gitignore for generated files.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Add .gitignore for generated files.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Convert test to use TAP13 ksft framework. Output after conversion:

TAP version 13
# [RUN]	Testing sync framework
ok 1 [RUN]	test_alloc_timeline
ok 2 [RUN]	test_alloc_fence
ok 3 [RUN]	test_alloc_fence_negative
ok 4 [RUN]	test_fence_one_timeline_wait
ok 5 [RUN]	test_fence_one_timeline_merge
ok 6 [RUN]	test_fence_merge_same_fence
ok 7 [RUN]	test_fence_multi_timeline_wait
ok 8 [RUN]	test_stress_two_threads_shared_timeline
ok 9 [RUN]	test_consumer_stress_multi_producer_single_consumer
ok 10 [RUN]	test_merge_stress_random_merge
Pass 10 Fail 0 Xfail 0 Xpass 0 Skip 0
1..10
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>

Some tests print final pass/fail message based on fail count. Add
ksft_get_*_cnt() API to kselftest framework to return counts.

Update ksft_print_cnts() to print the test results summary message with
individual pass, fail, ... counters.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>

Sync test doesn't differentiate between sync unsupported and test run
by non-root user and treats both as unsupported cases.

Fix it to add handling for these two different scenarios.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>

Use [ ! -z "$VAR" ] instead of [ "$VAR" ] to check
whether the given string variable is not zero-length
since it obviously shows what it means.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Acked-by: Steven Rostedt (VMware) <srostedt@goodmis.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Output logs only to console if "-" is given to --logdir
option. In this case, ftracetest doesn't record any log
on the disk, and all logs immediately shown (including
all command logs.) Since there is no "tee" in the middle
of command and console, it outputs the log really soon.

This option is useful only when the console is logged.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Acked-by: Steven Rostedt (VMware) <srostedt@goodmis.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Add 3-level verbosity for showing traced command log
on console immediately. Since some test cases can cause
kernel pacic if there is a probrem (like regression etc.),
we can not know which command caused the problem without
traced command log. This verbosity (-vvv) solves that
because it shows the log on console immediately. User
can get continuous command/error log.

Note that this is a kind of kernel debug mode, if you
don't see any kernel related issue, you don't need this
verbosity.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Acked-by: Steven Rostedt (VMware) <srostedt@goodmis.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Add --fail-unsupported option to fail the test result if
ftracetest gets UNSUPPORTED result. UNSUPPORTED usually
happens when the kernel is old (e.g. stable tree) or some
kernel feature is disabled.

However, if newer kernel has any bug or regression, it
can make test results in UNSUPPORTED too. This option
can detect such kernel regression.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Acked-by: Steven Rostedt (VMware) <srostedt@goodmis.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Do not return failure exit code (1) for unsupported testcases,
since it is expected for stable kernels.

Previously, ftracetest is expected to run only on current
release for avoiding regressions. However, nowadays we run
it on stable kernels. This means some test cases must return
unsupported result. In such case, we should NOT exit
ftracetest with error status for unsupported results so that
kselftest (upper tests wrapper) shows it passed correctly.

Note that we continue to treat unresolved results as failure,
if test writers would like to notice user that the test result
should be reviewed, they can use exit_unresolved.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Acked-by: Steven Rostedt (VMware) <srostedt@goodmis.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Add missing line breaks between the last two tests.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Simplify the Makefile rules so that the test is
automatically installed (and cleaned) by leveraging
the TEST_GEN_PROGS_EXTENDED definition.
Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Signed-off-by: Daniel Díaz <daniel.diaz@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Some watchdog drivers implement WDIOF_CARDRESET feature. As example,
see commit b6ef36d2 ("watchdog: qcom: Report reboot reason").
This option allows reporting to userspace the cause of the last boot
(POR/watchdog reset), being helpful in e.g. automated test-cases.

Add support for WDIOC_GETBOOTSTATUS in the test code, to be able to:
- check if watchdog drivers properly implement WDIOF_CARDRESET.
- check the last boot status, if WDIOF_CARDRESET is implemented.

Make the `-b, --bootstatus` option one-shot. That means, skip the
keepalive mechanism if `-b` is provided on the command line, as we
are only interested in the boot status information.

Tested on Rcar-H3 Salvator-X board:

********************** Cold boot finished
salvator-x:/home/root# ./watchdog-test -h
Usage: ./watchdog-test [options]
 -b, --bootstatus    Get last boot status (Watchdog/POR)
 -d, --disable       Turn off the watchdog timer
 -e, --enable        Turn on the watchdog timer
 -h, --help          Print the help message
 -p, --pingrate=P    Set ping rate to P seconds (default 1)
 -t, --timeout=T     Set timeout to T seconds

Parameters are parsed left-to-right in real-time.
Example: ./watchdog-test -d -t 10 -p 5 -e
salvator-x:/home/root#
salvator-x:/home/root# ./watchdog-test -b
Last boot is caused by: Power-On-Reset.
salvator-x:/home/root#
salvator-x:/home/root# ./watchdog-test -d -t 1 -p 2 -e
Watchdog card disabled.
Watchdog timeout set to 1 seconds.
Watchdog ping rate set to 2 seconds.
Watchdog card enabled.
Watchdog Ticking Away!
********************** Reboot due to watchdog trigger finished
salvator-x:/home/root# ./watchdog-test -b
Last boot is caused by: Watchdog.
salvator-x:/home/root#
salvator-x:/home/root# reboot
********************** Reboot due to user action finished
salvator-x:/home/root# ./watchdog-test -b
Last boot is caused by: Power-On-Reset.
salvator-x:/home/root#
Signed-off-by: Eugeniu Rosca <erosca@de.adit-jv.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Calling `watchdog-test [options] -p 0` results in flooding the kernel
with WDIOC_KEEPALIVE. Fix this by enforcing 1 second as minimal/default
keepalive/ping rate.
Signed-off-by: Eugeniu Rosca <erosca@de.adit-jv.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Report the failure of WDIOC_SETOPTIONS/WDIOC_SETTIMEOUT ioctls.
Signed-off-by: Eugeniu Rosca <erosca@de.adit-jv.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

commit f15d7114 ("Documentation/watchdog: add timeout and ping rate
control to watchdog-test.c") used both atoi() and strtoul() for string
to integer conversion. As usage of atoi() is discouraged in newer code,
replace it with strtoul() for consistency.
Signed-off-by: Eugeniu Rosca <erosca@de.adit-jv.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Switch from manual argv[] parsing to getopt_long() argument processing.
This creates more readable code and allows easier feature addition.
This also fixes some segmentation faults introduced by
commit 1dbdcc81 ("selftests: watchdog: accept multiple params on
command line"), when options -t or -p are not given the required value:

./watchdog-test -p 1 -t
./watchdog-test -t 1 -p

No changes are intended in the way watchdog-test interacts with the
kernel. The only noticible changes, tightly related to the addition
of getopt (and done for easier maintenance),  are:
- help message has been reworked and migrated to a dedicated function.
- all short/long options and the help message are sorted alphabetically.
- all case statements inside the getopt loop are sorted alphabetically.

Fixes: 1dbdcc81 ("selftests: watchdog: accept multiple params on command line")
Signed-off-by: Eugeniu Rosca <erosca@de.adit-jv.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Convert spaces to tabs for checkpatch compliance. Quick way to verify
this is by running `git show -w <commit-id>`, which returns an empty
commit body. No functional change intended.
Signed-off-by: Eugeniu Rosca <erosca@de.adit-jv.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>

Create a config fragment for nsfs to enable additional config options.
The config fragments can be used with the help of
scripts/kconfig/merge_config.sh.
Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Acked-by: Andrei Vagin <avagin@virtuozzo.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>