Split normal memfd and hugetlbfs tests to improve the test reporting.
Remove run_fuse_test.sh and memfd_test from run_tests.sh and add them
to the Makefile.

Add memfd_test to TEST_GEN_PROGS to be run separately.
Rename run_tests.sh to run_hugetlbfs_test.sh
Add run_fuse_test.sh and run_hugetlbfs_test.sh to TEST_PROGS

The report for non-root run wth this change is:

TAP version 13
selftests: memfd: memfd_test
========================================
memfd: CREATE
memfd: BASIC
memfd: SEAL-WRITE
memfd: SEAL-SHRINK
memfd: SEAL-GROW
memfd: SEAL-RESIZE
memfd: SHARE-DUP
memfd: SHARE-MMAP
memfd: SHARE-OPEN
memfd: SHARE-FORK
memfd: SHARE-DUP (shared file-table)
memfd: SHARE-MMAP (shared file-table)
memfd: SHARE-OPEN (shared file-table)
memfd: SHARE-FORK (shared file-table)
memfd: DONE
ok 1..1 selftests: memfd: memfd_test [PASS]
selftests: memfd: run_fuse_test.sh
========================================
opening: ./mnt/memfd
fuse: DONE
ok 1..2 selftests: memfd: run_fuse_test.sh [PASS]
selftests: memfd: run_hugetlbfs_test.sh
========================================
Please run memfd with hugetlbfs test as root
not ok 1..3 selftests: memfd: run_hugetlbfs_test.sh [SKIP]
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When net test is skipped because of unmet dependencies and/or unsupported
configuration, it returns 0 which is treated as a pass by the Kselftest
framework. This leads to false positive result even when the test could
not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.

Change psock_tpacket to use ksft_exit_skip() when a non-root user runs
the test and add an explicit check for root and a clear message, instead
of failing the test when /sys/power/state file open fails.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When mqueue test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as
a fail by the Kselftest framework. This leads to false negative
result even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Change it to use ksft_exit_skip() when the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When memory-hotplug test is skipped because of unmet dependencies and/or
unsupported configuration, it returns non-zero value hich is treated as a
fail by the Kselftest framework. This leads to false negative result even
when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When memfd test is skipped because of unmet dependencies and/or unsupported
configuration, it returns non-zero value which is treated as a fail by the
Kselftest framework. This leads to false negative result even when the test
could not be run.

Change it to return kselftest skip code when a test gets skipped to clearly
report that the test could not be run.

Added an explicit check for root user at the start of memfd hugetlbfs test
and return skip code if a non-root user attempts to run it.

In addition, return skip code when not enough huge pages are available to
run the test.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When membarrier test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as a
fail by the Kselftest framework. This leads to false negative result
even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Change it to use ksft_exit_skip() when the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When media_tests test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as a
fail by the Kselftest framework. This leads to false negative result
even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Change it to use ksft_exit_skip() when the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When locking test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as
a fail by the Kselftest framework. This leads to false negative
result even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Added an explicit search for ww_mutex module and return skip code if
it isn't found to differentiate between the failure to load the module
condition and module not found condition.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Add Makefile for locking test.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When lib test(s) is skipped because of unmet dependencies and/or
unsupported configuration, it returns non-zero value hich is treated
as a fail by the Kselftest framework. This leads to false negative result
even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

prime_numbers.sh is not included in TEST_PROGS. Add it.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When kvm test is skipped because of unmet dependencies and/or unsupported
configuration, it exits with error which is treated as a fail by the
Kselftest framework. This leads to false negative result even when the test
could not be run.

Change it to return kselftest skip code when a test gets skipped to clearly
report that the test could not be run.

Change it to use ksft_exit_skip() when the test is skipped. In addition,
refine test_assert() message to include strerror() string and add explicit
check for EACCES to cleary identify when test doesn't run when access is
denied to resources required e.g: open /dev/kvm failed, rc: -1 errno: 13
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When kmod test is skipped because of unmet dependencies and/or unsupported
configuration, it returns 0 which is treated as a pass by the Kselftest
framework. This leads to false positive result even when the test could
not be run. It returns fail in some cases when test is skipped. Either way,
it is incorrect and incosnistent reporting.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Reviewed-by: Luis R. Rodriguez <mcgrof@kernel.org>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When ipc test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as
a fail by the Kselftest framework. This leads to false negative
result even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Change it to use ksft_exit_skip() when the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When intel_pstate test is skipped because of unmet dependencies and/or
unsupported configuration, it returns 0 which is treated as a pass
by the Kselftest framework. This leads to false positive result even
when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When gpio test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as
a fail by the Kselftest framework. This leads to false negative
result even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When firmware test(s) get skipped because of unmet dependencies and/or
unsupported configuration, it returns 0 which is treated as a pass
by the Kselftest framework. This leads to false positive result even
when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Reviewed-by: Luis R. Rodriguez <mcgrof@kernel.org>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When devpts_pts test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as
a fail by the Kselftest framework. This leads to false negative
result even when the test could not be run.

In another case, it returns pass for a skipped test reporting a false
postive.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Change it to use ksft_exit_skip() when test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When execveat test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as
a fail by the Kselftest framework. This leads to false negative
result even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Change it to use ksft_exit_skip() when kernel doesn't support execveat.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When efivarfs test is skipped because of unmet dependencies and/or
unsupported configuration, it returns 0 which is treated as a pass
by the Kselftest framework. This leads to false positive result even
when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When cpufreq test is skipped because of unmet dependencies and/or
unsupported configuration, it exits with error which is treated as
a fail by the Kselftest framework. This leads to false negative
result even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When cpu-on-off-test is skipped because of unmet dependencies and/or
unsupported configuration, it returns 0 which is treated as a pass
by the Kselftest framework. This leads to false positive result even
when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When step_after_suspend_test is skipped because of unmet dependencies
and/or unsupported configuration, it exits with error which is treated
as a fail by the Kselftest framework. This leads to false negative result
even when the test could not be run.

Change it to return kselftest skip code when a test gets skipped to clearly
report that the test could not be run.

Change it to use ksft_exit_skip() when a non-root user runs the test and
add an explicit check for root and a clear message, instead of failing
the test when /sys/power/state file open fails.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

When ion test is skipped because of unmet dependencies and/or unsupported
configuration, it returns 0 which is treated as a pass by the Kselftest
framework. This leads to false positive result even when the test could
not be run.

Change it to return kselftest skip code when a test gets skipped to
clearly report that the test could not be run.

Kselftest framework SKIP code is 4 and the framework prints appropriate
messages to indicate that the test is skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Acked-by: Pintu Agarwal <pintu.ping@gmail.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Delete RUN_TESTS and EMIT_TESTS overrides and use common defines in
lib.mk. Common defines work after making the change the test to run
with ratio=2 as the default mode to be able to invoke the test without
the "-r 2" argument from the common RUN_TESTS and EMIT_TESTS.

The run_full_tests target now calls the test with "-r 10".
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Reviewed-by: Anders Roxell <anders.roxell@linaro.org>
Reviewed-by: Lei.Yang@windriver.com
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Delete RUN_TESTS and EMIT_TESTS overrides and use common defines in lib.mk.
The overrides are in place to call mq_open_tests with queue name argument.
The change to delete overrides is coupled with a change to mq_open_tests
to use default queue name when it is called without one.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Reviewed-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Delete RUN_TESTS and EMIT_TESTS overrides and use common defines in
lib.mk. Add new run_tests.sh to do the dependency checks the custom
RUN_TESTS did. Common defines work with the run_tests.sh set as the
TEST_PROGS and defining unprivileged-remount-test in TEST_GEN_FILES.

Kselftest framework builds and installs TEST_GEN_FILES and doesn't run
them via RUN_TESTS and include it in EMIT_TESTS. With this change the
new run_tests.sh runs the test after checking dependencies.

This change also adds Skip handling to return kselftest skip code when
test is skipped to clearly identify when the test is skipped instead of
reporting it as failed.

Output with this change:

TAP version 13
selftests: mount: run_tests.sh
========================================
WARN: No /proc/self/uid_map exist, test skipped.
not ok 1..1 selftests: mount: run_tests.sh [SKIP]
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Reviewed-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Delete RUN_TESTS and EMIT_TESTS overrides and use common defines in
lib.mk. Common defines work just fine and there is no need to define
custom overrides.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Reviewed-by: Anders Roxell <anders.roxell@linaro.org>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Delete RUN_TESTS and EMIT_TESTS overrides and use common defines in
lib.mk. Common defines work just fine and there is no need to define
custom overrides.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

KSFT_SKIP points to KSFT_PASS resulting in reporting skipped tests as
Passed, when test programs exit with KSFT_SKIP or call ksft_exit_skip().
If tests are skipped because of unmet dependencies and/or unsupported
configuration, reporting them as passed leads to too many false positives.

Fix it to return a skip code of 4 to clearly differentiate the skipped
tests.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Similar to what RUN_TESTS does, change EMIT_TESTS to check for execute
bit and emit code to print warnings if test isn't executable to the
the run_kselftest.sh.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

EMIT_TESTS which is the common function that implements run_tests target,
treats all non-zero return codes from tests as failures. When tests are
skipped with non-zero return code, because of unmet dependencies and/or
unsupported configuration, it reports them as failed. This will lead to
too many false negatives even on the tests that couldn't be run.

EMIT_TESTS is changed to test for SKIP=4 return from tests to enable
the framework for individual tests to return special SKIP code.

Tests will be changed as needed to report SKIP instead FAIL/PASS when
they get skipped.

Currently just the test name is printed in the RUN_TESTS output. For
example, when raw_skew sub-test from timers tests in run, the output
shows just raw_skew. Include main test name when printing sub-test
results.

In addition, remove duplicate strings for printing common information with
a new for the test header information.

With this change run_kelftest.sh output for breakpoints test will be:

TAP version 13
Running tests in breakpoints
========================================
selftests: breakpoints: step_after_suspend_test
not ok 1..1 selftests: breakpoints: step_after_suspend_test [SKIP]
selftests: breakpoints: breakpoint_test
ok 1..2 selftests: breakpoints: breakpoint_test [PASS]
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Currently just the test name is printed in the RUN_TESTS output. For
example, when raw_skew sub-test from timers tests in run, the output
shows just raw_skew. Include main test name when printing sub-test
results.

In addition, remove duplicate strings for printing common information
with a new for the test header information.

Before the change:

selftests: raw_skew
========================================
WARNING: ADJ_OFFSET in progress, this will cause inaccurate results
Estimating clock drift: -20.616(est) -20.586(act)       [OK]
Pass 0 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0
1..0
ok 1..7 selftests: raw_skew [PASS]

After the change:

selftests: timers: raw_skew
========================================
WARNING: ADJ_OFFSET in progress, this will cause inaccurate results
Estimating clock drift: -19.794(est) -19.896(act)       [OK]
Pass 0 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0
1..0
ok 1..7 selftests: timers: raw_skew [PASS]
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

RUN_TESTS function has grown and becoming harder to maintain. Move
the code that runs and tests for returns codes to a new function
and call it from RUN_TESTS.

A new RUN_TEST_PRINT_RESULT is created to simplify RUN_TESTS and make it
easier to add handling for other return codes as needed.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

RUN_TESTS which is the common function that implements run_tests target,
treats all non-zero return codes from tests as failures. When tests are
skipped with non-zero return code, because of unmet dependencies and/or
unsupported configuration, it reports them as failed. This will lead to
too many false negatives even on the tests that couldn't be run.

RUN_TESTS is changed to test for SKIP=4 return from tests to enable the
framework for individual tests to return special SKIP code.

Tests will be changed as needed to report SKIP instead FAIL/PASS when
they get skipped.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Refine RUN_TESTS define's output block for summary and non-summary code
to remove duplicate code and make it readable.

cd `dirname $$TEST` > /dev/null; and cd - > /dev/null; are moved
to common code block and indentation fixed.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Rework rtctest to use the test harness to better handle skipping tests
(e.g. when alarms are not available). Also, it now handles timeout so it
will not block expecting an alarm that never comes.
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Move the RTC tests out of the timers folder as they are mostly unrelated.
Keep rtcpie in timers as it only test hrtimers.
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

After the test ends, restore the PIE rate to its previous value to be less
disruptive.
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>

Since commit 6610e089 ("RTC: Rework RTC code to use timerqueue for
events"), PIE are completely handled using hrtimers, without actually using
any underlying hardware RTC.

Move PIE testing out of rtctest. It still depends on the presence of an RTC
(to access the device file) but doesn't depend on it actually working.
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>