New mtr option --skip-not-found makes it to show not found tests
as skipped

main.a                                   [ skipped ]  not found

(but only if the test was specified with the suite name)
and not error out early with

mysql-test-run: *** ERROR: Could not find 'a' in 'main' suite

This is useful in buildbot, on builders that generate the list
of tests dynamically.