Drop excess jobs while still making sure there is a good variation of
running all test suites, gcc and clang versions.

Also introduce testing on architectures arm64 and ppc64le.