This adds a hardcoded value for now, but in succeeding issues will be
replaced with an actual application limit.

We can't rely on test_suite.total_count because it counts test cases
uniquely by their classname + name. So if there's a malicious file
that are just a bunch of duplicate test cases, we won't be able to
catch it.