ci: Skip flaky tests automatically and allow to opt-out

Signed-off-by: Rémy Coutable <remy@rymai.me>

.gitlab-ci.yml

@@ -80,6 +80,7 @@ variables:
   CACHE_CLASSES: "true"
   CHECK_PRECOMPILED_ASSETS: "true"
   FF_USE_FASTZIP: "true"
+  SKIP_FLAKY_TESTS_AUTOMATICALLY: "true"
 
   DOCS_REVIEW_APPS_DOMAIN: "178.62.207.141.nip.io"
   DOCS_GITLAB_REPO_SUFFIX: "ee"

doc/development/pipelines.md

@@ -170,10 +170,9 @@ After that, the next pipeline uses the up-to-date `knapsack/report-master.json`
 
 ### Flaky tests
 
-Tests that are [known to be flaky](testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection) are:
-
-- skipped if the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is set to `true` (`false` by default)
-- run if `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is not set to `true` or if the `~"pipeline:run-flaky-tests"` label is set on the MR
+Tests that are [known to be flaky](testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection) are
+skipped unless the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is set to `false` or if the `~"pipeline:run-flaky-tests"`
+label is set on the MR.
 
 ### Monitoring
 

spec/support/flaky_tests.rb

 # frozen_string_literal: true
 
 return unless ENV['CI']
-return unless ENV['SKIP_FLAKY_TESTS_AUTOMATICALLY'] == "true"
+return if ENV['SKIP_FLAKY_TESTS_AUTOMATICALLY'] == "false"
 return if ENV['CI_MERGE_REQUEST_LABELS'].to_s.include?('pipeline:run-flaky-tests')
 
 require_relative '../../tooling/rspec_flaky/report'