Allow to specify number of retries with an env variable

Simply set `RETRIES=1 bin/rspec ...` and the example will be retried
once.

This can be useful when locally debugging examples that only pass on
retries on the CI, without having to set `CI=1`, which has side-effects.
Signed-off-by: Rémy Coutable <remy@rymai.me>

doc/development/testing_guide/best_practices.md

@@ -38,13 +38,13 @@ To run rspec tests:
 
 ```shell
 # run all tests
-bundle exec rspec
+bin/rspec
 
 # run test for path
-bundle exec rspec spec/[path]/[to]/[spec].rb
+bin/rspec spec/[path]/[to]/[spec].rb
 ```
 
-Use [guard](https://github.com/guard/guard) to continuously monitor for changes and only run matching tests:
+Use [Guard](https://github.com/guard/guard) to continuously monitor for changes and only run matching tests:
 
 ```shell
 bundle exec guard
@@ -130,7 +130,7 @@ Note: `live_debug` only works on JavaScript enabled specs.
 Run the spec with `CHROME_HEADLESS=0`, e.g.:
 
 ```
-CHROME_HEADLESS=0 bundle exec rspec some_spec.rb
+CHROME_HEADLESS=0 bin/rspec some_spec.rb
 ```
 
 The test will go by quickly, but this will give you an idea of what's happening.
@@ -382,8 +382,8 @@ this trait should be either fixed to not rely on Sidekiq processing jobs, or the
 the processing of background jobs is needed/expected.
 
 NOTE: **Note:**
-The usage of `perform_enqueued_jobs` is currently useless since our
-workers aren't inheriting from `ApplicationJob` / `ActiveJob::Base`.
+The usage of `perform_enqueued_jobs` is only useful for testing delayed mail
+deliveries since our Sidekiq workers aren't inheriting from `ApplicationJob` / `ActiveJob::Base`.
 
 #### DNS
 

doc/development/testing_guide/flaky_tests.md

@@ -50,6 +50,9 @@ is detected in any other branch (`flaky-examples-check` job). In the future, the
 
 This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/13021>.
 
+If you want to enable retries locally, you can use the `RETRIES` env variable.
+For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once.
+
 [rspec-retry]: https://github.com/NoRedInk/rspec-retry
 [`spec/spec_helper.rb`]: https://gitlab.com/gitlab-org/gitlab/blob/master/spec/spec_helper.rb
 

spec/spec_helper.rb

@@ -121,9 +121,9 @@ RSpec.configure do |config|
   config.include ExpectRequestWithStatus, type: :request
   config.include RailsHelpers
 
-  if ENV['CI']
+  if ENV['CI'] || ENV['RETRIES']
     # This includes the first try, i.e. tests will be run 4 times before failing.
-    config.default_retry_count = 4
+    config.default_retry_count = ENV.fetch('RETRIES', 3).to_i + 1
     config.reporter.register_listener(
       RspecFlaky::Listener.new,
       :example_passed,