[very specific cases](https://about.gitlab.com/handbook/engineering/workflow/#criteria-for-merging-during-broken-master).
For other cases, follow these [handbook instructions](https://about.gitlab.com/handbook/engineering/workflow/#merging-during-broken-master).
- If the latest pipeline was created before the merge request was approved, start a new pipeline to ensure that full RSpec suite has been run. You may skip this step only if the merge request does not contain any backend change.
- If the **latest [Pipeline for Merged Results](../ci/pipelines/pipelines_for_merged_results.md)** finished less than 2 hours ago, you
might merge without starting a new pipeline as the merge request is close
@@ -421,58 +421,18 @@ This experiment is only enabled when the CI/CD variable `RSPEC_FAIL_FAST_ENABLED
The test files related to the Merge Request are determined using the [`test_file_finder`](https://gitlab.com/gitlab-org/ci-cd/test_file_finder) gem.
We are using a custom mapping between source file to test files, maintained in the `tests.yml` file.
### RSpec minimal job experiment
### RSpec minimal jobs
As part of the objective to improve overall pipeline duration, we are experimenting with a minimal set of RSpec tests.
The purpose of this experiment is to verify if we are able to run a minimal set of RSpec tests in a Merge Request pipeline,
without resulting in increased number of broken main branch.
Before a merge request is approved, the pipeline will run a minimal set of RSpec tests that are related to the merge request changes.
This is to reduce the pipeline cost and shorten the job duration.
To identify the minimal set of tests needed, we use [Crystalball gem](https://github.com/toptal/crystalball) to create a test mapping.
The test mapping contains a map of each source files to a list of test files which is dependent of the source file.
This mapping is currently generated using a combination of test coverage tracing and a static mapping.
In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current Merge Request.
In this experiment, each `rspec` job is accompanied with a `minimal` version.
For example, `rspec unit` job has a corresponding `rspec unit minimal` job.
During the experiment, each Merge Request pipeline will contain both versions of the job, running in parallel.
To illustrate this:
```mermaid
graph LR
A --"artifact: list of test files"--> C1 & D1 & E1 & F1
subgraph "prepare stage";
A["detect-tests"];
end
subgraph "test stage";
C["rspec migration"];
C1["rspec migration minimal"];
D["rspec unit"];
D1["rspec unit minimal"];
E["rspec integration"];
E1["rspec integration minimal"];
F["rspec system"];
F1["rspec system minimal"];
end
```
The result of both set of jobs in the pipeline is then compared to identify any false positive.
A list of such pipeline can be found in [Sisense](https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=10492739&udv=833427).
A false positive is defined as a pipeline where the `minimal` jobs passed, but the non-`minimal` jobs failed.
This indicates that the changeset resulted in a test failure, which was not detected by the `minimal` jobs.
Consequently, this signifies a gap in the test mapping used, which would need to be rectified.
#### Findings
After a round of the experiment done in December 2020,
we discovered that it was challenging to achieve a mapping that gives high confidence at the moment,
because of 2 reasons:
- Each identified gap in the test mapping is unique, each needing its own investigation and improvement to the creation of the test mapping.
- There is a large number of flaky tests which added a lot of noise in the data, slowing down the investigation process.
After a merge request has been approved, the pipeline would contain the full RSpec tests. This will ensure that all tests
