Merge branch 'qa-make-review-qa-reliable-mandatory' into 'master'

Disallow review-qa-reliable job to fail

See merge request gitlab-org/gitlab!76802

.gitlab/ci/review-apps/qa.gitlab-ci.yml

@@ -155,6 +155,15 @@ allure-report-qa-smoke:
     ALLURE_REPORT_PATH_PREFIX: gitlab-review-smoke
     ALLURE_JOB_NAME: review-qa-smoke
 
+allure-report-qa-reliable:
+  extends:
+    - .allure-report-base
+    - .review:rules:review-qa-reliable-report
+  needs: ["review-qa-reliable"]
+  variables:
+    ALLURE_REPORT_PATH_PREFIX: gitlab-review-reliable
+    ALLURE_JOB_NAME: review-qa-reliable
+
 allure-report-qa-all:
   extends:
     - .allure-report-base

.gitlab/ci/rules.gitlab-ci.yml

@@ -1661,15 +1661,20 @@
 .review:rules:review-qa-reliable:
   rules:
     - when: on_success
-      allow_failure: true
+
+# Since `review-qa-reliable ` isn't allowed to fail, we need to use `when: always` for `review-qa-reliable-report`.
+# See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76756#qa-job-that-run-on_success-and-not-allowed-to-fail-eg-report-qa-smoke
+.review:rules:review-qa-reliable-report:
+  rules:
+    - when: always
 
 .review:rules:review-qa-all:
   rules:
-    - <<: *if-merge-request-labels-run-review-app  # we explicitely don't allow the job to fail in that case
+    - <<: *if-merge-request-labels-run-review-app  # we explicitly don't allow the job to fail in that case
     - <<: *if-dot-com-gitlab-org-merge-request
       changes: *code-patterns
       when: manual
-      allow_failure: true # manual jobs needs to be allowd to fail, otherwise they block the pipeline
+      allow_failure: true # manual jobs needs to be allowed to fail, otherwise they block the pipeline
     - when: on_success
       allow_failure: true
 

doc/development/testing_guide/end_to_end/rspec_metadata_tests.md

@@ -34,7 +34,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
 | `:packages`       | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. |
 | `:quarantine`     | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). |
 | `:relative_url`   | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). |
-| `:reliable`       | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
+| `:reliable`       | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
 | `:repository_storage` |  The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. |
 | `:requires_admin` | The test requires an administrator account. Tests with the tag are excluded when run against Canary and Production environments. |
 | `:requires_git_protocol_v2`   | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. |

doc/development/testing_guide/review_apps.md

@@ -21,13 +21,13 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be
 ## QA runs on Review Apps
 
 On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the
-`review` stage), the `review-qa-smoke` job is automatically started and it runs
-the QA smoke suite.
+`review` stage), the `review-qa-smoke` and `review-qa-reliable` jobs are automatically started. The `review-qa-smoke` runs
+the QA smoke suite and the `review-qa-reliable` executes E2E tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests).
 
 You can also manually start the `review-qa-all`: it runs the full QA suite.
 
 After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by
-the `allure-report-qa-smoke` and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request.
+the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request.
 
 ## Performance Metrics
 
@@ -121,7 +121,7 @@ graph TD
   B[review-build-cng];
   C[review-deploy];
   D[CNG-mirror];
-  E[review-qa-smoke];
+  E[review-qa-smoke, review-qa-reliable];
 
   A -->|once the `prepare` stage is done| B
   B -.->|triggers a CNG-mirror pipeline and wait for it to be done| D
@@ -142,7 +142,7 @@ subgraph "3. gitlab `review` stage"
   end
 
 subgraph "4. gitlab `qa` stage"
-  E[review-qa-smoke<br><br>gitlab-qa runs the smoke suite against the Review App.]
+  E[review-qa-smoke, review-qa-reliable<br><br>gitlab-qa runs the smoke and reliable suites against the Review App.]
   end
 
 subgraph "CNG-mirror pipeline"
@@ -196,7 +196,7 @@ subgraph "CNG-mirror pipeline"
   issue with a link to your merge request. Note that the deployment failure can
   reveal an actual problem introduced in your merge request (that is, this isn't
   necessarily a transient failure)!
-- If the `review-qa-smoke` job keeps failing (note that we already retry it twice),
+- If the `review-qa-smoke` or `review-qa-reliable` job keeps failing (note that we already retry them once),
   please check the job's logs: you could discover an actual problem introduced in
   your merge request. You can also download the artifacts to see screenshots of
   the page at the time the failures occurred. If you don't find the cause of the
@@ -250,7 +250,7 @@ Leading indicators may be health check failures leading to restarts or majority
 The [Review Apps Overview dashboard](https://console.cloud.google.com/monitoring/classic/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d)
 aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy.
 
-### Database related errors in `review-deploy` or `review-qa-smoke`
+### Database related errors in `review-deploy`, `review-qa-smoke`, or `review-qa-reliable`
 
 Occasionally the state of a Review App's database could diverge from the database schema. This could be caused by
 changes to migration files or schema, such as a migration being renamed or deleted. This typically manifests in migration errors such as: