@@ -21,7 +21,7 @@ the noise (due to constantly failing tests, flaky tests, and so on) so that new
...
@@ -21,7 +21,7 @@ the noise (due to constantly failing tests, flaky tests, and so on) so that new
- [ ] Quarantine test check-list
- [ ] Quarantine test check-list
-[ ] Follow the [Quarantining Tests guide](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests).
-[ ] Follow the [Quarantining Tests guide](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests).
-[ ] Confirm the test has a [`quarantine:` tag with the specified quarantine type](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantined-test-types).
-[ ] Confirm the test has a [`quarantine:` tag with the specified quarantine type](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantined-test-types).
-[ ] Note if the test should be [quarantined for a specific environment](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/environment_selection.html#quarantining-a-test-for-a-specific-environment).
-[ ] Note if the test should be [quarantined for a specific environment](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/execution_context_selection.html#quarantine-a-test-for-a-specific-environment).
- [ ] Dequarantine test check-list
- [ ] Dequarantine test check-list
-[ ] Follow the [Dequarantining Tests guide](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#dequarantining-tests).
-[ ] Follow the [Dequarantining Tests guide](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#dequarantining-tests).
- [ ] Confirm the test consistently passes on the target GitLab environment(s).
- [ ] Confirm the test consistently passes on the target GitLab environment(s).
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
---
# Environment selection
This file was moved to [another location](execution_context_selection.md).
Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines).
<!-- This redirect file can be deleted after <2021-08-14>. -->
We can specify what environments or pipelines to run tests against using the `only` metadata.
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
## Available switches
| Switch | Function | Type |
| -------| ------- | ----- |
| `tld` | Set the top-level domain matcher | `String` |
| `subdomain` | Set the subdomain matcher | `Array` or `String` |
| `domain` | Set the domain matcher | `String` |
| `production` | Match against production | `Static` |
| `pipeline` | Match against a pipeline | `Array` or `Static`|
WARNING:
You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously.
These options are mutually exclusive. If you want to specify production, you
can control the `tld` and `domain` independently.
## Examples
| Environment or pipeline | Key | Matches (regex for environments, string matching for pipelines) |
it 'runs only in production environment', only: :production do; end
it 'runs only in staging environment', only: { subdomain: :staging } do; end
it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {}
it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end
it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end
end
```
If the test has a `before` or `after`, you must add the `only` metadata
to the outer `RSpec.describe`.
If you want to run an `only: { :pipeline }` tagged test on your local GDK make sure either the `CI_PROJECT_NAME` CI/CD variable is unset, or that the `CI_PROJECT_NAME` variable matches the specified pipeline in the `only: { :pipeline }` tag, or just delete the `only: { :pipeline }` tag.
## Quarantining a test for a specific environment
Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
For instance, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against staging.
## Excluding a test from running in a particular job
Sometimes we need to skip a test in a particular job but allow it to run in other jobs of the same pipeline or environment.
We can do it with the help of `exclude` metadata.
Examples:
```ruby
RSpec.describe 'Excluding' do
it 'skips given a single named job', exclude: { job: 'ee:instance-image' } do; end
it 'skips given a single regex pattern', exclude: { job: '.*:instance-image' } do; end
it 'skips given an array of jobs', exclude: { job: %w[ee:instance-image qa-schedules-browser_ui-3_create] } do; end
it 'skips given an array of regex patterns', exclude: { job: %w[ee:.* qa-schedules-browser_ui.*] } do; end
it 'skips given a mix of strings and regex patterns', exclude: { job: %w[ee:instance-image qa-schedules-browser_ui.*] } do; end
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Execution context selection
Some tests are designed to be run against specific environments, or in specific [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines) or jobs. We can specify the test execution context using the `only` and `except` metadata.
## Available switches
| Switch | Function | Type |
| -------| ------- | ----- |
| `tld` | Set the top-level domain matcher | `String` |
| `subdomain` | Set the subdomain matcher | `Array` or `String` |
| `domain` | Set the domain matcher | `String` |
| `production` | Match the production environment | `Static` |
| `pipeline` | Match a pipeline | `Array` or `Static`|
| `job` | Match a job | `Array` or `Static`|
WARNING:
You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously.
These options are mutually exclusive. If you want to specify production, you
| The `nightly` and `canary` pipelines | `except: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) |
| The `ee:instance` job | `except: { job: 'ee:instance' }` | The `ee:instance` job in any pipeline |
| Any `quarantine` job | `except: { job: '.*quarantine' }` | Any job ending in `quarantine` in any pipeline |
```ruby
RSpec.describe'Area'do
it'runs in any execution context except the production environment',except: :productiondo;end
it'runs in any execution context except the staging environment',except: {subdomain: :staging}do;end
it'runs in any execution context except the nightly pipeline',except: {pipeline: :nightly}do;end
it'runs in any execution context except the ee:instance job',except: {job: 'ee:instance'}do;end
end
```
## Usage notes
If the test has a `before` or `after` block, you must add the `only` or `except` metadata to the outer `RSpec.describe` block.
To run a test tagged with `only` on your local GitLab instance, you can do one of the following:
- Make sure you **do not** have the `CI_PROJECT_NAME` or `CI_JOB_NAME` environment variables set.
- Set the appropriate variable to match the metadata. For example, if the metadata is `only: { pipeline: :nightly }` then set `CI_PROJECT_NAME=nightly`. If the metadata is `only: { job: 'ee:instance' }` then set `CI_JOB_NAME=ee:instance`.
- Temporarily remove the metadata.
To run a test tagged with `except` locally, you can either:
- Make sure you **do not** have the `CI_PROJECT_NAME` or `CI_JOB_NAME` environment variables set.
- Temporarily remove the metadata.
## Quarantine a test for a specific environment
Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
For example, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against `staging`.
@@ -14,7 +14,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
...
@@ -14,7 +14,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| Tag | Description |
| Tag | Description |
|-----|-------------|
|-----|-------------|
| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions)[`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions)[`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
| `:exclude` | The test is excluded from running in a specific job. See [Environment selection](environment_selection.md#excluding-a-test-from-running-in-a-particular-job) for more information. |
| `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. |
| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. |
| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. |
| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. |
| `:github` | The test requires a GitHub personal access token. |
| `:github` | The test requires a GitHub personal access token. |
...
@@ -27,10 +27,10 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
...
@@ -27,10 +27,10 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. |
| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. |
| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. |
| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. |
| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. |
| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. |
| `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. |
| `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). |
| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). |
| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. |
| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). |
| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). |
| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). |
| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. |
| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. |
# The following example is excluded from running in `review-qa-smoke` job
# The following example is excluded from running in `review-qa-smoke` job
# as it proved to be flaky when running against Review App
# as it proved to be flaky when running against Review App
# See https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/11568#note_621999351
# See https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/11568#note_621999351
it'comments on an issue with an attachment',testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/1742',exclude: {job: 'review-qa-smoke'}do
it'comments on an issue with an attachment',testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/1742',except: {job: 'review-qa-smoke'}do
Page::Project::Issue::Show.performdo|show|
Page::Project::Issue::Show.performdo|show|
show.comment('See attached image for scale',attachment: file_to_attach)
show.comment('See attached image for scale',attachment: file_to_attach)
