Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
a2b6a6b9
Commit
a2b6a6b9
authored
Jan 18, 2022
by
Rémy Coutable
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
doc: Improve the Flaky tests page
parent
2d74977a
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
9 additions
and
19 deletions
+9
-19
doc/development/testing_guide/flaky_tests.md
doc/development/testing_guide/flaky_tests.md
+9
-19
No files found.
doc/development/testing_guide/flaky_tests.md
View file @
a2b6a6b9
...
@@ -14,14 +14,14 @@ eventually.
...
@@ -14,14 +14,14 @@ eventually.
## Quarantined tests
## Quarantined tests
When a test frequently fails in
`main`
,
When a test frequently fails in
`main`
,
[
a ~"master:broken" issue
](
https://about.gitlab.com/handbook/engineering/workflow/#broken-master
)
create
[
a ~"failure::flaky-test" issue
](
https://about.gitlab.com/handbook/engineering/workflow/#broken-master
)
.
should be created.
If the test cannot be fixed in a timely fashion, there is an impact on the
If the test cannot be fixed in a timely fashion, there is an impact on the
productivity of all the developers, so it should be
placed in quarantine
by
productivity of all the developers, so it should be
quarantined
by
assigning the
`:quarantine`
metadata with the issue URL.
assigning the
`:quarantine`
metadata with the issue URL
, and add the
`~"quarantined test"`
label to the issue
.
```
ruby
```
ruby
it
's
hould succeed
'
,
quarantine:
'https://gitlab.com/gitlab-org/gitlab/-/issues/12345'
do
it
's
ucceeds
'
,
quarantine:
'https://gitlab.com/gitlab-org/gitlab/-/issues/12345'
do
expect
(
response
).
to
have_gitlab_http_status
(
:ok
)
expect
(
response
).
to
have_gitlab_http_status
(
:ok
)
end
end
```
```
...
@@ -32,23 +32,13 @@ This means it is skipped unless run with `--tag quarantine`:
...
@@ -32,23 +32,13 @@ This means it is skipped unless run with `--tag quarantine`:
bin/rspec
--tag
quarantine
bin/rspec
--tag
quarantine
```
```
**
Before putting a test in quarantine, you should make sure that a
~"master:broken" issue exists for it so it doesn't stay in quarantine forever.
**
Once a test is in quarantine, there are 3 choices:
Once a test is in quarantine, there are 3 choices:
-
Should the test be fixed (that is, get rid of its flakiness)?
-
Fix the test (that is, get rid of its flakiness).
-
Should the test be moved to a lower level of testing?
-
Move the test to a lower level of testing.
-
Should the test be removed
entirely (for example, because there's already a
-
Remove the test
entirely (for example, because there's already a
lower-level test, or it's duplicating another same-level test, or it's testing
lower-level test, or it's duplicating another same-level test, or it's testing
too much etc.)?
too much etc.).
### Quarantine tests on the CI
Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
-
`rspec-pg-quarantine`
(CE & EE)
-
`rspec-pg-quarantine-ee`
(EE only)
## Automatic retries and flaky tests detection
## Automatic retries and flaky tests detection
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment