Commit 7ff0dbc5 authored by Marcia Ramos's avatar Marcia Ramos

Merge branch 'docs-junit-test-reports' into 'master'

Documentation for JUnit XML Test Summary In MR widget

Closes #49867

See merge request gitlab-org/gitlab-ce!21061
parents 218fc3e3 6e9e61dc
...@@ -133,6 +133,7 @@ scales to run your tests faster. ...@@ -133,6 +133,7 @@ scales to run your tests faster.
- [GitLab CI/CD](ci/README.md): Explore the features and capabilities of Continuous Integration, Continuous Delivery, and Continuous Deployment with GitLab. - [GitLab CI/CD](ci/README.md): Explore the features and capabilities of Continuous Integration, Continuous Delivery, and Continuous Deployment with GitLab.
- [Review Apps](ci/review_apps/index.md): Preview changes to your app right from a merge request. - [Review Apps](ci/review_apps/index.md): Preview changes to your app right from a merge request.
- [Pipeline Graphs](ci/pipelines.md#pipeline-graphs) - [Pipeline Graphs](ci/pipelines.md#pipeline-graphs)
- [JUnit test reports](ci/junit_test_reports.md)
### Package ### Package
......
...@@ -43,6 +43,10 @@ There's also a collection of repositories with [example projects](https://gitlab ...@@ -43,6 +43,10 @@ There's also a collection of repositories with [example projects](https://gitlab
- [Using `dpl` as deployment tool](deployment/README.md) - [Using `dpl` as deployment tool](deployment/README.md)
- [The `.gitlab-ci.yml` file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) - [The `.gitlab-ci.yml` file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml)
## Test Reports
[Collect test reports in Verify stage](../junit_test_reports.md).
## Code Quality analysis ## Code Quality analysis
**(Starter)** [Analyze your project's Code Quality](code_quality.md). **(Starter)** [Analyze your project's Code Quality](code_quality.md).
......
# JUnit test reports
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45318) in GitLab 11.2.
Requires GitLab Runner 11.2 and above.
## Overview
It is very common that a [CI/CD pipeline](pipelines.md) contains a
test job that will verify your code.
If the tests fail, the pipeline fails and users get notified. The person that
works on the merge request will have to check the job logs and see where the
tests failed so that they can fix them.
You can configure your job to use JUnit test reports, and GitLab will display a
report on the merge request so that it's easier and faster to identify the
failure without having to check the entire log.
## Use cases
Consider the following workflow:
1. Your `master` branch is rock solid, your project is using GitLab CI/CD and
your pipelines indicate that there isn't anything broken.
1. Someone from you team submits a merge request, a test fails and the pipeline
gets the known red icon. To investigate more, you have to go through the job
logs to figure out the cause of the failed test, which usually contain
thousands of lines.
1. You configure the JUnit test reports and immediately GitLab collects and
exposes them in the merge request. No more searching in the job logs.
1. Your development and debugging workflow becomes easier, faster and efficient.
## How it works
First, GitLab Runner uploads all JUnit XML files as artifacts to GitLab. Then,
when you visit a merge request, GitLab starts comparing the head and base branch's
JUnit test reports, where:
- The base branch is the target branch (usually `master`).
- The head branch is the source branch (the latest pipeline in each merge request).
The reports panel has a summary showing how many tests failed and how many were fixed.
If no comparison can be done because data for the base branch is not available,
the panel will just show the list of failed tests for head.
There are three types of results:
1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch
1. **Existing failures:** Test cases which failed on base branch and failed on head branch
1. **Resolved failures:** Test cases which failed on base branch and passed on head branch
Each entry in the panel will show the test name and its type from the list
above. Clicking on the test name will open a modal window with details of its
execution time and the error output.
![Test Reports Widget](img/junit_test_report.png)
## How to set it up
NOTE: **Note:**
For a list of supported languages on JUnit tests, check the
[Wikipedia article](https://en.wikipedia.org/wiki/JUnit#Ports).
To enable the JUnit reports in merge requests, you need to add
[`artifacts:reports:junit`](yaml/README.md#artifacts-reports-junit)
in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports.
In the following examples, the job in the `test` stage runs and GitLab
collects the JUnit test report from each job. After each job is executed, the
XML reports are stored in GitLab as artifacts and their results are shown in the
merge request widget.
### Ruby example
Use the following job in `.gitlab-ci.yml`:
```yaml
## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec
ruby:
stage: test
script:
- bundle install
- rspec spec/lib/ --format RspecJunitFormatter --out rspec.xml
artifacts:
reports:
junit: rspec.xml
```
### Go example
Use the following job in `.gitlab-ci.yml`:
```yaml
## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go
golang:
stage: test
script:
- go get -u github.com/jstemmer/go-junit-report
- go test -v 2>&1 | go-junit-report > report.xml
artifacts:
reports:
junit: report.xml
```
...@@ -1092,6 +1092,52 @@ job: ...@@ -1092,6 +1092,52 @@ job:
expire_in: 1 week expire_in: 1 week
``` ```
### `artifacts:reports`
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20390) in
GitLab 11.2. Requires GitLab Runner 11.2 and above.
The `reports` keyword is used for collecting test reports from jobs and
exposing them in GitLab's UI (merge requests, pipeline views). Read how to use
this with [JUnit reports](#artifacts-reports-junit).
NOTE: **Note:**
The test reports are collected regardless of the job results (success or failure).
You can use [`artifacts:expire_in`](#artifacts-expire_in) to set up an expiration
date for their artifacts.
#### `artifacts:reports:junit`
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20390) in
GitLab 11.2. Requires GitLab Runner 11.2 and above.
The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html)
as artifacts. Although JUnit was originally developed in Java, there are many
[third party ports](https://en.wikipedia.org/wiki/JUnit#Ports) for other
languages like Javascript, Python, Ruby, etc.
Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool:
```yaml
rspec:
stage: test
script:
- bundle install
- rspec --format RspecJunitFormatter --out rspec.xml
artifacts:
reports:
junit: rspec.xml
```
The collected JUnit reports will be uploaded to GitLab as an artifact and will
be automatically [shown in merge requests](../junit_test_reports.md).
NOTE: **Note:**
In case the JUnit tool you use exports to multiple XML files, you can specify
multiple test report paths within a single job
(`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`) and they will be automatically
concatenated into a single file.
## `dependencies` ## `dependencies`
> Introduced in GitLab 8.6 and GitLab Runner v1.1.1. > Introduced in GitLab 8.6 and GitLab Runner v1.1.1.
......
...@@ -43,8 +43,7 @@ A. Consider you are a software developer working in a team: ...@@ -43,8 +43,7 @@ A. Consider you are a software developer working in a team:
1. You checkout a new branch, and submit your changes through a merge request 1. You checkout a new branch, and submit your changes through a merge request
1. You gather feedback from your team 1. You gather feedback from your team
1. You work on the implementation optimizing code with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** 1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD
1. You build and test your changes with GitLab CI/CD
1. You request the approval from your manager 1. You request the approval from your manager
1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter)
1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#manual-actions) for GitLab CI/CD 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#manual-actions) for GitLab CI/CD
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment