pipelines.md 11.3 KB
Newer Older
1 2
# Pipelines for the GitLab project

3
Pipelines for <https://gitlab.com/gitlab-org/gitlab> and <https://gitlab.com/gitlab-org/gitlab-foss> (as well as the
4 5 6 7 8 9 10 11 12 13 14 15 16 17
`dev` instance's mirrors) are configured in the usual
[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml)
which itself includes files under
[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/ci)
for easier maintenance.

We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfooding)
GitLab [CI/CD features and best-practices](../ci/yaml/README.md)
as much as possible.

## Stages

The current stages are:

18 19
- `sync`: This stage is used to synchronize changes from <https://gitlab.com/gitlab-org/gitlab> to
  <https://gitlab.com/gitlab-org/gitlab-foss>.
20 21 22 23 24 25
- `prepare`: This stage includes jobs that prepare artifacts that are needed by
  jobs in subsequent stages.
- `quick-test`: This stage includes test jobs that should run first and fail the
  pipeline early (currently used to run Geo tests when the branch name starts
  with `geo-`, `geo/`, or ends with `-geo`).
- `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs.
26 27
- `post-test`: This stage includes jobs that build reports or gather data from
  the `test` stage's jobs (e.g. coverage, Knapsack metadata etc.).
28 29 30 31 32 33
- `review-prepare`: This stage includes a job that build the CNG images that are
  later used by the (Helm) Review App deployment (see
  [Review Apps](testing_guide/review_apps.md) for details).
- `review`: This stage includes jobs that deploy the GitLab and Docs Review Apps.
- `qa`: This stage includes jobs that perform QA tasks against the Review App
  that is deployed in the previous stage.
34 35
- `post-qa`: This stage includes jobs that build reports or gather data from
  the `qa` stage's jobs (e.g. Review App performance report).
36
- `notification`: This stage includes jobs that sends notifications about pipeline status.
37
- `pages`: This stage includes a job that deploys the various reports as
38
  GitLab Pages (e.g. <https://gitlab-org.gitlab.io/gitlab/coverage-ruby/>,
39 40 41 42 43 44
  <https://gitlab-org.gitlab.io/gitlab/coverage-javascript/>,
  <https://gitlab-org.gitlab.io/gitlab/webpack-report/>).

## Default image

The default image is currently
45
`registry.gitlab.com/gitlab-org/gitlab-build-images:ruby-2.6.3-golang-1.12-git-2.24-lfs-2.9-chrome-73.0-node-12.x-yarn-1.16-postgresql-9.6-graphicsmagick-1.3.33`.
46

47
It includes Ruby 2.6.3, Go 1.12, Git 2.24, Git LFS 2.9, Chrome 73, Node 12, Yarn 1.16,
48 49 50 51 52 53 54 55
PostgreSQL 9.6, and Graphics Magick 1.3.33.

The images used in our pipelines are configured in the
[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images)
project, which is push-mirrored to <https://dev.gitlab.org/gitlab/gitlab-build-images>
for redundancy.

The current version of the build images can be found in the
56
["Used by GitLab section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml).
57 58 59 60

## Default variables

In addition to the [predefined variables](../ci/variables/predefined_variables.md),
61 62
each pipeline includes default variables defined in
<https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml>.
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

## Common job definitions

Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends)
that are scoped to a single
[configuration parameter](../ci/yaml/README.md#configuration-parameters).

These common definitions are:

- `.default-tags`: Ensures a job has the `gitlab-org` tag to ensure it's using
  our dedicated runners.
- `.default-retry`: Allows a job to retry upon `unknown_failure`, `api_failure`,
  `runner_system_failure`.
- `.default-before_script`: Allows a job to use a default `before_script` definition
  suitable for Ruby/Rails tasks that may need a database running (e.g. tests).
- `.default-cache`: Allows a job to use a default `cache` definition suitable for
  Ruby/Rails and frontend tasks.
- `.default-only`: Restricts the cases where a job is created. This currently
  includes `master`, `/^[\d-]+-stable(-ee)?$/` (stable branches),
82
  `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), `merge_requests`, `tags`.
83
  Note that jobs won't be created for branches with this default configuration.
84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103
- `.only:variables-canonical-dot-com`: Only creates a job if the project is
  located under <https://gitlab.com/gitlab-org>.
- `.only:variables_refs-canonical-dot-com-schedules`: Same as
  `.only:variables-canonical-dot-com` but add the condition that pipeline is scheduled.
- `.except:refs-deploy`: Don't create a job if the `ref` is an auto-deploy branch.
- `.except:refs-master-tags-stable-deploy`: Don't create a job if the `ref` is one of:
  - `master`
  - a tag
  - a stable branch
  - an auto-deploy branch
- `.only:kubernetes`: Only creates a job if a Kubernetes integration is enabled
  on the project.
- `.only-review`: This extends from:
  - `.only:variables-canonical-dot-com`
  - `.only:kubernetes`
  - `.except:refs-master-tags-stable-deploy`
- `.only-review-schedules`: This extends from:
  - `.only:variables_refs-canonical-dot-com-schedules`
  - `.only:kubernetes`
  - `.except:refs-deploy`
104 105 106 107 108 109
- `.use-pg9`: Allows a job to use the `postgres:9.6` and `redis:alpine` services.
- `.use-pg10`: Allows a job to use the `postgres:10.9` and `redis:alpine` services.
- `.use-pg9-ee`: Same as `.use-pg9` but also use the
  `docker.elastic.co/elasticsearch/elasticsearch:5.6.12` services.
- `.use-pg10-ee`: Same as `.use-pg10` but also use the
  `docker.elastic.co/elasticsearch/elasticsearch:5.6.12` services.
110
- `.only-ee`: Only creates a job for the `gitlab` or `gitlab-ee` project.
111
- `.only-ee-as-if-foss`: Same as `.only-ee` but simulate the FOSS project by
112
  setting the `FOSS_ONLY='1'` environment variable.
113 114 115 116 117 118 119 120

## Changes detection

If a job extends from `.default-only` (and most of the jobs should), it can restrict
the cases where it should be created
[based on the changes](../ci/yaml/README.md#onlychangesexceptchanges)
from a commit or MR by extending from the following CI definitions:

121 122 123 124 125 126 127
- `.only:changes-code`: Allows a job to only be created upon code-related changes.
- `.only:changes-qa`: Allows a job to only be created upon QA-related changes.
- `.only:changes-docs`: Allows a job to only be created upon docs-related changes.
- `.only:changes-graphql`: Allows a job to only be created upon GraphQL-related changes.
- `.only:changes-code-backstage`: Allows a job to only be created upon code-related or backstage-related (e.g. Danger, RuboCop, specs) changes.
- `.only:changes-code-qa`: Allows a job to only be created upon code-related or QA-related changes.
- `.only:changes-code-backstage-qa`: Allows a job to only be created upon code-related, backstage-related (e.g. Danger, RuboCop, specs) or QA-related changes.
128 129 130 131

**See <https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml>
for the list of exact patterns.**

132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189
## Rules conditions and changes patterns

We're making use of the [`rules` keyword](https://docs.gitlab.com/ee/ci/yaml/#rules) but we're currently
duplicating the `if` conditions and `changes` patterns lists since they cannot be shared accross
`include`d files as we do with `extends`.

**If you update an `if` condition or `changes`
patterns list, make sure to mass-update those accross all the CI config files (i.e. `.gitlab/ci/*.yml`).**

### Canonical commits only

This condition limits jobs creation to commits under the `gitlab-org/` top-level group
on GitLab.com only. This is similar to the `.only:variables-canonical-dot-com` CI definition:

```yaml
.if-canonical-gitlab: &if-canonical-gitlab
  if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/)/'
```

### Canonical merge requests only

Same as the "Canonical commits only" condition above but further limits jobs creation
to merge requests only (i.e. this won't run for `master`, stable or auto-deploy branches).
This is similar to the `.only:variables-canonical-dot-com` + `.except:refs-master-tags-stable-deploy`
CI definitions:

```yaml
.if-canonical-gitlab-and-merge-request: &if-canonical-gitlab-and-merge-request
  if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/)/ && $CI_MERGE_REQUEST_IID'
```

### Code changes patterns

Similar patterns as for `.only:changes-code`:

```yaml
.code-patterns: &code-patterns
  - ...
```

### QA changes patterns

Similar patterns as for `.only:changes-qa`:

```yaml
.qa-patterns: &qa-patterns
  - ...
```

### Code and QA changes patterns

Similar patterns as for `.only:changes-code-qa`:

```yaml
.code-qa-patterns: &code-qa-patterns
  - ...
```

190 191 192 193 194 195 196 197
## Directed acyclic graph

We're using the [`needs:`](../ci/yaml/README.md#needs) keyword to
execute jobs out of order for the following jobs:

```mermaid
graph RL;
  A[setup-test-env];
198
  B["gitlab:assets:compile pull-push-cache<br/>(master only)"];
199 200 201 202 203 204 205 206
  C[gitlab:assets:compile pull-cache];
  D["cache gems<br/>(master and tags only)"];
  E[review-build-cng];
  F[build-qa-image];
  G[review-deploy];
  G2["schedule:review-deploy<br/>(master only)"];
  H[karma];
  I[jest];
207
  J["compile-assets pull-push-cache<br/>(master only)"];
208 209 210 211 212
  K[compile-assets pull-cache];
  L[webpack-dev-server];
  M[coverage];
  N[pages];
  O[static-analysis];
213
  P["schedule:package-and-qa<br/>(master schedule only)"];
214 215
  Q[package-and-qa];
  R[package-and-qa-manual];
216 217
  S["RSpec<br/>(e.g. rspec unit pg9)"]
  T[retrieve-tests-metadata];
218 219 220

subgraph "`prepare` stage"
    A
221 222
    B
    C
223 224
    F
    K
225 226
    J
    T
227 228 229 230
    end

subgraph "`test` stage"
    D --> |needs| A;
231 232 233 234 235 236 237 238 239 240 241
    H -.-> |needs and depends on| A;
    H -.-> |needs and depends on| K;
    I -.-> |needs and depends on| A;
    I -.-> |needs and depends on| K;
    L -.-> |needs and depends on| A;
    L -.-> |needs and depends on| K;
    O -.-> |needs and depends on| A;
    O -.-> |needs and depends on| K;
    S -.-> |needs and depends on| A;
    S -.-> |needs and depends on| K;
    S -.-> |needs and depends on| T;
242 243 244 245
    downtime_check --> |needs and depends on| A;
    db:* --> |needs| A;
    gitlab:setup --> |needs| A;
    downtime_check --> |needs and depends on| A;
246
    graphql-docs-verify --> |needs| A;
247 248 249 250
    end

subgraph "`review-prepare` stage"
    E --> |needs| C;
251
    X["schedule:review-build-cng<br/>(master schedule only)"] --> |needs| C;
252 253 254
    end

subgraph "`review` stage"
255 256
    G
    G2
257 258 259 260 261 262 263
    end

subgraph "`qa` stage"
    Q --> |needs| C;
    Q --> |needs| F;
    R --> |needs| C;
    R --> |needs| F;
264
    P --> |needs| C;
265
    P --> |needs| F;
266 267 268 269 270
    review-qa-smoke -.-> |needs and depends on| G;
    review-qa-all -.-> |needs and depends on| G;
    review-performance -.-> |needs and depends on| G;
    X2["schedule:review-performance<br/>(master only)"] -.-> |needs and depends on| G2;
    dast -.-> |needs and depends on| G;
271 272
    end

273 274 275 276 277
subgraph "`notification` stage"
    NOTIFICATION1["schedule:package-and-qa:notify-success<br>(on_success)"] -.-> |needs| P;
    NOTIFICATION2["schedule:package-and-qa:notify-failure<br>(on_failure)"] -.-> |needs| P;
    end

278 279 280 281 282
subgraph "`post-test` stage"
    M
    end

subgraph "`pages` stage"
283
    N -.-> |depends on| C;
284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300
    N -.-> |depends on| H;
    N -.-> |depends on| M;
    end
```

## Test jobs

Consult [GitLab tests in the Continuous Integration (CI) context](testing_guide/ci.md)
for more information.

## Review app jobs

Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information.

---

[Return to Development documentation](README.md)