index.md 56.2 KB
Newer Older
1 2
# Auto DevOps

3 4
> - [Introduced][ce-37115] in GitLab 10.0.
> - Generally available on GitLab 11.0.
5

6 7 8
Auto DevOps provides pre-defined CI/CD configuration which allows you to automatically detect, build, test,
deploy, and monitor your applications. Leveraging CI/CD best practices and tools, Auto DevOps aims
to simplify the setup and execution of a mature & modern software development lifecycle.
9 10 11

## Overview

12 13 14 15 16 17
With Auto DevOps, the software development process becomes easier to set up
as every project can have a complete workflow from verification to monitoring
with minimal configuration. Just push your code and GitLab takes
care of everything else. This makes it easier to start new projects and brings
consistency to how applications are set up throughout a company.

18
For an introduction to Auto DevOps, watch [AutoDevOps in GitLab 11.0](https://youtu.be/0Tc0YYBxqi4).
19

20 21
## Enabled by default

22 23
Starting with GitLab 11.3, the Auto DevOps pipeline is enabled by default for all
projects. If it has not been explicitly enabled for the project, Auto DevOps will be automatically
24 25
disabled on the first pipeline failure. Your project will continue to use an alternative
[CI/CD configuration file](../../ci/yaml/README.md) if one is found. A GitLab
26
administrator can [change this setting](../../user/admin_area/settings/continuous_integration.md#auto-devops-core-only)
27 28
in the admin area.

29 30 31
## Quick start

If you are using GitLab.com, see the [quick start guide](quick_start_guide.md)
32 33 34 35 36 37 38
for how to use Auto DevOps with GitLab.com and a Kubernetes cluster on Google Kubernetes
Engine (GKE).

If you are using a self-hosted instance of GitLab, you will need to configure the
[Google OAuth2 OmniAuth Provider](../../integration/google.md) before
you can configure a cluster on GKE. Once this is set up, you can follow the steps on the
[quick start guide](quick_start_guide.md) to get started.
39

40 41
## Comparison to application platforms and PaaS

42 43
Auto DevOps provides functionality that is often included in an application
platform or a Platform as a Service (PaaS). It takes inspiration from the
44
innovative work done by [Heroku](https://www.heroku.com/) and goes beyond it
45
in multiple ways:
46

47 48 49 50 51 52 53 54 55 56
- Auto DevOps works with any Kubernetes cluster; you're not limited to running
  on GitLab's infrastructure. (Note that many features also work without Kubernetes).
- There is no additional cost (no markup on the infrastructure costs), and you
  can use a self-hosted Kubernetes cluster or Containers as a Service on any
  public cloud (for example, [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/)).
- Auto DevOps has more features including security testing, performance testing,
  and code quality testing.
- Auto DevOps offers an incremental graduation path. If you need advanced customizations,
  you can start modifying the templates without having to start over on a
  completely different platform. Review the [customizing](#customizing) section for more information.
57 58

## Features
59

60
Comprised of a set of stages, Auto DevOps brings these best practices to your
61
project in a simple and automatic way:
62

63 64
1. [Auto Build](#auto-build)
1. [Auto Test](#auto-test)
65 66 67 68 69
1. [Auto Code Quality](#auto-code-quality-starter) **(STARTER)**
1. [Auto SAST (Static Application Security Testing)](#auto-sast-ultimate) **(ULTIMATE)**
1. [Auto Dependency Scanning](#auto-dependency-scanning-ultimate) **(ULTIMATE)**
1. [Auto License Management](#auto-license-management-ultimate) **(ULTIMATE)**
1. [Auto Container Scanning](#auto-container-scanning-ultimate) **(ULTIMATE)**
70
1. [Auto Review Apps](#auto-review-apps)
71
1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast-ultimate) **(ULTIMATE)**
72
1. [Auto Deploy](#auto-deploy)
73
1. [Auto Browser Performance Testing](#auto-browser-performance-testing-premium) **(PREMIUM)**
74
1. [Auto Monitoring](#auto-monitoring)
75

76 77
As Auto DevOps relies on many different components, it's good to have a basic
knowledge of the following:
78

79 80 81 82 83
- [Kubernetes](https://kubernetes.io/docs/home/)
- [Helm](https://docs.helm.sh/)
- [Docker](https://docs.docker.com)
- [GitLab Runner](https://docs.gitlab.com/runner/)
- [Prometheus](https://prometheus.io/docs/introduction/overview/)
84

85 86
Auto DevOps provides great defaults for all the stages; you can, however,
[customize](#customizing) almost everything to your needs.
87

88 89
For an overview on the creation of Auto DevOps, read the blog post [From 2/3 of the Self-Hosted Git Market, to the Next-Generation CI System, to Auto DevOps](https://about.gitlab.com/2017/06/29/whats-next-for-gitlab-ci/).

90
## Requirements
91 92

To make full use of Auto DevOps, you will need:
93

94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
- **GitLab Runner** (needed for all stages) - Your Runner needs to be
  configured to be able to run Docker. Generally this means using the
  [Docker](https://docs.gitlab.com/runner/executors/docker.html) or [Kubernetes
  executor](https://docs.gitlab.com/runner/executors/kubernetes.html), with
  [privileged mode enabled](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode).
  The Runners do not need to be installed in the Kubernetes cluster, but the
  Kubernetes executor is easy to use and is automatically autoscaling.
  Docker-based Runners can be configured to autoscale as well, using [Docker
  Machine](https://docs.gitlab.com/runner/install/autoscaling.html). Runners
  should be registered as [shared Runners](../../ci/runners/README.md#registering-a-shared-runner)
  for the entire GitLab instance, or [specific Runners](../../ci/runners/README.md#registering-a-specific-runner)
  that are assigned to specific projects.
- **Base domain** (needed for Auto Review Apps and Auto Deploy) - You will need
  a domain configured with wildcard DNS which is going to be used by all of your
  Auto DevOps applications. [Read the specifics](#auto-devops-base-domain).
- **Kubernetes** (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring) -
  To enable deployments, you will need Kubernetes 1.5+. You need a [Kubernetes cluster][kubernetes-clusters]
  for the project, or a Kubernetes [default service template](../../user/project/integrations/services_templates.md)
  for the entire GitLab installation.
  - **A load balancer** - You can use NGINX ingress by deploying it to your
    Kubernetes cluster using the
    [`nginx-ingress`](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress)
    Helm chart.
- **Prometheus** (needed for Auto Monitoring) - To enable Auto Monitoring, you
  will need Prometheus installed somewhere (inside or outside your cluster) and
  configured to scrape your Kubernetes cluster. To get response metrics
  (in addition to system metrics), you need to
  [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring).
  The [Prometheus service](../../user/project/integrations/prometheus.md)
  integration needs to be enabled for the project, or enabled as a
  [default service template](../../user/project/integrations/services_templates.md)
  for the entire GitLab installation.
126

127 128 129
If you do not have Kubernetes or Prometheus installed, then Auto Review Apps,
Auto Deploy, and Auto Monitoring will be silently skipped.

130
## Auto DevOps base domain
131 132

The Auto DevOps base domain is required if you want to make use of [Auto
133
Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It can be defined
134
in any of the following places:
135

136
- either under the cluster's settings, whether for [projects](../../user/project/clusters/index.md#base-domain) or [groups](../../user/group/clusters/index.md#base-domain)
137
- or in instance-wide settings in the **admin area > Settings** under the "Continuous Integration and Delivery" section
138 139 140
- or at the project level as a variable: `KUBE_INGRESS_BASE_DOMAIN`
- or at the group level as a variable: `KUBE_INGRESS_BASE_DOMAIN`.

141
The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of precedence
142
as other environment [variables](../../ci/variables/README.md#priority-of-environment-variables).
143

144 145 146 147
NOTE: **Note**
`AUTO_DEVOPS_DOMAIN` environment variable is deprecated and
[is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959).

148
A wildcard DNS A record matching the base domain(s) is required, for example,
149 150
given a base domain of `example.com`, you'd need a DNS entry like:

151
```text
152 153 154
*.example.com   3600     A     1.2.3.4
```

155
In this case, `example.com` is the domain name under which the deployed apps will be served,
156
and `1.2.3.4` is the IP address of your load balancer; generally NGINX
157
([see requirements](#requirements)). How to set up the DNS record is beyond
158 159
the scope of this document; you should check with your DNS provider.

160 161 162
Alternatively you can use free public services like [nip.io](http://nip.io)
which provide automatic wildcard DNS without any configuration. Just set the
Auto DevOps base domain to `1.2.3.4.nip.io`.
163

164 165
Once set up, all requests will hit the load balancer, which in turn will route
them to the Kubernetes pods that run your application(s).
166

167 168 169 170 171
NOTE: **Note:**
From GitLab 11.8, `KUBE_INGRESS_BASE_DOMAIN` replaces `AUTO_DEVOPS_DOMAIN`.
Support for `AUTO_DEVOPS_DOMAIN` was [removed in GitLab
12.0](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959).

172
## Using multiple Kubernetes clusters **(PREMIUM)**
173 174 175

When using Auto DevOps, you may want to deploy different environments to
different Kubernetes clusters. This is possible due to the 1:1 connection that
176
[exists between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters-premium).
177

178 179
In the [Auto DevOps template] (used behind the scenes by Auto DevOps), there
are currently 3 defined environment names that you need to know:
180 181 182 183 184 185 186 187

- `review/` (every environment starting with `review/`)
- `staging`
- `production`

Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so
except for the environment scope, they would also need to have a different
domain they would be deployed to. This is why you need to define a separate
188
`KUBE_INGRESS_BASE_DOMAIN` variable for all the above
189
[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium).
190 191 192 193

The following table is an example of how the three different clusters would
be configured.

194
| Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes |
195 196 197
| ------------ | -------------- | ----------------------------- | ------------- | ------ |
| review       |  `review/*`    | `review.example.com`  | `review/*`      | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. |
| staging      |  `staging`     | `staging.example.com` | `staging`       | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). |
198
| production   |  `production`  | `example.com`         | `production`    | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](#incremental-rollout-to-production-premium). |
199 200 201 202 203 204

To add a different cluster for each environment:

1. Navigate to your project's **Operations > Kubernetes** and create the Kubernetes clusters
   with their respective environment scope as described from the table above.

205
   ![Auto DevOps multiple clusters](img/autodevops_multiple_clusters.png)
206 207

1. After the clusters are created, navigate to each one and install Helm Tiller
208
   and Ingress. Wait for the Ingress IP address to be assigned.
209 210
1. Make sure you have [configured your DNS](#auto-devops-base-domain) with the
   specified Auto DevOps domains.
211 212
1. Navigate to each cluster's page, through **Operations > Kubernetes**,
   and add the domain based on its Ingress IP address.
213 214 215 216 217 218

Now that all is configured, you can test your setup by creating a merge request
and verifying that your app is deployed as a review app in the Kubernetes
cluster with the `review/*` environment scope. Similarly, you can check the
other environments.

219
## Enabling/Disabling Auto DevOps
220

221 222
When first using Auto Devops, review the [requirements](#requirements) to ensure all necessary components to make
full use of Auto DevOps are available. If this is your fist time, we recommend you follow the
223
[quick start guide](quick_start_guide.md).
224

225
GitLab.com users can enable/disable Auto DevOps at the project-level only. Self-managed users
226
can enable/disable Auto DevOps at the project-level, group-level or instance-level.
227

228
### At the instance level (Administrators only)
229

230 231 232
Even when disabled at the instance level, group owners and project maintainers can still enable
Auto DevOps at the group and project level, respectively.

233 234 235 236 237
1. Go to **Admin area > Settings > Continuous Integration and Deployment**.
1. Toggle the checkbox labeled **Default to Auto DevOps pipeline for all projects**.
1. If enabling, optionally set up the Auto DevOps [base domain](#auto-devops-base-domain) which will be used for Auto Deploy and Auto Review Apps.
1. Click **Save changes** for the changes to take effect.

238
### At the group level
239 240 241

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/52447) in GitLab 11.10.

242 243
Only administrators and group owners can enable or disable Auto DevOps at the group level.

244 245 246 247 248 249 250 251 252 253
To enable or disable Auto DevOps at the group-level:

1. Go to group's **Settings > CI/CD > Auto DevOps** page.
1. Toggle the **Default to Auto DevOps pipeline** checkbox (checked to enable, unchecked to disable).
1. Click **Save changes** button for the changes to take effect.

When enabling or disabling Auto DevOps at group-level, group configuration will be implicitly used for
the subgroups and projects inside that group, unless Auto DevOps is specifically enabled or disabled on
the subgroup or project.

254
### At the project level
255

256 257
If enabling, check that your project doesn't have a `.gitlab-ci.yml`, or if one exists, remove it.

258
1. Go to your project's **Settings > CI/CD > Auto DevOps**.
259 260
1. Toggle the **Default to Auto DevOps pipeline** checkbox (checked to enable, unchecked to disable)
1. When enabling, it's optional but recommended to add in the [base domain](#auto-devops-base-domain)
261 262 263
   that will be used by Auto DevOps to [deploy your application](#auto-deploy)
   and choose the [deployment strategy](#deployment-strategy).
1. Click **Save changes** for the changes to take effect.
264

265
When the feature has been enabled, an Auto DevOps pipeline is triggered on the default branch.
266

267
### Enable for a percentage of projects
268

269 270 271 272 273
There is also a feature flag to enable Auto DevOps by default to your chosen percentage of projects.

This can be enabled from the console with the following, which uses the example of 10%:

`Feature.get(:force_autodevops_on_by_default).enable_percentage_of_actors(10)`
274

275
### Deployment strategy
276

277 278 279 280
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/38542) in GitLab 11.0.

You can change the deployment strategy used by Auto DevOps by going to your
project's **Settings > CI/CD > Auto DevOps**.
281 282 283

The available options are:

284 285 286
- **Continuous deployment to production**: Enables [Auto Deploy](#auto-deploy)
  with `master` branch directly deployed to production.
- **Continuous deployment to production using timed incremental rollout**: Sets the
287
  [`INCREMENTAL_ROLLOUT_MODE`](#timed-incremental-rollout-to-production-premium) variable
288 289 290
  to `timed`, and production deployment will be executed with a 5 minute delay between
  each increment in rollout.
- **Automatic deployment to staging, manual deployment to production**: Sets the
291
  [`STAGING_ENABLED`](#deploy-policy-for-staging-and-production-environments) and
292
  [`INCREMENTAL_ROLLOUT_MODE`](#incremental-rollout-to-production-premium) variables
293 294 295 296
  to `1` and `manual`. This means:

  - `master` branch is directly deployed to staging.
  - Manual actions are provided for incremental rollout to production.
297

298 299
## Stages of Auto DevOps

300 301
The following sections describe the stages of Auto DevOps. Read them carefully
to understand how each one works.
302 303 304

### Auto Build

305 306
Auto Build creates a build of the application using an existing `Dockerfile` or
Heroku buildpacks.
307 308

Either way, the resulting Docker image is automatically pushed to the
walkafwalka's avatar
walkafwalka committed
309
[Container Registry][container-registry] and tagged with the commit SHA or tag.
310

311 312 313 314 315
#### Auto Build using a Dockerfile

If a project's repository contains a `Dockerfile`, Auto Build will use
`docker build` to create a Docker image.

316 317
If you are also using Auto Review Apps and Auto Deploy and choose to provide
your own `Dockerfile`, make sure you expose your application to port
Evan Read's avatar
Evan Read committed
318
`5000` as this is the port assumed by the
319
[default Helm chart](https://gitlab.com/gitlab-org/charts/auto-deploy-app). Alternatively you can override the default values by [customizing the Auto Deploy helm chart](#custom-helm-chart)
320

321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343
#### Auto Build using Heroku buildpacks

Auto Build builds an application using a project's `Dockerfile` if present, or
otherwise it will use [Herokuish](https://github.com/gliderlabs/herokuish)
and [Heroku buildpacks](https://devcenter.heroku.com/articles/buildpacks)
to automatically detect and build the application into a Docker image.

Each buildpack requires certain files to be in your project's repository for
Auto Build to successfully build your application. For example, the following
files are required at the root of your application's repository, depending on
the language:

- A `Pipfile` or `requirements.txt` file for Python projects.
- A `Gemfile` or `Gemfile.lock` file for Ruby projects.

For the requirements of other languages and frameworks, read the
[buildpacks docs](https://devcenter.heroku.com/articles/buildpacks#officially-supported-buildpacks).

TIP: **Tip:**
If Auto Build fails despite the project meeting the buildpack requirements, set
a project variable `TRACE=true` to enable verbose logging, which may help to
troubleshoot.

344 345
### Auto Test

346
Auto Test automatically runs the appropriate tests for your application using
347
[Herokuish](https://github.com/gliderlabs/herokuish) and [Heroku
348 349 350 351 352
buildpacks](https://devcenter.heroku.com/articles/buildpacks) by analyzing
your project to detect the language and framework. Several languages and
frameworks are detected automatically, but if your language is not detected,
you may succeed with a [custom buildpack](#custom-buildpacks). Check the
[currently supported languages](#currently-supported-languages).
353 354 355 356

Auto Test uses tests you already have in your application. If there are no
tests, it's up to you to add them.

357
### Auto Code Quality **(STARTER)**
358

359 360
Auto Code Quality uses the
[Code Quality image](https://gitlab.com/gitlab-org/security-products/codequality) to run
361 362
static analysis and other code checks on the current code. The report is
created, and is uploaded as an artifact which you can later download and check
363 364
out.

365
Any differences between the source and target branches are also
366
[shown in the merge request widget](../../user/project/merge_requests/code_quality.md).
367

368
### Auto SAST **(ULTIMATE)**
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
369

370
> Introduced in [GitLab Ultimate][ee] 10.3.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
371 372

Static Application Security Testing (SAST) uses the
373
[SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) to run static
374
analysis on the current code and checks for potential security issues. The
375
Auto SAST stage will be skipped on licenses other than Ultimate and requires GitLab Runner 11.5 or above.
376 377

Once the report is created, it's uploaded as an artifact which you can later download and
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
378 379
check out.

380
Any security warnings are also shown in the merge request widget. Read more how
381
[SAST works](../../user/application_security/sast/index.md).
382

383
### Auto Dependency Scanning **(ULTIMATE)**
384 385 386 387 388

> Introduced in [GitLab Ultimate][ee] 10.7.

Dependency Scanning uses the
[Dependency Scanning Docker image](https://gitlab.com/gitlab-org/security-products/dependency-scanning)
389 390 391 392 393
to run analysis on the project dependencies and checks for potential security issues.
The Auto Dependency Scanning stage will be skipped on licenses other than Ultimate
and requires GitLab Runner 11.5 or above.

Once the
394 395 396
report is created, it's uploaded as an artifact which you can later download and
check out.

397
Any security warnings are also shown in the merge request widget. Read more about
398
[Dependency Scanning](../../user/application_security/dependency_scanning/index.md).
399

400
### Auto License Management **(ULTIMATE)**
401 402 403 404

> Introduced in [GitLab Ultimate][ee] 11.0.

License Management uses the
405
[License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management)
406 407 408 409
to search the project dependencies for their license. The Auto License Management stage
will be skipped on licenses other than Ultimate.

Once the
410 411 412
report is created, it's uploaded as an artifact which you can later download and
check out.

413
Any licenses are also shown in the merge request widget. Read more how
414
[License Management works](../../user/application_security/license_management/index.md).
415

416
### Auto Container Scanning **(ULTIMATE)**
417 418 419 420 421

> Introduced in GitLab 10.4.

Vulnerability Static Analysis for containers uses
[Clair](https://github.com/coreos/clair) to run static analysis on a
422 423 424 425
Docker image and checks for potential security issues. The Auto Container Scanning stage
will be skipped on licenses other than Ultimate.

Once the report is
426 427 428
created, it's uploaded as an artifact which you can later download and
check out.

429
Any security warnings are also shown in the merge request widget. Read more how
430
[Container Scanning works](../../user/application_security/container_scanning/index.md).
431

432 433 434
### Auto Review Apps

This is an optional step, since many projects do not have a Kubernetes cluster
435
available. If the [requirements](#requirements) are not met, the job will
436 437 438 439 440 441 442
silently be skipped.

[Review Apps][review-app] are temporary application environments based on the
branch's code so developers, designers, QA, product managers, and other
reviewers can actually see and interact with code changes as part of the review
process. Auto Review Apps create a Review App for each branch.

443 444 445
Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster
is available, no deployment will occur.

446
The Review App will have a unique URL based on the project ID, the branch or tag
447
name, and a unique number, combined with the Auto DevOps base domain. For
448 449
example, `13083-review-project-branch-123456.example.com`. A link to the Review App shows
up in the merge request widget for easy discovery. When the branch or tag is deleted,
450 451 452
for example after the merge request is merged, the Review App will automatically
be deleted.

453
Review apps are deployed using the
454
[auto-deploy-app](https://gitlab.com/gitlab-org/charts/auto-deploy-app) chart with
455
Helm, which can be [customized](#custom-helm-chart). The app will be deployed into the [Kubernetes
456 457 458 459 460 461 462 463 464 465 466 467 468 469 470
namespace](../../user/project/clusters/index.md#deployment-variables)
for the environment.

Since GitLab 11.4, a [local
Tiller](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22036) is
used. Previous versions of GitLab had a Tiller installed in the project
namespace.

CAUTION: **Caution:**
Your apps should *not* be manipulated outside of Helm (using Kubernetes directly).
This can cause confusion with Helm not detecting the change and subsequent
deploys with Auto DevOps can undo your changes. Also, if you change something
and want to undo it by deploying again, Helm may not detect that anything changed
in the first place, and thus not realize that it needs to re-apply the old config.

471
### Auto DAST **(ULTIMATE)**
472

473
> Introduced in [GitLab Ultimate][ee] 10.4.
474 475 476 477

Dynamic Application Security Testing (DAST) uses the
popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy)
to perform an analysis on the current code and checks for potential security
478 479 480
issues. The Auto DAST stage will be skipped on licenses other than Ultimate.

Once the report is created, it's uploaded as an artifact which you can
481 482
later download and check out.

483
Any security warnings are also shown in the merge request widget. Read how
484
[DAST works](../../user/application_security/dast/index.md).
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
485

486
### Auto Browser Performance Testing **(PREMIUM)**
Joshua Lambert's avatar
Joshua Lambert committed
487

488
> Introduced in [GitLab Premium][ee] 10.4.
Joshua Lambert's avatar
Joshua Lambert committed
489 490 491

Auto Browser Performance Testing utilizes the [Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) to measure the performance of a web page. A JSON report is created and uploaded as an artifact, which includes the overall performance score for each page. By default, the root page of Review and Production environments will be tested. If you would like to add additional URL's to test, simply add the paths to a file named `.gitlab-urls.txt` in the root directory, one per line. For example:

492
```text
Joshua Lambert's avatar
Joshua Lambert committed
493 494 495 496 497
/
/features
/direction
```

498
Any performance differences between the source and target branches are also
499
[shown in the merge request widget](../../user/project/merge_requests/browser_performance_testing.md).
Joshua Lambert's avatar
Joshua Lambert committed
500

501 502 503
### Auto Deploy

This is an optional step, since many projects do not have a Kubernetes cluster
504
available. If the [requirements](#requirements) are not met, the job will
505 506 507 508 509 510
silently be skipped.

After a branch or merge request is merged into the project's default branch (usually
`master`), Auto Deploy deploys the application to a `production` environment in
the Kubernetes cluster, with a namespace based on the project name and unique
project ID, for example `project-4321`.
511 512

Auto Deploy doesn't include deployments to staging or canary by default, but the
513
[Auto DevOps template] contains job definitions for these tasks if you want to
514 515
enable them.

516
You can make use of [environment variables](#environment-variables) to automatically
517
scale your pod replicas and to apply custom arguments to the Auto DevOps `helm upgrade` commands. This is an easy way to [customize the Auto Deploy helm chart](#custom-helm-chart).
518

519
Apps are deployed using the
520
[auto-deploy-app](https://gitlab.com/gitlab-org/charts/auto-deploy-app) chart with
521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536
Helm. The app will be deployed into the [Kubernetes
namespace](../../user/project/clusters/index.md#deployment-variables)
for the environment.

Since GitLab 11.4, a [local
Tiller](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22036) is
used. Previous versions of GitLab had a Tiller installed in the project
namespace.

CAUTION: **Caution:**
Your apps should *not* be manipulated outside of Helm (using Kubernetes directly).
This can cause confusion with Helm not detecting the change and subsequent
deploys with Auto DevOps can undo your changes. Also, if you change something
and want to undo it by deploying again, Helm may not detect that anything changed
in the first place, and thus not realize that it needs to re-apply the old config.

537 538 539 540
> [Introduced][ce-19507] in GitLab 11.0.

For internal and private projects a [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token)
will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved. This Deploy Token
541
can be used for permanent access to the registry. When the GitLab Deploy Token has been manually revoked, it won't be automatically created.
542 543 544 545 546 547 548 549 550

If the GitLab Deploy Token cannot be found, `CI_REGISTRY_PASSWORD` is
used. Note that `CI_REGISTRY_PASSWORD` is only valid during deployment.
This means that Kubernetes will be able to successfully pull the
container image during deployment but in cases where the image needs to
be pulled again, e.g. after pod eviction, Kubernetes will fail to do so
as it will be attempting to fetch the image using
`CI_REGISTRY_PASSWORD`.

551 552
#### Migrations

553 554
> [Introduced][ce-21955] in GitLab 11.4

555 556
Database initialization and migrations for PostgreSQL can be configured to run
within the application pod by setting the project variables `DB_INITIALIZE` and
557 558
`DB_MIGRATE` respectively.

559 560 561 562 563 564 565 566 567
If present, `DB_INITIALIZE` will be run as a shell command within an
application pod as a helm post-install hook. As some applications will
not run without a successful database initialization step, GitLab will
deploy the first release without the application deployment and only the
database initialization step. After the database initialization completes,
GitLab will deploy a second release with the application deployment as
normal.

Note that a post-install hook means that if any deploy succeeds,
568 569 570 571 572
`DB_INITIALIZE` will not be processed thereafter.

If present, `DB_MIGRATE` will be run as a shell command within an application pod as
a helm pre-upgrade hook.

573 574
For example, in a Rails application in an image built with
[Herokuish](https://github.com/gliderlabs/herokuish):
575

576 577
- `DB_INITIALIZE` can be set to `RAILS_ENV=production /bin/herokuish procfile exec bin/rails db:setup`
- `DB_MIGRATE` can be set to `RAILS_ENV=production /bin/herokuish procfile exec bin/rails db:migrate`
578

579
Unless you have a `Dockerfile` in your repo, your image is built with
580 581
Herokuish, and you must prefix commands run in these images with `/bin/herokuish
procfile exec` to replicate the environment where your application will run.
582

583 584
### Auto Monitoring

585
See the [requirements](#requirements) for Auto Monitoring to enable this stage.
586

587 588 589 590 591 592
Once your application is deployed, Auto Monitoring makes it possible to monitor
your application's server and response metrics right out of the box. Auto
Monitoring uses [Prometheus](../../user/project/integrations/prometheus.md) to
get system metrics such as CPU and memory usage directly from
[Kubernetes](../../user/project/integrations/prometheus_library/kubernetes.md),
and response metrics such as HTTP error rates, latency, and throughput from the
593
[NGINX server](../../user/project/integrations/prometheus_library/nginx_ingress.md).
594

595
The metrics include:
596

597 598
- **Response Metrics:** latency, throughput, error rate
- **System Metrics:** CPU utilization, memory utilization
599

600
In order to make use of monitoring you need to:
601

602
1. [Deploy Prometheus](../../user/project/integrations/prometheus.md) into your Kubernetes cluster
603 604
1. If you would like response metrics, ensure you are running at least version
   0.9.0 of NGINX Ingress and
605
   [enable Prometheus metrics](https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/customization/custom-vts-metrics-prometheus/nginx-vts-metrics-conf.yaml).
606 607 608
1. Finally, [annotate](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)
   the NGINX Ingress deployment to be scraped by Prometheus using
   `prometheus.io/scrape: "true"` and `prometheus.io/port: "10254"`.
609

610 611
To view the metrics, open the
[Monitoring dashboard for a deployed environment](../../ci/environments.md#monitoring-environments).
612

613
![Auto Metrics](img/auto_monitoring.png)
614

615
## Customizing
616

617 618 619
While Auto DevOps provides great defaults to get you started, you can customize
almost everything to fit your needs; from custom [buildpacks](#custom-buildpacks),
to [`Dockerfile`s](#custom-dockerfile), [Helm charts](#custom-helm-chart), or
620
even copying the complete [CI/CD configuration](#customizing-gitlab-ciyml)
621
into your project to enable staging and canary deployments, and more.
622

623
### Custom buildpacks
624 625

If the automatic buildpack detection fails for your project, or if you want to
626 627
use a custom buildpack, you can override the buildpack(s) using a project variable
or a `.buildpacks` file in your project:
628 629 630

- **Project variable** - Create a project variable `BUILDPACK_URL` with the URL
  of the buildpack to use.
631
- **`.buildpacks` file** - Add a file in your project's repo called  `.buildpacks`
632
  and add the URL of the buildpack to use on a line in the file. If you want to
633
  use multiple buildpacks, you can enter them in, one on each line.
634

635 636
CAUTION: **Caution:**
Using multiple buildpacks isn't yet supported by Auto DevOps.
637 638 639 640 641 642 643 644 645 646

### Custom `Dockerfile`

If your project has a `Dockerfile` in the root of the project repo, Auto DevOps
will build a Docker image based on the Dockerfile rather than using buildpacks.
This can be much faster and result in smaller images, especially if your
Dockerfile is based on [Alpine](https://hub.docker.com/_/alpine/).

### Custom Helm Chart

647 648 649
Auto DevOps uses [Helm](https://helm.sh/) to deploy your application to Kubernetes.
You can override the Helm chart used by bundling up a chart into your project
repo or by specifying a project variable:
650

651
- **Bundled chart** - If your project has a `./chart` directory with a `Chart.yaml`
652
  file in it, Auto DevOps will detect the chart and use it instead of the [default
653
  one](https://gitlab.com/gitlab-org/charts/auto-deploy-app).
654
  This can be a great way to control exactly how your application is deployed.
655
- **Project variable** - Create a [project variable](../../ci/variables/README.md#gitlab-cicd-environment-variables)
656
  `AUTO_DEVOPS_CHART` with the URL of a custom chart to use or create two project variables `AUTO_DEVOPS_CHART_REPOSITORY` with the URL of a custom chart repository and `AUTO_DEVOPS_CHART` with the path to the chart.
657

658
You can also make use of the `HELM_UPGRADE_EXTRA_ARGS` environment variable to override the default values in the `values.yaml` file in the [default Helm chart](https://gitlab.com/gitlab-org/charts/auto-deploy-app).
659
To apply your own `values.yaml` file to all Helm upgrade commands in Auto Deploy set `HELM_UPGRADE_EXTRA_ARGS` to `--values my-values.yaml`.
660

661
### Custom Helm chart per environment **(PREMIUM)**
662

Evan Read's avatar
Evan Read committed
663
You can specify the use of a custom Helm chart per environment by scoping the environment variable
664
to the desired environment. See [Limiting environment scopes of variables](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium).
665

666
### Customizing `.gitlab-ci.yml`
667

668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693
Everything about Auto DevOps is customizable since the [Auto DevOps template]
is just an example of a [`.gitlab-ci.yml`](../../ci/yaml/README.md) and uses
only features that are available to any `.gitlab-ci.yml`.

Auto DevOps is completely customizable because the [Auto DevOps template]:

- Is just an implementation of a [`.gitlab-ci.yml`](../../ci/yaml/README.md) file.
- Uses only features available to any implementation of `.gitlab-ci.yml`.

If you want to modify the CI/CD pipeline used by Auto DevOps, you can [`include`
the template](../../ci/yaml/README.md#includetemplate) and customize as
needed. To do this, add a `.gitlab-ci.yml` file to the root of your repository
containing the following:

```yml
include:
  - template: Auto-DevOps.gitlab-ci.yml
```

Then add any extra changes you want. Your additions will be merged with the
[Auto DevOps template] using the behaviour described for
[`include`](../../ci/yaml/README.md#include).

It is also possible to copy and paste the contents of the [Auto DevOps
template] into your project and edit this as needed. You may prefer to do it
that way if you want to specifically remove any part of it.
694

695 696 697 698 699 700 701 702 703 704 705 706 707 708 709
### Using components of Auto-DevOps

If you only require a subset of the features offered by Auto-DevOps, you can include
individual Auto-DevOps jobs into your own `.gitlab-ci.yml`.

For example, to make use of [Auto Build](#auto-build), you can add the following to
your `.gitlab-ci.yml`:

```yaml
include:
  - template: Jobs/Build.gitlab-ci.yml
```

Consult the [Auto DevOps template] for information on available jobs.

710 711 712 713 714 715 716 717 718 719 720 721 722 723 724
### PostgreSQL database support

In order to support applications that require a database,
[PostgreSQL][postgresql] is provisioned by default. The credentials to access
the database are preconfigured, but can be customized by setting the associated
[variables](#environment-variables). These credentials can be used for defining a
`DATABASE_URL` of the format:

```yaml
postgres://user:password@postgres-host:postgres-port/postgres-database
```

### Environment variables

The following variables can be used for setting up the Auto DevOps domain,
725
providing a custom Helm chart, or scaling your application. PostgreSQL can
726 727
also be customized, and you can easily use a [custom buildpack](#custom-buildpacks).

728 729
| **Variable**                 | **Description**                                                                                                                                                                                                               |
| ------------                 | ---------------                                                                                                                                                                                                               |
730
| `AUTO_DEVOPS_CHART`          | The Helm Chart used to deploy your apps; defaults to the one [provided by GitLab](https://gitlab.com/gitlab-org/charts/auto-deploy-app).                                                             |
731
| `AUTO_DEVOPS_CHART_REPOSITORY` | The Helm Chart repository used to search for charts; defaults to `https://charts.gitlab.io`. |
732 733 734
| `AUTO_DEVOPS_CHART_REPOSITORY_NAME` | From Gitlab 11.11, this variable can be used to set the name of the helm repository; defaults to "gitlab" |
| `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From Gitlab 11.11, this variable can be used to set a username to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD) |
| `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From Gitlab 11.11, this variable can be used to set a password to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_USERNAME) |
735
| `REPLICAS`                   | The number of replicas to deploy; defaults to 1.                                                                                                                                                                              |
736
| `PRODUCTION_REPLICAS`        | The number of replicas to deploy in the production environment. Takes precedence over `REPLICAS` and defaults to 1. For zero downtime upgrades, set to 2 or greater.                                |
737 738
| `CANARY_REPLICAS`            | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md); defaults to 1                                                                              |
| `CANARY_PRODUCTION_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. This takes precedence over `CANARY_REPLICAS`; defaults to 1  |
739 740
| `ADDITIONAL_HOSTS`           | Fully qualified domain names specified as a comma-separated list that are added to the ingress hosts.                                                                                                                         |
| `<ENVIRONMENT>_ADDITIONAL_HOSTS` | For a specific environment, the fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. This takes precedence over `ADDITIONAL_HOSTS`.                                      |
741 742 743
| `POSTGRES_ENABLED`           | Whether PostgreSQL is enabled; defaults to `"true"`. Set to `false` to disable the automatic deployment of PostgreSQL.                                                                                                        |
| `POSTGRES_USER`              | The PostgreSQL user; defaults to `user`. Set it to use a custom username.                                                                                                                                                     |
| `POSTGRES_PASSWORD`          | The PostgreSQL password; defaults to `testing-password`. Set it to use a custom password.                                                                                                                                     |
744
| `POSTGRES_DB`                | The PostgreSQL database name; defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/README.md#predefined-environment-variables). Set it to use a custom database name.                               |
745
| `POSTGRES_VERSION`           | Tag for the [`postgres` Docker image](https://hub.docker.com/_/postgres) to use. Defaults to `9.6.2`. |
746
| `BUILDPACK_URL`              | The buildpack's full URL. It can point to either Git repositories or a tarball URL. For Git repositories, it is possible to point to a specific `ref`, for example `https://github.com/heroku/heroku-buildpack-ruby.git#v142` |
747 748
| `SAST_CONFIDENCE_LEVEL`      | The minimum confidence level of security issues you want to be reported; `1` for Low, `2` for Medium, `3` for High; defaults to `3`.|
| `DEP_SCAN_DISABLE_REMOTE_CHECKS` | Whether remote Dependency Scanning checks are disabled; defaults to `"false"`. Set to `"true"` to disable checks that send data to GitLab central servers. [Read more about remote checks](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).|
749 750
| `DB_INITIALIZE`              | From GitLab 11.4, this variable can be used to specify the command to run to initialize the application's PostgreSQL database. It runs inside the application pod. |
| `DB_MIGRATE`                 | From GitLab 11.4, this variable can be used to specify the command to run to migrate the application's PostgreSQL database. It runs inside the application pod. |
751
| `STAGING_ENABLED`            | From GitLab 10.8, this variable can be used to define a [deploy policy for staging and production environments](#deploy-policy-for-staging-and-production-environments). |
752
| `CANARY_ENABLED`             | From GitLab 11.0, this variable can be used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments-premium). |
753
| `INCREMENTAL_ROLLOUT_MODE`| From GitLab 11.4, this variable, if present, can be used to enable an [incremental rollout](#incremental-rollout-to-production-premium) of your application for the production environment. Set to `manual` for manual deployment jobs or `timed` for automatic rollout deployments with a 5 minute delay each one. |
754
| `TEST_DISABLED`              | From GitLab 11.0, this variable can be used to disable the `test` job. If the variable is present, the job will not be created. |
755
| `CODE_QUALITY_DISABLED`       | From GitLab 11.0, this variable can be used to disable the `codequality` job. If the variable is present, the job will not be created. |
756
| `LICENSE_MANAGEMENT_DISABLED` | From GitLab 11.0, this variable can be used to disable the `license_management` job. If the variable is present, the job will not be created. |
757 758
| `SAST_DISABLED`              | From GitLab 11.0, this variable can be used to disable the `sast` job. If the variable is present, the job will not be created. |
| `DEPENDENCY_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dependency_scanning` job. If the variable is present, the job will not be created. |
759
| `CONTAINER_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `sast:container` job. If the variable is present, the job will not be created. |
760 761 762
| `REVIEW_DISABLED`            | From GitLab 11.0, this variable can be used to disable the `review` and the manual `review:stop` job. If the variable is present, these jobs will not be created. |
| `DAST_DISABLED`              | From GitLab 11.0, this variable can be used to disable the `dast` job. If the variable is present, the job will not be created. |
| `PERFORMANCE_DISABLED`       | From GitLab 11.0, this variable can be used to disable the `performance` job. If the variable is present, the job will not be created. |
763
| `K8S_SECRET_*`               | From GitLab 11.7, any variable prefixed with [`K8S_SECRET_`](#application-secret-variables) will be made available by Auto DevOps as environment variables to the deployed application. |
764
| `KUBE_INGRESS_BASE_DOMAIN`   | From GitLab 11.8, this variable can be used to set a domain per cluster. See [cluster domains](../../user/project/clusters/index.md#base-domain) for more information. |
765
| `ROLLOUT_RESOURCE_TYPE`     | From GitLab 11.9, this variable allows specification of the resource type being deployed when using a custom helm chart. Default value is `deployment`. |
766
| `ROLLOUT_STATUS_DISABLED`   | From GitLab 12.0, this variable allows to disable rollout status check because it doesn't support all resource types, for example, `cronjob`. |
767
| `HELM_UPGRADE_EXTRA_ARGS`   | From GitLab 11.11, this variable allows extra arguments in `helm` commands when deploying the application. Note that using quotes will not prevent word splitting. **Tip:** you can use this variable to [customize the Auto Deploy helm chart](https://docs.gitlab.com/ee/topics/autodevops/index.html#custom-helm-chart) by applying custom override values with `--values my-values.yaml`. |
768
| `HELM_RELEASE_NAME`   | From GitLab 12.1, this variable allows the `helm` release name to be overridden, this can be used to assign unique release names when deploying multiple projects to a single namespace  |
769 770 771

TIP: **Tip:**
Set up the replica variables using a
772
[project variable](../../ci/variables/README.md#gitlab-cicd-environment-variables)
773 774 775 776 777 778 779
and scale your application by just redeploying it!

CAUTION: **Caution:**
You should *not* scale your application using Kubernetes directly. This can
cause confusion with Helm not detecting the change, and subsequent deploys with
Auto DevOps can undo your changes.

780
#### Application secret variables
781 782 783 784

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/49056) in GitLab 11.7.

Some applications need to define secret variables that are
785
accessible by the deployed application. Auto DevOps detects variables where the key starts with
786 787 788 789 790 791
`K8S_SECRET_` and make these prefixed variables available to the
deployed application, as environment variables.

To configure your application variables:

1. Go to your project's **Settings > CI/CD**, then expand the section
792
   called **Variables**.
793

794
1. Create a CI Variable, ensuring the key is prefixed with
795 796 797
   `K8S_SECRET_`. For example, you can create a variable with key
`K8S_SECRET_RAILS_MASTER_KEY`.

798
1. Run an Auto Devops pipeline either by manually creating a new
799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827
   pipeline or by pushing a code change to GitLab.

Auto DevOps pipelines will take your application secret variables to
populate a Kubernetes secret. This secret is unique per environment.
When deploying your application, the secret is loaded as environment
variables in the container running the application. Following the
example above, you can see the secret below containing the
`RAILS_MASTER_KEY` variable.

```sh
$ kubectl get secret production-secret -n minimal-ruby-app-54 -o yaml
apiVersion: v1
data:
  RAILS_MASTER_KEY: MTIzNC10ZXN0
kind: Secret
metadata:
  creationTimestamp: 2018-12-20T01:48:26Z
  name: production-secret
  namespace: minimal-ruby-app-54
  resourceVersion: "429422"
  selfLink: /api/v1/namespaces/minimal-ruby-app-54/secrets/production-secret
  uid: 57ac2bfd-03f9-11e9-b812-42010a9400e4
type: Opaque
```

Environment variables are generally considered immutable in a Kubernetes
pod. Therefore, if you update an application secret without changing any
code then manually create a new pipeline, you will find that any running
application pods will not have the updated secrets. In this case, you
828 829
can either push a code update to GitLab to force the Kubernetes
Deployment to recreate pods or manually delete running pods to
830 831
cause Kubernetes to create new pods with updated secrets.

832 833 834 835
NOTE: **Note:**
Variables with multiline values are not currently supported due to
limitations with the current Auto DevOps scripting environment.

836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854
#### Advanced replica variables setup

Apart from the two replica-related variables for production mentioned above,
you can also use others for different environments.

There's a very specific mapping between Kubernetes' label named `track`,
GitLab CI/CD environment names, and the replicas environment variable.
The general rule is: `TRACK_ENV_REPLICAS`. Where:

- `TRACK`: The capitalized value of the `track`
  [Kubernetes label](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
  in the Helm Chart app definition. If not set, it will not be taken into account
  to the variable name.
- `ENV`: The capitalized environment name of the deploy job that is set in
  `.gitlab-ci.yml`.

That way, you can define your own `TRACK_ENV_REPLICAS` variables with which
you will be able to scale the pod's replicas easily.

855 856 857
In the example below, the environment's name is `qa` and it deploys the track
`foo` which would result in looking for the `FOO_QA_REPLICAS` environment
variable:
858 859 860 861 862 863 864

```yaml
QA testing:
  stage: deploy
  environment:
    name: qa
  script:
865
  - deploy foo
866 867
```

868 869
The track `foo` being referenced would also need to be defined in the
application's Helm chart, like:
870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890

```yaml
replicaCount: 1
image:
  repository: gitlab.example.com/group/project
  tag: stable
  pullPolicy: Always
  secrets:
    - name: gitlab-registry
application:
  track: foo
  tier: web
service:
  enabled: true
  name: web
  type: ClusterIP
  url: http://my.host.com/
  externalPort: 5000
  internalPort: 5000
```

891 892 893 894 895
#### Deploy policy for staging and production environments

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/160)
in GitLab 10.8.

896 897 898
TIP: **Tip:**
You can also set this inside your [project's settings](#deployment-strategy).

899 900 901 902 903 904 905
The normal behavior of Auto DevOps is to use Continuous Deployment, pushing
automatically to the `production` environment every time a new pipeline is run
on the default branch. However, there are cases where you might want to use a
staging environment and deploy to production manually. For this scenario, the
`STAGING_ENABLED` environment variable was introduced.

If `STAGING_ENABLED` is defined in your project (e.g., set `STAGING_ENABLED` to
906
`1` as a CI/CD variable), then the application will be automatically deployed
907 908 909
to a `staging` environment, and a  `production_manual` job will be created for
you when you're ready to manually deploy to production.

910
#### Deploy policy for canary environments **(PREMIUM)**
911 912 913 914

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/171)
in GitLab 11.0.

915
A [canary environment](../../user/project/canary_deployments.md) can be used
916 917 918
before any changes are deployed to production.

If `CANARY_ENABLED` is defined in your project (e.g., set `CANARY_ENABLED` to
919
`1` as a CI/CD variable) then two manual jobs will be created:
920 921 922 923 924

- `canary` which will deploy the application to the canary environment
- `production_manual` which is to be used by you when you're ready to manually
  deploy to production.

925
#### Incremental rollout to production **(PREMIUM)**
926 927 928

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5415) in GitLab 10.8.

929 930 931
TIP: **Tip:**
You can also set this inside your [project's settings](#deployment-strategy).

932 933 934 935 936
When you have a new version of your app to deploy in production, you may want
to use an incremental rollout to replace just a few pods with the latest code.
This will allow you to first check how the app is behaving, and later manually
increasing the rollout up to 100%.

937 938
If `INCREMENTAL_ROLLOUT_MODE` is set to `manual` in your project, then instead
of the standard `production` job, 4 different
939
[manual jobs](../../ci/pipelines.md#manual-actions-from-pipeline-graphs)
940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955
will be created:

1. `rollout 10%`
1. `rollout 25%`
1. `rollout 50%`
1. `rollout 100%`

The percentage is based on the `REPLICAS` variable and defines the number of
pods you want to have for your deployment. If you say `10`, and then you run
the `10%` rollout job, there will be `1` new pod + `9` old ones.

To start a job, click on the play icon next to the job's name. You are not
required to go from `10%` to `100%`, you can jump to whatever job you want.
You can also scale down by running a lower percentage job, just before hitting
`100%`. Once you get to `100%`, you cannot scale down, and you'd have to roll
back by redeploying the old version using the
956
[rollback button](../../ci/environments.md#retrying-and-rolling-back) in the
957 958 959 960 961
environment page.

Below, you can see how the pipeline will look if the rollout or staging
variables are defined.

962 963 964 965 966
Without `INCREMENTAL_ROLLOUT_MODE` and without `STAGING_ENABLED`:

![Staging and rollout disabled](img/rollout_staging_disabled.png)

Without `INCREMENTAL_ROLLOUT_MODE` and with `STAGING_ENABLED`:
967

968
![Staging enabled](img/staging_enabled.png)
969

970
With `INCREMENTAL_ROLLOUT_MODE` set to `manual` and without `STAGING_ENABLED`:
971

972
![Rollout enabled](img/rollout_enabled.png)
973

974 975 976 977 978 979 980 981 982
With `INCREMENTAL_ROLLOUT_MODE` set to `manual` and with `STAGING_ENABLED`

![Rollout and staging enabled](img/rollout_staging_enabled.png)

CAUTION: **Caution:**
Before GitLab 11.4 this feature was enabled by the presence of the
`INCREMENTAL_ROLLOUT_ENABLED` environment variable.
This configuration is deprecated and will be removed in the future.

983
#### Timed incremental rollout to production **(PREMIUM)**
984 985 986 987 988

> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7545) in GitLab 11.4.

TIP: **Tip:**
You can also set this inside your [project's settings](#deployment-strategy).
989

990
This configuration based on
991
[incremental rollout to production](#incremental-rollout-to-production-premium).
992

993
Everything behaves the same way, except:
994

995 996 997 998 999 1000
- It's enabled by setting the `INCREMENTAL_ROLLOUT_MODE` variable to `timed`.
- Instead of the standard `production` job, the following jobs with a 5 minute delay between each are created:
    1. `timed rollout 10%`
    1. `timed rollout 25%`
    1. `timed rollout 50%`
    1. `timed rollout 100%`
1001

1002 1003
## Currently supported languages

1004
Note that not all buildpacks support Auto Test yet, as it's a relatively new
1005 1006 1007 1008 1009 1010 1011 1012
enhancement. All of Heroku's [officially supported
languages](https://devcenter.heroku.com/articles/heroku-ci#currently-supported-languages)
support it, and some third-party buildpacks as well e.g., Go, Node, Java, PHP,
Python, Ruby, Gradle, Scala, and Elixir all support Auto Test, but notably the
multi-buildpack does not.

As of GitLab 10.0, the supported buildpacks are:

1013
```text
1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026
- heroku-buildpack-multi     v1.0.0
- heroku-buildpack-ruby      v168
- heroku-buildpack-nodejs    v99
- heroku-buildpack-clojure   v77
- heroku-buildpack-python    v99
- heroku-buildpack-java      v53
- heroku-buildpack-gradle    v23
- heroku-buildpack-scala     v78
- heroku-buildpack-play      v26
- heroku-buildpack-php       v122
- heroku-buildpack-go        v72
- heroku-buildpack-erlang    fa17af9
- buildpack-nginx            v8
1027 1028
```

1029 1030 1031 1032 1033 1034 1035
## Limitations

The following restrictions apply.

### Private project support

CAUTION: **Caution:** Private project support in Auto DevOps is experimental.
1036 1037 1038 1039 1040 1041 1042 1043 1044

When a project has been marked as private, GitLab's [Container
Registry][container-registry] requires authentication when downloading
containers. Auto DevOps will automatically provide the required authentication
information to Kubernetes, allowing temporary access to the registry.
Authentication credentials will be valid while the pipeline is running, allowing
for a successful initial deployment.

After the pipeline completes, Kubernetes will no longer be able to access the
1045 1046
Container Registry. **Restarting a pod, scaling a service, or other actions which
require on-going access to the registry may fail**. On-going secure access is
1047 1048 1049 1050 1051 1052 1053 1054 1055
planned for a subsequent release.

## Troubleshooting

- Auto Build and Auto Test may fail in detecting your language/framework. There
  may be no buildpack for your application, or your application may be missing the
  key files the buildpack is looking for. For example, for ruby apps, you must
  have a `Gemfile` to be properly detected, even though it is possible to write a
  Ruby app without a `Gemfile`. Try specifying a [custom
1056
  buildpack](#custom-buildpacks).
1057 1058
- Auto Test may fail because of a mismatch between testing frameworks. In this
  case, you may need to customize your `.gitlab-ci.yml` with your test commands.
Evan Read's avatar
Evan Read committed
1059 1060 1061
- Auto Deploy will fail if GitLab can not create a Kubernetes namespace and
  service account for your project. For help debugging this issue, see
  [Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting-failed-deployment-jobs).
1062

1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074
### Disable the banner instance wide

If an administrator would like to disable the banners on an instance level, this
feature can be disabled either through the console:

```sh
sudo gitlab-rails console
```

Then run:

```ruby
1075
Feature.get(:auto_devops_banner_disabled).enable
1076 1077 1078 1079 1080
```

Or through the HTTP API with an admin access token:

```sh
1081
curl --data "value=true" --header "PRIVATE-TOKEN: personal_access_token" https://gitlab.example.com/api/v4/features/auto_devops_banner_disabled
1082 1083
```

1084
[ce-37115]: https://gitlab.com/gitlab-org/gitlab-ce/issues/37115
1085
[kubernetes-clusters]: ../../user/project/clusters/index.md
1086 1087 1088 1089
[docker-in-docker]: ../../docker/using_docker_build.md#use-docker-in-docker-executor
[review-app]: ../../ci/review_apps/index.md
[container-registry]: ../../user/project/container_registry.md
[postgresql]: https://www.postgresql.org/
1090
[Auto DevOps template]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml
1091
[ee]: https://about.gitlab.com/pricing/
1092
[ce-21955]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21955
1093
[ce-19507]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19507
1094 1095 1096 1097

## Development guides

Configuring [GDK for Auto DevOps](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/auto_devops.md).