Auto DevOps provides great defaults for all the stages; you can, however,
[customize](#customizing) almost everything to your needs.
For an overview on the creation of Auto DevOps, read the blog post [From 2/3 of the Self-Hosted Git Market, to the Next-Generation CI System, to Auto DevOps](https://about.gitlab.com/2017/06/29/whats-next-for-gitlab-ci/).
## Requirements
TIP: **Tip:**
For self-hosted installations, the easiest way to make use of Auto DevOps is to
install GitLab inside a Kubernetes cluster using the [GitLab Omnibus Helm Chart]
which automatically installs and configures everything you need!
To make full use of Auto DevOps, you will need:
1.**GitLab Runner** (needed for all stages) - Your Runner needs to be
configured to be able to run Docker. Generally this means using the
[Docker](https://docs.gitlab.com/runner/executors/docker.html) or [Kubernetes
executor](https://docs.gitlab.com/runner/executors/kubernetes.html), with
Helm chart to your Kubernetes cluster to automatically issue certificates
for your domains using Let's Encrypt.
1.**Prometheus** (needed for Auto Monitoring) - To enable Auto Monitoring, you
will need Prometheus installed somewhere (inside or outside your cluster) and
configured to scrape your Kubernetes cluster. To get response metrics
(in addition to system metrics), you need to
[configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-prometheus-to-monitor-for-nginx-ingress-metrics).
The [Prometheus service](../../user/project/integrations/prometheus.md)
integration needs to be enabled for the project, or enabled as a
[default service template](../../user/project/integrations/services_templates.md)
for the entire GitLab installation.
NOTE: **Note:**
If you do not have Kubernetes or Prometheus installed, then Auto Review Apps,
Auto Deploy, and Auto Monitoring will be silently skipped.
## Auto DevOps base domain
The Auto DevOps base domain is required if you want to make use of [Auto
Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It can be defined
in three places:
- either under the project's CI/CD settings while [enabling Auto DevOps](#enabling-auto-devops)
There are two options when adding a new cluster to your project; either associate
- or in instance-wide settings in the **admin area > Settings** under the "Continuous Integration and Delivery" section
your account with Google Kubernetes Engine (GKE) so that you can [create new
- or at the project or group level as a variable: `AUTO_DEVOPS_DOMAIN` (required if you want to use [multiple clusters](#using-multiple-kubernetes-clusters))
clusters](#adding-and-creating-a-new-gke-cluster-via-gitlab) from within GitLab,
or provide the credentials to an [existing Kubernetes cluster](#adding-an-existing-kubernetes-cluster).
A wildcard DNS A record matching the base domain(s) is required, for example,
## Adding and creating a new GKE cluster via GitLab
given a base domain of `example.com`, you'd need a DNS entry like:
```
*.example.com 3600 A 1.2.3.4
```
In this case, `example.com` is the domain name under which the deployed apps will be served,
and `1.2.3.4` is the IP address of your load balancer; generally NGINX
([see requirements](#requirements)). How to set up the DNS record is beyond
the scope of this document; you should check with your DNS provider.
Alternatively you can use free public services like [nip.io](http://nip.io) or
[nip.io](http://nip.io) which provide automatic wildcard DNS without any
configuration. Just set the Auto DevOps base domain to `1.2.3.4.nip.io` or
`1.2.3.4.nip.io`.
Once set up, all requests will hit the load balancer, which in turn will route
them to the Kubernetes pods that run your application(s).
NOTE: **Note:**
NOTE: **Note:**
If GitLab is installed using the [GitLab Omnibus Helm Chart], there are two
You need Maintainer [permissions] and above to access the Kubernetes page.
options: provide a static IP, or have one assigned. For more information see the
relevant docs on the [network prerequisites](../../install/kubernetes/gitlab_omnibus.md#networking-prerequisites).
Before proceeding, make sure the following requirements are met:
## Using multiple Kubernetes clusters **[PREMIUM]**
- The [Google authentication integration](../../../integration/google.md) must
be enabled in GitLab at the instance level. If that's not the case, ask your
When using Auto DevOps, you may want to deploy different environments to
GitLab administrator to enable it.
different Kubernetes clusters. This is possible due to the 1:1 connection that
- Your associated Google account must have the right privileges to manage
[exists between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters).
-**Environment scope** - The [associated environment](#setting-the-environment-scope) to this cluster.
| review | `review/*` | `review.example.com` | `review/*` | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. |
-**Google Cloud Platform project** - The project you created in your GCP
| staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). |
console that will host the Kubernetes cluster. This must **not** be confused
| production | `production` | `example.com` | `production` | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](#incremental-rollout-to-production). |
with the project ID. Learn more about [Google Cloud Platform projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects).
-**Zone** - The [zone](https://cloud.google.com/compute/docs/regions-zones/)
To add a different cluster for each environment:
under which the cluster will be created.
-**Number of nodes** - The number of nodes you wish the cluster to have.
1. Navigate to your project's **Operations > Kubernetes** and create the Kubernetes clusters
-**Machine type** - The [machine type](https://cloud.google.com/compute/docs/machine-types)
with their respective environment scope as described from the table above.
of the Virtual Machine instance that the cluster will be based on.
1. Finally, click the **Create Kubernetes cluster** button.
GitLab Runners have the [privileged mode](https://docs.gitlab.com/runner/executors/docker.html#the-privileged-mode)
enabled by default, which allows them to execute special commands and running
Docker in Docker. This functionality is needed to run some of the [Auto DevOps]
jobs. This implies the containers are running in privileged mode and you should,
therefore, be aware of some important details.
The privileged flag gives all capabilities to the running container, which in
turn can do almost everything that the host can do. Be aware of the
inherent security risk associated with performing `docker run` operations on
arbitrary images as they effectively have root access.
If you don't want to use GitLab Runner in privileged mode, first make sure that
you don't have it installed via the applications, and then use the
[Runner's Helm chart](../../../install/kubernetes/gitlab_runner_chart.md) to
install it manually.
## Installing applications
GitLab provides a one-click install for various applications which will be
added directly to your configured cluster. Those applications are needed for
[Review Apps](../../../ci/review_apps/index.md) and [deployments](../../../ci/environments.md).
| Application | GitLab version | Description |
| ----------- | :------------: | ----------- |
| [Helm Tiller](https://docs.helm.sh/) | 10.2+ | Helm is a package manager for Kubernetes and is required to install all the other applications. It will be automatically installed as a dependency when you try to install a different app. It is installed in its own pod inside the cluster which can run the `helm` CLI in a safe environment. |
| [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) | 10.2+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps] or deploy your own web apps. |
| [Prometheus](https://prometheus.io/docs/introduction/overview/) | 10.4+ | Prometheus is an open-source monitoring and alerting system useful to supervise your deployed applications |
| [GitLab Runner](https://docs.gitlab.com/runner/) | 10.6+ | GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/), the open-source continuous integration service included with GitLab that coordinates the jobs. When installing the GitLab Runner via the applications, it will run in **privileged mode** by default. Make sure you read the [security implications](#security-implications) before doing so. |
| [JupyterHub](http://jupyter.org/) | 11.0+ | [JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a multi-user service for managing notebooks across a team. [Jupyter Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) provide a web-based interactive programming environment used for data analysis, visualization, and machine learning. **Note**: Authentication will be enabled for any user of the GitLab server via OAuth2. HTTPS will be supported in a future release. |
## Getting the external IP address
NOTE: **Note:**
NOTE: **Note:**
Auto Test uses tests you already have in your application. If there are no
You need a load balancer installed in your cluster in order to obtain the
tests, it's up to you to add them.
external IP address with the following procedure. It can be deployed using the
and select the proper project and cluster. Then click on **Connect** and execute
the `gcloud` command in a local terminal or using the **Cloud Shell**.
> Introduced in GitLab 10.4.
If the cluster is not on GKE, follow the specific instructions for your
Kubernetes provider to configure `kubectl` with the right credentials.
Vulnerability Static Analysis for containers uses
If you installed the Ingress [via the **Applications**](#installing-applications),
[Clair](https://github.com/coreos/clair) to run static analysis on a
run the following command:
Docker image and checks for potential security issues. Once the report is
created, it's uploaded as an artifact which you can later download and
check out.
In GitLab Ultimate, any security warnings are also
```bash
[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/container_scanning.html).
kubectl get svc --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -ojsonpath='{.status.loadBalancer.ingress[0].ip} '
```
### Auto Review Apps
NOTE: **Note:**
This is an optional step, since many projects do not have a Kubernetes cluster
available. If the [requirements](#requirements) are not met, the job will
silently be skipped.
CAUTION: **Caution:**
Your apps should *not* be manipulated outside of Helm (using Kubernetes directly.)
This can cause confusion with Helm not detecting the change, and subsequent
deploys with Auto DevOps can undo your changes. Also, if you change something
and want to undo it by deploying again, Helm may not detect that anything changed
in the first place, and thus not realize that it needs to re-apply the old config.
[Review Apps][review-app] are temporary application environments based on the
branch's code so developers, designers, QA, product managers, and other
reviewers can actually see and interact with code changes as part of the review
process. Auto Review Apps create a Review App for each branch.
The Review App will have a unique URL based on the project name, the branch
name, and a unique number, combined with the Auto DevOps base domain. For
example, `user-project-branch-1234.example.com`. A link to the Review App shows
up in the merge request widget for easy discovery. When the branch is deleted,
for example after the merge request is merged, the Review App will automatically
be deleted.
### Auto DAST **[ULTIMATE]**
> Introduced in [GitLab Ultimate][ee] 10.4.
Otherwise, you can list the IP addresses of all load balancers:
Dynamic Application Security Testing (DAST) uses the
```bash
popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy)
kubectl get svc --all-namespaces-ojsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} '
to perform an analysis on the current code and checks for potential security
```
issues. Once the report is created, it's uploaded as an artifact which you can
later download and check out.
In GitLab Ultimate, any security warnings are also
> **Note**: Some Kubernetes clusters return a hostname instead, like [Amazon EKS](https://aws.amazon.com/eks/). For these platforms, run:
[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dast.html).
> ```bash
> kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps -o jsonpath="{.status.loadBalancer.ingress[0].hostname}"`.
> ```
### Auto Browser Performance Testing **[PREMIUM]**
The output is the external IP address of your cluster. This information can then
be used to set up DNS entries and forwarding rules that allow external access to
your deployed applications.
> Introduced in [GitLab Premium][ee] 10.4.
### Using a static IP
Auto Browser Performance Testing utilizes the [Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) to measure the performance of a web page. A JSON report is created and uploaded as an artifact, which includes the overall performance score for each page. By default, the root page of Review and Production environments will be tested. If you would like to add additional URL's to test, simply add the paths to a file named `.gitlab-urls.txt` in the root directory, one per line. For example:
By default, an ephemeral external IP address is associated to the cluster's load
balancer. If you associate the ephemeral IP with your DNS and the IP changes,
your apps will not be able to be reached, and you'd have to change the DNS
record again. In order to avoid that, you should change it into a static
reserved IP.
```
[Read how to promote an ephemeral external IP address in GKE.](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip)
/
/features
/direction
```
In GitLab Premium, performance differences between the source
### Pointing your DNS at the cluster IP
and target branches are [shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html).
### Auto Deploy
Once you've set up the static IP, you should associate it to a [wildcard DNS
record](https://en.wikipedia.org/wiki/Wildcard_DNS_record), in order to be able
to reach your apps. This heavily depends on your domain provider, but in case
you aren't sure, just create an A record with a wildcard host like
`*.example.com.`.
NOTE: **Note:**
## Setting the environment scope
This is an optional step, since many projects do not have a Kubernetes cluster
available. If the [requirements](#requirements) are not met, the job will
silently be skipped.
CAUTION: **Caution:**
Your apps should *not* be manipulated outside of Helm (using Kubernetes directly.)
This can cause confusion with Helm not detecting the change, and subsequent
deploys with Auto DevOps can undo your changes. Also, if you change something
and want to undo it by deploying again, Helm may not detect that anything changed
in the first place, and thus not realize that it needs to re-apply the old config.
After a branch or merge request is merged into the project's default branch (usually
`master`), Auto Deploy deploys the application to a `production` environment in
the Kubernetes cluster, with a namespace based on the project name and unique
project ID, for example `project-4321`.
Auto Deploy doesn't include deployments to staging or canary by default, but the
[Auto DevOps template] contains job definitions for these tasks if you want to
enable them.
You can make use of [environment variables](#helm-chart-variables) to automatically
scale your pod replicas.
It's important to note that when a project is deployed to a Kubernetes cluster,
it relies on a Docker image that has been pushed to the
-**System Metrics:** CPU utilization, memory utilization
If GitLab has been deployed using the [GitLab Omnibus Helm Chart], no
configuration is required.
If you have installed GitLab using a different method, you need to:
1.[Deploy Prometheus](../../user/project/integrations/prometheus.md#configuring-your-own-prometheus-server-within-kubernetes) into your Kubernetes cluster
1. If you would like response metrics, ensure you are running at least version
The following variables can be used for setting up the Auto DevOps domain,
script:sh test
providing a custom Helm chart, or scaling your application. PostgreSQL can be
also be customized, and you can easily use a [custom buildpack](#custom-buildpacks).
| **Variable** | **Description** |
| ------------ | --------------- |
| `AUTO_DEVOPS_DOMAIN` | The [Auto DevOps domain](#auto-devops-domain); by default set automatically by the [Auto DevOps setting](#enabling-auto-devops). |
| `AUTO_DEVOPS_CHART` | The Helm Chart used to deploy your apps; defaults to the one [provided by GitLab](https://gitlab.com/charts/charts.gitlab.io/tree/master/charts/auto-deploy-app). |
| `REPLICAS` | The number of replicas to deploy; defaults to 1. |
| `PRODUCTION_REPLICAS` | The number of replicas to deploy in the production environment. This takes precedence over `REPLICAS`; defaults to 1. |
| `CANARY_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html); defaults to 1 |
| `CANARY_PRODUCTION_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) in the production environment. This takes precedence over `CANARY_REPLICAS`; defaults to 1 |
| `POSTGRES_ENABLED` | Whether PostgreSQL is enabled; defaults to `"true"`. Set to `false` to disable the automatic deployment of PostgreSQL. |
| `POSTGRES_USER` | The PostgreSQL user; defaults to `user`. Set it to use a custom username. |
| `POSTGRES_PASSWORD` | The PostgreSQL password; defaults to `testing-password`. Set it to use a custom password. |
| `POSTGRES_DB` | The PostgreSQL database name; defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/README.md#predefined-variables-environment-variables). Set it to use a custom database name. |
| `BUILDPACK_URL` | The buildpack's full URL. It can point to either Git repositories or a tarball URL. For Git repositories, it is possible to point to a specific `ref`, for example `https://github.com/heroku/heroku-buildpack-ruby.git#v142` |
| `SAST_CONFIDENCE_LEVEL` | The minimum confidence level of security issues you want to be reported; `1` for Low, `2` for Medium, `3` for High; defaults to `3`.|
| `DEP_SCAN_DISABLE_REMOTE_CHECKS` | Whether remote Dependency Scanning checks are disabled; defaults to `"false"`. Set to `"true"` to disable checks that send data to GitLab central servers. [Read more about remote checks](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).|
| `STAGING_ENABLED` | From GitLab 10.8, this variable can be used to define a [deploy policy for staging and production environments](#deploy-policy-for-staging-and-production-environments). |
| `CANARY_ENABLED` | From GitLab 11.0, this variable can be used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments). |
| `INCREMENTAL_ROLLOUT_ENABLED`| From GitLab 10.8, this variable can be used to enable an [incremental rollout](#incremental-rollout-to-production) of your application for the production environment. |
| `TEST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `test` job. If the variable is present, the job will not be created. |
| `CODEQUALITY_DISABLED` | From GitLab 11.0, this variable can be used to disable the `codequality` job. If the variable is present, the job will not be created. |
| `SAST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `sast` job. If the variable is present, the job will not be created. |
| `DEPENDENCY_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dependency_scanning` job. If the variable is present, the job will not be created. |
| `CONTAINER_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `sast:container` job. If the variable is present, the job will not be created. |
| `REVIEW_DISABLED` | From GitLab 11.0, this variable can be used to disable the `review` and the manual `review:stop` job. If the variable is present, these jobs will not be created. |
| `DAST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dast` job. If the variable is present, the job will not be created. |
| `PERFORMANCE_DISABLED` | From GitLab 11.0, this variable can be used to disable the `performance` job. If the variable is present, the job will not be created. |
A [canary environment](https://docs.gitlab.com/ee/user/project/canary_deployments.html) can be used
With GitLab Premium, you can associate more than one Kubernetes clusters to your
before any changes are deployed to production.
project. That way you can have different clusters for different environments,
like dev, staging, production, etc.
If `CANARY_ENABLED` is defined in your project (e.g., set `CANARY_ENABLED` to
Simply add another cluster, like you did the first time, and make sure to
`1` as a secret variable) then two manual jobs will be created:
[set an environment scope](#setting-the-environment-scope) that will
differentiate the new cluster with the rest.
-`canary` which will deploy the application to the canary environment
## Deployment variables
-`production_manual` which is to be used by you when you're ready to manually
deploy to production.
#### Incremental rollout to production **[PREMIUM]**
The Kubernetes cluster integration exposes the following
[deployment variables](../../../ci/variables/README.md#deployment-variables) in the
GitLab CI/CD build environment.
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5415) in GitLab 10.8.
| Variable | Description |
| -------- | ----------- |
| `KUBE_URL` | Equal to the API URL. |
| `KUBE_TOKEN` | The Kubernetes token. |
| `KUBE_NAMESPACE` | The Kubernetes namespace is auto-generated if not specified. The default value is `<project_name>-<project_id>`. You can overwrite it to use different one if needed, otherwise the `KUBE_NAMESPACE` variable will receive the default value. |
| `KUBE_CA_PEM_FILE` | Only present if a custom CA bundle was specified. Path to a file containing PEM data. |
| `KUBE_CA_PEM` | (**deprecated**) Only if a custom CA bundle was specified. Raw PEM data. |
| `KUBECONFIG` | Path to a file containing `kubeconfig` for this deployment. CA bundle would be embedded if specified. |
TIP: **Tip:**
## Enabling or disabling the Kubernetes cluster integration
You can also set this inside your [project's settings](#deployment-strategy).
When you have a new version of your app to deploy in production, you may want
After you have successfully added your cluster information, you can enable the
to use an incremental rollout to replace just a few pods with the latest code.
Kubernetes cluster integration:
This will allow you to first check how the app is behaving, and later manually
increasing the rollout up to 100%.
If `INCREMENTAL_ROLLOUT_ENABLED` is defined in your project (e.g., set
1. Click the "Enabled/Disabled" switch
`INCREMENTAL_ROLLOUT_ENABLED` to `1` as a secret variable), then instead of the
