Commit a1816ff0 authored by Evan Read's avatar Evan Read Committed by Achilleas Pipinellis

SSoT audit fixes

Implements part of the single source of truth audit
fixes for the CI section.
parent a80c6a68
---
type: index, concepts, howto
---
# Cache dependencies in GitLab CI/CD # Cache dependencies in GitLab CI/CD
GitLab CI/CD provides a caching mechanism that can be used to save time GitLab CI/CD provides a caching mechanism that can be used to save time
...@@ -60,7 +64,7 @@ In summary: ...@@ -60,7 +64,7 @@ In summary:
- Caches are disabled if not defined globally or per job (using `cache:`). - Caches are disabled if not defined globally or per job (using `cache:`).
- Caches are available for all jobs in your `.gitlab-ci.yml` if enabled globally. - Caches are available for all jobs in your `.gitlab-ci.yml` if enabled globally.
- Caches can be used by subsequent pipelines of that very same job (a script in - Caches can be used by subsequent pipelines of that same job (a script in
a stage) in which the cache was created (if not defined globally). a stage) in which the cache was created (if not defined globally).
- Caches are stored where the Runner is installed **and** uploaded to S3 if - Caches are stored where the Runner is installed **and** uploaded to S3 if
[distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching).
...@@ -87,7 +91,7 @@ cache, when declaring `cache` in your jobs, use one or a mix of the following: ...@@ -87,7 +91,7 @@ cache, when declaring `cache` in your jobs, use one or a mix of the following:
that share their cache. that share their cache.
- [Use sticky Runners](../runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) - [Use sticky Runners](../runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects)
that will be only available to a particular project. that will be only available to a particular project.
- [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (e.g., - [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example,
different caches on each branch). For that, you can take advantage of the different caches on each branch). For that, you can take advantage of the
[CI/CD predefined variables](../variables/README.md#predefined-environment-variables). [CI/CD predefined variables](../variables/README.md#predefined-environment-variables).
...@@ -420,7 +424,7 @@ mismatch and a few ideas how to fix it. ...@@ -420,7 +424,7 @@ mismatch and a few ideas how to fix it.
Let's explore some examples. Let's explore some examples.
--- #### Examples
Let's assume you have only one Runner assigned to your project, so the cache Let's assume you have only one Runner assigned to your project, so the cache
will be stored in the Runner's machine by default. If two jobs, A and B, will be stored in the Runner's machine by default. If two jobs, A and B,
...@@ -462,8 +466,6 @@ job B: ...@@ -462,8 +466,6 @@ job B:
To fix that, use different `keys` for each job. To fix that, use different `keys` for each job.
---
In another case, let's assume you have more than one Runners assigned to your In another case, let's assume you have more than one Runners assigned to your
project, but the distributed cache is not enabled. We want the second time the project, but the distributed cache is not enabled. We want the second time the
pipeline is run, `job A` and `job B` to re-use their cache (which in this case pipeline is run, `job A` and `job B` to re-use their cache (which in this case
...@@ -526,3 +528,15 @@ Behind the scenes, this works by increasing a counter in the database, and the ...@@ -526,3 +528,15 @@ Behind the scenes, this works by increasing a counter in the database, and the
value of that counter is used to create the key for the cache by appending an value of that counter is used to create the key for the cache by appending an
integer to it: `-1`, `-2`, etc. After a push, a new key is generated and the integer to it: `-1`, `-2`, etc. After a push, a new key is generated and the
old cache is not valid anymore. old cache is not valid anymore.
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
---
type: index, concepts, howto
---
# GitLab ChatOps # GitLab ChatOps
> **Notes:** > **Notes:**
...@@ -10,7 +14,10 @@ GitLab ChatOps provides a method to interact with CI/CD jobs through chat servic ...@@ -10,7 +14,10 @@ GitLab ChatOps provides a method to interact with CI/CD jobs through chat servic
## How it works ## How it works
GitLab ChatOps is built upon two existing features, [GitLab CI/CD](../README.md) and [Slack Slash Commmands](../../user/project/integrations/slack_slash_commands.md). GitLab ChatOps is built upon two existing features:
- [GitLab CI/CD](../README.md).
- [Slack Slash Commands](../../user/project/integrations/slack_slash_commands.md).
A new `run` action has been added to the [slash commands](../../integration/slash_commands.md), which takes two arguments: a `<job name>` to execute and the `<job arguments>`. When executed, ChatOps will look up the specified job name and attempt to match it to a corresponding job in [.gitlab-ci.yml](../yaml/README.md). If a matching job is found on `master`, a pipeline containing just that job is scheduled. Two additional [CI/CD variables](../variables/README.md#predefined-environment-variables) are passed to the job: `CHAT_INPUT` contains any additional arguments, and `CHAT_CHANNEL` is set to the name of channel the action was triggered in. A new `run` action has been added to the [slash commands](../../integration/slash_commands.md), which takes two arguments: a `<job name>` to execute and the `<job arguments>`. When executed, ChatOps will look up the specified job name and attempt to match it to a corresponding job in [.gitlab-ci.yml](../yaml/README.md). If a matching job is found on `master`, a pipeline containing just that job is scheduled. Two additional [CI/CD variables](../variables/README.md#predefined-environment-variables) are passed to the job: `CHAT_INPUT` contains any additional arguments, and `CHAT_CHANNEL` is set to the name of channel the action was triggered in.
...@@ -22,9 +29,9 @@ After the job has finished, its output is sent back to Slack provided it has com ...@@ -22,9 +29,9 @@ After the job has finished, its output is sent back to Slack provided it has com
Since ChatOps is built upon GitLab CI/CD, the job has all the same features and functions available. There a few best practices to consider however when creating ChatOps jobs: Since ChatOps is built upon GitLab CI/CD, the job has all the same features and functions available. There a few best practices to consider however when creating ChatOps jobs:
* It is strongly recommended to set `only: [chat]` so the job does not run as part of the standard CI pipeline. - It is strongly recommended to set `only: [chat]` so the job does not run as part of the standard CI pipeline.
* If the job is set to `when: manual`, the pipeline will be created however the job will wait to be started. - If the job is set to `when: manual`, the pipeline will be created however the job will wait to be started.
* It is important to keep in mind that there is very limited support for access control. If the user who triggered the slash command is a developer in the project, the job will run. The job itself can utilize existing [CI/CD variables](../variables/README.html#predefined-environment-variables) like `GITLAB_USER_ID` to perform additional rights validation, however these variables can be [overridden](../variables/README.html#priority-of-environment-variables). - It is important to keep in mind that there is limited support for access control. If the user who triggered the slash command is a developer in the project, the job will run. The job itself can utilize existing [CI/CD variables](../variables/README.html#predefined-environment-variables) like `GITLAB_USER_ID` to perform additional rights validation, however these variables can be [overridden](../variables/README.html#priority-of-environment-variables).
### Controlling the ChatOps reply ### Controlling the ChatOps reply
...@@ -59,3 +66,15 @@ You can find and download the official GitLab ChatOps icon here. ...@@ -59,3 +66,15 @@ You can find and download the official GitLab ChatOps icon here.
![GitLab ChatOps bot icon](img/gitlab-chatops-icon-small.png) ![GitLab ChatOps bot icon](img/gitlab-chatops-icon-small.png)
[Download bigger image](img/gitlab-chatops-icon.png) [Download bigger image](img/gitlab-chatops-icon.png)
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
---
type: howto
---
# Using GitLab CI/CD with a Bitbucket Cloud repository **[PREMIUM]** # Using GitLab CI/CD with a Bitbucket Cloud repository **[PREMIUM]**
GitLab CI/CD can be used with Bitbucket Cloud by creating a GitLab CI/CD can be used with Bitbucket Cloud by:
[CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html) and connecting
your Git repository via URL. 1. Creating a [CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html).
1. Connecting your Git repository via URL.
To use GitLab CI/CD with a Bitbucket Cloud repository:
1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and 1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and
create the project. create the project.
...@@ -16,13 +23,13 @@ your Git repository via URL. ...@@ -16,13 +23,13 @@ your Git repository via URL.
with `api` scope. This will be used to authenticate requests from the web with `api` scope. This will be used to authenticate requests from the web
hook that will be created in Bitbucket to notify GitLab of new commits. hook that will be created in Bitbucket to notify GitLab of new commits.
1. In Bitbucket from **Settings > Webhooks** create a new web hook to notify 1. In Bitbucket, from **Settings > Webhooks**, create a new web hook to notify
GitLab of new commits. GitLab of new commits.
The web hook URL should be set to the GitLab API to trigger pull mirroring, The web hook URL should be set to the GitLab API to trigger pull mirroring,
using the Personal Access Token we just generated for authentication. using the Personal Access Token we just generated for authentication.
``` ```text
https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN>
``` ```
...@@ -33,27 +40,27 @@ your Git repository via URL. ...@@ -33,27 +40,27 @@ your Git repository via URL.
After saving, test the web hook by pushing a change to your Bitbucket After saving, test the web hook by pushing a change to your Bitbucket
repository. repository.
1. In Bitbucket create an **App Password** from **Bitbucket Settings > App 1. In Bitbucket, create an **App Password** from **Bitbucket Settings > App
Passwords** to authenticate the build status script setting commit build Passwords** to authenticate the build status script setting commit build
statuses in Bitbucket. Repository write permissions are required. statuses in Bitbucket. Repository write permissions are required.
![Bitbucket Cloud webhook](img/bitbucket_app_password.png) ![Bitbucket Cloud webhook](img/bitbucket_app_password.png)
1. In GitLab from **Settings > CI/CD > Environment variables** add variables to allow 1. In GitLab, from **Settings > CI/CD > Environment variables**, add variables to allow
communication with Bitbucket via the Bitbucket API. communication with Bitbucket via the Bitbucket API:
`BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above.
`BITBUCKET_USERNAME`: the username of the Bitbucket account `BITBUCKET_USERNAME`: the username of the Bitbucket account.
`BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ.
`BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ.
1. In Bitbucket add a script to push the pipeline status to Bitbucket. 1. In Bitbucket, add a script to push the pipeline status to Bitbucket.
> Note: changes made in GitLab will be overwritten by any changes made > Note: changes made in GitLab will be overwritten by any changes made
upstream in Bitbucket. > upstream in Bitbucket.
Create a file `build_status` and insert the script below and run Create a file `build_status` and insert the script below and run
`chmod +x build_status` in your terminal to make the script executable. `chmod +x build_status` in your terminal to make the script executable.
...@@ -111,7 +118,7 @@ your Git repository via URL. ...@@ -111,7 +118,7 @@ your Git repository via URL.
1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push 1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push
pipeline success and failures to Bitbucket. pipeline success and failures to Bitbucket.
``` ```yaml
stages: stages:
- test - test
- ci_status - ci_status
...@@ -145,3 +152,15 @@ GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines ...@@ -145,3 +152,15 @@ GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines
configured in `.gitlab-ci.yml` and push the status to Bitbucket. configured in `.gitlab-ci.yml` and push the status to Bitbucket.
[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter [pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
---
type: howto
---
# Using GitLab CI/CD with a GitHub repository **[PREMIUM]** # Using GitLab CI/CD with a GitHub repository **[PREMIUM]**
GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by
...@@ -15,7 +19,7 @@ administrator: ...@@ -15,7 +19,7 @@ administrator:
NOTE: **Note:** NOTE: **Note:**
Due to a 10-token limitation on the [GitHub OAuth Implementation](https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps/#creating-multiple-tokens-for-oauth-apps), Due to a 10-token limitation on the [GitHub OAuth Implementation](https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps/#creating-multiple-tokens-for-oauth-apps),
if you import more than 10 times, your oldest imported project's token will be if you import more than 10 times, your oldest imported project's token will be
revoked. See issue [#9147](https://gitlab.com/gitlab-org/gitlab-ee/issues/9147) revoked. See issue [#9147](https://gitlab.com/gitlab-org/gitlab-ee/issues/9147)
for more information. for more information.
...@@ -31,23 +35,27 @@ for more information. ...@@ -31,23 +35,27 @@ for more information.
1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). 1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md).
GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable GitLab will:
[GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html), and create a web hook
on GitHub to notify GitLab of new commits. 1. Import the project.
1. Enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter).
1. Enable [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html).
1. Create a web hook on GitHub to notify GitLab of new commits.
## Connect with Personal Access Token ## Connect with Personal Access Token
NOTE: **Note:** Personal access tokens can only be used to connect GitHub.com NOTE: **Note:**
Personal access tokens can only be used to connect GitHub.com
repositories to GitLab. repositories to GitLab.
If you are not using the [GitHub integration](../../integration/github.md), you can If you are not using the [GitHub integration](../../integration/github.md), you can
still perform a one-off authorization with GitHub to grant GitLab access your still perform a one-off authorization with GitHub to grant GitLab access your
repositories: repositories:
1. Open https://github.com/settings/tokens/new to create a **Personal Access 1. Open <https://github.com/settings/tokens/new> to create a **Personal Access
Token**. This token with be used to access your repository and push commit Token**. This token with be used to access your repository and push commit
statuses to GitHub. statuses to GitHub.
The `repo` and `admin:repo_hook` should be enable to allow GitLab access to The `repo` and `admin:repo_hook` should be enable to allow GitLab access to
your project, update commit statuses, and create a web hook to notify your project, update commit statuses, and create a web hook to notify
GitLab of new commits. GitLab of new commits.
...@@ -62,20 +70,23 @@ repositories: ...@@ -62,20 +70,23 @@ repositories:
1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). 1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md).
GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable GitLab will:
[GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html), and create a web hook
on GitHub to notify GitLab of new commits. 1. Import the project.
1. Enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter).
1. Enable [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html).
1. Create a web hook on GitHub to notify GitLab of new commits.
## Connect manually ## Connect manually
If the [GitHub integration](../../integration/github.md) is not enabled, or is enabled If the [GitHub integration](../../integration/github.md) is not enabled, or is enabled
for a different GitHub instance, you GitLab CI/CD can be manually enabled for for a different GitHub instance, you GitLab CI/CD can be manually enabled for
your repository. your repository:
1. In GitHub open https://github.com/settings/tokens/new create a **Personal 1. In GitHub open <https://github.com/settings/tokens/new> create a **Personal
Access Token.** GitLab will use this token to access your repository and Access Token.** GitLab will use this token to access your repository and
push commit statuses. push commit statuses.
Enter a **Token description** and update the scope to allow: Enter a **Token description** and update the scope to allow:
`repo` so that GitLab can access your project and update commit statuses `repo` so that GitLab can access your project and update commit statuses
...@@ -109,3 +120,15 @@ your repository. ...@@ -109,3 +120,15 @@ your repository.
![Create web hook](img/github_push_webhook.png) ![Create web hook](img/github_push_webhook.png)
1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD. 1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD.
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
---
type: index, howto
---
# GitLab CI/CD for external repositories **[PREMIUM]** # GitLab CI/CD for external repositories **[PREMIUM]**
NOTE: **Note:** NOTE: **Note:**
...@@ -6,22 +10,26 @@ GitLab.com users until September 22nd, 2019. ...@@ -6,22 +10,26 @@ GitLab.com users until September 22nd, 2019.
>[Introduced][ee-4642] in [GitLab Premium][eep] 10.6. >[Introduced][ee-4642] in [GitLab Premium][eep] 10.6.
GitLab CI/CD can be used with GitHub or any other Git server. GitLab CI/CD can be used with:
- [GitHub](github_integration.md).
- [Bitbucket Cloud](bitbucket_integration.md).
- Any other Git server.
Instead of moving your entire project to GitLab, you can connect your Instead of moving your entire project to GitLab, you can connect your
external repository to get the benefits of GitLab CI/CD. external repository to get the benefits of GitLab CI/CD.
- [GitHub](github_integration.md)
- [Bitbucket Cloud](bitbucket_integration.md)
Connecting an external repository will set up [repository mirroring][mirroring] Connecting an external repository will set up [repository mirroring][mirroring]
and create a lightweight project where issues, merge requests, wiki, and and create a lightweight project where issues, merge requests, wiki, and
snippets disabled. These features snippets disabled. These features
[can be re-enabled later][settings]. [can be re-enabled later][settings].
1. From your GitLab dashboard click **New project** To connect to an external repository:
1. Switch to the **CI/CD for external repo** tab
1. Choose **GitHub** or **Repo by URL** 1. From your GitLab dashboard, click **New project**.
1. The next steps are similar to the [import flow](../../user/project/import/index.md) 1. Switch to the **CI/CD for external repo** tab.
1. Choose **GitHub** or **Repo by URL**.
1. The next steps are similar to the [import flow](../../user/project/import/index.md).
![CI/CD for external repository project creation](img/ci_cd_for_external_repo.png) ![CI/CD for external repository project creation](img/ci_cd_for_external_repo.png)
......
--- ---
comments: false comments: false
type: index
--- ---
# Docker integration # Docker integration
GitLab CI/CD can be combined with [Docker](https://www.docker.com) to enable
integration between the two.
The following documentation is available for using GitLab CI/CD with Docker: The following documentation is available for using GitLab CI/CD with Docker:
- [Using Docker images](using_docker_images.md). - [Using Docker images](using_docker_images.md).
......
---
type: concepts, howto
---
# Building Docker images with GitLab CI/CD # Building Docker images with GitLab CI/CD
GitLab CI/CD allows you to use Docker Engine to build and test docker-based projects. GitLab CI/CD allows you to use Docker Engine to build and test docker-based projects.
...@@ -5,10 +9,10 @@ GitLab CI/CD allows you to use Docker Engine to build and test docker-based proj ...@@ -5,10 +9,10 @@ GitLab CI/CD allows you to use Docker Engine to build and test docker-based proj
One of the new trends in Continuous Integration/Deployment is to: One of the new trends in Continuous Integration/Deployment is to:
1. Create an application image 1. Create an application image.
1. Run tests against the created image 1. Run tests against the created image.
1. Push image to a remote registry 1. Push image to a remote registry.
1. Deploy to a server from the pushed image 1. Deploy to a server from the pushed image.
It's also useful when your application already has the `Dockerfile` that can be It's also useful when your application already has the `Dockerfile` that can be
used to create and test an image: used to create and test an image:
...@@ -113,7 +117,7 @@ In order to do that, follow the steps: ...@@ -113,7 +117,7 @@ In order to do that, follow the steps:
The above command will create a `config.toml` entry similar to this: The above command will create a `config.toml` entry similar to this:
``` ```toml
[[runners]] [[runners]]
url = "https://gitlab.com/" url = "https://gitlab.com/"
token = TOKEN token = TOKEN
...@@ -227,7 +231,7 @@ In order to do that, follow the steps: ...@@ -227,7 +231,7 @@ In order to do that, follow the steps:
The above command will create a `config.toml` entry similar to this: The above command will create a `config.toml` entry similar to this:
``` ```toml
[[runners]] [[runners]]
url = "https://gitlab.com/" url = "https://gitlab.com/"
token = REGISTRATION_TOKEN token = REGISTRATION_TOKEN
...@@ -270,9 +274,9 @@ aware of the following implications: ...@@ -270,9 +274,9 @@ aware of the following implications:
create containers with specific names, they may conflict with each other. create containers with specific names, they may conflict with each other.
- Sharing files and directories from the source repo into containers may not - Sharing files and directories from the source repo into containers may not
work as expected since volume mounting is done in the context of the host work as expected since volume mounting is done in the context of the host
machine, not the build container, e.g.: machine, not the build container. For example:
``` ```sh
docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests
``` ```
...@@ -337,7 +341,7 @@ NOTE: **Note:** ...@@ -337,7 +341,7 @@ NOTE: **Note:**
The shared Runners on GitLab.com use the `overlay2` driver by default. The shared Runners on GitLab.com use the `overlay2` driver by default.
By default, when using `docker:dind`, Docker uses the `vfs` storage driver which By default, when using `docker:dind`, Docker uses the `vfs` storage driver which
copies the filesystem on every run. This is a very disk-intensive operation copies the filesystem on every run. This is a disk-intensive operation
which can be avoided if a different driver is used, for example `overlay2`. which can be avoided if a different driver is used, for example `overlay2`.
### Requirements ### Requirements
...@@ -345,13 +349,13 @@ which can be avoided if a different driver is used, for example `overlay2`. ...@@ -345,13 +349,13 @@ which can be avoided if a different driver is used, for example `overlay2`.
1. Make sure a recent kernel is used, preferably `>= 4.2`. 1. Make sure a recent kernel is used, preferably `>= 4.2`.
1. Check whether the `overlay` module is loaded: 1. Check whether the `overlay` module is loaded:
``` ```sh
sudo lsmod | grep overlay sudo lsmod | grep overlay
``` ```
If you see no result, then it isn't loaded. To load it use: If you see no result, then it isn't loaded. To load it use:
``` ```sh
sudo modprobe overlay sudo modprobe overlay
``` ```
...@@ -359,7 +363,7 @@ which can be avoided if a different driver is used, for example `overlay2`. ...@@ -359,7 +363,7 @@ which can be avoided if a different driver is used, for example `overlay2`.
On Ubuntu systems, this is done by editing `/etc/modules`. Just add the On Ubuntu systems, this is done by editing `/etc/modules`. Just add the
following line into it: following line into it:
``` ```text
overlay overlay
``` ```
...@@ -367,7 +371,7 @@ which can be avoided if a different driver is used, for example `overlay2`. ...@@ -367,7 +371,7 @@ which can be avoided if a different driver is used, for example `overlay2`.
You can enable the driver for each project individually by editing the project's `.gitlab-ci.yml`: You can enable the driver for each project individually by editing the project's `.gitlab-ci.yml`:
``` ```yaml
variables: variables:
DOCKER_DRIVER: overlay2 DOCKER_DRIVER: overlay2
``` ```
...@@ -571,3 +575,15 @@ deploy: ...@@ -571,3 +575,15 @@ deploy:
[docker-cap]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities [docker-cap]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities
[2fa]: ../../user/profile/account/two_factor_authentication.md [2fa]: ../../user/profile/account/two_factor_authentication.md
[pat]: ../../user/profile/personal_access_tokens.md [pat]: ../../user/profile/personal_access_tokens.md
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
---
type: concepts, howto
---
# Using Docker images # Using Docker images
GitLab CI in conjunction with [GitLab Runner](../runners/README.md) can use GitLab CI in conjunction with [GitLab Runner](../runners/README.md) can use
...@@ -45,10 +49,10 @@ The `image` keyword is the name of the Docker image the Docker executor ...@@ -45,10 +49,10 @@ The `image` keyword is the name of the Docker image the Docker executor
will run to perform the CI tasks. will run to perform the CI tasks.
By default, the executor will only pull images from [Docker Hub][hub], By default, the executor will only pull images from [Docker Hub][hub],
but this can be configured in the `gitlab-runner/config.toml` by setting however this can be configured in the `gitlab-runner/config.toml` by setting
the [Docker pull policy][] to allow using local images. the [Docker pull policy][] to allow using local images.
For more information about images and Docker Hub please read For more information about images and Docker Hub, please read
the [Docker Fundamentals][] documentation. the [Docker Fundamentals][] documentation.
## What is a service ## What is a service
...@@ -95,8 +99,8 @@ required for the CI/CD job to proceed and is accessed by network. ...@@ -95,8 +99,8 @@ required for the CI/CD job to proceed and is accessed by network.
To make sure this works, the Runner: To make sure this works, the Runner:
1. checks which ports are exposed from the container by default 1. Checks which ports are exposed from the container by default.
1. starts a special container that waits for these ports to be accessible 1. Starts a special container that waits for these ports to be accessible.
When the second stage of the check fails, either because there is no opened port in the When the second stage of the check fails, either because there is no opened port in the
service, or the service was not started properly before the timeout and the port is not service, or the service was not started properly before the timeout and the port is not
...@@ -106,7 +110,7 @@ In most cases it will affect the job, but there may be situations when the job ...@@ -106,7 +110,7 @@ In most cases it will affect the job, but there may be situations when the job
will still succeed even if that warning was printed. For example: will still succeed even if that warning was printed. For example:
- The service was started a little after the warning was raised, and the job is - The service was started a little after the warning was raised, and the job is
not using the linked service from the very beginning. In that case, when the not using the linked service from the beginning. In that case, when the
job needed to access the service, it may have been already there waiting for job needed to access the service, it may have been already there waiting for
connections. connections.
- The service container is not providing any networking service, but it's doing - The service container is not providing any networking service, but it's doing
...@@ -143,9 +147,9 @@ job: ...@@ -143,9 +147,9 @@ job:
If you need to have `php`, `node` and `go` available for your script, you should If you need to have `php`, `node` and `go` available for your script, you should
either: either:
- choose an existing Docker image that contains all required tools, or - Choose an existing Docker image that contains all required tools.
- create your own Docker image, which will have all the required tools included - Create your own Docker image, which will have all the required tools included
and use that in your job and use that in your job.
### Accessing the services ### Accessing the services
...@@ -167,18 +171,18 @@ access to it from your build container under two hostnames to choose from: ...@@ -167,18 +171,18 @@ access to it from your build container under two hostnames to choose from:
- `tutum-wordpress` - `tutum-wordpress`
- `tutum__wordpress` - `tutum__wordpress`
>**Note:** NOTE: **Note:**
Hostnames with underscores are not RFC valid and may cause problems in 3rd party Hostnames with underscores are not RFC valid and may cause problems in 3rd party
applications. applications.
The default aliases for the service's hostname are created from its image name The default aliases for the service's hostname are created from its image name
following these rules: following these rules:
- Everything after the colon (`:`) is stripped - Everything after the colon (`:`) is stripped.
- Slash (`/`) is replaced with double underscores (`__`) and the primary alias - Slash (`/`) is replaced with double underscores (`__`) and the primary alias
is created is created.
- Slash (`/`) is replaced with a single dash (`-`) and the secondary alias is - Slash (`/`) is replaced with a single dash (`-`) and the secondary alias is
created (requires GitLab Runner v1.1.0 or higher) created (requires GitLab Runner v1.1.0 or higher).
To override the default behavior, you can To override the default behavior, you can
[specify a service alias](#available-settings-for-services). [specify a service alias](#available-settings-for-services).
...@@ -333,7 +337,7 @@ services: ...@@ -333,7 +337,7 @@ services:
``` ```
The Runner will still start two containers using the `mysql:latest` image, The Runner will still start two containers using the `mysql:latest` image,
but now each of them will also be accessible with the alias configured however now each of them will also be accessible with the alias configured
in `.gitlab-ci.yml` file. in `.gitlab-ci.yml` file.
### Setting a command for the service ### Setting a command for the service
...@@ -408,8 +412,6 @@ you should check which one your Runner is using. Specifically: ...@@ -408,8 +412,6 @@ you should check which one your Runner is using. Specifically:
The syntax of `image:entrypoint` is similar to [Dockerfile's `ENTRYPOINT`][entrypoint]. The syntax of `image:entrypoint` is similar to [Dockerfile's `ENTRYPOINT`][entrypoint].
----
Let's assume you have a `super/sql:experimental` image with some SQL database Let's assume you have a `super/sql:experimental` image with some SQL database
inside it and you would like to use it as a base image for your job because you inside it and you would like to use it as a base image for your job because you
want to execute some tests with this database binary. Let's also assume that want to execute some tests with this database binary. Let's also assume that
...@@ -443,7 +445,7 @@ image: ...@@ -443,7 +445,7 @@ image:
Look for the `[runners.docker]` section: Look for the `[runners.docker]` section:
``` ```toml
[runners.docker] [runners.docker]
image = "ruby:2.1" image = "ruby:2.1"
services = ["mysql:latest", "postgres:latest"] services = ["mysql:latest", "postgres:latest"]
...@@ -469,11 +471,11 @@ image which is private and requires you to login into a private container regist ...@@ -469,11 +471,11 @@ image which is private and requires you to login into a private container regist
Let's also assume that these are the login credentials: Let's also assume that these are the login credentials:
| Key | Value | | Key | Value |
|----------|---------------------------| |:---------|:----------------------------|
| registry | registry.example.com:5000 | | registry | `registry.example.com:5000` |
| username | my_username | | username | `my_username` |
| password | my_password | | password | `my_password` |
To configure access for `registry.example.com:5000`, follow these steps: To configure access for `registry.example.com:5000`, follow these steps:
...@@ -534,7 +536,8 @@ To configure access for `registry.example.com:5000`, follow these steps: ...@@ -534,7 +536,8 @@ To configure access for `registry.example.com:5000`, follow these steps:
You can add configuration for as many registries as you want, adding more You can add configuration for as many registries as you want, adding more
registries to the `"auths"` hash as described above. registries to the `"auths"` hash as described above.
NOTE: **Note:** The full `hostname:port` combination is required everywhere NOTE: **Note:**
The full `hostname:port` combination is required everywhere
for the Runner to match the `DOCKER_AUTH_CONFIG`. For example, if for the Runner to match the `DOCKER_AUTH_CONFIG`. For example, if
`registry.example.com:5000/namespace/image:tag` is specified in `.gitlab-ci.yml`, `registry.example.com:5000/namespace/image:tag` is specified in `.gitlab-ci.yml`,
then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`. then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`.
...@@ -551,8 +554,9 @@ service containers. ...@@ -551,8 +554,9 @@ service containers.
For all possible configuration variables check the documentation of each image For all possible configuration variables check the documentation of each image
provided in their corresponding Docker hub page. provided in their corresponding Docker hub page.
*Note: All variables will be passed to all services containers. It's not NOTE: **Note:**
designed to distinguish which variable should go where.* All variables will be passed to all services containers. It's not
designed to distinguish which variable should go where.
### PostgreSQL service example ### PostgreSQL service example
...@@ -582,8 +586,9 @@ time. ...@@ -582,8 +586,9 @@ time.
## How to debug a job locally ## How to debug a job locally
*Note: The following commands are run without root privileges. You should be NOTE: **Note:**
able to run Docker with your regular user account.* The following commands are run without root privileges. You should be
able to run Docker with your regular user account.
First start with creating a file named `build_script`: First start with creating a file named `build_script`:
...@@ -602,7 +607,7 @@ is specific to your project. ...@@ -602,7 +607,7 @@ is specific to your project.
Then create some service containers: Then create some service containers:
``` ```sh
docker run -d --name service-mysql mysql:latest docker run -d --name service-mysql mysql:latest
docker run -d --name service-postgres postgres:latest docker run -d --name service-postgres postgres:latest
``` ```
...@@ -614,7 +619,7 @@ respectively. They will both run in the background (`-d`). ...@@ -614,7 +619,7 @@ respectively. They will both run in the background (`-d`).
Finally, create a build container by executing the `build_script` file we Finally, create a build container by executing the `build_script` file we
created earlier: created earlier:
``` ```sh
docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.1 /bin/bash < build_script docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.1 /bin/bash < build_script
``` ```
...@@ -626,7 +631,7 @@ piped using STDIN to the bash interpreter which in turn executes the ...@@ -626,7 +631,7 @@ piped using STDIN to the bash interpreter which in turn executes the
When you finish testing and no longer need the containers, you can remove them When you finish testing and no longer need the containers, you can remove them
with: with:
``` ```sh
docker rm -f -v build service-mysql service-postgres docker rm -f -v build service-mysql service-postgres
``` ```
......
---
type: howto
---
# Building images with kaniko and GitLab CI/CD # Building images with kaniko and GitLab CI/CD
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45512) in GitLab 11.2. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45512) in GitLab 11.2.
...@@ -9,17 +13,18 @@ container images from a Dockerfile, inside a container or Kubernetes cluster. ...@@ -9,17 +13,18 @@ container images from a Dockerfile, inside a container or Kubernetes cluster.
kaniko solves two problems with using the kaniko solves two problems with using the
[docker-in-docker build](using_docker_build.md#use-docker-in-docker-executor) method: [docker-in-docker build](using_docker_build.md#use-docker-in-docker-executor) method:
1. Docker-in-docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) - Docker-in-docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities)
in order to function, which is a significant security concern. in order to function, which is a significant security concern.
1. Docker-in-docker generally incurs a performance penalty and can be quite slow. - Docker-in-docker generally incurs a performance penalty and can be quite slow.
## Requirements ## Requirements
In order to utilize kaniko with GitLab, a [GitLab Runner](https://docs.gitlab.com/runner/) In order to utilize kaniko with GitLab, a [GitLab Runner](https://docs.gitlab.com/runner/)
using either the [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html), using one of the following executors is required:
[Docker](https://docs.gitlab.com/runner/executors/docker.html), or
[Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html) - [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html).
executors is required. - [Docker](https://docs.gitlab.com/runner/executors/docker.html).
- [Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html).
## Building a Docker image with kaniko ## Building a Docker image with kaniko
...@@ -34,14 +39,17 @@ few important details: ...@@ -34,14 +39,17 @@ few important details:
- A Docker `config.json` file needs to be created with the authentication - A Docker `config.json` file needs to be created with the authentication
information for the desired container registry. information for the desired container registry.
--- In the following example, kaniko is used to:
1. Build a Docker image.
1. Then push it to [GitLab Container Registry](../../user/project/container_registry.md).
In the following example, kaniko is used to build a Docker image and then push
it to [GitLab Container Registry](../../user/project/container_registry.md).
The job will run only when a tag is pushed. A `config.json` file is created under The job will run only when a tag is pushed. A `config.json` file is created under
`/kaniko/.docker` with the needed GitLab Container Registry credentials taken from the `/kaniko/.docker` with the needed GitLab Container Registry credentials taken from the
[environment variables](../variables/README.md#predefined-environment-variables) [environment variables](../variables/README.md#predefined-environment-variables)
GitLab CI/CD provides. In the last step, kaniko uses the `Dockerfile` under the GitLab CI/CD provides.
In the last step, kaniko uses the `Dockerfile` under the
root directory of the project, builds the Docker image and pushes it to the root directory of the project, builds the Docker image and pushes it to the
project's Container Registry while tagging it with the Git tag: project's Container Registry while tagging it with the Git tag:
...@@ -80,3 +88,15 @@ store: ...@@ -80,3 +88,15 @@ store:
... ...
-----END CERTIFICATE-----" >> /kaniko/ssl/certs/ca-certificates.crt -----END CERTIFICATE-----" >> /kaniko/ssl/certs/ca-certificates.crt
``` ```
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
--- ---
type: reference type: howto
--- ---
# How to enable or disable GitLab CI/CD # How to enable or disable GitLab CI/CD
To effectively use GitLab CI/CD, you need a valid [`.gitlab-ci.yml`](yaml/README.md) To effectively use GitLab CI/CD, you need:
file present at the root directory of your project and a
[runner](runners/README.md) properly set up. You can read our - A valid [`.gitlab-ci.yml`](yaml/README.md) file present at the root directory
[quick start guide](quick_start/README.md) to get you started. of your project.
- A [runner](runners/README.md) properly set up.
You can read our [quick start guide](quick_start/README.md) to get you started.
If you are using an external CI/CD server like Jenkins or Drone CI, it is advised If you are using an external CI/CD server like Jenkins or Drone CI, it is advised
to disable GitLab CI/CD in order to not have any conflicts with the commits status to disable GitLab CI/CD in order to not have any conflicts with the commits status
API. API.
---
GitLab CI/CD is exposed via the `/pipelines` and `/jobs` pages of a project. GitLab CI/CD is exposed via the `/pipelines` and `/jobs` pages of a project.
Disabling GitLab CI/CD in a project does not delete any previous jobs. Disabling GitLab CI/CD in a project does not delete any previous jobs.
In fact, the `/pipelines` and `/jobs` pages can still be accessed, although In fact, the `/pipelines` and `/jobs` pages can still be accessed, although
it's hidden from the left sidebar menu. it's hidden from the left sidebar menu.
GitLab CI/CD is enabled by default on new installations and can be disabled either GitLab CI/CD is enabled by default on new installations and can be disabled
individually under each project's settings, or site-wide by modifying the either:
settings in `gitlab.yml` and `gitlab.rb` for source and Omnibus installations
respectively. - Individually under each project's settings.
- Site-wide by modifying the settings in `gitlab.yml` and `gitlab.rb` for source
and Omnibus installations respectively.
## Per-project user setting ## Per-project user setting
...@@ -40,10 +43,10 @@ and `gitlab.rb` for source and Omnibus installations respectively. ...@@ -40,10 +43,10 @@ and `gitlab.rb` for source and Omnibus installations respectively.
Two things to note: Two things to note:
1. Disabling GitLab CI/CD, will affect only newly-created projects. Projects that - Disabling GitLab CI/CD, will affect only newly-created projects. Projects that
had it enabled prior to this modification, will work as before. had it enabled prior to this modification, will work as before.
1. Even if you disable GitLab CI/CD, users will still be able to enable it in the - Even if you disable GitLab CI/CD, users will still be able to enable it in the
project's settings. project's settings.
For installations from source, open `gitlab.yml` with your editor and set For installations from source, open `gitlab.yml` with your editor and set
`builds` to `false`: `builds` to `false`:
...@@ -58,12 +61,32 @@ default_projects_features: ...@@ -58,12 +61,32 @@ default_projects_features:
builds: false builds: false
``` ```
Save the file and restart GitLab: `sudo service gitlab restart`. Save the file and restart GitLab:
```sh
sudo service gitlab restart
```
For Omnibus installations, edit `/etc/gitlab/gitlab.rb` and add the line: For Omnibus installations, edit `/etc/gitlab/gitlab.rb` and add the line:
``` ```ruby
gitlab_rails['gitlab_default_projects_features_builds'] = false gitlab_rails['gitlab_default_projects_features_builds'] = false
``` ```
Save the file and reconfigure GitLab: `sudo gitlab-ctl reconfigure`. Save the file and reconfigure GitLab:
```sh
sudo gitlab-ctl reconfigure
```
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
...@@ -92,7 +92,7 @@ deploy_staging: ...@@ -92,7 +92,7 @@ deploy_staging:
- master - master
``` ```
We have defined 3 [stages](yaml/README.md#stages): We have defined three [stages](yaml/README.md#stages):
- `test` - `test`
- `build` - `build`
...@@ -135,7 +135,7 @@ In summary, with the above `.gitlab-ci.yml` we have achieved the following: ...@@ -135,7 +135,7 @@ In summary, with the above `.gitlab-ci.yml` we have achieved the following:
> the name given in `.gitlab-ci.yml` (with any variables expanded), while the > the name given in `.gitlab-ci.yml` (with any variables expanded), while the
> second is a "cleaned-up" version of the name, suitable for use in URLs, DNS, > second is a "cleaned-up" version of the name, suitable for use in URLs, DNS,
> etc. > etc.
>
> Starting with GitLab 9.3, the environment URL is exposed to the Runner via > Starting with GitLab 9.3, the environment URL is exposed to the Runner via
> `$CI_ENVIRONMENT_URL`. The URL is expanded from `.gitlab-ci.yml`, or if > `$CI_ENVIRONMENT_URL`. The URL is expanded from `.gitlab-ci.yml`, or if
> the URL was not defined there, the external URL from the environment is used. > the URL was not defined there, the external URL from the environment is used.
......
---
type: concepts, howto
---
# Protected Environments **[PREMIUM]** # Protected Environments **[PREMIUM]**
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6303) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6303) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.3.
...@@ -47,3 +51,15 @@ Maintainers can: ...@@ -47,3 +51,15 @@ Maintainers can:
- Update existing protected environments at any time by changing the access in the - Update existing protected environments at any time by changing the access in the
**Allowed to Deploy** dropdown menu. **Allowed to Deploy** dropdown menu.
- Unprotect a protected environment by clicking the **Unprotect** button for that environment. - Unprotect a protected environment by clicking the **Unprotect** button for that environment.
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g. `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment