Skip to content

G
gitlab-ce
Commit 594147b3 authored by Marcel Amirault's avatar Marcel Amirault Committed by Suzanne Selhorn
Browse files
Options

Move runner variable config docs 

Some runner features can be configured with variables,
but these details should be in the runner docs, not
the CI YAML ref.
parent 7b798038
Hide whitespace changes
Inline Side-by-side
Showing with 406 additions and 367 deletions
doc/ci/environments/index.md
View file @ 594147b3
......@@ -697,7 +697,7 @@ stop_review:
```
If you can't use [Pipelines for merge requests](../merge_request_pipelines/index.md),
setting the [`GIT_STRATEGY`](../yaml/README.md#git-strategy) to `none` is necessary in the
setting the [`GIT_STRATEGY`](../runners/README.md#git-strategy) to `none` is necessary in the
`stop_review` job so that the [runner](https://docs.gitlab.com/runner/) won't
try to check out the code after the branch is deleted.
......
doc/ci/git_submodules.md
View file @ 594147b3
......@@ -78,7 +78,7 @@ correctly with your CI jobs:
GIT_SUBMODULE_STRATEGY: recursive
```
See the [`.gitlab-ci.yml` reference](yaml/README.md#git-submodule-strategy)
See the [GitLab Runner documentation](runners/README.md#git-submodule-strategy)
for more details about `GIT_SUBMODULE_STRATEGY`.
1. If you are using an older version of `gitlab-runner`, then use
......
doc/ci/large_repositories/index.md
View file @ 594147b3
......@@ -78,7 +78,7 @@ done by GitLab, requiring you to do them.
> Introduced in GitLab Runner 11.10.
[`GIT_CLONE_PATH`](../yaml/README.md#custom-build-directories) allows you to
[`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) allows you to
control where you clone your sources. This can have implications if you
heavily use big repositories with fork workflow.
......@@ -90,7 +90,7 @@ In such cases, ideally you want to make the GitLab Runner executor be used only
for the given project and not shared across different projects to make this
process more efficient.
The [`GIT_CLONE_PATH`](../yaml/README.md#custom-build-directories) has to be
The [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) has to be
within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path
from disk.
......@@ -98,12 +98,12 @@ from disk.
> Introduced in GitLab Runner 11.10.
[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags) allows you to control
[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) allows you to control
whether or not you require the `git clean` command to be executed for each CI
job. By default, GitLab ensures that you have your worktree on the given SHA,
and that your repository is clean.
[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags) is disabled when set
[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) is disabled when set
to `none`. On very big repositories, this might be desired because `git
clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx
-e .build/` (for example) allows you to control and disable removal of some
......@@ -112,7 +112,7 @@ the incremental builds. This has the biggest effect if you re-use existing
machines and have an existing worktree that you can re-use for builds.
For exact parameters accepted by
[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags), see the documentation
[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags), see the documentation
for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters
are dependent on Git version.
......@@ -120,14 +120,14 @@ are dependent on Git version.
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1.
[`GIT_FETCH_EXTRA_FLAGS`](../yaml/README.md#git-fetch-extra-flags) allows you
[`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) allows you
to modify `git fetch` behavior by passing extra flags.
For example, if your project contains a large number of tags that your CI jobs don't rely on,
you could add [`--no-tags`](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---no-tags)
to the extra flags to make your fetches faster and more compact.
See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../yaml/README.md#git-fetch-extra-flags)
See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/README.md#git-fetch-extra-flags)
for more information.
## Fork-based workflow
......
doc/ci/pipelines/settings.md
View file @ 594147b3
......@@ -29,7 +29,7 @@ There are two options. Using:
- `git fetch`, which is faster as it re-uses the local working copy (falling
back to clone if it doesn't exist).
The default Git strategy can be overridden by the [GIT_STRATEGY variable](../yaml/README.md#git-strategy)
The default Git strategy can be overridden by the [GIT_STRATEGY variable](../runners/README.md#git-strategy)
in `.gitlab-ci.yml`.
## Git shallow clone
......
doc/ci/runners/README.md
View file @ 594147b3
......@@ -457,6 +457,342 @@ Example 2:
1. A job that has no tags defined is executed and run.
1. A second job that has a `docker` tag defined is stuck.
## Configure runner behavior with variables
You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior
globally or for individual jobs:
- [`GIT_STRATEGY`](#git-strategy)
- [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy)
- [`GIT_CHECKOUT`](#git-checkout)
- [`GIT_CLEAN_FLAGS`](#git-clean-flags)
- [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags)
- [`GIT_DEPTH`](#shallow-cloning) (shallow cloning)
- [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories)
You can also use variables to configure how many times a runner
[attempts certain stages of job execution](#job-stages-attempts).
### Git strategy
> - Introduced in GitLab 8.9 as an experimental feature.
> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+.
You can set the `GIT_STRATEGY` used for getting recent application code, either
globally or per-job in the [`variables`](../yaml/README.md#variables) section. If left
unspecified, the default from the project settings is used.
There are three possible values: `clone`, `fetch`, and `none`.
`clone` is the slowest option. It clones the repository from scratch for every
job, ensuring that the local working copy is always pristine.
```yaml
variables:
GIT_STRATEGY: clone
```
`fetch` is faster as it re-uses the local working copy (falling back to `clone`
if it does not exist). `git clean` is used to undo any changes made by the last
job, and `git fetch` is used to retrieve commits made since the last job ran.
```yaml
variables:
GIT_STRATEGY: fetch
```
`none` also re-uses the local working copy. However, it skips all Git operations,
including GitLab Runner's pre-clone script, if present.
It's useful for jobs that operate exclusively on artifacts, like a deployment job.
Git repository data may be present, but it's likely out-of-date. You should only
rely on files brought into the local working copy from cache or artifacts.
```yaml
variables:
GIT_STRATEGY: none
```
NOTE: **Note:**
`GIT_STRATEGY` is not supported for
[Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html),
but may be in the future. See the [support Git strategy with Kubernetes executor feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847)
for updates.
### Git submodule strategy
> Requires GitLab Runner v1.10+.
The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git
submodules are included when fetching the code before a build. You can set them
globally or per-job in the [`variables`](../yaml/README.md#variables) section.
There are three possible values: `none`, `normal`, and `recursive`:
- `none` means that submodules are not included when fetching the project
code. This is the default, which matches the pre-v1.10 behavior.
- `normal` means that only the top-level submodules are included. It's
equivalent to:
```shell
git submodule sync
git submodule update --init
```
- `recursive` means that all submodules (including submodules of submodules)
are included. This feature needs Git v1.8.1 and later. When using a
GitLab Runner with an executor not based on Docker, make sure the Git version
meets that requirement. It's equivalent to:
```shell
git submodule sync --recursive
git submodule update --init --recursive
```
For this feature to work correctly, the submodules must be configured
(in `.gitmodules`) with either:
- the HTTP(S) URL of a publicly-accessible repository, or
- a relative path to another repository on the same GitLab server. See the
[Git submodules](../git_submodules.md) documentation.
### Git checkout
> Introduced in GitLab Runner 9.3.
The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either
`clone` or `fetch` to specify whether a `git checkout` should be run. If not
specified, it defaults to true. You can set them globally or per-job in the
[`variables`](../yaml/README.md#variables) section.
If set to `false`, the runner:
- when doing `fetch` - updates the repository and leaves the working copy on
the current revision,
- when doing `clone` - clones the repository and leaves the working copy on the
default branch.
If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way.
The runner checks out the working copy of a revision related
to the CI pipeline:
```yaml
variables:
GIT_STRATEGY: clone
GIT_CHECKOUT: "false"
script:
- git checkout -B master origin/master
- git merge $CI_COMMIT_SHA
```
### Git clean flags
> Introduced in GitLab Runner 11.10
The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of
`git clean` after checking out the sources. You can set it globally or per-job in the
[`variables`](../yaml/README.md#variables) section.
`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean)
command.
`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified.
If `GIT_CLEAN_FLAGS` is:
- Not specified, `git clean` flags default to `-ffdx`.
- Given the value `none`, `git clean` is not executed.
For example:
```yaml
variables:
GIT_CLEAN_FLAGS: -ffdx -e cache/
script:
- ls -al cache/
```
### Git fetch extra flags
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1.
The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of
`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section.
`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified.
The default flags are:
- [GIT_DEPTH](#shallow-cloning).
- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec).
- A remote called `origin`.
If `GIT_FETCH_EXTRA_FLAGS` is:
- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags.
- Given the value `none`, `git fetch` is executed only with the default flags.
For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`:
```yaml
variables:
GIT_FETCH_EXTRA_FLAGS: --prune
script:
- ls -al cache/
```
The configuration above results in `git fetch` being called this way:
```shell
git fetch origin $REFSPECS --depth 50 --prune
```
Where `$REFSPECS` is a value provided to the runner internally by GitLab.
### Shallow cloning
> Introduced in GitLab 8.9 as an experimental feature.
You can specify the depth of fetching and cloning using `GIT_DEPTH`.
`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning.
It can be helpful for repositories with a large number of commits or old, large binaries. The value is
passed to `git fetch` and `git clone`.
In GitLab 12.0 and later, newly-created projects automatically have a
[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone).
If you use a depth of `1` and have a queue of jobs or retry
jobs, jobs may fail.
Git fetching and cloning is based on a ref, such as a branch name, so runners
can't clone a specific commit SHA. If multiple jobs are in the queue, or
you're retrying an old job, the commit to be tested must be within the
Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make
it impossible to run these old commits and `unresolved reference` is displayed in
job logs. You should then reconsider changing `GIT_DEPTH` to a higher value.
Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is
set since only part of the Git history is present.
To fetch or clone only the last 3 commits:
```yaml
variables:
GIT_DEPTH: "3"
```
You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section.
### Custom build directories
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10.
By default, GitLab Runner clones the repository in a unique subpath of the
`$CI_BUILDS_DIR` directory. However, your project might require the code in a
specific directory (Go projects, for example). In that case, you can specify
the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the
repository in:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name
test:
script:
- pwd
```
The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR`
is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
setting.
This can only be used when `custom_build_dir` is enabled in the
[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section).
This is the default configuration for the `docker` and `kubernetes` executors.
#### Handling concurrency
An executor that uses a concurrency greater than `1` might lead
to failures. Multiple jobs might be working on the same directory if the `builds_dir`
is shared between jobs.
The runner does not try to prevent this situation. It's up to the administrator
and developers to comply with the requirements of runner configuration.
To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner
exposes two additional variables that provide a unique `ID` of concurrency:
- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor.
- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project.
The most stable configuration that should work well in any scenario and on any executor
is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name
test:
script:
- pwd
```
The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH`
as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH
test:
script:
- pwd
```
#### Nested paths
The value of `GIT_CLONE_PATH` is expanded once and nesting variables
within is not supported.
For example, you define both the variables below in your
`.gitlab-ci.yml` file:
```yaml
variables:
GOPATH: $CI_BUILDS_DIR/go
GIT_CLONE_PATH: $GOPATH/src/namespace/project
```
The value of `GIT_CLONE_PATH` is expanded once into
`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure
because `$CI_BUILDS_DIR` is not expanded.
### Job stages attempts
> Introduced in GitLab, it requires GitLab Runner v1.9+.
You can set the number of attempts that the running job tries to execute
the following stages:
| Variable | Description |
|---------------------------------|--------------------------------------------------------|
| `ARTIFACT_DOWNLOAD_ATTEMPTS` | Number of attempts to download artifacts running a job |
| `EXECUTOR_JOB_SECTION_ATTEMPTS` | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). |
| `GET_SOURCES_ATTEMPTS` | Number of attempts to fetch sources running a job |
| `RESTORE_CACHE_ATTEMPTS` | Number of attempts to restore the cache running a job |
The default is one single attempt.
Example:
```yaml
variables:
GET_SOURCES_ATTEMPTS: 3
```
You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section.
## System calls not available on GitLab.com shared runners
GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library.
doc/ci/yaml/README.md
View file @ 594147b3
......@@ -2802,6 +2802,32 @@ URI-encoded `%2F`. A value made only of dots (`.`, `%2E`) is also forbidden.
You can specify a [fallback cache key](#fallback-cache-key) to use if the specified `cache:key` is not found.
#### Fallback cache key
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1534) in GitLab Runner 13.4.
You can use the `$CI_COMMIT_REF_SLUG` [variable](#variables) to specify your [`cache:key`](#cachekey).
For example, if your `$CI_COMMIT_REF_SLUG` is `test` you can set a job
to download cache that's tagged with `test`.
If a cache with this tag is not found, you can use `CACHE_FALLBACK_KEY` to
specify a cache to use when none exists.
For example:
```yaml
variables:
CACHE_FALLBACK_KEY: fallback-key
cache:
key: "$CI_COMMIT_REF_SLUG"
paths:
- binaries/
```
In this example, if the `$CI_COMMIT_REF_SLUG` is not found, the job uses the key defined
by the `CACHE_FALLBACK_KEY` variable.
##### `cache:key:files`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18986) in GitLab v12.5.
......@@ -3485,7 +3511,7 @@ Possible values for `when` are:
- `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner.
- `data_integrity_failure`: Retry if there was a structural integrity problem detected.
You can specify the number of [retry attempts for certain stages of job execution](#job-stages-attempts) using variables.
You can specify the number of [retry attempts for certain stages of job execution](../runners/README.md#job-stages-attempts) using variables.
### `timeout`
......@@ -4147,8 +4173,8 @@ Read more on [GitLab Pages user documentation](../../user/project/pages/index.md
> Introduced in GitLab Runner v0.5.0.
Variables are configurable values that are passed to jobs. They can be set
globally and per-job.
[CI/CD variables](../variables/README.md) are configurable values that are passed to jobs.
They can be set globally and per-job.
There are two types of variables.
......@@ -4165,372 +4191,49 @@ Variables are meant for non-sensitive project configuration, for example:
```yaml
variables:
DATABASE_URL: "postgres://postgres@postgres/my_database"
DEPLOY_SITE: "https://example.com/"
deploy_job:
stage: deploy
script:
- deploy-script --url $DEPLOY_SITE --path "/"
deploy_review_job:
stage: deploy
variables:
REVIEW_PATH: "/review"
script:
- deploy-review-script --url $DEPLOY_SITE --path $REVIEW_PATH
```
You can use integers and strings for the variable's name and value.
You cannot use floats.
You can use only integers and strings for the variable's name and value.
If you define a variable at the top level of the `gitlab-ci.yml` file, it is global,
meaning it applies to all jobs.
If you define a variable within a job, it's available to that job only.
meaning it applies to all jobs. If you define a variable within a job, it's available
to that job only.
If a variable of the same name is defined globally and for a specific job, the
[job-specific variable is used](../variables/README.md#priority-of-environment-variables).
All YAML-defined variables are also set to any linked
[service containers](../docker/using_docker_images.md#what-is-a-service).
[YAML anchors for variables](#yaml-anchors-for-variables) are available.
Learn more about [variables and their priority](../variables/README.md).
### Git strategy
> - Introduced in GitLab 8.9 as an experimental feature.
> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+.
You can set the `GIT_STRATEGY` used for getting recent application code, either
globally or per-job in the [`variables`](#variables) section. If left
unspecified, the default from the project settings is used.
There are three possible values: `clone`, `fetch`, and `none`.
`clone` is the slowest option. It clones the repository from scratch for every
job, ensuring that the local working copy is always pristine.
```yaml
variables:
GIT_STRATEGY: clone
```
`fetch` is faster as it re-uses the local working copy (falling back to `clone`
if it does not exist). `git clean` is used to undo any changes made by the last
job, and `git fetch` is used to retrieve commits made since the last job ran.
```yaml
variables:
GIT_STRATEGY: fetch
```
`none` also re-uses the local working copy. However, it skips all Git operations,
including GitLab Runner's pre-clone script, if present.
It's useful for jobs that operate exclusively on artifacts, like a deployment job.
Git repository data may be present, but it's likely out-of-date. You should only
rely on files brought into the local working copy from cache or artifacts.
```yaml
variables:
GIT_STRATEGY: none
```
NOTE: **Note:**
`GIT_STRATEGY` is not supported for
[Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html),
but may be in the future. See the [support Git strategy with Kubernetes executor feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847)
for updates.
### Git submodule strategy
> Requires GitLab Runner v1.10+.
The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git
submodules are included when fetching the code before a build. You can set them
globally or per-job in the [`variables`](#variables) section.
There are three possible values: `none`, `normal`, and `recursive`:
- `none` means that submodules are not included when fetching the project
code. This is the default, which matches the pre-v1.10 behavior.
- `normal` means that only the top-level submodules are included. It's
equivalent to:
```shell
git submodule sync
git submodule update --init
```
- `recursive` means that all submodules (including submodules of submodules)
are included. This feature needs Git v1.8.1 and later. When using a
GitLab Runner with an executor not based on Docker, make sure the Git version
meets that requirement. It's equivalent to:
```shell
git submodule sync --recursive
git submodule update --init --recursive
```
For this feature to work correctly, the submodules must be configured
(in `.gitmodules`) with either:
- the HTTP(S) URL of a publicly-accessible repository, or
- a relative path to another repository on the same GitLab server. See the
[Git submodules](../git_submodules.md) documentation.
### Git checkout
> Introduced in GitLab Runner 9.3.
The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either
`clone` or `fetch` to specify whether a `git checkout` should be run. If not
specified, it defaults to true. You can set them globally or per-job in the
[`variables`](#variables) section.
If set to `false`, the runner:
- when doing `fetch` - updates the repository and leaves the working copy on
the current revision,
- when doing `clone` - clones the repository and leaves the working copy on the
default branch.
If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way.
The runner checks out the working copy of a revision related
to the CI pipeline:
```yaml
variables:
GIT_STRATEGY: clone
GIT_CHECKOUT: "false"
script:
- git checkout -B master origin/master
- git merge $CI_COMMIT_SHA
```
### Git clean flags
> Introduced in GitLab Runner 11.10
The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of
`git clean` after checking out the sources. You can set it globally or per-job in the
[`variables`](#variables) section.
`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean)
command.
`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified.
If `GIT_CLEAN_FLAGS` is:
- Not specified, `git clean` flags default to `-ffdx`.
- Given the value `none`, `git clean` is not executed.
For example:
```yaml
variables:
GIT_CLEAN_FLAGS: -ffdx -e cache/
script:
- ls -al cache/
```
### Git fetch extra flags
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1.
The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of
`git fetch`. You can set it globally or per-job in the [`variables`](#variables) section.
`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified.
The default flags are:
[Docker service containers](../docker/using_docker_images.md#what-is-a-service).
- [GIT_DEPTH](#shallow-cloning).
- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec).
- A remote called `origin`.
You can use [YAML anchors for variables](#yaml-anchors-for-variables).
If `GIT_FETCH_EXTRA_FLAGS` is:
### Configure runner behavior with variables
- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags.
- Given the value `none`, `git fetch` is executed only with the default flags.
You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior:
For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`:
```yaml
variables:
GIT_FETCH_EXTRA_FLAGS: --prune
script:
- ls -al cache/
```
The configuration above results in `git fetch` being called this way:
```shell
git fetch origin $REFSPECS --depth 50 --prune
```
Where `$REFSPECS` is a value provided to the runner internally by GitLab.
### Job stages attempts
> Introduced in GitLab, it requires GitLab Runner v1.9+.
You can set the number of attempts that the running job tries to execute
the following stages:
| Variable | Description |
|-----------------------------------|--------------------------------------------------------|
| **ARTIFACT_DOWNLOAD_ATTEMPTS** | Number of attempts to download artifacts running a job |
| **EXECUTOR_JOB_SECTION_ATTEMPTS** | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). |
| **GET_SOURCES_ATTEMPTS** | Number of attempts to fetch sources running a job |
| **RESTORE_CACHE_ATTEMPTS** | Number of attempts to restore the cache running a job |
The default is one single attempt.
Example:
```yaml
variables:
GET_SOURCES_ATTEMPTS: 3
```
You can set them globally or per-job in the [`variables`](#variables) section.
### Fallback cache key
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1534) in GitLab Runner 13.4.
You can use the `$CI_COMMIT_REF_SLUG` variable to specify your [`cache:key`](#cachekey).
For example, if your `$CI_COMMIT_REF_SLUG` is `test` you can set a job
to download cache that's tagged with `test`.
If a cache with this tag is not found, you can use `CACHE_FALLBACK_KEY` to
specify a cache to use when none exists.
For example:
```yaml
variables:
CACHE_FALLBACK_KEY: fallback-key
cache:
key: "$CI_COMMIT_REF_SLUG"
paths:
- binaries/
```
In this example, if the `$CI_COMMIT_REF_SLUG` is not found, the job uses the key defined
by the `CACHE_FALLBACK_KEY` variable.
### Shallow cloning
> Introduced in GitLab 8.9 as an experimental feature.
You can specify the depth of fetching and cloning using `GIT_DEPTH`.
`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning.
It can be helpful for repositories with a large number of commits or old, large binaries. The value is
passed to `git fetch` and `git clone`.
In GitLab 12.0 and later, newly-created projects automatically have a
[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone).
If you use a depth of `1` and have a queue of jobs or retry
jobs, jobs may fail.
Git fetching and cloning is based on a ref, such as a branch name, so runners
can't clone a specific commit SHA. If multiple jobs are in the queue, or
you're retrying an old job, the commit to be tested must be within the
Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make
it impossible to run these old commits and `unresolved reference` is displayed in
job logs. You should then reconsider changing `GIT_DEPTH` to a higher value.
Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is
set since only part of the Git history is present.
To fetch or clone only the last 3 commits:
```yaml
variables:
GIT_DEPTH: "3"
```
You can set it globally or per-job in the [`variables`](#variables) section.
### Custom build directories
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10.
By default, GitLab Runner clones the repository in a unique subpath of the
`$CI_BUILDS_DIR` directory. However, your project might require the code in a
specific directory (Go projects, for example). In that case, you can specify
the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the
repository in:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name
test:
script:
- pwd
```
The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR`
is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
setting.
This can only be used when `custom_build_dir` is enabled in the
[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section).
This is the default configuration for the `docker` and `kubernetes` executors.
#### Handling concurrency
An executor that uses a concurrency greater than `1` might lead
to failures. Multiple jobs might be working on the same directory if the `builds_dir`
is shared between jobs.
The runner does not try to prevent this situation. It's up to the administrator
and developers to comply with the requirements of runner configuration.
To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner
exposes two additional variables that provide a unique `ID` of concurrency:
- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor.
- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project.
The most stable configuration that should work well in any scenario and on any executor
is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name
test:
script:
- pwd
```
The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH`
as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH
test:
script:
- pwd
```
#### Nested paths
The value of `GIT_CLONE_PATH` is expanded once and nesting variables
within is not supported.
For example, you define both the variables below in your
`.gitlab-ci.yml` file:
```yaml
variables:
GOPATH: $CI_BUILDS_DIR/go
GIT_CLONE_PATH: $GOPATH/src/namespace/project
```
- [`GIT_STRATEGY`](../runners/README.md#git-strategy)
- [`GIT_SUBMODULE_STRATEGY`](../runners/README.md#git-submodule-strategy)
- [`GIT_CHECKOUT`](../runners/README.md#git-checkout)
- [`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags)
- [`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags)
- [`GIT_DEPTH`](../runners/README.md#shallow-cloning) (shallow cloning)
- [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) (custom build directories)
The value of `GIT_CLONE_PATH` is expanded once into
`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure
because `$CI_BUILDS_DIR` is not expanded.
You can also use variables to configure how many times a runner
[attempts certain stages of job execution](../runners/README.md#job-stages-attempts).
## Special YAML features
......
doc/user/application_security/container_scanning/index.md
View file @ 594147b3
......@@ -415,7 +415,7 @@ automatically generates.
To enable remediation support, the scanning tool _must_ have access to the `Dockerfile` specified by
the [`DOCKERFILE_PATH`](#available-variables) environment variable. To ensure that the scanning tool
has access to this
file, it's necessary to set [`GIT_STRATEGY: fetch`](../../../ci/yaml/README.md#git-strategy) in
file, it's necessary to set [`GIT_STRATEGY: fetch`](../../../ci/runners/README.md#git-strategy) in
your `.gitlab-ci.yml` file by following the instructions described in this document's
[overriding the container scanning template](#overriding-the-container-scanning-template) section.
......
doc/user/application_security/dast/index.md
View file @ 594147b3
......@@ -603,7 +603,7 @@ variables:
### Cloning the project's repository
The DAST job does not require the project's repository to be present when running, so by default
[`GIT_STRATEGY`](../../../ci/yaml/README.md#git-strategy) is set to `none`.
[`GIT_STRATEGY`](../../../ci/runners/README.md#git-strategy) is set to `none`.
### Debugging DAST jobs
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7