Commit 03bbd847 authored by Marcia Ramos's avatar Marcia Ramos

Merge branch 'docs/ci-caching' into 'master'

Add docs on CI caching

Closes #32515, #18945, and gitlab-runner#1621

See merge request gitlab-org/gitlab-ce!12183
parents 9027d023 d24ee39e
...@@ -65,7 +65,8 @@ learn how to leverage its potential even more. ...@@ -65,7 +65,8 @@ learn how to leverage its potential even more.
environments and use them for different purposes like testing, building and environments and use them for different purposes like testing, building and
deploying deploying
- [Job artifacts](../user/project/pipelines/job_artifacts.md) - [Job artifacts](../user/project/pipelines/job_artifacts.md)
- [Git submodules](git_submodules.md): How to run your CI jobs when Git - [Caching dependencies](caching/index.md)
- [Git submodules](git_submodules.md) - How to run your CI jobs when Git
submodules are involved submodules are involved
- [Use SSH keys in your build environment](ssh_keys/README.md) - [Use SSH keys in your build environment](ssh_keys/README.md)
- [Trigger pipelines through the GitLab API](triggers/README.md) - [Trigger pipelines through the GitLab API](triggers/README.md)
......
This diff is collapsed.
...@@ -146,24 +146,7 @@ To protect/unprotect Runners: ...@@ -146,24 +146,7 @@ To protect/unprotect Runners:
## Manually clearing the Runners cache ## Manually clearing the Runners cache
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41249) in GitLab 10.4. Read [clearing the cache](../caching/index.md#clearing-the-cache).
GitLab Runners use [cache](../yaml/README.md#cache) to speed up the execution
of your jobs by reusing existing data. This however, can sometimes lead to an
inconsistent behavior.
To start with a fresh copy of the cache, you can easily do it via GitLab's UI:
1. Navigate to your project's **CI/CD > Pipelines** page.
1. Click on the **Clear Runner caches** to clean up the cache.
1. On the next push, your CI/CD job will use a new cache.
That way, you don't have to change the [cache key](../yaml/README.md#cache-key)
in your `.gitlab-ci.yml`.
Behind the scenes, this works by increasing a counter in the database, and the
value of that counter is used to create the key for the cache. After a push, a
new key is generated and the old cache is not valid anymore.
## How shared Runners pick jobs ## How shared Runners pick jobs
...@@ -227,15 +210,16 @@ that it may encounter on the projects it's shared over. This would be ...@@ -227,15 +210,16 @@ that it may encounter on the projects it's shared over. This would be
problematic for large amounts of projects, if it wasn't for tags. problematic for large amounts of projects, if it wasn't for tags.
By tagging a Runner for the types of jobs it can handle, you can make sure By tagging a Runner for the types of jobs it can handle, you can make sure
shared Runners will only run the jobs they are equipped to run. shared Runners will [only run the jobs they are equipped to run](../yaml/README.md#tags).
For instance, at GitLab we have Runners tagged with "rails" if they contain For instance, at GitLab we have Runners tagged with "rails" if they contain
the appropriate dependencies to run Rails test suites. the appropriate dependencies to run Rails test suites.
### Preventing Runners with tags from picking jobs without tags ### Preventing Runners with tags from picking jobs without tags
You can configure a Runner to prevent it from picking jobs with tags when You can configure a Runner to prevent it from picking
the Runner does not have tags assigned. This setting can be enabled the first [jobs with tags](../yaml/README.md#tags) when the Runner does not have tags
assigned. This setting can be enabled the first
time you [register a Runner][register] and can be changed afterwards under time you [register a Runner][register] and can be changed afterwards under
each Runner's settings. each Runner's settings.
......
...@@ -674,6 +674,10 @@ as Review Apps. You can see a simple example using Review Apps at ...@@ -674,6 +674,10 @@ as Review Apps. You can see a simple example using Review Apps at
by default. by default.
- From GitLab 9.2, caches are restored before [artifacts](#artifacts). - From GitLab 9.2, caches are restored before [artifacts](#artifacts).
TIP: **Learn more:**
Read how caching works and find out some good practices in the
[caching dependencies documentation](../caching/index.md).
`cache` is used to specify a list of files and directories which should be `cache` is used to specify a list of files and directories which should be
cached between jobs. You can only use paths that are within the project cached between jobs. You can only use paths that are within the project
workspace. workspace.
...@@ -681,35 +685,20 @@ workspace. ...@@ -681,35 +685,20 @@ workspace.
If `cache` is defined outside the scope of jobs, it means it is set If `cache` is defined outside the scope of jobs, it means it is set
globally and all jobs will use that definition. globally and all jobs will use that definition.
Cache all files in `binaries` and `.config`: ### `cache:paths`
```yaml Use the `paths` directive to choose which files or directories will be cached.
rspec: Wildcards can be used as well.
script: test
cache:
paths:
- binaries/
- .config
```
Cache all Git untracked files: Cache all files in `binaries` that end in `.apk` and the `.config` file:
```yaml ```yaml
rspec: rspec:
script: test script: test
cache: cache:
untracked: true
```
Cache all Git untracked files and files in `binaries`:
```yaml
rspec:
script: test
cache:
untracked: true
paths: paths:
- binaries/ - binaries/*.apk
- .config
``` ```
Locally defined cache overrides globally defined options. The following `rspec` Locally defined cache overrides globally defined options. The following `rspec`
...@@ -723,33 +712,26 @@ cache: ...@@ -723,33 +712,26 @@ cache:
rspec: rspec:
script: test script: test
cache: cache:
key: rspec
paths: paths:
- binaries/ - binaries/
``` ```
Note that since cache is shared between jobs, if you're using different
paths for different jobs, you should also set a different **cache:key**
otherwise cache content can be overwritten.
NOTE: **Note:**
The cache is provided on a best-effort basis, so don't expect that the cache
will be always present.
### `cache:key` ### `cache:key`
> Introduced in GitLab Runner v1.0.0. > Introduced in GitLab Runner v1.0.0.
The `key` directive allows you to define the affinity of caching Since the cache is shared between jobs, if you're using different
between jobs, allowing to have a single cache for all jobs, paths for different jobs, you should also set a different `cache:key`
cache per-job, cache per-branch or any other way that fits your needs. otherwise cache content can be overwritten.
This way, you can fine tune caching, allowing you to cache data between The `key` directive allows you to define the affinity of caching between jobs,
different jobs or even different branches. allowing to have a single cache for all jobs, cache per-job, cache per-branch
or any other way that fits your workflow. This way, you can fine tune caching,
allowing you to cache data between different jobs or even different branches.
The `cache:key` variable can use any of the The `cache:key` variable can use any of the
[predefined variables](../variables/README.md), and the default key, if not set, [predefined variables](../variables/README.md), and the default key, if not set,
is set as `$CI_JOB_NAME-$CI_COMMIT_REF_NAME` which translates as "per-job and is `$CI_JOB_NAME-$CI_COMMIT_REF_NAME` which translates as "per-job and
per-branch". It is the default across the project, therefore everything is per-branch". It is the default across the project, therefore everything is
shared between pipelines and jobs running on the same branch by default. shared between pipelines and jobs running on the same branch by default.
...@@ -757,56 +739,56 @@ NOTE: **Note:** ...@@ -757,56 +739,56 @@ NOTE: **Note:**
The `cache:key` variable cannot contain the `/` character, or the equivalent The `cache:key` variable cannot contain the `/` character, or the equivalent
URI-encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden. URI-encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden.
**Example configurations** For example, to enable per-branch caching:
To enable per-job caching:
```yaml
cache:
key: "$CI_JOB_NAME"
untracked: true
```
To enable per-branch caching:
```yaml ```yaml
cache: cache:
key: "$CI_COMMIT_REF_SLUG" key: "$CI_COMMIT_REF_SLUG"
untracked: true paths:
- binaries/
``` ```
To enable per-job and per-branch caching: If you use **Windows Batch** to run your shell scripts you need to replace
`$` with `%`:
```yaml ```yaml
cache: cache:
key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%"
untracked: true paths:
- binaries/
``` ```
To enable per-branch and per-stage caching: If you use **Windows PowerShell** to run your shell scripts you need to replace
`$` with `$env:`:
```yaml ```yaml
cache: cache:
key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG"
untracked: true paths:
- binaries/
``` ```
If you use **Windows Batch** to run your shell scripts you need to replace ### `cache:untracked`
`$` with `%`:
Set `untracked: true` to cache all files that are untracked in your Git
repository:
```yaml ```yaml
cache: rspec:
key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%" script: test
cache:
untracked: true untracked: true
``` ```
If you use **Windows PowerShell** to run your shell scripts you need to replace Cache all Git untracked files and files in `binaries`:
`$` with `$env:`:
```yaml ```yaml
cache: rspec:
key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG" script: test
cache:
untracked: true untracked: true
paths:
- binaries/
``` ```
### `cache:policy` ### `cache:policy`
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment