Skip to content

G
gitlab-ce
Commit 7d025a66 authored by Marcel Amirault's avatar Marcel Amirault Committed by Evan Read
Browse files
Options

Rename job traces to job logs 

Already in use as job logs in various places,
finish renaming job traces to job logs in docs
parent a06b8041
Hide whitespace changes
Inline Side-by-side
Showing with 196 additions and 189 deletions
doc/administration/gitaly/index.md
View file @ 7d025a66
......@@ -549,7 +549,7 @@ a few things that you need to do:
1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md)
to eliminate the need for a shared authorized_keys file.
1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage)
including [live tracing](../job_traces.md#new-live-trace-architecture).
including [incremental logging](../job_logs.md#new-incremental-logging-architecture).
1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only).
......
doc/administration/index.md
View file @ 7d025a66
......@@ -154,7 +154,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#site-wide-admin-setting): Enable or disable GitLab CI/CD for your instance.
- [GitLab CI/CD admin settings](../user/admin_area/settings/continuous_integration.md): Enable or disable Auto DevOps site-wide and define the artifacts' max size and expiration time.
- [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully).
- [Job traces](job_traces.md): Information about the job traces (logs).
- [Job logs](job_logs.md): Information about the job logs.
- [Register Shared and specific Runners](../ci/runners/README.md#registering-a-shared-runner): Learn how to register and configure Shared and specific Runners to your own instance.
- [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota-starter-only): Limit the usage of pipeline minutes for Shared Runners. **(STARTER ONLY)**
- [Enable/disable Auto DevOps](../topics/autodevops/index.md#enablingdisabling-auto-devops): Enable or disable Auto DevOps for your instance.
......
doc/administration/job_artifacts.md
View file @ 7d025a66
......@@ -90,7 +90,7 @@ This configuration relies on valid AWS credentials to be configured already.
Use an object storage option like AWS S3 to store job artifacts.
DANGER: **Danger:**
If you're enabling S3 in [GitLab HA](high_availability/README.md), you will need to have an [NFS mount set up for CI traces and artifacts](high_availability/nfs.md#a-single-nfs-mount) or enable [live tracing](job_traces.md#new-live-trace-architecture). If these settings are not set, you will risk job traces disappearing or not being saved.
If you're enabling S3 in [GitLab HA](high_availability/README.md), you will need to have an [NFS mount set up for CI logs and artifacts](high_availability/nfs.md#a-single-nfs-mount) or enable [incremental logging](job_logs.md#new-incremental-logging-architecture). If these settings are not set, you will risk job logs disappearing or not being saved.
#### Object Storage Settings
......
doc/administration/job_logs.md 0 → 100644
View file @ 7d025a66
# Job logs
> [Renamed from Job Traces to Job logs](https://gitlab.com/gitlab-org/gitlab/issues/29121) in 12.4.
Job logs (traces) are sent by GitLab Runner while it's processing a job. You can see
logs in job pages, pipelines, email notifications, etc.
## Data flow
In general, there are two states for job logs: `log` and `archived log`.
In the following table you can see the phases a log goes through:
| Phase | State | Condition | Data flow | Stored path |
| -------------- | ------------ | ----------------------- | -----------------------------------------| ----------- |
| 1: patching | log | When a job is running | GitLab Runner => Unicorn => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` |
| 2: overwriting | log | When a job is finished | GitLab Runner => Unicorn => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` |
| 3: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` |
| 4: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` |
The `ROOT_PATH` varies per environment. For Omnibus GitLab it
would be `/var/opt/gitlab`, and for installations from source
it would be `/home/git/gitlab`.
## Changing the job logs local location
To change the location where the job logs will be stored, follow the steps below.
**In Omnibus installations:**
1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line:
```ruby
gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds'
```
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
---
**In installations from source:**
1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
```yaml
gitlab_ci:
# The location where build logs are stored (default: builds/).
# Relative paths are relative to Rails.root.
builds_path: path/to/builds/
```
1. Save the file and [restart GitLab][] for the changes to take effect.
[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
## Uploading logs to object storage
Archived logs are considered as [job artifacts](job_artifacts.md).
Therefore, when you [set up the object storage integration](job_artifacts.md#object-storage-settings),
job logs are automatically migrated to it along with the other job artifacts.
See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process.
## How to remove job logs
There isn't a way to automatically expire old job logs, but it's safe to remove
them if they're taking up too much space. If you remove the logs manually, the
job output in the UI will be empty.
## New incremental logging architecture
> [Introduced][ce-18169] in GitLab 10.4.
> [Announced as General availability][ce-46097] in GitLab 11.0.
NOTE: **Note:**
This feature is off by default. See below for how to [enable or disable](#enabling-incremental-logging) it.
By combining the process with object storage settings, we can completely bypass
the local file storage. This is a useful option if GitLab is installed as
cloud-native, for example on Kubernetes.
The data flow is the same as described in the [data flow section](#data-flow)
with one change: _the stored path of the first two phases is different_. This incremental
log architecture stores chunks of logs in Redis and a persistent store (object storage or database) instead of
file storage. Redis is used as first-class storage, and it stores up-to 128KB
of data. Once the full chunk is sent, it is flushed to a persistent store, either object storage(temporary directory) or database.
After a while, the data in Redis and a persitent store will be archived to [object storage](#uploading-logs-to-object-storage).
The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`.
Here is the detailed data flow:
1. GitLab Runner picks a job from GitLab
1. GitLab Runner sends a piece of log to GitLab
1. GitLab appends the data to Redis
1. Once the data in Redis reach 128KB, the data is flushed to a persistent store (object storage or the database).
1. The above steps are repeated until the job is finished.
1. Once the job is finished, GitLab schedules a Sidekiq worker to archive the log.
1. The Sidekiq worker archives the log to object storage and cleans up the log
in Redis and a persistent store (object storage or the database).
### Enabling incremental logging
The following commands are to be issued in a Rails console:
```sh
# Omnibus GitLab
gitlab-rails console
# Installation from source
cd /home/git/gitlab
sudo -u git -H bin/rails console RAILS_ENV=production
```
**To check if incremental logging (trace) is enabled:**
```ruby
Feature.enabled?('ci_enable_live_trace')
```
**To enable incremental logging (trace):**
```ruby
Feature.enable('ci_enable_live_trace')
```
NOTE: **Note:**
The transition period will be handled gracefully. Upcoming logs will be
generated with the incremental architecture, and on-going logs will stay with the
legacy architecture, which means that on-going logs won't be forcibly
re-generated with the incremental architecture.
**To disable incremental logging (trace):**
```ruby
Feature.disable('ci_enable_live_trace')
```
NOTE: **Note:**
The transition period will be handled gracefully. Upcoming logs will be generated
with the legacy architecture, and on-going incremental logs will stay with the incremental
architecture, which means that on-going incremental logs won't be forcibly re-generated
with the legacy architecture.
### Potential implications
In some cases, having data stored on Redis could incur data loss:
1. **Case 1: When all data in Redis are accidentally flushed**
- On going incremental logs could be recovered by re-sending logs (this is
supported by all versions of the GitLab Runner).
- Finished jobs which have not archived incremental logs will lose the last part
(~128KB) of log data.
1. **Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that
prevents archiving process, Sidekiq inconsistency, etc.)**
- Currently all log data in Redis will be deleted after one week. If the
Sidekiq workers can't finish by the expiry date, the part of log data will be lost.
Another issue that might arise is that it could consume all memory on the Redis
instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed.
Also, it could pressure the database replication lag. `INSERT`s are generated to
indicate that we have log chunk. `UPDATE`s with 128KB of data is issued once we
receive multiple chunks.
[ce-18169]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/18169
[ce-21193]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/21193
[ce-46097]: https://gitlab.com/gitlab-org/gitlab-foss/issues/46097
doc/administration/job_traces.md
View file @ 7d025a66
# Job traces (logs)
Job traces are sent by GitLab Runner while it's processing a job. You can see
traces in job pages, pipelines, email notifications, etc.
## Data flow
In general, there are two states in job traces: "live trace" and "archived trace".
In the following table you can see the phases a trace goes through.
| Phase | State | Condition | Data flow | Stored path |
| ----- | ----- | --------- | --------- | ----------- |
| 1: patching | Live trace | When a job is running | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
| 2: overwriting | Live trace | When a job is finished | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`|
| 4: uploading | Archived trace | After a trace is archived | Sidekiq moves archived trace to [object storage](#uploading-traces-to-object-storage) (if configured) |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`|
The `ROOT_PATH` varies per your environment. For Omnibus GitLab it
would be `/var/opt/gitlab`, whereas for installations from source
it would be `/home/git/gitlab`.
## Changing the job traces local location
To change the location where the job logs will be stored, follow the steps below.
**In Omnibus installations:**
1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line:
```ruby
gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds'
```
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
---
redirect_to: 'job_logs.md'
---
**In installations from source:**
1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
```yaml
gitlab_ci:
# The location where build traces are stored (default: builds/).
# Relative paths are relative to Rails.root.
builds_path: path/to/builds/
```
1. Save the file and [restart GitLab][] for the changes to take effect.
[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
## Uploading traces to object storage
Archived traces are considered as [job artifacts](job_artifacts.md).
Therefore, when you [set up the object storage integration](job_artifacts.md#object-storage-settings),
job traces are automatically migrated to it along with the other job artifacts.
See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process.
## How to remove job traces
There isn't a way to automatically expire old job logs, but it's safe to remove
them if they're taking up too much space. If you remove the logs manually, the
job output in the UI will be empty.
## New live trace architecture
> [Introduced][ce-18169] in GitLab 10.4.
> [Announced as General availability][ce-46097] in GitLab 11.0.
NOTE: **Note:**
This feature is off by default. Check below how to [enable/disable](#enabling-live-trace) it.
By combining the process with object storage settings, we can completely bypass
the local file storage. This is a useful option if GitLab is installed as
cloud-native, for example on Kubernetes.
The data flow is the same as described in the [data flow section](#data-flow)
with one change: _the stored path of the first two phases is different_. This new live
trace architecture stores chunks of traces in Redis and a persistent store (object storage or database) instead of
file storage. Redis is used as first-class storage, and it stores up-to 128KB
of data. Once the full chunk is sent, it is flushed a persistent store, either object storage(temporary directory) or database.
After a while, the data in Redis and a persitent store will be archived to [object storage](#uploading-traces-to-object-storage).
The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`.
Here is the detailed data flow:
1. GitLab Runner picks a job from GitLab
1. GitLab Runner sends a piece of trace to GitLab
1. GitLab appends the data to Redis
1. Once the data in Redis reach 128KB, the data is flushed to a persistent store (object storage or the database).
1. The above steps are repeated until the job is finished.
1. Once the job is finished, GitLab schedules a Sidekiq worker to archive the trace.
1. The Sidekiq worker archives the trace to object storage and cleans up the trace
in Redis and a persistent store (object storage or the database).
### Enabling live trace
The following commands are to be issues in a Rails console:
```sh
# Omnibus GitLab
gitlab-rails console
# Installation from source
cd /home/git/gitlab
sudo -u git -H bin/rails console RAILS_ENV=production
```
**To check if live trace is enabled:**
```ruby
Feature.enabled?('ci_enable_live_trace')
```
**To enable live trace:**
```ruby
Feature.enable('ci_enable_live_trace')
```
NOTE: **Note:**
The transition period will be handled gracefully. Upcoming traces will be
generated with the new architecture, and on-going live traces will stay with the
legacy architecture, which means that on-going live traces won't be forcibly
re-generated with the new architecture.
**To disable live trace:**
```ruby
Feature.disable('ci_enable_live_trace')
```
NOTE: **Note:**
The transition period will be handled gracefully. Upcoming traces will be generated
with the legacy architecture, and on-going live traces will stay with the new
architecture, which means that on-going live traces won't be forcibly re-generated
with the legacy architecture.
### Potential implications
In some cases, having data stored on Redis could incur data loss:
1. **Case 1: When all data in Redis are accidentally flushed**
- On going live traces could be recovered by re-sending traces (this is
supported by all versions of the GitLab Runner).
- Finished jobs which have not archived live traces will lose the last part
(~128KB) of trace data.
1. **Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that
prevents archiving process, Sidekiq inconsistency, etc.)**
- Currently all trace data in Redis will be deleted after one week. If the
Sidekiq workers can't finish by the expiry date, the part of trace data will be lost.
Another issue that might arise is that it could consume all memory on the Redis
instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed.
Also, it could pressure the database replication lag. `INSERT`s are generated to
indicate that we have trace chunk. `UPDATE`s with 128KB of data is issued once we
receive multiple chunks.
[ce-18169]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/18169
[ce-21193]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/21193
[ce-46097]: https://gitlab.com/gitlab-org/gitlab-foss/issues/46097
This document was moved to [another location](job_logs.md).
doc/api/jobs.md
View file @ 7d025a66
......@@ -537,9 +537,9 @@ Possible response status codes:
| 400 | Invalid path provided |
| 404 | Build not found or no file/artifacts |
## Get a trace file
## Get a log file
Get a trace of a specific job of a project
Get a log (trace) of a specific job of a project:
```
GET /projects/:id/jobs/:job_id/trace
......@@ -556,10 +556,10 @@ curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.ex
Possible response status codes:
| Status | Description |
|-----------|-----------------------------------|
| 200 | Serves the trace file |
| 404 | Build not found or no trace file |
| Status | Description |
|-----------|-------------------------------|
| 200 | Serves the log file |
| 404 | Job not found or no log file |
## Cancel a job
......@@ -661,7 +661,7 @@ Example of response
## Erase a job
Erase a single job of a project (remove job artifacts and a job trace)
Erase a single job of a project (remove job artifacts and a job log)
```
POST /projects/:id/jobs/:job_id/erase
......
doc/ci/pipelines.md
View file @ 7d025a66
......@@ -283,11 +283,11 @@ You can also access pipelines for a merge request by navigating to its **Pipelin
When you access a pipeline, you can see the related jobs for that pipeline.
Clicking on an individual job will show you its job trace, and allow you to:
Clicking on an individual job will show you its job log, and allow you to:
- Cancel the job.
- Retry the job.
- Erase the job trace.
- Erase the job log.
### Seeing the failure reason for jobs
......
doc/ci/ssh_keys/README.md
View file @ 7d025a66
......@@ -35,8 +35,8 @@ with any type of [executor](https://docs.gitlab.com/runner/executors/)
if you are accessing a private GitLab repository.
NOTE: **Note:**
The private key will not be displayed in the job trace, unless you enable
[debug tracing](../variables/README.md#debug-tracing). You might also want to
The private key will not be displayed in the job log, unless you enable
[debug logging](../variables/README.md#debug-logging). You might also want to
check the [visibility of your pipelines](../../user/project/pipelines/settings.md#visibility-of-pipelines).
## SSH keys when using the Docker executor
......
doc/ci/variables/README.md
View file @ 7d025a66
......@@ -568,7 +568,7 @@ Below you can find supported syntax reference:
Precedence of operators follows standard Ruby 2.5 operation
[precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html).
## Debug tracing
## Debug logging
> Introduced in GitLab Runner 1.7.
......@@ -576,24 +576,24 @@ CAUTION: **Warning:**
Enabling debug tracing can have severe security implications. The
output **will** contain the content of all your variables and any other
secrets! The output **will** be uploaded to the GitLab server and made visible
in job traces!
in job logs!
By default, GitLab Runner hides most of the details of what it is doing when
processing a job. This behavior keeps job traces short, and prevents secrets
from being leaked into the trace unless your script writes them to the screen.
processing a job. This behavior keeps job logs short, and prevents secrets
from being leaked into the log unless your script writes them to the screen.
If a job isn't working as expected, this can make the problem difficult to
investigate; in these cases, you can enable debug tracing in `.gitlab-ci.yml`.
Available on GitLab Runner v1.7+, this feature enables the shell's execution
trace, resulting in a verbose job trace listing all commands that were run,
log, resulting in a verbose job log listing all commands that were run,
variables that were set, etc.
Before enabling this, you should ensure jobs are visible to
[team members only](../../user/permissions.md#project-features). You should
also [erase](../pipelines.md#accessing-individual-jobs) all generated job traces
also [erase](../pipelines.md#accessing-individual-jobs) all generated job logs
before making them visible again.
To enable debug traces, set the `CI_DEBUG_TRACE` variable to `true`:
To enable debug logs (traces), set the `CI_DEBUG_TRACE` variable to `true`:
```yaml
job_name:
......@@ -601,7 +601,7 @@ job_name:
CI_DEBUG_TRACE: "true"
```
Example truncated output with debug trace set to true:
Example truncated output with `CI_DEBUG_TRACE` set to `true`:
```bash
...
......
doc/ci/variables/predefined_variables.md
View file @ 7d025a66
......@@ -40,7 +40,7 @@ future GitLab releases.**
| `CI_COMMIT_TAG` | 9.0 | 0.5 | The commit tag name. Present only when building tags. |
| `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit - the full first line of the message |
| `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` |
| `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug tracing](README.md#debug-tracing) is enabled |
| `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug logging (tracing)](README.md#debug-logging) is enabled |
| `CI_DEPLOY_PASSWORD` | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.|
| `CI_DEPLOY_USER` | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.|
| `CI_DISPOSABLE_ENVIRONMENT` | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. |
......
doc/install/installation.md
View file @ 7d025a66
......@@ -448,7 +448,7 @@ sudo -u git -H mkdir -p public/uploads/
# now that files in public/uploads are served by gitlab-workhorse
sudo chmod 0700 public/uploads
# Change the permissions of the directory where CI job traces are stored
# Change the permissions of the directory where CI job logs are stored
sudo chmod -R u+rwX builds/
# Change the permissions of the directory where CI artifacts are stored
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7