Commit f41f8eca authored by Shinya Maeda's avatar Shinya Maeda Committed by Achilleas Pipinellis

Fix trace documentation

parent 266451e6
# Job traces (logs) # Job traces (logs)
By default, all job traces (logs) are saved to `/var/opt/gitlab/shared/asrtifacts` Job traces are sent by gitlab-runner while it's processing a job. You can see traces in job pages, pipelines, email notifications, etc.
and `/home/git/gitlab/builds` for Omnibus packages and installations from source Basically, there are two states in job traces. One is "Live trace", and another one is "Archived trace";
respectively. The job logs are organized by year and month (for example, `2017_03`),
and then by project ID. |state|condition|step|data flow|stored path|
|---|---|---|---|---|
|Live trace|when a job is running|1: patching| gitlab-runner => gitlab-unicorn => file storage|`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
|Live trace|when a job is finished|2: overwtiring| gitlab-runner => gitlab-unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
|Archived trace|After a job is finished|3: archiving| sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`|
The `ROOT_PATH` varies per your enviroment. For example, if you used omnibus packages, it would be `/var/opt/gitlab/gitlab-ci`,
whereas if you used source instlation, it would be `/home/git/gitlab`.
There isn't a way to automatically expire old job logs, but it's safe to remove There isn't a way to automatically expire old job logs, but it's safe to remove
them if they're taking up too much space. If you remove the logs manually, the them if they're taking up too much space. If you remove the logs manually, the
...@@ -41,24 +48,46 @@ To change the location where the job logs will be stored, follow the steps below ...@@ -41,24 +48,46 @@ To change the location where the job logs will be stored, follow the steps below
[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" [restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
## Upload traces to object storage
Archived trace is one of [job artifacts](job_artifacts.md).
If you set up [object storage settings](https://docs.gitlab.com/ce/administration/job_artifacts.html#object-storage-settings),
job traces are automatically migrated to object storage as well as other job artifacts.
Here is the data flow;
|state|condition|step|data flow|stored path|
|---|---|---|---|---|
|Live trace|when a job is running|1: patching| gitlab-runner => gitlab-unicorn => file storage|`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
|Live trace|when a job is finished|2: overwtiring| gitlab-runner => gitlab-unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
|Archived trace|After a job is finished|3: archiving| sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`|
|Archived trace|After a trace is archived|4: uploading| sidekiq moves archived trace to object storage |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`|
## New live trace architecture ## New live trace architecture
> [Introduced][ce-18169] in GitLab 10.4. > [Introduced][ce-18169] in GitLab 10.4.
> [Announced as General availability][ce-46097] in GitLab 11.0.
> **Notes**: > **Notes**:
- This feature is still Beta, which could impact GitLab.com/on-premises instances, and in the worst case scenario, traces will be lost. - Performance improvements are scheduled in [11.1](https://gitlab.com/gitlab-org/gitlab-ce/issues/47125).
- This feature is still being discussed in [an issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/46097) for the performance improvements.
- This feature is off by default. Please check below how to enable/disable this featrue. - This feature is off by default. Please check below how to enable/disable this featrue.
**What is "live trace"?** **For cloud-native compatible application**
By combining the process with object storage settings, we can completely bypass file storage. This is useful option in cloud-native GitLab installtion.
Here is the data flow;
Job trace that is sent by runner while jobs are running. You can see live trace in job pages UI. |state|condition|step|data flow|stored path|
The live traces are archived once job finishes. |---|---|---|---|---|
|Live trace|when a job is running|1: patching| gitlab-runner => gitlab-unicorn => redis and database|- (Stored in Redis and Database, instead)|
|Live trace|when a job is finished|2: overwtiring| gitlab-runner => gitlab-unicorn => redis and database |- (Stored in Redis and Database, instead)|
|Archived trace|After a job is finished|3: archiving| sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`|
|Archived trace|After a trace is archived|4: uploading| sidekiq moves archived trace to object storage |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`|
**What is new architecture?** (Step 3 is scheduled to be improved in https://gitlab.com/gitlab-org/gitlab-ce/issues/44663)
So far, when GitLab Runner sends a job trace to GitLab-Rails, traces have been saved to file storage as text files. **The detailed mechanizm**
This was a problem for [Cloud Native-compatible GitLab application](https://gitlab.com/gitlab-com/migration/issues/23) where GitLab had to rely on File Storage.
This new live trace architecture stores chunks of traces in Redis and database instead of file storage. This new live trace architecture stores chunks of traces in Redis and database instead of file storage.
Redis is used as first-class storage, and it stores up-to 128kB. Once the full chunk is sent it will be flushed to database. Afterwhile, the data in Redis and database will be archived to ObjectStorage. Redis is used as first-class storage, and it stores up-to 128kB. Once the full chunk is sent it will be flushed to database. Afterwhile, the data in Redis and database will be archived to ObjectStorage.
...@@ -134,4 +163,5 @@ We're currently evaluating this feature on dev.gitalb.org or staging.gitlab.com ...@@ -134,4 +163,5 @@ We're currently evaluating this feature on dev.gitalb.org or staging.gitlab.com
- TBD - TBD
[ce-18169]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18169 [ce-18169]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18169
\ No newline at end of file [ce-46097]: https://gitlab.com/gitlab-org/gitlab-ce/issues/46097
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment