This file logs information related to the Elasticsearch Integration, including
`/var/log/gitlab/gitlab-rails/elasticsearch.log` for Omnibus GitLab
errors during indexing or searching Elasticsearch. It's stored at:
packages or in `/home/git/gitlab/log/elasticsearch.log` for installations
from source.
It logs information related to the Elasticsearch Integration including
-`/var/log/gitlab/gitlab-rails/elasticsearch.log` for Omnibus GitLab packages.
errors during indexing or searching Elasticsearch.
-`/home/git/gitlab/log/elasticsearch.log` for installations from source.
Each line contains a JSON line that can be ingested by services like Elasticsearch and Splunk.
Each line contains a JSON line that can be ingested by services like Elasticsearch and Splunk.
Line breaks have been added to the following example line for clarity:
Line breaks have been added to the following example line for clarity:
...
@@ -785,12 +788,12 @@ Line breaks have been added to the following example line for clarity:
...
@@ -785,12 +788,12 @@ Line breaks have been added to the following example line for clarity:
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17819) in GitLab 12.6.
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17819) in GitLab 12.6.
This file lives in
This file logs the information about exceptions being tracked by
`/var/log/gitlab/gitlab-rails/exceptions_json.log` for Omnibus GitLab
`Gitlab::ErrorTracking`, which provides a standard and consistent way of
packages or in `/home/git/gitlab/log/exceptions_json.log` for installations
[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling). This file is stored in:
from source.
It logs the information about exceptions being tracked by `Gitlab::ErrorTracking` which provides a standard and consistent way of [processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling).
-`/var/log/gitlab/gitlab-rails/exceptions_json.log` for Omnibus GitLab packages.
-`/home/git/gitlab/log/exceptions_json.log` for installations from source.
Each line contains a JSON line that can be ingested by Elasticsearch. For example:
Each line contains a JSON line that can be ingested by Elasticsearch. For example:
...
@@ -820,7 +823,7 @@ Omnibus GitLab packages or in `/home/git/gitlab/log/service_measurement.log` for
...
@@ -820,7 +823,7 @@ Omnibus GitLab packages or in `/home/git/gitlab/log/service_measurement.log` for
installations from source.
installations from source.
It contains only a single structured log with measurements for each service execution.
It contains only a single structured log with measurements for each service execution.
It will contain measurements such as the number of SQL calls, `execution_time`, `gc_stats`, and `memory usage`.
It contains measurements such as the number of SQL calls, `execution_time`, `gc_stats`, and `memory usage`.
For example:
For example:
...
@@ -945,7 +948,7 @@ For Omnibus GitLab installations, Alertmanager logs reside in `/var/log/gitlab/a
...
@@ -945,7 +948,7 @@ For Omnibus GitLab installations, Alertmanager logs reside in `/var/log/gitlab/a
## Crond Logs
## Crond Logs
For Omnibus GitLab installations, crond logs reside in `/var/log/gitlab/crond/`.
For Omnibus GitLab installations, `crond` logs reside in `/var/log/gitlab/crond/`.
## Grafana Logs
## Grafana Logs
...
@@ -973,7 +976,7 @@ from a GitLab instance.
...
@@ -973,7 +976,7 @@ from a GitLab instance.
If the bug or error is readily reproducible, save the main GitLab logs
If the bug or error is readily reproducible, save the main GitLab logs
[to a file](troubleshooting/linux_cheat_sheet.md#files--dirs) while reproducing the
[to a file](troubleshooting/linux_cheat_sheet.md#files--dirs) while reproducing the
problem once or more times:
problem a few times:
```shell
```shell
sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log
sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log
...
@@ -986,7 +989,7 @@ Conclude the log gathering with <kbd>Ctrl</kbd> + <kbd>C</kbd>.
...
@@ -986,7 +989,7 @@ Conclude the log gathering with <kbd>Ctrl</kbd> + <kbd>C</kbd>.
If performance degradations or cascading errors occur that can't readily be attributed to one
If performance degradations or cascading errors occur that can't readily be attributed to one
of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/)
of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/)
can provide a perspective spanning all of Omnibus GitLab. For more details and instructions
can provide a perspective spanning all of Omnibus GitLab. For more details and instructions
to run it, see[the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos).
to run it, read[the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos).
NOTE: **Note:**
NOTE: **Note:**
GitLab Support likes to use this custom-made tool.
GitLab Support likes to use this custom-made tool.
...
@@ -995,5 +998,5 @@ GitLab Support likes to use this custom-made tool.
...
@@ -995,5 +998,5 @@ GitLab Support likes to use this custom-made tool.
[Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool
[Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool
for creating and comparing performance statistics from GitLab logs.
for creating and comparing performance statistics from GitLab logs.
For more details and instructions to run it, see
For more details and instructions to run it, read the
[read the documentation for fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage).
[documentation for fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage).