Commit e21dc168 authored by Katrin Leinweber's avatar Katrin Leinweber Committed by Achilleas Pipinellis

Crosslink Geo errors example

A follow-up to https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83004
parent 7482ed82
......@@ -111,6 +111,9 @@ http://secondary.example.com/
Last status report was: 2 minutes ago
```
To find more details about failed items, check
[the `gitlab-rails/geo.log` file](../../troubleshooting/log_parsing.md#find-most-common-geo-sync-errors)
### Check if PostgreSQL replication is working
To check if PostgreSQL replication is working, check if:
......
......@@ -16,7 +16,8 @@ Before troubleshooting, see the Gitaly and Gitaly Cluster
The following sections provide possible solutions to Gitaly errors.
See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings.
See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings,
and our advice on [parsing the `gitaly/current` file](../troubleshooting/log_parsing.md#parsing-gitalycurrent).
### Check versions when using standalone Gitaly servers
......
......@@ -11,6 +11,11 @@ but if they are not available you can still quickly parse
[GitLab logs](../logs.md) in JSON format
(the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/).
NOTE:
Spefically for summarising error events and basic usage statistics,
the GitLab Support Team provides the specialised
[`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it).
## What is JQ?
As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples
......@@ -18,6 +23,10 @@ include use cases targeted for parsing GitLab log files.
## Parsing Logs
The examples listed below address their respective log files by
their relative Omnibus paths and default filenames.
Find the respective full paths in the [GitLab logs sections](../logs.md#production_jsonlog).
### General Commands
#### Pipe colorized `jq` output into `less`
......@@ -61,7 +70,7 @@ zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time'
grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)' | less -R
```
### Parsing `production_json.log` and `api_json.log`
### Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log`
#### Find all requests with a 5XX status code
......@@ -111,7 +120,7 @@ jq 'select(.queue_duration > 10000)' <FILE>
jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE>
```
### Parsing `production_json.log`
### Parsing `gitlab-rails/production_json.log`
#### Print the top three controller methods by request volume and their three longest durations
......@@ -127,7 +136,7 @@ CT: 2435 METHOD: MetricsController#index DURS: 299.29, 284.01, 158.57
CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.39
```
### Parsing `api_json.log`
### Parsing `gitlab-rails/api_json.log`
#### Print top three routes with request count and their three longest durations
......@@ -157,6 +166,8 @@ jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv
### Parsing `gitaly/current`
The following examples are useful to [troubleshoot Gitaly](../gitaly/troubleshooting.md).
#### Find all Gitaly requests sent from web UI
```shell
......@@ -197,7 +208,7 @@ jq --raw-output --slurp '
.[2]."grpc.time_ms",
.[0]."grpc.request.glProjectPath"
]
| @sh' /var/log/gitlab/gitaly/current \
| @sh' current \
| awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" }
{ printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }'
```
......@@ -215,12 +226,12 @@ jq --raw-output --slurp '
#### Find all projects affected by a fatal Git problem
```shell
grep "fatal: " /var/log/gitlab/gitaly/current | \
grep "fatal: " current | \
jq '."grpc.request.glProjectPath"' | \
sort | uniq
```
### Parsing `gitlab-shell.log`
### Parsing `gitlab-shell/gitlab-shell.log`
For investigating Git calls via SSH, from [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367).
......@@ -238,7 +249,7 @@ jq --raw-output --slurp '
| sort_by(-length)
| limit(20; .[])
| "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \
/var/log/gitlab/gitlab-shell/gitlab-shell.log
gitlab-shell.log
```
Find the top 20 calls by project, user, and command:
......@@ -256,5 +267,5 @@ jq --raw-output --slurp '
| sort_by(-length)
| limit(20; .[])
| "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \
/var/log/gitlab/gitlab-shell/gitlab-shell.log
gitlab-shell.log
```
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment