Merge branch 'improve-log-parsing-with-jq-page' into 'master'

Improve jq log parsing page

See merge request gitlab-org/gitlab!85080

doc/administration/geo/replication/troubleshooting.md

@@ -128,6 +128,9 @@ http://secondary.example.com/
                 Last status report was: 1 minute ago
 ```
 
+To find more details about failed items, check
+[the `gitlab-rails/geo.log` file](../../troubleshooting/log_parsing.md#find-most-common-geo-sync-errors)
+
 ### Check if PostgreSQL replication is working
 
 To check if PostgreSQL replication is working, check if:

doc/administration/gitaly/troubleshooting.md

@@ -16,7 +16,8 @@ Before troubleshooting, see the Gitaly and Gitaly Cluster
 
 The following sections provide possible solutions to Gitaly errors.
 
-See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings.
+See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings,
+and our advice on [parsing the `gitaly/current` file](../troubleshooting/log_parsing.md#parsing-gitalycurrent).
 
 ### Check versions when using standalone Gitaly servers
 

doc/administration/troubleshooting/log_parsing.md

@@ -11,6 +11,11 @@ but if they are not available you can still quickly parse
 [GitLab logs](../logs.md) in JSON format
 (the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/).
 
+NOTE:
+Spefically for summarising error events and basic usage statistics,
+the GitLab Support Team provides the specialised
+[`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it).
+
 ## What is JQ?
 
 As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples
@@ -18,6 +23,10 @@ include use cases targeted for parsing GitLab log files.
 
 ## Parsing Logs
 
+The examples listed below address their respective log files by
+their relative Omnibus paths and default filenames.
+Find the respective full paths in the [GitLab logs sections](../logs.md#production_jsonlog).
+
 ### General Commands
 
 #### Pipe colorized `jq` output into `less`
@@ -61,7 +70,7 @@ zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time'
 grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)'  | less -R
 ```
 
-### Parsing `production_json.log` and `api_json.log`
+### Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log`
 
 #### Find all requests with a 5XX status code
 
@@ -111,7 +120,7 @@ jq 'select(.queue_duration > 10000)' <FILE>
 jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE>
 ```
 
-### Parsing `production_json.log`
+### Parsing `gitlab-rails/production_json.log`
 
 #### Print the top three controller methods by request volume and their three longest durations
 
@@ -127,7 +136,7 @@ CT: 2435   METHOD: MetricsController#index DURS: 299.29,  284.01,  158.57
 CT: 1328   METHOD: Projects::NotesController#index DURS: 403.99,  386.29,  384.39
 ```
 
-### Parsing `api_json.log`
+### Parsing `gitlab-rails/api_json.log`
 
 #### Print top three routes with request count and their three longest durations
 
@@ -157,6 +166,8 @@ jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv
 
 ### Parsing `gitaly/current`
 
+The following examples are useful to [troubleshoot Gitaly](../gitaly/troubleshooting.md).
+
 #### Find all Gitaly requests sent from web UI
 
 ```shell
@@ -197,7 +208,7 @@ jq --raw-output --slurp '
       .[2]."grpc.time_ms",
       .[0]."grpc.request.glProjectPath"
     ]
-  | @sh' /var/log/gitlab/gitaly/current \
+  | @sh' current \
 | awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" }
   { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }'
 ```
@@ -215,12 +226,12 @@ jq --raw-output --slurp '
 #### Find all projects affected by a fatal Git problem
 
 ```shell
-grep "fatal: " /var/log/gitlab/gitaly/current | \
+grep "fatal: " current | \
     jq '."grpc.request.glProjectPath"' | \
     sort | uniq
 ```
 
-### Parsing `gitlab-shell.log`
+### Parsing `gitlab-shell/gitlab-shell.log`
 
 For investigating Git calls via SSH, from [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367).
 
@@ -238,7 +249,7 @@ jq --raw-output --slurp '
   | sort_by(-length)
   | limit(20; .[])
   | "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \
-  /var/log/gitlab/gitlab-shell/gitlab-shell.log
+  gitlab-shell.log
 ```
 
 Find the top 20 calls by project, user, and command:
@@ -256,5 +267,5 @@ jq --raw-output --slurp '
   | sort_by(-length)
   | limit(20; .[])
   | "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \
-  /var/log/gitlab/gitlab-shell/gitlab-shell.log
+  gitlab-shell.log
 ```