Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
a243ce52
Commit
a243ce52
authored
Jul 02, 2020
by
Cody West
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Minor wording changes & formatting
parent
ff6dba73
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
17 additions
and
6 deletions
+17
-6
doc/administration/troubleshooting/tracing_correlation_id.md
doc/administration/troubleshooting/tracing_correlation_id.md
+17
-6
No files found.
doc/administration/troubleshooting/tracing_correlation_id.md
View file @
a243ce52
...
@@ -5,9 +5,9 @@ type: reference
...
@@ -5,9 +5,9 @@ type: reference
# Finding relevant log entries with a correlation ID
# Finding relevant log entries with a correlation ID
Since GitLab 11.6, a unique tracking ID, known as the "correlation ID" has been
Since GitLab 11.6, a unique tracking ID, known as the "correlation ID" has been
logged for
all requests in GitLab. This is great for tracking the logged
logged for
most requests in GitLab. This makes it easier to trace behavior in a
behavior in a precise manner. Without this approach and
can be difficult or
distributed system. Without this ID it
can be difficult or
impossible to match
up
log entries together.
impossible to match
correlating
log entries together.
## Getting the correlation ID
## Getting the correlation ID
...
@@ -46,12 +46,22 @@ you reproduce an action in GitLab. You could tail the GitLab logs, filtering
...
@@ -46,12 +46,22 @@ you reproduce an action in GitLab. You could tail the GitLab logs, filtering
to requests by your user, and then watch the requests until you see what you're
to requests by your user, and then watch the requests until you see what you're
interested in.
interested in.
### Getting the correlation ID from curl
If you're using curl then you can use the verbose option to show request and response headers, as well as other debug info.
```
shell
➜ ~ curl
-v
https://gitlab.example.com/api/v4/projects
# look for a line that looks like this
< x-request-id: 4rAMkV3gof4
```
#### Using jq
#### Using jq
This example uses
[
jq
](
https://stedolan.github.io/jq/
)
to filter results and
This example uses
[
jq
](
https://stedolan.github.io/jq/
)
to filter results and
display values we most likely care about.
display values we most likely care about.
```
sh
```
sh
ell
sudo
gitlab-ctl
tail
gitlab-rails/production_json.log | jq
'select(.username == "bob") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"'
sudo
gitlab-ctl
tail
gitlab-rails/production_json.log | jq
'select(.username == "bob") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"'
```
```
...
@@ -65,7 +75,7 @@ sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username ==
...
@@ -65,7 +75,7 @@ sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username ==
This example uses only grep and tr, which are more likely to be installed than jq.
This example uses only grep and tr, which are more likely to be installed than jq.
```
sh
```
sh
ell
sudo
gitlab-ctl
tail
gitlab-rails/production_json.log |
grep
'"username":"bob"'
|
tr
','
'\n'
| egrep
'method|path|correlation_id'
sudo
gitlab-ctl
tail
gitlab-rails/production_json.log |
grep
'"username":"bob"'
|
tr
','
'\n'
| egrep
'method|path|correlation_id'
```
```
...
@@ -91,7 +101,7 @@ entries. You can simply filter the lines by the correlation ID itself since it
...
@@ -91,7 +101,7 @@ entries. You can simply filter the lines by the correlation ID itself since it
is unique enough that there shouldn't be any duplicates. A simple find & grep
is unique enough that there shouldn't be any duplicates. A simple find & grep
combo should find what you're looking for.
combo should find what you're looking for.
```
sh
```
sh
ell
# find <gitlab log directory> -type f -mtime -0 exec grep '<correlation ID>' '{}' '+'
# find <gitlab log directory> -type f -mtime -0 exec grep '<correlation ID>' '{}' '+'
find /var/log/gitlab
-type
f
-mtime
0
-exec
grep
'LOt9hgi1TV4'
'{}'
'+'
find /var/log/gitlab
-type
f
-mtime
0
-exec
grep
'LOt9hgi1TV4'
'{}'
'+'
```
```
...
@@ -107,5 +117,6 @@ find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+'
...
@@ -107,5 +117,6 @@ find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+'
If you have done some horizontal scaling in your GitLab infrastructure, then
If you have done some horizontal scaling in your GitLab infrastructure, then
you will need to search across _all_ of your GitLab nodes. You can do this with
you will need to search across _all_ of your GitLab nodes. You can do this with
some sort of log aggregation software like Loki, ELK, Splunk, or others.
some sort of log aggregation software like Loki, ELK, Splunk, or others.
Something like Ansible or pssh (parellel ssh), that can execute commands across your servers in
Something like Ansible or pssh (parellel ssh), that can execute commands across your servers in
parallel, could be used to execute the same search command. Or you could craft your own solution.
parallel, could be used to execute the same search command. Or you could craft your own solution.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment