Commit 7382b749 authored by Marcel Amirault's avatar Marcel Amirault Committed by Evan Read

Move 3 references from debug project to docs

Move rails and linux cheat sheets, and test_environment
reference docs to the docs project
parent a15a69cf
...@@ -187,13 +187,29 @@ Learn how to install, configure, update, and maintain your GitLab instance. ...@@ -187,13 +187,29 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong
- [Log system](logs.md): Where to look for logs. - [Log system](logs.md): Where to look for logs.
- [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs.
- Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab - [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md)
Support team.
- [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md): Tips to troubleshoot ElasticSearch. ### Support Team Docs
- [Kubernetes troubleshooting](troubleshooting/kubernetes_cheat_sheet.md): Commands and tips useful
for troubleshooting Kubernetes-related issues. The GitLab Support Team has collected a lot of information about troubleshooting GitLab
- Useful links from the Support Team: instances. These documents are normally used by the Support Team itself, or by customers
- [GitLab Developer Docs](https://docs.gitlab.com/ee/development/README.html). with direct guidance from a Support Team member. GitLab administrators may find the
- [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html). information useful for troubleshooting, but if you are experiencing trouble with your
- [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html). GitLab instance, you should check your [support options](https://about.gitlab.com/support/)
- [Strace zine](https://wizardzines.com/zines/strace/). before referring to these documents.
CAUTION: **Warning:**
Using the commands listed in the documentation below could result in data loss or
other damage to a GitLab instance, and should only be used by experienced administrators
who are aware of the risks.
- [Useful diagnostics tools](troubleshooting/diagnostics_tools.md)
- [Useful Linux commands](troubleshooting/linux_cheat_sheet.md)
- [Troubleshooting Kubernetes](troubleshooting/kubernetes_cheat_sheet.md)
- [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers)
- [GitLab rails console commands](troubleshooting/gitlab_rails_cheat_sheet.md) (for Support Engineers)
- Useful links:
- [GitLab Developer Docs](../development/README.md)
- [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html)
- [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html)
- [Strace zine](https://wizardzines.com/zines/strace/)
This diff is collapsed.
---
type: reference
---
# Linux Cheat Sheet
This is the GitLab Support Team's collection of information regarding Linux, that they
sometimes use while troubleshooting. It is listed here for transparency,
and it may be useful for users with experience with Linux. If you are currently
having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/)
first, before attempting to use this information.
CAUTION: **CAUTION:**
If you are administering GitLab you are expected to know these commands for your distribution
of choice. If you are a GitLab Support Engineer, consider this a cross-reference to
translate `yum` -> `apt-get` and the like.
Note: **Note:**
Most of the commands below have not been labeled as to which distribution they work
on. Contributions are welcome to help add them.
## System Commands
### Distro Information
```bash
# Debian/Ubuntu
uname -a
lsb_release -a
# CentOS/RedHat
cat /etc/centos-release
cat /etc/redhat-release
# This will provide a lot more information
cat /etc/os-release
```
### Shut down or Reboot
```bash
shutdown -h now
reboot
```
### Permissions
```bash
# change the user:group ownership of a file/dir
chown root:git <file_or_dir>
# make a file executable
chmod u+x <file>
```
### Files & Dirs
```bash
# create a new directory and all subdirectories
mkdir -p dir/dir2/dir3
# Send a command's output to file.txt, no STDOUT
ls > file.txt
# Send a command's output to file.txt AND see it in STDOUT
ls | tee /tmp/file.txt
# Search and Replace within a file
sed -i 's/original-text/new-text/g' <filename>
```
### See all set environment variables
```bash
env
```
## Searching
### File names
```bash
# search for a file in a filesystem
find . -name 'filename.rb' -print
# locate a file
locate <filename>
# see command history
history
# search CLI history
<ctrl>-R
```
### File contents
```bash
# -B/A = show 2 lines before/after search_term
grep -B 2 -A 2 search_term <filename>
# -<number> shows both before and after
grep -2 search_term <filename>
# Search on all files in directory (recursively)
grep -r search_term <directory>
# search through *.gz files is the same except with zgrep
zgrep search_term <filename>
# Fast grep printing lines containing a string pattern
fgrep -R string_pattern <filename or directory>
```
### CLI
```bash
# View command history
history
# Run last command that started with 'his' (3 letters min)
!his
# Search through command history
<ctrl>-R
# Execute last command with sudo
sudo !!
```
## Managing resources
### Memory, Disk, & CPU usage
```bash
# disk space info. The '-h' gives the data in human-readable values
df -h
# size of each file/dir and its contents in the current dir
du -hd 1
# or alternative
du -h --max-depth=1
# find files greater than certain size(k, M, G) and list them in order
# get rid of the + for exact, - for less than
find / -type f -size +100M -print0 | xargs -0 du -hs | sort -h
# Find free memory on a system
free -m
# Find what processes are using memory/CPU and organize by it
# Load average is 1/CPU for 1, 5, and 15 minutes
top -o %MEM
top -o %CPU
```
### Strace
```bash
# strace a process
strace -tt -T -f -y -s 1024 -p <pid>
# -tt print timestamps with microsecond accuracy
# -T print the time spent in each syscall
# -f also trace any child processes that forked
# -y print the path associated with file handles
# -s max string length to print for an event
# -o output file
# run strace on all unicorn processes
ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -s 1024 -o /tmp/unicorn.txt
```
See the [strace zine](https://wizardzines.com/zines/strace/) for a quick walkthrough.
Brendan Gregg has a more detailed explanation of [how to use strace](http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html).
Be aware that strace can have major impacts to system performance when it is running.
### The Strace Parser tool
Our [strace-parser tool](https://gitlab.com/wchandler/strace-parser) can be used to
provide a high level summary of the `strace` output. It is similar to `strace -C`,
but provides much more detailed statistics.
MacOS and Linux binaries [are available](https://gitlab.com/gitlab-com/support/toolbox/strace-parser/-/tags),
or you can build it from source if you have the Rust compiler.
#### How to use the tool
First run the tool with no arguments other than the strace output file name to get
a summary of the top processes sorted by time spent actively performing tasks. You
can also sort based on total time, # of syscalls made, PID #, and # of child processes
using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but
can be changed using the `-c`/`--count` option. See `--help` for full details.
```sh
$ ./strace-parser strace.txt
Top 25 PIDs
-----------
pid active (ms) wait (ms) total (ms) % active syscalls
---------- ---------- --------- --------- --------- ---------
8795 689.072 45773.832 46462.902 16.89% 23018
13408 679.432 55910.891 56590.320 16.65% 28593
6423 554.822 13175.485 13730.308 13.60% 13735
...
```
Based on the summary, you can then view the details of syscalls made by one or more
procsses using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags for
a sorted list. `--stats` takes the same sorting and count options as summary.
```sh
$ ./strace-parse strace.text -p 6423
PID 6423
13735 syscalls, active time: 554.822ms, total time: 13730.308ms
syscall count total max avg min errors
(ms) (ms) (ms) (ms)
--------------- -------- ---------- ---------- ---------- ---------- --------
epoll_wait 628 13175.485 21.259 20.980 0.020
clock_gettime 7326 199.500 0.249 0.027 0.013
stat 2101 110.768 19.056 0.053 0.017 ENOENT: 2076
...
---------------
Parent PID: 495
Child PIDs: 8383, 8418, 8419, 8420, 8421
Slowest file access times for PID 6423:
open (ms) timestamp error file name
----------- --------------- --------------- ----------
29.818 10:53:11.528954 /srv/gitlab-data/builds/2018_08/6174/954448.log
12.309 10:53:46.708274 /srv/gitlab-data/builds/2018_08/5342/954186.log
0.039 10:53:49.222110 /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_note.html.haml
0.035 10:53:49.125115 /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_push.html.haml
...
```
In the example above, we can see that file opening times on `/srv/gitlab-data` are
extremely slow, about 100X slower than `/opt/gitlab`.
When nothing stands out in the results, a good way to get more context is to run `strace`
on your own GitLab instance while performing the action performed by the customer,
then compare summaries of both results and dive into the differences.
#### Stats for the open syscall
Rough numbers for calls to `open` and `openat` (used to access files) on various configurations.
Slow storage can cause the dreaded `DeadlineExceeded` error in Gitaly.
Also [see this entry](https://docs.gitlab.com/ee/administration/operations/filesystem_benchmarking.html)
in the handbook for quick tests customers can perform to check their filesystem performance.
Keep in mind that timing information from `strace` is often somewhat inaccurate, so
small differences should not be considered significant.
|Setup | access times |
|:--------------|:--------------|
| EFS | 10 - 30ms |
| Local Storage | 0.01 - 1ms |
## Networking
### Ports
```bash
# Find the programs that are listening on ports
netstat -plnt
ss -plnt
lsof -i -P | grep <port>
```
### Internet/DNS
```bash
# Show domain IP address
dig +short example.com
nslookup example.com
# Check DNS using specific nameserver
# 8.8.8.8 = google, 1.1.1.1 = cloudflare, 208.67.222.222 = opendns
dig @8.8.8.8 example.com
nslookup example.com 1.1.1.1
# Find host provider
whois <ip_address> | grep -i "orgname\|netname"
# Curl headers with redirect
curl --head --location https://example.com
```
## Package Management
```bash
# Debian/Ubuntu
# List packages
dpkg -l
apt list --installed
# Find an installed package
dpkg -l | grep <package>
apt list --installed | grep <package>
# Install a package
dpkg -i <package_name>.deb
apt-get install <package>
apt install <package>
# CentOS/RedHat
# Install a package
yum install <package>
dnf install <package> # RHEL/CentOS 8+
rpm -ivh <package_name>.rpm
# Find an installed package
rpm -qa | grep <package>
```
## Logs
```bash
# Print last lines in log file where 'n'
# is the number of lines to print
tail -n /path/to/log/file
```
---
type: reference
---
# Apps for a Testing Environment
This is the GitLab Support Team's collection of information regarding testing environments,
for use while troubleshooting. It is listed here for transparency, and it may be useful
for users with experience with these tools. If you are currently having an issue with
GitLab, you may want to check your [support options](https://about.gitlab.com/support/)
first, before attempting to use this information.
NOTE: **Note:**
This page was initially written for Support Engineers, so some of the links
are only available internally at GitLab.
## Docker
The following were tested on docker containers running in the cloud. Support Engineers,
please see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers)
on how to run Docker containers on `dev-resources`. Other setups haven't been tested,
but contributions are welcome.
### GitLab
Please see [our Docker test environment docs](https://docs.gitlab.com/ee/install/digitaloceandocker.html#create-new-gitlab-container)
for how to run GitLab on Docker. When spinning this up with `docker-machine`, ensure
you change a few things:
1. Update the name of the `docker-machine` host. You can see a list of hosts
with `docker-machine ls`.
1. Expose the necessary ports using the `-p` flag. Docker normally doesn't
allow access to any ports it uses outside of the container, so they must be
explicitly exposed.
1. Add any necessary `gitlab.rb` configuration to the
`GITLAB_OMNIBUS_CONFIG` variable.
For example, when the `docker-machine` host we want to use is `do-docker`:
```sh
docker run --detach --name gitlab \
--env GITLAB_OMNIBUS_CONFIG="external_url 'http://$(docker-machine ip do-docker)'; gitlab_rails['gitlab_shell_ssh_port'] = 2222;" \
--hostname $(docker-machine ip do-docker) \
-p 80:80 -p 2222:22 \
gitlab/gitlab-ee:11.5.3-ee.0
```
### SAML
#### SAML for Authentication
We can use the [test-saml-idp Docker image](https://hub.docker.com/r/jamedjo/test-saml-idp)
to do the work for us:
```sh
docker run --name gitlab_saml -p 8080:8080 -p 8443:8443 \
-e SIMPLESAMLPHP_SP_ENTITY_ID=<GITLAB_IP_OR_DOMAIN> \
-e SIMPLESAMLPHP_SP_ASSERTION_CONSUMER_SERVICE=<GITLAB_IP_OR_DOMAIN>/users/auth/saml/callback \
-d jamedjo/test-saml-idp
```
The following will also need to go in your `/etc/gitlab/gitlab.rb`. See [our SAML docs](https://docs.gitlab.com/ee/integration/saml.html)
for more, as well as the list of [default usernames, passwords, and emails](https://hub.docker.com/r/jamedjo/test-saml-idp/#usage).
```ruby
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
gitlab_rails['omniauth_sync_email_from_provider'] = 'saml'
gitlab_rails['omniauth_sync_profile_from_provider'] = ['saml']
gitlab_rails['omniauth_sync_profile_attributes'] = ['email']
gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml'
gitlab_rails['omniauth_block_auto_created_users'] = false
gitlab_rails['omniauth_auto_link_ldap_user'] = false
gitlab_rails['omniauth_auto_link_saml_user'] = true
gitlab_rails['omniauth_providers'] = [
{
"name" => "saml",
"label" => "SAML",
"args" => {
assertion_consumer_service_url: '<GITLAB_IP_OR_DOMAIN>/users/auth/saml/callback',
idp_cert_fingerprint: '119b9e027959cdb7c662cfd075d9e2ef384e445f',
idp_sso_target_url: '<SAML_IP_OR_DOMAIN>:8080/simplesaml/saml2/idp/SSOService.php',
issuer: '<GITLAB_IP_OR_DOMAIN>',
name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent'
}
}
]
```
#### GroupSAML for GitLab.com
See [the GDK SAML documentation](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/saml.md).
### ElasticSearch
```sh
docker run -d --name elasticsearch \
-p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" \
docker.elastic.co/elasticsearch/elasticsearch:5.5.1
```
Then confirm it works in the browser at `curl http://<IP_ADDRESS>:9200/_cat/health`.
ElasticSearch's default username is `elastic` and password is `changeme`.
### PlantUML
See [our PlantUML docs](../integration/plantuml.md#docker)
on running PlantUML in Docker.
### Jira
```sh
docker run -d -p 8081:8080 cptactionhank/atlassian-jira:latest
```
Then go to `<IP_ADDRESS>:8081` in the browser to set it up. This requires a
Jira license.
### Grafana
```sh
docker run -d --name grafana -e "GF_SECURITY_ADMIN_PASSWORD=gitlab" -p 3000:3000 grafana/grafana
```
Access it at `<IP_ADDRESS>:3000`.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment