Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
8ee6c088
Commit
8ee6c088
authored
Aug 08, 2019
by
Marcel Amirault
Committed by
Evan Read
Aug 08, 2019
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Bring k8s cheat sheet to docs
Bring the support team k8s cheat sheet into the main documentation
parent
e7a997aa
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
253 additions
and
0 deletions
+253
-0
doc/administration/index.md
doc/administration/index.md
+2
-0
doc/administration/troubleshooting/kubernetes_cheat_sheet.md
doc/administration/troubleshooting/kubernetes_cheat_sheet.md
+251
-0
No files found.
doc/administration/index.md
View file @
8ee6c088
...
...
@@ -188,3 +188,5 @@ Learn how to install, configure, update, and maintain your GitLab instance.
-
Useful
[
diagnostics tools
](
troubleshooting/diagnostics_tools.md
)
that are sometimes used by the GitLab
Support team.
-
[
Troubleshooting ElasticSearch
](
troubleshooting/elasticsearch.md
)
: Tips to troubleshoot ElasticSearch.
-
[
Kubernetes troubleshooting
](
troubleshooting/kubernetes_cheat_sheet.md
)
: Commands and tips useful
for troubleshooting Kubernetes-related issues.
doc/administration/troubleshooting/kubernetes_cheat_sheet.md
0 → 100644
View file @
8ee6c088
---
type
:
reference
---
# Kubernetes, GitLab and You
This is a list of useful information regarding Kubernetes that the GitLab Support
Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone
can make use of the Support team's collected knowledge
CAUTION:
**Caution:**
These commands
**can alter or break**
your Kubernetes components so use these at your own risk.
If you are on a
[
paid tier
](
https://about.gitlab.com/pricing/
)
and are not sure how
to use these commands, it is best to
[
contact Support
](
https://about.gitlab.com/support/
)
and they will assist you with any issues you are having.
## Generic kubernetes commands
-
How to authorize to your GCP project (can be especially useful if you have projects
under different GCP accounts):
```
bash
gcloud auth login
```
-
How to access Kubernetes dashboard:
```
bash
# for minikube:
minikube dashboard —url
# for non-local installations if access via kubectl is configured:
kubectl proxy
```
-
How to ssh to a Kubernetes node and enter the container as root
<https://github.com/kubernetes/kubernetes/issues/30656>
:
-
For GCP, you may find the node name and run
`gcloud compute ssh node-name`
.
-
List containers using
`docker ps`
.
-
Enter container using
`docker exec --user root -ti container-id bash`
.
-
How to copy a file from local machine to a pod:
```
bash
kubectl
cp
file-name pod-name:./destination-path
```
-
What to do with pods in
`CrashLoopBackoff`
status:
-
Check logs via Kubernetes dashboard.
-
Check logs via
`kubectl`
:
```bash
kubectl logs <unicorn pod> -c dependencies
```
-
How to tail all Kubernetes cluster events in real time:
```
bash
kubectl get events
-w
--all-namespaces
```
-
How to get logs of the previously terminated pod instance:
```
bash
kubectl logs <pod-name>
--previous
```
NOTE:
**Note:**
No logs are kept in the containers/pods themselves, everything is written to stdout.
This is the principle of Kubernetes, read
[
Twelve-factor app
](
https://12factor.net/
)
for details.
## Gitlab-specific kubernetes information
-
Minimal config that can be used to test a Kubernetes helm chart can be found
[
here
](
https://gitlab.com/charts/gitlab/issues/620
)
.
-
Tailing logs of a separate pod. An example for a unicorn pod:
```
bash
kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs
-c
unicorn
```
-
It is not possible to get all the logs via
`kubectl`
at once, like with
`gitlab-ctl tail`
,
but a number of third-party tools can be used to do it:
-
[
Kubetail
](
https://github.com/johanhaleby/kubetail
)
-
[
kail: kubernetes tail
](
https://github.com/boz/kail
)
-
[
stern
](
https://github.com/wercker/stern
)
-
Check all events in the
`gitlab`
namespace (the namespace name can be different if you
specified a different one when deploying the helm chart):
```
bash
kubectl get events
-w
--namespace
=
gitlab
```
-
Most of the useful GitLab tools (console, rake tasks, etc) are found in the task-runner
pod. You may enter it and run commands inside or run them from the outside:
```
bash
# find the pod
kubectl get pods |
grep
task-runner
# enter it
kubectl
exec
-it
<task-runner-pod-name> bash
# open rails console
# rails console can be also called from other GitLab pods
/srv/gitlab/bin/rails console
# source-style commands should also work
/srv/gitlab
&&
bundle
exec
rake gitlab:check
RAILS_ENV
=
production
# run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart
/usr/local/bin/gitlab-rake gitlab:check
# open console without entering pod
kubectl
exec
-it
<task-runner-pod-name> /srv/gitlab/bin/rails console
# check the status of DB migrations
kubectl
exec
-it
<task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status
```
You can also use
`gitlab-rake`
, instead of
`/usr/local/bin/gitlab-rake`
.
-
Troubleshooting
**Operations > Kubernetes**
integration:
-
Check the output of
`kubectl get events -w --all-namespaces`
.
-
Check the logs of pods within
`gitlab-managed-apps`
namespace.
-
On the side of GitLab check sidekiq log and kubernetes log. When GitLab is installed
via helm chart, kubernetes.log can be found inside the sidekiq pod.
-
How to get your initial admin password
<https://docs.gitlab.com/charts/installation/deployment.html#initial-login>
:
```
bash
# find the name of the secret containing the password
kubectl get secrets |
grep
initial-root
# decode it
kubectl get secret <secret-name>
-ojsonpath
={
.data.password
}
|
base64
--decode
;
echo
```
-
How to connect to a GitLab postgres database:
```
bash
kubectl
exec
-it
<task-runner-pod-name>
--
/srv/gitlab/bin/rails dbconsole
-p
```
-
How to get info about helm installation status:
```
bash
helm status name-of-installation
```
-
How to update GitLab installed using helm chart:
```
bash
helm repo upgrade
# get current values and redirect them to yaml file (analogue of gitlab.rb values)
helm get values <release name>
>
gitlab.yaml
# run upgrade itself
helm upgrade <release name> <chart path>
-f
gitlab.yaml
```
After
<https://canary.gitlab.com/charts/gitlab/issues/780>
is fixed, it should
be possible to use
[
Updating GitLab using the Helm Chart
](
https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#updating-gitlab-using-the-helm-chart
)
for upgrades.
-
How to apply changes to GitLab config:
-
Modify the
`gitlab.yaml`
file.
-
Run the following command to apply changes:
```bash
helm upgrade <release name> <chart path> -f gitlab.yaml
```
## Installation of minimal GitLab config via minukube on macOS
This section is based on
[
Developing for Kubernetes with Minikube
](
https://gitlab.com/charts/gitlab/blob/master/doc/minikube/index.md
)
and
[
Helm
](
https://gitlab.com/charts/gitlab/blob/master/doc/helm/index.md
)
. Refer
to those documents for details.
-
Install kubectl via Homebrew:
```
bash
brew
install
kubernetes-cli
```
-
Install minikube via Homebrew:
```
bash
brew cask
install
minikube
```
-
Start minikube and configure it. If minikube cannot start, try running
`minikube delete && minikube start`
and repeat the steps:
```
bash
minikube start
--cpus
3
--memory
8192
# minimum amount for GitLab to work
minikube addons
enable
ingress
minikube addons
enable
kube-dns
```
-
Install helm via Homebrew and initialize it:
```
bash
brew
install
kubernetes-helm
helm init
--service-account
tiller
```
-
Copy the file
<https://gitlab.com/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml>
to your workstation.
-
Find the IP address in the output of
`minikube ip`
and update the yaml file with
this IP address.
-
Install the GitLab helm chart:
```
bash
helm repo add gitlab https://charts.gitlab.io
helm
install
--name
gitlab
-f
<path-to-yaml-file> gitlab/gitlab
```
If you want to modify some GitLab settings, you can use the above-mentioned config
as a base and create your own yaml file.
-
Monitor the installation progress via
`helm status gitlab`
and
`minikube dashboard`
.
The installation could take up to 20-30 minutes depending on the amount of resources
on your workstation.
-
When all the pods show either a
`Running`
or
`Completed`
status, get the GitLab password as
described in
[
Initial login
](
https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#initial-login
)
,
and log in to GitLab via the UI. It will be accessible via
`https://gitlab.domain`
where
`domain`
is the value provided in the yaml file.
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, e.g.
`### Getting error message X`
.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment