@@ -63,3 +63,29 @@ You can add custom metrics in the self monitoring project by:
1.[Duplicating](../../../user/project/integrations/prometheus.md#duplicating-a-gitlab-defined-dashboard) the default dashboard.
1.[Editing](../../../user/project/integrations/prometheus.md#view-and-edit-the-source-file-of-a-custom-dashboard) the newly created dashboard file and configuring it with [dashboard YAML properties](../../../user/project/integrations/prometheus.md#dashboard-yaml-properties).
## Troubleshooting
### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don’t have permission to create groups."]`
There is [a bug](https://gitlab.com/gitlab-org/gitlab/issues/208676) which causes
project creation to fail with the following error (which appears in the log file)
| `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | |
| `sidekiq_concurrency` | Gauge | 12.5 | Maximum number of Sidekiq jobs | |
| `sidekiq_jobs_cpu_seconds` | Histogram | 12.4 | Seconds of cpu time to run Sidekiq job | queue, boundary, external_dependencies, feature_category, job_status, urgency |
| `sidekiq_jobs_db_seconds` | Histogram | 12.9 | Seconds of DB time to run Sidekiq job | queue, boundary, external_dependencies, feature_category, job_status, urgency |
| `sidekiq_jobs_gitaly_seconds` | Histogram | 12.9 | Seconds of Gitaly time to run Sidekiq job | queue, boundary, external_dependencies, feature_category, job_status, urgency |
| `sidekiq_jobs_queue_duration_seconds` | Histogram | 12.5 | Duration in seconds that a Sidekiq job was queued before being executed | queue, boundary, external_dependencies, feature_category, urgency |
| `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. |
| `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. |
| `CI_COMMIT_REF_NAME` | 9.0 | all | The branch or tag name for which project is built |
| `CI_COMMIT_REF_PROTECTED` | 11.11 | all | `true` if the job is running on a protected branch, `false` if not |
| `CI_COMMIT_REF_PROTECTED` | 11.11 | all | `true` if the job is running on a protected reference, `false` if not |
| `CI_COMMIT_REF_SLUG` | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. |
| `CI_COMMIT_SHA` | 9.0 | all | The commit revision for which project is built |
| `CI_COMMIT_SHORT_SHA` | 11.7 | all | The first eight characters of `CI_COMMIT_SHA` |
@@ -327,6 +327,46 @@ On the Route 53 dashboard, click **Hosted zones** in the left navigation bar:
1. Click **Create**.
1. Update your DNS records with your domain registrar. The steps for doing this vary depending on which registrar you use and is beyond the scope of this guide.
## Setting up Bastion Hosts
Since our GitLab instances will be in private subnets, we need a way to connect to these instances via SSH to make configuration changes, perform upgrades, etc. One way of doing this is via a [bastion host](https://en.wikipedia.org/wiki/Bastion_host), sometimes also referred to as a jump box.
TIP: **Tip:** If you do not want to maintain bastion hosts, you can set up [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) for access to instances. This is beyond the scope of this document.
### Create Bastion Host A
1. Navigate to the EC2 Dashboard and click on **Launch instance**.
1. Select the **Ubuntu Server 18.04 LTS (HVM)** AMI.
1. Choose an instance type. We'll use a `t2.micro` as we'll only use the bastion host to SSH into our other instances.
1. Click **Configure Instance Details**.
1. Under **Network**, select the `gitlab-vpc` from the dropdown menu.
1. Under **Subnet**, select the public subnet we created earlier (`gitlab-public-10.0.0.0`).
1. Double check that under **Auto-assign Public IP** you have **Use subnet setting (Enable)** selected.
1. Leave everything else as default and click **Add Storage**.
1. For storage, we'll leave everything as default and only add an 8GB root volume. We won't store anything on this instance.
1. Click **Add Tags** and on the next screen click **Add Tag**.
1. We’ll only set `Key: Name` and `Value: Bastion Host A`.
1. Click **Configure Security Group**.
1. Select **Create a new security group**, enter a **Security group name** (we'll use `bastion-sec-group`), and add a description.
1. We'll enable SSH access from anywhere (`0.0.0.0/0`). If you want stricter security, specify a single IP address or an IP address range in CIDR notation.
1. Click **Review and Launch**
1. Review all your settings and, if you're happy, click **Launch**.
1. Acknowledge that you have access to an existing key pair or create a new one. Click **Launch Instance**.
Confirm that you can SHH into the instance:
1. On the EC2 Dashboard, click on **Instances** in the left menu.
1. Select **Bastion Host A** from your list of instances.
1. Click **Connect** and follow the connection instructions.
1. If you are able to connect successfully, let's move on to setting up our second bastion host for redundancy.
### Create Bastion Host B
1. Create an EC2 instance following the same steps as above with the following changes:
1. For the **Subnet**, select the second public subnet we created earlier (`gitlab-public-10.0.2.0`).
1. Under the **Add Tags** section, we’ll set `Key: Name` and `Value: Bastion Host B` so that we can easily identify our two instances.
1. For the security group, select the existing `bastion-sec-group` we created above.
## Deploying GitLab inside an auto scaling group
We'll use AWS's wizard to deploy GitLab and then SSH into the instance to
@@ -825,7 +825,7 @@ management project. Refer to the
available configuration options.
NOTE: **Note:**
In this alpha implementation of installing Elastic Stack through CI, reading the environment pod logs through Elasticsearch is unsupported. This is supported if [installed via the UI](#elastic-stack).
In this alpha implementation of installing Elastic Stack through CI, reading the environment logs through Elasticsearch is unsupported. This is supported if [installed via the UI](#elastic-stack).
- Run serverless workloads on [Kubernetes with Knative](serverless/index.md).
### Deploy Boards **(PREMIUM)**
...
...
@@ -48,11 +48,11 @@ the need to leave GitLab.
[Read more about Canary Deployments](../canary_deployments.md)
### Pod logs **(ULTIMATE)**
### Logs **(ULTIMATE)**
GitLab makes it easy to view the logs of running pods in connected Kubernetes clusters. By displaying the logs directly in GitLab, developers can avoid having to manage console tools or jump to a different interface.
[Read more about Kubernetes pod logs](kubernetes_pod_logs.md)
[Read more about Kubernetes logs](kubernetes_pod_logs.md)
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/122013) in GitLab 12.8.
If you have [Pod Logs](../clusters/kubernetes_pod_logs.md) enabled,
you can navigate from the charts in the dashboard to view Pod Logs by
If you have [Logs](../clusters/kubernetes_pod_logs.md) enabled,
you can navigate from the charts in the dashboard to view Logs by
clicking on the context menu in the upper-right corner.
If you use the **Timeline zoom** function at the bottom of the chart, logs will narrow down to the time range you selected.
...
...
@@ -710,7 +710,7 @@ Prometheus server.
> [Introduced][ce-29691] in GitLab 12.2.
It is possible to display metrics charts within [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). The maximum number of embeds allowed in a GitLab Flavored Markdown field is 100.
It is possible to display metrics charts within [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm) fields such as issue or merge request descriptions. The maximum number of embedded charts allowed in a GitLab Flavored Markdown field is 100.
This can be useful if you are sharing an application incident or performance
metrics to others and want to have relevant information directly available.
...
...
@@ -748,6 +748,25 @@ It is also possible to embed either the default dashboard metrics or individual
![Embedded Metrics in issue templates](img/embed_metrics_issue_template.png)
### Embedding Cluster Health Charts **(ULTIMATE)**
> [Introduced](<https://gitlab.com/gitlab-org/gitlab/issues/40997>) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9.
[Cluster Health Metrics](../clusters/index.md#monitoring-your-kubernetes-cluster-ultimate) can also be embedded in [GitLab-flavored Markdown](../../markdown.md).
To embed a metric chart, include a link to that chart in the form `https://<root_url>/<project>/-/cluster/<cluster_id>?<query_params>` anywhere that GitLab-flavored Markdown is supported. To generate and copy a link to the chart, follow the instructions in the [Cluster Health Metric documentation](../clusters/index.md#monitoring-your-kubernetes-cluster-ultimate).
The following requirements must be met for the metric to unfurl:
- The `<cluster_id>` must correspond to a real cluster.
- Prometheus must be monitoring the cluster.
- The user must be allowed access to the project cluster metrics.
- The dashboards must be reporting data on the [Cluster Health Page](../clusters/index.md#monitoring-your-kubernetes-cluster-ultimate)
If the above requirements are met, then the metric will unfurl as seen below.
![Embedded Cluster Metric in issue descriptions](img/prometheus_cluster_health_embed_v12_9.png)
### Embedding Grafana charts
Grafana metrics can be embedded in [GitLab Flavored Markdown](../../markdown.md).
sidekiq_jobs_cpu_seconds: ::Gitlab::Metrics.histogram(:sidekiq_jobs_cpu_seconds,'Seconds of cpu time to run Sidekiq job',{},SIDEKIQ_LATENCY_BUCKETS),
sidekiq_jobs_completion_seconds: ::Gitlab::Metrics.histogram(:sidekiq_jobs_completion_seconds,'Seconds to complete Sidekiq job',{},SIDEKIQ_LATENCY_BUCKETS),
sidekiq_jobs_db_seconds: ::Gitlab::Metrics.histogram(:sidekiq_jobs_db_seconds,'Seconds of database time to run Sidekiq job',{},SIDEKIQ_LATENCY_BUCKETS),
sidekiq_jobs_gitaly_seconds: ::Gitlab::Metrics.histogram(:sidekiq_jobs_gitaly_seconds,'Seconds of Gitaly time to run Sidekiq job',{},SIDEKIQ_LATENCY_BUCKETS),
sidekiq_jobs_queue_duration_seconds: ::Gitlab::Metrics.histogram(:sidekiq_jobs_queue_duration_seconds,'Duration in seconds that a Sidekiq job was queued before being executed',{},SIDEKIQ_LATENCY_BUCKETS),
sidekiq_jobs_failed_total: ::Gitlab::Metrics.counter(:sidekiq_jobs_failed_total,'Sidekiq jobs failed'),
sidekiq_jobs_retried_total: ::Gitlab::Metrics.counter(:sidekiq_jobs_retried_total,'Sidekiq jobs retried'),