Skip to content

G
gitlab-ce
Commit 8e70838c authored by Nick Thomas's avatar Nick Thomas
Browse files
Options

Merge branch 'evn-ssh-clarify-docs' into 'master' 

Combine ssh docs and rename the doc

Closes #4267

See merge request gitlab-org/gitlab-ee!3753
parents 9d570c8c 14c66d23
Hide whitespace changes
Inline Side-by-side
Showing with 214 additions and 211 deletions
doc/administration/operations/fast_ssh_key_lookup.md 0 → 100644
View file @ 8e70838c
# Fast lookup of authorized SSH keys in the database
Regular SSH operations become slow as the number of users grows because OpenSSH
searches for a key to authorize a user via a linear search. In the worst case,
such as when the user is not authorized to access GitLab, OpenSSH will scan the
entire file to search for a key. This can take significant time and disk I/O,
which will delay users attempting to push or pull to a repository. Making
matters worse, if users add or remove keys frequently, the operating system may
not be able to cache the `authorized_keys` file, which causes the disk to be
accessed repeatedly.
GitLab Shell solves this by providing a way to authorize SSH users via a fast,
indexed lookup in the GitLab database. This page describes how to enable the fast
lookup of authorized SSH keys.
> **Warning:** OpenSSH version 6.9+ is required because
`AuthorizedKeysCommand` must be able to accept a fingerprint. These
instructions will break installations using older versions of OpenSSH, such as
those included with CentOS 6 as of September 2017. If you want to use this
feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing.
## Fast lookup is required for GitLab Geo
By default, GitLab manages an `authorized_keys` file, which contains all the
public SSH keys for users allowed to access GitLab. However, to maintain a
single source of truth, [Geo](../../gitlab-geo/README.md) needs to be configured to perform SSH fingerprint
lookups via database lookup.
As part of [setting up GitLab Geo](../../gitlab-geo/README.md#setup-instructions),
you will be required to follow the steps outlined below for both the primary and
secondary nodes, but note that the `Write to "authorized keys" file` checkbox
only needs to be unchecked on the primary node since it will be reflected
automatically on the secondary if database replication is working.
## Setting up fast lookup via GitLab Shell
GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup
to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to
check whether the user is authorized to access GitLab.
Create the directory `/opt/gitlab-shell` first:
```bash
sudo mkdir -p /opt/gitlab-shell
```
Create this file at `/opt/gitlab-shell/authorized_keys`:
```
#!/bin/bash
if [[ "$1" == "git" ]]; then
/opt/gitlab/embedded/service/gitlab-shell/bin/authorized_keys $2
fi
```
Set appropriate ownership and permissions:
```
sudo chown root:git /opt/gitlab-shell/authorized_keys
sudo chmod 0650 /opt/gitlab-shell/authorized_keys
```
Add the following to `/etc/ssh/sshd_config` or to `/assets/sshd_config` if you
are using Omnibus Docker:
```
AuthorizedKeysCommand /opt/gitlab-shell/authorized_keys %u %k
AuthorizedKeysCommandUser git
```
Reload OpenSSH:
```bash
# Debian or Ubuntu installations
sudo service ssh reload
# CentOS installations
sudo service sshd reload
```
Confirm that SSH is working by removing your user's SSH key in the UI, adding a
new one, and attempting to pull a repo.
> **Warning:** Do not disable writes until SSH is confirmed to be working
perfectly because the file will quickly become out-of-date.
In the case of lookup failures (which are not uncommon), the `authorized_keys`
file will still be scanned. So git SSH performance will still be slow for many
users as long as a large file exists.
You can disable any more writes to the `authorized_keys` file by unchecking
`Write to "authorized_keys" file` in the Application Settings of your GitLab
installation.
![Write to authorized keys setting](img/write_to_authorized_keys_setting.png)
Again, confirm that SSH is working by removing your user's SSH key in the UI,
adding a new one, and attempting to pull a repo.
Then you can backup and delete your `authorized_keys` file for best performance.
## How to go back to using the `authorized_keys` file
This is a brief overview. Please refer to the above instructions for more context.
1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file)
1. Enable writes to the `authorized_keys` file in Application Settings
1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker.
1. Reload sshd: `sudo service sshd reload`
1. Remove the `/opt/gitlab-shell/authorized_keys` file
## Compiling a custom version of OpenSSH for CentOS 6
Building a custom version of OpenSSH is not necessary for Ubuntu 16.04 users,
since Ubuntu 16.04 ships with OpenSSH 7.2.
It is also unnecessary for CentOS 7.4 users, as that version ships with
OpenSSH 7.4. If you are using CentOS 7.0 - 7.3, we strongly recommend that you
upgrade to CentOS 7.4 instead of following this procedure. This should be as
simple as running `yum update`.
CentOS 6 users must build their own OpenSSH package to enable SSH lookups via
the database. The following instructions can be used to build OpenSSH 7.5:
1. First, download the package and install the required packages:
```
sudo su -
cd /tmp
curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz
tar xzvf openssh-7.5p1.tar.gz
yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel
```
3. Prepare the build by copying files to the right place:
```
mkdir -p /root/rpmbuild/{SOURCES,SPECS}
cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/
cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/
cd /root/rpmbuild/SPECS
```
3. Next, set the spec settings properly:
```
sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec
sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec
sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec
```
3. Build the RPMs:
```
rpmbuild -bb openssh.spec
```
4. Ensure the RPMs were built:
```
ls -al /root/rpmbuild/RPMS/x86_64/
```
You should see something as the following:
```
total 1324
drwxr-xr-x. 2 root root 4096 Jun 20 19:37 .
drwxr-xr-x. 3 root root 19 Jun 20 19:37 ..
-rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm
```
5. Install the packages. OpenSSH packages will replace `/etc/pam.d/sshd`
with its own version, which may prevent users from logging in, so be sure
that the file is backed up and restored after installation:
```
timestamp=$(date +%s)
cp /etc/pam.d/sshd pam-ssh-conf-$timestamp
rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm
yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd
```
6. Verify the installed version. In another window, attempt to login to the server:
```
ssh -v <your-centos-machine>
```
You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5"
If not, you may need to restart sshd (e.g. `systemctl restart sshd.service`).
7. *IMPORTANT!* Open a new SSH session to your server before exiting to make
sure everything is working! If you need to downgrade, simple install the
older package:
```
# Only run this if you run into a problem logging in
yum downgrade openssh-server openssh openssh-clients
```
doc/administration/operations/index.md
View file @ 8e70838c
......@@ -15,5 +15,4 @@ that to prioritize important jobs.
to restart Sidekiq.
- **(EES/EEP)** [Extra Sidekiq operations](extra_sidekiq_processes.md): Configure an extra set of Sidekiq processes to ensure certain queues always have dedicated workers, no matter the amount of jobs that need to be processed.
- [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer.
- **(EES/EEP)** [Speed up SSH operations](speed_up_ssh.md): Authorize SSH users via a fast, indexed lookup to the GitLab database.
- [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer.
- **(EES/EEP)** [Speed up SSH operations](fast_ssh_key_lookup.md): Authorize SSH users via a fast, indexed lookup to the GitLab database.
doc/administration/operations/speed_up_ssh.md
View file @ 8e70838c
# Speed up SSH operations
>
- [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/250) in GitLab Enterprise Edition 8.7.
- Available in GitLab Enterprise Edition Starter.
## The problem
SSH operations become slow as the number of users grows.
## The reason
OpenSSH searches for a key to authorize a user via a linear search. In the worst case, such as when the user is not authorized to access GitLab, OpenSSH will scan the entire file to search for a key. This can take significant time and disk I/O, which will delay users attempting to push or pull to a repository. Making matters worse, if users add or remove keys frequently, the operating system may not be able to cache the authorized_keys file, which causes the disk to be accessed repeatedly.
## The solution
GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to check whether the user is authorized to access GitLab.
> **Warning:** OpenSSH version 6.9+ is required because
`AuthorizedKeysCommand` must be able to accept a fingerprint. These
instructions will break installations using older versions of OpenSSH, such as
those included with CentOS 6 as of September 2017. If you want to use this
feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package]
(#compiling-a-custom-version-of-openssh-for-centos-6) before continuing.
Create the directory `/opt/gitlab-shell` first:
```bash
sudo mkdir -p /opt/gitlab-shell
```
Create this file at `/opt/gitlab-shell/authorized_keys`:
```
#!/bin/bash
if [[ "$1" == "git" ]]; then
/opt/gitlab/embedded/service/gitlab-shell/bin/authorized_keys $2
fi
```
Set appropriate ownership and permissions:
```
sudo chown root:git /opt/gitlab-shell/authorized_keys
sudo chmod 0650 /opt/gitlab-shell/authorized_keys
```
Add the following to `/etc/ssh/sshd_config` or to `/assets/sshd_config` if you are using Omnibus Docker:
```
AuthorizedKeysCommand /opt/gitlab-shell/authorized_keys %u %k
AuthorizedKeysCommandUser git
```
Reload OpenSSH:
```bash
# Debian or Ubuntu installations
sudo service ssh reload
# CentOS installations
sudo service sshd reload
```
Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo.
> **Warning:** Do not disable writes until SSH is confirmed to be working perfectly because the file will quickly become out-of-date.
In the case of lookup failures (which are not uncommon), the `authorized_keys` file will still be scanned. So git SSH performance will still be slow for many users as long as a large file exists.
You can disable any more writes to the `authorized_keys` file by unchecking `Write to "authorized_keys" file` in the Application Settings of your GitLab installation.
![Write to authorized keys setting](img/write_to_authorized_keys_setting.png)
Again, confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo.
Then you can backup and delete your `authorized_keys` file for best performance.
## How to go back to using the `authorized_keys` file
This is a brief overview. Please refer to the above instructions for more context.
1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file)
1. Enable writes to the `authorized_keys` file in Application Settings
1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker.
1. Reload sshd: `sudo service sshd reload`
1. Remove the `/opt/gitlab-shell/authorized_keys` file
## Compiling a custom version of OpenSSH for CentOS 6
Building a custom version of OpenSSH is not necessary for Ubuntu 16.04 users,
since Ubuntu 16.04 ships with OpenSSH 7.2.
It is also unnecessary for CentOS 7.4 users, as that version ships with
OpenSSH 7.4. If you are using CentOS 7.0 - 7.3, we strongly recommend that you
upgrade to CentOS 7.4 instead of following this procedure. This should be as
simple as running `yum update`.
CentOS 6 users must build their own OpenSSH package to enable SSH lookups via
the database. The following instructions can be used to build OpenSSH 7.5:
1. First, download the package and install the required packages:
```
sudo su -
cd /tmp
curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz
tar xzvf openssh-7.5p1.tar.gz
yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel
```
3. Prepare the build by copying files to the right place:
```
mkdir -p /root/rpmbuild/{SOURCES,SPECS}
cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/
cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/
cd /root/rpmbuild/SPECS
```
3. Next, set the spec settings properly:
```
sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec
sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec
sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec
```
3. Build the RPMs:
```
rpmbuild -bb openssh.spec
```
4. Ensure the RPMs were built:
```
ls -al /root/rpmbuild/RPMS/x86_64/
```
You should see something as the following:
```
total 1324
drwxr-xr-x. 2 root root 4096 Jun 20 19:37 .
drwxr-xr-x. 3 root root 19 Jun 20 19:37 ..
-rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm
```
5. Install the packages. OpenSSH packages will replace `/etc/pam.d/sshd`
with its own version, which may prevent users from logging in, so be sure
that the file is backed up and restored after installation:
```
timestamp=$(date +%s)
cp /etc/pam.d/sshd pam-ssh-conf-$timestamp
rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm
yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd
```
6. Verify the installed version. In another window, attempt to login to the server:
```
ssh -v <your-centos-machine>
```
You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5"
If not, you may need to restart sshd (e.g. `systemctl restart sshd.service`).
7. *IMPORTANT!* Open a new SSH session to your server before exiting to make
sure everything is working! If you need to downgrade, simple install the
older package:
```
# Only run this if you run into a problem logging in
yum downgrade openssh-server openssh openssh-clients
```
This document was moved to [another location](fast_ssh_key_lookup.md).
doc/gitlab-geo/README.md
View file @ 8e70838c
......@@ -86,7 +86,7 @@ current version of OpenSSH:
Note that CentOS 6 and 7.0 ship with an old version of OpenSSH that do not
support a feature that Geo requires. See the [documentation on GitLab Geo SSH
access](ssh.md) for more details.
access](../administration/operations/fast_ssh_key_lookup.md) for more details.
### LDAP
......@@ -145,8 +145,8 @@ If you installed GitLab using the Omnibus packages (highly recommended):
Geo node to unlock GitLab Geo.
1. [Setup the database replication](database.md) (`primary (read-write) <->
secondary (read-only)` topology).
1. [Lookup authorized SSH keys in the database](../administration/operations/speed_up_ssh.html),
do this step for both primary AND secondary nodes.
1. [Configure fast lookup of authorized SSH keys in the database](../administration/operations/fast_ssh_key_lookup.md),
this step is required and needs to be done on both the primary AND secondary nodes.
1. [Configure GitLab](configuration.md) to set the primary and secondary nodes.
1. Optional: [Configure a secondary LDAP server](../administration/auth/ldap.md)
for the secondary. See [notes on LDAP](#ldap).
......@@ -165,7 +165,7 @@ If you installed GitLab from source:
Geo node to unlock GitLab Geo.
1. [Setup the database replication](database_source.md) (`primary (read-write)
<-> secondary (read-only)` topology).
1. [Lookup authorized SSH keys in the database](../administration/operations/speed_up_ssh.html),
1. [Configure fast lookup of authorized SSH keys in the database](../administration/operations/fast_ssh_key_lookup.md),
do this step for both primary AND secondary nodes.
1. [Configure GitLab](configuration_source.md) to set the primary and secondary
nodes.
......
doc/gitlab-geo/ssh.md
View file @ 8e70838c
# GitLab Geo SSH access
By default, GitLab manages an `authorized_keys` file, which contains all the
public SSH keys for users allowed to access GitLab. However, to maintain a
single source of truth, Geo needs to be configured to perform SSH fingerprint
lookups via database lookup. This approach is also much faster than scanning a
file.
>**Note:**
GitLab 10.0 and higher require database lookups for SSH keys.
Note this feature is only available on operating systems that support OpenSSH
6.9 and above. For CentOS 6, see the [instructions on building custom
version of OpenSSH for your server]
(../administration/operations/speed_up_ssh.html#compiling-a-custom-version-of-openssh-for-centos-6).
For both primary AND secondary nodes, follow the instructions on
[looking up authorized SSH keys in the database](../administration/operations/speed_up_ssh.html).
Note that the 'Write to "authorized keys" file' checkbox only needs
to be unchecked on the primary node since it will be reflected automatically
in the secondary if database replication is working.
This document was moved to [another location](../administration/operations/fast_ssh_key_lookup.md).
doc/gitlab-geo/updating_the_geo_nodes.md
View file @ 8e70838c
......@@ -93,7 +93,7 @@ for existing repositories was added in GitLab 10.1.
## Upgrading to GitLab 10.0
Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via
the database](ssh.md) to avoid having to maintain consistency of the
the database](../administration/operations/fast_ssh_key_lookup.md) to avoid having to maintain consistency of the
`authorized_keys` file for SSH access. Failing to do this will prevent users
from being able to clone via SSH.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7