Commit 41f446a0 authored by Amy Qualls's avatar Amy Qualls

Merge branch 'eread/remove-future-tense-from-gitaly-docs' into 'master'

Remove future tense from Gitaly documentation

See merge request gitlab-org/gitlab!47982
parents f5379925 39d11976
...@@ -434,8 +434,8 @@ server (with `gitaly_address`) unless you setup with special ...@@ -434,8 +434,8 @@ server (with `gitaly_address`) unless you setup with special
``` ```
NOTE: **Note:** NOTE: **Note:**
`/some/local/path` should be set to a local folder that exists, however no data will be stored in `/some/local/path` should be set to a local folder that exists, however no data is stored in
this folder. This will no longer be necessary after this folder. This requirement is scheduled to be removed when
[this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved.
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
...@@ -461,7 +461,7 @@ GitLab can reside on the same server as one of many Gitaly servers, but doesn't ...@@ -461,7 +461,7 @@ GitLab can reside on the same server as one of many Gitaly servers, but doesn't
configuration that mixes local and remote configuration. The following setup is incorrect, because: configuration that mixes local and remote configuration. The following setup is incorrect, because:
- All addresses must be reachable from the other Gitaly servers. - All addresses must be reachable from the other Gitaly servers.
- `storage1` will be assigned a Unix socket for `gitaly_address` which is - `storage1` is assigned a Unix socket for `gitaly_address` which is
invalid for some of the Gitaly servers. invalid for some of the Gitaly servers.
```ruby ```ruby
...@@ -493,7 +493,7 @@ gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" ...@@ -493,7 +493,7 @@ gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
``` ```
`path` can only be included for storage shards on the local Gitaly server. `path` can only be included for storage shards on the local Gitaly server.
If it's excluded, default Git storage directory will be used for that storage shard. If it's excluded, default Git storage directory is used for that storage shard.
### Disable Gitaly where not required (optional) ### Disable Gitaly where not required (optional)
...@@ -590,7 +590,7 @@ To configure Gitaly with TLS: ...@@ -590,7 +590,7 @@ To configure Gitaly with TLS:
``` ```
1. Copy all Gitaly server certificates (or their certificate authority) to 1. Copy all Gitaly server certificates (or their certificate authority) to
`/etc/gitlab/trusted-certs` so that Gitaly servers will trust the certificate when calling into themselves `/etc/gitlab/trusted-certs` so that Gitaly servers trust the certificate when calling into themselves
or other Gitaly servers: or other Gitaly servers:
```shell ```shell
...@@ -647,8 +647,8 @@ To configure Gitaly with TLS: ...@@ -647,8 +647,8 @@ To configure Gitaly with TLS:
``` ```
NOTE: **Note:** NOTE: **Note:**
`/some/local/path` should be set to a local folder that exists, however no data will be stored `/some/local/path` should be set to a local folder that exists, however no data is stored
in this folder. This will no longer be necessary after in this folder. This requirement is scheduled to be removed when
[Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. [Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved.
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
...@@ -668,7 +668,7 @@ To configure Gitaly with TLS: ...@@ -668,7 +668,7 @@ To configure Gitaly with TLS:
``` ```
1. Copy all Gitaly server certificates (or their certificate authority) to the system trusted 1. Copy all Gitaly server certificates (or their certificate authority) to the system trusted
certificates folder so Gitaly server will trust the certificate when calling into itself or other Gitaly certificates folder so Gitaly server trusts the certificate when calling into itself or other Gitaly
servers. servers.
```shell ```shell
...@@ -806,7 +806,7 @@ You can observe the behavior of this queue using the Gitaly logs and Prometheus: ...@@ -806,7 +806,7 @@ You can observe the behavior of this queue using the Gitaly logs and Prometheus:
NOTE: **Note:** NOTE: **Note:**
Though the name of the Prometheus metric contains `rate_limiting`, it is a concurrency limiter, not Though the name of the Prometheus metric contains `rate_limiting`, it is a concurrency limiter, not
a rate limiter. If a Gitaly client makes 1000 requests in a row very quickly, concurrency will not a rate limiter. If a Gitaly client makes 1000 requests in a row very quickly, concurrency does not
exceed 1 and the concurrency limiter has no effect. exceed 1 and the concurrency limiter has no effect.
## Rotate Gitaly authentication token ## Rotate Gitaly authentication token
...@@ -835,7 +835,7 @@ behavior of your GitLab installation using Prometheus. Use the following Prometh ...@@ -835,7 +835,7 @@ behavior of your GitLab installation using Prometheus. Use the following Prometh
sum(rate(gitaly_authentications_total[5m])) by (enforced, status) sum(rate(gitaly_authentications_total[5m])) by (enforced, status)
``` ```
In a system where authentication is configured correctly and where you have live traffic, you will In a system where authentication is configured correctly and where you have live traffic, you
see something like this: see something like this:
```prometheus ```prometheus
...@@ -891,7 +891,7 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi ...@@ -891,7 +891,7 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi
``` ```
If you run your [Prometheus query](#verify-authentication-monitoring) while this change is If you run your [Prometheus query](#verify-authentication-monitoring) while this change is
being rolled out, you will see non-zero values for the `enforced="false",status="denied"` counter. being rolled out, you see non-zero values for the `enforced="false",status="denied"` counter.
### Ensure there are no authentication failures ### Ensure there are no authentication failures
...@@ -1003,7 +1003,7 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks: ...@@ -1003,7 +1003,7 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks:
- Is the feature flag for this patch set in the database? If so, the feature flag setting controls - Is the feature flag for this patch set in the database? If so, the feature flag setting controls
GitLab's use of "Rugged patch" code. GitLab's use of "Rugged patch" code.
- If the feature flag is not set, GitLab tries accessing the filesystem underneath the - If the feature flag is not set, GitLab tries accessing the filesystem underneath the
Gitaly server directly. If it can, it will use the "Rugged patch": Gitaly server directly. If it can, it uses the "Rugged patch":
- If using Unicorn. - If using Unicorn.
- If using Puma and [thread count](../../install/requirements.md#puma-threads) is set - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set
to `1`. to `1`.
...@@ -1076,7 +1076,7 @@ gitaly-debug -h ...@@ -1076,7 +1076,7 @@ gitaly-debug -h
remote: GitLab: 401 Unauthorized remote: GitLab: 401 Unauthorized
``` ```
You will need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab You need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab
app nodes). app nodes).
### Client side gRPC logs ### Client side gRPC logs
......
This diff is collapsed.
...@@ -75,7 +75,7 @@ the `gitaly_authentications_total` metric. ...@@ -75,7 +75,7 @@ the `gitaly_authentications_total` metric.
### TLS ### TLS
Gitaly supports TLS encryption. You will need to bring your own certificates as Gitaly supports TLS encryption. You need to bring your own certificates as
this isn't provided automatically. this isn't provided automatically.
| Name | Type | Required | Description | | Name | Type | Required | Description |
...@@ -128,7 +128,7 @@ The following values can be set in the `[git]` section of the configuration file ...@@ -128,7 +128,7 @@ The following values can be set in the `[git]` section of the configuration file
| Name | Type | Required | Description | | Name | Type | Required | Description |
| ---- | ---- | -------- | ----------- | | ---- | ---- | -------- | ----------- |
| `bin_path` | string | no | Path to Git binary. If not set, will be resolved using `PATH`. | | `bin_path` | string | no | Path to Git binary. If not set, is resolved using `PATH`. |
| `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. | | `catfile_cache_size` | integer | no | Maximum number of cached [cat-file processes](#cat-file-cache). Default is `100`. |
#### `cat-file` cache #### `cat-file` cache
...@@ -192,8 +192,8 @@ For historical reasons ...@@ -192,8 +192,8 @@ For historical reasons
[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains
the Git hooks that allow GitLab to validate and react to Git pushes. the Git hooks that allow GitLab to validate and react to Git pushes.
Because Gitaly "owns" Git pushes, GitLab Shell must therefore be Because Gitaly "owns" Git pushes, GitLab Shell must therefore be
installed alongside Gitaly. This will be [simplified in the installed alongside Gitaly. We plan to
future](https://gitlab.com/gitlab-org/gitaly/-/issues/1226). [simplify this](https://gitlab.com/gitlab-org/gitaly/-/issues/1226).
| Name | Type | Required | Description | | Name | Type | Required | Description |
| ---- | ---- | -------- | ----------- | | ---- | ---- | -------- | ----------- |
...@@ -238,9 +238,11 @@ The following values configure logging in Gitaly under the `[logging]` section. ...@@ -238,9 +238,11 @@ The following values configure logging in Gitaly under the `[logging]` section.
While the main Gitaly application logs go to `stdout`, there are some extra log While the main Gitaly application logs go to `stdout`, there are some extra log
files that go to a configured directory, like the GitLab Shell logs. files that go to a configured directory, like the GitLab Shell logs.
GitLab Shell does not support `panic` or `trace` level logs. `panic` will fall GitLab Shell does not support `panic` or `trace` level logs:
back to `error`, while `trace` will fall back to `debug`. Any other invalid log
levels will default to `info`. - `panic` falls back to `error`.
- `trace` falls back to `debug`.
- Any other invalid log levels default to `info`.
Example: Example:
......
...@@ -50,7 +50,7 @@ storage2: ...@@ -50,7 +50,7 @@ storage2:
> structure than Omnibus. In `gitlab.yml` you indicate the path for the > structure than Omnibus. In `gitlab.yml` you indicate the path for the
> repositories, for example `/home/git/repositories`, while in Omnibus you > repositories, for example `/home/git/repositories`, while in Omnibus you
> indicate `git_data_dirs`, which for the example above would be `/home/git`. > indicate `git_data_dirs`, which for the example above would be `/home/git`.
> Then, Omnibus will create a `repositories` directory under that path to use with > Then, Omnibus creates a `repositories` directory under that path to use with
> `gitlab.yml`. > `gitlab.yml`.
> >
> This little detail matters because while restoring a backup, the current > This little detail matters because while restoring a backup, the current
...@@ -90,8 +90,7 @@ This example uses NFS. We do not recommend using EFS for storage as it may impac ...@@ -90,8 +90,7 @@ This example uses NFS. We do not recommend using EFS for storage as it may impac
1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
NOTE: **Note:** NOTE: **Note:**
The [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` will be We plan to replace [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` with `repositories: storages`. If you
deprecated and replaced by `repositories: storages` in the future, so if you
are upgrading from a version prior to 8.10, make sure to add the configuration are upgrading from a version prior to 8.10, make sure to add the configuration
as described in the step above. After you make the changes and confirm they are as described in the step above. After you make the changes and confirm they are
working, you can remove the `repos_path` line. working, you can remove the `repos_path` line.
...@@ -112,16 +111,15 @@ working, you can remove the `repos_path` line. ...@@ -112,16 +111,15 @@ working, you can remove the `repos_path` line.
Note that Omnibus stores the repositories in a `repositories` subdirectory Note that Omnibus stores the repositories in a `repositories` subdirectory
of the `git-data` directory. of the `git-data` directory.
## Choose where new repositories will be stored ## Choose where new repositories are stored
Once you set the multiple storage paths, you can choose where new repositories Once you set the multiple storage paths, you can choose where new repositories
will be stored under **Admin Area > Settings > Repository > are stored in the Admin Area under **Settings > Repository > Repository storage > Storage nodes for new repositories**.
Repository storage > Storage nodes for new repositories**.
Each storage can be assigned a weight from 0-100. When a new project is created, these Each storage can be assigned a weight from 0-100. When a new project is created, these
weights are used to determine the storage location the repository will be created on. weights are used to determine the storage location the repository is created on.
![Choose repository storage path in Admin Area](img/repository_storages_admin_ui_v13_1.png) ![Choose repository storage path in Admin Area](img/repository_storages_admin_ui_v13_1.png)
Beginning with GitLab 8.13.4, multiple paths can be chosen. New repositories Beginning with GitLab 8.13.4, multiple paths can be chosen. New repositories
will be randomly placed on one of the selected paths. are randomly placed on one of the selected paths.
...@@ -19,7 +19,7 @@ locations that can be: ...@@ -19,7 +19,7 @@ locations that can be:
- Accessed via [Gitaly](gitaly/index.md) on its own machine. - Accessed via [Gitaly](gitaly/index.md) on its own machine.
In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})`
configuration hash. The storage layouts discussed here will apply to any shard configuration hash. The storage layouts discussed here apply to any shard
defined in it. defined in it.
The `default` repository shard that is available in any installations The `default` repository shard that is available in any installations
...@@ -30,22 +30,22 @@ Anything discussed below is expected to be part of that folder. ...@@ -30,22 +30,22 @@ Anything discussed below is expected to be part of that folder.
NOTE: **Note:** NOTE: **Note:**
In GitLab 13.0, hashed storage is enabled by default and the legacy storage is In GitLab 13.0, hashed storage is enabled by default and the legacy storage is
deprecated. Support for legacy storage will be removed in GitLab 14.0. deprecated. Support for legacy storage is scheduled to be removed in GitLab 14.0.
If you haven't migrated yet, check the If you haven't migrated yet, check the
[migration instructions](raketasks/storage.md#migrate-to-hashed-storage). [migration instructions](raketasks/storage.md#migrate-to-hashed-storage).
The option to choose between hashed and legacy storage in the admin area has The option to choose between hashed and legacy storage in the admin area has
been disabled. been disabled.
Hashed storage is the storage behavior we rolled out with 10.0. Instead Hashed storage is the storage behavior we rolled out with 10.0. Instead
of coupling project URL and the folder structure where the repository will be of coupling project URL and the folder structure where the repository is
stored on disk, we are coupling a hash, based on the project's ID. This makes stored on disk, we are coupling a hash, based on the project's ID. This makes
the folder structure immutable, and therefore eliminates any requirement to the folder structure immutable, and therefore eliminates any requirement to
synchronize state from URLs to disk structure. This means that renaming a group, synchronize state from URLs to disk structure. This means that renaming a group,
user, or project will cost only the database transaction, and will take effect user, or project costs only the database transaction, and takes effect
immediately. immediately.
The hash also helps to spread the repositories more evenly on the disk, so the The hash also helps to spread the repositories more evenly on the disk, so the
top-level directory will contain less folders than the total amount of top-level top-level directory contains fewer folders than the total number of top-level
namespaces. namespaces.
The hash format is based on the hexadecimal representation of SHA256: The hash format is based on the hexadecimal representation of SHA256:
...@@ -64,7 +64,7 @@ by another folder with the next 2 characters. They are both stored in a special ...@@ -64,7 +64,7 @@ by another folder with the next 2 characters. They are both stored in a special
### Translating hashed storage paths ### Translating hashed storage paths
Troubleshooting problems with the Git repositories, adding hooks, and other Troubleshooting problems with the Git repositories, adding hooks, and other
tasks will require you translate between the human readable project name tasks requires you translate between the human readable project name
and the hashed storage path. and the hashed storage path.
#### From project name to hashed path #### From project name to hashed path
...@@ -102,7 +102,7 @@ To translate from a hashed storage path to a project name: ...@@ -102,7 +102,7 @@ To translate from a hashed storage path to a project name:
ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project
``` ```
The quoted string in that command is the directory tree you'll find on your The quoted string in that command is the directory tree you can find on your
GitLab server. For example, on a default Omnibus installation this would be GitLab server. For example, on a default Omnibus installation this would be
`/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`
with `.git` from the end of the directory name removed. with `.git` from the end of the directory name removed.
...@@ -135,7 +135,7 @@ when housekeeping is run on the source project. ...@@ -135,7 +135,7 @@ when housekeeping is run on the source project.
### Hashed storage coverage migration ### Hashed storage coverage migration
Files stored in an S3 compatible endpoint will not have the downsides Files stored in an S3-compatible endpoint do not have the downsides
mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`,
which is true for CI Cache and LFS Objects. which is true for CI Cache and LFS Objects.
...@@ -183,7 +183,7 @@ NOTE: **Deprecated:** ...@@ -183,7 +183,7 @@ NOTE: **Deprecated:**
In GitLab 13.0, hashed storage is enabled by default and the legacy storage is In GitLab 13.0, hashed storage is enabled by default and the legacy storage is
deprecated. If you haven't migrated yet, check the deprecated. If you haven't migrated yet, check the
[migration instructions](raketasks/storage.md#migrate-to-hashed-storage). [migration instructions](raketasks/storage.md#migrate-to-hashed-storage).
Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab Support for legacy storage is scheduled to be removed in GitLab 14.0. If you're on GitLab
13.0 and later, switching new projects to legacy storage is not possible. 13.0 and later, switching new projects to legacy storage is not possible.
The option to choose between hashed and legacy storage in the admin area has The option to choose between hashed and legacy storage in the admin area has
been disabled. been disabled.
...@@ -199,7 +199,7 @@ easy for Administrators to find where the repository is stored. ...@@ -199,7 +199,7 @@ easy for Administrators to find where the repository is stored.
On the other hand this has some drawbacks: On the other hand this has some drawbacks:
Storage location will concentrate huge amount of top-level namespaces. The Storage location concentrates a huge number of top-level namespaces. The
impact can be reduced by the introduction of impact can be reduced by the introduction of
[multiple storage paths](repository_storage_paths.md). [multiple storage paths](repository_storage_paths.md).
...@@ -209,6 +209,6 @@ an old removed or renamed project sharing the same URL. This means that ...@@ -209,6 +209,6 @@ an old removed or renamed project sharing the same URL. This means that
`mygroup/myproject` from your backup may not be the same original project that `mygroup/myproject` from your backup may not be the same original project that
is at that same URL today. is at that same URL today.
Any change in the URL will need to be reflected on disk (when groups / users or Any change in the URL needs to be reflected on disk (when groups / users or
projects are renamed). This can add a lot of load in big installations, projects are renamed). This can add a lot of load in big installations,
especially if using any type of network based filesystem. especially if using any type of network based filesystem.
...@@ -331,7 +331,7 @@ listed in the descriptions of the relevant settings. ...@@ -331,7 +331,7 @@ listed in the descriptions of the relevant settings.
| `receive_max_input_size` | integer | no | Maximum push size (MB). | | `receive_max_input_size` | integer | no | Maximum push size (MB). |
| `repository_checks_enabled` | boolean | no | GitLab will periodically run `git fsck` in all project and wiki repositories to look for silent disk corruption issues. | | `repository_checks_enabled` | boolean | no | GitLab will periodically run `git fsck` in all project and wiki repositories to look for silent disk corruption issues. |
| `repository_size_limit` | integer | no | **(PREMIUM)** Size limit per repository (MB) | | `repository_size_limit` | integer | no | **(PREMIUM)** Size limit per repository (MB) |
| `repository_storages_weighted` | hash of strings to integers | no | (GitLab 13.1 and later) Hash of names of taken from `gitlab.yml` to [weights](../administration/repository_storage_paths.md#choose-where-new-repositories-will-be-stored). New projects are created in one of these stores, chosen by a weighted random selection. | | `repository_storages_weighted` | hash of strings to integers | no | (GitLab 13.1 and later) Hash of names of taken from `gitlab.yml` to [weights](../administration/repository_storage_paths.md#choose-where-new-repositories-are-stored). New projects are created in one of these stores, chosen by a weighted random selection. |
| `repository_storages` | array of strings | no | (GitLab 13.0 and earlier) List of names of enabled storage paths, taken from `gitlab.yml`. New projects are created in one of these stores, chosen at random. | | `repository_storages` | array of strings | no | (GitLab 13.0 and earlier) List of names of enabled storage paths, taken from `gitlab.yml`. New projects are created in one of these stores, chosen at random. |
| `require_admin_approval_after_user_signup` | boolean | no | When enabled, any user that signs up for an account using the registration form is placed under a **Pending approval** state and has to be explicitly [approved](../user/admin_area/approving_users.md) by an administrator. | | `require_admin_approval_after_user_signup` | boolean | no | When enabled, any user that signs up for an account using the registration form is placed under a **Pending approval** state and has to be explicitly [approved](../user/admin_area/approving_users.md) by an administrator. |
| `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. | | `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. |
......
...@@ -84,7 +84,7 @@ If your test-suite is failing with Gitaly issues, as a first step, try running: ...@@ -84,7 +84,7 @@ If your test-suite is failing with Gitaly issues, as a first step, try running:
rm -rf tmp/tests/gitaly rm -rf tmp/tests/gitaly
``` ```
During RSpec tests, the Gitaly instance will write logs to `gitlab/log/gitaly-test.log`. During RSpec tests, the Gitaly instance writes logs to `gitlab/log/gitaly-test.log`.
## Legacy Rugged code ## Legacy Rugged code
...@@ -124,23 +124,23 @@ Most of this code exists in the `lib/gitlab/git/rugged_impl` directory. ...@@ -124,23 +124,23 @@ Most of this code exists in the `lib/gitlab/git/rugged_impl` directory.
NOTE: **Note:** NOTE: **Note:**
You should NOT need to add or modify code related to You should NOT need to add or modify code related to
Rugged unless explicitly discussed with the [Gitaly Rugged unless explicitly discussed with the [Gitaly
Team](https://gitlab.com/groups/gl-gitaly/group_members). This code will Team](https://gitlab.com/groups/gl-gitaly/group_members). This code does
NOT work on GitLab.com or other GitLab instances that do not use NFS. NOT work on GitLab.com or other GitLab instances that do not use NFS.
## `TooManyInvocationsError` errors ## `TooManyInvocationsError` errors
During development and testing, you may experience `Gitlab::GitalyClient::TooManyInvocationsError` failures. During development and testing, you may experience `Gitlab::GitalyClient::TooManyInvocationsError` failures.
The `GitalyClient` will attempt to block against potential n+1 issues by raising this error The `GitalyClient` attempts to block against potential n+1 issues by raising this error
when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution. when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution.
As a temporary measure, export `GITALY_DISABLE_REQUEST_LIMITS=1` to suppress the error. This will disable the n+1 detection As a temporary measure, export `GITALY_DISABLE_REQUEST_LIMITS=1` to suppress the error. This disables the n+1 detection
in your development environment. in your development environment.
Please raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly Please raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly
~performance ~"technical debt". Please ensure that the issue contains the full stack trace and error message of the ~performance ~"technical debt". Please ensure that the issue contains the full stack trace and error message of the
`TooManyInvocationsError`. Also include any known failing tests if possible. `TooManyInvocationsError`. Also include any known failing tests if possible.
Isolate the source of the n+1 problem. This will normally be a loop that results in Gitaly being called for each Isolate the source of the n+1 problem. This is normally a loop that results in Gitaly being called for each
element in an array. If you are unable to isolate the problem, please contact a member element in an array. If you are unable to isolate the problem, please contact a member
of the [Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members) for assistance. of the [Gitaly Team](https://gitlab.com/groups/gl-gitaly/group_members) for assistance.
...@@ -154,7 +154,7 @@ Gitlab::GitalyClient.allow_n_plus_1_calls do ...@@ -154,7 +154,7 @@ Gitlab::GitalyClient.allow_n_plus_1_calls do
end end
``` ```
Once the code is wrapped in this block, this code-path will be excluded from n+1 detection. Once the code is wrapped in this block, this code path is excluded from n+1 detection.
## Request counts ## Request counts
...@@ -184,12 +184,12 @@ branches and SHA to use a custom commit in <https://gitlab.com/gitlab-org/gitaly ...@@ -184,12 +184,12 @@ branches and SHA to use a custom commit in <https://gitlab.com/gitlab-org/gitaly
NOTE: **Note:** NOTE: **Note:**
With the introduction of auto-deploy for Gitaly, the format of With the introduction of auto-deploy for Gitaly, the format of
`GITALY_SERVER_VERSION` was aligned with Omnibus syntax. `GITALY_SERVER_VERSION` was aligned with Omnibus syntax.
It no longer supports `=revision`, it will evaluate the file content as a Git It no longer supports `=revision`, it evaluates the file content as a Git
reference (branch or SHA), only if it matches a semver it will prepend a `v`. reference (branch or SHA). Only if it matches a semver does it prepend a `v`.
If you want to run tests locally against a modified version of Gitaly you If you want to run tests locally against a modified version of Gitaly you
can replace `tmp/tests/gitaly` with a symlink. This is much faster can replace `tmp/tests/gitaly` with a symlink. This is much faster
because if will avoid a Gitaly re-install each time you run `rspec`. because it avoids a Gitaly re-install each time you run `rspec`.
```shell ```shell
rm -rf tmp/tests/gitaly rm -rf tmp/tests/gitaly
...@@ -197,12 +197,12 @@ ln -s /path/to/gitaly tmp/tests/gitaly ...@@ -197,12 +197,12 @@ ln -s /path/to/gitaly tmp/tests/gitaly
``` ```
Make sure you run `make` in your local Gitaly directory before running Make sure you run `make` in your local Gitaly directory before running
tests. Otherwise, Gitaly will fail to boot. tests. Otherwise, Gitaly fails to boot.
If you make changes to your local Gitaly in between test runs you need If you make changes to your local Gitaly in between test runs you need
to manually run `make` again. to manually run `make` again.
Note that CI tests will not use your locally modified version of Note that CI tests do not use your locally modified version of
Gitaly. To use a custom Gitaly version in CI you need to update Gitaly. To use a custom Gitaly version in CI you need to update
GITALY_SERVER_VERSION as described at the beginning of this paragraph. GITALY_SERVER_VERSION as described at the beginning of this paragraph.
......
...@@ -87,7 +87,7 @@ download all the advertised refs. ...@@ -87,7 +87,7 @@ download all the advertised refs.
git push origin --force 'refs/heads/*' git push origin --force 'refs/heads/*'
``` ```
[Protected branches](../protected_branches.md) will cause this to fail. To proceed, you must [Protected branches](../protected_branches.md) cause this to fail. To proceed, you must
remove branch protection, push, and then re-enable protected branches. remove branch protection, push, and then re-enable protected branches.
1. To remove large files from tagged releases, force push your changes to all tags on GitLab: 1. To remove large files from tagged releases, force push your changes to all tags on GitLab:
...@@ -96,7 +96,7 @@ download all the advertised refs. ...@@ -96,7 +96,7 @@ download all the advertised refs.
git push origin --force 'refs/tags/*' git push origin --force 'refs/tags/*'
``` ```
[Protected tags](../protected_tags.md) will cause this to fail. To proceed, you must remove tag [Protected tags](../protected_tags.md) cause this to fail. To proceed, you must remove tag
protection, push, and then re-enable protected tags. protection, push, and then re-enable protected tags.
1. To prevent dead links to commits that no longer exist, push the `refs/replace` created by `git filter-repo`. 1. To prevent dead links to commits that no longer exist, push the `refs/replace` created by `git filter-repo`.
...@@ -131,7 +131,7 @@ To purge files from GitLab storage: ...@@ -131,7 +131,7 @@ To purge files from GitLab storage:
tar xzf project-backup.tar.gz tar xzf project-backup.tar.gz
``` ```
This will contain a `project.bundle` file, which was created by This contains a `project.bundle` file, which was created by
[`git bundle`](https://git-scm.com/docs/git-bundle). [`git bundle`](https://git-scm.com/docs/git-bundle).
1. Clone a fresh copy of the repository from the bundle: 1. Clone a fresh copy of the repository from the bundle:
...@@ -141,12 +141,12 @@ To purge files from GitLab storage: ...@@ -141,12 +141,12 @@ To purge files from GitLab storage:
``` ```
1. Using `git filter-repo`, purge any files from the history of your repository. Because we are 1. Using `git filter-repo`, purge any files from the history of your repository. Because we are
trying to remove internal refs, we will rely on the `commit-map` produced by each run to tell us trying to remove internal refs, we rely on the `commit-map` produced by each run to tell us
which internal refs to remove. which internal refs to remove.
NOTE: **Note:** NOTE: **Note:**
`git filter-repo` creates a new `commit-map` file every run, and overwrite the `commit-map` from `git filter-repo` creates a new `commit-map` file every run, and overwrite the `commit-map` from
the previous run. You will need this file from **every** run. Do the next step every time you run the previous run. You need this file from **every** run. Do the next step every time you run
`git filter-repo`. `git filter-repo`.
To purge all large files, the `--strip-blobs-bigger-than` option can be used: To purge all large files, the `--strip-blobs-bigger-than` option can be used:
...@@ -178,7 +178,7 @@ To purge files from GitLab storage: ...@@ -178,7 +178,7 @@ To purge files from GitLab storage:
git push origin --force 'refs/heads/*' git push origin --force 'refs/heads/*'
``` ```
[Protected branches](../protected_branches.md) will cause this to fail. To proceed, you must [Protected branches](../protected_branches.md) cause this to fail. To proceed, you must
remove branch protection, push, and then re-enable protected branches. remove branch protection, push, and then re-enable protected branches.
1. To remove large files from tagged releases, force push your changes to all tags on GitLab: 1. To remove large files from tagged releases, force push your changes to all tags on GitLab:
...@@ -187,7 +187,7 @@ To purge files from GitLab storage: ...@@ -187,7 +187,7 @@ To purge files from GitLab storage:
git push origin --force 'refs/tags/*' git push origin --force 'refs/tags/*'
``` ```
[Protected tags](../protected_tags.md) will cause this to fail. To proceed, you must remove tag [Protected tags](../protected_tags.md) cause this to fail. To proceed, you must remove tag
protection, push, and then re-enable protected tags. protection, push, and then re-enable protected tags.
1. To prevent dead links to commits that no longer exist, push the `refs/replace` created by `git filter-repo`. 1. To prevent dead links to commits that no longer exist, push the `refs/replace` created by `git filter-repo`.
...@@ -205,12 +205,12 @@ To purge files from GitLab storage: ...@@ -205,12 +205,12 @@ To purge files from GitLab storage:
NOTE: **Note:** NOTE: **Note:**
Safely cleaning the repository requires it to be made read-only for the duration Safely cleaning the repository requires it to be made read-only for the duration
of the operation. This happens automatically, but submitting the cleanup request of the operation. This happens automatically, but submitting the cleanup request
will fail if any writes are ongoing, so cancel any outstanding `git push` fails if any writes are ongoing, so cancel any outstanding `git push`
operations before continuing. operations before continuing.
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19376) in GitLab 11.6. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19376) in GitLab 11.6.
Repository cleanup allows you to upload a text file of objects and GitLab will remove internal Git Repository cleanup allows you to upload a text file of objects and GitLab removes internal Git
references to these objects. You can use references to these objects. You can use
[`git filter-repo`](https://github.com/newren/git-filter-repo) to produce a list of objects (in a [`git filter-repo`](https://github.com/newren/git-filter-repo) to produce a list of objects (in a
`commit-map` file) that can be used with repository cleanup. `commit-map` file) that can be used with repository cleanup.
...@@ -230,25 +230,25 @@ To clean up a repository: ...@@ -230,25 +230,25 @@ To clean up a repository:
1. Click **Start cleanup**. 1. Click **Start cleanup**.
This will: This:
- Remove any internal Git references to old commits. - Removes any internal Git references to old commits.
- Run `git gc` against the repository to remove unreferenced objects. Repacking your repository will temporarily - Runs `git gc` against the repository to remove unreferenced objects. Repacking your repository temporarily
cause the size of your repository to increase significantly, because the old pack files are not removed until the causes the size of your repository to increase significantly, because the old pack files are not removed until the
new pack files have been created. new pack files have been created.
- Unlink any unused LFS objects currently attached to your project, freeing up storage space. - Unlinks any unused LFS objects currently attached to your project, freeing up storage space.
- Recalculate the size of your repository on disk. - Recalculates the size of your repository on disk.
You will receive an email notification with the recalculated repository size after the cleanup has completed. GitLab sends an email notification with the recalculated repository size after the cleanup has completed.
When using repository cleanup, note: When using repository cleanup, note:
- Project statistics are cached. You may need to wait 5-10 minutes to see a reduction in storage utilization. - Project statistics are cached. You may need to wait 5-10 minutes to see a reduction in storage utilization.
- Housekeeping prunes loose objects older than 2 weeks. This means objects added in the last 2 weeks - Housekeeping prunes loose objects older than 2 weeks. This means objects added in the last 2 weeks
will not be removed immediately. If you have access to the are not be removed immediately. If you have access to the
[Gitaly](../../../administration/gitaly/index.md) server, you may run `git gc --prune=now` to [Gitaly](../../../administration/gitaly/index.md) server, you may run `git gc --prune=now` to
prune all loose objects immediately. prune all loose objects immediately.
- This process will remove some copies of the rewritten commits from GitLab's cache and database, - This process removes some copies of the rewritten commits from GitLab's cache and database,
but there are still numerous gaps in coverage and some of the copies may persist indefinitely. but there are still numerous gaps in coverage and some of the copies may persist indefinitely.
[Clearing the instance cache](../../../administration/raketasks/maintenance.md#clear-redis-cache) [Clearing the instance cache](../../../administration/raketasks/maintenance.md#clear-redis-cache)
may help to remove some of them, but it should not be depended on for security purposes! may help to remove some of them, but it should not be depended on for security purposes!
...@@ -290,7 +290,7 @@ history. We recommend the open-source community-maintained tool ...@@ -290,7 +290,7 @@ history. We recommend the open-source community-maintained tool
[`git filter-repo`](https://github.com/newren/git-filter-repo). [`git filter-repo`](https://github.com/newren/git-filter-repo).
NOTE: **Note:** NOTE: **Note:**
Until `git gc` runs on the GitLab side, the "removed" commits and blobs will still exist. You also Until `git gc` runs on the GitLab side, the "removed" commits and blobs still exist. You also
must be able to push the rewritten history to GitLab, which may be impossible if you've already must be able to push the rewritten history to GitLab, which may be impossible if you've already
exceeded the maximum size limit. exceeded the maximum size limit.
...@@ -304,7 +304,7 @@ increased, your only option is to: ...@@ -304,7 +304,7 @@ increased, your only option is to:
CAUTION: **Caution:** CAUTION: **Caution:**
This process is not suitable for removing sensitive data like password or keys from your repository. This process is not suitable for removing sensitive data like password or keys from your repository.
Information about commits, including file content, is cached in the database, and will remain Information about commits, including file content, is cached in the database, and remain
visible even after they have been removed from the repository. visible even after they have been removed from the repository.
## Troubleshooting ## Troubleshooting
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment