Skip to content

G
gitlab-ce
Commit c2ff5a0b authored by Stan Hu's avatar Stan Hu
Browse files
Options

Add documentation for consolidated object storage config 

This documents the changes needed for
https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368 and
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35480.
parent f67f0ef4
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 470 additions and 189 deletions
doc/administration/job_artifacts.md
View file @ c2ff5a0b
......@@ -106,6 +106,10 @@ If you configure GitLab to store CI logs and artifacts on object storage, you mu
#### Object Storage Settings
NOTE: **Note** In GitLab 13.2 and later, we recommend using the
[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
This section describes the earlier configuration format.
For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`.
| Setting | Description | Default |
......@@ -117,22 +121,9 @@ For source installations the following settings are nested under `artifacts:` an
| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
| `connection` | Various connection options described below | |
##### S3 compatible connection settings
The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
#### Connection settings
| Setting | Description | Default |
|---------|-------------|---------|
| `provider` | Always `AWS` for compatible hosts | AWS |
| `aws_access_key_id` | AWS credentials, or compatible | |
| `aws_secret_access_key` | AWS credentials, or compatible | |
| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 |
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true |
| `region` | AWS region | us-east-1 |
| `host` | S3 compatible host for when not using AWS, for example `localhost` or `storage.example.com` | s3.amazonaws.com |
| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
See [the available connection settings for different providers](object_storage.md#connection-settings).
**In Omnibus installations:**
......@@ -210,19 +201,9 @@ JUnit test report artifact (`junit.xml.gz`) migration
[is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/27698)
by the `gitlab:artifacts:migrate` script.
### OpenStack compatible connection settings
### OpenStack example
The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
| Setting | Description | Default |
|---------|-------------|---------|
| `provider` | Always `OpenStack` for compatible hosts | OpenStack |
| `openstack_username` | OpenStack username | |
| `openstack_api_key` | OpenStack API key | |
| `openstack_temp_url_key` | OpenStack key for generating temporary URLs | |
| `openstack_auth_url` | OpenStack authentication endpoint | |
| `openstack_region` | OpenStack region | |
| `openstack_tenant_id` | OpenStack tenant ID |
See [the available connection settings for OpenStack](object_storage.md#openstack-compatible-connection-settings).
**In Omnibus installations:**
......
doc/administration/lfs/index.md
View file @ c2ff5a0b
......@@ -63,6 +63,10 @@ GitLab provides two different options for the uploading mechanism: "Direct uploa
[Read more about using object storage with GitLab](../object_storage.md).
NOTE: **Note** In GitLab 13.2 and later, we recommend using the
[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
This section describes the earlier configuration format.
**Option 1. Direct upload**
1. User pushes an `lfs` file to the GitLab instance
......@@ -86,54 +90,10 @@ The following general settings are supported.
| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
| `connection` | Various connection options described below | |
The `connection` settings match those provided by [Fog](https://github.com/fog).
See [the available connection settings for different providers](../object_storage.md#connection-settings).
Here is a configuration example with S3.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | AWS |
| `aws_access_key_id` | AWS credentials, or compatible | `ABC123DEF456` |
| `aws_secret_access_key` | AWS credentials, or compatible | `ABC123DEF456ABC123DEF456ABC123DEF456` |
| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 |
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true |
| `region` | AWS region | us-east-1 |
| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
Here is a configuration example with GCS.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | `Google` |
| `google_project` | GCP project name | `gcp-project-12345` |
| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` |
| `google_json_key_location` | The JSON key path | `/path/to/gcp-project-12345-abcde.json` |
NOTE: **Note:**
The service account must have permission to access the bucket.
[See more](https://cloud.google.com/storage/docs/authentication)
Here is a configuration example with Rackspace Cloud Files.
| Setting | Description | example |
|---------|-------------|---------|
| `provider` | The provider name | `Rackspace` |
| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` |
| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` |
| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` |
| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` |
NOTE: **Note:**
Regardless of whether the container has public access enabled or disabled, Fog will
use the TempURL method to grant access to LFS objects. If you see errors in logs referencing
instantiating storage with a `temp-url-key`, ensure that you have set the key properly
on the Rackspace API and in `gitlab.rb`. You can verify the value of the key Rackspace
has set by sending a GET request with token header to the service access endpoint URL
and comparing the output of the returned headers.
### Manual uploading to an object storage
There are two ways to manually do the same thing as automatic uploading (described above).
......
doc/administration/merge_request_diffs.md
View file @ c2ff5a0b
......@@ -72,6 +72,10 @@ be configured already.
## Object Storage Settings
NOTE: **Note** In GitLab 13.2 and later, we recommend using the
[consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration).
This section describes the earlier configuration format.
For source installations, these settings are nested under `external_diffs:` and
then `object_store:`. On Omnibus installations, they are prefixed by
`external_diffs_object_store_`.
......@@ -87,20 +91,7 @@ then `object_store:`. On Omnibus installations, they are prefixed by
### S3 compatible connection settings
The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
| Setting | Description | Default |
|---------|-------------|---------|
| `provider` | Always `AWS` for compatible hosts | AWS |
| `aws_access_key_id` | AWS credentials, or compatible | |
| `aws_secret_access_key` | AWS credentials, or compatible | |
| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 |
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true |
| `region` | AWS region | us-east-1 |
| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
See [the available connection settings for different providers](object_storage.md#connection-settings).
**In Omnibus installations:**
......
doc/administration/object_storage.md
View file @ c2ff5a0b
This diff is collapsed.
doc/administration/packages/dependency_proxy.md
View file @ c2ff5a0b
......@@ -87,6 +87,10 @@ store the blobs of the dependency proxy.
[Read more about using object storage with GitLab](../object_storage.md).
NOTE: **Note** In GitLab 13.2 and later, we recommend using the
[consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration).
This section describes the earlier configuration format.
**Omnibus GitLab installations**
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where
......
doc/administration/packages/index.md
View file @ c2ff5a0b
......@@ -99,6 +99,8 @@ store packages.
[Read more about using object storage with GitLab](../object_storage.md).
NOTE: **Note** We recommend using the [consolidated object storage settings](../object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original config format.
**Omnibus GitLab installations**
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where
......
doc/administration/terraform_state.md
View file @ c2ff5a0b
......@@ -68,20 +68,7 @@ The following settings are:
### S3-compatible connection settings
The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
| Setting | Description | Default |
|---------|-------------|---------|
| `provider` | Always `AWS` for compatible hosts | `AWS` |
| `aws_access_key_id` | Credentials for AWS or compatible provider | |
| `aws_secret_access_key` | Credentials for AWS or compatible provider | |
| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 |
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | `true` |
| `region` | AWS region | us-east-1 |
| `host` | S3-compatible host when not using AWS. For example, `localhost` or `storage.example.com` | `s3.amazonaws.com` |
| `endpoint` | Can be used when configuring an S3-compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | `false` |
| `use_iam_profile` | For AWS S3, set to true to use an IAM profile instead of access keys | `false` |
See [the available connection settings for different providers](object_storage.md#connection-settings).
**In Omnibus installations:**
......
doc/administration/uploads.md
View file @ c2ff5a0b
......@@ -57,6 +57,8 @@ This configuration relies on valid AWS credentials to be configured already.
[Read more about using object storage with GitLab](object_storage.md).
NOTE: **Note** We recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original config format.
## Object Storage Settings
For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`.
......@@ -70,22 +72,9 @@ For source installations the following settings are nested under `uploads:` and
| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
| `connection` | Various connection options described below | |
### S3-compatible connection settings
The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
### Connection settings
| Setting | Description | Default |
|---------|-------------|---------|
| `provider` | Always `AWS` for compatible hosts | `AWS` |
| `aws_access_key_id` | AWS credentials, or compatible | |
| `aws_secret_access_key` | AWS credentials, or compatible | |
| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` |
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` |
| `region` | AWS region | us-east-1 |
| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | `s3.amazonaws.com` |
| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false` |
| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys | false
See [the available connection settings for different providers](object_storage.md#connection-settings).
**In Omnibus installations:**
......@@ -143,35 +132,7 @@ _The uploads are stored by default in
1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md).
### Oracle Cloud S3 connection settings
Note that Oracle Cloud S3 must be sure to use the following settings:
| Setting | Value |
|---------|-------|
| `enable_signature_v4_streaming` | `false` |
| `path_style` | `true` |
If `enable_signature_v4_streaming` is set to `true`, you may see the
following error:
```plaintext
STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported
```
### OpenStack compatible connection settings
The connection settings match those provided by [Fog](https://github.com/fog), and are as follows:
| Setting | Description | Default |
|---------|-------------|---------|
| `provider` | Always `OpenStack` for compatible hosts | `OpenStack` |
| `openstack_username` | OpenStack username | |
| `openstack_api_key` | OpenStack API key | |
| `openstack_temp_url_key` | OpenStack key for generating temporary URLs | |
| `openstack_auth_url` | OpenStack authentication endpoint | |
| `openstack_region` | OpenStack region | |
| `openstack_tenant` | OpenStack tenant ID |
### OpenStack example
**In Omnibus installations:**
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7