Commit a4a33415 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'docs-bprescott-objstorage-20200407' into 'master'

Improve object storage page and link to it

See merge request gitlab-org/gitlab!29075
parents e4c8de23 98058990
......@@ -73,7 +73,7 @@ for Wiki and Design Repository cases.
GitLab stores files and blobs such as Issue attachments or LFS objects into either:
- The filesystem in a specific location.
- An Object Storage solution. Object Storage solutions can be:
- An [Object Storage](../../object_storage.md) solution. Object Storage solutions can be:
- Cloud based like Amazon S3 Google Cloud Storage.
- Hosted by you (like MinIO).
- A Storage Appliance that exposes an Object Storage-compatible API.
......
......@@ -12,6 +12,8 @@ To have:
- GitLab manage replication, follow [Enabling GitLab replication](#enabling-gitlab-managed-object-storage-replication).
- Third-party services manage replication, follow [Third-party replication services](#third-party-replication-services).
[Read more about using object storage with GitLab](../../object_storage.md).
## Enabling GitLab managed object storage replication
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/10586) in GitLab 12.4.
......
......@@ -92,6 +92,8 @@ Use an object storage option like AWS S3 to store job artifacts.
DANGER: **Danger:**
If you configure GitLab to store CI logs and artifacts on object storage, you must also enable [incremental logging](job_logs.md#new-incremental-logging-architecture). Otherwise, job logs will disappear or not be saved.
[Read more about using object storage with GitLab](object_storage.md).
#### Object Storage Settings
For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`.
......
......@@ -61,6 +61,8 @@ You can also use external object storage in a private local network. For example
GitLab provides two different options for the uploading mechanism: "Direct upload" and "Background upload".
[Read more about using object storage with GitLab](../object_storage.md).
**Option 1. Direct upload**
1. User pushes an `lfs` file to the GitLab instance
......
......@@ -68,6 +68,8 @@ Instead of storing the external diffs on disk, we recommended the use of an obje
store like AWS S3 instead. This configuration relies on valid AWS credentials to
be configured already.
[Read more about using object storage with GitLab](object_storage.md).
## Object Storage Settings
For source installations, these settings are nested under `external_diffs:` and
......
......@@ -21,9 +21,6 @@ Object storage options that GitLab has tested, or is aware of customers using in
For configuring GitLab to use Object Storage refer to the following guides:
1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk.
1. Configure [database lookup of SSH keys](operations/fast_ssh_key_lookup.md)
to eliminate the need for a shared `authorized_keys` file.
1. Configure [object storage for backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage).
1. Configure [object storage for job artifacts](job_artifacts.md#using-object-storage)
including [incremental logging](job_logs.md#new-incremental-logging-architecture).
......@@ -36,6 +33,19 @@ For configuring GitLab to use Object Storage refer to the following guides:
1. Configure [object storage for Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)**
1. Configure [object storage for Pseudonymizer](pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)**
1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance).
1. Configure [object storage for Terraform state files](terraform_state.md#using-object-storage-core-only)
### Other alternatives to filesystem storage
If you're working to [scale out](scaling/index.md) your GitLab implementation,
or add [fault tolerance and redundancy](high_availability/README.md) you may be
looking at removing dependencies on block or network filesystems.
See the following guides and
[note that Pages requires disk storage](#gitlab-pages-requires-nfs):
1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk.
1. Configure [database lookup of SSH keys](operations/fast_ssh_key_lookup.md)
to eliminate the need for a shared `authorized_keys` file.
## Warnings, limitations, and known issues
......@@ -67,8 +77,9 @@ with the Fog library that GitLab uses. Symptoms include:
### GitLab Pages requires NFS
If you're working to [scale out](high_availability/README.md) your GitLab implementation and
one of your requirements is [GitLab Pages](../user/project/pages/index.md) this currently requires
If you're working to add more GitLab servers for [scaling](scaling/index.md) or
[fault tolerance](high_availability/README.md) and one of your requirements
is [GitLab Pages](../user/project/pages/index.md) this currently requires
NFS. There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196)
to remove this dependency. In the future, GitLab Pages may use
[object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/208135).
......
......@@ -367,6 +367,8 @@ The different supported drivers are:
Read more about the individual driver's config options in the
[Docker Registry docs](https://docs.docker.com/registry/configuration/#storage).
[Read more about using object storage with GitLab](../object_storage.md).
CAUTION: **Warning:** GitLab will not backup Docker images that are not stored on the
filesystem. Remember to enable backups with your object storage provider if
desired.
......
......@@ -77,7 +77,9 @@ To change the local storage path:
### Using object storage
Instead of relying on the local storage, you can use an object storage to
upload the blobs of the dependency proxy:
store the blobs of the dependency proxy.
[Read more about using object storage with GitLab](../object_storage.md).
**Omnibus GitLab installations**
......
......@@ -86,7 +86,9 @@ To change the local storage path:
### Using object storage
Instead of relying on the local storage, you can use an object storage to
upload packages:
store packages.
[Read more about using object storage with GitLab](../object_storage.md).
**Omnibus GitLab installations**
......
......@@ -26,6 +26,8 @@ To configure the pseudonymizer, you need to:
Alternatively, you can use an absolute file path.
- Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option.
[Read more about using object storage with GitLab](object_storage.md).
**For Omnibus installations:**
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
......
......@@ -7,6 +7,8 @@ After [configuring the object storage](../../uploads.md#using-object-storage-cor
>**Note:**
All of the processing will be done in a background worker and requires **no downtime**.
[Read more about using object storage with GitLab](../../object_storage.md).
### All-in-one Rake task
GitLab provides a wrapper Rake task that migrates all uploaded files - avatars,
......
......@@ -51,6 +51,8 @@ Instead of storing Terraform state files on disk, we recommend the use of an obj
store that is S3-compatible instead. This configuration relies on valid credentials to
be configured already.
[Read more about using object storage with GitLab](object_storage.md).
### Object storage settings
The following settings are:
......
......@@ -55,6 +55,8 @@ If you don't want to use the local disk where GitLab is installed to store the
uploads, you can use an object storage provider like AWS S3 instead.
This configuration relies on valid AWS credentials to be configured already.
[Read more about using object storage with GitLab](object_storage.md).
## Object Storage Settings
For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`.
......
......@@ -555,7 +555,7 @@ In `/etc/ssh/sshd_config` update the following:
#### Amazon S3 object storage
Since we're not using NFS for shared storage, we will use [Amazon S3](https://aws.amazon.com/s3/) buckets to store backups, artifacts, LFS objects, uploads, merge request diffs, container registry images, and more. For instructions on how to configure each of these, please see [Cloud Object Storage](../../administration/high_availability/object_storage.md).
Since we're not using NFS for shared storage, we will use [Amazon S3](https://aws.amazon.com/s3/) buckets to store backups, artifacts, LFS objects, uploads, merge request diffs, container registry images, and more. Our [documentation includes configuration instructions](../../administration/object_storage.md) for each of these, and other information about using object storage with GitLab.
Remember to run `sudo gitlab-ctl reconfigure` after saving the changes to the `gitlab.rb` file.
......
......@@ -309,6 +309,8 @@ In the example below we use Amazon S3 for storage, but Fog also lets you use
for AWS, Google, OpenStack Swift, Rackspace and Aliyun as well. A local driver is
[also available](#uploading-to-locally-mounted-shares).
[Read more about using object storage with GitLab](../administration/object_storage.md).
#### Using Amazon S3
For Omnibus GitLab packages:
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment