Add Object Storage blueprint

doc/architecture/blueprints/object_storage/index.md

+---
+stage: none
+group: unassigned
+comments: false
+description: 'Object storage: direct_upload consolidation - architecture blueprint.'
+---
+
+# Object storage: `direct_upload` consolidation
+
+## Abstract
+
+GitLab stores three classes of user data: database records, Git
+repositories, and user-uploaded files (which are referred to as
+file storage throughout the blueprint).
+
+The user and contributor experience for our file
+storage has room for significant improvement:
+
+- Initial GitLab setup experience requires creation and setup of 13
+  buckets, instead of just 1.
+- Features using file storage require contributors to think about both local
+  storage and object storage, which leads to friction and
+  complexity. This often results in broken features and security issues.
+- Contributors who work on file storage often also have to write code
+  for Workhorse, Omnibus, and cloud native GitLab (CNG).
+
+## Problem definition
+
+Object storage is a fundamental component of GitLab, providing the
+underlying implementation for shared, distributed, highly-available
+(HA) file storage.
+
+Over time, we have built support for object storage across the
+application, solving specific problems in a [multitude of
+iterations](https://about.gitlab.com/company/team/structure/working-groups/object-storage/#company-efforts-on-uploads). This
+has led to increased complexity across the board, from development
+(new features and bug fixes) to installation:
+
+- New GitLab installations require the creation and configuration of
+  several object storage buckets instead of just one, as each group of
+  features requires its own. This has an impact on the installation
+  experience and new feature adoption, and takes us further away from
+  boring solutions.
+- The release of cloud native GitLab required the removal of NFS
+  shared storage and the development of direct upload, a feature that
+  was expanded, milestone after milestone, to several type of uploads,
+  but never enabled globally.
+- Today, GitLab supports both local storage and object storage. Local
+  storage only works on single box installations or with a NFS, which
+  [we no longer recommend](../../../administration/nfs.md) to our
+  users and is no longer in use on GitLab.com.
+- Understanding all the moving parts and the flow is extremely
+  complicated: we have CarrierWave, Fog, Golang S3/Azure SDKs, all
+  being used, and that complicates testing as well.
+- Fog and CarrierWave are not maintained to the level of the native
+  SDKs (for example, AWS S3 SDK), so we have to maintain or monkey
+  patch those tools to support requested customer features
+  (for example, [issue #242245](https://gitlab.com/gitlab-org/gitlab/-/issues/242245))
+  that would normally be "free".
+- In many cases, we copy around object storage files needlessly
+  (for example, [issue #285597](https://gitlab.com/gitlab-org/gitlab/-/issues/285597)).
+  Large files (LFS, packages, and so on) are slow to finalize or don't work
+  at all as a result.
+
+## Improvements over the current situation
+
+The following is a brief description of the main directions we can take to
+remove the pain points affecting our object storage implementation.
+
+This is also available as [a YouTube
+video](https://youtu.be/X9V_w8hsM8E) recorded for the [Object Storage
+Working
+Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/).
+
+### Simplify GitLab architecture by shipping MinIO
+
+In the beginning, object storage support was a Premium feature, not
+part of our CE distribution. Because of that, we had to support both
+local storage and object storage.
+
+With local storage, there is the assumption of a shared storage
+between components. This can be achieved by having a single box
+installation, without HA, or with a NFS, which [we no longer
+recommend](../../../administration/nfs.md).
+
+We have a testing gap on object storage. It also requires Workhorse
+and MinIO, which are not present in our pipelines, so too much is
+replaced by a mock implementation. Furthermore, the presence of a
+shared disk, both in CI and in local development, often hides broken
+implementations until we deploy on an HA environment.
+
+Shipping MinIO as part of the product will reduce the differences
+between a cloud and a local installation, standardizing our file
+storage on a single technology.
+
+The removal of local disk operations will reduce the complexity of
+development as well as mitigate several security attack vectors as
+we no longer write user-provided data on the local storage.
+
+It will also reduce human errors as we will always run a local object
+storage in development mode and any local file disk access should
+raise a red flag during the merge request review.
+
+This effort is described in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6099).
+
+### Enable direct upload by default on every upload
+
+Because every group of features requires its own bucket, we don't have
+direct upload enabled everywhere. Contributing a new upload requires
+coding it in both Ruby on Rails and Go.
+
+Implementing a new feature that does not yet have a dedicated bucket
+requires the developer to also create a merge request in Omnibus
+and CNG, as well as coordinate with SREs to configure the new bucket
+for our own environments.
+
+This also slows down feature adoptions, because our users need to
+reconfigure GitLab and prepare a new bucket in their
+infrastructure. It also makes the initial installation more complex
+feature after feature.
+
+Implementing a direct upload by default, with a
+[consolidated object storage configuration](../../../administration/object_storage.md#consolidated-object-storage-configuration)
+will reduce the number of merge requests needed to ship a new feature
+from four to only one. It will also remove the need for SRE
+intervention as the bucket will always be the same.
+
+This will simplify our development and review processes, as well as
+the GitLab configuration file. And every user will immediately have
+access to new features without infrastructure chores.
+
+### Simplify object storage code
+
+Our implementation is built on top of a 3rd-party framework where
+every object storage client is a 3rd-party library. Unfortunately some
+of them are unmaintained. [We have customers who cannot push 5GB Git
+LFS objects](https://gitlab.com/gitlab-org/gitlab/-/issues/216442),
+but with such a vital feature implemented in 3rd-party libraries we
+are slowed down in fixing it, and we also rely on external maintainers
+to merge and release fixes.
+
+Before the introduction of direct upload, using the
+[CarrierWave](https://github.com/carrierwaveuploader/carrierwave)
+library, _"a gem that provides a simple and extremely flexible way to
+upload files from Ruby applications."_, was the boring solution.
+However this is no longer our use-case, as we upload files from
+Workhorse, and we had to [patch CarrierWave's
+internals](https://gitlab.com/gitlab-org/gitlab/-/issues/285597#note_452696638)
+to support direct upload.
+
+A brief proposal covering CarrierWave removal and a new streamlined
+internal upload API is described
+[in this issue comment](https://gitlab.com/gitlab-org/gitlab/-/issues/213288#note_325358026).
+
+Ideally, we wouldn't need to duplicate object storage clients in Go
+and Ruby. By removing CarrierWave, we can make use of the officially
+supported native clients when the provider S3 compatibility level is
+not sufficient.
+
+## Iterations
+
+In this section we list some possible iterations. This is not
+intended to be the final roadmap, but is a conversation started for the
+Object Storage Working Group.
+
+1. Create a new catchall bucket and a unified internal API for
+   authorization without CarrierWave.
+1. Ship MinIO with Omnibus (CNG images already include it).
+1. Expand GitLab-QA to cover all the supported configurations.
+1. Deprecate local disk access.
+1. Deprecate configurations with multiple buckets.
+1. Implement a bucket-to-bucket migration.
+1. Migrate the current CarrierWave uploads to the new implementation.
+1. On the next major release: Remove support for local disk access and
+   configurations with multiple buckets.
+
+### Benefits of the current iteration plan
+
+The current plan is designed to provide tangible benefits from the
+first step.
+
+With the introduction of the catchall bucket, every upload currently
+not subject to direct upload will get its benefits, and new features
+could be shipped with a single merge request.
+
+Shipping MinIO with Omnibus will allow us to default new installations
+to object storage, and Omnibus could take care of creating
+buckets. This will simplify HA installation outside of Kubernetes.
+
+Then we can migrate each CarrierWave uploader to the new
+implementation, up to a point where GitLab installation will only
+require one bucket.
+
+## Additional reading materials
+
+- [Uploads development documentation: The problem description](../../../development/uploads.md#the-problem-description).
+- [Speed up the monolith, building a smart reverse proxy in Go](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/): a presentation explaining a bit of workhorse history and the challenge we faced in releasing the first cloud-native installation.
+- [Object Storage improvements epic](https://gitlab.com/groups/gitlab-org/-/epics/483).
+- We are moving to GraphQL API, but [we do not support direct upload](https://gitlab.com/gitlab-org/gitlab/-/issues/280819).
+
+## Who
+
+Proposal:
+
+<!-- vale gitlab.Spelling = NO -->
+
+| Role                           | Who                     |
+|--------------------------------|-------------------------|
+| Author                         | Alessio Caiazza         |
+| Architecture Evolution Coach   | Gerardo Lopez-Fernandez |
+| Engineering Leader             | Marin Jankovski         |
+| Domain Expert / Object storage | Stan Hu                 |
+| Domain Expert / Security       | Joern Schneeweisz       |
+
+DRIs:
+
+The DRI for this blueprint is the [Object Storage Working
+Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/).
+
+<!-- vale gitlab.Spelling = YES -->