1. 17 Jun, 2020 1 commit
    • Stan Hu's avatar
      Consolidate object storage config in one place · 95870df2
      Stan Hu authored
      Currently each object type has its own section in gitlab.yml. For
      example:
      
      ```
        artifacts:
          path: tmp/tests/artifacts
          enabled: true
          object_store:
            enabled: false
            remote_directory: artifacts # The bucket name
            background_upload: false
            connection:
              provider: AWS
              aws_access_key_id: AWS_ACCESS_KEY_ID
              aws_secret_access_key: AWS_SECRET_ACCESS_KEY
              region: us-east-1
      ```
      
      External diffs, LFS, uploads, packages, etc. all have similar
      independent configuration object storage sections. While this redundancy
      makes it possible to configure each bucket with different providers or
      credentials, this causes a configuration explosion that makes GitLab
      hard to manage.
      
      This change preserves the legacy format but adds a new `gitlab.yml`
      section that enforces a single, common object storage provider for all
      object storage types. This will make it possible for the S3 client in
      Workhorse to operate with one credential and simplify the configuration
      for the end user. An example config:
      
      ```
        object_store:
          enabled: true
          connection:
            provider: AWS
            aws_access_key_id: AWS_ACCESS_KEY_ID
            aws_secret_access_key: AWS_SECRET_ACCESS_KEY
            region: us-east-1
          proxy_download: true
          objects:
            artifacts:
              bucket: artifacts
              proxy_download: false
            external_diffs:
              bucket: external-diffs
            lfs:
              bucket: lfs-objects
            uploads:
              bucket: uploads
            packages:
              bucket: packages
            dependency_proxy:
              bucket: dependency_proxy
      ```
      
      Note that:
      
      1. The consolidated config only gets used if `object_store` is NOT
         defined within the types themselves.
      2. A bucket needs to be defined for each object type.
      3. Only bucket, enabled, and proxy_download can be overridden from the
      common configuration.
      
      Consolidating support for a single bucket for all types is a larger and
      more involved change.
      
      First step of https://gitlab.com/gitlab-org/gitlab/-/issues/23345
      95870df2
  2. 16 Jun, 2020 39 commits