Resize images on-demand with `gm convert`

See merge request gitlab-org/gitlab-workhorse!546

via graphicsmagick

[ci skip]

Add Azure Blob Storage support

See merge request gitlab-org/gitlab-workhorse!555

This merge request introduces a client for Azure Blob Storage in
Workhorse. Currently customers wanting to use Azure Blob Storage have to
set up a Minio Gateway
(https://docs.gitlab.com/charts/advanced/external-object-storage/azure-minio-gateway.html),
which isn't ideal because it requires customers to maintain their own
proxy server for Azure. We have a number of customers who want native
support for Azure Blob Storage.

Unlike AWS and Google, Azure needs to use an Azure client inside
Workhorse to support direct uploads. Using standard HTTP transfers with
pre-signed URLs with the Azure Put Blob API
(https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob)
doesn't work because Azure doesn't support chunked transfer encoding.
However, Azure does support uploading files in segments via the Put
Block and Put Block List API
(https://docs.microsoft.com/en-us/rest/api/storageservices/put-block),
but this requires an Azure client that can speak this API.

Instead of embedding the Microsoft Azure client directly, we use the Go
Cloud Development Kit (https://godoc.org/gocloud.dev/blob) to make it
easier to add other object storage providers later. For example, GitLab
Rails might return this JSON payload in the
`/internal/uploads/authorize` call:

```json
{
   "UseWorkhorseClient":true,
   "ObjectStorage":{
      "Provider":"AzureRM",
      "GoCloudConfig":{
         "URL":"azblob://test-bucket"
      }
   }
}
```

The `azblob` scheme is managed by the Go Cloud `URLMux`
(https://godoc.org/gocloud.dev/blob#URLMux).

Converting our existing S3 client with Go Cloud should be done later
(https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/275).

This changes requires
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38882 to work.

Omnibus configuration changes are in
https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4505.

Part of https://gitlab.com/gitlab-org/gitlab/-/issues/25877

[ci skip]

Add project level route for conan package uploads

See merge request gitlab-org/gitlab-workhorse!558

Security release process

See merge request gitlab-org/gitlab-workhorse!507

Reduce code duplication in LFS upload preparer

See merge request gitlab-org/gitlab-workhorse!560

We reuse the standard object storage upload preparer to avoid
duplication of code and add a test. This is done in prepration for
adding new fields to the filestore options.

Update staticcheck and ignore Protobuf v1 deprecation warnings

See merge request gitlab-org/gitlab-workhorse!559

This update is done in preparation for importing the Go Cloud SDK.  To
fix this properly, we will have to update Gitaly to use Protobuf v2:
https://blog.golang.org/protobuf-apiv2

See https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/274.

Refactor uploaders to use different upload strategies

See merge request gitlab-org/gitlab-workhorse!553

Fix unnecessary fmt.Sprintf (S1039)

See merge request gitlab-org/gitlab-workhorse!556

This was flagged in
https://gitlab.com/gitlab-org/gitlab-workhorse/-/jobs/684053889.

Previously it was particularly tricky to add a new object storage method
because you had to be aware of how to deal with different Goroutines and
contexts to handle the Workhorse upload flow
(https://docs.gitlab.com/ee/development/uploads.html#direct-upload). In
addition, the execution engine to handle this was duplicated across
multiple files. The execution engine essentially did the following:

1. Set up an upload context with a deadline
2. Record upload metrics
3. Initialize cleanup functions
4. Initiate upload
5. Validate upload ETag
6. Do cleanup (e.g. delete the temporary file)

To reduce code duplication and to make it easier to add new object
stores, the common execution sequence is now encapsulated in the
`uploader` `Execute()` method. We also introduce an `UploadStrategy`
interface that handles the details of the uploads, and `Execute()` calls
methods on this interface.

Now adding a new object storage type is a matter of implementing the
`UploadStrategy` interface without needing to understand the details of
the execution engine.

[ci skip]

Cache references in file

See merge request gitlab-org/gitlab-workhorse!544

Fix HTTP Range Requests not working on some S3 providers

Closes gitlab#223806

See merge request gitlab-org/gitlab-workhorse!549

This is to improve RAM usage when parsing references utilizing the
existing `cache` struct.

Reduces memory usage when parsing LSIF file of gitlab-workhorse
from 7MB to 4MB.

According to
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35:

Origin servers that accept byte-range requests MAY send Accept-Ranges:
bytes, but are not required to do so. Clients MAY generate byte-range
requests without having received this header for the resource involved.

Since some servers (e.g. Dell ECS) don't send an `Accept-Ranges: bytes`
header and we are already checking that range requests are supported
when we call `HttpReadSeeker.rangeRequest`, `canSeek` only becomes a
problem and should be removed.

Since it's not clear whether https://github.com/jfbus/httprs is actively
maintained, this commit applies https://github.com/jfbus/httprs/pull/6
to our vendored module.

Closes https://gitlab.com/gitlab-org/gitlab/-/issues/223806

Vendor httprs module

See merge request gitlab-org/gitlab-workhorse!550

Bump Go version in go.mod to v1.13

See merge request gitlab-org/gitlab-workhorse!552

Release v8.38.0

See merge request gitlab-org/gitlab-workhorse!551

Since we dropped support for v1.12 in
https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/532, we
should set the go.mod version to the minimum supported version, which is
Go v1.13.

It's not clear whether this source code is maintained anymore. This is
done in preparation to apply https://github.com/jfbus/httprs/pull/6,
which is 2 years old.

Add References struct

See merge request gitlab-org/gitlab-workhorse!540

Add support for AWS S3 Server Side Encryption (SSE-KMS)

See merge request gitlab-org/gitlab-workhorse!537

Extract references parsing logic to a separate file and struct.
Also modified `Item` struct to have `Line` as `int32` so it'll
be fixed size.

This is in preparation for another MR wherein references will be
stored in file instead of memory.

Prior to this change, uploads to AWS S3 were only encrypted on the
server if a default encryption were specified on the bucket.

With this change, admins can now configure the encryption and the AWS
Key Management Service (KMS) key ID in GitLab Rails, and the
configuration will be used in uploads. Bucket policies to enforce
encryption can now be used since Workhorse sends the required headers
(`x-amz-server-side-encryption` and
`x-amz-server-side-encryption-aws-kms-key-id`). The bucket policy cannot
be enforced with default encryption, since that is applied after the
check.

This requires the changes in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240 to work.

Part of https://gitlab.com/gitlab-org/gitlab/-/issues/22200

Add changelog for MR 529

See merge request gitlab-org/gitlab-workhorse!545

Added configuration option PropagateCorrelationID

See merge request gitlab-org/gitlab-workhorse!529