[ci skip]

Support Workhorse directly uploading files to S3 [3/3]

See merge request gitlab-org/gitlab-workhorse!466

This adds the AWS client directly to Workhorse and a new configuration
section for specifying credentials. This makes it possible to use S3
buckets with KMS encryption and proper MD5 checksums.

This is disabled by default. For this to be used:

1. GitLab Rails needs to send the `UseWorkhorseClient` and
`RemoteTempObjectID` in the `/authorize`
endpoint. (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29389)

2. S3 configuration must be specified in `config.toml`, or Rails must be
configured to use IAM instance profiles (`use_iam_profile` in Fog
connection parameters).

S3 sessions are created lazily and cached for 10 minutes to avoid
unnecessary local I/O access. When IAM instance profiles are used, this
also cuts down the number of HTTP requests needed to request AWS
credentials.

Related issues:

1. https://gitlab.com/gitlab-org/gitlab-workhorse/issues/222
2. https://gitlab.com/gitlab-org/gitlab-workhorse/issues/185
3. https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/210

Add configuration to support an S3 client inside Workhorse [2/3]

See merge request gitlab-org/gitlab-workhorse!516

Refactor Preparer and SaveFileOpts handling [1/3]

See merge request gitlab-org/gitlab-workhorse!515

This commit adds the configuration needed to support the Preparer
objects to extract the S3 bucket, endpoint, and other parameters needed
from Rails for Workhorse to upload a file. It does not yet use this
configuration.

This is in preparation for adding an S3 client in
https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466.

This commit ensures that a Preparer is always passed into an upload
handler and eliminates the possibility of a nil handler.

SaveFileOpts is now passed directly to downstream handlers to ensure a
cleaner interface separation between the Preparer and the handlers.

This is split out from the work for
https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466.

Disable compression for openHttpArchive

See merge request gitlab-org/gitlab-workhorse!508

[ci skip]

Gather gitlab-zip-cat/metadata stderr and log output in the current context

Closes #249

See merge request gitlab-org/gitlab-workhorse!498

Accept string or int type for LSIF ids

See merge request gitlab-org/gitlab-workhorse!499

Sync v13.0.1 security release

See merge request gitlab-org/gitlab-workhorse!512

Use labkit 96e583c57891

See merge request gitlab-org/gitlab-workhorse!506

Add test for reading artifact zips over HTTP

See merge request gitlab-org/gitlab-workhorse!509

Define Id type with custom marshalling which accepts either
int or string, because some languages (for example, Javascript)
define ids in LSIF as ints, but some languages (like Golang) use
string type for ids

It also changes int to uint32 for easier binary conversion

Add route for group imports via the UI

See merge request gitlab-org/gitlab-workhorse!493

Group imports via the UI has been implemented in the GitLab application
as part of https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29270.
Ensure that the workhorse configuration is setup for this at the same
time.

Relates to: https://gitlab.com/gitlab-org/gitlab/-/issues/211807

Specify client name when sending requests to Gitaly

See merge request gitlab-org/gitlab-workhorse!505

Release v8.32.0

See merge request gitlab-org/gitlab-workhorse!504

Process LSIF document before sending it to GitLab

See merge request gitlab-org/gitlab-workhorse!492

LSIF document is converted into multiple JSON files and archived
as a ZIP file

https://gitlab.com/gitlab-org/gitlab/-/issues/212384

Delay PostUploadPack response until request is fully read

Closes #258

See merge request gitlab-org/gitlab-workhorse!494

Git and Gitaly stream upload-pack response data as soon as it is
available. Some HTTP clients do not support the response being streamed
back while the server is still reading from the request body. To avoid
this, the request was buffered in to a temporary file before passing it
to Gitaly for handling. The buffer had a maximum size of 10MB, which is
now reached by requests to large repositories. These requests were then
truncated, causing Git to fail.

This commit fixes the problem by removing the request buffering and the
maximum size. Instead, the response is buffered in to a temporary file
until the request body is fully read, thus avoiding the problem of
streaming the request and the response simultaneously.

Update make to generate `testdata/scratch` dir

See merge request gitlab-org/gitlab-workhorse!495

When running the gitaly tests for the first time to see an error message
to run `make prepare-tests` but then after running the tests again you
start seeing the following error message:

```
Error:  Received unexpected error:
listen unix testdata/scratch/gitaly-8674665223082153551.sock: bind: no
such file or directory
```

This is because the directory `testdata/scratch` is not created.

[ci skip]

Add a new JWT signed field for uploads

See merge request gitlab-org/gitlab-workhorse!490

It contains all the parameters generated by workhorse during a file upload