- 19 Aug, 2020 1 commit
-
-
Stan Hu authored
This merge request introduces a client for Azure Blob Storage in Workhorse. Currently customers wanting to use Azure Blob Storage have to set up a Minio Gateway (https://docs.gitlab.com/charts/advanced/external-object-storage/azure-minio-gateway.html), which isn't ideal because it requires customers to maintain their own proxy server for Azure. We have a number of customers who want native support for Azure Blob Storage. Unlike AWS and Google, Azure needs to use an Azure client inside Workhorse to support direct uploads. Using standard HTTP transfers with pre-signed URLs with the Azure Put Blob API (https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob) doesn't work because Azure doesn't support chunked transfer encoding. However, Azure does support uploading files in segments via the Put Block and Put Block List API (https://docs.microsoft.com/en-us/rest/api/storageservices/put-block), but this requires an Azure client that can speak this API. Instead of embedding the Microsoft Azure client directly, we use the Go Cloud Development Kit (https://godoc.org/gocloud.dev/blob) to make it easier to add other object storage providers later. For example, GitLab Rails might return this JSON payload in the `/internal/uploads/authorize` call: ```json { "UseWorkhorseClient":true, "ObjectStorage":{ "Provider":"AzureRM", "GoCloudConfig":{ "URL":"azblob://test-bucket" } } } ``` The `azblob` scheme is managed by the Go Cloud `URLMux` (https://godoc.org/gocloud.dev/blob#URLMux). Converting our existing S3 client with Go Cloud should be done later (https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/275). This changes requires https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38882 to work. Omnibus configuration changes are in https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4505. Part of https://gitlab.com/gitlab-org/gitlab/-/issues/25877
-
- 18 Aug, 2020 7 commits
-
-
Nick Thomas authored
-
Nick Thomas authored
[ci skip]
-
Nick Thomas authored
Add project level route for conan package uploads See merge request gitlab-org/gitlab-workhorse!558
-
Jacob Vosmaer authored
Security release process See merge request gitlab-org/gitlab-workhorse!507
-
Steve Abrams authored
-
Nick Thomas authored
Reduce code duplication in LFS upload preparer See merge request gitlab-org/gitlab-workhorse!560
-
Stan Hu authored
We reuse the standard object storage upload preparer to avoid duplication of code and add a test. This is done in prepration for adding new fields to the filestore options.
-
- 17 Aug, 2020 1 commit
-
-
Nick Thomas authored
Update staticcheck and ignore Protobuf v1 deprecation warnings See merge request gitlab-org/gitlab-workhorse!559
-
- 15 Aug, 2020 1 commit
-
-
Stan Hu authored
This update is done in preparation for importing the Go Cloud SDK. To fix this properly, we will have to update Gitaly to use Protobuf v2: https://blog.golang.org/protobuf-apiv2 See https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/274.
-
- 13 Aug, 2020 4 commits
-
-
Nick Thomas authored
Refactor uploaders to use different upload strategies See merge request gitlab-org/gitlab-workhorse!553
-
Jacob Vosmaer authored
Fix unnecessary fmt.Sprintf (S1039) See merge request gitlab-org/gitlab-workhorse!556
-
Stan Hu authored
This was flagged in https://gitlab.com/gitlab-org/gitlab-workhorse/-/jobs/684053889.
-
Stan Hu authored
Previously it was particularly tricky to add a new object storage method because you had to be aware of how to deal with different Goroutines and contexts to handle the Workhorse upload flow (https://docs.gitlab.com/ee/development/uploads.html#direct-upload). In addition, the execution engine to handle this was duplicated across multiple files. The execution engine essentially did the following: 1. Set up an upload context with a deadline 2. Record upload metrics 3. Initialize cleanup functions 4. Initiate upload 5. Validate upload ETag 6. Do cleanup (e.g. delete the temporary file) To reduce code duplication and to make it easier to add new object stores, the common execution sequence is now encapsulated in the `uploader` `Execute()` method. We also introduce an `UploadStrategy` interface that handles the details of the uploads, and `Execute()` calls methods on this interface. Now adding a new object storage type is a matter of implementing the `UploadStrategy` interface without needing to understand the details of the execution engine.
-
- 12 Aug, 2020 5 commits
-
-
Nick Thomas authored
-
Nick Thomas authored
[ci skip]
-
Jacob Vosmaer authored
Cache references in file See merge request gitlab-org/gitlab-workhorse!544
-
Jacob Vosmaer authored
Fix HTTP Range Requests not working on some S3 providers Closes gitlab#223806 See merge request gitlab-org/gitlab-workhorse!549
-
Patrick Bajao authored
This is to improve RAM usage when parsing references utilizing the existing `cache` struct. Reduces memory usage when parsing LSIF file of gitlab-workhorse from 7MB to 4MB.
-
- 11 Aug, 2020 4 commits
-
-
Stan Hu authored
According to https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35: Origin servers that accept byte-range requests MAY send Accept-Ranges: bytes, but are not required to do so. Clients MAY generate byte-range requests without having received this header for the resource involved. Since some servers (e.g. Dell ECS) don't send an `Accept-Ranges: bytes` header and we are already checking that range requests are supported when we call `HttpReadSeeker.rangeRequest`, `canSeek` only becomes a problem and should be removed. Since it's not clear whether https://github.com/jfbus/httprs is actively maintained, this commit applies https://github.com/jfbus/httprs/pull/6 to our vendored module. Closes https://gitlab.com/gitlab-org/gitlab/-/issues/223806
-
Jacob Vosmaer authored
Vendor httprs module See merge request gitlab-org/gitlab-workhorse!550
-
Nick Thomas authored
Bump Go version in go.mod to v1.13 See merge request gitlab-org/gitlab-workhorse!552
-
Nick Thomas authored
Release v8.38.0 See merge request gitlab-org/gitlab-workhorse!551
-
- 10 Aug, 2020 7 commits
-
-
Stan Hu authored
-
Stan Hu authored
-
Stan Hu authored
Since we dropped support for v1.12 in https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/532, we should set the go.mod version to the minimum supported version, which is Go v1.13.
-
Stan Hu authored
-
Stan Hu authored
It's not clear whether this source code is maintained anymore. This is done in preparation to apply https://github.com/jfbus/httprs/pull/6, which is 2 years old.
-
Jacob Vosmaer authored
Add References struct See merge request gitlab-org/gitlab-workhorse!540
-
Jacob Vosmaer authored
Add support for AWS S3 Server Side Encryption (SSE-KMS) See merge request gitlab-org/gitlab-workhorse!537
-
- 07 Aug, 2020 1 commit
-
-
Patrick Bajao authored
Extract references parsing logic to a separate file and struct. Also modified `Item` struct to have `Line` as `int32` so it'll be fixed size. This is in preparation for another MR wherein references will be stored in file instead of memory.
-
- 06 Aug, 2020 5 commits
-
-
Stan Hu authored
Prior to this change, uploads to AWS S3 were only encrypted on the server if a default encryption were specified on the bucket. With this change, admins can now configure the encryption and the AWS Key Management Service (KMS) key ID in GitLab Rails, and the configuration will be used in uploads. Bucket policies to enforce encryption can now be used since Workhorse sends the required headers (`x-amz-server-side-encryption` and `x-amz-server-side-encryption-aws-kms-key-id`). The bucket policy cannot be enforced with default encryption, since that is applied after the check. This requires the changes in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240 to work. Part of https://gitlab.com/gitlab-org/gitlab/-/issues/22200
-
Nick Thomas authored
Add changelog for MR 529 See merge request gitlab-org/gitlab-workhorse!545
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
Added configuration option PropagateCorrelationID See merge request gitlab-org/gitlab-workhorse!529
-
Mahmoud Rahbar Azad authored
added configuration option PropagateCorrelationID and corresponding flag argument propagateCorrelationID to enable correlation.InboundHandlerOption: WithPropagation() for correlation.InjectCorrelationID
-
- 31 Jul, 2020 1 commit
-
-
Alessio Caiazza authored
Improve test report for performance_test.go See merge request gitlab-org/gitlab-workhorse!535
-
- 30 Jul, 2020 1 commit
-
-
Igor Drozdov authored
-
- 24 Jul, 2020 1 commit
-
-
Jacob Vosmaer authored
Drop Go v1.12 support See merge request gitlab-org/gitlab-workhorse!532
-
- 23 Jul, 2020 1 commit
-
-
Igor Drozdov authored
We officially use Go v1.13+ for everything (https://docs.gitlab.com/ee/install/requirements.html#go-versions), and dropping Go v1.12 allows us to use new methods such as `http.NewRequestWithContext`.
-