- 05 Oct, 2016 1 commit
-
-
Jacob Vosmaer (GitLab) authored
Allow to queue API requests and limit given capacity This MR implements an API queueing on Workhorse side. It's meant to better control given capacity for different resources. This is meant to solve: https://gitlab.com/gitlab-com/infrastructure/issues/320. And make a large number of requests easier to handle: https://gitlab.com/gitlab-org/gitlab-ce/issues/21698 It fulfils these requirements: - allow to limit capacity given to API, specifically to allow to process up to N-number of requests at single time, - allow to queue API requests and timeout them, specifically it allows to slow down processing of API calls if the Unicorn can process the current API requests in reasonable time The implementation is made as constant cost and it's dead simple. It should not inflate the memory / CPU usage of Workhorse. It works like this: - we hook into processing of requests, - we try to acquire slot for our request by pushing to buffered channel. The buffered channel actually limits number of processed requests at single time, - if we can't push to channel it means that all concurrent slots are in use and we have to wait, - we block on buffered channel for the free a slot, secondly we wait on timer to timeout on channel, - we generate 502 if timeout occurs, - we process request if we manage to push to channel, - we pop from channel when we finish processing of requests, allowing other requests to fire, - if there's already too many request (over `apiQueueLimit`) we return 429, This introduces 3 extra parameters (off by default): - `apiLimit` - limit number of concurrent API requests, - `apiQueueLimit` - limit the backlog for queueing, - `apiQueueTimeout` - duration after we timeout requests if they sit too long in queue. This allows: - limit used capacity to any number of available workers, ex. allowing for API to use at most 25% of capacity, - slowly process requests in case of slowness, - better manage the API calls then rate limiting requests, - by slowing down we are automatically backing off all services using API, See merge request !65
-
- 04 Oct, 2016 6 commits
-
-
Kamil Trzcinski authored
-
Kamil Trzcinski authored
-
Kamil Trzcinski authored
-
Kamil Trzcinski authored
-
Kamil Trzcinski authored
-
Kamil Trzcinski authored
Expose three options to configure queueing limits: - apiLimit - apiQueueLimit - apiQueueTimeout
-
- 03 Oct, 2016 2 commits
-
-
Jacob Vosmaer (GitLab) authored
Suppress upload-pack Wait() error on shallow clone Closes https://gitlab.com/gitlab-org/gitlab-workhorse/issues/36 It is a [known bug](https://gitlab.com/gitlab-org/gitlab-workhorse/issues/36#note_4761298) in git-upload-pack that it exits unsuccessfully when the client does a 'shallow clone'. We don't want this non-zero process exit status to clutter our logs. In this change we buffer the first 4kB of the request body for git upload-pack and scan for a 'deepen' message (which indicates the clients wants to do a shallow clone). If deepen is found we suppress error reporting for a non-zero exit status of git upload-pack. See merge request !71
-
Jacob Vosmaer authored
-
- 30 Sep, 2016 6 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
- 27 Sep, 2016 1 commit
-
-
Jacob Vosmaer (GitLab) authored
Improve errors - Add request metadata (method, path) - Sentry improvement: custom types for common errors Closes https://gitlab.com/gitlab-org/gitlab-workhorse/issues/64 See merge request !69
-
- 23 Sep, 2016 2 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
- 22 Sep, 2016 1 commit
-
-
Jacob Vosmaer authored
-
- 21 Sep, 2016 2 commits
-
-
-
Jacob Vosmaer authored
- Add request metadata (method, path) - Sentry improvement: custom types for common errors
-
- 16 Sep, 2016 4 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer (GitLab) authored
Support more archive extensions in SendArchive Closes https://gitlab.com/gitlab-org/gitlab-workhorse/issues/60 See merge request !66
-
-
- 12 Sep, 2016 1 commit
-
-
Jacob Vosmaer authored
-
- 09 Sep, 2016 8 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer (GitLab) authored
Pass version as 'release' to Sentry See merge request !64
-
Jacob Vosmaer authored
-
Jacob Vosmaer (GitLab) authored
Add Sentry remote error collection Closes https://gitlab.com/gitlab-org/gitlab-workhorse/issues/57 Vendors: - https://github.com/getsentry/raven-go (unnamed permissive license) - https://github.com/certifi/gocertifi (Mozilla Public License 2.0) Discussed licenses with @JobV; they seem OK. See merge request !63
-
Jacob Vosmaer authored
-
- 08 Sep, 2016 3 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-
Jacob Vosmaer (GitLab) authored
API safeties Increase security and sanity checks to allow for new features. - ability to exchange signed information with gitlab-rails via a shared secret - content-type checks to prevent leaking internal API data Signed messages are implemented using JWT. To make this possible we vendor the jwt-go library. Companion MR in gitlab-rails: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5907 See merge request !60
-
- 05 Sep, 2016 1 commit
-
-
Jacob Vosmaer authored
-
- 29 Aug, 2016 2 commits
-
-
Jacob Vosmaer authored
-
Jacob Vosmaer authored
-