Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
e840749b
Commit
e840749b
authored
Nov 10, 2016
by
Patricio Cano
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Refactored Sidekiq Throttler and updated documentation
parent
20853049
Changes
6
Show whitespace changes
Inline
Side-by-side
Showing
6 changed files
with
20 additions
and
23 deletions
+20
-23
app/models/application_setting.rb
app/models/application_setting.rb
+1
-1
app/views/admin/application_settings/_form.html.haml
app/views/admin/application_settings/_form.html.haml
+2
-2
changelogs/unreleased/sidekiq-job-throttling.yml
changelogs/unreleased/sidekiq-job-throttling.yml
+1
-1
doc/administration/operations/sidekiq_job_throttling.md
doc/administration/operations/sidekiq_job_throttling.md
+8
-7
lib/gitlab/sidekiq_throttler.rb
lib/gitlab/sidekiq_throttler.rb
+8
-6
spec/lib/gitlab/sidekiq_throttler_spec.rb
spec/lib/gitlab/sidekiq_throttler_spec.rb
+0
-6
No files found.
app/models/application_setting.rb
View file @
e840749b
...
@@ -19,7 +19,7 @@ class ApplicationSetting < ActiveRecord::Base
...
@@ -19,7 +19,7 @@ class ApplicationSetting < ActiveRecord::Base
serialize
:domain_whitelist
,
Array
serialize
:domain_whitelist
,
Array
serialize
:domain_blacklist
,
Array
serialize
:domain_blacklist
,
Array
serialize
:repository_storages
serialize
:repository_storages
serialize
:sidekiq_throttling_queues
serialize
:sidekiq_throttling_queues
,
Array
cache_markdown_field
:sign_in_text
cache_markdown_field
:sign_in_text
cache_markdown_field
:help_page_text
cache_markdown_field
:help_page_text
...
...
app/views/admin/application_settings/_form.html.haml
View file @
e840749b
...
@@ -304,9 +304,9 @@
...
@@ -304,9 +304,9 @@
.form-group
.form-group
=
f
.
label
:sidekiq_throttling_factor
,
'Throttling Factor'
,
class:
'control-label col-sm-2'
=
f
.
label
:sidekiq_throttling_factor
,
'Throttling Factor'
,
class:
'control-label col-sm-2'
.col-sm-10
.col-sm-10
=
f
.
number_field
:sidekiq_throttling_factor
,
class:
'form-control'
,
min:
'0'
,
max:
'0.99'
,
step:
'0.01'
=
f
.
number_field
:sidekiq_throttling_factor
,
class:
'form-control'
,
min:
'0
.01
'
,
max:
'0.99'
,
step:
'0.01'
.help-block
.help-block
The factor by which the queues should be throttled. A value between 0.
1 and 0.9
.
The factor by which the queues should be throttled. A value between 0.
0 and 1.0, exclusive
.
%fieldset
%fieldset
%legend
Spam and Anti-bot Protection
%legend
Spam and Anti-bot Protection
...
...
changelogs/unreleased/sidekiq-job-throttling.yml
View file @
e840749b
---
---
title
:
Added ability to throttle Sidekiq Jobs
title
:
Added ability to throttle Sidekiq Jobs
merge_request
:
7292
merge_request
:
7292
author
:
Patricio Cano
author
:
doc/administration/operations/sidekiq_job_throttling.md
View file @
e840749b
...
@@ -3,14 +3,15 @@
...
@@ -3,14 +3,15 @@
> Note: Introduced with GitLab 8.14
> Note: Introduced with GitLab 8.14
When your GitLab installation needs to handle tens of thousands of background
When your GitLab installation needs to handle tens of thousands of background
jobs, it can be convenient to prioritize queues that need to be executed
jobs, it can be convenient to throttle queues that do not need to be executed
immediately, e.g. user initiated actions like merging a Merge Request.
immediately, e.g. long running jobs like Pipelines, thus allowing jobs that do
need to be executed immediately to have access to more resources.
In order to accomplish this, you can limit the amount of workers that certain
In order to accomplish this, you can limit the amount of workers that certain
slow running queues
get
can have available. This is what we call Sidekiq Job
slow running queues can have available. This is what we call Sidekiq Job
Throttling. Depending on your infrastructure, you might have different slow
Throttling. Depending on your infrastructure, you might have different slow
running queues, which is why you can choose which queues
to throttle and by
running queues, which is why you can choose which queues
you want to throttle
how much you want to throttle them.
and by
how much you want to throttle them.
These settings are available in the Application Settings of your GitLab
These settings are available in the Application Settings of your GitLab
installation.
installation.
...
@@ -24,8 +25,8 @@ and rounded up to the closest full integer.
...
@@ -24,8 +25,8 @@ and rounded up to the closest full integer.
So, for example, you set the
`:concurrency`
to 25 and the
`Throttling factor`
to
So, for example, you set the
`:concurrency`
to 25 and the
`Throttling factor`
to
0.
1, the maximum workers assigned to the selected queues would be 3.
0.
1, the maximum workers assigned to the selected queues would be 3.
```
```
ruby
limit = (factor * Sidekiq.options[:concurrency]).ceil
queue_
limit
=
(
factor
*
Sidekiq
.
options
[
:concurrency
]).
ceil
```
```
After enabling the job throttling, you will need to restart your GitLab
After enabling the job throttling, you will need to restart your GitLab
...
...
lib/gitlab/sidekiq_throttler.rb
View file @
e840749b
...
@@ -3,19 +3,21 @@ module Gitlab
...
@@ -3,19 +3,21 @@ module Gitlab
class
<<
self
class
<<
self
def
execute!
def
execute!
if
Gitlab
::
CurrentSettings
.
sidekiq_throttling_enabled?
if
Gitlab
::
CurrentSettings
.
sidekiq_throttling_enabled?
current_application_settings
.
sidekiq_throttling_queues
.
each
do
|
queue
|
Gitlab
::
CurrentSettings
.
current_application_settings
.
sidekiq_throttling_queues
.
each
do
|
queue
|
Sidekiq
::
Queue
[
queue
].
limit
=
set
_limit
Sidekiq
::
Queue
[
queue
].
limit
=
queue
_limit
end
end
end
end
end
end
private
private
def
set_limit
def
queue_limit
factor
=
current_application_settings
.
sidekiq_throttling_factor
@queue_limit
||=
begin
factor
=
Gitlab
::
CurrentSettings
.
current_application_settings
.
sidekiq_throttling_factor
(
factor
*
Sidekiq
.
options
[
:concurrency
]).
ceil
(
factor
*
Sidekiq
.
options
[
:concurrency
]).
ceil
end
end
end
end
end
end
end
end
end
spec/lib/gitlab/sidekiq_throttler_spec.rb
View file @
e840749b
...
@@ -11,12 +11,6 @@ describe Gitlab::SidekiqThrottler do
...
@@ -11,12 +11,6 @@ describe Gitlab::SidekiqThrottler do
)
)
end
end
describe
'#set_limit'
do
it
'returns the correct limit'
do
expect
(
Gitlab
::
SidekiqThrottler
.
send
(
:set_limit
)).
to
eq
4
end
end
describe
'#execute!'
do
describe
'#execute!'
do
it
'sets limits on the selected queues'
do
it
'sets limits on the selected queues'
do
Gitlab
::
SidekiqThrottler
.
execute!
Gitlab
::
SidekiqThrottler
.
execute!
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment