Commit 3a426da1 authored by Sean McGivern's avatar Sean McGivern

Rename queue-selector to experimental-queue-selector

We want the ability to react quickly to changes, so while this is
documented, we need to make sure that people are aware that using this
while it's still experimental is a significant risk of having to deal
with breaking changes.
parent 95a74031
...@@ -86,9 +86,18 @@ you list: ...@@ -86,9 +86,18 @@ you list:
> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. > [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8.
In addition to selecting queues by name, as above, the `queue_selector` CAUTION: **Caution:**
option allows queue groups to be selected in a more general way using As this is marked as **experimental**, it is subject to change at any
the following components: time, including **breaking backwards compatibility**. This is so that we
can react to changes we need for our GitLab.com deployment. We have a
tracking issue open to [remove the experimental
designation](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/147)
from this feature; please comment there if you are interested in using
this in your own deployment.
In addition to selecting queues by name, as above, the
`experimental_queue_selector` option allows queue groups to be selected
in a more general way using the following components:
- Attributes that can be selected. - Attributes that can be selected.
- Operators used to construct a query. - Operators used to construct a query.
...@@ -97,7 +106,8 @@ the following components: ...@@ -97,7 +106,8 @@ the following components:
From the [list of all available From the [list of all available
attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml), attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml),
`queue_selector` allows selecting of queues by the following attributes: `experimental_queue_selector` allows selecting of queues by the
following attributes:
- `feature_category` - the [GitLab feature - `feature_category` - the [GitLab feature
category](https://about.gitlab.com/direction/maturity/#category-maturity) the category](https://about.gitlab.com/direction/maturity/#category-maturity) the
...@@ -122,8 +132,8 @@ considered false. ...@@ -122,8 +132,8 @@ considered false.
### Available operators ### Available operators
`queue_selector` supports the following operators, listed from highest `experimental_queue_selector` supports the following operators, listed
to lowest precedence: from highest to lowest precedence:
- `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a` - `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a`
and `query_b` are queries made up of the other operators here) will include and `query_b` are queries made up of the other operators here) will include
...@@ -150,7 +160,7 @@ In `/etc/gitlab/gitlab.rb`: ...@@ -150,7 +160,7 @@ In `/etc/gitlab/gitlab.rb`:
```ruby ```ruby
sidekiq_cluster['enable'] = true sidekiq_cluster['enable'] = true
sidekiq_cluster['queue_selector'] = true sidekiq_cluster['experimental_queue_selector'] = true
sidekiq_cluster['queue_groups'] = [ sidekiq_cluster['queue_groups'] = [
# Run all non-CPU-bound queues that are latency sensitive # Run all non-CPU-bound queues that are latency sensitive
'resource_boundary!=cpu&latency_sensitive=true', 'resource_boundary!=cpu&latency_sensitive=true',
......
...@@ -45,7 +45,7 @@ module Gitlab ...@@ -45,7 +45,7 @@ module Gitlab
queue_names = SidekiqConfig::CliMethods.worker_queues(@rails_path) queue_names = SidekiqConfig::CliMethods.worker_queues(@rails_path)
queue_groups = queue_groups =
if @queue_selector if @experimental_queue_selector
# When using the experimental queue query syntax, we treat # When using the experimental queue query syntax, we treat
# each queue group as a worker attribute query, and resolve # each queue group as a worker attribute query, and resolve
# the queues for the queue group using this query. # the queues for the queue group using this query.
...@@ -160,8 +160,8 @@ module Gitlab ...@@ -160,8 +160,8 @@ module Gitlab
@rails_path = path @rails_path = path
end end
opt.on('--queue-selector', 'EXPERIMENTAL: Run workers based on the provided selector') do |queue_selector| opt.on('--experimental-queue-selector', 'EXPERIMENTAL: Run workers based on the provided selector') do |experimental_queue_selector|
@queue_selector = queue_selector @experimental_queue_selector = experimental_queue_selector
end end
opt.on('-n', '--negate', 'Run workers for all queues in sidekiq_queues.yml except the given ones') do opt.on('-n', '--negate', 'Run workers for all queues in sidekiq_queues.yml except the given ones') do
......
...@@ -7,7 +7,7 @@ describe 'ee/bin/sidekiq-cluster' do ...@@ -7,7 +7,7 @@ describe 'ee/bin/sidekiq-cluster' do
where(:args, :included, :excluded) do where(:args, :included, :excluded) do
%w[--negate cronjob] | '-qdefault,1' | '-qcronjob,1' %w[--negate cronjob] | '-qdefault,1' | '-qcronjob,1'
%w[--queue-selector resource_boundary=cpu] | '-qupdate_merge_requests,1' | '-qdefault,1' %w[--experimental-queue-selector resource_boundary=cpu] | '-qupdate_merge_requests,1' | '-qdefault,1'
end end
with_them do with_them do
......
...@@ -79,7 +79,7 @@ describe Gitlab::SidekiqCluster::CLI do ...@@ -79,7 +79,7 @@ describe Gitlab::SidekiqCluster::CLI do
end end
end end
context 'with --queue-selector' do context 'with --experimental-queue-selector' do
where do where do
{ {
'memory-bound queues' => { 'memory-bound queues' => {
...@@ -125,7 +125,7 @@ describe Gitlab::SidekiqCluster::CLI do ...@@ -125,7 +125,7 @@ describe Gitlab::SidekiqCluster::CLI do
[] []
end end
cli.run(%W(--queue-selector #{query})) cli.run(%W(--experimental-queue-selector #{query}))
end end
it 'works when negated' do it 'works when negated' do
...@@ -137,7 +137,7 @@ describe Gitlab::SidekiqCluster::CLI do ...@@ -137,7 +137,7 @@ describe Gitlab::SidekiqCluster::CLI do
[] []
end end
cli.run(%W(--negate --queue-selector #{query})) cli.run(%W(--negate --experimental-queue-selector #{query}))
end end
end end
...@@ -147,13 +147,13 @@ describe Gitlab::SidekiqCluster::CLI do ...@@ -147,13 +147,13 @@ describe Gitlab::SidekiqCluster::CLI do
.with([['chat_notification'], ['project_export']], default_options) .with([['chat_notification'], ['project_export']], default_options)
.and_return([]) .and_return([])
cli.run(%w(--queue-selector feature_category=chatops&latency_sensitive=true resource_boundary=memory&feature_category=source_code_management)) cli.run(%w(--experimental-queue-selector feature_category=chatops&latency_sensitive=true resource_boundary=memory&feature_category=importers))
end end
it 'errors on an invalid query multiple queue groups correctly' do it 'errors on an invalid query multiple queue groups correctly' do
expect(Gitlab::SidekiqCluster).not_to receive(:start) expect(Gitlab::SidekiqCluster).not_to receive(:start)
expect { cli.run(%w(--queue-selector unknown_field=chatops)) } expect { cli.run(%w(--experimental-queue-selector unknown_field=chatops)) }
.to raise_error(Gitlab::SidekiqConfig::CliMethods::QueryError) .to raise_error(Gitlab::SidekiqConfig::CliMethods::QueryError)
end end
end end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment