Commit 5373d661 authored by Craig Miskell's avatar Craig Miskell Committed by Quang-Minh Nguyen

Apply 10 suggestion(s) to 1 file(s)

parent 61089fa8
...@@ -116,81 +116,10 @@ you list: ...@@ -116,81 +116,10 @@ you list:
> - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. > - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10.
> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. > - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
In addition to selecting queues by name, as above, the `queue_selector` In addition to selecting queues by name, as above, the `queue_selector` option
option allows queue groups to be selected in a more general way using allows queue groups to be selected in a more general way using [worker matching
the following components: query](extra_sidekiq_routing.md#worker-matching-query). After `queue_selector`
is set, all `queue_groups` must follow the aforementioned syntax.
- Attributes that can be selected.
- Operators used to construct a query.
When `queue_selector` is set, all `queue_groups` must be in the queue
selector syntax.
### Available attributes
- [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1, `tags`.
From the [list of all available
attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml),
`queue_selector` allows selecting of queues by the following attributes:
- `feature_category` - the [GitLab feature
category](https://about.gitlab.com/direction/maturity/#category-maturity) the
queue belongs to. For example, the `merge` queue belongs to the
`source_code_management` category.
- `has_external_dependencies` - whether or not the queue connects to external
services. For example, all importers have this set to `true`.
- `urgency` - how important it is that this queue's jobs run
quickly. Can be `high`, `low`, or `throttled`. For example, the
`authorized_projects` queue is used to refresh user permissions, and
is high urgency.
- `worker_name` - the worker name. The other attributes are typically more useful as
they are more general, but this is available in case a particular worker needs
to be selected.
- `name` - the queue name. Similarly, this is available in case a particular queue needs
to be selected.
- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or
`unknown`. For example, the `project_export` queue is memory bound as it has
to load data in memory before saving it for export.
- `tags` - short-lived annotations for queues. These are expected to frequently
change from release to release, and may be removed entirely.
`has_external_dependencies` is a boolean attribute: only the exact
string `true` is considered true, and everything else is considered
false.
`tags` is a set, which means that `=` checks for intersecting sets, and
`!=` checks for disjoint sets. For example, `tags=a,b` selects queues
that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
neither of those tags.
### Available operators
`queue_selector` supports the following operators, listed from highest
to lowest precedence:
- `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a`
and `query_b` are queries made up of the other operators here) will include
queues that match either query.
- `&` - the logical AND operator. For example, `query_a&query_b` (where
`query_a` and `query_b` are queries made up of the other operators here) will
only include queues that match both queries.
- `!=` - the NOT IN operator. For example, `feature_category!=issue_tracking`
excludes all queues from the `issue_tracking` feature category.
- `=` - the IN operator. For example, `resource_boundary=cpu` includes all
queues that are CPU bound.
- `,` - the concatenate set operator. For example,
`feature_category=continuous_integration,pages` includes all queues from
either the `continuous_integration` category or the `pages` category. This
example is also possible using the OR operator, but allows greater brevity, as
well as being lower precedence.
The operator precedence for this syntax is fixed: it's not possible to make AND
have higher precedence than OR.
[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and
later, as with the standard queue group syntax above, a single `*` as the
entire queue group selects all queues.
In `/etc/gitlab/gitlab.rb`: In `/etc/gitlab/gitlab.rb`:
......
...@@ -11,27 +11,27 @@ some scalability issues. One of them is the length of the queue tends to get ...@@ -11,27 +11,27 @@ some scalability issues. One of them is the length of the queue tends to get
longer. High-urgency jobs have to wait longer until other less urgent jobs longer. High-urgency jobs have to wait longer until other less urgent jobs
finish. This head-of-line blocking situation may eventually affect the finish. This head-of-line blocking situation may eventually affect the
responsiveness of the system, especially critical actions. In another scenario, responsiveness of the system, especially critical actions. In another scenario,
the performances of some jobs are degraded due to other CPU-intensive jobs the performance of some jobs is degraded due to other long running or CPU-intensive jobs
(computing or rendering ones) in the same machine. (computing or rendering ones) in the same machine.
To encounter the aforementioned issues, one effective solution is to split To counter the aforementioned issues, one effective solution is to split
Sidekiq jobs into different queues and assign machines handling each queue Sidekiq jobs into different queues and assign machines handling each queue
exclusively. For example, all CPU-intensive jobs are routed to `cpu-bound` exclusively. For example, all CPU-intensive jobs could be routed to the
queue and handled by a fleet of AWS CPU optimized instances. The queue topology `cpu-bound` queue and handled by a fleet of CPU optimized instances. The queue
differs between companies depending on the workloads and usage patterns. topology differs between companies depending on the workloads and usage
Therefore, GitLab supports a flexible mechanism for the administrator to route patterns. Therefore, GitLab supports a flexible mechanism for the
the jobs based on their characteristics. administrator to route the jobs based on their characteristics.
Opposed to [Queue selector](extra_sidekiq_processes.md#queue-selector), which As an alternative to [Queue selector](extra_sidekiq_processes.md#queue-selector), which
allows watching a set of workers or queues when starting a Sidekiq cluster, configures Sidekiq cluster to listen to a specific set of workers or queues,
GitLab also supports routing a job from a worker to the desired queue before it GitLab also supports routing a job from a worker to the desired queue when it
is scheduled. Sidekiq clients try to match a job against a configured list of is scheduled. Sidekiq clients try to match a job against a configured list of
routing rules. Rules are evaluated from first to last, and as soon as we find a routing rules. Rules are evaluated from first to last, and as soon as we find a
match for a given worker we stop processing for that worker (first match wins). match for a given worker we stop processing for that worker (first match wins).
If the worker doesn't match any rule, it falls back to the queue name generated If the worker doesn't match any rule, it falls back to the queue name generated
from the worker name. from the worker name.
By default, the routing rules are not configured (or denoted with an empty By default, if the routing rules are not configured (or denoted with an empty
array), all the jobs are routed to the queue generated from the worker name. array), all the jobs are routed to the queue generated from the worker name.
## Example configuration ## Example configuration
...@@ -64,11 +64,12 @@ is nil, or an empty string, the worker is routed to the queue generated ...@@ -64,11 +64,12 @@ is nil, or an empty string, the worker is routed to the queue generated
by the name of the worker instead. by the name of the worker instead.
The query supports wildcard matching `*`, which matches all workers. As a The query supports wildcard matching `*`, which matches all workers. As a
result, the wildcard query must stay at the end of the list or the rules behind result, the wildcard query must stay at the end of the list or the rules after it
are ignored. are ignored.
> Important: Queue routing rules are not compatible with Queue selector. Please > Important: Mixing queue routing rules and queue selectors requires care to
> consider using either one of them. > ensure all jobs that are scheduled and picked up by appropriate Sidekiq
> workers
## Worker matching query ## Worker matching query
...@@ -172,3 +173,11 @@ feature_category=database&urgency=throttled ...@@ -172,3 +173,11 @@ feature_category=database&urgency=throttled
# Match default and mailers queues # Match default and mailers queues
queue=default|queue=mailers queue=default|queue=mailers
``` ```
### Migration
After the Sidekiq routing rules are changed, administrators need to take care
with the migration to avoid losing jobs entirely, especially in a system with
long queues of jobs. The migration can be done by following the migration steps
mentioned in [Sidekiq job
migration](../../raketasks/sidekiq_job_migration.md)
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment