Commit 90330542 authored by Oswaldo Ferreira's avatar Oswaldo Ferreira Committed by Marcia Ramos

Add docs for idempotent workers

This mostly explains how to set a worker as
idempotent and add tests for it.
parent 1d9484d1
...@@ -64,6 +64,63 @@ the extra jobs will take resources away from jobs from workers that were already ...@@ -64,6 +64,63 @@ the extra jobs will take resources away from jobs from workers that were already
there, if the resources available to the Sidekiq process handling the namespace there, if the resources available to the Sidekiq process handling the namespace
are not adjusted appropriately. are not adjusted appropriately.
## Idempotent Jobs
It's known that a job can fail for multiple reasons, for example, network outages or bugs.
In order to address this, Sidekiq has a built-in retry mechanism that is
used by default by most workers within GitLab.
It's expected that a job can run again after a failure without major side-effects for the
application or users, which is why Sidekiq encourages
jobs to be [idempotent and transactional](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional).
As a general rule, a worker can be considered idempotent if:
- It can safely run multiple times with the same arguments.
- Application side-effects are expected to happen only once
(or side-effects of a second run are not impactful).
A good example of that would be a cache expiration worker.
### Ensuring a worker is idempotent
Make sure the worker tests pass using the following shared example:
```ruby
include_examples 'an idempotent worker' do
it 'marks the MR as merged' do
# Using subject inside this block will process the job multiple times
subject
expect(merge_request.state).to eq('merged')
end
end
```
Use the `perform_multiple` method directly instead of `job.perform` (this
helper method is automatically included for workers).
### Declaring a worker as idempotent
```ruby
class IdempotentWorker
include ApplicationWorker
# Declares a worker is idempotent and can
# safely run multiple times.
idempotent!
# ...
end
```
It's encouraged to only have the `idempotent!` call in the top-most worker class, even if
the `perform` method is defined in another class or module.
NOTE: **Note:**
Note that a cop will fail if the worker class is not marked as idempotent.
Consider skipping the cop if you're not confident your job can safely run multiple times.
## Latency Sensitive Jobs ## Latency Sensitive Jobs
If a large number of background jobs get scheduled at once, queueing of jobs may If a large number of background jobs get scheduled at once, queueing of jobs may
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment