Commit 0db134b7 authored by Matthias Kaeppler's avatar Matthias Kaeppler

Fix for sidekiq prometheus files being wiped

Due to a race condition during sidekiq process start-up, we might end
up losing prometheus db files due to doing a cleanup after first
creating them. By re-initializing after every cleanup, we side-step
the issue, but it's not really a clean solution.
parent 753fa91a
...@@ -34,6 +34,12 @@ Sidekiq.configure_server do |config| ...@@ -34,6 +34,12 @@ Sidekiq.configure_server do |config|
config.on(:startup) do config.on(:startup) do
# webserver metrics are cleaned up in config.ru: `warmup` block # webserver metrics are cleaned up in config.ru: `warmup` block
Prometheus::CleanupMultiprocDirService.new.execute Prometheus::CleanupMultiprocDirService.new.execute
# In production, sidekiq is run in a multi-process setup where processes might interfere
# with each other cleaning up and reinitializing prometheus database files, which is why
# we're re-doing the work every time here.
# A cleaner solution would be to run the cleanup pre-fork, and the initialization once
# after all workers have forked, but I don't know how at this point.
::Prometheus::Client.reinitialize_on_pid_change(force: true)
Gitlab::Metrics::Exporter::SidekiqExporter.instance.start Gitlab::Metrics::Exporter::SidekiqExporter.instance.start
end end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment