Fix: Sidekiq workers delete each other's metrics

When we moved the logic that wipes the Prometheus metrics
dir out of the Rackup file and into the initializer, all
Sidekiq workers would call this and potentially enter a
race condition where they deleted each other's database
files.

Changelog: fixed

config.ru

@@ -9,6 +9,12 @@ def master_process?
 end
 
 warmup do |app|
+  # The following is necessary to ensure stale Prometheus metrics don't accumulate over time.
+  # It needs to be done as early as here to ensure metrics files aren't deleted.
+  # After we hit our app in `warmup`, first metrics and corresponding files already being created,
+  # for example in `lib/gitlab/metrics/requests_rack_middleware.rb`.
+  Prometheus::CleanupMultiprocDirService.new.execute if master_process?
+
   client = Rack::MockRequest.new(app)
   client.get('/')
 end

config/initializers/7_prometheus_metrics.rb

@@ -40,9 +40,6 @@ if !Rails.env.test? && Gitlab::Metrics.prometheus_metrics_enabled?
   # When running Puma in a Single mode, `on_master_start` and `on_worker_start` are the same.
   # Thus, we order these events to run `reinitialize_on_pid_change` with `force: true` first.
   Gitlab::Cluster::LifecycleEvents.on_master_start do
-    # Ensure that stale Prometheus metrics don't accumulate over time
-    ::Prometheus::CleanupMultiprocDirService.new.execute
-
     ::Prometheus::Client.reinitialize_on_pid_change(force: true)
 
     if Gitlab::Runtime.puma?