Add a sidekiq_client.log logfile

Sidekiq client, in our case Rails, can also emit log messages related
to sidekiq jobs.

Instead of having these messages printed to STDOUT (which results in
`production.log`), this changes the Sidekiq.logger to a separate
writing to `log/sidekiq_client`. This allows us to ingest these
messages in the same index as the messages the sidekiq process
emits. Having them in the same index allows us to gather statistics
about jobs that haven't been processed.

config/initializers/sidekiq.rb

@@ -108,6 +108,11 @@ end
 
 Sidekiq.configure_client do |config|
   config.redis = queues_config_hash
+  # We only need to do this for other clients. If Sidekiq-server is the
+  # client scheduling jobs, we have access to the regular sidekiq logger that
+  # writes to STDOUT
+  Sidekiq.logger = Gitlab::SidekiqLogging::ClientLogger.build
+  Sidekiq.logger.formatter = Gitlab::SidekiqLogging::JSONFormatter.new if enable_json_logs
 
   config.client_middleware(&Gitlab::SidekiqMiddleware.client_configurator)
 end

doc/administration/logs.md

@@ -400,6 +400,21 @@ For source installations, edit the `gitlab.yml` and set the Sidekiq
     log_format: json
 ```
 
+## `sidekiq_client.log`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26586) in GitLab 12.9.
+
+This file lives in `/var/log/gitlab/gitlab-rails/sidekiq_client.log` for
+Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_client.log` for
+installations from source.
+
+This file contains logging information about jobs before they are start
+being processed by Sidekiq, for example before being enqueued.
+
+This logfile follows the same structure as
+[`sidekiq.log`](#sidekiqlog), so it will be structured as JSON if
+you've configured this for Sidekiq as mentioned above.
+
 ## `gitlab-shell.log`
 
 This file lives in `/var/log/gitlab/gitaly/gitlab-shell.log` for

lib/gitlab/sidekiq_logging/client_logger.rb

+# frozen_string_literal: true
+
+module Gitlab
+  module SidekiqLogging
+    class ClientLogger < Gitlab::Logger
+      def self.file_name_noext
+        'sidekiq_client'
+      end
+    end
+  end
+end