Disable Sendfile interface for serving Sidekiq Web assets

Sidekiq bundles its own CSS and JavaScript assets in the gem, and
registers these via `Rack::Static`. By default, `Rack::Sendfile` will
use the Sendfile interface if available to offload the sending to
Workhorse. However, Workhorse might not have these assets, as they are
bundled with the installed Ruby gem.

Cloud Native GitLab worked around this issue by routing `/admin/sidekiq`
directly to the webservice container, but this doesn't work if the NGINX
ingress controller isn't used. Also, users have reported that the
ordering of the rules may matter when another load balancer is used
(https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2731).

To avoid these kinds of issues leading to 404s, we now disable the use
of Sendfile for `admin/sidekiq` routes. These assets are small and
served infrequently, so it should be fine to do this.

Relates to
https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14150

Changelog: changed

config/initializers/static_files.rb

@@ -2,6 +2,10 @@
 
 app = Rails.application
 
+# Disable Sendfile for Sidekiq Web assets since Workhorse won't
+# always have access to these files.
+app.config.middleware.insert_before(Rack::Sendfile, Gitlab::Middleware::SidekiqWebStatic)
+
 if app.config.public_file_server.enabled
   # The `ActionDispatch::Static` middleware intercepts requests for static files
   # by checking if they exist in the `/public` directory.

lib/gitlab/middleware/sidekiq_web_static.rb

+# frozen_string_literal: true
+
+# This module removes the X-Sendfile-Type header for /admin/sidekiq
+# assets since Workhorse isn't always guaranteed to have the assets
+# present on disk, such as when using Cloud Native GitLab
+# containers. These assets are also small and served infrequently so it
+# should be fine to do this.
+module Gitlab
+  module Middleware
+    class SidekiqWebStatic
+      SIDEKIQ_REGEX = %r{\A/admin/sidekiq/}.freeze
+
+      def initialize(app)
+        @app = app
+      end
+
+      def call(env)
+        env.delete('HTTP_X_SENDFILE_TYPE') if env['PATH_INFO'] =~ SIDEKIQ_REGEX
+
+        @app.call(env)
+      end
+    end
+  end
+end

spec/lib/gitlab/middleware/sidekiq_web_static_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+RSpec.describe Gitlab::Middleware::SidekiqWebStatic do
+  let(:app) { double(:app) }
+  let(:middleware) { described_class.new(app) }
+  let(:env) { {} }
+
+  describe '#call' do
+    before do
+      env['HTTP_X_SENDFILE_TYPE'] = 'X-Sendfile'
+      env['PATH_INFO'] = path
+    end
+
+    context 'with an /admin/sidekiq route' do
+      let(:path) { '/admin/sidekiq/javascripts/application.js'}
+
+      it 'deletes the HTTP_X_SENDFILE_TYPE header' do
+        expect(app).to receive(:call)
+
+        middleware.call(env)
+
+        expect(env['HTTP_X_SENDFILE_TYPE']).to be_nil
+      end
+    end
+
+    context 'with some static asset route' do
+      let(:path) { '/assets/test.png' }
+
+      it 'keeps the HTTP_X_SENDFILE_TYPE header' do
+        expect(app).to receive(:call)
+
+        middleware.call(env)
+
+        expect(env['HTTP_X_SENDFILE_TYPE']).to eq('X-Sendfile')
+      end
+    end
+  end
+end