Add exclusive lock on project in ElasticCommitIndexerWorker

Indexing the same project twice at the same time causes conflict errors
in Elasticsearch and this usually leads to failures which can lead to a
project never being successfully indexed correctly. The simplest
solution is to use a lock in the worker and just fail immediately if
it's already running and let Sideikiq retries requeue the project if it
does have more updates that need indexing. When it next runs it will
check if there are new commits that need indexing.

Closes https://gitlab.com/gitlab-org/gitlab/-/issues/32648

ee/app/workers/elastic_commit_indexer_worker.rb

@@ -3,6 +3,7 @@
 class ElasticCommitIndexerWorker
   include ApplicationWorker
   prepend Elastic::IndexingControl
+  include Gitlab::ExclusiveLeaseHelpers
 
   feature_category :global_search
   sidekiq_options retry: 2
@@ -24,6 +25,8 @@ class ElasticCommitIndexerWorker
     project = Project.find(project_id)
     return true unless project.use_elasticsearch?
 
+    in_lock("#{self.class.name}/#{project_id}/#{wiki}", ttl: 1.hour, retries: 0) do
       Gitlab::Elastic::Indexer.new(project, wiki: wiki).run
     end
+  end
 end

ee/changelogs/unreleased/32648-dont-index-the-same-project-at-the-same-time.yml

+---
+title: Don't index the same project in Elasticsearch in parallel
+merge_request: 35842
+author:
+type: performance

ee/spec/workers/elastic_commit_indexer_worker_spec.rb

@@ -34,5 +34,14 @@ RSpec.describe ElasticCommitIndexerWorker do
 
       subject.perform(project.id, nil, nil, true)
     end
+
+    it 'does not run index when it is locked' do
+      expect(subject).to receive(:in_lock) # Mock and don't yield
+        .with("ElasticCommitIndexerWorker/#{project.id}/false", ttl: 1.hour, retries: 0)
+
+      expect(Gitlab::Elastic::Indexer).not_to receive(:new)
+
+      subject.perform(project.id)
+    end
   end
 end