Allow GC to run if deduplication service runs into an error

We noticed that the Gitaly `FetchIntoObjectPool` RPC can fail in
Gitaly Cluster. As a result, garbage collection was blocked,
preventing cleanup from happening.

We now allow GC to continue if GRPC either times out or raise some
internal error.

Relates to https://gitlab.com/gitlab-org/gitaly/-/issues/4022

Changelog: fixed

app/workers/projects/git_garbage_collect_worker.rb

@@ -16,7 +16,15 @@ module Projects
     def before_gitaly_call(task, resource)
       return unless gc?(task)
 
-      ::Projects::GitDeduplicationService.new(resource).execute
+      # Don't block garbage collection if we can't fetch into an object pool
+      # due to some gRPC error because we don't want to accumulate cruft.
+      # See https://gitlab.com/gitlab-org/gitaly/-/issues/4022.
+      begin
+        ::Projects::GitDeduplicationService.new(resource).execute
+      rescue Gitlab::Git::CommandTimedOut, GRPC::Internal => e
+        Gitlab::ErrorTracking.track_exception(e)
+      end
+
       cleanup_orphan_lfs_file_references(resource)
     end
 

spec/workers/projects/git_garbage_collect_worker_spec.rb

@@ -32,6 +32,21 @@ RSpec.describe Projects::GitGarbageCollectWorker do
 
         subject.perform(*params)
       end
+
+      context 'when deduplication service runs into a GRPC internal error' do
+        before do
+          allow_next_instance_of(::Projects::GitDeduplicationService) do |instance|
+            expect(instance).to receive(:execute).and_raise(GRPC::Internal)
+          end
+        end
+
+        it_behaves_like 'can collect git garbage' do
+          let(:resource) { project }
+          let(:statistics_service_klass) { Projects::UpdateStatisticsService }
+          let(:statistics_keys) { [:repository_size, :lfs_objects_size] }
+          let(:expected_default_lease) { "projects:#{resource.id}" }
+        end
+      end
     end
 
     context 'LFS object garbage collection' do