Make Geo repository verification flag opt out by default

parent b7aef3ae
# Automatic background verification **[PREMIUM ONLY]** # Automatic background verification **[PREMIUM ONLY]**
NOTE: **Note:** NOTE: **Note:**
Automatic background verification of repositories and wikis was added in GitLab Automatic background verification of repositories and wikis was added in
EE 10.6, but is disabled by default. To enable it, run GitLab EE 10.6 but is enabled by default only on Gitlab EE 11.1. You can
`sudo gitlab-rails runner 'Feature.enable(:geo_repository_verification)'` on disable or enable this feature manually by following
the **primary**. Until [issue #5699][ee-5699] is completed, we need to reset [these instructions][feature-flag].
the cache for this feature flag on each **secondary**, to do this run
`sudo gitlab-rails runner 'Rails.cache.expire('flipper/v1/feature/geo_repository_verification', 0)'`.
Automatic backgorund verification ensures that the transferred data matches a Automatic backgorund verification ensures that the transferred data matches a
calculated checksum, proving that the content on the **secondary** matches that calculated checksum, proving that the content on the **secondary** matches that
...@@ -30,6 +28,43 @@ sudo gitlab-rails runner 'Geo::ProjectRegistry.verification_failed.delete_all' ...@@ -30,6 +28,43 @@ sudo gitlab-rails runner 'Geo::ProjectRegistry.verification_failed.delete_all'
If verification is lagging significantly behind replication, consider giving If verification is lagging significantly behind replication, consider giving
the node more time before scheduling a planned failover. the node more time before scheduling a planned failover.
### Disabling or enabling the automatic background verification
The following commands are to be issues in a Rails console on
the **primary**:
```sh
# Omnibus GitLab
gitlab-rails console
# Installation from source
cd /home/git/gitlab
sudo -u git -H bin/rails console RAILS_ENV=production
```
**To check if automatic background verification is enabled:**
```ruby
Feature.enabled?('geo_repository_verification')
```
**To disable automatic background verification:**
```ruby
Feature.disable('geo_repository_verification')
```
**To enable automatic background verification:**
```ruby
Feature.enable('geo_repository_verification')
```
NOTE: **Note:**
Until [issue #5699][ee-5699] is completed, we need to reset the cache for this
feature flag on each **secondary**, to do this run
`sudo gitlab-rails runner 'Rails.cache.expire('flipper/v1/feature/geo_repository_verification', 0)'`.
# Repository verification # Repository verification
Visit the **Admin Area ➔ Geo nodes** dashboard on the **primary** and expand Visit the **Admin Area ➔ Geo nodes** dashboard on the **primary** and expand
...@@ -59,13 +94,14 @@ for every node after every update to make sure that they are all in sync. ...@@ -59,13 +94,14 @@ for every node after every update to make sure that they are all in sync.
Until [issue #5064][ee-5064] is completed, background verification doesn't cover Until [issue #5064][ee-5064] is completed, background verification doesn't cover
CI job artifacts and traces, LFS objects, or user uploads in file storage. CI job artifacts and traces, LFS objects, or user uploads in file storage.
Verifytheir integrity manually by following [these instructions][foreground-verification] Verify their integrity manually by following [these instructions][foreground-verification]
on both nodes, and comparing the output between them. on both nodes, and comparing the output between them.
Data in object storage is **not verified**, as the object store is responsible Data in object storage is **not verified**, as the object store is responsible
for ensuring the integrity of the data. for ensuring the integrity of the data.
[disaster-recovery]: index.md [disaster-recovery]: index.md
[feature-flag]: background_verification.md#enabling-or-disabling-the-automatic-background-verification
[foreground-verification]: ../../raketasks/check.md [foreground-verification]: ../../raketasks/check.md
[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 [ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064
[ee-5699]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5699 [ee-5699]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5699
......
...@@ -130,7 +130,7 @@ class GeoNodeStatus < ActiveRecord::Base ...@@ -130,7 +130,7 @@ class GeoNodeStatus < ActiveRecord::Base
end end
def initialize_feature_flags def initialize_feature_flags
self.repository_verification_enabled = Feature.enabled?('geo_repository_verification') self.repository_verification_enabled = Gitlab::Geo.repository_verification_enabled?
end end
def update_cache! def update_cache!
......
...@@ -3,7 +3,7 @@ module Geo ...@@ -3,7 +3,7 @@ module Geo
module Primary module Primary
class BatchWorker < Geo::Scheduler::Primary::PerShardSchedulerWorker class BatchWorker < Geo::Scheduler::Primary::PerShardSchedulerWorker
def perform def perform
return unless Feature.enabled?('geo_repository_verification') return unless Gitlab::Geo.repository_verification_enabled?
super super
end end
......
...@@ -3,7 +3,7 @@ module Geo ...@@ -3,7 +3,7 @@ module Geo
module Secondary module Secondary
class SchedulerWorker < Geo::Scheduler::Secondary::PerShardSchedulerWorker class SchedulerWorker < Geo::Scheduler::Secondary::PerShardSchedulerWorker
def perform def perform
return unless Feature.enabled?('geo_repository_verification') return unless Gitlab::Geo.repository_verification_enabled?
super super
end end
......
...@@ -109,5 +109,16 @@ module Gitlab ...@@ -109,5 +109,16 @@ module Gitlab
# urlsafe_base64 may return a string of size * 4/3 # urlsafe_base64 may return a string of size * 4/3
SecureRandom.urlsafe_base64(size)[0, size] SecureRandom.urlsafe_base64(size)[0, size]
end end
def self.repository_verification_enabled?
feature = Feature.get(:geo_repository_verification)
# If the feature has been set, always evaluate
if Feature.persisted?(feature)
return feature.enabled?
else
true
end
end
end end
end end
...@@ -252,7 +252,7 @@ namespace :geo do ...@@ -252,7 +252,7 @@ namespace :geo do
print "#{current_node_status.repositories_synced_count}/#{current_node_status.repositories_count} " print "#{current_node_status.repositories_synced_count}/#{current_node_status.repositories_count} "
puts using_percentage(current_node_status.repositories_synced_in_percentage) puts using_percentage(current_node_status.repositories_synced_in_percentage)
if Feature.enabled?(:geo_repository_verification) if Gitlab::Geo.repository_verification_enabled?
print 'Verified Repositories: '.rjust(COLUMN_WIDTH) print 'Verified Repositories: '.rjust(COLUMN_WIDTH)
show_failed_value(current_node_status.repositories_verification_failed_count) show_failed_value(current_node_status.repositories_verification_failed_count)
print "#{current_node_status.repositories_verified_count}/#{current_node_status.repositories_count} " print "#{current_node_status.repositories_verified_count}/#{current_node_status.repositories_count} "
...@@ -264,7 +264,7 @@ namespace :geo do ...@@ -264,7 +264,7 @@ namespace :geo do
print "#{current_node_status.wikis_synced_count}/#{current_node_status.wikis_count} " print "#{current_node_status.wikis_synced_count}/#{current_node_status.wikis_count} "
puts using_percentage(current_node_status.wikis_synced_in_percentage) puts using_percentage(current_node_status.wikis_synced_in_percentage)
if Feature.enabled?(:geo_repository_verification) if Gitlab::Geo.repository_verification_enabled?
print 'Verified Wikis: '.rjust(COLUMN_WIDTH) print 'Verified Wikis: '.rjust(COLUMN_WIDTH)
show_failed_value(current_node_status.wikis_verification_failed_count) show_failed_value(current_node_status.wikis_verification_failed_count)
print "#{current_node_status.wikis_verified_count}/#{current_node_status.wikis_count} " print "#{current_node_status.wikis_verified_count}/#{current_node_status.wikis_count} "
......
...@@ -174,4 +174,34 @@ describe Gitlab::Geo, :geo do ...@@ -174,4 +174,34 @@ describe Gitlab::Geo, :geo do
described_class.configure_cron_jobs! described_class.configure_cron_jobs!
end end
end end
describe '.repository_verification_enabled?' do
context "when the feature flag hasn't been set" do
it 'returns true' do
expect(described_class.repository_verification_enabled?).to eq true
end
end
context "when the feature flag has been set" do
before do
allow(Feature).to receive(:persisted?).and_return(true)
end
context "when the feature flag is set to enabled" do
it 'returns true' do
Feature.enable(:geo_repository_verification)
expect(described_class.repository_verification_enabled?).to eq true
end
end
context "when the feature flag is set to disabled" do
it 'returns false' do
Feature.disable(:geo_repository_verification)
expect(described_class.repository_verification_enabled?).to eq false
end
end
end
end
end end
...@@ -533,7 +533,6 @@ describe GeoNodeStatus, :geo do ...@@ -533,7 +533,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of checksummed repositories' do it 'returns the right number of checksummed repositories' do
stub_feature_flags(geo_repository_verification: true)
create(:repository_state, :repository_verified) create(:repository_state, :repository_verified)
create(:repository_state, :repository_verified) create(:repository_state, :repository_verified)
...@@ -541,7 +540,7 @@ describe GeoNodeStatus, :geo do ...@@ -541,7 +540,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag is off' do it 'returns existing value when feature flag is off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: primary) create(:geo_node_status, :healthy, geo_node: primary)
expect(subject.repositories_checksummed_count).to eq(600) expect(subject.repositories_checksummed_count).to eq(600)
...@@ -554,7 +553,6 @@ describe GeoNodeStatus, :geo do ...@@ -554,7 +553,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of failed repositories' do it 'returns the right number of failed repositories' do
stub_feature_flags(geo_repository_verification: true)
create(:repository_state, :repository_failed) create(:repository_state, :repository_failed)
create(:repository_state, :repository_failed) create(:repository_state, :repository_failed)
...@@ -562,7 +560,7 @@ describe GeoNodeStatus, :geo do ...@@ -562,7 +560,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: primary) create(:geo_node_status, :healthy, geo_node: primary)
expect(subject.repositories_checksum_failed_count).to eq(120) expect(subject.repositories_checksum_failed_count).to eq(120)
...@@ -597,7 +595,6 @@ describe GeoNodeStatus, :geo do ...@@ -597,7 +595,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of checksummed wikis' do it 'returns the right number of checksummed wikis' do
stub_feature_flags(geo_repository_verification: true)
create(:repository_state, :wiki_verified) create(:repository_state, :wiki_verified)
create(:repository_state, :wiki_verified) create(:repository_state, :wiki_verified)
...@@ -605,7 +602,7 @@ describe GeoNodeStatus, :geo do ...@@ -605,7 +602,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: primary) create(:geo_node_status, :healthy, geo_node: primary)
expect(subject.wikis_checksummed_count).to eq(585) expect(subject.wikis_checksummed_count).to eq(585)
...@@ -618,7 +615,6 @@ describe GeoNodeStatus, :geo do ...@@ -618,7 +615,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of failed wikis' do it 'returns the right number of failed wikis' do
stub_feature_flags(geo_repository_verification: true)
create(:repository_state, :wiki_failed) create(:repository_state, :wiki_failed)
create(:repository_state, :wiki_failed) create(:repository_state, :wiki_failed)
...@@ -626,7 +622,7 @@ describe GeoNodeStatus, :geo do ...@@ -626,7 +622,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: primary) create(:geo_node_status, :healthy, geo_node: primary)
expect(subject.wikis_checksum_failed_count).to eq(55) expect(subject.wikis_checksum_failed_count).to eq(55)
...@@ -661,7 +657,6 @@ describe GeoNodeStatus, :geo do ...@@ -661,7 +657,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of verified repositories' do it 'returns the right number of verified repositories' do
stub_feature_flags(geo_repository_verification: true)
create(:geo_project_registry, :repository_verified) create(:geo_project_registry, :repository_verified)
create(:geo_project_registry, :repository_verified) create(:geo_project_registry, :repository_verified)
...@@ -669,7 +664,7 @@ describe GeoNodeStatus, :geo do ...@@ -669,7 +664,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: secondary) create(:geo_node_status, :healthy, geo_node: secondary)
expect(subject.repositories_verified_count).to eq(501) expect(subject.repositories_verified_count).to eq(501)
...@@ -682,7 +677,6 @@ describe GeoNodeStatus, :geo do ...@@ -682,7 +677,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of repositories that checksum mismatch' do it 'returns the right number of repositories that checksum mismatch' do
stub_feature_flags(geo_repository_verification: true)
create(:geo_project_registry, :repository_checksum_mismatch) create(:geo_project_registry, :repository_checksum_mismatch)
create(:geo_project_registry, :repository_verification_failed) create(:geo_project_registry, :repository_verification_failed)
create(:geo_project_registry, :repository_verified) create(:geo_project_registry, :repository_verified)
...@@ -691,7 +685,7 @@ describe GeoNodeStatus, :geo do ...@@ -691,7 +685,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: secondary) create(:geo_node_status, :healthy, geo_node: secondary)
expect(subject.repositories_checksum_mismatch_count).to eq(15) expect(subject.repositories_checksum_mismatch_count).to eq(15)
...@@ -704,7 +698,6 @@ describe GeoNodeStatus, :geo do ...@@ -704,7 +698,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of failed repositories' do it 'returns the right number of failed repositories' do
stub_feature_flags(geo_repository_verification: true)
create(:geo_project_registry, :repository_verification_failed) create(:geo_project_registry, :repository_verification_failed)
create(:geo_project_registry, :repository_verification_failed) create(:geo_project_registry, :repository_verification_failed)
...@@ -712,7 +705,7 @@ describe GeoNodeStatus, :geo do ...@@ -712,7 +705,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: secondary) create(:geo_node_status, :healthy, geo_node: secondary)
expect(subject.repositories_verification_failed_count).to eq(100) expect(subject.repositories_verification_failed_count).to eq(100)
...@@ -725,7 +718,6 @@ describe GeoNodeStatus, :geo do ...@@ -725,7 +718,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of verified wikis' do it 'returns the right number of verified wikis' do
stub_feature_flags(geo_repository_verification: true)
create(:geo_project_registry, :wiki_verified) create(:geo_project_registry, :wiki_verified)
create(:geo_project_registry, :wiki_verified) create(:geo_project_registry, :wiki_verified)
...@@ -733,7 +725,7 @@ describe GeoNodeStatus, :geo do ...@@ -733,7 +725,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: secondary) create(:geo_node_status, :healthy, geo_node: secondary)
expect(subject.wikis_verified_count).to eq(499) expect(subject.wikis_verified_count).to eq(499)
...@@ -746,7 +738,6 @@ describe GeoNodeStatus, :geo do ...@@ -746,7 +738,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of wikis that checksum mismatch' do it 'returns the right number of wikis that checksum mismatch' do
stub_feature_flags(geo_repository_verification: true)
create(:geo_project_registry, :wiki_checksum_mismatch) create(:geo_project_registry, :wiki_checksum_mismatch)
create(:geo_project_registry, :wiki_verification_failed) create(:geo_project_registry, :wiki_verification_failed)
create(:geo_project_registry, :wiki_verified) create(:geo_project_registry, :wiki_verified)
...@@ -755,7 +746,7 @@ describe GeoNodeStatus, :geo do ...@@ -755,7 +746,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: secondary) create(:geo_node_status, :healthy, geo_node: secondary)
expect(subject.wikis_checksum_mismatch_count).to eq(10) expect(subject.wikis_checksum_mismatch_count).to eq(10)
...@@ -768,7 +759,6 @@ describe GeoNodeStatus, :geo do ...@@ -768,7 +759,6 @@ describe GeoNodeStatus, :geo do
end end
it 'returns the right number of failed wikis' do it 'returns the right number of failed wikis' do
stub_feature_flags(geo_repository_verification: true)
create(:geo_project_registry, :wiki_verification_failed) create(:geo_project_registry, :wiki_verification_failed)
create(:geo_project_registry, :wiki_verification_failed) create(:geo_project_registry, :wiki_verification_failed)
...@@ -776,7 +766,7 @@ describe GeoNodeStatus, :geo do ...@@ -776,7 +766,7 @@ describe GeoNodeStatus, :geo do
end end
it 'returns existing value when feature flag if off' do it 'returns existing value when feature flag if off' do
stub_feature_flags(geo_repository_verification: false) allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
create(:geo_node_status, :healthy, geo_node: secondary) create(:geo_node_status, :healthy, geo_node: secondary)
expect(subject.wikis_verification_failed_count).to eq(99) expect(subject.wikis_verification_failed_count).to eq(99)
......
...@@ -17,66 +17,58 @@ describe Geo::RepositoryVerification::Primary::BatchWorker, :postgresql, :clean_ ...@@ -17,66 +17,58 @@ describe Geo::RepositoryVerification::Primary::BatchWorker, :postgresql, :clean_
end end
describe '#perform' do describe '#perform' do
context 'when geo_repository_verification is enabled' do it 'skips backfill for repositories on other shards' do
before do create(:project, repository_storage: 'broken')
stub_feature_flags(geo_repository_verification: true) unhealthy_outdated = create(:project, repository_storage: 'broken')
end
it 'skips backfill for repositories on other shards' do
create(:project, repository_storage: 'broken')
unhealthy_outdated = create(:project, repository_storage: 'broken')
create(:repository_state, :repository_outdated, project: unhealthy_outdated) create(:repository_state, :repository_outdated, project: unhealthy_outdated)
allow(Gitlab::GitalyClient).to receive(:call) do allow(Gitlab::GitalyClient).to receive(:call) do
raise GRPC::Unavailable.new('No Gitaly available') raise GRPC::Unavailable.new('No Gitaly available')
end end
expect(Geo::RepositoryVerification::Primary::ShardWorker).not_to receive(:perform_async).with('broken') expect(Geo::RepositoryVerification::Primary::ShardWorker).not_to receive(:perform_async).with('broken')
subject.perform subject.perform
end end
it 'skips backfill for projects on missing shards' do it 'skips backfill for projects on missing shards' do
missing_not_verified = create(:project) missing_not_verified = create(:project)
missing_not_verified.update_column(:repository_storage, 'unknown') missing_not_verified.update_column(:repository_storage, 'unknown')
missing_outdated = create(:project) missing_outdated = create(:project)
missing_outdated.update_column(:repository_storage, 'unknown') missing_outdated.update_column(:repository_storage, 'unknown')
create(:repository_state, :repository_outdated, project: missing_outdated) create(:repository_state, :repository_outdated, project: missing_outdated)
# hide the 'broken' storage for this spec # hide the 'broken' storage for this spec
stub_storage_settings({}) stub_storage_settings({})
expect(Geo::RepositoryVerification::Primary::ShardWorker).to receive(:perform_async).with(healthy_shard) expect(Geo::RepositoryVerification::Primary::ShardWorker).to receive(:perform_async).with(healthy_shard)
expect(Geo::RepositoryVerification::Primary::ShardWorker).not_to receive(:perform_async).with('unknown') expect(Geo::RepositoryVerification::Primary::ShardWorker).not_to receive(:perform_async).with('unknown')
subject.perform subject.perform
end end
it 'skips backfill for projects with downed Gitaly server' do it 'skips backfill for projects with downed Gitaly server' do
create(:project, repository_storage: 'broken') create(:project, repository_storage: 'broken')
unhealthy_outdated = create(:project, repository_storage: 'broken') unhealthy_outdated = create(:project, repository_storage: 'broken')
create(:repository_state, :repository_outdated, project: unhealthy_outdated) create(:repository_state, :repository_outdated, project: unhealthy_outdated)
# Report only one healthy shard # Report only one healthy shard
expect(Gitlab::HealthChecks::GitalyCheck).to receive(:readiness) expect(Gitlab::HealthChecks::GitalyCheck).to receive(:readiness)
.and_return([result(true, healthy_shard), result(false, 'broken')]) .and_return([result(true, healthy_shard), result(false, 'broken')])
expect(Geo::RepositoryVerification::Primary::ShardWorker).to receive(:perform_async).with(healthy_shard) expect(Geo::RepositoryVerification::Primary::ShardWorker).to receive(:perform_async).with(healthy_shard)
expect(Geo::RepositoryVerification::Primary::ShardWorker).not_to receive(:perform_async).with('broken') expect(Geo::RepositoryVerification::Primary::ShardWorker).not_to receive(:perform_async).with('broken')
subject.perform subject.perform
end
end end
context 'when geo_repository_verification is disabled' do context 'when geo_repository_verification is disabled' do
before do
stub_feature_flags(geo_repository_verification: false)
end
it 'does not schedule jobs' do it 'does not schedule jobs' do
allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
expect(Geo::RepositoryVerification::Primary::ShardWorker) expect(Geo::RepositoryVerification::Primary::ShardWorker)
.not_to receive(:perform_async).with(healthy_shard) .not_to receive(:perform_async).with(healthy_shard)
......
...@@ -17,68 +17,60 @@ describe Geo::RepositoryVerification::Secondary::SchedulerWorker, :postgresql, : ...@@ -17,68 +17,60 @@ describe Geo::RepositoryVerification::Secondary::SchedulerWorker, :postgresql, :
end end
describe '#perform' do describe '#perform' do
context 'when geo_repository_verification is enabled' do it 'skips verification for repositories on other shards' do
before do create(:project, repository_storage: 'broken')
stub_feature_flags(geo_repository_verification: true)
end
it 'skips verification for repositories on other shards' do
create(:project, repository_storage: 'broken')
allow(Gitlab::GitalyClient).to receive(:call) do allow(Gitlab::GitalyClient).to receive(:call) do
raise GRPC::Unavailable.new('No Gitaly available') raise GRPC::Unavailable.new('No Gitaly available')
end end
expect(Geo::RepositoryVerification::Secondary::ShardWorker).not_to receive(:perform_async).with('broken') expect(Geo::RepositoryVerification::Secondary::ShardWorker).not_to receive(:perform_async).with('broken')
subject.perform subject.perform
end end
it 'skips verification for projects on missing shards' do it 'skips verification for projects on missing shards' do
missing_not_verified = create(:project) missing_not_verified = create(:project)
missing_not_verified.update_column(:repository_storage, 'unknown') missing_not_verified.update_column(:repository_storage, 'unknown')
# hide the 'broken' storage for this spec # hide the 'broken' storage for this spec
stub_storage_settings({}) stub_storage_settings({})
expect(Geo::RepositoryVerification::Secondary::ShardWorker).to receive(:perform_async).with(healthy_shard) expect(Geo::RepositoryVerification::Secondary::ShardWorker).to receive(:perform_async).with(healthy_shard)
expect(Geo::RepositoryVerification::Secondary::ShardWorker).not_to receive(:perform_async).with('unknown') expect(Geo::RepositoryVerification::Secondary::ShardWorker).not_to receive(:perform_async).with('unknown')
subject.perform subject.perform
end end
it 'skips verification for projects with downed Gitaly server' do it 'skips verification for projects with downed Gitaly server' do
create(:project, repository_storage: 'broken') create(:project, repository_storage: 'broken')
expect(Gitlab::HealthChecks::GitalyCheck).to receive(:readiness) expect(Gitlab::HealthChecks::GitalyCheck).to receive(:readiness)
.and_return([result(true, healthy_shard), result(false, 'broken')]) .and_return([result(true, healthy_shard), result(false, 'broken')])
expect(Geo::RepositoryVerification::Secondary::ShardWorker).to receive(:perform_async).with(healthy_shard) expect(Geo::RepositoryVerification::Secondary::ShardWorker).to receive(:perform_async).with(healthy_shard)
expect(Geo::RepositoryVerification::Secondary::ShardWorker).not_to receive(:perform_async).with('broken') expect(Geo::RepositoryVerification::Secondary::ShardWorker).not_to receive(:perform_async).with('broken')
subject.perform subject.perform
end end
it 'skips verification for projects on shards excluded by selective sync' do it 'skips verification for projects on shards excluded by selective sync' do
secondary.update!(selective_sync_type: 'shards', selective_sync_shards: [healthy_shard]) secondary.update!(selective_sync_type: 'shards', selective_sync_shards: [healthy_shard])
# Report both shards as healthy # Report both shards as healthy
expect(Gitlab::HealthChecks::GitalyCheck).to receive(:readiness) expect(Gitlab::HealthChecks::GitalyCheck).to receive(:readiness)
.and_return([result(true, healthy_shard), result(true, 'broken')]) .and_return([result(true, healthy_shard), result(true, 'broken')])
expect(Geo::RepositoryVerification::Secondary::ShardWorker).to receive(:perform_async).with(healthy_shard) expect(Geo::RepositoryVerification::Secondary::ShardWorker).to receive(:perform_async).with(healthy_shard)
expect(Geo::RepositoryVerification::Secondary::ShardWorker).not_to receive(:perform_async).with('broken') expect(Geo::RepositoryVerification::Secondary::ShardWorker).not_to receive(:perform_async).with('broken')
subject.perform subject.perform
end
end end
context 'when geo_repository_verification is disabled' do context 'when geo_repository_verification is disabled' do
before do
stub_feature_flags(geo_repository_verification: false)
end
it 'does not schedule jobs' do it 'does not schedule jobs' do
allow(Gitlab::Geo).to receive(:repository_verification_enabled?).and_return(false)
expect(Geo::RepositoryVerification::Secondary::ShardWorker) expect(Geo::RepositoryVerification::Secondary::ShardWorker)
.not_to receive(:perform_async).with(healthy_shard) .not_to receive(:perform_async).with(healthy_shard)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment