Simplify adding new Redis instance

The main sticking point was `config/README.md`, which went into a lot of
detail about each instance. It was quite easy to make a copy and paste
error, as some instances had default URLs and others didn't, some had
fallback instances and others didn't, etc.

This changes a number of things:

1. `config/README.md` now has a table for the Redis instances with a
   single list of instructions that apply to all instances.
2. There are now specs to ensure that each Redis instance has a default
   URL set when no Redis config at all is present.
3. Instances that use a fallback instance for config will automatically
   use the default URL of their fallback instance. They should not
   define their own default URL.
4. The `.gitignore` file does not need to be updated for each new Redis
   instance.
5. The example config files are now gone as `request.yml.example` should
   be sufficient.

.gitignore

@@ -40,11 +40,7 @@ eslint-report.html
 /config/initializers/smtp_settings.rb
 /config/initializers/relative_url.rb
 /config/resque.yml
-/config/redis.cache.yml
-/config/redis.queues.yml
-/config/redis.shared_state.yml
-/config/redis.trace_chunks.yml
-/config/redis.rate_limiting.yml
+/config/redis.*.yml
 /config/unicorn.rb
 /config/puma.rb
 /config/secrets.yml

config/redis.cache.yml.example

-# If you change this file in a merge request, please also create
-# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
-#
-development:
-  url: redis://localhost:6379/10
-  #
-  # url: redis://localhost:6380
-  # sentinels:
-  #   -
-  #     host: localhost
-  #     port: 26380 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26380 # point to sentinel, not to redis port
-test:
-  url: redis://localhost:6379/10
-  #
-  # url: redis://localhost:6380
-production:
-  # Redis (single instance)
-  url: unix:/var/run/redis/redis.cache.sock
-  ##
-  # Redis + Sentinel (for HA)
-  #
-  # Please read instructions carefully before using it as you may lose data:
-  # http://redis.io/topics/sentinel
-  #
-  # You must specify a list of a few sentinels that will handle client connection
-  # please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
-  ##
-  # url: redis://master:6380
-  # sentinels:
-  #   -
-  #     host: replica1
-  #     port: 26380 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26380 # point to sentinel, not to redis port

config/redis.queues.yml.example

-# If you change this file in a merge request, please also create
-# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
-#
-development:
-  url: redis://localhost:6379/11
-  #
-  # url: redis://localhost:6381
-  # sentinels:
-  #   -
-  #     host: localhost
-  #     port: 26381 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26381 # point to sentinel, not to redis port
-test:
-  url: redis://localhost:6379/11
-  #
-  # url: redis://localhost:6381
-production:
-  # Redis (single instance)
-  url: unix:/var/run/redis/redis.queues.sock
-  ##
-  # Redis + Sentinel (for HA)
-  #
-  # Please read instructions carefully before using it as you may lose data:
-  # http://redis.io/topics/sentinel
-  #
-  # You must specify a list of a few sentinels that will handle client connection
-  # please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
-  ##
-  # url: redis://master:6381
-  # sentinels:
-  #   -
-  #     host: replica1
-  #     port: 26381 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26381 # point to sentinel, not to redis port

config/redis.rate_limiting.yml.example

-# If you change this file in a merge request, please also create
-# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
-#
-development:
-  url: redis://localhost:6379/4
-  #
-  # url: redis://localhost:6382
-  # sentinels:
-  #   -
-  #     host: localhost
-  #     port: 26382 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26382 # point to sentinel, not to redis port
-test:
-  url: redis://localhost:6379/14
-  #
-  # url: redis://localhost:6382
-production:
-  # Redis (single instance)
-  url: unix:/var/run/redis/redis.rate_limiting.sock
-  ##
-  # Redis + Sentinel (for HA)
-  #
-  # Please read instructions carefully before using it as you may lose data:
-  # http://redis.io/topics/sentinel
-  #
-  # You must specify a list of a few sentinels that will handle client connection
-  # please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
-  ##
-  # url: redis://master:6382
-  # sentinels:
-  #   -
-  #     host: replica1
-  #     port: 26382 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26382 # point to sentinel, not to redis port

config/redis.shared_state.yml.example

-# If you change this file in a merge request, please also create
-# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
-#
-development:
-  url: redis://localhost:6379/12
-  #
-  # url: redis://localhost:6382
-  # sentinels:
-  #   -
-  #     host: localhost
-  #     port: 26382 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26382 # point to sentinel, not to redis port
-test:
-  url: redis://localhost:6379/12
-  #
-  # url: redis://localhost:6382
-production:
-  # Redis (single instance)
-  url: unix:/var/run/redis/redis.shared_state.sock
-  ##
-  # Redis + Sentinel (for HA)
-  #
-  # Please read instructions carefully before using it as you may lose data:
-  # http://redis.io/topics/sentinel
-  #
-  # You must specify a list of a few sentinels that will handle client connection
-  # please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
-  ##
-  # url: redis://master:6382
-  # sentinels:
-  #   -
-  #     host: replica1
-  #     port: 26382 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26382 # point to sentinel, not to redis port

config/redis.trace_chunks.yml.example

-# If you change this file in a merge request, please also create
-# a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
-#
-development:
-  url: redis://localhost:6379/13
-  #
-  # url: redis://localhost:6382
-  # sentinels:
-  #   -
-  #     host: localhost
-  #     port: 26382 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26382 # point to sentinel, not to redis port
-test:
-  url: redis://localhost:6379/13
-  #
-  # url: redis://localhost:6382
-production:
-  # Redis (single instance)
-  url: unix:/var/run/redis/redis.trace_chunks.sock
-  ##
-  # Redis + Sentinel (for HA)
-  #
-  # Please read instructions carefully before using it as you may lose data:
-  # http://redis.io/topics/sentinel
-  #
-  # You must specify a list of a few sentinels that will handle client connection
-  # please read here for more information: https://docs.gitlab.com/ee/administration/redis/index.html
-  ##
-  # url: redis://master:6382
-  # sentinels:
-  #   -
-  #     host: replica1
-  #     port: 26382 # point to sentinel, not to redis port
-  #   -
-  #     host: replica2
-  #     port: 26382 # point to sentinel, not to redis port

lib/gitlab/redis/cache.rb

@@ -4,14 +4,6 @@ module Gitlab
   module Redis
     class Cache < ::Gitlab::Redis::Wrapper
       CACHE_NAMESPACE = 'cache:gitlab'
-
-      private
-
-      def raw_config_hash
-        config = super
-        config[:url] = 'redis://localhost:6380' if config[:url].blank?
-        config
-      end
     end
   end
 end

lib/gitlab/redis/queues.rb

@@ -2,21 +2,12 @@
 
 # We need this require for MailRoom
 require_relative 'wrapper' unless defined?(::Gitlab::Redis::Wrapper)
-require 'active_support/core_ext/object/blank'
 
 module Gitlab
   module Redis
     class Queues < ::Gitlab::Redis::Wrapper
       SIDEKIQ_NAMESPACE = 'resque:gitlab'
       MAILROOM_NAMESPACE = 'mail_room:gitlab'
-
-      private
-
-      def raw_config_hash
-        config = super
-        config[:url] = 'redis://localhost:6381' if config[:url].blank?
-        config
-      end
     end
   end
 end

lib/gitlab/redis/shared_state.rb

@@ -7,14 +7,6 @@ module Gitlab
       USER_SESSIONS_NAMESPACE = 'session:user:gitlab'
       USER_SESSIONS_LOOKUP_NAMESPACE = 'session:lookup:user:gitlab'
       IP_SESSIONS_LOOKUP_NAMESPACE = 'session:lookup:ip:gitlab2'
-
-      private
-
-      def raw_config_hash
-        config = super
-        config[:url] = 'redis://localhost:6382' if config[:url].blank?
-        config
-      end
     end
   end
 end

lib/gitlab/redis/wrapper.rb

@@ -6,6 +6,7 @@
 # Rails.
 require 'active_support/core_ext/hash/keys'
 require 'active_support/core_ext/module/delegation'
+require 'active_support/core_ext/object/blank'
 require 'active_support/core_ext/string/inflections'
 
 # Explicitly load Redis::Store::Factory so we can read Redis configuration in
@@ -150,11 +151,35 @@ module Gitlab
       def raw_config_hash
         config_data = fetch_config
 
+        config_hash =
           if config_data
             config_data.is_a?(String) ? { url: config_data } : config_data.deep_symbolize_keys
           else
             { url: '' }
           end
+
+        if config_hash[:url].blank?
+          config_hash[:url] = legacy_fallback_urls[self.class.store_name] || legacy_fallback_urls[self.class.config_fallback.store_name]
+        end
+
+        config_hash
+      end
+
+      # These URLs were defined for cache, queues, and shared_state in
+      # code. They are used only when no config file exists at all for a
+      # given instance. The configuration does not seem particularly
+      # useful - it uses different ports on localhost - but we cannot
+      # confidently delete it as we don't know if any instances rely on
+      # this.
+      #
+      # DO NOT ADD new instances here. All new instances should define a
+      # `.config_fallback`, which will then be used to look up this URL.
+      def legacy_fallback_urls
+        {
+          'Cache' => 'redis://localhost:6380',
+          'Queues' => 'redis://localhost:6381',
+          'SharedState' => 'redis://localhost:6382'
+        }
       end
 
       def fetch_config

scripts/prepare_build.sh

@@ -38,21 +38,6 @@ sed -i 's|url:.*$|url: redis://redis:6379|g' config/cable.yml
 cp config/resque.yml.example config/resque.yml
 sed -i 's|url:.*$|url: redis://redis:6379|g' config/resque.yml
 
-cp config/redis.cache.yml.example config/redis.cache.yml
-sed -i 's|url:.*$|url: redis://redis:6379/10|g' config/redis.cache.yml
-
-cp config/redis.queues.yml.example config/redis.queues.yml
-sed -i 's|url:.*$|url: redis://redis:6379/11|g' config/redis.queues.yml
-
-cp config/redis.shared_state.yml.example config/redis.shared_state.yml
-sed -i 's|url:.*$|url: redis://redis:6379/12|g' config/redis.shared_state.yml
-
-cp config/redis.trace_chunks.yml.example config/redis.trace_chunks.yml
-sed -i 's|url:.*$|url: redis://redis:6379/13|g' config/redis.trace_chunks.yml
-
-cp config/redis.rate_limiting.yml.example config/redis.rate_limiting.yml
-sed -i 's|url:.*$|url: redis://redis:6379/14|g' config/redis.rate_limiting.yml
-
 if [ "$SETUP_DB" != "false" ]; then
   setup_db
 elif getent hosts postgres; then

spec/support/redis/redis_shared_examples.rb

@@ -327,6 +327,12 @@ RSpec.shared_examples "redis_shared_examples" do
 
       expect(subject.send(:fetch_config)).to eq false
     end
+
+    it 'has a value for the legacy default URL' do
+      allow(subject).to receive(:fetch_config) { false }
+
+      expect(subject.send(:raw_config_hash)).to include(url: a_string_matching(%r{\Aredis://localhost:638[012]\Z}))
+    end
   end
 
   def clear_raw_config