Merge branch 'rename-table-without-downtime' into 'master'

Rename table without downtime helper

See merge request gitlab-org/gitlab!54354

config/initializers/active_record_renamed_table.rb

+# frozen_string_literal: true
+
+ActiveSupport.on_load(:active_record) do
+  ActiveRecord::ConnectionAdapters::SchemaCache.prepend(Gitlab::Database::SchemaCacheWithRenamedTable)
+end

doc/development/avoiding_downtime_in_migrations.md

@@ -365,6 +365,12 @@ if the application no longer uses the table.
 Renaming tables requires downtime as an application may continue
 using the old table name during/after a database migration.
 
+If the table and the ActiveRecord model is not in use yet, removing the old
+table and creating a new one is the preferred way to "rename" the table.
+
+Renaming a table is possible without downtime by following our multi-release
+[rename table process](database/rename_database_tables.md#rename-table-without-downtime).
+
 ## Adding Foreign Keys
 
 Adding foreign keys usually works in 3 steps:

doc/development/database/rename_database_tables.md

+---
+stage: Enablement
+group: Database
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Rename table without downtime
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54354) in GitLab 13.12.
+
+With our database helper methods built into GitLab, it's possible to rename a database table without downtime.
+
+The technique builds on top of database views, using the following steps:
+
+1. Rename the database table.
+1. Create a database view using the old table name by pointing to the new table name.
+1. Add workaround for ActiveRecord's schema cache.
+
+For example, consider that we are renaming the `issues` table name to `tickets`. Run:
+
+```sql
+BEGIN;
+  ALTER TABLE issues RENAME TO tickets;
+  CREATE VIEW issues AS SELECT * FROM tickets;
+COMMIT;
+```
+
+As database views do not expose the underlying table schema (default values, not null
+constraints, and indexes), we need further steps to update the application to use the new
+table name. ActiveRecord heavily relies on this data, for example, to initialize new
+models.
+
+To work around this limitation, we need to tell ActiveRecord to acquire this information
+from a different table using the new table name.
+
+## Migration strategy breakdown
+
+### Release N.M: Mark the ActiveRecord model's table
+
+Consider the current release as "Release N.M".
+
+In this release, register the database table so that it instructs ActiveRecord to fetch the
+database table information (for `SchemaCache`) using the new table name (if it's present). Otherwise, fall back
+to the old table name. This is necessary to avoid errors during a zero-downtime deployment.
+
+1. Edit the `TABLES_TO_BE_RENAMED` constant in: `lib/gitlab/database.rb`
+
+   ```ruby
+   TABLES_TO_BE_RENAMED = {
+     'issues' => 'tickets'
+   }.freeze
+   ```
+
+Note that, in this release (N.M), the `tickets` database table does not exist yet. This step is preparing for the actual table rename in release N.M+1.
+
+### Release N.M+1: Rename the database table
+
+Consider the next release as "Release N.M".
+
+Execute a standard migration (not a post-migration):
+
+```ruby
+  include Gitlab::Database::MigrationHelpers
+
+  def up
+    rename_table_safely(:issues, :tickets)
+  end
+
+  def down
+    undo_rename_table_safely(:issues, :tickets)
+  end
+```
+
+**Important notes:**
+
+- Let other developers know that the table is going to be renamed.
+  - Ping the `@gl-database` group in your merge request.
+  - Add a note in the Engineering Week-in-Review document: `table_name` is going to be renamed in N.M. Modifications to this table are not allowed in release N.M and N.M+1.
+- The helper method uses the standard `rename_table` helper from Rails for renaming the table.
+- The helper renames the sequence and the indexes. Sometimes it diverges from the standard Rails convention
+when naming indexes, so there is a possibility that not all indexes are properly renamed. After running
+the migration locally, check if there are inconsistently named indexes (`db/structure.sql`). Those can be
+renamed manually in a separate migration, which can be also part of the release M.N+1.
+- Foreign key columns might still contain the old table name. For smaller tables, follow our [standard column 
+rename process](../avoiding_downtime_in_migrations.md#renaming-columns)
+- Avoid renaming database tables which are using with triggers.
+- Table modifications (add or remove columns) are not allowed during the rename process, please make sure that all changes to the table happen before the rename migration is started (or in the next release).
+- As the index names might change, verify that the model does not use bulk insert
+(for example, `insert_all` and `upsert_all`) with the `unique_by: index_name` option.
+Renaming an index while using these methods may break functionality.
+- Modify the model code to point to the new database table. Do this by
+renaming the model directly or setting the `self.table_name` variable.
+
+At this point, we don't have applications using the old database table name in their queries.
+
+1. Remove the database view through a post-migration:
+
+   ```ruby
+     include Gitlab::Database::MigrationHelpers
+
+     def up
+       finalize_table_rename(:issues, :tickets)
+     end
+
+     def down
+       undo_finalize_table_rename(:issues, :tickets)
+     end
+   ```
+
+1. Additionally the table definition from `TABLES_TO_BE_RENAMED` **must** be removed.
+
+To do so, edit the `TABLES_TO_BE_RENAMED` constant in `lib/gitlab/database.rb`:
+
+   From:
+
+   ```ruby
+   TABLES_TO_BE_RENAMED = {
+     'issues' => 'tickets'
+   }.freeze
+   ```
+
+   To:
+
+   ```ruby
+   TABLES_TO_BE_RENAMED = {}.freeze
+   ```
+
+#### Zero-downtime deployments
+
+When the application is upgraded without downtime, there can be application instances
+running the old code. The old code still references the old database table. The queries
+still function without any problems, because the backward-compatible database view is
+in place.
+
+In case the old version of the application needs to be restarted or reconnected to the
+database, ActiveRecord fetches the column information again. At this time, our previously
+marked table (`TABLES_TO_BE_RENAMED`) instructs ActiveRecord to use the new database table name
+when fetching the database table information.
+
+The new version of the application will use the new database table.

lib/gitlab/database.rb

@@ -2,6 +2,14 @@
 
 module Gitlab
   module Database
+    # This constant is used when renaming tables concurrently.
+    # If you plan to rename a table using the `rename_table_safely` method, add your table here one milestone before the rename.
+    # Example:
+    # TABLES_TO_BE_RENAMED = {
+    #   'old_name' => 'new_name'
+    # }.freeze
+    TABLES_TO_BE_RENAMED = {}.freeze
+
     # Minimum PostgreSQL version requirement per documentation:
     # https://docs.gitlab.com/ee/install/requirements.html#postgresql-requirements
     MINIMUM_POSTGRES_VERSION = 11

lib/gitlab/database/migration_helpers.rb

@@ -5,6 +5,7 @@ module Gitlab
     module MigrationHelpers
       include Migrations::BackgroundMigrationHelpers
       include DynamicModelHelpers
+      include Migrations::RenameTableHelpers
 
       # https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS
       MAX_IDENTIFIER_NAME_LENGTH = 63

lib/gitlab/database/rename_table_helpers.rb

+# frozen_string_literal: true
+
+module Gitlab
+  module Database
+    module RenameTableHelpers
+      def rename_table_safely(old_table_name, new_table_name)
+        with_lock_retries do
+          rename_table(old_table_name, new_table_name)
+          execute("CREATE VIEW #{old_table_name} AS SELECT * FROM #{new_table_name}")
+        end
+      end
+
+      def undo_rename_table_safely(old_table_name, new_table_name)
+        with_lock_retries do
+          execute("DROP VIEW IF EXISTS #{old_table_name}")
+          rename_table(new_table_name, old_table_name)
+        end
+      end
+
+      def finalize_table_rename(old_table_name, new_table_name)
+        with_lock_retries do
+          execute("DROP VIEW IF EXISTS #{old_table_name}")
+        end
+      end
+
+      def undo_finalize_table_rename(old_table_name, new_table_name)
+        with_lock_retries do
+          execute("CREATE VIEW #{old_table_name} AS SELECT * FROM #{new_table_name}")
+        end
+      end
+    end
+  end
+end

lib/gitlab/database/schema_cache_with_renamed_table.rb

+# frozen_string_literal: true
+
+module Gitlab
+  module Database
+    module SchemaCacheWithRenamedTable
+      # Override methods in ActiveRecord::ConnectionAdapters::SchemaCache
+
+      def clear!
+        super
+
+        clear_renamed_tables_cache!
+      end
+
+      def clear_data_source_cache!(name)
+        super(name)
+
+        clear_renamed_tables_cache!
+      end
+
+      def primary_keys(table_name)
+        super(underlying_table(table_name))
+      end
+
+      def columns(table_name)
+        super(underlying_table(table_name))
+      end
+
+      def columns_hash(table_name)
+        super(underlying_table(table_name))
+      end
+
+      def indexes(table_name)
+        super(underlying_table(table_name))
+      end
+
+      private
+
+      def underlying_table(table_name)
+        renamed_tables_cache.fetch(table_name, table_name)
+      end
+
+      def renamed_tables_cache
+        @renamed_tables ||= begin
+          Gitlab::Database::TABLES_TO_BE_RENAMED.select do |old_name, new_name|
+            ActiveRecord::Base.connection.view_exists?(old_name)
+          end
+        end
+      end
+
+      def clear_renamed_tables_cache!
+        @renamed_tables = nil # rubocop:disable Gitlab/ModuleWithInstanceVariables
+      end
+    end
+  end
+end

spec/lib/gitlab/database/schema_cache_with_renamed_table_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+RSpec.describe Gitlab::Database::SchemaCacheWithRenamedTable do
+  let(:old_model) do
+    Class.new(ActiveRecord::Base) do
+      self.table_name = 'projects'
+    end
+  end
+
+  let(:new_model) do
+    Class.new(ActiveRecord::Base) do
+      self.table_name = 'projects_new'
+    end
+  end
+
+  before do
+    stub_const('Gitlab::Database::TABLES_TO_BE_RENAMED', { 'projects' => 'projects_new' })
+  end
+
+  context 'when table is not renamed yet' do
+    before do
+      old_model.reset_column_information
+      ActiveRecord::Base.connection.schema_cache.clear!
+    end
+
+    it 'uses the original table to look up metadata' do
+      expect(old_model.primary_key).to eq('id')
+    end
+  end
+
+  context 'when table is renamed' do
+    before do
+      ActiveRecord::Base.connection.execute("ALTER TABLE projects RENAME TO projects_new")
+      ActiveRecord::Base.connection.execute("CREATE VIEW projects AS SELECT * FROM projects_new")
+
+      old_model.reset_column_information
+      ActiveRecord::Base.connection.schema_cache.clear!
+    end
+
+    it 'uses the renamed table to look up metadata' do
+      expect(old_model.primary_key).to eq('id')
+    end
+
+    it 'has primary key' do
+      expect(old_model.primary_key).to eq('id')
+      expect(old_model.primary_key).to eq(new_model.primary_key)
+    end
+
+    it 'has the same column definitions' do
+      expect(old_model.columns).to eq(new_model.columns)
+    end
+
+    it 'has the same indexes' do
+      indexes_for_old_table = ActiveRecord::Base.connection.schema_cache.indexes('projects')
+      indexes_for_new_table = ActiveRecord::Base.connection.schema_cache.indexes('projects_new')
+
+      expect(indexes_for_old_table).to eq(indexes_for_new_table)
+    end
+
+    it 'has the same column_hash' do
+      columns_hash_for_old_table = ActiveRecord::Base.connection.schema_cache.columns_hash('projects')
+      columns_hash_for_new_table = ActiveRecord::Base.connection.schema_cache.columns_hash('projects_new')
+
+      expect(columns_hash_for_old_table).to eq(columns_hash_for_new_table)
+    end
+
+    describe 'when the table behind a model is actually a view' do
+      let(:group) { create(:group) }
+      let(:project_attributes) { attributes_for(:project, namespace_id: group.id).except(:creator) }
+      let(:record) { old_model.create!(project_attributes) }
+
+      it 'can persist records' do
+        expect(record.reload.attributes).to eq(new_model.find(record.id).attributes)
+      end
+
+      it 'can find records' do
+        expect(old_model.find_by_id(record.id)).not_to be_nil
+      end
+    end
+  end
+end