Merge branch '215065-update-database-guides-on-text-columns-and-check-constraints' into 'master'

Update database guides on text data type usage

Closes #214256 and #215065

See merge request gitlab-org/gitlab!30929

doc/development/README.md

@@ -113,50 +113,7 @@ Complementary reads:
 
 ## Database guides
 
-### Tooling
-
-- [Understanding EXPLAIN plans](understanding_explain_plans.md)
-- [explain.depesz.com](https://explain.depesz.com/) for visualizing the output
-  of `EXPLAIN`
-- [pgFormatter](http://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier
-
-### Migrations
-
-- [What requires downtime?](what_requires_downtime.md)
-- [SQL guidelines](sql.md) for working with SQL queries
-- [Migrations style guide](migration_style_guide.md) for creating safe SQL migrations
-- [Testing Rails migrations](testing_guide/testing_migrations_guide.md) guide
-- [Post deployment migrations](post_deployment_migrations.md)
-- [Background migrations](background_migrations.md)
-- [Swapping tables](swapping_tables.md)
-- [Deleting migrations](deleting_migrations.md)
-
-### Debugging
-
-- Tracing the source of an SQL query using query comments with [Marginalia](database_query_comments.md)
-- Tracing the source of an SQL query in Rails console using [Verbose Query Logs](https://guides.rubyonrails.org/debugging_rails_applications.html#verbose-query-logs)
-
-### Best practices
-
-- [Adding database indexes](adding_database_indexes.md)
-- [Foreign keys & associations](foreign_keys.md)
-- [Single table inheritance](single_table_inheritance.md)
-- [Polymorphic associations](polymorphic_associations.md)
-- [Serializing data](serializing_data.md)
-- [Hash indexes](hash_indexes.md)
-- [Storing SHA1 hashes as binary](sha1_as_binary.md)
-- [Iterating tables in batches](iterating_tables_in_batches.md)
-- [Insert into tables in batches](insert_into_tables_in_batches.md)
-- [Ordering table columns](ordering_table_columns.md)
-- [Verifying database capabilities](verifying_database_capabilities.md)
-- [Database Debugging and Troubleshooting](database_debugging.md)
-- [Query Count Limits](query_count_limits.md)
-- [Creating enums](creating_enums.md)
-
-### Case studies
-
-- [Database case study: Filtering by label](filtering_by_label.md)
-- [Database case study: Namespaces storage statistics](namespaces_storage_statistics.md)
+See [database guidelines](database/index.md).
 
 ## Integration guides
 

doc/development/database/index.md

+# Database guides
+
+## Tooling
+
+- [Understanding EXPLAIN plans](../understanding_explain_plans.md)
+- [explain.depesz.com](https://explain.depesz.com/) for visualizing the output
+  of `EXPLAIN`
+- [pgFormatter](http://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier
+
+## Migrations
+
+- [What requires downtime?](../what_requires_downtime.md)
+- [SQL guidelines](../sql.md) for working with SQL queries
+- [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations
+- [Testing Rails migrations](../testing_guide/testing_migrations_guide.md) guide
+- [Post deployment migrations](../post_deployment_migrations.md)
+- [Background migrations](../background_migrations.md)
+- [Swapping tables](../swapping_tables.md)
+- [Deleting migrations](../deleting_migrations.md)
+
+## Debugging
+
+- Tracing the source of an SQL query using query comments with [Marginalia](../database_query_comments.md)
+- Tracing the source of an SQL query in Rails console using [Verbose Query Logs](https://guides.rubyonrails.org/debugging_rails_applications.html#verbose-query-logs)
+
+## Best practices
+
+- [Adding database indexes](../adding_database_indexes.md)
+- [Foreign keys & associations](../foreign_keys.md)
+- [Adding a foreign key constraint to an existing column](add_foreign_key_to_existing_column.md)
+- [Strings and the Text data type](strings_and_the_text_data_type.md)
+- [Single table inheritance](../single_table_inheritance.md)
+- [Polymorphic associations](../polymorphic_associations.md)
+- [Serializing data](../serializing_data.md)
+- [Hash indexes](../hash_indexes.md)
+- [Storing SHA1 hashes as binary](../sha1_as_binary.md)
+- [Iterating tables in batches](../iterating_tables_in_batches.md)
+- [Insert into tables in batches](../insert_into_tables_in_batches.md)
+- [Ordering table columns](../ordering_table_columns.md)
+- [Verifying database capabilities](../verifying_database_capabilities.md)
+- [Database Debugging and Troubleshooting](../database_debugging.md)
+- [Query Count Limits](../query_count_limits.md)
+- [Creating enums](../creating_enums.md)
+
+## Case studies
+
+- [Database case study: Filtering by label](../filtering_by_label.md)
+- [Database case study: Namespaces storage statistics](../namespaces_storage_statistics.md)

doc/development/database/strings_and_the_text_data_type.md

+# Strings and the Text data type
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30453) in GitLab 13.0.
+
+When adding new columns that will be used to store strings or other textual information:
+
+1. We always use the `text` data type instead of the `string` data type.
+1. `text` columns should always have a limit set by using the `add_text_limit` migration helper.
+
+The `text` data type can not be defined with a limit, so `add_text_limit` is enforcing that by
+adding a [check constraint](https://www.postgresql.org/docs/11/ddl-constraints.html) on the
+column and then validating it at a followup step.
+
+## Background info
+
+The reason we always want to use `text` instead of `string` is that `string` columns have the
+disadvantage that if you want to update their limit, you have to run an `ALTER TABLE ...` command.
+
+While a limit is added, the `ALTER TABLE ...` command requires an `EXCLUSIVE LOCK` on the table, which
+is held throughout the process of updating the column and while validating all existing records, a
+process that can take a while for large tables.
+
+On the other hand, texts are [more or less equivalent to strings](https://www.depesz.com/2010/03/02/charx-vs-varcharx-vs-varchar-vs-text/) in PostgreSQL,
+while having the additional advantage that adding a limit on an existing column or updating their
+limit does not require the very costly `EXCLUSIVE LOCK` to be held throughout the validation phase.
+We can start by updating the constraint with the valid option off, which requires an `EXCLUSIVE LOCK`
+but only for updating the declaration of the columns. We can then validate it at a later step using
+`VALIDATE CONSTRAINT`, which requires only a `SHARE UPDATE EXCLUSIVE LOCK` (only conflicts with other
+validations and index creation while it allows reads and writes).
+
+## Create a new table with text columns
+
+When adding a new table, the limits for all text columns should be added in the same migration as
+the table creation.
+
+For example, consider a migration that creates a table with two text columns,
+**db/migrate/20200401000001_create_db_guides.rb**:
+
+```ruby
+class CreateDbGuides < ActiveRecord::Migration[6.0]
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    unless table_exists?(:db_guides)
+      create_table :db_guides do |t|
+        t.bigint :stars, default: 0, null: false
+        t.text :title
+        t.text :notes
+      end
+    end
+
+    # The following add the constraints and validate them immediately (no data in the table)
+    add_text_limit :db_guides, :title, 128
+    add_text_limit :db_guides, :notes, 1024
+  end
+
+  def down
+    # No need to drop the constraints, drop_table takes care of everything
+    drop_table :db_guides
+  end
+end
+```
+
+Adding a check constraint requires an exclusive lock while the `ALTER TABLE` that adds is running.
+As we don't want the exclusive lock to be held for the duration of a transaction, `add_text_limit`
+must always run in a migration with `disable_ddl_transaction!`.
+
+Also, note that we have to add a check that the table exists so that the migration can be repeated
+in case of a failure.
+
+## Add a text column to an existing table
+
+Adding a column to an existing table requires an exclusive lock for that table. Even though that lock
+is held for a brief amount of time, the time `add_column` needs to complete its execution can vary
+depending on how frequently the table is accessed. For example, acquiring an exclusive lock for a very
+frequently accessed table may take minutes in GitLab.com and requires the use of `with_lock_retries`.
+
+For these reasons, it is advised to add the text limit on a separate migration than the `add_column` one.
+
+For example, consider a migration that adds a new text column `extended_title` to table `sprints`,
+**db/migrate/20200501000001_add_extended_title_to_sprints.rb**:
+
+```ruby
+class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0]
+  DOWNTIME = false
+
+  # rubocop:disable Migration/AddLimitToTextColumns
+  # limit is added in 20200501000002_add_text_limit_to_sprints_extended_title
+  def change
+    add_column :sprints, :extended_title, :text
+  end
+  # rubocop:enable Migration/AddLimitToTextColumns
+end
+```
+
+A second migration should follow the first one with a limit added to `extended_title`,
+**db/migrate/20200501000002_add_text_limit_to_sprints_extended_title.rb**:
+
+```ruby
+class AddTextLimitToSprintsExtendedTitle < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    add_text_limit :sprints, :extended_title, 512
+  end
+
+  def down
+    # Down is required as `add_text_limit` is not reversible
+    remove_text_limit :sprints, :extended_title
+  end
+end
+```
+
+## Add a text limit constraint to an existing column
+
+Adding text limits to existing database columns requires multiple steps split into at least two different releases:
+
+1. Release `N.M` (current release)
+
+   - Add a post-deployment migration to add the limit to the text column with `validate: false`.
+   - Add a post-deployment migration to fix the existing records.
+
+     NOTE: **Note:**
+     Depending on the size of the table, a background migration for cleanup could be required in the next release.
+     See [text limit constraints on large tables](strings_and_the_text_data_type.md#text-limit-constraints-on-large-tables) for more information.
+
+   - Create an issue for the next milestone to validate the text limit.
+
+1. Release `N.M+1` (next release)
+
+   - Validate the text limit using a post-deployment migration.
+
+### Example
+
+Let's assume we want to add a `1024` limit to `issues.title_html` for a given release milestone,
+such as 13.0.
+
+Issues is a pretty busy and large table with more than 25 million rows, so we don't want to lock all
+other processes that try to access it while running the update.
+
+Also, after checking our production database, we know that there are `issues` with more characters in
+their title than the 1024 character limit, so we can not add and validate the constraint in one step.
+
+NOTE: **Note:**
+Even if we did not have any record with a title larger than the provided limit, another
+instance of GitLab could have such records, so we would follow the same process either way.
+
+#### Prevent new invalid records (current release)
+
+We first add the limit as a `NOT VALID` check constraint to the table, which enforces consistency when
+new records are inserted or current records are updated.
+
+In the example above, the existing issues with more than 1024 characters in their title will not be
+affected and you'll be still able to update records in the `issues` table. However, when you'd try
+to update the `title_html` with a title that has more than 1024 characters, the constraint causes
+a database error.
+
+Adding or removing a constraint to an existing attribute requires that any application changes are
+deployed _first_, [otherwise servers still in the old version of the application may try to update the
+attribute with invalid values](../multi_version_compatibility.md#ci-artifact-uploads-were-failing).
+For these reasons, `add_text_limit` should run in a post-deployment migration.
+
+Still in our example, for the 13.0 milestone (current), consider that the following validation
+has been added to model `Issue`:
+
+```ruby
+validates :title_html, length: { maximum: 1024 }
+```
+
+We can also update the database in the same milestone by adding the text limit with `validate: false`
+in a post-deployment migration,
+**db/post_migrate/20200501000001_add_text_limit_migration.rb**:
+
+```ruby
+class AddTextLimitMigration < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    # This will add the constraint WITHOUT validating it
+    add_text_limit :issues, :title_html, 1024, validate: false
+  end
+
+  def down
+    # Down is required as `add_text_limit` is not reversible
+    remove_text_limit :issues, :title_html
+  end
+end
+```
+
+#### Data migration to fix existing records (current release)
+
+The approach here depends on the data volume and the cleanup strategy. The number of records that must
+be fixed on GitLab.com is a nice indicator that will help us decide whether to use a post-deployment
+migration or a background data migration:
+
+- If the data volume is less than `1,000` records, then the data migration can be executed within the post-migration.
+- If the data volume is higher than `1,000` records, it's advised to create a background migration.
+
+When unsure about which option to use, please contact the Database team for advice.
+
+Back to our example, the issues table is considerably large and frequently accessed, so we are going
+to add a background migration for the 13.0 milestone (current),
+**db/post_migrate/20200501000002_schedule_cap_title_length_on_issues.rb**:
+
+```ruby
+class ScheduleCapTitleLengthOnIssues < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+
+  # Info on how many records will be affected on GitLab.com
+  # time each batch needs to run on average, etc ...
+  BATCH_SIZE = 5000
+  DELAY_INTERVAL = 2.minutes.to_i
+
+  # Background migration will update issues whose title is longer than 1024 limit
+  ISSUES_BACKGROUND_MIGRATION = 'CapTitleLengthOnIssues'.freeze
+
+  disable_ddl_transaction!
+
+  class Issue < ActiveRecord::Base
+    include EachBatch
+
+    self.table_name = 'issues'
+  end
+
+  def up
+    queue_background_migration_jobs_by_range_at_intervals(
+      Issue.where('char_length(title_html) > 1024'),
+      ISSUES_MIGRATION,
+      DELAY_INTERVAL,
+      batch_size: BATCH_SIZE
+    )
+  end
+
+  def down
+    # no-op : the part of the title_html after the limit is lost forever
+  end
+end
+```
+
+To keep this guide short, we skipped the definition of the background migration and only
+provided a high level example of the post-deployment migration that is used to schedule the batches.
+You can find more info on the guide about [background migrations](../background_migrations.md)
+
+#### Validate the text limit (next release)
+
+Validating the text limit will scan the whole table and make sure that each record is correct.
+
+Still in our example, for the 13.1 milestone (next), we run the `validate_text_limit` migration
+helper in a final post-deployment migration,
+**db/post_migrate/20200601000001_validate_text_limit_migration.rb**:
+
+```ruby
+class ValidateTextLimitMigration < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    validate_text_limit :issues, :title_html
+  end
+
+  def down
+    # no-op
+  end
+end
+```
+
+## Text limit constraints on large tables
+
+If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/migration_helpers.rb#L12)
+(for example, the `artifacts` in `ci_builds`), your background migration will go on for a while and
+it will need an additional [background migration cleaning up](../background_migrations.md#cleaning-up)
+in the release after adding the data migration.
+
+In that rare case you will need 3 releases end-to-end:
+
+1. Release `N.M` - Add the text limit and the background migration to fix the existing records.
+1. Release `N.M+1` - Cleanup the background migration.
+1. Release `N.M+2` - Validate the text limit.

doc/development/migration_style_guide.md

@@ -727,6 +727,12 @@ Rails migration example:
 add_column(:projects, :foo, :integer, default: 10, limit: 8)
 ```
 
+## Strings and the Text data type
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30453) in GitLab 13.0.
+
+See the [text data type](database/strings_and_the_text_data_type.md) style guide for more information.
+
 ## Timestamp column type
 
 By default, Rails uses the `timestamp` data type that stores timestamp data