Commit 94c1828b authored by Ben Prescott @bprescott_↙ ☺'s avatar Ben Prescott @bprescott_↙ ☺ Committed by Marcia Ramos

Clarify 14.1 on the upgrade path

parent a90adab3
...@@ -12,6 +12,11 @@ GitLab version is, if you're upgrading to a major version, and so on. ...@@ -12,6 +12,11 @@ GitLab version is, if you're upgrading to a major version, and so on.
Make sure to read the whole page as it contains information related to every upgrade method. Make sure to read the whole page as it contains information related to every upgrade method.
NOTE:
Upgrade GitLab to the latest available patch release, for example `13.8.8` rather than `13.8.0`.
This includes [versions you must stop at on the upgrade path](#upgrade-paths) as there may
be fixes for issues relating to the upgrade process.
The [maintenance policy documentation](../policy/maintenance.md) The [maintenance policy documentation](../policy/maintenance.md)
has additional information about upgrading, including: has additional information about upgrading, including:
...@@ -76,22 +81,44 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md). ...@@ -76,22 +81,44 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md).
## Checking for background migrations before upgrading ## Checking for background migrations before upgrading
Certain major/minor releases may require different migrations to be Certain releases may require different migrations to be
finished before you update to the newer version. finished before you update to the newer version. Additionally check
[batched migrations](#batched-background-migrations) from GitLab 14.0.
Decrease the time required to complete these migrations by increasing the number of Decrease the time required to complete these migrations by increasing the number of
[Sidekiq workers](../administration/operations/extra_sidekiq_processes.md) [Sidekiq workers](../administration/operations/extra_sidekiq_processes.md)
that can process jobs in the `background_migration` queue. that can process jobs in the `background_migration` queue.
**For GitLab 14.0 and newer** **For Omnibus installations:**
```shell
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
```
**For installations from source:**
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
```
### Batched background migrations
GitLab 14.0 introduced [batched background migrations](../user/admin_area/monitoring/background_migrations.md).
Some installations [may need to run GitLab 14.0 for at least a day](#1400) to complete the database changes introduced by that upgrade.
To check the status of [batched background migrations](../user/admin_area/monitoring/background_migrations.md): #### Check the status of batched background migrations
To check the status of batched background migrations:
1. On the top bar, select **Menu > Admin**. 1. On the top bar, select **Menu > Admin**.
1. On the left sidebar, select **Monitoring > Background Migrations**. 1. On the left sidebar, select **Monitoring > Background Migrations**.
![queued batched background migrations table](img/batched_background_migrations_queued_v14_0.png) ![queued batched background migrations table](img/batched_background_migrations_queued_v14_0.png)
All migrations must have a `Finished` status before you upgrade GitLab.
The status of batched background migrations can also be queried directly in the database. The status of batched background migrations can also be queried directly in the database.
1. Log into a `psql` prompt according to the directions for your instance's installation method 1. Log into a `psql` prompt according to the directions for your instance's installation method
...@@ -102,20 +129,14 @@ The status of batched background migrations can also be queried directly in the ...@@ -102,20 +129,14 @@ The status of batched background migrations can also be queried directly in the
select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3; select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
``` ```
**For Omnibus installations** If the migrations are not finished and you try to update to a later version,
GitLab prompts you with an error:
You can also run:
```shell ```plaintext
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
``` ```
**For installations from source** If you get this error, [check the batched background migration options](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished) to complete the upgrade.
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
```
### What do I do if my background migrations are stuck? ### What do I do if my background migrations are stuck?
...@@ -148,6 +169,10 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background ...@@ -148,6 +169,10 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background
pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }
``` ```
**Batched migrations (GitLab 14.0 and newer):**
See [troubleshooting batched background migrations](../user/admin_area/monitoring/background_migrations.md#troubleshooting).
## Dealing with running CI/CD pipelines and jobs ## Dealing with running CI/CD pipelines and jobs
If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. However, depending on the error, the GitLab Runner either retries or eventually terminates job handling. If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. However, depending on the error, the GitLab Runner either retries or eventually terminates job handling.
...@@ -197,7 +222,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab ...@@ -197,7 +222,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab
accordingly, while also consulting the accordingly, while also consulting the
[version-specific upgrade instructions](#version-specific-upgrading-instructions): [version-specific upgrade instructions](#version-specific-upgrading-instructions):
`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [latest `13.12.Z`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) `8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/)
The following table, while not exhaustive, shows some examples of the supported The following table, while not exhaustive, shows some examples of the supported
upgrade paths. upgrade paths.
...@@ -320,9 +345,11 @@ Git 2.33.x and later is required. We recommend you use the ...@@ -320,9 +345,11 @@ Git 2.33.x and later is required. We recommend you use the
### 14.3.0 ### 14.3.0
- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading
to 14.3.Z from earlier GitLab 14 releases.
- Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) - Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby)
for how to proceed. for how to proceed.
- GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below:
- [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245) - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245)
...@@ -343,13 +370,10 @@ for how to proceed. ...@@ -343,13 +370,10 @@ for how to proceed.
### 14.2.0 ### 14.2.0
- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
[not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - Ensure [batched background migrations finish](#batched-background-migrations) before upgrading
for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) to 14.2.Z from earlier GitLab 14 releases.
and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need
to update to at least 14.1.0 that contains the same fix before you update to 14.2.
- GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below:
- [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216) - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216)
- [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123) - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123)
- [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433) - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433)
...@@ -372,35 +396,24 @@ for how to proceed. ...@@ -372,35 +396,24 @@ for how to proceed.
### 14.1.0 ### 14.1.0
Due to an issue where `BatchedBackgroundMigrationWorkers` were - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases)
[not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) but can upgrade to 14.1.Z.
for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z.
and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need 14.1 is included on the upgrade path for the broadest compatibility
to update to at least 14.1.0 that contains the same fix before you update to with self-managed installations, and ensure 14.0.0-14.0.4 installations do not
a later version. encounter issues with [batched background migrations](#batched-background-migrations).
After you update to 14.1.0,
[batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations)
before you update to a later version.
If the migrations are not finished and you try to update to a later version,
you'll see an error like:
```plaintext
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
```
See how to [resolve this error](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished).
### 14.0.0 ### 14.0.0
- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances.
These [batched background migrations](#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or higher.
- Due to an issue where `BatchedBackgroundMigrationWorkers` were - Due to an issue where `BatchedBackgroundMigrationWorkers` were
[not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345)
for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106)
that requires an update to at least 14.0.5. that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410).
After you update to 14.0.5 or a later 14.0 patch version, After you update to 14.0.5 or a later 14.0 patch version,
[batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) [batched background migrations need to finish](#batched-background-migrations)
before you update to a later version. before you update to a later version.
If the migrations are not finished and you try to update to a later version, If the migrations are not finished and you try to update to a later version,
...@@ -419,6 +432,16 @@ See how to [resolve this error](../user/admin_area/monitoring/background_migrati ...@@ -419,6 +432,16 @@ See how to [resolve this error](../user/admin_area/monitoring/background_migrati
You should instead follow a [supported upgrade path](#upgrade-paths). You should instead follow a [supported upgrade path](#upgrade-paths).
- The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0.
#### Upgrading to later 14.Y releases
- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later,
because of [batched background migrations](#batched-background-migrations).
1. Upgrade first to either:
- 14.0.5 or a later 14.0.Z patch release.
- 14.1.0 or a later 14.1.Z patch release.
1. [Batched background migrations need to finish](#batched-background-migrations)
before you update to a later version [and may take longer than usual](#1400).
### 13.11.0 ### 13.11.0
Git 2.31.x and later is required. We recommend you use the Git 2.31.x and later is required. We recommend you use the
......
...@@ -24,7 +24,7 @@ prevent integer overflow for some tables. ...@@ -24,7 +24,7 @@ prevent integer overflow for some tables.
## Check the status of background migrations ## Check the status of background migrations
All migrations must have a `Finished` status before you [upgrade GitLab](../../../update/index.md). All migrations must have a `Finished` status before you [upgrade GitLab](../../../update/index.md).
You can [check the status of existing migrations](../../../update/index.md#checking-for-background-migrations-before-upgrading). You can [check the status of existing migrations](../../../update/index.md#batched-background-migrations).
## Enable or disable batched background migrations ## Enable or disable batched background migrations
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment