Merge branch 'bbodenmiller-master-patch-97329' into 'master'

Improve Job artifacts migration to object storage details

See merge request gitlab-org/gitlab!75989

doc/administration/job_artifacts.md

@@ -154,37 +154,7 @@ _The artifacts are stored by default in
    ```
 
 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-1. Migrate any existing local artifacts to the object storage:
-
-   ```shell
-   gitlab-rake gitlab:artifacts:migrate
-   ```
-
-1. Optional. Verify all files migrated properly.
-   From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database)
-   (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts:
-
-   ```shell
-   gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts;
-
-   total | filesystem | objectstg
-   ------+------------+-----------
-    2409 |          0 |      2409
-   ```
-
-   Verify no files on disk in `artifacts` folder:
-
-   ```shell
-   sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l
-   ```
-
-   In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files)
-   to clean up orphaned artifacts.
-
-WARNING:
-JUnit test report artifact (`junit.xml.gz`) migration
-[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991)
-by the `gitlab:artifacts:migrate` script.
+1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage).
 
 **In installations from source:**
 
@@ -208,36 +178,7 @@ _The artifacts are stored by default in
    ```
 
 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
-1. Migrate any existing local artifacts to the object storage:
-
-   ```shell
-   sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production
-   ```
-
-1. Optional: Verify all files migrated properly.
-   From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts:
-
-   ```shell
-   gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts;
-
-   total | filesystem | objectstg
-   ------+------------+-----------
-    2409 |          0 |      2409
-   ```
-
-   Verify no files on disk in `artifacts` folder:
-
-   ```shell
-   sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l
-   ```
-
-   In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files)
-   to clean up orphaned artifacts.
-
-WARNING:
-JUnit test report artifact (`junit.xml.gz`) migration
-[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991)
-by the `gitlab:artifacts:migrate` script.
+1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage).
 
 ### OpenStack example
 
@@ -267,11 +208,7 @@ _The uploads are stored by default in
    ```
 
 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-1. Migrate any existing local artifacts to the object storage:
-
-   ```shell
-   gitlab-rake gitlab:artifacts:migrate
-   ```
+1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage).
 
 ---
 
@@ -302,11 +239,55 @@ _The uploads are stored by default in
    ```
 
 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
-1. Migrate any existing local artifacts to the object storage:
+1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage).
 
-   ```shell
-   sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production
-   ```
+### Migrating to object storage
+
+After [configuring the object storage](#using-object-storage), use the following task to
+migrate existing job artifacts from the local storage to the remote storage.
+The processing is done in a background worker and requires **no downtime**.
+
+**In Omnibus installations:**
+
+```shell
+gitlab-rake gitlab:artifacts:migrate
+```
+
+**In installations from source:**
+
+```shell
+sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production
+```
+
+You can optionally track progress and verify that all job artifacts migrated successfully using the
+[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database):
+
+- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances.
+- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances.
+
+Verify `objectstg` below (where `store=2`) has count of all job artifacts:
+
+```shell
+gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts;
+
+total | filesystem | objectstg
+------+------------+-----------
+   19 |          0 |        19
+```
+
+Verify that there are no files on disk in the `artifacts` folder:
+
+```shell
+sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp | wc -l
+```
+
+In some cases, you need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files)
+to clean up orphaned artifacts.
+
+WARNING:
+JUnit test report artifact (`junit.xml.gz`) migration
+[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991)
+by the `gitlab:artifacts:migrate` Rake task.
 
 ### Migrating from object storage to local storage