Commit e634623d authored by Evan Read's avatar Evan Read

Merge branch 'bbodenmiller-master-patch-97329' into 'master'

Improve Job artifacts migration to object storage details

See merge request gitlab-org/gitlab!75989
parents 60cd1b1f 9867cf52
......@@ -154,37 +154,7 @@ _The artifacts are stored by default in
```
1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Migrate any existing local artifacts to the object storage:
```shell
gitlab-rake gitlab:artifacts:migrate
```
1. Optional. Verify all files migrated properly.
From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database)
(`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts:
```shell
gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts;
total | filesystem | objectstg
------+------------+-----------
2409 | 0 | 2409
```
Verify no files on disk in `artifacts` folder:
```shell
sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l
```
In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files)
to clean up orphaned artifacts.
WARNING:
JUnit test report artifact (`junit.xml.gz`) migration
[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991)
by the `gitlab:artifacts:migrate` script.
1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage).
**In installations from source:**
......@@ -208,36 +178,7 @@ _The artifacts are stored by default in
```
1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
1. Migrate any existing local artifacts to the object storage:
```shell
sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production
```
1. Optional: Verify all files migrated properly.
From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts:
```shell
gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts;
total | filesystem | objectstg
------+------------+-----------
2409 | 0 | 2409
```
Verify no files on disk in `artifacts` folder:
```shell
sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l
```
In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files)
to clean up orphaned artifacts.
WARNING:
JUnit test report artifact (`junit.xml.gz`) migration
[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991)
by the `gitlab:artifacts:migrate` script.
1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage).
### OpenStack example
......@@ -267,11 +208,7 @@ _The uploads are stored by default in
```
1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Migrate any existing local artifacts to the object storage:
```shell
gitlab-rake gitlab:artifacts:migrate
```
1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage).
---
......@@ -302,11 +239,55 @@ _The uploads are stored by default in
```
1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect.
1. Migrate any existing local artifacts to the object storage:
1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage).
```shell
sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production
```
### Migrating to object storage
After [configuring the object storage](#using-object-storage), use the following task to
migrate existing job artifacts from the local storage to the remote storage.
The processing is done in a background worker and requires **no downtime**.
**In Omnibus installations:**
```shell
gitlab-rake gitlab:artifacts:migrate
```
**In installations from source:**
```shell
sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production
```
You can optionally track progress and verify that all job artifacts migrated successfully using the
[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database):
- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances.
- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances.
Verify `objectstg` below (where `store=2`) has count of all job artifacts:
```shell
gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts;
total | filesystem | objectstg
------+------------+-----------
19 | 0 | 19
```
Verify that there are no files on disk in the `artifacts` folder:
```shell
sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp | wc -l
```
In some cases, you need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files)
to clean up orphaned artifacts.
WARNING:
JUnit test report artifact (`junit.xml.gz`) migration
[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991)
by the `gitlab:artifacts:migrate` Rake task.
### Migrating from object storage to local storage
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment