diff options
Diffstat (limited to 'doc/administration/job_artifacts.md')
| -rw-r--r-- | doc/administration/job_artifacts.md | 127 |
1 files changed, 124 insertions, 3 deletions
diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 1e9f20ded55..d79f322c3f5 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -200,7 +200,8 @@ sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production You can optionally track progress and verify that all job artifacts migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. Verify `objectstg` below (where `store=2`) has count of all job artifacts: @@ -301,7 +302,7 @@ I/O. It instead inspects the metadata file which contains all the relevant information. This is especially important when there is a lot of artifacts, or an archive is a very large file. -When clicking on a specific file, [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) extracts it +When selecting a specific file, [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) extracts it from the archive and the download begins. This implementation saves space, memory and disk I/O. @@ -315,12 +316,132 @@ reasons are: - Users have configured job artifacts expiration to be longer than necessary. - The number of jobs run, and hence artifacts generated, is higher than expected. - Job logs are larger than expected, and have accumulated over time. +- The file system might run out of inodes because + [empty directories are left behind by artifact housekeeping](https://gitlab.com/gitlab-org/gitlab/-/issues/17465). + [The Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files) + removes these. +- Artifact files might be left on disk and not deleted by housekeeping. Run the + [Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files) + to remove these. This script should always find work to do, as it also removes empty directories (see above). +- [Artifact housekeeping was changed significantly](#artifacts-housekeeping-disabled-in-gitlab-146-to-152), + and you might need to enable a feature flag to used the updated system. In these and other cases, identify the projects most responsible for disk space usage, figure out what types of artifacts are using the most space, and in some cases, manually delete job artifacts to reclaim disk space. -One possible first step is to [clean up _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files). +#### Artifacts housekeeping disabled in GitLab 14.6 to 15.2 + +Artifact housekeeping was significantly changed in GitLab 14.10, and the changes +were back ported to GitLab 14.6 and later. The updated housekeeping must be +enabled with feature flags [until GitLab 15.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92931). + +To check if the feature flags are enabled: + +1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). + +1. Check if the feature flags are enabled. + + - GitLab 14.10 and earlier: + + ```ruby + Feature.enabled?(:ci_detect_wrongly_expired_artifacts, default_enabled: :yaml) + Feature.enabled?(:ci_update_unlocked_job_artifacts, default_enabled: :yaml) + Feature.enabled?(:ci_destroy_unlocked_job_artifacts, default_enabled: :yaml) + ``` + + - GitLab 15.00 and later: + + ```ruby + Feature.enabled?(:ci_detect_wrongly_expired_artifacts) + Feature.enabled?(:ci_update_unlocked_job_artifacts) + Feature.enabled?(:ci_destroy_unlocked_job_artifacts) + ``` + +1. If any of the feature flags are disabled, enable them: + + ```ruby + Feature.enable(:ci_detect_wrongly_expired_artifacts) + Feature.enable(:ci_update_unlocked_job_artifacts) + Feature.enable(:ci_destroy_unlocked_job_artifacts) + ``` + +These changes include switching artifacts from `unlocked` to `locked` if +they [should be retained](../ci/pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). + +Artifacts created before this feature was introduced have a status of `unknown`. After they expire, +these artifacts are not processed by the new housekeeping jobs. + +You can check the database to confirm if your instance has artifacts with the `unknown` status: + +1. Start a database console, on Omnibus: + + ```shell + sudo gitlab-psql + ``` + +1. Run this query: + + ```sql + select expire_at, file_type, locked, count(*) from ci_job_artifacts + where expire_at is not null and + file_type != 3 + group by expire_at, file_type, locked having count(*) > 1; + ``` + +If records are returned, then there are artifacts which the housekeeping job +is unable to process. For example: + +```plaintext + expire_at | file_type | locked | count +-------------------------------+-----------+--------+-------- + 2021-06-21 22:00:00+00 | 1 | 2 | 73614 + 2021-06-21 22:00:00+00 | 2 | 2 | 73614 + 2021-06-21 22:00:00+00 | 4 | 2 | 3522 + 2021-06-21 22:00:00+00 | 9 | 2 | 32 + 2021-06-21 22:00:00+00 | 12 | 2 | 163 +``` + +Artifacts with locked status `2` are `unknown`. Check +[issue #346261](https://gitlab.com/gitlab-org/gitlab/-/issues/346261#note_1028871458) +for more details. + +The Sidekiq worker that processes all `unknown` artifacts is enabled by default in +GitLab 15.3 and later. It analyzes the artifacts returned by the above database query and +determines which should be `locked` or `unlocked`. Artifacts are then deleted +by that worker if needed. + +The worker can be enabled on self-managed instances running GitLab 14.10 and later: + +1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). + +1. Check if the feature is enabled. + + - GitLab 14.10: + + ```ruby + Feature.enabled?(:ci_job_artifacts_backlog_work, default_enabled: :yaml) + ``` + + - GitLab 15.0 and later: + + ```ruby + Feature.enabled?(:ci_job_artifacts_backlog_work) + ``` + +1. Enable the feature, if needed: + + ```ruby + Feature.enable(:ci_job_artifacts_backlog_work) + ``` + +The worker processes 10,000 `unknown` artifacts every seven minutes, or roughly two million +in 24 hours. + +There is a related `ci_job_artifacts_backlog_large_loop_limit` feature flag +which causes the worker to process `unknown` artifacts +[in batches that are five times larger](https://gitlab.com/gitlab-org/gitlab/-/issues/356319). +This flag is not recommended for use on self-managed instances. #### List projects and builds with artifacts with a specific expiration (or no expiration) |