diff options
Diffstat (limited to 'doc/administration/job_artifacts.md')
| -rw-r--r-- | doc/administration/job_artifacts.md | 425 |
1 files changed, 0 insertions, 425 deletions
diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 106334b226d..b3778e89b19 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -399,428 +399,3 @@ an archive is a very large file. When selecting a specific file, [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) extracts it from the archive and the download begins. This implementation saves space, memory and disk I/O. - -## Troubleshooting - -### Job artifacts using too much disk space - -Job artifacts can fill up your disk space quicker than expected. Some possible -reasons are: - -- Users have configured job artifacts expiration to be longer than necessary. -- The number of jobs run, and hence artifacts generated, is higher than expected. -- Job logs are larger than expected, and have accumulated over time. -- The file system might run out of inodes because - [empty directories are left behind by artifact housekeeping](https://gitlab.com/gitlab-org/gitlab/-/issues/17465). - [The Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files) - removes these. -- Artifact files might be left on disk and not deleted by housekeeping. Run the - [Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files) - to remove these. This script should always find work to do, as it also removes empty directories (see above). -- [Artifact housekeeping was changed significantly](#artifacts-housekeeping-disabled-in-gitlab-146-to-152), - and you might need to enable a feature flag to used the updated system. -- The [keep latest artifacts from most recent success jobs](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs) - feature is enabled. - -In these and other cases, identify the projects most responsible -for disk space usage, figure out what types of artifacts are using the most -space, and in some cases, manually delete job artifacts to reclaim disk space. - -#### Artifacts housekeeping disabled in GitLab 14.6 to 15.2 - -Artifact housekeeping was significantly changed in GitLab 14.10, and the changes -were back ported to GitLab 14.6 and later. The updated housekeeping must be -enabled with feature flags [until GitLab 15.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92931). - -To check if the feature flags are enabled: - -1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). - -1. Check if the feature flags are enabled. - - - GitLab 14.10 and earlier: - - ```ruby - Feature.enabled?(:ci_detect_wrongly_expired_artifacts, default_enabled: :yaml) - Feature.enabled?(:ci_update_unlocked_job_artifacts, default_enabled: :yaml) - Feature.enabled?(:ci_destroy_unlocked_job_artifacts, default_enabled: :yaml) - ``` - - - GitLab 15.00 and later: - - ```ruby - Feature.enabled?(:ci_detect_wrongly_expired_artifacts) - Feature.enabled?(:ci_update_unlocked_job_artifacts) - Feature.enabled?(:ci_destroy_unlocked_job_artifacts) - ``` - -1. If any of the feature flags are disabled, enable them: - - ```ruby - Feature.enable(:ci_detect_wrongly_expired_artifacts) - Feature.enable(:ci_update_unlocked_job_artifacts) - Feature.enable(:ci_destroy_unlocked_job_artifacts) - ``` - -These changes include switching artifacts from `unlocked` to `locked` if -they [should be retained](../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). - -Artifacts created before this feature was introduced have a status of `unknown`. After they expire, -these artifacts are not processed by the new housekeeping jobs. - -You can check the database to confirm if your instance has artifacts with the `unknown` status: - -1. Start a database console: - - ::Tabs - - :::TabTitle Linux package (Omnibus) - - ```shell - sudo gitlab-psql - ``` - - :::TabTitle Helm chart (Kubernetes) - - ```shell - # Find the toolbox pod - kubectl --namespace <namespace> get pods -lapp=toolbox - # Connect to the PostgreSQL console - kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main - ``` - - :::TabTitle Docker - - ```shell - sudo docker exec -it <container_name> /bin/bash - gitlab-psql - ``` - - :::TabTitle Self-compiled (source) - - ```shell - sudo -u git -H psql -d gitlabhq_production - ``` - - ::EndTabs - -1. Run the following query: - - ```sql - select expire_at, file_type, locked, count(*) from ci_job_artifacts - where expire_at is not null and - file_type != 3 - group by expire_at, file_type, locked having count(*) > 1; - ``` - -If records are returned, then there are artifacts which the housekeeping job -is unable to process. For example: - -```plaintext - expire_at | file_type | locked | count --------------------------------+-----------+--------+-------- - 2021-06-21 22:00:00+00 | 1 | 2 | 73614 - 2021-06-21 22:00:00+00 | 2 | 2 | 73614 - 2021-06-21 22:00:00+00 | 4 | 2 | 3522 - 2021-06-21 22:00:00+00 | 9 | 2 | 32 - 2021-06-21 22:00:00+00 | 12 | 2 | 163 -``` - -Artifacts with locked status `2` are `unknown`. Check -[issue #346261](https://gitlab.com/gitlab-org/gitlab/-/issues/346261#note_1028871458) -for more details. - -The Sidekiq worker that processes all `unknown` artifacts is enabled by default in -GitLab 15.3 and later. It analyzes the artifacts returned by the above database query and -determines which should be `locked` or `unlocked`. Artifacts are then deleted -by that worker if needed. - -The worker can be enabled on self-managed instances running GitLab 14.10 and later: - -1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). - -1. Check if the feature is enabled. - - - GitLab 14.10: - - ```ruby - Feature.enabled?(:ci_job_artifacts_backlog_work, default_enabled: :yaml) - ``` - - - GitLab 15.0 and later: - - ```ruby - Feature.enabled?(:ci_job_artifacts_backlog_work) - ``` - -1. Enable the feature, if needed: - - ```ruby - Feature.enable(:ci_job_artifacts_backlog_work) - ``` - -The worker processes 10,000 `unknown` artifacts every seven minutes, or roughly two million -in 24 hours. - -There is a related `ci_job_artifacts_backlog_large_loop_limit` feature flag -which causes the worker to process `unknown` artifacts -[in batches that are five times larger](https://gitlab.com/gitlab-org/gitlab/-/issues/356319). -This flag is not recommended for use on self-managed instances. - -#### List projects and builds with artifacts with a specific expiration (or no expiration) - -Using a [Rails console](operations/rails_console.md), you can find projects that have job artifacts with either: - -- No expiration date. -- An expiration date more than 7 days in the future. - -Similar to [deleting artifacts](#delete-job-artifacts-from-jobs-completed-before-a-specific-date), use the following example time frames -and alter them as needed: - -- `7.days.from_now` -- `10.days.from_now` -- `2.weeks.from_now` -- `3.months.from_now` - -Each of the following scripts also limits the search to 50 results with `.limit(50)`, but this number can also be changed as needed: - -```ruby -# Find builds & projects with artifacts that never expire -builds_with_artifacts_that_never_expire = Ci::Build.with_downloadable_artifacts.where(artifacts_expire_at: nil).limit(50) -builds_with_artifacts_that_never_expire.find_each do |build| - puts "Build with id #{build.id} has artifacts that don't expire and belongs to project #{build.project.full_path}" -end - -# Find builds & projects with artifacts that expire after 7 days from today -builds_with_artifacts_that_expire_in_a_week = Ci::Build.with_downloadable_artifacts.where('artifacts_expire_at > ?', 7.days.from_now).limit(50) -builds_with_artifacts_that_expire_in_a_week.find_each do |build| - puts "Build with id #{build.id} has artifacts that expire at #{build.artifacts_expire_at} and belongs to project #{build.project.full_path}" -end -``` - -#### List projects by total size of job artifacts stored - -List the top 20 projects, sorted by the total size of job artifacts stored, by -running the following code in the Rails console (`sudo gitlab-rails console`): - -```ruby -include ActionView::Helpers::NumberHelper -ProjectStatistics.order(build_artifacts_size: :desc).limit(20).each do |s| - puts "#{number_to_human_size(s.build_artifacts_size)} \t #{s.project.full_path}" -end -``` - -You can change the number of projects listed by modifying `.limit(20)` to the -number you want. - -#### List largest artifacts in a single project - -List the 50 largest job artifacts in a single project by running the following -code in the Rails console (`sudo gitlab-rails console`): - -```ruby -include ActionView::Helpers::NumberHelper -project = Project.find_by_full_path('path/to/project') -Ci::JobArtifact.where(project: project).order(size: :desc).limit(50).map { |a| puts "ID: #{a.id} - #{a.file_type}: #{number_to_human_size(a.size)}" } -``` - -You can change the number of job artifacts listed by modifying `.limit(50)` to -the number you want. - -#### List artifacts in a single project - -List the artifacts for a single project, sorted by artifact size. The output includes the: - -- ID of the job that created the artifact -- artifact size -- artifact file type -- artifact creation date -- on-disk location of the artifact - -```ruby -p = Project.find_by_id(<project_id>) -arts = Ci::JobArtifact.where(project: p) - -list = arts.order(size: :desc).limit(50).each do |art| - puts "Job ID: #{art.job_id} - Size: #{art.size}b - Type: #{art.file_type} - Created: #{art.created_at} - File loc: #{art.file}" -end -``` - -To change the number of job artifacts listed, change the number in `limit(50)`. - -#### Delete job artifacts from jobs completed before a specific date - -WARNING: -These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. - -If you need to manually remove job artifacts associated with multiple jobs while -**retaining their job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): - -1. Select jobs to be deleted: - - To select all jobs with artifacts for a single project: - - ```ruby - project = Project.find_by_full_path('path/to/project') - builds_with_artifacts = project.builds.with_downloadable_artifacts - ``` - - To select all jobs with artifacts across the entire GitLab instance: - - ```ruby - builds_with_artifacts = Ci::Build.with_downloadable_artifacts - ``` - -1. Delete job artifacts older than a specific date: - - NOTE: - This step also erases artifacts that users have chosen to - ["keep"](../ci/jobs/job_artifacts.md#download-job-artifacts). - - ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) - builds_to_clear.find_each do |build| - Ci::JobArtifacts::DeleteService.new(build).execute - build.update!(artifacts_expire_at: Time.now) - end - ``` - - In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/372537), use the following instead: - - ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) - builds_to_clear.find_each do |build| - build.artifacts_expire_at = Time.now - build.erase_erasable_artifacts! - end - ``` - - `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new - date or time in the past. Other valid examples are: - - - `7.days.ago` - - `3.months.ago` - - `1.year.ago` - - `erase_erasable_artifacts!` is a synchronous method, and upon execution the artifacts are immediately removed; - they are not scheduled by a background queue. - -#### Delete job artifacts and logs from jobs completed before a specific date - -WARNING: -These commands remove data permanently from both the database and from disk. Before running them, we highly recommend seeking guidance from a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. - -If you need to manually remove **all** job artifacts associated with multiple jobs, -**including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): - -1. Select the jobs to be deleted: - - To select jobs with artifacts for a single project: - - ```ruby - project = Project.find_by_full_path('path/to/project') - builds_with_artifacts = project.builds.with_existing_job_artifacts(Ci::JobArtifact.trace) - ``` - - To select jobs with artifacts across the entire GitLab instance: - - ```ruby - builds_with_artifacts = Ci::Build.with_existing_job_artifacts(Ci::JobArtifact.trace) - ``` - -1. Select the user which is mentioned in the web UI as erasing the job: - - ```ruby - admin_user = User.find_by(username: 'username') - ``` - -1. Erase the job artifacts and logs older than a specific date: - - ```ruby - builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) - builds_to_clear.find_each do |build| - print "Ci::Build ID #{build.id}... " - - if build.erasable? - Ci::BuildEraseService.new(build, admin_user).execute - puts "Erased" - else - puts "Skipped (Nothing to erase or not erasable)" - end - end - ``` - - In [GitLab 15.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/369132), replace - `Ci::BuildEraseService.new(build, admin_user).execute` with `build.erase(erased_by: admin_user)`. - - `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new - date or time in the past. Other valid examples are: - - - `7.days.ago` - - `3.months.ago` - - `1.year.ago` - -### Job artifact upload fails with error 500 - -If you are using object storage for artifacts and a job artifact fails to upload, -review: - -- The job log for an error message similar to: - - ```plaintext - WARNING: Uploading artifacts as "archive" to coordinator... failed id=12345 responseStatus=500 Internal Server Error status=500 token=abcd1234 - ``` - -- The [workhorse log](logs/index.md#workhorse-logs) for an error message similar to: - - ```json - {"error":"MissingRegion: could not find region configuration","level":"error","msg":"error uploading S3 session","time":"2021-03-16T22:10:55-04:00"} - ``` - -In both cases, you might need to add `region` to the job artifact [object storage configuration](object_storage.md). - -### Job artifact upload fails with `500 Internal Server Error (Missing file)` - -Bucket names that include folder paths are not supported with [consolidated object storage](object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). -For example, `bucket/path`. If a bucket name has a path in it, you might receive an error similar to: - -```plaintext -WARNING: Uploading artifacts as "archive" to coordinator... POST https://gitlab.example.com/api/v4/jobs/job_id/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day: 500 Internal Server Error (Missing file) -FATAL: invalid argument -``` - -If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type. - -### Job artifacts fail to upload with `FATAL: invalid argument` when using Windows mount - -If you are using a Windows mount with CIFS for job artifacts, you may see an -`invalid argument` error when the runner attempts to upload artifacts: - -```plaintext -WARNING: Uploading artifacts as "dotenv" to coordinator... POST https://<your-gitlab-instance>/api/v4/jobs/<JOB_ID>/artifacts: 500 Internal Server Error id=1296 responseStatus=500 Internal Server Error status=500 token=***** -FATAL: invalid argument -``` - -To work around this issue, you can try: - -- Switching to an ext4 mount instead of CIFS. -- Upgrading to at least Linux kernel 5.15 which contains a number of important bug fixes - relating to CIFS file leases. -- For older kernels, using the `nolease` mount option to disable file leasing. - -For more information, [see the investigation details](https://gitlab.com/gitlab-org/gitlab/-/issues/389995). - -### Usage quota shows incorrect artifact storage usage - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238536) in GitLab 14.10. - -Sometimes the [artifacts storage usage](../user/usage_quotas.md) displays an incorrect -value for the total storage space used by artifacts. To recalculate the artifact -usage statistics for all projects in the instance, you can run this background script: - -```shell -bin/rake 'gitlab:refresh_project_statistics_build_artifacts_size[file.csv]' -``` - -The artifact usage value can fluctuate to `0` while the script is running. After -recalculation, usage should display as expected again. |