diff options
Diffstat (limited to 'doc/development/import_export.md')
| -rw-r--r-- | doc/development/import_export.md | 108 |
1 files changed, 1 insertions, 107 deletions
diff --git a/doc/development/import_export.md b/doc/development/import_export.md index f864dd3b678..9aab7f38dbb 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -6,116 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Import/Export development documentation -Troubleshooting and general development guidelines and tips for the [Import/Export feature](../user/project/settings/import_export.md). +General development guidelines and tips for the [Import/Export feature](../user/project/settings/import_export.md). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> This document is originally based on the [Import/Export 201 presentation available on YouTube](https://www.youtube.com/watch?v=V3i1OfExotE). -## Troubleshooting commands - -Finds information about the status of the import and further logs using the JID: - -```ruby -# Rails console -Project.find_by_full_path('group/project').import_state.slice(:jid, :status, :last_error) -> {"jid"=>"414dec93f941a593ea1a6894", "status"=>"finished", "last_error"=>nil} -``` - -```shell -# Logs -grep JID /var/log/gitlab/sidekiq/current -grep "Import/Export error" /var/log/gitlab/sidekiq/current -grep "Import/Export backtrace" /var/log/gitlab/sidekiq/current -tail /var/log/gitlab/gitlab-rails/importer.log -``` - -## Troubleshooting performance issues - -Read through the current performance problems using the Import/Export below. - -### OOM errors - -Out of memory (OOM) errors are normally caused by the [Sidekiq Memory Killer](../administration/operations/sidekiq_memory_killer.md): - -```shell -SIDEKIQ_MEMORY_KILLER_MAX_RSS = 2000000 -SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS = 3000000 -SIDEKIQ_MEMORY_KILLER_GRACE_TIME = 900 -``` - -An import status `started`, and the following Sidekiq logs signal a memory issue: - -```shell -WARN: Work still in progress <struct with JID> -``` - -### Timeouts - -Timeout errors occur due to the `Gitlab::Import::StuckProjectImportJobsWorker` marking the process as failed: - -```ruby -module Gitlab - module Import - class StuckProjectImportJobsWorker - include Gitlab::Import::StuckImportJob - # ... - end - end -end - -module Gitlab - module Import - module StuckImportJob - # ... - IMPORT_JOBS_EXPIRATION = 15.hours.to_i - # ... - def perform - stuck_imports_without_jid_count = mark_imports_without_jid_as_failed! - stuck_imports_with_jid_count = mark_imports_with_jid_as_failed! - - track_metrics(stuck_imports_with_jid_count, stuck_imports_without_jid_count) - end - # ... - end - end -end -``` - -```shell -Marked stuck import jobs as failed. JIDs: xyz -``` - -```plaintext - +-----------+ +-----------------------------------+ - |Export Job |--->| Calls ActiveRecord `as_json` and | - +-----------+ | `to_json` on all project models | - +-----------------------------------+ - - +-----------+ +-----------------------------------+ - |Import Job |--->| Loads all JSON in memory, then | - +-----------+ | inserts into the DB in batches | - +-----------------------------------+ -``` - -### Problems and solutions - -| Problem | Possible solutions | -| -------- | -------- | -| [Slow JSON](https://gitlab.com/gitlab-org/gitlab/-/issues/25251) loading/dumping models from the database | [split the worker](https://gitlab.com/gitlab-org/gitlab/-/issues/25252) | -| | Batch export -| | Optimize SQL -| | Move away from `ActiveRecord` callbacks (difficult) -| High memory usage (see also some [analysis](https://gitlab.com/gitlab-org/gitlab/-/issues/18857) | DB Commit sweet spot that uses less memory | -| | [Netflix Fast JSON API](https://github.com/Netflix/fast_jsonapi) may help | -| | Batch reading/writing to disk and any SQL - -### Temporary solutions - -While the performance problems are not tackled, there is a process to workaround -importing big projects, using a foreground import: - -[Foreground import](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/5384) of big projects for customers. -(Using the import template in the [infrastructure tracker](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues)) - ## Security The Import/Export feature is constantly updated (adding new things to export), however |