diff options
Diffstat (limited to 'doc/development/github_importer.md')
| -rw-r--r-- | doc/development/github_importer.md | 38 |
1 files changed, 33 insertions, 5 deletions
diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 9e9216c4f66..676da95eb7e 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -95,7 +95,7 @@ For every collaborator, we schedule a job for the `Gitlab::GithubImport::ImportC NOTE: This stage is optional (controlled by `Gitlab::GithubImport::Settings`) and is selected by default. -### 6. Stage::ImportPullRequestsMergedByWorker +### 6. Stage::ImportPullRequestsMergedByWorker (deprecated) This worker imports the pull requests' _merged-by_ user information. The [_List pull requests_](https://docs.github.com/en/rest/pulls#list-pull-requests) @@ -104,20 +104,33 @@ individually to import this information. A `Gitlab::GithubImport::PullRequests::ImportMergedByWorker` job is scheduled for each fetched pull request. -### 7. Stage::ImportPullRequestsReviewRequestsWorker +NOTE: +This stage is skipped when `github_import_extended_events` feature flag is enabled as pull requests merged by information +is imported in the `10. Stage::ImportIssueEventsWorker` stage. This stage will be removed along with the feature flag. + +### 7. Stage::ImportPullRequestsReviewRequestsWorker (deprecated) This worker imports assigned reviewers of pull requests. For each pull request, this worker: - Fetches all assigned review requests. - Schedules a `Gitlab::GithubImport::PullRequests::ImportReviewRequestWorker` job for each fetched review request. -### 8. Stage::ImportPullRequestsReviewsWorker +NOTE: +This stage is skipped when `github_import_extended_events` feature flag is enabled as pull requests review requests +information is imported in the `10. Stage::ImportIssueEventsWorker` stage. This stage will be removed along with the +feature flag. + +### 8. Stage::ImportPullRequestsReviewsWorker (deprecated) This worker imports reviews of pull requests. For each pull request, this worker: - Fetches all the pages of reviews. - Schedules a `Gitlab::GithubImport::PullRequests::ImportReviewWorker` job for each fetched review. +NOTE: +This stage is skipped when `github_import_extended_events` feature flag is enabled as pull requests reviews information +is imported in the`10. Stage::ImportIssueEventsWorker` stage. This stage will be removed along with the feature flag. + ### 9. Stage::ImportIssuesAndDiffNotesWorker This worker imports all issues and pull request comments. For every issue, we @@ -147,10 +160,21 @@ GitHub are stored in a single table. Therefore, they have globally-unique IDs an Therefore, both issues and pull requests have a common API for most related things. +When the feature flag `github_import_extended_events` is enabled, this stage is used to import +`pull request merged by`, `pull request reviews`, `pull request review requests` and `notes`. This is possible because +[timeline events endpoint](https://docs.github.com/en/rest/issues/timeline?apiVersion=2022-11-28#list-timeline-events-for-an-issue) +also contains such information. + +To facilitate the import of `pull request review requests` using the timeline events endpoint, +events must be processed sequentially. Given that import workers do not execute in a guaranteed order, +the `pull request review requests` events are initially placed in a Redis ordered list. Subsequently, they are consumed +in sequence by the `Gitlab::GithubImport::ReplayEventsWorker`. + NOTE: -This stage is optional and can consume significant extra import time (controlled by `Gitlab::GithubImport::Settings`). +This stage is mandatory when `github_import_extended_events` feature flag is enabled. Otherwise the stage is optional +and can executed using the [import options](../user/project/import/github.md#select-additional-items-to-import). -### 11. Stage::ImportNotesWorker +### 11. Stage::ImportNotesWorker (deprecated) This worker imports regular comments for both issues and pull requests. For every comment, we schedule a job for the @@ -161,6 +185,10 @@ returns comments for both issues and pull requests. This means we have to wait for all issues and pull requests to be imported before we can import regular comments. +NOTE: +This stage is skipped when `github_import_extended_events` feature flag is enabled as notes are imported in the +`10. Stage::ImportIssueEventsWorker` stage. This stage will be removed along with the feature flag. + ### 12. Stage::ImportAttachmentsWorker This worker imports note attachments that are linked inside Markdown. |