diff options
Diffstat (limited to 'doc/user/project/repository/mirror')
| -rw-r--r-- | doc/user/project/repository/mirror/bidirectional.md | 5 | ||||
| -rw-r--r-- | doc/user/project/repository/mirror/img/mirror_error_v16_3.png | bin | 0 -> 7610 bytes | |||
| -rw-r--r-- | doc/user/project/repository/mirror/index.md | 193 | ||||
| -rw-r--r-- | doc/user/project/repository/mirror/pull.md | 3 | ||||
| -rw-r--r-- | doc/user/project/repository/mirror/push.md | 3 | ||||
| -rw-r--r-- | doc/user/project/repository/mirror/troubleshooting.md | 217 |
6 files changed, 228 insertions, 193 deletions
diff --git a/doc/user/project/repository/mirror/bidirectional.md b/doc/user/project/repository/mirror/bidirectional.md index 550d4535adb..fade9e1b63c 100644 --- a/doc/user/project/repository/mirror/bidirectional.md +++ b/doc/user/project/repository/mirror/bidirectional.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Bidirectional mirroring **(PREMIUM)** +# Bidirectional mirroring **(PREMIUM ALL)** > Moved to GitLab Premium in 13.9. @@ -138,7 +138,7 @@ This sample has a few limitations: - The script circumvents the Git hook quarantine environment because the update of `$TARGET_REPO` is seen as a ref update, and Git displays warnings about it. -## Mirror with Perforce Helix with Git Fusion **(PREMIUM)** +## Mirror with Perforce Helix with Git Fusion **(PREMIUM ALL)** > Moved to GitLab Premium in 13.9. @@ -167,4 +167,5 @@ Read about [Git Fusion settings on Perforce.com](https://www.perforce.com/manual ## Related topics +- [Troubleshooting](troubleshooting.md) for repository mirroring. - [Configure server hooks](../../../../administration/server_hooks.md) diff --git a/doc/user/project/repository/mirror/img/mirror_error_v16_3.png b/doc/user/project/repository/mirror/img/mirror_error_v16_3.png Binary files differnew file mode 100644 index 00000000000..7d3c03534ef --- /dev/null +++ b/doc/user/project/repository/mirror/img/mirror_error_v16_3.png diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index 58c6c343282..7ade27e556d 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Repository mirroring **(FREE)** +# Repository mirroring **(FREE ALL)** You can _mirror_ a repository to and from external sources. You can select which repository serves as the source. Branches, tags, and commits are synced automatically. @@ -43,6 +43,7 @@ Prerequisites: 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. 1. Select **Settings > Repository**. 1. Expand **Mirroring repositories**. +1. Select **Add new**. 1. Enter a **Git repository URL**. For security reasons, the URL to the original repository is only displayed to users with the Maintainer role or the Owner role for the mirrored project. @@ -76,7 +77,7 @@ non-protected branches in the mirroring project are not mirrored and can diverge To use this option, select **Only mirror protected branches** when you create a repository mirror. -### Mirror specific branches **(PREMIUM)** +### Mirror specific branches **(PREMIUM ALL)** > - Mirroring branches matching a regex [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/102608) in GitLab 15.8 [with a flag](../../../../administration/feature_flags.md) named `mirror_only_branches_match_regex`. Disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/381667) in GitLab 16.0. @@ -205,193 +206,7 @@ Older versions of SSH may require you to remove `-E md5` from the command. ## Related topics +- [Troubleshooting](troubleshooting.md) for repository mirroring. - Configure a [Pull Mirroring Interval](../../../../administration/instance_limits.md#pull-mirroring-interval) - [Disable mirrors for a project](../../../../administration/settings/visibility_and_access_controls.md#enable-project-mirroring) - [Secrets file and mirroring](../../../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost) - -## Troubleshooting - -Should an error occur during a push, GitLab displays an **Error** highlight for that repository. Details on the error can then be seen by hovering over the highlight text. - -### Received RST_STREAM with error code 2 with GitHub - -If you receive this message while mirroring to a GitHub repository: - -```plaintext -13:Received RST_STREAM with error code 2 -``` - -One of these issues might be occurring: - -1. Your GitHub settings might be set to block pushes that expose your email address - used in commits. To fix this problem, either: - - Set your GitHub email address to public. - - Disable the [Block command line pushes that expose my email](https://github.com/settings/emails) - setting. -1. Your repository exceeds GitHub's file size limit of 100 MB. To fix this problem, - check the file size limit configured for on GitHub, and consider using - [Git Large File Storage](https://git-lfs.github.com) to manage large files. - -### Deadline Exceeded - -When upgrading GitLab, a change in how usernames are represented means that you -must update your mirroring username and password to ensure that `%40` characters are replaced with `@`. - -### Connection blocked because server only allows public key authentication - -The connection between GitLab and the remote repository is blocked. Even if a -[TCP Check](../../../../administration/raketasks/maintenance.md#check-tcp-connectivity-to-a-remote-site) -is successful, you must check any networking components in the route from GitLab -to the remote server for blockage. - -This error can occur when a firewall performs a `Deep SSH Inspection` on outgoing packets. - -### Could not read username: terminal prompts disabled - -If you receive this error after creating a new project using -[GitLab CI/CD for external repositories](../../../../ci/ci_cd_for_external_repos/index.md): - -- In Bitbucket Cloud: - - ```plaintext - "2:fetch remote: "fatal: could not read Username for 'https://bitbucket.org': - terminal prompts disabled\n": exit status 128." - ``` - -- In Bitbucket Server (self-managed): - - ```plaintext - "2:fetch remote: "fatal: could not read Username for 'https://lab.example.com': - terminal prompts disabled\n": exit status 128. - ``` - -Check if the repository owner is specified in the URL of your mirrored repository: - -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > Repository**. -1. Expand **Mirroring repositories**. -1. If no repository owner is specified, delete and add the URL again in this format, - replacing `OWNER`, `ACCOUNTNAME`, `PATH_TO_REPO`, and `REPONAME` with your values: - - - In Bitbucket Cloud: - - ```plaintext - https://OWNER@bitbucket.org/ACCOUNTNAME/REPONAME.git - ``` - - - In Bitbucket Server (self-managed): - - ```plaintext - https://OWNER@lab.example.com/PATH_TO_REPO/REPONAME.git - ``` - -When connecting to the Cloud or self-managed Bitbucket repository for mirroring, the repository owner is required in the string. - -### Pull mirror is missing LFS files - -In some cases, pull mirroring does not transfer LFS files. This issue occurs when: - -- You use an SSH repository URL. The workaround is to use an HTTPS repository URL instead. - An issue exists [to fix this problem for SSH URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/11997). -- You're using GitLab 14.0 or older, and the source repository is a public Bitbucket URL. - [Fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/335123) in GitLab 14.0.6. -- You mirror an external repository using object storage. - An issue exists [to fix this problem](https://gitlab.com/gitlab-org/gitlab/-/issues/335495). - -### `The repository is being updated`, but neither fails nor succeeds visibly - -In rare cases, mirroring slots on Redis can become exhausted, -possibly because Sidekiq workers are reaped due to out-of-memory (OoM) events. -When this occurs, mirroring jobs start and complete quickly, but they neither -fail nor succeed. They also do not leave a clear log. To check for this problem: - -1. Enter the [Rails console](../../../../administration/operations/rails_console.md) - and check Redis' mirroring capacity: - - ```ruby - current = Gitlab::Redis::SharedState.with { |redis| redis.scard('MIRROR_PULL_CAPACITY') }.to_i - maximum = Gitlab::CurrentSettings.mirror_max_capacity - available = maximum - current - ``` - -1. If the mirroring capacity is `0` or very low, you can drain all stuck jobs with: - - ```ruby - Gitlab::Redis::SharedState.with { |redis| redis.smembers('MIRROR_PULL_CAPACITY') }.each do |pid| - Gitlab::Redis::SharedState.with { |redis| redis.srem('MIRROR_PULL_CAPACITY', pid) } - end - ``` - -1. After you run the command, the [background jobs page](../../../../administration/admin_area.md#background-jobs) - should show new mirroring jobs being scheduled, especially when - [triggered manually](#update-a-mirror). - -### Invalid URL - -If you receive this error while setting up mirroring over [SSH](#ssh-authentication), make sure the URL is in a valid format. - -Mirroring does not support the short version of SSH clone URLs (`git@gitlab.com:gitlab-org/gitlab.git`) -and requires the full version including the protocol (`ssh://git@gitlab.com/gitlab-org/gitlab.git`). - -Make sure that host and project path are separated using `/` instead of `:`. - -### Host key verification failed - -This error is returned when the target host public SSH key changes. -Public SSH keys rarely, if ever, change. If host key verification fails, -but you suspect the key is still valid, you can refresh the key's information. - -Prerequisites: - -- You must have at least the Maintainer role for a project. - -To resolve the issue: - -1. [Verify the host key](#verify-a-host-key). -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. Select **Settings > Repository**. -1. Expand **Mirroring repositories**. -1. To refresh the keys, either: - - - Select **Detect host keys** for GitLab to fetch the host keys from the server, and display the fingerprints. - - Select **Input host keys manually**, and enter the host key into the **SSH host key** field. - -- Select **Mirror repository**. - -### Transfer mirror users and tokens to a single service account in Rails console - -This requires access to the [GitLab Rails console](../../../../administration/operations/rails_console.md#starting-a-rails-console-session). - -Use case: If you have multiple users using their own GitHub credentials to set up -repository mirroring, mirroring breaks when people leave the company. Use this -script to migrate disparate mirroring users and tokens into a single service account: - -WARNING: -Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. - -```ruby -svc_user = User.find_by(username: 'ourServiceUser') -token = 'githubAccessToken' - -Project.where(mirror: true).each do |project| - import_url = project.import_url - - # The url we want is https://token@project/path.git - repo_url = if import_url.include?('@') - # Case 1: The url is something like https://23423432@project/path.git - import_url.split('@').last - elsif import_url.include?('//') - # Case 2: The url is something like https://project/path.git - import_url.split('//').last - end - - next unless repo_url - - final_url = "https://#{token}@#{repo_url}" - - project.mirror_user = svc_user - project.import_url = final_url - project.username_only_import_url = final_url - project.save -end -``` diff --git a/doc/user/project/repository/mirror/pull.md b/doc/user/project/repository/mirror/pull.md index 56e85157c03..ba54c18f8ee 100644 --- a/doc/user/project/repository/mirror/pull.md +++ b/doc/user/project/repository/mirror/pull.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Pull from a remote repository **(PREMIUM)** +# Pull from a remote repository **(PREMIUM ALL)** > Moved to GitLab Premium in 13.9. @@ -145,5 +145,6 @@ end ## Related topics +- [Troubleshooting](troubleshooting.md) for repository mirroring. - [Pull mirroring intervals](../../../../administration/instance_limits.md#pull-mirroring-interval) - [Pull mirroring API](../../../../api/projects.md#configure-pull-mirroring-for-a-project) diff --git a/doc/user/project/repository/mirror/push.md b/doc/user/project/repository/mirror/push.md index 26a60002f0e..cd4fe68b01b 100644 --- a/doc/user/project/repository/mirror/push.md +++ b/doc/user/project/repository/mirror/push.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Push mirroring **(FREE)** +# Push mirroring **(FREE ALL)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40137) in GitLab 13.5: LFS support over HTTPS. @@ -209,4 +209,5 @@ If it isn't working correctly, a red `error` tag appears, and shows the error me ## Related topics +- [Troubleshooting](troubleshooting.md) for repository mirroring. - [Remote mirrors API](../../../../api/remote_mirrors.md) diff --git a/doc/user/project/repository/mirror/troubleshooting.md b/doc/user/project/repository/mirror/troubleshooting.md new file mode 100644 index 00000000000..5817aab5fc7 --- /dev/null +++ b/doc/user/project/repository/mirror/troubleshooting.md @@ -0,0 +1,217 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting repository mirroring **(FREE ALL)** + +When mirroring fails, project maintainers can see a link similar to **{warning-solid}** **Pull mirroring failed 1 hour ago.** +on the project details page. Select this link to go directly to the mirroring settings, +where GitLab displays an **Error** badge for the mirrored repository. You can hover your mouse cursor +over the badge to display the text of the error: + + + +## Received RST_STREAM with error code 2 with GitHub + +If you receive this message while mirroring to a GitHub repository: + +```plaintext +13:Received RST_STREAM with error code 2 +``` + +One of these issues might be occurring: + +1. Your GitHub settings might be set to block pushes that expose your email address + used in commits. To fix this problem, either: + - Set your GitHub email address to public. + - Disable the [Block command line pushes that expose my email](https://github.com/settings/emails) + setting. +1. Your repository exceeds GitHub's file size limit of 100 MB. To fix this problem, + check the file size limit configured for on GitHub, and consider using + [Git Large File Storage](https://git-lfs.github.com) to manage large files. + +## Deadline Exceeded + +When upgrading GitLab, a change in how usernames are represented means that you +must update your mirroring username and password to ensure that `%40` characters are replaced with `@`. + +## Connection blocked because server only allows public key authentication + +The connection between GitLab and the remote repository is blocked. Even if a +[TCP Check](../../../../administration/raketasks/maintenance.md#check-tcp-connectivity-to-a-remote-site) +is successful, you must check any networking components in the route from GitLab +to the remote server for blockage. + +This error can occur when a firewall performs a `Deep SSH Inspection` on outgoing packets. + +## Could not read username: terminal prompts disabled + +If you receive this error after creating a new project using +[GitLab CI/CD for external repositories](../../../../ci/ci_cd_for_external_repos/index.md): + +- In Bitbucket Cloud: + + ```plaintext + "2:fetch remote: "fatal: could not read Username for 'https://bitbucket.org': + terminal prompts disabled\n": exit status 128." + ``` + +- In Bitbucket Server (self-managed): + + ```plaintext + "2:fetch remote: "fatal: could not read Username for 'https://lab.example.com': + terminal prompts disabled\n": exit status 128. + ``` + +Check if the repository owner is specified in the URL of your mirrored repository: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. Select **Settings > Repository**. +1. Expand **Mirroring repositories**. +1. If no repository owner is specified, delete and add the URL again in this format, + replacing `OWNER`, `ACCOUNTNAME`, `PATH_TO_REPO`, and `REPONAME` with your values: + + - In Bitbucket Cloud: + + ```plaintext + https://OWNER@bitbucket.org/ACCOUNTNAME/REPONAME.git + ``` + + - In Bitbucket Server (self-managed): + + ```plaintext + https://OWNER@lab.example.com/PATH_TO_REPO/REPONAME.git + ``` + +When connecting to the Cloud or self-managed Bitbucket repository for mirroring, the repository owner is required in the string. + +## Pull mirror is missing LFS files + +In some cases, pull mirroring does not transfer LFS files. This issue occurs when: + +- You use an SSH repository URL. The workaround is to use an HTTPS repository URL instead. + An issue exists [to fix this problem for SSH URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/11997). +- You're using GitLab 14.0 or older, and the source repository is a public Bitbucket URL. + [Fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/335123) in GitLab 14.0.6. +- You mirror an external repository using object storage. + An issue exists [to fix this problem](https://gitlab.com/gitlab-org/gitlab/-/issues/335495). + +## Pull mirroring is not triggering pipelines + +Pipelines might not run for multiple reasons: + +- [Trigger pipelines for mirror updates](pull.md#trigger-pipelines-for-mirror-updates) + might not be enabled. This setting can only be enabled when initially + [configuring pull mirroring](pull.md#configure-pull-mirroring). The status + [is not displayed](https://gitlab.com/gitlab-org/gitlab/-/issues/346630) + when checking the project afterwards. + + When mirroring is set up using [CI/CD for external repositories](../../../../ci/ci_cd_for_external_repos/index.md) + this setting is enabled by default. If repository mirroring is manually reconfigured, triggering pipelines + is off by default and this could be why pipelines stop running. +- [`rules`](../../../../ci/yaml/index.md#rules) configuration prevents any jobs from + being added to the pipeline. +- Pipelines are triggered using [the account that set up the pull mirror](https://gitlab.com/gitlab-org/gitlab/-/issues/13697). + If the account is no longer valid, pipelines do not run. +- [Branch protection](../../protected_branches.md#run-pipelines-on-protected-branches) + might prevent the account that set up mirroring from running pipelines. + +## `The repository is being updated`, but neither fails nor succeeds visibly + +In rare cases, mirroring slots on Redis can become exhausted, +possibly because Sidekiq workers are reaped due to out-of-memory (OoM) events. +When this occurs, mirroring jobs start and complete quickly, but they neither +fail nor succeed. They also do not leave a clear log. To check for this problem: + +1. Enter the [Rails console](../../../../administration/operations/rails_console.md) + and check Redis' mirroring capacity: + + ```ruby + current = Gitlab::Redis::SharedState.with { |redis| redis.scard('MIRROR_PULL_CAPACITY') }.to_i + maximum = Gitlab::CurrentSettings.mirror_max_capacity + available = maximum - current + ``` + +1. If the mirroring capacity is `0` or very low, you can drain all stuck jobs with: + + ```ruby + Gitlab::Redis::SharedState.with { |redis| redis.smembers('MIRROR_PULL_CAPACITY') }.each do |pid| + Gitlab::Redis::SharedState.with { |redis| redis.srem('MIRROR_PULL_CAPACITY', pid) } + end + ``` + +1. After you run the command, the [background jobs page](../../../../administration/admin_area.md#background-jobs) + should show new mirroring jobs being scheduled, especially when + [triggered manually](index.md#update-a-mirror). + +## Invalid URL + +If you receive this error while setting up mirroring over [SSH](index.md#ssh-authentication), make sure the URL is in a valid format. + +Mirroring does not support the short version of SSH clone URLs (`git@gitlab.com:gitlab-org/gitlab.git`) +and requires the full version including the protocol (`ssh://git@gitlab.com/gitlab-org/gitlab.git`). + +Make sure that host and project path are separated using `/` instead of `:`. + +## Host key verification failed + +This error is returned when the target host public SSH key changes. +Public SSH keys rarely, if ever, change. If host key verification fails, +but you suspect the key is still valid, you can refresh the key's information. + +Prerequisites: + +- You must have at least the Maintainer role for a project. + +To resolve the issue: + +1. [Verify the host key](index.md#verify-a-host-key). +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. Select **Settings > Repository**. +1. Expand **Mirroring repositories**. +1. To refresh the keys, either: + + - Select **Detect host keys** for GitLab to fetch the host keys from the server, and display the fingerprints. + - Select **Input host keys manually**, and enter the host key into the **SSH host key** field. + +- Select **Mirror repository**. + +## Transfer mirror users and tokens to a single service account in Rails console + +This requires access to the [GitLab Rails console](../../../../administration/operations/rails_console.md#starting-a-rails-console-session). + +Use case: If you have multiple users using their own GitHub credentials to set up +repository mirroring, mirroring breaks when people leave the company. Use this +script to migrate disparate mirroring users and tokens into a single service account: + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +```ruby +svc_user = User.find_by(username: 'ourServiceUser') +token = 'githubAccessToken' + +Project.where(mirror: true).each do |project| + import_url = project.import_url + + # The url we want is https://token@project/path.git + repo_url = if import_url.include?('@') + # Case 1: The url is something like https://23423432@project/path.git + import_url.split('@').last + elsif import_url.include?('//') + # Case 2: The url is something like https://project/path.git + import_url.split('//').last + end + + next unless repo_url + + final_url = "https://#{token}@#{repo_url}" + + project.mirror_user = svc_user + project.import_url = final_url + project.username_only_import_url = final_url + project.save +end +``` |
