diff options
7 files changed, 153 insertions, 104 deletions
diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 2dd87eff90b..57bad6177d9 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -122,18 +122,24 @@ gitlab-rake gitlab:geo:check If any failures are reported on either site, they should be resolved **before** scheduling a planned failover. -### Check that secrets match between sites +### Check that secrets and SSH host keys match between nodes The SSH host keys and `/etc/gitlab/gitlab-secrets.json` files should be -identical on all sites. Check this by running the following on all sites and +identical on all nodes. Check this by running the following on all nodes and comparing the output: ```shell sudo sha256sum /etc/ssh/ssh_host* /etc/gitlab/gitlab-secrets.json ``` -If any files differ, replace the content on the **secondary** site with the -content from the **primary** site. +If any files differ, [manually replicate GitLab secrets](../replication/configuration.md#step-1-manually-replicate-secret-gitlab-values) and [replicate SSH host keys](../replication/configuration.md#step-2-manually-replicate-the-primary-sites-ssh-host-keys) +to the **secondary** site as necessary. + +### Check that the correct certificates are installed for HTTPS + +This step can be safely skipped if the **primary** site and all external sites accessed by the **primary** site use public CA-issued certificates. + +If the **primary** site uses custom or self-signed TLS certificates to secure inbound connections or if the **primary** site connects to external services that use custom or self-signed certificates, the correct certificates should also be installed on the **secondary** site. Follow instructions for [using custom certificates](../replication/configuration.md#step-4-optional-using-custom-certificates) with **secondary** sites. ### Ensure Geo replication is up-to-date diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 631cf293722..59134df8e7f 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -84,7 +84,7 @@ failed to replicate is **lost**. You can use the [Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. -A common cause of replication failures is the data being missing on the +A common cause of replication failures is data that is missing on the **primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index f8afd71bd36..adf89c24a4e 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -241,15 +241,60 @@ that the **secondary** site can act on those notifications immediately. Be sure the _secondary_ site is running and accessible. You can sign in to the _secondary_ site with the same credentials as were used with the _primary_ site. -### Step 4. (Optional) Configuring the **secondary** site to trust the **primary** site +### Step 4. (Optional) Using custom certificates -You can safely skip this step if your **primary** site uses a CA-issued HTTPS certificate. +You can safely skip this step if: -If your **primary** site is using a self-signed certificate for *HTTPS* support, you -need to add that certificate to the **secondary** site's trust store. Retrieve the -certificate from the **primary** site and follow -[these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) -on the **secondary** site. +- Your **primary** site uses a public CA-issued HTTPS certificate. +- Your **primary** site only connects to external services with CA-issued (not self-signed) HTTPS certificates. + +#### Custom or self-signed certificate for inbound connections + +If your GitLab Geo **primary** site uses a custom or [self-signed certificate to secure inbound HTTPS connections](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates), this certificate can either be single-domain certificate or multi-domain. + +Install the correct certificate based on your certificate type: + +- **Multi-domain certificate** that includes both primary and secondary site domains: Install the certificate at `/etc/gitlab/ssl` on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site. +- **Single-domain certificate** where the certificates are specific to each Geo site domain: Generate a valid certificate for your **secondary** site's domain and install it at `/etc/gitlab/ssl` per [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site. + +#### Connecting to external services that use customer certificates + +A copy of the self-signed certificate for the external service needs to be added to the trust store on all the **primary** site's nodes that require access to the service. + +For the **secondary** site to be able to access the same external services, these certificates *must* be added to the **secondary** site's trust store. + +If your **primary** site is using a [custom or self-signed certificate for inbound HTTPS connections](#custom-or-self-signed-certificate-for-inbound-connections), the **primary** site's certificate needs to be added to the **secondary** site's trust store: + +1. SSH into each **Rails, Sidekiq, and Gitaly node on your secondary** site and login as root: + + ```shell + sudo -i + ``` + +1. Copy the trusted certs from the **primary** site: + + If you can access one of the nodes on your **primary** site serving SSH traffic using the root user: + + ```shell + scp root@<primary_site_node_fqdn>:/etc/gitlab/trusted-certs/* /etc/gitlab/trusted-certs + ``` + + If you only have access through a user with sudo privileges: + + ```shell + # Run this from the node on your primary site: + sudo tar --transform 's/.*\///g' -zcvf ~/geo-trusted-certs.tar.gz /etc/gitlab/trusted-certs/* + + # Run this on each node on your secondary site: + scp <user_with_sudo>@<primary_site_node_fqdn>:geo-trusted-certs.tar.gz . + tar zxvf ~/geo-trusted-certs.tar.gz -C /etc/gitlab/trusted-certs + ``` + +1. Reconfigure each updated **Rails, Sidekiq, and Gitaly node in your secondary** site: + + ```shell + sudo gitlab-ctl reconfigure + ``` ### Step 5. Enable Git access over HTTP/HTTPS diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 585d65889c4..1e09dada36e 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -163,7 +163,7 @@ We also collect metrics specific to [Geo](../../administration/geo/index.md) sec ] ``` -### Enable or disable service ping metadata reporting +### Enable or disable service ping metadata reporting Service Ping timing metadata reporting is under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. @@ -444,7 +444,7 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta ``` 1. Connect to console host: - + ```shell ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal ``` @@ -457,14 +457,14 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta 1. To detach from screen, press `ctrl + A`, `ctrl + D`. 1. Exit from bastion: - + ```shell exit ``` 1. Get the metrics duration from logs: -Search in Google Console logs for `time_elapsed`. Query example [here](https://cloudlogging.app.goo.gl/nWheZvD8D3nWazNe6). +Search in Google Console logs for `time_elapsed`. Query example [here](https://cloudlogging.app.goo.gl/nWheZvD8D3nWazNe6). ### Verification (After approx 30 hours) @@ -495,7 +495,7 @@ Search in Google Console logs for `time_elapsed`. Query example [here](https://c ``` 1. Check the last payload in `raw_usage_data` table: - + ```shell RawUsageData.last.payload ``` @@ -519,89 +519,6 @@ ServicePing::SubmitService.new(skip_db_write: true).execute Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). -## Troubleshooting - -### Cannot disable Service Ping using the configuration file - -The method to disable Service Ping using the GitLab configuration file does not work in -GitLab versions 9.3.0 to 13.12.3. To disable it, you must use the Admin Area in -the GitLab UI instead. For more information, see -[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/333269). - -GitLab functionality and application settings cannot override or circumvent -restrictions at the network layer. If Service Ping is blocked by your firewall, -you are not impacted by this bug. - -#### Check if you are affected - -You can check if you were affected by this bug by using the Admin Area or by -checking the configuration file of your GitLab instance: - -- Using the Admin Area: - - 1. On the top bar, select **Menu > Admin**. - 1. On the left sidebar, select **Settings > Metrics and profiling**. - 1. Expand **Usage Statistics**. - 1. Are you able to check or uncheck the checkbox to disable Service Ping? - - - If _yes_, your GitLab instance is not affected by this bug. - - If you can't check or uncheck the checkbox, you are affected by this bug. - See the steps on [how to fix this](#how-to-fix-the-cannot-disable-service-ping-bug). - -- Checking your GitLab instance configuration file: - - To check whether you're impacted by this bug, check your instance configuration - settings. The configuration file in which Service Ping can be disabled depends - on your installation and deployment method, but is typically one of the following: - - - `/etc/gitlab/gitlab.rb` for Omnibus GitLab Linux Package and Docker. - - `charts.yaml` for GitLab Helm and cloud-native Kubernetes deployments. - - `gitlab.yml` for GitLab installations from source. - - To check the relevant configuration file for strings that indicate whether - Service Ping is disabled, you can use `grep`: - - ```shell - # Linux package - grep "usage_ping_enabled'\] = false" /etc/gitlab/gitlab.rb - - # Kubernetes charts - grep "enableUsagePing: false" values.yaml - - # From source - grep "usage_ping_enabled'\] = false" gitlab/config.yml - ``` - - If you see any output after running the relevant command, your GitLab instance - may be affected by the bug. Otherwise, your instance is not affected. - -#### How to fix the "Cannot disable Service Ping" bug - -To work around this bug, you have two options: - -- [Update](../../update/index.md) to GitLab 13.12.4 or newer to fix this bug. -- If you can't update to GitLab 13.12.4 or newer, enable Service Ping in the - configuration file, then disable Service Ping in the UI. For example, if you're - using the Linux package: - - 1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['usage_ping_enabled'] = true - ``` - - 1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. In GitLab, on the top bar, select **Menu > Admin**. - 1. On the left sidebar, select **Settings > Metrics and profiling**. - 1. Expand **Usage Statistics**. - 1. Clear the **Enable Service Ping** checkbox. - 1. Select **Save Changes**. - ## Related topics - [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md index 15bc01f1270..2764ef41f98 100644 --- a/doc/development/service_ping/troubleshooting.md +++ b/doc/development/service_ping/troubleshooting.md @@ -22,10 +22,91 @@ The alert compares the current daily value with the daily value from previous we You can use [this query](https://gitlab.com/gitlab-org/gitlab/-/issues/347298#note_836685350) as an example, to start detecting when the drop started. -### Troubleshooting GitLab application layer +### Troubleshoot the GitLab application layer For results about an investigation conducted into an unexpected drop in Service ping Payload events volume, see [this issue](https://gitlab.com/gitlab-data/analytics/-/issues/11071). -### Troubleshooting data warehouse layer +### Troubleshoot the data warehouse layer Reach out to the [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us). + +### Cannot disable Service Ping with the configuration file + +The method to disable Service Ping with the GitLab configuration file does not work in +GitLab versions 9.3.0 to 13.12.3. To disable it, you must use the Admin Area in +the GitLab UI instead. For more information, see +[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/333269). + +GitLab functionality and application settings cannot override or circumvent +restrictions at the network layer. If Service Ping is blocked by your firewall, +you are not impacted by this bug. + +#### Check if you are affected + +You can check if you were affected by this bug by using the Admin Area or by +checking the configuration file of your GitLab instance: + +- Using the Admin Area: + + 1. On the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Settings > Metrics and profiling**. + 1. Expand **Usage Statistics**. + 1. Are you able to check or uncheck the checkbox to disable Service Ping? + + - If _yes_, your GitLab instance is not affected by this bug. + - If you can't check or uncheck the checkbox, you are affected by this bug. + See the steps on [how to fix this](#how-to-fix-the-cannot-disable-service-ping-bug). + +- Checking your GitLab instance configuration file: + + To check whether you're impacted by this bug, check your instance configuration + settings. The configuration file in which Service Ping can be disabled depends + on your installation and deployment method, but is typically one of the following: + + - `/etc/gitlab/gitlab.rb` for Omnibus GitLab Linux Package and Docker. + - `charts.yaml` for GitLab Helm and cloud-native Kubernetes deployments. + - `gitlab.yml` for GitLab installations from source. + + To check the relevant configuration file for strings that indicate whether + Service Ping is disabled, you can use `grep`: + + ```shell + # Linux package + grep "usage_ping_enabled'\] = false" /etc/gitlab/gitlab.rb + + # Kubernetes charts + grep "enableUsagePing: false" values.yaml + + # From source + grep "usage_ping_enabled'\] = false" gitlab/config.yml + ``` + + If you see any output after running the relevant command, your GitLab instance + may be affected by the bug. Otherwise, your instance is not affected. + +#### How to fix the "Cannot disable Service Ping" bug + +To work around this bug, you have two options: + +- [Update](../../update/index.md) to GitLab 13.12.4 or newer to fix this bug. +- If you can't update to GitLab 13.12.4 or newer, enable Service Ping in the + configuration file, then disable Service Ping in the UI. For example, if you're + using the Linux package: + + 1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['usage_ping_enabled'] = true + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + + 1. In GitLab, on the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Settings > Metrics and profiling**. + 1. Expand **Usage Statistics**. + 1. Clear the **Enable Service Ping** checkbox. + 1. Select **Save Changes**. diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index 08f800b84a6..ce949999fb8 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -123,7 +123,7 @@ To enable or disable Service Ping and version check: NOTE: The method to disable Service Ping in the GitLab configuration file does not work in -GitLab versions 9.3 to 13.12.3. For more information about how to disable it, see [troubleshooting](../../../development/service_ping/index.md#cannot-disable-service-ping-using-the-configuration-file). +GitLab versions 9.3 to 13.12.3. For more information about how to disable it, see [troubleshooting](../../../development/service_ping/troubleshooting.md#cannot-disable-service-ping-with-the-configuration-file). To disable Service Ping and prevent it from being configured in the future through the Admin Area: diff --git a/spec/features/merge_request/user_edits_merge_request_spec.rb b/spec/features/merge_request/user_edits_merge_request_spec.rb index 364af8d8a76..0b4b9d7452a 100644 --- a/spec/features/merge_request/user_edits_merge_request_spec.rb +++ b/spec/features/merge_request/user_edits_merge_request_spec.rb @@ -92,7 +92,7 @@ RSpec.describe 'User edits a merge request', :js do select2('merge-test', from: '#merge_request_target_branch') click_button('Save changes') - expect(page).to have_content("Request to merge #{merge_request.source_branch} into merge-test") + expect(page).to have_content("requested to merge #{merge_request.source_branch} into merge-test") expect(page).to have_content("changed target branch from #{merge_request.target_branch} to merge-test") end |