diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-11-19 11:27:35 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-11-19 11:27:35 +0300 |
| commit | 7e9c479f7de77702622631cff2628a9c8dcbc627 (patch) | |
| tree | c8f718a08e110ad7e1894510980d2155a6549197 /doc/administration | |
| parent | e852b0ae16db4052c1c567d9efa4facc81146e88 (diff) | |
Add latest changes from gitlab-org/gitlab@13-6-stable-eev13.6.0-rc42
Diffstat (limited to 'doc/administration')
104 files changed, 1323 insertions, 524 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index ac972e2e33e..67a3a3c1539 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -11,6 +11,8 @@ GitLab offers a way to view the changes made within the GitLab server for owners GitLab system administrators can also take advantage of the logs located on the file system. See [the logs system documentation](logs.md) for more details. +You can generate an [Audit report](audit_reports.md) of audit events. + ## Overview **Audit Events** is a tool for GitLab owners and administrators @@ -96,9 +98,10 @@ From there, you can see the following actions: - Permission to approve merge requests by authors was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) - Number of required approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) - Added or removed users and groups from project approval groups ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213603) in GitLab 13.2) -- Project CI/CD variable added, removed, or protected status changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4. +- Project CI/CD variable added, removed, or protected status changed ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.4) +- User was approved via Admin Area ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6) -Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events) +Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events). ### Instance events **(PREMIUM ONLY)** @@ -113,8 +116,8 @@ To view the server-wide administrator log, visit **Admin Area > Monitoring > Aud In addition to the group and project events, the following user actions are also recorded: -- Failed Logins - Sign-in events and the authentication type (such as standard, LDAP, or OmniAuth) +- Failed sign-ins - Added SSH key - Added or removed email - Changed password @@ -127,6 +130,8 @@ recorded: - User was blocked via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) - User was blocked via API ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9) - Failed second-factor authentication attempt ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in GitLab 13.5) +- A user's personal access token was successfully created or revoked ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6) +- A failed attempt to create or revoke a user's personal access token ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6) It's possible to filter particular actions by choosing an audit data type from the filter dropdown box. You can further filter by specific group, project, or user @@ -134,7 +139,7 @@ the filter dropdown box. You can further filter by specific group, project, or u  -Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events) +Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). ### Missing events @@ -175,6 +180,12 @@ the steps bellow. Feature.enable(:repository_push_audit_event) ``` +## Retention policy + +On GitLab.com, Audit Event records become subject to deletion after 400 days, or when your license is downgraded to a tier that does not include access to Audit Events. Data that is subject to deletion will be deleted at GitLab's discretion, possibly without additional notice. + +If you require a longer retention period, you should independently archive your Audit Event data, which you can retrieve through the [Audit Events API](../api/audit_events.md). + ## Export to CSV **(PREMIUM ONLY)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. @@ -228,7 +239,7 @@ The first row contains the headers, which are listed in the following table alon ### Limitation -The Audit Log CSV file size is limited to a maximum of `15 MB`. +The Audit Log CSV file size is limited to a maximum of `100,000` events. The remaining records are truncated when this limit is reached. ### Enable or disable Audit Log Export to CSV diff --git a/doc/administration/audit_reports.md b/doc/administration/audit_reports.md index 83fbeda26aa..9c772302375 100644 --- a/doc/administration/audit_reports.md +++ b/doc/administration/audit_reports.md @@ -18,12 +18,12 @@ needs. ## APIs -- `https://docs.gitlab.com/ee/api/audit_events.html` -- `https://docs.gitlab.com/ee/api/graphql/reference/#user` -- `https://docs.gitlab.com/ee/api/graphql/reference/#groupmember` -- `https://docs.gitlab.com/ee/api/graphql/reference/#projectmember` +- [Audit events](../api/audit_events.md) +- [GraphQL - User](../api/graphql/reference/index.md#user) +- [GraphQL - GroupMember](../api/graphql/reference/index.md#groupmember) +- [GraphQL - ProjectMember](../api/graphql/reference/index.md#projectmember) ## Features -- `https://docs.gitlab.com/ee/administration/audit_events.html` -- `https://docs.gitlab.com/ee/administration/logs.html` +- [Audit events](audit_events.md) +- [Log system](logs.md) diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 254bd259344..6e3dbcb68b3 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -82,6 +82,8 @@ If you see an error message like the one below when you sign in after Crowd auth could not authorize you from Crowd because invalid credentials ``` -Please make sure the Crowd users who need to login to GitLab are authorized to [the application](#configure-a-new-crowd-application) in the step of **Authorisation**. This could be verified by try "Authentication test" for Crowd as of 2.11. +Ensure the Crowd users who need to sign in to GitLab are authorized to the +[application](#configure-a-new-crowd-application) in the **Authorisation** step. +This could be verified by trying "Authentication test" for Crowd (as of 2.11).  diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 3df85babc94..9d88d66bf46 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -17,7 +17,7 @@ This integration works with most LDAP-compliant directory servers, including: - Open LDAP - 389 Server -Users added through LDAP take a [licensed seat](../../../subscriptions/self_managed/index.md#choose-the-number-of-users). +Users added through LDAP take a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). GitLab Enterprise Editions (EE) include enhanced integration, including group membership syncing as well as multiple LDAP servers support. @@ -97,7 +97,8 @@ library. `start_tls` corresponds to StartTLS, not to be confused with regular TL Normally, if you specify `simple_tls` it will be on port 636, while `start_tls` (StartTLS) would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced with `start_tls` and `ssl` was replaced with `simple_tls`. -LDAP users must have an email address set, regardless of whether it is used to sign-in. +LDAP users must have a set email address, regardless of whether or not it's used +to sign in. ### Example Configurations **(CORE ONLY)** @@ -444,10 +445,10 @@ account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) has bit 2 set. For more information, see <https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/> -The user will be set to `ldap_blocked` state in GitLab if the above conditions -fail. This means the user will not be able to sign-in or push/pull code. +The user is set to an `ldap_blocked` state in GitLab if the previous conditions +fail. This means the user won't be able to sign in or push/pull code. -The process will also update the following user information: +The process also updates the following user information: - Email address. - If `sync_ssh_keys` is set, SSH public keys. @@ -460,16 +461,12 @@ The LDAP sync process: ### Adjusting LDAP user sync schedule **(STARTER ONLY)** -NOTE: **Note:** -These are cron formatted values. You can use a crontab generator to create -these values, for example <http://www.crontabgenerator.com/>. - By default, GitLab runs a worker once per day at 01:30 a.m. server time to check and update GitLab users against LDAP. You can manually configure LDAP user sync times by setting the following configuration values, in cron format. If needed, you can -use a [crontab generator](http://crontabgenerator.com). +use a [crontab generator](http://www.crontabgenerator.com). The example below shows how to set LDAP user sync to run once every 12 hours at the top of the hour. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index c6558bf1791..8920479949d 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -13,10 +13,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w #### Connection refused -If you are getting `Connection Refused` errors when trying to connect to the -LDAP server please double-check the LDAP `port` and `encryption` settings used by -GitLab. Common combinations are `encryption: 'plain'` and `port: 389`, OR -`encryption: 'simple_tls'` and `port: 636`. +If you're getting `Connection Refused` error messages when attempting to +connect to the LDAP server, review the LDAP `port` and `encryption` settings +used by GitLab. Common combinations are `encryption: 'plain'` and `port: 389`, +or `encryption: 'simple_tls'` and `port: 636`. #### Connection times out @@ -69,10 +69,10 @@ options = { # :filter is optional # 'cn' looks for all "cn"s under :base # '*' is the search string - here, it's a wildcard - filter: Net::Ldap::Filter.eq('cn', '*'), + filter: Net::LDAP::Filter.eq('cn', '*'), # :attributes is optional - # the attributes we want to get returned + # the attributes we want to get returnedk attributes: %w(dn cn memberuid member submember uniquemember memberof) } adapter.ldap_search(options) @@ -103,16 +103,16 @@ A user can have trouble signing in for any number of reasons. To get started, here are some questions to ask yourself: - Does the user fall under the [configured `base`](index.md#configuration) in - LDAP? The user must fall under this `base` to sign-in. + LDAP? The user must fall under this `base` to sign in. - Does the user pass through the [configured `user_filter`](index.md#set-up-ldap-user-filter)? If one is not configured, this question can be ignored. If it is, then the - user must also pass through this filter to be allowed to sign-in. + user must also pass through this filter to be allowed to sign in. - Refer to our docs on [debugging the `user_filter`](#debug-ldap-user-filter). If the above are both okay, the next place to look for the problem is the logs themselves while reproducing the issue. -- Ask the user to sign-in and let it fail. +- Ask the user to sign in and let it fail. - [Look through the output](#gitlab-logs) for any errors or other messages about the sign-in. You may see one of the other error messages on this page, in which case that section can help resolve the issue. @@ -159,7 +159,7 @@ The user should now be able to sign in. #### Email has already been taken -A user tries to sign-in with the correct LDAP credentials, is denied access, +A user tries to sign in with the correct LDAP credentials, is denied access, and the [production.log](../../logs.md#productionlog) shows an error that looks like this: ```plaintext @@ -649,8 +649,7 @@ ldapsearch -D "cn=admin,dc=ldap-testing,dc=example,dc=com" \ Note that the `bind_dn`, `password`, `port`, `host`, and `base` are all identical to what's configured in the `gitlab.rb`. -Please see [the official -`ldapsearch` documentation](https://linux.die.net/man/1/ldapsearch) for more. +For more information, see the [official `ldapsearch` documentation](https://linux.die.net/man/1/ldapsearch). ### Using **AdFind** (Windows) @@ -675,15 +674,15 @@ adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -u ### Rails console CAUTION: **Caution:** -Please note that it is very easy to create, read, modify, and destroy data on the -rails console, so please be sure to run commands exactly as listed. +It is very easy to create, read, modify, and destroy data with the rails +console. Be sure to run commands exactly as listed. The rails console is a valuable tool to help debug LDAP problems. It allows you to directly interact with the application by running commands and seeing how GitLab responds to them. -Please refer to [this guide](../../operations/rails_console.md#starting-a-rails-console-session) -for instructions on how to use the rails console. +For instructions about how to use the rails console, refer to this +[guide](../../operations/rails_console.md#starting-a-rails-console-session). #### Enable debug output diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 7a55e127733..06b50be2647 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -14,7 +14,9 @@ The following documentation enables Okta as a SAML provider. ## Configure the Okta application -1. On Okta go to the admin section and choose to **Add an App**. +The following guidance is based on this Okta article, on adding a [SAML Application with an Okta Developer account](https://support.okta.com/help/s/article/Why-can-t-I-add-a-SAML-Application-with-an-Okta-Developer-account?language=en_US): + +1. On Okta admin section, make sure to select Classic UI view in the top left corner. From there, choose to **Add an App**. 1. When the app screen comes up you see another button to **Create an App** and choose SAML 2.0 on the next screen. 1. Now, very important, add a logo diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index a696d0499a4..d2937d36a35 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 44270283308..4cb2c002015 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/consul.md b/doc/administration/consul.md index 4eed020c284..491251bc842 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -17,12 +17,9 @@ a service networking solution that you can manage by using `/etc/gitlab/gitlab.r ## Configure the Consul nodes -NOTE: **Important:** -Before proceeding, refer to the -[available reference architectures](reference_architectures/index.md#available-reference-architectures) -to find out how many Consul server nodes you should have. - -On **each** Consul server node perform the following: +After you review the [reference architecture](reference_architectures/index.md#available-reference-architectures) +documentation to determine the number of Consul server nodes you should have, +on _each_ Consul server node: 1. Follow the instructions to [install](https://about.gitlab.com/install/) GitLab by choosing your preferred platform, but do not supply the @@ -87,16 +84,15 @@ Consul nodes communicate using the raft protocol. If the current leader goes offline, there needs to be a leader election. A leader node must exist to facilitate synchronization across the cluster. If too many nodes go offline at the same time, the cluster will lose quorum and not elect a leader due to -[broken consensus](https://www.consul.io/docs/internals/consensus.html). +[broken consensus](https://www.consul.io/docs/architecture/consensus). Consult the [troubleshooting section](#troubleshooting-consul) if the cluster is not able to recover after the upgrade. The [outage recovery](#outage-recovery) may be of particular interest. -NOTE: **Note:** -GitLab uses Consul to store only transient data that is easily regenerated. If -the bundled Consul was not used by any process other than GitLab itself, then -[rebuilding the cluster from scratch](#recreate-from-scratch) is fine. +GitLab uses Consul to store only easily regenerated, transient data. If the +bundled Consul wasn't used by any process other than GitLab itself, you can +[rebuild the cluster from scratch](#recreate-from-scratch). ## Troubleshooting Consul diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index e88f88a0427..2b25ee4bab8 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 4129677f134..2a5260a1342 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -36,8 +36,8 @@ error, it's very important that you [**provide feedback**](https://gitlab.com/gi as possible so we can improve or fix it while behind a flag. When you upgrade GitLab to an earlier version, the feature flag status may change. -NOTE: **Note:** -Mind that features deployed behind feature flags may not be ready for +CAUTION: **Caution:** +Features deployed behind feature flags may not be ready for production use. However, disabling features behind flags that were deployed enabled by default may also present a risk. If they're enabled, we recommend you leave them as-is. diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 83081e2cef6..b73d3ba52a2 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -45,9 +45,12 @@ To bring the former **primary** node up to date: all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) during this procedure. -1. [Setup database replication](../setup/database.md). Note that in this - case, **primary** node refers to the current **primary** node, and **secondary** node refers to the - former **primary** node. +1. [Set up database replication](../setup/database.md). In this case, the **secondary** node + refers to the former **primary** node. + 1. If [PgBouncer](../../postgresql/pgbouncer.md) was enabled on the **current secondary** node + (when it was a primary node) disable it by editing `/etc/gitlab/gitlab.rb` + and running `sudo gitlab-ctl reconfigure`. + 1. You can then set up database replication on the **secondary** node. If you have lost your original **primary** node, follow the [setup instructions](../setup/index.md) to set up a new **secondary** node. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 5876a8c36e6..1e9b430834c 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -133,6 +133,12 @@ Note the following when promoting a secondary: ``` 1. Promote the **secondary** node to the **primary** node. + DANGER: **Warning:** + In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the + secondary is paused fails. Do not pause replication before promoting a + secondary. If the node is paused, be sure to resume before promoting. This + issue has been fixed in GitLab 13.4 and later. + CAUTION: **Caution:** If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. @@ -167,8 +173,14 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. +DANGER: **Warning:** +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + CAUTION: **Caution:** -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs + If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index c89b7929b13..3864445bbf4 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -227,6 +227,12 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. +DANGER: **Warning:** +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + CAUTION: **Caution:** If the secondary node [has been paused](../../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 0bfdf1d2d65..02b907ae237 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -49,7 +49,7 @@ Geo provides: - Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams. - Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. -- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. +- An intuitive UI: **Secondary** nodes use the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. ## How it works @@ -116,6 +116,7 @@ The following are required to run Geo: - [Ubuntu](https://ubuntu.com) 16.04+ - PostgreSQL 11+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ +- Git-lfs 2.4.2+ on the user side when using LFS - All nodes must run the same GitLab version. Additionally, check GitLab's [minimum requirements](../../install/requirements.md), @@ -195,6 +196,12 @@ For information on how to update your Geo nodes to the latest GitLab version, se > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +DANGER: **Warning:** +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + CAUTION: **Caution:** Pausing and resuming of replication is currently only supported for Geo installations using an Omnibus GitLab-managed database. External databases are currently not supported. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 8c818bcfbe2..43a422c4bc3 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -229,19 +229,10 @@ replicating missing data from the **primary** node in a process known as **backf Meanwhile, the **primary** node will start to notify each **secondary** node of any changes, so that the **secondary** node can act on those notifications immediately. -Make sure the **secondary** node is running and accessible. -You can login to the **secondary** node with the same credentials as used for the **primary** node. +Be sure the _secondary_ node is running and accessible. You can sign in to the +_secondary_ node with the same credentials as were used with the _primary_ node. -### Step 4. Enabling Hashed Storage - -Using Hashed Storage significantly improves Geo replication. Project and group -renames no longer require synchronization between nodes. - -1. Visit the **primary** node's **Admin Area > Settings > Repository** - (`/admin/application_settings/repository`) in your browser. -1. In the **Repository storage** section, check **Use hashed storage paths for newly created and renamed projects**. - -### Step 5. (Optional) Configuring the **secondary** node to trust the **primary** node +### Step 4. (Optional) Configuring the **secondary** node to trust the **primary** node You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. @@ -251,21 +242,23 @@ certificate from the **primary** node and follow [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html) on the **secondary** node. -### Step 6. Enable Git access over HTTP/HTTPS +### Step 5. Enable Git access over HTTP/HTTPS Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone -method to be enabled. Navigate to **Admin Area > Settings** -(`/admin/application_settings/general`) on the **primary** node, and set -`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. +method to be enabled. This is enabled by default, but if converting an existing node to Geo it should be checked: + +1. Navigate to **Admin Area > Settings** (`/admin/application_settings/general`) on the **primary** node. +1. Expand "Visibility and access controls". +1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". -### Step 7. Verify proper functioning of the **secondary** node +### Step 6. Verify proper functioning of the **secondary** node Your **secondary** node is now configured! -You can login to the **secondary** node with the same credentials you used for the -**primary** node. Visit the **secondary** node's **Admin Area > Geo** -(`/admin/geo/nodes`) in your browser to check if it's correctly identified as a -**secondary** Geo node and if Geo is enabled. +You can sign in to the _secondary_ node with the same credentials you used with +the _primary_ node. Visit the _secondary_ node's **Admin Area > Geo** +(`/admin/geo/nodes`) in your browser to determine if it's correctly identified +as a _secondary_ Geo node, and if Geo is enabled. The initial replication, or 'backfill', will probably still be in progress. You can monitor the synchronization process on each Geo node from the **primary** diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index d1fa2d503be..9e896f2c07c 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -127,8 +127,11 @@ and verification status on a **secondary** node. You can keep track of the progress to implement the missing items in these epics/issues: -- [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) -- [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) +- [Geo: Build a scalable, self-service Geo replication and verification framework](https://gitlab.com/groups/gitlab-org/-/epics/2161) +- [Geo: Improve the self-service Geo replication framework](https://gitlab.com/groups/gitlab-org/-/epics/3761) +- [Geo: Move existing blobs to framework](https://gitlab.com/groups/gitlab-org/-/epics/3588) +- [Geo: Add unreplicated data types](https://gitlab.com/groups/gitlab-org/-/epics/893) +- [Geo: Support GitLab Pages](https://gitlab.com/groups/gitlab-org/-/epics/589) ### Replicated data types behind a feature flag @@ -157,18 +160,19 @@ To enable, such as for package file replication: Feature.enable(:geo_package_file_replication) ``` -DANGER: **Danger:** +DANGER: **Warning:** Features not on this list, or with **No** in the **Replicated** column, are not replicated on the **secondary** node. Failing over without manually replicating data from those features will cause the data to be **lost**. If you wish to use those features on a **secondary** node, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (please see [Geo with Object Storage](object_storage.md)) | Notes | +| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | |:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | | [Project repository](../../..//user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | -| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | +| [Group wiki repository](../../../user/group/index.md#group-wikis) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | | | [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | | [LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | | [Personal snippets](../../../user/snippets.md#personal-snippets) | **Yes** (10.2) | **Yes** (10.2) | No | | @@ -176,7 +180,7 @@ successfully, you must replicate their data using some other means. | [CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta) . | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | | [Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | | [Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](./docker_registry.md) to enable. | +| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | | [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | | [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | | [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | @@ -187,7 +191,7 @@ successfully, you must replicate their data using some other means. | [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | -| [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | | +| [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | | [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | | [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | | [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 14a11d9c1e3..dabf4499f9d 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -25,7 +25,7 @@ To disable Geo, follow these steps: ## Remove all secondary Geo nodes To disable Geo, you need to first remove all your secondary Geo nodes, which means replication will not happen -anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](./remove_geo_node.md). +anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](remove_geo_node.md). If the current node that you want to keep using is a secondary node, you need to first promote it to primary. You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node) diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index ddcdea736e7..82fda8b0fc2 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -43,8 +43,8 @@ In any case, you require: - A working GitLab **secondary** node. - A Route53 Hosted Zone managing your domain. -If you have not yet setup a Geo **primary** node and **secondary** node, please consult -[the Geo setup instructions](../index.md#setup-instructions). +If you haven't yet set up a Geo _primary_ node and _secondary_ node, see the +[Geo setup instructions](../index.md#setup-instructions). ## Create a traffic policy diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 642d31f6298..3f5ba1939b8 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -72,7 +72,7 @@ If you are using Google Cloud Storage, consider using Or you can use the [Storage Transfer Service](https://cloud.google.com/storage-transfer/docs/), although this only supports daily synchronization. -For manual synchronization, or scheduled by `cron`, please have a look at: +For manual synchronization, or scheduled by `cron`, see: - [`s3cmd sync`](https://s3tools.org/s3cmd-sync) - [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync) diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index b62c5c6f460..15e3d3ff0a8 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -720,7 +720,8 @@ GitLab cannot find or doesn't have permission to access the `database_geo.yml` c In an Omnibus GitLab installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. If it doesn't exist or inadvertent changes have been made to it, run `sudo gitlab-ctl reconfigure` to restore it to its correct state. -If this path is mounted on a remote volume, please check your volume configuration and that it has correct permissions. +If this path is mounted on a remote volume, ensure your volume configuration +has the correct permissions. ### An existing tracking database cannot be reused @@ -782,3 +783,13 @@ To resolve this issue: using IPv6 to send its status to the **primary** node. If it is, add an entry to the **primary** node using IPv4 in the `/etc/hosts` file. Alternatively, you should [enable IPv6 on the **primary** node](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). + +## Fixing client errors + +### Authorization errors from LFS HTTP(s) client requests + +You may have problems if you're running a version of [Git LFS](https://git-lfs.github.com/) before 2.4.2. +As noted in [this authentication issue](https://github.com/git-lfs/git-lfs/issues/3025), +requests redirected from the secondary to the primary node do not properly send the +Authorization header. This may result in either an infinite `Authorization <-> Redirect` +loop, or Authorization errors. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 55ddccb1d98..af59e45250c 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -8,7 +8,9 @@ type: howto # Updating the Geo nodes **(PREMIUM ONLY)** CAUTION: **Warning:** -Please ensure you read these sections carefully before updating your Geo nodes! Not following version-specific update steps may result in unexpected downtime. Please [contact support](https://about.gitlab.com/support/#contact-support) if you have any specific questions. +Read these sections carefully before updating your Geo nodes. Not following +version-specific update steps may result in unexpected downtime. If you have +any specific questions, [contact Support](https://about.gitlab.com/support/#contact-support). Updating Geo nodes involves performing: @@ -47,4 +49,4 @@ everything is working correctly: 1. Test the data replication by pushing code to the **primary** node and see if it is received by **secondary** nodes. -If you encounter any issues, please consult the [Geo troubleshooting guide](troubleshooting.md). +If you encounter any issues, see the [Geo troubleshooting guide](troubleshooting.md). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 85c869eff92..9ea5283812e 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -24,34 +24,41 @@ DROP SERVER gitlab_secondary CASCADE; DROP EXTENSION IF EXISTS postgres_fdw; ``` +DANGER: **Warning:** +In GitLab 13.3, promoting a secondary node to a primary while the secondary is +paused fails. Do not pause replication before promoting a secondary. If the +node is paused, be sure to resume before promoting. To avoid this issue, +upgrade to GitLab 13.4 or later. + +## Updating to GitLab 13.2 + +In GitLab 13.2, promoting a secondary node to a primary while the secondary is paused fails. **Do not pause replication before promoting a secondary.** If the node is paused, please resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. + ## Updating to GitLab 13.0 Upgrading to GitLab 13.0 requires GitLab 12.10 to already be using PostgreSQL -version 11. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +version 11. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Updating to GitLab 12.10 -GitLab 12.10 does not attempt to automatically update the embedded PostgreSQL -server when using Geo, because the PostgreSQL upgrade requires downtime on -secondaries while reinitializing streaming replication. It must be upgraded -manually. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +GitLab 12.10 doesn't attempt to update the embedded PostgreSQL server when +using Geo, because the PostgreSQL upgrade requires downtime for secondaries +while reinitializing streaming replication. It must be upgraded manually. For +the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Updating to GitLab 12.9 CAUTION: **Warning:** GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). -The issue is fixed in GitLab 12.9.4. Please upgrade to GitLab 12.9.4 or later. +The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. By default, GitLab 12.9 will attempt to automatically update the embedded PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +while reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -63,9 +70,8 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade By default, GitLab 12.8 will attempt to automatically update the embedded PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -75,7 +81,7 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.7 -DANGER: **Danger:** +DANGER: **Warning:** Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 through 12.7.4 because there is [an initialization order bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo @@ -84,10 +90,9 @@ fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was shipped in 12.7.5. By default, GitLab 12.7 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while -reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -98,10 +103,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.6 By default, GitLab 12.6 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while -reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -112,10 +116,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.5 By default, GitLab 12.5 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while -reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -126,10 +129,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.4 By default, GitLab 12.4 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while -reinitializing streaming replication. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. For the recommended procedure, see +the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). This can be temporarily disabled by running the following before updating: @@ -139,35 +141,31 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.3 -DANGER: **Danger:** +DANGER: **Warning:** If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or newer. By default, GitLab 12.3 will attempt to -automatically update the embedded PostgreSQL server to 10.9 from 9.6. In -certain circumstances, it will fail. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for more information. +upgrade to GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the +embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +fail. For more information, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade -requires downtime on secondaries while reinitializing streaming replication. -Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade +requires downtime for secondaries while reinitializing streaming replication. +For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Updating to GitLab 12.2 -DANGER: **Danger:** +DANGER: **Warning:** If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or newer. By default, GitLab 12.2 will attempt to -automatically update the embedded PostgreSQL server to 10.9 from 9.6. In -certain circumstances, it will fail. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for more information. +upgrade to GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the +embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +fail. For more information, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade -requires downtime on secondaries while reinitializing streaming replication. -Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +requires downtime for secondaries while reinitializing streaming replication. +For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). GitLab 12.2 includes the following minor PostgreSQL updates: @@ -187,33 +185,31 @@ The restart avoids a version mismatch when PostgreSQL tries to load the FDW exte ## Updating to GitLab 12.1 -DANGER: **Danger:** +DANGER: **Warning:** If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or newer. By default, GitLab 12.1 will attempt to -automatically update the embedded PostgreSQL server to 10.9 from 9.6. In -certain circumstances, it will fail. Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for more information. +upgrade to GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the +embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +fail. For more information, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade -requires downtime on secondaries while reinitializing streaming replication. -Please see -[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) -for the recommended procedure. +Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade +requires downtime for secondaries while reinitializing streaming replication. +For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). ## Updating to GitLab 12.0 CAUTION: **Warning:** -This version is affected by [a bug that results in new LFS objects not being replicated to -Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed -in GitLab 12.1. Please upgrade to GitLab 12.1 or later. +This version is affected by a [bug that results in new LFS objects not being +replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 11.11 CAUTION: **Warning:** -This version is affected by [a bug that results in new LFS objects not being replicated to -Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed -in GitLab 12.1. Please upgrade to GitLab 12.1 or later. +This version is affected by a [bug that results in new LFS objects not being +replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 10.8 @@ -329,7 +325,7 @@ In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, support is removed entirely. All installations will switch to the HTTP/HTTPS cloning method instead. Before updating, ensure that all your Geo nodes are configured to use this method and that it works for your installation. In -particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps). +particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-5-enable-git-access-over-httphttps). Synchronizing repositories over the public Internet using HTTP is insecure, so you should ensure that you have HTTPS configured before updating. Note that @@ -433,7 +429,7 @@ required because replication slots are used by default. However, if you started with GitLab 9.3 and updated later, you should still follow the instructions below. -When in doubt, it does not hurt to do a resync. The easiest way to do this in +When in doubt, it doesn't hurt to do a resync. The easiest way to do this in Omnibus is the following: 1. Make sure you have Omnibus GitLab on the **primary** server. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 09b9c71aeb7..24e55d26997 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -474,6 +474,81 @@ high-availability configuration with a cluster of nodes supporting a Geo **primary** node and another cluster of nodes supporting a Geo **secondary** node. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). +## Patroni support + +Support for Patroni is intended to replace `repmgr` as a +[highly availabile PostgreSQL solution](../../postgresql/replication_and_failover.md) +on the primary node, but it can also be used for PostgreSQL HA on a secondary +node. + +Starting with GitLab 13.5, Patroni is available for _experimental_ use with Geo +primary and secondary nodes. Due to its experimental nature, Patroni support is +subject to change without notice. + +This experimental implementation has the following limitations: + +- Whenever a new Leader is elected, the PgBouncer instance must be reconfigured + to point to the new Leader. +- Whenever a new Leader is elected on the primary node, the Standby Leader on + the secondary needs to be reconfigured to point to the new Leader. +- Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a + chance the node will be demoted due to the required short-time restart. To + avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`. + After a reconfigure, it unpauses on its own. + +For instructions about how to set up Patroni on the primary node, see the +[PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. + +A production-ready and secure setup requires at least three Patroni instances on +the primary, and a similar configuration on the secondary nodes. Be sure to use +password credentials and other database best practices. + +Similar to `repmgr`, using Patroni on a secondary node is optional. + +To set up database replication with Patroni on a secondary node, configure a +_permanent replication slot_ on the primary node's Patroni cluster, and ensure +password authentication is used. + +On Patroni instances for the primary node, add the following to the +`/etc/gitlab/gitlab.rb` file: + +```ruby +# You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints: +# +# Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' }, +# We don't support setting a permanent replication slot for logical replication type +patroni['replication_slots'] = { + 'geo_secondary' => { 'type' => 'physical' } +} + +postgresql['md5_auth_cidr_addresses'] = [ + 'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32', + 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32' # we list all secondary instances as they can all become a Standby Leader + # any other instance that needs access to the database as per documentation +] + +postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' +postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' +postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' +``` + +On Patroni instances for the secondary node, add the following to the +`/etc/gitlab/gitlab.rb` file: + +```ruby +postgresql['md5_auth_cidr_addresses'] = [ + 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', + # any other instance that needs access to the database as per documentation +] + +patroni['enable'] = true +patroni['standby_cluster']['enable'] = true +patroni['standby_cluster']['host'] = 'PATRONI_PRIMARY_LEADER_IP' # this needs to be changed anytime the primary Leader changes +patroni['standby_cluster']['port'] = 5432 +patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # or the unique replication slot name you setup before +patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' +``` + ## Troubleshooting Read the [troubleshooting document](../replication/troubleshooting.md). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 59a6f2596da..0d9c761e078 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -121,7 +121,7 @@ The following list depicts the network architecture of Gitaly: - Authentication is done through a static token which is shared among the Gitaly and GitLab Rails nodes. -DANGER: **Danger:** +DANGER: **Warning:** Gitaly servers must not be exposed to the public internet as Gitaly's network traffic is unencrypted by default. The use of firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#enable-tls-support). @@ -373,7 +373,7 @@ As the final step, you must update Gitaly clients to switch from using local Git the Gitaly servers you just configured. This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly -servers will cause all Gitaly requests to fail. For example, any sort of network, firewall, or name +servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name resolution problems. Additionally, you must [disable Rugged](../nfs.md#improving-nfs-performance-with-gitlab) @@ -450,7 +450,7 @@ server (with `gitaly_address`) unless you setup with special When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS. -DANGER: **Danger:** +DANGER: **Warning:** If you have [server hooks](../server_hooks.md) configured, either per repository or globally, you must move these to the Gitaly servers. If you have multiple Gitaly servers, copy your server hooks to all Gitaly servers. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index bd075e86a15..20dce6d68df 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -42,6 +42,8 @@ a mail server feature where any email to `user+arbitrary_tag@example.com` will e in the mailbox for `user@example.com` . It is supported by providers such as Gmail, Google Apps, Yahoo! Mail, Outlook.com, and iCloud, as well as the [Postfix mail server](reply_by_email_postfix_setup.md), which you can run on-premises. +Microsoft Exchange Server [does not support sub-addressing](#microsoft-exchange-server), +and Microsoft Office 365 [does not support sub-addressing by default](#microsoft-office-365) TIP: **Tip:** If your provider or server supports email sub-addressing, we recommend using it. @@ -326,11 +328,11 @@ incoming_email: #### Microsoft Exchange Server -Example configurations for Microsoft Exchange Server with IMAP enabled. Since +Example configurations for Microsoft Exchange Server with IMAP enabled. Because Exchange does not support sub-addressing, only two options exist: -- Catch-all mailbox (recommended for Exchange-only) -- Dedicated email address (supports Reply by Email only) +- [Catch-all mailbox](#catch-all-mailbox) (recommended for Exchange-only) +- [Dedicated email address](#dedicated-email-address) (supports Reply by Email only) ##### Catch-all mailbox @@ -417,7 +419,8 @@ Example for source installs: incoming_email: enabled: true - # Exchange does not support sub-addressing, and we're not using a catch-all mailbox so %{key} is not used here + # Exchange does not support sub-addressing, + # and we're not using a catch-all mailbox so %{key} is not used here address: "incoming@exchange.example.com" # Email account username @@ -433,3 +436,180 @@ incoming_email: # Whether the IMAP server uses SSL ssl: true ``` + +#### Microsoft Office 365 + +Example configurations for Microsoft Office 365 with IMAP enabled. + +##### Sub-addressing mailbox + +NOTE: **Note:** +As of September 2020 sub-addressing support +[has been added to Office 365](https://office365.uservoice.com/forums/273493-office-365-admin/suggestions/18612754-support-for-dynamic-email-aliases-in-office-36). This feature is not +enabled by default, and must be enabled through PowerShell. + +This series of PowerShell commands enables [sub-addressing](#email-sub-addressing) +at the organization level in Office 365. This allows all mailboxes in the organization +to receive sub-addressed mail: + +NOTE: **Note:** +This series of commands will enable sub-addressing at the organization +level in Office 365. This will allow all mailboxes in the organization +to receive sub-addressed mail. + +```powershell +Set-ExecutionPolicy RemoteSigned -Scope CurrentUser + +$UserCredential = Get-Credential + +$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection + +Import-PSSession $Session -DisableNameChecking + +Set-OrganizationConfig -AllowPlusAddressInRecipients $true +``` + +This example for Omnibus GitLab assumes the mailbox `incoming@office365.example.com`: + +```ruby +gitlab_rails['incoming_email_enabled'] = true + +# The email address including the `%{key}` placeholder that will be replaced +# to reference the item being replied to. The placeholder can be omitted, but if +# present, it must appear in the "user" part of the address (before the `@`). +gitlab_rails['incoming_email_address'] = "incoming+%{key}@office365.example.com" + +# Email account username +# Typically this is the userPrincipalName (UPN) +gitlab_rails['incoming_email_email'] = "incoming@office365.example.com" +# Email account password +gitlab_rails['incoming_email_password'] = "[REDACTED]" + +# IMAP server host +gitlab_rails['incoming_email_host'] = "outlook.office365.com" +# IMAP server port +gitlab_rails['incoming_email_port'] = 993 +# Whether the IMAP server uses SSL +gitlab_rails['incoming_email_ssl'] = true +``` + +This example for source installs assumes the mailbox `incoming@office365.example.com`: + +```yaml +incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced + # to reference the item being replied to. The placeholder can be omitted, but + # if present, it must appear in the "user" part of the address (before the `@`). + address: "incoming+%{key}@office365.example.comm" + + # Email account username + # Typically this is the userPrincipalName (UPN) + user: "incoming@office365.example.comm" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "outlook.office365.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true +``` + +##### Catch-all mailbox + +This example for Omnibus installs assumes the catch-all mailbox `incoming@office365.example.com`: + +```ruby +gitlab_rails['incoming_email_enabled'] = true + +# The email address including the `%{key}` placeholder that will be replaced to +# reference the item being replied to. The placeholder can be omitted, but if present, +# it must appear in the "user" part of the address (before the `@`). +gitlab_rails['incoming_email_address'] = "incoming-%{key}@office365.example.com" + +# Email account username +# Typically this is the userPrincipalName (UPN) +gitlab_rails['incoming_email_email'] = "incoming@office365.example.com" +# Email account password +gitlab_rails['incoming_email_password'] = "[REDACTED]" + +# IMAP server host +gitlab_rails['incoming_email_host'] = "outlook.office365.com" +# IMAP server port +gitlab_rails['incoming_email_port'] = 993 +# Whether the IMAP server uses SSL +gitlab_rails['incoming_email_ssl'] = true +``` + +This example for source installs assumes the catch-all mailbox `incoming@office365.example.com`: + +```yaml +incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced + # to reference the item being replied to. The placeholder can be omitted, but + # if present, it must appear in the "user" part of the address (before the `@`). + address: "incoming-%{key}@office365.example.com" + + # Email account username + # Typically this is the userPrincipalName (UPN) + user: "incoming@ad-domain.example.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "outlook.office365.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true +``` + +##### Dedicated email address + +This example for Omnibus installs assumes the dedicated email address `incoming@office365.example.com`: + +```ruby +gitlab_rails['incoming_email_enabled'] = true + +gitlab_rails['incoming_email_address'] = "incoming@office365.example.com" + +# Email account username +# Typically this is the userPrincipalName (UPN) +gitlab_rails['incoming_email_email'] = "incoming@office365.example.com" +# Email account password +gitlab_rails['incoming_email_password'] = "[REDACTED]" + +# IMAP server host +gitlab_rails['incoming_email_host'] = "outlook.office365.com" +# IMAP server port +gitlab_rails['incoming_email_port'] = 993 +# Whether the IMAP server uses SSL +gitlab_rails['incoming_email_ssl'] = true +``` + +This example for source installs assumes the dedicated email address `incoming@office365.example.com`: + +```yaml +incoming_email: + enabled: true + + address: "incoming@office365.example.com" + + # Email account username + # Typically this is the userPrincipalName (UPN) + user: "incoming@office365.example.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "outlook.office365.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true +``` diff --git a/doc/administration/index.md b/doc/administration/index.md index 07aa3b50bc6..0aa94b86371 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" description: 'Learn how to install, configure, update, and maintain your GitLab instance.' --- @@ -143,7 +143,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Container Registry](packages/container_registry.md): Configure Container Registry with GitLab. - [Package Registry](packages/index.md): Enable GitLab to act as an NPM Registry and a Maven Repository. -- [Dependency Proxy](packages/dependency_proxy.md): Configure the Dependency Proxy, a local proxy for frequently used upstream images/packages. **(PREMIUM ONLY)** +- [Dependency Proxy](packages/dependency_proxy.md): Configure the Dependency Proxy, a local proxy for frequently used upstream images/packages. ### Repository settings diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index cb37be8d9dd..ec56a59fdc2 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -179,7 +179,6 @@ Plan.default.actual_limits.update!(project_hooks: 100) Plan.default.actual_limits.update!(group_hooks: 100) ``` -NOTE: **Note:** Set the limit to `0` to disable it. ## Incoming emails from auto-responders @@ -217,7 +216,6 @@ Plan.default.actual_limits.update!(offset_pagination_limit: 10000) - **Default offset pagination limit:** 50000 -NOTE: **Note:** Set the limit to `0` to disable it. ## CI/CD limits @@ -250,7 +248,6 @@ To set this limit on a self-managed installation, run the following in the Plan.default.actual_limits.update!(ci_active_jobs: 500) ``` -NOTE: **Note:** Set the limit to `0` to disable it. ### Number of CI/CD subscriptions to a project @@ -273,7 +270,6 @@ To set this limit on a self-managed installation, run the following in the Plan.default.actual_limits.update!(ci_project_subscriptions: 500) ``` -NOTE: **Note:** Set the limit to `0` to disable it. ### Number of pipeline schedules @@ -351,7 +347,7 @@ setting is used: | `ci_max_artifact_size_license_management` | 0 | | `ci_max_artifact_size_license_scanning` | 0 | | `ci_max_artifact_size_load_performance` | 0 | -| `ci_max_artifact_size_lsif` | 20 MB ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3) | +| `ci_max_artifact_size_lsif` | 100 MB ([Introduced at 20 MB](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3 and [raised to 100 MB](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46980) in GitLab 13.6.) | | `ci_max_artifact_size_metadata` | 0 | | `ci_max_artifact_size_metrics_referee` | 0 | | `ci_max_artifact_size_metrics` | 0 | @@ -462,11 +458,10 @@ Setting a limit helps reduce the memory usage of the indexing processes as well as the overall index size. This value defaults to `1024 KiB` (1 MiB) as any text files larger than this likely aren't meant to be read by humans. -NOTE: **Note:** -You must set a limit, as an unlimited file size is not supported. Setting this -value to be greater than the amount of memory on GitLab's Sidekiq nodes will -lead to GitLab's Sidekiq nodes running out of memory as they will pre-allocate -this amount of memory during indexing. +You must set a limit, as unlimited file sizes aren't supported. Setting this +value to be greater than the amount of memory on GitLab's Sidekiq nodes causes +GitLab's Sidekiq nodes to run out of memory, as they will pre-allocate this +amount of memory during indexing. ### Maximum field length @@ -486,7 +481,6 @@ indexed](#maximum-file-size-indexed)). This limit can be configured for self-managed installations when [enabling Elasticsearch](../integration/elasticsearch.md#enabling-advanced-search). -NOTE: **Note:** Set the limit to `0` to disable it. ## Wiki limits diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index 7eadb54804b..b00586b281b 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -4,15 +4,15 @@ group: Conversion info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Instance Review **(CORE ONLY)** +# Instance Review > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in [GitLab Core](https://about.gitlab.com/pricing/) 11.3. -If you are running a medium size instance (50+ users) of -[GitLab Core](https://about.gitlab.com/pricing/) edition, you are qualified for a -free Instance Review. +If you run a medium-sized self-managed instance (50+ users) of a free version of GitLab, +[either Community Edition or unlicensed Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/), +you qualify for a free Instance Review. -1. Sign in as a user with Admin [permissions](../user/permissions.md). +1. Sign in as a user with administrator [permissions](../user/permissions.md). 1. In the top menu, click your user icon, and select **Get a free instance review**: diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 8069b12e0b9..a010903013e 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -103,7 +103,7 @@ If you configure GitLab to store artifacts on object storage, you may also want [eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage). In both cases, job logs are archived and moved to object storage when the job completes. -DANGER: **Danger:** +DANGER: **Warning:** In a multi-server setup you must use one of the options to [eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage), or job logs could be lost. diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index c89ffb8da8b..23343d309de 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -43,6 +43,45 @@ To change the location where the job logs will be stored, follow the steps below 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +Alternatively, if you have existing job logs you can follow +these steps to move the logs to a new location without losing any data. + +1. Pause continuous integration data processing by updating this setting in `/etc/gitlab/gitlab.rb`. + Jobs in progress are not affected, based on how [data flow](#data-flow) works. + + ```ruby + sidekiq['experimental_queue_selector'] = true + sidekiq['queue_groups'] = [ + "feature_category!=continuous_integration" + ] + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the + changes to take effect. +1. Set the new storage location in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the + changes to take effect. +1. Use `rsync` to move job logs from the current location to the new location: + + ```shell + sudo rsync -avzh --remove-source-files --ignore-existing --progress /var/opt/gitlab/gitlab-ci/builds/ /mnt/to/gitlab-ci/builds` + ``` + + Use `--ignore-existing` so you don't override new job logs with older versions of the same log. +1. Unpause continuous integration data processing by editing `/etc/gitlab/gitlab.rb` and removing the `sidekiq` setting you updated earlier. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the + changes to take effect. +1. Remove the old job logs storage location: + + ```shell + sudo rm -rf /var/opt/gitlab/gitlab-ci/builds` + ``` + **In installations from source:** 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -82,7 +121,7 @@ job output in the UI will be empty. For example, to delete all job logs older than 60 days, run the following from a shell in your GitLab instance: -DANGER: **Danger:** +DANGER: **Warning:** This command will permanently delete the log files and is irreversible. ```shell @@ -104,7 +143,7 @@ The data flow is the same as described in the [data flow section](#data-flow) with one change: _the stored path of the first two phases is different_. This incremental log architecture stores chunks of logs in Redis and a persistent store (object storage or database) instead of file storage. Redis is used as first-class storage, and it stores up-to 128KB -of data. Once the full chunk is sent, it is flushed to a persistent store, either object storage (temporary directory) or database. +of data. After the full chunk is sent, it is flushed to a persistent store, either object storage (temporary directory) or database. After a while, the data in Redis and a persistent store will be archived to [object storage](#uploading-logs-to-object-storage). The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`. @@ -114,9 +153,9 @@ Here is the detailed data flow: 1. The runner picks a job from GitLab 1. The runner sends a piece of log to GitLab 1. GitLab appends the data to Redis -1. Once the data in Redis reach 128KB, the data is flushed to a persistent store (object storage or the database). +1. After the data in Redis reaches 128KB, the data is flushed to a persistent store (object storage or the database). 1. The above steps are repeated until the job is finished. -1. Once the job is finished, GitLab schedules a Sidekiq worker to archive the log. +1. After the job is finished, GitLab schedules a Sidekiq worker to archive the log. 1. The Sidekiq worker archives the log to object storage and cleans up the log in Redis and a persistent store (object storage or the database). diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index ae4fa83662a..410381ff2b0 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -114,7 +114,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. ## Readiness check -It is strongly recommend that multi-node deployments configure load balancers to utilize the [readiness check](../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma will not accept requests. +It is strongly recommend that multi-node deployments configure load balancers to use the [readiness check](../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma will not accept requests. <!-- ## Troubleshooting diff --git a/doc/administration/logs.md b/doc/administration/logs.md index c9e0cca807e..e5523ba67aa 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -14,8 +14,8 @@ Find more about them [in Audit Events documentation](audit_events.md). System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. -[Read more about how to customise logging on Omnibus GitLab -installations](https://docs.gitlab.com/omnibus/settings/logs.html) +Read more about how to +[customize logging on Omnibus GitLab installations](https://docs.gitlab.com/omnibus/settings/logs.html) including adjusting log retention, log forwarding, switching logs from JSON to plain text logging, and more. @@ -87,7 +87,7 @@ In addition, the log contains the originating IP address, (`remote_ip`), the user's ID (`user_id`), and username (`username`). Some endpoints such as `/search` may make requests to Elasticsearch if using -[Advanced Search](../user/search/advanced_global_search.md). These will +[Advanced Search](../user/search/advanced_global_search.md). These additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`, which correspond to: @@ -367,9 +367,9 @@ After GitLab version 12.2, this file was renamed from `githost.log` to `git_json.log` and stored in JSON format. GitLab has to interact with Git repositories, but in some rare cases -something can go wrong, and in this case you may need know what exactly +something can go wrong. If this happens, you need to know exactly what happened. This log file contains all failed requests from GitLab to Git -repositories. In the majority of cases this file will be useful for developers +repositories. In the majority of cases this file is useful for developers only. For example: ```json @@ -482,7 +482,7 @@ This file contains logging information about jobs before Sidekiq starts processing them, such as before being enqueued. This log file follows the same structure as -[`sidekiq.log`](#sidekiqlog), so it will be structured as JSON if +[`sidekiq.log`](#sidekiqlog), so it is structured as JSON if you've configured this for Sidekiq as mentioned above. ## `gitlab-shell.log` @@ -582,6 +582,12 @@ This file lives in `/var/log/gitlab/gitaly/current` and is produced by [runit](h This file lives in `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab packages. Native [gRPC](https://grpc.io/) logging used by Gitaly. +### `gitaly_ruby_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2678) in GitLab 13.6. + +This file lives in `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is produced by [`gitaly-ruby`](gitaly/reference.md#gitaly-ruby). It contains an access log of gRPC calls made by Gitaly to `gitaly-ruby`. + ## Puma Logs ### `puma_stdout.log` @@ -599,7 +605,7 @@ installations from source. ## Unicorn Logs Starting with GitLab 13.0, Puma is the default web server used in GitLab -all-in-one package based installations as well as GitLab Helm chart deployments. +all-in-one package based installations, and GitLab Helm chart deployments. ### `unicorn_stdout.log` @@ -720,21 +726,26 @@ was initiated, such as `1509705644.log` ## `sidekiq_exporter.log` and `web_exporter.log` If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq -will start a Web server and listen to the defined port (default: +starts a Web server and listen to the defined port (default: `8082`). By default, Sidekiq Exporter access logs are disabled but can -be enabled via the `sidekiq['exporter_log_enabled'] = true` option in `/etc/gitlab/gitlab.rb` -for Omnibus installations, or via the `sidekiq_exporter.log_enabled` option -in `gitlab.yml` for installations from source. When enabled, -access logs will be generated in +be enabled: + +- For Omnibus GitLab installations, using the `sidekiq['exporter_log_enabled'] = true` + option in `/etc/gitlab/gitlab.rb`. +- For installations from source, using the `sidekiq_exporter.log_enabled` option + in `gitlab.yml`. + +When enabled, access logs are generated in `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for installations from source. -If Prometheus metrics and the Web Exporter are both enabled, Puma/Unicorn will -start a Web server and listen to the defined port (default: `8083`). Access logs -will be generated in `/var/log/gitlab/gitlab-rails/web_exporter.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/web_exporter.log` for -installations from source. +If Prometheus metrics and the Web Exporter are both enabled, Puma/Unicorn +starts a Web server and listen to the defined port (default: `8083`), and access logs +are generated: + +- For Omnibus GitLab packages, in `/var/log/gitlab/gitlab-rails/web_exporter.log`. +- For installations from source, in `/home/git/gitlab/log/web_exporter.log`. ## `database_load_balancing.log` **(PREMIUM ONLY)** @@ -750,13 +761,11 @@ It's stored at: > Introduced in GitLab 12.6. -This file lives in -`/var/log/gitlab/gitlab-rails/elasticsearch.log` for Omnibus GitLab -packages or in `/home/git/gitlab/log/elasticsearch.log` for installations -from source. +This file logs information related to the Elasticsearch Integration, including +errors during indexing or searching Elasticsearch. It's stored at: -It logs information related to the Elasticsearch Integration including -errors during indexing or searching Elasticsearch. +- `/var/log/gitlab/gitlab-rails/elasticsearch.log` for Omnibus GitLab packages. +- `/home/git/gitlab/log/elasticsearch.log` for installations from source. Each line contains a JSON line that can be ingested by services like Elasticsearch and Splunk. Line breaks have been added to the following example line for clarity: @@ -779,12 +788,12 @@ Line breaks have been added to the following example line for clarity: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17819) in GitLab 12.6. -This file lives in -`/var/log/gitlab/gitlab-rails/exceptions_json.log` for Omnibus GitLab -packages or in `/home/git/gitlab/log/exceptions_json.log` for installations -from source. +This file logs the information about exceptions being tracked by +`Gitlab::ErrorTracking`, which provides a standard and consistent way of +[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling). This file is stored in: -It logs the information about exceptions being tracked by `Gitlab::ErrorTracking` which provides a standard and consistent way of [processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling). +- `/var/log/gitlab/gitlab-rails/exceptions_json.log` for Omnibus GitLab packages. +- `/home/git/gitlab/log/exceptions_json.log` for installations from source. Each line contains a JSON line that can be ingested by Elasticsearch. For example: @@ -814,7 +823,7 @@ Omnibus GitLab packages or in `/home/git/gitlab/log/service_measurement.log` for installations from source. It contains only a single structured log with measurements for each service execution. -It will contain measurements such as the number of SQL calls, `execution_time`, `gc_stats`, and `memory usage`. +It contains measurements such as the number of SQL calls, `execution_time`, `gc_stats`, and `memory usage`. For example: @@ -847,7 +856,9 @@ This file is stored in: - `/var/log/gitlab/gitlab-rails/update_mirror_service_json.log` for Omnibus GitLab installations. - `/home/git/gitlab/log/update_mirror_service_json.log` for installations from source. -This file contains information about any errors that occurred during project mirroring. +This file contains information about LFS errors that occurred during project mirroring. +While we work to move other project mirroring errors into this log, the [general log](#productionlog) +can be used. ```json { @@ -939,7 +950,7 @@ For Omnibus GitLab installations, Alertmanager logs reside in `/var/log/gitlab/a ## Crond Logs -For Omnibus GitLab installations, crond logs reside in `/var/log/gitlab/crond/`. +For Omnibus GitLab installations, `crond` logs reside in `/var/log/gitlab/crond/`. ## Grafana Logs @@ -957,6 +968,11 @@ For Omnibus GitLab installations, GitLab Monitor logs reside in `/var/log/gitlab For Omnibus GitLab installations, GitLab Exporter logs reside in `/var/log/gitlab/gitlab-exporter/`. +## GitLab Kubernetes Agent Server + +For Omnibus GitLab installations, GitLab Kubernetes Agent Server logs reside +in `/var/log/gitlab/gitlab-kas/`. + ## Gathering logs When [troubleshooting](troubleshooting/index.md) issues that aren't localized to one of the @@ -967,7 +983,7 @@ from a GitLab instance. If the bug or error is readily reproducible, save the main GitLab logs [to a file](troubleshooting/linux_cheat_sheet.md#files--dirs) while reproducing the -problem once or more times: +problem a few times: ```shell sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log @@ -980,7 +996,7 @@ Conclude the log gathering with <kbd>Ctrl</kbd> + <kbd>C</kbd>. If performance degradations or cascading errors occur that can't readily be attributed to one of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) can provide a perspective spanning all of Omnibus GitLab. For more details and instructions -to run it, see [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos). +to run it, read [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos). NOTE: **Note:** GitLab Support likes to use this custom-made tool. @@ -989,5 +1005,5 @@ GitLab Support likes to use this custom-made tool. [Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool for creating and comparing performance statistics from GitLab logs. -For more details and instructions to run it, see -[read the documentation for fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage). +For more details and instructions to run it, read the +[documentation for fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats#usage). diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index 64f45001438..da7796c05f0 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -17,11 +17,10 @@ monitor the health and progress of the importer. |------------------------------------------|-----------| | `github_importer_total_duration_seconds` | histogram | -This metric tracks the total time spent (in seconds) importing a project (from +This metric tracks the total time, in seconds, spent importing a project (from project creation until the import process finishes), for every imported project. - The name of the project is stored in the `project` label in the format -`namespace/name` (e.g. `gitlab-org/gitlab`). +`namespace/name` (such as `gitlab-org/gitlab`). ## Number of imported projects @@ -59,7 +58,7 @@ projects. This metric does not expose any labels. This metric tracks the number of imported issues across all projects. The name of the project is stored in the `project` label in the format -`namespace/name` (e.g. `gitlab-org/gitlab`). +`namespace/name` (such as `gitlab-org/gitlab`). ## Number of imported pull requests @@ -70,7 +69,7 @@ The name of the project is stored in the `project` label in the format This metric tracks the number of imported pull requests across all projects. The name of the project is stored in the `project` label in the format -`namespace/name` (e.g. `gitlab-org/gitlab`). +`namespace/name` (such as `gitlab-org/gitlab`). ## Number of imported comments @@ -81,7 +80,7 @@ The name of the project is stored in the `project` label in the format This metric tracks the number of imported comments across all projects. The name of the project is stored in the `project` label in the format -`namespace/name` (e.g. `gitlab-org/gitlab`). +`namespace/name` (such as `gitlab-org/gitlab`). ## Number of imported pull request review comments @@ -92,7 +91,7 @@ The name of the project is stored in the `project` label in the format This metric tracks the number of imported comments across all projects. The name of the project is stored in the `project` label in the format -`namespace/name` (e.g. `gitlab-org/gitlab`). +`namespace/name` (such as `gitlab-org/gitlab`). ## Number of imported repositories diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 6677eb73664..eb9dab1af59 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -7,21 +7,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Configuration GitLab Performance Monitoring is disabled by default. To enable it and change any of its -settings, navigate to **Admin Area > Settings > Metrics and profiling** -(`/admin/application_settings/metrics_and_profiling`). +settings: - +1. Navigate to **Admin Area > Settings > Metrics and profiling** + (`/admin/application_settings/metrics_and_profiling`): -Finally, a restart of all GitLab processes is required for the changes to take -effect: +  -```shell -# For Omnibus installations -sudo gitlab-ctl restart +1. You must restart all GitLab processes for the changes to take effect: -# For installations from source -sudo service gitlab restart -``` + - For Omnibus GitLab installations: `sudo gitlab-ctl restart` + - For installations from source: `sudo service gitlab restart` ## Pending Migrations diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 77ed5d442e6..dc77a35e44b 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -113,7 +113,7 @@ If you require access to your old Grafana data but don't meet one of these crite 1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#exporting-a-dashboard) you need. 1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#importing-a-dashboard). -DANGER: **Danger:** +DANGER: **Warning:** These actions pose a temporary vulnerability while your old Grafana data is in use. Deciding to take any of these actions should be weighed carefully with your need to access existing data and dashboards. diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 68b89b837ac..6ea756619f5 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Performance Monitoring GitLab comes with its own application performance measuring system as of GitLab -8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the +8.4, called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the Community and Enterprise editions. Apart from this introduction, you are advised to read through the following @@ -42,7 +42,7 @@ Two types of metrics are collected: Transaction metrics are metrics that can be associated with a single transaction. This includes statistics such as the transaction duration, timings -of any executed SQL queries, time spent rendering HAML views, etc. These metrics +of any executed SQL queries, time spent rendering HAML views, and so on. These metrics are collected for every Rack request and Sidekiq job processed. ### Sampled Metrics @@ -59,5 +59,5 @@ parts: The actual interval can be anywhere between a half of the defined interval and a half above the interval. For example, for a user defined interval of 15 seconds the actual interval can be anywhere between 7.5 and 22.5. The interval is -re-generated for every sampling run instead of being generated once and re-used +re-generated for every sampling run instead of being generated one time and reused for the duration of the process' lifetime. diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 971dafb4ba2..aa2eb9f3415 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -13,7 +13,6 @@ The [GitLab exporter](https://gitlab.com/gitlab-org/gitlab-exporter) enables you measure various GitLab metrics pulled from Redis and the database in Omnibus GitLab instances. -NOTE: **Note:** For installations from source you must install and configure it yourself. To enable the GitLab exporter in an Omnibus GitLab instance: diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index adb1f719f3c..5c9268b54e1 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -13,7 +13,6 @@ To enable the GitLab Prometheus metrics: 1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. -NOTE: **Note:** For installations from source you must configure it yourself. ## Collecting the metrics @@ -124,8 +123,6 @@ The following metrics can be controlled by feature flags: |:---------------------------------------------------------------|:-------------------------------------------------------------------| | `gitlab_method_call_duration_seconds` | `prometheus_metrics_method_instrumentation` | | `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` | -| `gitlab_issuable_fast_count_by_state_total` | `soft_fail_count_by_state` | -| `gitlab_issuable_fast_count_by_state_failures_total` | `soft_fail_count_by_state` | ## Sidekiq metrics @@ -211,6 +208,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` | | `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | | `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | +| `destroyed_job_artifacts_count_total` | Counter | 13.6 | Number of destroyed expired job artifacts | | ## Database load balancing metrics **(PREMIUM ONLY)** diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 63231996dcc..91f810dc681 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -26,12 +26,11 @@ access to high quality time-series monitoring of GitLab services. Prometheus works by periodically connecting to data sources and collecting their performance metrics through the [various exporters](#bundled-software-metrics). To view and work with the monitoring data, you can either -[connect directly to Prometheus](#viewing-performance-metrics) or utilize a +[connect directly to Prometheus](#viewing-performance-metrics) or use a dashboard tool like [Grafana](https://grafana.com). ## Configuring Prometheus -NOTE: **Note:** For installations from source, you must install and configure it yourself. Prometheus and its exporters are on by default, starting with GitLab 9.0. @@ -54,7 +53,7 @@ To disable Prometheus and all of its exporters, as well as any added in the futu ### Changing the port and address Prometheus listens on -NOTE: **Note:** +CAUTION: **Caution:** The following change was added in [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1261). Although possible, it's not recommended to change the port Prometheus listens on, as this might affect or conflict with other services running on the GitLab @@ -178,7 +177,6 @@ The next step is to tell all the other nodes where the monitoring node is: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** After monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both `consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` @@ -186,7 +184,7 @@ will result in errors. ### Using an external Prometheus server -NOTE: **Note:** +CAUTION: **Caution:** Prometheus and most exporters don't support authentication. We don't recommend exposing them outside the local network. A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for [GitLab deployments with multiple nodes](../../reference_architectures/index.md). diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index dae1f02b196..fea78a3685c 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -9,7 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w The [node exporter](https://github.com/prometheus/node_exporter) enables you to measure various machine resources such as memory, disk and CPU utilization. -NOTE: **Note:** For installations from source you must install and configure it yourself. To enable the node exporter: diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index 4554bc06401..ff0cfc65e10 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -11,7 +11,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w The [PgBouncer exporter](https://github.com/prometheus-community/pgbouncer_exporter) enables you to measure various [PgBouncer](https://www.pgbouncer.org/) metrics. -NOTE: **Note:** For installations from source you must install and configure it yourself. To enable the PgBouncer exporter: diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 9eb9ba3c59f..f7368556235 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -8,7 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w The [PostgreSQL Server Exporter](https://github.com/wrouesnel/postgres_exporter) allows you to export various PostgreSQL metrics. -NOTE: **Note:** For installations from source you must install and configure it yourself. To enable the PostgreSQL Server Exporter: diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index 16a758c9804..41a84f1f3ed 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -10,7 +10,6 @@ The [Redis exporter](https://github.com/oliver006/redis_exporter) enables you to various [Redis](https://redis.io) metrics. For more information on what is exported, [read the upstream documentation](https://github.com/oliver006/redis_exporter/blob/master/README.md#whats-exported). -NOTE: **Note:** For installations from source you must install and configure it yourself. To enable the Redis exporter: diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index fabbfb2552e..3e16b173003 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -19,11 +19,10 @@ From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, support for NFS for Git repositories is scheduled to be removed. Upgrade to [Gitaly Cluster](gitaly/praefect.md) as soon as possible. -NOTE: **Note:** -Filesystem performance has a big impact on overall GitLab -performance, especially for actions that read or write to Git repositories. See -[Filesystem Performance Benchmarking](operations/filesystem_benchmarking.md) -for steps to test filesystem performance. +Filesystem performance can impact overall GitLab performance, especially for +actions that read or write to Git repositories. For steps you can use to test +filesystem performance, see +[Filesystem Performance Benchmarking](operations/filesystem_benchmarking.md). ## Known kernel version incompatibilities @@ -119,7 +118,7 @@ To disable NFS server delegation, do the following: 1. Restart the NFS server process. For example, on CentOS run `service nfs restart`. -NOTE: **Important note:** +NOTE: **Note:** The kernel bug may be fixed in [more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). Red Hat Enterprise 7 [shipped a kernel update](https://access.redhat.com/errata/RHSA-2019:2029) @@ -250,9 +249,9 @@ gitlab_rails['shared_path'] = '/gitlab-nfs/gitlab-data/shared' gitlab_ci['builds_directory'] = '/gitlab-nfs/gitlab-data/builds' ``` -Run `sudo gitlab-ctl reconfigure` to start using the central location. Please -be aware that if you had existing data you will need to manually copy/rsync it -to these new locations and then restart GitLab. +Run `sudo gitlab-ctl reconfigure` to start using the central location. Be aware +that if you had existing data, you'll need to manually copy or rsync it to +these new locations, and then restart GitLab. ### Bind mounts @@ -307,8 +306,12 @@ by testing the following commands: ```shell sudo mkdir /gitlab-nfs/test-dir sudo chown git /gitlab-nfs/test-dir -sudo chgrp gitlab-www /gitlab-nfs/test-dir sudo chgrp root /gitlab-nfs/test-dir +sudo chmod 0700 /gitlab-nfs/test-dir +sudo chgrp gitlab-www /gitlab-nfs/test-dir +sudo chmod 0751 /gitlab-nfs/test-dir +sudo chgrp git /gitlab-nfs/test-dir +sudo chmod 2770 /gitlab-nfs/test-dir sudo chmod 2755 /gitlab-nfs/test-dir sudo -u git mkdir /gitlab-nfs/test-dir/test2 sudo -u git chmod 2755 /gitlab-nfs/test-dir/test2 @@ -323,7 +326,7 @@ Any `Operation not permitted` errors means you should investigate your NFS serve If the traffic between your NFS server and NFS client(s) is subject to port filtering by a firewall, then you will need to reconfigure that firewall to allow NFS communication. -[This guide from TDLP](http://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) +[This guide from TDLP](https://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to search for and review the specific documentation for your operating system or distribution and your firewall software. @@ -396,8 +399,8 @@ Additionally, this configuration is specifically warned against in the >system semantics, this can cause reliability problems. Specifically, delayed (asynchronous) writes >to the NFS server can cause data corruption problems. -For supported database architecture, please see our documentation on -[Configuring a Database for GitLab HA](postgresql/replication_and_failover.md). +For supported database architecture, see our documentation about +[configuring a database for replication and failover](postgresql/replication_and_failover.md). <!-- ## Troubleshooting diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 0ce1ff447ec..2b714c36d93 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -61,18 +61,17 @@ must be enabled, only the following providers can be used: - [Google Cloud Storage](#google-cloud-storage-gcs) - [Azure Blob storage](#azure-blob-storage) -Background upload is not supported with the consolidated object storage -configuration. We recommend enabling direct upload mode because it does -not require a shared folder, and [this setting may become the +Background upload isn't supported with the consolidated object storage +configuration. We recommend enabling direct upload mode because it doesn't +require a shared folder, and [this setting may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331). -NOTE: **Note:** -Consolidated object storage configuration cannot be used for -backups or Mattermost. See [the full table for a complete list](#storage-specific-configuration). +Consolidated object storage configuration can't be used for backups or +Mattermost. See the [full table for a complete list](#storage-specific-configuration). -NOTE: **Note:** -Enabling consolidated object storage will enable object storage for all object types. -If you wish to use local storage for specific object types, you can [selectively disable object storages](#selectively-disabling-object-storage). +Enabling consolidated object storage enables object storage for all object +types. If you want to use local storage for specific object types, you can +[selectively disable object storages](#selectively-disabling-object-storage). Most types of objects, such as CI artifacts, LFS files, upload attachments, and so on can be saved in object storage by specifying a single @@ -117,7 +116,7 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details. gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = '<terraform-state>' ``` - NOTE: For GitLab 9.4 or later, if you're using AWS IAM profiles, be sure to omit the + For GitLab 9.4 or later, if you're using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. For example: ```ruby @@ -262,10 +261,11 @@ Here are the valid connection parameters for GCS: | `google_project` | GCP project name | `gcp-project-12345` | | `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | | `google_json_key_location` | The JSON key path | `/path/to/gcp-project-12345-abcde.json` | +| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | -NOTE: **Note:** -The service account must have permission to access the bucket. -[See more](https://cloud.google.com/storage/docs/authentication) +The service account must have permission to access the bucket. Learn more +in Google's +[Cloud Storage authentication documentation](https://cloud.google.com/storage/docs/authentication). ##### Google example (consolidated form) @@ -280,6 +280,33 @@ gitlab_rails['object_store']['connection'] = { } ``` +##### Google example with ADC (consolidated form) + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275979) in GitLab 13.6. + +Google Cloud Application Default Credentials (ADC) are typically +used with GitLab to use the default service account. This eliminates the +need to supply credentials for the instance. For example: + +```ruby +gitlab_rails['object_store']['connection'] = { + 'provider' => 'Google', + 'google_project' => '<GOOGLE PROJECT>', + 'google_application_default' => true +} +``` + +If you use ADC, be sure that: + +- The service account that you use has the +[`iam.serviceAccounts.signBlob` permission](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob). + Typically this is done by granting the `Service Account Token Creator` role to the service account. +- Your virtual machines have the [correct access scopes to access Google Cloud APIs](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes). If the machines do not have the right scope, the error logs may show: + + ```markdown + Google::Apis::ClientError (insufficientPermissions: Request had insufficient authentication scopes.) + ``` + #### Azure Blob storage > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4. @@ -288,6 +315,11 @@ Although Azure uses the word `container` to denote a collection of blobs, GitLab standardizes on the term `bucket`. Be sure to configure Azure container names in the `bucket` settings. +Azure Blob storage can only be used with the [consolidated form](#consolidated-object-storage-configuration) +because a single set of credentials are used to access multiple +containers. The [storage-specific form](#storage-specific-configuration) +is not supported. For more details, see [how to transition to consolidated form](#transition-to-consolidated-form). + The following are the valid connection parameters for Azure. Read the [Azure Blob storage documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) to learn more. @@ -314,10 +346,9 @@ gitlab_rails['object_store']['connection'] = { ###### Azure Workhorse settings (source installs only) -NOTE: **Note:** -For source installations, Workhorse needs to be configured with the -Azure credentials as well. This is not needed in Omnibus installs because -the Workhorse settings are populated from the settings above. +For source installations, Workhorse also needs to be configured with Azure +credentials. This isn't needed in Omnibus installs, because the Workhorse +settings are populated from the previous settings. 1. Edit `/home/git/gitlab-workhorse/config.toml` and add or amend the following lines: @@ -337,14 +368,14 @@ GitLab Rails and Workhorse. #### OpenStack-compatible connection settings -NOTE: **Note:** -This is not compatible with the consolidated object storage form. -OpenStack Swift is only supported with the storage-specific form. See the -[S3 settings](#s3-compatible-connection-settings) if you want to use the consolidated form. +Although OpenStack Swift provides S3 compatibility, some users may want to use +the [Swift API](https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html). + +This isn't compatible with the consolidated object storage form. OpenStack Swift +is supported only with the storage-specific form. If you want to use the +consolidated form, see the [S3 settings](#s3-compatible-connection-settings). -While OpenStack Swift provides S3 compatibility, some users may want to use the -[Swift API](https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html). -Here are the valid connection settings below for the Swift API, provided by +Here are the valid connection settings for the Swift API, provided by [fog-openstack](https://github.com/fog/fog-openstack): | Setting | Description | Default | @@ -359,12 +390,11 @@ Here are the valid connection settings below for the Swift API, provided by #### Rackspace Cloud Files -NOTE: **Note:** -This is not compatible with the consolidated object -storage form. Rackspace Cloud is only supported with the storage-specific form. +The following table describes the valid connection parameters for +Rackspace Cloud, provided by [fog-rackspace](https://github.com/fog/fog-rackspace/). -Here are the valid connection parameters for Rackspace Cloud, provided by -[fog-rackspace](https://github.com/fog/fog-rackspace/): +This isn't compatible with the consolidated object storage form. +Rackspace Cloud is supported only with the storage-specific form. | Setting | Description | example | |---------|-------------|---------| @@ -374,13 +404,13 @@ Here are the valid connection parameters for Rackspace Cloud, provided by | `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | | `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | -NOTE: **Note:** -Regardless of whether the container has public access enabled or disabled, Fog will -use the TempURL method to grant access to LFS objects. If you see errors in logs referencing -instantiating storage with a `temp-url-key`, ensure that you have set the key properly -on the Rackspace API and in `gitlab.rb`. You can verify the value of the key Rackspace -has set by sending a GET request with token header to the service access endpoint URL -and comparing the output of the returned headers. +Regardless of whether the container has public access enabled or disabled, Fog +uses the TempURL method to grant access to LFS objects. If you see error +messages in logs that refer to instantiating storage with a `temp-url-key`, +be sure you have set the key properly both in the Rackspace API and in +`gitlab.rb`. You can verify the value of the key Rackspace has set by sending a +GET request with token header to the service access endpoint URL and comparing +the output of the returned headers. ### Object-specific configuration @@ -488,15 +518,16 @@ gitlab_rails['uploads_object_store_remote_directory'] = 'uploads' gitlab_rails['uploads_object_store_connection'] = { 'provider' => 'AWS', 'aws_access_key_id' => 'access_key', 'aws_secret_access_key' => 'secret' } ``` -While this provides flexibility in that it makes it possible for GitLab +Although this provides flexibility in that it makes it possible for GitLab to store objects across different cloud providers, it also creates additional complexity and unnecessary redundancy. Since both GitLab Rails and Workhorse components need access to object storage, the consolidated form avoids excessive duplication of credentials. -NOTE: **Note:** -The consolidated object storage configuration is **only** used if all -lines from the original form is omitted. To move to the consolidated form, remove the original configuration (for example, `artifacts_object_store_enabled`, `uploads_object_store_connection`, and so on.) +The consolidated object storage configuration is used _only_ if all lines from +the original form is omitted. To move to the consolidated form, remove the +original configuration (for example, `artifacts_object_store_enabled`, or +`uploads_object_store_connection`) ## Storage-specific configuration diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index a94f88439f2..8f90231a713 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -34,7 +34,7 @@ rcli() { # This example works for Omnibus installations of GitLab 7.3 or newer. For an # installation from source you will have to change the socket path and the # path to redis-cli. - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.shared_state.socket "$@" + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" } # test the new shell function; the response should be PONG diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 76dc9bf5510..5de79882083 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -38,7 +38,7 @@ To start multiple processes: process, and values in each item determine the queues it works on. For example, the following setting creates three Sidekiq processes, one to run on - `elastic_indexer`, one to run on `mailers`, and one process running all on queues: + `elastic_indexer`, one to run on `mailers`, and one process running on all queues: ```ruby sidekiq['queue_groups'] = [ @@ -110,34 +110,29 @@ you list: sudo gitlab-ctl reconfigure ``` -## Queue selector (experimental) +## Queue selector -> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. +> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. +> - [Sidekiq cluster including queue selector moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 12.10. +> - [Marked as supported](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 13.6. Renamed from `experimental_queue_selector` to `queue_selector`. -CAUTION: **Caution:** -As this is marked as **experimental**, it is subject to change at any -time, including **breaking backwards compatibility**. This is so that we -can react to changes we need for our GitLab.com deployment. We have a -tracking issue open to [remove the experimental -designation](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) -from this feature; please comment there if you are interested in using -this in your own deployment. - -In addition to selecting queues by name, as above, the -`experimental_queue_selector` option allows queue groups to be selected -in a more general way using the following components: +In addition to selecting queues by name, as above, the `queue_selector` +option allows queue groups to be selected in a more general way using +the following components: - Attributes that can be selected. - Operators used to construct a query. +When `queue_selector` is set, all `queue_groups` must be in the queue +selector syntax. + ### Available attributes - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1, `tags`. From the [list of all available attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml), -`experimental_queue_selector` allows selecting of queues by the -following attributes: +`queue_selector` allows selecting of queues by the following attributes: - `feature_category` - the [GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the @@ -169,8 +164,8 @@ neither of those tags. ### Available operators -`experimental_queue_selector` supports the following operators, listed -from highest to lowest precedence: +`queue_selector` supports the following operators, listed from highest +to lowest precedence: - `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a` and `query_b` are queries made up of the other operators here) will include @@ -201,7 +196,7 @@ In `/etc/gitlab/gitlab.rb`: ```ruby sidekiq['enable'] = true -sidekiq['experimental_queue_selector'] = true +sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ # Run all non-CPU-bound queues that are high urgency 'resource_boundary!=cpu&urgency=high', @@ -230,7 +225,7 @@ All of the aforementioned configuration options for `sidekiq` are available. By default, they will be configured as follows: ```ruby -sidekiq['experimental_queue_selector'] = false +sidekiq['queue_selector'] = false sidekiq['interval'] = nil sidekiq['max_concurrency'] = 50 sidekiq['min_concurrency'] = nil @@ -327,9 +322,9 @@ Running Sidekiq cluster is the default in GitLab 13.0 and later. ``` `min_concurrency` and `max_concurrency` are independent; one can be set without -the other. Setting `min_concurrency` to 0 will disable the limit. +the other. Setting `min_concurrency` to `0` will disable the limit. -For each queue group, let N be one more than the number of queues. The +For each queue group, let `N` be one more than the number of queues. The concurrency factor will be set to: 1. `N`, if it's between `min_concurrency` and `max_concurrency`. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 88ef69b29f2..c8830a45fb2 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -196,7 +196,8 @@ the database. The following instructions can be used to build OpenSSH 7.5: yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd ``` -1. Verify the installed version. In another window, attempt to login to the server: +1. Verify the installed version. In another window, attempt to sign in to the + server: ```shell ssh -v <your-centos-machine> @@ -204,7 +205,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5" - If not, you may need to restart `sshd` (e.g. `systemctl restart sshd.service`). + If not, you may need to restart `sshd` (for example, `systemctl restart sshd.service`). 1. *IMPORTANT!* Open a new SSH session to your server before exiting to make sure everything is working! If you need to downgrade, simple install the diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index aaeb8c723d0..864bbb3233e 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -13,6 +13,9 @@ Keep your GitLab instance up and running smoothly. you have been running a large GitLab server (thousands of users) since before GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis database after you upgrade to GitLab 7.3. +- [Rake tasks](../../raketasks/README.md): Tasks for common administration and operational processes such as + [cleaning up unneeded items from GitLab instance](../../raketasks/cleanup.md), integrity checks, + and more. - [Moving repositories](moving_repositories.md): Moving all repositories managed by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller @@ -28,6 +31,8 @@ Keep your GitLab instance up and running smoothly. performance can have a big impact on GitLab performance, especially for actions that read or write Git repositories. This information will help benchmark filesystem performance against known good and bad real-world systems. +- [The Rails Console](rails_console.md): Provides a way to interact with your GitLab instance from the command line. + Used for troubleshooting a problem or retrieving some data that can only be done through direct access to GitLab. - [ChatOps Scripts](https://gitlab.com/gitlab-com/chatops): The GitLab.com Infrastructure team uses this repository to house common ChatOps scripts they use to troubleshoot and maintain the production instance of GitLab.com. These scripts are likely useful to administrators of GitLab instances of all sizes. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 94eea1e25b8..b311bee1a5b 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -1,26 +1,69 @@ --- -stage: none -group: unassigned +stage: Create +group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference --- -# Moving repositories managed by GitLab +# Moving repositories managed by GitLab **(CORE ONLY)** Sometimes you need to move all repositories managed by GitLab to -another filesystem or another server. In this document we will look +another file system or another server. + +## Moving data within a GitLab instance + +The GitLab API is the recommended way to move Git repositories: + +- Between servers. +- Between different storage. +- From single-node Gitaly to Gitaly Cluster. + +For more information, see: + +- [Configuring additional storage for Gitaly](../gitaly/index.md#network-architecture). Within this + example, additional storage called `storage1` and `storage2` is configured. +- [The API documentation](../../api/project_repository_storage_moves.md) details the endpoints for + querying and scheduling repository moves. +- [Migrate existing repositories to Gitaly Cluster](../gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). + +### Limitations + +Read more in the [API documentation](../../api/project_repository_storage_moves.md#limitations). + +## Migrating to another GitLab instance + +[Using the API](#moving-data-within-a-gitlab-instance) isn't an option if you are migrating to a new +GitLab environment, for example: + +- From a single-node GitLab to a scaled-out architecture. +- From a GitLab instance in your private datacenter to a cloud provider. + +The rest of the document looks at some of the ways you can copy all your repositories from `/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`. -We will look at three scenarios: the target directory is empty, the -target directory contains an outdated copy of the repositories, and -how to deal with thousands of repositories. +We look at three scenarios: + +- The target directory is empty. +- The target directory contains an outdated copy of the repositories. +- How to deal with thousands of repositories. + +DANGER: **Warning:** +Each of the approaches we list can or does overwrite data in the target directory +`/mnt/gitlab/repositories`. Do not mix up the source and the target. + +### Recommended approach in all cases + +GitLab's [backup and restore capability](../../raketasks/backup_restore.md) should be used. Git +repositories are accessed, managed, and stored on GitLab servers by Gitaly as a database. Data loss +can result from directly accessing and copying Gitaly's files using tools like `rsync`. -DANGER: **Danger:** -Each of the approaches we list can/will overwrite data in the -target directory `/mnt/gitlab/repositories`. Do not mix up the -source and the target. +- From GitLab 13.3, backup performance can be improved by + [processing multiple repositories concurrently](../../raketasks/backup_restore.md#back-up-git-repositories-concurrently). +- Backups can be created of just the repositories using the + [skip feature](../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup). -## Target directory is empty: use a `tar` pipe +### Target directory is empty: use a `tar` pipe If the target directory `/mnt/gitlab/repositories` is empty the simplest thing to do is to use a `tar` pipe. This method has low @@ -35,7 +78,7 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ If you want to see progress, replace `-xf` with `-xvf`. -### `tar` pipe to another server +#### `tar` pipe to another server You can also use a `tar` pipe to copy data to another server. If your `git` user has SSH access to the new server as `git@newserver`, you @@ -47,15 +90,19 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ ``` If you want to compress the data before it goes over the network -(which will cost you CPU cycles) you can replace `ssh` with `ssh -C`. +(which costs you CPU cycles) you can replace `ssh` with `ssh -C`. -## The target directory contains an outdated copy of the repositories: use `rsync` +### The target directory contains an outdated copy of the repositories: use `rsync` + +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). If the target directory already contains a partial / outdated copy of the repositories it may be wasteful to copy all the data again with `tar`. In this scenario it is better to use `rsync`. This utility is either already installed on your system or easily installable -via `apt`, `yum`, etc. +via `apt`, `yum`, and so on. ```shell sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ @@ -66,7 +113,11 @@ The `/.` in the command above is very important, without it you can easily get the wrong directory structure in the target directory. If you want to see progress, replace `-a` with `-av`. -### Single `rsync` to another server +#### Single `rsync` to another server + +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). If the `git` user on your source system has SSH access to the target server you can send the repositories over the network with `rsync`. @@ -76,7 +127,11 @@ sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ git@newserver:/mnt/gitlab/repositories' ``` -## Thousands of Git repositories: use one `rsync` per repository +### Thousands of Git repositories: use one `rsync` per repository + +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). Every time you start an `rsync` job it has to inspect all files in the source directory, all files in the target directory, and then @@ -86,20 +141,20 @@ for your GitLab server. In cases like this you can make `rsync`'s life easier by dividing its work in smaller pieces, and sync one repository at a time. -In addition to `rsync` we will use [GNU -Parallel](http://www.gnu.org/software/parallel/). This utility is -not included in GitLab so you need to install it yourself with `apt` -or `yum`. Also note that the GitLab scripts we used below were added -in GitLab 8.1. +In addition to `rsync` we use [GNU Parallel](http://www.gnu.org/software/parallel/). +This utility is not included in GitLab so you need to install it yourself with `apt` +or `yum`. Also note that the GitLab scripts we used below were added in GitLab 8.1. **This process does not clean up repositories at the target location that no -longer exist at the source.** If you start using your GitLab instance with -`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos` -after switching to the new repository storage directory. +longer exist at the source.** -### Parallel `rsync` for all repositories known to GitLab +#### Parallel `rsync` for all repositories known to GitLab -This will sync repositories with 10 `rsync` processes at a time. We keep +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). + +This syncs repositories with 10 `rsync` processes at a time. We keep track of progress so that the transfer can be restarted if necessary. First we create a new directory, owned by `git`, to hold transfer @@ -154,7 +209,11 @@ cat /home/git/transfer-logs/* | sort | uniq -u |\ ` ``` -### Parallel `rsync` only for repositories with recent activity +#### Parallel `rsync` only for repositories with recent activity + +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). Suppose you have already done one sync that started after 2015-10-1 12:00 UTC. Then you might only want to sync repositories that were changed via GitLab diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 2d53a790428..5104b65c86d 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -33,7 +33,7 @@ will _not_ carry over automatically, due to differences between the two applicat deployments, see [Configuring Puma Settings](https://docs.gitlab.com/omnibus/settings/puma.html#configuring-puma-settings). For Helm based deployments, see the [Webservice Chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). -Additionally we strongly recommend that multi-node deployments [configure their load balancers to utilize the readiness check](../load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. +Additionally we strongly recommend that multi-node deployments [configure their load balancers to use the readiness check](../load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. ## Performance caveat when using Puma with Rugged diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index 9db9a885baf..dac36135a8e 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # The Rails Console The [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index d99468411e3..0de8b681dd8 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 56b7f01e1ad..541bd99084c 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -170,7 +170,7 @@ If your certificate provider provides the CA Bundle certificates, append them to 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). -Users should now be able to login to the Container Registry with their GitLab +Users should now be able to sign in to the Container Registry with their GitLab credentials using: ```shell @@ -234,7 +234,7 @@ registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/certificate.key" 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). -Users should now be able to login to the Container Registry using their GitLab +Users should now be able to sign in to the Container Registry using their GitLab credentials: ```shell @@ -397,6 +397,20 @@ To configure the `s3` storage driver in Omnibus: } ``` + To avoid using static credentials, use an + [IAM role](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) + and omit `accesskey` and `secretkey`. Make sure that your IAM profile follows + [the permissions documented by Docker](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). + + ```ruby + registry['storage'] = { + 's3' => { + 'bucket' => 'your-s3-bucket', + 'region' => 'your-s3-region' + } + } + ``` + - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO. It takes a URL such as `http://127.0.0.1:9000`. - `your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. @@ -412,8 +426,8 @@ when you [deployed your Docker registry](https://docs.docker.com/registry/deploy ```yaml storage: s3: - accesskey: 's3-access-key' - secretkey: 's3-secret-key-for-access-key' + accesskey: 's3-access-key' # Not needed if IAM role used + secretkey: 's3-secret-key-for-access-key' # Not needed if IAM role used bucket: 'your-s3-bucket' region: 'your-s3-region' regionendpoint: 'your-s3-regionendpoint' @@ -471,7 +485,7 @@ you can pull from the Container Registry, but you cannot push. [`--dryrun`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) flag and run the command. - DANGER: **Danger:** + DANGER: **Warning:** The [`--delete`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) flag deletes files that exist in the destination but not in the source. If you swap the source and destination, all data in the Registry is deleted. @@ -584,7 +598,7 @@ on how to achieve that. ## Use an external container registry with GitLab as an auth endpoint If you use an external container registry, some features associated with the -container registry may be unavailable or have [inherent risks](./../../user/packages/container_registry/index.md#use-with-external-container-registries). +container registry may be unavailable or have [inherent risks](../../user/packages/container_registry/index.md#use-with-external-container-registries). **Omnibus GitLab** @@ -815,23 +829,23 @@ If you changed the location of the Container Registry `config.yml`: sudo gitlab-ctl registry-garbage-collect /path/to/config.yml ``` -You may also [remove all unreferenced manifests](#removing-unused-layers-not-referenced-by-manifests), +You may also [remove all untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers), although this is a way more destructive operation, and you should first understand the implications. -### Removing unused layers not referenced by manifests +### Removing untagged manifests and unreferenced layers > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3097) in Omnibus GitLab 11.10. -DANGER: **Danger:** +DANGER: **Warning:** This is a destructive operation. The GitLab Container Registry follows the same default workflow as Docker Distribution: -retain all layers, even ones that are unreferenced directly to allow all content -to be accessed using context addressable identifiers. +retain untagged manifests and all layers, even ones that are not referenced directly. All content +can be accessed by using context addressable identifiers. -However, in most workflows, you don't care about old layers if they are not directly -referenced by the registry tag. The `registry-garbage-collect` command supports the +However, in most workflows, you don't care about untagged manifests and old layers if they are not directly +referenced by a tagged manifest. The `registry-garbage-collect` command supports the `-m` switch to allow you to remove all unreferenced manifests and layers that are not directly accessible via `tag`: @@ -839,6 +853,8 @@ not directly accessible via `tag`: sudo gitlab-ctl registry-garbage-collect -m ``` +Without the `-m` flag, the Container Registry only removes layers that are not referenced by any manifest, tagged or not. + Since this is a way more destructive operation, this behavior is disabled by default. You are likely expecting this way of operation, but before doing that, ensure that you have backed up all registry data. @@ -932,6 +948,8 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 5 4 * * 0 root gitlab-ctl registry-garbage-collect ``` +You may want to add the `-m` flag to [remove untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers). + ## Troubleshooting Before diving in to the following sections, here's some basic troubleshooting: @@ -1234,8 +1252,8 @@ This will launch the Docker daemon and proxy all connections through mitmproxy. #### Running the Docker client -Now that we have mitmproxy and Docker running, we can attempt to login and push -a container image. You may need to run as root to do this. For example: +Now that we have mitmproxy and Docker running, we can attempt to sign in and +push a container image. You may need to run as root to do this. For example: ```shell docker login s3-testing.myregistry.com:5050 diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index fba3d51f741..56b39658dc2 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -4,11 +4,12 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab Dependency Proxy administration **(PREMIUM ONLY)** +# GitLab Dependency Proxy administration -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) to [GitLab Core](https://about.gitlab.com/pricing/) in GitLab 13.6. -GitLab can be utilized as a dependency proxy for a variety of common package managers. +GitLab can be used as a dependency proxy for a variety of common package managers. This is the administration documentation. If you want to learn how to use the dependency proxies, see the [user guide](../../user/packages/dependency_proxy/index.md). diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 0b3a7ae9fa5..4af0de864ca 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -6,26 +6,50 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Package Registry administration -GitLab Packages allows organizations to utilize GitLab as a private repository +GitLab Packages allows organizations to use GitLab as a private repository for a variety of common package managers. Users are able to build and publish packages, which can be easily consumed as a dependency in downstream projects. The Packages feature allows GitLab to act as a repository for the following: -| Software repository | Description | Available in GitLab version | -| ------------------- | ----------- | --------------------------- | -| [PyPI Repository](../../user/packages/pypi_repository/index.md) | The GitLab PyPI Repository enables every project in GitLab to have its own space to store [PyPI](https://pypi.org/) packages. | 12.10+ | -| [Composer Repository](../../user/packages/composer_repository/index.md) | The GitLab Composer Repository enables every project in GitLab to have its own space to store [Composer](https://getcomposer.org/) packages. | 13.1+ | -| [NuGet Repository](../../user/packages/nuget_repository/index.md) | The GitLab NuGet Repository enables every project in GitLab to have its own space to store [NuGet](https://www.nuget.org/) packages. | 12.8+ | -| [Conan Repository](../../user/packages/conan_repository/index.md) | The GitLab Conan Repository enables every project in GitLab to have its own space to store [Conan](https://conan.io/) packages. | 12.4+ | -| [Maven Repository](../../user/packages/maven_repository/index.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | -| [NPM Registry](../../user/packages/npm_registry/index.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | -| [Go Proxy](../../user/packages/go_proxy/index.md) | The Go proxy for GitLab enables every project in GitLab to be fetched with the [Go proxy protocol](https://proxy.golang.org/). | 13.1+ | -| [Generic packages](../../user/packages/generic_packages/index.md) | Store arbitrary files, like release binaries. | 13.5+ | - -Don't you see your package management system supported yet? -Please consider contributing -to GitLab. This [development documentation](../../development/packages.md) will guide you through the process. +The Package Registry supports the following formats: + +<div class="row"> +<div class="col-md-9"> +<table align="left" style="width:50%"> +<tr style="background:#dfdfdf"><th>Package type</th><th>GitLab version</th></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/composer_repository/index.html">Composer</a></td><td>13.2+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/conan_repository/index.html">Conan</a></td><td>12.6+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/go_proxy/index.html">Go</a></td><td>13.1+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/maven_repository/index.html">Maven</a></td><td>11.3+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/npm_registry/index.html">NPM</a></td><td>11.7+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/nuget_repository/index.html">NuGet</a></td><td>12.8+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/pypi_repository/index.html">PyPI</a></td><td>12.10+</td></tr> +<tr><td><a href="https://docs.gitlab.com/ee/user/packages/generic_packages/index.html">Generic packages</a></td><td>13.5+</td></tr> +</table> +</div> +</div> + +## Accepting contributions + +The below table lists formats that are not supported, but are accepting Community contributions for. Consider contributing to GitLab. This [development documentation](../../development/packages.md) will +guide you through the process. + +| Format | Status | +| ------ | ------ | +| Chef | [#36889](https://gitlab.com/gitlab-org/gitlab/-/issues/36889) | +| CocoaPods | [#36890](https://gitlab.com/gitlab-org/gitlab/-/issues/36890) | +| Conda | [#36891](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | +| CRAN | [#36892](https://gitlab.com/gitlab-org/gitlab/-/issues/36892) | +| Debian | [WIP: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44746) | +| Opkg | [#36894](https://gitlab.com/gitlab-org/gitlab/-/issues/36894) | +| P2 | [#36895](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | +| Puppet | [#36897](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | +| RPM | [#5932](https://gitlab.com/gitlab-org/gitlab/-/issues/5932) | +| RubyGems | [#803](https://gitlab.com/gitlab-org/gitlab/-/issues/803) | +| SBT | [#36898](https://gitlab.com/gitlab-org/gitlab/-/issues/36898) | +| Terraform | [WIP: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18834) | +| Vagrant | [#36899](https://gitlab.com/gitlab-org/gitlab/-/issues/36899) | ## Enabling the Packages feature @@ -58,6 +82,18 @@ To enable the Packages feature: 1. [Restart GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +**Helm Chart installations** + +1. After the installation is complete, you will have to configure the `packages` + section in `global.appConfig.packages`. Set to `true` to enable it: + + ```yaml + packages: + enabled: true + ``` + +1. [Restart GitLab](../restart_gitlab.md#helm-chart-installations "How to reconfigure Helm GitLab") for the changes to take effect. + ## Changing the storage path By default, the packages are stored locally, but you can change the default diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 9f72293a730..0ebeb96d247 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -415,9 +415,6 @@ internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: ### Using a custom Certificate Authority (CA) -NOTE: **Note:** -[Before 13.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4411), when using Omnibus, a [workaround was required](https://docs.gitlab.com/13.1/ee/administration/pages/index.html#using-a-custom-certificate-authority-ca). - When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md#gitlab-pages-access-control) and the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#browsing-artifacts) will fail to work if the custom CA is not recognized. @@ -462,6 +459,34 @@ are stored. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +Alternatively, if you have existing Pages deployed you can follow +the below steps to do a no downtime transfer to a new storage location. + +1. Pause Pages deployments by setting the following in `/etc/gitlab/gitlab.rb`: + + ```ruby + sidekiq['experimental_queue_selector'] = true + sidekiq['queue_groups'] = [ + "feature_category!=pages" + ] + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. `rsync` contents from the current storage location to the new storage location: `sudo rsync -avzh --progress /var/opt/gitlab/gitlab-rails/shared/pages/ /mnt/storage/pages` +1. Set the new storage location in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['pages_path'] = "/mnt/storage/pages" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Verify Pages are still being served up as expected. +1. Unpause Pages deployments by removing from `/etc/gitlab/gitlab.rb` the `sidekiq` setting set above. +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Trigger a new Pages deployment and verify it's working as expected. +1. Remove the old Pages storage location: `sudo rm -rf /var/opt/gitlab/gitlab-rails/shared/pages` +1. Verify Pages are still being served up as expected. + ## Configure listener for reverse proxy requests Follow the steps below to configure the proxy listener of GitLab Pages. [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in @@ -514,7 +539,7 @@ your main application server. To configure GitLab Pages on a separate server: -DANGER: **Danger:** +DANGER: **Warning:** The following procedure includes steps to back up and edit the `gitlab-secrets.json` file. This file contains secrets that control database encryption. Proceed with caution. @@ -528,7 +553,6 @@ database encryption. Proceed with caution. 1. On the **GitLab server**, to enable Pages, add the following to `/etc/gitlab/gitlab.rb`: ```ruby - gitlab_pages['enable'] = true pages_external_url "http://<pages_server_URL>" ``` @@ -559,18 +583,11 @@ database encryption. Proceed with caution. to include: ```ruby - external_url 'http://<gitlab_server_IP_or_URL>' + roles ['pages_role'] + pages_external_url "http://<pages_server_URL>" - postgresql['enable'] = false - redis['enable'] = false - prometheus['enable'] = false - puma['enable'] = false - sidekiq['enable'] = false - gitlab_workhorse['enable'] = false - gitaly['enable'] = false - alertmanager['enable'] = false - node_exporter['enable'] = false - gitlab_rails['auto_migrate'] = false + + gitlab_pages['gitlab_server'] = 'http://<gitlab_server_IP_or_URL>' ``` 1. Create a backup of the secrets file on the **Pages server**: diff --git a/doc/administration/polling.md b/doc/administration/polling.md index a1077614677..17b8045b51f 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Polling configuration The GitLab UI polls for updates for different resources (issue notes, issue diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 0a40b61fd3c..a29815672e5 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -579,18 +579,19 @@ in the Troubleshooting section before proceeding. ### Ensure GitLab is running -At this point, your GitLab instance should be up and running. Verify you are -able to login, and create issues and merge requests. If you have troubles check +At this point, your GitLab instance should be up and running. Verify you're able +to sign in, and create issues and merge requests. If you encounter issues, see the [Troubleshooting section](#troubleshooting). ## Example configuration -Here we'll show you some fully expanded example configurations. +This section describes several fully expanded example configurations. ### Example recommended setup -This example uses 3 Consul servers, 3 PgBouncer servers (with associated internal load balancer), -3 PostgreSQL servers, and 1 application node. +This example uses three Consul servers, three PgBouncer servers (with an +associated internal load balancer), three PostgreSQL servers, and one +application node. We start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 22543a5c743..41a7ec087ac 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Pseudonymizer **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.1. @@ -44,7 +50,6 @@ To configure the pseudonymizer, you need to: } ``` - NOTE: **Note:** If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. ```ruby diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index bdccd6d5fe9..f61a3107b3d 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Integrity check Rake task **(CORE ONLY)** GitLab provides Rake tasks to check the integrity of various components. diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index c97aa5a4de1..e6a6751f199 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Doctor Rake tasks **(CORE ONLY)** This is a collection of tasks to help investigate and repair @@ -15,7 +21,6 @@ Automatic resolution is not yet implemented. If you have values that cannot be decrypted, you can follow steps to reset them, see our docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). -NOTE: **Note:** This can take a very long time, depending on the size of your database, as it checks all rows in all tables. diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 885d19903ed..0ee6fd8fc75 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Geo Rake Tasks **(PREMIUM ONLY)** The following Rake tasks are for [Geo installations](../geo/index.md). diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 7f673a2c850..d549110c32f 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitHub import **(CORE ONLY)** > [Introduced]( https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10308) in GitLab 9.1. diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 6d04a786d3a..821a72291fd 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # LDAP Rake tasks **(CORE ONLY)** The following are LDAP-related Rake tasks. diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 553afba78e3..b93442be0a1 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Maintenance Rake tasks **(CORE ONLY)** GitLab provides Rake tasks for general maintenance. @@ -124,7 +130,6 @@ sudo gitlab-rake gitlab:check bundle exec rake gitlab:check RAILS_ENV=production ``` -NOTE: **Note:** Use `SANITIZE=true` for `gitlab:check` if you want to omit project names from the output. Example output: @@ -267,7 +272,7 @@ clear it. To clear all exclusive leases: -DANGER: **Danger:** +DANGER: **Warning:** Don't run it while GitLab or Sidekiq is running ```shell diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 51e0e2e46b6..d83ab6e5cee 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Project import/export administration **(CORE ONLY)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3050) in GitLab 8.9. diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 896eafeb5f3..9b15f5ed4de 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Repository storage Rake tasks **(CORE ONLY)** This is a collection of Rake tasks to help you list and migrate @@ -68,7 +74,7 @@ To have a summary and then a list of projects and their attachments using hashed ## Migrate to hashed storage -NOTE: **Note:** +DANGER: **Deprecated:** In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) is enabled by default and the legacy storage is deprecated. Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab @@ -109,7 +115,6 @@ If you find it necessary, you can run this migration script again to schedule mi Any error or warning will be logged in Sidekiq's log file. -NOTE: **Note:** If [Geo](../geo/index.md) is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes. @@ -118,7 +123,7 @@ commands below that helps you inspect projects and attachments in both legacy an ## Rollback from hashed storage to legacy storage -NOTE: **Deprecated:** +DANGER: **Deprecated:** In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) is enabled by default and the legacy storage is deprecated. Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index 8d61beea597..075b27fb70d 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Uploads migrate Rake tasks **(CORE ONLY)** There is a Rake task for migrating uploads between different storage types. @@ -10,11 +16,10 @@ There is a Rake task for migrating uploads between different storage types. After [configuring the object storage](../../uploads.md#using-object-storage) for GitLab's uploads, use this task to migrate existing uploads from the local storage to the remote storage. -Read more about using [object storage with GitLab](../../object_storage.md). - -NOTE: **Note:** All of the processing will be done in a background worker and requires **no downtime**. +Read more about using [object storage with GitLab](../../object_storage.md). + ### All-in-one Rake task GitLab provides a wrapper Rake task that migrates all uploaded files (for example avatars, logos, @@ -93,7 +98,6 @@ gitlab-rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, De **Source Installation** -NOTE: **Note:** Use `RAILS_ENV=production` for every task. ```shell diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 54aa62059cf..4f7c99babd8 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Uploads sanitize Rake tasks **(CORE ONLY)** In GitLab 11.9 and later, EXIF data is automatically stripped from JPG or TIFF image uploads. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 7910905ce54..9f522e0d599 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -836,6 +836,12 @@ a node and change its status from primary to replica (and vice versa). # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + # Set the Redis Cache instance as an LRU + # 90% of available RAM in MB + redis['maxmemory'] = '13500mb' + redis['maxmemory_policy'] = "allkeys-lru" + redis['maxmemory_samples'] = 5 + ## Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -897,6 +903,12 @@ You can specify multiple roles, like sentinel and Redis, as: # to `6379`. #redis['master_port'] = 6379 + # Set the Redis Cache instance as an LRU + # 90% of available RAM in MB + redis['maxmemory'] = '13500mb' + redis['maxmemory_policy'] = "allkeys-lru" + redis['maxmemory_samples'] = 5 + ## Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1335,6 +1347,13 @@ To configure the Sentinel Queues server: ## Configure Gitaly +NOTE: **Note:** +[Gitaly Cluster](../gitaly/praefect.md) support +for the Reference Architectures is being +worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified +some Architecture specs will likely change as a result to support the new +and improved designed. + [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended that a Gitaly server node stores no more than 5 TB of data. Although this @@ -1988,7 +2007,7 @@ on what features you intend to use: | [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | | [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index bb6e2eb4376..b106f7bced1 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -836,6 +836,12 @@ a node and change its status from primary to replica (and vice versa). # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + # Set the Redis Cache instance as an LRU + # 90% of available RAM in MB + redis['maxmemory'] = '13500mb' + redis['maxmemory_policy'] = "allkeys-lru" + redis['maxmemory_samples'] = 5 + ## Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -897,6 +903,12 @@ You can specify multiple roles, like sentinel and Redis, as: # to `6379`. #redis['master_port'] = 6379 + # Set the Redis Cache instance as an LRU + # 90% of available RAM in MB + redis['maxmemory'] = '13500mb' + redis['maxmemory_policy'] = "allkeys-lru" + redis['maxmemory_samples'] = 5 + ## Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1225,7 +1237,7 @@ To configure the Sentinel Queues server: 1. SSH in to the server that will host Sentinel. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus package, with the same version + on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: @@ -1335,6 +1347,13 @@ To configure the Sentinel Queues server: ## Configure Gitaly +NOTE: **Note:** +[Gitaly Cluster](../gitaly/praefect.md) support +for the Reference Architectures is being +worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified +some Architecture specs will likely change as a result to support the new +and improved designed. + [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended that a Gitaly server node stores no more than 5 TB of data. Although this @@ -1988,7 +2007,7 @@ on what features you intend to use: | [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | | [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 4863329b695..f4842a8568b 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -356,6 +356,13 @@ are supported and can be added if needed. ## Configure Gitaly +NOTE: **Note:** +[Gitaly Cluster](../gitaly/praefect.md) support +for the Reference Architectures is being +worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified +some Architecture specs will likely change as a result to support the new +and improved designed. + [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended that a Gitaly server node stores no more than 5TB of data. Although this @@ -859,7 +866,7 @@ on what features you intend to use: | [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | | [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 70d0cae6dfd..b5b3e4e0300 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -1058,6 +1058,13 @@ Refer to your preferred Load Balancer's documentation for further guidance. ## Configure Gitaly +NOTE: **Note:** +[Gitaly Cluster](../gitaly/praefect.md) support +for the Reference Architectures is being +worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified +some Architecture specs will likely change as a result to support the new +and improved designed. + [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended that a Gitaly server node stores no more than 5 TB of data. Although this @@ -1735,7 +1742,7 @@ on what features you intend to use: | [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | | [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 44fbe2db504..152eb9cb90d 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -836,6 +836,12 @@ a node and change its status from primary to replica (and vice versa). # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + # Set the Redis Cache instance as an LRU + # 90% of available RAM in MB + redis['maxmemory'] = '13500mb' + redis['maxmemory_policy'] = "allkeys-lru" + redis['maxmemory_samples'] = 5 + ## Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -897,6 +903,12 @@ You can specify multiple roles, like sentinel and Redis, as: # to `6379`. #redis['master_port'] = 6379 + # Set the Redis Cache instance as an LRU + # 90% of available RAM in MB + redis['maxmemory'] = '13500mb' + redis['maxmemory_policy'] = "allkeys-lru" + redis['maxmemory_samples'] = 5 + ## Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1335,6 +1347,13 @@ To configure the Sentinel Queues server: ## Configure Gitaly +NOTE: **Note:** +[Gitaly Cluster](../gitaly/praefect.md) support +for the Reference Architectures is being +worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified +some Architecture specs will likely change as a result to support the new +and improved designed. + [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended that a Gitaly server node stores no more than 5 TB of data. Although this @@ -1988,7 +2007,7 @@ on what features you intend to use: | [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | | [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 9f83950a927..f023971bdc0 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -1057,6 +1057,13 @@ Refer to your preferred Load Balancer's documentation for further guidance. ## Configure Gitaly +NOTE: **Note:** +[Gitaly Cluster](../gitaly/praefect.md) support +for the Reference Architectures is being +worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified +some Architecture specs will likely change as a result to support the new +and improved designed. + [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended that a Gitaly server node stores no more than 5 TB of data. Although this @@ -1734,7 +1741,7 @@ on what features you intend to use: | [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | | [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index 62645ad17a1..4108635ba2c 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -46,4 +46,4 @@ If it finds a reply key, it will be able to leave your reply as a comment on the entity the notification was about (issue, merge request, commit...). For more details about the `Message-ID`, `In-Reply-To`, and `References headers`, -please consult [RFC 5322](https://tools.ietf.org/html/rfc5322#section-3.6.4). +see [RFC 5322](https://tools.ietf.org/html/rfc5322#section-3.6.4). diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index f4fcbefa403..aaadeef8bf5 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Set up Postfix for incoming email This document will take you through the steps of setting up a basic Postfix mail @@ -85,9 +91,10 @@ The instructions make the assumption that you will be using the email address `i quit ``` - _**Note:** The `.` is a literal period on its own line._ + NOTE: **Note:** + The `.` is a literal period on its own line. - _**Note:** If you receive an error after entering `rcpt to: incoming@localhost` + If you receive an error after entering `rcpt to: incoming@localhost` then your Postfix `my_network` configuration is not correct. The error will say 'Temporary lookup failure'. See [Configure Postfix to receive email from the Internet](#configure-postfix-to-receive-email-from-the-internet)._ @@ -158,11 +165,11 @@ Courier, which we will install later to add IMAP authentication, requires mailbo q ``` - _**Note:** If `mail` returns an error `Maildir: Is a directory` then your + If `mail` returns an error `Maildir: Is a directory` then your version of `mail` doesn't support Maildir style mailboxes. Install `heirloom-mailx` by running `sudo apt-get install heirloom-mailx`. Then, try the above steps again, substituting `heirloom-mailx` for the `mail` - command._ + command. 1. Sign out of the `incoming` account, and go back to being `root`: @@ -265,7 +272,8 @@ Courier, which we will install later to add IMAP authentication, requires mailbo quit ``` - (Note: The `.` is a literal period on its own line) + NOTE: **Note:** + The `.` is a literal period on its own line. 1. Check if the `incoming` user received the email: diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index b272c4b463e..d2d492c8f1a 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -117,7 +117,7 @@ The output includes the project ID and the project name: > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1606) in GitLab 12.1. -DANGER: **Danger:** +DANGER: **Warning:** Do not run `git prune` or `git gc` in pool repositories! This can cause data loss in "real" repositories that depend on the pool in question. diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 57f53fd6cc4..f6a4a39ad60 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # How to restart GitLab Depending on how you installed GitLab, there are different methods to restart diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index 23e870dbb82..2cc6acc8c87 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -92,7 +95,6 @@ you want using steps 1 and 2 from the GitLab downloads page. 1. Run `gitlab-ctl reconfigure`. -NOTE: **Note:** You will need to restart the Sidekiq nodes after an update has occurred and database migrations performed. @@ -111,7 +113,6 @@ prometheus['enable'] = false gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false -gitlab_monitor['enable'] = false gitlab_workhorse['enable'] = false nginx['enable'] = false postgres_exporter['enable'] = false diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index 6ded7fdd7d0..f5b40210e62 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -1,10 +1,15 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Signing outgoing email with S/MIME Notification emails sent by GitLab can be signed with S/MIME for improved security. -NOTE: **Note:** -Please be aware that S/MIME certificates and TLS/SSL certificates are not the +Be aware that S/MIME certificates and TLS/SSL certificates are not the same and are used for different purposes: TLS creates a secure channel, whereas S/MIME signs and/or encrypts the message itself @@ -21,7 +26,7 @@ files must be provided: Optionally, you can also provide a bundle of CA certs (PEM-encoded) to be included on each signature. This will typically be an intermediate CA. -NOTE: **Note:** +CAUTION: **Caution:** Be mindful of the access levels for your private keys and visibility to third parties. @@ -39,7 +44,6 @@ third parties. 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** The key needs to be readable by the GitLab system user (`git` by default). **For installations from source:** @@ -63,7 +67,6 @@ The key needs to be readable by the GitLab system user (`git` by default). 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -NOTE: **Note:** The key needs to be readable by the GitLab system user (`git` by default). ### How to convert S/MIME PKCS#12 / PFX format to PEM encoding diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index 3cfee8630b7..bec82f66948 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Changing your time zone The global time zone configuration parameter can be changed in `config/gitlab.yml`: @@ -15,7 +21,7 @@ To see all available time zones, run `bundle exec rake time:zones:all`. For Omnibus installations, run `gitlab-rake time:zones:all`. NOTE: **Note:** -Currently, this Rake task does not list timezones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). +This Rake task does not list timezones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). ## Changing time zone in Omnibus installations diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index ef95bdc8602..d67c9963092 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Debugging Tips Sometimes things don't work the way they should. Here are some tips on debugging issues out diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 5912981ba6e..132f8524ca1 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 6de5bb71d75..12aa91e6f14 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Global Search +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Troubleshooting Elasticsearch To install and configure Elasticsearch, and for common and known issues, @@ -165,7 +171,7 @@ The first step is to confirm GitLab is using Elasticsearch for the search functi To do this: 1. Confirm the integration is enabled in **Admin Area > Settings > General**. -1. Confirm searches utilize Elasticsearch by accessing the rails console +1. Confirm searches use Elasticsearch by accessing the rails console (`sudo gitlab-rails console`) and running the following commands: ```rails diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index dbda889d370..cae4aa96f75 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -1,8 +1,11 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- -# GitLab Rails Console Cheat Sheet +# GitLab Rails Console Cheat Sheet **(CORE ONLY)** This is the GitLab Support Team's collection of information regarding the GitLab Rails console, for use while troubleshooting. It is listed here for transparency, @@ -43,6 +46,40 @@ instance_of_object.method(:foo).source_location project.method(:private?).source_location ``` +## Attributes + +View available attributes, formatted using pretty print (`pp`). + +For example, determine what attributes contain users' names and email addresses: + +```ruby +u = User.find_by_username('someuser') +pp u.attributes +``` + +Partial output: + +```plaintext +{"id"=>1234, + "email"=>"someuser@example.com", + "sign_in_count"=>99, + "name"=>"S User", + "username"=>"someuser", + "first_name"=>nil, + "last_name"=>nil, + "bot_type"=>nil} +``` + +Then make use of the attributes, [testing SMTP, for example](https://docs.gitlab.com/omnibus/settings/smtp.html#testing-the-smtp-configuration): + +```ruby +e = u.email +n = u.name +Notify.test_email(e, "Test email for #{n}", 'Test email').deliver_now +# +Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now +``` + ## Query the database using an ActiveRecord Model ```ruby @@ -145,7 +182,7 @@ project.repository.expire_exists_cache Project.update_all(visibility_level: 0) ``` -### Find & remove projects that are pending deletion +### Find projects that are pending deletion ```ruby # @@ -174,8 +211,6 @@ project = Project.find_by_full_path('group-changeme/project-changeme') ::Projects::DestroyService.new(project, user, {}).execute ``` -Next, run `sudo gitlab-rake gitlab:cleanup:repos` on the command line to finish. - ### Destroy a project ```ruby @@ -273,11 +308,11 @@ pp p.statistics # compare with earlier values ### Recreate -A Projects Wiki can be recreated by - -NOTE: **Note:** +CAUTION: **Caution:** This is a destructive operation, the Wiki will be empty. +A Projects Wiki can be recreated by this command: + ```ruby p = Project.find_by_full_path('<username-or-group>/<project-name>') ### enter your projects path @@ -423,7 +458,7 @@ user.skip_reconfirmation! User.active.count # Users taking a seat on the instance -License.current.current_active_users_count +User.billable.count # The historical max on the instance as of the past year ::HistoricalData.max_historical_user_count @@ -484,6 +519,16 @@ user.max_member_access_for_group group.id ## Groups +### Transfer group to another location + +```ruby +user = User.find_by_username('<username>') +group = Group.find_by_name("<group_name>") +parent_group = Group.find_by(id: "") # empty string amounts to root as parent +service = ::Groups::TransferService.new(group, user) +service.execute(parent_group) +``` + ### Count unique users in a group and sub-groups ```ruby @@ -813,7 +858,7 @@ Find this content in the [Container Registry troubleshooting docs](../packages/c ## Sidekiq -This content has been moved to the [Troubleshooting Sidekiq docs](./sidekiq.md). +This content has been moved to the [Troubleshooting Sidekiq docs](sidekiq.md). ## Redis @@ -954,3 +999,19 @@ project = Project.find_by_full_path('<group/project>') Geo::RepositorySyncService.new(project).execute ``` + +### Generate usage ping + +#### Generate or get the cached usage ping + +```ruby +Gitlab::UsageData.to_json +``` + +#### Generate a fresh new usage ping + +This will also refresh the cached usage ping displayed in the admin area + +```ruby +Gitlab::UsageData.to_json(force_refresh: true) +``` diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index 7492688ded0..efc8eaab198 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -1,4 +1,7 @@ --- +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png b/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png Binary files differindex cd1b9db31d9..e24c994e996 100644 --- a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png +++ b/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png diff --git a/doc/administration/troubleshooting/img/OneLogin-encryption.png b/doc/administration/troubleshooting/img/OneLogin-encryption.png Binary files differdeleted file mode 100644 index 2b811409bd0..00000000000 --- a/doc/administration/troubleshooting/img/OneLogin-encryption.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/network_monitor_xid.png b/doc/administration/troubleshooting/img/network_monitor_xid.png Binary files differindex 5392f77327f..7fc2cf47ea0 100644 --- a/doc/administration/troubleshooting/img/network_monitor_xid.png +++ b/doc/administration/troubleshooting/img/network_monitor_xid.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 50f192b1983..8a4a0a4caac 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Troubleshooting a GitLab installation Below are some resources to help you troubleshoot a GitLab installation diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 7c5a9e0d79f..21fd183dfd0 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -67,8 +70,7 @@ and they will assist you with any issues you are having. kubectl logs <pod-name> --previous ``` - NOTE: **Note:** - No logs are kept in the containers/pods themselves, everything is written to stdout. + No logs are kept in the containers/pods themselves. Everything is written to stdout. This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) for details. diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index f24234e1aff..b1042a9402b 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -15,7 +18,6 @@ If you are administering GitLab you are expected to know these commands for your of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like. -Note: **Note:** Most of the commands below have not been labeled as to which distribution they work on. Contributions are welcome to help add them. diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 7914628a756..4e9e8cd591f 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -1,8 +1,14 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Parsing GitLab logs with `jq` We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, but if they are not available you can still quickly parse -[GitLab logs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26311) in JSON format +[GitLab logs](../logs.md) in JSON format (the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). ## What is JQ? diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index a1485249b0e..475f3d56836 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Navigating GitLab via Rails console At the heart of GitLab is a web application [built using the Ruby on Rails @@ -379,7 +385,7 @@ User.find_by(username: 'root') User.find_by_any_email('user@example.com') ``` -Note: `find_by_any_email` is a custom method added by GitLab developers rather +The `find_by_any_email` method is a custom method added by GitLab developers rather than a Rails-provided default method. **Get a collection of admin users:** @@ -388,7 +394,7 @@ than a Rails-provided default method. User.admins ``` -Note: `admins` is a [scope convenience method](https://guides.rubyonrails.org/active_record_querying.html#scopes) +`admins` is a [scope convenience method](https://guides.rubyonrails.org/active_record_querying.html#scopes) which does `where(admin: true)` under the hood. **Get a project by its path:** @@ -397,7 +403,7 @@ which does `where(admin: true)` under the hood. Project.find_by_full_path('group/subgroup/project') ``` -Note: `find_by_full_path` is a custom method added by GitLab developers rather +`find_by_full_path` is a custom method added by GitLab developers rather than a Rails-provided default method. **Get a project's issue or merge request by its numeric ID:** @@ -408,7 +414,7 @@ project.issues.find_by(iid: 42) project.merge_requests.find_by(iid: 42) ``` -Note: `iid` means "internal ID" and is how we keep issue and merge request IDs +`iid` means "internal ID" and is how we keep issue and merge request IDs scoped to each GitLab project. **Get a group by its path:** @@ -448,7 +454,7 @@ Ci::Pipeline.find(4151) Ci::Build.find(66124) ``` -Note: The pipeline and job #ID numbers increment globally across your GitLab +The pipeline and job ID numbers increment globally across your GitLab instance, so there's no need to use an internal ID attribute to look them up, unlike with issues or merge requests. diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 91ff6f6524a..d22e76a505a 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -145,4 +148,4 @@ It may take a little while to respond. ``` NOTE: **Note:** -These are Omnibus settings. If an external database, such as a customer's PostgreSQL installation or Amazon RDS is being used, these values don't get set, and would have to be set externally. +These are Omnibus GitLab settings. If an external database, such as a customer's PostgreSQL installation or Amazon RDS is being used, these values don't get set, and would have to be set externally. diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index b7762f8ac3e..d415aa0d980 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Troubleshooting Sidekiq Sidekiq is the background job processor GitLab uses to asynchronously run @@ -7,12 +13,10 @@ may be filling up. Users will notice when this happens because new branches may not show up and merge requests may not be updated. The following are some troubleshooting steps that will help you diagnose the bottleneck. -NOTE: **Note:** GitLab administrators/users should consider working through these debug steps with GitLab Support so the backtraces can be analyzed by our team. It may reveal a bug or necessary improvement in GitLab. -NOTE: **Note:** In any of the backtraces, be wary of suspecting cases where every thread appears to be waiting in the database, Redis, or waiting to acquire a mutex. This **may** mean there's contention in the database, for example, @@ -22,19 +26,11 @@ preventing other threads from continuing. ## Log arguments to Sidekiq jobs -If you want to see what arguments are being passed to Sidekiq jobs you can set -the `SIDEKIQ_LOG_ARGUMENTS` [environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html) to `1` (true). - -Example: - -```ruby -gitlab_rails['env'] = {"SIDEKIQ_LOG_ARGUMENTS" => "1"} -``` - -This does not log all job arguments. To avoid logging sensitive -information (for instance, password reset tokens), it logs numeric -arguments for all workers, with overrides for some specific workers -where their arguments are not sensitive. +[In GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44853) +some arguments passed to Sidekiq jobs are logged by default. +To avoid logging sensitive information (for instance, password reset tokens), +GitLab logs numeric arguments for all workers, with overrides for some specific +workers where their arguments are not sensitive. Example log output: @@ -49,6 +45,17 @@ arguments logs are limited to a maximum size of 10 kilobytes of text; any arguments after this limit will be discarded and replaced with a single argument containing the string `"..."`. +You can set `SIDEKIQ_LOG_ARGUMENTS` [environment variable](https://docs.gitlab.com/omnibus/settings/environment-variables.html) +to `0` (false) to disable argument logging. + +Example: + +```ruby +gitlab_rails['env'] = {"SIDEKIQ_LOG_ARGUMENTS" => "0"} +``` + +In GitLab 13.5 and earlier, set `SIDEKIQ_LOG_ARGUMENTS` to `1` to start logging arguments passed to Sidekiq. + ## Thread dump Send the Sidekiq process ID the `TTIN` signal and it will output thread @@ -127,7 +134,6 @@ corresponding Ruby code where this is happening. `gdb` can be another effective tool for debugging Sidekiq. It gives you a little more interactive way to look at each thread and see what's causing problems. -NOTE: **Note:** Attaching to a process with `gdb` will suspends the normal operation of the process (Sidekiq will not process jobs while `gdb` is attached). @@ -278,15 +284,15 @@ end ### Remove Sidekiq jobs for given parameters (destructive) -The general method to kill jobs conditionally is the following: +The general method to kill jobs conditionally is the following command, which +will remove jobs that are queued but not started. Running jobs will not be killed. ```ruby queue = Sidekiq::Queue.new('<queue name>') queue.each { |job| job.delete if <condition>} ``` -NOTE: **Note:** -This will remove jobs that are queued but not started, running jobs will not be killed. Have a look at the section below for cancelling running jobs. +Have a look at the section below for cancelling running jobs. In the method above, `<queue-name>` is the name of the queue that contains the job(s) you want to delete and `<condition>` will decide which jobs get deleted. @@ -294,7 +300,6 @@ Commonly, `<condition>` references the job arguments, which depend on the type o For example, `repository_import` has `project_id` as the job argument, while `update_merge_requests` has `project_id, user_id, oldrev, newrev, ref`. -NOTE: **Note:** Arguments need to be referenced by their sequence ID using `job.args[<id>]` because `job.args` is a list of all arguments provided to the Sidekiq job. Here are some examples: diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 4d4b9755fa9..996856521b7 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 80ccd15aa22..9855a27ca30 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index ae9ebd90951..8840c2706d6 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 91de089c45e..cd15301edf6 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Uploads administration **(CORE ONLY)** Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled. @@ -16,7 +22,7 @@ were before. This change is deployed behind a feature flag that is **enabled by default**. If you experience any issues with upload, -[GitLab administrators with access to the GitLab Rails console](./feature_flags.md) +[GitLab administrators with access to the GitLab Rails console](feature_flags.md) can opt to disable it. To enable it: @@ -33,16 +39,15 @@ Feature.disable(:upload_middleware_jwt_params_handler) ## Using local storage -NOTE: **Note:** -This is the default configuration +This is the default configuration. To change the location where the uploads are +stored locally, use the steps in this section based on your installation method: -To change the location where the uploads are stored locally, follow the steps -below. - -**In Omnibus installations:** +**In Omnibus GitLab installations:** NOTE: **Note:** -For historical reasons, uploads are stored into a base directory, which by default is `uploads/-/system`. It is strongly discouraged to change this configuration option on an existing GitLab installation. +For historical reasons, uploads are stored into a base directory, which by +default is `uploads/-/system`. It's strongly discouraged to change this +configuration option for an existing GitLab installation. _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ @@ -86,7 +91,6 @@ This configuration relies on valid AWS credentials to be configured already. [Read more about using object storage with GitLab](object_storage.md). -NOTE: **Note:** We recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original configuration format. ## Object Storage Settings @@ -125,7 +129,6 @@ _The uploads are stored by default in } ``` - NOTE: **Note:** If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. ```ruby diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index 7a3e1d86712..94ce1debfea 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Modifying global user settings GitLab administrators can modify user settings for the entire GitLab instance. |
