diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-07-19 17:16:28 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-07-19 17:16:28 +0300 |
commit | e4384360a16dd9a19d4d2d25d0ef1f2b862ed2a6 (patch) | |
tree | 2fcdfa7dcdb9db8f5208b2562f4b4e803d671243 /doc/administration | |
parent | ffda4e7bcac36987f936b4ba515995a6698698f0 (diff) |
Add latest changes from gitlab-org/gitlab@16-2-stable-eev16.2.0-rc42
Diffstat (limited to 'doc/administration')
212 files changed, 10779 insertions, 1362 deletions
diff --git a/doc/administration/admin_area.md b/doc/administration/admin_area.md new file mode 100644 index 00000000000..1e103bb55c8 --- /dev/null +++ b/doc/administration/admin_area.md @@ -0,0 +1,489 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# GitLab Admin Area **(FREE SELF)** + +The Admin Area provides a web UI to manage and configure features of GitLab +self-managed instances. If you are an administrator,to access the Admin Area: + +- In GitLab 16.1 and later: on the left sidebar, expand the top-most chevron (**{chevron-down}**), then select **Admin Area**. +- In GitLab 16.0 and earlier: on the top bar, select **Main menu > Admin**. + +NOTE: +Only administrators can access the Admin Area. + +## Administering projects + +You can administer all projects in the GitLab instance from the Admin Area's Projects page. + +To access the Projects page: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Projects**. +1. Select the **All**, **Private**, **Internal**, or **Public** tab to list only + projects of that criteria. + +By default, all projects are listed, in reverse order of when they were last updated. For each +project, the following information is listed: + +- Name +- Namespace +- Description +- Size, updated every 15 minutes at most + +Projects can be edited or deleted. + +To edit a project's name or description: + +1. In the Projects overview, next to the project you want to edit, select **Edit**. +1. Edit the **Project name** or **Project description**. +1. Select **Save Changes**. + +To delete a project: + +1. In the Projects overview, next to the project you want to delete, select **Delete**. + +The list of projects can be sorted by: + +- Updated date +- Last created +- Name +- Most stars +- Oldest created +- Oldest updated +- Largest repository + +A user can choose to hide or show archived projects in the list. + +In the **Filter by name** field, type the project name you want to find, and GitLab filters +them as you type. + +To filter only projects in that namespace, select from the **Namespace** dropdown list. + +You can combine the filter options. For example, to list only public projects with `score` in their name: + +1. Select the **Public** tab. +1. Enter `score` in the **Filter by name...** input box. + +## Administering users + +You can administer all users in the GitLab instance from the Admin Area's Users page: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. + +To list users matching a specific criteria, select one of the following tabs on the **Users** page: + +- **Active** +- **Admins** +- **2FA Enabled** +- **2FA Disabled** +- **External** +- **[Blocked](../administration/moderate_users.md#block-a-user)** +- **[Deactivated](../administration/moderate_users.md#deactivate-a-user)** +- **Without projects** + +For each user, the following are listed: + +1. Username +1. Email address +1. Project membership count +1. Group membership count ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276215) in GitLab 13.12) +1. Date of account creation +1. Date of last activity + +To edit a user, in the user's row, select **Edit**. To delete the user, or delete the user and their contributions, select the cog dropdown list in +that user's row, and select the desired option. + +To change the sort order: + +1. Select the sort dropdown list. +1. Select the desired order. + +By default the sort dropdown list shows **Name**. + +To search for users, enter your criteria in the search field. The user search is case +insensitive, and applies partial matching to name and username. To search for an email address, +you must provide the complete email address. + +### User impersonation + +An administrator can "impersonate" any other user, including other administrators. +This allows the administrator to "see what the user sees," and take actions on behalf of the user. +You can impersonate a user in the following ways: + +- Through the UI: + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. + 1. On the left sidebar, select **Overview > Users**. + 1. From the list of users, select a user. + 1. Select **Impersonate**. +- With the API, using [impersonation tokens](../api/rest/index.md#impersonation-tokens). + +All impersonation activities are [captured with audit events](audit_events.md#user-impersonation). + +By default, impersonation is enabled. GitLab can be configured to [disable impersonation](../api/rest/index.md#disable-impersonation). + +![user impersonation button](img/impersonate_user_button_v13_8.png) + +### User identities + +> The ability to see a user's SCIM identity was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294608) in GitLab 15.3. + +When using authentication providers, administrators can see the identities for a user: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. From the list of users, select a user. +1. Select **Identities**. + +This list shows the user's identities, including SCIM identities. Administrators can use this information to troubleshoot SCIM-related issues and confirm +the identities being used for an account. + +### User Permission Export **(PREMIUM SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1772) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292436) in GitLab 13.9. + +An administrator can export user permissions for all users in the GitLab instance from the Admin Area's Users page. +The export lists direct membership the users have in groups and projects. + +The following data is included in the export: + +- Username +- Email +- Type +- Path +- Access level ([Project](../user/permissions.md#project-members-permissions) and [Group](../user/permissions.md#group-members-permissions)) +- Date of last activity ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345388) in GitLab 14.6). For a list of activities that populate this column, see the [Users API documentation](../api/users.md#get-user-activities). + +Only the first 100,000 user accounts are exported. + +![user permission export button](img/export_permissions_v13_11.png) + +### Users statistics + +The **Users statistics** page provides an overview of user accounts by role. These statistics are +calculated daily, so user changes made since the last update are not reflected. + +The following totals are also included: + +- Billable users +- Blocked users +- Total users + +GitLab billing is based on the number of [**Billable users**](../subscriptions/self_managed/index.md#billable-users). + +### Add email to user + +You must be an administrator to manually add emails to users: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Locate the user and select them. +1. Select **Edit**. +1. In **Email**, enter the new email address. This adds the new email address to the + user and sets the previous email address to be a secondary. +1. Select **Save changes**. + +## User cohorts + +The [Cohorts](user_cohorts.md) tab displays the monthly cohorts of new users and their activities over time. + +## Prevent a user from creating groups + +By default, users can create groups. To prevent a user from creating a top level group: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Locate the user and select them. +1. Select **Edit**. +1. Clear the **Can create group** checkbox. +1. Select **Save changes**. + +It is also possible to [limit which roles can create a subgroup within a group](../user/group/subgroups/index.md#change-who-can-create-subgroups). + +## Administering groups + +You can administer all groups in the GitLab instance from the Admin Area's Groups page. + +To access the Groups page: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Groups**. + +For each group, the page displays their name, description, size, number of projects in the group, +number of members, and whether the group is private, internal, or public. To edit a group, in the group's row, select **Edit**. To delete the group, in the group's row, select **Delete**. + +To change the sort order, select the sort dropdown list and select the desired order. The default +sort order is by **Last created**. + +To search for groups by name, enter your criteria in the search field. The group search is case +insensitive, and applies partial matching. + +To [Create a new group](../user/group/index.md#create-a-group) select **New group**. + +## Administering topics + +- > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340920) in GitLab 14.4. +- > Merging topics [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366884) in GitLab 15.5. + +[Topics](../user/project/working_with_projects.md#explore-topics) are used to categorize and find similar projects. + +### View all topics + +To view all topics in the GitLab instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Topics**. + +For each topic, the page displays its name and the number of projects labeled with the topic. + +### Search for topics + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Topics**. +1. In the search box, enter your search criteria. + The topic search is case-insensitive and applies partial matching. + +### Create a topic + +To create a topic: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Topics**. +1. Select **New topic**. +1. Enter the **Topic slug (name)** and **Topic title**. +1. Optional. Enter a **Description** and add a **Topic avatar**. +1. Select **Save changes**. + +The created topics are displayed on the **Explore topics** page. + +NOTE: +The assigned topics are visible only to everyone with access to the project, +but everyone can see which topics exist on the GitLab instance. +Do not include sensitive information in the name of a topic. + +### Edit a topic + +You can edit a topic's name, title, description, and avatar at any time. +To edit a topic: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Topics**. +1. Select **Edit** in that topic's row. +1. Edit the topic slug (name), title, description, or avatar. +1. Select **Save changes**. + +### Remove a topic + +If you no longer need a topic, you can permanently remove it. +To remove a topic: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Topics**. +1. To remove a topic, select **Remove** in that topic's row. + +### Merge topics + +You can move all projects assigned to a topic to another topic. +The source topic is then permanently deleted. +After a merged topic is deleted, you cannot restore it. + +To merge topics: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Topics**. +1. Select **Merge topics**. +1. From the **Source topic** dropdown list, select the topic you want to merge and remove. +1. From the **Target topic** dropdown list, select the topic you want to merge the source topic into. +1. Select **Merge**. + +## Administering Gitaly servers + +You can list all Gitaly servers in the GitLab instance from the Admin Area's **Gitaly Servers** +page. For more details, see [Gitaly](gitaly/index.md). + +To access the **Gitaly Servers** page: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Gitaly Servers**. + +For each Gitaly server, the following details are listed: + +| Field | Description | +|----------------|-------------| +| Storage | Repository storage | +| Address | Network address on which the Gitaly server is listening | +| Server version | Gitaly version | +| Git version | Version of Git installed on the Gitaly server | +| Up to date | Indicates if the Gitaly server version is the latest version available. A green dot indicates the server is up to date. | + +## CI/CD section + +### Administering runners + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/340859) from **Overview > Runners** to **CI/CD > Runners** in GitLab 15.8. + +You can administer all runners in the GitLab instance from the Admin Area's **Runners** page. See +[GitLab Runner](https://docs.gitlab.com/runner/) for more information. + +To access the **Runners** page: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Runners**. + +#### Search and filter runners + +To search runners' descriptions: + +1. In the **Search or filter results...** field, type the description of the runner you want to + find. +1. Press <kbd>Enter</kbd>. + +You can also filter runners by status, type, and tag. To filter: + +1. Select a tab or the **Search or filter results...** field. +1. Select any **Type**, or filter by **Status** or **Tags**. +1. Select or enter your search criteria. + +![Attributes of a runner, with the **Search or filter results...** field active](img/index_runners_search_or_filter_v14_5.png) + +#### Bulk delete runners + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/370241) in GitLab 15.4. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/353981) in GitLab 15.5. + +You can delete multiple runners at the same time. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Runners**. +1. To the left of the runners you want to delete, select the checkbox. + To select all of the runners on the page, select the checkbox above + the list. +1. Select **Delete selected**. + +#### Runner attributes + +For each runner, the following attributes are listed: + +| Attribute | Description | +|--------------|-------------| +| Status | The status of the runner. In [GitLab 15.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22224), for the **Ultimate** tier, the upgrade status is available. | +| Runner details | Information about the runner, including partial token and details about the computer the runner was registered from. | +| Version | GitLab Runner version. | +| Jobs | Total number of jobs run by the runner. | +| Tags | Tags associated with the runner. | +| Last contact | Timestamp indicating when the runner last contacted the GitLab instance. | + +You can also edit, pause, or remove each runner. + +### Administering Jobs + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/386311) from **Overview > Jobs** to **CI/CD > Jobs** in GitLab 15.8. + +You can administer all jobs in the GitLab instance from the Admin Area's Jobs page. + +To access the Jobs page: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **CI/CD > Jobs**. All jobs are listed, in descending order of job ID. +1. Select the **All** tab to list all jobs. Select the **Pending**, **Running**, or **Finished** + tab to list only jobs of that status. + +For each job, the following details are listed: + +| Field | Description | +|----------|-------------| +| Status | Job status, either **passed**, **skipped**, or **failed**. | +| Job | Includes links to the job, branch, and the commit that started the job. | +| Pipeline | Includes a link to the specific pipeline. | +| Project | Name of the project, and organization, to which the job belongs. | +| Runner | Name of the CI runner assigned to execute the job. | +| Stage | Stage that the job is declared in a `.gitlab-ci.yml` file. | +| Name | Name of the job specified in a `.gitlab-ci.yml` file. | +| Timing | Duration of the job, and how long ago the job completed. | +| Coverage | Percentage of tests coverage. | + +## Monitoring section + +The following topics document the **Monitoring** section of the Admin Area. + +### System Information + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341248) in GitLab 15.2, support for relative time. "Uptime" statistic was renamed to "System started". + +The **System Info** page provides the following statistics: + +| Field | Description | +|:---------------|:--------------------------------------------------| +| CPU | Number of CPU cores available | +| Memory Usage | Memory in use, and total memory available | +| Disk Usage | Disk space in use, and total disk space available | +| System started | When the system hosting GitLab was started. In GitLab 15.1 and earlier, this was an uptime statistic. | + +These statistics are updated only when you navigate to the **System Info** page, or you refresh the page in your browser. + +### Background Jobs + +The **Background Jobs** page displays the Sidekiq dashboard. Sidekiq is used by GitLab to +perform processing in the background. + +The Sidekiq dashboard consists of the following elements: + +- A tab per jobs' status. +- A breakdown of background job statistics. +- A live graph of **Processed** and **Failed** jobs, with a selectable polling interval. +- An historical graph of **Processed** and **Failed** jobs, with a selectable time span. +- Redis statistics, including: + - Version number + - Uptime, measured in days + - Number of connections + - Current memory usage, measured in MB + - Peak memory usage, measured in MB + +### Logs + +Since GitLab 13.0, **Log** view has been removed from the Admin Area dashboard since the logging does not work in multi-node setups and could cause confusion for administrators by displaying partial information. + +For multi-node systems we recommend ingesting the logs into services like Elasticsearch and Splunk. + +| Log file | Contents | +|:------------------------|:---------| +| `application_json.log` | GitLab user activity | +| `git_json.log` | Failed GitLab interaction with Git repositories | +| `production.log` | Requests received from Puma, and the actions taken to serve those requests | +| `sidekiq.log` | Background jobs | +| `repocheck.log` | Repository activity | +| `integrations_json.log` | Activity between GitLab and integrated systems | +| `kubernetes.log` | Kubernetes activity | + +The contents of these log files can be useful when troubleshooting a problem. + +For details of these log files and their contents, see [Log system](logs/index.md). + +The content of each log file is listed in chronological order. To minimize performance issues, a maximum 2000 lines of each log file are shown. + +### Audit Events **(PREMIUM SELF)** + +The **Audit Events** page lists changes made within the GitLab server. With this information you can control, analyze, and track every change. diff --git a/doc/administration/analytics/dev_ops_reports.md b/doc/administration/analytics/dev_ops_reports.md new file mode 100644 index 00000000000..1dcee5ec03d --- /dev/null +++ b/doc/administration/analytics/dev_ops_reports.md @@ -0,0 +1,74 @@ +--- +stage: Plan +group: Optimize +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# DevOps Reports **(FREE SELF)** + +DevOps Reports give you an overview of your entire instance's adoption of +[Concurrent DevOps](https://about.gitlab.com/topics/concurrent-devops/) +from planning to monitoring. + +To see DevOps Reports: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Analytics > DevOps Reports**. + +## DevOps Score + +> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) from Conversational Development Index in GitLab 12.6. + +NOTE: +To see the DevOps score, you must activate your GitLab instance's [Service Ping](../settings/usage_statistics.md#service-ping). DevOps Score is a comparative tool, so your score data must be centrally processed by GitLab Inc. first. + +You can use the DevOps score to compare your DevOps status to other organizations. + +The DevOps Score tab displays usage of major GitLab features on your instance over +the last 30 days, averaged over the number of billable users in that time period. +You can also see the Leader usage score, calculated from top-performing instances based on +[Service Ping data](../settings/usage_statistics.md#service-ping) that GitLab has collected. +Your score is compared to the lead score of each feature and then expressed +as a percentage at the bottom of said feature. Your overall **DevOps Score** is an average of your +feature scores. + +Service Ping data is aggregated on GitLab servers for analysis. Your usage +information is **not sent** to any other GitLab instances. +If you have just started using GitLab, it might take a few weeks for data to be collected before this +feature is available. + +## DevOps Adoption **(ULTIMATE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247112) in GitLab 13.7 as a [Beta feature](../../policy/experiment-beta-support.md#beta). +> - The Overview tab [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330401) in GitLab 14.1. +> - DAST and SAST metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328033) in GitLab 14.1. +> - Fuzz Testing metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/330398) in GitLab 14.2. +> - Dependency Scanning metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328034) in GitLab 14.2. +> - Multi-select [added](https://gitlab.com/gitlab-org/gitlab/-/issues/333586) in GitLab 14.2. +> - Overview table [added](https://gitlab.com/gitlab-org/gitlab/-/issues/335638) in GitLab 14.3. + +DevOps Adoption shows feature adoption for development, security, and operations. + +| Category | Feature | +| --- | --- | +| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | +| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | +| Operations | Deployments<br>Pipelines<br>Runners | + +You can use Group DevOps Adoption to: + +- Identify specific subgroups that are lagging in their adoption of GitLab features, so you can guide them on +their DevOps journey. +- Find subgroups that have adopted certain features, and provide guidance to other subgroups on +how to use those features. +- Verify if you are getting the return on investment that you expected from GitLab. + +## Add or remove a group + +To add or remove a subgroup from the DevOps Adoption report: + +1. Select **Add or remove groups**. +1. Select the subgroup you want to add or remove and select **Save changes**. + +![DevOps Adoption](img/admin_devops_adoption_v14_2.png) diff --git a/doc/administration/analytics/img/admin_devops_adoption_v14_2.png b/doc/administration/analytics/img/admin_devops_adoption_v14_2.png Binary files differnew file mode 100644 index 00000000000..666e03f1d9d --- /dev/null +++ b/doc/administration/analytics/img/admin_devops_adoption_v14_2.png diff --git a/doc/administration/analytics/img/instance_activity_pipelines_chart_v13_6_a.png b/doc/administration/analytics/img/instance_activity_pipelines_chart_v13_6_a.png Binary files differnew file mode 100644 index 00000000000..bd02065556c --- /dev/null +++ b/doc/administration/analytics/img/instance_activity_pipelines_chart_v13_6_a.png diff --git a/doc/administration/analytics/index.md b/doc/administration/analytics/index.md new file mode 100644 index 00000000000..6441cd866c8 --- /dev/null +++ b/doc/administration/analytics/index.md @@ -0,0 +1,26 @@ +--- +stage: Plan +group: Optimize +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Instance-level analytics **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41416) in GitLab 11.2. + +Instance-level analytics provide insights into the feature and data usage of your entire instance. + +## View instance-level analytics + +Prerequisite: + +- You must have administrator access to the instance. + +To view instance-level analytics: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Analytics**, then one of the available analytics: + + - [DevOps Reports](dev_ops_reports.md): Provides an overview of your entire instance's feature usage. + - [Usage Trends](usage_trends.md): Shows how much data your instance contains, and how the data is changing. diff --git a/doc/administration/analytics/usage_trends.md b/doc/administration/analytics/usage_trends.md new file mode 100644 index 00000000000..49e82f71a3a --- /dev/null +++ b/doc/administration/analytics/usage_trends.md @@ -0,0 +1,46 @@ +--- +stage: Plan +group: Optimize +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Usage Trends **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235754) in GitLab 13.5 behind a feature flag, disabled by default. +> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46962) in GitLab 13.6. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/285220) from Instance Statistics to Usage Trends in GitLab 13.6. +> - It's enabled on GitLab.com. +> - It's recommended for production use. + +Usage Trends gives you an overview of how much data your instance contains, and how quickly this volume is changing over time. +Usage Trends data refreshes daily. + +## View Usage Trends + +To view Usage Trends: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Analytics > Usage Trends**. + +## Total counts + +At the top of the page, Usage Trends shows total counts for: + +- Users +- Projects +- Groups +- Issues +- Merge requests +- Pipelines + +These figures can be useful for understanding how much data your instance contains in total. + +## Past year trend charts + +Usage Trends also displays line charts that show total counts per month, over the past 12 months, +in the categories shown in [Total counts](#total-counts). + +These charts help you visualize how rapidly these records are being created on your instance. + +![Instance Activity Pipelines chart](img/instance_activity_pipelines_chart_v13_6_a.png) diff --git a/doc/administration/appearance.md b/doc/administration/appearance.md new file mode 100644 index 00000000000..c5c50d95eb6 --- /dev/null +++ b/doc/administration/appearance.md @@ -0,0 +1,123 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GitLab Appearance **(FREE SELF)** + +Several options are available for customizing the appearance of a self-managed instance +of GitLab. To access these settings: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Appearance**. + +## Navigation bar + +By default, the navigation bar has the GitLab logo, but this can be customized with +any image desired. It is optimized for images 28px high (any width), but any image can be +used (less than 1 MB) and it is automatically resized. + +After you select and upload an image, select **Update appearance settings** at the bottom +of the page to activate it in the GitLab instance. + +NOTE: +GitLab pipeline emails also display the custom logo, unless the logo is in SVG format. If the +custom logo is in SVG format, the default logo is used instead because the SVG format is not +supported by many email clients. + +## Favicon + +By default, the favicon (used by the browser as the tab icon, as well as the CI status icon) +uses the GitLab logo. This can be customized with any icon desired. It must be a +32x32 `.png` or `.ico` image. + +After you select and upload an icon, select **Update appearance settings** at the bottom +of the page to activate it in the GitLab instance. + +## System header and footer messages + +> **Enable header and footer in emails** checkbox [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344819) in GitLab 15.9. + +You can add a small header message, a small footer message, or both, to the interface +of your GitLab instance. These messages appear on all projects and pages of the +instance, including the sign in / sign up page. The default color is white text on +an orange background, but this can be customized by selecting **Customize colors**. + +Limited [Markdown](../user/markdown.md) is supported, such as bold, italics, and links, for +example. Other Markdown features, including lists, images, and quotes are not supported +as the header and footer messages can only be a single line. + +You can select **Enable header and footer in emails** to have the text of +the header and footer added to all emails sent by the GitLab instance. + +After you add a message, select **Update appearance settings** at the bottom of the page +to activate it in the GitLab instance. + +## Sign in / Sign up pages + +You can replace the default message on the sign in / sign up page with your own message +and logo. You can make full use of [Markdown](../user/markdown.md) in the description. + +The optimal size for the logo is 128 x 128 pixels, but any image can be used (below 1 MB) +and it is resized automatically. The logo image appears between the title and +the description, on the left of the sign-up page. + +After you add a message, select **Update appearance settings** at the bottom of the page +to activate it in the GitLab instance. You can also select **Sign-in page**, +to review the saved appearance settings: + +NOTE: +You can add also add a [customized hcelp message](settings/help_page.md) below the sign in message or add [a Sign in text message](settings/sign_in_restrictions.md#sign-in-information). + +## Progressive Web App + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/375708) in GitLab 15.9. + +GitLab can be installed as a [Progressive Web App](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps) (PWA). +Use the Progressive Web App settings to customize its appearance, including its name, +description, and icon. + +### Configure the PWA settings + +To configure the PWA settings: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Appearance**. +1. Scroll to the **Progressive Web App (PWA)** section. +1. Complete the fields. + - **Icon**: If you use the standard GitLab icon, it is available in sizes 192x192 pixels, + 512x512 pixels, also as a maskable icon. If you use a custom icon, it must be in either size + 192x192 pixels, or 512x512 pixels. +1. Select **Update appearance settings**. + +## New project pages + +You can add a new project guidelines message to the **New project page** in GitLab. +You can make full use of [Markdown](../user/markdown.md) in the description: + +The message is displayed below the **New Project** message, on the left side +of the **New project page**. + +After you add a message, select **Update appearance settings** at the bottom of the page +to activate it in the GitLab instance. You can also select **New project page**, +which brings you to the new project page so you can review the change. + +## Libravatar + +[Libravatar](https://www.libravatar.org) is supported by GitLab for avatar images, but you must +[manually enable Libravatar support on the GitLab instance](../administration/libravatar.md) to use the service. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/audit_event_streaming/graphql_api.md b/doc/administration/audit_event_streaming/graphql_api.md index f5a31f073dc..9f8fef0e3ca 100644 --- a/doc/administration/audit_event_streaming/graphql_api.md +++ b/doc/administration/audit_event_streaming/graphql_api.md @@ -14,6 +14,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Custom HTTP headers API [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/366524) in GitLab 15.3. [Feature flag `streaming_audit_event_headers`](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) removed. > - User-specified verification token API support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360813) in GitLab 15.4. > - APIs for custom HTTP headers for instance level streaming destinations [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/404560) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. +> - User-specified destination name API support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/413894) in GitLab 16.2. Audit event streaming destinations can be maintained using a GraphQL API. @@ -38,6 +40,7 @@ mutation { errors externalAuditEventDestination { id + name destinationUrl verificationToken group { @@ -59,6 +62,28 @@ mutation { errors externalAuditEventDestination { id + name + destinationUrl + verificationToken + group { + name + } + } + } +} +``` + +You can optionally specify your own destination name (instead of the default GitLab-generated one) using the GraphQL +`externalAuditEventDestinationCreate` +mutation. Name length must not exceed 72 characters and trailing whitespace are not trimmed. This value should be unique scoped to a group. For example: + +```graphql +mutation { + externalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", name: "destination-name-here", groupPath: "my-group" }) { + errors + externalAuditEventDestination { + id + name destinationUrl verificationToken group { @@ -90,11 +115,12 @@ The header is created if the returned `errors` object is empty. ### Instance streaming destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named -`ff_external_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. Prerequisites: @@ -110,6 +136,7 @@ mutation { instanceExternalAuditEventDestination { destinationUrl id + name verificationToken } } @@ -121,6 +148,24 @@ Event streaming is enabled if: - The returned `errors` object is empty. - The API responds with `200 OK`. +You can optionally specify your own destination name (instead of the default GitLab-generated one) using the GraphQL +`instanceExternalAuditEventDestinationCreate` +mutation. Name length must not exceed 72 characters and trailing whitespace are not trimmed. This value should be unique. For example: + +```graphql +mutation { + instanceExternalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", name: "destination-name-here"}) { + errors + instanceExternalAuditEventDestination { + destinationUrl + id + name + verificationToken + } + } +} +``` + Instance administrators can add an HTTP header using the GraphQL `auditEventsStreamingInstanceHeadersCreate` mutation. You can retrieve the destination ID by [listing all the streaming destinations](#list-streaming-destinations) for the instance or from the mutation above. @@ -144,6 +189,39 @@ mutation { The header is created if the returned `errors` object is empty. +### Google Cloud Logging streaming + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409422) in GitLab 16.1. + +Prerequisites: + +- Owner role for a top-level group. +- A Google Cloud project with the necessary permissions to create service accounts and enable Google Cloud Logging. + +To enable streaming and add a configuration, use the +`googleCloudLoggingConfigurationCreate` mutation in the GraphQL API. + +```graphql +mutation { + googleCloudLoggingConfigurationCreate(input: { groupPath: "my-group", googleProjectIdName: "my-google-project", clientEmail: "my-email@my-google-project.iam.gservice.account.com", privateKey: "YOUR_PRIVATE_KEY", logIdName: "audit-events" } ) { + errors + googleCloudLoggingConfiguration { + id + googleProjectIdName + logIdName + privateKey + clientEmail + } + errors + } +} +``` + +Event streaming is enabled if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + ## List streaming destinations List new streaming destinations for top-level groups or an entire instance. @@ -166,6 +244,7 @@ query { destinationUrl verificationToken id + name headers { nodes { key @@ -184,11 +263,12 @@ If the resulting list is empty, then audit streaming is not enabled for that gro ### Instance streaming destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named -`ff_external_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. Prerequisites: @@ -202,6 +282,7 @@ query { instanceExternalAuditEventDestinations { nodes { id + name destinationUrl verificationToken headers { @@ -220,6 +301,38 @@ If the resulting list is empty, then audit streaming is not enabled for the inst You need the ID values returned by this query for the update and delete mutations. +### Google Cloud Logging configurations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409422) in GitLab 16.1. + +Prerequisite: + +- Owner role for a top-level group. + +You can view a list of streaming configurations for a top-level group using the `googleCloudLoggingConfigurations` query +type. + +```graphql +query { + group(fullPath: "my-group") { + id + googleCloudLoggingConfigurations { + nodes { + id + logIdName + googleProjectIdName + privateKey + clientEmail + } + } + } +} +``` + +If the resulting list is empty, then audit streaming is not enabled for the group. + +You need the ID values returned by this query for the update and delete mutations. + ## Update streaming destinations Update streaming destinations for a top-level group or an entire instance. @@ -236,8 +349,20 @@ by [listing all the custom HTTP headers](#list-streaming-destinations) for the g ```graphql mutation { - externalAuditEventDestinationDestroy(input: { id: destination }) { + externalAuditEventDestinationUpdate(input: { + id:"gid://gitlab/AuditEvents::ExternalAuditEventDestination/1", + destinationUrl: "https://www.new-domain.com/webhook", + name: "destination-name"} ) { errors + externalAuditEventDestination { + id + name + destinationUrl + verificationToken + group { + name + } + } } } ``` @@ -262,11 +387,12 @@ The header is deleted if the returned `errors` object is empty. ### Instance streaming destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named -`ff_external_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. Prerequisites: @@ -278,11 +404,15 @@ by [listing all the external destinations](#list-streaming-destinations) for the ```graphql mutation { - instanceExternalAuditEventDestinationUpdate(input: { id: "gid://gitlab/AuditEvents::InstanceExternalAuditEventDestination/1", destinationUrl: "https://www.new-domain.com/webhook"}) { + instanceExternalAuditEventDestinationUpdate(input: { + id: "gid://gitlab/AuditEvents::InstanceExternalAuditEventDestination/1", + destinationUrl: "https://www.new-domain.com/webhook", + name: "destination-name"}) { errors instanceExternalAuditEventDestination { destinationUrl id + name verificationToken } } @@ -313,6 +443,40 @@ mutation { The header is updated if the returned `errors` object is empty. +### Google Cloud Logging configurations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409422) in GitLab 16.1. + +Prerequisite: + +- Owner role for a top-level group. + +To update streaming configuration for a top-level group, use the +`googleCloudLoggingConfigurationUpdate` mutation type. You can retrieve the configuration ID +by [listing all the external destinations](#list-streaming-destinations). + +```graphql +mutation { + googleCloudLoggingConfigurationUpdate( + input: {id: "gid://gitlab/AuditEvents::GoogleCloudLoggingConfiguration/1", googleProjectIdName: "my-google-project", clientEmail: "my-email@my-google-project.iam.gservice.account.com", privateKey: "YOUR_PRIVATE_KEY", logIdName: "audit-events"} + ) { + errors + googleCloudLoggingConfiguration { + id + logIdName + privateKey + googleProjectIdName + clientEmail + } + } +} +``` + +Streaming configuration is updated if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + ## Delete streaming destinations Delete streaming destinations for a top-level group or an entire instance. @@ -357,11 +521,12 @@ The header is deleted if the returned `errors` object is empty. ### Instance streaming destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335175) in GitLab 16.0 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named -`ff_external_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. Prerequisites: @@ -384,6 +549,45 @@ Streaming destination is deleted if: - The returned `errors` object is empty. - The API responds with `200 OK`. +To remove an HTTP header, use the GraphQL `auditEventsStreamingInstanceHeadersDestroy` mutation. +To retrieve the header ID, +[list all the custom HTTP headers](#list-streaming-destinations) for the instance. + +```graphql +mutation { + auditEventsStreamingInstanceHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::InstanceHeader/<id>" }) { + errors + } +} +``` + +The header is deleted if the returned `errors` object is empty. + +### Google Cloud Logging configurations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/409422) in GitLab 16.1. + +Prerequisite: + +- Owner role for a top-level group. + +Users with the Owner role for a group can delete streaming configurations using the +`googleCloudLoggingConfigurationDestroy` mutation type. You can retrieve the configurations ID +by [listing all the streaming destinations](#list-streaming-destinations) for the group. + +```graphql +mutation { + googleCloudLoggingConfigurationDestroy(input: { id: "gid://gitlab/AuditEvents::GoogleCloudLoggingConfiguration/1" }) { + errors + } +} +``` + +Streaming configuration is deleted if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + ## Event type filters > Event type filters API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344845) in GitLab 15.7. diff --git a/doc/administration/audit_event_streaming/index.md b/doc/administration/audit_event_streaming/index.md index 44c6cff7455..622d29fa9a7 100644 --- a/doc/administration/audit_event_streaming/index.md +++ b/doc/administration/audit_event_streaming/index.md @@ -41,7 +41,7 @@ To add streaming destinations to a top-level group: 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. -1. Select **Add streaming destination** to show the section for adding destinations. +1. Select **Add streaming destination** and select **HTTP endpoint** to show the section for adding destinations. 1. Enter the destination URL to add. 1. Optional. Locate the **Custom HTTP headers** table. 1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the @@ -50,13 +50,14 @@ To add streaming destinations to a top-level group: 20 headers per streaming destination. 1. After all headers have been filled out, select **Add** to add the new streaming destination. -### Instance streaming destinations +### Instance streaming destinations **(ULTIMATE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `instance_streaming_audit_events`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named -`instance_streaming_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. Prerequisites: @@ -68,8 +69,30 @@ To add a streaming destination for an instance: 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select **Streams** tab. -1. Select **Add streaming destination** to show the section for adding destinations. +1. Select **Add streaming destination** and select **HTTP endpoint** to show the section for adding destinations. 1. Enter the destination URL to add. +1. Optional. To add custom HTTP headers, select **Add header** to create a new name and value pair, and input their values. Repeat this step for as many name and value pairs are required. You can add up to 20 headers per streaming destination. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the + **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509). +1. Select **Add header** to create a new name and value pair. Repeat this step for as many name and value pairs are required. You can add up to + 20 headers per streaming destination. +1. After all headers have been filled out, select **Add** to add the new streaming destination. + +### Google Cloud Logging streaming + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124384) in GitLab 16.2. + +Prerequisites: + +- Owner role for a top-level group. + +To add Google Cloud Logging streaming destinations to a top-level group: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. Select **Add streaming destination** and select **Google Cloud Logging** to show the section for adding destinations. +1. Enter the Google Project ID, Google Client Email, Log ID, and Google Private Key to add. 1. Select **Add** to add the new streaming destination. ## List streaming destinations @@ -87,15 +110,16 @@ To list the streaming destinations for a top-level group: 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. -1. To the right of the item, select **Edit** (**{pencil}**) to see all the custom HTTP headers. +1. Select the stream URL to expand it and see all the custom HTTP headers. ### For instance streaming destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `instance_streaming_audit_events`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named -`instance_streaming_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. Prerequisites: @@ -108,6 +132,21 @@ To list the streaming destinations for an instance: 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select **Streams** tab. +### Google Cloud Logging streaming + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124384) in GitLab 16.2. + +Prerequisites: + +- Owner role for a top-level group. + +To list Google Cloud Logging streaming destinations for a top-level group: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. Select the Google Cloud Logging stream to expand and see all the fields. + ## Update streaming destinations Update streaming destinations for a top-level group or an entire instance. @@ -123,6 +162,33 @@ To update a streaming destination's custom HTTP headers: 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. 1. Select **Secure > Audit events**. 1. On the main area, select **Streams** tab. +1. Select the stream URL to expand. +1. Locate the **Custom HTTP headers** table. +1. Locate the header that you wish to update. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the + **Active** checkbox, see [issue 367509](https://gitlab.com/gitlab-org/gitlab/-/issues/367509). +1. Select **Add header** to create a new name and value pair. Enter as many name and value pairs as required. You can add up to + 20 headers per streaming destination. +1. Select **Save** to update the streaming destination. + +### Instance streaming destinations **(ULTIMATE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. + +FLAG: +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. + +Prerequisites: + +- Administrator access on the instance. + +To update the streaming destinations for an instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Secure > Audit events**. +1. On the main area, select the **Streams** tab. 1. To the right of the item, select **Edit** (**{pencil}**). 1. Locate the **Custom HTTP headers** table. 1. Locate the header that you wish to update. @@ -132,6 +198,23 @@ To update a streaming destination's custom HTTP headers: 20 headers per streaming destination. 1. Select **Save** to update the streaming destination. +### Google Cloud Logging streaming + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124384) in GitLab 16.2. + +Prerequisites: + +- Owner role for a top-level group. + +To update Google Cloud Logging streaming destinations to a top-level group: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select **Streams** tab. +1. Select the Google Cloud Logging stream to expand. +1. Enter the Google Project ID, Google Client Email, Log ID, and Google Private Key to update. +1. Select **Save** to update the streaming destination. + ## Delete streaming destinations Delete streaming destinations for a top-level group or an entire instance. When the last destination is successfully @@ -148,14 +231,16 @@ To delete a streaming destination: 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. 1. Select **Secure > Audit events**. 1. On the main area, select the **Streams** tab. -1. To the right of the item, select **Delete** (**{remove}**). +1. Select the stream URL to expand. +1. Select **Delete destination**. +1. Confirm by selecting **Delete destination** in the modal. To delete only the custom HTTP headers for a streaming destination: 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. 1. Select **Secure > Audit events**. 1. On the main area, select the **Streams** tab. -1. To the right of the item, **Edit** (**{pencil}**). +1. Select the stream URL to expand. 1. Locate the **Custom HTTP headers** table. 1. Locate the header that you wish to remove. 1. To the right of the header, select **Delete** (**{remove}**). @@ -163,11 +248,12 @@ To delete only the custom HTTP headers for a streaming destination: ### Instance streaming destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `instance_streaming_audit_events`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named -`instance_streaming_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. Prerequisites: @@ -179,7 +265,38 @@ To delete the streaming destinations for an instance: 1. Select **Admin Area**. 1. On the left sidebar, select **Monitoring > Audit Events**. 1. On the main area, select the **Streams** tab. -1. To the right of the item, select **Delete** (**{remove}**). +1. Select the stream URL to expand. +1. Select **Delete destination**. +1. Confirm by selecting **Delete destination** in the modal. + +To delete only the custom HTTP headers for a streaming destination: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. On the left sidebar, select **Monitoring > Audit Events**. +1. On the main area, select the **Streams** tab. +1. To the right of the item, **Edit** (**{pencil}**). +1. Locate the **Custom HTTP headers** table. +1. Locate the header that you wish to remove. +1. To the right of the header, select **Delete** (**{remove}**). +1. Select **Save** to update the streaming destination. + +### Google Cloud Logging streaming + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124384) in GitLab 16.2. + +Prerequisites: + +- Owner role for a top-level group. + +To delete Google Cloud Logging streaming destinations to a top-level group: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select the **Streams** tab. +1. Select the Google Cloud Logging stream to expand. +1. Select **Delete destination**. +1. Confirm by selecting **Delete destination** in the modal. ## Verify event authenticity @@ -204,15 +321,17 @@ To list streaming destinations and see the verification tokens: 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. 1. Select **Secure > Audit events**. 1. On the main area, select the **Streams**. -1. View the verification token on the right side of each item. +1. Select the stream URL to expand. +1. Locate the **Verification token** input. ### Instance streaming destinations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `instance_streaming_audit_events`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/398107) in GitLab 16.1 [with a flag](../feature_flags.md) named `ff_external_audit_events`. Disabled by default. +> - [Feature flag `ff_external_audit_events`](https://gitlab.com/gitlab-org/gitlab/-/issues/393772) enabled by default in GitLab 16.2. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) named -`instance_streaming_audit_events`. On GitLab.com, this feature is not available. The feature is not ready for production use. +On self-managed GitLab, by default this feature is enabled. To disable it, an administrator can [disable the feature flag](../feature_flags.md) named +`ff_external_audit_events`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is ready for production use. Prerequisites: @@ -226,6 +345,25 @@ To list streaming destinations for an instance and see the verification tokens: 1. On the main area, select the **Streams**. 1. View the verification token on the right side of each item. +## Event type filters + +> Event type filtering in the UI with a defined list of audit event types [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/413581) in GitLab 16.1. + +When this feature is enabled for a group, you can permit users to filter streamed audit events per destination. +If the feature is enabled with no filters, the destination receives all audit events. + +A streaming destination that has an event type filter set has a **filtered** (**{filter}**) label. + +To update a streaming destination's event filters: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. Select **Secure > Audit events**. +1. On the main area, select the **Streams** tab. +1. Select the stream URL to expand. +1. Locate the **Filter by audit event type** dropdown. +1. Select the dropdown list and select or clear the required event types. +1. Select **Save** to update the event filters. + ## Override default content type header By default, streaming destinations use a `content-type` header of `application/x-www-form-urlencoded`. However, you diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index e924da39145..69c97982562 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -49,7 +49,7 @@ You can view audit events scoped to a group or project. To view a group's audit events: 1. Go to the group. -1. On the left sidebar, select **Security and Compliance > Audit Events**. +1. On the left sidebar, select **Secure > Audit events**. Group events do not include project audit events. Group events can also be accessed using the [Group Audit Events API](../api/audit_events.md#group-audit-events). Group event queries are limited to a maximum of 30 @@ -58,7 +58,7 @@ days. To view a project's audit events: 1. Go to the project. -1. On the left sidebar, select **Security & Compliance > Audit Events**. +1. On the left sidebar, select **Secure > Audit events**. Project events can also be accessed using the [Project Audit Events API](../api/audit_events.md#project-audit-events). Project event queries are limited to a maximum of 30 days. @@ -119,7 +119,7 @@ The first row contains the headers, which are listed in the following table alon Successful sign-in events are the only audit events available at all tiers. To see successful sign-in events: -1. Select your avatar. +1. On the left sidebar, select your avatar. 1. Select **Edit profile > Authentication log**. After upgrading to a paid tier, you can also see successful sign-in events on audit event pages. @@ -140,7 +140,7 @@ From audit events pages, different filters are available depending on the page y > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/536) in GitLab 13.0. > - Impersonation session events included in group audit events in GitLab 14.8. -When a user is [impersonated](../user/admin_area/index.md#user-impersonation), their actions are logged as audit events +When a user is [impersonated](../administration/admin_area.md#user-impersonation), their actions are logged as audit events with additional details: - Audit events include information about the impersonating administrator. These audit events are visible in audit event @@ -362,6 +362,10 @@ GitLab generates audit events when a cluster agent token is created or revoked. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276250) in GitLab 13.6, audit events for when a user is approved using the Admin Area. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276921) in GitLab 13.6, audit events for when a user's personal access token is successfully or unsuccessfully created or revoked. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298783) in GitLab 13.9, audit events for when a user requests access to an instance or is rejected using the Admin Area. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in + GitLab 15.1, audit events when a user's two-factor authentication is disabled. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124169) in GitLab 16.2, audit events when a user's access is locked. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124973) in GitLab 16.2, audit events when a user's access is unlocked. The following user actions on a GitLab instance generate instance audit events: @@ -373,8 +377,9 @@ The following user actions on a GitLab instance generate instance audit events: - Grant OAuth access. - Failed second-factor authentication attempt. - A user's personal access token was successfully or unsuccessfully created or revoked. -- A user's two-factor authentication was disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238177) in - GitLab 15.1. +- A user's two-factor authentication was disabled. +- A user's access is locked. +- A user's access is unlocked. #### User management @@ -400,6 +405,13 @@ The following user actions on a GitLab instance generate instance audit events: Instance events can also be accessed using the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). +#### Application settings **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/282428) in GitLab 16.2. + +When a user changes an application setting in an instance, project, or group, +that change and the user that made the change are recorded in the audit log. + ### GitLab Runner events > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335509) in GitLab 14.8, audit events for when a runner is registered. diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 3b6992c92e0..e8ed0eb4313 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Users with auditor access have read-only access to all groups, projects, and other resources except: -- The [Admin Area](../user/admin_area/index.md). +- The [Admin Area](../administration/admin_area.md). - Project and group settings. For more information, see [Auditor user permissions and restrictions](#auditor-user-permissions-and-restrictions) @@ -50,9 +50,9 @@ users with auditor access have the same [permissions](../user/permissions.md) as If you are signed in with auditor access, you: -- Have full access to projects you own. -- Have read-only access to projects you aren't a member of. -- Have [permissions](../user/permissions.md) based on your role to projects you are a member of. For example, if you have the Developer role, +- Have full access to the projects and groups you own. +- Have read-only access to the projects and groups you are not a member of. +- Have [permissions](../user/permissions.md) based on your role to projects and groups you are a member of. For example, if you have the Developer role, you can push commits or comment on issues. - Can access the same resources using the GitLab UI or API. - Can't view the Admin Area, or perform any administration actions. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index a4484da5940..1905a009eb6 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -25,7 +25,7 @@ Users added through LDAP: - Usually use a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). - Can authenticate with Git using either their GitLab username or their email and LDAP password, even if password authentication for Git - [is disabled](../../../user/admin_area/settings/sign_in_restrictions.md#password-authentication-enabled). + [is disabled](../../settings/sign_in_restrictions.md#password-authentication-enabled). The LDAP DN is associated with existing GitLab users when: @@ -1000,13 +1000,13 @@ authenticated with the TLS protocol. Users deleted from the LDAP server: - Are immediately blocked from signing in to GitLab. -- [No longer consume a license](../../../user/admin_area/moderate_users.md). +- [No longer consume a license](../../../administration/moderate_users.md). However, these users can continue to use Git with SSH until the next time the [LDAP check cache runs](ldap_synchronization.md#adjust-ldap-user-sync-schedule). To delete the account immediately, you can manually -[block the user](../../../user/admin_area/moderate_users.md#block-a-user). +[block the user](../../../administration/moderate_users.md#block-a-user). ## Update user email addresses diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 9fb3888b995..8cffff7b725 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -378,7 +378,7 @@ things to debug the situation. an LDAP DN as the `Identifier`. If not, this user hasn't signed in with LDAP yet and must do so first. - You've waited an hour or [the configured interval](ldap_synchronization.md#adjust-ldap-group-sync-schedule) for - the group to sync. To speed up the process, either go to the GitLab group **Group information > Members** + the group to sync. To speed up the process, either go to the GitLab group **Manage > Members** and press **Sync now** (sync one group) or [run the group sync Rake task](../../raketasks/ldap.md#run-a-group-sync) (sync all groups). @@ -447,7 +447,7 @@ A displayed error can identify the problem and point to a solution. For example, ```ruby irb(main):018:0> group.members.map(&:errors).map(&:full_messages) -=> [["The member's email address is not allowed for this group. Go to the group’s 'Settings > General' page, and check 'Restrict membership by email domain'."]] +=> [["The member's email address is not allowed for this group. Go to the group's 'Settings > General' page, and check 'Restrict membership by email domain'."]] ``` This error showed that an Administrator chose to [restrict group membership by email domain](../../../user/group/access_and_permissions.md#restrict-group-access-by-domain), @@ -737,7 +737,7 @@ To resolve this error, you must apply a new license to the GitLab instance witho 1. Remove or comment out the GitLab configuration lines for all non-primary LDAP servers. 1. [Reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) so that it temporarily uses only one LDAP server. -1. Enter the [Rails console and add the license key](../../../user/admin_area/license_file.md#add-a-license-through-the-console). +1. Enter the [Rails console and add the license key](../../../administration/license_file.md#add-a-license-through-the-console). 1. Re-enable the additional LDAP servers in the GitLab configuration and reconfigure GitLab again. ## Users are being removed from group and re-added again diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index e4b43feacc2..9d9ed563fe5 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -40,7 +40,7 @@ For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/ The process also updates the following user information: - Name. Because of a [sync issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342598), `name` is not synchronized if - [**Prevent users from changing their profile name**](../../../user/admin_area/settings/account_and_limit_settings.md#disable-user-profile-name-changes) is enabled or `sync_name` is set to `false`. + [**Prevent users from changing their profile name**](../../../administration/settings/account_and_limit_settings.md#disable-user-profile-name-changes) is enabled or `sync_name` is set to `false`. - Email address. - SSH public keys if `sync_ssh_keys` is set. - Kerberos identity if Kerberos is enabled. @@ -627,7 +627,7 @@ sync to run once every two hours at the top of the hour. ### External groups Using the `external_groups` setting allows you to mark all users belonging -to these groups as [external users](../../../user/admin_area/external_users.md). +to these groups as [external users](../../../administration/external_users.md). Group membership is checked periodically through the `LdapGroupSync` background task. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 23d2ab512db..d48de109bd0 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -757,8 +757,8 @@ For more information, see the [GitLab API user method documentation](https://pyt You can configure OIDC group membership to: - Require users to be members of a certain group. -- Assign users [external roles](../../user/admin_area/external_users.md), or as - administrators based on group membership. +- Assign users [external](../external_users.md), administrator or + [auditor](../auditor_users.md) roles based on group membership. GitLab checks these groups on each sign in and updates user attributes as necessary. This feature **does not** allow you to automatically add users to GitLab @@ -845,12 +845,12 @@ For self-compiled installations: ### External groups Your IdP must pass group information to GitLab in the OIDC response. To use this -response to identify users as [external users](../../user/admin_area/external_users.md) +response to identify users as [external users](../external_users.md) based on group membership, configure GitLab to identify: - Where to look for the groups in the OIDC response, using the `groups_attribute` setting. - Which group memberships should identify a user as an - [external user](../../user/admin_area/external_users.md), using the + [external user](../external_users.md), using the `external_groups` setting. For Linux package installations: @@ -921,6 +921,83 @@ For self-compiled installations: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +### Auditor groups **(PREMIUM SELF)** + +Your IdP must pass group information to GitLab in the OIDC response. To use this +response to assign users as auditors based on group membership, configure GitLab to identify: + +- Where to look for the groups in the OIDC response, using the `groups_attribute` setting. +- Which group memberships grant the user auditor access, using the `auditor_groups` + setting. + +For Linux package installations: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "openid_connect", + label: "Provider name", + args: { + name: "openid_connect", + scope: ["openid","profile","email","groups"], + response_type: "code", + issuer: "<your_oidc_url>", + discovery: true, + client_auth_method: "query", + uid_field: "<uid_field>", + client_options: { + identifier: "<your_oidc_client_id>", + secret: "<your_oidc_client_secret>", + redirect_uri: "<your_gitlab_url>/users/auth/openid_connect/callback", + gitlab: { + groups_attribute: "groups", + auditor_groups: ["Auditor"] + } + } + } + } + ] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect. + +For self-compiled installations: + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + omniauth: + providers: + - { name: 'openid_connect', + label: 'Provider name', + args: { + name: 'openid_connect', + scope: ['openid','profile','email','groups'], + response_type: 'code', + issuer: '<your_oidc_url>', + discovery: true, + client_auth_method: 'query', + uid_field: '<uid_field>', + client_options: { + identifier: '<your_oidc_client_id>', + secret: '<your_oidc_client_secret>', + redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback', + gitlab: { + groups_attribute: "groups", + auditor_groups: ["Auditor"] + } + } + } + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + ### Administrator groups Your IdP must pass group information to GitLab in the OIDC response. To use this diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 9432a627ed7..5802db78dd6 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -17,7 +17,7 @@ By default, existing users can continue to sign in with a username and password authentication is enabled. To force existing users to use only smartcard authentication, -[disable username and password authentication](../../user/admin_area/settings/sign_in_restrictions.md#password-authentication-enabled). +[disable username and password authentication](../settings/sign_in_restrictions.md#password-authentication-enabled). ## Authentication methods diff --git a/doc/administration/backup_restore/backup_gitlab.md b/doc/administration/backup_restore/backup_gitlab.md new file mode 100644 index 00000000000..24b7b453517 --- /dev/null +++ b/doc/administration/backup_restore/backup_gitlab.md @@ -0,0 +1,1925 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Back up GitLab **(FREE SELF)** + +The exact procedure for backing up GitLab depends on many factors. Your particular deployment's usage and configuration determine what kind of data exists, where it is located, and how much there is. These factors influence your options for how to perform a back up, how to store it, and how to restore it. + +## Simple back up procedure + +As a rough guideline, if you are using a [1k reference architecture](../reference_architectures/1k_users.md) with less than 100 GB of data, then follow these steps: + +1. Run the [backup command](#backup-command). +1. Back up [object storage](#object-storage), if applicable. +1. Manually back up [configuration files](#storing-configuration-files). + +## Scaling backups + +As the volume of GitLab data grows, the [backup command](#backup-command) takes longer to execute. At some point, the execution time becomes impractical. For example, it can take 24 hours or more. + +For more information, see [alternative backup strategies](#alternative-backup-strategies). + +## What data needs to be backed up? + +- [PostgreSQL databases](#postgresql-databases) +- [Git repositories](#git-repositories) +- [Blobs](#blobs) +- [Configuration files](#storing-configuration-files) +- [Other data](#other-data) + +### PostgreSQL databases + +In the simplest case, GitLab has one PostgreSQL database in one PostgreSQL server on the same VM as all other GitLab services. But depending on configuration, GitLab may use multiple PostgreSQL databases in multiple PostgreSQL servers. + +In general, this data is the single source of truth for most user-generated content in the Web interface, such as issue and merge request content, comments, permissions, and credentials. + +PostgreSQL also holds some cached data like HTML-rendered Markdown, and by default, merge request diffs. +However, merge request diffs can also be configured to be offloaded to the file system or object storage, see [Blobs](#blobs). + +Gitaly Cluster's Praefect service uses a PostgreSQL database as a single source of truth to manage its Gitaly nodes. + +A common PostgreSQL utility, [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html), produces a backup file which can be used to restore a PostgreSQL database. The [backup command](#backup-command) uses this utility under the hood. + +Unfortunately, the larger the database, the longer it takes `pg_dump` to execute. Depending on your situation, the duration becomes impractical at some point (days, for example). If your database is over 100 GB, `pg_dump`, and by extension the [backup command](#backup-command), is likely not usable. For more information, see [alternative backup strategies](#alternative-backup-strategies). + +### Git repositories + +A GitLab instance can have one or more repository shards. Each shard is a Gitaly instance or Gitaly Cluster that +is responsible for allowing access and operations on the locally stored Git repositories. Gitaly can run +on a machine: + +- With a single disk. +- With multiple disks mounted as a single mount-point (like with a RAID array). +- Using LVM. + +Each project can have up to 3 different repositories: + +- A project repository, where the source code is stored. +- A wiki repository, where the wiki content is stored. +- A design repository, where design artifacts are indexed (assets are actually in LFS). + +They all live in the same shard and share the same base name with a `-wiki` and `-design` suffix +for Wiki and Design Repository cases. + +Personal and project snippets, and group wiki content, are stored in Git repositories. + +Project forks are deduplicated in live a GitLab site using pool repositories. + +The [backup command](#backup-command) produces a Git bundle for each repository and tars them all up. This duplicates pool repository data into every fork. In [our testing](https://gitlab.com/gitlab-org/gitlab/-/issues/396343), 100 GB of Git repositories took a little over 2 hours to back up and upload to S3. At around 400 GB of Git data, the backup command is likely not viable for regular backups. For more information, see [alternative backup strategies](#alternative-backup-strategies). + +### Blobs + +GitLab stores blobs (or files) such as issue attachments or LFS objects into either: + +- The file system in a specific location. +- An [Object Storage](../object_storage.md) solution. Object Storage solutions can be: + - Cloud based like Amazon S3 and Google Cloud Storage. + - Hosted by you (like MinIO). + - A Storage Appliance that exposes an Object Storage-compatible API. + +#### Object storage + +The [backup command](#backup-command) doesn't back up blobs that aren't stored on the file system. If you're using [object storage](../object_storage.md), be sure to enable backups with your object storage provider. For example, see: + +- [Amazon S3 backups](https://docs.aws.amazon.com/aws-backup/latest/devguide/s3-backups.html) +- [Google Cloud Storage Transfer Service](https://cloud.google.com/storage-transfer-service) and [Google Cloud Storage Object Versioning](https://cloud.google.com/storage/docs/object-versioning) + +### Storing configuration files + +WARNING: +The [backup Rake task](#back-up-gitlab) GitLab provides does _not_ store your configuration files. The primary reason for this is that your database contains items including encrypted information for two-factor authentication and the CI/CD _secure variables_. Storing encrypted information in the same location as its key defeats the purpose of using encryption in the first place. For example, the secrets file contains your database encryption key. If you lose it, then the GitLab application will not be able to decrypt any encrypted values in the database. + +WARNING: +The secrets file may change after upgrades. + +You should back up the configuration directory. At the very **minimum**, you must back up: + +::Tabs + +:::TabTitle Linux package + +- `/etc/gitlab/gitlab-secrets.json` +- `/etc/gitlab/gitlab.rb` + +For more information, see [Backup and restore Linux package (Omnibus) configuration](https://docs.gitlab.com/omnibus/settings/backups.html#backup-and-restore-omnibus-gitlab-configuration). + +:::TabTitle Self-compiled + +- `/home/git/gitlab/config/secrets.yml` +- `/home/git/gitlab/config/gitlab.yml` + +:::TabTitle Docker + +- Back up the volume where the configuration files are stored. If you created +the GitLab container according to the documentation, it should be in the +`/srv/gitlab/config` directory. + +:::TabTitle GitLab Helm chart + +- Follow the [Back up the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#back-up-the-secrets) +instructions. + +::EndTabs + +You may also want to back up any TLS keys and certificates (`/etc/gitlab/ssl`, `/etc/gitlab/trusted-certs`), and your +[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) +to avoid man-in-the-middle attack warnings if you have to perform a full machine restore. + +In the unlikely event that the secrets file is lost, see the +[troubleshooting section](#when-the-secrets-file-is-lost). + +### Other data + +GitLab uses Redis both as a cache store and to hold persistent data for our background jobs system, Sidekiq. The provided [backup command](#backup-command) does _not_ back up Redis data. This means that in order to take a consistent backup with the [backup command](#backup-command), there must be no pending or running background jobs. It is possible to [manually back up Redis](https://redis.io/docs/management/persistence/#backing-up-redis-data). + +Elasticsearch is an optional database for advanced search. It can improve search +in both source-code level, and user generated content in issues, merge requests, and discussions. The [backup command](#backup-command) does _not_ back up Elasticsearch data. Elasticsearch data can be regenerated from PostgreSQL data after a restore. It is possible to [manually back up Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html). + +## Command line interface + +GitLab provides a command line interface to back up your entire instance, +including: + +- Database +- Attachments +- Git repositories data +- CI/CD job output logs +- CI/CD job artifacts +- LFS objects +- Terraform states ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331806) in GitLab 14.7) +- Container Registry images +- GitLab Pages content +- Packages ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332006) in GitLab 14.7) +- Snippets +- [Group wikis](../../user/project/wiki/group.md) +- Project-level Secure Files ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121142) in GitLab 16.1) + +Backups do not include: + +- [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage) +- Redis (and thus Sidekiq jobs) +- [Object storage](#object-storage) + +WARNING: +GitLab does not back up any configuration files (`/etc/gitlab`), TLS keys and certificates, or system +files. You are highly advised to read about [storing configuration files](#storing-configuration-files). + +### Requirements + +To be able to back up and restore, ensure that Rsync is installed on your +system. If you installed GitLab: + +- Using the Linux package, Rsync is already installed. +- Using self-compiled, check if `rsync` is installed. If Rsync is not installed, install it. For example: + + ```shell + # Debian/Ubuntu + sudo apt-get install rsync + + # RHEL/CentOS + sudo yum install rsync + ``` + +### Backup command + +WARNING: +The backup command does not back up items in [object storage](#object-storage). + +WARNING: +The backup command requires [additional parameters](#back-up-and-restore-for-installations-using-pgbouncer) when +your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster. + +WARNING: +Before GitLab 15.5.0, the backup command doesn't verify if another backup is already running, as described in +[issue 362593](https://gitlab.com/gitlab-org/gitlab/-/issues/362593). We strongly recommend +you make sure that all backups are complete before starting a new one. + +NOTE: +You can only restore a backup to **exactly the same version and type (CE/EE)** +of GitLab on which it was created. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-backup create +``` + +:::TabTitle Helm chart (Kubernetes) + +Run the backup task by using `kubectl` to run the `backup-utility` script on the GitLab toolbox pod. For more details, see the [charts backup documentation](https://docs.gitlab.com/charts/backup-restore/backup.html). + +:::TabTitle Docker + +Run the backup from the host. + +- GitLab 12.2 or later: + +```shell +docker exec -t <container name> gitlab-backup create +``` + +:::TabTitle Self-compiled + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +::EndTabs + +If your GitLab deployment has multiple nodes, you need to pick a node for running the backup command. You must ensure that the designated node: + +- is persistent, and not subject to auto-scaling. +- has the GitLab Rails application already installed. If Puma or Sidekiq is running, then Rails is installed. +- has sufficient storage and memory to produce the backup file. + +Example output: + +```plaintext +Dumping database tables: +- Dumping table events... [DONE] +- Dumping table issues... [DONE] +- Dumping table keys... [DONE] +- Dumping table merge_requests... [DONE] +- Dumping table milestones... [DONE] +- Dumping table namespaces... [DONE] +- Dumping table notes... [DONE] +- Dumping table projects... [DONE] +- Dumping table protected_branches... [DONE] +- Dumping table schema_migrations... [DONE] +- Dumping table services... [DONE] +- Dumping table snippets... [DONE] +- Dumping table taggings... [DONE] +- Dumping table tags... [DONE] +- Dumping table users... [DONE] +- Dumping table users_projects... [DONE] +- Dumping table web_hooks... [DONE] +- Dumping table wikis... [DONE] +Dumping repositories: +- Dumping repository abcd... [DONE] +Creating backup archive: $TIMESTAMP_gitlab_backup.tar [DONE] +Deleting tmp directories...[DONE] +Deleting old backups... [SKIPPING] +``` + +### Backup timestamp + +The backup archive is saved in `backup_path`, which is specified in the +`config/gitlab.yml` file. The default path is `/var/opt/gitlab/backups`. The filename is `[TIMESTAMP]_gitlab_backup.tar`, +where `TIMESTAMP` identifies the time at which each backup was created, plus +the GitLab version. The timestamp is needed if you need to restore GitLab and +multiple backups are available. + +For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`, +the timestamp is `1493107454_2018_04_25_10.6.4-ce`. + +### Backup options + +The command line tool GitLab provides to back up your instance can accept more +options. + +#### Backup strategy option + +The default backup strategy is to essentially stream data from the respective +data locations to the backup using the Linux command `tar` and `gzip`. This works +fine in most cases, but can cause problems when data is rapidly changing. + +When data changes while `tar` is reading it, the error `file changed as we read +it` may occur, and causes the backup process to fail. In that case, you can use +the backup strategy called `copy`. The strategy copies data files +to a temporary location before calling `tar` and `gzip`, avoiding the error. + +A side-effect is that the backup process takes up to an additional 1X disk +space. The process does its best to clean up the temporary files at each stage +so the problem doesn't compound, but it could be a considerable change for large +installations. + +To use the `copy` strategy instead of the default streaming strategy, specify +`STRATEGY=copy` in the Rake task command. For example: + +```shell +sudo gitlab-backup create STRATEGY=copy +``` + +#### Backup filename + +WARNING: +If you use a custom backup filename, you can't +[limit the lifetime of the backups](#limit-backup-lifetime-for-local-files-prune-old-backups). + +By default, a backup file is created according to the specification in the +previous [Backup timestamp](#backup-timestamp) section. You can, however, +override the `[TIMESTAMP]` portion of the filename by setting the `BACKUP` +environment variable. For example: + +```shell +sudo gitlab-backup create BACKUP=dump +``` + +The resulting file is named `dump_gitlab_backup.tar`. This is useful for +systems that make use of rsync and incremental backups, and results in +considerably faster transfer speeds. + +#### Confirm archive can be transferred + +To ensure the generated archive is transferable by rsync, you can set the `GZIP_RSYNCABLE=yes` +option. This sets the `--rsyncable` option to `gzip`, which is useful only in +combination with setting [the Backup filename option](#backup-filename). + +The `--rsyncable` option in `gzip` isn't guaranteed to be available +on all distributions. To verify that it's available in your distribution, run +`gzip --help` or consult the man pages. + +```shell +sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes +``` + +#### Excluding specific directories from the backup + +You can exclude specific directories from the backup by adding the environment variable `SKIP`, whose values are a comma-separated list of the following options: + +- `db` (database) +- `uploads` (attachments) +- `builds` (CI job output logs) +- `artifacts` (CI job artifacts) +- `lfs` (LFS objects) +- `terraform_state` (Terraform states) +- `registry` (Container Registry images) +- `pages` (Pages content) +- `repositories` (Git repositories data) +- `packages` (Packages) +- `ci_secure_files` (Project-level Secure Files) + +NOTE: +When [backing up and restoring Helm Charts](https://docs.gitlab.com/charts/architecture/backup-restore.html), there is an additional option `packages`, which refers to any packages managed by the GitLab [package registry](../../user/packages/package_registry/index.md). +For more information see [command line arguments](https://docs.gitlab.com/charts/architecture/backup-restore.html#command-line-arguments). + +All wikis are backed up as part of the `repositories` group. Non-existent +wikis are skipped during a backup. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-backup create SKIP=db,uploads +``` + +:::TabTitle Self-compiled + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production +``` + +::EndTabs + +`SKIP=` is also used to: + +- [Skip creation of the tar file](#skipping-tar-creation) (`SKIP=tar`). +- [Skip uploading the backup to remote storage](#skip-uploading-backups-to-remote-storage) (`SKIP=remote`). + +#### Skipping tar creation + +NOTE: +It is not possible to skip the tar creation when using [object storage](#upload-backups-to-a-remote-cloud-storage) for backups. + +The last part of creating a backup is generation of a `.tar` file containing all the parts. In some cases, creating a `.tar` file might be wasted effort or even directly harmful, so you can skip this step by adding `tar` to the `SKIP` environment variable. Example use-cases: + +- When the backup is picked up by other backup software. +- To speed up incremental backups by avoiding having to extract the backup every time. (In this case, `PREVIOUS_BACKUP` and `BACKUP` must not be specified, otherwise the specified backup is extracted, but no `.tar` file is generated at the end.) + +Adding `tar` to the `SKIP` variable leaves the files and directories containing the +backup in the directory used for the intermediate files. These files are +overwritten when a new backup is created, so you should make sure they are copied +elsewhere, because you can only have one backup on the system. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-backup create SKIP=tar +``` + +:::TabTitle Self-compiled + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=production +``` + +::EndTabs + +#### Back up Git repositories concurrently + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37158) in GitLab 13.3. +> - [Concurrent restore introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69330) in GitLab 14.3 + +When using [multiple repository storages](../repository_storage_paths.md), +repositories can be backed up or restored concurrently to help fully use CPU time. The +following variables are available to modify the default behavior of the Rake +task: + +- `GITLAB_BACKUP_MAX_CONCURRENCY`: The maximum number of projects to back up at + the same time. Defaults to the number of logical CPUs (in GitLab 14.1 and + earlier, defaults to `1`). +- `GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY`: The maximum number of projects to + back up at the same time on each storage. This allows the repository backups + to be spread across storages. Defaults to `2` (in GitLab 14.1 and earlier, + defaults to `1`). + +For example, with 4 repository storages: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-backup create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1 +``` + +:::TabTitle Self-compiled + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1 +``` + +::EndTabs + +#### Incremental repository backups + +> - Introduced in GitLab 14.9 [with a flag](../feature_flags.md) named `incremental_repository_backup`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/355945) in GitLab 14.10. +> - `PREVIOUS_BACKUP` option [introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4184) in GitLab 15.0. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, an administrator can [disable the feature flag](../feature_flags.md) named `incremental_repository_backup`. +On GitLab.com, this feature is not available. + +NOTE: +Only repositories support incremental backups. Therefore, if you use `INCREMENTAL=yes`, the task +creates a self-contained backup tar archive. This is because all subtasks except repositories are +still creating full backups (they overwrite the existing full backup). +See [issue 19256](https://gitlab.com/gitlab-org/gitlab/-/issues/19256) for a feature request to +support incremental backups for all subtasks. + +Incremental repository backups can be faster than full repository backups because they only pack changes since the last backup into the backup bundle for each repository. +The incremental backup archives are not linked to each other: each archive is a self-contained backup of the instance. There must be an existing backup +to create an incremental backup from: + +- In GitLab 14.9 and 14.10, use the `BACKUP=<timestamp_of_backup>` option to choose the backup to use. The chosen previous backup is overwritten. +- In GitLab 15.0 and later, use the `PREVIOUS_BACKUP=<timestamp_of_backup>` option to choose the backup to use. By default, a backup file is created + as documented in the [Backup timestamp](#backup-timestamp) section. You can override the `[TIMESTAMP]` portion of the filename by setting the + [`BACKUP` environment variable](#backup-filename). + +To create an incremental backup, run: + +- In GitLab 15.0 or later: + + ```shell + sudo gitlab-backup create INCREMENTAL=yes PREVIOUS_BACKUP=<timestamp_of_backup> + ``` + +- In GitLab 14.9 and 14.10: + + ```shell + sudo gitlab-backup create INCREMENTAL=yes BACKUP=<timestamp_of_backup> + ``` + +To create an [untarred](#skipping-tar-creation) incremental backup from a tarred backup, use `SKIP=tar`: + +```shell +sudo gitlab-backup create INCREMENTAL=yes SKIP=tar +``` + +#### Back up specific repository storages + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86896) in GitLab 15.0. + +When using [multiple repository storages](../repository_storage_paths.md), +repositories from specific repository storages can be backed up separately +using the `REPOSITORIES_STORAGES` option. The option accepts a comma-separated list of +storage names. + +For example: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-backup create REPOSITORIES_STORAGES=storage1,storage2 +``` + +:::TabTitle Self-compiled + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_STORAGES=storage1,storage2 +``` + +::EndTabs + +#### Back up specific repositories + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. + +You can back up specific repositories using the `REPOSITORIES_PATHS` option. +Similarly, you can use `SKIP_REPOSITORIES_PATHS` to skip certain repositories. +Both options accept a comma-separated list of project or group paths. If you +specify a group path, all repositories in all projects in the group and +descendent groups are included or skipped, depending on which option you used. + +For example, to back up all repositories for all projects in **Group A** (`group-a`), the repository for **Project C** in **Group B** (`group-b/project-c`), +and skip the **Project D** in **Group A** (`group-a/project-d`): + +::Tabs + +:::TabTitle Linux package (Omnibus) + + ```shell + sudo gitlab-backup create REPOSITORIES_PATHS=group-a,group-b/project-c SKIP_REPOSITORIES_PATHS=group-a/project-d + ``` + +:::TabTitle Self-compiled + + ```shell + sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_PATHS=group-a,group-b/project-c SKIP_REPOSITORIES_PATHS=group-a/project-d + ``` + +::EndTabs + +#### Upload backups to a remote (cloud) storage + +NOTE: +It is not possible to [skip the tar creation](#skipping-tar-creation) when using object storage for backups. + +You can let the backup script upload (using the [Fog library](https://fog.io/)) +the `.tar` file it creates. In the following example, we use Amazon S3 for +storage, but Fog also lets you use [other storage providers](https://fog.io/storage/). +GitLab also [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/-/blob/da46c9655962df7d49caef0e2b9f6bbe88462a02/Gemfile#L113) +for AWS, Google, and Aliyun. A local driver is +[also available](#upload-to-locally-mounted-shares). + +[Read more about using object storage with GitLab](../object_storage.md). + +##### Using Amazon S3 + +For Linux package (Omnibus): + +1. Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-west-1', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123' + # If using an IAM Profile, don't configure aws_access_key_id & aws_secret_access_key + # 'use_iam_profile' => true + } + gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' + # Consider using multipart uploads when file size reaches 100MB. Enter a number in bytes. + # gitlab_rails['backup_multipart_chunk_size'] = 104857600 + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect + +##### S3 Encrypted Buckets + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64765) in GitLab 14.3. + +AWS supports these [modes for server side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html): + +- Amazon S3-Managed Keys (SSE-S3) +- Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS) +- Customer-Provided Keys (SSE-C) + +Use your mode of choice with GitLab. Each mode has similar, but slightly +different, configuration methods. + +###### SSE-S3 + +To enable SSE-S3, in the backup storage options set the `server_side_encryption` +field to `AES256`. For example, in the Linux package (Omnibus): + +```ruby +gitlab_rails['backup_upload_storage_options'] = { + 'server_side_encryption' => 'AES256' +} +``` + +###### SSE-KMS + +To enable SSE-KMS, you need the +[KMS key via its Amazon Resource Name (ARN) in the `arn:aws:kms:region:acct-id:key/key-id` format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). +Under the `backup_upload_storage_options` configuration setting, set: + +- `server_side_encryption` to `aws:kms`. +- `server_side_encryption_kms_key_id` to the ARN of the key. + +For example, in the Linux package (Omnibus): + +```ruby +gitlab_rails['backup_upload_storage_options'] = { + 'server_side_encryption' => 'aws:kms', + 'server_side_encryption_kms_key_id' => 'arn:aws:<YOUR KMS KEY ID>:' +} +``` + +###### SSE-C + +SSE-C requires you to set these encryption options: + +- `backup_encryption`: AES256. +- `backup_encryption_key`: Unencoded, 32-byte (256 bits) key. The upload fails if this isn't exactly 32 bytes. + +For example, in the Linux package (Omnibus): + +```ruby +gitlab_rails['backup_encryption'] = 'AES256' +gitlab_rails['backup_encryption_key'] = '<YOUR 32-BYTE KEY HERE>' +``` + +If the key contains binary characters and cannot be encoded in UTF-8, +instead, specify the key with the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable. +For example: + +```ruby +gitlab_rails['env'] = { 'GITLAB_BACKUP_ENCRYPTION_KEY' => "\xDE\xAD\xBE\xEF" * 8 } +``` + +##### Digital Ocean Spaces + +This example can be used for a bucket in Amsterdam (AMS3): + +1. Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'ams3', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123', + 'endpoint' => 'https://ams3.digitaloceanspaces.com' + } + gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect + +If you see a `400 Bad Request` error message when using Digital Ocean Spaces, +the cause may be the use of backup encryption. Because Digital Ocean Spaces +doesn't support encryption, remove or comment the line that contains +`gitlab_rails['backup_encryption']`. + +##### Other S3 Providers + +Not all S3 providers are fully compatible with the Fog library. For example, +if you see a `411 Length Required` error message after attempting to upload, +you may need to downgrade the `aws_signature_version` value from the default +value to `2`, [due to this issue](https://github.com/fog/fog-aws/issues/428). + +For installations from source: + +1. Edit `home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + # snip + upload: + # Fog storage connection settings, see https://fog.io/storage/ . + connection: + provider: AWS + region: eu-west-1 + aws_access_key_id: AKIAKIAKI + aws_secret_access_key: 'secret123' + # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty + # ie. aws_access_key_id: '' + # use_iam_profile: 'true' + # The remote 'directory' to store your backups. For S3, this would be the bucket name. + remote_directory: 'my.s3.bucket' + # Specifies Amazon S3 storage class to use for backups, this is optional + # storage_class: 'STANDARD' + # + # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional + # 'encryption' must be set in order for this to have any effect. + # 'encryption_key' should be set to the 256-bit encryption key for Amazon S3 to use to encrypt or decrypt. + # To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY` your data. + # encryption: 'AES256' + # encryption_key: '<key>' + # + # + # Turns on AWS Server-Side Encryption with Amazon S3-Managed keys (optional) + # https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html + # For SSE-S3, set 'server_side_encryption' to 'AES256'. + # For SS3-KMS, set 'server_side_encryption' to 'aws:kms'. Set + # 'server_side_encryption_kms_key_id' to the ARN of customer master key. + # storage_options: + # server_side_encryption: 'aws:kms' + # server_side_encryption_kms_key_id: 'arn:aws:kms:YOUR-KEY-ID-HERE' + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect + +If you're uploading your backups to S3, you should create a new +IAM user with restricted access rights. To give the upload user access only for +uploading backups create the following IAM profile, replacing `my.s3.bucket` +with the name of your bucket: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "Stmt1412062044000", + "Effect": "Allow", + "Action": [ + "s3:AbortMultipartUpload", + "s3:GetBucketAcl", + "s3:GetBucketLocation", + "s3:GetObject", + "s3:GetObjectAcl", + "s3:ListBucketMultipartUploads", + "s3:PutObject", + "s3:PutObjectAcl" + ], + "Resource": [ + "arn:aws:s3:::my.s3.bucket/*" + ] + }, + { + "Sid": "Stmt1412062097000", + "Effect": "Allow", + "Action": [ + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "*" + ] + }, + { + "Sid": "Stmt1412062128000", + "Effect": "Allow", + "Action": [ + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::my.s3.bucket" + ] + } + ] +} +``` + +##### Using Google Cloud Storage + +To use Google Cloud Storage to save backups, you must first create an +access key from the Google console: + +1. Go to the [Google storage settings page](https://console.cloud.google.com/storage/settings). +1. Select **Interoperability**, and then create an access key. +1. Make note of the **Access Key** and **Secret** and replace them in the + following configurations. +1. In the buckets advanced settings ensure the Access Control option + **Set object-level and bucket-level permissions** is selected. +1. Ensure you have already created a bucket. + +For the Linux package (Omnibus): + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_storage_access_key_id' => 'Access Key', + 'google_storage_secret_access_key' => 'Secret', + + ## If you have CNAME buckets (foo.example.com), you might run into SSL issues + ## when uploading backups ("hostname foo.example.com.storage.googleapis.com + ## does not match the server certificate"). In that case, uncomnent the following + ## setting. See: https://github.com/fog/fog/issues/2834 + #'path_style' => true + } + gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect + +For installations from source: + +1. Edit `home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + upload: + connection: + provider: 'Google' + google_storage_access_key_id: 'Access Key' + google_storage_secret_access_key: 'Secret' + remote_directory: 'my.google.bucket' + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect + +##### Using Azure Blob storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AzureRM', + 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', + 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', + 'azure_storage_domain' => 'blob.core.windows.net', # Optional + } + gitlab_rails['backup_upload_remote_directory'] = '<AZURE BLOB CONTAINER>' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect + +:::TabTitle Self-compiled + +1. Edit `home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + upload: + connection: + provider: 'AzureRM' + azure_storage_account_name: '<AZURE STORAGE ACCOUNT NAME>' + azure_storage_access_key: '<AZURE STORAGE ACCESS KEY>' + remote_directory: '<AZURE BLOB CONTAINER>' + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect + +::EndTabs + +For more details, see the [table of Azure parameters](../object_storage.md#azure-blob-storage). + +##### Specifying a custom directory for backups + +This option works only for remote storage. If you want to group your backups, +you can pass a `DIRECTORY` environment variable: + +```shell +sudo gitlab-backup create DIRECTORY=daily +sudo gitlab-backup create DIRECTORY=weekly +``` + +#### Skip uploading backups to remote storage + +If you have configured GitLab to [upload backups in a remote storage](#upload-backups-to-a-remote-cloud-storage), +you can use the `SKIP=remote` option to skip uploading your backups to the remote storage. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-backup create SKIP=remote +``` + +:::TabTitle Self-compiled + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create SKIP=remote RAILS_ENV=production +``` + +::EndTabs + +#### Upload to locally-mounted shares + +You can send backups to a locally-mounted share (for example, `NFS`,`CIFS`, or `SMB`) using the Fog +[`Local`](https://github.com/fog/fog-local#usage) storage provider. + +To do this, you must set the following configuration keys: + +- `backup_upload_connection.local_root`: mounted directory that backups are copied to. +- `backup_upload_remote_directory`: subdirectory of the `backup_upload_connection.local_root` directory. It is created if it doesn't exist. + If you want to copy the tarballs to the root of your mounted directory, use `.`. + +When mounted, the directory set in the `local_root` key must be owned by either: + +- The `git` user. So, mounting with the `uid=` of the `git` user for `CIFS` and `SMB`. +- The user that you are executing the backup tasks as. For the Linux package (Omnibus), this is the `git` user. + +Because file system performance may affect overall GitLab performance, +[we don't recommend using cloud-based file systems for storage](../nfs.md#avoid-using-cloud-based-file-systems). + +##### Avoid conflicting configuration + +Don't set the following configuration keys to the same path: + +- `gitlab_rails['backup_path']` (`backup.path` for source installations). +- `gitlab_rails['backup_upload_connection'].local_root` (`backup.upload.connection.local_root` for source installations). + +The `backup_path` configuration key sets the local location of the backup file. The `upload` configuration key is +intended for use when the backup file is uploaded to a separate server, perhaps for archival purposes. + +If these configuration keys are set to the same location, the upload feature fails because a backup already exists at +the upload location. This failure causes the upload feature to delete the backup because it assumes it's a residual file +remaining after the failed upload attempt. + +##### Configure uploads to locally-mounted shares + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + :provider => 'Local', + :local_root => '/mnt/backups' + } + + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect. + +:::TabTitle Self-compiled + +1. Edit `home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + upload: + # Fog storage connection settings, see https://fog.io/storage/ . + connection: + provider: Local + local_root: '/mnt/backups' + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + remote_directory: 'gitlab_backups' + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + +::EndTabs + +#### Backup archive permissions + +The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`) +have the owner/group `git`/`git` and 0600 permissions by default. This is +meant to avoid other system users reading GitLab data. If you need the backup +archives to have different permissions, you can use the `archive_permissions` +setting. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect. + +:::TabTitle Self-compiled + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + archive_permissions: 0644 # Makes the backup archives world-readable + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + +::EndTabs + +#### Configuring cron to make daily backups + +WARNING: +The following cron jobs do not [back up your GitLab configuration files](#storing-configuration-files) +or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). + +You can schedule a cron job that backs up your repositories and GitLab metadata. + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit the crontab for the `root` user: + + ```shell + sudo su - + crontab -e + ``` + +1. There, add the following line to schedule the backup for everyday at 2 AM: + + ```plaintext + 0 2 * * * /opt/gitlab/bin/gitlab-backup create CRON=1 + ``` + +:::TabTitle Self-compiled + +1. Edit the crontab for the `git` user: + + ```shell + sudo -u git crontab -e + ``` + +1. Add the following lines at the bottom: + + ```plaintext + # Create a full backup of the GitLab repositories and SQL database every day at 2am + 0 2 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1 + ``` + +::EndTabs + +The `CRON=1` environment setting directs the backup script to hide all progress +output if there aren't any errors. This is recommended to reduce cron spam. +When troubleshooting backup problems, however, replace `CRON=1` with `--trace` to log verbosely. + +#### Limit backup lifetime for local files (prune old backups) + +WARNING: +The process described in this section doesn't work if you used a [custom filename](#backup-filename) +for your backups. + +To prevent regular backups from using all your disk space, you may want to set a limited lifetime +for backups. The next time the backup task runs, backups older than the `backup_keep_time` are +pruned. + +This configuration option manages only local files. GitLab doesn't prune old +files stored in a third-party [object storage](#upload-backups-to-a-remote-cloud-storage) +because the user may not have permission to list and delete files. It's +recommended that you configure the appropriate retention policy for your object +storage (for example, [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html)). + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + ## Limit backup lifetime to 7 days - 604800 seconds + gitlab_rails['backup_keep_time'] = 604800 + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect. + +:::TabTitle Self-compiled + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + ## Limit backup lifetime to 7 days - 604800 seconds + keep_time: 604800 + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect. + +::EndTabs + +#### Back up and restore for installations using PgBouncer + +Do not back up or restore GitLab through a PgBouncer connection. These +tasks must [bypass PgBouncer and connect directly to the PostgreSQL primary database node](#bypassing-pgbouncer), +or they cause a GitLab outage. + +When the GitLab backup or restore task is used with PgBouncer, the +following error message is shown: + +```ruby +ActiveRecord::StatementInvalid: PG::UndefinedTable +``` + +Each time the GitLab backup runs, GitLab starts generating 500 errors and errors about missing +tables will [be logged by PostgreSQL](../logs/index.md#postgresql-logs): + +```plaintext +ERROR: relation "tablename" does not exist at character 123 +``` + +This happens because the task uses `pg_dump`, which +[sets a null search path and explicitly includes the schema in every SQL query](https://gitlab.com/gitlab-org/gitlab/-/issues/23211) +to address [CVE-2018-1058](https://www.postgresql.org/about/news/postgresql-103-968-9512-9417-and-9322-released-1834/). + +Because connections are reused with PgBouncer in transaction pooling mode, +PostgreSQL fails to search the default `public` schema. As a result, +this clearing of the search path causes tables and columns to appear +missing. + +##### Bypassing PgBouncer + +There are two ways to fix this: + +1. [Use environment variables to override the database settings](#environment-variable-overrides) for the backup task. +1. Reconfigure a node to [connect directly to the PostgreSQL primary database node](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + +###### Environment variable overrides + +By default, GitLab uses the database configuration stored in a +configuration file (`database.yml`). However, you can override the database settings +for the backup and restore task by setting environment +variables that are prefixed with `GITLAB_BACKUP_`: + +- `GITLAB_BACKUP_PGHOST` +- `GITLAB_BACKUP_PGUSER` +- `GITLAB_BACKUP_PGPORT` +- `GITLAB_BACKUP_PGPASSWORD` +- `GITLAB_BACKUP_PGSSLMODE` +- `GITLAB_BACKUP_PGSSLKEY` +- `GITLAB_BACKUP_PGSSLCERT` +- `GITLAB_BACKUP_PGSSLROOTCERT` +- `GITLAB_BACKUP_PGSSLCRL` +- `GITLAB_BACKUP_PGSSLCOMPRESSION` + +For example, to override the database host and port to use 192.168.1.10 +and port 5432 with the Linux package (Omnibus): + +```shell +sudo GITLAB_BACKUP_PGHOST=192.168.1.10 GITLAB_BACKUP_PGPORT=5432 /opt/gitlab/bin/gitlab-backup create +``` + +See the [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-envars.html) +for more details on what these parameters do. + +#### `gitaly-backup` for repository backup and restore + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.2. +> - [Deployed behind a feature flag](../../user/feature_flags.md), enabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.10. [Feature flag `gitaly_backup`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83254) removed. + +The `gitaly-backup` binary is used by the backup Rake task to create and restore repository backups from Gitaly. +`gitaly-backup` replaces the previous backup method that directly calls RPCs on Gitaly from GitLab. + +The backup Rake task must be able to find this executable. In most cases, you don't need to change +the path to the binary as it should work fine with the default path `/opt/gitlab/embedded/bin/gitaly-backup`. +If you have a specific reason to change the path, it can be configured in the Linux package (Omnibus): + +1. Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_gitaly_backup_path'] = '/path/to/gitaly-backup' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) + for the changes to take effect. + +## Alternative backup strategies + +Because every deployment may have different capabilities, you should first review [what data needs to be backed up](#what-data-needs-to-be-backed-up) to better understand if, and how, you can leverage them. + +For example, if you use Amazon RDS, you might choose to use its built-in backup and restore features to handle your GitLab [PostgreSQL data](#postgresql-databases), and [exclude PostgreSQL data](#excluding-specific-directories-from-the-backup) when using the [backup command](#backup-command). + +In the following cases, consider using file system data transfer or snapshots as part of your backup strategy: + +- Your GitLab instance contains a lot of Git repository data and the GitLab backup script is too slow. +- Your GitLab instance has a lot of forked projects and the regular backup task duplicates the Git data for all of them. +- Your GitLab instance has a problem and using the regular backup and import Rake tasks isn't possible. + +WARNING: +Gitaly Cluster [does not support snapshot backups](../gitaly/index.md#snapshot-backup-and-recovery-limitations). + +When considering using file system data transfer or snapshots: + +- Don't use these methods to migrate from one operating system to another. The operating systems of the source and destination should be as similar as possible. For example, + don't use these methods to migrate from Ubuntu to Fedora. +- Data consistency is very important. You should stop GitLab with `sudo gitlab-ctl stop` before taking doing a file system transfer (with `rsync`, for example) or taking a + snapshot. + +Example: Amazon Elastic Block Store (EBS) + +- A GitLab server using the Linux package (Omnibus) hosted on Amazon AWS. +- An EBS drive containing an ext4 file system is mounted at `/var/opt/gitlab`. +- In this case you could make an application backup by taking an EBS snapshot. +- The backup includes all repositories, uploads and PostgreSQL data. + +Example: Logical Volume Manager (LVM) snapshots + rsync + +- A GitLab server using the Linux package (Omnibus), with an LVM logical volume mounted at `/var/opt/gitlab`. +- Replicating the `/var/opt/gitlab` directory using rsync would not be reliable because too many files would change while rsync is running. +- Instead of rsync-ing `/var/opt/gitlab`, we create a temporary LVM snapshot, which we mount as a read-only file system at `/mnt/gitlab_backup`. +- Now we can have a longer running rsync job which creates a consistent replica on the remote server. +- The replica includes all repositories, uploads and PostgreSQL data. + +If you're running GitLab on a virtualized server, you can possibly also create +VM snapshots of the entire GitLab server. It's not uncommon however for a VM +snapshot to require you to power down the server, which limits this solution's +practical use. + +### Back up repository data separately + +First, ensure you back up existing GitLab data while [skipping repositories](#excluding-specific-directories-from-the-backup): + +::Tabs + +:::TabTitle Linux package (Omnibus) + +```shell +sudo gitlab-backup create SKIP=repositories +``` + +:::TabTitle Self-compiled + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create SKIP=repositories RAILS_ENV=production +``` + +::EndTabs + +For manually backing up the Git repository data on disk, there are multiple possible strategies: + +- Use snapshots, such as the previous examples of Amazon EBS drive snapshots, or LVM snapshots + rsync. +- Use [GitLab Geo](../geo/index.md) and rely on the repository data on a Geo secondary site. +- [Prevent writes and copy the Git repository data](#prevent-writes-and-copy-the-git-repository-data). +- [Create an online backup by marking repositories as read-only (experimental)](#online-backup-through-marking-repositories-as-read-only-experimental). + +#### Prevent writes and copy the Git repository data + +Git repositories must be copied in a consistent way. They should not be copied during concurrent write +operations, as this can lead to inconsistencies or corruption issues. For more details, +[issue #270422](https://gitlab.com/gitlab-org/gitlab/-/issues/270422 "Provide documentation on preferred method of migrating Gitaly servers") +has a longer discussion explaining the potential problems. + +To prevent writes to the Git repository data, there are two possible approaches: + +- Use [maintenance mode](../maintenance_mode/index.md) to place GitLab in a read-only state. +- Create explicit downtime by stopping all Gitaly services before backing up the repositories: + + ```shell + sudo gitlab-ctl stop gitaly + # execute git data copy step + sudo gitlab-ctl start gitaly + ``` + +You can copy Git repository data using any method, as long as writes are prevented on the data being copied +(to prevent inconsistencies and corruption issues). In order of preference and safety, the recommended methods are: + +1. Use `rsync` with archive-mode, delete, and checksum options, for example: + + ```shell + rsync -aR --delete --checksum source destination # be extra safe with the order as it will delete existing data if inverted + ``` + +1. Use a [`tar` pipe to copy the entire repository's directory to another server or location](../operations/moving_repositories.md#tar-pipe-to-another-server). + +1. Use `sftp`, `scp`, `cp`, or any other copying method. + +#### Online backup through marking repositories as read-only (experimental) + +One way of backing up repositories without requiring instance-wide downtime +is to programmatically mark projects as read-only while copying the underlying data. + +There are a few possible downsides to this: + +- Repositories are read-only for a period of time that scales with the size of the repository. +- Backups take a longer time to complete due to marking each project as read-only, potentially leading to inconsistencies. For example, + a possible date discrepancy between the last data available for the first project that gets backed up compared to + the last project that gets backed up. +- Fork networks should be entirely read-only while the projects inside get backed up to prevent potential changes to the pool repository. + +There is an **experimental** script that attempts to automate this process in +[the Geo team Runbooks project](https://gitlab.com/gitlab-org/geo-team/runbooks/-/tree/main/experimental-online-backup-through-rsync). + +## Troubleshooting + +The following are possible problems you might encounter, along with potential +solutions. + +### When the secrets file is lost + +If you didn't [back up the secrets file](#storing-configuration-files), you +must complete several steps to get GitLab working properly again. + +The secrets file is responsible for storing the encryption key for the columns +that contain required, sensitive information. If the key is lost, GitLab can't +decrypt those columns, preventing access to the following items: + +- [CI/CD variables](../../ci/variables/index.md) +- [Kubernetes / GCP integration](../../user/infrastructure/clusters/index.md) +- [Custom Pages domains](../../user/project/pages/custom_domains_ssl_tls_certification/index.md) +- [Project error tracking](../../operations/error_tracking.md) +- [Runner authentication](../../ci/runners/index.md) +- [Project mirroring](../../user/project/repository/mirror/index.md) +- [Integrations](../../user/project/integrations/index.md) +- [Web hooks](../../user/project/integrations/webhooks.md) + +In cases like CI/CD variables and runner authentication, you can experience +unexpected behaviors, such as: + +- Stuck jobs. +- 500 errors. + +In this case, you must reset all the tokens for CI/CD variables and +runner authentication, which is described in more detail in the following +sections. After resetting the tokens, you should be able to visit your project +and the jobs begin running again. + +WARNING: +The steps in this section can potentially lead to **data loss** on the above listed items. +Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. + +#### Verify that all values can be decrypted + +You can determine if your database contains values that can't be decrypted by using a +[Rake task](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). + +#### Take a backup + +You must directly modify GitLab data to work around your lost secrets file. + +WARNING: +Be sure to create a full database backup before attempting any changes. + +#### Disable user two-factor authentication (2FA) + +Users with 2FA enabled can't sign in to GitLab. In that case, you must +[disable 2FA for everyone](../../security/two_factor_authentication.md#for-all-users), +after which users must reactivate 2FA. + +#### Reset CI/CD variables + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For installations from source, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Examine the `ci_group_variables` and `ci_variables` tables: + + ```sql + SELECT * FROM public."ci_group_variables"; + SELECT * FROM public."ci_variables"; + ``` + + These are the variables that you need to delete. + +1. Delete all variables: + + ```sql + DELETE FROM ci_group_variables; + DELETE FROM ci_variables; + ``` + +1. If you know the specific group or project from which you wish to delete variables, you can include a `WHERE` statement to specify that in your `DELETE`: + + ```sql + DELETE FROM ci_group_variables WHERE group_id = <GROUPID>; + DELETE FROM ci_variables WHERE project_id = <PROJECTID>; + ``` + +You may need to reconfigure or restart GitLab for the changes to take effect. + +#### Reset runner registration tokens + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For installations from source, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Clear all tokens for projects, groups, and the entire instance: + + WARNING: + The final `UPDATE` operation stops the runners from being able to pick + up new jobs. You must register new runners. + + ```sql + -- Clear project tokens + UPDATE projects SET runners_token = null, runners_token_encrypted = null; + -- Clear group tokens + UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; + -- Clear instance tokens + UPDATE application_settings SET runners_registration_token_encrypted = null; + -- Clear key used for JWT authentication + -- This may break the $CI_JWT_TOKEN job variable: + -- https://gitlab.com/gitlab-org/gitlab/-/issues/325965 + UPDATE application_settings SET encrypted_ci_jwt_signing_key = null; + -- Clear runner tokens + UPDATE ci_runners SET token = null, token_encrypted = null; + ``` + +#### Reset pending pipeline jobs + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For installations from source, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Clear all the tokens for pending jobs: + + For GitLab 15.3 and earlier: + + ```sql + -- Clear build tokens + UPDATE ci_builds SET token = null, token_encrypted = null; + ``` + + For GitLab 15.4 and later: + + ```sql + -- Clear build tokens + UPDATE ci_builds SET token_encrypted = null; + ``` + +A similar strategy can be employed for the remaining features. By removing the +data that can't be decrypted, GitLab can be returned to operation, and the +lost data can be manually replaced. + +#### Fix integrations and webhooks + +If you've lost your secrets, the [integrations settings](../../user/project/integrations/index.md) +and [webhooks settings](../../user/project/integrations/webhooks.md) pages might display `500` error messages. Lost secrets might also produce `500` errors when you try to access a repository in a project with a previously configured integration or webhook. + +The fix is to truncate the affected tables (those containing encrypted columns). +This deletes all your configured integrations, webhooks, and related metadata. +You should verify that the secrets are the root cause before deleting any data. + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For installations from source, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + +1. Truncate the following tables: + + ```sql + -- truncate web_hooks table + TRUNCATE integrations, chat_names, issue_tracker_data, jira_tracker_data, slack_integrations, web_hooks, zentao_tracker_data, web_hook_logs CASCADE; + ``` + +### Container Registry push failures after restoring from a backup + +If you use the [Container Registry](../../user/packages/container_registry/index.md), +pushes to the registry may fail after restoring your backup on a Linux package (Omnibus) +instance after restoring the registry data. + +These failures mention permission issues in the registry logs, similar to: + +```plaintext +level=error +msg="response completed with error" +err.code=unknown +err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/...: permission denied" +err.message="unknown error" +``` + +This issue is caused by the restore running as the unprivileged user `git`, +which is unable to assign the correct ownership to the registry files during +the restore process ([issue #62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). + +To get your registry working again: + +```shell +sudo chown -R registry:registry /var/opt/gitlab/gitlab-rails/shared/registry/docker +``` + +If you changed the default file system location for the registry, run `chown` +against your custom location, instead of `/var/opt/gitlab/gitlab-rails/shared/registry/docker`. + +### Backup fails to complete with Gzip error + +When running the backup, you may receive a Gzip error message: + +```shell +sudo /opt/gitlab/bin/gitlab-backup create +... +Dumping ... +... +gzip: stdout: Input/output error + +Backup failed +``` + +If this happens, examine the following: + +- Confirm there is sufficient disk space for the Gzip operation. It's not uncommon for backups that + use the [default strategy](#backup-strategy-option) to require half the instance size + in free disk space during backup creation. +- If NFS is being used, check if the mount option `timeout` is set. The + default is `600`, and changing this to smaller values results in this error. + +### Backup fails with `File name too long` error + +During backup, you can get the `File name too long` error ([issue #354984](https://gitlab.com/gitlab-org/gitlab/-/issues/354984)). For example: + +```plaintext +Problem: <class 'OSError: [Errno 36] File name too long: +``` + +This problem stops the backup script from completing. To fix this problem, you must truncate the filenames causing the problem. A maximum of 246 characters, including the file extension, is permitted. + +WARNING: +The steps in this section can potentially lead to **data loss**. All steps must be followed strictly in the order given. +Consider opening a [Support Request](https://support.gitlab.com/hc/en-us/requests/new) if you're a Premium or Ultimate customer. + +Truncating filenames to resolve the error involves: + +- Cleaning up remote uploaded files that aren't tracked in the database. +- Truncating the filenames in the database. +- Rerunning the backup task. + +#### Clean up remote uploaded files + +A [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45425) caused object store uploads to remain after a parent resource was deleted. This issue was [resolved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18698). + +To fix these files, you must clean up all remote uploaded files that are in the storage but not tracked in the `uploads` database table. + +1. List all the object store upload files that can be moved to a lost and found directory if they don't exist in the GitLab database: + + ```shell + bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production + ``` + +1. If you are sure you want to delete these files and remove all non-referenced uploaded files, run: + + WARNING: + The following action is **irreversible**. + + ```shell + bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production DRY_RUN=false + ``` + +#### Truncate the filenames referenced by the database + +You must truncate the files referenced by the database that are causing the problem. The filenames referenced by the database are stored: + +- In the `uploads` table. +- In the references found. Any reference found from other database tables and columns. +- On the file system. + +Truncate the filenames in the `uploads` table: + +1. Enter the database console: + + For the Linux package (Omnibus) GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For the Linux package (Omnibus) GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + + For installations from source, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + +1. Search the `uploads` table for filenames longer than 246 characters: + + The following query selects the `uploads` records with filenames longer than 246 characters in batches of 0 to 10000. This improves the performance on large GitLab instances with tables having thousand of records. + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, id, path + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + SELECT + u.id, + u.path, + -- Current filename + (regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] AS current_filename, + -- New filename + CONCAT( + LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) AS new_filename, + -- New path + CONCAT( + COALESCE((regexp_match(u.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) AS new_path + FROM uploads_with_long_filenames AS u + WHERE u.row_id > 0 AND u.row_id <= 10000; + ``` + + Output example: + + ```postgresql + -[ RECORD 1 ]----+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + id | 34 + path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt + current_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt + new_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt + new_path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt + ``` + + Where: + + - `current_filename`: a filename that is currently more than 246 characters long. + - `new_filename`: a filename that has been truncated to 246 characters maximum. + - `new_path`: new path considering the `new_filename` (truncated). + + After you validate the batch results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +1. Rename the files found in the `uploads` table from long filenames to new truncated filenames. The following query rolls back the update so you can check the results safely in a transaction wrapper: + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + BEGIN; + WITH updated_uploads AS ( + UPDATE uploads + SET + path = + CONCAT( + COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) + FROM + uploads_with_long_filenames AS updatable_uploads + WHERE + uploads.id = updatable_uploads.id + AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000 + RETURNING uploads.* + ) + SELECT id, path FROM updated_uploads; + ROLLBACK; + ``` + + After you validate the batch update results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +1. Validate that the new filenames from the previous query are the expected ones. If you are sure you want to truncate the records found in the previous step to 246 characters, run the following: + + WARNING: + The following action is **irreversible**. + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + UPDATE uploads + SET + path = + CONCAT( + COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) + FROM + uploads_with_long_filenames AS updatable_uploads + WHERE + uploads.id = updatable_uploads.id + AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000; + ``` + + After you finish the batch update, you must change the batch size (`updatable_uploads.row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +Truncate the filenames in the references found: + +1. Check if those records are referenced somewhere. One way to do this is to dump the database and search for the parent directory name and filename: + + 1. To dump your database, you can use the following command as an example: + + ```shell + pg_dump -h /var/opt/gitlab/postgresql/ -d gitlabhq_production > gitlab-dump.tmp + ``` + + 1. Then you can search for the references using the `grep` command. Combining the parent directory and the filename can be a good idea. For example: + + ```shell + grep public/alongfilenamehere.txt gitlab-dump.tmp + ``` + +1. Replace those long filenames using the new filenames obtained from querying the `uploads` table. + +Truncate the filenames on the file system. You must manually rename the files in your file system to the new filenames obtained from querying the `uploads` table. + +#### Re-run the backup task + +After following all the previous steps, re-run the backup task. + +### Restoring database backup fails when `pg_stat_statements` was previously enabled + +The GitLab backup of the PostgreSQL database includes all SQL statements required to enable extensions that were +previously enabled in the database. + +The `pg_stat_statements` extension can only be enabled or disabled by a PostgreSQL user with `superuser` role. +As the restore process uses a database user with limited permissions, it can't execute the following SQL statements: + +```sql +DROP EXTENSION IF EXISTS pg_stat_statements; +CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; +``` + +When trying to restore the backup in a PostgreSQL instance that doesn't have the `pg_stats_statements` extension, +the following error message is displayed: + +```plaintext +ERROR: permission denied to create extension "pg_stat_statements" +HINT: Must be superuser to create this extension. +ERROR: extension "pg_stat_statements" does not exist +``` + +When trying to restore in an instance that has the `pg_stats_statements` extension enabled, the cleaning up step +fails with an error message similar to the following: + +```plaintext +rake aborted! +ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Caused by: +PG::InsufficientPrivilege: ERROR: must be owner of view pg_stat_statements +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:42:in `block (4 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `each' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:41:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Tasks: TOP => gitlab:db:drop_tables +(See full trace by running task with --trace) +``` + +#### Prevent the dump file to include `pg_stat_statements` + +To prevent the inclusion of the extension in the PostgreSQL dump file that is part of the backup bundle, +enable the extension in any schema except the `public` schema: + +```sql +CREATE SCHEMA adm; +CREATE EXTENSION pg_stat_statements SCHEMA adm; +``` + +If the extension was previously enabled in the `public` schema, move it to a new one: + +```sql +CREATE SCHEMA adm; +ALTER EXTENSION pg_stat_statements SET SCHEMA adm; +``` + +To query the `pg_stat_statements` data after changing the schema, prefix the view name with the new schema: + +```sql +SELECT * FROM adm.pg_stat_statements limit 0; +``` + +To make it compatible with third-party monitoring solutions that expect it to be enabled in the `public` schema, +you need to include it in the `search_path`: + +```sql +set search_path to public,adm; +``` + +#### Fix an existing dump file to remove references to `pg_stat_statements` + +To fix an existing backup file, do the following changes: + +1. Extract from the backup the following file: `db/database.sql.gz`. +1. Decompress the file or use an editor that is capable of handling it compressed. +1. Remove the following lines, or similar ones: + + ```sql + CREATE EXTENSION IF NOT EXISTS pg_stat_statements WITH SCHEMA public; + ``` + + ```sql + COMMENT ON EXTENSION pg_stat_statements IS 'track planning and execution statistics of all SQL statements executed'; + ``` + +1. Save the changes and recompress the file. +1. Update the backup file with the modified `db/database.sql.gz`. diff --git a/doc/administration/backup_restore/index.md b/doc/administration/backup_restore/index.md new file mode 100644 index 00000000000..72a0176adc1 --- /dev/null +++ b/doc/administration/backup_restore/index.md @@ -0,0 +1,232 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Back up and restore GitLab **(FREE SELF)** + +Your software or organization depends on the data in your GitLab instance. You need to ensure this data is protected from adverse events such as: + +- Corrupted data +- Accidental deletion of data +- Ransomware attacks +- Unexpected cloud provider downtime + +You can mitigate all of these risks with a disaster recovery plan that includes backups. + +## Back up GitLab + +For detailed information on backing up GitLab, see [Backup GitLab](backup_gitlab.md). + +## Restore GitLab + +For detailed information on restoring GitLab, see [Restore GitLab](restore_gitlab.md). + +## Migrate to a new server + +<!-- some details borrowed from GitLab.com move from Azure to GCP detailed at https://gitlab.com/gitlab-com/migration/-/blob/master/.gitlab/issue_templates/failover.md --> + +You can use GitLab backup and restore to migrate your instance to a new server. This section outlines a typical procedure for a GitLab deployment running on a single server. +If you're running GitLab Geo, an alternative option is [Geo disaster recovery for planned failover](../geo/disaster_recovery/planned_failover.md). + +WARNING: +Avoid uncoordinated data processing by both the new and old servers, where multiple +servers could connect concurrently and process the same data. For example, when using +[incoming email](../incoming_email.md), if both GitLab instances are +processing email at the same time, then both instances miss some data. +This type of problem can occur with other services as well, such as a +[non-packaged database](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server), +a non-packaged Redis instance, or non-packaged Sidekiq. + +Prerequisites: + +- Some time before your migration, consider notifying your users of upcoming + scheduled maintenance with a [broadcast message banner](../broadcast_messages.md). +- Ensure your backups are complete and current. Create a complete system-level backup, or + take a snapshot of all servers involved in the migration, in case destructive commands + (like `rm`) are run incorrectly. + +### Prepare the new server + +To prepare the new server: + +1. Copy the + [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) + from the old server to avoid man-in-the-middle attack warnings. + See [Manually replicate the primary site's SSH host keys](../geo/replication/configuration.md#step-2-manually-replicate-the-primary-sites-ssh-host-keys) for example steps. +1. [Install and configure GitLab](https://about.gitlab.com/install/) except + [incoming email](../incoming_email.md): + 1. Install GitLab. + 1. Configure by copying `/etc/gitlab` files from the old server to the new server, and update as necessary. + Read the + [Omnibus configuration backup and restore instructions](https://docs.gitlab.com/omnibus/settings/backups.html) for more detail. + 1. If applicable, disable [incoming email](../incoming_email.md). + 1. Block new CI/CD jobs from starting upon initial startup after the backup and restore. + Edit `/etc/gitlab/gitlab.rb` and set the following: + + ```ruby + nginx['custom_gitlab_server_config'] = "location = /api/v4/jobs/request {\n deny all;\n return 503;\n }\n" + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Stop GitLab to avoid any potential unnecessary and unintentional data processing: + + ```shell + sudo gitlab-ctl stop + ``` + +1. Configure the new server to allow receiving the Redis database and GitLab backup files: + + ```shell + sudo rm -f /var/opt/gitlab/redis/dump.rdb + sudo chown <your-linux-username> /var/opt/gitlab/redis /var/opt/gitlab/backups + ``` + +### Prepare and transfer content from the old server + +1. Ensure you have an up-to-date system-level backup or snapshot of the old server. +1. Enable [maintenance mode](../maintenance_mode/index.md), + if supported by your GitLab edition. +1. Block new CI/CD jobs from starting: + 1. Edit `/etc/gitlab/gitlab.rb`, and set the following: + + ```ruby + nginx['custom_gitlab_server_config'] = "location = /api/v4/jobs/request {\n deny all;\n return 503;\n }\n" + ``` + + 1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Disable periodic background jobs: + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, select **Cron** tab and then + **Disable All**. +1. Wait for the currently running CI/CD jobs to finish, or accept that jobs that have not completed may be lost. + To view jobs currently running, on the left sidebar, select **Overviews > Jobs**, + and then select **Running**. +1. Wait for Sidekiq jobs to finish: + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, select **Queues** and then **Live Poll**. + Wait for **Busy** and **Enqueued** to drop to 0. + These queues contain work that has been submitted by your users; + shutting down before these jobs complete may cause the work to be lost. + Make note of the numbers shown in the Sidekiq dashboard for post-migration verification. +1. Flush the Redis database to disk, and stop GitLab other than the services needed for migration: + + ```shell + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket save && sudo gitlab-ctl stop && sudo gitlab-ctl start postgresql && sudo gitlab-ctl start gitaly + ``` + +1. Create a GitLab backup: + + ```shell + sudo gitlab-backup create + ``` + +1. Disable the following GitLab services and prevent unintentional restarts by adding the following to the bottom of `/etc/gitlab/gitlab.rb`: + + ```ruby + alertmanager['enable'] = false + gitlab_exporter['enable'] = false + gitlab_pages['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + logrotate['enable'] = false + gitlab_rails['incoming_email_enabled'] = false + nginx['enable'] = false + node_exporter['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + puma['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + registry['enable'] = false + sidekiq['enable'] = false + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Verify everything is stopped, and confirm no services are running: + + ```shell + sudo gitlab-ctl status + ``` + +1. Transfer the Redis database and GitLab backups to the new server: + + ```shell + sudo scp /var/opt/gitlab/redis/dump.rdb <your-linux-username>@new-server:/var/opt/gitlab/redis + sudo scp /var/opt/gitlab/backups/your-backup.tar <your-linux-username>@new-server:/var/opt/gitlab/backups + ``` + +### Restore data on the new server + +1. Restore appropriate file system permissions: + + ```shell + sudo chown gitlab-redis /var/opt/gitlab/redis + sudo chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis/dump.rdb + sudo chown git:root /var/opt/gitlab/backups + sudo chown git:git /var/opt/gitlab/backups/your-backup.tar + ``` + +1. [Restore the GitLab backup](#restore-gitlab). +1. Verify that the Redis database restored correctly: + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. + 1. On the left sidebar, select **Monitoring > Background Jobs**. + 1. Under the Sidekiq dashboard, verify that the numbers + match with what was shown on the old server. + 1. While still under the Sidekiq dashboard, select **Cron** and then **Enable All** + to re-enable periodic background jobs. +1. Test that read-only operations on the GitLab instance work as expected. For example, browse through project repository files, merge requests, and issues. +1. Disable [Maintenance Mode](../maintenance_mode/index.md), if previously enabled. +1. Test that the GitLab instance is working as expected. +1. If applicable, re-enable [incoming email](../incoming_email.md) and test it is working as expected. +1. Update your DNS or load balancer to point at the new server. +1. Unblock new CI/CD jobs from starting by removing the custom NGINX configuration + you added previously: + + ```ruby + # The following line must be removed + nginx['custom_gitlab_server_config'] = "location = /api/v4/jobs/request {\n deny all;\n return 503;\n }\n" + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Remove the scheduled maintenance [broadcast message banner](../broadcast_messages.md). + +## Additional notes + +This documentation is for GitLab Community and Enterprise Edition. We back up +GitLab.com and ensure your data is secure. You can't, however, use these +methods to export or back up your data yourself from GitLab.com. + +Issues are stored in the database, and can't be stored in Git itself. + +## Related features + +- [Geo](../geo/index.md) +- [Disaster Recovery (Geo)](../geo/disaster_recovery/index.md) +- [Migrating GitLab groups](../../user/group/import/index.md) +- [Import and migrate projects](../../user/project/import/index.md) diff --git a/doc/administration/backup_restore/restore_gitlab.md b/doc/administration/backup_restore/restore_gitlab.md new file mode 100644 index 00000000000..2cc0c68c66b --- /dev/null +++ b/doc/administration/backup_restore/restore_gitlab.md @@ -0,0 +1,432 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Restore GitLab **(FREE SELF)** + +GitLab provides a command line interface to restore your entire installation, +and is flexible enough to fit your needs. + +The [restore prerequisites section](#restore-prerequisites) includes crucial +information. Be sure to read and test the complete restore process at least +once before attempting to perform it in a production environment. + +NOTE: +You can only restore a backup to **exactly the same version and type (CE/EE)** +of GitLab on which it was created (for example CE 15.1.4). + +If your backup is a different version than the current installation, you must +[downgrade](../../update/package/downgrade.md) or [upgrade](../../update/package/index.md#upgrade-to-a-specific-version-using-the-official-repositories) your GitLab installation +before restoring the backup. + +Each backup archive contains a full self-contained backup, including those created through the [incremental repository backup procedure](backup_gitlab.md#incremental-repository-backups). To restore an incremental repository backup, use the same instructions as restoring any other regular backup archive. + +## Restore prerequisites + +You need to have a working GitLab installation before you can perform a +restore. This is because the system user performing the restore actions (`git`) +is usually not allowed to create or delete the SQL database needed to import +data into (`gitlabhq_production`). All existing data is either erased +(SQL) or moved to a separate directory (such as repositories and uploads). +Restoring SQL data skips views owned by PostgreSQL extensions. + +To restore a backup, **you must also restore the GitLab secrets**. +These include the database encryption key, [CI/CD variables](../../ci/variables/index.md), and +variables used for [two-factor authentication](../../user/profile/account/two_factor_authentication.md). +Without the keys, [multiple issues occur](backup_gitlab.md#when-the-secrets-file-is-lost), +including loss of access by users with [two-factor authentication enabled](../../user/profile/account/two_factor_authentication.md), +and GitLab Runners cannot log in. + +Restore: + +- `/etc/gitlab/gitlab-secrets.json` (Linux package) +- `/home/git/gitlab/.secret` (self-compiled installations) +- Rails secret (cloud-native GitLab) + - [This can be converted to the Linux package format](https://docs.gitlab.com/charts/installation/migration/helm_to_package.html), if required. + +You may also want to restore your previous `/etc/gitlab/gitlab.rb` (for Omnibus packages) +or `/home/git/gitlab/config/gitlab.yml` (for installations from source) and +any TLS keys, certificates (`/etc/gitlab/ssl`, `/etc/gitlab/trusted-certs`), or +[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). + +Depending on your case, you might want to run the restore command with one or +more of the following options: + +- `BACKUP=timestamp_of_backup`: Required if more than one backup exists. + Read what the [backup timestamp is about](backup_gitlab.md#backup-timestamp). +- `force=yes`: Doesn't ask if the `authorized_keys` file should get regenerated, + and assumes 'yes' for warning about database tables being removed, + enabling the `Write to authorized_keys file` setting, and updating LDAP + providers. + +If you're restoring into directories that are mount points, you must ensure these directories are +empty before attempting a restore. Otherwise, GitLab attempts to move these directories before +restoring the new data, which causes an error. + +Read more about [configuring NFS mounts](../nfs.md) + +Restoring a backup from an instance using local storage restores to local storage even if the target instance uses object storage. +Migrations to object storage must be done before or after restoration. + +## Restore for Omnibus GitLab installations + +This procedure assumes that: + +- You have installed the **exact same version and type (CE/EE)** of GitLab + Omnibus with which the backup was created. +- You have run `sudo gitlab-ctl reconfigure` at least once. +- GitLab is running. If not, start it using `sudo gitlab-ctl start`. + +First ensure your backup tar file is in the backup directory described in the +`gitlab.rb` configuration `gitlab_rails['backup_path']`. The default is +`/var/opt/gitlab/backups`. The backup file needs to be owned by the `git` user. + +```shell +sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups/ +sudo chown git:git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar +``` + +Stop the processes that are connected to the database. Leave the rest of GitLab +running: + +```shell +sudo gitlab-ctl stop puma +sudo gitlab-ctl stop sidekiq +# Verify +sudo gitlab-ctl status +``` + +Next, ensure you have completed the [restore prerequisites](#restore-prerequisites) steps and have run `gitlab-ctl reconfigure` +after copying over the GitLab secrets file from the original installation. + +Next, restore the backup, specifying the timestamp of the backup you wish to +restore: + +```shell +# This command will overwrite the contents of your GitLab database! +# NOTE: "_gitlab_backup.tar" is omitted from the name +sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce +``` + +If there's a GitLab version mismatch between your backup tar file and the +installed version of GitLab, the restore command aborts with an error +message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/), +and then try again. + +WARNING: +The restore command requires [additional parameters](backup_gitlab.md#back-up-and-restore-for-installations-using-pgbouncer) when +your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster. + +Next, restart and [check](../raketasks/maintenance.md#check-gitlab-configuration) GitLab: + +```shell +sudo gitlab-ctl restart +sudo gitlab-rake gitlab:check SANITIZE=true +``` + +In GitLab 13.1 and later, check [database values can be decrypted](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets) +especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is +the target for the restore. + +```shell +sudo gitlab-rake gitlab:doctor:secrets +``` + +For added assurance, you can perform [an integrity check on the uploaded files](../raketasks/check.md#uploaded-files-integrity): + +```shell +sudo gitlab-rake gitlab:artifacts:check +sudo gitlab-rake gitlab:lfs:check +sudo gitlab-rake gitlab:uploads:check +``` + +## Restore for Docker image and GitLab Helm chart installations + +For GitLab installations using the Docker image or the GitLab Helm chart on a +Kubernetes cluster, the restore task expects the restore directories to be +empty. However, with Docker and Kubernetes volume mounts, some system level +directories may be created at the volume roots, such as the `lost+found` +directory found in Linux operating systems. These directories are usually owned +by `root`, which can cause access permission errors since the restore Rake task +runs as the `git` user. To restore a GitLab installation, users have to confirm +the restore target directories are empty. + +For both these installation types, the backup tarball has to be available in +the backup location (default location is `/var/opt/gitlab/backups`). + +### Restore for Helm chart installations + +The GitLab Helm chart uses the process documented in +[restoring a GitLab Helm chart installation](https://docs.gitlab.com/charts/backup-restore/restore.html#restoring-a-gitlab-installation) + +### Restore for Docker image installations + +If you're using [Docker Swarm](../../install/docker.md#install-gitlab-using-docker-swarm-mode), +the container might restart during the restore process because Puma is shut down, +and so the container health check fails. To work around this problem, +temporarily disable the health check mechanism. + +1. Edit `docker-compose.yml`: + + ```yaml + healthcheck: + disable: true + ``` + +1. Deploy the stack: + + ```shell + docker stack deploy --compose-file docker-compose.yml mystack + ``` + +For more information, see [issue 6846](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6846 "GitLab restore can fail owing to `gitlab-healthcheck`"). + +The restore task can be run from the host: + +```shell +# Stop the processes that are connected to the database +docker exec -it <name of container> gitlab-ctl stop puma +docker exec -it <name of container> gitlab-ctl stop sidekiq + +# Verify that the processes are all down before continuing +docker exec -it <name of container> gitlab-ctl status + +# Run the restore. NOTE: "_gitlab_backup.tar" is omitted from the name +docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce + +# Restart the GitLab container +docker restart <name of container> + +# Check GitLab +docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true +``` + +## Restore for installation from source + +First, ensure your backup tar file is in the backup directory described in the +`gitlab.yml` configuration: + +```yaml +## Backup settings +backup: + path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/) +``` + +The default is `/home/git/gitlab/tmp/backups`, and it needs to be owned by the `git` user. Now, you can begin the backup procedure: + +```shell +# Stop processes that are connected to the database +sudo service gitlab stop + +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +Example output: + +```plaintext +Unpacking backup... [DONE] +Restoring database tables: +-- create_table("events", {:force=>true}) + -> 0.2231s +[...] +- Loading fixture events...[DONE] +- Loading fixture issues...[DONE] +- Loading fixture keys...[SKIPPING] +- Loading fixture merge_requests...[DONE] +- Loading fixture milestones...[DONE] +- Loading fixture namespaces...[DONE] +- Loading fixture notes...[DONE] +- Loading fixture projects...[DONE] +- Loading fixture protected_branches...[SKIPPING] +- Loading fixture schema_migrations...[DONE] +- Loading fixture services...[SKIPPING] +- Loading fixture snippets...[SKIPPING] +- Loading fixture taggings...[SKIPPING] +- Loading fixture tags...[SKIPPING] +- Loading fixture users...[DONE] +- Loading fixture users_projects...[DONE] +- Loading fixture web_hooks...[SKIPPING] +- Loading fixture wikis...[SKIPPING] +Restoring repositories: +- Restoring repository abcd... [DONE] +- Object pool 1 ... +Deleting tmp directories...[DONE] +``` + +Next, restore `/home/git/gitlab/.secret` if necessary, [as previously mentioned](#restore-prerequisites). + +Restart GitLab: + +```shell +sudo service gitlab restart +``` + +## Restoring only one or a few projects or groups from a backup + +Although the Rake task used to restore a GitLab instance doesn't support +restoring a single project or group, you can use a workaround by restoring +your backup to a separate, temporary GitLab instance, and then export your +project or group from there: + +1. [Install a new GitLab](../../install/index.md) instance at the same version as + the backed-up instance from which you want to restore. +1. [Restore the backup](#restore-gitlab) into this new instance, then + export your [project](../../user/project/settings/import_export.md) + or [group](../../user/group/import/index.md#migrate-groups-by-uploading-an-export-file-deprecated). For more information about what is and isn't exported, see the export feature's documentation. +1. After the export is complete, go to the old instance and then import it. +1. After importing the projects or groups that you wanted is complete, you may + delete the new, temporary GitLab instance. + +A feature request to provide direct restore of individual projects or groups +is being discussed in [issue #17517](https://gitlab.com/gitlab-org/gitlab/-/issues/17517). + +## Restore options + +The command line tool GitLab provides to restore from backup can accept more +options. + +### Disabling prompts during restore + +During a restore from backup, the restore script may ask for confirmation before +proceeding. If you wish to disable these prompts, you can set the `GITLAB_ASSUME_YES` +environment variable to `1`. + +For Omnibus GitLab packages: + +```shell +sudo GITLAB_ASSUME_YES=1 gitlab-backup restore +``` + +For installations from source: + +```shell +sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +### Excluding tasks on restore + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19347) in GitLab 14.10. + +You can exclude specific tasks on restore by adding the environment variable `SKIP`, whose values are a comma-separated list of the following options: + +- `db` (database) +- `uploads` (attachments) +- `builds` (CI job output logs) +- `artifacts` (CI job artifacts) +- `lfs` (LFS objects) +- `terraform_state` (Terraform states) +- `registry` (Container Registry images) +- `pages` (Pages content) +- `repositories` (Git repositories data) +- `packages` (Packages) + +For Omnibus GitLab packages: + +```shell +sudo gitlab-backup restore BACKUP=timestamp_of_backup SKIP=db,uploads +``` + +For installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup SKIP=db,uploads RAILS_ENV=production +``` + +### Restore specific repository storages + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86896) in GitLab 15.0. + +When using [multiple repository storages](../repository_storage_paths.md), +repositories from specific repository storages can be restored separately +using the `REPOSITORIES_STORAGES` option. The option accepts a comma-separated list of +storage names. + +For example, for Omnibus GitLab installations: + +```shell +sudo gitlab-backup restore BACKUP=timestamp_of_backup REPOSITORIES_STORAGES=storage1,storage2 +``` + +For example, for installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup REPOSITORIES_STORAGES=storage1,storage2 +``` + +### Restore specific repositories + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. + +You can restore specific repositories using the `REPOSITORIES_PATHS` and the `SKIP_REPOSITORIES_PATHS` options. +Both options accept a comma-separated list of project and group paths. If you +specify a group path, all repositories in all projects in the group and +descendent groups are included or skipped, depending on which option you used. The project and group repositories must exist within the specified backup. + +For example, to restore all repositories for all projects in **Group A** (`group-a`), the repository for **Project C** in **Group B** (`group-b/project-c`), +and skip the **Project D** in **Group A** (`group-a/project-d`): + +- Omnibus GitLab installations: + + ```shell + sudo gitlab-backup restore BACKUP=timestamp_of_backup REPOSITORIES_PATHS=group-a,group-b/project-c SKIP_REPOSITORIES_PATHS=group-a/project-d + ``` + +- Installations from source: + + ```shell + sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup REPOSITORIES_PATHS=group-a,group-b/project-c SKIP_REPOSITORIES_PATHS=group-a/project-d + ``` + +### Restore untarred backups + +If an [untarred backup](backup_gitlab.md#skipping-tar-creation) (made with `SKIP=tar`) is found, +and no backup is chosen with `BACKUP=<timestamp>`, the untarred backup is used. + +For example, for Omnibus GitLab installations: + +```shell +sudo gitlab-backup restore +``` + +For example, for installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:restore +``` + +## Troubleshooting + +The following are possible problems you might encounter, along with potential +solutions. + +### Restoring database backup using Omnibus packages outputs warnings + +If you're using backup restore procedures, you may encounter the following +warning messages: + +```plaintext +ERROR: must be owner of extension pg_trgm +ERROR: must be owner of extension btree_gist +ERROR: must be owner of extension plpgsql +WARNING: no privileges could be revoked for "public" (two occurrences) +WARNING: no privileges were granted for "public" (two occurrences) +``` + +Be advised that the backup is successfully restored in spite of these warning +messages. + +The Rake task runs this as the `gitlab` user, which doesn't have superuser +access to the database. When restore is initiated, it also runs as the `gitlab` +user, but it also tries to alter the objects it doesn't have access to. +Those objects have no influence on the database backup or restore, but display +a warning message. + +For more information, see: + +- PostgreSQL issue tracker: + - [Not being a superuser](https://www.postgresql.org/message-id/201110220712.30886.adrian.klaver@gmail.com). + - [Having different owners](https://www.postgresql.org/message-id/2039.1177339749@sss.pgh.pa.us). + +- Stack Overflow: [Resulting errors](https://stackoverflow.com/questions/4368789/error-must-be-owner-of-language-plpgsql). diff --git a/doc/administration/broadcast_messages.md b/doc/administration/broadcast_messages.md new file mode 100644 index 00000000000..556edbd3e5e --- /dev/null +++ b/doc/administration/broadcast_messages.md @@ -0,0 +1,120 @@ +--- +stage: Growth +group: Acquisition +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# Broadcast messages **(FREE SELF)** + +> - Target roles [introduced](https://gitlab.com/gitlab-org/growth/team-tasks/-/issues/461) in GitLab 14.8 [with a flag](../administration/feature_flags.md) named `role_targeted_broadcast_messages`. Disabled by default. +> - Theme [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83251) and background color removed in GitLab 14.10. + +GitLab can display broadcast messages to users of a GitLab instance. There are two types of broadcast messages: + +- Banners +- Notifications + +Broadcast messages can be managed using the [broadcast messages API](../api/broadcast_messages.md). + +## Banners + +Banners are shown on the top of a page and optionally in the command line as a Git remote response. + +![Broadcast Message Banner](img/broadcast_messages_banner_v15_0.png) + +```shell +$ git push +... +remote: +remote: **Welcome** to GitLab :wave: +remote: +... +``` + +If more than one banner is active at one time, they are displayed at the top of the page in order of creation. In the command line, only the latest banner is shown. + +## Notifications + +Notifications are shown on the bottom right of a page and can contain placeholders. A placeholder is replaced with an attribute of the active user. Placeholders must be surrounded by curly braces, for example `{{name}}`. +The available placeholders are: + +- `{{email}}` +- `{{name}}` +- `{{user_id}}` +- `{{username}}` +- `{{instance_id}}` + +If the user is not signed in, user related values are empty. + +![Broadcast Message Notification](img/broadcast_messages_notification_v12_10.png) + +If more than one notification is active at one time, only the newest is shown. + +## Add a broadcast message + +To display messages to users on your GitLab instance, add a broadcast message. + +To add a broadcast message: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Messages**. +1. Add the text for the message to the **Message** field. You can style a message's content using Markdown, emoji, and the `a` and `br` HTML tags. + The `br` tag inserts a line break. The `a` HTML tag accepts `class` and `style` attributes with the following CSS properties: + - `color` + - `border` + - `background` + - `padding` + - `margin` + - `text-decoration` +1. Select a **Theme**. The default theme is `indigo`. +1. Select the **Dismissable** checkbox to enable users to dismiss the broadcast message. +1. Optional. Clear the **Git remote responses** checkbox to prevent broadcast messages from being displayed in the command line as Git remote responses. +1. Optional. Select **Target roles** to only show the broadcast message to users with the selected roles. The message displays on group, subgroup, and project pages, and does not display in Git remote responses. +1. If required, add a **Target Path** to only show the broadcast message on URLs matching that path. You can use the wildcard character `*` to match multiple URLs, for example `mygroup/myproject*`. +1. Select a date and time (UTC) for the message to start and end. +1. Select **Add broadcast message**. + +When a broadcast message expires, it no longer displays in the user interface but is still listed in the +list of broadcast messages. + +## Edit a broadcast message + +If you must make changes to a broadcast message, you can edit it. + +To edit a broadcast message: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Messages**. +1. From the list of broadcast messages, select the edit button for the message. +1. After making the required changes, select **Update broadcast message**. + +Expired messages can be made active again by changing their end date. + +## Delete a broadcast message + +If you no longer require a broadcast message, you can delete it. +You can delete a broadcast message while it's active. + +To delete a broadcast message: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Messages**. +1. From the list of broadcast messages, select the delete button for the message. + +When a broadcast message is deleted, it's removed from the list of broadcast messages. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index b44c9571715..583a6401c05 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -142,10 +142,9 @@ For details, see [how to use the GitLab-KAS chart](https://docs.gitlab.com/chart ## Kubernetes API proxy cookie -> Introduced in GitLab 15.10 [with feature flags](../feature_flags.md) named `kas_user_access` and `kas_user_access_project`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flags](../feature_flags.md) named `kas_user_access` and `kas_user_access_project`. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104504) in GitLab 15.10 [with feature flags](../feature_flags.md) named `kas_user_access` and `kas_user_access_project`. Disabled by default. +> - Feature flags `kas_user_access` and `kas_user_access_project` [enabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/123479) in GitLab 16.1. +> - Feature flags `kas_user_access` and `kas_user_access_project` [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/125835) in GitLab 16.2. KAS proxies Kubernetes API requests to the GitLab agent with either: diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 79133d5a6c7..978e43b2e2c 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -20,7 +20,7 @@ and secure supply chain best practices: | Feature | Instances | Groups | Projects | Description | |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Credentials inventory](../user/admin_area/credentials_inventory.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Keep track of the credentials used by all of the users in a GitLab instance. | +| [Credentials inventory](credentials_inventory.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Keep track of the credentials used by all of the users in a GitLab instance. | | [Granular user roles<br/>and flexible permissions](../user/permissions.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker. | | [Merge request approvals](../user/project/merge_requests/approvals/index.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Configure approvals required for merge requests. | | [Push rules](../user/project/repository/push_rules.md) | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Control pushes to your repositories. | @@ -65,10 +65,10 @@ These features can also help with compliance requirements: | Feature | Instances | Groups | Projects | Description | |:------------------------------------------------------------------------------------------------------------------------------------|:-----------------------|:-----------------------|:-----------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Email all users of a project,<br/>group, or entire server](../user/admin_area/email_from_gitlab.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Email groups of users based on project or group membership, or email everyone using the GitLab instance. These emails are great for scheduled maintenance or upgrades. | -| [Enforce ToS acceptance](../user/admin_area/settings/terms.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Enforce your users accepting new terms of service by blocking GitLab traffic. | +| [Email all users of a project,<br/>group, or entire server](email_from_gitlab.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Email groups of users based on project or group membership, or email everyone using the GitLab instance. These emails are great for scheduled maintenance or upgrades. | +| [Enforce ToS acceptance](settings/terms.md) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Enforce your users accepting new terms of service by blocking GitLab traffic. | | [External Status Checks](../user/project/merge_requests/status_checks.md) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Interface with third-party systems you already use during development to ensure you remain compliant. | -| [Generate reports on permission<br/>levels of users](../user/admin_area/index.md#user-permission-export) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Generate a report listing all users' access permissions for groups and projects in the instance. | +| [Generate reports on permission<br/>levels of users](../administration/admin_area.md#user-permission-export) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Generate a report listing all users' access permissions for groups and projects in the instance. | | [License compliance](../user/compliance/license_compliance/index.md) | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Search dependencies for their licenses. This lets you determine if the licenses of your project's dependencies are compatible with your project's license. | | [Lock project membership to group](../user/group/access_and_permissions.md#prevent-members-from-being-added-to-projects-in-a-group) | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Group owners can prevent new members from being added to projects in a group. | | [LDAP group sync](auth/ldap/ldap_synchronization.md#group-sync) | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Automatically synchronize groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools. | diff --git a/doc/administration/configure.md b/doc/administration/configure.md index d68e81ebf69..8ff2ae5aa9d 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -11,7 +11,7 @@ Customize and configure your self-managed GitLab installation. - [Authentication](auth/index.md) - [CI/CD](../administration/cicd.md) -- [Configuration](../user/admin_area/index.md) +- [Configuration](../administration/admin_area.md) - [Consul](../administration/consul.md) - [Environment variables](../administration/environment_variables.md) - [File hooks](../administration/file_hooks.md) @@ -35,7 +35,7 @@ Customize and configure your self-managed GitLab installation. - [Agent server for Kubernetes](../administration/clusters/kas.md) - [Server hooks](../administration/server_hooks.md) - [Terraform state](../administration/terraform_state.md) -- [Terraform limits](../user/admin_area/settings/terraform_limits.md) +- [Terraform limits](settings/terraform_limits.md) - [Packages](../administration/packages/index.md) - [Web terminals](../administration/integration/terminal.md) - [Wikis](../administration/wikis/index.md) diff --git a/doc/administration/credentials_inventory.md b/doc/administration/credentials_inventory.md new file mode 100644 index 00000000000..482d46498e3 --- /dev/null +++ b/doc/administration/credentials_inventory.md @@ -0,0 +1,86 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto +--- + +# Credentials inventory **(ULTIMATE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20912) in GitLab 12.6. +> - [Bot-created access tokens not displayed in personal access token list](https://gitlab.com/gitlab-org/gitlab/-/issues/351759) in GitLab 14.9. + +GitLab administrators are responsible for the overall security of their instance. To assist, GitLab +provides a Credentials inventory to keep track of all the credentials that can be used to access +their self-managed instance. + +Use Credentials inventory to see for your GitLab instance all: + +- Personal access tokens (PAT). +- Project access tokens (GitLab 14.8 and later). +- SSH keys. +- GPG keys. + +You can also [revoke](#revoke-a-users-personal-access-token) and [delete](#delete-a-users-ssh-key) and see: + +- Who they belong to. +- Their access scope. +- Their usage pattern. +- When they expire. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214809) in GitLab 13.2. +- When they were revoked. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214809) in GitLab 13.2. + +To access the Credentials inventory: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Credentials**. + +## Revoke a user's personal access token + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214811) in GitLab 13.4. + +If you see a **Revoke** button, you can revoke that user's PAT. Whether you see a **Revoke** button depends on the token state, and if an expiration date has been set. For more information, see the following table: + +| Token state | Show Revoke button? | Comments | +|-------------|---------------------|----------------------------------------------------------------------------| +| Active | Yes | Allows administrators to revoke the PAT, such as for a compromised account | +| Expired | No | Not applicable; token is already expired | +| Revoked | No | Not applicable; token is already revoked | + +When a PAT is revoked from the credentials inventory, the instance notifies the user by email. + +![Credentials inventory page - Personal access tokens](img/credentials_inventory_personal_access_tokens_v14_9.png) + +## Revoke a user's project access token + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/243833) in GitLab 14.8. + +The **Revoke** button next to a project access token can be selected to revoke that particular project access token. This both: + +- Revokes the token project access token. +- Enqueues a background worker to delete the project bot user. + +![Credentials inventory page - Project access tokens](img/credentials_inventory_project_access_tokens_v14_9.png) + +## Delete a user's SSH key + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225248) in GitLab 13.5. + +You can **Delete** a user's SSH key by navigating to the credentials inventory's SSH Keys tab. +The instance then notifies the user. + +![Credentials inventory page - SSH keys](img/credentials_inventory_ssh_keys_v14_9.png) + +## Review existing GPG keys + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/282429) in GitLab 13.10. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292961) in GitLab 13.12. + +You can view all existing GPG in your GitLab instance by navigating to the +credentials inventory GPG Keys tab, as well as the following properties: + +- Who the GPG key belongs to. +- The ID of the GPG key. +- Whether the GPG key is [verified or unverified](../user/project/repository/gpg_signed_commits/index.md) + +![Credentials inventory page - GPG keys](img/credentials_inventory_gpg_keys_v14_9.png) diff --git a/doc/administration/custom_project_templates.md b/doc/administration/custom_project_templates.md new file mode 100644 index 00000000000..2bbbb5649e6 --- /dev/null +++ b/doc/administration/custom_project_templates.md @@ -0,0 +1,73 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Custom instance-level project templates **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6860) in GitLab 11.2. + +GitLab administrators can set a group to be the source of project templates that are +selectable when a new project is created on the instance. These templates can be selected +when you go to **New project > Create from template** and select the **Instance** tab. + +Every project in the group, but not its subgroups, can be selected when a new project +is created, based on the user's access permissions: + +- Public projects can be selected by any authenticated user as a template for a new project, + if all enabled [project features](../user/project/settings/index.md#configure-project-visibility-features-and-permissions) + except for **GitLab Pages** and **Security and Compliance** are set to **Everyone With Access**. + The same applies to internal projects. +- Private projects can be selected only by users who are members of the projects. + +The **Metrics Dashboard** is set to **Only Project Members** when you create a new project. Make +sure you change it to **Everyone With Access** before making it a project template. + +Repository and database information that are copied over to each new project are +identical to the data exported with the [GitLab Project Import/Export](../user/project/settings/import_export.md). + +To set project templates at the group level, see [Custom group-level project templates](../user/group/custom_project_templates.md). + +## Select instance-level project template group + +To select the group to use as the source for the project templates: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Templates**. +1. Expand **Custom project templates**. +1. Select a group to use. +1. Select **Save changes**. + +Projects in subgroups of the template group are **not** included in the template list. + +## What is copied from the templates + +The entire custom instance-level project templates repository is copied, including: + +- Branches +- Commits +- Tags + +If the user: + +- Has the Owner role on the custom instance-level project templates project or is a GitLab administrator, all project settings are copied over to the new + project. +- Doesn't have the Owner role or is not a GitLab administrator, project [deploy keys](../user/project/deploy_keys/index.md#view-deploy-keys) and project + [webhooks](../user/project/integrations/webhooks.md) aren't copied over because they contain sensitive data. + +To learn more about what is migrated, see +[Items that are exported](../user/project/settings/import_export.md#items-that-are-exported). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/dedicated/index.md b/doc/administration/dedicated/index.md index f2caa5fa368..db30684ab1c 100644 --- a/doc/administration/dedicated/index.md +++ b/doc/administration/dedicated/index.md @@ -14,7 +14,11 @@ The instructions on this page guide you through: 1. Onboarding and initial setup of your GitLab Dedicated instance. 1. Configuring your GitLab Dedicated instance including enabling and updating the settings for [available functionality](../../subscriptions/gitlab_dedicated/index.md#available-features). -Any functionality in the GitLab application that is not controlled by the SaaS environment can be configured by using the [Admin Panel](../../user/admin_area/index.md). +Any functionality in the GitLab application that is not controlled by the SaaS environment can be configured by using the [Admin Panel](../../administration/admin_area.md). + +Examples of SaaS environment settings include `gitlab.rb` configurations and access to shell, Rails console, and PostgreSQL console. +These environment settings cannot be changed by tenants. +GitLab Dedicated Engineers also don't have direct access to tenant environments, except for [break glass situations](../../subscriptions/gitlab_dedicated/index.md#access-controls). ## Onboarding @@ -68,8 +72,9 @@ To create a KMS key using the AWS Console: 1. Key material origin: **KMS** 1. Regionality: **Multi-Region key** 1. Enter your values for key alias, description, and tags. -1. Select Key administrators (optionally allow or deny key administrators to delete the key). -1. For Key usage permissions, add the GitLab AWS account using the **Other AWS accounts** dialog. +1. Select key administrators. +1. Optional. Allow or prevent key administrators from deleting the key. +1. On the **Define key usage permissions** page, under **Other AWS accounts**, add the GitLab AWS account. The last page asks you to confirm the KMS key policy. It should look similar to the following example, populated with your account IDs and usernames: diff --git a/doc/administration/diff_limits.md b/doc/administration/diff_limits.md new file mode 100644 index 00000000000..48b52950ee3 --- /dev/null +++ b/doc/administration/diff_limits.md @@ -0,0 +1,53 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Diff limits administration **(FREE SELF)** + +You can set a maximum size for display of diff files (patches). + +For details about diff files, [view changes between files](../user/project/merge_requests/changes.md). +Read more about the [built-in limits for merge requests and diffs](../administration/instance_limits.md#merge-requests). + +## Configure diff limits + +WARNING: +These settings are experimental. An increased maximum increases resource +consumption of your instance. Keep this in mind when adjusting the maximum. + +To speed the loading time of merge request views and branch comparison views +on your instance, you can configure three instance-level maximum values for diffs: + +| Value | Definition | Default value | Maximum value | +| ----- | ---------- | :-----------: | :-----------: | +| **Maximum diff patch size** | The total size, in bytes, of the entire diff. | 200 KB | 500 KB | +| **Maximum diff files** | The total number of files changed in a diff. | 1000 | 3000 | +| **Maximum diff lines** | The total number of lines changed in a diff. | 50,000 | 100,000 | + +When a diff reaches 10% of any of these values, the files are shown in a +collapsed view, with a link to expand the diff. Diffs that exceed any of the +set values are presented as **Too large** are cannot be expanded in the UI. + +To configure these values: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Diff limits**. +1. Enter a value for the diff limit. +1. Select **Save changes**. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 54f8c15a922..1eecc139d1f 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -163,7 +163,7 @@ To extract the HTML files of the documentation site: ## Redirect the `/help` links to the new Docs site After your local product documentation site is running, -[redirect the help links](../user/admin_area/settings/help_page.md#redirect-help-pages) +[redirect the help links](../administration/settings/help_page.md#redirect-help-pages) in the GitLab application to your local site, by using the fully qualified domain name as the documentation URL. For example, if you used the [Docker method](#self-host-the-product-documentation-with-docker), enter `http://0.0.0.0:4000`. @@ -173,9 +173,9 @@ documentation URL requests as needed. For example, if your GitLab version is 14.5: - The GitLab documentation URL becomes `http://0.0.0.0:4000/14.5/`. -- The link in GitLab displays as `<instance_url>/help/user/admin_area/settings/help_page#destination-requirements`. +- The link in GitLab displays as `<instance_url>/help/administration/settings/help_page#destination-requirements`. - When you select the link, you are redirected to -`http://0.0.0.0:4000/14.5/ee/user/admin_area/settings/help_page/#destination-requirements`. +`http://0.0.0.0:4000/14.5/ee/administration/settings/help_page/#destination-requirements`. To test the setting, in GitLab, select a **Learn more** link. For example: diff --git a/doc/administration/email_from_gitlab.md b/doc/administration/email_from_gitlab.md new file mode 100644 index 00000000000..9272f58d2b9 --- /dev/null +++ b/doc/administration/email_from_gitlab.md @@ -0,0 +1,61 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto, reference +--- + +# Email from GitLab **(PREMIUM SELF)** + +GitLab provides a tool to administrators for emailing all users, or users of +a chosen group or project, right from the Admin Area. Users receive the email +at their primary email address. + +For information about email notifications originating from GitLab, read +[GitLab notification emails](../user/profile/notifications.md). + +## Use-cases + +- Notify your users about a new project, a new feature, or a new product launch. +- Notify your users about a new deployment, or that downtime is expected + for a particular reason. + +## Sending emails to users from GitLab + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select **Send email to users**. + + ![administrators](img/email1.png) + +1. Compose an email and choose where to send it (all users or users of a + chosen group or project). The email body only supports plain text messages. + HTML, Markdown, and other rich text formats are not supported, and is + sent as plain text to users. + + ![compose an email](img/email2.png) + +NOTE: +[Starting with GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/31509), email notifications can be sent only once every 10 minutes. This helps minimize performance issues. + +## Unsubscribing from emails + +Users can choose to unsubscribe from receiving emails from GitLab by following +the unsubscribe link in the email. Unsubscribing is unauthenticated in order +to keep this feature simple. + +On unsubscribe, users receive an email notification that unsubscribe happened. +The endpoint that provides the unsubscribe option is rate-limited. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/external_users.md b/doc/administration/external_users.md new file mode 100644 index 00000000000..f8ca379d10c --- /dev/null +++ b/doc/administration/external_users.md @@ -0,0 +1,80 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# External users **(FREE SELF)** + +In cases where it is desired that a user has access only to some internal or +private projects, there is the option of creating **External Users**. This +feature may be useful when for example a contractor is working on a given +project and should only have access to that project. + +External users: + +- Cannot create project, groups, and snippets in their personal namespaces. +- Can only create projects (including forks), subgroups, and snippets within top-level groups to which they are explicitly granted access. +- Can only access public projects and projects to which they are explicitly granted access, + thus hiding all other internal or private ones from them (like being + logged out). +- Can only access public groups and groups to which they are explicitly granted access, + thus hiding all other internal or private ones from them (like being + logged out). +- Can only access public snippets. + +Access can be granted by adding the user as member to the project or group. +Like usual users, they receive a role in the project or group with all +the abilities that are mentioned in the [permissions table](../user/permissions.md#project-members-permissions). +For example, if an external user is added as Guest, and your project is internal or +private, they do not have access to the code; you need to grant the external +user access at the Reporter level or above if you want them to have access to the code. You should +always take into account the +[project's visibility and permissions settings](../user/project/settings/index.md#configure-project-visibility-features-and-permissions) +as well as the permission level of the user. + +NOTE: +External users still count towards a license seat. + +An administrator can flag a user as external by either of the following methods: + +- [Through the API](../api/users.md#user-modification). +- Using the GitLab UI: + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. + 1. On the left sidebar, select **Overview > Users** to create a new user or edit an existing one. + There, you can find the option to flag the user as external. + +Additionally, users can be set as external users using: + +- [SAML groups](../integration/saml.md#external-groups). +- [LDAP groups](../administration/auth/ldap/ldap_synchronization.md#external-groups). +- the [External providers list](../integration/omniauth.md#create-an-external-providers-list). + +## Set a new user to external + +By default, new users are not set as external users. This behavior can be changed +by an administrator: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Account and limit** section. + +If you change the default behavior of creating new users as external, you +have the option to narrow it down by defining a set of internal users. +The **Internal users** field allows specifying an email address regex pattern to +identify default internal users. New users whose email address matches the regex +pattern are set to internal by default rather than an external collaborator. + +The regex pattern format is in Ruby, but it needs to be convertible to JavaScript, +and the ignore case flag is set (`/regex pattern/i`). Here are some examples: + +- Use `\.internal@domain\.com$` to mark email addresses ending with + `.internal@domain.com` as internal. +- Use `^(?:(?!\.ext@domain\.com).)*$\r?` to mark users with email addresses + not including `.ext@domain.com` as internal. + +WARNING: +Be aware that this regex could lead to a +[regular expression denial of service (ReDoS) attack](https://en.wikipedia.org/wiki/ReDoS). diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 13e0938fa59..6ac67c3d21e 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -56,12 +56,12 @@ site you are about to failover to: rsync --archive --perms --delete root@<geo-primary>:/var/opt/gitlab/gitlab-rails/shared/registry/. /var/opt/gitlab/gitlab-rails/shared/registry ``` -Alternatively, you can [back up](../../../raketasks/backup_restore.md#back-up-gitlab) +Alternatively, you can [back up](../../../administration/backup_restore/index.md#back-up-gitlab) the container registry on the primary site and restore it onto the secondary site: 1. On your primary site, back up only the registry and - [exclude specific directories from the backup](../../../raketasks/backup_gitlab.md#excluding-specific-directories-from-the-backup): + [exclude specific directories from the backup](../../../administration/backup_restore/backup_gitlab.md#excluding-specific-directories-from-the-backup): ```shell # Create a backup in the /var/opt/gitlab/backups folder @@ -71,7 +71,7 @@ site: 1. Copy the backup tarball generated from your primary site to the `/var/opt/gitlab/backups` folder on your secondary site. -1. On your secondary site, restore the registry following the [Restore GitLab](../../../raketasks/backup_restore.md#restore-gitlab) +1. On your secondary site, restore the registry following the [Restore GitLab](../../../administration/backup_restore/index.md#restore-gitlab) documentation. ## Preflight checks diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index 9abd7ea9347..2e9a637eb5c 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -19,19 +19,20 @@ these definitions yet. We provide example diagrams and statements to demonstrate correct usage of terms. -| Term | Definition | Scope | Discouraged synonyms | -|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|-------------------------------------------------| -| Node | An individual server that runs GitLab either with a specific role or as a whole (for example a Rails application node). In a cloud context this can be a specific machine type. | GitLab | instance, server | -| Site | One or a collection of nodes running a single GitLab application. A site can be single-node or multi-node. | GitLab | deployment, installation instance | -| Single-node site | A specific configuration of GitLab that uses exactly one node. | GitLab | single-server, single-instance -| Multi-node site | A specific configuration of GitLab that uses more than one node. | GitLab | multi-server, multi-instance, high availability | -| Primary site | A GitLab site whose data is being replicated by at least one secondary site. There can only be a single primary site. | Geo-specific | Geo deployment, Primary node | -| Secondary site | A GitLab site that is configured to replicate the data of a primary site. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | -| Geo deployment | A collection of two or more GitLab sites with exactly one primary site being replicated by one or more secondary sites. | Geo-specific | | -| Reference architecture | A [specified configuration of GitLab for a number of users](../reference_architectures/index.md), possibly including multiple nodes and multiple sites. | GitLab | | -| Promoting | Changing the role of a site from secondary to primary. | Geo-specific | | -| Demoting | Changing the role of a site from primary to secondary. | Geo-specific | | -| Failover | The entire process that shifts users from a primary Site to a secondary site. This includes promoting a secondary, but contains other parts as well. For example, scheduling maintenance. | Geo-specific | | +| Term | Definition | Scope | Discouraged synonyms | +|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|-------------------------------------------------| +| Node | An individual server that runs GitLab either with a specific role or as a whole (for example a Rails application node). In a cloud context this can be a specific machine type. | GitLab | instance, server | +| Site | One or a collection of nodes running a single GitLab application. A site can be single-node or multi-node. | GitLab | deployment, installation instance | +| Single-node site | A specific configuration of GitLab that uses exactly one node. | GitLab | single-server, single-instance | +| Multi-node site | A specific configuration of GitLab that uses more than one node. | GitLab | multi-server, multi-instance, high availability | +| Primary site | A GitLab site whose data is being replicated by at least one secondary site. There can only be a single primary site. | Geo-specific | Geo deployment, Primary node | +| Secondary site | A GitLab site that is configured to replicate the data of a primary site. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | +| Geo deployment | A collection of two or more GitLab sites with exactly one primary site being replicated by one or more secondary sites. | Geo-specific | | +| Reference architecture | A [specified configuration of GitLab for a number of users](../reference_architectures/index.md), possibly including multiple nodes and multiple sites. | GitLab | | +| Promoting | Changing the role of a site from secondary to primary. | Geo-specific | | +| Demoting | Changing the role of a site from primary to secondary. | Geo-specific | | +| Failover | The entire process that shifts users from a primary Site to a secondary site. This includes promoting a secondary, but contains other parts as well. For example, scheduling maintenance. | Geo-specific | | +| Replication | Also called "synchronization". The uni-directional process that updates a resource on a secondary site to match the resource on the primary site. | Geo-specific | | ## Examples diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index ae2cc262160..0ab24cc4fb8 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -203,6 +203,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - [Disaster recovery](disaster_recovery/index.md) for multi-secondary sites causes downtime due to the complete re-synchronization and re-configuration of all non-promoted secondaries. - For Git over SSH, to make the project clone URL display correctly regardless of which site you are browsing, secondary sites must use the same port as the primary. [GitLab issue #339262](https://gitlab.com/gitlab-org/gitlab/-/issues/339262) proposes to remove this limitation. - Git push over SSH against a secondary site does not work for pushes over 1.86 GB. [GitLab issue #413109](https://gitlab.com/gitlab-org/gitlab/-/issues/413109) tracks this bug. +- Backups [cannot be run on secondaries](replication/troubleshooting.md#message-error-canceling-statement-due-to-conflict-with-recovery). ### Limitations on replication/verification @@ -217,7 +218,7 @@ If you try to view replication data on the primary site, you receive a warning t The only way to view projects replication data for a particular secondary site is to visit that secondary site directly. For example, `https://<IP of your secondary site>/admin/geo/replication/projects`. An [epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4623) to fix this limitation. -Keep in mind that mentioned URLs don't work when [Admin Mode](../../user/admin_area/settings/sign_in_restrictions.md#admin-mode) is enabled. +Keep in mind that mentioned URLs don't work when [Admin Mode](../settings/sign_in_restrictions.md#admin-mode) is enabled. When using Unified URL, visiting the secondary site directly means you must route your requests to the secondary site. Exactly how this might be done depends on your networking configuration. If using DNS to route requests to the appropriate site, then you can, for example, edit your local machine's `/etc/hosts` file to route your requests to the desired secondary site. If the Geo sites are all behind a load balancer, then depending on the load balancer, you might be able to configure all requests from your IP to go to a particular secondary site. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 18d0440965e..a5d85187812 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -190,7 +190,7 @@ keys must be manually replicated to the **secondary** site. ```ruby ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_nodes.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' ``` @@ -332,7 +332,7 @@ the **primary** site. After you sign in: 1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. -The initial replication may take some time. The status of the site or the ‘backfill’ may still in progress. You +The initial replication may take some time. The status of the site or the 'backfill' may still in progress. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Sites** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index f038bfd705c..b25700ccd29 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -80,10 +80,8 @@ on a machine: - With multiple disks mounted as a single mount-point (like with a RAID array). - Using LVM. -GitLab does not require a special file system and can work with: - -- NFS. -- A mounted Storage Appliance (there may be performance limitations when using a remote file system). +GitLab does not require a special file system and can work with a mounted Storage Appliance. However, there can be +performance limitations and consistency issues when using a remote file system. Geo triggers garbage collection in Gitaly to [deduplicate forked repositories](../../../development/git_object_deduplication.md#git-object-deduplication-and-gitlab-geo) on Geo secondary sites. @@ -111,7 +109,7 @@ GitLab stores files and blobs such as Issue attachments or LFS objects into eith - The file system in a specific location. - An [Object Storage](../../object_storage.md) solution. Object Storage solutions can be: - - Cloud based like Amazon S3 Google Cloud Storage. + - Cloud based like Amazon S3 and Google Cloud Storage. - Hosted by you (like MinIO). - A Storage Appliance that exposes an Object Storage-compatible API. @@ -192,7 +190,7 @@ successfully, you must replicate their data using some other means. |:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:---------------------------------------------------------------------------|:--------------------------------------------------------------------|:----------------------------------------------------------------|:------| |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | N/A | N/A | | |[Project repository](../../../user/project/repository/index.md) | **Yes** (10.2) | **Yes** (10.7) | N/A | N/A | | -|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2)<sup>2</sup> | **Yes** (10.7)<sup>2</sup> | N/A | N/A | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag geo_project_wiki_repository_replication, enabled by default in (15.11). | +|[Project wiki repository](../../../user/project/wiki/index.md) | **Yes** (10.2)<sup>2</sup> | **Yes** (10.7)<sup>2</sup> | N/A | N/A | Migrated to [self-service framework](../../../development/geo/framework.md) in 15.11. See GitLab issue [#367925](https://gitlab.com/gitlab-org/gitlab/-/issues/367925) for more details.<br /><br />Behind feature flag `geo_project_wiki_repository_replication`, enabled by default in (15.11). | |[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | N/A | N/A | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | |[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | @@ -203,7 +201,7 @@ successfully, you must replicate their data using some other means. |[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | **Yes** (12.3)<sup>1</sup> | **Yes** (15.10) | See [instructions](container_registry.md) to set up the Container Registry replication. | |[Terraform Module Registry](../../../user/packages/terraform_module_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | **Yes** (16.1) | N/A | N/A | Designs also require replication of LFS objects and Uploads. Replication is behind the feature flag geo_design_management_repository_replication, enabled by default.| +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | **Yes** (16.1) | N/A | N/A | Designs also require replication of LFS objects and Uploads. Replication is behind the feature flag `geo_design_management_repository_replication`, enabled by default.| |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | **Yes** (13.12) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 4a3f9c86041..a3abc945288 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -107,7 +107,7 @@ You can customize the: - SSH remote URL to use the location-aware `git.example.com`. To do so, change the SSH remote URL host by setting `gitlab_rails['gitlab_ssh_host']` in `gitlab.rb` of web nodes. - HTTP remote URL as shown in - [Custom Git clone URL for HTTP(S)](../../../user/admin_area/settings/visibility_and_access_controls.md#customize-git-clone-url-for-https). + [Custom Git clone URL for HTTP(S)](../../settings/visibility_and_access_controls.md#customize-git-clone-url-for-https). ## Example Git request handling behavior diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 4e597a21922..29edac1be83 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -67,7 +67,7 @@ The following steps enable a GitLab site to serve as the Geo **primary** site. ```ruby ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_nodes.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' @@ -217,7 +217,7 @@ then make the following modifications: ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_nodes.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' @@ -318,7 +318,7 @@ application nodes above, with some changes to run only the `sidekiq` service: ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_nodes.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' diff --git a/doc/administration/geo/replication/single_sign_on.md b/doc/administration/geo/replication/single_sign_on.md index 55e77d5657c..15e24cdcefb 100644 --- a/doc/administration/geo/replication/single_sign_on.md +++ b/doc/administration/geo/replication/single_sign_on.md @@ -31,6 +31,10 @@ If you have configured SAML on the primary site correctly, then it should work o ### SAML with separate URL with proxying enabled +NOTE: +When proxying is enabled, SAML can only be used to sign in the secondary site if your SAML Identity Provider (IdP) allows an +application to have multiple callback URLs configured. Check with your IdP provider support team to confirm if this is the case. + If a secondary site uses a different `external_url` to the primary site, then configure your SAML Identity Provider (IdP) to allow the secondary site's SAML callback URL. For example, to configure Okta: 1. [Sign in to Okta](https://www.okta.com/login/). diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 4047167e4af..c63480db389 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -51,7 +51,7 @@ Geo::MetricsUpdateWorker.new.perform If it raises an error, then the error is probably also preventing the jobs from completing. If it takes longer than 10 minutes, then there may be a performance issue, and the UI may always show "Unhealthy" even if the status eventually does get updated. -If it successfully updates the status, then something may be wrong with Sidekiq. Is it running? Do the logs show errors? This job is supposed to be enqueued every minute. It takes an exclusive lease in Redis to ensure that only one of these jobs can run at a time. The primary site updates its status directly in the PostgreSQL database. Secondary sites send an HTTP Post request to the primary site with their status data. +If it successfully updates the status, then something may be wrong with Sidekiq. Is it running? Do the logs show errors? This job is supposed to be enqueued every minute and might not run if a [job deduplication idempotency](../../sidekiq/sidekiq_troubleshooting.md#clearing-a-sidekiq-job-deduplication-idempotency-key) key was not cleared properly. It takes an exclusive lease in Redis to ensure that only one of these jobs can run at a time. The primary site updates its status directly in the PostgreSQL database. Secondary sites send an HTTP Post request to the primary site with their status data. A site also shows as "Unhealthy" if certain health checks fail. You can reveal the failure by running the following in the [Rails console](../../operations/rails_console.md) on the affected secondary site: @@ -240,7 +240,7 @@ This machine's Geo node name matches a database record ... no ``` For more information about recommended site names in the description of the Name field, see -[Geo Admin Area Common Settings](../../../user/admin_area/geo_sites.md#common-settings). +[Geo Admin Area Common Settings](../../../administration/geo_sites.md#common-settings). ### Reverify all uploads (or any SSF data type which is verified) @@ -622,7 +622,7 @@ This happens on wrongly-formatted addresses in `postgresql['md5_auth_cidr_addres ``` To fix this, update the IP addresses in `/etc/gitlab/gitlab.rb` under `postgresql['md5_auth_cidr_addresses']` -to respect the CIDR format (that is, `1.2.3.4/32`). +to respect the CIDR format (for example, `10.0.0.1/32`). ### Message: `LOG: invalid IP mask "md5": Name or service not known` @@ -634,7 +634,7 @@ This happens when you have added IP addresses without a subnet mask in `postgres ``` To fix this, add the subnet mask in `/etc/gitlab/gitlab.rb` under `postgresql['md5_auth_cidr_addresses']` -to respect the CIDR format (that is, `1.2.3.4/32`). +to respect the CIDR format (for example, `10.0.0.1/32`). ### Message: `Found data in the gitlabhq_production database!` when running `gitlab-ctl replicate-geo-database` @@ -1295,7 +1295,7 @@ When [Geo proxying for secondary sites](../secondary_proxy/index.md) is enabled, Check the NGINX logs for errors similar to this example: ```plaintext -2022/01/26 00:02:13 [error] 26641#0: *829148 upstream sent too big header while reading response header from upstream, client: 1.2.3.4, server: geo.staging.gitlab.com, request: "POST /users/sign_in HTTP/2.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/users/sign_in", host: "geo.staging.gitlab.com", referrer: "https://geo.staging.gitlab.com/users/sign_in" +2022/01/26 00:02:13 [error] 26641#0: *829148 upstream sent too big header while reading response header from upstream, client: 10.0.2.2, server: geo.staging.gitlab.com, request: "POST /users/sign_in HTTP/2.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/users/sign_in", host: "geo.staging.gitlab.com", referrer: "https://geo.staging.gitlab.com/users/sign_in" ``` To resolve this issue: @@ -1345,15 +1345,8 @@ To fix this issue, set the primary site's internal URL to a URL that is: - Unique to the primary site. - Accessible from all secondary sites. -1. Enter the [Rails console](../../operations/rails_console.md) on the primary site. - -1. Run the following, replacing `https://unique.url.for.primary.site` with your specific internal URL. - For example, depending on your network configuration, you could use an IP address, like - `http://1.2.3.4`. - - ```ruby - GeoNode.where(primary: true).first.update!(internal_url: "https://unique.url.for.primary.site") - ``` +1. Visit the primary site. +1. [Set up the internal URLs](../../../administration/geo_sites.md#set-up-the-internal-urls). ### Secondary site returns `Received HTTP code 403 from proxy after CONNECT` @@ -1404,6 +1397,27 @@ In this case, make sure to update the changed URL on all your sites: 1. On the left sidebar, select **Geo > Sites**. 1. Change the URL and save the change. +### Message: `ERROR: canceling statement due to conflict with recovery` during backup + +Running a backup on a Geo **secondary** [is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/211668). + +When running a backup on a **secondary** you might encounter the following error message: + +```plaintext +Dumping PostgreSQL database gitlabhq_production ... +pg_dump: error: Dumping the contents of table "notes" failed: PQgetResult() failed. +pg_dump: error: Error message from server: ERROR: canceling statement due to conflict with recovery +DETAIL: User query might have needed to see row versions that must be removed. +pg_dump: error: The command was: COPY public.notes (id, note, [...], last_edited_at) TO stdout; +``` + +To prevent a database backup being made automatically during GitLab upgrades on your Geo **secondaries**, +create the following empty file: + +```shell +sudo touch /etc/gitlab/skip-auto-backup +``` + ## Fixing non-PostgreSQL replication failures If you notice replication failures in `Admin > Geo > Sites` or the [Sync status Rake task](#sync-status-rake-task), you can try to resolve the failures with the following general steps: @@ -1665,7 +1679,7 @@ Repository check failures on a Geo secondary site do not necessarily imply a rep 1. Find affected repositories as mentioned below, as well as their [logged errors](../../repository_checks.md#what-to-do-if-a-check-failed). 1. Try to diagnose specific `git fsck` errors. The range of possible errors is wide, try putting them into search engines. -1. Test normal functions of the affected repositories. Pull from the secondary, view the files. +1. Test typical functions of the affected repositories. Pull from the secondary, view the files. 1. Check if the primary site's copy of the repository has an identical `git fsck` error. If you are planning a failover, then consider prioritizing that the secondary site has the same information that the primary site has. Ensure you have a backup of the primary, and follow [planned failover guidelines](../disaster_recovery/planned_failover.md). 1. Push to the primary and check if the change gets replicated to the secondary site. 1. If replication is not automatically working, try to manually sync the repository. @@ -1806,3 +1820,30 @@ If the output differs on some hosts, PostgreSQL replication does not work proper A full index rebuild is required if the on-disk data is transferred 'at rest' to an operating system with an incompatible locale, or through replication. This check is also required when using a mixture of GitLab deployments. The locale might be different between an Linux package install, a GitLab Docker container, a Helm chart deployment, or external database services. + +## Investigate causes of database replication lag + +If the output of `sudo gitlab-rake geo:status` shows that `Database replication lag` remains significantly high over time, the primary node in database replication can be checked to determine the status of lag for +different parts of the database replication process. These values are known as `write_lag`, `flush_lag`, and `replay_lag`. For more information, see +[the official PostgreSQL documentation](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-REPLICATION-VIEW). + +Run the following command from the primary Geo node's database to provide relevant output: + +```shell +gitlab-psql -xc 'SELECT write_lag,flush_lag,replay_lag FROM pg_stat_replication;' + +-[ RECORD 1 ]--------------- +write_lag | 00:00:00.072392 +flush_lag | 00:00:00.108168 +replay_lag | 00:00:00.108283 +``` + +If one or more of these values is significantly high, this could indicate a problem and should be investigated further. When determining the cause, consider that: + +- `write_lag` indicates the time since when WAL bytes have been sent by the primary, then received to the secondary, but not yet flushed or applied. +- A high `write_lag` value may indicate degraded network performance or insufficient network speed between the primary and secondary nodes. +- A high `flush_lag` value may indicate degraded or sub-optimal disk I/O performance with the secondary node's storage device. +- A high `replay_lag` value may indicate long running transactions in PostgreSQL, or the saturation of a needed resource like the CPU. +- The difference in time between `write_lag` and `flush_lag` indicates that WAL bytes have been sent to the underlying storage system, but it has not reported that they were flushed. + This data is most likely not fully written to a persistent storage, and likely held in some kind of volatile write cache. +- The difference between `flush_lag` and `replay_lag` indicates WAL bytes that have been successfully persisted to storage, but could not be replayed by the database system. diff --git a/doc/administration/geo/replication/upgrading_the_geo_sites.md b/doc/administration/geo/replication/upgrading_the_geo_sites.md index 644232a2c9e..ce0ad736071 100644 --- a/doc/administration/geo/replication/upgrading_the_geo_sites.md +++ b/doc/administration/geo/replication/upgrading_the_geo_sites.md @@ -33,7 +33,7 @@ and all **secondary** sites: 1. SSH into each node of the **primary** site. 1. [Upgrade GitLab on the **primary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). 1. Perform testing on the **primary** site, particularly if you paused replication in step 1 to protect DR. [There are some suggestions for post-upgrade testing](../../../update/plan_your_upgrade.md#pre-upgrade-and-post-upgrade-checks) in the upgrade documentation. -1. Ensure that the secrets in the `/etc/gitlab/gitlab-secrets.json` file of both the primary site and the secondary site are the same. The file must be the same on all of a site’s nodes. +1. Ensure that the secrets in the `/etc/gitlab/gitlab-secrets.json` file of both the primary site and the secondary site are the same. The file must be the same on all of a site's nodes. 1. SSH into each node of **secondary** sites. 1. [Upgrade GitLab on each **secondary** site](../../../update/package/index.md#upgrade-using-the-official-repositories). 1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication). diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 3081d1c2485..2ab96a3d33d 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -131,6 +131,10 @@ and cannot be configured per Geo site. Therefore, all runners clone from the pri which Geo site they register on. For information about GitLab CI using a specific Geo secondary to clone from, see issue [3294](https://gitlab.com/gitlab-org/gitlab/-/issues/3294#note_1009488466). +- When secondary proxying is used together with separate URLs, + [signing in the secondary site using SAML](../replication/single_sign_on.md#saml-with-separate-url-with-proxying-enabled) + is only supported if the SAML Identity Provider (IdP) allows an application to be configured with multiple callback URLs. + ## Behavior of secondary sites when the primary Geo site is down Considering that web traffic is proxied to the primary, the behavior of the secondary sites differs when the primary diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 31d0fbdffe0..be6e327732d 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -75,7 +75,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ```ruby ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_nodes.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' ``` @@ -193,8 +193,8 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o | `postgresql['md5_auth_cidr_addresses']` | **Primary** and **Secondary** sites' public or VPC private addresses. | If you are using Google Cloud Platform, SoftLayer, or any other vendor that - provides a virtual private cloud (VPC), you can use the **primary** and **secondary** sites' - private addresses (corresponds to "internal address" for Google Cloud Platform) for + provides a virtual private cloud (VPC), we recommend using the **primary** + and **secondary** sites' "private" or "internal" addresses for `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. The `listen_address` option opens PostgreSQL up to network connections with the interface @@ -468,7 +468,8 @@ data before running `pg_basebackup`. sudo -i ``` -1. Choose a database-friendly name to use for your **secondary** site to +1. Choose a [database-friendly name](https://www.postgresql.org/docs/13/warm-standby.html#STREAMING-REPLICATION-SLOTS-MANIPULATION) + to use for your **secondary** site to use as the replication slot name. For example, if your domain is `secondary.geo.example.com`, use `secondary_example` as the slot name as shown in the commands below. diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 81541d1cb9c..50383546da3 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -39,7 +39,7 @@ developed and tested. We aim to be compatible with most external ## ## The unique identifier for the Geo site. See - ## https://docs.gitlab.com/ee/user/admin_area/geo_nodes.html#common-settings + ## https://docs.gitlab.com/ee/administration/geo_nodes.html#common-settings ## gitlab_rails['geo_node_name'] = '<site_name_here>' ``` diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 359f706a8aa..8ac64a963bb 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -34,8 +34,8 @@ If both Geo sites are based on the [1K reference architecture](../../reference_a - [Using Linux package PostgreSQL instances](database.md) . - [Using external PostgreSQL instances](external_database.md) 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. -1. Recommended: [Configure unified URLs](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. -1. Optional: [Configure Object storage replication](../../object_storage.md) +1. Recommended: [Configure unified URLs](../secondary_proxy/index.md#set-up-a-unified-url-for-geo-sites) to use a single, unified URL for all Geo sites. +1. Optional: [Configure Object storage replication](../replication/object_storage.md) 1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. See [notes on LDAP](../index.md#ldap). 1. Optional: [Configure Container Registry for the secondary site](../replication/container_registry.md). 1. Follow the [Using a Geo Site](../replication/usage.md) guide. diff --git a/doc/administration/geo_sites.md b/doc/administration/geo_sites.md new file mode 100644 index 00000000000..81cc3a87941 --- /dev/null +++ b/doc/administration/geo_sites.md @@ -0,0 +1,119 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Geo sites Admin Area **(PREMIUM SELF)** + +You can configure various settings for GitLab Geo sites. For more information, see +[Geo documentation](../administration/geo/index.md). + +On either the primary or secondary site: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Geo > Sites**. + +## Common settings + +All Geo sites have the following settings: + +| Setting | Description | +| --------| ----------- | +| Primary | This marks a Geo site as **primary** site. There can be only one **primary** site. | +| Name | The unique identifier for the Geo site. It's highly recommended to use a physical location as a name. Good examples are "London Office" or "us-east-1". Avoid words like "primary", "secondary", "Geo", or "DR". This makes the failover process easier because the physical location does not change, but the Geo site role can. All nodes in a single Geo site use the same site name. Nodes use the `gitlab_rails['geo_node_name']` setting in `/etc/gitlab/gitlab.rb` to lookup their Geo site record in the PostgreSQL database. If `gitlab_rails['geo_node_name']` is not set, the node's `external_url` with trailing slash is used as fallback. The value of `Name` is case-sensitive, and most characters are allowed. | +| URL | The instance's user-facing URL. | + +The site you're currently browsing is indicated with a blue `Current` label, and +the **primary** node is listed first as `Primary site`. + +## Secondary site settings + +**Secondary** sites have a number of additional settings available: + +| Setting | Description | +|---------------------------|-------------| +| Selective synchronization | Enable Geo [selective sync](../administration/geo/replication/configuration.md#selective-synchronization) for this **secondary** site. | +| Repository sync capacity | Number of concurrent requests this **secondary** site makes to the **primary** site when backfilling repositories. | +| File sync capacity | Number of concurrent requests this **secondary** site makes to the **primary** site when backfilling files. | + +## Geo backfill + +**Secondary** sites are notified of changes to repositories and files by the **primary** site, +and always attempt to synchronize those changes as quickly as possible. + +Backfill is the act of populating the **secondary** site with repositories and files that +existed *before* the **secondary** site was added to the database. Because there may be +extremely large numbers of repositories and files, it's not feasible to attempt to +download them all at once; so, GitLab places an upper limit on the concurrency of +these operations. + +How long the backfill takes is dependent on the maximum concurrency, but higher +values place more strain on the **primary** site. The limits are configurable. +If your **primary** site has lots of surplus capacity, +you can increase the values to complete backfill in a shorter time. If it's +under heavy load and backfill reduces its availability for standard requests, +you can decrease them. + +## Set up the internal URLs + +> Setting up internal URLs in secondary sites was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77179) in GitLab 14.7. + +You can set up a different URL for synchronization between the primary and secondary site. + +The **primary** site's Internal URL is used by **secondary** sites to contact it +(to sync repositories, for example). The name Internal URL distinguishes it from +[External URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab), +which is used by users. Internal URL does not need to be a private address. + +When [Geo secondary proxying](../administration/geo/secondary_proxy/index.md) is enabled, +the primary uses the secondary's internal URL to contact it directly. + +The internal URL defaults to external URL. To change it: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Geo > Sites**. +1. Select **Edit** on the site you want to customize. +1. Edit the internal URL. +1. Select **Save changes**. + +When enabled, the Admin Area for Geo shows replication details for each site directly +from the primary site's UI, and through the Geo secondary proxy, if enabled. + +WARNING: +We recommend using an HTTPS connection while configuring the Geo sites. To avoid +breaking communication between **primary** and **secondary** sites when using +HTTPS, customize your Internal URL to point to a load balancer with TLS +terminated at the load balancer. + +WARNING: +Starting with GitLab 13.3 and [until 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/325522), +if you use an internal URL that is not accessible to the users, the +OAuth authorization flow does not work properly, because users are redirected +to the internal URL instead of the external one. + +## Multiple secondary sites behind a load balancer + +**Secondary** sites can use identical external URLs if +a unique `name` is set for each Geo site. The `gitlab.rb` setting +`gitlab_rails['geo_node_name']` must: + +- Be set for each GitLab instance that runs `puma`, `sidekiq`, or `geo_logcursor`. +- Match a Geo site name. + +The load balancer must use sticky sessions to avoid authentication +failures and cross-site request errors. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 60291732a20..bf3d38657f8 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -41,7 +41,7 @@ Get started: - [Add members](../user/group/index.md#add-users-to-a-group) to the group. - Create a [subgroup](../user/group/subgroups/index.md#create-a-subgroup). - [Add members](../user/group/subgroups/index.md#subgroup-membership) to the subgroup. -- Enable [external authorization control](../user/admin_area/settings/external_authorization.md#configuration). +- Enable [external authorization control](../administration/settings/external_authorization.md#configuration). **More resources** @@ -76,16 +76,16 @@ While this isn't an exhaustive list, following these steps gives you a solid sta - Use a long root password, stored in a vault. - Install trusted SSL certificate and establish a process for renewal and revocation. - [Configure SSH key restrictions](../security/ssh_keys_restrictions.md#restrict-allowed-ssh-key-technologies-and-minimum-length) per your organization's guidelines. -- [Disable new sign-ups](../user/admin_area/settings/sign_up_restrictions.md#disable-new-sign-ups). +- [Disable new sign-ups](settings/sign_up_restrictions.md#disable-new-sign-ups). - Require email confirmation. - Set password length limit, configure SSO or SAML user management. - Limit email domains if allowing sign-up. - Require two-factor authentication (2FA). -- [Disable password authentication](../user/admin_area/settings/sign_in_restrictions.md#password-authentication-enabled) for Git over HTTPS. -- Set up [email notification for unknown sign-ins](../user/admin_area/settings/sign_in_restrictions.md#email-notification-for-unknown-sign-ins). +- [Disable password authentication](settings/sign_in_restrictions.md#password-authentication-enabled) for Git over HTTPS. +- Set up [email notification for unknown sign-ins](settings/sign_in_restrictions.md#email-notification-for-unknown-sign-ins). - Configure [user and IP rate limits](https://about.gitlab.com/blog/2020/05/20/gitlab-instance-security-best-practices/#user-and-ip-rate-limits). - Limit [webhooks local access](https://about.gitlab.com/blog/2020/05/20/gitlab-instance-security-best-practices/#webhooks). -- Set [rate limits for protected paths](../user/admin_area/settings/protected_paths.md). +- Set [rate limits for protected paths](settings/protected_paths.md). - Sign up for [Security Alerts](https://about.gitlab.com/company/preference-center/) from the Communication Preference Center. - Keep track of security best practices on our [blog page](https://about.gitlab.com/blog/2020/05/20/gitlab-instance-security-best-practices/). @@ -130,7 +130,7 @@ The routine differs, depending on whether you deployed with the Linux package or When backing up (single node) GitLab server installed using the Linux package, you can use a single Rake task. -Learn about [backing up Linux package or Helm variations](../raketasks/backup_restore.md). +Learn about [backing up Linux package or Helm variations](../administration/backup_restore/index.md). This process backs up your entire instance, but does not back up the configuration files. Ensure those are backed up separately. Keep your configuration files and backup archives in a separate location to ensure the encryption keys are not kept with the encrypted data. @@ -163,7 +163,7 @@ For more information about GitLab SaaS backups, see our [Backup FAQ page](https: ### Alternative backup strategies In some situations the Rake task for backups may not be the most optimal solution. Here are some -[alternatives](../raketasks/backup_restore.md) to consider if the Rake task does not work for you. +[alternatives](../administration/backup_restore/index.md) to consider if the Rake task does not work for you. #### Option 1: File system snapshot @@ -236,10 +236,10 @@ Rate limits also improve the security of your application. You can make changes to your default rate limits from the Admin Area. For more information about configuration, see the [Admin Area page](../security/rate_limits.md#configurable-limits). -- Define [issues rate limits](../user/admin_area/settings/rate_limit_on_issues_creation.md) to set a maximum number of issue creation requests per minute, per user. -- Enforce [user and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md) for unauthenticated web requests. -- Review the [rate limit on raw endpoints](../user/admin_area/settings/rate_limits_on_raw_endpoints.md). The default setting is 300 requests per minute for raw file access. -- Review the [import/export rate limits](../user/admin_area/settings/import_export_rate_limits.md) of the six active defaults. +- Define [issues rate limits](settings/rate_limit_on_issues_creation.md) to set a maximum number of issue creation requests per minute, per user. +- Enforce [user and IP rate limits](settings/user_and_ip_rate_limits.md) for unauthenticated web requests. +- Review the [rate limit on raw endpoints](settings/rate_limits_on_raw_endpoints.md). The default setting is 300 requests per minute for raw file access. +- Review the [import/export rate limits](settings/import_export_rate_limits.md) of the six active defaults. For more information about API and rate limits, see our [API page](../api/rest/index.md). diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 5c6fc370052..c7fa40014d0 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -377,9 +377,6 @@ This can be risky because anything that prevents your Gitaly clients from reachi servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name resolution problems. -Additionally, you must [disable Rugged](../nfs.md#improving-nfs-performance-with-gitlab) -if previously enabled manually. - Gitaly makes the following assumptions: - Your `gitaly1.internal` Gitaly server can be reached at `gitaly1.internal:8075` from your Gitaly @@ -835,7 +832,7 @@ For consistent and stable performance, you should first explore other options su adjusting node specifications, and [reviewing large repositories](../../user/project/repository/managing_large_repositories.md) or workloads. FLAG: -On self-managed GitLab, by default repository cgroups are not available. To make it available, ask an administrator to +On self-managed GitLab, by default repository cgroups are not available. To make it available, an administrator can [enable the feature flag](../feature_flags.md) named `gitaly_run_cmds_in_cgroup`. When enabling cgroups for memory, you should ensure that no swap is configured on the Gitaly nodes as diff --git a/doc/administration/gitaly/gitaly_geo_capabilities.md b/doc/administration/gitaly/gitaly_geo_capabilities.md index e4147eec162..a98477318f2 100644 --- a/doc/administration/gitaly/gitaly_geo_capabilities.md +++ b/doc/administration/gitaly/gitaly_geo_capabilities.md @@ -17,7 +17,6 @@ The following tables are intended to guide you to choose the right combination o |------------|--------------|----------------|-----------------|-------------|-----------------| |Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10 s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not support snapshot backups](../gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets | |Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs | -|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences | ## Geo capabilities diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 2a3c3da24b3..db11ac8c769 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -52,13 +52,10 @@ Before deploying Gitaly Cluster, review: - [Configuration guidance](configure_gitaly.md) and [Repository storage options](../repository_storage_paths.md) to make sure that Gitaly Cluster is the best setup for you. -If you have: +If you have not yet migrated to Gitaly Cluster, you have two options: -- Not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the service you are using. However, NFS - is [no longer supported](../../update/removals.md#nfs-as-git-repository-storage-is-no-longer-supported). -- Not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options: - - A sharded Gitaly instance. - - Gitaly Cluster. +- A sharded Gitaly instance. +- Gitaly Cluster. Contact your [Customer Success Manager](https://about.gitlab.com/job-families/sales/customer-success-management/) or customer support if you have any questions. @@ -72,15 +69,15 @@ the current status of these issues, refer to the referenced issues and epics. | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later) on your Geo primary site. In GitLab 15.3, the feature flag is enabled by default. | | Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform standard operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind results in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup:<br/><br/>1. [Shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui).<br/>2. Snapshot all Gitaly Cluster nodes at the same time.<br/>3. Take a database dump of the Praefect database. | -| Rebuilding or replacing an existing Gitaly Cluster node | There is no way to replace existing nodes in place because the Praefect database is relied on to determine the current state of each Gitaly node. Changing how Gitaly Cluster stores repositories is proposed in issue [4218](https://gitlab.com/gitlab-org/gitaly/-/issues/4218). Turning on [repository verification](praefect.md#repository-verification) is proposed in issue [4429](https://gitlab.com/gitlab-org/gitaly/-/issues/4429) so Praefect can detect that data is missing and needs to replicated to a new Gitaly node. | No known solution at this time. | +| Limitations when running in Kubernetes, Amazon ECS, or similar | Praefect (Gitaly Cluster) is not supported and Gitaly has known limitations. For more information, see [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127). | Use our [reference architectures](../reference_architectures/index.md). | ### Snapshot backup and recovery limitations Gitaly Cluster does not support snapshot backups. Snapshot backups can cause issues where the Praefect database becomes out of sync with the disk storage. Because of how Praefect rebuilds the replication metadata of Gitaly disk information -during a restore, you should use the [official backup and restore Rake tasks](../../raketasks/backup_restore.md). +during a restore, you should use the [official backup and restore Rake tasks](../../administration/backup_restore/index.md). -The [incremental backup method](../../raketasks/backup_gitlab.md#incremental-repository-backups) +The [incremental backup method](../../administration/backup_restore/backup_gitlab.md#incremental-repository-backups) can be used to speed up Gitaly Cluster backups. If you are unable to use either method, contact customer support for restoration help. @@ -175,7 +172,7 @@ best suited by using Gitaly Cluster. ### Backing up repositories -When backing up or syncing repositories using tools other than GitLab, you must [prevent writes](../../raketasks/backup_restore.md#prevent-writes-and-copy-the-git-repository-data) +When backing up or syncing repositories using tools other than GitLab, you must [prevent writes](../../administration/backup_restore/backup_gitlab.md#prevent-writes-and-copy-the-git-repository-data) while copying repository data. ## Gitaly Cluster @@ -641,7 +638,7 @@ code. This re-introduced code is informally referred to as the "Rugged patches". FLAG: On self-managed GitLab, by default automatic detection of whether Rugged should be used (per storage) is not available. -To make it available, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named +To make it available, an administrator can [disable the feature flag](../../administration/feature_flags.md) named `skip_rugged_auto_detect`. The Ruby methods that perform direct Git access are behind diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index f80a5192c55..9577c4a4cb2 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -19,6 +19,10 @@ Configure Gitaly Cluster using either: Smaller GitLab installations may need only [Gitaly itself](index.md). +NOTE: +Gitaly Cluster is not yet supported in Kubernetes, Amazon ECS, or similar container environments. For more information, see +[epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127). + ## Requirements The minimum recommended configuration for a Gitaly Cluster requires: @@ -1107,8 +1111,8 @@ For more information on Gitaly server configuration, see our ```ruby # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your front door GitLab URL or an internal load balancer. - # Examples: 'https://gitlab.example.com', 'http://1.2.3.4' - gitlab_rails['internal_api_url'] = 'http://GITLAB_HOST' + # Examples: 'https://gitlab.example.com', 'http://10.0.2.2' + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' ``` 1. Configure the storage location for Git data by setting `gitaly['configuration'][:storage]` in @@ -1487,7 +1491,7 @@ repository storage redundancy. For a replication factor: -- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. +- Of `1`: Gitaly and Gitaly Cluster have roughly the same storage requirements. - More than `1`: The amount of required storage is `used space * replication factor`. `used space` should include any planned future growth. @@ -1628,20 +1632,8 @@ You should use [repository-specific primary nodes](#repository-specific-primary- > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3492) in GitLab 13.12, with primary elections run when Praefect starts or the cluster's consensus of a Gitaly node's health changes. > - [Changed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3543) in GitLab 14.1, primary elections are run lazily. -Gitaly Cluster supports electing repository-specific primary Gitaly nodes. Repository-specific -Gitaly primary nodes are enabled in `/etc/gitlab/gitlab.rb` by setting -`praefect['failover_election_strategy'] = 'per_repository'`. - -Praefect's [deprecated election strategies](#deprecated-election-strategies): - -- Elected a primary Gitaly node for each virtual storage, which was used as the primary node for - each repository in the virtual storage. -- Prevented horizontal scaling of a virtual storage. The primary Gitaly node needed a replica of - each repository and thus became the bottleneck. - -The `per_repository` election strategy solves this problem by electing a primary Gitaly node separately for each -repository. Combined with [configurable replication factors](#configure-replication-factor), you can -horizontally scale storage capacity and distribute write load across Gitaly nodes. +Gitaly Cluster elects a primary Gitaly node separately for each repository. Combined with +[configurable replication factors](#configure-replication-factor), you can horizontally scale storage capacity and distribute write load across Gitaly nodes. Primary elections are run lazily. Praefect doesn't immediately elect a new primary node if the current one is unhealthy. A new primary is elected if a request must be served while the current primary is unavailable. diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index dbbed0f60ba..aa487917ef0 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -432,6 +432,11 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t > - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5658) in GitLab 14.4. > - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5789) in GitLab 14.6, support for immediate replication. +WARNING: +Because of a [known issue](https://gitlab.com/gitlab-org/gitaly/-/issues/5402), you can't add repositories to the +Praefect tracking database with Praefect-generated replica paths (`@cluster`). These repositories are not associated with the repository path used by GitLab and are +inaccessible. + The `track-repository` Praefect sub-command adds repositories on disk to the Praefect tracking database to be tracked. ```shell @@ -484,6 +489,11 @@ This command fails if: > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6319) in GitLab 15.4. +WARNING: +Because of a [known issue](https://gitlab.com/gitlab-org/gitaly/-/issues/5402), you can't add repositories to the +Praefect tracking database with Praefect-generated replica paths (`@cluster`). These repositories are not associated with the repository path used by GitLab and are +inaccessible. + Migrations using the API automatically add repositories to the Praefect tracking database. If you are instead manually copying repositories over from existing infrastructure, you can use the `track-repositories` diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index afef787e9c3..3d8110e1dab 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -12,7 +12,7 @@ Refer to the information below when troubleshooting Gitaly and Gitaly Cluster. The following sections provide possible solutions to Gitaly errors. -See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings, +See also [Gitaly timeout](../settings/gitaly_timeouts.md) settings, and our advice on [parsing the `gitaly/current` file](../logs/log_parsing.md#parsing-gitalycurrent). ### Check versions when using standalone Gitaly servers diff --git a/doc/administration/img/abuse_report_blocked_user.png b/doc/administration/img/abuse_report_blocked_user.png Binary files differnew file mode 100644 index 00000000000..435d8dfe821 --- /dev/null +++ b/doc/administration/img/abuse_report_blocked_user.png diff --git a/doc/administration/img/abuse_reports_page_v13_11.png b/doc/administration/img/abuse_reports_page_v13_11.png Binary files differnew file mode 100644 index 00000000000..ef57f45ab77 --- /dev/null +++ b/doc/administration/img/abuse_reports_page_v13_11.png diff --git a/doc/administration/img/admin_labels_v14_7.png b/doc/administration/img/admin_labels_v14_7.png Binary files differnew file mode 100644 index 00000000000..01a4ea0c2cc --- /dev/null +++ b/doc/administration/img/admin_labels_v14_7.png diff --git a/doc/administration/img/broadcast_messages_banner_v15_0.png b/doc/administration/img/broadcast_messages_banner_v15_0.png Binary files differnew file mode 100644 index 00000000000..e1b350142b3 --- /dev/null +++ b/doc/administration/img/broadcast_messages_banner_v15_0.png diff --git a/doc/administration/img/broadcast_messages_notification_v12_10.png b/doc/administration/img/broadcast_messages_notification_v12_10.png Binary files differnew file mode 100644 index 00000000000..fb03748c892 --- /dev/null +++ b/doc/administration/img/broadcast_messages_notification_v12_10.png diff --git a/doc/administration/img/cohorts_v13_9_a.png b/doc/administration/img/cohorts_v13_9_a.png Binary files differnew file mode 100644 index 00000000000..1a4590290b9 --- /dev/null +++ b/doc/administration/img/cohorts_v13_9_a.png diff --git a/doc/administration/img/credentials_inventory_gpg_keys_v14_9.png b/doc/administration/img/credentials_inventory_gpg_keys_v14_9.png Binary files differnew file mode 100644 index 00000000000..6c4c8f30df6 --- /dev/null +++ b/doc/administration/img/credentials_inventory_gpg_keys_v14_9.png diff --git a/doc/administration/img/credentials_inventory_personal_access_tokens_v14_9.png b/doc/administration/img/credentials_inventory_personal_access_tokens_v14_9.png Binary files differnew file mode 100644 index 00000000000..254d520d538 --- /dev/null +++ b/doc/administration/img/credentials_inventory_personal_access_tokens_v14_9.png diff --git a/doc/administration/img/credentials_inventory_project_access_tokens_v14_9.png b/doc/administration/img/credentials_inventory_project_access_tokens_v14_9.png Binary files differnew file mode 100644 index 00000000000..ae204ac09ef --- /dev/null +++ b/doc/administration/img/credentials_inventory_project_access_tokens_v14_9.png diff --git a/doc/administration/img/credentials_inventory_ssh_keys_v14_9.png b/doc/administration/img/credentials_inventory_ssh_keys_v14_9.png Binary files differnew file mode 100644 index 00000000000..8f2f11515eb --- /dev/null +++ b/doc/administration/img/credentials_inventory_ssh_keys_v14_9.png diff --git a/doc/administration/img/email1.png b/doc/administration/img/email1.png Binary files differnew file mode 100644 index 00000000000..e79ccc3e9a9 --- /dev/null +++ b/doc/administration/img/email1.png diff --git a/doc/administration/img/email2.png b/doc/administration/img/email2.png Binary files differnew file mode 100644 index 00000000000..d073c0e42da --- /dev/null +++ b/doc/administration/img/email2.png diff --git a/doc/administration/img/export_permissions_v13_11.png b/doc/administration/img/export_permissions_v13_11.png Binary files differnew file mode 100644 index 00000000000..d9bbe8c3daf --- /dev/null +++ b/doc/administration/img/export_permissions_v13_11.png diff --git a/doc/administration/img/impersonate_user_button_v13_8.png b/doc/administration/img/impersonate_user_button_v13_8.png Binary files differnew file mode 100644 index 00000000000..475315a0c0f --- /dev/null +++ b/doc/administration/img/impersonate_user_button_v13_8.png diff --git a/doc/administration/img/index_runners_search_or_filter_v14_5.png b/doc/administration/img/index_runners_search_or_filter_v14_5.png Binary files differnew file mode 100644 index 00000000000..10b8cc01103 --- /dev/null +++ b/doc/administration/img/index_runners_search_or_filter_v14_5.png diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index df364a3f737..679042c3114 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -19,11 +19,9 @@ Read more about [configuring rate limits](../security/rate_limits.md). ### Issue creation -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28129) in GitLab 12.10. - This setting limits the request rate to the issue creation endpoint. -Read more about [issue creation rate limits](../user/admin_area/settings/rate_limit_on_issues_creation.md). +Read more about [issue creation rate limits](settings/rate_limit_on_issues_creation.md). - **Default rate limit**: Disabled by default. @@ -31,17 +29,15 @@ Read more about [issue creation rate limits](../user/admin_area/settings/rate_li This setting limits the request rate per user or IP. -Read more about [User and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md). +Read more about [User and IP rate limits](settings/user_and_ip_rate_limits.md). - **Default rate limit**: Disabled by default. ### By raw endpoint -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30829) in GitLab 12.2. - This setting limits the request rate per endpoint. -Read more about [raw endpoint rate limits](../user/admin_area/settings/rate_limits_on_raw_endpoints.md). +Read more about [raw endpoint rate limits](settings/rate_limits_on_raw_endpoints.md). - **Default rate limit**: 300 requests per project, per commit and per file path. @@ -63,16 +59,14 @@ GitLab rate limits the following paths by default: '/admin/session' ``` -Read more about [protected path rate limits](../user/admin_area/settings/protected_paths.md). +Read more about [protected path rate limits](settings/protected_paths.md). - **Default rate limit**: After 10 requests, the client must wait 60 seconds before trying again. ### Package Registry -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57029) in GitLab 13.12. - This setting limits the request rate on the Packages API per user or IP. For more information, see -[Package Registry Rate Limits](../user/admin_area/settings/package_registry_rate_limits.md). +[Package Registry Rate Limits](settings/package_registry_rate_limits.md). - **Default rate limit**: Disabled by default. @@ -92,7 +86,7 @@ requests per user. For more information, read > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75918) in GitLab 14.6. [Feature flag `files_api_throttling`](https://gitlab.com/gitlab-org/gitlab/-/issues/338903) removed. This setting limits the request rate on the Packages API per user or IP address. For more information, read -[Files API rate limits](../user/admin_area/settings/files_api_rate_limits.md). +[Files API rate limits](settings/files_api_rate_limits.md). - **Default rate limit**: Disabled by default. @@ -101,26 +95,24 @@ This setting limits the request rate on the Packages API per user or IP address. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68645) in GitLab 14.4. This setting limits the request rate on deprecated API endpoints per user or IP address. For more information, read -[Deprecated API rate limits](../user/admin_area/settings/deprecated_api_rate_limits.md). +[Deprecated API rate limits](settings/deprecated_api_rate_limits.md). - **Default rate limit**: Disabled by default. ### Import/Export -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35728) in GitLab 13.2. - This setting limits the import/export actions for groups and projects. | Limit | Default (per minute per user) | |-------------------------|-------------------------------| -| Project Import | 6 | -| Project Export | 6 | -| Project Export Download | 1 | -| Group Import | 6 | -| Group Export | 6 | -| Group Export Download | 1 | +| Project Import | 6 | +| Project Export | 6 | +| Project Export Download | 1 | +| Group Import | 6 | +| Group Export | 6 | +| Group Export Download | 1 | -Read more about [import/export rate limits](../user/admin_area/settings/import_export_rate_limits.md). +Read more about [import/export rate limits](settings/import_export_rate_limits.md). ### Member Invitations @@ -162,10 +154,10 @@ Set the limit to `0` to disable it. This setting limits search requests as follows: -| Limit | Default (requests per minute) | -|-------------------------|-------------------------------| -| Authenticated user | 300 | -| Unauthenticated user | 100 | +| Limit | Default (requests per minute) | +|----------------------|-------------------------------| +| Authenticated user | 300 | +| Unauthenticated user | 100 | Search requests that exceed the search rate limit per minute return the following error: @@ -179,7 +171,7 @@ This endpoint has been requested too many times. Try again later. This setting limits the request rate to the pipeline creation endpoints. -Read more about [pipeline creation rate limits](../user/admin_area/settings/rate_limit_on_pipelines_creation.md). +Read more about [pipeline creation rate limits](settings/rate_limit_on_pipelines_creation.md). ## Gitaly concurrency limit @@ -191,8 +183,6 @@ Read more about [Gitaly concurrency limits](gitaly/configure_gitaly.md#limit-rpc ## Number of comments per issue, merge request, or commit -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22388) in GitLab 12.4. - There's a limit to the number of comments that can be submitted on an issue, merge request, or commit. When the limit is reached, system notes can still be added so that the history of events is not lost, but the user-submitted @@ -202,8 +192,6 @@ comment fails. ## Size of comments and descriptions of issues, merge requests, and epics -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61974) in GitLab 12.2. - There is a limit to the size of comments and descriptions of issues, merge requests, and epics. Attempting to add a body of text larger than the limit, results in an error, and the item is also not created. @@ -214,8 +202,6 @@ It's possible that this limit changes to a lower number in the future. ## Size of commit titles and descriptions -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292039) in GitLab 13.9. - Commits with arbitrarily large messages may be pushed to GitLab, but the following display limits apply: @@ -231,9 +217,6 @@ are processed. ## Number of issues in the milestone overview -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39453) in GitLab 12.10. -> - [Set](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/58168) to 500 in GitLab 13.11. - The maximum number of issues loaded on the milestone overview page is 500. When the number exceeds the limit the page displays an alert and links to a paginated [issue list](../user/project/issues/managing_issues.md) of all issues in the milestone. @@ -242,8 +225,6 @@ When the number exceeds the limit the page displays an alert and links to a pagi ## Number of pipelines per Git push -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51401) in GitLab 11.10. - When pushing multiple changes with a single Git push, like multiple tags or branches, only four tag or branch pipelines can be triggered. This limit prevents the accidental creation of a large number of pipelines when using `git push --all` or `git push --mirror`. @@ -259,12 +240,10 @@ instance if too many changes are pushed at once and a flood of pipelines are cre ## Retention of activity history -Activity history for projects and individuals' profiles was limited to one year until [GitLab 11.4](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52246) when it was extended to two years, and in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/33840) to three years. +Activity history for projects and individuals' profiles is limited to three years. ## Number of embedded metrics -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14939) in GitLab 12.7. - There is a limit when embedding metrics in GitLab Flavored Markdown (GLFM) for performance reasons. - **Max limit**: 100 embeds. @@ -343,8 +322,6 @@ Blocked recursive webhook calls are logged in `auth.log` with the message `"Recu ## Pull Mirroring Interval -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/237891) in GitLab 13.7. - The [minimum wait time between pull refreshes](../user/project/repository/mirror/index.md) defaults to 300 seconds (5 minutes). For example, a pull refresh only runs once in a given 300 second period, regardless of how many times you trigger it. @@ -362,8 +339,6 @@ Plan.default.actual_limits.update!(pull_mirror_interval_seconds: 200) ## Incoming emails from auto-responders -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30327) in GitLab 12.4. - GitLab ignores all incoming emails sent from auto-responders by looking for the `X-Autoreply` header. Such emails don't create comments on issues or merge requests. @@ -376,8 +351,6 @@ and to limit memory consumption. ## Max offset allowed by the REST API for offset-based pagination -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34565) in GitLab 13.0. - When using offset-based pagination in the REST API, there is a limit to the maximum requested offset into the set of results. This limit is only applied to endpoints that also support keyset-based pagination. More information about pagination options can be @@ -401,8 +374,6 @@ Set the limit to `0` to disable it. ### Number of jobs in active pipelines -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32823) in GitLab 12.6. - The total number of jobs in active pipelines can be limited per project. This limit is checked each time a new pipeline is created. An active pipeline is any pipeline in one of the following states: @@ -433,8 +404,6 @@ Set the limit to `0` to disable it. ### Maximum time jobs can run -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16777) in GitLab 12.3. - The default maximum time that jobs can run for is 60 minutes. Jobs that run for more than 60 minutes time out. @@ -447,8 +416,6 @@ You can change the maximum time a job can run before it times out: ### Maximum number of deployment jobs in a pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46931) in GitLab 13.7. - You can limit the maximum number of deployment jobs in a pipeline. A deployment is any job with an [`environment`](../ci/environments/index.md) specified. The number of deployments in a pipeline is checked at pipeline creation. Pipelines that have @@ -470,8 +437,6 @@ Set the limit to `0` to disable it. ### Number of CI/CD subscriptions to a project -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab 12.9. - The total number of subscriptions can be limited per project. This limit is checked each time a new subscription is created. @@ -516,8 +481,6 @@ This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd). ### Number of pipeline schedules -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29566) in GitLab 12.10. - The total number of pipeline schedules can be limited per project. This limit is checked each time a new pipeline schedule is created. If a new pipeline schedule would cause the total number of pipeline schedules to exceed the limit, the @@ -585,8 +548,6 @@ This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd). ### Number of instance level variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216097) in GitLab 13.1. - The total number of instance level CI/CD variables is limited at the instance level. This limit is checked each time a new instance level variable is created. If a new variable would cause the total number of variables to exceed the limit, the new variable is not created. @@ -639,12 +600,10 @@ Plan.default.actual_limits.update!(project_ci_variables: 10000) ### Maximum file size per type of artifact -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3. - Job artifacts defined with [`artifacts:reports`](../ci/yaml/index.md#artifactsreports) that are uploaded by the runner are rejected if the file size exceeds the maximum file size limit. The limit is determined by comparing the project's -[maximum artifact size setting](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size) +[maximum artifact size setting](../administration/settings/continuous_integration.md#maximum-artifacts-size) with the instance limit for the given artifact type, and choosing the smaller value. Limits are set in megabytes, so the smallest possible value that can be defined is `1 MB`. @@ -655,35 +614,35 @@ setting is used: | Artifact limit name | Default value | |---------------------------------------------|---------------| -| `ci_max_artifact_size_accessibility` | 0 | -| `ci_max_artifact_size_api_fuzzing` | 0 | -| `ci_max_artifact_size_archive` | 0 | -| `ci_max_artifact_size_browser_performance` | 0 | -| `ci_max_artifact_size_cluster_applications` | 0 | -| `ci_max_artifact_size_cobertura` | 0 | -| `ci_max_artifact_size_codequality` | 0 | -| `ci_max_artifact_size_container_scanning` | 0 | -| `ci_max_artifact_size_coverage_fuzzing` | 0 | -| `ci_max_artifact_size_dast` | 0 | -| `ci_max_artifact_size_dependency_scanning` | 0 | -| `ci_max_artifact_size_dotenv` | 0 | -| `ci_max_artifact_size_junit` | 0 | -| `ci_max_artifact_size_license_management` | 0 | -| `ci_max_artifact_size_license_scanning` | 0 | -| `ci_max_artifact_size_load_performance` | 0 | -| `ci_max_artifact_size_lsif` | 100 MB ([Introduced at 20 MB](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3 and [raised to 100 MB](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46980) in GitLab 13.6.) | -| `ci_max_artifact_size_metadata` | 0 | -| `ci_max_artifact_size_metrics_referee` | 0 | -| `ci_max_artifact_size_metrics` | 0 | -| `ci_max_artifact_size_network_referee` | 0 | -| `ci_max_artifact_size_performance` | 0 | -| `ci_max_artifact_size_requirements` | 0 | -| `ci_max_artifact_size_requirements_v2` | 0 | -| `ci_max_artifact_size_sast` | 0 | -| `ci_max_artifact_size_secret_detection` | 0 | -| `ci_max_artifact_size_terraform` | 5 MB ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37018) in GitLab 13.3) | -| `ci_max_artifact_size_trace` | 0 | -| `ci_max_artifact_size_cyclonedx` | 1 MB | +| `ci_max_artifact_size_accessibility` | 0 | +| `ci_max_artifact_size_api_fuzzing` | 0 | +| `ci_max_artifact_size_archive` | 0 | +| `ci_max_artifact_size_browser_performance` | 0 | +| `ci_max_artifact_size_cluster_applications` | 0 | +| `ci_max_artifact_size_cobertura` | 0 | +| `ci_max_artifact_size_codequality` | 0 | +| `ci_max_artifact_size_container_scanning` | 0 | +| `ci_max_artifact_size_coverage_fuzzing` | 0 | +| `ci_max_artifact_size_dast` | 0 | +| `ci_max_artifact_size_dependency_scanning` | 0 | +| `ci_max_artifact_size_dotenv` | 0 | +| `ci_max_artifact_size_junit` | 0 | +| `ci_max_artifact_size_license_management` | 0 | +| `ci_max_artifact_size_license_scanning` | 0 | +| `ci_max_artifact_size_load_performance` | 0 | +| `ci_max_artifact_size_lsif` | 100 MB | +| `ci_max_artifact_size_metadata` | 0 | +| `ci_max_artifact_size_metrics_referee` | 0 | +| `ci_max_artifact_size_metrics` | 0 | +| `ci_max_artifact_size_network_referee` | 0 | +| `ci_max_artifact_size_performance` | 0 | +| `ci_max_artifact_size_requirements` | 0 | +| `ci_max_artifact_size_requirements_v2` | 0 | +| `ci_max_artifact_size_sast` | 0 | +| `ci_max_artifact_size_secret_detection` | 0 | +| `ci_max_artifact_size_terraform` | 5 MB | +| `ci_max_artifact_size_trace` | 0 | +| `ci_max_artifact_size_cyclonedx` | 1 MB | For example, to set the `ci_max_artifact_size_junit` limit to 10 MB on a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -732,10 +691,10 @@ GitLab SaaS subscribers have different limits defined per plan, affecting all pr Self-managed GitLab Premium and Ultimate limits are defined by a default plan that affects all projects: -| Runner scope | Default value | -|---------------------------------------------|---------------| -| `ci_registered_group_runners` | 1000 | -| `ci_registered_project_runners` | 1000 | +| Runner scope | Default value | +|---------------------------------|---------------| +| `ci_registered_group_runners` | 1000 | +| `ci_registered_project_runners` | 1000 | To update these limits, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -808,7 +767,7 @@ You can change these limits in the [GitLab Rails console](operations/rails_conso You can set a limit on the maximum number of variables inside of a dotenv artifact. This limit is checked every time a dotenv file is exported as an artifact. -Set the limit to `0` to disable it. Defaults to `0` on self-managed instances. +Set the limit to `0` to disable it. Defaults to `20` on self-managed instances. To set this limit to `100` on a self-managed instance, run the following command in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -839,28 +798,20 @@ Plan.default.actual_limits.update!(dotenv_size: 5.kilobytes) ### Limit inbound incident management alerts -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17859) in GitLab 12.5. - This setting limits the number of inbound alert payloads over a period of time. -Read more about [incident management rate limits](../user/admin_area/settings/rate_limit_on_pipelines_creation.md). +Read more about [incident management rate limits](settings/rate_limit_on_pipelines_creation.md). ### Prometheus Alert JSON payloads -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19940) in GitLab 12.6. - Prometheus alert payloads sent to the `notify.json` endpoint are limited to 1 MB in size. ### Generic Alert JSON payloads -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16441) in GitLab 12.4. - Alert payloads sent to the `notify.json` endpoint are limited to 1 MB in size. ### Metrics dashboard YAML files -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34834) in GitLab 13.2. - The memory occupied by a parsed metrics dashboard YAML file cannot exceed 1 MB. The maximum depth of each YAML file is limited to 100. The maximum depth of a YAML @@ -891,8 +842,6 @@ panel_groups: ## Environment Dashboard limits **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33895) in GitLab 13.4. - See [Environment Dashboard](../ci/environments/environments_dashboard.md#adding-a-project-to-the-dashboard) for the maximum number of displayed projects. ## Environment data on deploy boards @@ -907,7 +856,7 @@ Kubernetes aren't shown. GitLab has limits around: -- The patch size for a single file. [This is configurable on self-managed instance](../user/admin_area/diff_limits.md). +- The patch size for a single file. [This is configurable on self-managed instance](../administration/diff_limits.md). - The total size of all the diffs for a merge request. An upper and lower limit applies to each of these: @@ -932,8 +881,6 @@ Reports that go over the 20 MB limit aren't loaded. Affected reports: ### Maximum file size indexed -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8638) in GitLab 13.3. - You can set a limit on the content of repository files that are indexed in Elasticsearch. Any files larger than this limit only index the filename. The file content is neither indexed nor searchable. @@ -949,8 +896,6 @@ is pre-allocated during indexing. ### Maximum field length -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201826) in GitLab 12.8. - You can set a limit on the content of text fields indexed for advanced search. Setting a maximum helps to reduce the load of the indexing processes. If any text field exceeds this limit, then the text is truncated to this number of @@ -983,12 +928,10 @@ See the limits in the [Add a design to an issue](../user/project/issues/design_m ### Max push size -The maximum allowed [push size](../user/admin_area/settings/account_and_limit_settings.md#max-push-size) is set to 5 GB. +The maximum allowed [push size](../administration/settings/account_and_limit_settings.md#max-push-size) is set to 5 GB. ### Webhooks and Project Services -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31009) in GitLab 12.4. - Total number of changes (branches or tags) in a single push. If changes are more than the specified limit, hooks are not executed. @@ -999,19 +942,15 @@ More information can be found in these documentations: ### Activities -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31007) in GitLab 12.4. - Total number of changes (branches or tags) in a single push to determine whether individual push events or a bulk push event are created. -More information can be found in the [Push event activities limit and bulk push events documentation](../user/admin_area/settings/push_event_activities_limit.md). +More information can be found in the [Push event activities limit and bulk push events documentation](settings/push_event_activities_limit.md). ## Package Registry Limits ### File Size Limits -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218017) in GitLab 13.4. - The default maximum file size for a package that's uploaded to the [GitLab Package Registry](../user/packages/package_registry/index.md) varies by format: - Conan: 3 GB @@ -1074,14 +1013,6 @@ varies by file type: - Image blob: 5 GB - Image manifest: 10 MB -## Branch retargeting on merge - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/320902) in GitLab 13.9. - -If a branch is merged while open merge requests still point to it, GitLab can -retarget merge requests pointing to the now-merged branch. For more information, see -[Update merge requests when target branch merges](../user/project/merge_requests/index.md#update-merge-requests-when-target-branch-merges). - ## Maximum number of assignees and reviewers > - Maximum assignees [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/368936) in GitLab 15.6. @@ -1117,3 +1048,108 @@ The [secure files API](../api/secure_files.md) enforces the following limits: The [changelog API](../api/repositories.md#add-changelog-data-to-a-changelog-file) enforces the following limits: - The commit range between `from` and `to` cannot exceed 15000 commits. + +## Value Stream Analytics limits + +- Each namespace (such as a group or a project) can have a maximum of 50 value streams. +- Each value stream can have a maximum of 15 stages. + +## List all instance limits + +To list all instance limit values, run the following from the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits +``` + +Sample output: + +```ruby +id: 1, +plan_id: 1, +ci_pipeline_size: 0, +ci_active_jobs: 0, +project_hooks: 100, +group_hooks: 50, +ci_project_subscriptions: 3, +ci_pipeline_schedules: 10, +offset_pagination_limit: 50000, +ci_instance_level_variables: "[FILTERED]", +storage_size_limit: 0, +ci_max_artifact_size_lsif: 100, +ci_max_artifact_size_archive: 0, +ci_max_artifact_size_metadata: 0, +ci_max_artifact_size_trace: "[FILTERED]", +ci_max_artifact_size_junit: 0, +ci_max_artifact_size_sast: 0, +ci_max_artifact_size_dependency_scanning: 350, +ci_max_artifact_size_container_scanning: 150, +ci_max_artifact_size_dast: 0, +ci_max_artifact_size_codequality: 0, +ci_max_artifact_size_license_management: 0, +ci_max_artifact_size_license_scanning: 100, +ci_max_artifact_size_performance: 0, +ci_max_artifact_size_metrics: 0, +ci_max_artifact_size_metrics_referee: 0, +ci_max_artifact_size_network_referee: 0, +ci_max_artifact_size_dotenv: 0, +ci_max_artifact_size_cobertura: 0, +ci_max_artifact_size_terraform: 5, +ci_max_artifact_size_accessibility: 0, +ci_max_artifact_size_cluster_applications: 0, +ci_max_artifact_size_secret_detection: "[FILTERED]", +ci_max_artifact_size_requirements: 0, +ci_max_artifact_size_coverage_fuzzing: 0, +ci_max_artifact_size_browser_performance: 0, +ci_max_artifact_size_load_performance: 0, +ci_needs_size_limit: 2, +conan_max_file_size: 3221225472, +maven_max_file_size: 3221225472, +npm_max_file_size: 524288000, +nuget_max_file_size: 524288000, +pypi_max_file_size: 3221225472, +generic_packages_max_file_size: 5368709120, +golang_max_file_size: 104857600, +debian_max_file_size: 3221225472, +project_feature_flags: 200, +ci_max_artifact_size_api_fuzzing: 0, +ci_pipeline_deployments: 500, +pull_mirror_interval_seconds: 300, +daily_invites: 0, +rubygems_max_file_size: 3221225472, +terraform_module_max_file_size: 1073741824, +helm_max_file_size: 5242880, +ci_registered_group_runners: 1000, +ci_registered_project_runners: 1000, +ci_daily_pipeline_schedule_triggers: 0, +ci_max_artifact_size_cluster_image_scanning: 0, +ci_jobs_trace_size_limit: "[FILTERED]", +pages_file_entries: 200000, +dast_profile_schedules: 1, +external_audit_event_destinations: 5, +dotenv_variables: "[FILTERED]", +dotenv_size: 5120, +pipeline_triggers: 25000, +project_ci_secure_files: 100, +repository_size: 0, +security_policy_scan_execution_schedules: 0, +web_hook_calls_mid: 0, +web_hook_calls_low: 0, +project_ci_variables: "[FILTERED]", +group_ci_variables: "[FILTERED]", +ci_max_artifact_size_cyclonedx: 1, +rpm_max_file_size: 5368709120, +pipeline_hierarchy_size: 1000, +ci_max_artifact_size_requirements_v2: 0, +enforcement_limit: 0, +notification_limit: 0, +dashboard_limit_enabled_at: nil, +web_hook_calls: 0, +project_access_token_limit: 0, +google_cloud_logging_configurations: 5, +ml_model_max_file_size: 10737418240, +limits_history: {} +``` + +Some limit values display as `[FILTERED]` in the list due to +[filtering in the Rails console](operations/rails_console.md#filtered-console-output). diff --git a/doc/administration/integration/diagrams_net.md b/doc/administration/integration/diagrams_net.md index a4e8528fb25..335b26565e6 100644 --- a/doc/administration/integration/diagrams_net.md +++ b/doc/administration/integration/diagrams_net.md @@ -8,7 +8,7 @@ type: reference, howto # Diagrams.net **(FREE)** With the [diagrams.net](https://www.diagrams.net/) integration, you can create and embed SVG diagrams in wikis. -The diagram editor is available in both the Markdown editor and the content editor. +The diagram editor is available in both the plain text editor and the rich text editor. On GitLab.com, this integration is enabled for all SaaS users and does not require any additional configuration. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 9c5e07eedaa..f05197d45e7 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -157,7 +157,6 @@ using Tomcat or Jetty. Prerequisites: - JRE/JDK version 11 or later. -- Apache Maven version 3.0.2 or later. - (Recommended) Jetty version 11 or later. - (Recommended) Tomcat version 10 or later. @@ -167,11 +166,11 @@ PlantUML recommends to install Tomcat 10 or above. The scope of this page only includes setting up a basic Tomcat server. For more production-ready configurations, see the [Tomcat Documentation](https://tomcat.apache.org/tomcat-10.1-doc/index.html). -1. Install JDK/JRE 11 and Maven: +1. Install JDK/JRE 11: ```shell sudo apt update - sudo apt-get install graphviz default-jdk git-core maven + sudo apt-get install graphviz default-jdk git-core ``` 1. Add a user for Tomcat: @@ -183,8 +182,8 @@ see the [Tomcat Documentation](https://tomcat.apache.org/tomcat-10.1-doc/index.h 1. Install and configure Tomcat 10: ```shell - cd /tmp & wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.9/bin/apache-tomcat-10.1.9.tar.gz - sudo tar xzvf apache-tomcat-10*tar.gz -C /opt/tomcat --strip-components=1 + wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.9/bin/apache-tomcat-10.1.9.tar.gz -P /tmp + sudo tar xzvf /tmp/apache-tomcat-10*tar.gz -C /opt/tomcat --strip-components=1 sudo chown -R tomcat:tomcat /opt/tomcat/ sudo chmod -R u+x /opt/tomcat/bin ``` @@ -248,7 +247,7 @@ see the [Tomcat Documentation](https://tomcat.apache.org/tomcat-10.1-doc/index.h tcp6 0 0 :::8005 :::* LISTEN 14935/java ``` -1. Modify your NGINX configuration. The `proxy_pass` port matches the Connector port in the `server.xml`: +1. Modify your NGINX configuration in `/etc/gitlab/gitlab.rb`. Ensure the `proxy_pass` port matches the Connector port in `server.xml`: ```shell nginx['custom_gitlab_server_config'] = "location /-/plantuml { @@ -269,19 +268,22 @@ see the [Tomcat Documentation](https://tomcat.apache.org/tomcat-10.1-doc/index.h 1. Install PlantUML and copy the `.war` file: + Use the [latest release](https://github.com/plantuml/plantuml-server/releases) of plantuml-jsp (example: plantuml-jsp-v1.2023.8.war). For context, see [this issue](https://github.com/plantuml/plantuml-server/issues/265). + ```shell - cd / & git clone https://github.com/plantuml/plantuml-server.git - cd plantuml-server - mvn package - cp /plantuml-server/target/plantuml.war /opt/tomcat/webapps/plantuml.war - chown tomcat:tomcat /opt/tomcat/webapps/plantuml.war - systemctl restart tomcat + cd / + wget https://github.com/plantuml/plantuml-server/releases/download/v1.2023.8/plantuml-jsp-v1.2023.8.war + sudo cp plantuml-jsp-v1.2023.8.war /opt/tomcat/webapps/plantuml.war + sudo chown tomcat:tomcat /opt/tomcat/webapps/plantuml.war + sudo systemctl restart tomcat ``` The Tomcat service should restart. After the restart is complete, the PlantUML integration is ready and listening for requests on port `8005`: `http://localhost:8005/plantuml` +To test if the PlantUML server is working, run `curl --location --verbose "http://localhost:8005/plantuml/"`. + To change the Tomcat defaults, edit the `/opt/tomcat/conf/server.xml` file. NOTE: @@ -293,6 +295,7 @@ the configuration below accordingly. PlantUML has features that allow fetching network resources. If you self-host the PlantUML server, put network controls in place to isolate it. +For example, make use of PlantUML's [security profiles](https://plantuml.com/security). ```plaintext @startuml @@ -320,7 +323,7 @@ these steps: - For PlantUML servers running v1.2020.9 and above, such as [plantuml.com](https://plantuml.com), you must set the `PLANTUML_ENCODING` environment variable to enable the `deflate` - compression. In Linux package installations, you can set this value in `/etc/gitlab.rb` with + compression. In Linux package installations, you can set this value in `/etc/gitlab/gitlab.rb` with this command: ```ruby @@ -337,6 +340,6 @@ these steps: PLANTUML_ENCODING: deflate ``` -- For GitLab versions 13.1 and later, PlantUML integration now - [requires a header prefix in the URL](https://github.com/plantuml/plantuml/issues/117#issuecomment-6235450160) +- `deflate` is the default encoding type for PlantUML. To use a different encoding type, PlantUML integration + [requires a header prefix in the URL](https://plantuml.com/text-encoding) to distinguish different encoding types. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index e5920520be7..1ab45d6ce99 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -13,7 +13,7 @@ WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `certificate_based_clusters`. +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `certificate_based_clusters`. - Read more about the non-deprecated [Web Terminals accessible through the Web IDE](../../user/project/web_ide/index.md). - Read more about the non-deprecated [Web Terminals accessible from a running CI job](../../ci/interactive_web_terminal/index.md). @@ -114,5 +114,5 @@ lifetime in your GitLab instance: 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). 1. Select **Admin Area**. -1. Select [**Settings > Web terminal**](../../user/admin_area/settings/index.md#general). +1. Select [**Settings > Web terminal**](../../administration/settings/index.md#general). 1. Set a `max session time`. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 099e7ec9e6f..106334b226d 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -282,7 +282,7 @@ To migrate back to local storage, you must If [`artifacts:expire_in`](../ci/yaml/index.md#artifactsexpire_in) is used to set an expiry for the artifacts, they are marked for deletion right after that date passes. -Otherwise, they expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md). +Otherwise, they expire per the [default artifacts expiration setting](../administration/settings/continuous_integration.md). Artifacts are cleaned up by the `expire_build_artifacts_worker` cron job which Sidekiq runs every 7 minutes (`*/7 * * * *` in [Cron](../topics/cron/index.md) syntax). @@ -376,7 +376,7 @@ To change the default schedule on which the artifacts are expired: ## Set the maximum file size of the artifacts If artifacts are enabled, you can change the maximum file size of the -artifacts through the [Admin Area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). +artifacts through the [Admin Area settings](../administration/settings/continuous_integration.md#maximum-artifacts-size). ## Storage statistics diff --git a/doc/administration/labels.md b/doc/administration/labels.md new file mode 100644 index 00000000000..adc621a2982 --- /dev/null +++ b/doc/administration/labels.md @@ -0,0 +1,28 @@ +--- +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Labels administration **(FREE SELF)** + +To manage labels for the GitLab instance, select **Labels** (**{labels}**) from the Admin Area sidebar. For more details on how to manage labels, see [Labels](../user/project/labels.md). + +Labels created in the Admin Area are automatically added to new projects. +They are not available to new groups. +Updating or adding labels in the Admin Area does not modify labels in existing projects. + +![Default label set](img/admin_labels_v14_7.png) + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index e4475e877d8..4659917cf8b 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -391,9 +391,11 @@ To delete these references: ls -al /var/opt/gitlab/gitlab-rails/shared/lfs-objects/00/66/22269c61b41bf14a22bbe0e43be3acf86a4a446afb4250c3794ea47541a7 ``` -1. If the file is not present, remove the database record via the rails console: +1. If the file is not present, remove the database records via the rails console: ```ruby + # First delete the parent records and then destroy the record itself + lfs_object.lfs_objects_projects.destroy_all lfs_object.destroy ``` diff --git a/doc/administration/license.md b/doc/administration/license.md new file mode 100644 index 00000000000..732c2840217 --- /dev/null +++ b/doc/administration/license.md @@ -0,0 +1,83 @@ +--- +stage: Fulfillment +group: Provision +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Activate GitLab Enterprise Edition (EE) **(PREMIUM SELF)** + +When you install a new GitLab instance without a license, only Free features +are enabled. To enable more features in GitLab Enterprise Edition (EE), activate +your instance with an activation code. + +## Activate GitLab EE + +In GitLab Enterprise Edition 14.1 and later, you need an activation code to activate +your instance. + +Prerequisite: + +- You must [purchase a subscription](https://about.gitlab.com/pricing/). +- You must be running GitLab Enterprise Edition (EE). +- You must have GitLab 14.1 or later. +- Your instance must be connected to the internet. + +To activate your instance with an activation code: + +1. Copy the activation code, a 24-character alphanumeric string, from either: + - Your subscription confirmation email. + - The [Customers Portal](https://customers.gitlab.com/customers/sign_in), on the **Manage Purchases** page. +1. Sign in to your GitLab self-managed instance. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Subscription**. +1. Paste the activation code in **Activation code**. +1. Read and accept the terms of service. +1. Select **Activate**. + +The subscription is activated. + +If you have an offline environment, +[activate GitLab EE with a license file or key](license_file.md) instead. + +If you have questions or need assistance activating your instance, +[contact GitLab Support](https://about.gitlab.com/support/#contact-support). + +When [the license expires](../administration/license_file.md#what-happens-when-your-license-expires), +some functionality is locked. + +## Verify your GitLab edition + +To verify the edition, sign in to GitLab and select +**Help** (**{question-o}**) > **Help**. The GitLab edition and version are listed +at the top of the page. + +If you are running GitLab Community Edition, you can upgrade your installation to GitLab +EE. For more details, see [Upgrading between editions](../update/index.md#upgrading-between-editions). +If you have questions or need assistance upgrading from GitLab Community Edition (CE) to EE, +[contact GitLab Support](https://about.gitlab.com/support/#contact-support). + +## Troubleshooting + +### Cannot activate instance due to connectivity error + +This error occurs when you use an activation code to activate your instance, but your instance is unable to connect to the GitLab servers. + +You may have connectivity issues due to the following reasons: + +- **You have an offline environment**: + - Configure your setup to allow connection to GitLab servers. If connection to GitLab servers is not possible, contact your Sales Representative to request a license key. You can also contact [GitLab support](https://about.gitlab.com/support/#contact-support) if you need help finding your Sales Representative. +- **Customers Portal is not operational**: + - To check for performance or service disruptions, check the Customers Portal [status](https://status.gitlab.com/). +- **Firewall settings**: + - Check if your GitLab instance has an encrypted connection to `customers.gitlab.com` (with IP addresses 172.64.146.11 and 104.18.41.245) on port 443: + + ```shell + curl --verbose "https://customers.gitlab.com/" + ``` + + - If the curl command returns a failure, either: + - [Configure a proxy](https://docs.gitlab.com/omnibus/settings/environment-variables.html) in `gitlab.rb` to point to your server. + - Contact your network administrator to make changes to the proxy. + - If an SSL inspection appliance is used, you must add the appliance's root CA certificate to `/etc/gitlab/trusted-certs` on the server, then run `gitlab-ctl reconfigure`. +
\ No newline at end of file diff --git a/doc/administration/license_file.md b/doc/administration/license_file.md new file mode 100644 index 00000000000..5f82536698b --- /dev/null +++ b/doc/administration/license_file.md @@ -0,0 +1,269 @@ +--- +stage: Fulfillment +group: Provision +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +<!-- To promote the workflow described in license.md, this page is not included in global left nav. --> + +# Activate GitLab EE with a license file or key + +If you receive a license file from GitLab (for example, for a trial), you can +upload it to your instance or add it during installation. The license file is +a base64-encoded ASCII text file with a `.gitlab-license` extension. + +The first time you sign in to your GitLab instance, a note with a +link to the **Add license** page should be displayed. + +Otherwise, to add your license: + +1. Sign in to GitLab as an administrator. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. In the **Add License** area, add a license by either uploading the file or entering the key. +1. Select the **Terms of Service** checkbox. +1. Select **Add license**. + +NOTE: +In GitLab 14.7.x to 14.9.x, you can add the license file with the UI. +In GitLab 14.1.x to 14.7, if you have already activated your subscription with an activation code, you cannot access **Add License** from the Admin Area. You must access **Add License** directly from the URL, `<YourGitLabURL>/admin/license/new`. + +## Activate subscription during installation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114572) in GitLab 16.0. + +To activate your subscription during installation, set the `GITLAB_ACTIVATION_CODE` environment variable with the activation code: + +```shell +export GITLAB_ACTIVATION_CODE=your_activation_code +``` + +## Add license file during installation + +If you have a license, you can also import it when you install GitLab. + +- For self-compiled installations: + - Place the `Gitlab.gitlab-license` file in the `config/` directory. + - To specify a custom location and filename for the license, set the + `GITLAB_LICENSE_FILE` environment variable with the path to the file: + + ```shell + export GITLAB_LICENSE_FILE="/path/to/license/file" + ``` + +- For Linux package installations: + - Place the `Gitlab.gitlab-license` file in the `/etc/gitlab/` directory. + - To specify a custom location and filename for the license, add this entry to `gitlab.rb`: + + ```ruby + gitlab_rails['initial_license_file'] = "/path/to/license/file" + ``` + +WARNING: +These methods only add a license at the time of installation. To renew or upgrade +a license, add the license in the **Admin Area** in the web user interface. + +## Submit license usage data + +If you use a license file or key to activate your instance in an offline environment, you must submit your license +usage data monthly. +To submit the data, [export your license usage](../subscriptions/self_managed/index.md#export-your-license-usage) +and send it by email to the renewals service, `renewals-service@customers.gitlab.com`. + +If you don't submit your data each month after your subscription start date, an email is sent to the address +associated with your subscription and a banner displays to remind you to submit your data. The banner displays +in the **Admin Area** on the **Dashboard** and on the **Subscription** pages. You can only dismiss it until the +following month after you submit your license usage data. + +## What happens when your license expires + +Fifteen days before the license expires, a notification banner with the upcoming expiration +date displays to GitLab administrators. + +When your license expires, GitLab locks features, like Git pushes +and issue creation. Your instance becomes read-only and +an expiration message displays to all administrators. You have a 14-day grace period +before this occurs. + +To resume functionality, [renew your subscription](../subscriptions/self_managed/index.md#renew-subscription-manually). + +If the license has been expired for more than 30 days, you must purchase a [new subscription](../subscriptions/self_managed/index.md) to resume functionality. + +To go back to Free features, [delete all expired licenses](#remove-a-license). + +## Remove a license + +To remove a license from a self-managed instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Subscription**. +1. Select **Remove license**. + +Repeat these steps to remove all licenses, including those applied in the past. + +## View license details and history + +To view your license details: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Subscription**. + +You can add and view more than one license, but only the latest license in +the current date range is the active license. + +When you add a future-dated license, it doesn't take effect until its applicable date. +You can view all active subscriptions in the **Subscription history** table. + +You can also [export](../subscriptions/self_managed/index.md) your license usage information to a CSV file. + +NOTE: +In GitLab 13.6 and earlier, a banner about an expiring license may continue to display +when you add a new license. This happens when the start date of the new license +is in the future and the expiring one is still active. +The banner disappears after the new license becomes active. + +## License commands in the Rails console + +The following commands can be run in the [Rails console](../administration/operations/rails_console.md#starting-a-rails-console-session). + +WARNING: +Any command that changes data directly could be damaging if not run correctly, or under the right conditions. +We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. + +### See current license information + +```ruby +# License information (name, company, email address) +License.current.licensee + +# Plan: +License.current.plan + +# Uploaded: +License.current.created_at + +# Started: +License.current.starts_at + +# Expires at: +License.current.expires_at + +# Is this a trial license? +License.current.trial? + +# License ID for lookup on CustomersDot +License.current.license_id + +# License data in Base64-encoded ASCII format +License.current.data + +# Confirm the current billable seat count excluding guest users. This is useful for customers who use an Ultimate subscription tier where Guest seats are not counted. +User.active.without_bots.excluding_guests.count + +``` + +#### Interaction with licenses that start in the future + +```ruby +# Future license data follows the same format as current license data it just uses a different modifier for the License prefix +License.future_dated +``` + +### Check if a project feature is available on the instance + +Features listed in [`features.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/gitlab_subscriptions/features.rb). + +```ruby +License.current.feature_available?(:jira_dev_panel_integration) +``` + +#### Check if a project feature is available in a project + +Features listed in [`features.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/gitlab_subscriptions/features.rb). + +```ruby +p = Project.find_by_full_path('<group>/<project>') +p.feature_available?(:jira_dev_panel_integration) +``` + +### Add a license through the console + +#### Using a `key` variable + +```ruby +key = "<key>" +license = License.new(data: key) +license.save +License.current # check to make sure it applied +``` + +#### Using a license file + +```ruby +license_file = File.open("/tmp/Gitlab.license") + +key = license_file.read.gsub("\r\n", "\n").gsub(/\n+$/, '') + "\n" + +license = License.new(data: key) +license.save +License.current # check to make sure it applied +``` + +These snippets can be saved to a file and executed [using the Rails Runner](operations/rails_console.md#using-the-rails-runner) so the +license can be applied via shell automation scripts. + +This is needed for example in a known edge-case with +[expired license and multiple LDAP servers](../administration/auth/ldap/ldap-troubleshooting.md#expired-license-causes-errors-with-multiple-ldap-servers). + +### Remove licenses + +To clean up the [License History table](../administration/license_file.md#view-license-details-and-history): + +```ruby +TYPE = :trial? +# or :expired? + +License.select(&TYPE).each(&:destroy!) + +# or even License.all.each(&:destroy!) +``` + +## Troubleshooting + +### No Subscription area in the Admin Area + +You cannot add your license because there is no **Subscription** area. +This issue might occur if: + +- You're running GitLab Community Edition. Before you add your license, you + must [upgrade to Enterprise Edition](../update/index.md#community-to-enterprise-edition). +- You're using GitLab.com. You cannot add a self-managed license to GitLab.com. + To use paid features on GitLab.com, [purchase a separate subscription](../subscriptions/gitlab_com/index.md). + +### Users exceed license limit upon renewal + +GitLab displays a message prompting you to purchase +additional users. This issue occurs if you add a license that does not have enough +users to cover the number of users in your instance. + +To fix this issue, purchase additional seats to cover those users. +For more information, read the [licensing FAQ](https://about.gitlab.com/pricing/licensing-faq/). + +In GitLab 14.2 and later, for instances that use a license file, the following +rules apply: + +- If the users over license are less than or equal to 10% of the users in the license + file, the license is applied and you pay the overage in the next renewal. +- If the users over license are more than 10% of the users in the license file, + you cannot apply the license without purchasing more users. + +For example, if you purchase a license for 100 users, you can have 110 users when you add +your license. However, if you have 111 users, you must purchase more users before you can add +the license. + +### `Start GitLab Ultimate trial` still displays after adding license + +To fix this issue, restart [Puma or your entire GitLab instance](../administration/restart_gitlab.md). diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index e43fe851aa2..a862fd46a3f 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -113,7 +113,7 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. ## Readiness check -It is strongly recommend that multi-node deployments configure load balancers to use the [readiness check](../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma doesn't accept requests. +It is strongly recommend that multi-node deployments configure load balancers to use the [readiness check](../administration/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma doesn't accept requests. WARNING: Using the `all=1` parameter with the readiness check in GitLab versions 15.4 to 15.8 may cause [increased Praefect memory usage](https://gitlab.com/gitlab-org/gitaly/-/issues/4751) and lead to memory errors. diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index 8dcb25e22df..449f33fbbef 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -754,8 +754,8 @@ This file is located at: This log records: -- Requests over the [Rate Limit](../../user/admin_area/settings/rate_limits_on_raw_endpoints.md) on raw endpoints. -- [Protected paths](../../user/admin_area/settings/protected_paths.md) abusive requests. +- Requests over the [Rate Limit](../settings/rate_limits_on_raw_endpoints.md) on raw endpoints. +- [Protected paths](../settings/protected_paths.md) abusive requests. - In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and later, user ID and username, if available. @@ -1129,7 +1129,7 @@ GitLab also tracks [Prometheus metrics for Praefect](../gitaly/monitoring.md#mon For Omnibus installations, the backup log is located at `/var/log/gitlab/gitlab-rails/backup_json.log`. -This log is populated when a [GitLab backup is created](../../raketasks/backup_restore.md). You can use this log to understand how the backup process performed. +This log is populated when a [GitLab backup is created](../../administration/backup_restore/index.md). You can use this log to understand how the backup process performed. ## Performance bar stats diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 3bbebe7ecce..336067d1891 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -128,8 +128,12 @@ For most JSON requests, `POST`, `PUT`, `PATCH`, and `DELETE` are blocked, and th ### GraphQL API +> The `GeoRegistriesUpdate` mutation addition in the allowlist was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124259) in GitLab 16.2. + `POST /api/graphql` requests are allowed but mutations are blocked with the error message `You cannot perform write operations on a read-only instance`. +The only mutation that is allowed is the `GeoRegistriesUpdate` which is used to resync and reverify registries. + ### Continuous Integration - No new jobs or pipelines start, scheduled or otherwise. @@ -194,7 +198,8 @@ When primary is in Maintenance Mode, secondary also automatically goes into Main It is important that you do not disable replication before enabling Maintenance Mode. -Replication and verification continues to work but proxied Git pushes to primary do not work. +Replication, verification and manual actions to resync and reverify registries through the Admin UI +continue to work, but proxied Git pushes to primary don't. ### Secure features diff --git a/doc/administration/merge_requests_approvals.md b/doc/administration/merge_requests_approvals.md new file mode 100644 index 00000000000..6cd0edf22eb --- /dev/null +++ b/doc/administration/merge_requests_approvals.md @@ -0,0 +1,43 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, concepts +--- + +# Merge request approvals **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39060) in GitLab 12.8. + +Merge request approval rules prevent users from overriding certain settings on the project level. +When enabled at the instance level, these settings [cascade](../user/project/merge_requests/approvals/settings.md#settings-cascading) +and can no longer be changed: + +- In projects. +- In groups. Cascading to groups was [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) + in GitLab 14.5. + +To enable merge request approval settings for an instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Push Rules**. +1. Expand **Merge request approvals**. +1. Choose the required options. +1. Select **Save changes**. + +## Available rules + +Merge request approval settings that can be set at an instance level are: + +- **Prevent approval by author**. Prevents project maintainers from allowing request authors to + merge their own merge requests. +- **Prevent approvals by users who add commits**. Prevents project maintainers from allowing users + to approve merge requests if they have submitted any commits to the source branch. +- **Prevent editing approval rules in projects and merge requests**. Prevents users from modifying + the approvers list in project settings or in individual merge requests. + +See also the following, which are affected by instance-level rules: + +- [Project merge request approval rules](../user/project/merge_requests/approvals/index.md). +- [Group merge request approval settings](../user/group/manage.md#group-merge-request-approval-settings) available in GitLab 13.9 and later. diff --git a/doc/administration/moderate_users.md b/doc/administration/moderate_users.md new file mode 100644 index 00000000000..42f1f26586f --- /dev/null +++ b/doc/administration/moderate_users.md @@ -0,0 +1,393 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto +--- + +# Moderate users (administration) **(FREE SELF)** + +This is the administration documentation. For information about moderating users at the group level, see the [group-level documentation](../user/group/moderate_users.md). + +GitLab administrators can moderate user access by approving, blocking, banning, or deactivating +users. + +## Users pending approval + +A user in _pending approval_ state requires action by an administrator. A user sign up can be in a +pending approval state because an administrator has enabled any of the following options: + +- [Require administrator approval for new sign-ups](../administration/settings/sign_up_restrictions.md#require-administrator-approval-for-new-sign-ups) setting. +- [User cap](../administration/settings/sign_up_restrictions.md#user-cap). +- [Block auto-created users (OmniAuth)](../integration/omniauth.md#configure-common-settings) +- [Block auto-created users (LDAP)](../administration/auth/ldap/index.md#basic-configuration-settings) + +When a user registers for an account while this setting is enabled: + +- The user is placed in a **Pending approval** state. +- The user sees a message telling them their account is awaiting approval by an administrator. + +A user pending approval: + +- Is functionally identical to a [blocked](#block-a-user) user. +- Cannot sign in. +- Cannot access Git repositories or the GitLab API. +- Does not receive any notifications from GitLab. +- Does not consume a [seat](../subscriptions/self_managed/index.md#billable-users). + +An administrator must [approve their sign up](#approve-or-reject-a-user-sign-up) to allow them to +sign in. + +### View user sign ups pending approval + +To view user sign ups pending approval: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Pending approval** tab. + +### Approve or reject a user sign up + +A user sign up pending approval can be approved or rejected from the Admin Area. + +To approve or reject a user sign up: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Pending approval** tab. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Approve** or **Reject**. + +Approving a user: + +- Activates their account. +- Changes the user's state to active. +- Consumes a subscription [seat](../subscriptions/self_managed/index.md#billable-users). + +## Block and unblock users + +GitLab administrators can block and unblock users. + +### Block a user + +To completely prevent access of a user to the GitLab instance, +administrators can choose to block the user. + +Users can be blocked [via an abuse report](../administration/review_abuse_reports.md#blocking-users), +by removing them in LDAP, or directly from the Admin Area. To do this: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Block**. + +A blocked user: + +- Cannot sign in. +- Cannot access Git repositories or the API. +- Does not receive any notifications from GitLab. +- Cannot use [slash commands](../user/project/integrations/gitlab_slack_application.md#slash-commands). +- Does not consume a [seat](../subscriptions/self_managed/index.md#billable-users). + +Personal projects, and group and user history of the blocked user are left intact. + +NOTE: +Users can also be blocked using the [GitLab API](../api/users.md#block-user). + +### Unblock a user + +A blocked user can be unblocked from the Admin Area. To do this: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Blocked** tab. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Unblock**. + +The user's state is set to active and they consume a +[seat](../subscriptions/self_managed/index.md#billable-users). + +NOTE: +Users can also be unblocked using the [GitLab API](../api/users.md#unblock-user). + +The unblock option may be unavailable for LDAP users. To enable the unblock option, +the LDAP identity first needs to be deleted: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Blocked** tab. +1. Select a user. +1. Select the **Identities** tab. +1. Find the LDAP provider and select **Delete**. + +## Activate and deactivate users + +GitLab administrators can deactivate and activate users. + +### Deactivate a user + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22257) in GitLab 12.4. + +To temporarily prevent access by a GitLab user that has no recent activity, +administrators can choose to deactivate the user. + +Deactivating a user is functionally identical to [blocking a user](#block-and-unblock-users), +with the following differences: + +- It does not prohibit the user from logging back in via the UI. +- Once a deactivated user logs back into the GitLab UI, their account is set to active. + +A deactivated user: + +- Cannot access Git repositories or the API. +- Does not receive any notifications from GitLab. +- Cannot use [slash commands](../user/project/integrations/gitlab_slack_application.md#slash-commands). +- Does not consume a [seat](../subscriptions/self_managed/index.md#billable-users). + +Personal projects, and group and user history of the deactivated user are left intact. + +NOTE: +Users are notified about account deactivation if +[user deactivation emails](../administration/settings/email.md#user-deactivation-emails) are enabled. + +A user can be deactivated from the Admin Area. To do this: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Deactivate**. + +For the deactivation option to be visible to an administrator, the user: + +- Must have a state of active. +- Must be [dormant](#automatically-deactivate-dormant-users). + +NOTE: +Users can also be deactivated using the [GitLab API](../api/users.md#deactivate-user). + +### Automatically deactivate dormant users + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/320875) in GitLab 14.0. +> - Exclusion of GitLab generate bots [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340346) in GitLab 14.5 +> - Customizable time period [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336747) in GitLab 15.4 +> - The lower limit for inactive period set to 90 days [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/100793) in GitLab 15.5 + +Administrators can enable automatic deactivation of users who either: + +- Were created more than a week ago and have not signed in. +- Have no activity for a specified period of time (default and minimum is 90 days). + +To do this: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Account and limit** section. +1. Under **Dormant users**, check **Deactivate dormant users after a period of inactivity**. +1. Under **Days of inactivity before deactivation**, enter the number of days before deactivation. Minimum value is 90 days. +1. Select **Save changes**. + +When this feature is enabled, GitLab runs a job once a day to deactivate the dormant users. + +A maximum of 100,000 users can be deactivated per day. + +NOTE: +GitLab generated bots are excluded from the automatic deactivation of dormant users. + +### Automatically delete unconfirmed users **(PREMIUM SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352514) in GitLab 16.1 [with a flag](../administration/feature_flags.md) named `delete_unconfirmed_users_setting`. Disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/124982) in GitLab 16.2. + +Prerequisites: + +- You must be an administrator. + +You can enable automatic deletion of users who both: + +- Never confirmed their email address. +- Signed up for GitLab more than a specified number of days in the past. + +You can configure these settings using either the [Settings API](../api/settings.md) or in a Rails console: + +```ruby + Gitlab::CurrentSettings.update(delete_unconfirmed_users: true) + Gitlab::CurrentSettings.update(unconfirmed_users_delete_after_days: 365) +``` + +When the `delete_unconfirmed_users` setting is enabled, GitLab runs a job once an hour to delete the unconfirmed users. +The job only deletes users who signed up more than `unconfirmed_users_delete_after_days` days in the past. + +This job only runs when the `email_confirmation_setting` is set to `soft` or `hard`. + +A maximum of 240,000 users can be deleted per day. + +### Activate a user + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22257) in GitLab 12.4. + +A deactivated user can be activated from the Admin Area. + +To do this: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Deactivated** tab. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Activate**. + +The user's state is set to active and they consume a +[seat](../subscriptions/self_managed/index.md#billable-users). + +NOTE: +A deactivated user can also activate their account themselves by logging back in via the UI. +Users can also be activated using the [GitLab API](../api/users.md#activate-user). + +## Ban and unban users + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327353) in GitLab 14.2 [with a flag](../administration/feature_flags.md) named `ban_user_feature_flag`. Disabled by default. +> - Ban and unban users [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/327353) in GitLab 14.8. Feature flag `ban_user_feature_flag` removed. +> - Hiding merge requests of banned users [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107836) in GitLab 15.8 [with a flag](../administration/feature_flags.md) named `hide_merge_requests_from_banned_users`. Disabled by default. +> - Hiding comments of banned users [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112973) in GitLab 15.11 [with a flag](../administration/feature_flags.md) named `hidden_notes`. Disabled by default. +> - Hiding projects of banned users [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/121488) in GitLab 16.2 [with a flag](../administration/feature_flags.md) named `hide_projects_of_banned_users`. Disabled by default. + +GitLab administrators can ban and unban users. Banned users are blocked, and their projects, issues, merge requests, and comments are hidden. + +### Ban a user + +To block a user and hide their contributions, administrators can ban the user. + +Users can be banned using the Admin Area. To do this: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Ban user**. + +The banned user does not consume a [seat](../subscriptions/self_managed/index.md#billable-users). + +### Unban a user + +A banned user can be unbanned using the Admin Area. To do this: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Banned** tab. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Unban user**. + +The user's state is set to active and they consume a +[seat](../subscriptions/self_managed/index.md#billable-users). + +### Delete a user + +Use the Admin Area to delete users. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Banned** tab. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Delete user**. +1. Type the username. +1. Select **Delete user**. + +NOTE: +You can only delete a user if there are inherited or direct owners of a group. You cannot delete a user if they are the only group owner. + +You can also delete a user and their contributions, such as merge requests, issues, and groups of which they are the only group owner. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Banned** tab. +1. Optional. Select a user. +1. Select the **{settings}** **User administration** dropdown list. +1. Select **Delete user and contributions**. +1. Type the username. +1. Select **Delete user and contributions**. + +NOTE: +Before 15.1, additionally groups of which deleted user were the only owner among direct members were deleted. + +## Troubleshooting + +When moderating users, you may need to perform bulk actions on them based on certain conditions. The following rails console scripts show some examples of this. You may [start a rails console session](../administration/operations/rails_console.md#starting-a-rails-console-session) and use scripts similar to the following: + +### Deactivate users that have no recent activity + +Administrators can deactivate users that have no recent activity. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +```ruby +days_inactive = 90 +inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) + +inactive_users.each do |user| + puts "user '#{user.username}': #{user.last_activity_on}" + user.deactivate! +end +``` + +### Block users that have no recent activity + +Administrators can block users that have no recent activity. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +```ruby +days_inactive = 90 +inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) + +inactive_users.each do |user| + puts "user '#{user.username}': #{user.last_activity_on}" + user.block! +end +``` + +### Block or delete users that have no projects or groups + +Administrators can block or delete users that have no projects or groups. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +```ruby +users = User.where('id NOT IN (select distinct(user_id) from project_authorizations)') + +# How many users are removed? +users.count + +# If that count looks sane: + +# You can either block the users: +users.each { |user| user.blocked? ? nil : user.block! } + +# Or you can delete them: + # need 'current user' (your user) for auditing purposes +current_user = User.find_by(username: '<your username>') + +users.each do |user| + DeleteUserWorker.perform_async(current_user.id, user.id) +end +``` diff --git a/doc/administration/monitoring/health_check.md b/doc/administration/monitoring/health_check.md new file mode 100644 index 00000000000..4dbbdf6f3c9 --- /dev/null +++ b/doc/administration/monitoring/health_check.md @@ -0,0 +1,146 @@ +--- +stage: Monitor +group: Respond +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Health Check **(FREE SELF)** + +GitLab provides liveness and readiness probes to indicate service health and +reachability to required services. These probes report on the status of the +database connection, Redis connection, and access to the file system. These +endpoints [can be provided to schedulers like Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) to hold +traffic until the system is ready or restart the container as needed. + +## IP allowlist + +To access monitoring resources, the requesting client IP needs to be included in the allowlist. +For details, see [how to add IPs to the allowlist for the monitoring endpoints](../../administration/monitoring/ip_allowlist.md). + +## Using the endpoints locally + +With default allowlist settings, the probes can be accessed from localhost using the following URLs: + +```plaintext +GET http://localhost/-/health +``` + +```plaintext +GET http://localhost/-/readiness +``` + +```plaintext +GET http://localhost/-/liveness +``` + +## Health + +Checks whether the application server is running. +It does not verify the database or other services +are running. This endpoint circumvents Rails Controllers +and is implemented as additional middleware `BasicHealthCheck` +very early into the request processing lifecycle. + +```plaintext +GET /-/health +``` + +Example request: + +```shell +curl "https://gitlab.example.com/-/health" +``` + +Example response: + +```plaintext +GitLab OK +``` + +## Readiness + +The readiness probe checks whether the GitLab instance is ready +to accept traffic via Rails Controllers. The check by default +does validate only instance-checks. + +If the `all=1` parameter is specified, the check also validates +the dependent services (Database, Redis, Gitaly etc.) +and gives a status for each. + +```plaintext +GET /-/readiness +GET /-/readiness?all=1 +``` + +Example request: + +```shell +curl "https://gitlab.example.com/-/readiness" +``` + +Example response: + +```json +{ + "master_check":[{ + "status":"failed", + "message": "unexpected Master check result: false" + }], + ... +} +``` + +On failure, the endpoint returns a `503` HTTP status code. + +This check is being exempt from Rack Attack. + +## Liveness + +WARNING: +In GitLab [12.4](https://about.gitlab.com/upcoming-releases/) +the response body of the Liveness check was changed +to match the example below. + +Checks whether the application server is running. +This probe is used to know if Rails Controllers +are not deadlocked due to a multi-threading. + +```plaintext +GET /-/liveness +``` + +Example request: + +```shell +curl "https://gitlab.example.com/-/liveness" +``` + +Example response: + +On success, the endpoint returns a `200` HTTP status code, and a response like below. + +```json +{ + "status": "ok" +} +``` + +On failure, the endpoint returns a `503` HTTP status code. + +This check is being exempt from Rack Attack. + +## Sidekiq + +Learn how to configure the [Sidekiq health checks](../../administration/sidekiq/sidekiq_health_check.md). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index 1b23d6b7f49..622a772a638 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -15,7 +15,7 @@ Explore our features to monitor your GitLab instance: products. - [GitHub imports](github_imports.md): Monitor the health and progress of the GitHub importer with various Prometheus metrics. -- [Monitoring uptime](../../user/admin_area/monitoring/health_check.md): Check the +- [Monitoring uptime](health_check.md): Check the server status using the health check endpoint. - [IP allowlists](ip_allowlist.md): Configure GitLab for monitoring endpoints that provide health check information when probed. diff --git a/doc/administration/monitoring/ip_allowlist.md b/doc/administration/monitoring/ip_allowlist.md index 72640cd6218..364c1b27d33 100644 --- a/doc/administration/monitoring/ip_allowlist.md +++ b/doc/administration/monitoring/ip_allowlist.md @@ -6,13 +6,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w # IP whitelist **(FREE SELF)** -GitLab provides some [monitoring endpoints](../../user/admin_area/monitoring/health_check.md) +GitLab provides some [monitoring endpoints](health_check.md) that provide health check information when probed. To control access to those endpoints via IP whitelisting, you can add single hosts or use IP ranges: -**Omnibus** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Open `/etc/gitlab/gitlab.rb` and add or uncomment the following: @@ -22,9 +24,7 @@ hosts or use IP ranges: 1. Save the file and [reconfigure](../restart_gitlab.md#reconfigure-a-linux-package-installation) GitLab for the changes to take effect. ---- - -**Helm chart** +:::TabTitle Helm chart (Kubernetes) You can set the required IPs under the `gitlab.webservice.monitoring.ipWhitelist` key. For example: @@ -37,9 +37,7 @@ gitlab: - 0.0.0.0/0 # Default ``` ---- - -**Source** +:::TabTitle Self-compiled (source) 1. Edit `config/gitlab.yml`: @@ -52,3 +50,5 @@ gitlab: ``` 1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. + +::EndTabs diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 0d2037f3a92..a1def4764f6 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -16,8 +16,8 @@ settings: 1. Add the necessary configuration changes. 1. Restart all GitLab for the changes to take effect: - - For Omnibus GitLab installations: `sudo gitlab-ctl restart` - - For installations from source: `sudo service gitlab restart` + - For Linux package installations: `sudo gitlab-ctl restart` + - For self-compiled installations: `sudo service gitlab restart` NOTE: Removed [in GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30786). Use the diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index b448ac461c8..8b3720ca8a9 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -18,10 +18,10 @@ and Grafana allows you to query the data to display graphs. ## Deprecation of bundled Grafana -Bundled Grafana was an optional Omnibus GitLab service that provided a user interface to GitLab metrics. +Bundled Grafana was an optional service for Linux package installations that provided a user interface to GitLab metrics. -The version of Grafana that is bundled with Omnibus GitLab is no longer supported. If you're using the bundled Grafana, you -should switch to a newer version from [Grafana Labs](https://grafana.com/grafana/). +The version of Grafana that is bundled with Linux package installations is no longer supported. If you're using the +bundled Grafana, you should switch to a newer version from [Grafana Labs](https://grafana.com/grafana/). ### Switch to new Grafana instance @@ -34,7 +34,8 @@ To switch away from bundled Grafana to a newer version of Grafana from Grafana L ### Temporary workaround -In GitLab versions 16.0 to 16.2, you can still force Omnibus GitLab to enable and configure Grafana by setting the following: +In GitLab versions 16.0 to 16.2, you can still force Linux package installations to enable and configure Grafana by +setting the following: - `grafana['enable'] = true`. - `grafana['enable_deprecated_service'] = true`. @@ -92,9 +93,9 @@ GitLab sidebar: and expand **Metrics - Grafana**. 1. Select the **Add a link to Grafana** checkbox. 1. Configure the **Grafana URL**: - - *If Grafana is enabled through Omnibus GitLab and on the same server,* + - If Grafana is enabled through a Linux package installation and on the same server, leave **Grafana URL** unchanged. It should be `/-/grafana`. - - *Otherwise,* enter the full URL of the Grafana instance. + - Otherwise, enter the full URL of the Grafana instance. 1. Select **Save changes**. GitLab displays your link in the **Main menu > Admin > Monitoring > Metrics Dashboard**. diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 3fdd4c24177..8afec54dab2 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -22,7 +22,7 @@ From left to right, the performance bar displays: - **Current Host**: the current host serving the page. - **Database queries**: the time taken (in milliseconds) and the total number of database queries, displayed in the format `00ms / 00 (00 cached) pg`. Select to display - a modal window with more details. You can use this to see the following + a dialog with more details. You can use this to see the following details for each query: - **In a transaction**: shows up below the query if it was executed in the context of a transaction @@ -38,8 +38,7 @@ From left to right, the performance bar displays: [Gitaly](../../gitaly/index.md) calls. Select to display a modal window with more details. - **Rugged calls**: the time taken (in milliseconds) and the total number of - [Rugged](../../nfs.md#improving-nfs-performance-with-gitlab) calls. - Select to display a modal window with more details. + Rugged calls. Select to display a modal window with more details. - **Redis calls**: the time taken (in milliseconds) and the total number of Redis calls. Select to display a modal window with more details. - **Elasticsearch calls**: the time taken (in milliseconds) and the total number of diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 0bd13fe5a87..22b73378cab 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -9,12 +9,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Renamed from `GitLab monitor exporter` to `GitLab exporter` in [GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16511). The [GitLab exporter](https://gitlab.com/gitlab-org/gitlab-exporter) enables you to -measure various GitLab metrics pulled from Redis and the database in Omnibus GitLab +measure various GitLab metrics pulled from Redis and the database in Linux package instances. For installations from source you must install and configure it yourself. -To enable the GitLab exporter in an Omnibus GitLab instance: +To enable the GitLab exporter in a Linux package instance: 1. [Enable Prometheus](index.md#configuring-prometheus). 1. Edit `/etc/gitlab/gitlab.rb`. @@ -32,7 +32,7 @@ the GitLab exporter exposed at `localhost:9168`. ## Use a different Rack server -> - Introduced in [Omnibus GitLab 13.8](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4896). +> - Introduced in [GitLab 13.8](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4896). > - WEBrick is now the default Rack server instead of Puma. By default, the GitLab exporter runs on [WEBrick](https://github.com/ruby/webrick), a single-threaded Ruby web server. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 5d6827b79ee..713a1fb3b5d 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -23,8 +23,8 @@ GitLab monitors its own internal service metrics, and makes them available at th `/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, to access the metrics, the client IP address must be [explicitly allowed](../ip_allowlist.md). -These metrics are enabled and collected for [Omnibus GitLab](https://docs.gitlab.com/omnibus/) -and Chart installations. For source installations, these metrics must be enabled +These metrics are enabled and collected for [Linux package](https://docs.gitlab.com/omnibus/) +and Helm chart installations. For self-compiled installations, these metrics must be enabled manually and collected by a Prometheus server. For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#sidekiq-metrics). @@ -206,7 +206,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `sidekiq_jobs_dead_total` | Counter | 13.7 | Sidekiq dead jobs (jobs that have run out of retries) | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_redis_requests_total` | Counter | 13.1 | Redis requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_elasticsearch_requests_total` | Counter | 13.1 | Elasticsearch requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | -| `sidekiq_jobs_deferred_total` | Counter | 16.1 | Number of jobs being deferred when `defer_sidekiq_jobs` feature flag is enabled | `worker` | +| `sidekiq_jobs_skipped_total` | Counter | 16.2 | Number of jobs being skipped (dropped or deferred) when `drop_sidekiq_jobs` feature flag is enabled or `run_sidekiq_jobs` feature flag is disabled | `worker`, `action` | | `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_concurrency` | Gauge | 12.5 | Maximum number of Sidekiq jobs | | | `sidekiq_mem_total_bytes` | Gauge | 15.3 | Number of bytes allocated for both objects consuming an object slot and objects that required a malloc'| | @@ -371,6 +371,17 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_project_wiki_repositories_verification_total` | Gauge | 15.10 | Number of Project Wiki Repositories to attempt to verify on secondary | `url` | | `geo_project_wiki_repositories_verified` | Gauge | 15.10 | Number of Project Wiki Repositories successfully verified on secondary | `url` | | `geo_project_wiki_repositories_verification_failed` | Gauge | 15.10 | Number of Project Wiki Repositories that failed verification on secondary | `url` | +| `geo_project_repositories` | Gauge | 16.2 | Number of Project Repositories on primary | `url` | +| `geo_project_repositories_checksum_total` | Gauge | 16.2 | Number of Project Repositories to checksum on primary | `url` | +| `geo_project_repositories_checksummed` | Gauge | 16.2 | Number of Project Repositories that successfully calculated the checksum on primary | `url` | +| `geo_project_repositories_checksum_failed` | Gauge | 16.2 | Number of Project Repositories that failed to calculate the checksum on primary | `url` | +| `geo_project_repositories_synced` | Gauge | 16.2 | Number of syncable Project Repositories synced on secondary | `url` | +| `geo_project_repositories_failed` | Gauge | 16.2 | Number of syncable Project Repositories failed to sync on secondary | `url` | +| `geo_project_repositories_registry` | Gauge | 16.2 | Number of Project Repositories in the registry | `url` | +| `geo_project_repositories_verification_total` | Gauge | 16.2 | Number of Project Repositories to attempt to verify on secondary | `url` | +| `geo_project_repositories_verified` | Gauge | 16.2 | Number of Project Repositories successfully verified on secondary | `url` | +| `geo_project_repositories_verification_failed` | Gauge | 16.2 | Number of Project Repositories that failed verification on secondary | `url` | + | `gitlab_memwd_violations_total` | Counter | 15.9 | Total number of times a Sidekiq process violated a memory threshold | | | `gitlab_memwd_violations_handled_total` | Counter | 15.9 | Total number of times Sidekiq process memory violations were handled | | | `sidekiq_watchdog_running_jobs_total` | Counter | 15.9 | Current running jobs when RSS limit was reached | `worker_class` | @@ -460,7 +471,6 @@ Some basic Ruby runtime metrics are available: | `puma_pool_capacity` | Gauge | 12.0 | Number of requests the worker is capable of taking right now | | `puma_max_threads` | Gauge | 12.0 | Maximum number of worker threads | | `puma_idle_threads` | Gauge | 12.0 | Number of spawned threads which are not processing a request | -| `puma_killer_terminations_total` | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | ## Redis metrics @@ -487,5 +497,5 @@ metrics can't function correctly. This directory's location is configured using environment variable `prometheus_multiproc_dir`. For best performance, create this directory in `tmpfs`. -If GitLab is installed using [Omnibus GitLab](https://docs.gitlab.com/omnibus/) +If GitLab is installed using the [Linux package](https://docs.gitlab.com/omnibus/) and `tmpfs` is available, then GitLab configures the metrics directory for you. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 0e8315e528a..a9b393aab33 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -9,12 +9,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w [Prometheus](https://prometheus.io) is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. -GitLab provides out-of-the-box monitoring with Prometheus, providing easy -access to high quality time-series monitoring of GitLab services. +GitLab provides out-of-the-box monitoring with Prometheus, providing access to high quality time-series monitoring of +GitLab services. -Prometheus and the various exporters listed in this page are bundled in the -Omnibus GitLab package. Check each exporter's documentation for the timeline -they got added. For installations from source you must install them +Prometheus and the various exporters listed in this page are bundled in Linux packages. Check each exporter's +documentation for the timeline they got added. For installations from source you must install them yourself. Over subsequent releases additional GitLab metrics are captured. Prometheus services are on by default. @@ -85,7 +84,7 @@ listens on: ### Adding custom scrape configurations -You can configure additional scrape targets for the Omnibus GitLab-bundled +You can configure additional scrape targets for the Linux package-bundled Prometheus by editing `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` using the [Prometheus scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E) syntax. @@ -108,16 +107,16 @@ prometheus['scrape_configs'] = [ ] ``` -### Standalone Prometheus using Omnibus GitLab +### Standalone Prometheus using the Linux package -The Omnibus GitLab package can be used to configure a standalone Monitoring node running Prometheus and [Grafana](../performance/grafana_configuration.md). +The Linux package can be used to configure a standalone Monitoring node running Prometheus and [Grafana](../performance/grafana_configuration.md). -The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with Omnibus GitLab: +The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with the Linux +package: 1. SSH into the Monitoring node. -1. [Install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page, but - do not follow the remaining steps. +1. [Install](https://about.gitlab.com/install/) the Linux package you want using **steps 1 and 2** from the GitLab + downloads page, but do not follow the remaining steps. 1. Make sure to collect the IP addresses or DNS records of the Consul server nodes, for the next step. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: @@ -388,8 +387,7 @@ Read more about the [GitLab Metrics](gitlab_metrics.md). ## Bundled software metrics -Many of the GitLab dependencies bundled in Omnibus GitLab are preconfigured to -export Prometheus metrics. +Many of the GitLab dependencies bundled in the Linux package are preconfigured to export Prometheus metrics. ### Node exporter @@ -435,22 +433,6 @@ The GitLab exporter allows you to measure various GitLab metrics, pulled from Re [Read more about the GitLab exporter](gitlab_exporter.md). -## Configuring Prometheus to monitor Kubernetes - -If your GitLab server is running within Kubernetes, Prometheus collects metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration](../../../user/project/integrations/prometheus.md) to monitor them. - -To disable the monitoring of Kubernetes: - -1. Edit `/etc/gitlab/gitlab.rb`. -1. Add (or find and uncomment) the following line and set it to `false`: - - ```ruby - prometheus['monitor_kubernetes'] = false - ``` - -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to - take effect. - ## Troubleshooting ### `/var/opt/gitlab/prometheus` consumes too much disk space diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 3e3712c9645..0273c4b03b1 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -15,35 +15,10 @@ is recommended over NFS where possible, due to better performance. When eliminating the usage of NFS, there are [additional steps you need to take](object_storage.md#alternatives-to-file-system-storage) in addition to moving to Object Storage. -File system performance can impact overall GitLab performance, especially for -actions that read or write to Git repositories. For steps you can use to test -file system performance, see -[File System Performance Benchmarking](operations/filesystem_benchmarking.md). - -## Gitaly with NFS not supported - -Technical and engineering support for using NFS to store Git repository data is officially at end-of-life. No product -changes or troubleshooting is provided through engineering, security or paid support channels. - -If an issue is reproducible, or if it happens intermittently but regularly, GitLab Support can investigate providing the -issue can be reproduced without NFS. To reproduce without NFS, migrate the affected repositories to a different Gitaly -shard. For example, a Gitaly Cluster or a standalone Gitaly VM, backed with block storage. - -## Known kernel version incompatibilities - -RedHat Enterprise Linux (RHEL) and CentOS v7.7 and v7.8 ship with kernel -version `3.10.0-1127`, which [contains a bug](https://bugzilla.redhat.com/show_bug.cgi?id=1783554) that causes -[uploads to fail to copy over NFS](https://gitlab.com/gitlab-org/gitlab/-/issues/218999). The -following GitLab versions include a fix to work properly with that -kernel version: - -- [12.10.12](https://about.gitlab.com/releases/2020/06/25/gitlab-12-10-12-released/) -- [13.0.7](https://about.gitlab.com/releases/2020/06/25/gitlab-13-0-7-released/) -- [13.1.1](https://about.gitlab.com/releases/2020/06/24/gitlab-13-1-1-released/) -- 13.2 and up +NFS cannot be used for repository storage. -If you are using that kernel version, be sure to upgrade GitLab to avoid -errors. +For steps you can use to test file system performance, see +[File System Performance Benchmarking](operations/filesystem_benchmarking.md). ## Fast lookup of authorized SSH keys @@ -59,26 +34,6 @@ is moved to NFS. We are investigating the use of [fast lookup as the default](https://gitlab.com/groups/gitlab-org/-/epics/3104). -## Improving NFS performance with GitLab - -NFS performance with GitLab can in some cases be improved with -[direct Git access](gitaly/index.md#direct-access-to-git-in-gitlab) using [Rugged](https://github.com/libgit2/rugged). - -Depending on the GitLab version, GitLab [automatically detects](gitaly/index.md#automatic-detection) if Rugged can and should -be used per storage. - -If the Rugged feature flag is explicitly set to either `true` or `false`, GitLab uses the value explicitly set. If you -previously enabled Rugged using the feature flag and you want to use automatic detection instead, you must unset -the feature flag: - -```shell -sudo gitlab-rake gitlab:features:unset_rugged -``` - -From GitLab 12.7, Rugged is only automatically enabled for use with Puma if the -[Puma thread count is set to `1`](../install/requirements.md#puma-settings). To use Rugged with a Puma thread count of -more than `1`, enable Rugged using the [feature flag](../development/gitaly.md#legacy-rugged-code). - ## NFS server Installing the `nfs-kernel-server` package allows you to share directories with @@ -293,7 +248,7 @@ NFS mount point is `/gitlab-nfs`. Then, add the following bind mounts in Using bind mounts requires you to manually make sure the data directories are empty before attempting a restore. Read more about the -[restore prerequisites](../raketasks/backup_restore.md). +[restore prerequisites](../administration/backup_restore/index.md). ### Multiple NFS mounts @@ -315,7 +270,7 @@ provide configuration for [UDP log shipping](https://docs.gitlab.com/omnibus/set Having multiple NFS mounts requires you to manually make sure the data directories are empty before attempting a restore. Read more about the -[restore prerequisites](../raketasks/backup_restore.md). +[restore prerequisites](../administration/backup_restore/index.md). ## Testing NFS @@ -361,33 +316,6 @@ sudo ufw allow from <client_ip_address> to any port nfs ## Known issues -### Upgrade to Gitaly Cluster or disable caching if experiencing data loss - -WARNING: -Engineering support for NFS for Git repositories -[is unavailable](../update/removals.md#nfs-as-git-repository-storage-is-no-longer-supported). - -Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories. -For example, we have seen: - -- [Inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). -- `git ls-remote` [returning the wrong (or no branches)](https://gitlab.com/gitlab-org/gitaly/-/issues/3083) -causing Jenkins to intermittently re-run all pipelines for a repository. - -The problem may be partially mitigated by adjusting caching using the following NFS client mount options: - -| Setting | Description | -| ------- | ----------- | -| `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. -| `actimeo=0` | Sets the time to zero that the NFS client caches files and directories before requesting fresh information from a server. | -| `noac` | Tells the NFS client not to cache file attributes and forces application writes to become synchronous so that local changes to a file become visible on the server immediately. | - -WARNING: -The `actimeo=0` and `noac` options both result in a significant reduction in performance, possibly leading to timeouts. -You may be able to avoid timeouts and data loss using `actimeo=0` and `lookupcache=positive` _without_ `noac`, however -we expect the performance reduction is still significant. Upgrade to -[Gitaly Cluster](gitaly/praefect.md) as soon as possible. - ### Avoid using cloud-based file systems GitLab strongly recommends against using cloud-based file systems such as: @@ -447,10 +375,3 @@ On Ubuntu 16.04, use: ```shell sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma) ``` - -### Warnings `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in Gitaly logs - -If you find any warnings like `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in [Gitaly logs](logs/index.md#gitaly-logs), -this problem occurs if `lookupcache=positive` is not set, which we recommend as an -[NFS mount option](#mount-options). -See [Gitaly issue #3175](https://gitlab.com/gitlab-org/gitaly/-/issues/3175) for more details. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index f2b966bd180..2bf3ef0275c 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -68,7 +68,7 @@ automatically. Thus, only the following providers can be used: The consolidated form configuration can't be used for backups or Mattermost. Backups can be configured with -[server side encryption](../raketasks/backup_gitlab.md#s3-encrypted-buckets) +[server side encryption](../administration/backup_restore/backup_gitlab.md#s3-encrypted-buckets) separately. See the [table for a complete list](#configure-each-object-type-to-define-its-own-storage-connection-storage-specific-form) of supported object storage types. @@ -163,7 +163,7 @@ supported by consolidated form, refer to the following guides: | Object storage type | Supported by consolidated form? | |---------------------|------------------------------------------| | [Project-level Secure Files](secure_files.md#using-object-storage) | **{dotted-circle}** No | -| [Backups](../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | +| [Backups](../administration/backup_restore/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | | [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | @@ -174,7 +174,7 @@ supported by consolidated form, refer to the following guides: | [Packages](packages/index.md#use-object-storage) (optional feature) | **{check-circle}** Yes | | [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) | **{check-circle}** Yes | | [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | -| [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | +| [Pages content](pages/index.md#object-storage-settings) | **{check-circle}** Yes | ## Configure the connection settings @@ -825,7 +825,7 @@ See the following additional guides: ### Objects are not included in GitLab backups -As noted in [the backup documentation](../raketasks/backup_restore.md), +As noted in [the backup documentation](../administration/backup_restore/index.md), objects are not included in GitLab backups. You can enable backups with your object storage provider instead. @@ -848,7 +848,7 @@ Helm-based installs require separate buckets to ### S3 API compatibility issues -Not all S3 providers [are fully compatible](../raketasks/backup_gitlab.md#other-s3-providers) +Not all S3 providers [are fully compatible](../administration/backup_restore/backup_gitlab.md#other-s3-providers) with the Fog library that GitLab uses. Symptoms include an error in `production.log`: ```plaintext diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index d54d286c19d..8382f3aa8b5 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -27,8 +27,8 @@ lookup of authorized SSH keys. ## Fast lookup is required for Geo **(PREMIUM)** -Unlike [Cloud Native GitLab](https://docs.gitlab.com/charts/), Omnibus GitLab by default -manages an `authorized_keys` file that is located in the +Unlike [Cloud Native GitLab](https://docs.gitlab.com/charts/), by default Linux package installations +manage an `authorized_keys` file that is located in the `git` user's home directory. For most installations, this file is located under `/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to locate the `authorized_keys` on your system: @@ -74,7 +74,7 @@ able to accept a fingerprint. Check the version of OpenSSH on your server with ` Add the following to your `sshd_config` file. This file is usually located at `/etc/ssh/sshd_config`, but it is at `/assets/sshd_config` if you're using -Omnibus Docker: +Docker from a Linux package installation: ```plaintext Match User git # Apply the AuthorizedKeysCommands to the git user only @@ -146,7 +146,8 @@ This overview is brief. Refer to the above instructions for more context. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. 1. Select the **Use authorized_keys file to authenticate SSH keys** checkbox. -1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker. +1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Docker + from a Linux package installation. 1. Reload `sshd`: `sudo service sshd reload`. ## SELinux support and limitations diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index cd4ab1a9cf8..bd37bd4b1a8 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -11,7 +11,7 @@ especially for actions that read or write to Git repositories. This information helps benchmark file system performance against known good and bad real-world systems. -Normally when talking about file system performance the biggest concern is +When talking about file system performance the biggest concern is with Network File Systems (NFS). However, even some local disks can have slow I/O. The information on this page can be used for either scenario. diff --git a/doc/administration/operations/gitlab_sshd.md b/doc/administration/operations/gitlab_sshd.md index 5c4af32fc3d..2707c8f08a0 100644 --- a/doc/administration/operations/gitlab_sshd.md +++ b/doc/administration/operations/gitlab_sshd.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # `gitlab-sshd` **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5 as an Experiment for self-managed customers. -> - Ready for production use with [Cloud Native GitLab in GitLab 15.1](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2540) and [Omnibus GitLab in GitLab 15.9](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5937). +> - Ready for production use with [Cloud Native GitLab in GitLab 15.1](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2540) and [Linux packages in GitLab 15.9](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5937). `gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd) written in Go. It is provided as a part of the `gitlab-shell` package. It has a lower memory @@ -47,7 +47,7 @@ The following instructions enable `gitlab-sshd` on a different port than OpenSSH gitlab_sshd['listen_address'] = '[::]:2222' # Adjust the port accordingly ``` -1. Optional. By default, Omnibus GitLab generates SSH host keys for `gitlab-sshd` if +1. Optional. By default, Linux package installations generate SSH host keys for `gitlab-sshd` if they do not exist in `/var/opt/gitlab/gitlab-sshd`. If you wish to disable this automatic generation, add this line: ```ruby diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 867ef3236ee..be90b0a073f 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Keep your GitLab instance up and running. - [Housekeeping](../../administration/housekeeping.md) -- [Activate GitLab EE with license](../../user/admin_area/license_file.md) +- [Activate GitLab EE with license](../../administration/license_file.md) - [Fast SSH key lookup](../../administration/operations/fast_ssh_key_lookup.md) - [File system benchmarking](../../administration/operations/filesystem_benchmarking.md) - [`gitlab-sshd`](../../administration/operations/gitlab_sshd.md) @@ -17,7 +17,7 @@ Keep your GitLab instance up and running. - [Use SSH certificates](../../administration/operations/ssh_certificates.md) - [Enable encrypted configuration](../../administration/encrypted_configuration.md) - [Rake tasks](../../raketasks/index.md) -- [Backup and restore](../../raketasks/backup_restore.md) +- [Backup and restore](../../administration/backup_restore/index.md) - [Inactive project deletion](../../administration/inactive_project_deletion.md) - [Move repositories](../../administration/operations/moving_repositories.md) - [Read-only state](../../administration/read_only_gitlab.md) diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index e9d829f3f08..c27bedd39de 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -188,14 +188,14 @@ Each of the approaches we list can or does overwrite data in the target director ### Recommended approach in all cases -For either Gitaly or Gitaly Cluster targets, the GitLab [backup and restore capability](../../raketasks/backup_restore.md) +For either Gitaly or Gitaly Cluster targets, the GitLab [backup and restore capability](../../administration/backup_restore/index.md) should be used. Git repositories are accessed, managed, and stored on GitLab servers by Gitaly as a database. Data loss can result from directly accessing and copying Gitaly files using tools like `rsync`. - From GitLab 13.3, backup performance can be improved by - [processing multiple repositories concurrently](../../raketasks/backup_gitlab.md#back-up-git-repositories-concurrently). + [processing multiple repositories concurrently](../../administration/backup_restore/backup_gitlab.md#back-up-git-repositories-concurrently). - Backups can be created of just the repositories using the - [skip feature](../../raketasks/backup_gitlab.md#excluding-specific-directories-from-the-backup). + [skip feature](../../administration/backup_restore/backup_gitlab.md#excluding-specific-directories-from-the-backup). No other method works for Gitaly Cluster targets. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index d7d6f6228f9..f471dcd44b0 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -173,7 +173,7 @@ optimal configuration: ## Configuring Puma to listen over SSL -Puma, when deployed with Omnibus GitLab, listens over a Unix socket by +Puma, when deployed with a Linux package installation, listens over a Unix socket by default. To configure Puma to listen over an HTTPS port instead, follow the steps below: @@ -283,7 +283,7 @@ For Helm-based deployments, see the [`webservice` chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). Starting with GitLab 13.0, Puma is the default web server and Unicorn has been disabled. -In GitLab 14.0, [Unicorn was removed](../../update/removals.md#unicorn-in-gitlab-self-managed) +In GitLab 14.0, [Unicorn was removed](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html) from the Linux package and is no longer supported. Puma has a multi-thread architecture that uses less memory than a multi-process @@ -386,7 +386,7 @@ downtime. Otherwise, skip to the next section. GDB reports an error if the Puma process terminates before you can run these commands. To buy more time, you can always raise the -Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and +Puma worker timeout. For Linux package installation users, you can edit `/etc/gitlab/gitlab.rb` and increase it from 60 seconds to 600: ```ruby diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index ac550d30566..ac0a7e5870b 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -25,28 +25,34 @@ Rails experience is useful but not required. ## Starting a Rails console session -**For Omnibus installations** +The process for starting a Rails console session depends on the type of GitLab installation. + +::Tabs + +:::TabTitle Linux package (Omnibus) ```shell sudo gitlab-rails console ``` -**For Docker installations** +:::TabTitle Docker ```shell docker exec -it <container-id> gitlab-rails console ``` -**For installations from source** +:::TabTitle Self-compiled (source) ```shell sudo -u git -H bundle exec rails console -e production ``` -**For Kubernetes deployments** +:::TabTitle Helm chart (Kubernetes) The console is in the toolbox pod. Refer to our [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) for details. +::EndTabs + To exit the console, type: `quit`. ## Enable Active Record logging @@ -130,31 +136,31 @@ environment, you can do so using the [Rails Runner](https://guides.rubyonrails.o When executing a script file, the script must be accessible by the `git` user. When the command or script completes, the Rails Runner process finishes. -It is useful for running within other scripts or cron jobs for example. +It is useful for running in other scripts or cron jobs for example. -**For Omnibus installations** +- For Linux package installations: -```shell -sudo gitlab-rails runner "RAILS_COMMAND" + ```shell + sudo gitlab-rails runner "RAILS_COMMAND" -# Example with a two-line Ruby script -sudo gitlab-rails runner "user = User.first; puts user.username" + # Example with a two-line Ruby script + sudo gitlab-rails runner "user = User.first; puts user.username" -# Example with a ruby script file (make sure to use the full path) -sudo gitlab-rails runner /path/to/script.rb -``` + # Example with a ruby script file (make sure to use the full path) + sudo gitlab-rails runner /path/to/script.rb + ``` -**For installations from source** +- For self-compiled installations: -```shell -sudo -u git -H bundle exec rails runner -e production "RAILS_COMMAND" + ```shell + sudo -u git -H bundle exec rails runner -e production "RAILS_COMMAND" -# Example with a two-line Ruby script -sudo -u git -H bundle exec rails runner -e production "user = User.first; puts user.username" + # Example with a two-line Ruby script + sudo -u git -H bundle exec rails runner -e production "user = User.first; puts user.username" -# Example with a ruby script file (make sure to use the full path) -sudo -u git -H bundle exec rails runner -e production /path/to/script.rb -``` + # Example with a ruby script file (make sure to use the full path) + sudo -u git -H bundle exec rails runner -e production /path/to/script.rb + ``` Rails Runner does not produce the same output as the console. @@ -180,68 +186,6 @@ Some basic knowledge of Ruby is very useful. Try [this 30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction. Rails experience is helpful but not essential. -### Troubleshooting Rails Runner - -The `gitlab-rails` command executes Rails Runner using a non-root account and group, by default: `git:git`. - -If the non-root account cannot find the Ruby script filename passed to `gitlab-rails runner` -you may get a syntax error, not an error that the file couldn't be accessed. - -A common reason for this is that the script has been put in the root account's home directory. - -`runner` tries to parse the path and file parameter as Ruby code. - -For example: - -```plaintext -[root ~]# echo 'puts "hello world"' > ./helloworld.rb -[root ~]# sudo gitlab-rails runner ./helloworld.rb -Please specify a valid ruby command or the path of a script to run. -Run 'rails runner -h' for help. - -/opt/gitlab/..../runner_command.rb:45: syntax error, unexpected '.' -./helloworld.rb -^ -[root ~]# sudo gitlab-rails runner /root/helloworld.rb -Please specify a valid ruby command or the path of a script to run. -Run 'rails runner -h' for help. - -/opt/gitlab/..../runner_command.rb:45: unknown regexp options - hllwrld -[root ~]# mv ~/helloworld.rb /tmp -[root ~]# sudo gitlab-rails runner /tmp/helloworld.rb -hello world -``` - -A meaningful error should be generated if the directory can be accessed, but the file cannot: - -```plaintext -[root ~]# chmod 400 /tmp/helloworld.rb -[root ~]# sudo gitlab-rails runner /tmp/helloworld.rb -Traceback (most recent call last): - [traceback removed] -/opt/gitlab/..../runner_command.rb:42:in `load': cannot load such file -- /tmp/helloworld.rb (LoadError) -``` - -In case you encounter a similar error to this: - -```plaintext -[root ~]# sudo gitlab-rails runner helloworld.rb -Please specify a valid ruby command or the path of a script to run. -Run 'rails runner -h' for help. - -undefined local variable or method `helloworld' for main:Object -``` - -You can either move the file to the `/tmp` directory or create a new directory owned by the user `git` and save the script in that directory as illustrated below: - -```shell -sudo mkdir /scripts -sudo mv /script_path/helloworld.rb /scripts -sudo chown -R git:git /scripts -sudo chmod 700 /scripts -sudo gitlab-rails runner /scripts/helloworld.rb -``` - ## Find specific methods for an object ```ruby @@ -264,7 +208,7 @@ Adding a semicolon(`;`) and a follow-up statement at the end of a statement prev ```ruby puts ActiveRecord::Base.descendants; :ok -Project.select(&:pages_deployed?).each {|p| puts p.pages_url }; true +Project.select(&:pages_deployed?).each {|p| puts p.path }; true ``` ## Get or store the result of last operation @@ -756,3 +700,84 @@ project.irb irb(#<Project>)> web_url # => "https://gitlab-example/root/discard" ``` + +## Troubleshooting + +### Rails Runner `syntax error` + +The `gitlab-rails` command executes Rails Runner using a non-root account and group, by default: `git:git`. + +If the non-root account cannot find the Ruby script filename passed to `gitlab-rails runner` +you may get a syntax error, not an error that the file couldn't be accessed. + +A common reason for this is that the script has been put in the root account's home directory. + +`runner` tries to parse the path and file parameter as Ruby code. + +For example: + +```plaintext +[root ~]# echo 'puts "hello world"' > ./helloworld.rb +[root ~]# sudo gitlab-rails runner ./helloworld.rb +Please specify a valid ruby command or the path of a script to run. +Run 'rails runner -h' for help. + +/opt/gitlab/..../runner_command.rb:45: syntax error, unexpected '.' +./helloworld.rb +^ +[root ~]# sudo gitlab-rails runner /root/helloworld.rb +Please specify a valid ruby command or the path of a script to run. +Run 'rails runner -h' for help. + +/opt/gitlab/..../runner_command.rb:45: unknown regexp options - hllwrld +[root ~]# mv ~/helloworld.rb /tmp +[root ~]# sudo gitlab-rails runner /tmp/helloworld.rb +hello world +``` + +A meaningful error should be generated if the directory can be accessed, but the file cannot: + +```plaintext +[root ~]# chmod 400 /tmp/helloworld.rb +[root ~]# sudo gitlab-rails runner /tmp/helloworld.rb +Traceback (most recent call last): + [traceback removed] +/opt/gitlab/..../runner_command.rb:42:in `load': cannot load such file -- /tmp/helloworld.rb (LoadError) +``` + +In case you encounter a similar error to this: + +```plaintext +[root ~]# sudo gitlab-rails runner helloworld.rb +Please specify a valid ruby command or the path of a script to run. +Run 'rails runner -h' for help. + +undefined local variable or method `helloworld' for main:Object +``` + +You can either move the file to the `/tmp` directory or create a new directory owned by the user `git` and save the script in that directory as illustrated below: + +```shell +sudo mkdir /scripts +sudo mv /script_path/helloworld.rb /scripts +sudo chown -R git:git /scripts +sudo chmod 700 /scripts +sudo gitlab-rails runner /scripts/helloworld.rb +``` + +### Filtered console output + +Some output in the console might be filtered by default to prevent leaks of certain values +like variables, logs, or secrets. This output displays as `[FILTERED]`. For example: + +```plain_text +> Plan.default.actual_limits +=> ci_instance_level_variables: "[FILTERED]", +``` + +To work around the filtering, read the values directly from the object. For example: + +```plain_text +> Plan.default.limits.ci_instance_level_variables +=> 25 +``` diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md index ac183afdc2f..f85ada3c782 100644 --- a/doc/administration/package_information/defaults.md +++ b/doc/administration/package_information/defaults.md @@ -11,8 +11,7 @@ the package assumes the defaults as noted below. ## Ports -See the table below for the list of ports that the Omnibus GitLab assigns -by default: +See the table below for the list of ports that the Linux package assigns by default: | Component | On by default | Communicates via | Alternative | Connection port | |:--------------------:|:-------------:|:----------------:|:-----------:|:------------------------------------------:| @@ -47,7 +46,7 @@ by default: | Mattermost | No | Port | X | 8065 | | Mattermost | No | Port | X | 80 or 443 | | PgBouncer | No | Port | X | 6432 | -| Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] | +| Consul | No | Port | X | 8300, 8301(TCP and UDP), 8500, 8600[^Consul-notes] | | Patroni | No | Port | X | 8008 | | GitLab KAS | Yes | Port | X | 8150 | | Gitaly | Yes | Socket | Port (8075) | 8075 or 9999 (TLS) | diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index d8f4551ca09..f2d23da2b7e 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -4,9 +4,9 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Omnibus GitLab deprecation policy **(FREE SELF)** +# Linux package deprecation policy **(FREE SELF)** -The Omnibus GitLab packages come with number of different libraries and services which offers users plethora of configuration options. +The Linux packages come with number of different libraries and services which offers users plethora of configuration options. As libraries and services get updated, their configuration options change and become obsolete. To increase maintainability and preserve a working @@ -16,7 +16,7 @@ setup, various configuration requires removal. ### Policy -The Omnibus GitLab package retains configuration for at least **one major** +The Linux package retains configuration for at least **one major** version. We can't guarantee that deprecated configuration is available in the next major release. See [example](#example) for more details. @@ -49,7 +49,8 @@ Deprecation procedure is similar for both `sensitive` and `regular` configuratio Common steps: -1. Create an issue at the [Omnibus GitLab issue tracker](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues) with details on deprecation type and other necessary information. Apply the label `deprecation`. +1. Create an issue at the [`omnibus-gitlab` issue tracker](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues) with + details on deprecation type and other necessary information. Apply the label `deprecation`. 1. Decide on the removal target for the deprecated configuration 1. Formulate deprecation notice for each item as noted in [Notice section](#notice) diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md index bfa751a051f..503fdfe3ba8 100644 --- a/doc/administration/package_information/index.md +++ b/doc/administration/package_information/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Package information **(FREE SELF)** -The Omnibus GitLab package is bundled with all dependencies required for GitLab +The Linux package is bundled with all dependencies required for GitLab to function correctly. More details can be found at [bundling dependencies document](omnibus_packages.md). @@ -14,11 +14,11 @@ at [bundling dependencies document](omnibus_packages.md). The released package versions are in the format `MAJOR.MINOR.PATCH-EDITION.OMNIBUS_RELEASE` -| Component | Meaning | Example | -|---------------------|---------|---------| -| `MAJOR.MINOR.PATCH` | The GitLab version this corresponds to. | `13.3.0` | -| `EDITION` | The edition of GitLab this corresponds to. | `ee` | -| `OMNIBUS_RELEASE` | The Omnibus GitLab release. Usually, this is 0. This is incremented if we need to build a new package without changing the GitLab version. | `0` | +| Component | Meaning | Example | +|:--------------------|:------------------------------------------------------------------------------------------------------------------------------------------|:---------| +| `MAJOR.MINOR.PATCH` | The GitLab version this corresponds to. | `13.3.0` | +| `EDITION` | The edition of GitLab this corresponds to. | `ee` | +| `OMNIBUS_RELEASE` | The Linux package release. Usually, this is `0`. We increment this if we need to build a new package without changing the GitLab version. | `0` | ## Licenses @@ -26,7 +26,7 @@ See [licensing](licensing.md) ## Defaults -The Omnibus GitLab package requires various configuration to get the components +The Linux package requires various configuration to get the components in working order. If the configuration is not provided, the package uses the default values assumed in the package. @@ -34,10 +34,10 @@ These defaults are noted in the package [defaults document](defaults.md). ## Checking the versions of bundled software -After the Omnibus GitLab package is installed, you can find the version of +After the Linux package is installed, you can find the version of GitLab and all bundled libraries in `/opt/gitlab/version-manifest.txt`. -If you don't have the package installed, you can always check the Omnibus GitLab +If you don't have the package installed, you can always check the Linux package [source repository](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master), specifically the [configuration directory](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/config). @@ -51,15 +51,14 @@ Documentation on package signatures can be found at [Signed Packages](signed_pac ## Checking for newer configuration options on upgrade -Configuration file in `/etc/gitlab/gitlab.rb` is created on initial installation -of the Omnibus GitLab package. On subsequent package upgrades, the configuration -file is not updated with new configuration. This is done to avoid -accidental overwrite of user configuration provided in `/etc/gitlab/gitlab.rb`. +The `/etc/gitlab/gitlab.rb` configuration file is created when the Linux package is initially installed. +To avoid accidental overwrites of user configuration, the `/etc/gitlab/gitlab.rb` configuration file is not updated +with new configuration when the Linux package installation is upgraded. New configuration options are noted in the [`gitlab.rb.template` file](https://gitlab.com/gitlab-org/omnibus-gitlab/raw/master/files/gitlab-config-template/gitlab.rb.template). -The Omnibus GitLab package also provides convenience command which +The Linux package also provides a convenience command which compares the existing user configuration with the latest version of the template contained in the package. @@ -76,7 +75,7 @@ characters on each line. ## Init system detection -Omnibus GitLab attempts to query the underlying system to +The Linux package attempts to query the underlying system to check which init system it uses. This manifests itself as a `WARNING` during the `sudo gitlab-ctl reconfigure` run. diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md index bbc0e864e95..cad9c0e6075 100644 --- a/doc/administration/package_information/licensing.md +++ b/doc/administration/package_information/licensing.md @@ -8,12 +8,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## License -While GitLab itself is MIT, the Omnibus GitLab sources are licensed under the Apache-2.0. +While GitLab itself is MIT, the Linux package sources are licensed under the Apache-2.0. ## License file location -Starting with version 8.11, the Omnibus GitLab package contains license -information of all software that is bundled within the package. +Starting with version 8.11, the Linux package contains license +information of all software that is bundled in the package. After installing the package, licenses for each individual bundled library can be found in `/opt/gitlab/LICENSES` directory. @@ -21,11 +21,11 @@ can be found in `/opt/gitlab/LICENSES` directory. There is also one `LICENSE` file which contains all licenses compiled together. This compiled license can be found in `/opt/gitlab/LICENSE` file. -Starting with version 9.2, the Omnibus GitLab package ships a +Starting with version 9.2, the Linux package ships with a `dependency_licenses.json` file containing version and license information of all bundled software, including software libraries, Ruby gems that the rails application uses, and JavaScript libraries that is required for the frontend -components. Because it's in JSON format, GitLab can parse this file easily and use it for automated checks or validations. The file may be found at +components. Because it's in JSON format, GitLab can parse this file and use it for automated checks or validations. The file may be found at `/opt/gitlab/dependency_licenses.json`. Starting with version 11.3, we have also made the license information available @@ -33,28 +33,28 @@ online, at: <https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html> ## Checking licenses -The Omnibus GitLab package is made up of many pieces of software, comprising code +The Linux package is made up of many pieces of software, comprising code that is covered by many different licenses. Those licenses are provided and compiled as stated above. Starting with version 8.13, GitLab has placed an additional step into -Omnibus GitLab. The `license_check` step calls +Linux package installation. The `license_check` step calls `lib/gitlab/tasks/license_check.rake`, which checks the compiled `LICENSE` file against the current list of approved and questionable licenses as denoted in the arrays at the top of the script. This script outputs one of `Good`, `Unknown` or `Check` for each piece of software that is a part of the -Omnibus GitLab package. +Linux package. -- `Good`: denotes a license that is approved for all usage types, within GitLab and - Omnibus GitLab. +- `Good`: denotes a license that is approved for all usage types, in GitLab and + the Linux package. - `Unknown`: denotes a license that is not recognized in the list of 'good' or 'bad', which should be immediately reviewed for implications of use. - `Check`: denotes a license that has the potential be incompatible with GitLab itself, - and thus should be checked for how it is used as a part of the Omnibus GitLab package + and thus should be checked for how it is used as a part of the Linux package to ensure compliance. -This list is currently sourced from the [GitLab development documentation on licensing](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/doc/development/licensing.md). -However, due to the nature of the Omnibus GitLab package the licenses may not apply +This list is sourced from the [GitLab development documentation on licensing](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/doc/development/licensing.md). +However, due to the nature of the Linux package, the licenses may not apply in the same way. Such as with `git` and `rsync`. See the [GNU License FAQ](https://www.gnu.org/licenses/gpl-faq.en.html#MereAggregation) ## License acknowledgements diff --git a/doc/administration/package_information/omnibus_packages.md b/doc/administration/package_information/omnibus_packages.md index 222341c7308..d16783034dc 100644 --- a/doc/administration/package_information/omnibus_packages.md +++ b/doc/administration/package_information/omnibus_packages.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Omnibus based packages and images **(FREE SELF)** +# Packages and images from the Linux package **(FREE SELF)** Below you can find some basic information on why GitLab provides packages and a Docker image that come with bundled dependencies. @@ -19,7 +19,7 @@ We have a few core goals with these packages: 1. Support for a wide variety of operating systems 1. Wide support of cloud service providers -## Omnibus GitLab Architecture +## Linux package architecture GitLab in its core is a Ruby on Rails project. However, GitLab as a whole application is more complex and has multiple components. If these components are @@ -72,7 +72,7 @@ Some drawbacks of a package with bundled dependencies: 1. Duplication with possibly existing software. 1. Less flexibility in configuration. -## Why would you install an omnibus package when you can use a system package? +## Why would you install a package from the Linux package when you can use a system package? The answer can be simplified to: less maintenance required. Instead of handling multiple packages that *can* break existing functionality if the versions are @@ -83,23 +83,24 @@ Keeping configuration in sync can be error prone. If you have the skill set to maintain all current dependencies and enough time to handle any future dependencies that might get introduced, the above listed -reasons might not be good enough for you to not use the omnibus package. +reasons might not be good enough for you to not use a package from the Linux package. There are two things to keep in mind before going down this route: 1. Getting support for any problems you encounter might be more difficult due to the number of possibilities that exist when using a library version that is not tested by majority of users. -1. Omnibus package also allows shutting off of any services that you do not need, +1. Packages from the Linux package also allow shutting off of any services that you do not need, if you need to run a component independently. For example, you can use a - [non-bundled PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server) with the omnibus package. + [non-bundled PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server) + with a Linux package installation. -Keep in mind that a non-standard solution like the omnibus package +Keep in mind that a non-standard solution like the Linux package might be a better fit when the application has a number of moving parts. ## Docker image with multiple services -[GitLab Docker image](../../install/docker.md#gitlab-docker-images) is based on the omnibus package. +[GitLab Docker image](../../install/docker.md#gitlab-docker-images) is based on the Linux package. Considering that container spawned from this image contains multiple processes, these types of containers are also referred to as 'fat containers'. diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 44032883eb4..101e1549d19 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -4,23 +4,24 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# PostgreSQL versions shipped with Omnibus GitLab **(FREE SELF)** +# PostgreSQL versions shipped with the Linux package **(FREE SELF)** NOTE: This table lists only GitLab versions where a significant change happened in the package regarding PostgreSQL versions, not all. Usually, PostgreSQL versions change with major or minor GitLab releases. However, patch versions -of Omnibus GitLab sometimes update the patch level of PostgreSQL. We've established a +of the Linux package sometimes update the patch level of PostgreSQL. We've established a [yearly cadence for PostgreSQL upgrades](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/postgresql-upgrade-cadence.html) and trigger automatic database upgrades in the release before the new version is required. For example: -- Omnibus 12.7.6 shipped with PostgreSQL 9.6.14 and 10.9. -- Omnibus 12.7.7 shipped with PostgreSQL 9.6.17 and 10.12. +- Linux package 12.7.6 shipped with PostgreSQL 9.6.14 and 10.9. +- Linux package 12.7.7 shipped with PostgreSQL 9.6.17 and 10.12. -[Find out which versions of PostgreSQL (and other components) ship with each Omnibus GitLab release](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html). +Find out [which versions of PostgreSQL (and other components) ship](https://gitlab-org.gitlab.io/omnibus-gitlab/licenses.html) +with each Linux package release. The lowest supported PostgreSQL versions are listed in the [installation requirements](../../install/requirements.md#postgresql-requirements). @@ -29,13 +30,16 @@ Read more about update policies and warnings in the PostgreSQL [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server). | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | -| -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | -| 16.0 | 13.11 | 13.11 | 13.11 | | +| -------------- | ------------------- | ---------------------------------- | ---------------------------- | ----- | +| 16.0.2 | 13.11 | 13.11 | 13.11 | | +| 16.0.0 | 13.8 | 13.8 | 13.8 | | +| 15.11.7 | 13.11 | 13.11 | 12.12 | | +| 15.10.8 | 13.11 | 13.11 | 12.12 | | | 15.6 | 12.12, 13.8 | 13.8 | 12.12 | For upgrades, users can manually upgrade to 13.8 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | | 15.0 | 12.10, 13.6 | 13.6 | 12.10 | For upgrades, users can manually upgrade to 13.6 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | -| 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). -| 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and are prevented from upgrading to Omnibus GitLab 14.0 | -| 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster.). | +| 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-for-linux-package-installations). +| 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and are prevented from upgrading to Linux package 14.0 | +| 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster. | | 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | | 13.4 | 11.9, 12.4 | 11.9 | 11.9 | Package upgrades aborted if users not running PostgreSQL 11 already | | 13.3 | 11.7, 12.3 | 11.7 | 11.7 | Package upgrades aborted if users not running PostgreSQL 11 already | diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md index 638dd7820b8..b2aab96e52c 100644 --- a/doc/administration/package_information/signed_packages.md +++ b/doc/administration/package_information/signed_packages.md @@ -6,13 +6,22 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Package Signatures **(FREE SELF)** -Omnibus GitLab packages produced by GitLab are created via the [Omnibus](https://github.com/chef/omnibus) tool, for which GitLab has added DEB signing via `debsigs` in [our own fork](https://gitlab.com/gitlab-org/omnibus). This addition, combined with the existing functionality of RPM signing, allows GitLab to provide signed packages for all supported distributions using DEB or RPM. +Linux packages produced by GitLab are created using [Omnibus](https://github.com/chef/omnibus), for which GitLab +has added DEB signing using `debsigs` in [our own fork](https://gitlab.com/gitlab-org/omnibus). -These packages are produced by the GitLab CI process, as found in the [Omnibus GitLab project](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.gitlab-ci.yml), prior to their delivery to <https://packages.gitlab.com> to ensure provide assurance that the packages are not altered prior to delivery to our community. +Combined with the existing functionality of RPM signing, this addition allows GitLab to provide signed packages for all +supported distributions using DEB or RPM. + +These packages are produced by the GitLab CI process, as found in the +[`omnibus-gitlab` project](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.gitlab-ci.yml), +prior to their delivery to <https://packages.gitlab.com> to provide assurance that the packages are not altered prior +to delivery to our community. ## GnuPG Public Keys -All packages are signed with [GnuPG](https://www.gnupg.org/), in a method appropriate for their format. The key used to sign these packages can be found on [MIT PGP Public Key Server](https://pgp.mit.edu) at [0x3cfcf9baf27eab47](https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3CFCF9BAF27EAB47) +All packages are signed with [GnuPG](https://www.gnupg.org/), in a method appropriate for their format. The key used to +sign these packages can be found on [MIT PGP Public Key Server](https://pgp.mit.edu) at +[`0x3cfcf9baf27eab47`](https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3CFCF9BAF27EAB47). ## Verifying Signatures diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 8ad3a656e05..16f4c2cdeab 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -83,7 +83,7 @@ running GitLab on ARM. ## OS Versions that are no longer supported -GitLab provides omnibus packages for operating systems only until their +GitLab provides Linux packages for operating systems only until their EOL (End-Of-Life). After the EOL date of the OS, GitLab stops releasing official packages. The list of deprecated operating systems and the final GitLab release for them can be found below: diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index e3867a25846..007072647a2 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -16,9 +16,11 @@ Registry, see the [user documentation](../../user/packages/container_registry/in ## Enable the Container Registry -**Omnibus GitLab installations** +The process for enabling the Container Registry depends on the type of installation you use. -If you installed GitLab by using the Omnibus installation package, the Container Registry +### Linux package installations + +If you installed GitLab by using the Linux package, the Container Registry may or may not be available by default. The Container Registry is automatically enabled and available on your GitLab domain, port 5050 if @@ -34,9 +36,9 @@ but it's not recommended and is beyond the scope of this document. Read the [insecure Registry documentation](https://docs.docker.com/registry/insecure/) if you want to implement this. -**Installations from source** +### Self-compiled installations -If you have installed GitLab from source: +If you self-compiled your GitLab installation: 1. You must [deploy a registry](https://docs.docker.com/registry/deploying/) using the image corresponding to the version of GitLab you are installing @@ -115,8 +117,8 @@ certificate. If the GitLab domain is `https://gitlab.example.com` and the port to the outside world is `5050`, to configure the Container Registry: -- Edit `gitlab.rb` if you are using Omnibus GitLab. -- Edit `gitlab.yml` if you installed GitLab from source. +- Edit `gitlab.rb` if you are using a Linux package installation. +- Edit `gitlab.yml` if you are using a self-compiled installation. Ensure you choose a port different than the one that Registry listens to (`5000` by default), otherwise conflicts occur. @@ -126,7 +128,9 @@ Host and container firewall rules must be configured to allow traffic in through under the `registry_external_url` line, rather than the port listed under `gitlab_rails['registry_port']` (default `5000`). -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Your `/etc/gitlab/gitlab.rb` should contain the Registry URL as well as the path to the existing TLS certificate and key used by GitLab: @@ -171,7 +175,7 @@ registry_nginx['redirect_http_to_https'] = true registry_nginx['listen_port'] = 5678 ``` -**Installations from source** +:::TabTitle Self-compiled (source) 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and configure it with the following settings: @@ -186,6 +190,8 @@ registry_nginx['listen_port'] = 5678 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). +::EndTabs + Users should now be able to sign in to the Container Registry with their GitLab credentials using: @@ -202,12 +208,14 @@ domain. For example, `*.gitlab.example.com`, is a wildcard that matches `registr and is distinct from `*.example.com`. As well as manually generated SSL certificates (explained here), certificates automatically -generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl/index.html). +generated by Let's Encrypt are also [supported in Linux package installations](https://docs.gitlab.com/omnibus/settings/ssl/index.html). Let's assume that you want the container Registry to be accessible at `https://registry.gitlab.example.com`. -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Place your TLS certificate and key in `/etc/gitlab/ssl/registry.gitlab.example.com.crt` and @@ -237,7 +245,7 @@ registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/certificate.pem" registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/certificate.key" ``` -**Installations from source** +:::TabTitle Self-compiled (source) 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and configure it with the following settings: @@ -251,6 +259,8 @@ registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/certificate.key" 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). +::EndTabs + Users should now be able to sign in to the Container Registry using their GitLab credentials: @@ -264,7 +274,9 @@ When you disable the Registry by following these steps, you do not remove any existing Docker images. Docker image removal is handled by the Registry application itself. -**Omnibus GitLab** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Open `/etc/gitlab/gitlab.rb` and set `registry['enable']` to `false`: @@ -274,7 +286,7 @@ Registry application itself. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +:::TabTitle Self-compiled (source) 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and set `enabled` to `false`: @@ -286,13 +298,17 @@ Registry application itself. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +::EndTabs + ## Disable Container Registry for new projects site-wide If the Container Registry is enabled, then it should be available on all new projects. To disable this function and let the owners of a project to enable the Container Registry by themselves, follow the steps below. -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -302,7 +318,7 @@ the Container Registry by themselves, follow the steps below. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +:::TabTitle Self-compiled (source) 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `default_projects_features` entry and configure it so that `container_registry` is set to `false`: @@ -320,6 +336,8 @@ the Container Registry by themselves, follow the steps below. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +::EndTabs + ### Increase token duration In GitLab, tokens for the Container Registry expire every five minutes. @@ -381,9 +399,11 @@ This path is accessible to: All GitLab, Registry, and web server users must have access to this directory. -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) -The default location where images are stored in Omnibus, is +The default location where images are stored in Linux package installations is `/var/opt/gitlab/gitlab-rails/shared/registry`. To change it: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -394,9 +414,9 @@ The default location where images are stored in Omnibus, is 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +:::TabTitle Self-compiled (source) -The default location where images are stored in source installations, is +The default location where images are stored in self-compiled installations is `/home/git/gitlab/shared/registry`. To change it: 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and @@ -409,6 +429,8 @@ The default location where images are stored in source installations, is 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +::EndTabs + ### Use object storage If you want to store your images on object storage, you can change the storage @@ -421,9 +443,9 @@ GitLab does not back up Docker images that are not stored on the file system. Enable backups with your object storage provider if desired. -**Omnibus GitLab installations** +#### Linux package installations -To configure the `s3` storage driver in Omnibus: +To configure the `s3` storage driver for a Linux package installation: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -493,7 +515,7 @@ To configure the `s3` storage driver in Omnibus: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +#### Self-compiled installations Configuring the storage driver is done in the registry configuration YAML file created when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). @@ -602,7 +624,9 @@ When moving from an existing file system or another object storage provider to A Configure it by setting [`trimlegacyrootprefix: true`](https://gitlab.com/gitlab-org/container-registry/-/blob/a3f64464c3ec1c5a599c0a2daa99ebcbc0100b9a/docs-gitlab/README.md#azure-storage-driver) in the Azure storage driver section of the registry configuration. Without this configuration, the Azure storage driver uses `//` instead of `/` as the first section of the root path, rendering the migrated images inaccessible. -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) ```ruby registry['storage'] = { @@ -616,7 +640,7 @@ registry['storage'] = { } ``` -**Installations from source** +:::TabTitle Self-compiled (source) ```yaml storage: @@ -628,6 +652,8 @@ storage: trimlegacyrootprefix: true ``` +::EndTabs + By default, Azure Storage Driver uses the `core.windows.net` realm. You can set another value for `realm` in the `azure` section (for example, `core.usgovcloudapi.net` for Azure Government Cloud). For more information, see the [Docker documentation](https://docs.docker.com/registry/storage-drivers/azure/). ### Disable redirect for storage driver @@ -636,7 +662,9 @@ By default, users accessing a registry configured with a remote backend are redi However, this behavior is undesirable for registries used by internal hosts that usually can't access public servers. To disable redirects and [proxy download](../object_storage.md#proxy-download), set the `disable` flag to true as follows. This makes all traffic always go through the Registry service. This results in improved security (less surface attack as the storage backend is not publicly accessible), but worse performance (all traffic is redirected via the service). -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb`: @@ -657,7 +685,7 @@ However, this behavior is undesirable for registries used by internal hosts that 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +:::TabTitle Self-compiled (source) 1. Add the `redirect` flag to your registry configuration YAML file: @@ -679,6 +707,8 @@ However, this behavior is undesirable for registries used by internal hosts that 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +::EndTabs + #### Encrypted S3 buckets You can use server-side encryption with AWS KMS for S3 buckets that have @@ -689,7 +719,9 @@ encryption keys in every request. For SSE-S3, you must enable the `encrypt` option in the registry settings. How you do this depends on how you installed GitLab. Follow the instructions here that match your installation method. -For Omnibus GitLab installations: +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb`: @@ -709,7 +741,7 @@ For Omnibus GitLab installations: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -For installations from source: +:::TabTitle Self-compiled (source) 1. Edit your registry configuration YAML file: @@ -727,6 +759,8 @@ For installations from source: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +::EndTabs + ### Storage limitations There is no storage limitation, which means a user can upload an @@ -739,7 +773,9 @@ The Registry server listens on localhost at port `5000` by default, which is the address for which the Registry server should accept connections. In the examples below we set the Registry's port to `5010`. -**Omnibus GitLab** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Open `/etc/gitlab/gitlab.rb` and set `registry['registry_http_addr']`: @@ -749,7 +785,7 @@ In the examples below we set the Registry's port to `5010`. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +:::TabTitle Self-compiled (source) 1. Open the configuration file of your Registry server and edit the [`http:addr`](https://docs.docker.com/registry/configuration/#http) value: @@ -761,6 +797,8 @@ In the examples below we set the Registry's port to `5010`. 1. Save the file and restart the Registry server. +::EndTabs + ## Disable Container Registry per project If Registry is enabled in your GitLab instance, but you don't need it for your @@ -799,7 +837,7 @@ under the project hierarchy, like `registry.example.com/group/project/my/image-name:tag`, and only recognizes `registry.example.com/group/project:tag`. -**Omnibus GitLab** +### Linux package installations You can use GitLab as an auth endpoint with an external container registry. @@ -832,14 +870,14 @@ You can use GitLab as an auth endpoint with an external container registry. # Example: registry['internal_key'] = "---BEGIN RSA PRIVATE KEY---\nMIIEpQIBAA\n" - # Optionally define a custom file for Omnibus GitLab to write the contents + # Optionally define a custom file for a Linux package installation to write the contents # of registry['internal_key'] to. gitlab_rails['registry_key_path'] = "/custom/path/to/registry-key.key" ``` Each time reconfigure is executed, the file specified at `registry_key_path` gets populated with the content specified by `internal_key`. If - no file is specified, Omnibus GitLab defaults it to + no file is specified, Linux package installations default it to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and populates it. @@ -854,7 +892,7 @@ You can use GitLab as an auth endpoint with an external container registry. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +### Self-compiled installations 1. Open `/home/git/gitlab/config/gitlab.yml`, and edit the configuration settings under `registry`: @@ -885,9 +923,11 @@ Read more about the Container Registry notifications configuration options in th You can configure multiple endpoints for the Container Registry. -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) -To configure a notification endpoint in Omnibus: +To configure a notification endpoint for a Linux package installation: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -908,7 +948,7 @@ To configure a notification endpoint in Omnibus: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +:::TabTitle Self-compiled (source) Configuring the notification endpoint is done in your registry configuration YAML file created when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). @@ -927,6 +967,8 @@ notifications: backoff: 1000 ``` +::EndTabs + ## Run the Cleanup policy now WARNING: @@ -1007,28 +1049,27 @@ NOTE: Retention policies in your object storage provider, such as Amazon S3 Lifecycle, may prevent objects from being properly deleted. -Container Registry can use considerable amounts of disk space. To clear up -some unused layers, the registry includes a garbage collect command. +The container registry can use considerable amounts of storage space, and you might want to +[reduce storage usage](../../user/packages/container_registry/reduce_container_registry_storage.md). +Among the listed options, deleting tags is the most effective option. However, tag deletion +alone does not delete image layers, it only leaves the underlying image manifests untagged. + +To more effectively free up space, the Container Registry has a garbage collector that can +delete unreferenced layers and (optionally) untagged manifests. -GitLab offers a set of APIs to manipulate the Container Registry and aid the process -of removing unused tags. Currently, this is exposed using the API, but in the future, -these controls should migrate to the GitLab interface. +To start the garbage collector, use the `registry-garbage-collect` command provided by `gitlab-ctl`. -Users who have the [Maintainer role](../../user/permissions.md) for the project can -[delete Container Registry tags in bulk](../../api/container_registry.md#delete-registry-repository-tags-in-bulk) -periodically based on their own criteria. However, deleting the tags alone does not recycle data, -it only unlinks tags from manifests and image blobs. To recycle the Container -Registry data in the whole GitLab instance, you can use the built-in garbage collection command -provided by `gitlab-ctl`. +WARNING: +This command shuts down the Container Registry prior to the garbage collection and +only starts it again after garbage collection completes. If you prefer to avoid downtime, +you can manually set the Container Registry to [read-only mode and bypass `gitlab-ctl`](#performing-garbage-collection-without-downtime). + +The time required to perform garbage collection is proportional to the Container Registry data size. Prerequisites: -- You must have installed GitLab by using an Omnibus package or the +- You must have installed GitLab by using a Linux package or the [GitLab Helm chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). -- You must set the Registry to [read-only mode](#performing-garbage-collection-without-downtime). - Running garbage collection causes downtime for the Container Registry. When you run this command - on an instance in an environment where another instance is still writing to the Registry storage, - referenced manifests are removed. ### Understanding the content-addressable layers @@ -1053,16 +1094,11 @@ Due to the architecture of registry, this data is still accessible when pulling image `my.registry.com/my.group/my.project@sha256:111111`, though it is no longer directly accessible via the `:latest` tag. -### Recycling unused tags - -Before you run the built-in command, note the following: +### Remove unreferenced layers -- The built-in command stops the registry before it starts the garbage collection. -- The garbage collect command takes some time to complete, depending on the - amount of data that exists. -- If you changed the location of registry configuration file, you must - specify its path. -- After the garbage collection is done, the registry should start automatically. +Image layers are the bulk of the Container Registry storage. A layer is considered +unreferenced when no image manifest references it. Unreferenced layers are the +default target of the Container Registry garbage collector. If you did not change the default location of the configuration file, run: @@ -1070,51 +1106,37 @@ If you did not change the default location of the configuration file, run: sudo gitlab-ctl registry-garbage-collect ``` -This command takes some time to complete, depending on the amount of -layers you have stored. - If you changed the location of the Container Registry `config.yml`: ```shell sudo gitlab-ctl registry-garbage-collect /path/to/config.yml ``` -You may also [remove all untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers), -although this is a way more destructive operation, and you should first -understand the implications. +You can also [remove all untagged manifests and unreferenced layers](#removing-untagged-manifests-and-unreferenced-layers) +to recover additional space. ### Removing untagged manifests and unreferenced layers -WARNING: -This is a destructive operation. - -The GitLab Container Registry follows the same default workflow as Docker Distribution: -retain untagged manifests and all layers, even ones that are not referenced directly. All content -can be accessed by using context addressable identifiers. +By default the Container Registry garbage collector ignores images that are untagged, +and users can keep pulling untagged images by digest. Users can also re-tag images +in the future, making them visible again in the GitLab UI and API. -However, in most workflows, you don't care about untagged manifests and old layers if they are not directly -referenced by a tagged manifest. The `registry-garbage-collect` command supports the -`-m` switch to allow you to remove all unreferenced manifests and layers that are -not directly accessible via `tag`: +If you do not care about untagged images and the layers exclusively referenced by these images, +you can delete them all. Use the `-m` flag on the `registry-garbage-collect` command: ```shell sudo gitlab-ctl registry-garbage-collect -m ``` -Since this is a way more destructive operation, this behavior is disabled by default. -You are likely expecting this way of operation, but before doing that, ensure -that you have backed up all registry data. - -When the command is used without the `-m` flag, the Container Registry only removes layers that are not referenced by any manifest, tagged or not. +If you are unsure about deleting untagged images, back up your registry data before proceeding. ### Performing garbage collection without downtime -You can perform garbage collection without stopping the Container Registry by putting -it in read-only mode and by not using the built-in command. On large instances -this could require Container Registry to be in read-only mode for a while. -During this time, -you are able to pull from the Container Registry, but you are not able to -push. +To do garbage collection while keeping the Container Registry online, put the registry +in read-only mode and bypass the built-in `gitlab-ctl registry-garbage-collect` command. + +You can pull but not push images while the Container Registry is in read-only mode. The Container +Registry must remain in read-only for the full duration of the garbage collection. By default, the [registry storage path](#configure-storage-for-the-container-registry) is `/var/opt/gitlab/gitlab-rails/shared/registry`. @@ -1146,18 +1168,15 @@ To enable the read-only mode: 1. Next, trigger one of the garbage collect commands: - WARNING: - You must use `/opt/gitlab/embedded/bin/registry` to recycle unused tags. If you use `gitlab-ctl registry-garbage-collect`, **the container registry goes down**. - ```shell - # Recycling unused tags + # Remove unreferenced layers sudo /opt/gitlab/embedded/bin/registry garbage-collect /var/opt/gitlab/registry/config.yml - # Removing unused layers not referenced by manifests + # Remove untagged manifests and unreferenced layers sudo /opt/gitlab/embedded/bin/registry garbage-collect -m /var/opt/gitlab/registry/config.yml ``` - This command starts the garbage collection, which might take some time to complete. + This command starts the garbage collection. The time to complete is proportional to the registry data size. 1. Once done, in `/etc/gitlab/gitlab.rb` change it back to read-write mode: @@ -1212,13 +1231,13 @@ itself on the system so that the `gitlab-ctl` command can bring the registry ser Also, there's no way to save progress or results during the mark phase of the process. Only once blobs start being deleted is anything permanent done. -## Configuring GitLab and Registry to run on separate nodes (Omnibus GitLab) +## Configure GitLab and Registry to run on separate nodes (Linux package installations) By default, package assumes that both services are running on the same node. To get GitLab and Registry to run on a separate nodes, separate configuration is necessary for Registry and GitLab. -### Configuring Registry +### Configure Registry Below you can find configuration options you should set in `/etc/gitlab/gitlab.rb`, for Registry to run separately from GitLab: @@ -1235,7 +1254,7 @@ for Registry to run separately from GitLab: - `registry['health_storagedriver_enabled']`, default [set programmatically](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-7-stable/files/gitlab-cookbooks/gitlab/libraries/registry.rb#L88). Configure whether health checks on the configured storage driver are enabled. - `gitlab_rails['registry_issuer']`, [default value](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/10-3-stable/files/gitlab-cookbooks/gitlab/attributes/default.rb#L153). This setting needs to be set the same between Registry and GitLab. -### Configuring GitLab +### Configure GitLab Below you can find configuration options you should set in `/etc/gitlab/gitlab.rb`, for GitLab to run separately from Registry: @@ -1435,7 +1454,9 @@ level=error msg="response completed with error" err.code=unknown err.detail="une To resolve the error specify a `chunksize` value in the Registry configuration. Start with a value between `25000000` (25 MB) and `50000000` (50 MB). -**For Omnibus installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb`: @@ -1452,7 +1473,7 @@ Start with a value between `25000000` (25 MB) and `50000000` (50 MB). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**For installations from source** +:::TabTitle Self-compiled (source) 1. Edit `config/gitlab.yml`: @@ -1467,16 +1488,20 @@ Start with a value between `25000000` (25 MB) and `50000000` (50 MB). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +::EndTabs + ### Supporting older Docker clients The Docker container registry shipped with GitLab disables the schema1 manifest by default. If you are still using older Docker clients (1.9 or older), you may experience an error pushing images. See -[omnibus-4145](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4145) for more details. +[issue 4145](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4145) for more details. You can add a configuration option for backwards compatibility. -**For Omnibus installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb`: @@ -1486,7 +1511,7 @@ You can add a configuration option for backwards compatibility. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**For installations from source** +:::TabTitle Self-compiled (source) 1. Edit the YAML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: @@ -1498,6 +1523,8 @@ You can add a configuration option for backwards compatibility. 1. Restart the registry for the changes to take affect. +::EndTabs + ### Docker connection error A Docker connection error can occur when there are special characters in either the group, @@ -1522,7 +1549,9 @@ offloaded to a third party reverse proxy. This problem was discussed in a [Docker project issue](https://github.com/docker/distribution/issues/970) and a simple solution would be to enable relative URLs in the Registry. -**For Omnibus installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb`: @@ -1534,7 +1563,7 @@ and a simple solution would be to enable relative URLs in the Registry. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**For installations from source** +:::TabTitle Self-compiled (source) 1. Edit the YAML configuration file you created when you [deployed the registry](https://docs.docker.com/registry/deploying/). Add the following snippet: @@ -1545,6 +1574,8 @@ and a simple solution would be to enable relative URLs in the Registry. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +::EndTabs + ### Enable the Registry debug server You can use the Container Registry debug server to diagnose problems. The debug endpoint can monitor metrics and health, as well as do profiling. @@ -1602,7 +1633,9 @@ for more information. The following sections provide additional details about each installation method. -#### Helm chart installations +::Tabs + +:::TabTitle Helm chart (Kubernetes) For Helm chart installations: @@ -1614,9 +1647,9 @@ For Helm chart installations: No other registry configuration changes are required. -#### Omnibus installations +:::TabTitle Linux package (Omnibus) -For Omnibus installations: +For Linux package installations: 1. Temporarily replace the registry binary that ships with GitLab 13.9+ for one prior to `v3.0.0-gitlab`. To do so, pull a previous version of the Docker image for the GitLab Container @@ -1629,20 +1662,21 @@ For Omnibus installations: docker rm $id ``` -1. Replace the binary embedded in the Omnibus install, located at +1. Replace the binary embedded in the Linux package installation located at `/opt/gitlab/embedded/bin/registry`, with `registry-2.13.1-gitlab`. Make sure to start by backing - up the original binary embedded in Omnibus, and restore it after performing the - [image upgrade](#images-upgrade)) steps. You should [stop](https://docs.gitlab.com/omnibus/maintenance/#starting-and-stopping) + up the original binary embedded in the Linux package, and restore it after performing the + [image upgrade](#images-upgrade) steps. You should [stop](https://docs.gitlab.com/omnibus/maintenance/#starting-and-stopping) the registry service before replacing its binary and start it right after. No registry configuration changes are required. -#### Source installations +:::TabTitle Self-compiled (source) For source installations, locate your `registry` binary and temporarily replace it with the one -obtained from `v3.0.0-gitlab`, as explained for [Omnibus installations](#omnibus-installations). +obtained from `v3.0.0-gitlab`, as explained for Linux package installations. Make sure to start by backing up the original registry binary, and restore it after performing the -[images upgrade](#images-upgrade)) -steps. +[images upgrade](#images-upgrade) steps. + +::EndTabs #### Images upgrade diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 5e770c2464b..ee352a82058 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -25,13 +25,11 @@ The GitLab Dependency Proxy: The Dependency Proxy is enabled by default. If you are an administrator, you can turn off the Dependency Proxy. To turn off the Dependency Proxy, follow the instructions that -correspond to your GitLab installation: +correspond to your GitLab installation. -- [Omnibus GitLab installations](#omnibus-gitlab-installations) -- [Helm chart installations](#helm-chart-installations) -- [Installations from source](#installations-from-source) +::Tabs -### Omnibus GitLab installations +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -42,7 +40,7 @@ correspond to your GitLab installation: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -### Helm chart installations +:::TabTitle Helm chart (Kubernetes) After the installation is complete, update the global `appConfig` to turn off the Dependency Proxy: @@ -59,7 +57,7 @@ global: For more information, see [Configure Charts using Globals](https://docs.gitlab.com/charts/charts/globals.html#configure-appconfig-settings). -### Installations from source +:::TabTitle Self-compiled (source) 1. After the installation is complete, configure the `dependency_proxy` section in `config/gitlab.yml`. Set `enabled` to `false` to turn off the Dependency Proxy: @@ -69,13 +67,13 @@ For more information, see [Configure Charts using Globals](https://docs.gitlab.c enabled: false ``` -1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") - for the changes to take effect. +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +::EndTabs ### Multi-node GitLab installations -Follow the steps for [Omnibus GitLab installations](#omnibus-gitlab-installations) -for each Web and Sidekiq node. +Follow the steps for Linux package installations for each Web and Sidekiq node. ## Turn on the Dependency Proxy @@ -91,12 +89,13 @@ local location or even use object storage. ### Changing the local storage path -The Dependency Proxy files for Omnibus GitLab installations are stored under +The Dependency Proxy files for Linux package installations are stored under `/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/` and for source installations under `shared/dependency_proxy/` (relative to the Git home directory). -To change the local storage path: -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -106,7 +105,7 @@ To change the local storage path: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +:::TabTitle Self-compiled (source) 1. Edit the `dependency_proxy` section in `config/gitlab.yml`: @@ -116,7 +115,9 @@ To change the local storage path: storage_path: shared/dependency_proxy ``` -1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect. +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +::EndTabs ### Using object storage @@ -127,7 +128,9 @@ This section describes the earlier configuration format. [Migration steps still [Read more about using object storage with GitLab](../object_storage.md). -**Omnibus GitLab installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where necessary): @@ -158,7 +161,7 @@ This section describes the earlier configuration format. [Migration steps still 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -**Installations from source** +:::TabTitle Self-compiled (source) 1. Edit the `dependency_proxy` section in `config/gitlab.yml` (uncomment where necessary): @@ -190,7 +193,9 @@ This section describes the earlier configuration format. [Migration steps still # path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. ``` -1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect. +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +::EndTabs #### Migrate local Dependency Proxy blobs and manifests to object storage @@ -200,24 +205,24 @@ After [configuring object storage](#using-object-storage), use the following task to migrate existing Dependency Proxy blobs and manifests from local storage to remote storage. The processing is done in a background worker and requires no downtime. -For Omnibus GitLab: +- For Linux package installations: -```shell -sudo gitlab-rake "gitlab:dependency_proxy:migrate" -``` + ```shell + sudo gitlab-rake "gitlab:dependency_proxy:migrate" + ``` -For installations from source: +- For installations from source: -```shell -RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:dependency_proxy:migrate -``` + ```shell + RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:dependency_proxy:migrate + ``` You can optionally track progress and verify that all Dependency Proxy blobs and manifests migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. -- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. -- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. +- `sudo gitlab-rails dbconsole` for Linux package installations running version 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Linux package installations running version 14.2 and later. +- `sudo -u git -H psql -d gitlabhq_production` for self-compiled instances. Verify that `objectstg` (where `file_store = '2'`) has the count of all Dependency Proxy blobs and manifests for each respective query: diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index b735204c323..77730384623 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -55,7 +55,7 @@ guides you through the process. When downloading packages as dependencies in downstream projects, many requests are made through the Packages API. You may therefore reach enforced user and IP rate limits. To address this issue, you -can define specific rate limits for the Packages API. For more details, see [Package Registry Rate Limits](../../user/admin_area/settings/package_registry_rate_limits.md). +can define specific rate limits for the Packages API. For more details, see [Package Registry Rate Limits](../settings/package_registry_rate_limits.md). ## Enable or disable the Package Registry diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index e86726524d0..726307229d6 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -10,9 +10,8 @@ GitLab Pages allows for hosting of static sites. It must be configured by an administrator. Separate [user documentation](../../user/project/pages/index.md) is available. NOTE: -This guide is for Omnibus GitLab installations. If you have installed -GitLab from source, see -[GitLab Pages administration for source installations](source.md). +This guide is for Linux package installations. If you have a self-compiled GitLab installation, see +[GitLab Pages administration for self-compiled installations](source.md). ## The GitLab Pages daemon @@ -225,88 +224,89 @@ This setup is primarily intended to be used when [installing a GitLab POC on Ama ### Global settings -Below is a table of all configuration settings known to Pages in Omnibus GitLab, +Below is a table of all configuration settings known to Pages in a Linux package installation, and what they do. These options can be adjusted in `/etc/gitlab/gitlab.rb`, and take effect after you [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). Most of these settings don't have to be configured manually unless you need more granular control over how the Pages daemon runs and serves content in your environment. -| Setting | Description | -|-----------------------------------------|-------------| +| Setting | Description | +|-----------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `pages_external_url` | The URL where GitLab Pages is accessible, including protocol (HTTP / HTTPS). If `https://` is used, additional configuration is required. See [Wildcard domains with TLS support](#wildcard-domains-with-tls-support) and [Custom domains with TLS support](#custom-domains-with-tls-support) for details. | -| **`gitlab_pages[]`** | | -| `access_control` | Whether to enable [access control](index.md#access-control). | -| `api_secret_key` | Full path to file with secret key used to authenticate with the GitLab API. Auto-generated when left unset. | -| `artifacts_server` | Enable viewing [artifacts](../job_artifacts.md) in GitLab Pages. | -| `artifacts_server_timeout` | Timeout (in seconds) for a proxied request to the artifacts server. | -| `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server), this URL must point to the main GitLab server's API. | -| `auth_redirect_uri` | Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`. | -| `auth_secret` | Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration. | -| `dir` | Working directory for configuration and secrets files. | -| `enable` | Enable or disable GitLab Pages on the current system. | -| `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. | -| `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. | -| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: `30s`). | -| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: `10s`). | -| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: `30s`). | -| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: `600s`). | -| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: `60s`). | -| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: `60s`). | -| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: `30s`). | -| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: `1s`). | -| `gitlab_retrieval_retries` | The maximum number of times to retry to resolve a domain's configuration via the API (default: 3). | -| `domain_config_source` | This parameter was removed in 14.0, on earlier versions it can be used to enable and test API domain configuration source | -| `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | -| `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | -| `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | -| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: `10m`). A value of `0` means the cookie is deleted after the browser session ends. | -| `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. | -| `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` | -| `enable_disk` | Allows the GitLab Pages daemon to serve content from disk. Shall be disabled if shared disk storage isn't available. | -| `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. | -| `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. | -| `listen_proxy` | The addresses to listen on for reverse-proxy requests. Pages binds to these addresses' network sockets and receives incoming requests from them. Sets the value of `proxy_pass` in `$nginx-dir/conf/gitlab-pages.conf`. | -| `log_directory` | Absolute path to a log directory. | -| `log_format` | The log output format: `text` or `json`. | -| `log_verbose` | Verbose logging, true/false. | -| `propagate_correlation_id` | Set to true (false by default) to re-use existing Correlation ID from the incoming request header `X-Request-ID` if present. If a reverse proxy sets this header, the value is propagated in the request chain. | -| `max_connections` | Limit on the number of concurrent connections to the HTTP, HTTPS or proxy listeners. | +| **`gitlab_pages[]`** | | +| `access_control` | Whether to enable [access control](index.md#access-control). | +| `api_secret_key` | Full path to file with secret key used to authenticate with the GitLab API. Auto-generated when left unset. | +| `artifacts_server` | Enable viewing [artifacts](../job_artifacts.md) in GitLab Pages. | +| `artifacts_server_timeout` | Timeout (in seconds) for a proxied request to the artifacts server. | +| `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server), this URL must point to the main GitLab server's API. | +| `auth_redirect_uri` | Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`. | +| `auth_secret` | Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration. | +| `dir` | Working directory for configuration and secrets files. | +| `enable` | Enable or disable GitLab Pages on the current system. | +| `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. | +| `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. | +| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: `30s`). | +| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: `10s`). | +| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: `30s`). | +| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: `600s`). | +| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: `60s`). | +| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: `60s`). | +| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: `30s`). | +| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: `1s`). | +| `gitlab_retrieval_retries` | The maximum number of times to retry to resolve a domain's configuration via the API (default: 3). | +| `domain_config_source` | This parameter was removed in 14.0, on earlier versions it can be used to enable and test API domain configuration source | +| `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | +| `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | +| `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | +| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: `10m`). A value of `0` means the cookie is deleted after the browser session ends. | +| `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. | +| `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` | +| `enable_disk` | Allows the GitLab Pages daemon to serve content from disk. Shall be disabled if shared disk storage isn't available. | +| `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. | +| `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. | +| `listen_proxy` | The addresses to listen on for reverse-proxy requests. Pages binds to these addresses' network sockets and receives incoming requests from them. Sets the value of `proxy_pass` in `$nginx-dir/conf/gitlab-pages.conf`. | +| `log_directory` | Absolute path to a log directory. | +| `log_format` | The log output format: `text` or `json`. | +| `log_verbose` | Verbose logging, true/false. | +| `propagate_correlation_id` | Set to true (false by default) to re-use existing Correlation ID from the incoming request header `X-Request-ID` if present. If a reverse proxy sets this header, the value is propagated in the request chain. | +| `max_connections` | Limit on the number of concurrent connections to the HTTP, HTTPS or proxy listeners. | | `max_uri_length` | The maximum length of URIs accepted by GitLab Pages. Set to 0 for unlimited length. [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5729) in GitLab 14.5. -| `metrics_address` | The address to listen on for metrics requests. | -| `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. | -| `redirects_max_config_size` | The maximum size of the `_redirects` file, in bytes (default: 65536). | -| `redirects_max_path_segments` | The maximum number of path segments allowed in `_redirects` rules URLs (default: 25). | -| `redirects_max_rule_count` | The maximum number of rules allowed in `_redirects` (default: 1000). | -| `sentry_dsn` | The address for sending Sentry crash reporting to. | -| `sentry_enabled` | Enable reporting and logging with Sentry, true/false. | -| `sentry_environment` | The environment for Sentry crash reporting. | -| `status_uri` | The URL path for a status page, for example, `/@status`. | -| `tls_max_version` | Specifies the maximum TLS version ("tls1.2" or "tls1.3"). | -| `tls_min_version` | Specifies the minimum TLS version ("tls1.2" or "tls1.3"). | -| `use_http2` | Enable HTTP2 support. | -| **`gitlab_pages['env'][]`** | | -| `http_proxy` | Configure GitLab Pages to use an HTTP Proxy to mediate traffic between Pages and GitLab. Sets an environment variable `http_proxy` when starting Pages daemon. | -| **`gitlab_rails[]`** | | -| `pages_domain_verification_cron_worker` | Schedule for verifying custom GitLab Pages domains. | -| `pages_domain_ssl_renewal_cron_worker` | Schedule for obtaining and renewing SSL certificates through Let's Encrypt for GitLab Pages domains. | -| `pages_domain_removal_cron_worker` | Schedule for removing unverified custom GitLab Pages domains. | -| `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. | -| **`pages_nginx[]`** | | -| `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). | -| `FF_ENABLE_PLACEHOLDERS` | Feature flag for rewrites (enabled by default). See [Rewrites](../../user/project/pages/redirects.md#rewrites) for more information. | -| `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Removed in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | -| `rate_limit_source_ip` | Rate limit per source IP in number of requests per second. Set to `0` to disable this feature. | -| `rate_limit_source_ip_burst` | Rate limit per source IP maximum burst allowed per second. | -| `rate_limit_domain` | Rate limit per domain in number of requests per second. Set to `0` to disable this feature. | -| `rate_limit_domain_burst` | Rate limit per domain maximum burst allowed per second. | -| `rate_limit_tls_source_ip` | Rate limit per source IP in number of TLS connections per second. Set to `0` to disable this feature. | -| `rate_limit_tls_source_ip_burst` | Rate limit per source IP maximum TLS connections burst allowed per second. | -| `rate_limit_tls_domain` | Rate limit per domain in number of TLS connections per second. Set to `0` to disable this feature. | -| `rate_limit_tls_domain_burst` | Rate limit per domain maximum TLS connections burst allowed per second. | -| `server_read_timeout` | Maximum duration to read the request headers and body. For no timeout, set to `0` or a negative value. Default: `5s` | -| `server_read_header_timeout` | Maximum duration to read the request headers. For no timeout, set to `0` or a negative value. Default: `1s` | -| `server_write_timeout` | Maximum duration to write all files in the response. Larger files require more time. For no timeout, set to `0` or a negative value. Default: `0` | -| `server_keep_alive` | The `Keep-Alive` period for network connections accepted by this listener. If `0`, `Keep-Alive` is enabled if supported by the protocol and operating system. If negative, `Keep-Alive` is disabled. Default: `15s` | +| `metrics_address` | The address to listen on for metrics requests. | +| `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. | +| `redirects_max_config_size` | The maximum size of the `_redirects` file, in bytes (default: 65536). | +| `redirects_max_path_segments` | The maximum number of path segments allowed in `_redirects` rules URLs (default: 25). | +| `redirects_max_rule_count` | The maximum number of rules allowed in `_redirects` (default: 1000). | +| `sentry_dsn` | The address for sending Sentry crash reporting to. | +| `sentry_enabled` | Enable reporting and logging with Sentry, true/false. | +| `sentry_environment` | The environment for Sentry crash reporting. | +| `status_uri` | The URL path for a status page, for example, `/@status`. | +| `tls_max_version` | Specifies the maximum TLS version ("tls1.2" or "tls1.3"). | +| `tls_min_version` | Specifies the minimum TLS version ("tls1.2" or "tls1.3"). | +| `use_http2` | Enable HTTP2 support. | +| **`gitlab_pages['env'][]`** | | +| `http_proxy` | Configure GitLab Pages to use an HTTP Proxy to mediate traffic between Pages and GitLab. Sets an environment variable `http_proxy` when starting Pages daemon. | +| **`gitlab_rails[]`** | | +| `pages_domain_verification_cron_worker` | Schedule for verifying custom GitLab Pages domains. | +| `pages_domain_ssl_renewal_cron_worker` | Schedule for obtaining and renewing SSL certificates through Let's Encrypt for GitLab Pages domains. | +| `pages_domain_removal_cron_worker` | Schedule for removing unverified custom GitLab Pages domains. | +| `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. | +| **`pages_nginx[]`** | | +| `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). | +| `FF_CONFIGURABLE_ROOT_DIR` | Feature flag to [customize the default folder](../../user/project/pages/introduction.md#customize-the-default-folder) (enabled by default). | +| `FF_ENABLE_PLACEHOLDERS` | Feature flag for rewrites (enabled by default). See [Rewrites](../../user/project/pages/redirects.md#rewrites) for more information. | +| `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Removed in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | +| `rate_limit_source_ip` | Rate limit per source IP in number of requests per second. Set to `0` to disable this feature. | +| `rate_limit_source_ip_burst` | Rate limit per source IP maximum burst allowed per second. | +| `rate_limit_domain` | Rate limit per domain in number of requests per second. Set to `0` to disable this feature. | +| `rate_limit_domain_burst` | Rate limit per domain maximum burst allowed per second. | +| `rate_limit_tls_source_ip` | Rate limit per source IP in number of TLS connections per second. Set to `0` to disable this feature. | +| `rate_limit_tls_source_ip_burst` | Rate limit per source IP maximum TLS connections burst allowed per second. | +| `rate_limit_tls_domain` | Rate limit per domain in number of TLS connections per second. Set to `0` to disable this feature. | +| `rate_limit_tls_domain_burst` | Rate limit per domain maximum TLS connections burst allowed per second. | +| `server_read_timeout` | Maximum duration to read the request headers and body. For no timeout, set to `0` or a negative value. Default: `5s` | +| `server_read_header_timeout` | Maximum duration to read the request headers. For no timeout, set to `0` or a negative value. Default: `1s` | +| `server_write_timeout` | Maximum duration to write all files in the response. Larger files require more time. For no timeout, set to `0` or a negative value. Default: `0` | +| `server_keep_alive` | The `Keep-Alive` period for network connections accepted by this listener. If `0`, `Keep-Alive` is enabled if supported by the protocol and operating system. If negative, `Keep-Alive` is disabled. Default: `15s` | ## Advanced configuration @@ -530,7 +530,7 @@ This usually results in this error: For installation from source, this can be fixed by installing the custom Certificate Authority (CA) in the system certificate store. -For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). +For Linux package installations, this is fixed by [installing a custom CA](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates). ### ZIP serving and cache configuration @@ -590,6 +590,27 @@ gitlab_pages['redirects_max_path_segments'] = 50 gitlab_pages['redirects_max_rule_count'] = 2000 ``` +## Use environment variables + +You can pass an environment variable to the Pages daemon (for example, +to enable or disable a feature flag). + +To disable the configurable directory feature: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['env'] = { + 'FF_CONFIGURABLE_ROOT_DIR' => "false" + } + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + ## Activate verbose logging for daemon Follow the steps below to configure verbose logging of GitLab Pages daemon. @@ -698,10 +719,7 @@ Prerequisite: To set the maximum size of GitLab Pages site in a project, overriding the inherited setting: 1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. -1. On the left sidebar, select **Settings > Pages**. - - If this path is not visible, select **Deployments > Pages**. - [This location is part of an experiment](../../user/project/pages/index.md#menu-position-test). +1. On the left sidebar, select **Deploy > Pages**. 1. In **Maximum size of pages**, enter the size in MB. 1. Select **Save changes**. @@ -758,7 +776,7 @@ database encryption. Proceed with caution. gitlab_pages['access_control'] = true ``` -1. Configure [the object storage and migrate pages data to it](#using-object-storage). +1. Configure [the object storage and migrate pages data to it](#object-storage-settings). 1. [Reconfigure the **GitLab server**](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. The `gitlab-secrets.json` file is now updated with the @@ -766,7 +784,7 @@ database encryption. Proceed with caution. 1. Set up a new server. This becomes the **Pages server**. -1. On the **Pages server**, install Omnibus GitLab and modify `/etc/gitlab/gitlab.rb` +1. On the **Pages server**, install GitLab by using the Linux package and modify `/etc/gitlab/gitlab.rb` to include: ```ruby @@ -832,45 +850,12 @@ This approach had several disadvantages and was replaced with GitLab Pages using every time a new domain is requested. The domain information is also cached by the Pages daemon to speed up subsequent requests. -From [GitLab 13.3 to GitLab 13.12](#domain-source-configuration-before-140) GitLab Pages supported both ways of obtaining domain information. - Starting from [GitLab 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages uses API by default and fails to start if it can't connect to it. For common issues, see [troubleshooting](troubleshooting.md#failed-to-connect-to-the-internal-gitlab-api). For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). -### Domain source configuration before 14.0 - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3. - -WARNING: -`domain_config_source` parameter is removed and has no effect starting from [GitLab 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) - -From [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) to [GitLab 13.12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages can either use `disk` or `gitlab` domain configuration source. - -We highly advise you to use `gitlab` configuration source as it makes transitions to newer versions easier. - -To explicitly enable API source: - -1. Add the following to your `/etc/gitlab/gitlab.rb` file: - - ```ruby - gitlab_pages['domain_config_source'] = "gitlab" - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. - -Or if you want to use legacy configuration source you can: - -1. Add the following to your `/etc/gitlab/gitlab.rb` file: - - ```ruby - gitlab_pages['domain_config_source'] = "disk" - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. - ### GitLab API cache configuration > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/520) in GitLab 13.10. @@ -909,18 +894,12 @@ only when there is an error response from the API, for example a connection time - Decreasing `gitlab_retrieval_retries` reduces the number of times a domain's configuration is tried to be resolved automatically before reporting an error. -## Using object storage - -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5577) in GitLab 13.6. +## Object storage settings -For more information, see [object storage](../object_storage.md). +The following [object storage](../object_storage.md) settings are: -### Object storage settings - -The following settings are: - -- Nested under `pages:` and then `object_store:` on source installations. -- Prefixed by `pages_object_store_` on Omnibus GitLab installations. +- Nested under `pages:` and then `object_store:` on self-compiled installations. +- Prefixed by `pages_object_store_` on Linux package installations. | Setting | Description | Default | |---------|-------------|---------| @@ -932,7 +911,7 @@ NOTE: If you want to stop using and disconnect the NFS server, you need to [explicitly disable local storage](#disable-pages-local-storage), and it's only possible after upgrading to GitLab 13.11. -#### S3-compatible connection settings +### S3-compatible connection settings In GitLab 13.2 and later, you should use the [consolidated object storage settings](../object_storage.md#configure-a-single-storage-connection-for-all-object-types-consolidated-form). @@ -940,7 +919,9 @@ This section describes the earlier configuration format. See [the available connection settings for different providers](../object_storage.md#configure-the-connection-settings). -In Omnibus installations: +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Add the following lines to `/etc/gitlab/gitlab.rb` and replace the values with the ones you want: @@ -971,7 +952,7 @@ In Omnibus installations: 1. [Migrate existing Pages deployments to object storage.](#migrate-pages-deployments-to-object-storage) -In installations from source: +:::TabTitle Self-compiled (source) 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -992,75 +973,14 @@ In installations from source: 1. [Migrate existing Pages deployments to object storage.](#migrate-pages-deployments-to-object-storage) -## ZIP storage - -In GitLab 14.0 the underlying storage format of GitLab Pages is changing from -files stored directly in disk to a single ZIP archive per project. - -These ZIP archives can be stored either locally on disk storage or on [object storage](#using-object-storage) if it is configured. - -[Starting from GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/245308) ZIP archives are stored every time pages site is updated. - -### Migrate legacy storage to ZIP storage - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59003) in GitLab 13.11. - -GitLab tries to -[automatically migrate](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54578) -the old storage format to the new ZIP-based one when you upgrade to GitLab 13.11 or further. -However, some projects may fail to be migrated for different reasons. -To verify that all projects have been migrated successfully, you can manually run the migration: - -```shell -sudo gitlab-rake gitlab:pages:migrate_legacy_storage -``` - -It's safe to interrupt this task and run it multiple times. - -There are two most common problems this task can report: - -- `Missing public directory` error: - - ```txt - E, [2021-04-09T13:11:52.534768 #911919] ERROR -- : project_id: 1 /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test failed to be migrated in 0.07 seconds: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test - ``` - - In this case, you should verify that these projects don't have pages deployed, and re-run the migration with an additional flag to mark those projects as not deployed with GitLab Pages: - - ```shell - sudo PAGES_MIGRATION_MARK_PROJECTS_AS_NOT_DEPLOYED=true gitlab-rake gitlab:pages:migrate_legacy_storage - ``` - -- File `is invalid` error: - - ```txt - E, [2021-04-09T14:43:05.821767 #923322] ERROR -- : project_id: 1 /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test failed to be migrated: /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test/public/link is invalid, input_dir: /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test - ``` - - This error indicates invalid files on disk storage, most commonly symlinks leading outside of the `public` directory. - You can manually remove these files, or just ignore them during migration: - - ```shell - sudo PAGES_MIGRATION_IGNORE_INVALID_ENTRIES=true gitlab-rake gitlab:pages:migrate_legacy_storage - ``` - -### Rolling back ZIP migration - -If you find that migrated data is invalid, you can remove all migrated data by running: - -```shell -sudo gitlab-rake gitlab:pages:clean_migrated_zip_storage -``` - -This does not remove any data from the legacy disk storage and the GitLab Pages daemon automatically falls back -to using that. +::EndTabs ### Migrate Pages deployments to object storage -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325285) in GitLab 13.11. +Existing Pages deployment objects (zip archives) can be stored in either: -Existing Pages deployment objects (which store [ZIP archives](#zip-storage)) can similarly be -migrated to [object storage](#using-object-storage). +- Local storage +- [Object storage](../object_storage.md) Migrate your existing Pages deployments from local storage to object storage: @@ -1071,9 +991,9 @@ sudo gitlab-rake gitlab:pages:deployments:migrate_to_object_storage You can track progress and verify that all Pages deployments migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. -- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. -- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. +- `sudo gitlab-rails dbconsole` for Linux package installations running GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Linux package installations running 14.2 and later. +- `sudo -u git -H psql -d gitlabhq_production` for self-compiled installations. Verify `objectstg` below (where `store=2`) has count of all Pages deployments: @@ -1100,7 +1020,7 @@ sudo gitlab-rake gitlab:pages:deployments:migrate_to_local > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301159) in GitLab 13.11. -If you use [object storage](#using-object-storage), you can disable local storage to avoid unnecessary disk usage/writes: +If you use [object storage](#object-storage-settings), you can disable local storage to avoid unnecessary disk usage/writes: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -1110,34 +1030,18 @@ If you use [object storage](#using-object-storage), you can disable local storag 1. [Reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation) for the changes to take effect. -Starting from GitLab 13.12, this setting also disables the [legacy storage](#migrate-legacy-storage-to-zip-storage), so if you were using NFS to serve Pages, you can completely disconnect from it. - -## Prepare GitLab Pages for 14.0 - -In GitLab 14.0 a number of breaking changes were introduced which may require some user intervention. -The steps below describe the best way to migrate without causing any downtime for your GitLab instance. +## ZIP storage -A GitLab instance running on a single server typically upgrades to 14.0 smoothly, and there should be minimal issues after the upgrade is complete. -Regardless, you should follow the migration steps to ensure a successful upgrade. -For common issues, see [troubleshooting](troubleshooting.md). +In GitLab 14.0 the underlying storage format of GitLab Pages changed from +files stored directly in disk to a single ZIP archive per project. -If your current GitLab version is lower than 13.12, then you must first update to 13.12. -Updating directly to 14.0 is [not supported](../../update/index.md#upgrade-paths) -and may cause downtime for some web-sites hosted on GitLab Pages. After you update to 13.12, -migrate GitLab Pages to prepare them for GitLab 14.0: +These ZIP archives can be stored either locally on disk storage or on [object storage](#object-storage-settings) if it is configured. -1. Set [`domain_config_source` to `gitlab`](#domain-source-configuration-before-140), which -is the default starting from GitLab 14.0. Skip this step if you're already running GitLab 14.0 or above. -1. If you want to store your pages content in [object storage](#using-object-storage), make sure to configure it. -If you want to store the pages content locally or continue using an NFS server, skip this step. -1. [Migrate legacy storage to ZIP storage.](#migrate-legacy-storage-to-zip-storage) -1. If you have configured GitLab to store your pages content in [object storage](#using-object-storage), - [migrate Pages deployments to object storage](#migrate-pages-deployments-to-object-storage) -1. Upgrade GitLab to 14.0. +[Starting from GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/245308) ZIP archives are stored every time pages site is updated. ## Backup -GitLab Pages are part of the [regular backup](../../raketasks/backup_restore.md), so there is no separate backup to configure. +GitLab Pages are part of the [regular backup](../../administration/backup_restore/index.md), so there is no separate backup to configure. ## Security diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 2ee9dd653f0..26dedd47473 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -4,21 +4,17 @@ group: Knowledge info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab Pages administration for source installations **(FREE SELF)** +# GitLab Pages administration for self-compiled installations **(FREE SELF)** NOTE: Before attempting to enable GitLab Pages, first make sure you have [installed GitLab](../../install/installation.md) successfully. -This document explains how to configure GitLab Pages when you have installed -GitLab from source and not the Omnibus packages. +This document explains how to configure GitLab Pages for self-compiled GitLab installations. -You are encouraged to read the [Omnibus documentation](index.md) as it provides -invaluable information about the configuration of GitLab Pages. +For more information about configuring GitLab Pages for Linux Package installations (recommended), see the [Linux package documentation](index.md). -We also highly recommend that you use the Omnibus GitLab packages. We -optimize them specifically for GitLab, and we take care of upgrading GitLab -Pages to the latest supported version. +The advantage of using the Linux package installation is that it contains the latest supported version of GitLab Pages. ## How GitLab Pages works @@ -491,7 +487,7 @@ To change this value: ## Backup -Pages are part of the [regular backup](../../raketasks/backup_restore.md) so there is nothing to configure. +Pages are part of the [regular backup](../../administration/backup_restore/index.md) so there is nothing to configure. ## Security diff --git a/doc/administration/pages/troubleshooting.md b/doc/administration/pages/troubleshooting.md index e829943f270..7e184631515 100644 --- a/doc/administration/pages/troubleshooting.md +++ b/doc/administration/pages/troubleshooting.md @@ -250,7 +250,7 @@ the shared pages directory is mounted on a different path on the main GitLab ser GitLab Pages server. In that case, it's highly recommended you to configure -[object storage and migrate any existing pages data to it](index.md#using-object-storage). +[object storage and migrate any existing pages data to it](index.md#object-storage-settings). Alternatively, you can mount the GitLab Pages shared directory to the same path on both servers. @@ -259,7 +259,7 @@ both servers. GitLab 14.0 introduces a number of changes to GitLab Pages which may require manual intervention. -1. Firstly [follow the migration guide](index.md#prepare-gitlab-pages-for-140). +1. Firstly [follow the migration guide](https://archives.docs.gitlab.com/14.10/ee/administration/pages/#prepare-gitlab-pages-for-140). 1. Try to upgrade to GitLab 14.3 or above. Some of the issues were fixed in GitLab 14.1, 14.2 and 14.3. 1. If it doesn't work, see [GitLab Pages logs](#how-to-see-gitlab-pages-logs), and if you see any errors there then search them on this page. diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index cc550dbe389..f8b6be1fb21 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -64,7 +64,7 @@ To enable Database Load Balancing, make sure that: - The HA PostgreSQL setup has one or more secondary nodes replicating the primary. - Each PostgreSQL node is connected with the same credentials and on the same port. -For Omnibus GitLab, you also need PgBouncer configured on each PostgreSQL node to pool +For Linux package installations, you also need PgBouncer configured on each PostgreSQL node to pool all load-balanced connections when [configuring a multi-node setup](replication_and_failover.md). ## Configuring Database Load Balancing @@ -112,7 +112,7 @@ checks a DNS `A` record, using the IPs returned by this record as the addresses for the secondaries. For service discovery to work, all you need is a DNS server and an `A` record containing the IP addresses of your secondaries. -When using Omnibus GitLab the provided [Consul](../consul.md) service works as +When using a Linux package installation, the provided [Consul](../consul.md) service works as a DNS server and returns PostgreSQL addresses via the `postgresql-ha.service.consul` record. For example: diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 8f664f9809e..9c44d53213b 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -11,7 +11,7 @@ managed service for PostgreSQL. For example, AWS offers a managed Relational Database Service (RDS) that runs PostgreSQL. Alternatively, you may opt to manage your own PostgreSQL instance or cluster -separate from the Omnibus GitLab package. +separate from the Linux package. If you use a cloud-managed service, or provide your own PostgreSQL instance: diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index e829397abc1..af0a86c3d72 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -13,32 +13,32 @@ be used with GitLab in one of our [reference architectures](../reference_archite Choose one of the following PostgreSQL configuration options: -### Standalone PostgreSQL using Omnibus GitLab +### Standalone PostgreSQL for Linux package installations -This setup is for when you have installed the -[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE), +This setup is for when you have installed GitLab by using the +[Linux package](https://about.gitlab.com/install/) (CE or EE), to use the bundled PostgreSQL having only its service enabled. -Read how to [set up a standalone PostgreSQL instance](standalone.md) using Omnibus GitLab. +Read how to [set up a standalone PostgreSQL instance](standalone.md) for Linux package installations. ### Provide your own PostgreSQL instance This setup is for when you have installed GitLab using the -[Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE), -or installed it [from source](../../install/installation.md), but you want to use +[Linux package](https://about.gitlab.com/install/) (CE or EE), +or [self-compiled](../../install/installation.md) your installation, but you want to use your own external PostgreSQL server. Read how to [set up an external PostgreSQL instance](external.md). -### PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)** +### PostgreSQL replication and failover for Linux package installations **(PREMIUM SELF)** This setup is for when you have installed GitLab using the -[Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). +[Linux **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). All the tools that are needed like PostgreSQL, PgBouncer, and Patroni are bundled in the package, so you can use it to set up the whole PostgreSQL infrastructure (primary, replica). -Read how to [set up PostgreSQL replication and failover](replication_and_failover.md) using Omnibus GitLab. +Read how to [set up PostgreSQL replication and failover](replication_and_failover.md) for Linux package installations. ## Related topics diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md index f03781d0ee2..5dcb080d707 100644 --- a/doc/administration/postgresql/multiple_databases.md +++ b/doc/administration/postgresql/multiple_databases.md @@ -65,7 +65,7 @@ copy the database across. 1. Configure GitLab to [use multiple databases](#set-up-multiple-databases). -### Existing Omnibus installation +### Existing Linux package installations 1. Stop GitLab, except for PostgreSQL: @@ -103,7 +103,7 @@ the other way around. 1. For existing installations, [migrate the data](#migrate-existing-installations) first. -1. [Back up GitLab](../../raketasks/backup_restore.md) +1. [Back up GitLab](../../administration/backup_restore/index.md) in case of unforeseen issues. 1. Stop GitLab: @@ -152,12 +152,12 @@ the other way around. sudo service gitlab restart ``` -### Omnibus GitLab installations +### Linux package installations 1. For existing installations, [migrate the data](#migrate-existing-installations) first. -1. [Back up GitLab](../../raketasks/backup_restore.md) +1. [Back up GitLab](../../administration/backup_restore/index.md) in case of unforeseen issues. 1. Stop GitLab: diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 59aac9141a4..7e46933113b 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -25,9 +25,9 @@ This content has been moved to a [new location](replication_and_failover.md#conf ## PgBouncer as part of a non-fault-tolerant GitLab installation -1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer` +1. Generate `PGBOUNCER_USER_PASSWORD_HASH` with the command `gitlab-ctl pg-password-md5 pgbouncer` -1. Generate SQL_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 gitlab`. Enter the plaintext SQL_USER_PASSWORD later. +1. Generate `SQL_USER_PASSWORD_HASH` with the command `gitlab-ctl pg-password-md5 gitlab`. Enter the plaintext SQL_USER_PASSWORD later. 1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb` @@ -88,7 +88,7 @@ This content has been moved to a [new location](replication_and_failover.md#conf Do not backup or restore GitLab through a PgBouncer connection: it causes a GitLab outage. -[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer). +[Read more about this and how to reconfigure backups](../../administration/backup_restore/backup_gitlab.md#back-up-and-restore-for-installations-using-pgbouncer). ## Enable Monitoring @@ -119,7 +119,7 @@ If you enable Monitoring, it must be enabled on **all** PgBouncer servers. ## Administrative console -As part of Omnibus GitLab, a command is provided to automatically connect to the +In Linux package installations, a command is provided to automatically connect to the PgBouncer administrative console. See the [PgBouncer documentation](https://www.pgbouncer.org/usage.html#admin-console) for detailed instructions on how to interact with the console. @@ -174,11 +174,11 @@ ote_pid | tls ## Procedure for bypassing PgBouncer -### Omnibus installations +### Linux package installations Some database changes have to be done directly, and not through PgBouncer. -The main affected tasks are [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) +The main affected tasks are [database restores](../../administration/backup_restore/backup_gitlab.md#back-up-and-restore-for-installations-using-pgbouncer) and [GitLab upgrades with database migrations](../../update/zero_downtime.md#postgresql). 1. To find the primary node, run the following on a database node: @@ -208,8 +208,8 @@ After you've performed the tasks or procedure, switch back to using PgBouncer: ### Helm chart installations -High-availability deployments also need to bypass PgBouncer for the same reasons as Omnibus-based ones. -For this type of installation: +High-availability deployments also need to bypass PgBouncer for the same reasons as Linux package-based ones. +For Helm chart installations: - Database backup and restore tasks are performed by the toolbox container. - Migration tasks are performed by the migrations container. @@ -226,7 +226,7 @@ In specific cases you may want to change the performance-specific and resource-s throughput or to limit resource utilization that could cause memory exhaustion on the database. You can find the parameters and respective documentation on the [official PgBouncer documentation](https://www.pgbouncer.org/config.html). -Listed below are the most relevant ones and their defaults on an Omnibus GitLab installation: +Listed below are the most relevant ones and their defaults on a Linux package installation: - `pgbouncer['max_client_conn']` (default: `2048`, depends on server file descriptor limits) This is the "frontend" pool in PgBouncer: connections from Rails to PgBouncer. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 392f9f2b89c..05ff6fa8a4a 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -4,21 +4,21 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)** +# PostgreSQL replication and failover for Linux package installations **(PREMIUM SELF)** If you're a Free user of GitLab self-managed, consider using a cloud-hosted solution. This document doesn't cover installations from source. If a setup with replication and failover isn't what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) -for the Omnibus GitLab packages. +for the Linux packages. It's recommended to read this document fully before attempting to configure PostgreSQL with replication and failover for GitLab. ## Architecture -The Omnibus GitLab recommended configuration for a PostgreSQL cluster with +The Linux pacakage-recommended configuration for a PostgreSQL cluster with replication failover requires: - A minimum of three PostgreSQL nodes. @@ -73,9 +73,9 @@ sure you have redundant connectivity between all Database and GitLab instances to avoid the network becoming a single point of failure. NOTE: -As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is supported only with +As of GitLab 13.3, PostgreSQL 12 is shipped with Linux package installations. Clustering for PostgreSQL 12 is supported only with Patroni. See the [Patroni](#patroni) section for further details. Starting with GitLab 14.0, only PostgreSQL 12 is -shipped with Omnibus GitLab, and thus Patroni becomes mandatory for replication and failover. +shipped with Linux package installations, and thus Patroni becomes mandatory for replication and failover. ### Database node @@ -152,7 +152,7 @@ This is why you need: When using default setup, minimum configuration requires: -- `CONSUL_USERNAME`. The default user for Omnibus GitLab is `gitlab-consul` +- `CONSUL_USERNAME`. The default user for Linux package installations is `gitlab-consul` - `CONSUL_DATABASE_PASSWORD`. Password for the database user. - `CONSUL_PASSWORD_HASH`. This is a hash generated out of Consul username/password pair. It can be generated with: @@ -192,7 +192,7 @@ server nodes on hand. You need the following password information for the application's database user: -- `POSTGRESQL_USERNAME`. The default user for Omnibus GitLab is `gitlab` +- `POSTGRESQL_USERNAME`. The default user for Linux package installations is `gitlab` - `POSTGRESQL_USER_PASSWORD`. The password for the database user - `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. It can be generated with: @@ -212,7 +212,7 @@ You need the following password information for the Patroni API: When using a default setup, the minimum configuration requires: -- `PGBOUNCER_USERNAME`. The default user for Omnibus GitLab is `pgbouncer` +- `PGBOUNCER_USERNAME`. The default user for Linux package installations is `pgbouncer` - `PGBOUNCER_PASSWORD`. This is a password for PgBouncer service. - `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of PgBouncer username/password pair. It can be generated with: @@ -230,10 +230,9 @@ Few things to remember about the service itself: - `/etc/gitlab/gitlab.rb`: hashed, and in plain text - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed -### Installing Omnibus GitLab +### Installing the Linux package -First, make sure to [download/install](https://about.gitlab.com/install/) -Omnibus GitLab **on each node**. +First, make sure to [download and install](https://about.gitlab.com/install/) the Linux package **on each node**. Make sure you install the necessary dependencies from step 1, add GitLab package repository from step 2. @@ -373,7 +372,7 @@ patroni['tls_key_password'] = 'private-key-password' # This is the plain-text pa ``` If you are using a self-signed certificate or an internal CA, you need to either disable the TLS verification or pass the certificate of the -internal CA, otherwise you may run into an unexpected error when using the `gitlab-ctl patroni ....` commands. Omnibus ensures that Patroni API +internal CA, otherwise you may run into an unexpected error when using the `gitlab-ctl patroni ....` commands. The Linux package ensures that Patroni API clients honor this configuration. TLS certificate verification is enabled by default. To disable it: @@ -415,7 +414,7 @@ authentication mode (`patroni['tls_client_mode']`), must each have the same valu 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. -1. One each node, edit the `/etc/gitlab/gitlab.rb` configuration file and replace values noted in the `# START user configuration` section as below: +1. On each node, edit the `/etc/gitlab/gitlab.rb` configuration file and replace values noted in the `# START user configuration` section as below: ```ruby # Disable all components except PgBouncer and Consul agent @@ -571,7 +570,7 @@ in the Troubleshooting section before proceeding. Do not backup or restore GitLab through a PgBouncer connection: this causes a GitLab outage. -[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer). +[Read more about this and how to reconfigure backups](../../administration/backup_restore/backup_gitlab.md#back-up-and-restore-for-installations-using-pgbouncer). ### Ensure GitLab is running @@ -837,7 +836,7 @@ Read more about the data returned by the replica ### Selecting the appropriate Patroni replication method -[Review the Patroni documentation carefully](https://patroni.readthedocs.io/en/latest/SETTINGS.html#postgresql) +[Review the Patroni documentation carefully](https://patroni.readthedocs.io/en/latest/yaml_configuration.html#postgresql) before making changes as **_some of the options carry a risk of potential data loss if not fully understood_**. The [replication mode](https://patroni.readthedocs.io/en/latest/replication_modes.html) configured determines the amount of tolerable data loss. @@ -845,7 +844,7 @@ configured determines the amount of tolerable data loss. WARNING: Replication is not a backup strategy! There is no replacement for a well-considered and tested backup solution. -Omnibus GitLab defaults [`synchronous_commit`](https://www.postgresql.org/docs/11/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) to `on`. +Linux package installations default [`synchronous_commit`](https://www.postgresql.org/docs/11/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) to `on`. ```ruby postgresql['synchronous_commit'] = 'on' @@ -854,7 +853,7 @@ gitlab['geo-postgresql']['synchronous_commit'] = 'on' #### Customizing Patroni failover behavior -Omnibus GitLab exposes several options allowing more control over the [Patroni restoration process](#recovering-the-patroni-cluster). +Linux package installations expose several options allowing more control over the [Patroni restoration process](#recovering-the-patroni-cluster). Each option is shown below with its default value in `/etc/gitlab/gitlab.rb`. @@ -864,7 +863,7 @@ patroni['remove_data_directory_on_rewind_failure'] = false patroni['remove_data_directory_on_diverged_timelines'] = false ``` -[The upstream documentation is always more up to date](https://patroni.readthedocs.io/en/latest/SETTINGS.html#postgresql), but the table below should provide a minimal overview of functionality. +[The upstream documentation is always more up to date](https://patroni.readthedocs.io/en/latest/patroni_configuration.html), but the table below should provide a minimal overview of functionality. |Setting|Overview| |-|-| @@ -996,7 +995,9 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with ### Upgrading PostgreSQL major version in a Patroni cluster -As of GitLab 14.1, PostgreSQL 12.6 and 13.3 are both shipped with Omnibus GitLab by default. As of GitLab 15.0, PostgreSQL 13 is the default. If you want to upgrade to PostgreSQL 13 in versions prior to GitLab 15.0, you must ask for it explicitly. +As of GitLab 14.1, PostgreSQL 12.6 and 13.3 are both shipped with the Linux package by default. As of GitLab 15.0, +PostgreSQL 13 is the default. If you want to upgrade to PostgreSQL 13 in versions prior to GitLab 15.0, you must ask for +it explicitly. WARNING: The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. @@ -1466,9 +1467,9 @@ Workarounds: - If set to enforcing, SELinux may also prevent these operations. Verify the issue is fixed by setting SELinux to permissive. -Patroni first shipped in Omnibus GitLab 13.1, along with a build of Python 3.7. +Patroni first shipped in the Linux package for GitLab 13.1, along with a build of Python 3.7. The code which causes this was removed in Python 3.8: this fix shipped in -[Omnibus GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5547) +[the Linux package for GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5547) and later, removing the need for a workaround. ### Errors running `gitlab-ctl` diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index d00310ecee0..d6f3460e255 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -4,19 +4,19 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Standalone PostgreSQL using Omnibus GitLab **(FREE SELF)** +# Standalone PostgreSQL for Linux package installations **(FREE SELF)** If you wish to have your database service hosted separately from your GitLab application servers, you can do this using the PostgreSQL binaries packaged -together with Omnibus GitLab. This is recommended as part of our +together with the Linux package. This is recommended as part of our [reference architecture for up to 2,000 users](../reference_architectures/2k_users.md). ## Setting it up 1. SSH in to the PostgreSQL server. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using *steps 1 and 2* from the GitLab downloads page. - - Do not complete any other steps on the download page. +1. [Download and install](https://about.gitlab.com/install/) the Linux + package you want using *steps 1 and 2* from the GitLab downloads page. Do not complete any other steps on the + download page. 1. Generate a password hash for PostgreSQL. This assumes you are using the default username of `gitlab` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index e55a0f1c8a7..9ced19b53b7 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -42,17 +42,17 @@ This task loops through the project code repositories and runs the integrity che described previously. If a project uses a pool repository, that is also checked. Other types of Git repositories [are not checked](https://gitlab.com/gitlab-org/gitaly/-/issues/3643). -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:git:fsck -``` + ```shell + sudo gitlab-rake gitlab:git:fsck + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake gitlab:git:fsck RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake gitlab:git:fsck RAILS_ENV=production + ``` ## Checksum of repository refs @@ -73,17 +73,17 @@ checksums in the format `<PROJECT ID>,<CHECKSUM>`. - If a repository exists but is empty, the output checksum is `0000000000000000000000000000000000000000`. - Projects which don't exist are skipped. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:git:checksum_projects -``` + ```shell + sudo gitlab-rake gitlab:git:checksum_projects + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake gitlab:git:checksum_projects RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake gitlab:git:checksum_projects RAILS_ENV=production + ``` For example, if: @@ -124,23 +124,23 @@ Integrity checks are supported for the following types of file: - Project-level Secure Files (introduced in GitLab 16.1.0) - User uploads (introduced in GitLab 10.6.0) -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:artifacts:check -sudo gitlab-rake gitlab:ci_secure_files:check -sudo gitlab-rake gitlab:lfs:check -sudo gitlab-rake gitlab:uploads:check -``` + ```shell + sudo gitlab-rake gitlab:artifacts:check + sudo gitlab-rake gitlab:ci_secure_files:check + sudo gitlab-rake gitlab:lfs:check + sudo gitlab-rake gitlab:uploads:check + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production -sudo -u git -H bundle exec rake gitlab:ci_secure_files:check RAILS_ENV=production -sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production -sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:ci_secure_files:check RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production + ``` These tasks also accept some environment variables which you can use to override certain values: @@ -215,22 +215,22 @@ secrets file (`gitlab-secrets.json`). Automatic resolution is not yet implemented. If you have values that cannot be decrypted, you can follow steps to reset them, see our -documentation on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). +documentation on what to do [when the secrets file is lost](../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost). This can take a very long time, depending on the size of your database, as it checks all rows in all tables. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:doctor:secrets -``` + ```shell + sudo gitlab-rake gitlab:doctor:secrets + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:doctor:secrets RAILS_ENV=production -``` + ```shell + bundle exec rake gitlab:doctor:secrets RAILS_ENV=production + ``` **Example output** @@ -251,17 +251,17 @@ I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! To get more detailed information about which rows and columns can't be decrypted, you can pass a `VERBOSE` environment variable: -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1 -``` + ```shell + sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1 + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1 -``` + ```shell + bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1 + ``` **Example verbose output** diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 03b09e00f1c..c6bc891f529 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -18,17 +18,17 @@ next repository sync in a **secondary** node: This is equivalent of running `git repack -d` on a _bare_ repository. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake geo:git:housekeeping:incremental_repack -``` + ```shell + sudo gitlab-rake geo:git:housekeeping:incremental_repack + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_ENV=production + ``` ### Full Repack @@ -36,48 +36,48 @@ This is equivalent of running `git repack -d -A --pack-kept-objects` on a _bare_ repository which optionally, writes a reachability bitmap index when this is enabled in GitLab. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake geo:git:housekeeping:full_repack -``` + ```shell + sudo gitlab-rake geo:git:housekeeping:full_repack + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake geo:git:housekeeping:full_repack RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake geo:git:housekeeping:full_repack RAILS_ENV=production + ``` ### GC This is equivalent of running `git gc` on a _bare_ repository, optionally writing a reachability bitmap index when this is enabled in GitLab. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake geo:git:housekeeping:gc -``` + ```shell + sudo gitlab-rake geo:git:housekeeping:gc + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake geo:git:housekeeping:gc RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake geo:git:housekeeping:gc RAILS_ENV=production + ``` ## Remove orphaned project registries Under certain conditions your project registry can contain obsolete records, you can remove them using the Rake task `geo:run_orphaned_project_registry_cleaner`: -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake geo:run_orphaned_project_registry_cleaner -``` + ```shell + sudo gitlab-rake geo:run_orphaned_project_registry_cleaner + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake geo:run_orphaned_project_registry_cleaner RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake geo:run_orphaned_project_registry_cleaner RAILS_ENV=production + ``` diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index f6c5f84c500..4a657b04bdc 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -15,17 +15,17 @@ The LDAP check Rake task tests the `bind_dn` and `password` credentials executed as part of the `gitlab:check` task, but can run independently using the command below. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:ldap:check -``` + ```shell + sudo gitlab-rake gitlab:ldap:check + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake gitlab:ldap:check RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake gitlab:ldap:check RAILS_ENV=production + ``` By default, the task returns a sample of 100 LDAP users. Change this limit by passing a number to the check task: @@ -47,17 +47,17 @@ If you'd like to change the frequency at which a group sync is performed, [adjust the cron schedule](../auth/ldap/ldap_synchronization.md#adjust-ldap-group-sync-schedule) instead. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:ldap:group_sync -``` + ```shell + sudo gitlab-rake gitlab:ldap:group_sync + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:ldap:group_sync -``` + ```shell + bundle exec rake gitlab:ldap:group_sync + ``` ## Rename a provider @@ -86,17 +86,17 @@ If you input an incorrect new provider, users cannot sign in. If this happens, run the task again with the incorrect provider as the `old_provider` and the correct provider as the `new_provider`. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:ldap:rename_provider[old_provider,new_provider] -``` + ```shell + sudo gitlab-rake gitlab:ldap:rename_provider[old_provider,new_provider] + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:ldap:rename_provider[old_provider,new_provider] RAILS_ENV=production -``` + ```shell + bundle exec rake gitlab:ldap:rename_provider[old_provider,new_provider] RAILS_ENV=production + ``` ### Example @@ -123,17 +123,17 @@ User identities were successfully updated If you do not specify an `old_provider` and `new_provider` the task prompts you for them: -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:ldap:rename_provider -``` + ```shell + sudo gitlab-rake gitlab:ldap:rename_provider + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:ldap:rename_provider RAILS_ENV=production -``` + ```shell + bundle exec rake gitlab:ldap:rename_provider RAILS_ENV=production + ``` **Example output:** @@ -158,17 +158,17 @@ The following Rake tasks are provided for updating the contents of the encrypted Show the contents of the current LDAP secrets. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:ldap:secret:show -``` + ```shell + sudo gitlab-rake gitlab:ldap:secret:show + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:ldap:secret:show RAILS_ENV=production -``` + ```shell + bundle exec rake gitlab:ldap:secret:show RAILS_ENV=production + ``` **Example output:** @@ -182,33 +182,33 @@ main: Opens the secret contents in your editor, and writes the resulting content to the encrypted secret file when you exit. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:ldap:secret:edit EDITOR=vim -``` + ```shell + sudo gitlab-rake gitlab:ldap:secret:edit EDITOR=vim + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:ldap:secret:edit RAILS_ENV=production EDITOR=vim -``` + ```shell + bundle exec rake gitlab:ldap:secret:edit RAILS_ENV=production EDITOR=vim + ``` ### Write raw secret Write new secret content by providing it on STDIN. -**Omnibus Installation** +- Linux package installations: -```shell -echo -e "main:\n password: '123'" | sudo gitlab-rake gitlab:ldap:secret:write -``` + ```shell + echo -e "main:\n password: '123'" | sudo gitlab-rake gitlab:ldap:secret:write + ``` -**Source Installation** +- Self-compiled installations: -```shell -echo -e "main:\n password: '123'" | bundle exec rake gitlab:ldap:secret:write RAILS_ENV=production -``` + ```shell + echo -e "main:\n password: '123'" | bundle exec rake gitlab:ldap:secret:write RAILS_ENV=production + ``` ### Secrets examples diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 50c4b004f9c..c2c6429ba8b 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -13,17 +13,17 @@ GitLab provides Rake tasks for general maintenance. This command gathers information about your GitLab installation and the system it runs on. These may be useful when asking for help or reporting issues. In a multi-node environment, run this command on nodes running GitLab Rails to avoid PostgreSQL socket errors. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:env:info -``` + ```shell + sudo gitlab-rake gitlab:env:info + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:env:info RAILS_ENV=production -``` + ```shell + bundle exec rake gitlab:env:info RAILS_ENV=production + ``` Example output: @@ -69,24 +69,24 @@ GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20501) in GitLab 12.6. > - Moved to GitLab Premium in 13.9. -This command shows information about your [GitLab license](../../user/admin_area/license.md) and +This command shows information about your [GitLab license](../../administration/license.md) and how many seats are used. It is only available on GitLab Enterprise installations: a license cannot be installed into GitLab Community Edition. These may be useful when raising tickets with Support, or for programmatically checking your license parameters. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:license:info -``` + ```shell + sudo gitlab-rake gitlab:license:info + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:license:info RAILS_ENV=production -``` + ```shell + bundle exec rake gitlab:license:info RAILS_ENV=production + ``` Example output: @@ -117,24 +117,24 @@ If you're running Geo, see also the [Geo Health check Rake task](../geo/replicat You may also have a look at our troubleshooting guides for: -- [GitLab](../troubleshooting/index.md) -- [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting) +- [GitLab](../troubleshooting/index.md). +- [Linux package installations](https://docs.gitlab.com/omnibus/index.html#troubleshooting). Additionally you should also [verify database values can be decrypted using the current secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). To run `gitlab:check`, run: -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:check -``` + ```shell + sudo gitlab-rake gitlab:check + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:check RAILS_ENV=production -``` + ```shell + bundle exec rake gitlab:check RAILS_ENV=production + ``` Use `SANITIZE=true` for `gitlab:check` if you want to omit project names from the output. @@ -190,18 +190,18 @@ for example, if after an upgrade you receive `Permission denied (publickey)` whe and find `404 Key Not Found` errors in [the `gitlab-shell.log` file](../logs/index.md#gitlab-shelllog). To rebuild `authorized_keys`, run: -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:shell:setup -``` + ```shell + sudo gitlab-rake gitlab:shell:setup + ``` -**Source Installation** +- Self-compiled installations: -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production -``` + ```shell + cd /home/git/gitlab + sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production + ``` Example output: @@ -216,18 +216,18 @@ Do you want to continue (yes/no)? yes If for some reason the dashboard displays the wrong information, you might want to clear Redis' cache. To do this, run: -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake cache:clear -``` + ```shell + sudo gitlab-rake cache:clear + ``` -**Source Installation** +- Self-compiled installations: -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production -``` + ```shell + cd /home/git/gitlab + sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production + ``` ## Precompile the assets @@ -235,24 +235,24 @@ Sometimes during version upgrades you might end up with some wrong CSS or missing some icons. In that case, try to precompile the assets again. This Rake task only applies to source installations. [Read more](../../update/package/index.md#missing-asset-files) -about troubleshooting this problem when running the Omnibus GitLab package. -The guidance for Omnibus GitLab might be applicable for Kubernetes and Docker Omnibus +about troubleshooting this problem when running the Linux package. +The guidance for Linux package might be applicable for Kubernetes and Docker deployments of GitLab, though in general, container-based installations don't have issues with missing assets. -**Source Installation** +- Self-compiled installations: -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production -``` + ```shell + cd /home/git/gitlab + sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production + ``` -For omnibus versions, the unoptimized assets (JavaScript, CSS) are frozen at -the release of upstream GitLab. The omnibus version includes optimized versions +For Linux package installations, the unoptimized assets (JavaScript, CSS) are frozen at +the release of upstream GitLab. The Linux package installation includes optimized versions of those assets. Unless you are modifying the JavaScript / CSS code on your production machine after installing the package, there should be no reason to redo `rake gitlab:assets:compile` on the production machine. If you suspect that assets -have been corrupted, you should reinstall the omnibus package. +have been corrupted, you should reinstall the Linux package. ## Check TCP connectivity to a remote site @@ -261,18 +261,18 @@ service on another machine (for example a PostgreSQL or web server) to troubleshoot proxy issues. A Rake task is included to help you with this. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:tcp_check[example.com,80] -``` + ```shell + sudo gitlab-rake gitlab:tcp_check[example.com,80] + ``` -**Source Installation** +- Self-compiled installations: -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:tcp_check[example.com,80] RAILS_ENV=production -``` + ```shell + cd /home/git/gitlab + sudo -u git -H bundle exec rake gitlab:tcp_check[example.com,80] RAILS_ENV=production + ``` ## Clear exclusive lease (DANGER) @@ -410,3 +410,16 @@ To re-import the metrics you can run: ```shell sudo gitlab-rake metrics:setup_common_metrics ``` + +## Troubleshooting + +### Advisory lock connection information + +After running the `db:migrate` Rake task, you may see output like the following: + +```shell +main: == [advisory_lock_connection] object_id: 173580, pg_backend_pid: 5532 +main: == [advisory_lock_connection] object_id: 173580, pg_backend_pid: 5532 +``` + +The messages returned are informational and can be ignored. diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index 1f9eb06f567..ba2a135cb01 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -20,14 +20,14 @@ Rake tasks are available for projects that have been created on Praefect storage Run this Rake task on the node that GitLab is installed and not on the node that Praefect is installed. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake "gitlab:praefect:replicas[project_id]" -``` + ```shell + sudo gitlab-rake "gitlab:praefect:replicas[project_id]" + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo -u git -H bundle exec rake "gitlab:praefect:replicas[project_id]" RAILS_ENV=production -``` + ```shell + sudo -u git -H bundle exec rake "gitlab:praefect:replicas[project_id]" RAILS_ENV=production + ``` diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 17a0eb46a30..36013803999 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -33,7 +33,7 @@ Parameters: bundle exec rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" ``` -If you're running Omnibus, run the following Rake task: +If you're running a Linux package installation, run the following Rake task: ```shell gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" @@ -99,10 +99,10 @@ To fix the issue: 1. Change the file permissions to `0400`. 1. Move the file to a public folder (for example `/tmp/`). -### `Name can contain only letters, digits, emojis ...` +### `Name can contain only letters, digits, emoji ...` ```plaintext -Name can contain only letters, digits, emojis, '_', '.', '+', dashes, or spaces. It must start with a letter, +Name can contain only letters, digits, emoji, '_', '.', '+', dashes, or spaces. It must start with a letter, digit, emoji, or '_', and Path can contain only letters, digits, '_', '-', or '.'. It cannot start with '-', end in '.git', or end in '.atom'. ``` diff --git a/doc/administration/raketasks/service_desk_email.md b/doc/administration/raketasks/service_desk_email.md index 1cbdec35171..9bf0846fef2 100644 --- a/doc/administration/raketasks/service_desk_email.md +++ b/doc/administration/raketasks/service_desk_email.md @@ -12,7 +12,7 @@ The following are Service Desk email-related Rake tasks. ## Secrets -GitLab can use [Service Desk email](../../user/project/service_desk.md#configure-a-custom-mailbox) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file. +GitLab can use [Service Desk email](../../user/project/service_desk.md#configure-service-desk-alias-email) secrets read from an encrypted file instead of storing them in plaintext in the file system. The following Rake tasks are provided for updating the contents of the encrypted file. ### Show secret diff --git a/doc/administration/raketasks/smtp.md b/doc/administration/raketasks/smtp.md index 5e9e3544902..3cb161345cb 100644 --- a/doc/administration/raketasks/smtp.md +++ b/doc/administration/raketasks/smtp.md @@ -18,17 +18,17 @@ GitLab can use SMTP configuration secrets to read from an encrypted file. The fo Show the contents of the current SMTP secrets. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:smtp:secret:show -``` + ```shell + sudo gitlab-rake gitlab:smtp:secret:show + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:smtp:secret:show RAILS_ENV=production -``` + ```shell + bundle exec rake gitlab:smtp:secret:show RAILS_ENV=production + ``` **Example output:** @@ -41,33 +41,33 @@ user_name: 'gitlab-inst' Opens the secret contents in your editor, and writes the resulting content to the encrypted secret file when you exit. -**Omnibus Installation** +- Linux package installations: -```shell -sudo gitlab-rake gitlab:smtp:secret:edit EDITOR=vim -``` + ```shell + sudo gitlab-rake gitlab:smtp:secret:edit EDITOR=vim + ``` -**Source Installation** +- Self-compiled installations: -```shell -bundle exec rake gitlab:smtp:secret:edit RAILS_ENV=production EDITOR=vim -``` + ```shell + bundle exec rake gitlab:smtp:secret:edit RAILS_ENV=production EDITOR=vim + ``` ### Write raw secret Write new secret content by providing it on `STDIN`. -**Omnibus Installation** +- Linux package installations: -```shell -echo -e "password: '123'" | sudo gitlab-rake gitlab:smtp:secret:write -``` + ```shell + echo -e "password: '123'" | sudo gitlab-rake gitlab:smtp:secret:write + ``` -**Source Installation** +- Self-compiled installations: -```shell -echo -e "password: '123'" | bundle exec rake gitlab:smtp:secret:write RAILS_ENV=production -``` + ```shell + echo -e "password: '123'" | bundle exec rake gitlab:smtp:secret:write RAILS_ENV=production + ``` ### Secrets examples diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 3664a79bf43..9e0a89fa7cb 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -20,7 +20,7 @@ available on legacy and hashed storage. To have a summary and then a list of projects and their attachments using legacy storage: -- **Omnibus installation** +- Linux package installations: ```shell # Projects @@ -32,7 +32,7 @@ To have a summary and then a list of projects and their attachments using legacy sudo gitlab-rake gitlab:storage:list_legacy_attachments ``` -- **Source installation** +- Self-compiled installations: ```shell # Projects @@ -48,7 +48,7 @@ To have a summary and then a list of projects and their attachments using legacy To have a summary and then a list of projects and their attachments using hashed storage: -- **Omnibus installation** +- Linux package installations: ```shell # Projects @@ -60,7 +60,7 @@ To have a summary and then a list of projects and their attachments using hashed sudo gitlab-rake gitlab:storage:list_hashed_attachments ``` -- **Source installation** +- Self-compiled installations: ```shell # Projects @@ -86,13 +86,13 @@ This task must be run on any machine that has Rails/Sidekiq configured, and the schedules all your existing projects and attachments associated with it to be migrated to the **Hashed** storage type: -- **Omnibus installation** +- Linux package installations: ```shell sudo gitlab-rake gitlab:storage:migrate_to_hashed ``` -- **Source installation** +- Self-compiled installations: ```shell sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production @@ -101,7 +101,7 @@ migrated to the **Hashed** storage type: If you have any existing integration, you may want to do a small rollout first, to validate. You can do so by specifying an ID range with the operation by using the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout -to project IDs 50 to 100 in an Omnibus GitLab installation: +to project IDs 50 to 100 in a Linux package installation: ```shell sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 @@ -139,13 +139,13 @@ been disabled. This task schedules all your existing projects and associated attachments to be rolled back to the legacy storage type. -- **Omnibus installation** +- Linux package installations: ```shell sudo gitlab-rake gitlab:storage:rollback_to_legacy ``` -- **Source installation** +- Self-compiled installations: ```shell sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production @@ -154,7 +154,7 @@ legacy storage type. If you have any existing integration, you may want to do a small rollback first, to validate. You can do so by specifying an ID range with the operation by using the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout -to project IDs 50 to 100 in an Omnibus GitLab installation: +to project IDs 50 to 100 in a Linux package installation: ```shell sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 @@ -244,9 +244,9 @@ If destroying the project generates a stack trace relating to encryption or the 1. [Verify your GitLab secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). 1. If the affected projects have secrets that cannot be decrypted it will be necessary to remove those specific secrets. - [Our documentation for dealing with lost secrets](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost) + [Our documentation for dealing with lost secrets](../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost) is for loss of all secrets, but it's possible for specific projects to be affected. For example, - to [reset specific runner registration tokens](../../raketasks/backup_restore.md#reset-runner-registration-tokens) + to [reset specific runner registration tokens](../../administration/backup_restore/backup_gitlab.md#reset-runner-registration-tokens) for a specific project ID: ```sql diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index 1f6e7fda082..a1826d3e5df 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -30,24 +30,24 @@ These [individual Rake tasks](#individual-rake-tasks) are described in the next To migrate all uploads from local storage to object storage, run: -**Omnibus Installation** +- Linux package installations: -```shell -gitlab-rake "gitlab:uploads:migrate:all" -``` + ```shell + gitlab-rake "gitlab:uploads:migrate:all" + ``` -**Source Installation** +- Self-compiled installations: -```shell -sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all -``` + ```shell + sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all + ``` You can optionally track progress and verify that all uploads migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. -- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. -- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. +- `sudo gitlab-rails dbconsole` for Linux package installations running GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Linux package installations running GitLab 14.2 and later. +- `sudo -u git -H psql -d gitlabhq_production` for self-compiled installations. Verify `objectstg` below (where `store=2`) has count of all artifacts: @@ -92,7 +92,9 @@ the default batch size: The following shows how to run `gitlab:uploads:migrate` for individual types of uploads. -**Omnibus Installation** +::Tabs + +:::TabTitle Linux package (Omnibus) ```shell # gitlab-rake gitlab:uploads:migrate[uploader_class, model_class, mount_point] @@ -120,7 +122,7 @@ gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" gitlab-rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action, :image_v432x230]" ``` -**Source Installation** +:::TabTitle Self-compiled (source) Use `RAILS_ENV=production` for every task. @@ -150,6 +152,8 @@ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeReque sudo -u git -H bundle exec rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action]" ``` +::EndTabs + ## Migrate to local storage If you need to disable [object storage](../../object_storage.md) for any reason, you must first @@ -172,17 +176,17 @@ Keep in mind the task name in this case is `gitlab:uploads:migrate_to_local`. To migrate uploads from object storage to local storage, run the following Rake task: -**Omnibus GitLab installation** +- Linux package installations: -```shell -gitlab-rake "gitlab:uploads:migrate_to_local:all" -``` + ```shell + gitlab-rake "gitlab:uploads:migrate_to_local:all" + ``` -**Source installation** +- Self-compiled installations: -```shell -sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all -``` + ```shell + sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all + ``` After running the Rake task, you can disable object storage by undoing the changes described in the instructions to [configure object storage](../../uploads.md#using-object-storage). diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 567a20a37f3..f86b4406a9e 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -15,8 +15,8 @@ can remove EXIF data from existing images that were uploaded to an earlier versi To run this Rake task, you need `exiftool` installed on your system. If you installed GitLab: -- Using the Omnibus package, you're all set. -- From source, make sure `exiftool` is installed: +- By using the Linux package, you're all set. +- By using the self-compiled installation, make sure `exiftool` is installed: ```shell # Debian/Ubuntu diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index 3842cf0846b..adc6f42271c 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -66,7 +66,7 @@ sudo gitlab-ctl start puma If you want to allow users to use the GitLab UI, ensure that the database is read-only: -1. Take a [GitLab backup](../raketasks/backup_restore.md) +1. Take a [GitLab backup](../administration/backup_restore/index.md) in case things don't go as expected. 1. Enter PostgreSQL on the console as an administrator user: diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 4a6f58a8d6a..1db5b82e7dc 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -659,7 +659,7 @@ persistence classes. | `shared_state` | Store session-related and other persistent data. | | `actioncable` | Pub/Sub queue backend for ActionCable. | | `trace_chunks` | Store [CI trace chunks](../job_logs.md#enable-or-disable-incremental-logging) data. | -| `rate_limiting` | Store [rate limiting](../../user/admin_area/settings/user_and_ip_rate_limits.md) state. | +| `rate_limiting` | Store [rate limiting](../settings/user_and_ip_rate_limits.md) state. | | `sessions` | Store [sessions](../../../ee/development/session.md#gitlabsession). | | `repository_cache` | Store cache data specific to repositories. | diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index c3cf7c599a3..e9a77d15a6c 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -232,7 +232,7 @@ spread connections equally in practice. ### Readiness checks Ensure the external load balancer only routes to working services with built -in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +in monitoring endpoints. The [readiness checks](../monitoring/health_check.md) all require [additional configuration](../monitoring/ip_allowlist.md) on the nodes being checked, otherwise, the external load balancer will not be able to connect. @@ -2265,7 +2265,7 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more ### Cluster topology The following tables and diagram detail the hybrid environment using the same formats -as the normal environment above. +as the typical environment above. First are the components that run in Kubernetes. These run across several node groups, although you can change the overall makeup as desired as long as the minimum CPU and Memory requirements are observed. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 37571ed5771..0e608a953b8 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -243,7 +243,7 @@ spread connections equally in practice. ### Readiness checks Ensure the external load balancer only routes to working services with built -in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +in monitoring endpoints. The [readiness checks](../monitoring/health_check.md) all require [additional configuration](../monitoring/ip_allowlist.md) on the nodes being checked, otherwise, the external load balancer will not be able to connect. @@ -2283,7 +2283,7 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more ### Cluster topology The following tables and diagram detail the hybrid environment using the same formats -as the normal environment above. +as the typical environment above. First are the components that run in Kubernetes. These run across several node groups, although you can change the overall makeup as desired as long as the minimum CPU and Memory requirements are observed. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 455b0fbafd1..e3e361db133 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -134,7 +134,7 @@ spread connections equally in practice. ### Readiness checks Ensure the external load balancer only routes to working services with built -in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +in monitoring endpoints. The [readiness checks](../monitoring/health_check.md) all require [additional configuration](../monitoring/ip_allowlist.md) on the nodes being checked, otherwise, the external load balancer will not be able to connect. @@ -963,7 +963,7 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more ### Cluster topology The following tables and diagram detail the hybrid environment using the same formats -as the normal environment above. +as the typical environment above. First are the components that run in Kubernetes. These run across several node groups, although you can change the overall makeup as desired as long as the minimum CPU and Memory requirements are observed. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 6a7d9864376..4b7563d9d8d 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -243,7 +243,7 @@ spread connections equally in practice. ### Readiness checks Ensure the external load balancer only routes to working services with built -in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +in monitoring endpoints. The [readiness checks](../monitoring/health_check.md) all require [additional configuration](../monitoring/ip_allowlist.md) on the nodes being checked, otherwise, the external load balancer will not be able to connect. @@ -2272,7 +2272,7 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more ### Cluster topology The following tables and diagram detail the hybrid environment using the same formats -as the normal environment above. +as the typical environment above. First are the components that run in Kubernetes. These run across several node groups, although you can change the overall makeup as desired as long as the minimum CPU and Memory requirements are observed. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 9a2c354f27c..6dc7f7affab 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -241,7 +241,7 @@ spread connections equally in practice. ### Readiness checks Ensure the external load balancer only routes to working services with built -in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +in monitoring endpoints. The [readiness checks](../monitoring/health_check.md) all require [additional configuration](../monitoring/ip_allowlist.md) on the nodes being checked, otherwise, the external load balancer will not be able to connect. @@ -2282,7 +2282,7 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more ### Cluster topology The following tables and diagram detail the hybrid environment using the same formats -as the normal environment above. +as the typical environment above. First are the components that run in Kubernetes. These run across several node groups, although you can change the overall makeup as desired as long as the minimum CPU and Memory requirements are observed. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index b0bc70aaf00..754a844df3f 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -240,7 +240,7 @@ spread connections equally in practice. ### Readiness checks Ensure the external load balancer only routes to working services with built -in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) +in monitoring endpoints. The [readiness checks](../monitoring/health_check.md) all require [additional configuration](../monitoring/ip_allowlist.md) on the nodes being checked, otherwise, the external load balancer will not be able to connect. @@ -2240,7 +2240,7 @@ Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more ### Cluster topology The following tables and diagram detail the hybrid environment using the same formats -as the normal environment above. +as the typical environment above. First are the components that run in Kubernetes. These run across several node groups, although you can change the overall makeup as desired as long as the minimum CPU and Memory requirements are observed. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 08aeb149454..8fc9fbce119 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -63,7 +63,7 @@ This section explains the designs you can choose from. It begins with the least ### Standalone (non-HA) -For environments serving 2,000 or fewer users, we generally recommend a standalone approach by deploying a non-highly available single or multi-node environment. With this approach, you can employ strategies such as [automated backups](../../raketasks/backup_gitlab.md#configuring-cron-to-make-daily-backups) for recovery to provide a good level of RPO / RTO while avoiding the complexities that come with HA. +For environments serving 2,000 or fewer users, we generally recommend a standalone approach by deploying a non-highly available single or multi-node environment. With this approach, you can employ strategies such as [automated backups](../../administration/backup_restore/backup_gitlab.md#configuring-cron-to-make-daily-backups) for recovery to provide a good level of RPO / RTO while avoiding the complexities that come with HA. *[RTO]: Recovery time objective *[RPO]: Recovery point objective @@ -312,7 +312,7 @@ Additionally, the following cloud provider services are validated and supported </tr> <tr> <td>Redis</td> - <td></td> + <td>🟢 <a href="https://cloud.google.com/memorystore" target="_blank" rel="noopener noreferrer">Memorystore</a></td> <td>🟢 <a href="https://aws.amazon.com/elasticache/" target="_blank" rel="noopener noreferrer">ElastiCache</a></td> <td></td> </tr> @@ -633,8 +633,8 @@ Most setups would only need vertical scaling, but there are some specific areas - Gitaly to Gitaly Cluster w/ Praefect - From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded. -Conversely, if you have robust metrics in place that show the environment is over-provisioned you can apply the same process for -scaling downloads. It's recommended to take an iterative approach when scaling downwards however to ensure there are no issues. +Conversely, if you have robust metrics in place that show the environment is over-provisioned, you can apply the same process for +scaling downwards. You should take an iterative approach when scaling downwards, however, to ensure there are no issues. ### How to monitor your environment diff --git a/doc/administration/reporting/git_abuse_rate_limit.md b/doc/administration/reporting/git_abuse_rate_limit.md new file mode 100644 index 00000000000..270a7cb4800 --- /dev/null +++ b/doc/administration/reporting/git_abuse_rate_limit.md @@ -0,0 +1,49 @@ +--- +stage: Anti-Abuse +group: Anti-Abuse +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Git abuse rate limit (administration) **(ULTIMATE SELF)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8066) in GitLab 15.2 [with a flag](../../administration/feature_flags.md) named `git_abuse_rate_limit_feature_flag`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/394996) in GitLab 15.11. Feature flag `git_abuse_rate_limit_feature_flag` removed. + +This is the administration documentation. For information about Git abuse rate limiting at the group level, see the [group-level documentation](../../user/group/reporting/git_abuse_rate_limit.md). + +Git abuse rate limiting is a feature to automatically [ban users](../../administration/moderate_users.md#ban-and-unban-users) who download, clone, or fork more than a specified number of repositories in any project in the instance in a given time frame. Banned users cannot sign in to the instance and cannot access any non-public group via HTTP or SSH. The rate limit also applies to users who authenticate with a [personal](../../user/profile/personal_access_tokens.md) or [group access token](../../user/group/settings/group_access_tokens.md). + +Git abuse rate limiting does not apply to instance administrators, [deploy tokens](../../user/project/deploy_tokens/index.md), or [deploy keys](../../user/project/deploy_keys/index.md). + +How GitLab determines a user's rate limit is under development. +GitLab team members can view more information in this confidential epic: +`https://gitlab.com/groups/gitlab-org/modelops/anti-abuse/-/epics/14`. + +## Configure Git abuse rate limiting + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Reporting**. +1. Expand **Git abuse rate limit**. +1. Update the Git abuse rate limit settings: + 1. Enter a number in the **Number of repositories** field, greater than or equal to `0` and less than or equal to `10,000`. This number specifies the maximum amount of unique repositories a user can download in the specified time period before they're banned. When set to `0`, Git abuse rate limiting is disabled. + 1. Enter a number in the **Reporting time period (seconds)** field, greater than or equal to `0` and less than or equal to `86,400` (10 days). This number specifies the time in seconds a user can download the maximum amount of repositories before they're banned. When set to `0`, Git abuse rate limiting is disabled. + 1. Optional. Exclude up to `100` users by adding them to the **Excluded users** field. Excluded users are not automatically banned. + 1. Add up to `100` users to the **Send notifications to** field. You must select at least one user. All application administrators are selected by default. + 1. Optional. Turn on the **Automatically ban users from this namespace when they exceed the specified limits** toggle to enable automatic banning. +1. Select **Save changes**. + +## Automatic ban notifications + +If automatic banning is disabled, a user is not banned automatically when they exceed the limit. However, notifications are still sent to the users listed under **Send notifications to**. You can use this setup to determine the correct values of the rate limit settings before enabling automatic banning. + +If automatic banning is enabled, an email notification is sent when a user is about to be banned, and the user is automatically banned from the GitLab instance. + +## Unban a user + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Banned** tab and search for the account you want to unban. +1. From the **User administration** dropdown list select **Unban user**. +1. On the confirmation dialog, select **Unban user**. diff --git a/doc/administration/reporting/ip_addr_restrictions.md b/doc/administration/reporting/ip_addr_restrictions.md new file mode 100644 index 00000000000..5b749c62c30 --- /dev/null +++ b/doc/administration/reporting/ip_addr_restrictions.md @@ -0,0 +1,33 @@ +--- +stage: Anti-Abuse +group: Anti-Abuse +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# IP address restrictions **(FREE SELF)** + +IP address restrictions help prevent malicious users hiding their activities behind multiple IP addresses. + +GitLab maintains a list of the unique IP addresses used by a user to make requests over a specified period. When the +specified limit is reached, any requests made by the user from a new IP address are rejected with a `403 Forbidden` error. + +IP addresses are cleared from the list when no further requests have been made by the user from the IP address in the specified time period. + +NOTE: +When a runner runs a CI/CD job as a particular user, the runner IP address is also stored against the user's list of +unique IP addresses. Therefore, the IP addresses per user limit should take into account the number of configured active runners. + +## Configure IP address restrictions + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Reporting**. +1. Expand **Spam and Anti-bot Protection**. +1. Update the IP address restrictions settings: + 1. Select the **Limit sign in from multiple IP addresses** checkbox to enable IP address restrictions. + 1. Enter a number in the **IP addresses per user** field, greater than or equal to `1`. This number specifies the + maximum number of unique IP addresses a user can access GitLab from in the specified time period before requests + from a new IP address are rejected. + 1. Enter a number in the **IP address expiration time** field, greater than or equal to `0`. This number specifies the + time in seconds an IP address counts towards the limit for a user, taken from the time the last request was made. +1. Select **Save changes**. diff --git a/doc/administration/reporting/spamcheck.md b/doc/administration/reporting/spamcheck.md new file mode 100644 index 00000000000..8e478729299 --- /dev/null +++ b/doc/administration/reporting/spamcheck.md @@ -0,0 +1,69 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Spamcheck anti-spam service **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6259) in GitLab 14.8. + +WARNING: +Spamcheck is available to all tiers, but only on instances using GitLab Enterprise Edition (EE). For [licensing reasons](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6259#note_726605397), it is not included in the GitLab Community Edition (CE) package. You can [migrate from CE to EE](../../update/package/convert_to_ee.md). + +[Spamcheck](https://gitlab.com/gitlab-org/spamcheck) is an anti-spam engine +developed by GitLab originally to combat rising amount of spam in GitLab.com, +and later made public to be used in self-managed GitLab instances. + +## Enable Spamcheck + +Spamcheck is only available for package-based installations: + +1. Edit `/etc/gitlab/gitlab.rb` and enable Spamcheck: + + ```ruby + spamcheck['enable'] = true + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Verify that the new services `spamcheck` and `spam-classifier` are + up and running: + + ```shell + sudo gitlab-ctl status + ``` + +## Configure GitLab to use Spamcheck + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Reporting**. +1. Expand **Spam and Anti-bot Protection**. +1. Update the Spam Check settings: + 1. Check the "Enable Spam Check via external API endpoint" checkbox. + 1. For **URL of the external Spam Check endpoint** use `grpc://localhost:8001`. + 1. Leave **Spam Check API key** blank. +1. Select **Save changes**. + +NOTE: +In single-node instances, Spamcheck runs over `localhost`, and hence is running +in an unauthenticated mode. If on multi-node instances where GitLab runs on one +server and Spamcheck runs on another server listening over a public endpoint, it +is recommended to enforce some sort of authentication using a reverse proxy in +front of the Spamcheck service that can be used along with an API key. One +example would be to use `JWT` authentication for this and specifying a bearer +token as the API key. +[Native authentication for Spamcheck is in the works](https://gitlab.com/gitlab-com/gl-security/engineering-and-research/automation-team/spam/spamcheck/-/issues/171). + +## Running Spamcheck over TLS + +Spamcheck service on its own cannot communicate directly over TLS with GitLab. +However, Spamcheck can be deployed behind a reverse proxy which performs TLS +termination. In such a scenario, GitLab can be made to communicate with +Spamcheck over TLS by specifying `tls://` scheme for the external Spamcheck URL +instead of `grpc://` in the Admin Area settings. diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 1a83a05c3dd..9967b623773 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -45,8 +45,8 @@ For more information on: WARNING: The following information is for configuring GitLab to directly access repositories. This -configuration option is deprecated in favor of using [Gitaly](gitaly/index.md). Gitaly issue -[1690](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) proposes to remove this configuration option. +configuration option is deprecated in favor of using [Gitaly](gitaly/index.md). +[Issue 403318](https://gitlab.com/gitlab-org/gitlab/-/issues/403318) proposes to remove this configuration option. To configure repository storage paths: @@ -73,7 +73,7 @@ For repository storage paths: ### Configure for backups -For [backups](../raketasks/backup_restore.md) to work correctly: +For [backups](../administration/backup_restore/index.md) to work correctly: - The repository storage path cannot be a mount point. - The GitLab user must have correct permissions for the parent directory of the path. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 3bd73b4df94..e1512038286 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -73,7 +73,7 @@ translate between the human-readable project name and the hashed storage path. Y Administrators can look up a project's hashed path from its name or ID using: -- The [Admin Area](../user/admin_area/index.md#administering-projects). +- The [Admin Area](../administration/admin_area.md#administering-projects). - A Rails console. To look up a project's hash path in the Admin Area: diff --git a/doc/administration/review_abuse_reports.md b/doc/administration/review_abuse_reports.md new file mode 100644 index 00000000000..e3891cbe68a --- /dev/null +++ b/doc/administration/review_abuse_reports.md @@ -0,0 +1,96 @@ +--- +stage: Anti-Abuse +group: Anti-Abuse +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference, howto +--- + +# Review abuse reports **(FREE SELF)** + +View and resolve abuse reports from GitLab users. + +GitLab administrators can view and [resolve](#resolving-abuse-reports) abuse +reports in the Admin Area. + +## Receive notification of abuse reports by email + +To receive notifications of new abuse reports by email: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Reporting**. +1. Expand the **Abuse reports** section. +1. Provide an email address and select **Save changes**. + +The notification email address can also be set and retrieved +[using the API](../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls). + +## Reporting abuse + +To find out more about reporting abuse, see +[abuse reports user documentation](../user/report_abuse.md). + +## Resolving abuse reports + +To access abuse reports: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Abuse Reports**. + +There are 3 ways to resolve an abuse report, with a button for each method: + +- Remove user & report. This: + - [Deletes the reported user](../user/profile/account/delete_account.md) from the + instance. + - Removes the abuse report from the list. +- [Block user](#blocking-users). +- Remove report. This: + - Removes the abuse report from the list. + - Removes access restrictions for the reported user. + +The following is an example of the **Abuse Reports** page: + +![abuse-reports-page-image](img/abuse_reports_page_v13_11.png) + +### Blocking users + +A blocked user cannot sign in or access any repositories, but all of their data +remains. + +Blocking a user: + +- Leaves them in the abuse report list. +- Changes the **Block user** button to a disabled **Already blocked** button. + +The user is notified with the following message: + +```plaintext +Your account has been blocked. If you believe this is in error, contact a staff member. +``` + +After blocking, you can still either: + +- Remove the user and report if necessary. +- Remove the report. + +The following is an example of a blocked user listed on the **Abuse Reports** +page: + +![abuse-report-blocked-user-image](img/abuse_report_blocked_user.png) + +NOTE: +Users can be [blocked](../api/users.md#block-user) and +[unblocked](../api/users.md#unblock-user) using the GitLab API. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/account_and_limit_settings.md b/doc/administration/settings/account_and_limit_settings.md new file mode 100644 index 00000000000..ca56a322237 --- /dev/null +++ b/doc/administration/settings/account_and_limit_settings.md @@ -0,0 +1,399 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +type: reference +--- + +# Account and limit settings **(FREE SELF)** + +## Default projects limit + +You can configure the default maximum number of projects new users can create in their +personal namespace. This limit affects only new user accounts created after you change +the setting. This setting is not retroactive for existing users, but you can separately edit +the [project limits for existing users](#projects-limit-for-a-user). + +To configure the maximum number of projects in personal namespaces for new users: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Increase or decrease that **Default projects limit** value. + +If you set **Default projects limit** to 0, users are not allowed to create projects +in their users personal namespace. However, projects can still be created in a group. + +### Projects limit for a user + +You can edit a specific user, and change the maximum number of projects this user +can create in their personal namespace: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview** > **Users**. +1. From the list of users, select a user. +1. Select **Edit**. +1. Increase or decrease the **Projects limit** value. + +## Max attachment size + +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/20061) from 10 MB to 100 MB in GitLab 15.7. + +The maximum file size for attachments in GitLab comments and replies is 100 MB. +To change the maximum attachment size: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Increase or decrease by changing the value in **Maximum attachment size (MB)**. + +If you choose a size larger than the configured value for the web server, +you may receive errors. Read the [troubleshooting section](#troubleshooting) for more +details. + +For GitLab.com repository size limits, read [accounts and limit settings](../../user/gitlab_com/index.md#account-and-limit-settings). + +## Max push size + +You can change the maximum push size for your instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Increase or decrease by changing the value in **Maximum push size (MB)**. + +For GitLab.com push size limits, read [accounts and limit settings](../../user/gitlab_com/index.md#account-and-limit-settings). + +NOTE: +When you [add files to a repository](../../user/project/repository/web_editor.md#create-a-file) +through the web UI, the maximum **attachment** size is the limiting factor, +because the [web server](../../development/architecture.md#components) +must receive the file before GitLab can generate the commit. +Use [Git LFS](../../topics/git/lfs/index.md) to add large files to a repository. + +## Max export size + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86124) in GitLab 15.0. + +To modify the maximum file size for exports in GitLab: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**, then expand **Account and limit**. +1. Increase or decrease by changing the value in **Maximum export size (MB)**. + +## Max import size + +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to unlimited in GitLab 13.8. + +To modify the maximum file size for imports in GitLab: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Increase or decrease by changing the value in **Maximum import size (MB)**. + +This setting applies only to repositories +[imported from a GitLab export file](../../user/project/settings/import_export.md#import-a-project-and-its-data). + +If you choose a size larger than the configured value for the web server, +you may receive errors. See the [troubleshooting section](#troubleshooting) for more +details. + +For GitLab.com repository size limits, read [accounts and limit settings](../../user/gitlab_com/index.md#account-and-limit-settings). + +## Personal access token prefix + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20968) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342327) in GitLab 14.5, a default prefix. + +You can specify a prefix for personal access tokens. You might use a prefix +to find tokens more quickly, or for use with automation tools. + +The default prefix is `glpat-` but administrators can change it. + +[Project access tokens](../../user/project/settings/project_access_tokens.md) and +[group access tokens](../../user/group/settings/group_access_tokens.md) also inherit this prefix. + +### Set a prefix + +To change the default global prefix: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Account and limit** section. +1. Fill in the **Personal Access Token prefix** field. +1. Select **Save changes**. + +You can also configure the prefix by using the +[settings API](../../api/settings.md). + +## Repository size limit **(PREMIUM SELF)** + +Repositories in your GitLab instance can grow quickly, especially if you are +using LFS. Their size can grow exponentially, rapidly consuming available storage. +To prevent this from happening, you can set a hard limit for your repositories' size. +This limit can be set globally, per group, or per project, with per project limits +taking the highest priority. + +There are numerous use cases where you might set up a limit for repository size. +For instance, consider the following workflow: + +1. Your team develops apps which require large files to be stored in + the application repository. +1. Although you have enabled [Git LFS](../../topics/git/lfs/index.md#git-large-file-storage-lfs) + to your project, your storage has grown significantly. +1. Before you exceed available storage, you set up a limit of 10 GB + per repository. + +NOTE: +For GitLab.com repository size limits, read [accounts and limit settings](../../user/gitlab_com/index.md#account-and-limit-settings). + +### How it works + +Only a GitLab administrator can set those limits. Setting the limit to `0` means +there are no restrictions. + +These settings can be found in: + +- Each project's settings: + 1. From the Project's homepage, navigate to **Settings > General**. + 1. Fill in the **Repository size limit (MB)** field in the **Naming, topics, avatar** section. + 1. Select **Save changes**. +- Each group's settings: + 1. From the Group's homepage, navigate to **Settings > General**. + 1. Fill in the **Repository size limit (MB)** field in the **Naming, visibility** section. + 1. Select **Save changes**. +- GitLab global settings: + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. + 1. Select **Settings > General**. + 1. Expand the **Account and limit** section. + 1. Fill in the **Size limit per repository (MB)** field. + 1. Select **Save changes**. + +The first push of a new project, including LFS objects, is checked for size. +If the sum of their sizes exceeds the maximum allowed repository size, the push +is rejected. + +NOTE: +The repository size limit includes repository files and LFS, but does not include artifacts, uploads, +wiki, packages, or snippets. The repository size limit applies to both private and public projects. + +For details on manually purging files, see [reducing the repository size using Git](../../user/project/repository/reducing_the_repo_size_using_git.md). + +## Session duration + +### Customize the default session duration + +You can change how long users can remain signed in without activity. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. The set duration is in **Session duration (minutes)**. + +If [Remember me](#turn-remember-me-on-or-off) is enabled, users' sessions can remain active for an indefinite period of time. + +For details, see [cookies used for sign-in](../../user/profile/index.md#cookies-used-for-sign-in). + +### Turn **Remember me** on or off + +> Ability to turn the **Remember me** setting on and off [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/369133) in GitLab 16.0. + +Users can select the **Remember me** checkbox on sign-in, and their session will remain active for an indefinite period of time when accessed from that specific browser. You can turn off this setting if you need sessions to expire for security or compliance purposes. Turning off this setting will ensure users' sessions expire after the number of minutes of inactivity set when you [customize your session duration](#customize-the-default-session-duration). + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Select or clear the **Remember me** checkbox to turn this setting on or off. + +### Customize session duration for Git Operations when 2FA is enabled **(PREMIUM SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/296669) in GitLab 13.9. +> - It's deployed behind a feature flag, disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../administration/feature_flags.md) named `two_factor_for_cli`. On GitLab.com, this feature is not available. This feature is not ready for production use. This feature flag also affects [2FA for Git over SSH operations](../../security/two_factor_authentication.md#2fa-for-git-over-ssh-operations). + +GitLab administrators can choose to customize the session duration (in minutes) for Git operations when 2FA is enabled. The default is 15 and this can be set to a value between 1 and 10080. + +To set a limit on how long these sessions are valid: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Account and limit** section. +1. Fill in the **Session duration for Git operations when 2FA is enabled (minutes)** field. +1. Select **Save changes**. + +## Limit the lifetime of SSH keys **(ULTIMATE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1007) in GitLab 14.6 [with a flag](../../administration/feature_flags.md) named `ff_limit_ssh_key_lifetime`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/346753) in GitLab 14.6. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/1007) in GitLab 14.7. [Feature flag `ff_limit_ssh_key_lifetime`](https://gitlab.com/gitlab-org/gitlab/-/issues/347408) removed. + +Users can optionally specify a lifetime for +[SSH keys](../../user/ssh.md). +This lifetime is not a requirement, and can be set to any arbitrary number of days. + +SSH keys are user credentials to access GitLab. +However, organizations with security requirements may want to enforce more protection by +requiring the regular rotation of these keys. + +### Set a lifetime + +Only a GitLab administrator can set a lifetime. Leaving it empty means +there are no restrictions. + +To set a lifetime on how long SSH keys are valid: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Account and limit** section. +1. Fill in the **Maximum allowable lifetime for SSH keys (days)** field. +1. Select **Save changes**. + +Once a lifetime for SSH keys is set, GitLab: + +- Requires users to set an expiration date that is no later than the allowed lifetime on new + SSH keys. +- Applies the lifetime restriction to existing SSH keys. Keys with no expiry or a lifetime + greater than the maximum immediately become invalid. + +NOTE: +When a user's SSH key becomes invalid they can delete and re-add the same key again. + +## Limit the lifetime of access tokens **(ULTIMATE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3649) in GitLab 12.6. + +Users can optionally specify a maximum lifetime in days for +access tokens, this includes [personal](../../user/profile/personal_access_tokens.md), +[group](../../user/group/settings/group_access_tokens.md), and [project](../../user/project/settings/project_access_tokens.md) access tokens. +This lifetime is not a requirement, and can be set to any value greater than 0 and less than or equal to 365. If this setting is left blank, the default allowable lifetime of access tokens is 365 days. + +Access tokens are the only tokens needed for programmatic access to GitLab. +However, organizations with security requirements may want to enforce more protection by +requiring the regular rotation of these tokens. + +### Set a lifetime + +Only a GitLab administrator can set a lifetime. Leaving it empty means +there are no restrictions. + +To set a lifetime on how long access tokens are valid: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Account and limit** section. +1. Fill in the **Maximum allowable lifetime for access tokens (days)** field. +1. Select **Save changes**. + +Once a lifetime for access tokens is set, GitLab: + +- Applies the lifetime for new personal access tokens, and require users to set an expiration date + and a date no later than the allowed lifetime. +- After three hours, revoke old tokens with no expiration date or with a lifetime longer than the + allowed lifetime. Three hours is given to allow administrators to change the allowed lifetime, + or remove it, before revocation takes place. + +## Disable user profile name changes **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24605) in GitLab 12.7. + +To maintain integrity of user details in [Audit Events](../../administration/audit_events.md), GitLab administrators can choose to disable a user's ability to change their profile name. + +To do this: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Select the **Prevent users from changing their profile name** checkbox. + +NOTE: +When this ability is disabled, GitLab administrators can still use the +[Admin Area](../../administration/admin_area.md#administering-users) or the +[API](../../api/users.md#user-modification) to update usernames. + +## Prevent new users from creating top-level groups + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/367754) in GitLab 15.5. + +By default, new users can create top-level groups. GitLab administrators can prevent new users from creating top-level groups: + +- In GitLab 15.5 and later, using either: + - The GitLab UI using the steps in this section. + - The [application setting API](../../api/settings.md#change-application-settings). +- In GitLab 15.4 and earlier, a [configuration file](../../administration/user_settings.md#use-configuration-files-to-prevent-new-users-from-creating-top-level-groups). + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Clear the **Allow new users to create top-level groups** checkbox. + +## Set profiles of new users to private by default + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231301) in GitLab 15.8. + +By default, newly created users have a public profile. GitLab administrators can set new users to have a private profile by default: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Select the **Make new users' profiles private by default** checkbox. + +## Prevent users from deleting their accounts **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26053) in GitLab 16.1 [with a flag](../../administration/feature_flags.md) named `deleting_account_disabled_for_users`. Enabled by default. + +By default, users can delete their own accounts. GitLab administrators can prevent +users from deleting their own accounts: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Account and limit**. +1. Clear the **Allows users to delete their own accounts** checkbox. + +## Troubleshooting + +### 413 Request Entity Too Large + +When attaching a file to a comment or reply in GitLab displays a `413 Request Entity Too Large` +error, the [max attachment size](#max-attachment-size) +is probably larger than the web server's allowed value. + +To increase the max attachment size to 200 MB in a +[Linux package](https://docs.gitlab.com/omnibus/) install, you may need to +add the line below to `/etc/gitlab/gitlab.rb` before increasing the max attachment size: + +```ruby +nginx['client_max_body_size'] = "200m" +``` + +### This repository has exceeded its size limit + +If you receive intermittent push errors in your [Rails exceptions log](../../administration/logs/index.md#exceptions_jsonlog), like this: + +```plaintext +Your push has been rejected, because this repository has exceeded its size limit. +``` + +[Housekeeping](../../administration/housekeeping.md) tasks may be causing your repository size to grow. +To resolve this problem, either of these options helps in the short- to middle-term: + +- Increase the [repository size limit](#repository-size-limit). +- [Reduce the repository size](../../user/project/repository/reducing_the_repo_size_using_git.md). diff --git a/doc/administration/settings/continuous_integration.md b/doc/administration/settings/continuous_integration.md new file mode 100644 index 00000000000..eaa240d4c96 --- /dev/null +++ b/doc/administration/settings/continuous_integration.md @@ -0,0 +1,429 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Continuous Integration and Deployment Admin Area settings **(FREE SELF)** + +The [Admin Area](index.md) has the instance settings for Auto DevOps, runners, and +job artifacts. + +## Auto DevOps + +To enable (or disable) [Auto DevOps](../../topics/autodevops/index.md) +for all projects: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Check (or uncheck to disable) the box that says **Default to Auto DevOps pipeline for all projects**. +1. Optionally, set up the [Auto DevOps base domain](../../topics/autodevops/requirements.md#auto-devops-base-domain) + which is used for Auto Deploy and Auto Review Apps. +1. Select **Save changes** for the changes to take effect. + +From now on, every existing project and newly created ones that don't have a +`.gitlab-ci.yml` use the Auto DevOps pipelines. + +If you want to disable it for a specific project, you can do so in +[its settings](../../topics/autodevops/index.md#enable-or-disable-auto-devops). + +## Enable shared runners for new projects + +You can set all new projects to have the instance's shared runners available by default. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand **Continuous Integration and Deployment**. +1. Select the **Enable shared runners for new projects** checkbox. + +Any time a new project is created, the shared runners are available. + +## Shared runners compute quota + +As an administrator you can set either a global or namespace-specific +limit on the number of [compute minutes](../../ci/pipelines/cicd_minutes.md) you can use. + +## Enable a project runner for multiple projects + +If you have already registered a [project runner](../../ci/runners/runners_scope.md#project-runners) +you can assign that runner to other projects. + +To enable a project runner for more than one project: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. From the left sidebar, select **CI/CD > Runners**. +1. Select the runner you want to edit. +1. In the upper-right corner, select **Edit** (**{pencil}**). +1. Under **Restrict projects for this runner**, search for a project. +1. To the left of the project, select **Enable**. +1. Repeat this process for each additional project. + +## Add a message for shared runners + +To display details about the instance's shared runners in all projects' +runner settings: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand **Continuous Integration and Deployment**. +1. Enter text, including Markdown if you want, in the **Shared runner details** field. For example: + + ![Shared runner details input](img/continuous_integration_shared_runner_details_input_v14_10.png) + +To view the rendered details: + +1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project or group. +1. Select **Settings > CI/CD**. +1. Expand **Runners**. + +![Shared runner details example](img/continuous_integration_shared_runner_details_v14_10.png) + +## Maximum artifacts size + +The maximum size of the [job artifacts](../../administration/job_artifacts.md) +can be set at: + +- The instance level. +- [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/21688), the project and group level. + +For the setting on GitLab.com, see [Artifacts maximum size](../../user/gitlab_com/index.md#gitlab-cicd). + +The value is in MB and the default is 100 MB per job. To change it at the: + +- Instance level: + + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. + 1. On the left sidebar, select **Settings > CI/CD > Continuous Integration and Deployment**. + 1. Change the value of **Maximum artifacts size (MB)**. + 1. Select **Save changes** for the changes to take effect. + +- Group level (this overrides the instance setting): + + 1. Go to the group's **Settings > CI/CD > General Pipelines**. + 1. Change the value of **Maximum artifacts size** (in MB). + 1. Select **Save changes** for the changes to take effect. + +- Project level (this overrides the instance and group settings): + + 1. Go to the project's **Settings > CI/CD > General Pipelines**. + 1. Change the value of **maximum artifacts size** (in MB). + 1. Select **Save changes** for the changes to take effect. + +NOTE: +The setting at all levels is only available to GitLab administrators. + +## Default artifacts expiration + +The default expiration time of the [job artifacts](../../administration/job_artifacts.md) +can be set in the Admin Area of your GitLab instance. The syntax of duration is +described in [`artifacts:expire_in`](../../ci/yaml/index.md#artifactsexpire_in) +and the default value is `30 days`. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Change the value of default expiration time. +1. Select **Save changes** for the changes to take effect. + +This setting is set per job and can be overridden in +[`.gitlab-ci.yml`](../../ci/yaml/index.md#artifactsexpire_in). +To disable the expiration, set it to `0`. The default unit is in seconds. + +NOTE: +Any changes to this setting applies to new artifacts only. The expiration time is not +be updated for artifacts created before this setting was changed. +The administrator may need to manually search for and expire previously-created +artifacts, as described in the [troubleshooting documentation](../../administration/job_artifacts.md#delete-job-artifacts-from-jobs-completed-before-a-specific-date). + +## Keep the latest artifacts for all jobs in the latest successful pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50889) in GitLab 13.9. + +When enabled (default), the artifacts of the most recent pipeline for each Git ref +([branches and tags](https://git-scm.com/book/en/v2/Git-Internals-Git-References)) +are locked against deletion and kept regardless of the expiry time. + +When disabled, the latest artifacts for any **new** successful or fixed pipelines +are allowed to expire. + +This setting takes precedence over the [project level setting](../../ci/jobs/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). +If disabled at the instance level, you cannot enable this per-project. + +To disable the setting: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand **Continuous Integration and Deployment**. +1. Clear the **Keep the latest artifacts for all jobs in the latest successful pipelines** checkbox. +1. Select **Save changes** + +When you disable the feature, the latest artifacts do not immediately expire. +A new pipeline must run before the latest artifacts can expire and be deleted. + +NOTE: +All application settings have a [customizable cache expiry interval](../../administration/application_settings_cache.md) which can delay the settings affect. + +## Archive jobs + +Archiving jobs is useful for reducing the CI/CD footprint on the system by removing some +of the capabilities of the jobs (metadata stored in the database needed to run the job), +but persisting the traces and artifacts for auditing purposes. + +To set the duration for which the jobs are considered as old and expired: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand the **Continuous Integration and Deployment** section. +1. Set the value of **Archive jobs**. +1. Select **Save changes** for the changes to take effect. + +After that time passes, the jobs are archived in the background and no longer able to be +retried. Make it empty to never expire jobs. It has to be no less than 1 day, +for example: <code>15 days</code>, <code>1 month</code>, <code>2 years</code>. + +For the value set for GitLab.com, see [Scheduled job archiving](../../user/gitlab_com/index.md#gitlab-cicd). + +## Protect CI/CD variables by default + +To set all new [CI/CD variables](../../ci/variables/index.md) as +[protected](../../ci/variables/index.md#protect-a-cicd-variable) by default: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Select **Protect CI/CD variables by default**. + +## Maximum includes + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207270) in GitLab 16.0. + +The maximum number of [includes](../../ci/yaml/includes.md) per pipeline can be set at the instance level. +The default is `150`. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Change the value of **Maximum includes**. +1. Select **Save changes** for the changes to take effect. + +## Default CI/CD configuration file + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18073) in GitLab 12.5. + +The default CI/CD configuration file and path for new projects can be set in the Admin Area +of your GitLab instance (`.gitlab-ci.yml` if not set): + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Input the new file and path in the **Default CI/CD configuration file** field. +1. Select **Save changes** for the changes to take effect. + +It is also possible to specify a [custom CI/CD configuration file for a specific project](../../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file). + +## Set CI/CD limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352175) in GitLab 14.10. + +You can configure some [CI/CD limits](../../administration/instance_limits.md#cicd-limits) +from the Admin Area: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand the **Continuous Integration and Deployment** section. +1. In the **CI/CD limits** section, you can set the following limits: + - **Maximum number of jobs in a single pipeline** + - **Total number of jobs in currently active pipelines** + - **Maximum number of active pipelines per project** + - **Maximum number of pipeline subscriptions to and from a project** + - **Maximum number of pipeline schedules** + - **Maximum number of DAG dependencies that a job can have** + - **Maximum number of runners registered per group** + - **Maximum number of runners registered per project** + - **Maximum number of downstream pipelines in a pipeline's hierarchy tree** + +## Enable or disable the pipeline suggestion banner + +By default, a banner displays in merge requests with no pipeline suggesting a +walkthrough on how to add one. + +![Suggest pipeline banner](img/suggest_pipeline_banner_v14_5.png) + +To enable or disable the banner: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Select or clear the **Enable pipeline suggestion banner** checkbox. +1. Select **Save changes**. + +## Required pipeline configuration **(ULTIMATE SELF)** + +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/352316) from GitLab Premium to GitLab Ultimate in 15.0. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/389467) in GitLab 15.9. + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/389467) in GitLab 15.9 +and is planned for removal in 17.0. Use [compliance pipelines](../../user/group/compliance_frameworks.md#compliance-pipelines) +instead. This change is a breaking change. + +You can set a [CI/CD template](../../ci/examples/index.md#cicd-templates) +as a required pipeline configuration for all projects on a GitLab instance. You can +use a template from: + +- The default CI/CD templates. +- A custom template stored in an [instance template repository](instance_template_repository.md). + + NOTE: + When you use a configuration defined in an instance template repository, + nested [`include:`](../../ci/yaml/index.md#include) keywords + (including `include:file`, `include:local`, `include:remote`, and `include:template`) + [do not work](https://gitlab.com/gitlab-org/gitlab/-/issues/35345). + +The project CI/CD configuration merges into the required pipeline configuration when +a pipeline runs. The merged configuration is the same as if the required pipeline configuration +added the project configuration with the [`include` keyword](../../ci/yaml/index.md#include). +To view a project's full merged configuration, [View full configuration](../../ci/pipeline_editor/index.md#view-full-configuration) +in the pipeline editor. + +To select a CI/CD template for the required pipeline configuration: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand the **Required pipeline configuration** section. +1. Select a CI/CD template from the dropdown list. +1. Select **Save changes**. + +## Package Registry configuration + +### Maven Forwarding **(PREMIUM SELF)** + +GitLab administrators can disable the forwarding of Maven requests to [Maven Central](https://search.maven.org/). + +To disable forwarding Maven requests: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand the **Package Registry** section. +1. Clear the checkbox **Forward Maven package requests to the Maven Registry if the packages are not found in the GitLab Package Registry**. +1. Select **Save changes**. + +### npm Forwarding **(PREMIUM SELF)** + +GitLab administrators can disable the forwarding of npm requests to [npmjs.com](https://www.npmjs.com/). + +To disable it: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand the **Package Registry** section. +1. Clear the checkbox **Forward npm package requests to the npm Registry if the packages are not found in the GitLab Package Registry**. +1. Select **Save changes**. + +### PyPI Forwarding **(PREMIUM SELF)** + +GitLab administrators can disable the forwarding of PyPI requests to [pypi.org](https://pypi.org/). + +To disable it: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand the **Package Registry** section. +1. Clear the checkbox **Forward PyPI package requests to the PyPI Registry if the packages are not found in the GitLab Package Registry**. +1. Select **Save changes**. + +### Package file size limits + +GitLab administrators can adjust the maximum allowed file size for each package type. + +To set the maximum file size: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand the **Package Registry** section. +1. Find the package type you would like to adjust. +1. Enter the maximum file size, in bytes. +1. Select **Save size limits**. + +## Restrict runner registration by all users in an instance + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22225) in GitLab 14.1. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/368008) in GitLab 15.5. + +GitLab administrators can adjust who is allowed to register runners, by showing and hiding areas of the UI. + +When the registration sections are hidden in the UI, members of the project or group must contact administrators to enable runner registration in the group or project. If you plan to prevent registration, ensure users have access to the runners they need to run jobs. + +By default, all members of a project and group are able to register runners. + +To restrict all users in an instance from registering runners: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand **Runners**. +1. In the **Runner registration** section, clear the **Members of the project can register runners** and + **Members of the group can register runners** checkboxes to remove runner registration from the UI. +1. Select **Save changes**. + +NOTE: +After you disable runner registration by members of a project, the registration +token automatically rotates. The token is no longer valid and you must +use the new registration token for the project. + +## Restrict runner registration by all members in a group + +Prerequisites: + +- Runner registration must be enabled for [all users in the instance](#restrict-runner-registration-by-all-users-in-an-instance). + +GitLab administrators can adjust group permissions to restrict runner registration by group members. + +To restrict runner registration by members in a specific group: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Groups** and find your group. +1. Select **Edit**. +1. Clear the **New group runners can be registered** checkbox if you want to disable runner registration by all members in the group. If the setting is read-only, you must enable runner registration for the [instance](#restrict-runner-registration-by-all-users-in-an-instance). +1. Select **Save changes**. + +## Disable runner version management + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/114041) in GitLab 15.10. + +By default, GitLab instances periodically fetch official runner version data from GitLab.com to [determine whether the runners need upgrades](../../ci/runners/runners_scope.md#determine-which-runners-need-to-be-upgraded). + +To disable your instance fetching this data: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > CI/CD**. +1. Expand **Runners**. +1. In the **Runner version management** section, clear the **Fetch GitLab Runner release version data from GitLab.com** checkbox. +1. Select **Save changes**. + +## Troubleshooting + +### 413 Request Entity Too Large + +When build jobs fail with the following error, +increase the [maximum artifacts size](#maximum-artifacts-size). + +```plaintext +Uploading artifacts as "archive" to coordinator... too large archive <job-id> responseStatus=413 Request Entity Too Large status=413" at end of a build job on pipeline when trying to store artifacts to <object-storage>. +``` diff --git a/doc/administration/settings/deprecated_api_rate_limits.md b/doc/administration/settings/deprecated_api_rate_limits.md new file mode 100644 index 00000000000..f8db0810af5 --- /dev/null +++ b/doc/administration/settings/deprecated_api_rate_limits.md @@ -0,0 +1,54 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Deprecated API rate limits **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68645) in GitLab 14.4. + +Deprecated API endpoints are those which have been replaced with alternative +functionality, but cannot be removed without breaking backward compatibility. +Setting a restrictive rate limit on these endpoints can encourage users to +switch to the alternatives. + +## Deprecated API endpoints + +Not all deprecated API endpoints are included in this rate limit - just those +that might have a performance impact: + +- [`GET /groups/:id`](../../api/groups.md#details-of-a-group) **without** the `with_projects=0` query parameter. + +## Define Deprecated API rate limits + +Rate limits for deprecated API endpoints are disabled by default. When enabled, they supersede +the general user and IP rate limits for requests to deprecated endpoints. You can keep any general user +and IP rate limits already in place, and increase or decrease the rate limits +for deprecated API endpoints. No other new features are provided by this override. + +Prerequisite: + +- You must have administrator access to the instance. + +To override the general user and IP rate limits for requests to deprecated API endpoints: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Deprecated API Rate Limits**. +1. Select the checkboxes for the types of rate limits you want to enable: + - **Unauthenticated API request rate limit** + - **Authenticated API request rate limit** +1. If you selected **unauthenticated**: + 1. Select the **Maximum unauthenticated API requests per period per IP**. + 1. Select the **Unauthenticated API rate limit period in seconds**. +1. If you selected **authenticated**: + 1. Select the **Maximum authenticated API requests per period per user**. + 1. Select the **Authenticated API rate limit period in seconds**. + +## Related topics + +- [Rate limits](../../security/rate_limits.md) +- [User and IP rate limits](../settings/user_and_ip_rate_limits.md) diff --git a/doc/administration/settings/email.md b/doc/administration/settings/email.md new file mode 100644 index 00000000000..e4972897aab --- /dev/null +++ b/doc/administration/settings/email.md @@ -0,0 +1,125 @@ +--- +type: reference +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Email **(FREE SELF)** + +You can customize some of the content in emails sent from your GitLab instance. + +## Custom logo + +The logo in the header of some emails can be customized, see the [logo customization section](../../administration/appearance.md#navigation-bar). + +## Include author name in email notification email body **(PREMIUM SELF)** + +By default, GitLab overrides the email address in notification emails with the email address +of the issue, merge request, or comment author. Enable this setting to include the author's email +address in the body of the email instead. + +To include the author's email address in the email body: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Email**. +1. Select the **Include author name in email notification email body** checkbox. +1. Select **Save changes**. + +## Enable multipart email **(PREMIUM SELF)** + +GitLab can send email in multipart format (HTML and plain text) or plain text only. + +To enable multipart email: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Email**. +1. Select **Enable multipart email**. +1. Select **Save changes**. + +## Custom hostname for private commit emails **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22560) in GitLab 11.5. + +This configuration option sets the email hostname for [private commit emails](../../user/profile/index.md#use-an-automatically-generated-private-commit-email). + By default it is set to `users.noreply.YOUR_CONFIGURED_HOSTNAME`. + +To change the hostname used in private commit emails: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Email**. +1. Enter the desired hostname in the **Custom hostname (for private commit emails)** field. +1. Select **Save changes**. + +NOTE: +After the hostname is configured, every private commit email using the previous hostname is not +recognized by GitLab. This can directly conflict with certain [Push rules](../../user/project/repository/push_rules.md) such as +`Check whether author is a GitLab user` and `Check whether committer is the current authenticated user`. + +## Custom additional text **(PREMIUM SELF)** + +You can add additional text at the bottom of any email that GitLab sends. This additional text +can be used for legal, auditing, or compliance reasons, for example. + +To add additional text to emails: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Email**. +1. Enter your text in the **Additional text** field. +1. Select **Save changes**. + +## User deactivation emails + +GitLab sends email notifications to users when their account has been deactivated. + +To disable these notifications: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Email**. +1. Clear the **Enable user deactivation emails** checkbox. +1. Select **Save changes**. + +### Custom additional text in deactivation emails **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355964) in GitLab 15.9 [with a flag](../../administration/feature_flags.md) named `deactivation_email_additional_text`. Disabled by default. +> - [Enabled on self-managed and GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/111882) in GitLab 15.9. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an +administrator to [disable the feature flag](../../administration/feature_flags.md) named +`deactivation_email_additional_text`. + +You can add additional text at the bottom of the email that GitLab sends to users when their account +is deactivated. This email text is separate from the [custom additional text](#custom-additional-text) +setting. + +To add additional text to deactivation emails: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Email**. +1. Enter your text in the **Additional text for deactivation email** field. +1. Select **Save changes**. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/external_authorization.md b/doc/administration/settings/external_authorization.md new file mode 100644 index 00000000000..45887fdccb8 --- /dev/null +++ b/doc/administration/settings/external_authorization.md @@ -0,0 +1,144 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# External authorization control **(FREE SELF)** + +> [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27056) from GitLab Premium to GitLab Free in 11.10. + +In highly controlled environments, it may be necessary for access policy to be +controlled by an external service that permits access based on project +classification and user access. GitLab provides a way to check project +authorization with your own defined service. + +After the external service is configured and enabled, when a project is +accessed, a request is made to the external service with the user information +and project classification label assigned to the project. When the service +replies with a known response, the result is cached for six hours. + +If the external authorization is enabled, GitLab further blocks pages and +functionality that render cross-project data. That includes: + +- Most pages under Dashboard (Activity, Milestones, Snippets, Assigned merge + requests, Assigned issues, To-Do List). +- Under a specific group (Activity, Contribution analytics, Issues, Issue boards, + Labels, Milestones, Merge requests). +- Global and Group search are disabled. + +This is to prevent performing too many requests at once to the external +authorization service. + +Whenever access is granted or denied this is logged in a log file called +`external-policy-access-control.log`. Read more about the logs GitLab keeps in +the [Linux package documentation](https://docs.gitlab.com/omnibus/settings/logs.html). + +When using TLS Authentication with a self signed certificate, the CA certificate +needs to be trusted by the OpenSSL installation. When using GitLab installed +using the Linux package, learn to install a custom CA in the +[Linux package documentation](https://docs.gitlab.com/omnibus/settings/ssl/index.html). +Alternatively, learn where to install custom certificates by using +`openssl version -d`. + +## Configuration + +The external authorization service can be enabled by an administrator: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **External authorization**. +1. Complete the fields. +1. Select **Save changes**. + +### Allow external authorization with deploy tokens and deploy keys + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/386656) in GitLab 15.9. +> - Deploy tokens no longer being able to access container or package registries [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/387721) in GitLab 16.0. + +You can set your instance to allow external authorization for Git operations with +[deploy tokens](../../user/project/deploy_tokens/index.md) or [deploy keys](../../user/project/deploy_keys/index.md). + +Prerequisites: + +- You must be using classification labels without a service URL for external authorization. + +To allow authorization with deploy tokens and keys: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **External authorization**, and: + - Leave the service URL field empty. + - Select **Allow deploy tokens and deploy keys to be used with external authorization**. +1. Select **Save changes**. + +WARNING: +If you enable external authorization, deploy tokens cannot access container or package registries. If you use deploy tokens to access these registries, this measure breaks this use of these tokens. Disable external authorization to use tokens with container or package registries. + +## How it works + +When GitLab requests access, it sends a JSON POST request to the external +service with this body: + +```json +{ + "user_identifier": "jane@acme.org", + "project_classification_label": "project-label", + "user_ldap_dn": "CN=Jane Doe,CN=admin,DC=acme", + "identities": [ + { "provider": "ldap", "extern_uid": "CN=Jane Doe,CN=admin,DC=acme" }, + { "provider": "bitbucket", "extern_uid": "2435223452345" } + ] +} +``` + +The `user_ldap_dn` is optional and is only sent when the user is signed in +through LDAP. + +`identities` contains the details of all the identities associated with the +user. This is an empty array if there are no identities associated with the +user. + +When the external authorization service responds with a status code 200, the +user is granted access. When the external service responds with a status code +401 or 403, the user is denied access. In any case, the request is cached for +six hours. + +When denying access, a `reason` can be optionally specified in the JSON body: + +```json +{ + "reason": "You are not allowed access to this project." +} +``` + +Any other status code than 200, 401 or 403 also deny access to the user, but the +response isn't cached. + +If the service times out (after 500 ms), a message "External Policy Server did +not respond" is displayed. + +## Classification labels + +You can use your own classification label in the project's +**Settings > General > General project settings** page in the "Classification +label" box. When no classification label is specified on a project, the default +label defined in the [global settings](#configuration) is used. + +On all project pages, in the upper-right corner, the label appears. + +![classification label on project page](img/classification_label_on_project_page_v14_8.png) + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/files_api_rate_limits.md b/doc/administration/settings/files_api_rate_limits.md new file mode 100644 index 00000000000..cb5442c957f --- /dev/null +++ b/doc/administration/settings/files_api_rate_limits.md @@ -0,0 +1,51 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Rate limits on Repository files API **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75918) in GitLab 14.6. [Feature flag `files_api_throttling`](https://gitlab.com/gitlab-org/gitlab/-/issues/338903) removed. + +The [Repository files API](../../api/repository_files.md) enables you to +fetch, create, update, and delete files in your repository. To improve the security +and durability of your web application, you can enforce +[rate limits](../../security/rate_limits.md) on this API. Any rate limits you +create for the Files API override the [general user and IP rate limits](user_and_ip_rate_limits.md). + +## Define Files API rate limits + +Rate limits for the Files API are disabled by default. When enabled, they supersede +the general user and IP rate limits for requests to the +[Repository files API](../../api/repository_files.md). You can keep any general user +and IP rate limits already in place, and increase or decrease the rate limits +for the Files API. No other new features are provided by this override. + +Prerequisite: + +- You must have administrator access to the instance. + +To override the general user and IP rate limits for requests to the Repository files API: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Files API Rate Limits**. +1. Select the checkboxes for the types of rate limits you want to enable: + - **Unauthenticated API request rate limit** + - **Authenticated API request rate limit** +1. If you selected **unauthenticated**: + 1. Select the **Max unauthenticated API requests per period per IP**. + 1. Select the **Unauthenticated API rate limit period in seconds**. +1. If you selected **authenticated**: + 1. Select the **Max authenticated API requests per period per user**. + 1. Select the **Authenticated API rate limit period in seconds**. + +## Related topics + +- [Rate limits](../../security/rate_limits.md) +- [Repository files API](../../api/repository_files.md) +- [User and IP rate limits](user_and_ip_rate_limits.md) diff --git a/doc/administration/settings/floc.md b/doc/administration/settings/floc.md new file mode 100644 index 00000000000..6bd5a6dfed4 --- /dev/null +++ b/doc/administration/settings/floc.md @@ -0,0 +1,42 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Federated Learning of Cohorts (FLoC) **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60933) in GitLab 13.12. + +Federated Learning of Cohorts (FLoC) is a new feature of the Chrome browser. +It works by categorizing users into different cohorts, so that +advertisers can use this data to uniquely target and track users. For more +information, see the [FLoC repository](https://github.com/WICG/floc). + +To avoid users being tracked and categorized in any GitLab instance, FLoC is +disabled by default by sending the following header: + +```plaintext +Permissions-Policy: interest-cohort=() +``` + +To enable it: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Federated Learning of Cohorts (FLoC)**. +1. Select the **Participate in FLoC** checkbox. +1. Select **Save changes**. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/git_lfs_rate_limits.md b/doc/administration/settings/git_lfs_rate_limits.md new file mode 100644 index 00000000000..cb2cc80e397 --- /dev/null +++ b/doc/administration/settings/git_lfs_rate_limits.md @@ -0,0 +1,36 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Rate limits on Git LFS **(FREE SELF)** + +[Git LFS (Large File Storage)](../../topics/git/lfs/index.md) is a Git extension +for handling large files. If you use Git LFS in your repository, common Git operations +can generate many Git LFS requests. You can enforce +[general user and IP rate limits](../settings/user_and_ip_rate_limits.md), but you can also +override the general setting to enforce additional limits on Git LFS requests. This +override can improve the security and durability of your web application. Aside from +precedence, this configuration provides the same features as the general user and IP +rate limits. + +## Configure Git LFS rate limits + +Git LFS rate limits are disabled by default. If enabled and configured, these limits +supersede the [general user and IP rate limits](../settings/user_and_ip_rate_limits.md): + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Git LFS Rate Limits**. +1. Select **Enable authenticated Git LFS request rate limit**. +1. Enter a value for **Max authenticated Git LFS requests per period per user**. +1. Enter a value for **Authenticated Git LFS rate limit period in seconds**. +1. Select **Save changes**. + +## Related topics + +- [Rate limiting](../../security/rate_limits.md) +- [User and IP rate limits](../settings/user_and_ip_rate_limits.md) diff --git a/doc/administration/settings/gitaly_timeouts.md b/doc/administration/settings/gitaly_timeouts.md new file mode 100644 index 00000000000..49dc7763cd0 --- /dev/null +++ b/doc/administration/settings/gitaly_timeouts.md @@ -0,0 +1,27 @@ +--- +stage: Systems +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Gitaly timeouts **(FREE SELF)** + +[Gitaly](../gitaly/index.md) timeouts are configurable. The timeouts can be +configured to make sure that long-running Gitaly calls don't needlessly take up resources. + +To access Gitaly timeout settings: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand the **Gitaly timeouts** section. + +## Available timeouts + +The following timeouts are available. + +| Timeout | Default | Description | +|:--------|:-----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Default | 55 seconds | Timeout for most Gitaly calls (not enforced for `git` `fetch` and `push` operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made within a web request cannot exceed the entire request timeout. It should be shorter than the [worker timeout](../operations/puma.md#change-the-worker-timeout) that can be configured for [Puma](../../install/requirements.md#puma-settings). If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker. | +| Fast | 10 seconds | Timeout for fast Gitaly operations used within requests, sometimes multiple times. For example, checking if a repository exists on disk. If fast operations exceed this threshold, there may be a problem with a storage shard. Failing fast can help maintain the stability of the GitLab instance. | +| Medium | 30 seconds | Timeout for Gitaly operations that should be fast (possibly within requests) but preferably not used multiple times within a request. For example, loading blobs. Timeout that should be set between Default and Fast. | diff --git a/doc/administration/settings/help_page.md b/doc/administration/settings/help_page.md new file mode 100644 index 00000000000..46c2c395102 --- /dev/null +++ b/doc/administration/settings/help_page.md @@ -0,0 +1,111 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto +--- + +# Customize the Help and sign-in page messages **(FREE SELF)** + +In large organizations, it is useful to have information about who to contact or where +to go for help. You can customize and display this information on the GitLab `/help` page and on +the GitLab sign-in page. + +## Add a help message to the Help page + +You can add a help message, which is shown at the top of the GitLab `/help` page (for example, +<https://gitlab.com/help>): + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Sign-in and Help page**. +1. In **Additional text to show on the Help page**, enter the information you want to display on `/help`. +1. Select **Save changes**. + +You can now see the message on `/help`. + +NOTE: +By default, `/help` is visible to unauthenticated users. However, if the +[**Public** visibility level](visibility_and_access_controls.md#restrict-visibility-levels) +is restricted, `/help` is visible only to authenticated users. + +## Add a help message to the sign-in page + +You can add a help message, which is shown on the GitLab sign-in page. The message appears on the sign-in page: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Sign-in and Help page**. +1. In **Additional text to show on the sign-in page**, enter the information you want to + display on the sign-in page. +1. Select **Save changes**. + +You can now see the message on the sign-in page. + +## Hide marketing-related entries from the Help page + +GitLab marketing-related entries are occasionally shown on the Help page. To hide these entries: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Sign-in and Help page**. +1. Select the **Hide marketing-related entries from the Help page** checkbox. +1. Select **Save changes**. + +## Set a custom Support page URL + +You can specify a custom URL to which users are directed when they: + +- Select **Support** from the Help dropdown list. +- Select **See our website for help** on the Help page. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Sign-in and Help page**. +1. In the **Support page URL** field, enter the URL. +1. Select **Save changes**. + +## Redirect `/help` pages + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43157) in GitLab 13.5. +> - [Feature flag `help_page_documentation_redirect`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71737) removed in GitLab 14.4. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71737) in GitLab 14.4. + +You can redirect all `/help` links to a destination that meets the [necessary requirements](#destination-requirements). + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Sign-in and Help page**. +1. In the **Documentation pages URL** field, enter the URL. +1. Select **Save changes**. + +If the "Documentation pages URL" field is empty, the GitLab instance displays a basic version of the documentation sourced from the [`doc` directory](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) of GitLab. + +### Destination requirements + +When redirecting `/help`, GitLab: + +- Redirects requests to the specified URL. +- Appends `ee` and the documentation path, which includes the version number, to the URL. +- Appends `.html` to the URL, and removes `.md` if necessary. + +For example, if the URL is set to `https://docs.gitlab.com`, requests for +`/help/administration/settings/help_page.md` redirect to: +`https://docs.gitlab.com/${VERSION}/ee/administration/settings/help_page.html`. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/img/bulk_push_event_v12_4.png b/doc/administration/settings/img/bulk_push_event_v12_4.png Binary files differnew file mode 100644 index 00000000000..114e87a61f1 --- /dev/null +++ b/doc/administration/settings/img/bulk_push_event_v12_4.png diff --git a/doc/administration/settings/img/classification_label_on_project_page_v14_8.png b/doc/administration/settings/img/classification_label_on_project_page_v14_8.png Binary files differnew file mode 100644 index 00000000000..4bd2e7d389b --- /dev/null +++ b/doc/administration/settings/img/classification_label_on_project_page_v14_8.png diff --git a/doc/administration/settings/img/continuous_integration_shared_runner_details_input_v14_10.png b/doc/administration/settings/img/continuous_integration_shared_runner_details_input_v14_10.png Binary files differnew file mode 100644 index 00000000000..08451f36962 --- /dev/null +++ b/doc/administration/settings/img/continuous_integration_shared_runner_details_input_v14_10.png diff --git a/doc/administration/settings/img/continuous_integration_shared_runner_details_v14_10.png b/doc/administration/settings/img/continuous_integration_shared_runner_details_v14_10.png Binary files differnew file mode 100644 index 00000000000..64bd9cf6911 --- /dev/null +++ b/doc/administration/settings/img/continuous_integration_shared_runner_details_v14_10.png diff --git a/doc/administration/settings/img/custom_git_clone_url_for_https_v12_4.png b/doc/administration/settings/img/custom_git_clone_url_for_https_v12_4.png Binary files differnew file mode 100644 index 00000000000..22cdd15cc0c --- /dev/null +++ b/doc/administration/settings/img/custom_git_clone_url_for_https_v12_4.png diff --git a/doc/administration/settings/img/domain_denylist_v14_1.png b/doc/administration/settings/img/domain_denylist_v14_1.png Binary files differnew file mode 100644 index 00000000000..e27997fc2c2 --- /dev/null +++ b/doc/administration/settings/img/domain_denylist_v14_1.png diff --git a/doc/administration/settings/img/email_notification_for_unknown_sign_ins_v13_2.png b/doc/administration/settings/img/email_notification_for_unknown_sign_ins_v13_2.png Binary files differnew file mode 100644 index 00000000000..fdcc542c4d7 --- /dev/null +++ b/doc/administration/settings/img/email_notification_for_unknown_sign_ins_v13_2.png diff --git a/doc/administration/settings/img/file_template_user_dropdown.png b/doc/administration/settings/img/file_template_user_dropdown.png Binary files differnew file mode 100644 index 00000000000..8c9eb49f6c9 --- /dev/null +++ b/doc/administration/settings/img/file_template_user_dropdown.png diff --git a/doc/administration/settings/img/mirror_settings_v15_7.png b/doc/administration/settings/img/mirror_settings_v15_7.png Binary files differnew file mode 100644 index 00000000000..5da41dbeceb --- /dev/null +++ b/doc/administration/settings/img/mirror_settings_v15_7.png diff --git a/doc/administration/settings/img/protected_paths.png b/doc/administration/settings/img/protected_paths.png Binary files differnew file mode 100644 index 00000000000..2233a71a139 --- /dev/null +++ b/doc/administration/settings/img/protected_paths.png diff --git a/doc/administration/settings/img/push_event_activities_limit_v12_4.png b/doc/administration/settings/img/push_event_activities_limit_v12_4.png Binary files differnew file mode 100644 index 00000000000..ea618ad4c50 --- /dev/null +++ b/doc/administration/settings/img/push_event_activities_limit_v12_4.png diff --git a/doc/administration/settings/img/rate_limit_on_issues_creation_v14_2.png b/doc/administration/settings/img/rate_limit_on_issues_creation_v14_2.png Binary files differnew file mode 100644 index 00000000000..1a0a7548a91 --- /dev/null +++ b/doc/administration/settings/img/rate_limit_on_issues_creation_v14_2.png diff --git a/doc/administration/settings/img/rate_limits_on_raw_endpoints.png b/doc/administration/settings/img/rate_limits_on_raw_endpoints.png Binary files differnew file mode 100644 index 00000000000..c59f67df1ce --- /dev/null +++ b/doc/administration/settings/img/rate_limits_on_raw_endpoints.png diff --git a/doc/administration/settings/img/restricted_url.png b/doc/administration/settings/img/restricted_url.png Binary files differnew file mode 100644 index 00000000000..c71abf0a226 --- /dev/null +++ b/doc/administration/settings/img/restricted_url.png diff --git a/doc/administration/settings/img/sign_up_terms.png b/doc/administration/settings/img/sign_up_terms.png Binary files differnew file mode 100644 index 00000000000..4cac9d426c9 --- /dev/null +++ b/doc/administration/settings/img/sign_up_terms.png diff --git a/doc/administration/settings/img/suggest_pipeline_banner_v14_5.png b/doc/administration/settings/img/suggest_pipeline_banner_v14_5.png Binary files differnew file mode 100644 index 00000000000..0d9bfa4a173 --- /dev/null +++ b/doc/administration/settings/img/suggest_pipeline_banner_v14_5.png diff --git a/doc/administration/settings/img/two_factor_grace_period.png b/doc/administration/settings/img/two_factor_grace_period.png Binary files differnew file mode 100644 index 00000000000..e7fb52969aa --- /dev/null +++ b/doc/administration/settings/img/two_factor_grace_period.png diff --git a/doc/administration/settings/img/update-available.png b/doc/administration/settings/img/update-available.png Binary files differnew file mode 100644 index 00000000000..9887e06c7dc --- /dev/null +++ b/doc/administration/settings/img/update-available.png diff --git a/doc/administration/settings/import_export_rate_limits.md b/doc/administration/settings/import_export_rate_limits.md new file mode 100644 index 00000000000..99385c77cdf --- /dev/null +++ b/doc/administration/settings/import_export_rate_limits.md @@ -0,0 +1,31 @@ +--- +stage: Manage +group: Import and Integrate +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limits for imports and exports of project and groups **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35728) in GitLab 13.2. + +You can configure the rate limits for imports and exports of projects and groups: + +To change a rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Import and export rate limits**. +1. Change the value of any rate limit. The rate limits are per minute per user, not per IP address. + Set to `0` to disable a rate limit. + +| Limit | Default | +|-------------------------|---------| +| Project Import | 6 | +| Project Export | 6 | +| Project Export Download | 1 | +| Group Import | 6 | +| Group Export | 6 | +| Group Export Download | 1 | + +When a user exceeds a rate limit, it is logged in `auth.log`. diff --git a/doc/administration/settings/incident_management_rate_limits.md b/doc/administration/settings/incident_management_rate_limits.md new file mode 100644 index 00000000000..2a74c843107 --- /dev/null +++ b/doc/administration/settings/incident_management_rate_limits.md @@ -0,0 +1,39 @@ +--- +type: reference +stage: Monitor +group: Respond +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Incident management rate limits **(ULTIMATE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17859) in GitLab 12.5. + +You can limit the number of inbound alerts for [incidents](../../operations/incident_management/incidents.md) +that can be created in a period of time. The inbound [incident management](../../operations/incident_management/index.md) +alert limit can help prevent overloading your incident responders by reducing the +number of alerts or duplicate issues. + +As an example, if you set a limit of `10` requests every `60` seconds, +and `11` requests are sent to an [alert integration endpoint](../../operations/incident_management/integrations.md) within one minute, +the eleventh request is blocked. Access to the endpoint is allowed again after one minute. + +This limit is: + +- Applied independently per project. +- Not applied per IP address. +- Disabled by default. + +Requests that exceed the limit are logged into `auth.log`. + +## Set a limit on inbound alerts + +To set inbound incident management alert limits: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Incident Management Limits**. +1. Select the **Enable Incident Management inbound alert limit** checkbox. +1. Optional. Input a custom value for **Maximum requests per project per rate limit period**. Default is 3600. +1. Optional. Input a custom value for **Rate limit period**. Default is 3600 seconds. diff --git a/doc/administration/settings/index.md b/doc/administration/settings/index.md new file mode 100644 index 00000000000..5fc0c029c30 --- /dev/null +++ b/doc/administration/settings/index.md @@ -0,0 +1,215 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: index +--- + +# Admin Area settings **(FREE SELF)** + +As an administrator of a GitLab self-managed instance, you can manage the behavior of your +deployment. + +The **Admin Area** is not accessible on GitLab.com, and settings can only be changed by the +GitLab.com administrators. For the settings and limits on the GitLab.com instance, +read [GitLab.com settings](../../user/gitlab_com/index.md). + +## Access the Admin Area + +To access the **Admin Area**: + +1. Sign in to your GitLab instance as an administrator. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings**, and the group of settings to view: + - [General](#general) + - [Geo](#geo) + - [CI/CD](#cicd) + - [Integrations](#integrations) + - [Metrics and profiling](#metrics-and-profiling) + - [Network](#network) + - [Preferences](#preferences) + - [Reporting](#reporting) + - [Repository](#repository) + - [Templates](#templates) + +### General + +The **General** settings contain: + +- [Visibility and access controls](../settings/visibility_and_access_controls.md) - Set default and + restrict visibility levels. Configure import sources and Git access protocol. +- [Account and limit](../settings/account_and_limit_settings.md) - Set projects and maximum size limits, + session duration, user options, and check feature availability for namespace plan. +- [Diff limits](../diff_limits.md) - Diff content limits. +- [Sign-up restrictions](../settings/sign_up_restrictions.md) - Configure the way a user creates a new account. +- [Sign in restrictions](../settings/sign_in_restrictions.md) - Set requirements for a user to sign in. + Enable mandatory two-factor authentication. +- [Terms of Service and Privacy Policy](../settings/terms.md) - Include a Terms of Service agreement + and Privacy Policy that all users must accept. +- [External Authentication](../../administration/settings/external_authorization.md#configuration) - External Classification Policy Authorization. +- [Web terminal](../integration/terminal.md#limiting-websocket-connection-time) - + Set max session time for web terminal. +- [FLoC](floc.md) - Enable or disable + [Federated Learning of Cohorts (FLoC)](https://en.wikipedia.org/wiki/Federated_Learning_of_Cohorts) tracking. +- [GitLab for Slack app](../../user/admin_area/settings/slack_app.md) - Enable and configure the GitLab for Slack app. + +### CI/CD + +The **CI/CD** settings contain: + +- [Continuous Integration and Deployment](../../administration/settings/continuous_integration.md) - + Auto DevOps, runners and job artifacts. +- [Required pipeline configuration](../../administration/settings/continuous_integration.md#required-pipeline-configuration) - + Set an instance-wide auto included [pipeline configuration](../../ci/yaml/index.md). + This pipeline configuration is run after the project's own configuration. +- [Package Registry](../../administration/settings/continuous_integration.md#package-registry-configuration) - + Settings related to the use and experience of using the GitLab Package Registry. Some + [risks are involved](../../user/packages/container_registry/reduce_container_registry_storage.md#use-with-external-container-registries) + in enabling some of these settings. + +## Security and Compliance settings + +- [License compliance settings](security_and_compliance.md#choose-package-registry-metadata-to-sync): Enable or disable synchronization of package metadata by a registry type. + +### Geo **(PREMIUM SELF)** + +The **Geo** setting contains: + +- [Geo](../geo/index.md) - Replicate your GitLab instance to other + geographical locations. Redirects to **Admin Area > Geo > Settings** are no + longer available at **Admin Area > Settings > Geo** in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/36896). + +### Integrations + +The **Integrations** settings contain: + +- [Elasticsearch](../../integration/advanced_search/elasticsearch.md#enable-advanced-search) - + Elasticsearch integration. Elasticsearch AWS IAM. +- [Kroki](../integration/kroki.md#enable-kroki-in-gitlab) - + Allow rendering of diagrams in AsciiDoc and Markdown documents using [kroki.io](https://kroki.io). +- [Mailgun](../integration/mailgun.md) - Enable your GitLab instance + to receive invite email bounce events from Mailgun, if it is your email provider. +- [PlantUML](../integration/plantuml.md) - Allow rendering of PlantUML + diagrams in documents. +- [Customer experience improvement and third-party offers](../settings/third_party_offers.md) - + Control the display of customer experience improvement content and third-party offers. +- [Snowplow](../../development/internal_analytics/snowplow/index.md) - Configure the Snowplow integration. +- [Google GKE](../../user/project/clusters/add_gke_clusters.md) - Google GKE integration enables + you to provision GKE clusters from GitLab. +- [Amazon EKS](../../user/project/clusters/add_eks_clusters.md) - Amazon EKS integration enables + you to provision EKS clusters from GitLab. + +### Metrics and profiling + +The **Metrics and profiling** settings contain: + +- [Metrics - Prometheus](../monitoring/prometheus/gitlab_metrics.md) - + Enable and configure Prometheus metrics. +- [Metrics - Grafana](../monitoring/performance/grafana_configuration.md#integrate-with-gitlab-ui) - + Enable and configure Grafana. +- [Profiling - Performance bar](../monitoring/performance/performance_bar.md#enable-the-performance-bar-for-non-administrators) - + Enable access to the Performance Bar for non-administrator users in a given group. +- [Usage statistics](../settings/usage_statistics.md) - Enable or disable version check and Service Ping. + +### Network + +The **Network** settings contain: + +- Performance optimization - Various settings that affect GitLab performance, including: + - [Write to `authorized_keys` file](../operations/fast_ssh_key_lookup.md#set-up-fast-lookup). + - [Push event activities limit and bulk push events](../settings/push_event_activities_limit.md). +- [User and IP rate limits](../settings/user_and_ip_rate_limits.md) - Configure limits for web and API requests. + These rate limits can be overridden: + - [Package Registry Rate Limits](../settings/package_registry_rate_limits.md) - Configure specific + limits for Packages API requests that supersede the user and IP rate limits. + - [Git LFS Rate Limits](../settings/git_lfs_rate_limits.md) - Configure specific limits for + Git LFS requests that supersede the user and IP rate limits. + - [Files API Rate Limits](../settings/files_api_rate_limits.md) - Configure specific limits for + Files API requests that supersede the user and IP rate limits. + - [Search rate limits](../instance_limits.md#search-rate-limit) - Configure global search request rate limits for authenticated and unauthenticated users. + - [Deprecated API Rate Limits](../settings/deprecated_api_rate_limits.md) - Configure specific limits + for deprecated API requests that supersede the user and IP rate limits. +- [Outbound requests](../../security/webhooks.md) - Allow requests to the local network from webhooks and integrations, or deny all outbound requests. +- [Protected Paths](../settings/protected_paths.md) - Configure paths to be protected by Rack Attack. +- [Incident Management Limits](../../operations/incident_management/index.md) - Limit the + number of inbound alerts that can be sent to a project. +- [Notes creation limit](../settings/rate_limit_on_notes_creation.md) - Set a rate limit on the note creation requests. +- [Get single user limit](../settings/rate_limit_on_users_api.md) - Set a rate limit on users API endpoint to get a user by ID. +- [Projects API rate limits for unauthenticated requests](../settings/rate_limit_on_projects_api.md) - Set a rate limit on Projects list API endpoint for unauthenticated requests. + +### Preferences + +The **Preferences** settings contain: + +- [Email](../settings/email.md) - Various email settings. +- [What's new](../whats-new.md) - Configure **What's new** drawer and content. +- [Help page](help_page.md) - Help page text and support page URL. +- [Pages](../pages/index.md#custom-domain-verification) - + Size and domain settings for static websites. +- [Polling interval multiplier](../polling.md) - + Configure how frequently the GitLab UI polls for updates. +- [Gitaly timeouts](gitaly_timeouts.md) - Configure Gitaly timeouts. +- Localization: + - [Default first day of the week](../../user/profile/preferences.md). + - [Time tracking](../../user/project/time_tracking.md#limit-displayed-units-to-hours). +- [Sidekiq Job Limits](../settings/sidekiq_job_limits.md) - Limit the size of Sidekiq jobs stored in Redis. + +### Reporting + +The **Reporting** settings contain: + +- Spam and Anti-bot protection: + - Anti-spam services, such as [reCAPTCHA](../../integration/recaptcha.md), + [Akismet](../../integration/akismet.md), or [Spamcheck](../reporting/spamcheck.md). + - [IP address restrictions](../reporting/ip_addr_restrictions.md). +- [Abuse reports](../review_abuse_reports.md) - Set notification email for abuse reports. +- [Git abuse rate limit](../reporting/git_abuse_rate_limit.md) - Configure Git abuse rate limit settings. **(ULTIMATE SELF)** + +### Repository + +The **Repository** settings contain: + +- [Repository's custom initial branch name](../../user/project/repository/branches/default.md#instance-level-custom-initial-branch-name) - + Set a custom branch name for new repositories created in your instance. +- [Repository's initial default branch protection](../../user/project/repository/branches/default.md#instance-level-default-branch-protection) - + Configure the branch protections to apply to every repository's default branch. +- [Repository mirror](visibility_and_access_controls.md#enable-project-mirroring) - + Configure repository mirroring. +- [Repository storage](../repository_storage_types.md) - Configure storage path settings. +- Repository maintenance: + - [Repository checks](../repository_checks.md) - Configure + automatic Git checks on repositories. + - [Housekeeping](../housekeeping.md). Configure automatic + Git housekeeping on repositories. + - [Inactive project deletion](../inactive_project_deletion.md). Configure inactive + project deletion. +- [Repository static objects](../static_objects_external_storage.md) - + Serve repository static objects (for example, archives and blobs) from an external storage (for example, a CDN). + +### Templates **(PREMIUM SELF)** + +The **Templates** settings contain: + +- [Templates](instance_template_repository.md#configuration) - Set instance-wide template repository. +- [Custom project templates](../custom_project_templates.md) - Select the custom project template source group. + +## Default first day of the week + +You can change the [Default first day of the week](../../user/profile/preferences.md) +for the entire GitLab instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Scroll to the **Localization** section, and select your desired first day of the week. + +## Default language + +You can change the [Default language](../../user/profile/preferences.md) +for the entire GitLab instance: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Scroll to the **Localization** section, and select your desired default language. diff --git a/doc/administration/settings/instance_template_repository.md b/doc/administration/settings/instance_template_repository.md new file mode 100644 index 00000000000..8c8c9f44998 --- /dev/null +++ b/doc/administration/settings/instance_template_repository.md @@ -0,0 +1,84 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +type: reference +--- + +# Instance template repository **(PREMIUM SELF)** + +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52360) to behave like group-level templates in GitLab 13.9. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321247) in GitLab 14.0. + +In hosted systems, enterprises often have a need to share their own templates +across teams. This feature allows an administrator to pick a project to be the +instance-wide collection of file templates. These templates are then exposed to +all users through the [Web Editor](../../user/project/repository/web_editor.md) +while the project remains secure. + +## Configuration + +To select a project to serve as the custom template repository: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Templates**. +1. Expand **Templates** +1. From the dropdown list, select the project to use as the template repository. +1. Select **Save changes**. +1. Add custom templates to the selected repository. + +After you add templates, you can use them for the entire instance. +They are available in the [Web Editor](../../user/project/repository/web_editor.md) +and through the [API settings](../../api/settings.md). + +## Supported file types and locations + +Templates must be added to a specific subdirectory in the repository, +corresponding to the kind of template. The following types of custom templates +are supported: + +| Type | Directory | Extension | +| :---------------: | :-----------: | :-----------: | +| `Dockerfile` | `Dockerfile` | `.dockerfile` | +| `.gitignore` | `gitignore` | `.gitignore` | +| `.gitlab-ci.yml` | `gitlab-ci` | `.yml` | +| `LICENSE` | `LICENSE` | `.txt` | + +Each template must go in its respective subdirectory, have the correct +extension and not be empty. So, the hierarchy should look like this: + +```plaintext +|-- README.md +|-- Dockerfile + |-- custom_dockerfile.dockerfile + |-- another_dockerfile.dockerfile +|-- gitignore + |-- custom_gitignore.gitignore + |-- another_gitignore.gitignore +|-- gitlab-ci + |-- custom_gitlab-ci.yml + |-- another_gitlab-ci.yml +|-- LICENSE + |-- custom_license.txt + |-- another_license.txt +``` + +Your custom templates are displayed on the dropdown list when a new file is added through the GitLab UI: + +![Custom template dropdown list](img/file_template_user_dropdown.png) + +If this feature is disabled or no templates are present, +no **Custom** section displays in the selection dropdown. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/package_registry_rate_limits.md b/doc/administration/settings/package_registry_rate_limits.md new file mode 100644 index 00000000000..ffba5bbf15a --- /dev/null +++ b/doc/administration/settings/package_registry_rate_limits.md @@ -0,0 +1,57 @@ +--- +stage: Package +group: Package Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Package Registry Rate Limits **(FREE SELF)** + +With the [GitLab Package Registry](../../user/packages/package_registry/index.md), +you can use GitLab as a private or public registry for a variety of common package managers. You can +publish and share packages, which others can consume as a dependency in downstream projects through +the [Packages API](../../api/packages.md). + +If downstream projects frequently download such dependencies, many requests are made through the +Packages API. You may therefore reach enforced [user and IP rate limits](../settings/user_and_ip_rate_limits.md). +To address this issue, you can define specific rate limits for the Packages API: + +- [Unauthenticated requests (per IP)](#enable-unauthenticated-request-rate-limit-for-packages-api). +- [Authenticated API requests (per user)](#enable-authenticated-api-request-rate-limit-for-packages-api). + +These limits are disabled by default. + +When enabled, they supersede the general user and IP rate limits for requests to +the Packages API. You can therefore keep the general user and IP rate limits, and +increase the rate limits for the Packages API. Besides this precedence, there is +no difference in functionality compared to the general user and IP rate limits. + +## Enable unauthenticated request rate limit for packages API + +To enable the unauthenticated request rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Package registry rate limits**. +1. Select **Enable unauthenticated request rate limit**. + + - Optional. Update the **Maximum unauthenticated requests per rate limit period per IP** value. + Defaults to `800`. + - Optional. Update the **Unauthenticated rate limit period in seconds** value. + Defaults to `15`. + +## Enable authenticated API request rate limit for packages API + +To enable the authenticated API request rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network** +1. Expand **Package registry rate limits**. +1. Select **Enable authenticated API request rate limit**. + + - Optional. Update the **Maximum authenticated API requests per rate limit period per user** value. + Defaults to `1000`. + - Optional. Update the **Authenticated API rate limit period in seconds** value. + Defaults to `15`. diff --git a/doc/administration/settings/project_integration_management.md b/doc/administration/settings/project_integration_management.md new file mode 100644 index 00000000000..1bb4465020c --- /dev/null +++ b/doc/administration/settings/project_integration_management.md @@ -0,0 +1,138 @@ +--- +stage: Manage +group: Import and Integrate +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Project integration management **(FREE)** + +Project integrations can be configured and enabled by project administrators. As a GitLab instance +administrator, you can set default configuration parameters for a given integration that all projects +can inherit and use, enabling the integration for all projects that are not already using custom +settings. + +You can update these default settings at any time, changing the settings used for all projects that +are set to use instance-level or group-level defaults. Updating the default settings also enables the integration +for all projects that didn't have it already enabled. + +Only the complete settings for an integration can be inherited. Per-field inheritance is [planned](https://gitlab.com/groups/gitlab-org/-/epics/2137). + +## Manage instance-level default settings for a project integration **(FREE SELF)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2137) in GitLab 13.3 for project-level integrations. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2543) in GitLab 13.6 for group-level integrations. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Integrations**. +1. Select an integration. +1. Enter configuration details and select **Save changes**. + +WARNING: +This may affect all or most of the groups and projects on your GitLab instance. Review the details +below. + +If this is the first time you are setting up instance-level settings for an integration: + +- The integration is enabled for all groups and projects that don't already have this integration configured, + if you have the **Enable integration** toggle turned on in the instance-level settings. +- Groups and projects that already have the integration configured are not affected, but can choose to use the + inherited settings at any time. + +When you make further changes to the instance defaults: + +- They are immediately applied to all groups and projects that have the integration set to use default settings. +- They are immediately applied to newer groups and projects, created after you last saved defaults for the + integration. If your instance-level default setting has the **Enable integration** toggle turned + on, the integration is automatically enabled for all such groups and projects. +- Groups and projects with custom settings selected for the integration are not immediately affected and may + choose to use the latest defaults at any time. + +Only the complete settings for an integration can be inherited. Per-field inheritance +is [planned](https://gitlab.com/groups/gitlab-org/-/epics/2137). This would allow +administrators to update settings inherited by groups and projects without enabling the +integration on all non-configured groups and projects by default. + +### Remove an instance-level default setting + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Integrations**. +1. Select an integration. +1. Select **Reset** and confirm. + +Resetting an instance-level default setting removes the integration from all projects that have the integration set to use default settings. + +### View projects that override the default settings + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218252) in GitLab 14.2. + +You can view which projects in your instance use custom settings that [override the instance-level default settings](#use-custom-settings-for-a-group-or-project-integration) +for an integration. + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Integrations**. +1. Select an integration. +1. Select the **Projects using custom settings** tab. + +## Manage group-level default settings for a project integration + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2543) in GitLab 13.6. + +1. Navigate to the group's **Settings > Integrations**. +1. Select an integration. +1. Enter configuration details and select **Save changes**. + +WARNING: +This may affect all or most of the subgroups and projects belonging to the group. Review the details below. + +If this is the first time you are setting up group-level settings for an integration: + +- The integration is enabled for all subgroups and projects belonging to the group that don't already have + this integration configured, if you have the **Enable integration** toggle turned on in the group-level + settings. +- Subgroups and projects that already have the integration configured are not affected, but can choose to use + the inherited settings at any time. + +When you make further changes to the group defaults: + +- They are immediately applied to all subgroups and projects belonging to the group that have the integration + set to use default settings. +- They are immediately applied to newer subgroups and projects, even those created after you last saved defaults for the + integration. If your group-level default setting has the **Enable integration** toggle turned on, + the integration is automatically enabled for all such subgroups and projects. + +- Subgroups and projects with custom settings selected for the integration are not immediately affected and + may choose to use the latest defaults at any time. + +Only the complete settings for an integration can be inherited. Per-field inheritance +is [planned](https://gitlab.com/groups/gitlab-org/-/epics/2137). This would allow +administrators to update settings inherited by subgroups and projects without enabling the +integration on all non-configured groups and projects by default. + +### Remove a group-level default setting + +1. Navigate to the group's **Settings > Integrations**. +1. Select an integration. +1. Select **Reset** and confirm. + +Resetting a group-level default setting removes integrations that use default settings and belong to a project or subgroup of the group. + +## Use instance-level or group-level default settings for a project integration + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2543) in GitLab 13.6 for group-level settings. + +1. Navigate to **Project > Settings > Integrations**. +1. Choose the integration you want to enable or update. +1. From the dropdown list, select **Use default settings**. +1. Ensure the toggle is set to **Enable integration**. +1. Select **Save changes**. + +## Use custom settings for a group or project integration + +1. Navigate to project or group's **Settings > Integrations**. +1. Choose the integration you want to enable or update. +1. From the dropdown list, select **Use custom settings**. +1. Ensure the toggle is set to **Enable integration** and enter all required settings. +1. Select **Save changes**. diff --git a/doc/administration/settings/protected_paths.md b/doc/administration/settings/protected_paths.md new file mode 100644 index 00000000000..ba257983619 --- /dev/null +++ b/doc/administration/settings/protected_paths.md @@ -0,0 +1,43 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Protected paths **(FREE SELF)** + +Rate limiting is a technique that improves the security and durability of a web +application. For more details, see [Rate limits](../../security/rate_limits.md). + +You can rate limit (protect) specified paths. For these paths, GitLab responds with HTTP status +code `429` to POST requests at protected paths that exceed 10 requests per minute per IP address. + +For example, the following are limited to a maximum 10 requests per minute: + +- User sign-in +- User sign-up (if enabled) +- User password reset + +After 10 requests, the client must wait 60 seconds before it can try again. + +See also: + +- List of paths [protected by default](../instance_limits.md#by-protected-path). +- [User and IP rate limits](user_and_ip_rate_limits.md#response-headers) + for the headers returned to blocked requests. + +## Configure protected paths + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31246) in GitLab 12.4. + +Throttling of protected paths is enabled by default and can be disabled or +customized on **Admin > Network > Protected Paths**, along with these options: + +- Maximum number of requests per period per user. +- Rate limit period in seconds. +- Paths to be protected. + +![protected-paths](img/protected_paths.png) + +Requests over the rate limit are logged into `auth.log`. diff --git a/doc/administration/settings/push_event_activities_limit.md b/doc/administration/settings/push_event_activities_limit.md new file mode 100644 index 00000000000..117e7322e30 --- /dev/null +++ b/doc/administration/settings/push_event_activities_limit.md @@ -0,0 +1,38 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +type: reference +--- + +# Push event activities limit and bulk push events **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31007) in GitLab 12.4. + +Set the number of branches or tags to limit the number of single push events +allowed at once. If the number of events is greater than this, GitLab creates +bulk push event instead. + +For example, if 4 branches are pushed and the limit is currently set to 3, +the activity feed displays: + +![Bulk push event](img/bulk_push_event_v12_4.png) + +With this feature, when a single push includes a lot of changes (for example, 1,000 +branches), only 1 bulk push event is created instead of 1,000 push +events. This helps in maintaining good system performance and preventing spam on +the activity feed. + +To modify this setting: + +- In the Admin Area: + 1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). + 1. Select **Admin Area**. + 1. Select **Settings > Network**. + 1. Expand **Performance optimization**. +- Through the [Application settings API](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls) + as `push_event_activities_limit`. + +The default value is `3`, but the value can be greater than or equal to `0`. Setting this value to `0` does not disable throttling. + +![Push event activities limit](img/push_event_activities_limit_v12_4.png) diff --git a/doc/administration/settings/rate_limit_on_issues_creation.md b/doc/administration/settings/rate_limit_on_issues_creation.md new file mode 100644 index 00000000000..d8bc04fcdd3 --- /dev/null +++ b/doc/administration/settings/rate_limit_on_issues_creation.md @@ -0,0 +1,36 @@ +--- +type: reference +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limits on issue creation **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28129) in GitLab 12.10. + +This setting allows you to rate limit the requests to the issue and epic creation endpoints. +To can change its value: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Issues Rate Limits**. +1. Under **Max requests per minute**, enter the new value. +1. Select **Save changes**. + +For example, if you set a limit of 300, requests using the +[Projects::IssuesController#create](https://gitlab.com/gitlab-org/gitlab/blob/master/app/controllers/projects/issues_controller.rb) +action exceeding a rate of 300 per minute are blocked. Access to the endpoint is allowed after one minute. + +When using [epics](../../user/group/epics/index.md), epic creation shares this rate limit with issues. + +![Rate limits on issues creation](img/rate_limit_on_issues_creation_v14_2.png) + +This limit is: + +- Applied independently per project and per user. +- Not applied per IP address. +- Disabled by default. To enable it, set the option to any value other than `0`. + +Requests over the rate limit are logged into the `auth.log` file. diff --git a/doc/administration/settings/rate_limit_on_notes_creation.md b/doc/administration/settings/rate_limit_on_notes_creation.md new file mode 100644 index 00000000000..59548836e78 --- /dev/null +++ b/doc/administration/settings/rate_limit_on_notes_creation.md @@ -0,0 +1,35 @@ +--- +type: reference +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limits on note creation **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53637) in GitLab 13.9. + +You can configure the per-user rate limit for requests to the note creation endpoint. + +To change the note creation rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Notes rate limit**. +1. In the **Maximum requests per minute** box, enter the new value. +1. Optional. In the **Users to exclude from the rate limit** box, list users allowed to exceed the limit. +1. Select **Save changes**. + +This limit is: + +- Applied independently per user. +- Not applied per IP address. + +The default value is `300`. + +Requests over the rate limit are logged into the `auth.log` file. + +For example, if you set a limit of 300, requests using the +[Projects::NotesController#create](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/notes_controller.rb) +action exceeding a rate of 300 per minute are blocked. Access to the endpoint is allowed after one minute. diff --git a/doc/administration/settings/rate_limit_on_pipelines_creation.md b/doc/administration/settings/rate_limit_on_pipelines_creation.md new file mode 100644 index 00000000000..19e1410ef73 --- /dev/null +++ b/doc/administration/settings/rate_limit_on_pipelines_creation.md @@ -0,0 +1,35 @@ +--- +type: reference +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limits on pipeline creation **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362475) in GitLab 15.0. + +You can set a limit so that users and processes can't request more than a certain number of pipelines each minute. This limit can help save resources and improve stability. + +For example, if you set a limit of `10`, and `11` requests are sent to the [trigger API](../../ci/triggers/index.md) within one minute, +the eleventh request is blocked. Access to the endpoint is allowed again after one minute. + +This limit is: + +- Applied to the number of pipelines created for the same combination of project, commit, and user. +- Not applied per IP address. +- Disabled by default. + +Requests that exceed the limit are logged in the `application_json.log` file. + +## Set a pipeline request limit + +To limit the number of pipeline requests: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Pipelines Rate Limits**. +1. Under **Max requests per minute**, enter a value greater than `0`. +1. Select **Save changes**. +1. Enable `ci_enforce_throttle_pipelines_creation` feature flag to enable the rate limit. diff --git a/doc/administration/settings/rate_limit_on_projects_api.md b/doc/administration/settings/rate_limit_on_projects_api.md new file mode 100644 index 00000000000..2192e4355c0 --- /dev/null +++ b/doc/administration/settings/rate_limit_on_projects_api.md @@ -0,0 +1,37 @@ +--- +type: reference +stage: Data Stores +group: Tenant Scale +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limit on Projects API **(FREE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112283) in GitLab 15.10 with a [flag](../feature_flags.md) named `rate_limit_for_unauthenticated_projects_api_access`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/391922) on May 08, 2023. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/119603) in GitLab 16.0 by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/120445) in GitLab 16.0. Feature flag `rate_limit_for_unauthenticated_projects_api_access` removed. + +You can configure the rate limit per IP address for unauthenticated requests to the [list all projects API](../../api/projects.md#list-all-projects). + +To change the rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Projects API rate limit**. +1. In the **Maximum requests per 10 minutes per IP address** text box, enter the new value. +1. Select **Save changes**. + +The rate limit: + +- Applies per IP address. +- Doesn't apply to authenticated requests. +- Can be set to 0 to disable rate limiting. + +The default value of the rate limit is `400`. + +Requests over the rate limit are logged into the `auth.log` file. + +For example, if you set a limit of 400, unauthenticated requests to the `GET /projects` API endpoint that +exceed a rate of 400 within 10 minutes are blocked. Access to the endpoint is restored after ten minutes have elapsed. diff --git a/doc/administration/settings/rate_limit_on_users_api.md b/doc/administration/settings/rate_limit_on_users_api.md new file mode 100644 index 00000000000..9424e508d86 --- /dev/null +++ b/doc/administration/settings/rate_limit_on_users_api.md @@ -0,0 +1,34 @@ +--- +type: reference +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Rate limits on Users API **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78364) in GitLab 14.8. + +You can configure the per user rate limit for requests to [Users API](../../api/users.md). + +To change the rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Users API rate limit**. +1. In the **Maximum requests per 10 minutes** text box, enter the new value. +1. Optional. In the **Users to exclude from the rate limit** box, list users allowed to exceed the limit. +1. Select **Save changes**. + +This limit is: + +- Applied independently per user. +- Not applied per IP address. + +The default value is `300`. + +Requests over the rate limit are logged into the `auth.log` file. + +For example, if you set a limit of 300, requests to the `GET /users/:id` API endpoint +exceeding a rate of 300 per 10 minutes are blocked. Access to the endpoint is allowed after ten minutes have elapsed. diff --git a/doc/administration/settings/rate_limits_on_git_ssh_operations.md b/doc/administration/settings/rate_limits_on_git_ssh_operations.md new file mode 100644 index 00000000000..64acb15b8ac --- /dev/null +++ b/doc/administration/settings/rate_limits_on_git_ssh_operations.md @@ -0,0 +1,33 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Rate limits on Git SSH operations **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78373) in GitLab 14.7 [with a flag](../feature_flags.md) named `rate_limit_gitlab_shell`. Available by default without a feature flag from 15.8. + +GitLab applies rate limits to Git operations that use SSH by user account and project. When the rate limit is exceeded, GitLab rejects +further connection requests from that user for the project. + +The rate limit applies at the Git command ([plumbing](https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain)) level. +Each command has a rate limit of 600 per minute. For example: + +- `git push` has a rate limit of 600 per minute. +- `git pull` has its own rate limit of 600 per minute. + +Because the same commands are shared by `git-upload-pack`, `git pull`, and `git clone`, they share a rate limit. + +Users on self-managed GitLab can disable this rate limit. + +## Configure GitLab Shell operation limit + +`Git operations using SSH` is enabled by default. Defaults to 600 per user per minute. + +1. On the left sidebar, select **Your work > Admin Area**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Git SSH operations rate limit**. +1. Enter a value for **Maximum number of Git operations per minute**. +1. Select **Save changes**. diff --git a/doc/administration/settings/rate_limits_on_raw_endpoints.md b/doc/administration/settings/rate_limits_on_raw_endpoints.md new file mode 100644 index 00000000000..78e65f7ba7b --- /dev/null +++ b/doc/administration/settings/rate_limits_on_raw_endpoints.md @@ -0,0 +1,29 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Rate limits on raw endpoints **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30635) in GitLab 12.2. + +This setting defaults to `300` requests per minute, and allows you to rate limit the requests to raw endpoints: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **Performance optimization**. + +For example, requests over `300` per minute to `https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/controllers/application_controller.rb` are blocked. Access to the raw file is released after 1 minute. + +![Rate limits on raw endpoints](img/rate_limits_on_raw_endpoints.png) + +This limit is: + +- Applied independently per project, per file path. +- Not applied per IP address. +- Active by default. To disable, set the option to `0`. + +Requests over the rate limit are logged into `auth.log`. diff --git a/doc/administration/settings/scim_setup.md b/doc/administration/settings/scim_setup.md new file mode 100644 index 00000000000..6a02a5b832c --- /dev/null +++ b/doc/administration/settings/scim_setup.md @@ -0,0 +1,43 @@ +--- +type: reference, howto +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Configure SCIM for self-managed GitLab instances **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8902) in GitLab 15.8. + +You can use the open standard System for Cross-domain Identity Management (SCIM) to automatically: + +- Create users. +- Block users. + +The [internal GitLab SCIM API](../../development/internal_api/index.md#instance-scim-api) implements part of [the RFC7644 protocol](https://www.rfc-editor.org/rfc/rfc7644). + +If you are a GitLab.com user, see [configuring SCIM for GitLab.com groups](../../user/group/saml_sso/scim_setup.md). + +## Configure GitLab + +Prerequisites: + +- Configure [SAML single sign-on](../../integration/saml.md). + +To configure GitLab SCIM: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **SCIM Token** section and select **Generate a SCIM token**. +1. For configuration of your identity provider, save the: + - Token from the **Your SCIM token** field. + - URL from the **SCIM API endpoint URL** field. + +## Remove access + +Removing or deactivating a user on the identity provider blocks the user on +the GitLab instance, while the SCIM identity remains linked to the GitLab user. + +To update the user SCIM identity, use the +[internal GitLab SCIM API](../../development/internal_api/index.md#update-a-single-scim-provisioned-user-1). diff --git a/doc/administration/settings/security_and_compliance.md b/doc/administration/settings/security_and_compliance.md new file mode 100644 index 00000000000..2237866ad9c --- /dev/null +++ b/doc/administration/settings/security_and_compliance.md @@ -0,0 +1,25 @@ +--- +stage: Secure +group: Composition Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: howto +--- + +# Security and Compliance Admin Area settings **(ULTIMATE SELF)** + +The settings for package metadata synchronization are located in the [Admin Area](index.md). + +## Choose package registry metadata to sync + +WARNING: +The full package metadata sync can add up to 30 GB to the PostgreSQL database. Ensure you have provisioned enough disk space for the database before enabling this feature. +We are actively working on reducing this data size in [epic 10415](https://gitlab.com/groups/gitlab-org/-/epics/10415). + +To choose the packages you want to synchronize with the GitLab License Database for [License Compliance](../../user/compliance/license_scanning_of_cyclonedx_files/index.md): + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Security and Compliance**. +1. Expand **License Compliance**. +1. Select or clear checkboxes for the package registries that you want to sync. +1. Select **Save changes**. diff --git a/doc/administration/settings/sidekiq_job_limits.md b/doc/administration/settings/sidekiq_job_limits.md new file mode 100644 index 00000000000..d5cd24c5237 --- /dev/null +++ b/doc/administration/settings/sidekiq_job_limits.md @@ -0,0 +1,35 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Sidekiq job size limits **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68982) in GitLab 14.3. + +[Sidekiq](../sidekiq/index.md) jobs get stored in +Redis. To avoid excessive memory for Redis, we: + +- Compress job arguments before storing them in Redis. +- Reject jobs that exceed the specified threshold limit after compression. + +To access Sidekiq job size limits: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Sidekiq job size limits**. +1. Adjust the compression threshold or size limit. The compression can + be disabled by selecting the **Track** mode. + +## Available settings + +| Setting | Default | Description | +|-------------------------------------------|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Limiting mode | Compress | This mode compresses the jobs at the specified threshold and rejects them if they exceed the specified limit after compression. | +| Sidekiq job compression threshold (bytes) | 100 000 (100 KB) | When the size of arguments exceeds this threshold, they are compressed before being stored in Redis. | +| Sidekiq job size limit (bytes) | 0 | The jobs exceeding this size after compression are rejected. This avoids excessive memory usage in Redis leading to instability. Setting it to 0 prevents rejecting jobs. | + +After changing these values, [restart Sidekiq](../restart_gitlab.md). diff --git a/doc/administration/settings/sign_in_restrictions.md b/doc/administration/settings/sign_in_restrictions.md new file mode 100644 index 00000000000..393c7dd6aeb --- /dev/null +++ b/doc/administration/settings/sign_in_restrictions.md @@ -0,0 +1,203 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Sign-in restrictions **(FREE SELF)** + +You can use **Sign-in restrictions** to customize authentication restrictions for web interfaces as well as Git over HTTP(S). + +## Settings + +To access sign-in restriction settings: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Sign-in restrictions** section. + +## Password authentication enabled + +You can restrict the password authentication for web interface and Git over HTTP(S): + +- **Web interface**: When this feature is disabled, the **Standard** sign-in tab + is removed and an [external authentication provider](../auth/index.md) + must be used. +- **Git over HTTP(S)**: When this feature is disabled, a [Personal Access Token](../../user/profile/personal_access_tokens.md) + or LDAP password must be used to authenticate. + +In the event of an external authentication provider outage, use the [GitLab Rails console](../operations/rails_console.md) to [re-enable the standard web sign-in form](#re-enable-standard-web-sign-in-form-in-rails-console). This configuration can also be changed over the [Application settings REST API](../../api/settings.md#change-application-settings) while authenticating with an administrator account's personal access token. + +## Admin Mode + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2158) in GitLab 13.10. + +If you're an administrator, you might want to work in GitLab without administrator access. +You could either create a separate user account that does not have +administrator access or use Admin Mode. + +With Admin Mode, your account does not have administrator access by default. +You can continue to access groups and projects you're a member of. However, for administrative tasks, +you must authenticate (except for [certain features](#limitations-of-admin-mode)). + +When Admin Mode is enabled, it applies to all administrators on the instance. + +When Admin Mode is enabled for an instance, administrators: + +- Are allowed to access group and projects for which they are members. +- Cannot access the **Admin Area**. + +### Enable Admin Mode for your instance + +Administrators can enable Admin Mode though the API, the Rails console, or the UI. + +#### Use the API to enable Admin Mode + +Make the following request to your instance endpoint: + +```shell +curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab.example.com>/api/v4/application/settings?admin_mode=true" +``` + +Replace `<gitlab.example.com>` with your instance URL. + +For more information, see the [list of settings that can be accessed through API calls](../../api/settings.md). + +#### Use the Rails console to enable Admin Mode + +Open the [Rails console](../operations/rails_console.md) and run the following: + +```ruby +::Gitlab::CurrentSettings.update!(admin_mode: true) +``` + +#### Use the UI to enable Admin Mode + +To enable Admin Mode through the UI: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Sign-in restrictions**. +1. In the **Admin Mode** section, select the **Require additional authentication for administrative tasks** checkbox. + +### Turn on Admin Mode for your session + +To turn on Admin Mode for your current session and access potentially dangerous resources: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Enter Admin Mode**. +1. Try to access any part of the UI with `/admin` in the URL (which requires administrator access). + +When Admin Mode status is disabled or turned off, administrators cannot access resources unless +they've been explicitly granted access. For example, administrators get a `404` error +if they try to open a private group or project, unless they are members of that group or project. + +2FA should be enabled for administrators. 2FA, OmniAuth providers, and LDAP +authentication are supported by Admin Mode. Admin Mode status is stored in the current user session and remains active until either: + +- It is explicitly disabled. +- It is disabled automatically after six hours. + +### Turn off Admin Mode for your session + +To turn off Admin Mode for your current session: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Leave Admin Mode**. + +### Limitations of Admin Mode + +Admin Mode times out after six hours, and you cannot change this timeout limit. + +The following access methods are **not** protected by Admin Mode: + +- Git client access (SSH using public keys or HTTPS using Personal Access Tokens). + +In other words, administrators who are otherwise limited by Admin Mode can still use +Git clients without additional authentication steps. + +To use the GitLab REST- or GraphQL API, administrators must [create a personal access token](../../user/profile/personal_access_tokens.md#create-a-personal-access-token) with the [`admin_mode` scope](../../user/profile/personal_access_tokens.md#personal-access-token-scopes). + +If an administrator with a personal access token with the `admin_mode` scope loses their administrator access, that user cannot access the API as an administrator even though they still have the token with the `admin_mode` scope. + +We may address these limitations in the future. For more information see the following epic: +[Admin Mode for GitLab Administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158). + +Also, when GitLab Geo is enabled, you can't view the replication status of projects and designs while +on a secondary node. A fix is proposed when projects ([issue 367926](https://gitlab.com/gitlab-org/gitlab/-/issues/367926)) and designs ([issue 355660](https://gitlab.com/gitlab-org/gitlab/-/issues/355660)) move to the new Geo framework. + +### Troubleshooting Admin Mode + +If necessary, you can disable **Admin Mode** as an administrator by using one of these two methods: + +- **API**: + + ```shell + curl --request PUT --header "PRIVATE-TOKEN:$ADMIN_TOKEN" "<gitlab-url>/api/v4/application/settings?admin_mode=false" + ``` + +- [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session): + + ```ruby + ::Gitlab::CurrentSettings.update!(admin_mode: false) + ``` + +## Two-factor authentication + +When this feature is enabled, all users must use the [two-factor authentication](../../user/profile/account/two_factor_authentication.md). + +After the two-factor authentication is configured as mandatory, users are allowed +to skip forced configuration of two-factor authentication for the configurable grace +period in hours. + +![Two-factor grace period](img/two_factor_grace_period.png) + +## Email notification for unknown sign-ins + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218457) in GitLab 13.2. + +When enabled, GitLab notifies users of sign-ins from unknown IP addresses or devices. For more information, +see [Email notification for unknown sign-ins](../../user/profile/notifications.md#notifications-for-unknown-sign-ins). + +![Email notification for unknown sign-ins](img/email_notification_for_unknown_sign_ins_v13_2.png) + +## Sign-in information + +All users that are not logged in are redirected to the page represented by the configured +**Home page URL** if value is not empty. + +All users are redirected to the page represented by the configured **Sign-out page URL** +after sign out if value is not empty. + +In the **Sign-in restrictions** section, scroll to the **Sign-in text** field. You can add a +custom message for your users in Markdown format. + +For example, if you include the following information in the noted text box: + +```markdown +# Custom sign-in text + +To access this text box: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Sign-in restrictions** section. +``` + +Your users see the **Custom sign-in text** when they navigate to the sign-in screen for your +GitLab instance. + +## Troubleshooting + +### Re-enable standard web sign-in form in rails console + +Re-enable the standard username and password-based sign-in form if it was disabled as a [Sign-in restriction](#password-authentication-enabled). + +You can use this method through the [rails console](../operations/rails_console.md#starting-a-rails-console-session) when a configured external authentication provider (through SSO or an LDAP configuration) is facing an outage and direct sign-in access to GitLab is required. + +```ruby +Gitlab::CurrentSettings.update!(password_authentication_enabled_for_web: true) +``` diff --git a/doc/administration/settings/sign_up_restrictions.md b/doc/administration/settings/sign_up_restrictions.md new file mode 100644 index 00000000000..f213794eb75 --- /dev/null +++ b/doc/administration/settings/sign_up_restrictions.md @@ -0,0 +1,206 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Sign-up restrictions **(FREE SELF)** + +You can enforce the following restrictions on sign ups: + +- Disable new sign ups. +- Require administrator approval for new sign ups. +- Require user email confirmation. +- Allow or deny sign ups using specific email domains. + +## Disable new sign ups + +By default, any user visiting your GitLab domain can sign up for an account. For customers running +public-facing GitLab instances, we **highly** recommend that you consider disabling new sign ups if +you do not expect public users to sign up for an account. + +To disable sign ups: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Sign-up restrictions**. +1. Clear the **Sign-up enabled** checkbox, then select **Save changes**. + +## Require administrator approval for new sign ups + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4491) in GitLab 13.5. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/267568) in GitLab 13.6. + +When this setting is enabled, any user visiting your GitLab domain and signing up for a new account using the registration form +must be explicitly [approved](../../administration/moderate_users.md#approve-or-reject-a-user-sign-up) by an +administrator before they can start using their account. In GitLab 13.6 and later, this setting is +enabled by default for new GitLab instances. It is only applicable if sign ups are enabled. + +To require administrator approval for new sign ups: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Sign-up restrictions**. +1. Select the **Require admin approval for new sign-ups** checkbox, then select **Save changes**. + +In [GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/273258), if an administrator disables this setting, the users in pending approval state are +automatically approved in a background job. + +NOTE: +This setting doesn't apply to LDAP or OmniAuth users. To enforce approvals for new users +signing up using OmniAuth or LDAP, set `block_auto_created_users` to `true` in the +[OmniAuth configuration](../../integration/omniauth.md#configure-common-settings) or +[LDAP configuration](../auth/ldap/index.md#basic-configuration-settings). +A [user cap](#user-cap) can also be used to enforce approvals for new users. + +## Confirm user email + +> - Soft email confirmation [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47003) in GitLab 12.2 [with a flag](../../operations/feature_flags.md) named `soft_email_confirmation`. +> - Soft email confirmation [changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107302/diffs) from a feature flag to an application setting in GitLab 15.9. + +You can send confirmation emails during sign up and require that users confirm +their email address before they are allowed to sign in. + +For example, to enforce confirmation of the email address used for new sign ups: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Sign-up restrictions**. +1. Under **Email confirmation settings**, select **Hard**. + +The following settings are available: + +- **Hard** - Send a confirmation email during sign up. New users must confirm their email address before they can log in. +- **Soft** - Send a confirmation email during sign up. New users can log in immediately, but must confirm their email in three days. Unconfirmed accounts are deleted. +- **Off** - New users can sign up without confirming their email address. + +## User cap + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4315) in GitLab 13.7. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292600) in GitLab 13.9. + +When the number of billable users reaches the user cap, any user who is added or requests access must be +[approved](../../administration/moderate_users.md#approve-or-reject-a-user-sign-up) by an administrator before they can start using +their account. + +If an administrator [increases](#set-the-user-cap-number) or [removes](#remove-the-user-cap) the +user cap, the users in pending approval state are automatically approved in a background job. + +NOTE: +The amount of billable users [is updated once a day](../../subscriptions/self_managed/index.md#billable-users). +This means the user cap might apply only retrospectively after the cap has already been exceeded. +To ensure the cap is enabled immediately, set it to a low value below the current number of +billable users, for example: `1`. + +On instances that use LDAP or OmniAuth, enabling and disabling +[administrator approval for new sign ups](#require-administrator-approval-for-new-sign-ups) +involves changing the Rails configuration, and may require downtime. +User cap can be used instead. As noted above, set the cap to value that ensures it is enforced immediately. + +### Set the user cap number + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Sign-up restrictions**. +1. Enter a number in **User cap**. +1. Select **Save changes**. + +New user sign ups are subject to the user cap restriction. + +## Remove the user cap + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Sign-up restrictions**. +1. Remove the number from **User cap**. +1. Select **Save changes**. + +New users sign ups are not subject to the user cap restriction. Users in pending approval state are +automatically approved in a background job. + +## Minimum password length limit + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20661) in GitLab 12.6 + +You can [change](../../security/password_length_limits.md#modify-minimum-password-length) +the minimum number of characters a user must have in their password using the GitLab UI. + +### Password complexity requirements **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/354965) in GitLab 15.2. + +By default, the only requirement for user passwords is [minimum password length](#minimum-password-length-limit). +You can add additional complexity requirements. Changes to password complexity requirements apply to new passwords: + +- For new users that sign up. +- For existing users that reset their password. + +Existing passwords are unaffected. To change password complexity requirements: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Sign-up restrictions**. +1. Under **Minimum password length (number of characters)**, select additional password complexity requirements. You can require numbers, uppercase letters, lowercase letters, + and symbols. +1. Select **Save changes**. + +## Allow or deny sign ups using specific email domains + +You can specify an inclusive or exclusive list of email domains which can be used for user sign up. + +These restrictions are only applied during sign up from an external user. An administrator can add a +user through the administrator panel with a disallowed domain. The users can also change their +email addresses to disallowed domains after sign up. + +### Allowlist email domains + +You can restrict users only to sign up using email addresses matching the given +domains list. + +### Denylist email domains + +You can block users from signing up when using an email addresses of specific domains. This can +reduce the risk of malicious users creating spam accounts with disposable email addresses. + +### Create email domain allowlist or denylist + +To create an email domain allowlist or denylist: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Sign-up restrictions**. +1. For the allowlist, you must enter the list manually. For the denylist, you can enter the list + manually or upload a `.txt` file that contains list entries. + + Both the allowlist and denylist accept wildcards. For example, you can use +`*.company.com` to accept every `company.com` subdomain, or `*.io` to block all +domains ending in `.io`. Domains must be separated by a whitespace, +semicolon, comma, or a new line. + + ![Domain Denylist](img/domain_denylist_v14_1.png) + +## Set up LDAP user filter + +You can limit GitLab access to a subset of the LDAP users on your LDAP server. + +See the [documentation on setting up an LDAP user filter](../auth/ldap/index.md#set-up-ldap-user-filter) for more information. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/terms.md b/doc/administration/settings/terms.md new file mode 100644 index 00000000000..4b4972acc8e --- /dev/null +++ b/doc/administration/settings/terms.md @@ -0,0 +1,49 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Terms of Service and Privacy Policy **(FREE SELF)** + +An administrator can enforce acceptance of a terms of service and privacy policy. +When this option is enabled, new and existing users must accept the terms. + +When enabled, you can view the Terms of Service at the `-/users/terms` page on the instance, +for example `https://gitlab.example.com/-/users/terms`. + +## Enforce a Terms of Service and Privacy Policy + +To enforce acceptance of a Terms of Service and Privacy Policy: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Terms of Service and Privacy Policy** section. +1. Check the **All users must accept the Terms of Service and Privacy Policy to access GitLab** checkbox. +1. Input the text of the **Terms of Service and Privacy Policy**. You can use [Markdown](../../user/markdown.md) + in this text box. +1. Select **Save changes**. + +For each update to the terms, a new version is stored. When a user accepts or declines the terms, +GitLab records which version they accepted or declined. + +Existing users must accept the terms on their next GitLab interaction. +If an authenticated user declines the terms, they are signed out. + +When enabled, it adds a mandatory checkbox to the sign up page for new users: + +![Sign up form](img/sign_up_terms.png) + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/terraform_limits.md b/doc/administration/settings/terraform_limits.md new file mode 100644 index 00000000000..90ab1c25522 --- /dev/null +++ b/doc/administration/settings/terraform_limits.md @@ -0,0 +1,28 @@ +--- +stage: Deploy +group: Environments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Terraform limits **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352951) in GitLab 15.7. + +You can limit the total storage of [Terraform state files](../terraform_state.md). +The limit applies to each individual +state file version, and is checked whenever a new version is created. + +To add a storage limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Preferences**. +1. Expand **Terraform limits**. +1. Adjust the size limit. + +## Available settings + +| Setting | Default | Description | +|------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------| +| Terraform state size limit (bytes) | 0 | Terraform state files that exceed this size are not saved, and associated Terraform operations are rejected. Set to 0 to allow files of unlimited size. | diff --git a/doc/administration/settings/third_party_offers.md b/doc/administration/settings/third_party_offers.md new file mode 100644 index 00000000000..39e2275f411 --- /dev/null +++ b/doc/administration/settings/third_party_offers.md @@ -0,0 +1,38 @@ +--- +stage: Data Stores +group: Tenant Scale +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# Customer experience improvement and third-party offers **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20379) in GitLab 11.1. + +Within GitLab, we inform users of available third-party offers they might find valuable in order +to enhance the development of their projects. An example is the Google Cloud Platform free credit +for using [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/). + +Furthermore, we use content to improve customer experience. An example are the personalization +questions when creating a group. + +To toggle the display of customer experience improvement content and third-party offers: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand **Customer experience improvement and third-party offers**. +1. Select **Do not display content for customer experience improvement and offers from third parties**. +1. Select **Save changes**. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/usage_statistics.md b/doc/administration/settings/usage_statistics.md new file mode 100644 index 00000000000..ba15ee4e33e --- /dev/null +++ b/doc/administration/settings/usage_statistics.md @@ -0,0 +1,230 @@ +--- +stage: Analytics +group: Analytics Instrumentation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Usage statistics **(FREE SELF)** + +GitLab Inc. periodically collects information about your instance in order +to perform various actions. + +All usage statistics are [opt-out](#enable-or-disable-usage-statistics). + +## Service Ping + +Service Ping is a process that collects and sends a weekly payload to GitLab Inc. +For more information, see the [Service Ping guide](../../development/internal_analytics/service_ping/index.md). When Service Ping is enabled, GitLab gathers data from other instances and enables certain [instance-level analytics features](../../user/analytics/index.md) +that are dependent on Service Ping. + +### Why enable Service Ping? + +The main purpose of Service Ping is to build a better GitLab. We collect data about how GitLab is used +to understand feature or stage adoption and usage. This data gives an insight into how GitLab adds +value and helps our team understand the reasons why people use GitLab, and with this knowledge we're able to make better product decisions. + +There are several other benefits to enabling Service Ping: + +- Analyze the users' activities over time of your GitLab installation. +- A [DevOps Score](../analytics/dev_ops_reports.md) to give you an overview of your entire instance's adoption of concurrent DevOps from planning to monitoring. +- More proactive support (assuming that our [Customer Success Managers (CSMs)](https://handbook.gitlab.com/job-families/sales/customer-success-management/) and support organization used the data to deliver more value). +- Insight and advice into how to get the most value out of your investment in GitLab. +- Reports that show how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. +- Participation in our [Registration Features Program](#registration-features-program) to receive free paid features. + +## Registration Features Program + +> Introduced in GitLab 14.1. + +In GitLab versions 14.1 and later, GitLab Free customers with a self-managed instance running +GitLab Enterprise Edition can receive paid features by registering with GitLab and sending us +activity data through Service Ping. Features introduced here do not remove the feature from its paid +tier. Users can continue to access the features in a paid tier without sharing usage data. + +### Features available in 14.1 and later + +- [Email from GitLab](../email_from_gitlab.md). + +### Features available in 14.4 and later + +- [Repository size limit](../../administration/settings/account_and_limit_settings.md#repository-size-limit). +- [Group access restriction by IP address](../../user/group/access_and_permissions.md#restrict-group-access-by-ip-address). + +### Features available in 16.0 and later + +- [View description change history](../../user/discussions/index.md#view-description-change-history). +- [Maintenance mode](../maintenance_mode/index.md). +- [Configurable issue boards](../../user/project/issue_board.md#configurable-issue-boards). +- [Coverage-guided fuzz testing](../../user/application_security/coverage_fuzzing/index.md). +- [Password complexity requirements](../../administration/settings/sign_up_restrictions.md#password-complexity-requirements). + +NOTE: +Registration is not yet required for participation, but may be added in a future milestone. + +### Enable registration features + +1. Sign in as a user with administrator access. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Metrics and profiling**. +1. Expand the **Usage statistics** section. +1. If not enabled, select the **Enable Service Ping** checkbox. +1. Select the **Enable Registration Features** checkbox. +1. Select **Save changes**. + +## Version check + +If enabled, version check informs you if a new version is available and the +importance of it through a status. The status displays on the help pages (`/help`) +for all authenticated users, and on the Admin Area pages. The statuses are: + +- Green: You are running the latest version of GitLab. +- Orange: An updated version of GitLab is available. +- Red: The version of GitLab you are running is vulnerable. You should install + the latest version with security fixes as soon as possible. + +![Orange version check example](../settings/img/update-available.png) + +GitLab Inc. collects your instance's version and hostname (through the HTTP +referer) as part of the version check. No other information is collected. + +This information is used, among other things, to identify to which versions +patches must be backported, making sure active GitLab instances remain +secure. + +If you [disable version check](#enable-or-disable-usage-statistics), this information isn't collected. + +### Request flow example + +The following example shows a basic request/response flow between a +self-managed GitLab instance and the GitLab Version Application: + +```mermaid +sequenceDiagram + participant GitLab instance + participant Version Application + GitLab instance->>Version Application: Is there a version update? + loop Version Check + Version Application->>Version Application: Record version info + end + Version Application->>GitLab instance: Response (PNG/SVG) +``` + +## Configure your network + +To send usage statistics to GitLab Inc., you must allow network traffic from your +GitLab instance to the host `version.gitlab.com` on port `443`. + +If your GitLab instance is behind a proxy, set the appropriate +[proxy configuration variables](https://docs.gitlab.com/omnibus/settings/environment-variables.html). + +## Enable or disable usage statistics + +To enable or disable Service Ping and version check: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Metrics and profiling**. +1. Expand **Usage statistics**. +1. Select or clear the **Enable version check** and **Enable Service Ping** checkboxes. +1. Select **Save changes**. + +NOTE: +Service Ping settings only control whether the data is being shared with GitLab, or used only internally. +Even if you disable Service Ping, the `gitlab_service_ping_worker` background job still periodically generates a Service Ping payload for your instance. +The payload is available in the [Service Usage data](#manually-upload-service-ping-payload) admin section. + +## Disable usage statistics with the configuration file + +NOTE: +The method to disable Service Ping in the GitLab configuration file does not work in +GitLab versions 9.3 to 13.12.3. For more information about how to disable it, see [troubleshooting](../../development/internal_analytics/service_ping/troubleshooting.md#cannot-disable-service-ping-with-the-configuration-file). + +To disable Service Ping and prevent it from being configured in the future through +the Admin Area: + +**For installations using the Linux package:** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['usage_ping_enabled'] = false + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +**For installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + # ... + gitlab: + # ... + usage_ping_enabled: false + ``` + +1. Restart GitLab: + + ```shell + sudo service gitlab restart + ``` + +## View the Service Ping payload + +You can view the exact JSON payload sent to GitLab Inc. in the Admin Area. To view the payload: + +1. Sign in as a user with administrator access. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Metrics and profiling**. +1. Expand the **Usage statistics** section. +1. Select **Preview payload**. + +For an example payload, see [Example Service Ping payload](../../development/internal_analytics/service_ping/index.md#example-service-ping-payload). + +## Manually upload Service Ping payload + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7388) in GitLab 14.8 with a flag named `admin_application_settings_service_usage_data_center`. Disabled by default. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83265) in GitLab 14.10. + +You can upload the Service Ping payload to GitLab even if your instance doesn't have internet access, +or if the Service Ping [cron job](../../development/internal_analytics/service_ping/index.md#how-service-ping-works) is not enabled. + +To upload the payload manually: + +1. Sign in as a user with administrator access. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Service** usage data. +1. Select **Download payload**. +1. Save the JSON file. +1. Visit [Service usage data center](https://version.gitlab.com/usage_data/new). +1. Select **Choose file** and choose the file from p5. +1. Select **Upload**. + +The uploaded file is encrypted and sent using secure HTTPS protocol. HTTPS creates a secure +communication channel between web browser and the server, and protects transmitted data against man-in-the-middle attacks. + +If there are problems with the manual upload: + +1. Open a confidential issue in the [security fork of version app project](https://gitlab.com/gitlab-org/security/version.gitlab.com). +1. Attach the JSON payload if possible. +1. Tag `@gitlab-org/analytics-section/analytics-instrumentation` who will triage the issue. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/user_and_ip_rate_limits.md b/doc/administration/settings/user_and_ip_rate_limits.md new file mode 100644 index 00000000000..44bd08a8824 --- /dev/null +++ b/doc/administration/settings/user_and_ip_rate_limits.md @@ -0,0 +1,240 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# User and IP rate limits **(FREE SELF)** + +Rate limiting is a common technique used to improve the security and durability +of a web application. For more details, see +[Rate limits](../../security/rate_limits.md). + +The following limits are disabled by default: + +- [Unauthenticated API requests (per IP)](#enable-unauthenticated-api-request-rate-limit). +- [Unauthenticated web requests (per IP)](#enable-unauthenticated-web-request-rate-limit). +- [Authenticated API requests (per user)](#enable-authenticated-api-request-rate-limit). +- [Authenticated web requests (per user)](#enable-authenticated-web-request-rate-limit). + +NOTE: +By default, all Git operations are first tried unauthenticated. Because of this, HTTP Git operations +may trigger the rate limits configured for unauthenticated requests. + +NOTE: +[In GitLab 14.8 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/344807), +the rate limits for API requests don't affect requests made by the frontend, as these are always +counted as web traffic. + +## Enable unauthenticated API request rate limit + +To enable the unauthenticated request rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **User and IP rate limits**. +1. Select **Enable unauthenticated API request rate limit**. + + - Optional. Update the **Maximum unauthenticated API requests per rate limit period per IP** value. + Defaults to `3600`. + - Optional. Update the **Unauthenticated rate limit period in seconds** value. + Defaults to `3600`. + +## Enable unauthenticated web request rate limit + +To enable the unauthenticated request rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **User and IP rate limits**. +1. Select **Enable unauthenticated web request rate limit**. + + - Optional. Update the **Maximum unauthenticated web requests per rate limit period per IP** value. + Defaults to `3600`. + - Optional. Update the **Unauthenticated rate limit period in seconds** value. + Defaults to `3600`. + +## Enable authenticated API request rate limit + +To enable the authenticated API request rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **User and IP rate limits**. +1. Select **Enable authenticated API request rate limit**. + + - Optional. Update the **Maximum authenticated API requests per rate limit period per user** value. + Defaults to `7200`. + - Optional. Update the **Authenticated API rate limit period in seconds** value. + Defaults to `3600`. + +## Enable authenticated web request rate limit + +To enable the unauthenticated request rate limit: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **User and IP rate limits**. +1. Select **Enable authenticated web request rate limit**. + + - Optional. Update the **Maximum authenticated web requests per rate limit period per user** value. + Defaults to `7200`. + - Optional. Update the **Authenticated web rate limit period in seconds** value. + Defaults to `3600`. + +## Use a custom rate limit response + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50693) in GitLab 13.8. + +A request that exceeds a rate limit returns a `429` response code and a +plain-text body, which by default is `Retry later`. + +To use a custom response: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > Network**. +1. Expand **User and IP rate limits**. +1. In the **Plain-text response to send to clients that hit a rate limit** text box, + add the plain-text response message. + +## Response headers + +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/731) in GitLab 13.8, the `RateLimit` headers. `Retry-After` was introduced in an earlier version. + +When a client exceeds the associated rate limit, the following requests are +blocked. The server may respond with rate-limiting information allowing the +requester to retry after a specific period of time. These information are +attached into the response headers. + +| Header | Example | Description | +|:----------------------|:--------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `RateLimit-Limit` | `60` | The request quota for the client **each minute**. If the rate limit period set in the Admin Area is different from 1 minute, the value of this header is adjusted to approximately the nearest 60-minute period. | +| `RateLimit-Name` | `throttle_authenticated_web` | Name of the throttle blocking the requests. | +| `RateLimit-Observed` | `67` | Number of requests associated to the client in the time window. | +| `RateLimit-Remaining` | `0` | Remaining quota in the time window. The result of `RateLimit-Limit` - `RateLimit-Observed`. | +| `RateLimit-Reset` | `1609844400` | [Unix time](https://en.wikipedia.org/wiki/Unix_time)-formatted time when the request quota is reset. | +| `RateLimit-ResetTime` | `Tue, 05 Jan 2021 11:00:00 GMT` | [RFC2616](https://www.rfc-editor.org/rfc/rfc2616#section-3.3.1)-formatted date and time when the request quota is reset. | +| `Retry-After` | `30` | Remaining duration **in seconds** until the quota is reset. This is a [standard HTTP header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After). | + +## Use an HTTP header to bypass rate limiting + +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/622) in GitLab 13.6. + +Depending on the needs of your organization, you may want to enable rate limiting +but have some requests bypass the rate limiter. + +You can do this by marking requests that should bypass the rate limiter with a custom +header. You must do this somewhere in a load balancer or reverse proxy in front of +GitLab. For example: + +1. Pick a name for your bypass header. For example, `Gitlab-Bypass-Rate-Limiting`. +1. Configure your load balancer to set `Gitlab-Bypass-Rate-Limiting: 1` on requests + that should bypass GitLab rate limiting. +1. Configure your load balancer to either: + - Erase `Gitlab-Bypass-Rate-Limiting`. + - Set `Gitlab-Bypass-Rate-Limiting` to a value other than `1` on all requests that + should be affected by rate limiting. +1. Set the environment variable `GITLAB_THROTTLE_BYPASS_HEADER`. + - For [Linux package installations](https://docs.gitlab.com/omnibus/settings/environment-variables.html), + set `'GITLAB_THROTTLE_BYPASS_HEADER' => 'Gitlab-Bypass-Rate-Limiting'` in `gitlab_rails['env']`. + - For source installations, set `export GITLAB_THROTTLE_BYPASS_HEADER=Gitlab-Bypass-Rate-Limiting` + in `/etc/default/gitlab`. + +It is important that your load balancer erases or overwrites the bypass +header on all incoming traffic. Otherwise, you must trust your +users to not set that header and bypass the GitLab rate limiter. + +The bypass works only if the header is set to `1`. + +Requests that bypassed the rate limiter because of the bypass header +are marked with `"throttle_safelist":"throttle_bypass_header"` in +[`production_json.log`](../logs/index.md#production_jsonlog). + +To disable the bypass mechanism, make sure the environment variable +`GITLAB_THROTTLE_BYPASS_HEADER` is unset or empty. + +## Allow specific users to bypass authenticated request rate limiting + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49127) in GitLab 13.7. + +Similarly to the bypass header described above, it is possible to allow +a certain set of users to bypass the rate limiter. This only applies +to authenticated requests: with unauthenticated requests, by definition +GitLab does not know who the user is. + +The allowlist is configured as a comma-separated list of user IDs in +the `GITLAB_THROTTLE_USER_ALLOWLIST` environment variable. If you want +users 1, 53 and 217 to bypass the authenticated request rate limiter, +the allowlist configuration would be `1,53,217`. + +- For [Linux package installations](https://docs.gitlab.com/omnibus/settings/environment-variables.html), + set `'GITLAB_THROTTLE_USER_ALLOWLIST' => '1,53,217'` in `gitlab_rails['env']`. +- For source installations, set `export GITLAB_THROTTLE_USER_ALLOWLIST=1,53,217` + in `/etc/default/gitlab`. + +Requests that bypassed the rate limiter because of the user allowlist +are marked with `"throttle_safelist":"throttle_user_allowlist"` in +[`production_json.log`](../logs/index.md#production_jsonlog). + +At application startup, the allowlist is logged in [`auth.log`](../logs/index.md#authlog). + +## Try out throttling settings before enforcing them + +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/629) in GitLab 13.6. + +You can try out throttling settings by setting the `GITLAB_THROTTLE_DRY_RUN` environment variable to +a comma-separated list of throttle names. + +The possible names are: + +- `throttle_unauthenticated` + - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/335300) in GitLab 14.3. Use `throttle_unauthenticated_api` or `throttle_unauthenticated_web` instead. + `throttle_unauthenticated` is still supported and selects both of them. +- `throttle_unauthenticated_api` +- `throttle_unauthenticated_web` +- `throttle_authenticated_api` +- `throttle_authenticated_web` +- `throttle_unauthenticated_protected_paths` +- `throttle_authenticated_protected_paths_api` +- `throttle_authenticated_protected_paths_web` +- `throttle_unauthenticated_packages_api` +- `throttle_authenticated_packages_api` +- `throttle_authenticated_git_lfs` +- `throttle_unauthenticated_files_api` +- `throttle_authenticated_files_api` +- `throttle_unauthenticated_deprecated_api` +- `throttle_authenticated_deprecated_api` + +For example, to try out throttles for all authenticated requests to +non-protected paths can be done by setting +`GITLAB_THROTTLE_DRY_RUN='throttle_authenticated_web,throttle_authenticated_api'`. + +To enable dry run mode for all throttles, the variable can be set to `*`. + +Setting a throttle to dry run mode logs a message to the +[`auth.log`](../logs/index.md#authlog) when it would hit the limit, while letting the +request continue. The log message contains an `env` field set to `track`. The `matched` +field contains the name of throttle that was hit. + +It is important to set the environment variable **before** enabling +the rate limiting in the settings. The settings in the Admin Area +take effect immediately, while setting the environment variable +requires a restart of all the Puma processes. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/settings/visibility_and_access_controls.md b/doc/administration/settings/visibility_and_access_controls.md new file mode 100644 index 00000000000..5e1e35db244 --- /dev/null +++ b/doc/administration/settings/visibility_and_access_controls.md @@ -0,0 +1,363 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +type: reference +--- + +# Control access and visibility **(FREE SELF)** + +GitLab enables users with administrator access to enforce +specific controls on branches, projects, snippets, groups, and more. + +To access the visibility and access control options: + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. + +## Define which roles can create projects + +Instance-level protections for project creation define which roles can +[add projects to a group](../../user/group/index.md#specify-who-can-add-projects-to-a-group) +on the instance. To alter which roles have permission to create projects: + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. For **Default project creation protection**, select the desired roles: + - No one. + - Maintainers. + - Developers and Maintainers. +1. Select **Save changes**. + +## Restrict project deletion to administrators **(PREMIUM SELF)** + +> User interface [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/352960) in GitLab 15.1. + +By default both administrators and anyone with the **Owner** role can delete a project. To restrict project deletion to only administrators: + +1. Sign in to GitLab as a user with administrator access. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Scroll to: + - (GitLab 15.1 and later) **Allowed to delete projects**, and select **Administrators**. + - (GitLab 15.0 and earlier) **Default project deletion protection** and select **Only admins can delete project**. +1. Select **Save changes**. + +## Deletion protection **(PREMIUM SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255449) in GitLab 14.2 for groups created after August 12, 2021. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/352960) from default delayed project deletion in GitLab 15.1. +> - [Enabled for projects in personal namespaces](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89466) in GitLab 15.1. +> - [Disabled for projects in personal namespaces](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95495) in GitLab 15.3. +> - [Removed option to delete immediately](https://gitlab.com/gitlab-org/gitlab/-/issues/389557) in GitLab 15.11 [with a flag](../feature_flags.md) named `always_perform_delayed_deletion`. Disabled by default. +> - Enabled delayed deletion by default and removed the option to delete immediately [on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/393622) and [on self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/119606) in GitLab 16.0. + +Instance-level protection against accidental deletion of groups and projects. + +### Retention period + +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/352960) in GitLab 15.1. + +Groups and projects remain restorable within a defined retention period. By default this is 7 days but it can be changed. +Setting the retention period to `0` means that groups and project are removed immediately and cannot be restored. + +In GitLab 15.1 and later, the retention period must be between `1` and `90`. If the retention period was `0` before the 15.1 update, +then it gets automatically changed to `1` while also disabling deletion protection the next time any application setting is changed. + +### Delayed project deletion + +> - User interface [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/352960) in GitLab 15.1. +> - Enabled delayed deletion by default and removed the option to delete immediately [on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/393622) and [on self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/119606) in GitLab 16.0. + +To configure delayed project deletion: + +1. Sign in to GitLab as a user with administrator access. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Scroll to: + - (In GitLab 15.11 and later with `always_perform_delayed_deletion` feature flag enabled, or GitLab 16.0 and later) **Deletion protection** and set the retention period to a value between `1` and `90`. + - (GitLab 15.1 and later) **Deletion protection** and select keep deleted groups and projects, and select a retention period. + - (GitLab 15.0 and earlier) **Default delayed project protection** and select **Enable delayed project deletion by + default for newly-created groups.** Then set a retention period in **Default deletion delay**. +1. Select **Save changes**. + +Deletion protection is not available for projects only (without being also being enabled for groups). + +In GitLab 15.1, and later this setting is enforced on groups when disabled and it cannot be overridden. + +### Delayed group deletion + +> - User interface [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352960) in GitLab 15.1. +> - [Changed to default behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/389557) on the Premium and Ultimate tier in GitLab 16.0. + +Groups remain restorable if the retention period is `1` or more days. + +In GitLab 15.1 and later, delayed group deletion can be enabled by setting **Deletion projection** to **Keep deleted**. +In GitLab 15.11 and later with the `always_perform_delayed_deletion` feature flag enabled, or in GitLab 16.0 and later: + +- The **Keep deleted** option is removed. +- Delayed group deletion is the default. + +### Override defaults and delete immediately + +Alternatively, projects that are marked for removal can be deleted immediately. To do so: + +1. [Restore the project](../../user/project/settings/index.md#restore-a-project). +1. Delete the project as described in the + [Administering Projects page](../admin_area.md#administering-projects). + +## Configure project visibility defaults + +To set the default [visibility levels for new projects](../../user/public_access.md): + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Select the desired default project visibility: + - **Private** - Project access must be granted explicitly to each user. If this + project is part of a group, access is granted to members of the group. + - **Internal** - The project can be accessed by any authenticated user except external users. + - **Public** - The project can be accessed without any authentication. +1. Select **Save changes**. + +## Configure snippet visibility defaults + +To set the default visibility levels for new [snippets](../../user/snippets.md): + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Select the desired default snippet visibility. +1. Select **Save changes**. + +For more details on snippet visibility, read +[Project visibility](../../user/public_access.md). + +## Configure group visibility defaults + +To set the default visibility levels for new groups: + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Select the desired default group visibility: + - **Private** - The group and its projects can only be viewed by members. + - **Internal** - The group and any internal projects can be viewed by any authenticated user except external users. + - **Public** - The group and any public projects can be viewed without any authentication. +1. Select **Save changes**. + +For more details on group visibility, see +[Group visibility](../../user/group/index.md#group-visibility). + +## Restrict visibility levels + +When restricting visibility levels, consider how these restrictions interact +with permissions for subgroups and projects that inherit their visibility from +the item you're changing. + +To restrict visibility levels for groups, projects, snippets, and selected pages: + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. In the **Restricted visibility levels** section, select the desired visibility levels to restrict. + - If you restrict the **Public** level: + - Only administrators are able to create public groups, projects, and snippets. + - User profiles are only visible to authenticated users through the Web interface. + - User attributes through the GraphQL API are: + - Not visible in [GitLab 15.1 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88020). + - Only visible to authenticated users between [GitLab 13.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33195) and GitLab 15.0. + - If you restrict the **Internal** level: + - Only administrators are able to create internal groups, projects, and snippets. + - If you restrict the **Private** level: + - Only administrators are able to create private groups, projects, and snippets. +1. Select **Save changes**. + +For more details on project visibility, see +[Project visibility](../../user/public_access.md). + +## Configure allowed import sources + +Before you can import projects from other systems, you must enable the +[import source](../../user/gitlab_com/index.md#default-import-sources) for that system. + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Select each of **Import sources** to allow. +1. Select **Save changes**. + +## Enable project export + +To enable the export of +[projects and their data](../../user/project/settings/import_export.md#export-a-project-and-its-data): + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Scroll to **Project export**. +1. Select the **Enabled** checkbox. +1. Select **Save changes**. + +## Enable migration of groups and projects by direct transfer + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/383268) in GitLab 15.8. + +You can enable migration of groups by direct transfer using the UI. + +To enable migration of groups by direct transfer: + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Scroll to **Allow migrating GitLab groups and projects by direct transfer**. +1. Select the **Enabled** checkbox. +1. Select **Save changes**. + +The same setting +[is available](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls) in the API as the +`bulk_import_enabled` attribute. + +## Configure enabled Git access protocols + +With GitLab access restrictions, you can select the protocols users can use to +communicate with GitLab. Disabling an access protocol does not block port access to the +server itself. The ports used for the protocol, SSH or HTTP(S), are still accessible. +The GitLab restrictions apply at the application level. + +To specify the enabled Git access protocols: + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Select the desired Git access protocols: + - Both SSH and HTTP(S) + - Only SSH + - Only HTTP(S) +1. Select **Save changes**. + +When both SSH and HTTP(S) are enabled, users can choose either protocol. +If only one protocol is enabled: + +- The project page shows only the allowed protocol's URL, with no option to + change it. +- GitLab shows a tooltip when you hover over the protocol for the URL, if user action + (such as adding a SSH key or setting a password) is required: + + ![Project URL with SSH only access](../settings/img/restricted_url.png) + +GitLab only allows Git actions for the protocols you select. + +WARNING: +GitLab versions [10.7 and later](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18021), +allow the HTTP(S) protocol for Git clone or fetch requests done by GitLab Runner +from CI/CD jobs, even if you select **Only SSH**. + +## Customize Git clone URL for HTTP(S) + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18422) in GitLab 12.4. + +You can customize project Git clone URLs for HTTP(S), which affects the clone +panel: + +For example, if: + +- Your GitLab instance is at `https://example.com`, then project clone URLs are like + `https://example.com/foo/bar.git`. +- You want clone URLs that look like `https://git.example.com/gitlab/foo/bar.git` instead, + you can set this setting to `https://git.example.com/gitlab/`. + +![Custom Git clone URL for HTTP](../settings/img/custom_git_clone_url_for_https_v12_4.png) + +To specify a custom Git clone URL for HTTP(S): + +1. Enter a root URL for **Custom Git clone URL for HTTP(S)**. +1. Select **Save changes**. + +NOTE: +SSH clone URLs can be customized in `gitlab.rb` by setting `gitlab_rails['gitlab_ssh_host']` and +other related settings. + +## Configure defaults for RSA, DSA, ECDSA, ED25519, ECDSA_SK, ED25519_SK SSH keys + +These options specify the permitted types and lengths for SSH keys. + +To specify a restriction for each key type: + +1. Select the desired option from the dropdown list. +1. Select **Save changes**. + +For more details, see [SSH key restrictions](../../security/ssh_keys_restrictions.md). + +## Enable project mirroring + +This option is enabled by default. By disabling it, both +[pull mirroring](../../user/project/repository/mirror/pull.md) and [push mirroring](../../user/project/repository/mirror/push.md) no longer +work in every repository. They can only be re-enabled by an administrator user on a per-project basis. + +![Mirror settings](../settings/img/mirror_settings_v15_7.png) + +## Configure globally-allowed IP address ranges + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87579) in GitLab 15.1 [with a flag](../feature_flags.md) named `group_ip_restrictions_allow_global`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/366445) in GitLab 15.4. [Feature flag `group_ip_restrictions_allow_global`](https://gitlab.com/gitlab-org/gitlab/-/issues/366445) removed. + +Administrators can set IP address ranges to be combined with [group-level IP restrictions](../../user/group/access_and_permissions.md#restrict-group-access-by-ip-address). +Use globally-allowed IP addresses to allow aspects of the GitLab installation to work even when group-level IP address +restrictions are set. + +For example, if the GitLab Pages daemon runs on the `10.0.0.0/24` range, you can specify that range as globally-allowed. +This means GitLab Pages can still fetch artifacts from pipelines even if group-level IP address restrictions don't +include the `10.0.0.0/24` range. + +To add a IP address range to the group-level allowlist: + +1. Sign in to GitLab as a user with Administrator access level. +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. In **Globally-allowed IP ranges**, provide a list of IP address ranges. This list: + - Has no limit on the number of IP address ranges. + - Has a size limit of 1 GB. + - Applies to both SSH or HTTP authorized IP address ranges. You cannot split + this list by type of authorization. +1. Select **Save changes**. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, for example `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/sidekiq/extra_sidekiq_processes.md b/doc/administration/sidekiq/extra_sidekiq_processes.md index 31e713bbf06..3a522c7171d 100644 --- a/doc/administration/sidekiq/extra_sidekiq_processes.md +++ b/doc/administration/sidekiq/extra_sidekiq_processes.md @@ -105,8 +105,6 @@ concurrency is set to: When `min_concurrency` is greater than `max_concurrency`, it is treated as being equal to `max_concurrency`. -You can find example values used by GitLab.com by searching for `concurrency:` -in [the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl). The values vary according to the work each specific deployment of Sidekiq does. Any other specialized deployments with processes dedicated to specific queues should have the concurrency tuned according to: diff --git a/doc/administration/sidekiq/index.md b/doc/administration/sidekiq/index.md index bb517e21fd0..c3e1182cb05 100644 --- a/doc/administration/sidekiq/index.md +++ b/doc/administration/sidekiq/index.md @@ -383,7 +383,7 @@ blocking all jobs on that worker from proceeding. If Rugged calls performed by S background task processing. By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. -[Using Rugged is recommended when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if +Using Rugged is recommended when using NFS, but if you are using local storage, disabling Rugged can improve Sidekiq performance: ```shell diff --git a/doc/administration/sidekiq/sidekiq_troubleshooting.md b/doc/administration/sidekiq/sidekiq_troubleshooting.md index 315714cd00b..6802bb68b31 100644 --- a/doc/administration/sidekiq/sidekiq_troubleshooting.md +++ b/doc/administration/sidekiq/sidekiq_troubleshooting.md @@ -529,6 +529,32 @@ The list of available jobs can be found in the [workers](https://gitlab.com/gitl For more information about Sidekiq jobs, see the [Sidekiq-cron](https://github.com/sidekiq-cron/sidekiq-cron#work-with-job) documentation. +## Clearing a Sidekiq job deduplication idempotency key + +Occasionally, jobs that are expected to run (for example, cron jobs) are observed to not run at all. When checking the logs, there might be instances where jobs are seen to not run with a `"job_status": "deduplicated"`. + +This can happen when a job failed and the idempotency key was not cleared properly. For example, [stopping Sidekiq kills any remaining jobs after 25 seconds](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4918). + +[By default, the key expires after 6 hours](https://gitlab.com/gitlab-org/gitlab/-/blob/87c92f06eb92716a26679cd339f3787ae7edbdc3/lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb#L23), +but if you want to clear the idempotency key immediately, follow the following steps (the example provided is for `Geo::VerificationBatchWorker`): + +1. Find the worker class and `args` of the job in the Sidekiq logs: + + ```plaintext + { ... "class":"Geo::VerificationBatchWorker","args":["container_repository"] ... } + ``` + +1. Start a [Rails console session](../operations/rails_console.md#starting-a-rails-console-session). +1. Run the following snippet: + + ```ruby + worker_class = Geo::VerificationBatchWorker + args = ["container_repository"] + dj = Gitlab::SidekiqMiddleware::DuplicateJobs::DuplicateJob.new({ 'class' => worker_class.name, 'args' => args }, worker_class.queue) + dj.send(:idempotency_key) + dj.delete! + ``` + ## Omnibus GitLab 14.0 and later: remove the `sidekiq-cluster` service Omnibus GitLab instances that were configured to run `sidekiq-cluster` prior to GitLab 14.0 diff --git a/doc/administration/silent_mode/index.md b/doc/administration/silent_mode/index.md index 49c95d75768..0b04654beaa 100644 --- a/doc/administration/silent_mode/index.md +++ b/doc/administration/silent_mode/index.md @@ -71,6 +71,14 @@ Incoming emails still raise issues, but the users who sent the emails to [Servic Triggering webhook tests via the UI results in HTTP status 500 responses. +### Remote mirrors + +Updates on [remote mirrors](../../user/project/repository/mirror/index.md) (pushing to, and pulling from them) are suppressed. + +### Integrations + +Executable [integrations](../../user/project/integrations/index.md) are suppressed. + ### Outbound emails Outbound emails are suppressed. diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index 613d161a64c..9b485140070 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -25,7 +25,7 @@ content changes. ### Snippets size limit configuration -This setting is not available through the [Admin Area settings](../../user/admin_area/settings/index.md). +This setting is not available through the [Admin Area settings](../settings/index.md). To configure this setting, use either the Rails console or the [Application settings API](../../api/settings.md). diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 84f596dacf2..2b738f975ba 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -27,7 +27,7 @@ or because your instance doesn't use Terraform. When Terraform state administration is disabled: -- On the left sidebar, you cannot select **Infrastructure > Terraform states**. +- On the left sidebar, you cannot select **Operate > Terraform states**. - Any CI/CD jobs that access the Terraform state fail with this error: ```shell diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index fc319fad3e8..f164e8ccbad 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -50,7 +50,7 @@ This content has been moved to [Troubleshooting CI/CD](../../ci/troubleshooting. ## License -This content has been moved to [Activate GitLab EE with a license file or key](../../user/admin_area/license_file.md). +This content has been moved to [Activate GitLab EE with a license file or key](../../administration/license_file.md). ## Registry diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 120d566a7e7..88e0d4e6c6b 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -152,7 +152,7 @@ without having to [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux- which in this case would restart Puma and Sidekiq. For example, a backup may fail with the following errors in the output of the -[backup command](../../raketasks/backup_restore.md#back-up-gitlab) +[backup command](../../administration/backup_restore/index.md#back-up-gitlab) because the statement timeout was too short: ```plaintext diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 99f19821916..d8f82f13875 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -19,7 +19,7 @@ This is the default configuration. To change the location where the uploads are stored locally, use the steps in this section based on your installation method: NOTE: -For historical reasons, uploads for the whole instance (for example the [favicon](../user/admin_area/appearance.md#favicon)) are stored in a base directory, +For historical reasons, uploads for the whole instance (for example the [favicon](../administration/appearance.md#favicon)) are stored in a base directory, which by default is `uploads/-/system`. Changing the base directory on an existing GitLab installation is strongly discouraged. diff --git a/doc/administration/user_cohorts.md b/doc/administration/user_cohorts.md new file mode 100644 index 00000000000..6f2798f437c --- /dev/null +++ b/doc/administration/user_cohorts.md @@ -0,0 +1,40 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Cohorts **(FREE SELF)** + +You can analyze your users' GitLab activities over time. + +How do you interpret the user cohorts table? Let's review an example with the +following user cohorts: + +![User cohort example](img/cohorts_v13_9_a.png) + +For the cohort of March 2020, three users were added to this server and have +been active since this month. One month later (April 2020), two users are still +active. Five months later (August 2020), one user from this cohort is still +active, or 33% of the original cohort of three that joined in March. + +The **Inactive users** column shows the number of users who were added during +the month, but who never had any activity in the instance. + +How do we measure the activity of users? GitLab considers a user active if: + +- The user signs in. +- The user has Git activity (whether push or pull). +- The user visits pages related to dashboards, projects, issues, or merge + requests ([introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/54947) in GitLab 11.8). +- The user uses the API. +- The user uses the GraphQL API. + +## View user cohorts + +To view user cohorts: + +1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. Select **Admin Area**. +1. Select **Overview > Users**. +1. Select the **Cohorts** tab. diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index c4b00d05f9d..f480f05f6a2 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -14,7 +14,7 @@ By default, new users can create top-level groups. To disable new users' ability to create top-level groups (does not affect existing users' setting), GitLab administrators can modify this setting: - In GitLab 15.5 and later, using either: - - The [GitLab UI](../user/admin_area/settings/account_and_limit_settings.md#prevent-new-users-from-creating-top-level-groups). + - The [GitLab UI](../administration/settings/account_and_limit_settings.md#prevent-new-users-from-creating-top-level-groups). - The [application setting API](../api/settings.md#change-application-settings). - In GitLab 15.4 and earlier, in a configuration file by following the steps in this section. @@ -44,7 +44,7 @@ For self-compiled installations: Administrators can: -- Use the Admin Area to [prevent an existing user from creating top-level groups](../user/admin_area/index.md#prevent-a-user-from-creating-groups). +- Use the Admin Area to [prevent an existing user from creating top-level groups](../administration/admin_area.md#prevent-a-user-from-creating-groups). - Use the [modify an existing user API endpoint](../api/users.md#user-modification) to change the `can_create_group` setting. ## Prevent users from changing their usernames diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md index 540e50d5c70..fd7c1825e8d 100644 --- a/doc/administration/wikis/index.md +++ b/doc/administration/wikis/index.md @@ -26,7 +26,7 @@ is edited again and the content changes. ### Wiki page content size limit configuration -This setting is not available through the [Admin Area settings](../../user/admin_area/settings/index.md). +This setting is not available through the [Admin Area settings](../settings/index.md). To configure this setting, use either the Rails console or the [Application settings API](../../api/settings.md). @@ -76,7 +76,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ### Reduce wiki repository size -The wiki counts as part of the [namespace storage size](../../user/admin_area/settings/account_and_limit_settings.md), +The wiki counts as part of the [namespace storage size](../settings/account_and_limit_settings.md), so you should keep your wiki repositories as compact as possible. For more information about tools to compact repositories, |