diff options
Diffstat (limited to 'doc/administration')
40 files changed, 1908 insertions, 295 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 61ea673071e..ccb4ccbd525 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -59,6 +59,8 @@ From there, you can see the following actions: - 2FA enforcement/grace period changed - Roles allowed to create project changed +Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events-starter) + ### Project events **(STARTER)** NOTE: **Note:** @@ -107,6 +109,8 @@ the filter drop-down. You can further filter by specific group, project or user  +Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events-premium-only) + ### Missing events Some events are not being tracked in Audit Events. Please see the following diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index e02ce1c0a21..d449a5a72af 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -118,6 +118,7 @@ LDAP users must have an email address set, regardless of whether it is used to l ```ruby gitlab_rails['ldap_enabled'] = true +gitlab_rails['prevent_ldap_sign_in'] = false gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below ## ## 'main' is the GitLab 'provider ID' of this LDAP server @@ -357,6 +358,7 @@ production: # snip... ldap: enabled: false + prevent_ldap_sign_in: false servers: ## ## 'main' is the GitLab 'provider ID' of this LDAP server @@ -493,6 +495,38 @@ the configuration option `lowercase_usernames`. By default, this configuration o 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +## Disable LDAP web sign in + +It can be be useful to prevent using LDAP credentials through the web UI when +an alternative such as SAML is preferred. This allows LDAP to be used for group +sync, while also allowing your SAML identity provider to handle additional +checks like custom 2FA. + +When LDAP web sign in is disabled, users will not see a **LDAP** tab on the sign in page. +This does not disable [using LDAP credentials for Git access](#git-password-authentication). + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['prevent_ldap_sign_in'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + prevent_ldap_sign_in: true + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + ## Encryption ### TLS Server Authentication diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 34c668b5fb5..5caf1d53a2c 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -11,9 +11,8 @@ calculated checksum. If the checksum of the data on the **primary** node matches data on the **secondary** node, the data transferred successfully. Following a planned failover, any corrupted data may be **lost**, depending on the extent of the corruption. -If verification fails on the **primary** node, this indicates that Geo is -successfully replicating a corrupted object; restore it from backup or remove it -it from the **primary** node to resolve the issue. +If verification fails on the **primary** node, this indicates Geo is replicating a corrupted object. +You can restore it from backup or remove it from the **primary** node to resolve the issue. If verification succeeds on the **primary** node but fails on the **secondary** node, this indicates that the object was corrupted during the replication process. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index f09d9f20dab..44baab40153 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -187,14 +187,18 @@ keys must be manually replicated to the **secondary** node. 1. Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in your browser. 1. Click the **New node** button. -1. Add the **secondary** node. Use the **exact** name you inputed for `gitlab_rails['geo_node_name']` as the Name and the full URL as the URL. **Do NOT** check the - **This is a primary node** checkbox. -  +1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in + `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character + for character. +1. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb`. These + values must always match, but it doesn't matter if one ends with a `/` and + the other doesn't. +1. **Do NOT** check the **This is a primary node** checkbox. 1. Optionally, choose which groups or storage shards should be replicated by the **secondary** node. Leave blank to replicate all. Read more in [selective synchronization](#selective-synchronization). -1. Click the **Add node** button. +1. Click the **Add node** button to add the **secondary** node. 1. SSH into your GitLab **secondary** server and restart the services: ```sh diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index fa1b0f0e1d7..f7da4e14e9d 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -425,6 +425,9 @@ data before running `pg_basebackup`. --host=<primary_node_ip> ``` + NOTE: **Note:** + Replication slot names must only contain lowercase letters, numbers, and the underscore character. + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` user in the first step. @@ -454,7 +457,7 @@ The replication process is now complete. ## PgBouncer support (optional) -[PgBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool +[PgBouncer](https://www.pgbouncer.org/) may be used with GitLab Geo to pool PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a high-availability configuration with a cluster of nodes supporting a Geo **primary** node and another cluster of nodes supporting a Geo **secondary** node. For more diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index a9087abcbd9..3251a673e4e 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -30,7 +30,7 @@ To enable GitLab replication, you must: checkbox. For LFS, follow the documentation to -[set up LFS object storage](../../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). +[set up LFS object storage](../../lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). For CI job artifacts, there is similar documentation to configure [jobs artifact object storage](../../job_artifacts.md#using-object-storage) diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 4d64941411a..d2fe02abbab 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -115,11 +115,19 @@ Any **secondary** nodes should point only to read-only instances. #### Can Geo detect the current node correctly? -Geo uses the defined node from the **Admin Area > Geo** screen, and tries to match -it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file. -The relevant line looks like: `external_url "http://gitlab.example.com"`. +Geo finds the current machine's name in `/etc/gitlab/gitlab.rb` by first looking +for `gitlab_rails['geo_node_name']`. If it is not defined, then it defaults to +the external URL defined in e.g. `external_url "http://gitlab.example.com"`. To +get a machine's name, run: -To check if the node on the current machine is correctly detected type: +```sh +sudo gitlab-rails runner "puts GeoNode.current_node_name" +``` + +This name is used to look up the node with the same **Name** in +**Admin Area > Geo**. + +To check if current machine is correctly finding its node: ```sh sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect" @@ -134,6 +142,106 @@ and expect something like: By running the command above, `primary` should be `true` when executed in the **primary** node, and `false` on any **secondary** node. +## Fixing errors found when running the Geo check rake task + +When running this rake task, you may see errors if the nodes are not properly configured: + +```sh +sudo gitlab-rake gitlab:geo:check +``` + +1. Rails did not provide a password when connecting to the database + + ```text + Checking Geo ... + + GitLab Geo is available ... Exception: fe_sendauth: no password supplied + GitLab Geo is enabled ... Exception: fe_sendauth: no password supplied + ... + Checking Geo ... Finished + ``` + + - Ensure that you have the `gitlab_rails['db_password']` set to the plain text-password used when creating the hash for `postgresql['sql_user_password']`. + +1. Rails is unable to connect to the database + + ```text + Checking Geo ... + + GitLab Geo is available ... Exception: FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL on + FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL off + GitLab Geo is enabled ... Exception: FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL on + FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL off + ... + Checking Geo ... Finished + ``` + + - Ensure that you have the IP address of the rails node included in `postgresql['md5_auth_cidr_addresses']`. + - Ensure that you have included the subnet mask on the IP address: `postgresql['md5_auth_cidr_addresses'] = ['1.1.1.1/32']`. + +1. Rails has supplied the incorrect password + + ```text + Checking Geo ... + GitLab Geo is available ... Exception: FATAL: password authentication failed for user "gitlab" + FATAL: password authentication failed for user "gitlab" + GitLab Geo is enabled ... Exception: FATAL: password authentication failed for user "gitlab" + FATAL: password authentication failed for user "gitlab" + ... + Checking Geo ... Finished + ``` + + - Verify the correct password is set for `gitlab_rails['db_password']` that was used when creating the hash in `postgresql['sql_user_password']` by running `gitlab-ctl pg-password-md5 gitlab` and entering the password. + +1. Check returns not a secondary node + + ```text + Checking Geo ... + + GitLab Geo is available ... yes + GitLab Geo is enabled ... yes + GitLab Geo secondary database is correctly configured ... not a secondary node + Database replication enabled? ... not a secondary node + ... + Checking Geo ... Finished + ``` + + - Ensure that you have added the secondary node in the admin area of the primary node. + - Ensure that you entered the `external_url` or `gitlab_rails['geo_node_name']` when adding the secondary node in the admin are of the primary node. + - Prior to GitLab 12.4, edit the secondary node in the admin area of the primary node and ensure that there is a trailing `/` in the `Name` field. + +1. Check returns Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist + + ```text + Checking Geo ... + + GitLab Geo is available ... no + Try fixing it: + Upload a new license that includes the GitLab Geo feature + For more information see: + https://about.gitlab.com/features/gitlab-geo/ + GitLab Geo is enabled ... Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist + LINE 8: WHERE a.attrelid = '"geo_nodes"'::regclass + ^ + : SELECT a.attname, format_type(a.atttypid, a.atttypmod), + pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, + c.collname, col_description(a.attrelid, a.attnum) AS comment + FROM pg_attribute a + LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum + LEFT JOIN pg_type t ON a.atttypid = t.oid + LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation + WHERE a.attrelid = '"geo_nodes"'::regclass + AND a.attnum > 0 AND NOT a.attisdropped + ORDER BY a.attnum + ... + Checking Geo ... Finished + ``` + + When performing a Postgres major version (9 > 10) update this is expected. Follow: + + - [initiate-the-replication-process](https://docs.gitlab.com/ee/administration/geo/replication/database.html#step-3-initiate-the-replication-process) + - [Geo database has an outdated FDW remote schema](https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html#geo-database-has-an-outdated-fdw-remote-schema-error) + ## Fixing replication errors The following sections outline troubleshooting steps for fixing replication @@ -258,7 +366,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl tail sidekiq ``` -1. Rename repository storage folders and create new ones +1. Rename repository storage folders and create new ones. If you are not concerned about possible orphaned directories and files, then you can simply skip this step. ```sh mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old @@ -305,7 +413,9 @@ to start again from scratch, there are a few steps that can help you: 1. Reset the Tracking Database ```sh - gitlab-rake geo:db:reset + gitlab-rake geo:db:drop + gitlab-ctl reconfigure + gitlab-rake geo:db:setup ``` 1. Restart previously stopped services @@ -511,6 +621,20 @@ to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-a On a Geo **secondary** node, this command will also clean up all Geo registry record related to the orphan files on disk. +## Fixing sign in errors + +### Message: The redirect URI included is not valid + +If you are able to log in to the **primary** node, but you receive this error +when attempting to log into a **secondary**, you should check that the Geo +node's URL matches its external URL. + +1. On the primary, visit **Admin Area > Geo**. +1. Find the affected **secondary** and click **Edit**. +1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` + in `external_url "https://gitlab.example.com"` on the frontend server(s) of + the **secondary** node. + ## Fixing common errors This section documents common errors reported in the Admin UI and how to fix them. @@ -531,13 +655,6 @@ Geo cannot reuse an existing tracking database. It is safest to use a fresh secondary, or reset the whole secondary by following [Resetting Geo secondary node replication](#resetting-geo-secondary-node-replication). -If you are not concerned about possible orphaned directories and files, then you -can simply reset the existing tracking database with: - -```sh -sudo gitlab-rake geo:db:reset -``` - ### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node This error refers to a problem with the database replica on a **secondary** node, diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 55c7e78da92..37982f2756c 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -10,8 +10,12 @@ Example of the output you will see when pushing to a **secondary** node: ```bash $ git push -> GitLab: You're pushing to a Geo secondary. -> GitLab: We'll help you by proxying this request to the primary: ssh://git@primary.geo/user/repo.git +remote: +remote: You're pushing to a Geo secondary. We'll help you by proxying this +remote: request to the primary: +remote: +remote: ssh://git@primary.geo/user/repo.git +remote: Everything up-to-date ``` diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 6d550a49df4..5288cc6e186 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -4,6 +4,24 @@ Check this document if it includes instructions for the version you are updating These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 12.2 + +GitLab 12.2 includes the following minor PostgreSQL updates: + +- To version `9.6.14` if you run PostgreSQL 9.6. +- To version `10.9` if you run PostgreSQL 10. + +This update will occur even if major PostgreSQL updates are disabled. + +Before [refreshing Foreign Data Wrapper during a Geo HA upgrade](https://docs.gitlab.com/omnibus/update/README.html#run-post-deployment-migrations-and-checks), +restart the Geo tracking database: + +```sh +sudo gitlab-ctl restart geo-postgresql +``` + +The restart avoids a version mismatch when PostgreSQL tries to load the FDW extension. + ## Updating to GitLab 12.1 By default, GitLab 12.1 will attempt to automatically update the diff --git a/doc/administration/git_annex.md b/doc/administration/git_annex.md new file mode 100644 index 00000000000..52d848efa27 --- /dev/null +++ b/doc/administration/git_annex.md @@ -0,0 +1,242 @@ +--- +disqus_identifier: 'https://docs.gitlab.com/ee/workflow/git_annex.html' +--- + +# Git annex + +> **Warning:** GitLab has [completely +removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22). +Read through the [migration guide from git-annex to Git LFS][guide]. + +The biggest limitation of Git, compared to some older centralized version +control systems, has been the maximum size of the repositories. + +The general recommendation is to not have Git repositories larger than 1GB to +preserve performance. Although GitLab has no limit (some repositories in GitLab +are over 50GB!), we subscribe to the advice to keep repositories as small as +you can. + +Not being able to version control large binaries is a big problem for many +larger organizations. +Videos, photos, audio, compiled binaries and many other types of files are too +large. As a workaround, people keep artwork-in-progress in a Dropbox folder and +only check in the final result. This results in using outdated files, not +having a complete history and increases the risk of losing work. + +This problem is solved in GitLab Enterprise Edition by integrating the +[git-annex] application. + +`git-annex` allows managing large binaries with Git without checking the +contents into Git. +You check-in only a symlink that contains the SHA-1 of the large binary. If you +need the large binary, you can sync it from the GitLab server over `rsync`, a +very fast file copying tool. + +## GitLab git-annex Configuration + +`git-annex` is disabled by default in GitLab. Below you will find the +configuration options required to enable it. + +### Requirements + +`git-annex` needs to be installed both on the server and the client side. + +For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running: + +``` +sudo apt-get update && sudo apt-get install git-annex +``` + +For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running: + +``` +sudo yum install epel-release && sudo yum install git-annex +``` + +### Configuration for Omnibus packages + +For Omnibus GitLab packages, only one configuration setting is needed. +The Omnibus package will internally set the correct options in all locations. + +1. In `/etc/gitlab/gitlab.rb` add the following line: + + ```ruby + gitlab_shell['git_annex_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +### Configuration for installations from source + +There are 2 settings to enable git-annex on your GitLab server. + +One is located in `config/gitlab.yml` of the GitLab repository and the other +one is located in `config.yml` of GitLab Shell. + +1. In `config/gitlab.yml` add or edit the following lines: + + ```yaml + gitlab_shell: + git_annex_enabled: true + ``` + +1. In `config.yml` of GitLab Shell add or edit the following lines: + + ```yaml + git_annex_enabled: true + ``` + +1. Save the files and [restart GitLab][] for the changes to take effect. + +## Using GitLab git-annex + +> **Note:** +> Your Git remotes must be using the SSH protocol, not HTTP(S). + +Here is an example workflow of uploading a very large file and then checking it +into your Git repository: + +```bash +git clone git@example.com:group/project.git + +git annex init 'My Laptop' # initialize the annex project and give an optional description +cp ~/tmp/debian.iso ./ # copy a large file into the current directory +git annex add debian.iso # add the large file to git annex +git commit -am "Add Debian iso" # commit the file metadata +git annex sync --content # sync the Git repo and large file to the GitLab server +``` + +The output should look like this: + +``` +commit +On branch master +Your branch is ahead of 'origin/master' by 1 commit. + (use "git push" to publish your local commits) +nothing to commit, working tree clean +ok +pull origin +remote: Counting objects: 5, done. +remote: Compressing objects: 100% (4/4), done. +remote: Total 5 (delta 2), reused 0 (delta 0) +Unpacking objects: 100% (5/5), done. +From example.com:group/project + 497842b..5162f80 git-annex -> origin/git-annex +ok +(merging origin/git-annex into git-annex...) +(recording state in git...) +copy debian.iso (checking origin...) (to origin...) +SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso + 26,214,400 100% 638.88kB/s 0:00:40 (xfr#1, to-chk=0/1) +ok +pull origin +ok +(recording state in git...) +push origin +Counting objects: 15, done. +Delta compression using up to 4 threads. +Compressing objects: 100% (13/13), done. +Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done. +Total 15 (delta 1), reused 0 (delta 0) +To example.com:group/project.git + * [new branch] git-annex -> synced/git-annex + * [new branch] master -> synced/master +ok +``` + +Your files can be found in the `master` branch, but you'll notice that there +are more branches created by the `annex sync` command. + +Git Annex will also create a new directory at `.git/annex/` and will record the +tracked files in the `.git/config` file. The files you assign to be tracked +with `git-annex` will not affect the existing `.git/config` records. The files +are turned into symbolic links that point to data in `.git/annex/objects/`. + +The `debian.iso` file in the example will contain the symbolic link: + +``` +.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso +``` + +Use `git annex info` to retrieve the information about the local copy of your +repository. + +--- + +Downloading a single large file is also very simple: + +```bash +git clone git@gitlab.example.com:group/project.git + +git annex sync # sync Git branches but not the large file +git annex get debian.iso # download the large file +``` + +To download all files: + +```bash +git clone git@gitlab.example.com:group/project.git + +git annex sync --content # sync Git branches and download all the large files +``` + +By using `git-annex` without GitLab, anyone that can access the server can also +access the files of all projects, but GitLab Annex ensures that you can only +access files of projects you have access to (developer, maintainer, or owner role). + +## How it works + +Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great +integration point for `git-annex`. +There is a setting in GitLab Shell so you can disable GitLab Annex support +if you want to. + +## Troubleshooting tips + +Differences in version of `git-annex` on the GitLab server and on local machines +can cause `git-annex` to raise unpredicted warnings and errors. + +Consult the [Annex upgrade page][annex-upgrade] for more information about +the differences between versions. You can find out which version is installed +on your server by navigating to <https://pkgs.org/download/git-annex> and +searching for your distribution. + +Although there is no general guide for `git-annex` errors, there are a few tips +on how to go around the warnings. + +### `git-annex-shell: Not a git-annex or gcrypt repository` + +This warning can appear on the initial `git annex sync --content` and is caused +by differences in `git-annex-shell`. You can read more about it +[in this git-annex issue][issue]. + +One important thing to note is that despite the warning, the `sync` succeeds +and the files are pushed to the GitLab repository. + +If you get hit by this, you can run the following command inside the repository +that the warning was raised: + +``` +git config remote.origin.annex-ignore false +``` + +Consecutive runs of `git annex sync --content` **should not** produce this +warning and the output should look like this: + +``` +commit ok +pull origin +ok +pull origin +ok +push origin +``` + +[annex-upgrade]: https://git-annex.branchable.com/upgrades/ +[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab/issues/1648 +[git-annex]: https://git-annex.branchable.com/ "git-annex website" +[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository" +[guide]: lfs/migrate_from_git_annex_to_git_lfs.html +[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue" +[reconfigure GitLab]: restart_gitlab.md#omnibus-gitlab-reconfigure +[restart GitLab]: restart_gitlab.md#installations-from-source diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index d5749427f6e..82283650070 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -89,6 +89,10 @@ your GitLab installation has three repository storages: `default`, `storage1` and `storage2`. You can use as little as just one server with one repository storage if desired. +Note: **Note:** The token referred to throughout the Gitaly documentation is +just an arbitrary password selected by the administrator. It is unrelated to +tokens created for the GitLab API or other similar web API tokens. + ### 1. Installation First install Gitaly on each Gitaly server using either @@ -142,11 +146,6 @@ the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gi from an existing GitLab server to the Gitaly server. Without this shared secret, Git operations in GitLab will result in an API error. -NOTE: **Note:** -In most or all cases, the storage paths below end in `/repositories` which is -not the case with `path` in `git_data_dirs` of Omnibus GitLab installations. -Check the directory layout on your Gitaly server to be sure. - **For Omnibus GitLab** 1. Edit `/etc/gitlab/gitlab.rb`: @@ -193,24 +192,26 @@ Check the directory layout on your Gitaly server to be sure. On `gitaly1.internal`: ``` - gitaly['storage'] = [ - { 'name' => 'default' }, - { 'name' => 'storage1' }, - ] + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) ``` On `gitaly2.internal`: ``` - gitaly['storage'] = [ - { 'name' => 'storage2' }, - ] + git_data_dirs({ + 'storage2' => { + 'path' => '/srv/gitlab/git-data' + }, + }) ``` - NOTE: **Note:** - In some cases, you'll have to set `path` for `gitaly['storage']` in the - format `'path' => '/mnt/gitlab/<storage name>/repositories'`. - 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). **For installations from source** @@ -222,6 +223,11 @@ Check the directory layout on your Gitaly server to be sure. [auth] token = 'abc123secret' + + [logging] + format = 'json' + level = 'info' + dir = '/var/log/gitaly' ``` 1. Append the following to `/home/git/gitaly/config.toml` for each respective server: @@ -231,9 +237,11 @@ Check the directory layout on your Gitaly server to be sure. ```toml [[storage]] name = 'default' + path = '/var/opt/gitlab/git-data/repositories' [[storage]] name = 'storage1' + path = '/mnt/gitlab/git-data/repositories' ``` On `gitaly2.internal`: @@ -241,12 +249,9 @@ Check the directory layout on your Gitaly server to be sure. ```toml [[storage]] name = 'storage2' + path = '/srv/gitlab/git-data/repositories' ``` - NOTE: **Note:** - In some cases, you'll have to set `path` for each `[[storage]]` in the - format `path = '/mnt/gitlab/<storage name>/repositories'`. - 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). ### 4. Converting clients to use the Gitaly server @@ -327,6 +332,8 @@ When you tail the Gitaly logs on your Gitaly server you should see requests coming in. One sure way to trigger a Gitaly request is to clone a repository from your GitLab server over HTTP. +DANGER: **Danger:** If you have [custom server-side Git hooks](../custom_hooks.md#custom-server-side-git-hooks) configured, either per repository or globally, you must move these to the Gitaly node. If you have multiple Gitaly nodes, copy your custom hook(s) to all nodes. + ### Disabling the Gitaly service in a cluster environment If you are running Gitaly [as a remote @@ -404,11 +411,11 @@ To configure Gitaly with TLS: ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on client node(s). -1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +1. On the Gitaly server, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```sh sudo mkdir -p /etc/gitlab/ssl - sudo chmod 700 /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl sudo cp key.pem cert.pem /etc/gitlab/ssl/ ``` @@ -550,8 +557,11 @@ a few things that you need to do: to eliminate the need for a shared authorized_keys file. 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) including [incremental logging](../job_logs.md#new-incremental-logging-architecture). -1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). +1. Configure [object storage for LFS objects](../lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). 1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for Merge Request Diffs](../merge_request_diffs.md#using-object-storage). +1. Configure [object storage for Packages](../packages/index.md#using-object-storage) (Optional Feature). +1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (Optional Feature). NOTE: **Note:** One current feature of GitLab that still requires a shared directory (NFS) is diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 9038675a28f..83c9aa3f013 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -26,34 +26,42 @@ For this document, the following network topology is assumed: graph TB GitLab --> Gitaly; GitLab --> Praefect; - Praefect --> Preafect-Git-1; - Praefect --> Preafect-Git-2; - Praefect --> Preafect-Git-3; + Praefect --> Praefect-Gitaly-1; + Praefect --> Praefect-Gitaly-2; + Praefect --> Praefect-Gitaly-3; ``` Where `GitLab` is the collection of clients that can request Git operations. -`Gitaly` is a Gitaly server before using Praefect. The Praefect node has two +`Gitaly` is a Gitaly server before using Praefect. The Praefect node has three storage nodes attached. Praefect itself doesn't store data, but connects to -three Gitaly nodes, `Praefect-Git-1`, `Praefect-Git-2`, and `Praefect-Git-3`. -There should be no knowledge other than with Praefect about the existence of -the `Praefect-Git-X` nodes. +three Gitaly nodes, `Praefect-Gitaly-1`, `Praefect-Gitaly-2`, and `Praefect-Gitaly-3`. -### Setup +Praefect may be enabled on its own node or can be run on the GitLab server. +In the example below we will use a separate server, but the optimal configuration +for Praefect is still being determined. -In this setup guide, the Gitaly node will be added first, then Praefect, and -lastly we update the GitLab configuration. +Praefect will handle all Gitaly RPC requests to its child nodes. However, the child nodes +will still need to communicate with the GitLab server via its internal API for authentication +purposes. -#### Gitaly +### Setup -In their own machine, configure the Gitaly server as described in the -[gitaly documentation](index.md#3-gitaly-server-configuration). +In this setup guide we will start by configuring Praefect, then its child +Gitaly nodes, and lastly the GitLab server configuration. #### Praefect -Next, Praefect has to be enabled on its own node. Disable all other services, -and add each Gitaly node that will be connected to Praefect. In the example below, -the Gitaly nodes are named `praefect-git-X`. Note that one node is designated as -primary, by setting the primary to `true`: +On the Praefect node we disable all other services, including Gitaly. We list each +Gitaly node that will be connected to Praefect under `praefect['storage_nodes']`. + +In the example below, the Gitaly nodes are named `praefect-gitaly-N`. Note that one +node is designated as primary by setting the primary to `true`. + +`praefect['auth_token']` is the token used to authenticate with the GitLab server, +just like `gitaly['auth_token']` is used for a standard Gitaly server. + +The `token` field under each storage listed in `praefect['storage_nodes']` is used +to authenticate each child Gitaly node with Praefect. ```ruby # /etc/gitlab/gitlab.rb @@ -67,38 +75,111 @@ unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitaly['enable'] = false +``` + +##### Set up Praefect and its Gitaly nodes + +In the example below, the Gitaly nodes are named `praefect-git-X`. Note that one node is designated as +primary, by setting the primary to `true`: + +```ruby +# /etc/gitlab/gitlab.rb + +# Prevent database connections during 'gitlab-ctl reconfigure' +gitlab_rails['rake_cache_clear'] = false +gitlab_rails['auto_migrate'] = false + +praefect['enable'] = true + +# Make Praefect accept connections on all network interfaces. You must use +# firewalls to restrict access to this address/port. +praefect['listen_addr'] = '0.0.0.0:2305' # virtual_storage_name must match the same storage name given to praefect in git_data_dirs praefect['virtual_storage_name'] = 'praefect' -praefect['auth_token'] = 'super_secret_abc' -praefect['enable'] = true -praefect['storage_nodes'] = [ - { - 'storage' => 'praefect-git-1', - 'address' => 'tcp://praefect-git-1.internal', - 'token' => 'token1', + +# Authentication token to ensure only authorized servers can communicate with +# Praefect server +praefect['auth_token'] = 'praefect-token' +praefect['storage_nodes'] = { + 'praefect-gitaly-1' => { + 'address' => 'tcp://praefect-git-1.internal:8075', + 'token' => 'praefect-gitaly-token', 'primary' => true }, - { - 'storage' => 'praefect-git-2', - 'address' => 'tcp://praefect-git-2.internal', - 'token' => 'token2' + 'praefect-gitaly-2' => { + 'address' => 'tcp://praefect-git-2.internal:8075', + 'token' => 'praefect-gitaly-token' }, - { - 'storage' => 'praefect-git-3', - 'address' => 'tcp://praefect-git-3.internal', - 'token' => 'token3' + 'praefect-gitaly-3' => { + 'address' => 'tcp://praefect-git-3.internal:8075', + 'token' => 'praefect-gitaly-token' } -] +} ``` Save the file and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure). +#### Gitaly + +Next we will configure each Gitaly server assigned to Praefect. Configuration for these +is the same as a normal standalone Gitaly server, except that we use storage names and +auth tokens from Praefect instead of GitLab. + +Below is an example configuration for `praefect-gitaly-1`, the only difference for the +other Gitaly nodes is the storage name under `git_data_dirs`. + +Note that `gitaly['auth_token']` matches the `token` value listed under `praefect['storage_nodes']` +on the Praefect node. + +```ruby +# /etc/gitlab/gitlab.rb + +# Avoid running unnecessary services on the Gitaly server +postgresql['enable'] = false +redis['enable'] = false +nginx['enable'] = false +prometheus['enable'] = false +unicorn['enable'] = false +sidekiq['enable'] = false +gitlab_workhorse['enable'] = false + +# Prevent database connections during 'gitlab-ctl reconfigure' +gitlab_rails['rake_cache_clear'] = false +gitlab_rails['auto_migrate'] = false + +# Configure the gitlab-shell API callback URL. Without this, `git push` will +# fail. This can be your 'front door' GitLab URL or an internal load +# balancer. +# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. +gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + +# Authentication token to ensure only authorized servers can communicate with +# Gitaly server +gitaly['auth_token'] = 'praefect-gitaly-token' + +# Make Gitaly accept connections on all network interfaces. You must use +# firewalls to restrict access to this address/port. +# Comment out following line if you only want to support TLS connections +gitaly['listen_addr'] = "0.0.0.0:8075" + +git_data_dirs({ + "praefect-gitaly-1" => { + "path" => "/var/opt/gitlab/git-data" + } +}) +``` + +Note that just as with a standard Gitaly server, `/etc/gitlab/gitlab-secrets.json` must +be copied from the GitLab server to the Gitaly node for authentication purposes. + +For more information on Gitaly server configuration, see our [gitaly documentation](index.md#3-gitaly-server-configuration). + #### GitLab When Praefect is running, it should be exposed as a storage to GitLab. This is done through setting the `git_data_dirs`. Assuming the default storage -configuration is used, there would be two storages available to GitLab: +is present, there should be two storages available to GitLab: ```ruby git_data_dirs({ @@ -109,6 +190,28 @@ git_data_dirs({ "gitaly_address" => "tcp://praefect.internal:2305" } }) + +gitlab_rails['gitaly_token'] = 'praefect-token' ``` +Note that the storage name used is the same as the `praefect['virtual_storage_name']` set +on the Praefect node. + +Also, the `gitlab_rails['gitaly_token']` matches the value of `praefect['auth_token']` +on Praefect. + Restart GitLab using `gitlab-ctl restart` on the GitLab node. + +### Testing Praefect + +To test Praefect, first set it as the default storage node for new projects +using **Admin Area > Settings > Repository > Repository storage**. Next, +create a new project and check the "Initialize repository with a README" box. + +If you receive a 503 error, check `/var/log/gitlab/gitlab-rails/production.log`. +A `GRPC::Unavailable (14:failed to connect to all addresses)` error indicates +that GitLab was unable to connect to Praefect. + +If the project is created but the README is not, then ensure that the +`/etc/gitlab/gitlab-secrets.json` file from the GitLab server has been copied +to the Gitaly servers. diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 199944a160c..7f0b4056acc 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -38,14 +38,17 @@ The following components need to be considered for a scaled or highly-available environment. In many cases, components can be combined on the same nodes to reduce complexity. -- Unicorn/Workhorse - Web-requests (UI, API, Git over HTTP) +- GitLab application nodes (Unicorn / Puma, Workhorse) - Web-requests (UI, API, Git over HTTP) - Sidekiq - Asynchronous/Background jobs - PostgreSQL - Database - Consul - Database service discovery and health checks/failover - PgBouncer - Database pool manager - Redis - Key/Value store (User sessions, cache, queue for Sidekiq) - Sentinel - Redis health check/failover manager -- Gitaly - Provides high-level RPC access to Git repositories +- Gitaly - Provides high-level storage and RPC access to Git repositories +- S3 Object Storage service[^3] and / or NFS storage servers[^4] for entities such as Uploads, Artifacts, LFS Objects, etc... +- Load Balancer[^2] - Main entry point and handles load balancing for the GitLab application nodes. +- Monitor - Prometheus and Grafana monitoring with auto discovery. ## Scalable Architecture Examples @@ -67,8 +70,10 @@ larger one. - 1 PostgreSQL node - 1 Redis node -- 1 NFS/Gitaly storage server -- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq) +- 1 Gitaly node +- 1 or more Object Storage services[^3] and / or NFS storage server[^4] +- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq) +- 1 or more Load Balancer nodes[^2] - 1 Monitoring node (Prometheus, Grafana) #### Installation Instructions @@ -77,10 +82,12 @@ Complete the following installation steps in order. A link at the end of each section will bring you back to the Scalable Architecture Examples section so you can continue with the next step. -1. [PostgreSQL](database.md#postgresql-in-a-scaled-environment) +1. [PostgreSQL](database.md#postgresql-in-a-scaled-environment) with [PGBouncer](https://docs.gitlab.com/ee/administration/high_availability/pgbouncer.html) 1. [Redis](redis.md#redis-in-a-scaled-environment) -1. [Gitaly](gitaly.md) (recommended) or [NFS](nfs.md) +1. [Gitaly](gitaly.md) (recommended) and / or [NFS](nfs.md)[^4] 1. [GitLab application nodes](gitlab.md) + - With [Object Storage service enabled](../gitaly/index.md#eliminating-nfs-altogether)[^3] +1. [Load Balancer(s)](load_balancer.md)[^2] 1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) ### Full Scaling @@ -91,11 +98,13 @@ is split into separate Sidekiq and Unicorn/Workhorse nodes. One indication that this architecture is required is if Sidekiq queues begin to periodically increase in size, indicating that there is contention or there are not enough resources. -- 1 PostgreSQL node -- 1 Redis node -- 2 or more NFS/Gitaly storage servers +- 1 or more PostgreSQL nodes +- 1 or more Redis nodes +- 1 or more Gitaly storage servers +- 1 or more Object Storage services[^3] and / or NFS storage server[^4] - 2 or more Sidekiq nodes -- 2 or more GitLab application nodes (Unicorn, Workhorse) +- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq) +- 1 or more Load Balancer nodes[^2] - 1 Monitoring node (Prometheus, Grafana) ## High Availability Architecture Examples @@ -114,10 +123,10 @@ This may lead to the other nodes believing a failure has occurred and initiating automated failover. Isolating Redis and Consul from the services they monitor reduces the chances of a false positive that a failure has occurred. -The examples below do not really address high availability of NFS. Some enterprises -have access to NFS appliances that manage availability. This is the best case -scenario. In the future, GitLab may offer a more user-friendly solution to -[GitLab HA Storage](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2472). +The examples below do not address high availability of NFS for objects. We recommend a +S3 Object Storage service[^3] is used where possible over NFS but it's still required in +certain cases[^4]. Where NFS is to be used some enterprises have access to NFS appliances +that manage availability and this would be best case scenario. There are many options in between each of these examples. Work with GitLab Support to understand the best starting point for your workload and adapt from there. @@ -138,8 +147,10 @@ the contention. - 3 PostgreSQL nodes - 2 Redis nodes - 3 Consul/Sentinel nodes -- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PgBouncer) -- 1 NFS/Gitaly server +- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq) +- 1 Gitaly storage servers +- 1 Object Storage service[^3] and / or NFS storage server[^4] +- 1 or more Load Balancer nodes[^2] - 1 Monitoring node (Prometheus, Grafana)  @@ -156,8 +167,10 @@ contention due to certain workloads. - 2 Redis nodes - 3 Consul/Sentinel nodes - 2 or more Sidekiq nodes -- 2 or more GitLab application nodes (Unicorn, Workhorse) -- 1 or more NFS/Gitaly servers +- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq) +- 1 Gitaly storage servers +- 1 Object Storage service[^3] and / or NFS storage server[^4] +- 1 or more Load Balancer nodes[^2] - 1 Monitoring node (Prometheus, Grafana)  @@ -169,6 +182,7 @@ the basis of the GitLab.com architecture. While this scales well it also comes with the added complexity of many more nodes to configure, manage, and monitor. - 3 PostgreSQL nodes +- 1 or more PgBouncer nodes (with associated internal load balancers) - 4 or more Redis nodes (2 separate clusters for persistent and cache data) - 3 Consul nodes - 3 Sentinel nodes @@ -177,120 +191,116 @@ with the added complexity of many more nodes to configure, manage, and monitor. - 2 or more Git nodes (Git over SSH/Git over HTTP) - 2 or more API nodes (All requests to `/api`) - 2 or more Web nodes (All other web requests) -- 2 or more NFS/Gitaly servers +- 2 or more Gitaly storage servers +- 1 or more Object Storage services[^3] and / or NFS storage servers[^4] +- 1 or more Load Balancer nodes[^2] - 1 Monitoring node (Prometheus, Grafana)  -The following pages outline the steps necessary to configure each component -separately: +## Reference Architecture Examples -1. [Configure the database](database.md) -1. [Configure Redis](redis.md) - 1. [Configure Redis for GitLab source installations](redis_source.md) -1. [Configure NFS](nfs.md) - 1. [NFS Client and Host setup](nfs_host_client_setup.md) -1. [Configure the GitLab application servers](gitlab.md) -1. [Configure the load balancers](load_balancer.md) -1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) +The Support and Quality teams build, performance test, and validate Reference +Architectures that support set large numbers of users. The specifications below are a +representation of this work so far and may be adjusted in the future based on +additional testing and iteration. -## Reference Architecture Examples +The architectures have been tested with specific coded workloads. The throughputs +used for testing are calculated based on sample customer data. We test each endpoint +type with the following number of requests per second (RPS) per 1000 users: -These reference architecture examples rely on the general rule that approximately 2 requests per second (RPS) of load is generated for every 100 users. +- API: 20 RPS +- Web: 2 RPS +- Git: 2 RPS -The specifications here were performance tested against a specific coded -workload. Your exact needs may be more, depending on your workload. Your +Note that your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. ### 10,000 User Configuration - **Supported Users (approximate):** 10,000 -- **RPS:** 200 requests per second +- **Test RPS Rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS - **Known Issues:** While validating the reference architecture, slow API endpoints were discovered. For details, see the related issues list in [this issue](https://gitlab.com/gitlab-org/gitlab-foss/issues/64335). -The Support and Quality teams built, performance tested, and validated an -environment that supports about 10,000 users. The specifications below are a -representation of the work so far. The specifications may be adjusted in the -future based on additional testing and iteration. - -| Service | Configuration | GCP type | -| ------------------------------|-------------------------|----------------| -| 3 GitLab Rails <br> - Puma workers on each node set to 90% of available CPUs with 16 threads | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | -| 3 PostgreSQL | 4 vCPU, 15GB Memory | n1-standard-4 | -| 1 PgBouncer | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | -| X Gitaly[^1] <br> - Gitaly Ruby workers on each node set to 90% of available CPUs with 16 threads | 16 vCPU, 60GB Memory | n1-standard-16 | -| 3 Redis Cache + Sentinel <br> - Cache maxmemory set to 90% of available memory | 4 vCPU, 15GB Memory | n1-standard-4 | -| 3 Redis Persistent + Sentinel | 4 vCPU, 15GB Memory | n1-standard-4 | -| 4 Sidekiq | 4 vCPU, 15GB Memory | n1-standard-4 | -| 3 Consul | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | -| 1 NFS Server | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | -| 1 Monitoring node | 4 CPU, 3.6GB Memory | n1-highcpu-4 | -| 1 Load Balancing node[^2] . | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| Service | Nodes | Configuration | GCP type | +| ----------------------------|-------|-----------------------|---------------| +| GitLab Rails <br> - Puma workers on each node set to 90% of available CPUs with 16 threads | 3 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | +| PostgreSQL | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| Gitaly <br> - Gitaly Ruby workers on each node set to 20% of available CPUs | X[^1] . | 16 vCPU, 60GB Memory | n1-standard-16 | +| Redis Cache + Sentinel <br> - Cache maxmemory set to 90% of available memory | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Redis Persistent + Sentinel | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| NFS Server[^4] . | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | +| S3 Object Storage[^3] . | - | - | - | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | +| External load balancing node[^2] . | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| Internal load balancing node[^2] . | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | + +NOTE: **Note:** Memory values are given directly by GCP machine sizes. On different cloud +vendors a best effort like for like can be used. ### 25,000 User Configuration - **Supported Users (approximate):** 25,000 -- **RPS:** 500 requests per second +- **Test RPS Rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS - **Known Issues:** The slow API endpoints that were discovered during testing the 10,000 user architecture also affect the 25,000 user architecture. For details, see the related issues list in [this issue](https://gitlab.com/gitlab-org/gitlab-foss/issues/64335). -The GitLab Support and Quality teams built, performance tested, and validated an -environment that supports around 25,000 users. The specifications below are a -representation of the work so far. The specifications may be adjusted in the -future based on additional testing and iteration. - -NOTE: **Note:** The specifications here were performance tested against a -specific coded workload. Your exact needs may be more, depending on your -workload. Your workload is influenced by factors such as - but not limited to - -how active your users are, how much automation you use, mirroring, and -repo/change size. - -| Service | Configuration | GCP type | -| ------------------------------|-------------------------|----------------| -| 7 GitLab Rails <br> - Puma workers on each node set to 90% of available CPUs with 16 threads | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | -| 3 PostgreSQL | 8 vCPU, 30GB Memory | n1-standard-8 | -| 1 PgBouncer | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | -| X Gitaly[^1] <br> - Gitaly Ruby workers on each node set to 90% of available CPUs with 16 threads | 32 vCPU, 120GB Memory | n1-standard-32 | -| 3 Redis Cache + Sentinel <br> - Cache maxmemory set to 90% of available memory | 4 vCPU, 15GB Memory | n1-standard-4 | -| 3 Redis Persistent + Sentinel | 4 vCPU, 15GB Memory | n1-standard-4 | -| 4 Sidekiq | 4 vCPU, 15GB Memory | n1-standard-4 | -| 3 Consul | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | -| 1 NFS Server | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | -| 1 Monitoring node | 4 CPU, 3.6GB Memory | n1-highcpu-4 | -| 1 Load Balancing node[^2] . | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| Service | Nodes | Configuration | GCP type | +| ----------------------------|-------|-----------------------|---------------| +| GitLab Rails <br> - Puma workers on each node set to 90% of available CPUs with 16 threads | 7 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | +| PostgreSQL | 3 | 8 vCPU, 30GB Memory | n1-standard-8 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| Gitaly <br> - Gitaly Ruby workers on each node set to 20% of available CPUs | X[^1] . | 32 vCPU, 120GB Memory | n1-standard-32 | +| Redis Cache + Sentinel <br> - Cache maxmemory set to 90% of available memory | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Redis Persistent + Sentinel | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| NFS Server[^4] . | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | +| S3 Object Storage[^3] . | - | - | - | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | +| External load balancing node[^2] . | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| Internal load balancing node[^2] . | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | + +NOTE: **Note:** Memory values are given directly by GCP machine sizes. On different cloud +vendors a best effort like for like can be used. ### 50,000 User Configuration - **Supported Users (approximate):** 50,000 -- **RPS:** 1,000 requests per second +- **Test RPS Rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS - **Status:** Work-in-progress - **Related Issue:** See the [related issue](https://gitlab.com/gitlab-org/quality/performance/issues/66) for more information. -The Support and Quality teams are in the process of building and performance -testing an environment that will support around 50,000 users. The specifications -below are a very rough work-in-progress representation of the work so far. The -Quality team will be certifying this environment in late 2019. The -specifications may be adjusted prior to certification based on performance -testing. - -| Service | Configuration | GCP type | -| ------------------------------|-------------------------|----------------| -| 15 GitLab Rails <br> - Puma workers on each node set to 90% of available CPUs with 16 threads | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | -| 3 PostgreSQL | 8 vCPU, 30GB Memory | n1-standard-8 | -| 1 PgBouncer | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | -| X Gitaly[^1] <br> - Gitaly Ruby workers on each node set to 90% of available CPUs with 16 threads | 64 vCPU, 240GB Memory | n1-standard-64 | -| 3 Redis Cache + Sentinel <br> - Cache maxmemory set to 90% of available memory | 4 vCPU, 15GB Memory | n1-standard-4 | -| 3 Redis Persistent + Sentinel | 4 vCPU, 15GB Memory | n1-standard-4 | -| 4 Sidekiq | 4 vCPU, 15GB Memory | n1-standard-4 | -| 3 Consul | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | -| 1 NFS Server | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | -| 1 Monitoring node | 4 CPU, 3.6GB Memory | n1-highcpu-4 | -| 1 Load Balancing node[^2] . | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +NOTE: **Note:** This architecture is a work-in-progress of the work so far. The +Quality team will be certifying this environment in late 2019. The specifications +may be adjusted prior to certification based on performance testing. + +| Service | Nodes | Configuration | GCP type | +| ----------------------------|-------|-----------------------|---------------| +| GitLab Rails <br> - Puma workers on each node set to 90% of available CPUs with 16 threads | 15 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | +| PostgreSQL | 3 | 8 vCPU, 30GB Memory | n1-standard-8 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| Gitaly <br> - Gitaly Ruby workers on each node set to 20% of available CPUs | X[^1] . | 64 vCPU, 240GB Memory | n1-standard-64 | +| Redis Cache + Sentinel <br> - Cache maxmemory set to 90% of available memory | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Redis Persistent + Sentinel | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| NFS Server[^4] . | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | +| S3 Object Storage[^3] . | - | - | - | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | +| External load balancing node[^2] . | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | +| Internal load balancing node[^2] . | 1 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | + +NOTE: **Note:** Memory values are given directly by GCP machine sizes. On different cloud +vendors a best effort like for like can be used. [^1]: Gitaly node requirements are dependent on customer data. We recommend 2 nodes as an absolute minimum for performance at the 10,000 and 25,000 user @@ -298,5 +308,19 @@ testing. additional nodes should be considered in conjunction with a review of project counts and sizes. -[^2]: HAProxy is the only tested and recommended load balancer. Additional - options may be supported in the future. +[^2]: Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. However other reputable load balancers with similar feature sets + should also work here but be aware these aren't validated. + +[^3]: For data objects such as LFS, Uploads, Artifacts, etc... We recommend a S3 Object Storage + where possible over NFS due to better performance and availability. Several types of objects + are supported for S3 storage - [Job artifacts](../job_artifacts.md#using-object-storage), + [LFS](../lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage), + [Uploads](../uploads.md#using-object-storage-core-only), + [Merge Request Diffs](../merge_request_diffs.md#using-object-storage), + [Packages](../packages/index.md#using-object-storage) (Optional Feature), + [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (Optional Feature). + +[^4]: NFS storage server is still required for [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) + and optionally for CI Job Incremental Logging + ([can be switched to use Redis instead](https://docs.gitlab.com/ee/administration/job_logs.html#new-incremental-logging-architecture)). diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md index b01419200cc..392b9b76c31 100644 --- a/doc/administration/high_availability/consul.md +++ b/doc/administration/high_availability/consul.md @@ -102,6 +102,23 @@ To be safe, we recommend you only restart one server agent at a time to ensure t For larger clusters, it is possible to restart multiple agents at a time. See the [Consul consensus document](https://www.consul.io/docs/internals/consensus.html#deployment-table) for how many failures it can tolerate. This will be the number of simulateneous restarts it can sustain. +## Upgrades for bundled Consul + +Nodes running GitLab-bundled Consul should be: + +- Members of a healthy cluster prior to upgrading the GitLab Omnibus package. +- Upgraded one node at a time. + +NOTE: **NOTE:** +Running `curl http://127.0.0.1:8500/v1/health/state/critical` from any Consul node will identify existing health issues in the cluster. The command will return an empty array if the cluster is healthy. + +Consul clusters communicate using the raft protocol. If the current leader goes offline, there needs to be a leader election. A leader node must exist to facilitate synchronization across the cluster. If too many nodes go offline at the same time, the cluster will lose quorum and not elect a leader due to [broken consensus](https://www.consul.io/docs/internals/consensus.html). + +Consult the [troubleshooting section](#troubleshooting) if the cluster is not able to recover after the upgrade. The [outage recovery](#outage-recovery) may be of particular interest. + +NOTE: **NOTE:** +GitLab only uses Consul to store transient data that is easily regenerated. If the bundled Consul was not used by any process other than GitLab itself, then [rebuilding the cluster from scratch](#recreate-from-scratch) is fine. + ## Troubleshooting ### Consul server agents unable to communicate diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index a50cc0cbd03..02684f575d4 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -135,7 +135,8 @@ The recommended configuration for a PostgreSQL HA requires: - `repmgrd` - A service to monitor, and handle failover in case of a failure - `Consul` agent - Used for service discovery, to alert other nodes when failover occurs - A minimum of three `Consul` server nodes -- A minimum of one `pgbouncer` service node +- A minimum of one `pgbouncer` service node, but it's recommended to have one per database node + - An internal load balancer (TCP) is required when there is more than one `pgbouncer` service node You also need to take into consideration the underlying network topology, making sure you have redundant connectivity between all Database and GitLab instances, @@ -155,13 +156,13 @@ Database nodes run two services with PostgreSQL: On failure, the old master node is automatically evicted from the cluster, and should be rejoined manually once recovered. - Consul. Monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. -Alongside PgBouncer, there is a Consul agent that watches the status of the PostgreSQL service. If that status changes, Consul runs a script which updates the configuration and reloads PgBouncer +Alongside each PgBouncer, there is a Consul agent that watches the status of the PostgreSQL service. If that status changes, Consul runs a script which updates the configuration and reloads PgBouncer ##### Connection flow Each service in the package comes with a set of [default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports). You may need to make specific firewall rules for the connections listed below: -- Application servers connect to [PgBouncer default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) +- Application servers connect to either PgBouncer directly via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers. - PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) - Repmgr connects to the database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) - Postgres secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) @@ -499,7 +500,7 @@ attributes set, but the following need to be set. # Disable PostgreSQL on the application node postgresql['enable'] = false - gitlab_rails['db_host'] = 'PGBOUNCER_NODE' + gitlab_rails['db_host'] = 'PGBOUNCER_NODE' or 'INTERNAL_LOAD_BALANCER' gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' gitlab_rails['auto_migrate'] = false @@ -533,7 +534,8 @@ Here we'll show you some fully expanded example configurations. ##### Example recommended setup -This example uses 3 Consul servers, 3 PostgreSQL servers, and 1 application node. +This example uses 3 Consul servers, 3 PgBouncer servers (with associated internal load balancer), +3 PostgreSQL servers, and 1 application node. We start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. @@ -543,14 +545,16 @@ Here is a list and description of each machine and the assigned IP: - `10.6.0.11`: Consul 1 - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -- `10.6.0.21`: PostgreSQL master -- `10.6.0.22`: PostgreSQL secondary -- `10.6.0.23`: PostgreSQL secondary -- `10.6.0.31`: GitLab application - -All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. +- `10.6.0.20`: Internal Load Balancer +- `10.6.0.21`: PgBouncer 1 +- `10.6.0.22`: PgBouncer 2 +- `10.6.0.23`: PgBouncer 3 +- `10.6.0.31`: PostgreSQL master +- `10.6.0.32`: PostgreSQL secondary +- `10.6.0.33`: PostgreSQL secondary +- `10.6.0.41`: GitLab application -The external_url for GitLab is `http://gitlab.example.com` +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes and the external_url for GitLab is `http://gitlab.example.com`. Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. @@ -566,10 +570,45 @@ consul['configuration'] = { server: true, retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } +consul['monitoring_service_discovery'] = true +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup for PgBouncer servers + +On each server edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except Pgbouncer and Consul agent +roles ['pgbouncer_role'] + +# Configure PgBouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['watchers'] = %w(postgresql) +consul['enable'] = true +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +consul['monitoring_service_discovery'] = true ``` [Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. +##### Internal load balancer setup + +An internal load balancer (TCP) is then required to be setup to serve each PgBouncer node (in this example on the IP of `10.6.0.20`). An example of how to do this can be found in the [PgBouncer Configure Internal Load Balancer](pgbouncer.md#configure-the-internal-load-balancer) section. + ##### Example recommended setup for PostgreSQL servers ###### Primary node @@ -589,9 +628,6 @@ postgresql['shared_preload_libraries'] = 'repmgr_funcs' # Disable automatic database migrations gitlab_rails['auto_migrate'] = false -# Configure the Consul agent -consul['services'] = %w(postgresql) - postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' postgresql['max_wal_senders'] = 4 @@ -599,9 +635,13 @@ postgresql['max_wal_senders'] = 4 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +# Configure the Consul agent +consul['services'] = %w(postgresql) +consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } +consul['monitoring_service_discovery'] = true ``` [Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. @@ -626,18 +666,15 @@ On the server edit `/etc/gitlab/gitlab.rb`: ```ruby external_url 'http://gitlab.example.com' -gitlab_rails['db_host'] = '127.0.0.1' +gitlab_rails['db_host'] = '10.6.0.20' # Internal Load Balancer for PgBouncer nodes gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = 'toomanysecrets' gitlab_rails['auto_migrate'] = false postgresql['enable'] = false -pgbouncer['enable'] = true +pgbouncer['enable'] = false consul['enable'] = true -# Configure PgBouncer -pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - # Configure Consul agent consul['watchers'] = %w(postgresql) @@ -661,7 +698,7 @@ consul['configuration'] = { After deploying the configuration follow these steps: -1. On `10.6.0.21`, our primary database +1. On `10.6.0.31`, our primary database Enable the `pg_trgm` extension @@ -673,7 +710,7 @@ After deploying the configuration follow these steps: CREATE EXTENSION pg_trgm; ``` -1. On `10.6.0.22`, our first standby database +1. On `10.6.0.32`, our first standby database Make this node a standby of the primary @@ -681,7 +718,7 @@ After deploying the configuration follow these steps: gitlab-ctl repmgr standby setup 10.6.0.21 ``` -1. On `10.6.0.23`, our second standby database +1. On `10.6.0.33`, our second standby database Make this node a standby of the primary @@ -689,7 +726,7 @@ After deploying the configuration follow these steps: gitlab-ctl repmgr standby setup 10.6.0.21 ``` -1. On `10.6.0.31`, our application server +1. On `10.6.0.41`, our application server Set `gitlab-consul` user's PgBouncer password to `toomanysecrets` @@ -705,7 +742,7 @@ After deploying the configuration follow these steps: #### Example minimal setup -This example uses 3 PostgreSQL servers, and 1 application node. +This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer setup alongside). It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [Consul outage recovery](consul.md#outage-recovery) on the same set of machines. diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index e7479ad1ecb..09b33c3554a 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -4,13 +4,9 @@ type: reference # Working with the bundle PgBouncer service -As part of its High Availability stack, GitLab Premium includes a bundled version of [PgBouncer](https://pgbouncer.github.io/) that can be managed through `/etc/gitlab/gitlab.rb`. +As part of its High Availability stack, GitLab Premium includes a bundled version of [PgBouncer](https://pgbouncer.github.io/) that can be managed through `/etc/gitlab/gitlab.rb`. PgBouncer is used to seamlessly migrate database connections between servers in a failover scenario. Additionally, it can be used in a non-HA setup to pool connections, speeding up response time while reducing resource usage. -In a High Availability setup, PgBouncer is used to seamlessly migrate database connections between servers in a failover scenario. - -Additionally, it can be used in a non-HA setup to pool connections, speeding up response time while reducing resource usage. - -It is recommended to run PgBouncer alongside the `gitlab-rails` service, or on its own dedicated node in a cluster. +In a HA setup, it's recommended to run a PgBouncer node separately for each database node with an internal load balancer (TCP) serving each accordingly. ## Operations @@ -18,7 +14,7 @@ It is recommended to run PgBouncer alongside the `gitlab-rails` service, or on i 1. Make sure you collect [`CONSUL_SERVER_NODES`](database.md#consul-information), [`CONSUL_PASSWORD_HASH`](database.md#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](database.md#pgbouncer-information) before executing the next step. -1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: +1. One each node, edit the `/etc/gitlab/gitlab.rb` config file and replace values noted in the `# START user configuration` section as below: ```ruby # Disable all components except PgBouncer and Consul agent @@ -67,7 +63,7 @@ It is recommended to run PgBouncer alongside the `gitlab-rails` service, or on i #### PgBouncer Checkpoint -1. Ensure the node is talking to the current master: +1. Ensure each node is talking to the current master: ```sh gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD @@ -100,6 +96,41 @@ It is recommended to run PgBouncer alongside the `gitlab-rails` service, or on i (2 rows) ``` +#### Configure the internal load balancer + +If you're running more than one PgBouncer node as recommended, then at this time you'll need to set up a TCP internal load balancer to serve each correctly. This can be done with any reputable TCP load balancer. + +As an example here's how you could do it with [HAProxy](https://www.haproxy.org/): + +``` +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 <ip>:6432 check + server pgbouncer2 <ip>:6432 check + server pgbouncer3 <ip>:6432 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + ### Running PgBouncer as part of a non-HA GitLab installation 1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer` @@ -177,7 +208,7 @@ If you enable Monitoring, it must be enabled on **all** PgBouncer servers. #### Administrative console -As part of Omnibus GitLab, we provide a command `gitlab-ctl pgb-console` to automatically connect to the PgBouncer administrative console. Please see the [PgBouncer documentation](https://pgbouncer.github.io/usage.html#admin-console) for detailed instructions on how to interact with the console. +As part of Omnibus GitLab, we provide a command `gitlab-ctl pgb-console` to automatically connect to the PgBouncer administrative console. Please see the [PgBouncer documentation](https://www.pgbouncer.org/usage.html#admin-console) for detailed instructions on how to interact with the console. To start a session, run diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index ba4599e5bcd..72968cfed56 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -491,7 +491,7 @@ multiple machines with the Sentinel daemon. 1. **You can omit this step if the Sentinels will be hosted in the same node as the other Redis instances.** - [Download/install](https://about.gitlab.com/downloads-ee) the + [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the GitLab downloads page. - Make sure you select the correct Omnibus package, with the same version diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 88cf702cf0e..a0360f1d252 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -7,7 +7,7 @@ GitLab has several features based on receiving incoming emails: - [New issue by email](../user/project/issues/managing_issues.md#new-issue-via-email): allow GitLab users to create a new issue by sending an email to a user-specific email address. -- [New merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email): +- [New merge request by email](../user/project/merge_requests/creating_merge_requests.md#create-new-merge-requests-by-email): allow GitLab users to create a new merge request by sending an email to a user-specific email address. - [Service Desk](../user/project/service_desk.md): provide e-mail support to @@ -79,7 +79,7 @@ email address in order to sign up. If you also host a public-facing GitLab instance at `hooli.com` and set your incoming email domain to `hooli.com`, an attacker could abuse the "Create new issue by email" or -"[Create new merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email)" +"[Create new merge request by email](../user/project/merge_requests/creating_merge_requests.md#create-new-merge-requests-by-email)" features by using a project's unique address as the email when signing up for Slack, which would send a confirmation email, which would create a new issue or merge request on the project owned by the attacker, allowing them to click the diff --git a/doc/administration/index.md b/doc/administration/index.md index f90b9b2c7d5..bf21347fb99 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -43,7 +43,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Configuring GitLab -- [Adjust your instance's timezone](../workflow/timezone.md): Customize the default time zone of GitLab. +- [Adjust your instance's timezone](timezone.md): Customize the default time zone of GitLab. - [System hooks](../system_hooks/system_hooks.md): Notifications when users, projects and keys are changed. - [Security](../security/README.md): Learn what you can do to further secure your GitLab instance. - [Usage statistics, version check, and usage ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. @@ -51,7 +51,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Polling](polling.md): Configure how often the GitLab UI polls for updates. - [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. - [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). -- [Uploads configuration](uploads.md): Configure GitLab uploads storage. +- [Uploads administration](uploads.md): Configure GitLab uploads storage. - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. - [Plugins](plugins.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) @@ -68,11 +68,10 @@ Learn how to install, configure, update, and maintain your GitLab instance. #### Customizing GitLab's appearance -- [Header logo](../customization/branded_page_and_email_header.md): Change the logo on all pages and email headers. -- [Favicon](../customization/favicon.md): Change the default favicon to your own logo. -- [Branded login page](../customization/branded_login_page.md): Customize the login page with your own logo, title, and description. -- [Welcome message](../customization/welcome_message.md): Add a custom welcome message to the sign-in page. -- ["New Project" page](../customization/new_project_page.md): Customize the text to be displayed on the page that opens whenever your users create a new project. +- [Header logo](../user/admin_area/appearance.md#navigation-bar): Change the logo on all pages and email headers. +- [Favicon](../user/admin_area/appearance.md#favicon): Change the default favicon to your own logo. +- [Branded login page](../user/admin_area/appearance.md#sign-in--sign-up-pages): Customize the login page with your own logo, title, and description. +- ["New Project" page](../user/admin_area/appearance.md#new-project-pages): Customize the text to be displayed on the page that opens whenever your users create a new project. - [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text-premium-only): Add additional custom text to emails sent from GitLab. **(PREMIUM ONLY)** ### Maintaining GitLab @@ -105,7 +104,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## User settings and permissions - [Creating users](../user/profile/account/create_accounts.md): Create users manually or through authentication integrations. -- [Libravatar](../customization/libravatar.md): Use Libravatar instead of Gravatar for user avatars. +- [Libravatar](libravatar.md): Use Libravatar instead of Gravatar for user avatars. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS and additional providers. @@ -120,7 +119,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **(PREMIUM ONLY)** - [Incoming email](incoming_email.md): Configure incoming emails to allow users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/managing_issues.md#new-issue-via-email) and - [merge requests by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email), and to enable [Service Desk](../user/project/service_desk.md). + [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#create-new-merge-requests-by-email), and to enable [Service Desk](../user/project/service_desk.md). - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail server with IMAP authentication on Ubuntu for incoming emails. @@ -162,9 +161,10 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Git configuration options - [Custom Git hooks](custom_hooks.md): Custom Git hooks (on the filesystem) for when webhooks aren't enough. -- [Git LFS configuration](../workflow/lfs/lfs_administration.md): Learn how to configure LFS for GitLab. +- [Git LFS configuration](lfs/lfs_administration.md): Learn how to configure LFS for GitLab. - [Housekeeping](housekeeping.md): Keep your Git repositories tidy and fast. - [Configuring Git Protocol v2](git_protocol.md): Git protocol version 2 support. +- [Manage large files with `git-annex` (Deprecated)](git_annex.md) ## Monitoring GitLab diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index e595c640aac..23803b82543 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -96,7 +96,7 @@ To enable the redirection, add the following line in `/etc/gitlab/gitlab.rb`: nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" # Built from source -nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://127.0.0.1:8080/plantuml/; \n}\n" +nginx['custom_gitlab_server_config'] = "location /-/plantuml { \n rewrite ^/-/(plantuml.*) /$1 break;\n proxy_cache off; \n proxy_pass http://localhost:8080/plantuml; \n}\n" ``` To activate the changes, run the following command: diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index d6d56515ac6..6042786d101 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -1,8 +1,8 @@ # Job logs -> [Renamed from Job Traces to Job logs](https://gitlab.com/gitlab-org/gitlab/issues/29121) in 12.4. +> [Renamed from job traces to job logs](https://gitlab.com/gitlab-org/gitlab/issues/29121) in GitLab 12.5. -Job logs (traces) are sent by GitLab Runner while it's processing a job. You can see +Job logs are sent by GitLab Runner while it's processing a job. You can see logs in job pages, pipelines, email notifications, etc. ## Data flow @@ -33,9 +33,8 @@ To change the location where the job logs will be stored, follow the steps below gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. - ---- +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the + changes to take effect. **In installations from source:** @@ -48,10 +47,8 @@ To change the location where the job logs will be stored, follow the steps below builds_path: path/to/builds/ ``` -1. Save the file and [restart GitLab][] for the changes to take effect. - -[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" -[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes + to take effect. ## Uploading logs to object storage @@ -69,8 +66,8 @@ job output in the UI will be empty. ## New incremental logging architecture -> [Introduced][ce-18169] in GitLab 10.4. -> [Announced as General availability][ce-46097] in GitLab 11.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/18169) in GitLab 10.4. +> - [Announced as generally available](https://gitlab.com/gitlab-org/gitlab-foss/issues/46097) in GitLab 11.0. NOTE: **Note:** This feature is off by default. See below for how to [enable or disable](#enabling-incremental-logging) it. @@ -83,7 +80,7 @@ The data flow is the same as described in the [data flow section](#data-flow) with one change: _the stored path of the first two phases is different_. This incremental log architecture stores chunks of logs in Redis and a persistent store (object storage or database) instead of file storage. Redis is used as first-class storage, and it stores up-to 128KB -of data. Once the full chunk is sent, it is flushed to a persistent store, either object storage(temporary directory) or database. +of data. Once the full chunk is sent, it is flushed to a persistent store, either object storage (temporary directory) or database. After a while, the data in Redis and a persitent store will be archived to [object storage](#uploading-logs-to-object-storage). The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`. @@ -163,7 +160,3 @@ instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed. Also, it could pressure the database replication lag. `INSERT`s are generated to indicate that we have log chunk. `UPDATE`s with 128KB of data is issued once we receive multiple chunks. - -[ce-18169]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/18169 -[ce-21193]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/21193 -[ce-46097]: https://gitlab.com/gitlab-org/gitlab-foss/issues/46097 diff --git a/doc/administration/lfs/img/git-annex-branches.png b/doc/administration/lfs/img/git-annex-branches.png Binary files differnew file mode 100644 index 00000000000..3d614f68177 --- /dev/null +++ b/doc/administration/lfs/img/git-annex-branches.png diff --git a/doc/administration/lfs/img/lfs-icon.png b/doc/administration/lfs/img/lfs-icon.png Binary files differnew file mode 100644 index 00000000000..eef9a14187a --- /dev/null +++ b/doc/administration/lfs/img/lfs-icon.png diff --git a/doc/administration/lfs/lfs_administration.md b/doc/administration/lfs/lfs_administration.md new file mode 100644 index 00000000000..f3b8029f487 --- /dev/null +++ b/doc/administration/lfs/lfs_administration.md @@ -0,0 +1,273 @@ +--- +disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' +--- + +# GitLab Git LFS Administration + +Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](manage_large_binaries_with_git_lfs.md). + +## Requirements + +- Git LFS is supported in GitLab starting with version 8.2. +- Support for object storage, such as AWS S3, was introduced in 10.0. +- Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up. + +## Configuration + +Git LFS objects can be large in size. By default, they are stored on the server +GitLab is installed on. + +There are various configuration options to help GitLab server administrators: + +- Enabling/disabling Git LFS support +- Changing the location of LFS object storage +- Setting up object storage supported by [Fog](http://fog.io/about/provider_documentation.html) + +### Configuration for Omnibus installations + +In `/etc/gitlab/gitlab.rb`: + +```ruby +# Change to true to enable lfs - enabled by default if not defined +gitlab_rails['lfs_enabled'] = false + +# Optionally, change the storage path location. Defaults to +# `#{gitlab_rails['shared_path']}/lfs-objects`. Which evaluates to +# `/var/opt/gitlab/gitlab-rails/shared/lfs-objects` by default. +gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" +``` + +### Configuration for installations from source + +In `config/gitlab.yml`: + +```yaml +# Change to true to enable lfs + lfs: + enabled: false + storage_path: /mnt/storage/lfs-objects +``` + +## Storing LFS objects in remote object storage + +> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core in 10.7. + +It is possible to store LFS objects in remote object storage which allows you +to offload local hard disk R/W operations, and free up disk space significantly. +GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](http://fog.io/about/provider_documentation.html) +to check which storage services can be integrated with GitLab. +You can also use external object storage in a private local network. For example, +[MinIO](https://min.io/) is a standalone object storage service, is easy to set up, and works well with GitLab instances. + +GitLab provides two different options for the uploading mechanism: "Direct upload" and "Background upload". + +**Option 1. Direct upload** + +1. User pushes an lfs file to the GitLab instance +1. GitLab-workhorse uploads the file directly to the external object storage +1. GitLab-workhorse notifies GitLab-rails that the upload process is complete + +**Option 2. Background upload** + +1. User pushes an lfs file to the GitLab instance +1. GitLab-rails stores the file in the local file storage +1. GitLab-rails then uploads the file to the external object storage asynchronously + +The following general settings are supported. + +| Setting | Description | Default | +|---------|-------------|---------| +| `enabled` | Enable/disable object storage | `false` | +| `remote_directory` | The bucket name where LFS objects will be stored| | +| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | +| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `connection` | Various connection options described below | | + +The `connection` settings match those provided by [Fog](https://github.com/fog). + +Here is a configuration example with S3. + +| Setting | Description | example | +|---------|-------------|---------| +| `provider` | The provider name | AWS | +| `aws_access_key_id` | AWS credentials, or compatible | `ABC123DEF456` | +| `aws_secret_access_key` | AWS credentials, or compatible | `ABC123DEF456ABC123DEF456ABC123DEF456` | +| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | +| `region` | AWS region | us-east-1 | +| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | +| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | +| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false | +| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false + +Here is a configuration example with GCS. + +| Setting | Description | example | +|---------|-------------|---------| +| `provider` | The provider name | `Google` | +| `google_project` | GCP project name | `gcp-project-12345` | +| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | +| `google_json_key_location` | The json key path | `/path/to/gcp-project-12345-abcde.json` | + +NOTE: **Note:** +The service account must have permission to access the bucket. +[See more](https://cloud.google.com/storage/docs/authentication) + +Here is a configuration example with Rackspace Cloud Files. + +| Setting | Description | example | +|---------|-------------|---------| +| `provider` | The provider name | `Rackspace` | +| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | +| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | +| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` | + +NOTE: **Note:** +Regardless of whether the container has public access enabled or disabled, Fog will +use the TempURL method to grant access to LFS objects. If you see errors in logs referencing +instantiating storage with a temp-url-key, ensure that you have set they key properly +on the Rackspace API and in `gitlab.rb`. You can verify the value of the key Rackspace +has set by sending a GET request with token header to the service access endpoint URL +and comparing the output of the returned headers. + +### Manual uploading to an object storage + +There are two ways to manually do the same thing as automatic uploading (described above). + +**Option 1: rake task** + +```sh +rake gitlab:lfs:migrate +``` + +**Option 2: rails console** + +```sh +$ sudo gitlab-rails console # Login to rails console + +> # Upload LFS files manually +> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| +> lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? +> end +``` + +### S3 for Omnibus installations + +On Omnibus installations, the settings are prefixed by `lfs_object_store_`: + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['lfs_object_store_enabled'] = true + gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects" + gitlab_rails['lfs_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N', + 'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE', + # The below options configure an S3 compatible host instead of AWS + 'host' => 'localhost', + 'endpoint' => 'http://127.0.0.1:9000', + 'path_style' => true + } + ``` + +1. Save the file and [reconfigure GitLab]s for the changes to take effect. +1. Migrate any existing local LFS objects to the object storage: + + ```bash + gitlab-rake gitlab:lfs:migrate + ``` + + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless + `gitlab_rails['lfs_object_store_background_upload']` is set to false. + +### S3 for installations from source + +For source installations the settings are nested under `lfs:` and then +`object_store:`: + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + lfs: + enabled: true + object_store: + enabled: false + remote_directory: lfs-objects # Bucket name + connection: + provider: AWS + aws_access_key_id: 1ABCD2EFGHI34JKLM567N + aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE + region: eu-central-1 + # Use the following options to configure an AWS compatible host such as Minio + host: 'localhost' + endpoint: 'http://127.0.0.1:9000' + path_style: true + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. +1. Migrate any existing local LFS objects to the object storage: + + ```bash + sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production + ``` + + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless `background_upload` is set to + false. + +### Migrating back to local storage + +In order to migrate back to local storage: + +1. Set both `direct_upload` and `background_upload` to false under the LFS object storage settings. Don't forget to restart GitLab. +1. Run `rake gitlab:lfs:migrate_to_local` on your console. +1. Disable `object_storage` for LFS objects in `gitlab.rb`. Remember to restart GitLab afterwards. + +## Storage statistics + +You can see the total storage used for LFS objects on groups and projects +in the administration area, as well as through the [groups](../../api/groups.md) +and [projects APIs](../../api/projects.md). + +## Troubleshooting: `Google::Apis::TransmissionError: execution expired` + +If LFS integration is configred with Google Cloud Storage and background uploads (`background_upload: true` and `direct_upload: false`), +Sidekiq workers may encouter this error. This is because the uploading timed out with very large files. +LFS files up to 6Gb can be uploaded without any extra steps, otherwise you need to use the following workaround. + +```shell +$ sudo gitlab-rails console # Login to rails console + +> # Set up timeouts. 20 minutes is enough to upload 30GB LFS files. +> # These settings are only in effect for the same session, i.e. they are not effective for sidekiq workers. +> ::Google::Apis::ClientOptions.default.open_timeout_sec = 1200 +> ::Google::Apis::ClientOptions.default.read_timeout_sec = 1200 +> ::Google::Apis::ClientOptions.default.send_timeout_sec = 1200 + +> # Upload LFS files manually. This process does not use sidekiq at all. +> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| +> lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? +> end +``` + +See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/19581) + +## Known limitations + +- Support for removing unreferenced LFS objects was added in 8.14 onwards. +- LFS authentications via SSH was added with GitLab 8.12. +- Only compatible with the Git LFS client versions 1.1.0 and up, or 1.0.2. +- The storage statistics currently count each LFS object multiple times for + every project linking to it. + +[reconfigure gitlab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: ../restart_gitlab.md#installations-from-source "How to restart GitLab" +[eep]: https://about.gitlab.com/pricing/ "GitLab Premium" +[ee-2760]: https://gitlab.com/gitlab-org/gitlab/merge_requests/2760 diff --git a/doc/administration/lfs/manage_large_binaries_with_git_lfs.md b/doc/administration/lfs/manage_large_binaries_with_git_lfs.md new file mode 100644 index 00000000000..1fd3077ecb9 --- /dev/null +++ b/doc/administration/lfs/manage_large_binaries_with_git_lfs.md @@ -0,0 +1,266 @@ +--- +disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html' +--- + +# Git LFS + +Managing large files such as audio, video and graphics files has always been one +of the shortcomings of Git. The general recommendation is to not have Git repositories +larger than 1GB to preserve performance. + + + +An LFS icon is shown on files tracked by Git LFS to denote if a file is stored +as a blob or as an LFS pointer. + +## How it works + +Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication +to authorize client requests. Once the request is authorized, Git LFS client receives +instructions from where to fetch or where to push the large file. + +## GitLab server configuration + +Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md). + +## Requirements + +- Git LFS is supported in GitLab starting with version 8.2 +- Git LFS must be enabled under project settings +- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up + +## Known limitations + +- Git LFS v1 original API is not supported since it was deprecated early in LFS + development +- When SSH is set as a remote, Git LFS objects still go through HTTPS +- Any Git LFS request will ask for HTTPS credentials to be provided so a good Git + credentials store is recommended +- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have + to add the URL to Git config manually (see [troubleshooting](#troubleshooting)) + +NOTE: **Note:** +With 8.12 GitLab added LFS support to SSH. The Git LFS communication +still goes over HTTP, but now the SSH client passes the correct credentials +to the Git LFS client, so no action is required by the user. + +## Using Git LFS + +Lets take a look at the workflow when you need to check large files into your Git +repository with Git LFS. For example, if you want to upload a very large file and +check it into your Git repository: + +```bash +git clone git@gitlab.example.com:group/project.git +git lfs install # initialize the Git LFS project +git lfs track "*.iso" # select the file extensions that you want to treat as large files +``` + +Once a certain file extension is marked for tracking as a LFS object you can use +Git as usual without having to redo the command to track a file with the same extension: + +```bash +cp ~/tmp/debian.iso ./ # copy a large file into the current directory +git add . # add the large file to the project +git commit -am "Added Debian iso" # commit the file meta data +git push origin master # sync the git repo and large file to the GitLab server +``` + +**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git +LFS will not be working properly for people cloning the project: + +```bash +git add .gitattributes +``` + +Cloning the repository works the same as before. Git automatically detects the +LFS-tracked files and clones them via HTTP. If you performed the `git clone` +command with a SSH URL, you have to enter your GitLab credentials for HTTP +authentication. + +```bash +git clone git@gitlab.example.com:group/project.git +``` + +If you already cloned the repository and you want to get the latest LFS object +that are on the remote repository, eg. for a branch from origin: + +```bash +git lfs fetch origin master +``` + +### Migrate an existing repo to Git LFS + +Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md). + +## File Locking + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5. + +The first thing to do before using File Locking is to tell Git LFS which +kind of files are lockable. The following command will store PNG files +in LFS and flag them as lockable: + +```bash +git lfs track "*.png" --lockable +``` + +After executing the above command a file named `.gitattributes` will be +created or updated with the following content: + +```bash +*.png filter=lfs diff=lfs merge=lfs -text lockable +``` + +You can also register a file type as lockable without using LFS +(In order to be able to lock/unlock a file you need a remote server that implements the LFS File Locking API), +in order to do that you can edit the `.gitattributes` file manually: + +```bash +*.pdf lockable +``` + +After a file type has been registered as lockable, Git LFS will make +them readonly on the file system automatically. This means you will +need to lock the file before editing it. + +### Managing Locked Files + +Once you're ready to edit your file you need to lock it first: + +```bash +git lfs lock images/banner.png +Locked images/banner.png +``` + +This will register the file as locked in your name on the server: + +```bash +git lfs locks +images/banner.png joe ID:123 +``` + +Once you have pushed your changes, you can unlock the file so others can +also edit it: + +```bash +git lfs unlock images/banner.png +``` + +You can also unlock by id: + +```bash +git lfs unlock --id=123 +``` + +If for some reason you need to unlock a file that was not locked by you, +you can use the `--force` flag as long as you have a `maintainer` access on +the project: + +```bash +git lfs unlock --id=123 --force +``` + +## Troubleshooting + +### error: Repository or object not found + +There are a couple of reasons why this error can occur: + +- You don't have permissions to access certain LFS object + +Check if you have permissions to push to the project or fetch from the project. + +- Project is not allowed to access the LFS object + +LFS object you are trying to push to the project or fetch from the project is not +available to the project anymore. Probably the object was removed from the server. + +- Local Git repository is using deprecated LFS API + +### Invalid status for `<url>` : 501 + +Git LFS will log the failures into a log file. +To view this log file, while in project directory: + +```bash +git lfs logs last +``` + +If the status `error 501` is shown, it is because: + +- Git LFS is not enabled in project settings. Check your project settings and + enable Git LFS. + +- Git LFS support is not enabled on the GitLab server. Check with your GitLab + administrator why Git LFS is not enabled on the server. See + [LFS administration documentation](lfs_administration.md) for instructions + on how to enable LFS support. + +- Git LFS client version is not supported by GitLab server. Check your Git LFS + version with `git lfs version`. Check the Git config of the project for traces + of deprecated API with `git lfs -l`. If `batch = false` is set in the config, + remove the line and try to update your Git LFS client. Only version 1.0.1 and + newer are supported. + +### getsockopt: connection refused + +If you push a LFS object to a project and you receive an error similar to: +`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`, +the LFS client is trying to reach GitLab through HTTPS. However, your GitLab +instance is being served on HTTP. + +This behaviour is caused by Git LFS using HTTPS connections by default when a +`lfsurl` is not set in the Git config. + +To prevent this from happening, set the lfs url in project Git config: + +```bash +git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs" +``` + +### Credentials are always required when pushing an object + +NOTE: **Note:** +With 8.12 GitLab added LFS support to SSH. The Git LFS communication +still goes over HTTP, but now the SSH client passes the correct credentials +to the Git LFS client, so no action is required by the user. + +Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing +the LFS object on every push for every object, user HTTPS credentials are required. + +By default, Git has support for remembering the credentials for each repository +you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials). + +For example, you can tell Git to remember the password for a period of time in +which you expect to push the objects: + +```bash +git config --global credential.helper 'cache --timeout=3600' +``` + +This will remember the credentials for an hour after which Git operations will +require re-authentication. + +If you are using OS X you can use `osxkeychain` to store and encrypt your credentials. +For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases). + +More details about various methods of storing the user credentials can be found +on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). + +### LFS objects are missing on push + +GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab. + +Verify that LFS in installed locally and consider a manual push with `git lfs push --all`. + +If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../api/projects.md#edit-project). + +### Hosting LFS objects externally + +It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`. + +You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store, +GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled. + +To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS). diff --git a/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md new file mode 100644 index 00000000000..cf798472d62 --- /dev/null +++ b/doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md @@ -0,0 +1,254 @@ +# Migration guide from Git Annex to Git LFS + +>**Note:** +Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise +Edition 9.0 (2017/03/22). + +Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git. + +## History + +Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time +where Git LFS didn't yet exist. A few months later, GitLab brought support for +Git LFS in [GitLab 8.2][post-2] and is available for both Community and +Enterprise editions. + +## Differences between Git Annex and Git LFS + +Some items below are general differences between the two protocols and some are +ones that GitLab developed. + +- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS + (SSH support was added in GitLab 8.12). +- Annex files are stored in a sub-directory of the normal repositories, whereas + LFS files are stored outside of the repositories in a place you can define. +- Git Annex requires a more complex setup, but has much more options than Git + LFS. You can compare the commands each one offers by running `man git-annex` + and `man git-lfs`. +- Annex files cannot be browsed directly in GitLab's interface, whereas LFS + files can. + +## Migration steps + +>**Note:** +Since Git Annex files are stored in a sub-directory of the normal repositories +(`.git/annex/objects`) and LFS files are stored outside of the repositories, +they are not compatible as they are using a different scheme. Therefore, the +migration has to be done manually per repository. + +There are basically two steps you need to take in order to migrate from Git +Annex to Git LFS. + +### TL; DR + +If you know what you are doing and want to skip the reading, this is what you +need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab-git-annex) in your +repository and that you have made backups in case something goes wrong). +Fire up a terminal, navigate to your Git repository and: + +1. Disable `git-annex`: + + ```bash + git annex sync --content + git annex direct + git annex uninit + git annex indirect + ``` + +1. Enable `git-lfs`: + + ``` + git lfs install + git lfs track <files> + git add . + git commit -m "commit message" + git push + ``` + +### Disabling Git Annex in your repo + +Before changing anything, make sure you have a backup of your repository first. +There are a couple of ways to do that, but you can simply clone it to another +local path and maybe push it to GitLab if you want a remote backup as well. +Here you'll find a guide on +[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive]. + +Since Annex files are stored as objects with symlinks and cannot be directly +modified, we need to first remove those symlinks. + +NOTE: **Note:** +Make sure the you read about the [`direct` mode][annex-direct] as it contains +useful information that may fit in your use case. Note that `annex direct` is +deprecated in Git Annex version 6, so you may need to upgrade your repository +if the server also has Git Annex 6 installed. Read more in the +[Git Annex troubleshooting tips](../git_annex.md#troubleshooting-tips) section. + +1. Backup your repository + + ```bash + cd repository + git annex sync --content + cd .. + git clone repository repository-backup + cd repository-backup + git annex get + cd .. + ``` + +1. Use `annex direct`: + + ```bash + cd repository + git annex direct + ``` + + The output should be similar to this: + + ```bash + commit + On branch master + Your branch is up-to-date with 'origin/master'. + nothing to commit, working tree clean + ok + direct debian.iso ok + direct ok + ``` + +1. Disable Git Annex with [`annex uninit`][uninit]: + + ```bash + git annex uninit + ``` + + The output should be similar to this: + + ```bash + unannex debian.iso ok + Deleted branch git-annex (was 2534d2c). + ``` + + This will `unannex` every file in the repository, leaving the original files. + +1. Switch back to `indirect` mode: + + ```bash + git annex indirect + ``` + + The output should be similar to this: + + ```bash + (merging origin/git-annex into git-annex...) + (recording state in git...) + commit (recording state in git...) + + ok + (recording state in git...) + [master fac3194] commit before switching to indirect mode + 1 file changed, 1 deletion(-) + delete mode 120000 alpine-virt-3.4.4-x86_64.iso + ok + indirect ok + ok + ``` + +--- + +At this point, you have two options. Either add, commit and push the files +directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in +the next section. + +### Enabling Git LFS in your repo + +Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE, +GitLab.com), therefore, you don't need to do anything server-side. + +1. First, make sure you have `git-lfs` installed locally: + + ```bash + git lfs help + ``` + + If the terminal doesn't prompt you with a full response on `git-lfs` commands, + [install the Git LFS client][install-lfs] first. + +1. Inside the repo, run the following command to initiate LFS: + + ```bash + git lfs install + ``` + +1. Enable `git-lfs` for the group of files you want to track. You + can track specific files, all files containing the same extension, or an + entire directory: + + ```bash + git lfs track images/01.png # per file + git lfs track **/*.png # per extension + git lfs track images/ # per directory + ``` + + Once you do that, run `git status` and you'll see `.gitattributes` added + to your repo. It collects all file patterns that you chose to track via + `git-lfs`. + +1. Add the files, commit and push them to GitLab: + + ```bash + git add . + git commit -m "commit message" + git push + ``` + + If your remote is set up with HTTP, you will be asked to enter your login + credentials. If you have [2FA enabled](../../user/profile/account/two_factor_authentication.md), make sure to use a + [personal access token](../../user/profile/account/two_factor_authentication.md#personal-access-tokens) + instead of your password. + +## Removing the Git Annex branches + +After the migration finishes successfully, you can remove all `git-annex` +related branches from your repository. + +On GitLab, navigate to your project's **Repository ➔ Branches** and delete all +branches created by Git Annex: `git-annex`, and all under `synced/`. + + + +You can also do this on the command line with: + +```bash +git branch -d synced/master +git branch -d synced/git-annex +git push origin :synced/master +git push origin :synced/git-annex +git push origin :git-annex +git remote prune origin +``` + +If there are still some Annex objects inside your repository (`.git/annex/`) +or references inside `.git/config`, run `annex uninit` again: + +```bash +git annex uninit +``` + +## Further Reading + +- (Blog Post) [Getting Started with Git FLS][post-1] +- (Blog Post) [Announcing LFS Support in GitLab][post-2] +- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3] +- (GitLab Docs) [Git Annex](../git_annex.md) +- (GitLab Docs) [Git LFS](manage_large_binaries_with_git_lfs.md) + +[annex-direct]: https://git-annex.branchable.com/direct_mode/ +[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive +[Git Annex]: http://git-annex.branchable.com/ +[Git LFS]: https://git-lfs.github.com/ +[install-lfs]: https://git-lfs.github.com/ +[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab/issues/1648 +[lfs-track]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs +[post-1]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/ +[post-2]: https://about.gitlab.com/blog/2015/11/23/announcing-git-lfs-support-in-gitlab/ +[post-3]: https://about.gitlab.com/blog/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ +[uninit]: https://git-annex.branchable.com/git-annex-uninit/ diff --git a/doc/administration/logs.md b/doc/administration/logs.md index dae0dae8395..aa10cdd220c 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -42,6 +42,48 @@ User clone/fetch activity using http transport appears in this log as `action: g In addition, the log contains the IP address from which the request originated (`remote_ip`) as well as the user's ID (`user_id`), and username (`username`). +NOTE: **Note:** Starting with GitLab 12.5, if an error occurs, an +`exception` field is included with `class`, `message`, and +`backtrace`. Previous versions included an `error` field instead of +`exception.class` and `exception.message`. For example: + +```json +{ + "method": "GET", + "path": "/admin", + "format": "html", + "controller": "Admin::DashboardController", + "action": "index", + "status": 500, + "duration": 2584.11, + "view": 0, + "db": 9.21, + "time": "2019-11-14T13:12:46.156Z", + "params": [], + "remote_ip": "127.0.0.1", + "user_id": 1, + "username": "root", + "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0", + "queue_duration": 274.35, + "correlation_id": "KjDVUhNvvV3", + "cpu_s": 2.837645135999999, + "exception": { + "class": "NameError", + "message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>", + "backtrace": [ + "app/controllers/admin/dashboard_controller.rb:11:in `index'", + "ee/app/controllers/ee/admin/dashboard_controller.rb:14:in `index'", + "ee/lib/gitlab/ip_address_state.rb:10:in `with'", + "ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'", + "lib/gitlab/session.rb:11:in `with_session'", + "app/controllers/application_controller.rb:450:in `set_session_storage'", + "app/controllers/application_controller.rb:444:in `set_locale'", + "ee/lib/gitlab/jira/middleware.rb:19:in `call'" + ] + } +} +``` + ## `production.log` This file lives in `/var/log/gitlab/gitlab-rails/production.log` for diff --git a/doc/administration/monitoring/gitlab_instance_administration_project/index.md b/doc/administration/monitoring/gitlab_instance_administration_project/index.md index bb76ad59e3b..b07bbafaf7d 100644 --- a/doc/administration/monitoring/gitlab_instance_administration_project/index.md +++ b/doc/administration/monitoring/gitlab_instance_administration_project/index.md @@ -1,6 +1,7 @@ # GitLab instance administration project -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/56883) in GitLab 12.2. +NOTE: **Note:** +This feature is not yet available and is [planned for 12.6](https://gitlab.com/gitlab-org/gitlab/issues/32351). GitLab has been adding the ability for administrators to see insights into the health of their GitLab instance. In order to surface this experience in a native way, similar to how diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differindex d1187fd879a..acad60f863e 100644 --- a/doc/administration/monitoring/performance/img/performance_bar.png +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 53c08e32cb2..a52b6227e14 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -8,14 +8,17 @@ activated, it looks as follows: It allows you to see (from left to right): - the current host serving the page -- time taken and number of DB queries, click through for details of these queries +- time taken and number of DB queries; click through for details of these queries  -- time taken and number of [Gitaly] calls, click through for details of these calls +- time taken and number of [Gitaly] calls; click through for details of these calls  -- time taken and number of [Rugged] calls, click through for details of these calls +- time taken and number of [Rugged] calls; click through for details of these calls  -- time taken and number of Redis calls, click through for details of these calls +- time taken and number of Redis calls; click through for details of these calls  +- a link to add a request's details to the performance bar; the request can be + added by its full URL (authenticated as the current user), or by the value of + its `X-Request-Id` header On the far right is a request selector that allows you to view the same metrics (excluding the page timing and line profiler) for any requests made while the @@ -51,7 +54,7 @@ Make sure _Enable the Performance Bar_ is checked and hit **Save** to save the changes. Once the Performance Bar is enabled, you will need to press the [<kbd>p</kbd> + -<kbd>b</kbd> keyboard shortcut](../../../workflow/shortcuts.md) to actually +<kbd>b</kbd> keyboard shortcut](../../../user/shortcuts.md) to actually display it. You can toggle the Bar using the same shortcut. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index c35d6f505be..c0b563bd76e 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -78,6 +78,31 @@ To change the address/port that Prometheus listens on: 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect +### Adding custom scrape configs + +You can configure additional scrape targets for the GitLab Omnibus-bundled +Prometheus by editing `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` +using the [Prometheus scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E) +syntax. + +Here is an example configuration to scrape `http://1.1.1.1:8060/probe?param_a=test¶m_b=additional_test`: + +```ruby +prometheus['scrape_configs'] = [ + { + 'job_name': 'custom-scrape', + 'metrics_path': '/probe', + 'params' => { + 'param_a' => ['test'], + 'param_b' => ['additional_test'] + }, + 'static_configs' => [ + 'targets' => ['1.1.1.1:8060'], + ], + }, +] +``` + ### Using an external Prometheus server NOTE: **Note:** diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index df208b3f427..1d80e23eecf 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -21,3 +21,6 @@ Keep your GitLab instance up and running smoothly. performance can have a big impact on GitLab performance, especially for actions that read or write Git repositories. This information will help benchmark filesystem performance against known good and bad real-world systems. +- [ChatOps Scripts](https://gitlab.com/gitlab-com/chatops): The GitLab.com Infrastructure team uses this repository to house + common ChatOps scripts they use to troubleshoot and maintain the production instance of GitLab.com. + These scripts are likely useful to administrators of GitLab instances of all sizes. diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index 79e9fb778b6..6438dbb9dab 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -34,7 +34,7 @@ The MemoryKiller is controlled using environment variables. In _daemon_ mode, the MemoryKiller checks the Sidekiq process RSS every 3 seconds (defined by `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL`). -- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is greater +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS` (KB): if this variable is set, and its value is greater than 0, the MemoryKiller is enabled. Otherwise the MemoryKiller is disabled. `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defines the Sidekiq process allowed RSS. @@ -52,7 +52,7 @@ The MemoryKiller is controlled using environment variables. [in the Omnibus GitLab repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). -- `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`: is used by _daemon_ mode. If the Sidekiq +- `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS` (KB): is used by _daemon_ mode. If the Sidekiq process RSS (expressed in kilobytes) exceeds `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS`, an immediate graceful restart of Sidekiq is triggered. diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index bf86a549fda..a62e3ab603d 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -18,7 +18,9 @@ You can read more about the Docker Registry at **Omnibus GitLab installations** -All you have to do is configure the domain name under which the Container +If you are using the Omnibus GitLab built in [Let's Encrypt integration](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration), as of GitLab 12.5, the Container Registry will be automatically enabled on port 5050 of the default domain. + +If you would like to use a separate domain, all you have to do is configure the domain name under which the Container Registry will listen to. Read [#container-registry-domain-configuration](#container-registry-domain-configuration) and pick one of the two options that fits your case. @@ -353,7 +355,7 @@ configuration. NOTE: **Note:** Enabling a storage driver other than `filesystem` would mean that your Docker client needs to be able to access the storage backend directly. -In that case, you must use an address that resolves and is accessible outside GitLab server. +In that case, you must use an address that resolves and is accessible outside GitLab server. The Docker client will continue to authenticate via GitLab but data transfer will be direct to and from the storage backend. The different supported drivers are: @@ -877,6 +879,6 @@ The above image shows: - The HEAD request to the AWS bucket reported a 403 Unauthorized. What does this mean? This strongly suggests that the S3 user does not have the right -[permissions to perform a HEAD request](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html). +[permissions to perform a HEAD request](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). Once the right permissions were set, the error will go away. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index cacfb73451c..f51c375860b 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -120,7 +120,7 @@ The Pages daemon doesn't listen to the outside world. 1. Set the external URL for GitLab Pages in `/etc/gitlab/gitlab.rb`: - ```shell + ```ruby pages_external_url 'http://example.io' ``` @@ -145,7 +145,7 @@ outside world. 1. Place the certificate and key inside `/etc/gitlab/ssl` 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: - ```shell + ```ruby pages_external_url 'https://example.io' pages_nginx['redirect_http_to_https'] = true @@ -167,7 +167,7 @@ behavior: 1. Edit `/etc/gitlab/gitlab.rb`. 1. Set the `inplace_chroot` to `true` for GitLab Pages: - ```shell + ```ruby gitlab_pages['inplace_chroot'] = true ``` @@ -202,7 +202,7 @@ world. Custom domains are supported, but no TLS. 1. Edit `/etc/gitlab/gitlab.rb`: - ```shell + ```ruby pages_external_url "http://example.io" nginx['listen_addresses'] = ['192.0.2.1'] pages_nginx['enable'] = false @@ -233,7 +233,7 @@ world. Custom domains and TLS are supported. 1. Edit `/etc/gitlab/gitlab.rb`: - ```shell + ```ruby pages_external_url "https://example.io" nginx['listen_addresses'] = ['192.0.2.1'] pages_nginx['enable'] = false @@ -271,7 +271,7 @@ sites served under a custom domain. To enable it, you'll need to: -1. Choose an email on which you will recieve notifications about expiring domains. +1. Choose an email on which you will receive notifications about expiring domains. 1. Navigate to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. 1. Enter the email for receiving notifications and accept Let's Encrypt's Terms of Service as shown below. 1. Click **Save changes**. @@ -332,7 +332,7 @@ Follow the steps below to configure verbose logging of GitLab Pages daemon. If you wish to make it log events with level `DEBUG` you must configure this in `/etc/gitlab/gitlab.rb`: - ```shell + ```ruby gitlab_pages['log_verbose'] = true ``` @@ -347,7 +347,7 @@ are stored. If you wish to store them in another location you must set it up in `/etc/gitlab/gitlab.rb`: - ```shell + ```ruby gitlab_rails['pages_path'] = "/mnt/storage/pages" ``` @@ -363,14 +363,14 @@ Omnibus GitLab 11.1. If you wish to disable it you must configure this in `/etc/gitlab/gitlab.rb`: - ```shell + ```ruby gitlab_pages['listen_proxy'] = nil ``` If you wish to make it listen on a different port you must configure this also in `/etc/gitlab/gitlab.rb`: - ```shell + ```ruby gitlab_pages['listen_proxy'] = "localhost:10080" ``` @@ -382,21 +382,26 @@ The maximum size of the unpacked archive per project can be configured in the Admin area under the Application settings in the **Maximum size of pages (MB)**. The default is 100MB. -## Running GitLab Pages in a separate server +## Running GitLab Pages on a separate server -You may want to run GitLab Pages daemon on a separate server in order to decrease the load on your main application server. -Follow the steps below to configure GitLab Pages in a separate server. +You can run the GitLab Pages daemon on a separate server in order to decrease the load on your main application server. -1. Suppose you have the main GitLab application server named `app1`. Prepare - new Linux server (let's call it `app2`), create NFS share there and configure access to - this share from `app1`. Let's use the default GitLab Pages folder `/var/opt/gitlab/gitlab-rails/shared/pages` - as the shared folder on `app2` and mount it to `/mnt/pages` on `app1`. +To configure GitLab Pages on a separate server: -1. On `app2` install GitLab omnibus and modify `/etc/gitlab/gitlab.rb` this way: +1. Set up a new server. This will become the **Pages server**. - ```shell +1. Create an NFS share on the new server and configure this share to + allow access from your main **GitLab server**. For this example, we use the + default GitLab Pages folder `/var/opt/gitlab/gitlab-rails/shared/pages` + as the shared folder on the new server and we will mount it to `/mnt/pages` + on the **GitLab server**. + +1. On the **Pages server**, install Omnibus GitLab and modify `/etc/gitlab/gitlab.rb` + to include: + + ```ruby external_url 'http://<ip-address-of-the-server>' - pages_external_url "http://<your-pages-domain>" + pages_external_url "http://<your-pages-server-URL>" postgresql['enable'] = false redis['enable'] = false prometheus['enable'] = false @@ -409,20 +414,82 @@ Follow the steps below to configure GitLab Pages in a separate server. gitlab_rails['auto_migrate'] = false ``` -1. Run `sudo gitlab-ctl reconfigure`. -1. On `app1` apply the following changes to `/etc/gitlab/gitlab.rb`: +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - ```shell +1. On the **GitLab server**, make the following changes to `/etc/gitlab/gitlab.rb`: + + ```ruby gitlab_pages['enable'] = false - pages_external_url "http://<your-pages-domain>" + pages_external_url "http://<your-pages-server-URL>" gitlab_rails['pages_path'] = "/mnt/pages" ``` -1. Run `sudo gitlab-ctl reconfigure`. +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +It is possible to run GitLab Pages on multiple servers if you wish to distribute +the load. You can do this through standard load balancing practices such as +configuring your DNS server to return multiple IPs for your Pages server, +configuring a load balancer to work at the IP level, and so on. If you wish to +set up GitLab Pages on multiple servers, perform the above procedure for each +Pages server. + +### Access control when running GitLab Pages on a separate server + +If you are [running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server), +then you must use the following procedure to configure [access control](#access-control): + +1. On the **GitLab server**, add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['enable'] = true + gitlab_pages['access_control'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the + changes to take effect. The `gitlab-secrets.json` file is now updated with the + new configuration. + + DANGER: **Danger:** + The `gitlab-secrets.json` file contains secrets that control database encryption. + Do not edit or replace this file on the **GitLab server** or you might + experience permanent data loss. Make a backup copy of this file before proceeding, + as explained in the following steps. + +1. Create a backup of the secrets file on the **GitLab server**: + + ```shell + cp /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak + ``` + +1. Create a backup of the secrets file on the **Pages server**: + + ```shell + cp /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak + ``` + +1. Disable Pages on the **GitLab server** by setting the following in + `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['enable'] = false + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the **GitLab server** + to the **Pages server**. + +1. On your **Pages server**, add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['gitlab_server'] = "https://<your-gitlab-server-URL>" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ## Backup -Pages are part of the [regular backup][backup] so there is nothing to configure. +GitLab Pages are part of the [regular backup][backup], so there is no separate backup to configure. ## Security diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 7d3e36e9796..86998280b93 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -2,8 +2,8 @@ > [Introduced][ce-4578] in GitLab 8.10. -GitLab allows you to define multiple repository storage paths to distribute the -storage load between several mount points. +GitLab allows you to define multiple repository storage paths (sometimes called +storage shards) to distribute the storage load between several mount points. > **Notes:** > diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 227d6928baf..9c7b5bc6b87 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -5,8 +5,8 @@ Two different storage layouts can be used to store the repositories on disk and their characteristics. -GitLab can be configured to use one or multiple repository shard locations -that can be: +GitLab can be configured to use one or multiple repository storage paths/shard +locations that can be: - Mounted to the local disk - Exposed as an NFS shared volume @@ -34,8 +34,8 @@ easy for Administrators to find where the repository is stored. On the other hand this has some drawbacks: Storage location will concentrate huge amount of top-level namespaces. The -impact can be reduced by the introduction of [multiple storage -paths][storage-paths]. +impact can be reduced by the introduction of +[multiple storage paths](repository_storage_paths.md). Because backups are a snapshot of the same URL mapping, if you try to recover a very old backup, you need to verify whether any project has taken the place of @@ -183,7 +183,8 @@ CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in ##### LFS Objects -LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following Git own implementation: +[LFS Objects in GitLab](lfs/manage_large_binaries_with_git_lfs.md) implement a similar +storage pattern using 2 chars, 2 level folders, following Git's own implementation: ```ruby "shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}" @@ -192,10 +193,9 @@ LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, "shared/lfs-objects/89/09/029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c" ``` -They are also S3 compatible since **10.0** (GitLab Premium), and available in GitLab Core since **10.7**. +LFS objects are also [S3 compatible](lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). [ce-2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 [ce-28283]: https://gitlab.com/gitlab-org/gitlab-foss/issues/28283 [rake/migrate-to-hashed]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage -[storage-paths]: repository_storage_types.md [gitaly]: gitaly/index.md diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md new file mode 100644 index 00000000000..3594ba19181 --- /dev/null +++ b/doc/administration/timezone.md @@ -0,0 +1,37 @@ +# Changing your time zone + +The global time zone configuration parameter can be changed in `config/gitlab.yml`: + +```text +# time_zone: 'UTC' +``` + +Uncomment and customize if you want to change the default time zone of the GitLab application. + +## Viewing available timezones + +To see all available time zones, run `bundle exec rake time:zones:all`. + +For Omnibus installations, run `gitlab-rake time:zones:all`. + +NOTE: **Note:** +Currently, this rake task does not list timezones in TZInfo format required by GitLab Omnibus during a reconfigure: [#58672](https://gitlab.com/gitlab-org/gitlab-foss/issues/58672). + +## Changing time zone in Omnibus installations + +GitLab defaults its time zone to UTC. It has a global timezone configuration parameter in `/etc/gitlab/gitlab.rb`. + +To obtain a list of timezones, log in to your GitLab application server and run a command that generates a list of timezones in TZInfo format for the server. For example, install `timedatectl` and run `timedatectl list-timezones`. + +To update, add the timezone that best applies to your location. For example: + +```ruby +gitlab_rails['time_zone'] = 'America/New_York' +``` + +After adding the configuration parameter, reconfigure and restart your GitLab instance: + +```sh +gitlab-ctl reconfigure +gitlab-ctl restart +``` diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 34a5acbe7b7..dd220d0871d 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -442,7 +442,7 @@ personal_access_token = User.find(123).personal_access_tokens.create( scopes: [:api] ) -personal_access_token.token +puts personal_access_token.token ``` You might also want to manually set the token string: @@ -715,7 +715,7 @@ For more information, see the [confidential issue](../../user/project/issues/con ```ruby Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count -Ci::Pipeline.where(project_id: p.id).where(status: 'pending').each {|p| p.cancel} +Ci::Pipeline.where(project_id: p.id).where(status: 'pending').each {|p| p.cancel if p.stuck?} Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count ``` diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 04cebe568f3..bccfe7c542f 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -63,8 +63,8 @@ For source installations the following settings are nested under `uploads:` and |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Uploads will be stored| | -| `direct_upload` | Set to true to enable direct upload of Uploads without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `direct_upload` | Set to true to remove Unicorn from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Unicorn does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [what the direct_upload setting means](https://docs.gitlab.com/ee/development/uploads.html#what-does-the-direct_upload-setting-mean). | `false` | +| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 (if `direct_upload` is set to `true` it will override `background_upload`) | `true` | | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | |
