diff options
Diffstat (limited to 'doc')
72 files changed, 926 insertions, 183 deletions
diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 423a79c56d8..54279897e04 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -30,9 +30,9 @@ the LDAP server. ### User deletion -If a user is deleted from the LDAP server, they will be blocked in GitLab, as +If a user is deleted from the LDAP server, they will be blocked in GitLab as well. Users will be immediately blocked from logging in. However, there is an -LDAP check cache time (sync time) of one hour (see note). This means users that +LDAP check cache time of one hour (see note) which means users that are already logged in or are using Git over SSH will still be able to access GitLab for up to one hour. Manually block the user in the GitLab Admin area to immediately block all access. diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 7d2fd51f834..c7299b6e196 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -157,6 +157,42 @@ For wikis: sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production ``` +## Reconcile differences with checksum mismatches + +If the **primary** and **secondary** nodes have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: + +1. Navigate to the **Admin Area > Projects** dashboard on the **primary** node, find the + project that you want to check the checksum differences and click on the + **Edit** button: +  + +1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**: +  + +1. Navigate to the project's repository directory on both **primary** and **secondary** nodes. For an installation from source, the path is usually `/home/git/repositories`. For Omnibus installs, the path is usually `/var/opt/gitlab/git-data/repositories`. Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure. + + ```sh + cd /var/opt/gitlab/git-data/repositories + ``` + +1. Run the following command on the **primary** node, redirecting the output to a file: + + ```sh + git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > primary-node-refs + ``` + +1. Run the following command on the **secondary** node, redirecting the output to a file: + + ```sh + git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > secondary-node-refs + ``` + +1. Copy the files from the previous steps on the same system, and do a diff between the contents: + + ```sh + diff primary-node-refs secondary-node-refs + ``` + ## Current limitations Until [issue #5064][ee-5064] is completed, background verification doesn't cover diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png Binary files differnew file mode 100644 index 00000000000..fd51523104b --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png Binary files differnew file mode 100644 index 00000000000..b2a6da69d3d --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index de61c01991b..24db1c28778 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -145,14 +145,38 @@ environments this is a good architecture to consider if you foresee or do have contention due to certain workloads. - 3 PostgreSQL nodes +- 1 PgBouncer node - 2 Redis nodes - 3 Consul/Sentinel nodes - 2 or more Sidekiq nodes -- 2 or more Web nodes (Unicorn, Workhorse, PGBouncer) +- 2 or more GitLab application nodes (Unicorn, Workhorse) - 1 or more NFS/Gitaly servers +- 1 Monitoring node (Prometheus, Grafana)  +#### Reference Architecture + +- **Status:** Work-in-progress +- **Supported Users (approximate):** 10,000 +- **Related Issues:** [gitlab-com/support/support-team-meta#1513](https://gitlab.com/gitlab-com/support/support-team-meta/issues/1513), + [gitlab-org/quality/team-tasks#110](https://gitlab.com/gitlab-org/quality/team-tasks/issues/110) + +The Support and Quality teams are in the process of building and performance testing +an environment that will support about 10,000 users. The specifications below +are a work-in-progress representation of the work so far. Quality will be +certifying this environment in FY20-Q2. The specifications may be adjusted +prior to certification based on performance testing. + +- 3 PostgreSQL - 4 CPU, 8GB RAM +- 1 PgBouncer - 2 CPU, 4GB RAM +- 2 Redis - 2 CPU, 8GB RAM +- 3 Consul/Sentinel - 2 CPU, 2GB RAM +- 4 Sidekiq - 4 CPU, 8GB RAM +- 5 GitLab application nodes - 20 CPU, 64GB RAM +- 1 Gitaly - 20 CPU, 64GB RAM +- 1 Monitoring node - 4 CPU, 8GB RAM + ### Fully Distributed This architecture scales to hundreds of thousands of users and projects and is diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index d0e0e320019..caadec3ac4e 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -41,7 +41,6 @@ options: NOTE: **Note:** This is only available starting in certain versions of GitLab: - * 11.5.11 * 11.6.11 * 11.7.12 @@ -59,13 +58,13 @@ details. To do this, run the Rake task: ```sh -gitlab-rake gitlab:features:enable_rugged +sudo gitlab-rake gitlab:features:enable_rugged ``` If you need to undo this setting for some reason, run: ```sh -gitlab-rake gitlab:features:disable_rugged +sudo gitlab-rake gitlab:features:disable_rugged ``` ### Known issues diff --git a/doc/administration/index.md b/doc/administration/index.md index 02e88dbd2a6..797a7242bd0 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -40,6 +40,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **[PREMIUM ONLY]** - [Disaster Recovery](https://docs.gitlab.com/ee/administration/geo/disaster_recovery/index.html): Quickly fail-over to a different site with minimal effort in a disaster situation. **[PREMIUM ONLY]** - [Pivotal Tile](https://docs.gitlab.com/ee/install/pivotal/index.html): Deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **[PREMIUM ONLY]** +- [Add License](../user/admin_area/license.md): Upload a license at install time to unlock features that are in paid tiers of GitLab. **[STARTER ONLY]** ### Configuring GitLab diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index d383d1efe70..b7b820abb40 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -57,8 +57,7 @@ you can change these defaults by editing the `/etc/tomcat7/server.xml` file. You need to enable PlantUML integration from Settings under Admin Area. To do that, login with an Admin account and do following: - - in GitLab go to **Admin Area** and then **Settings** - - scroll to bottom of the page until PlantUML section + - in GitLab go to **Admin Area**->**Settings**->**Integrations**->**PlantUML** - check **Enable PlantUML** checkbox - set the PlantUML instance as **PlantUML URL** diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index c243dd9edbb..48bd709a2b7 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -43,10 +43,11 @@ The following metrics are available: | redis_ping_latency_seconds | Gauge | 9.4 | Round trip time of the redis ping | | user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | | upload_file_does_not_exist | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | -| failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | -| successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | -| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | -| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | +| failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | +| successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | +| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | +| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | +| unicorn_workers | Gauge | 11.11 | The number of Unicorn workers | ## Sidekiq Metrics available for Geo **[PREMIUM]** @@ -100,6 +101,10 @@ Some basic Ruby runtime metrics are available: | ruby_file_descriptors | Gauge | 11.1 | File descriptors per process | | ruby_memory_bytes | Gauge | 11.1 | Memory usage by process | | ruby_sampler_duration_seconds_total | Counter | 11.1 | Time spent collecting stats | +| ruby_process_cpu_seconds_total | Gauge | 11.11 | Total amount of CPU time per process | +| ruby_process_max_fds | Gauge | 11.11 | Maximum number of open file descriptors per process | +| ruby_process_resident_memory_bytes | Gauge | 11.11 | Memory usage by process, measured in bytes | +| ruby_process_start_time_seconds | Gauge | 11.11 | The elapsed time between system boot and the process started, measured in seconds | [GC.stat]: https://ruby-doc.org/core-2.3.0/GC.html#method-c-stat diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 67bbd4cc1ac..07a6201b10b 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -153,7 +153,8 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab ### Add note to existing issue discussion -Adds a new note to the discussion. +Adds a new note to the discussion. This can also +[create a discussion from a single comment](../user/discussions/#start-a-discussion-by-replying-to-a-standard-comment). ``` POST /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes @@ -652,7 +653,8 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab. ### Add note to existing merge request discussion -Adds a new note to the discussion. +Adds a new note to the discussion. This can also +[create a discussion from a single comment](../user/discussions/#start-a-discussion-by-replying-to-a-standard-comment). ``` POST /projects/:id/merge_requests/:merge_request_iid/discussions/:discussion_id/notes diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 6fcc06ea8cd..87c7f371de1 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -181,7 +181,7 @@ Currently gitlab-shell has a boolean return code, preventing GitLab from specify ## Delete existing file in repository -This allows you to delete a single file. For deleting multiple files with a singleh request see the [commits API](commits.html#create-a-commit-with-multiple-files-and-actions). +This allows you to delete a single file. For deleting multiple files with a single request, see the [commits API](commits.html#create-a-commit-with-multiple-files-and-actions). ``` DELETE /projects/:id/repository/files/:file_path diff --git a/doc/api/search.md b/doc/api/search.md index 6ee3d32d8bc..c2dd72479c1 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -556,6 +556,7 @@ GET /projects/:id/search | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `scope` | string | yes | The scope to search in | | `search` | string | yes | The search query | +| `ref` | string | no | The name of a repository branch or tag to search on. The project's default branch is used by default. This is only applicable for scopes: commits, blobs, and wiki_blobs. | Search the expression within the specified scope. Currently these scopes are supported: issues, merge_requests, milestones, notes, wiki_blobs, commits, blobs, users. @@ -850,7 +851,7 @@ Blobs searches are performed on both filenames and contents. Search results: times in the content. ```bash -curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/6/search?scope=blobs&search=installation +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/6/search?scope=blobs&search=installation&ref=feature ``` Example response: @@ -863,7 +864,7 @@ Example response: "data": "```\n\n## Installation\n\nQuick start using the [pre-built", "filename": "README.md", "id": null, - "ref": "master", + "ref": "feature", "startline": 46, "project_id": 6 } diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index e079483e2b5..3f72fe3e9eb 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -289,7 +289,7 @@ jobs inherit it. Gems are installed in `vendor/ruby/` and are cached per-branch: # # https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml # -image: ruby:2.5 +image: ruby:2.6 # Cache gems in between builds cache: diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 6b9f8181d1d..340a41c196b 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -18,30 +18,30 @@ Examples are available in several forms. As a collection of: The following table lists examples for different use cases: -| Use case | Resource | -|:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------| -| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | -| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | -| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | -| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). | -| Dependency scanning | [Dependency Scanning with GitLab CI/CD](dependency_scanning.md). **[ULTIMATE]** | -| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | -| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** | -| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | -| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | -| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | -| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | -| JUnit | [JUnit test reports](../junit_test_reports.md). | -| License management | [Dependencies license management with GitLab CI/CD](license_management.md) **[ULTIMATE]** | -| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | -| PHP | [Testing PHP projects](php.md). | -| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | -| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | -| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | -| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | -| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | -| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](sast.md) **[ULTIMATE]** | -| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | +| Use case | Resource | +|:-----------------------------------------------|:------------------------------------------------------------------------------------------------------------------------| +| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | +| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | +| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | +| Container scanning | [Container Scanning with GitLab CI/CD](../../user/application_security/container_scanning/index.md). **[ULTIMATE]** | +| Dependency scanning | [Dependency Scanning with GitLab CI/CD](../../user/application_security/dependency_scanning/index.md). **[ULTIMATE]** | +| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | +| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](../../user/application_security/dast/index.md). **[ULTIMATE]** | +| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | +| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | +| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | +| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | +| JUnit | [JUnit test reports](../junit_test_reports.md). | +| License management | [Dependencies license management with GitLab CI/CD](../../user/application_security/license_management/index.md). **[ULTIMATE]** | +| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | +| PHP | [Testing PHP projects](php.md). | +| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | +| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | +| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | +| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | +| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | +| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](../../user/application_security/sast/index.md). **[ULTIMATE]** | +| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | ### Contributing examples diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md index 99a4316ab0d..47d20a4e1c1 100644 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -9,7 +9,12 @@ You can checkout the [example source](https://gitlab.com/ayufan/python-getting-s This is what the `.gitlab-ci.yml` file looks like for this project: ```yaml +stages: + - test + - deploy + test: + stage: test script: # this configures Django application to use attached postgres database that is run on `postgres` host - export DATABASE_URL=postgres://postgres:@postgres:5432/python-test-app @@ -19,7 +24,7 @@ test: - python manage.py test staging: - type: deploy + stage: deploy script: - apt-get update -qy - apt-get install -y ruby-dev @@ -29,7 +34,7 @@ staging: - master production: - type: deploy + stage: deploy script: - apt-get update -qy - apt-get install -y ruby-dev diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index 14ea648c00b..0045fa2fb1f 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -156,7 +156,7 @@ Once you're happy with your implementation:  -GitLab CI/CD is capable of a doing a lot more, but this workflow +GitLab CI/CD is capable of doing a lot more, but this workflow exemplifies GitLab's ability to track the entire process, without the need of any external tool to deliver your software. And, most usefully, you can visualize all the steps through diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index 2902c30c7c0..0de6a93514a 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -17,8 +17,8 @@ services: variables: # Configure mysql environment variables (https://hub.docker.com/_/mysql/) - MYSQL_DATABASE: el_duderino - MYSQL_ROOT_PASSWORD: mysql_strong_password + MYSQL_DATABASE: "<your_mysql_database>" + MYSQL_ROOT_PASSWORD: "mysql_strong_password" ``` And then configure your application to use the database, for example: diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 99e4c64ff86..36b88ff12c2 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -337,6 +337,7 @@ In addition, `only` and `except` allow the use of special keywords: | `triggers` | For pipelines created using a trigger token. | | `web` | For pipelines created using **Run pipeline** button in GitLab UI (under your project's **Pipelines**). | | `merge_requests` | When a merge request is created or updated (See [pipelines for merge requests](../merge_request_pipelines/index.md)). | +| `chats` | For jobs created using a [GitLab ChatOps](../chatops/README.md) command. | In the example below, `job` will run only for refs that start with `issue-`, whereas all branches will be skipped: @@ -1446,7 +1447,7 @@ be automatically shown in merge requests. > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `sast` report collects [SAST vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html) +The `sast` report collects [SAST vulnerabilities](https://docs.gitlab.com/ee/user/application_security/sast/index.html) as artifacts. The collected SAST report will be uploaded to GitLab as an artifact and will @@ -1764,9 +1765,6 @@ TIP: **Tip:** Use merging to customize and override included CI/CD configurations with local definitions. -Recursive includes are not supported. Your external files should not use the -`include` keyword as it will be ignored. - NOTE: **Note:** Using YAML aliases across different YAML files sourced by `include` is not supported. You must only refer to aliases in the same file. Instead @@ -2078,8 +2076,8 @@ of the `group/my-project`: ```yaml include: - - local: : /templates/docker-build.yml - - local: : /templates/docker-testing.yml + - local: /templates/docker-build.yml + - local: /templates/docker-testing.yml ``` Our `/templates/docker-build.yml` present in `group/my-project` adds a `docker-build` job: diff --git a/doc/development/README.md b/doc/development/README.md index 83a1145c020..cfbbd5aaeaa 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -44,6 +44,7 @@ description: 'Learn how to contribute to GitLab.' - [`Gemfile` guidelines](gemfile.md) - [Pry debugging](pry_debugging.md) - [Sidekiq debugging](sidekiq_debugging.md) +- [Accessing session data](session.md) - [Gotchas](gotchas.md) to avoid - [Avoid modules with instance variables](module_with_instance_variables.md) if possible - [How to dump production data to staging](db_dump.md) diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 115c8cfb9ff..56c5056fd6d 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -16,9 +16,18 @@ This document is designed to be consumed by systems adminstrators and GitLab Sup When deployed, GitLab should be considered the amalgamation of the below processes. When troubleshooting or debugging, be as specific as possible as to which component you are referencing. That should increase clarity and reduce confusion. +### Layers + +GitLab can be considered to have two layers from a process perspective: + +- **Monitoring**: Anything from this layer is not required to deliver GitLab the application, but will allow administrators more insight into their infrastructure and what the service as a whole is doing. +- **Core**: Any process that is vital for the delivery of GitLab as a platform. If any of these processes halt there will be a GitLab outage. For the Core layer, you can further divide into: + - **Processors**: These processes are responsible for actually performing operations and presenting the service. + - **Data**: These services store/expose structured data for the GitLab service. + ### GitLab Process Descriptions -As of this writing, a fresh GitLab 11.3.0 install will show the following processes with `gitlab-ctl status`: +As of this writing, a fresh GitLab 11.3.0 install with default settings will show the following processes with `gitlab-ctl status`: ``` run: alertmanager: (pid 30829) 14207s; run: log: (pid 13906) 2432044s @@ -37,86 +46,77 @@ run: sidekiq: (pid 30953) 14205s; run: log: (pid 13810) 2432047s run: unicorn: (pid 30960) 14204s; run: log: (pid 13809) 2432047s ``` -### Layers - -GitLab can be considered to have two layers from a process perspective: - -- **Monitoring**: Anything from this layer is not required to deliver GitLab the application, but will allow administrators more insight into their infrastructure and what the service as a whole is doing. -- **Core**: Any process that is vital for the delivery of GitLab as a platform. If any of these processes halt there will be a GitLab outage. For the Core layer, you can further divide into: - - **Processors**: These processes are responsible for actually performing operations and presenting the service. - - **Data**: These services store/expose structured data for the GitLab service. - -### alertmanager +#### alertmanager - Omnibus configuration options - Layer: Monitoring -[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. +[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. -### gitaly +#### gitaly - [Omnibus configuration options](https://gitlab.com/gitlab-org/gitaly/tree/master/doc/configuration) - Layer: Core Service (Data) -Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (Think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). +Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). -### gitlab-monitor +#### gitlab-monitor - Omnibus configuration options - Layer: Monitoring -GitLab Monitor is a process designed in house that allows us to export metrics about GitLab application internals to prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor) +GitLab Monitor is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor). -### gitlab-workhorse +#### gitlab-workhorse - Omnibus configuration options - Layer: Core Service (Processor) -[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. -### logrotate +#### logrotate - [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) - Layer: Core Service -GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common opensource offering. +GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. -### nginx +#### nginx - [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/nginx.html) - Layer: Core Service (Processor) Nginx as an ingress port for all HTTP requests and routes them to the approriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver. -### node-exporter +#### node-exporter - [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/node_exporter.html) - Layer: Monitoring -[Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine. (Think CPU/Disk/Load) It's just a packaged version of the common open source offering from the Prometheus project. +[Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine (think CPU/Disk/Load). It's just a packaged version of the common open source offering from the Prometheus project. -### postgres-exporter +#### postgres-exporter - [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/postgres_exporter.html) - Layer: Monitoring -[Postgres-exporter](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about Postgres to prometheus for use in Grafana Dashboards. +[Postgres-exporter](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about Postgres to Prometheus for use in Grafana Dashboards. -### postgresql +#### postgresql - [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/database.html) - Layer: Core Service (Data) GitLab packages the popular Database to provide storage for Application meta data and user information. -### prometheus +#### prometheus - [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/) - Layer: Monitoring Prometheus is a time-series tool that helps GitLab administrators expose metrics about the individual processes used to provide GitLab the service. -### redis +#### redis - [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) - Layer: Core Service (Data) @@ -125,40 +125,42 @@ Redis is packaged to provide a place to store: - session data - temporary cache information -- background job queues. +- background job queues -### redis-exporter +#### redis-exporter - [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/redis_exporter.html) - Layer: Monitoring -[Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Graphana. +[Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Grafana. -### sidekiq +#### sidekiq - Omnibus configuration options - Layer: Core Service (Processor) Sidekiq is a Ruby background job processor that pulls jobs from the redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background. -### unicorn +#### unicorn - [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/unicorn.html) - Layer: Core Service (Processor) [Unicorn](https://bogomips.org/unicorn/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version. -### Additional Processes +#### Additional Processes + +The following processes will be running if corresponding feature is enabled. -### GitLab Pages +##### gitlab-pages TODO -### Mattermost +##### mattermost TODO -### Registry +##### registry The registry is what users use to store their own Docker images. The bundled registry uses nginx as a load balancer and GitLab as an authentication manager. @@ -181,10 +183,10 @@ It's important to understand the distinction as some processes are used in both ### GitLab Web HTTP Request Cycle -When making a request to an HTTP Endpoint (Think `/users/sign_in`) the request will take the following path through the GitLab Service: +When making a request to an HTTP Endpoint (think `/users/sign_in`) the request will take the following path through the GitLab Service: -- nginx - Acts as our first line reverse proxy -- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on unicorn. +- nginx - Acts as our first line reverse proxy. +- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. - unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn. - Postgres/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data. @@ -200,7 +202,7 @@ TODO ## System Layout -When referring to `~git` in the pictures it means the home directory of the git user which is typically /home/git. +When referring to `~git` in the pictures it means the home directory of the git user which is typically `/home/git`. GitLab is primarily installed within the `/home/git` user home directory as `git` user. Within the home directory is where the gitlabhq server software resides as well as the repositories (though the repository location is configurable). @@ -210,12 +212,128 @@ To serve repositories over SSH there's an add-on application called gitlab-shell ### Components -<img src="https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/pub?w=987&h=797"> +#### Component Diagram + +```mermaid +graph TB + + HTTP[HTTP/HTTPS] -- TCP 80, 443 --> NGINX[NGINX] + SSH -- TCP 22 --> GitLabShell[GitLab Shell] + SMTP[SMTP Gateway] + Geo[GitLab Geo Node] -- TCP 22, 80, 443 --> NGINX + + GitLabShell --TCP 8080 -->Unicorn["Unicorn (GitLab Rails)"] + GitLabShell --> Gitaly + GitLabShell --> Redis + Unicorn --> PgBouncer[PgBouncer] + Unicorn --> Redis + Unicorn --> Gitaly + Redis --> Sidekiq + Sidekiq["Sidekiq (GitLab Rails, ES Indexer)"] --> PgBouncer + GitLabWorkhorse[GitLab Workhorse] --> Unicorn + GitLabWorkhorse --> Redis + GitLabWorkhorse --> Gitaly + Gitaly --> Redis + NGINX --> GitLabWorkhorse + NGINX -- TCP 8090 --> GitLabPages[GitLab Pages] + NGINX --> Grafana[Grafana] + Grafana -- TCP 9090 --> Prometheus[Prometheus] + Prometheus -- TCP 80, 443 --> Unicorn + RedisExporter[Redis Exporter] --> Redis + Prometheus -- TCP 9121 --> RedisExporter + PostgreSQLExporter[PostgreSQL Exporter] --> PostgreSQL + PgBouncerExporter[PgBouncer Exporter] --> PgBouncer + Prometheus -- TCP 9187 --> PostgreSQLExporter + Prometheus -- TCP 9100 --> NodeExporter[Node Exporter] + Prometheus -- TCP 9168 --> GitLabMonito[GitLab Monitor] + Prometheus -- TCP 9127 --> PgBouncerExporter + GitLabMonitor --> PostgreSQL + GitLabMonitor --> GitLabShell + GitLabMonitor --> Sidekiq + PgBouncer --> Consul + PostgreSQL --> Consul + PgBouncer --> PostgreSQL + NGINX --> Registry + Unicorn --> Registry + NGINX --> Mattermost + Mattermost --- Unicorn + Prometheus --> Alertmanager + Migrations --> PostgreSQL + Runner -- TCP 443 --> NGINX + Unicorn -- TCP 9200 --> ElasticSearch + Sidekiq -- TCP 9200 --> ElasticSearch + Sidekiq -- TCP 80, 443 --> Sentry + Unicorn -- TCP 80, 443 --> Sentry + Sidekiq -- UDP 6831 --> Jaeger + Unicorn -- UDP 6831 --> Jaeger + Gitaly -- UDP 6831 --> Jaeger + GitLabShell -- UDP 6831 --> Jaeger + GitLabWorkhorse -- UDP 6831 --> Jaeger + Alertmanager -- TCP 25 --> SMTP + Sidekiq -- TCP 25 --> SMTP + Unicorn -- TCP 25 --> SMTP + Unicorn -- TCP 369 --> LDAP + Sidekiq -- TCP 369 --> LDAP + Unicorn -- TCP 443 --> ObjectStorage["Object Storage"] + Sidekiq -- TCP 443 --> ObjectStorage + GitLabWorkhorse -- TCP 443 --> ObjectStorage + Registry -- TCP 443 --> ObjectStorage + Geo -- TCP 5432 --> PostgreSQL -_[edit diagram (for GitLab team members only)](https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/edit)_ +``` + +#### Component Legend + +* ✅ - Automatically configured +* ⚙ - Requires additional configuration +* ⤓ - Additional software/service required +* ❌ - Not available + +#### Component Table + +| Component | Description | [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html) | [GitLab chart](https://docs.gitlab.com/charts/) | [Minikube Minimal](https://docs.gitlab.com/charts/development/minikube/#deploying-gitlab-with-minimal-settings) | [GitLab.com](https://gitlab.com) | CE/EE | +| --------- | ----------- |:--------------------:|:------------------:|:-----:|:--------:|:--------:| +| NGINX | Routes requests to appropriate components, terminates SSL | [✅](https://docs.gitlab.com/omnibus/settings/nginx.html) | [✅](https://docs.gitlab.com/charts/charts/nginx/index.html) | [⚙](https://docs.gitlab.com/charts/charts/nginx/index.html) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | +| Unicorn (GitLab Rails) | Handles requests for the web interface and API | [✅](https://docs.gitlab.com/omnibus/settings/unicorn.html) | [✅](https://docs.gitlab.com/charts/charts/gitlab/unicorn/index.html) | [✅](https://docs.gitlab.com/charts/charts/gitlab/unicorn/index.html) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#unicorn) | CE & EE | +| Sidekiq | Background jobs processor | [✅](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) | [✅](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [✅](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#sidekiq) | CE & EE | +| Gitaly | Git RPC service for handling all git calls made by GitLab | [✅](https://docs.gitlab.com/ee/administration/gitaly/) | [✅](https://docs.gitlab.com/charts/charts/gitlab/gitaly/index.html) | [✅](https://docs.gitlab.com/charts/charts/gitlab/gitaly/index.html) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | +| GitLab Workhorse | Smart reverse proxy, handles large HTTP requests | [✅](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) | [✅](https://docs.gitlab.com/charts/charts/gitlab/unicorn/index.html) | [✅](https://docs.gitlab.com/charts/charts/gitlab/unicorn/index.html) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | +| GitLab Shell | Handles `git` over SSH sessions | [✅](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) | [✅](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/index.html) | [✅](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/index.html) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | +| GitLab Pages | Hosts static websites | [⚙](https://docs.gitlab.com/ee/administration/pages/) | [❌](https://gitlab.com/charts/gitlab/issues/37) | [❌](https://gitlab.com/charts/gitlab/issues/37) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-pages) | CE & EE | +| Registry | Container registry, allows pushing and pulling of images | [⚙](https://docs.gitlab.com/ee/administration/container_registry.html#container-registry-domain-configuration) | [✅](https://docs.gitlab.com/charts/charts/registry/index.html) | [✅](https://docs.gitlab.com/charts/charts/registry/index.html) | [✅](https://docs.gitlab.com/ee/user/project/container_registry.html#build-and-push-images) | CE & EE | +| Redis | Caching service | [✅](https://docs.gitlab.com/omnibus/settings/redis.html) | [✅](https://docs.gitlab.com/charts/charts/redis/index.html) | [✅](https://docs.gitlab.com/charts/charts/redis/index.html) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | CE & EE | +| PostgreSQL | Database | [✅](https://docs.gitlab.com/omnibus/settings/database.html) | [✅](https://github.com/helm/charts/tree/master/stable/postgresql) | [✅](https://github.com/helm/charts/tree/master/stable/postgresql) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#postgresql) | CE & EE | +| PgBouncer | Database connection pooling, failover | [⚙](https://docs.gitlab.com/ee/administration/high_availability/pgbouncer.html) | [❌](https://docs.gitlab.com/charts/installation/deployment.html#postgresql) | [❌](https://docs.gitlab.com/charts/installation/deployment.html#postgresql) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#database-architecture) | EE Only | +| Consul | Database node discovery, failover | [⚙](https://docs.gitlab.com/ee/administration/high_availability/consul.html) | [❌](https://docs.gitlab.com/charts/installation/deployment.html#postgresql) | [❌](https://docs.gitlab.com/charts/installation/deployment.html#postgresql) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#consul) | EE Only | +| Prometheus | Time-series database, metrics collection, and query service | [✅](https://docs.gitlab.com/ee/administration/monitoring/prometheus/) | [✅](https://github.com/helm/charts/tree/master/stable/prometheus) | [⚙](https://github.com/helm/charts/tree/master/stable/prometheus) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#prometheus) | CE & EE | +| Prometheus Alertmanager | Deduplicates, groups, and routes alerts from Prometheus | [✅](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) | [✅](https://github.com/helm/charts/tree/master/stable/prometheus) | [✅](https://github.com/helm/charts/tree/master/stable/prometheus) | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | +| Grafana | Metrics dashboard | [⚙](https://docs.gitlab.com/ee/administration/monitoring/performance/grafana_configuration.html) | [⤓](https://github.com/helm/charts/tree/master/stable/grafana) | [⤓](https://github.com/helm/charts/tree/master/stable/grafana) | [✅](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s) | CE & EE | +| Redis Exporter | Prometheus endpoint with Redis metrics | [✅](https://docs.gitlab.com/ee/administration/monitoring/prometheus/redis_exporter.html) | [✅](https://docs.gitlab.com/charts/charts/redis/index.html) | [✅](https://docs.gitlab.com/charts/charts/redis/index.html) | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | +| PostgreSQL Exporter | Prometheus endpoint with PostgreSQL metrics | [✅](https://docs.gitlab.com/ee/administration/monitoring/prometheus/postgres_exporter.html) | [✅](https://github.com/helm/charts/tree/master/stable/postgresql) | [✅](https://github.com/helm/charts/tree/master/stable/postgresql) | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | +| PgBouncer Exporter | Prometheus endpoint with PgBouncer metrics | [⚙](https://docs.gitlab.com/ee/administration/monitoring/prometheus/pgbouncer_exporter.html) | [❌](https://docs.gitlab.com/charts/installation/deployment.html#postgresql) | [❌](https://docs.gitlab.com/charts/installation/deployment.html#postgresql) | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | +| GitLab Monitor | Generates a variety of GitLab metrics | [✅](https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_monitor_exporter.html) | [❌](https://gitlab.com/charts/gitlab/issues/319) | [❌](https://gitlab.com/charts/gitlab/issues/319) | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | CE & EE | +| Mattermost | Open-source Slack alternative | [⚙](https://docs.gitlab.com/omnibus/gitlab-mattermost/) | [⤓](https://docs.mattermost.com/install/install-mmte-helm-gitlab-helm.html) | [⤓](https://docs.mattermost.com/install/install-mmte-helm-gitlab-helm.html) | [⤓](https://docs.gitlab.com/ee/user/project/integrations/mattermost_slash_commands.html#manual-configuration), [⤓](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html) | CE & EE | +| Minio | Object storage service | [⤓](https://min.io/download) | [✅](https://docs.gitlab.com/charts/charts/minio/index.html) | [✅](https://docs.gitlab.com/charts/charts/minio/index.html) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#storage-architecture) | CE & EE | +| Runner | Executes GitLab CI jobs | [⤓](https://docs.gitlab.com/runner/) | [✅](https://docs.gitlab.com/runner/) | [⚙](https://docs.gitlab.com/runner/) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners) | CE & EE | +| DB Migrations | Database migrations | [✅](https://docs.gitlab.com/omnibus/settings/database.html#disabling-automatic-database-migration) | [✅](https://docs.gitlab.com/charts/charts/gitlab/migrations/index.html) | [✅](https://docs.gitlab.com/charts/charts/gitlab/migrations/index.html) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#database-architecture) | CE & EE | +| Certificate Management | TLS Settings, Let's Encrypt | [✅](https://docs.gitlab.com/omnibus/settings/ssl.html) | [✅](https://docs.gitlab.com/charts/installation/tls.html) | [⚙](https://docs.gitlab.com/charts/installation/tls.html) | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | CE & EE | +| GitLab Geo Node | Geographically distributed GitLab nodes | [⚙](https://docs.gitlab.com/ee/administration/geo/replication/index.html#setup-instructions) | [❌](https://gitlab.com/charts/gitlab/issues/8) | [❌](https://gitlab.com/charts/gitlab/issues/8) | ✅ | EE Only | +| LDAP Authentication | Authenticate users against centralized LDAP directory | [⤓](https://docs.gitlab.com/ee/administration/auth/ldap.html) | [⤓](https://docs.gitlab.com/charts/charts/globals.html#ldap) | [⤓](https://docs.gitlab.com/charts/charts/globals.html#ldap) | [❌](https://about.gitlab.com/pricing/#gitlab-com) | CE & EE | +| Outbound email (SMTP) | Send email messages to users | [⤓](https://docs.gitlab.com/omnibus/settings/smtp.html) | [⤓](https://docs.gitlab.com/charts/installation/command-line-options.html#outgoing-email-configuration) | [⤓](https://docs.gitlab.com/charts/installation/command-line-options.html#outgoing-email-configuration) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | CE & EE | +| Inbound email (SMTP) | Receive messages to update issues | [⤓](https://docs.gitlab.com/ee/administration/incoming_email.html) | [⤓](https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration) | [⤓](https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration) | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | CE & EE | +| ElasticSearch | Improved search within GitLab | [⤓](https://docs.gitlab.com/ee/integration/elasticsearch.html) | [⤓](https://docs.gitlab.com/ee/integration/elasticsearch.html) | [⤓](https://docs.gitlab.com/ee/integration/elasticsearch.html) | [❌](https://gitlab.com/groups/gitlab-org/-/epics/153) | EE Only | +| Sentry: GitLab instance | Track errors generated by the GitLab instance | [⤓](https://docs.gitlab.com/omnibus/settings/configuration.html#error-reporting-and-logging-with-sentry) | [❌](https://gitlab.com/charts/gitlab/issues/1319) | [❌](https://gitlab.com/charts/gitlab/issues/1319) | [✅](https://about.gitlab.com/handbook/support/workflows/services/gitlab_com/500_errors.html#searching-sentry) | CE & EE | +| Jaeger: GitLab instance | View traces generated by the GitLab instance | [❌](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104) | [❌](https://gitlab.com/charts/gitlab/issues/1320) | [❌](https://gitlab.com/charts/gitlab/issues/1320) | [❌](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104) | CE & EE | +| Sentry: deployed apps | Error tracking for deployed apps | [⤓](https://docs.gitlab.com/ee/user/project/operations/error_tracking.html) | [⤓](https://docs.gitlab.com/ee/user/project/operations/error_tracking.html) | [⤓](https://docs.gitlab.com/ee/user/project/operations/error_tracking.html) | [⤓](https://docs.gitlab.com/ee/user/project/operations/error_tracking.html) | CE & EE | +| Jaeger: deployed apps | Distributed tracing for deployed apps | [⤓](https://docs.gitlab.com/ee/user/project/operations/tracing.html) | [⤓](https://docs.gitlab.com/ee/user/project/operations/tracing.html) | [⤓](https://docs.gitlab.com/ee/user/project/operations/tracing.html) | [⤓](https://docs.gitlab.com/ee/user/project/operations/tracing.html) | EE Only | +| Kubernetes cluster apps | Deploy [Helm](https://docs.helm.sh/), [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/), [Cert-Manager](https://docs.cert-manager.io/en/latest/), [Prometheus](https://prometheus.io/docs/introduction/overview/), a [Runner](https://docs.gitlab.com/runner/), [JupyterHub](http://jupyter.org/), [Knative](https://cloud.google.com/knative) to a cluster | [✅](https://docs.gitlab.com/ee/user/project/clusters/#installing-applications) | [✅](https://docs.gitlab.com/ee/user/project/clusters/#installing-applications) | [✅](https://docs.gitlab.com/ee/user/project/clusters/#installing-applications) | [✅](https://docs.gitlab.com/ee/user/project/clusters/#installing-applications) | CE & EE | + +#### Component Overview A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incoming jobs. +We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/). + The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects. diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 52a4bb27817..5633c8313a3 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -52,6 +52,58 @@ Adhere to the [Documentation Style Guide](styleguide.md). If a style standard is See the [Structure](styleguide.md#structure) section of the [Documentation Style Guide](styleguide.md). +## Single codebase + +We currently maintain two sets of docs: one in the +[gitlab-ce](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc) repo and +one in [gitlab-ee](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc). +They are similar, and most pages are identical, but they are different repositories. +With the single codebase effort, we want to make those two sets identical, so when the +time comes to have only one codebase, we'll be ready. + +Here are some links to get you up to speed with the current effort: + +- [CE/EE codebases blueprint](https://about.gitlab.com/handbook/engineering/infrastructure/blueprint/ce-ee-codebases/) +- [CE/EE codebases merge design](https://about.gitlab.com/handbook/engineering/infrastructure/design/merge-ce-ee-codebases/) +- [Single docs codebase epic](https://gitlab.com/groups/gitlab-org/-/epics/199) +- [Issue board of related issues](https://gitlab.com/groups/gitlab-org/-/boards/981090?&label_name[]=Documentation&label_name[]=single%20codebase) +- [Related merge requests](https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=all&label_name[]=Documentation&label_name[]=single%20codebase) +- [Visualize the existing diffs](https://leipert-projects.gitlab.io/is-gitlab-pretty-yet/diff/?search=%5Edoc) + +### CE first + +After a given documentation path is aligned across CE and EE, all merge requests +affecting that path must be submitted to CE, regardless of the content it has. +This means that for EE-only features which are being added only to the EE codebase, +you have to submit a separate merge request in CE that contains the docs. +This might seem like a duplicate effort, but it's for the short term. +A list of the already aligned docs can be found in +[the epic description](https://gitlab.com/groups/gitlab-org/-/epics/199#ee-specific-lines-check). + +Since the docs will be combined, it's crucial to add the relevant +[product badges](styleguide.md#product-badges) for all EE documentation, so that +we can discern which features belong to which tier. + +### EE specific lines check + +There's a special test in place +([`ee_specific_check.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/ee_specific_check/ee_specific_check.rb)), +which, among others, checks and prevents creating/editing new files and directories +in EE under `doc/`. + +We have a long list of documentation paths that are either whitelisted or not. +Paths in the whitelist (not commented out) will not be subject to the test, +which means you are allowed to create/change docs content in EE for the time +being. The goal is to not have any doc whitelisted. + +At the time of this writing, the only items left to be aligned are: + +- `doc/api/*` ([issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/60045) / [merge request](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27491)) +- `doc/README.md` + +Eventually, once all docs are aligned, we'll remove any doc reference from that +script, so it catches everything. + ## Changing document location Changing a document's location requires specific steps to be followed to ensure that @@ -438,6 +490,10 @@ Currently, the following tests are in place: As CE is merged into EE once a day, it's important to avoid merge conflicts. Submitting an EE-equivalent merge request cherry-picking all commits from CE to EE is essential to avoid them. +1. [`ee-files-location-check`/`ee-specific-lines-check`](#ee-specific-lines-check) (runs on EE only): + This test ensures that no new files/directories are created/changed in EE. + All docs should be submitted in CE instead, regardless the tier they are on. + This is for the [single codebase](#single-codebase) effort. 1. In a full pipeline, tests for [`/help`](#gitlab-help-tests). ### Linting diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 9853b38b8e9..9db28bcddf8 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -446,6 +446,28 @@ The disadvantage of this: port `render_if_exists` to CE. - If we have typos in the partial name, it would be silently ignored. + +##### Caveats + +The `render_if_exists` view path argument must be relative to `app/views/` and `ee/app/views`. +Resolving an EE template path that is relative to the CE view path will not work. + +```haml +- # app/views/projects/index.html.haml + += render_if_exists 'button' # Will not render `ee/app/views/projects/_button` and will quietly fail += render_if_exists 'projects/button' # Will render `ee/app/views/projects/_button` +``` + +You should not explicitly set render options like `partial` or provide a `locals` hash. +The first argument should be a path string and the second can be a hash replacing `locals`. + +```ruby +render partial: 'projects/button', locals: { project: project } +# becomes +render_if_exists 'projects/button', project: project +``` + #### Using `render_ce` For `render` and `render_if_exists`, they search for the EE partial first, diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 5f6123b5f9b..8e06aa5d173 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -43,9 +43,9 @@ new Vue({ Read more about [Vue Apollo][vue-apollo] in the [Vue Apollo documentation][vue-apollo-docs]. -### Local state with `apollo-link-state` +### Local state with Apollo -It is possible to use our Apollo setup with [apollo-link-state][apollo-link-state] by passing +It is possible to manage an application state with Apollo by passing in a resolvers object when creating the default client. The default state can be set by writing to the cache after setting up the default client. @@ -76,6 +76,8 @@ const apolloProvider = new VueApollo({ }); ``` +Read more about local state management with Apollo in the [Vue Apollo documentation](https://vue-apollo.netlify.com/guide/local-state.html#local-state). + ### Testing With [Vue test utils][vue-test-utils] it is easy to quickly test components that @@ -92,6 +94,8 @@ it('tests apollo component', () => { }); ``` +Another possible way is testing queries with mocked GraphQL schema. Read more about this way in [Vue Apollo testing documentation](https://vue-apollo.netlify.com/guide/testing.html#tests-with-mocked-graqhql-schema) + ## Usage outside of Vue It is also possible to use GraphQL outside of Vue by directly importing diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 7d52cac5f7e..bf248b7f8af 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -186,7 +186,7 @@ Remember that actions only describe that something happened, they don't describe state.users.push(user); }, [types.REQUEST_ADD_USER_ERROR](state, error) { - state.isAddingUser = true; + state.isAddingUser = false; state.errorAddingUser = error; }, }; @@ -231,7 +231,7 @@ The store should be included in the main component of your application: ```javascript // app.vue - import store from 'store'; // it will include the index.js file + import store from './store'; // it will include the index.js file export default { name: 'application', diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index b9dc3797e5b..cf78b792ffd 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -40,7 +40,7 @@ of possible security breaches in our code: - SQL injections Remember to run -[SAST](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html) +[SAST](https://docs.gitlab.com/ee/user/application_security/sast/index) **[ULTIMATE]** on your project (or at least the [gosec analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), and to follow our [Security @@ -96,7 +96,7 @@ dependency should be argued in the merge request, as per our [Approval Guidelines](../code_review.md#approval-guidelines). Both [License Management](https://docs.gitlab.com/ee/user/project/merge_requests/license_management.html) **[ULTIMATE]** and [Dependency -Scanning](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html) +Scanning](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index) **[ULTIMATE]** should be activated on all projects to ensure new dependencies security status and license compatibility. diff --git a/doc/development/session.md b/doc/development/session.md new file mode 100644 index 00000000000..9edce3dbda0 --- /dev/null +++ b/doc/development/session.md @@ -0,0 +1,65 @@ +# Accessing session data + +Session data in GitLab is stored in Redis and can be accessed in a variety of ways. + +During a web request, for example: + +- Rails provides access to the session from within controllers through [`ActionDispatch::Session`](https://guides.rubyonrails.org/action_controller_overview.html#session). +- Outside of controllers, it is possible to access the session through `Gitlab::Session`. + +Outside of a web request it is still possible to access sessions stored in Redis. For example: + +- Session IDs and contents can be [looked up directly in Redis](#redis). +- Data about the UserAgent associated with the session can be accessed through `ActiveSession`. + +When storing values in a session it is best to: + +- Use simple primitives and avoid storing objects to avoid marshaling complications. +- Clean up after unneeded variables to keep memory usage in Redis down. + +## Gitlab::Session + +Sometimes you might want to persist data in the session instead of another store like the database. `Gitlab::Session` lets you access this without passing the session around extensively. For example, you could access it from within a policy without having to pass the session through to each place permissions are checked from. + +The session has a hash-like interface, just like when using it from a controller. There is also `NamespacedSessionStore` for storing key-value data in a hash. + +```ruby +# Lookup a value stored in the current session +Gitlab::Session.current[:my_feature] + +# Modify the current session stored in redis +Gitlab::Session.current[:my_feature] = value + +# Store key-value data namespaced under a key +Gitlab::NamespacedSessionStore.new(:my_feature)[some_key] = value + +# Set the session for a block of code, such as for tests +Gitlab::Session.with_session(my_feature: value) do + # Code that uses Session.current[:my_feature] +end +``` + +## Redis + +Session data can be accessed directly through Redis. This can let you check up on a browser session when debugging. + +```ruby +# Get a list of sessions +session_ids = Gitlab::Redis::SharedState.with do |redis| + redis.smembers("#{Gitlab::Redis::SharedState::USER_SESSIONS_LOOKUP_NAMESPACE}:#{user.id}") +end + +# Retrieve a specific session +session_data = Gitlab::Redis::SharedState.with { |redis| redis.get("#{Gitlab::Redis::SharedState::SESSION_NAMESPACE}:#{session_id}") } +Marshal.load(session_data) +``` + +## Getting device information with ActiveSession + +The [**Active Sessions** page on a user's profile](../user/profile/active_sessions.md) displays information about the device used to access each session. The methods used there to list sessions can also be useful for development. + +```ruby +# Get list of sessions for a given user +# Includes session_id and data from the UserAgent +ActiveSession.list(user) +``` diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index f58a8dcbcdc..9bd99e80357 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -187,6 +187,7 @@ export default function doSomething() { visitUrl('/foo/bar'); } ``` + ```js // my_module_spec.js import doSomething from '~/my_module'; @@ -213,7 +214,187 @@ Further documentation on the babel rewire pluign API can be found on #### Waiting in tests -If you cannot avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) in tests, please use the [Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). +Sometimes a test needs to wait for something to happen in the application before it continues. +Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) +because it makes the reason for waiting unclear and if passed a time larger than zero it will slow down our test suite. +Instead use one of the following approaches. + +##### Promises and Ajax calls + +Register handler functions to wait for the `Promise` to be resolved. + +```javascript +const askTheServer = () => { + return axios + .get('/endpoint') + .then(response => { + // do something + }) + .catch(error => { + // do something else + }); +}; +``` + +**in Jest:** + +```javascript +it('waits for an Ajax call', () => { + return askTheServer().then(() => { + expect(something).toBe('done'); + }); +}); +``` + +**in Karma:** + +```javascript +it('waits for an Ajax call', done => { + askTheServer() + .then(() => { + expect(something).toBe('done'); + }) + .then(done) + .catch(done.fail); +}); +``` + +If you are not able to register handlers to the `Promise`—for example because it is executed in a synchronous Vue life +cycle hook—you can flush all pending `Promise`s: + +**in Jest:** + +```javascript +it('waits for an Ajax call', () => { + synchronousFunction(); + jest.runAllTicks(); + + expect(something).toBe('done'); +}); +``` + +**in Karma:** + +You are out of luck. The following only works sometimes and may lead to flaky failures: + +```javascript +it('waits for an Ajax call', done => { + synchronousFunction(); + + // create a new Promise and hope that it resolves after the rest + Promise.resolve() + .then(() => { + expect(something).toBe('done'); + }) + .then(done) + .catch(done.fail); +}); +``` + +##### Vue rendering + +To wait until a Vue component is re-rendered, use either of the equivalent +[`Vue.nextTick()`](https://vuejs.org/v2/api/#Vue-nextTick) or `vm.$nextTick()`. + +**in Jest:** + +```javascript +it('renders something', () => { + wrapper.setProps({ value: 'new value' }); + + return wrapper.vm.$nextTick().then(() => { + expect(wrapper.text()).toBe('new value'); + }); +}); +``` + +**in Karma:** + +```javascript +it('renders something', done => { + wrapper.setProps({ value: 'new value' }); + + wrapper.vm + .$nextTick() + .then(() => { + expect(wrapper.text()).toBe('new value'); + }) + .then(done) + .catch(done.fail); +}); +``` + +##### `setTimeout()` / `setInterval()` in application + +If the application itself is waiting for some time, mock await the waiting. In Jest this is already +[done by default](https://gitlab.com/gitlab-org/gitlab-ce/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) +(see also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks)). In Karma you can use the +[Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). + +```javascript +const doSomethingLater = () => { + setTimeout(() => { + // do something + }, 4000); +}; +``` + +**in Jest:** + +```javascript +it('does something', () => { + doSomethingLater(); + jest.runAllTimers(); + + expect(something).toBe('done'); +}); +``` + +**in Karma:** + +```javascript +it('does something', () => { + jasmine.clock().install(); + + doSomethingLater(); + jasmine.clock().tick(4000); + + expect(something).toBe('done'); + jasmine.clock().uninstall(); +}); +``` + +##### Events + +If the application triggers an event that you need to wait for in your test, register an event handler which contains +the assertions: + +```javascript +it('waits for an event', done => { + eventHub.$once('someEvent', eventHandler); + + someFunction(); + + function eventHandler() { + expect(something).toBe('done'); + done(); + } +}); +``` + +In Jest you can also use a `Promise` for this: + +```javascript +it('waits for an event', () => { + const eventTriggered = new Promise(resolve => eventHub.$once('someEvent', resolve)); + + someFunction(); + + return eventTriggered.then(() => { + expect(something).toBe('done'); + }); +}); +``` #### Migrating flaky Karma tests to Jest diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 7ad33f05f77..f34793c11f4 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -78,17 +78,27 @@ subgraph CNG-mirror pipeline **Additional notes:** -- The Kubernetes cluster is connected to the `gitlab-{ce,ee}` projects using - [GitLab's Kubernetes integration][gitlab-k8s-integration]. This basically - allows to have a link to the Review App directly from the merge request - widget. -- If the Review App deployment fails, you can simply retry it (there's no need - to run the [`review-stop`][gitlab-ci-yml] job first). +- If the `review-deploy` job keep failing (note that we already retry it twice), + please post a message in the `#quality` channel and/or create a ~Quality ~bug + issue with a link to your merge request. Note that the deployment failure can + reveal an actual problem introduced in your merge request (i.e. this isn't + necessarily a transient failure)! +- If the `review-qa-smoke` job keep failing (note that we already retry it twice), + please check the job's logs: you could discover an actual problem introduced in + your merge request. You can also download the artifacts to see screenshots of + the page at the time the failures occurred. If you don't find the cause of the + failure or if it seems unrelated to your change, please post a message in the + `#quality` channel and/or create a ~Quality ~bug issue with a link to your + merge request. - The manual [`review-stop`][gitlab-ci-yml] in the `test` stage can be used to stop a Review App manually, and is also started by GitLab once a merge request's branch is deleted after being merged. - Review Apps are cleaned up regularly via a pipeline schedule that runs the [`schedule:review-cleanup`][gitlab-ci-yml] job. +- The Kubernetes cluster is connected to the `gitlab-{ce,ee}` projects using + [GitLab's Kubernetes integration][gitlab-k8s-integration]. This basically + allows to have a link to the Review App directly from the merge request + widget. ## QA runs diff --git a/doc/install/installation.md b/doc/install/installation.md index 60a8ffacd76..724f43e6ae2 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -111,7 +111,7 @@ sudo apt-get install -y libcurl4-openssl-dev libexpat1-dev gettext libz-dev libs # Download and compile from source cd /tmp curl --remote-name --location --progress https://www.kernel.org/pub/software/scm/git/git-2.21.0.tar.gz -echo '85eca51c7404da75e353eba587f87fea9481ba41e162206a6f70ad8118147bee' git-2.21.0.tar.gz' | shasum -a256 -c - && tar -xzf git-2.21.0.tar.gz +echo '85eca51c7404da75e353eba587f87fea9481ba41e162206a6f70ad8118147bee git-2.21.0.tar.gz' | shasum -a256 -c - && tar -xzf git-2.21.0.tar.gz cd git-2.21.0/ ./configure make prefix=/usr/local all @@ -163,9 +163,9 @@ Download Ruby and compile it: ```sh mkdir /tmp/ruby && cd /tmp/ruby -curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.3.tar.gz -echo 'f919a9fbcdb7abecd887157b49833663c5c15fda ruby-2.5.3.tar.gz' | shasum -c - && tar xzf ruby-2.5.3.tar.gz -cd ruby-2.5.3 +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.gz +echo '2347ed6ca5490a104ebd5684d2b9b5eefa6cd33c ruby-2.6.3.tar.gz' | shasum -c - && tar xzf ruby-2.6.3.tar.gz +cd ruby-2.6.3 ./configure --disable-install-rdoc make @@ -434,7 +434,8 @@ sudo -u git -H editor config/resque.yml ``` CAUTION: **Caution:** -Make sure to edit both `gitlab.yml` and `unicorn.rb` to match your setup. +Make sure to edit both `gitlab.yml` and `unicorn.rb` to match your setup. +If you want to use Puma web server, see [Using Puma](#using-puma) for the additional steps. NOTE: **Note:** If you want to use HTTPS, see [Using HTTPS](#using-https) for the additional steps. @@ -448,6 +449,18 @@ sudo -u git cp config/database.yml.postgresql config/database.yml # MySQL only: sudo -u git cp config/database.yml.mysql config/database.yml +# PostgreSQL only: +# Remove host, username, and password lines from config/database.yml. +# Once modified, the `production` settings will be as follows: +# +# production: +# adapter: postgresql +# encoding: unicode +# database: gitlabhq_production +# pool: 10 +# +sudo -u git -H editor config/database.yml + # MySQL and remote PostgreSQL only: # Update username/password in config/database.yml. # You only need to adapt the production settings (first part). @@ -565,6 +578,18 @@ sudo -u git -H editor config.toml For more information about configuring Gitaly see [doc/administration/gitaly](../administration/gitaly). +### Start Gitaly + +Gitaly must be running for the next section. + +```sh +gitlab_path=/home/git/gitlab +gitaly_path=/home/git/gitaly + +sudo -u git -H $gitlab_path/bin/daemon_with_pidfile $gitlab_path/tmp/pids/gitaly.pid \ + $gitaly_path/gitaly $gitaly_path/config.toml >> $gitlab_path/log/gitaly.log 2>&1 & +``` + ### Initialize Database and Activate Advanced Features ```sh @@ -579,10 +604,10 @@ sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production force=yes ``` NOTE: **Note:** -You can set the Administrator/root password and e-mail by supplying them in environmental variables, `GITLAB_ROOT_PASSWORD` and `GITLAB_ROOT_EMAIL` respectively, as seen below. If you don't set the password (and it is set to the default one), wait to expose GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login, you'll be forced to change the default password. +You can set the Administrator/root password and e-mail by supplying them in environmental variables, `GITLAB_ROOT_PASSWORD` and `GITLAB_ROOT_EMAIL` respectively, as seen below. If you don't set the password (and it is set to the default one), wait to expose GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login, you'll be forced to change the default password. An Enterprise Edition license may also be installed at this time by supplying a full path in the `GITLAB_LICENSE_FILE` environment variable. ```sh -sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword GITLAB_ROOT_EMAIL=youremail +sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword GITLAB_ROOT_EMAIL=youremail GITLAB_LICENSE_FILE="/path/to/license" ``` ### Secure secrets.yml @@ -640,6 +665,12 @@ sudo -u git -H yarn install --production --pure-lockfile sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production NODE_ENV=production ``` +If `rake` fails with `JavaScript heap out of memory` error, try to run it with `NODE_OPTIONS` set as follows. + +```sh +sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production NODE_ENV=production NODE_OPTIONS="--max_old_space_size=4096" +``` + ### Start Your GitLab Instance ```sh @@ -845,6 +876,25 @@ You also need to change the corresponding options (e.g. `ssh_user`, `ssh_host`, Apart from the always supported markdown style, there are other rich text files that GitLab can display. But you might have to install a dependency to do so. See the [github-markup gem README](https://github.com/gitlabhq/markup#markups) for more information. +### Using Puma + +Puma is a multi-threaded HTTP 1.1 server for Ruby applications. + +To use GitLab with Puma: + +1. Finish GitLab setup so you have it up and running. +1. Copy the supplied example Puma config file into place: + + ```sh + cd /home/git/gitlab + + # Copy config file for the web server + sudo -u git -H config/puma.rb.example config/puma.rb + ``` + +1. Edit the system `init.d` script to use `EXPERIMENTAL_PUMA=1` flag. If you have `/etc/default/gitlab`, then you should edit it instead. +1. Restart GitLab. + ## Troubleshooting ### "You appear to have cloned an empty repository." diff --git a/doc/install/ldap.md b/doc/install/ldap.md index a19f0342b65..d8d54864586 100644 --- a/doc/install/ldap.md +++ b/doc/install/ldap.md @@ -2,6 +2,4 @@ redirect_to: '../administration/auth/ldap.md' --- -# GitLab LDAP integration - -This document was moved under [`administration/auth/ldap`](../administration/auth/ldap.md). +This document was moved to [another location](../administration/auth/ldap.md). diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 17099c1d051..f6a52205a0e 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -87,7 +87,7 @@ if your available memory changes. We also recommend [configuring the kernel's sw to a low value like `10` to make the most of your RAM while still having the swap available when needed. -Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as `top` or `htop`) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about how many you need of those. +NOTE: **Note:** The 25 workers of Sidekiq will show up as separate processes in your process overview (such as `top` or `htop`) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about how many you need of those. ## Database @@ -169,7 +169,7 @@ So for a machine with 2 cores, 3 unicorn workers is ideal. For all machines that have 2GB and up we recommend a minimum of three unicorn workers. If you have a 1GB machine we recommend to configure only two Unicorn workers to prevent excessive swapping. -To change the Unicorn workers when you have the Omnibus package (which defaults to the recommendation above) please see [the Unicorn settings in the Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md#unicorn-settings). +To change the Unicorn workers when you have the Omnibus package (which defaults to the recommendation above) please see [the Unicorn settings in the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/unicorn.html). ## Redis and Sidekiq @@ -201,14 +201,13 @@ you decide to run GitLab Runner and the GitLab Rails application on the same machine. It is also not safe to install everything on a single machine, because of the -[security reasons] - especially when you plan to use shell executor with GitLab +[security reasons](https://docs.gitlab.com/runner/security/) +- especially when you plan to use shell executor with GitLab Runner. We recommend using a separate machine for each GitLab Runner, if you plan to use the CI features. -[security reasons]: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/security/index.md - ## Supported web browsers We support the current and the previous major release of: @@ -224,5 +223,5 @@ Support is only provided for the current minor version of the major version you Each time a new browser version is released, we begin supporting that version and stop supporting the third most recent version. -Note: We do not support running GitLab with JavaScript disabled in the browser and have no plans of supporting that +NOTE: **Note:** We do not support running GitLab with JavaScript disabled in the browser and have no plans of supporting that in the future because we have features such as Issue Boards which require JavaScript extensively. diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index 7cef664bc98..d1d12dfd064 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -85,7 +85,7 @@ To build and install the indexer, run: ```sh git clone https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer.git -cd /gitlab-elasticsearch-indexer +cd gitlab-elasticsearch-indexer make sudo make install ``` @@ -131,6 +131,8 @@ The following Elasticsearch settings are available: | `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). | | `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. | | `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., "http://host1, https://host2:9200"). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | +| `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#create-index-settings) | +| `Number of Elasticsearch replicas` | Each Elasticsearch shard can have a number of replicas. These are a complete copy of the shard, and can provide increased query performance or resilience against hardware failure. Increasing this value will greatly increase total disk space required by the index. | | `Limit namespaces and projects that can be indexed` | Enabling this will allow you to select namespaces and projects to index. All other namespaces and projects will use database search instead. Please note that if you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limiting-namespaces-and-projects). | `Using AWS hosted Elasticsearch with IAM credentials` | Sign your Elasticsearch requests using [AWS IAM authorization](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or [AWS EC2 Instance Profile Credentials](http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli). The policies must be configured to allow `es:*` actions. | | `AWS Region` | The AWS region your Elasticsearch service is located in. | diff --git a/doc/integration/img/salesforce_app_details.png b/doc/integration/img/salesforce_app_details.png Binary files differnew file mode 100644 index 00000000000..00e66f07282 --- /dev/null +++ b/doc/integration/img/salesforce_app_details.png diff --git a/doc/integration/img/salesforce_app_secret_details.png b/doc/integration/img/salesforce_app_secret_details.png Binary files differnew file mode 100644 index 00000000000..fad2a4a1f97 --- /dev/null +++ b/doc/integration/img/salesforce_app_secret_details.png diff --git a/doc/integration/img/salesforce_oauth_app_details.png b/doc/integration/img/salesforce_oauth_app_details.png Binary files differnew file mode 100644 index 00000000000..a5fb680cca6 --- /dev/null +++ b/doc/integration/img/salesforce_oauth_app_details.png diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index ef1f2df77f8..a13e9f73f48 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -35,6 +35,7 @@ contains some settings that are common for all providers. - [JWT](../administration/auth/jwt.md) - [OpenID Connect](../administration/auth/oidc.md) - [UltraAuth](ultra_auth.md) +- [SalesForce](salesforce.md) ## Initial OmniAuth Configuration diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md new file mode 100644 index 00000000000..8a99641a256 --- /dev/null +++ b/doc/integration/salesforce.md @@ -0,0 +1,79 @@ +# SalesForce OmniAuth Provider + +You can integrate your GitLab instance with [SalesForce](https://www.salesforce.com/) to enable users to login to your GitLab instance with their SalesForce account. + +## Create SalesForce Application + +To enable SalesForce OmniAuth provider, you must use SalesForce's credentials for your GitLab instance. +To get the credentials (a pair of Client ID and Client Secret), you must register an application on SalesForces. + +1. Sign in to [SalesForce](https://www.salesforce.com/). + +1. Navigate to **Platform Tools/Apps/App Manager** and click on **New Connected App**. + +1. Fill in the application details into the following fields: + - **Connected App Name** and **API Name**: Set to any value but consider something like `<Organization>'s GitLab`, `<Your Name>'s GitLab`, or something else that is descriptive. + - **Description**: Description for the application. + +  +1. Select **API (Enable OAuth Settings)** and click on **Enable OAuth Settings**. +1. Fill in the application details into the following fields: + - **Callback URL**: The call callback URL. For example, `https://gitlab.example.com/users/auth/salesforce/callback`. + - **Selected OAuth Scopes**: Move **Access your basic information (id, profile, email, address, phone)** and **Allow access to your unique identifier (openid)** to the right column. + +  +1. Click **Save**. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "salesforce", + "app_id" => "SALESFORCE_CLIENT_ID", + "app_secret" => "SALESFORCE_CLIENT_SECRET" + } + ] + ``` + + For installation from source: + + ``` + - { name: 'salesforce', + app_id: 'SALESFORCE_CLIENT_ID', + app_secret: 'SALESFORCE_CLIENT_SECRET' + } + ``` +1. Change `SALESFORCE_CLIENT_ID` to the Consumer Key from the SalesForce connected application page. +1. Change `SALESFORCE_CLIENT_SECRET` to the Consumer Secret from the SalesForce connected application page. +  + +1. Save the configuration file. +1. [Reconfigure GitLab]( ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure ) or [restart GitLab]( ../administration/restart_gitlab.md#installations-from-source ) for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. + +On the sign in page, there should now be a SalesForce icon below the regular sign in form. +Click the icon to begin the authentication process. SalesForce will ask the user to sign in and authorize the GitLab application. +If everything goes well, the user will be returned to GitLab and will be signed in. + +NOTE: **Note:** +GitLab requires the email address of each new user. Once the user is logged in using SalesForce, GitLab will redirect the user to the profile page where they will have to provide the email and verify the email. diff --git a/doc/license/README.md b/doc/license/README.md index 4cc387ba95f..b9281fd5299 100644 --- a/doc/license/README.md +++ b/doc/license/README.md @@ -2,4 +2,4 @@ redirect_to: 'https://docs.gitlab.com/ee/user/admin_area/license.html' --- -This document was moved to [user/admin_area/license](https://docs.gitlab.com/ee/user/admin_area/license.html). +This document was moved to [another location](https://docs.gitlab.com/ee/user/admin_area/license.html). diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 0531627b7b3..00b6c1dfdc2 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -17,11 +17,11 @@ This page gathers all the resources for the topic **Authentication** within GitL ## GitLab administrators - [LDAP (Community Edition)](../../administration/auth/ldap.md) -- [LDAP (Enterprise Edition)](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html) +- [LDAP (Enterprise Edition)](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html) **[STARTER]** - [Enforce Two-factor Authentication (2FA)](../../security/two_factor_authentication.md#enforce-two-factor-authentication-2fa) - **Articles:** - [How to Configure LDAP with GitLab CE](../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md) - - [How to Configure LDAP with GitLab EE](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html) + - [How to Configure LDAP with GitLab EE](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html) **[STARTER]** - [Feature Highlight: LDAP Integration](https://about.gitlab.com/2014/07/10/feature-highlight-ldap-sync/) - [Debugging LDAP](https://about.gitlab.com/handbook/support/workflows/support-engineering/ldap/debugging_ldap.html) - **Integrations:** @@ -30,10 +30,10 @@ This page gathers all the resources for the topic **Authentication** within GitL - [Atlassian Crowd OmniAuth Provider](../../administration/auth/crowd.md) - [CAS OmniAuth Provider](../../integration/cas.md) - [SAML OmniAuth Provider](../../integration/saml.md) - - [SAML for GitLab.com Groups](https://docs.gitlab.com/ee/user/group/saml_sso/index.html) - - [SCIM user provisioning for GitLab.com Groups](https://docs.gitlab.com/ee/user/group/saml_sso/scim_setup.html) + - [SAML for GitLab.com Groups](https://docs.gitlab.com/ee/user/group/saml_sso/index.html) **[SILVER ONLY]** + - [SCIM user provisioning for GitLab.com Groups](https://docs.gitlab.com/ee/user/group/saml_sso/scim_setup.html) **[SILVER ONLY]** - [Okta SSO provider](../../administration/auth/okta.md) - - [Kerberos integration (GitLab EE)](https://docs.gitlab.com/ee/integration/kerberos.html) + - [Kerberos integration (GitLab EE)](https://docs.gitlab.com/ee/integration/kerberos.html) **[STARTER]** ## API diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index e607dbceeea..b83abcd36f7 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -45,6 +45,22 @@ For other distributions, follow the instructions in PostgreSQL's [download page](https://www.postgresql.org/download/) to add their repository and then install `pgloader`. +If you are migrating to a Docker based installation, you will need to install +pgloader within the container as it is not included in the container image. + +1. Start a shell session in the context of the running container: + + ``` bash + docker exec -it gitlab bash + ``` + +1. Install pgloader: + + ``` bash + apt-get update + apt-get -y install pgloader + ``` + ## Omnibus GitLab installations For [Omnibus GitLab packages](https://about.gitlab.com/install/), you'll first diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index f90bff662e2..c2dff21b028 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -67,7 +67,7 @@ sudo -u git -H bundle exec rake gettext:pack RAILS_ENV=production sudo -u git -H bundle exec rake gettext:po_to_json RAILS_ENV=production # Clean up assets and cache -sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile cache:clear RAILS_ENV=production NODE_ENV=production +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile cache:clear RAILS_ENV=production NODE_ENV=production NODE_OPTIONS="--max_old_space_size=4096" ``` ### 4. Update gitlab-workhorse to the corresponding version diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index fea89669831..82a86f3f343 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -25,13 +25,7 @@ This section contains all the steps necessary to upgrade Community Edition or Enterprise Edition, regardless of the version you are upgrading to. Version specific guidelines (should there be any) are covered separately. -### 1. Stop server - -```bash -sudo service gitlab stop -``` - -### 2. Backup +### 1. Backup NOTE: If you installed GitLab from source, make sure `rsync` is installed. @@ -41,6 +35,12 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` +### 2. Stop server + +```bash +sudo service gitlab stop +``` + ### 3. Update Ruby NOTE: Beginning in GitLab 11.6, we only support Ruby 2.5 or higher, and dropped @@ -52,9 +52,9 @@ Download Ruby and compile it: ```bash mkdir /tmp/ruby && cd /tmp/ruby -curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.3.tar.gz -echo 'f919a9fbcdb7abecd887157b49833663c5c15fda ruby-2.5.3.tar.gz' | shasum -c - && tar xzf ruby-2.5.3.tar.gz -cd ruby-2.5.3 +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.gz +echo '2347ed6ca5490a104ebd5684d2b9b5eefa6cd33c ruby-2.6.3.tar.gz' | shasum -c - && tar xzf ruby-2.6.3.tar.gz +cd ruby-2.6.3 ./configure --disable-install-rdoc make @@ -314,7 +314,7 @@ sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production sudo -u git -H bundle exec rake gettext:compile RAILS_ENV=production # Update node dependencies and recompile assets -sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production NODE_OPTIONS="--max_old_space_size=4096" # Clean up cache sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index 45f986d480f..49959a9daef 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -2,7 +2,8 @@ To activate all GitLab Enterprise Edition (EE) functionality, you need to upload a license. Once you've received your license from GitLab Inc., you can upload it -by **signing into your GitLab instance as an admin**. +by **signing into your GitLab instance as an admin** or add it at +installation time. The license has the form of a base64 encoded ASCII text with a `.gitlab-license` extension and can be obtained when you [purchase one][pricing] or when you sign @@ -42,6 +43,36 @@ Otherwise, you can: "Enter license key" option, copy the license, paste it into the "License key" field and click **Upload license**. +## Add your license at install time + +The license may be automatically injected during installation using one of +two methods. + +The first requires a license file named `Gitlab.gitlab-release`. + +Place it in the `config/` directory if installing from source or in the +`/etc/gitlab/` directory if installing Omnibus. + +The second allows the administrator to configure the location and +filename of the license. + +Source installations should set the `GITLAB_LICENSE_FILE` environment +variable with the path to a valid GitLab Enterprise Edition license. + +```sh +export GITLAB_LICENSE_FILE="/path/to/license/file" +``` + +Omnibus installations should add this entry to `gitlab.rb`: + +```ruby +gitlab_rails['license_file'] = "/path/to/license/file" +``` + +CAUTION:: **Caution:** +These methods will only add a license at the time of installation. Use the +admin area in the web ui to renew or upgrade licenses. + --- Once the license is uploaded, all GitLab Enterprise Edition functionality diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 834a1e2423a..9dd476656ed 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -98,8 +98,7 @@ the group. NOTE: **Note:** Only available on GitLab.com. -If you have a Group with a [paid plan](https://about.gitlab.com/pricing/#gitlab-com) on GitLab.com, -then you can purchase additional CI minutes so your pipelines will not be blocked after you have +You can purchase additional CI minutes so your pipelines will not be blocked after you have used all your CI minutes from your main quota. In order to purchase additional minutes, you should follow these steps: @@ -119,7 +118,15 @@ will be synced to your Group and you can visualize it from the  -NOTE: **Important note**: If you have some minutes used over your default quota, these minutes will +Be aware that: + +1. If you have purchased extra CI minutes before the purchase of a paid plan, +we will calculate a pro-rated charge for your paid plan. That means you may +be charged for less than one year since your subscription was previously +created with the extra CI minutes. +1. Once the extra CI minutes has been assigned to a Group they cannot be transferred +to a different Group. +1. If you have some minutes used over your default quota, these minutes will be deducted from your Additional Minutes quota immediately after your purchase of additional minutes. diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index e165a120162..8b5d80efb0d 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -48,6 +48,8 @@ You can view the exact JSON payload in the administration panel. To view the pay 1. Expand **Settings** in the left sidebar and click on **Metrics and profiling**. 1. Expand **Usage statistics** and click on the **Preview payload** button. +You can see how [the usage ping data maps to different stages of the product](https://gitlab.com/gitlab-data/analytics/blob/master/transform/snowflake-dbt/data/ping_metrics_to_stage_mapping_data.csv). + ### Deactivate the usage ping The usage ping is opt-out. If you want to deactivate this feature, go to diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index ce86ade3c1a..5c635b09503 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -122,12 +122,13 @@ container_scanning: ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG CI_APPLICATION_TAG: $CI_COMMIT_SHA + CLAIR_LOCAL_SCAN_VERSION: v2.0.8_fe9b059d930314b54c78f75afe265955faf4fdc1 allow_failure: true services: - docker:stable-dind script: - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6 + - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:${CLAIR_LOCAL_SCAN_VERSION} - apk add -U wget ca-certificates - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 @@ -164,12 +165,13 @@ container_scanning: ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG CI_APPLICATION_TAG: $CI_COMMIT_SHA + CLAIR_LOCAL_SCAN_VERSION: v2.0.8_fe9b059d930314b54c78f75afe265955faf4fdc1 allow_failure: true services: - docker:stable-dind script: - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6 + - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:${CLAIR_LOCAL_SCAN_VERSION} - apk add -U wget ca-certificates - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 904c9e8fefe..f3b7d7fd471 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -138,7 +138,7 @@ variables: #### Customizing the DAST settings -The SAST settings can be changed through environment variables by using the +The DAST settings can be changed through environment variables by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. These variables are documented in the [DAST README](https://gitlab.com/gitlab-org/security-products/dast#settings). diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 248f8395db1..5d69efc3600 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -385,11 +385,6 @@ the Merge Request authored by the user that applied them.  - > **Note:** - The suggestion will only affect the commented line. Multi-line - suggestions are currently not supported. Will be introduced by - [#53310](https://gitlab.com/gitlab-org/gitlab-ce/issues/53310). - 1. In the comment, add your suggestion to the pre-populated code block:  @@ -401,13 +396,10 @@ the Merge Request authored by the user that applied them.  - > **Note:** - Discussions are _not_ automatically resolved. Will be introduced by - [#54405](https://gitlab.com/gitlab-org/gitlab-ce/issues/54405). - Once the author applies a suggestion, it will be marked with the **Applied** label, -and GitLab will create a new commit with the message `Apply suggestion to <file-name>` -and push the suggested change directly into the codebase in the merge request's branch. +the discussion will be automatically resolved, and GitLab will create a new commit +with the message `Apply suggestion to <file-name>` and push the suggested change +directly into the codebase in the merge request's branch. [Developer permission](../permissions.md) is required to do so. > **Note:** diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 8e101407ac0..f67325272a6 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -1,4 +1,4 @@ -# Custom group-level project templates **[PREMIUM ONLY]** +# Custom group-level project templates **[PREMIUM]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6861) in [GitLab Premium](https://about.gitlab.com/pricing) 11.6. diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index ee3137d032e..53116606201 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -22,8 +22,16 @@ SAML SSO for groups is used only as a convenient way to add users and does not s  -NOTE: **Note:** -Partial SSO enforcement was introduced in [11.8](https://gitlab.com/gitlab-org/gitlab-ee/issues/5291). With this option enabled, users must use your group's GitLab single sign on URL to be added to the group or be added via SCIM. Users can no longer be added manually. After a user has been added to the group, GitLab does not continue to enforce the use of SSO, but we'll [add a persistent check](https://gitlab.com/gitlab-org/gitlab-ee/issues/9255) in a later version. +### SSO enforcement + +SSO enforcement was: + +- [Introduced in GitLab 11.8](https://gitlab.com/gitlab-org/gitlab-ee/issues/5291). +- [Improved upon in GitLab 11.11 with ongoing enforcement in the GitLab UI](https://gitlab.com/gitlab-org/gitlab-ee/issues/9255). + +With this option enabled, users must use your group's GitLab single sign on URL to be added to the group or be added via SCIM. Users cannot be added manually, and may only access project/group resources via the UI by signing in through the SSO URL. + +We intend to add a similar SSO requirement for [Git and API activity](https://gitlab.com/gitlab-org/gitlab-ee/issues/9152) in the future. ### NameID diff --git a/doc/user/img/mermaid_diagram_render_gfm.png b/doc/user/img/mermaid_diagram_render_gfm.png Binary files differdeleted file mode 100644 index 9d192a30a85..00000000000 --- a/doc/user/img/mermaid_diagram_render_gfm.png +++ /dev/null diff --git a/doc/user/markdown.md b/doc/user/markdown.md index c1c2c036bae..5dad9621802 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -286,15 +286,15 @@ On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/he Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. ``` -Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/monkey.png" width="20px" height="20px" style="display:inline;margin:0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/star2.png" width="20px" height="20px" style="display:inline;margin:0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0">. Well we have a gift for you: +Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/monkey.png" width="20px" height="20px" style="display:inline;margin:0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/star2.png" width="20px" height="20px" style="display:inline;margin:0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0">. Well we have a gift for you: -<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/zap.png" width="20px" height="20px" style="display:inline;margin:0">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/v.png" width="20px" height="20px" style="display:inline;margin:0"> +<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/zap.png" width="20px" height="20px" style="display:inline;margin:0">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/v.png" width="20px" height="20px" style="display:inline;margin:0"> -You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/bug.png" width="20px" height="20px" style="display:inline;margin:0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/snail.png" width="20px" height="20px" style="display:inline;margin:0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/birthday.png" width="20px" height="20px" style="display:inline;margin:0">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/heart.png" width="20px" height="20px" style="display:inline;margin:0"> you for that. +You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/bug.png" width="20px" height="20px" style="display:inline;margin:0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/snail.png" width="20px" height="20px" style="display:inline;margin:0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/birthday.png" width="20px" height="20px" style="display:inline;margin:0">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/heart.png" width="20px" height="20px" style="display:inline;margin:0"> you for that. -If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/fearful.png" width="20px" height="20px" style="display:inline;margin:0">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/family.png" width="20px" height="20px" style="display:inline;margin:0">. All you need to do is to look up one of the supported codes. +If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/fearful.png" width="20px" height="20px" style="display:inline;margin:0">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/family.png" width="20px" height="20px" style="display:inline;margin:0">. All you need to do is to look up one of the supported codes. -Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0"> +Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0"> Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. @@ -480,7 +480,7 @@ GitLab 10.3. > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#mermaid -It is possible to generate diagrams and flowcharts from text using [Mermaid][mermaid]. +It is possible to generate diagrams and flowcharts from text using [Mermaid](https://mermaidjs.github.io/). In order to generate a diagram or flowchart, you should write your text inside the `mermaid` block. @@ -496,9 +496,15 @@ Example: Becomes: -<img src="./img/mermaid_diagram_render_gfm.png" width="200px" height="400px"> +```mermaid +graph TD; + A-->B; + A-->C; + B-->D; + C-->D; +``` -For details see the [Mermaid official page][mermaid]. +For details see the [Mermaid official page](https://mermaidjs.github.io/). ### Front matter @@ -1072,7 +1078,6 @@ A link starting with a `/` is relative to the wiki root. [^2]: This is my awesome footnote. [markdown.md]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md -[mermaid]: https://mermaidjs.github.io/ "Mermaid website" [rouge]: http://rouge.jneen.net/ "Rouge website" [redcarpet]: https://github.com/vmg/redcarpet "Redcarpet website" [katex]: https://github.com/Khan/KaTeX "KaTeX website" diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 9b298e4eb30..318053fdabb 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -11,7 +11,7 @@ project itself, the highest permission level is used. On public and internal projects the Guest role is not enforced. All users will be able to create issues, leave comments, and clone or download the project code. -When a member leaves the team all the assigned [Issues](project/issues/index.md) and [Merge Requests](project/merge_requests/index.md) +When a member leaves a team's project, all the assigned [Issues](project/issues/index.md) and [Merge Requests](project/merge_requests/index.md) will be unassigned automatically. GitLab [administrators](../administration/index.md) receive all permissions. diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 8e1603f9ec9..10eb3a4f3b7 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -321,8 +321,14 @@ X-Gitlab-Event: Issue Hook "group_id": 41 }], "changes": { - "updated_by_id": [null, 1], - "updated_at": ["2017-09-15 16:50:55 UTC", "2017-09-15 16:52:00 UTC"], + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current": "2017-09-15 16:52:00 UTC" + }, "labels": { "previous": [{ "id": 206, @@ -851,8 +857,14 @@ X-Gitlab-Event: Merge Request Hook "group_id": 41 }], "changes": { - "updated_by_id": [null, 1], - "updated_at": ["2017-09-15 16:50:55 UTC", "2017-09-15 16:52:00 UTC"], + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current":"2017-09-15 16:52:00 UTC" + }, "labels": { "previous": [{ "id": 206, diff --git a/doc/user/project/merge_requests/img/container_scanning.png b/doc/user/project/merge_requests/img/container_scanning.png Binary files differdeleted file mode 100644 index e47f62acd9d..00000000000 --- a/doc/user/project/merge_requests/img/container_scanning.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/create-issue-with-list-hover.png b/doc/user/project/merge_requests/img/create-issue-with-list-hover.png Binary files differdeleted file mode 100644 index 7d70e8299f5..00000000000 --- a/doc/user/project/merge_requests/img/create-issue-with-list-hover.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/dast_all.png b/doc/user/project/merge_requests/img/dast_all.png Binary files differdeleted file mode 100644 index b6edc928dc3..00000000000 --- a/doc/user/project/merge_requests/img/dast_all.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/dast_single.png b/doc/user/project/merge_requests/img/dast_single.png Binary files differdeleted file mode 100644 index 26ca4bde786..00000000000 --- a/doc/user/project/merge_requests/img/dast_single.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/dependency_scanning.png b/doc/user/project/merge_requests/img/dependency_scanning.png Binary files differdeleted file mode 100644 index 18df356f846..00000000000 --- a/doc/user/project/merge_requests/img/dependency_scanning.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/interactive_reports.png b/doc/user/project/merge_requests/img/interactive_reports.png Binary files differdeleted file mode 100644 index 9f9812dc69d..00000000000 --- a/doc/user/project/merge_requests/img/interactive_reports.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/license_management.png b/doc/user/project/merge_requests/img/license_management.png Binary files differdeleted file mode 100644 index cdce6b5fe38..00000000000 --- a/doc/user/project/merge_requests/img/license_management.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/license_management_decision.png b/doc/user/project/merge_requests/img/license_management_decision.png Binary files differdeleted file mode 100644 index 0763130c375..00000000000 --- a/doc/user/project/merge_requests/img/license_management_decision.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/license_management_pipeline_tab.png b/doc/user/project/merge_requests/img/license_management_pipeline_tab.png Binary files differdeleted file mode 100644 index 80ffca815b9..00000000000 --- a/doc/user/project/merge_requests/img/license_management_pipeline_tab.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/license_management_settings.png b/doc/user/project/merge_requests/img/license_management_settings.png Binary files differdeleted file mode 100644 index b5490e59074..00000000000 --- a/doc/user/project/merge_requests/img/license_management_settings.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/revert_changes_commit.png b/doc/user/project/merge_requests/img/revert_changes_commit.png Binary files differdeleted file mode 100644 index c9dd0019024..00000000000 --- a/doc/user/project/merge_requests/img/revert_changes_commit.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/revert_changes_mr.png b/doc/user/project/merge_requests/img/revert_changes_mr.png Binary files differdeleted file mode 100644 index 06b841b3002..00000000000 --- a/doc/user/project/merge_requests/img/revert_changes_mr.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/sast.png b/doc/user/project/merge_requests/img/sast.png Binary files differdeleted file mode 100644 index 2c75592c32a..00000000000 --- a/doc/user/project/merge_requests/img/sast.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/security_report.png b/doc/user/project/merge_requests/img/security_report.png Binary files differdeleted file mode 100644 index ba41b707238..00000000000 --- a/doc/user/project/merge_requests/img/security_report.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/vulnerability_solution.png b/doc/user/project/merge_requests/img/vulnerability_solution.png Binary files differdeleted file mode 100644 index 7443b9b6eea..00000000000 --- a/doc/user/project/merge_requests/img/vulnerability_solution.png +++ /dev/null diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md index 265871a7b4b..ea86633d9e2 100644 --- a/doc/user/project/merge_requests/merge_request_approvals.md +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -294,6 +294,18 @@ enabling [**Prevent approval of merge requests by their committers**](#prevent-a 1. Tick the checkbox **Prevent approval of merge requests by their committers**. 1. Click **Save changes**. +## Require authentication when approving a merge request **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5981) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.0. + +You can force the approver to enter a password in order to authenticate who is approving the merge request by +enabling **Require user password to approve**. This enables an Electronic Signature +for approvals such as the one defined by [CFR Part 11](https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1&subpartNode=21:1.0.1.1.8.3)): + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Tick the checkbox **Require user password to approve**. +1. Click **Save changes**. + ## Merge requests with different source branch and target branch projects If the merge request source branch and target branch belong to different diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index 6c3fa5eb463..d36312c9b8d 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -44,14 +44,14 @@ It is important to note that we have a few types of users: - **Administrators**: CI jobs created by Administrators will not have access to all GitLab projects, but only to projects and container images of projects - that the administrator is a member of.That means that if a project is either + that the administrator is a member of. That means that if a project is either public or internal users have access anyway, but if a project is private, the Administrator will have to be a member of it in order to have access to it via another project's job. - **External users**: CI jobs created by [external users](../permissions.md#external-users-permissions) will have access only to projects to which user has at least reporter access. This - rules out accessing all internal projects by default, + rules out accessing all internal projects by default. This allows us to make the CI and permission system more trustworthy. Let's consider the following scenario: diff --git a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md index ea22f3e905b..da1b7c59c8e 100644 --- a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md +++ b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md @@ -145,8 +145,8 @@ Now that your certificate has been issued, let's add it to your Pages site: 1. Visit your website at `https://example.com`. To force `https` connections on your site, navigate to your -project's **Settings > Pages** and check **Force domains with SSL -certificates to use HTTPS**. +project's **Settings > Pages** and check **Force HTTPS (requires +valid certificates)**. ## Renewal |
