diff options
Diffstat (limited to 'doc')
36 files changed, 871 insertions, 120 deletions
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index abef7a6cd33..0795d3dad40 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -52,10 +52,10 @@ is used. ### Network architecture - gitlab-rails shards repositories into "repository storages" -- gitlab-rails/config/gitlab.yml contains a map from storage names to +- `gitlab-rails/config/gitlab.yml` contains a map from storage names to (Gitaly address, Gitaly token) pairs - the `storage name` -\> `(Gitaly address, Gitaly token)` map in - gitlab.yml is the single source of truth for the Gitaly network + `gitlab.yml` is the single source of truth for the Gitaly network topology - a (Gitaly address, Gitaly token) corresponds to a Gitaly server - a Gitaly server hosts one or more storages @@ -65,7 +65,7 @@ is used. gitlab-shell, and Gitaly itself - special case: a Gitaly server must be able to make RPC calls **to itself** via its own (Gitaly address, Gitaly token) pair as - specified in gitlab-rails/config/gitlab.yml + specified in `gitlab-rails/config/gitlab.yml` - Gitaly servers must not be exposed to the public internet Gitaly network traffic is unencrypted so you should use a firewall to @@ -125,7 +125,7 @@ Omnibus installations: ```ruby # /etc/gitlab/gitlab.rb -# Avoid running unnecessary services on the gitaly server +# Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false @@ -153,7 +153,7 @@ gitaly['storage'] = [ { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' }, ] -# To use tls for gitaly you need to add +# To use TLS for Gitaly you need to add gitaly['tls_listen_addr'] = "0.0.0.0:9999" gitaly['certificate_path'] = "path/to/cert.pem" gitaly['key_path'] = "path/to/key.pem" @@ -239,11 +239,11 @@ repository from your GitLab server over HTTP. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22602) in GitLab 11.7. Gitaly supports TLS credentials for GRPC authentication. To be able to communicate -with a gitaly instance that listens for secure connections you will need to use `tls://` url +with a Gitaly instance that listens for secure connections you will need to use `tls://` url scheme in the `gitaly_address` of the corresponding storage entry in the gitlab configuration. The admin needs to bring their own certificate as we do not provide that automatically. -The certificate to be used needs to be installed on all gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) +The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) ### Example TLS configuration @@ -261,7 +261,7 @@ git_data_dirs({ gitlab_rails['gitaly_token'] = 'abc123secret' ``` -#### On gitaly server nodes: +#### On Gitaly server nodes: ```ruby gitaly['tls_listen_addr'] = "0.0.0.0:9999" @@ -289,7 +289,7 @@ gitlab: token: 'abc123secret' ``` -#### On gitaly server nodes: +#### On Gitaly server nodes: ```toml # /home/git/gitaly/config.toml diff --git a/doc/administration/index.md b/doc/administration/index.md index 0b673d61139..12fec2753bf 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -48,6 +48,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Third party offers](../user/admin_area/settings/third_party_offers.md) - [Compliance](compliance.md): A collection of features from across the application that you may configure to help ensure that your GitLab instance and DevOps workflow meet compliance standards. - [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering size limits of branch comparison pages. +- [Merge request diffs](merge_request_diffs.md): Configure the diffs shown on merge requests - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. #### Customizing GitLab's appearance @@ -64,6 +65,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Backup and restore](../raketasks/backup_restore.md): Backup and restore your GitLab instance. - [Operations](operations/index.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq MemoryKiller, Unicorn). - [Restart GitLab](restart_gitlab.md): Learn how to restart GitLab and its components. +- [Invalidate markdown cache](invalidate_markdown_cache.md): Invalidate any cached markdown. #### Updating GitLab diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md new file mode 100644 index 00000000000..ad64cb077c1 --- /dev/null +++ b/doc/administration/invalidate_markdown_cache.md @@ -0,0 +1,16 @@ +# Invalidate Markdown Cache + +For performance reasons, GitLab caches the HTML version of markdown text +(e.g. issue and merge request descriptions, comments). It's possible +that these cached versions become outdated, for example +when the `external_url` configuration option is changed - causing links +in the cached text to refer to the old URL. + +To avoid this problem, the administrator can invalidate the existing cache by +increasing the `local_markdown_version` setting in application settings. This can +be done by [changing the application settings through +the API](../api/settings.md#change-application-settings): + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/application/settings?local_markdown_version=<increased_number> +``` diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md new file mode 100644 index 00000000000..94620c3d3a0 --- /dev/null +++ b/doc/administration/merge_request_diffs.md @@ -0,0 +1,154 @@ +# Merge request diffs administration + +> **Notes:** +> - External merge request diffs introduced in GitLab 11.8 + +Merge request diffs are size-limited copies of diffs associated with merge +requests. When viewing a merge request, diffs are sourced from these copies +wherever possible as a performance optimization. + +By default, merge request diffs are stored in the database, in a table named +`merge_request_diff_files`. Larger installations may find this table grows too +large, in which case, switching to external storage is recommended. + +### Using external storage + +Merge request diffs can be stored on disk, or in object storage. In general, it +is better to store the diffs in the database than on disk. + +To enable external storage of merge request diffs: + +--- + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['external_diffs_enabled'] = true + ``` + +1. _The external diffs will be stored in in + `/var/opt/gitlab/gitlab-rails/shared/external-diffs`._ To change the path, + for example to `/mnt/storage/external-diffs`, edit `/etc/gitlab/gitlab.rb` + and add the following line: + + ```ruby + gitlab_rails['external_diffs_storage_path'] = "/mnt/storage/external-diffs" + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + external_diffs: + enabled: true + ``` + +1. _The external diffs will be stored in + `/home/git/gitlab/shared/external-diffs`._ To change the path, for example + to `/mnt/storage/external-diffs`, edit `/home/git/gitlab/config/gitlab.yml` + and add or amend the following lines: + + ```yaml + external_diffs: + enabled: true + storage_path: /mnt/storage/external-diffs + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. + +### Using object storage + +Instead of storing the external diffs on disk, we recommended you use an object +store like AWS S3 instead. This configuration relies on valid AWS credentials to +be configured already. + +### Object Storage Settings + +For source installations, these settings are nested under `external_diffs:` and +then `object_store:`. On omnibus installs, they are prefixed by +`external_diffs_object_store_`. + +| Setting | Description | Default | +|---------|-------------|---------| +| `enabled` | Enable/disable object storage | `false` | +| `remote_directory` | The bucket name where external diffs will be stored| | +| `direct_upload` | Set to true to enable direct upload of external diffs without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | +| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `connection` | Various connection options described below | | + +#### S3 compatible connection settings + +The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: + +| Setting | Description | Default | +|---------|-------------|---------| +| `provider` | Always `AWS` for compatible hosts | AWS | +| `aws_access_key_id` | AWS credentials, or compatible | | +| `aws_secret_access_key` | AWS credentials, or compatible | | +| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `region` | AWS region | us-east-1 | +| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | +| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | +| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false | +| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['external_diffs_enabled'] = true + gitlab_rails['external_diffs_object_store_enabled'] = true + gitlab_rails['external_diffs_object_store_remote_directory'] = "external-diffs" + gitlab_rails['external_diffs_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: if you are using AWS IAM profiles, be sure to omit the + AWS access key and secret access key/value pairs. For example: + + ```ruby + gitlab_rails['external_diffs_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + external_diffs: + enabled: true + object_store: + enabled: true + remote_directory: "external-diffs" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index c9a2778b3a4..6ea0ac0d495 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -48,6 +48,8 @@ The following metrics are available: | upload_file_does_not_exist | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | | failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | +| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | +| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | ### Ruby metrics diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index 0397452e650..c0c242733a2 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -7,13 +7,72 @@ systems. Normally when talking about filesystem performance the biggest concern is with Network Filesystems (NFS). However, even some local disks can have slow -IO. The information on this page can be used for either scenario. +I/O. The information on this page can be used for either scenario. -## Write Performance +## Executing benchmarks -The following one-line command is a quick benchmark for filesystem write +### Benchmarking with `fio` + +We recommend using +[fio](https://fio.readthedocs.io/en/latest/fio_doc.html) to test I/O +performance. This test should be run both on the NFS server and on the +application nodes that talk to the NFS server. + +To install: + +- On Ubuntu: `apt install fio`. +- On `yum`-managed environments: `yum install fio`. + +Then run the following: + +```sh +fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/path/to/git-data/testfile --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 +``` + +This will create a 4GB file in `/path/to/git-data/testfile`. It performs +4KB reads and writes using a 75%/25% split within the file, with 64 +operations running at a time. Be sure to delete the file after the test +completes. + +The output will vary depending on what version of `fio` installed. The following +is an example output from `fio` v2.2.10 on a networked solid-state drive (SSD): + +``` +test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64 + fio-2.2.10 + Starting 1 process + test: Laying out IO file(s) (1 file(s) / 1024MB) + Jobs: 1 (f=1): [m(1)] [100.0% done] [131.4MB/44868KB/0KB /s] [33.7K/11.3K/0 iops] [eta 00m:00s] + test: (groupid=0, jobs=1): err= 0: pid=10287: Sat Feb 2 17:40:10 2019 + read : io=784996KB, bw=133662KB/s, iops=33415, runt= 5873msec + write: io=263580KB, bw=44880KB/s, iops=11219, runt= 5873msec + cpu : usr=6.56%, sys=23.11%, ctx=266267, majf=0, minf=8 + IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% + issued : total=r=196249/w=65895/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 + latency : target=0, window=0, percentile=100.00%, depth=64 + + Run status group 0 (all jobs): + READ: io=784996KB, aggrb=133661KB/s, minb=133661KB/s, maxb=133661KB/s, mint=5873msec, maxt=5873msec + WRITE: io=263580KB, aggrb=44879KB/s, minb=44879KB/s, maxb=44879KB/s, mint=5873msec, maxt=5873msec +``` + +Notice the `iops` values in this output. In this example, the SSD +performed 33,415 read operations per second and 11,219 write operations +per second. A spinning disk might yield 2,000 and 700 read and write +operations per second. + +### Simple benchmarking + +NOTE: **Note:** This test is naive but may be useful if `fio` is not +available on the system. It's possible to receive good results on this +test but still have poor performance due to read speed and various other +factors. + +The following one-line commands provide a quick benchmark for filesystem write and read performance. This will write 1,000 small files to the directory in which it is -executed. +executed, and then read the same 1,000 files. 1. Change into the root of the appropriate [repository storage path](../repository_storage_paths.md). @@ -27,13 +86,18 @@ executed. ```sh time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done ``` +1. To benchmark read performance, run the command: + + ```sh + time for i in {0..1000}; do cat "test${i}.txt" > /dev/null; done + ``` 1. Remove the test files: ```sh cd ../; rm -rf test ``` -The output of the `time for ...` command will look similar to the following. The +The output of the `time for ...` commands will look similar to the following. The important metric is the `real` time. ```sh @@ -42,12 +106,13 @@ $ time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done real 0m0.116s user 0m0.025s sys 0m0.091s + +$ time for i in {0..1000}; do cat "test${i}.txt" > /dev/null; done + +real 0m3.118s +user 0m1.267s +sys 0m1.663s ``` From experience with multiple customers, this task should take under 10 -seconds to indicate good filesystem performance. - -NOTE: **Note:** -This test is naive and only evaluates write performance. It's possible to -receive good results on this test but still have poor performance due to read -speed and various other factors.
\ No newline at end of file +seconds to indicate good filesystem performance. diff --git a/doc/api/README.md b/doc/api/README.md index 692f63a400c..a060e0481bf 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -29,6 +29,7 @@ The following API resources are available: - [Group access requests](access_requests.md) - [Group badges](group_badges.md) - [Group issue boards](group_boards.md) + - [Group labels](group_labels.md) - [Group-level variables](group_level_variables.md) - [Group members](members.md) - [Group milestones](group_milestones.md) diff --git a/doc/api/group_labels.md b/doc/api/group_labels.md new file mode 100644 index 00000000000..c36d34b4af1 --- /dev/null +++ b/doc/api/group_labels.md @@ -0,0 +1,201 @@ +# Group Label API + +>**Note:** This feature was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21368) in GitLab 11.8. + +This API supports managing of [group labels](../user/project/labels.md#project-labels-and-group-labels). It allows to list, create, update, and delete group labels. Furthermore, users can subscribe and unsubscribe to and from group labels. + +## List group labels + +Get all labels for a given group. + +``` +GET /groups/:id/labels +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/labels +``` + +Example response: + +```json +[ + { + "id": 7, + "name": "bug", + "color": "#FF0000", + "description": null, + "open_issues_count": 0, + "closed_issues_count": 0, + "open_merge_requests_count": 0, + "subscribed": false + }, + { + "id": 4, + "name": "feature", + "color": "#228B22", + "description": null, + "open_issues_count": 0, + "closed_issues_count": 0, + "open_merge_requests_count": 0, + "subscribed": false + } +] +``` + +## Create a new group label + +Create a new group label for a given group. + +``` +POST /groups/:id/labels +``` + +| Attribute | Type | Required | Description | +| ------------- | ------- | -------- | ---------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the label | +| `color` | string | yes | The color of the label given in 6-digit hex notation with leading '#' sign (e.g. #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | +| `description` | string | no | The description of the label | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"name": "Feature Proposal", "color": "#FFA500", "description": "Describes new ideas" }' https://gitlab.example.com/api/v4/groups/5/labels +``` + +Example response: + +```json +{ + "id": 9, + "name": "Feature Proposal", + "color": "#FFA500", + "description": "Describes new ideas", + "open_issues_count": 0, + "closed_issues_count": 0, + "open_merge_requests_count": 0, + "subscribed": false +} +``` + +## Update a group label + +Updates an existing group label. At least one parameter is required, to update the group label. + +``` +PUT /groups/:id/labels +``` + +| Attribute | Type | Required | Description | +| ------------- | ------- | -------- | ---------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the label | +| `new_name` | string | no | The new name of the label | +| `color` | string | no | The color of the label given in 6-digit hex notation with leading '#' sign (e.g. #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | +| `description` | string | no | The description of the label | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"name": "Feature Proposal", "new_name": "Feature Idea" }' https://gitlab.example.com/api/v4/groups/5/labels +``` + +Example response: + +```json +{ + "id": 9, + "name": "Feature Idea", + "color": "#FFA500", + "description": "Describes new ideas", + "open_issues_count": 0, + "closed_issues_count": 0, + "open_merge_requests_count": 0, + "subscribed": false +} +``` + +## Delete a group label + +Deletes a group label with a given name. + +``` +DELETE /groups/:id/labels +``` + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | --------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the label | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/labels?name=bug +``` + +## Subscribe to a group label + +Subscribes the authenticated user to a group label to receive notifications. If +the user is already subscribed to the label, the status code `304` is returned. + +``` +POST /groups/:id/labels/:label_id/subscribe +``` + +| Attribute | Type | Required | Description | +| ---------- | ----------------- | -------- | ------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `label_id` | integer or string | yes | The ID or title of a group's label | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/labels/9/subscribe +``` + +Example response: + +```json +{ + "id": 9, + "name": "Feature Idea", + "color": "#FFA500", + "description": "Describes new ideas", + "open_issues_count": 0, + "closed_issues_count": 0, + "open_merge_requests_count": 0, + "subscribed": true +} +``` + +## Unsubscribe from a group label + +Unsubscribes the authenticated user from a group label to not receive +notifications from it. If the user is not subscribed to the label, the status +code `304` is returned. + +``` +POST /groups/:id/labels/:label_id/unsubscribe +``` + +| Attribute | Type | Required | Description | +| ---------- | ----------------- | -------- | ------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `label_id` | integer or string | yes | The ID or title of a group's label | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/labels/9/unsubscribe +``` + +Example response: + +```json +{ + "id": 9, + "name": "Feature Idea", + "color": "#FFA500", + "description": "Describes new ideas", + "open_issues_count": 0, + "closed_issues_count": 0, + "open_merge_requests_count": 0, + "subscribed": false +} +``` diff --git a/doc/api/issues.md b/doc/api/issues.md index 6d8683601f6..ed3165d95df 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -31,6 +31,7 @@ GET /issues?iids[]=42&iids[]=43 GET /issues?author_id=5 GET /issues?assignee_id=5 GET /issues?my_reaction_emoji=star +GET /issues?search=foo&in=title ``` | Attribute | Type | Required | Description | @@ -46,6 +47,7 @@ GET /issues?my_reaction_emoji=star | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search issues against their `title` and `description` | +| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | | `created_after` | datetime | no | Return issues created on or after the given time | | `created_before` | datetime | no | Return issues created on or before the given time | | `updated_after` | datetime | no | Return issues updated on or after the given time | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index b3548391228..d58cd45538d 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -24,6 +24,7 @@ GET /merge_requests?labels=bug,reproduced GET /merge_requests?author_id=5 GET /merge_requests?my_reaction_emoji=star GET /merge_requests?scope=assigned_to_me +GET /merge_requests?search=foo&in=title ``` Parameters: @@ -47,6 +48,7 @@ Parameters: | `source_branch` | string | no | Return merge requests with the given source branch | | `target_branch` | string | no | Return merge requests with the given target branch | | `search` | string | no | Search merge requests against their `title` and `description` | +| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | | `wip` | string | no | Filter merge requests against their `wip` status. `yes` to return *only* WIP merge requests, `no` to return *non* WIP merge requests | ```json @@ -992,6 +994,8 @@ Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user - `merge_request_iid` (required) - Internal ID of MR - `merge_commit_message` (optional) - Custom merge commit message +- `squash_commit_message` (optional) - Custom squash commit message +- `squash` (optional) - if `true` the commits will be squashed into a single commit on merge - `should_remove_source_branch` (optional) - if `true` removes the source branch - `merge_when_pipeline_succeeds` (optional) - if `true` the MR is merged when the pipeline succeeds - `sha` (optional) - if present, then this SHA must match the HEAD of the source branch, otherwise the merge will fail diff --git a/doc/api/settings.md b/doc/api/settings.md index c329e3cdf24..2e0a2a09133 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -57,11 +57,13 @@ Example response: "dsa_key_restriction": 0, "ecdsa_key_restriction": 0, "ed25519_key_restriction": 0, + "first_day_of_week": 0, "enforce_terms": true, "terms": "Hello world!", "performance_bar_allowed_group_id": 42, "instance_statistics_visibility_private": false, - "user_show_add_ssh_key_message": true + "user_show_add_ssh_key_message": true, + "local_markdown_version": 0 } ``` @@ -113,11 +115,13 @@ Example response: "dsa_key_restriction": 0, "ecdsa_key_restriction": 0, "ed25519_key_restriction": 0, + "first_day_of_week": 0, "enforce_terms": true, "terms": "Hello world!", "performance_bar_allowed_group_id": 42, "instance_statistics_visibility_private": false, - "user_show_add_ssh_key_message": true + "user_show_add_ssh_key_message": true, + "local_markdown_version": 0 } ``` @@ -157,6 +161,7 @@ are listed in the descriptions of the relevant settings. | `email_author_in_body` | boolean | no | Some email servers do not support overriding the email sender name. Enable this option to include the name of the author of the issue, merge request or comment in the email body instead. | | `enabled_git_access_protocol` | string | no | Enabled protocols for Git access. Allowed values are: `ssh`, `http`, and `nil` to allow both protocols. | | `enforce_terms` | boolean | no | (**If enabled, requires:** `terms`) Enforce application ToS to all users. | +| `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday and `1` for Monday. | | `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | @@ -235,3 +240,4 @@ are listed in the descriptions of the relevant settings. | `user_oauth_applications` | boolean | no | Allow users to register any application to use GitLab as an OAuth provider. | | `user_show_add_ssh_key_message` | boolean | no | When set to `false` disable the "You won't be able to pull or push project code via SSH" warning shown to users with no uploaded SSH key. | | `version_check_enabled` | boolean | no | Let GitLab inform you when an update is available. | +| `local_markdown_version` | integer | no | Increase this value when any cached markdown should be invalidated. | diff --git a/doc/api/users.md b/doc/api/users.md index 6000b9b900f..fd8778abb17 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -1212,6 +1212,7 @@ The activities that update the timestamp are: - Git HTTP/SSH activities (such as clone, push) - User logging in into GitLab + - User visiting pages related to Dashboards, Projects, Issues and Merge Requests ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/54947) in GitLab 11.8) By default, it shows the activity for all users in the last 6 months, but this can be amended by using the `from` parameter. diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 97e133a2e2f..32c73c4f398 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -98,7 +98,7 @@ future GitLab releases.** | **CI_PIPELINE_SOURCE** | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | | **CI_PIPELINE_TRIGGERED** | all | all | The flag to indicate that job was [triggered] | | **CI_PIPELINE_URL** | 11.1 | 0.5 | Pipeline details URL | -| **CI_PROJECT_DIR** | all | all | The full path where the repository is cloned and where the job is run | +| **CI_PROJECT_DIR** | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | | **CI_PROJECT_ID** | all | all | The unique id of the current project that GitLab CI uses internally | | **CI_PROJECT_NAME** | 8.10 | 0.5 | The project name that is currently being built (actually it is project folder name) | | **CI_PROJECT_NAMESPACE** | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | diff --git a/doc/development/README.md b/doc/development/README.md index 05715274a81..d5829e31343 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -123,3 +123,7 @@ description: 'Learn how to contribute to GitLab.' ## Compliance - [Licensing](licensing.md) for ensuring license compliance + +## Go guides + +- [Go Guidelines](go_guide/index.md) diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 24feb1378a1..c5344139ab4 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -20,7 +20,7 @@ All labels, their meaning and priority are defined on the If you come across an issue that has none of these, and you're allowed to set labels, you can _always_ add the team and type, and often also the subject. -[milestones-page]: https://gitlab.com/gitlab-org/gitlab-ce/milestones +[milestones-page]: https://gitlab.com/groups/gitlab-org/-/milestones ## Type labels diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 9bef0635e3f..19b6181c9a2 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -86,6 +86,9 @@ request is as follows: guidelines](../merge_request_performance_guidelines.md). 1. For tests that use Capybara or PhantomJS, see this [article on how to write reliable asynchronous tests](https://robots.thoughtbot.com/write-reliable-asynchronous-integration-tests-with-capybara). +1. If your merge request introduces changes that require additional steps when + installing GitLab from source, add them to `doc/install/installation.md` in + the same merge request. Please keep the change in a single MR **as small as possible**. If you want to contribute a large feature think very hard what the minimum viable change is. diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 6f1ba5d62a5..0eedef5e14f 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -21,6 +21,7 @@ of _prohibited this user from being saved due to the following errors:_ the text should be _sorry, we could not create your account because:_ 1. Code should be written in [US English][us-english] +1. [Go](../go_guide/index.md) This is also the style used by linting tools such as [RuboCop](https://github.com/bbatsov/rubocop) and [Hound CI](https://houndci.com). diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index b90dc90e424..597812c8c49 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -18,6 +18,7 @@ There are many places where file uploading is used, according to contexts: - Issues/MR/Notes Legacy Markdown attachments - CI Artifacts (archive, metadata, trace) - LFS Objects + - Merge request diffs ## Disk storage @@ -37,6 +38,7 @@ they are still not 100% standardized. You can see them below: | Issues/MR/Notes Legacy Markdown attachments | no | uploads/-/system/note/attachment/:id/:filename | `AttachmentUploader` | Note | | CI Artifacts (CE) | yes | shared/artifacts/:disk_hash[0..1]/:disk_hash[2..3]/:disk_hash/:year_:month_:date/:job_id/:job_artifact_id (:disk_hash is SHA256 digest of project_id) | `JobArtifactUploader` | Ci::JobArtifact | | LFS Objects (CE) | yes | shared/lfs-objects/:hex/:hex/:object_hash | `LfsObjectUploader` | LfsObject | +| External merge request diffs | yes | shared/external-diffs/merge_request_diffs/mr-:parent_id/diff-:id | `ExternalDiffUploader` | MergeRequestDiff | CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader` while in EE they inherit the `ObjectStorage` and store files in and S3 API compatible object store. diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md new file mode 100644 index 00000000000..cdc806a2d31 --- /dev/null +++ b/doc/development/go_guide/index.md @@ -0,0 +1,216 @@ +# Go standards and style guidelines + +This document describes various guidelines and best practices for GitLab +projects using the [Go language](https://golang.org). + +## Overview + +GitLab is built on top of [Ruby on Rails](https://rubyonrails.org/), but we're +also using Go for projects where it makes sense. Go is a very powerful +language, with many advantages, and is best suited for projects with a lot of +IO (disk/network access), HTTP requests, parallel processing, etc. Since we +have both Ruby on Rails and Go at GitLab, we should evaluate carefully which of +the two is best for the job. + +This page aims to define and organize our Go guidelines, based on our various +experiences. Several projects were started with different standards and they +can still have specifics. They will be described in their respective +`README.md` or `PROCESS.md` files. + +## Code Review + +We follow the common principles of +[Go Code Review Comments](https://github.com/golang/go/wiki/CodeReviewComments). + +Reviewers and maintainers should pay attention to: + +- `defer` functions: ensure the presence when needed, and after `err` check. +- Inject dependencies as parameters. +- Void structs when marshalling to JSON (generates `null` instead of `[]`). + +### Security + +Security is our top priority at GitLab. During code reviews, we must take care +of possible security breaches in our code: + +- XSS when using text/template +- CSRF Protection using Gorilla +- Use a Go version without known vulnerabilities +- Don't leak secret tokens +- SQL injections + +Remember to run +[SAST](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html) +**[ULTIMATE]** on your project (or at least the [gosec +analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), +and to follow our [Security +requirements](../code_review.md#security-requirements). + +Web servers can take advantages of middlewares like [Secure](https://github.com/unrolled/secure). + +### Finding a reviewer + +Many of our projects are too small to have full-time maintainers. That's why we +have a shared pool of Go reviewers at GitLab. To find a reviewer, use the +[Engineering Projects](https://about.gitlab.com/handbook/engineering/projects/) +page in the handbook. "GitLab Community Edition (CE)" and "GitLab Community +Edition (EE)" both have a "Go" section with its list of reviewers. + +To add yourself to this list, add the following to your profile in the +[team.yml](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/team.yml) +file and ask your manager to review and merge. + +```yaml +projects: + gitlab-ee: reviewer go + gitlab-ce: reviewer go +``` + +## Code style and format + +- Avoid global variables, even in packages. By doing so you will introduce side + effects if the package is included multiple times. +- Use `go fmt` before committing ([Gofmt](https://golang.org/cmd/gofmt/) is a + tool that automatically formats Go source code). + +### Automatic linting + +All Go projects should include these GitLab CI/CD jobs: + +```yaml +go lint: + image: golang:1.11 + script: + - go get -u golang.org/x/lint/golint + - golint -set_exit_status +``` + +Once [recursive includes](https://gitlab.com/gitlab-org/gitlab-ce/issues/56836) +become available, you will be able to share job templates like this +[analyzer](https://gitlab.com/gitlab-org/security-products/ci-templates/raw/master/includes-dev/analyzer.yml). + +## Dependencies + +Dependencies should be kept to the minimum. The introduction of a new +dependency should be argued in the merge request, as per our [Approval +Guidelines](../code_review.html#approval-guidelines). Both [License +Management](https://docs.gitlab.com/ee/user/project/merge_requests/license_management.html) +**[ULTIMATE]** and [Dependency +Scanning](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html) +**[ULTIMATE]** should be activated on all projects to ensure new dependencies +security status and license compatibility. + +### Modules + +Since Go 1.11, a standard dependency system is available behind the name [Go +Modules](https://github.com/golang/go/wiki/Modules). It provides a way to +define and lock dependencies for reproducible builds. It should be used +whenever possible. + +There was a [bug on modules +checksums](https://github.com/golang/go/issues/29278) in Go < v1.11.4, so make +sure to use at least this version to avoid `checksum mismatch` errors. + +### ORM + +We don't use object-relational mapping libraries (ORMs) at GitLab (except +[ActiveRecord](https://guides.rubyonrails.org/active_record_basics.html) in +Ruby on Rails). Projects can be structured with services to avoid them. +[PQ](https://github.com/lib/pq) should be enough to interact with PostgreSQL +databases. + +### Migrations + +In the rare event of managing a hosted database, it's necessary to use a +migration system like ActiveRecord is providing. A simple library like +[Journey](https://github.com/db-journey/journey), designed to be used in +`postgres` containers, can be deployed as long-running pods. New versions will +deploy a new pod, migrating the data automatically. + +## Testing + +We should not use any specific library or framework for testing, as the +[standard library](https://golang.org/pkg/) provides already everything to get +started. For example, some external dependencies might be worth considering in +case we decide to use a specific library or framework: + +- [Testify](https://github.com/stretchr/testify) +- [httpexpect](https://github.com/gavv/httpexpect) + +Use [subtests](https://blog.golang.org/subtests) whenever possible to improve +code readability and test output. + +### Benchmarks + +Programs handling a lot of IO or complex operations should always include +[benchmarks](https://golang.org/pkg/testing/#hdr-Benchmarks), to ensure +performance consistency over time. + +## CLIs + +Every Go program is launched from the command line. +[cli](https://github.com/urfave/cli) is a convenient package to create command +line apps. It should be used whether the project is a daemon or a simple cli +tool. Flags can be mapped to [environment +variables](https://github.com/urfave/cli#values-from-the-environment) directly, +which documents and centralizes at the same time all the possible command line +interactions with the program. Don't use `os.GetEnv`, it hides variables deep +in the code. + +## Daemons + +### Logging + +The usage of a logging library is strongly recommended for daemons. Even though +there is a `log` package in the standard library, we generally use +[logrus](https://github.com/sirupsen/logrus). Its plugin ("hooks") system +makes it a powerful logging library, with the ability to add notifiers and +formatters at the logger level directly. + +### Tracing and Correlation + +[LabKit](https://gitlab.com/gitlab-org/labkit) is a place to keep common +libraries for Go services. Currently it's vendored into two projects: +Workhorse and Gitaly, and it exports two main (but related) pieces of +functionality: + +- [`gitlab.com/gitlab-org/labkit/correlation`](https://gitlab.com/gitlab-org/labkit/tree/master/correlation): + for propagating and extracting correlation ids between services. +- [`gitlab.com/gitlab-org/labkit/tracing`](https://gitlab.com/gitlab-org/labkit/tree/master/tracing): + for instrumenting Go libraries for distributed tracing. + +This gives us a thin abstraction over underlying implementations that is +consistent across Workhorse, Gitaly, and, in future, other Go servers. For +example, in the case of `gitlab.com/gitlab-org/labkit/tracing` we can switch +from using Opentracing directly to using Zipkin or Gokit's own tracing wrapper +without changes to the application code, while still keeping the same +consistent configuration mechanism (i.e. the `GITLAB_TRACING` environment +variable). + +### Context + +Since daemons are long-running applications, they should have mechanisms to +manage cancellations, and avoid unnecessary resources consumption (which could +lead to DDOS vulnerabilities). [Go +Context](https://github.com/golang/go/wiki/CodeReviewComments#contexts) should +be used in functions that can block and passed as the first parameter. + +## Dockerfiles + +Every project should have a `Dockerfile` at the root of their repository, to +build and run the project. Since Go program are static binaries, they should +not require any external dependency, and shells in the final image are useless. +We encourage [Multistage +builds](https://docs.docker.com/develop/develop-images/multistage-build/): + +- They let the user build the project with the right Go version and + dependencies. +- They generate a small, self-contained image, derived from `Scratch`. + +Generated docker images should have the program at their `Entrypoint` to create +portable commands. That way, anyone can run the image, and without parameters +it will display its help message (if `cli` has been used). + +--- + +[Return to Development documentation](../README.md). diff --git a/doc/development/sql.md b/doc/development/sql.md index 06005a0a6f8..47519d39e74 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -256,32 +256,12 @@ violation, for example. Using transactions does not solve this problem. -The following pattern should be used to avoid the problem: +To solve this we've added the `ApplicationRecord.safe_find_or_create_by`. -```ruby -Project.transaction do - begin - User.find_or_create_by(username: "foo") - rescue ActiveRecord::RecordNotUnique - retry - end -end -``` - -If the above block is run inside a transaction and hits the race -condition, the transaction is aborted and we cannot simply retry (any -further queries inside the aborted transaction are going to fail). We -can employ [nested transactions](http://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html#module-ActiveRecord::Transactions::ClassMethods-label-Nested+transactions) -here to only rollback the "inner transaction". Note that `requires_new: true` is required here. +This method can be used just as you would the normal +`find_or_create_by` but it wraps the call in a *new* transaction and +retries if it were to fail because of an +`ActiveRecord::RecordNotUnique` error. -```ruby -Project.transaction do - begin - User.transaction(requires_new: true) do - User.find_or_create_by(username: "foo") - end - rescue ActiveRecord::RecordNotUnique - retry - end -end -``` +To be able to use this method, make sure the model you want to use +this on inherits from `ApplicationRecord`. diff --git a/doc/install/installation.md b/doc/install/installation.md index 1f65e3415d1..a8064ae046e 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -139,8 +139,8 @@ Then select 'Internet Site' and press enter to confirm the hostname. The Ruby interpreter is required to run GitLab. -**Note:** The current supported Ruby (MRI) version is 2.3.x. GitLab 9.0 dropped -support for Ruby 2.1.x. +**Note:** The current supported Ruby (MRI) version is 2.5.x. GitLab 11.6 + dropped support for Ruby 2.4.x. The use of Ruby version managers such as [RVM], [rbenv] or [chruby] with GitLab in production, frequently leads to hard to diagnose problems. For example, @@ -345,11 +345,15 @@ cd /home/git ```sh # Clone GitLab repository -sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 11-7-stable gitlab +sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b X-Y-stable gitlab ``` +Make sure to replace `X-Y-stable` with the stable branch that matches the +version you want to install. For example, if you want to install 11.8 you would +use the branch name `11-8-stable`. + CAUTION: **Caution:** -You can change `11-7-stable` to `master` if you want the *bleeding edge* version, but never install `master` on a production server! +You can change `X-Y-stable` to `master` if you want the *bleeding edge* version, but never install `master` on a production server! ### Configure It @@ -691,6 +695,11 @@ sudo nginx -t You should receive `syntax is okay` and `test is successful` messages. If you receive errors check your `gitlab` or `gitlab-ssl` Nginx config file for typos, etc. as indicated in the error message given. +NOTE: **Note:** +Verify that the installed version is greater than 1.12.1 by running `nginx -v`. If it's lower, you may receive the error below: +`nginx: [emerg] unknown "start$temp=[filtered]$rest" variable +nginx: configuration file /etc/nginx/nginx.conf test failed` + ### Restart ```sh diff --git a/doc/ssh/README.md b/doc/ssh/README.md index 09a97fcea07..1b53be15b44 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -19,7 +19,7 @@ comes pre-installed on GNU/Linux and macOS, but not on Windows. Depending on your Windows version, there are different methods to work with SSH keys. -### Installing the SSH client for Windows 10 +### Windows 10: Windows Subsystem for Linux Starting with Windows 10, you can [install the Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/install-win10) @@ -27,10 +27,10 @@ where you can run Linux distributions directly on Windows, without the overhead of a virtual machine. Once installed and set up, you'll have the Git and SSH clients at your disposal. -### Installing the SSH client for Windows 8.1 and Windows 7 +### Windows 10, 8.1, and 7: Git for Windows The easiest way to install Git and the SSH client on Windows 8.1 and Windows 7 -is [Git for Windows](https://gitforwindows.org). It provides a BASH +is [Git for Windows](https://gitforwindows.org). It provides a Bash emulation (Git Bash) used for running Git from the command line and the `ssh-keygen` command that is useful to create SSH keys as you'll learn below. @@ -114,7 +114,8 @@ To create a new SSH key pair: and want to tell which is which. It is optional. 1. Next, you will be prompted to input a file path to save your SSH key pair to. - If you don't already have an SSH key pair, use the suggested path by pressing + If you don't already have an SSH key pair and aren't generating a [deploy key](#deploy-keys), + use the suggested path by pressing <kbd>Enter</kbd>. Using the suggested path will normally allow your SSH client to automatically use the SSH key pair with no additional configuration. @@ -128,7 +129,7 @@ To create a new SSH key pair: <kbd>Enter</kbd> twice. If, in any case, you want to add or change the password of your SSH key pair, - you can use the `-p`flag: + you can use the `-p` flag: ``` ssh-keygen -p -o -f <keyname> @@ -258,7 +259,8 @@ Integration (CI) server. By using deploy keys, you don't have to set up a dummy user account. If you are a project maintainer or owner, you can add a deploy key in the -project settings under the section 'Repository'. Specify a title for the new +project's **Settings > Repository** page by expanding the +**Deploy Keys** section. Specify a title for the new deploy key and paste a public SSH key. After this, the machine that uses the corresponding private SSH key has read-only or read-write (if enabled) access to the project. @@ -300,8 +302,8 @@ of broader usage for something like "Anywhere you need to give read access to your repository". Once a GitLab administrator adds the Global Deployment key, project maintainers -and owners can add it in project's **Settings > Repository** section by expanding the -**Deploy Key** section and clicking **Enable** next to the appropriate key listed +and owners can add it in project's **Settings > Repository** page by expanding the +**Deploy Keys** section and clicking **Enable** next to the appropriate key listed under **Public deploy keys available to any project**. NOTE: **Note:** diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 325de50cab0..463bdd59282 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -126,14 +126,22 @@ Auto Deploy, and Auto Monitoring will be silently skipped. ## Auto DevOps base domain +NOTE: **Note** +`AUTO_DEVOPS_DOMAIN` environment variable is deprecated and +[is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959) in GitLab 12.0. + The Auto DevOps base domain is required if you want to make use of [Auto Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It can be defined -in three places: +in any of the following places: -- either under the project's CI/CD settings while [enabling Auto DevOps](#enabling-auto-devops) +- either under the cluster's settings, whether for [projects](../../user/project/clusters/index.md#base-domain) or [groups](../../user/group/clusters/index.md#base-domain) - or in instance-wide settings in the **admin area > Settings** under the "Continuous Integration and Delivery" section -- or at the project as a variable: `AUTO_DEVOPS_DOMAIN` (required if you want to use [multiple clusters](#using-multiple-kubernetes-clusters)) -- or at the group level as a variable: `AUTO_DEVOPS_DOMAIN` +- or at the project level as a variable: `KUBE_INGRESS_BASE_DOMAIN` +- or at the group level as a variable: `KUBE_INGRESS_BASE_DOMAIN`. + +NOTE: **Note** +The Auto DevOps base domain variable (`KUBE_INGRESS_BASE_DOMAIN`) follows the same order of precedence +as other environment [variables](../../ci/variables/README.md#priority-of-variables). A wildcard DNS A record matching the base domain(s) is required, for example, given a base domain of `example.com`, you'd need a DNS entry like: @@ -170,13 +178,13 @@ In the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab-ce/blob/maste Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so except for the environment scope, they would also need to have a different domain they would be deployed to. This is why you need to define a separate -`AUTO_DEVOPS_DOMAIN` variable for all the above +`KUBE_INGRESS_BASE_DOMAIN` variable for all the above [based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-variables). The following table is an example of how the three different clusters would be configured. -| Cluster name | Cluster environment scope | `AUTO_DEVOPS_DOMAIN` variable value | Variable environment scope | Notes | +| Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes | | ------------ | -------------- | ----------------------------- | ------------- | ------ | | review | `review/*` | `review.example.com` | `review/*` | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. | | staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). | @@ -190,14 +198,11 @@ To add a different cluster for each environment:  1. After the clusters are created, navigate to each one and install Helm Tiller - and Ingress. + and Ingress. Wait for the Ingress IP address to be assigned. 1. Make sure you have [configured your DNS](#auto-devops-base-domain) with the specified Auto DevOps domains. -1. Navigate to your project's **Settings > CI/CD > Environment variables** and add - the `AUTO_DEVOPS_DOMAIN` variables with their respective environment - scope. - -  +1. Navigate to each cluster's page, through **Operations > Kubernetes**, + and add the domain based on its Ingress IP address. Now that all is configured, you can test your setup by creating a merge request and verifying that your app is deployed as a review app in the Kubernetes @@ -205,10 +210,9 @@ cluster with the `review/*` environment scope. Similarly, you can check the other environments. NOTE: **Note:** -Auto DevOps is not supported for a group with multiple clusters, as it -is not possible to set `AUTO_DEVOPS_DOMAIN` per environment on the group -level. This will be resolved in the future with the [following issue]( -https://gitlab.com/gitlab-org/gitlab-ce/issues/52363). +From GitLab 11.8, `KUBE_INGRESS_BASE_DOMAIN` replaces `AUTO_DEVOPS_DOMAIN`. +`AUTO_DEVOPS_DOMAIN` [is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959) +in GitLab 12.0. ## Enabling/Disabling Auto DevOps @@ -681,7 +685,7 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | **Variable** | **Description** | | ------------ | --------------- | -| `AUTO_DEVOPS_DOMAIN` | The [Auto DevOps domain](#auto-devops-domain); by default set automatically by the [Auto DevOps setting](#enabling-auto-devops). | +| `AUTO_DEVOPS_DOMAIN` | The [Auto DevOps domain](#auto-devops-domain). By default, set automatically by the [Auto DevOps setting](#enabling-auto-devops). This variable is deprecated and [is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959) in GitLab 12.0. Use `KUBE_INGRESS_BASE_DOMAIN` instead. | | `AUTO_DEVOPS_CHART` | The Helm Chart used to deploy your apps; defaults to the one [provided by GitLab](https://gitlab.com/charts/auto-deploy-app). | | `AUTO_DEVOPS_CHART_REPOSITORY` | The Helm Chart repository used to search for charts; defaults to `https://charts.gitlab.io`. | | `REPLICAS` | The number of replicas to deploy; defaults to 1. | @@ -711,6 +715,7 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | `DAST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dast` job. If the variable is present, the job will not be created. | | `PERFORMANCE_DISABLED` | From GitLab 11.0, this variable can be used to disable the `performance` job. If the variable is present, the job will not be created. | | `K8S_SECRET_*` | From GitLab 11.7, any variable prefixed with [`K8S_SECRET_`](#application-secret-variables) will be made available by Auto DevOps as environment variables to the deployed application. | +| `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, this variable can be used to set a domain per cluster. See [cluster domains](../../user/project/clusters/index.md#base-domain) for more information. | TIP: **Tip:** Set up the replica variables using a diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 9f9b2da23e1..9fc50741407 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -59,11 +59,16 @@ Add another cluster similar to the first one and make sure to [set an environment scope](#environment-scopes) that will differentiate the new cluster from the rest. -NOTE: **Note:** -Auto DevOps is not supported for a group with multiple clusters, as it -is not possible to set `AUTO_DEVOPS_DOMAIN` per environment on the group -level. This will be resolved in the future with the [following issue]( -https://gitlab.com/gitlab-org/gitlab-ce/issues/52363). +## Base domain + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24580) in GitLab 11.8. + +Domains at the cluster level permit support for multiple domains +per [multiple Kubernetes clusters](#multiple-kubernetes-clusters-premium). When specifying a domain, +this will be automatically set as an environment variable (`KUBE_INGRESS_BASE_DOMAIN`) during +the [Auto DevOps](../../../topics/autodevops/index.md) stages. + +The domain should have a wildcard DNS configured to the Ingress IP address. ## Environment scopes **[PREMIUM]** diff --git a/doc/user/instance_statistics/user_cohorts.md b/doc/user/instance_statistics/user_cohorts.md index f52f24ef5f7..e76363a6d9f 100644 --- a/doc/user/instance_statistics/user_cohorts.md +++ b/doc/user/instance_statistics/user_cohorts.md @@ -25,3 +25,4 @@ How do we measure the activity of users? GitLab considers a user active if: - The user signs in. - The user has Git activity (whether push or pull). +- The user visits pages related to Dashboards, Projects, Issues and Merge Requests ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/54947) in GitLab 11.8). diff --git a/doc/user/markdown.md b/doc/user/markdown.md index f2448f240ca..9a01625f3ff 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -31,8 +31,10 @@ dependency to do so. Please see the [`github-markup` gem readme](https://github. > As of 11.1, GitLab uses the [CommonMark Ruby Library][commonmarker] for Markdown processing of all new issues, merge requests, comments, and other Markdown content in the GitLab system. As of 11.3, wiki pages and Markdown files (`.md`) in the -repositories are also processed with CommonMark. Older content in issues/comments -are still processed using the [Redcarpet Ruby library][redcarpet]. +repositories are also processed with CommonMark. As of 11.8, the [Redcarpet +Ruby library][redcarpet] has been removed and all issues/comments, including +those from pre-11.1, are now processed using [CommonMark Ruby +Library][commonmarker]. > > The documentation website had its [markdown engine migrated from Redcarpet to Kramdown](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108) in October 2018. @@ -41,11 +43,11 @@ in October 2018. ### Transitioning to CommonMark -You may have Markdown documents in your repository that were written using some -of the nuances of RedCarpet's version of Markdown. Since CommonMark uses a -slightly stricter syntax, these documents may now display a little strangely -since we've transitioned to CommonMark. Numbered lists with nested lists in -particular can be displayed incorrectly. +You may have older issues/merge requests or Markdown documents in your +repository that were written using some of the nuances of RedCarpet's version +of Markdown. Since CommonMark uses a slightly stricter syntax, these documents +may now display a little strangely since we've transitioned to CommonMark. +Numbered lists with nested lists in particular can be displayed incorrectly. It is usually quite easy to fix. In the case of a nested list such as this: @@ -65,11 +67,6 @@ simply add a space to each nested item: In the documentation below, we try to highlight some of the differences. -If you have a need to view a document using RedCarpet, you can add the token -`legacy_render=1` to the end of the url, like this: - -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md?legacy_render=1 - If you have a large volume of Markdown files, it can be tedious to determine if they will be displayed correctly or not. You can use the [diff_redcarpet_cmark](https://gitlab.com/digitalmoksha/diff_redcarpet_cmark) @@ -677,7 +674,7 @@ Becomes: + Or pluses If a list item contains multiple paragraphs, -each subsequent paragraph should be indented to the same level as the start of the list item text (_Redcarpet: paragraph should be indented with four spaces._) +each subsequent paragraph should be indented to the same level as the start of the list item text Example: @@ -841,7 +838,7 @@ These details <em>will</em> remain <strong>hidden</strong> until expanded. </details> </p> -**Note:** Markdown inside these tags is supported, as long as you have a blank line after the `</summary>` tag and before the `</details>` tag, as shown in the example. _Redcarpet does not support Markdown inside these tags. You can work around this by using HTML, for example you can use `<pre><code>` tags instead of [code fences](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#code-and-syntax-highlighting)._ +**Note:** Markdown inside these tags is supported, as long as you have a blank line after the `</summary>` tag and before the `</details>` tag, as shown in the example. ```html <details> diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index eb2d731343e..363d3db8db1 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -87,3 +87,11 @@ You can choose between 3 options: - Files and Readme (default) - Readme - Activity + +## Localization + +### First day of the week + +The first day of the week can be customised for calendar views and date pickers. + +You can choose **Sunday** or **Monday** as the first day of the week. If you select **System Default**, the system-wide default setting will be used. diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index bb815695cb1..85a4af24dc5 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -172,6 +172,17 @@ functionalities needed to successfully build and deploy a containerized application. Bear in mind that the same credentials are used for all the applications running on the cluster. +## Base domain + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24580) in GitLab 11.8. + +Domains at the cluster level permit support for multiple domains +per [multiple Kubernetes clusters](#multiple-kubernetes-clusters-premium). When specifying a domain, +this will be automatically set as an environment variable (`KUBE_INGRESS_BASE_DOMAIN`) during +the [Auto DevOps](../../../topics/autodevops/index.md) stages. + +The domain should have a wildcard DNS configured to the Ingress IP address. + ## Access controls When creating a cluster in GitLab, you will be asked if you would like to create an @@ -254,6 +265,12 @@ install it manually. ## Installing applications +NOTE: **Note:** +Before starting the installation of applications, make sure that time is synchronized +between your GitLab server and your Kubernetes cluster. Otherwise, installation could fail +and you may get errors like `Error: remote error: tls: bad certificate` +in the `stdout` of pods created by GitLab in your Kubernetes cluster. + GitLab provides a one-click install for various applications which can be added directly to your configured cluster. Those applications are needed for [Review Apps](../../../ci/review_apps/index.md) and @@ -449,6 +466,7 @@ GitLab CI/CD build environment. | `KUBE_CA_PEM_FILE` | Path to a file containing PEM data. Only present if a custom CA bundle was specified. | | `KUBE_CA_PEM` | (**deprecated**) Raw PEM data. Only if a custom CA bundle was specified. | | `KUBECONFIG` | Path to a file containing `kubeconfig` for this deployment. CA bundle would be embedded if specified. This config also embeds the same token defined in `KUBE_TOKEN` so you likely will only need this variable. This variable name is also automatically picked up by `kubectl` so you won't actually need to reference it explicitly if using `kubectl`. | +| `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, this variable can be used to set a domain per cluster. See [cluster domains](#base-domain) for more information. | NOTE: **NOTE:** Prior to GitLab 11.5, `KUBE_TOKEN` was the Kubernetes token of the main diff --git a/doc/user/project/merge_requests/img/squash_mr_message.png b/doc/user/project/merge_requests/img/squash_mr_message.png Binary files differnew file mode 100644 index 00000000000..8734cab29aa --- /dev/null +++ b/doc/user/project/merge_requests/img/squash_mr_message.png diff --git a/doc/user/project/merge_requests/squash_and_merge.md b/doc/user/project/merge_requests/squash_and_merge.md index 1b57331dbe7..4ff8ec3a7e6 100644 --- a/doc/user/project/merge_requests/squash_and_merge.md +++ b/doc/user/project/merge_requests/squash_and_merge.md @@ -18,12 +18,19 @@ Into a single commit on merge: ![A squashed commit followed by a merge commit][squashed-commit] -The squashed commit's commit message is the merge request title. And note that -the squashed commit is still followed by a merge commit, as the merge -method for this example repository uses a merge commit. Squashing also works -with the fast-forward merge strategy, see -[squashing and fast-forward merge](#squash-and-fast-forward-merge) for more -details. +The squashed commit's commit message will be either: + +- Taken from the first multi-line commit message in the merge. +- The merge request's title if no multi-line commit message is found. + +It can be customized before merging a merge request. + + + +NOTE: **Note:** +The squashed commit in this example is followed by a merge commit, as the merge method for this example repository uses a merge commit. + +Squashing also works with the fast-forward merge strategy, see [squashing and fast-forward merge](#squash-and-fast-forward-merge) for more details. ## Use cases @@ -34,7 +41,7 @@ you'd rather not include them in your target branch. With squash and merge, when the merge request is ready to be merged, all you have to do is enable squashing before you press merge to join -the commits include in the merge request into a single commit. +the commits in the merge request into a single commit. This way, the history of your base branch remains clean with meaningful commit messages and is simpler to [revert] if necessary. @@ -56,7 +63,7 @@ This can then be overridden at the time of accepting the merge request: The squashed commit has the following metadata: -- Message: the title of the merge request. +- Message: the message of the squash commit, or a customized message. - Author: the author of the merge request. - Committer: the user who initiated the squash. diff --git a/doc/user/project/operations/index.md b/doc/user/project/operations/index.md new file mode 100644 index 00000000000..b0f9936be5c --- /dev/null +++ b/doc/user/project/operations/index.md @@ -0,0 +1,11 @@ +# Project operations + +GitLab provides a variety of tools to help operate and maintain +your applications: + +- Collect [Prometheus metrics](../integrations/prometheus_library/index.md). +- Deploy to different [environments](../../../ci/environments.md). +- Connect your project to a [Kubernetes cluster](../clusters/index.md). +- Discover and view errors generated by your applications with [Error Tracking](error_tracking.md). +- Create, toggle, and remove [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html). **[PREMIUM]** +- [Trace](https://docs.gitlab.com/ee/user/project/operations/tracing.html) the performance and health of a deployed application. **[ULTIMATE]** diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index 68dd3330d7a..b2da1c85c62 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -79,11 +79,14 @@ running on your instance).  -NOTE: **Note:** -Note that if you use your root domain for your GitLab Pages website **only**, and if -your domain registrar supports this feature, you can add a DNS apex `CNAME` -record instead of an `A` record. The main advantage of doing so is that when GitLab Pages -IP on GitLab.com changes for whatever reason, you don't need to update your `A` record. +CAUTION: **Caution:** +Note that if you use your root domain for your GitLab Pages website +**only**, and if your domain registrar supports this feature, you can +add a DNS apex `CNAME` record instead of an `A` record. The main +advantage of doing so is that when GitLab Pages IP on GitLab.com +changes for whatever reason, you don't need to update your `A` record. +There may be a few exceptions, but **this method is not recommended** +as it most likely won't work if you set an `MX` record for your root domain. #### DNS CNAME record @@ -114,14 +117,16 @@ co-exist, so you need to place the TXT record in a special subdomain of its own. #### TL;DR -If the domain has multiple uses (e.g., you host email on it as well): +For root domains (`domain.com`), set a DNS `A` record and verify your +domain's ownership with a TXT record: | From | DNS Record | To | | ---- | ---------- | -- | | domain.com | A | 35.185.44.232 | | domain.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | -If the domain is dedicated to GitLab Pages use and no other services run on it: +For subdomains (`subdomain.domain.com`), set a DNS `CNAME` record and +verify your domain's ownership with a TXT record: | From | DNS Record | To | | ---- | ---------- | -- | diff --git a/doc/user/project/pages/getting_started_part_two.md b/doc/user/project/pages/getting_started_part_two.md index b0560c2f44c..8dbbdd051f1 100644 --- a/doc/user/project/pages/getting_started_part_two.md +++ b/doc/user/project/pages/getting_started_part_two.md @@ -31,12 +31,26 @@ The optional settings, custom domain, DNS records, and SSL/TLS certificates, are ## Project Your GitLab Pages project is a regular project created the -same way you do for the other ones. To get started with GitLab Pages, you have two ways: +same way you do for the other ones. To get started with GitLab Pages, you have three ways: +- Use one of the popular templates already in the app, - Fork one of the templates from Page Examples, or - Create a new project from scratch -Let's go over both options. +Let's go over each option. + +### Use one of the popular Pages templates bundled with GitLab + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/47857) +in GitLab 11.8. + +The simplest way to create a GitLab Pages site is to use one of the most +popular templates, which come already bundled and ready to go. To use one +of these templates: + +1. From the top navigation, click the **+** button and select **New project** +1. Select **Create from Template** +1. Choose one of the templates starting with **Pages** ### Fork a project to get started from diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index ce4fccdaff3..11f6165fcb4 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -91,8 +91,8 @@ site under the HTTPS protocol. ## Getting started -To get started with GitLab Pages, you can either [create a project from scratch](getting_started_part_two.md#create-a-project-from-scratch) -or quickly start from copying an existing example project, as follows: +To get started with GitLab Pages, you can either [create a project from scratch](getting_started_part_two.md#create-a-project-from-scratch), +use a [bundled template](getting_started_part_two.md#use-one-of-the-popular-pages-templates-bundled-with-gitlab), or copy any of our existing example projects: 1. Choose an [example project](https://gitlab.com/pages) to [fork](../../../gitlab-basics/fork-project.md#how-to-fork-a-project): by forking a project, you create a copy of the codebase you're forking from to start from a template instead of starting from scratch. diff --git a/doc/user/project/pipelines/job_artifacts.md b/doc/user/project/pipelines/job_artifacts.md index a8b47558c99..0d0575b1ab4 100644 --- a/doc/user/project/pipelines/job_artifacts.md +++ b/doc/user/project/pipelines/job_artifacts.md @@ -38,8 +38,9 @@ turn are defined with the `paths` keyword. All paths to files and directories are relative to the repository that was cloned during the build. These uploaded artifacts will be kept in GitLab for 1 week as defined by the `expire_in` definition. You have the option to keep the artifacts from expiring via the -[web interface](#browsing-job-artifacts). If you don't define an expiry date, -the artifacts will be kept forever. +[web interface](#browsing-job-artifacts). If the expiry time is not defined, +it defaults to the [instance wide +setting](../../admin_area/settings/continuous_integration.md#default-artifacts-expiration-core-only). For more examples on artifacts, follow the [artifacts reference in `.gitlab-ci.yml`](../../../ci/yaml/README.md#artifacts). diff --git a/doc/workflow/repository_mirroring.md b/doc/workflow/repository_mirroring.md index 1213474b7d8..8a2f4e1b40e 100644 --- a/doc/workflow/repository_mirroring.md +++ b/doc/workflow/repository_mirroring.md @@ -88,6 +88,14 @@ The mirrored repository will be listed. For example, `https://*****:*****@github The repository will push soon. To force a push, click the appropriate button. +## Setting up a push mirror to another GitLab instance with 2FA activated + +1. On the destination GitLab instance, create a [personal access token](../user/profile/personal_access_tokens.md) with `API` scope. +1. On the source GitLab instance: + 1. Fill in the **Git repository URL** field using this format: `https://oauth2@<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. + 1. Fill in **Password** field with the GitLab personal access token created on the destination GitLab instance. + 1. Click the **Mirror repository** button. + ## Pulling from a remote repository **[STARTER]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51) in GitLab Enterprise Edition 8.2. |
