diff options
| author | Constance Okoghenun <constanceokoghenun@gmail.com> | 2019-04-12 09:19:00 +0300 |
|---|---|---|
| committer | Constance Okoghenun <constanceokoghenun@gmail.com> | 2019-04-12 09:19:00 +0300 |
| commit | d7afdae4d14f1255a1c4f640c2aeaca0b20b6cbb (patch) | |
| tree | 8f0f9eea0d87596da001f723e84923bd5d72b37d /doc | |
| parent | e11fc352b1f44d5b607d16d35d6b3e4c85d4f5e1 (diff) | |
| parent | 336825c3be6984ca1d10e3e30efd39dcc53e5d05 (diff) | |
Merge branch 'master' of https://gitlab.com/gitlab-org/gitlab-ce into 57402-create-vue-components-to-refactor-issues-list
Diffstat (limited to 'doc')
246 files changed, 3920 insertions, 2874 deletions
diff --git a/doc/README.md b/doc/README.md index ecc214d97c8..14a1eeffda0 100644 --- a/doc/README.md +++ b/doc/README.md @@ -222,7 +222,7 @@ The following documentation relates to the DevOps **Verify** stage: |:---------------------------------------------------|:-----------------------------------------------------------------------------| | [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Integration with GitLab. | | [JUnit test reports](ci/junit_test_reports.md) | Display JUnit test reports on merge requests. | -| [Pipeline Graphs](ci/pipelines.md#pipeline-graphs) | Visualize builds. | +| [Pipeline Graphs](ci/pipelines.md#visualizing-pipelines) | Visualize builds. | | [Review Apps](ci/review_apps/index.md) | Preview changes to your application right from a merge request. | <div align="right"> @@ -288,7 +288,7 @@ The following documentation relates to the DevOps **Configure** stage: | [GitLab ChatOps](ci/chatops/README.md) | Interact with CI/CD jobs through chat services. | | [Installing Applications](user/project/clusters/index.md#installing-applications) | Deploy Helm, Ingress, and Prometheus on Kubernetes. | | [Mattermost slash commands](user/project/integrations/mattermost_slash_commands.md) | Enable and use slash commands from within Mattermost. | -| [Protected variables](ci/variables/README.md#protected-variables) | Restrict variables to protected branches and tags. | +| [Protected variables](ci/variables/README.md#protected-environment-variables) | Restrict variables to protected branches and tags. | | [Serverless](user/project/clusters/serverless/index.md) | Run serverless workloads on Kubernetes. | | [Slack slash commands](user/project/integrations/slack_slash_commands.md) | Enable and use slash commands from within Slack. | diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md new file mode 100644 index 00000000000..65a51fc4aa0 --- /dev/null +++ b/doc/administration/auth/google_secure_ldap.md @@ -0,0 +1,207 @@ +# Google Secure LDAP **[CORE ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/46391) in GitLab 11.9. + +[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure +LDAP service that can be configured with GitLab for authentication and group sync. + +Secure LDAP requires a slightly different configuration than standard LDAP servers. +The steps below cover: + +- Configuring the Secure LDAP Client in the Google Admin console. +- Required GitLab configuration. + +## Configuring Google LDAP client + +1. Navigate to https://admin.google.com and sign in as a GSuite domain administrator. + +1. Go to **Apps > LDAP > Add Client**. + +1. Provide an `LDAP client name` and an optional `Description`. Any descriptive + values are acceptable. For example, the name could be 'GitLab' and the + description could be 'GitLab LDAP Client'. Click the **Continue** button. + +  + +1. Set **Access Permission** according to your needs. You must choose either + 'Entire domain (GitLab)' or 'Selected organizational units' for both 'Verify user + credentials' and 'Read user information'. Select 'Add LDAP Client' + + TIP: **Tip:** If you plan to use GitLab [LDAP Group Sync](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#group-sync) + , turn on 'Read group information'. + +  + +1. Download the generated certificate. This is required for GitLab to + communicate with the Google Secure LDAP service. Save the downloaded certificates + for later use. After downloading, click the **Continue to Client Details** button. + +1. Expand the **Service Status** section and turn the LDAP client 'ON for everyone'. + After selecting 'Save', click on the 'Service Status' bar again to collapse + and return to the rest of the settings. + +1. Expand the **Authentication** section and choose 'Generate New Credentials'. + Copy/note these credentials for later use. After selecting 'Close', click + on the 'Authentication' bar again to collapse and return to the rest of the settings. + +Now the Google Secure LDAP Client configuration is finished. The screenshot below +shows an example of the final settings. Continue on to configure GitLab. + + + +## Configuring GitLab + +Edit GitLab configuration, inserting the access credentials and certificate +obtained earlier. + +The following are the configuration keys that need to be modified using the +values obtained during the LDAP client configuration earlier: + +- `bind_dn`: The access credentials username +- `password`: The access credentials password +- `cert`: The `.crt` file text from the downloaded certificate bundle +- `key`: The `.key` file text from the downloaded certificate bundle + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_enabled'] = true + gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below + main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'Google Secure LDAP' + + host: 'ldap.google.com' + port: 636 + uid: 'uid' + bind_dn: 'DizzyHorse' + password: 'd6V5H8nhMUW9AuDP25abXeLd' + encryption: 'simple_tls' + verify_certificates: true + + tls_options: + cert: | + -----BEGIN CERTIFICATE----- + MIIDbDCCAlSgAwIBAgIGAWlzxiIfMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ + bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE + CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAzMTIyMTE5 + MThaFw0yMjAzMTEyMTE5MThaMHcxFDASBgNVBAoTC0dvb2dsZSBJbmMuMRYwFAYDVQQHEw1Nb3Vu + dGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UECxMGR1N1aXRlMQswCQYDVQQG + EwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB + ALOTy4aC38dyjESk6N8fRsKk8DN23ZX/GaNFL5OUmmA1KWzrvVC881OzNdtGm3vNOIxr9clteEG/ + tQwsmsJvQT5U+GkBt+tGKF/zm7zueHUYqTP7Pg5pxAnAei90qkIRFi17ulObyRHPYv1BbCt8pxNB + 4fG/gAXkFbCNxwh1eiQXXRTfruasCZ4/mHfX7MVm8JmWU9uAVIOLW+DSWOFhrDQduJdGBXJOyC2r + Gqoeg9+tkBmNH/jjxpnEkFW8q7io9DdOUqqNgoidA1h9vpKTs3084sy2DOgUvKN9uXWx14uxIyYU + Y1DnDy0wczcsuRt7l+EgtCEgpsLiLJQbKW+JS1UCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAf60J + yazhbHkDKIH2gFxfm7QLhhnqsmafvl4WP7JqZt0u0KdnvbDPfokdkM87yfbKJU1MTI86M36wEC+1 + P6bzklKz7kXbzAD4GggksAzxsEE64OWHC+Y64Tkxq2NiZTw/76POkcg9StiIXjG0ZcebHub9+Ux/ + rTncip92nDuvgEM7lbPFKRIS/YMhLCk09B/U0F6XLsf1yYjyf5miUTDikPkov23b/YGfpc8kh6hq + 1kqdi6a1cYPP34eAhtRhMqcZU9qezpJF6s9EeN/3YFfKzLODFSsVToBRAdZgGHzj//SAtLyQTD4n + KCSvK1UmaMxNaZyTHg8JnMf0ZuRpv26iSg== + -----END CERTIFICATE----- + + key: | + -----BEGIN PRIVATE KEY----- + MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCzk8uGgt/HcoxEpOjfH0bCpPAz + dt2V/xmjRS+TlJpgNSls671QvPNTszXbRpt7zTiMa/XJbXhBv7UMLJrCb0E+VPhpAbfrRihf85u8 + 7nh1GKkz+z4OacQJwHovdKpCERYte7pTm8kRz2L9QWwrfKcTQeHxv4AF5BWwjccIdXokF10U367m + rAmeP5h31+zFZvCZllPbgFSDi1vg0ljhYaw0HbiXRgVyTsgtqxqqHoPfrZAZjR/448aZxJBVvKu4 + qPQ3TlKqjYKInQNYfb6Sk7N9POLMtgzoFLyjfbl1sdeLsSMmFGNQ5w8tMHM3LLkbe5fhILQhIKbC + 4iyUGylviUtVAgMBAAECggEAIPb0CQy0RJoX+q/lGbRVmnyJpYDf+115WNnl+mrwjdGkeZyqw4v0 + BPzkWYzUFP1esJRO6buBNFybQRFdFW0z5lvVv/zzRKq71aVUBPInxaMRyHuJ8D5lIL8nDtgVOwyE + 7DOGyDtURUMzMjdUwoTe7K+O6QBU4X/1pVPZYgmissYSMmt68LiP8k0p601F4+r5xOi/QEy44aVp + aOJZBUOisKB8BmUXZqmQ4Cy05vU9Xi1rLyzkn9s7fxnZ+JO6Sd1r0Thm1mE0yuPgxkDBh/b4f3/2 + GsQNKKKCiij/6TfkjnBi8ZvWR44LnKpu760g/K7psVNrKwqJG6C/8RAcgISWQQKBgQDop7BaKGhK + 1QMJJ/vnlyYFTucfGLn6bM//pzTys5Gop0tpcfX/Hf6a6Dd+zBhmC3tBmhr80XOX/PiyAIbc0lOI + 31rafZuD/oVx5mlIySWX35EqS14LXmdVs/5vOhsInNgNiE+EPFf1L9YZgG/zA7OUBmqtTeYIPDVC + 7ViJcydItQKBgQDFmK0H0IA6W4opGQo+zQKhefooqZ+RDk9IIZMPOAtnvOM7y3rSVrfsSjzYVuMS + w/RP/vs7rwhaZejnCZ8/7uIqwg4sdUBRzZYR3PRNFeheW+BPZvb+2keRCGzOs7xkbF1mu54qtYTa + HZGZj1OsD83AoMwVLcdLDgO1kw32dkS8IQKBgFRdgoifAHqqVah7VFB9se7Y1tyi5cXWsXI+Wufr + j9U9nQ4GojK52LqpnH4hWnOelDqMvF6TQTyLIk/B+yWWK26Ft/dk9wDdSdystd8L+dLh4k0Y+Whb + +lLMq2YABw+PeJUnqdYE38xsZVHoDjBsVjFGRmbDybeQxauYT7PACy3FAoGBAK2+k9bdNQMbXp7I + j8OszHVkJdz/WXlY1cmdDAxDwXOUGVKIlxTAf7TbiijILZ5gg0Cb+hj+zR9/oI0WXtr+mAv02jWp + W8cSOLS4TnBBpTLjIpdu+BwbnvYeLF6MmEjNKEufCXKQbaLEgTQ/XNlchBSuzwSIXkbWqdhM1+gx + EjtBAoGARAdMIiDMPWIIZg3nNnFebbmtBP0qiBsYohQZ+6i/8s/vautEHBEN6Q0brIU/goo+nTHc + t9VaOkzjCmAJSLPUanuBC8pdYgLu5J20NXUZLD9AE/2bBT3OpezKcdYeI2jqoc1qlWHlNtVtdqQ2 + AcZSFJQjdg5BTyvdEDhaYUKGdRw= + -----END PRIVATE KEY----- + EOS + ``` + +1. Save the file and [reconfigure] GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + ldap: + enabled: true + servers: + main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'Google Secure LDAP' + + host: 'ldap.google.com' + port: 636 + uid: 'uid' + bind_dn: 'DizzyHorse' + password: 'd6V5H8nhMUW9AuDP25abXeLd' + encryption: 'simple_tls' + verify_certificates: true + + tls_options: + cert: | + -----BEGIN CERTIFICATE----- + MIIDbDCCAlSgAwIBAgIGAWlzxiIfMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ + bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE + CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAzMTIyMTE5 + MThaFw0yMjAzMTEyMTE5MThaMHcxFDASBgNVBAoTC0dvb2dsZSBJbmMuMRYwFAYDVQQHEw1Nb3Vu + dGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UECxMGR1N1aXRlMQswCQYDVQQG + EwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB + ALOTy4aC38dyjESk6N8fRsKk8DN23ZX/GaNFL5OUmmA1KWzrvVC881OzNdtGm3vNOIxr9clteEG/ + tQwsmsJvQT5U+GkBt+tGKF/zm7zueHUYqTP7Pg5pxAnAei90qkIRFi17ulObyRHPYv1BbCt8pxNB + 4fG/gAXkFbCNxwh1eiQXXRTfruasCZ4/mHfX7MVm8JmWU9uAVIOLW+DSWOFhrDQduJdGBXJOyC2r + Gqoeg9+tkBmNH/jjxpnEkFW8q7io9DdOUqqNgoidA1h9vpKTs3084sy2DOgUvKN9uXWx14uxIyYU + Y1DnDy0wczcsuRt7l+EgtCEgpsLiLJQbKW+JS1UCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAf60J + yazhbHkDKIH2gFxfm7QLhhnqsmafvl4WP7JqZt0u0KdnvbDPfokdkM87yfbKJU1MTI86M36wEC+1 + P6bzklKz7kXbzAD4GggksAzxsEE64OWHC+Y64Tkxq2NiZTw/76POkcg9StiIXjG0ZcebHub9+Ux/ + rTncip92nDuvgEM7lbPFKRIS/YMhLCk09B/U0F6XLsf1yYjyf5miUTDikPkov23b/YGfpc8kh6hq + 1kqdi6a1cYPP34eAhtRhMqcZU9qezpJF6s9EeN/3YFfKzLODFSsVToBRAdZgGHzj//SAtLyQTD4n + KCSvK1UmaMxNaZyTHg8JnMf0ZuRpv26iSg== + -----END CERTIFICATE----- + + key: | + -----BEGIN PRIVATE KEY----- + MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCzk8uGgt/HcoxEpOjfH0bCpPAz + dt2V/xmjRS+TlJpgNSls671QvPNTszXbRpt7zTiMa/XJbXhBv7UMLJrCb0E+VPhpAbfrRihf85u8 + 7nh1GKkz+z4OacQJwHovdKpCERYte7pTm8kRz2L9QWwrfKcTQeHxv4AF5BWwjccIdXokF10U367m + rAmeP5h31+zFZvCZllPbgFSDi1vg0ljhYaw0HbiXRgVyTsgtqxqqHoPfrZAZjR/448aZxJBVvKu4 + qPQ3TlKqjYKInQNYfb6Sk7N9POLMtgzoFLyjfbl1sdeLsSMmFGNQ5w8tMHM3LLkbe5fhILQhIKbC + 4iyUGylviUtVAgMBAAECggEAIPb0CQy0RJoX+q/lGbRVmnyJpYDf+115WNnl+mrwjdGkeZyqw4v0 + BPzkWYzUFP1esJRO6buBNFybQRFdFW0z5lvVv/zzRKq71aVUBPInxaMRyHuJ8D5lIL8nDtgVOwyE + 7DOGyDtURUMzMjdUwoTe7K+O6QBU4X/1pVPZYgmissYSMmt68LiP8k0p601F4+r5xOi/QEy44aVp + aOJZBUOisKB8BmUXZqmQ4Cy05vU9Xi1rLyzkn9s7fxnZ+JO6Sd1r0Thm1mE0yuPgxkDBh/b4f3/2 + GsQNKKKCiij/6TfkjnBi8ZvWR44LnKpu760g/K7psVNrKwqJG6C/8RAcgISWQQKBgQDop7BaKGhK + 1QMJJ/vnlyYFTucfGLn6bM//pzTys5Gop0tpcfX/Hf6a6Dd+zBhmC3tBmhr80XOX/PiyAIbc0lOI + 31rafZuD/oVx5mlIySWX35EqS14LXmdVs/5vOhsInNgNiE+EPFf1L9YZgG/zA7OUBmqtTeYIPDVC + 7ViJcydItQKBgQDFmK0H0IA6W4opGQo+zQKhefooqZ+RDk9IIZMPOAtnvOM7y3rSVrfsSjzYVuMS + w/RP/vs7rwhaZejnCZ8/7uIqwg4sdUBRzZYR3PRNFeheW+BPZvb+2keRCGzOs7xkbF1mu54qtYTa + HZGZj1OsD83AoMwVLcdLDgO1kw32dkS8IQKBgFRdgoifAHqqVah7VFB9se7Y1tyi5cXWsXI+Wufr + j9U9nQ4GojK52LqpnH4hWnOelDqMvF6TQTyLIk/B+yWWK26Ft/dk9wDdSdystd8L+dLh4k0Y+Whb + +lLMq2YABw+PeJUnqdYE38xsZVHoDjBsVjFGRmbDybeQxauYT7PACy3FAoGBAK2+k9bdNQMbXp7I + j8OszHVkJdz/WXlY1cmdDAxDwXOUGVKIlxTAf7TbiijILZ5gg0Cb+hj+zR9/oI0WXtr+mAv02jWp + W8cSOLS4TnBBpTLjIpdu+BwbnvYeLF6MmEjNKEufCXKQbaLEgTQ/XNlchBSuzwSIXkbWqdhM1+gx + EjtBAoGARAdMIiDMPWIIZg3nNnFebbmtBP0qiBsYohQZ+6i/8s/vautEHBEN6Q0brIU/goo+nTHc + t9VaOkzjCmAJSLPUanuBC8pdYgLu5J20NXUZLD9AE/2bBT3OpezKcdYeI2jqoc1qlWHlNtVtdqQ2 + AcZSFJQjdg5BTyvdEDhaYUKGdRw= + -----END PRIVATE KEY----- + ``` + +1. Save the file and [restart] GitLab for the changes to take effect. + + +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: ../restart_gitlab.md#installations-from-source diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_1.png b/doc/administration/auth/img/google_secure_ldap_add_step_1.png Binary files differnew file mode 100644 index 00000000000..fd254443d75 --- /dev/null +++ b/doc/administration/auth/img/google_secure_ldap_add_step_1.png diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_2.png b/doc/administration/auth/img/google_secure_ldap_add_step_2.png Binary files differnew file mode 100644 index 00000000000..611a21ae03c --- /dev/null +++ b/doc/administration/auth/img/google_secure_ldap_add_step_2.png diff --git a/doc/administration/auth/img/google_secure_ldap_client_settings.png b/doc/administration/auth/img/google_secure_ldap_client_settings.png Binary files differnew file mode 100644 index 00000000000..3c0b3f3d4bd --- /dev/null +++ b/doc/administration/auth/img/google_secure_ldap_client_settings.png diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 440c2b1285a..2d057dc7509 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -48,6 +48,14 @@ LDAP-enabled users can always authenticate with Git using their GitLab username or email and LDAP password, even if password authentication for Git is disabled in the application settings. +## Google Secure LDAP **[CORE ONLY]** + +> Introduced in GitLab 11.9. + +[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure +LDAP service that can be configured with GitLab for authentication and group sync. +See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. + ## Configuration NOTE: **Note**: diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 638405126a5..aa4e1b0d2e0 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -92,18 +92,23 @@ Now that the Okta app is configured, it's time to enable it in GitLab. 1. Add the provider configuration. >**Notes:** + > >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab installation to generate the correct value). + > >- To get the `idp_cert_fingerprint` fingerprint, first download the certificate from the Okta app you registered and then run: `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` with the location of your certificate. + > >- Change the value of `idp_sso_target_url`, with the value of the **Identity Provider Single Sign-On URL** from the step when you configured the Okta app. - >- Change the value of `issuer` to a unique name, which will identify the application + > + >- Change the value of `issuer` to the value of the **Audience Restriction** from your Okta app configuration. This will identify GitLab to the IdP. + > >- Leave `name_identifier_format` as-is. **For Omnibus GitLab installations** diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index b21bfafc096..05beb724d4d 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -362,6 +362,10 @@ configuring a different storage driver. By default the GitLab Container Registry is configured to use the filesystem driver, which makes use of [storage path](#container-registry-storage-path) configuration. +NOTE: **Note:** Enabling a storage driver other than `filesystem` would mean +that your Docker client needs to be able to access the storage backend directly. +In that case, you must use an address that resolves and is accessible outside GitLab server. + The different supported drivers are: | Driver | Description | @@ -369,20 +373,16 @@ The different supported drivers are: | filesystem | Uses a path on the local filesystem | | azure | Microsoft Azure Blob Storage | | gcs | Google Cloud Storage | -| s3 | Amazon Simple Storage Service | +| s3 | Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct [S3 Permission Scopes](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). | | swift | OpenStack Swift Object Storage | | oss | Aliyun OSS | Read more about the individual driver's config options in the [Docker Registry docs][storage-config]. -> **Warning** GitLab will not backup Docker images that are not stored on the +CAUTION: **Warning:** GitLab will not backup Docker images that are not stored on the filesystem. Remember to enable backups with your object storage provider if desired. -> -> **Important** Enabling storage driver other than `filesystem` would mean -that your Docker client needs to be able to access the storage backend directly. -So you must use an address that resolves and is accessible outside GitLab server. --- diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 60ad4bf4e8f..28afaf84f5a 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -51,7 +51,7 @@ Hooks can be also placed in `hooks/<hook_name>.d` (global) or execution of the hooks. NOTE: **Note:** `<hook_name>.d` would need to be either `pre-receive.d`, -`post-receive.d`, or `update.d` to work properly. Any other names will be ignored. +`post-receive.d`, or `update.d` to work properly. Any other names will be ignored. To look in a different directory for the global custom hooks (those in `hooks/<hook_name.d>`), set `custom_hooks_dir` in gitlab-shell config. For @@ -76,9 +76,21 @@ first script exiting with a non-zero value. > [Introduced][5073] in GitLab 8.10. -If the commit is declined or an error occurs during the Git hook check, -the STDERR or STDOUT message of the hook will be present in GitLab's UI. -STDERR takes precedence over STDOUT. +To have custom error messages appear in GitLab's UI when the commit is +declined or an error occurs during the Git hook, your script should: + +- Send the custom error messages to either the script's `stdout` or `stderr`. +- Prefix each message with `GL-HOOK-ERR:` with no characters appearing before the prefix. + +### Example custom error message + +This hook script written in bash will generate the following message in GitLab's UI: + +```bash +#!/bin/sh +echo "GL-HOOK-ERR: My custom error message."; +exit 1 +```  diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 345e8646f9b..1780e1babe9 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -31,7 +31,7 @@ From the client side, `git` `v2.18.0` or newer must be installed. From the server side, if we want to configure SSH we need to set the `sshd` server to accept the `GIT_PROTOCOL` environment. -In installations using [GitLab Helm Charts](../install/kubernetes/gitlab_chart.md) +In installations using [GitLab Helm Charts](https://docs.gitlab.com/charts/) and [All-in-one docker image](https://docs.gitlab.com/omnibus/docker/), the SSH service is already configured to accept the `GIT_PROTOCOL` environment and users need not do anything more. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 2d4b5c65c46..f1cedb85455 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -241,12 +241,24 @@ repository from your GitLab server over HTTP. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22602) in GitLab 11.8. -Gitaly supports TLS credentials for GRPC authentication. To be able to communicate +Gitaly supports TLS encryption. To be able to communicate with a Gitaly instance that listens for secure connections you will need to use `tls://` url scheme in the `gitaly_address` of the corresponding storage entry in the gitlab configuration. The admin needs to bring their own certificate as we do not provide that automatically. -The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) +The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). + +Note that it is possible to configure Gitaly servers with both an +unencrypted listening address `listen_addr` and an encrypted listening +address `tls_listen_addr` at the same time. This allows you to do a +gradual transition from unencrypted to encrypted traffic, if necessary. + +To observe what type of connections are actually being used in a +production environment you can use the following Prometheus query: + +``` +sum(rate(gitaly_connections_total[5m])) by (type) +``` ### Example TLS configuration @@ -303,6 +315,66 @@ certificate_path = '/path/to/cert.pem' key_path = '/path/to/key.pem' ``` +## Gitaly-ruby + +Gitaly was developed to replace Ruby application code in gitlab-ce/ee. +In order to save time and/or avoid the risk of rewriting existing +application logic, in some cases we chose to copy some application code +from gitlab-ce into Gitaly almost as-is. To be able to run that code, we +made gitaly-ruby, which is a sidecar process for the main Gitaly Go +process. Some examples of things that are implemented in gitaly-ruby are +RPC's that deal with wiki's, and RPC's that create commits on behalf of +a user, such as merge commits. + +### Number of gitaly-ruby workers + +Gitaly-ruby has much less capacity than Gitaly itself. If your Gitaly +server has to handle a lot of request, the default setting of having +just 1 active gitaly-ruby sidecar might not be enough. If you see +ResourceExhausted errors from Gitaly it's very likely that you have not +enough gitaly-ruby capacity. + +You can increase the number of gitaly-ruby processes on your Gitaly +server with the following settings. + +Omnibus: + +```ruby +# /etc/gitlab/gitlab.rb +# Default is 2 workers. The minimum is 2; 1 worker is always reserved as +# a passive stand-by. +gitaly['ruby_num_workers'] = 4 +``` + +Source: + +```toml +# /home/git/gitaly/config.toml +[gitaly-ruby] +num_workers = 4 +``` + +### Observing gitaly-ruby traffic + +Gitaly-ruby is a somewhat hidden, internal implementation detail of +Gitaly. There is not that much visibility into what goes on inside +gitaly-ruby processes. + +If you have Prometheus set up to scrape your Gitaly process, you can see +request rates and error codes for individual RPC's in gitaly-ruby by +querying `grpc_client_handled_total`. Strictly speaking this metric does +not differentiate between gitaly-ruby and other RPC's, but in practice +(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal +calls from the main Gitaly process to one of its gitaly-ruby sidecars. + +Assuming your `grpc_client_handled_total` counter only observes Gitaly, +the following query shows you RPC's are (most likely) internally +implemented as calls to gitaly-ruby. + +``` +sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 +``` + ## Disabling or enabling the Gitaly service in a cluster environment If you are running Gitaly [as a remote diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index b1aaa3bca13..f406163aea0 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -71,9 +71,8 @@ bug](https://bugzilla.redhat.com/show_bug.cgi?id=1552203) that may be fixed in [more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). -Users encountering a similar issue may be advised to disable the NFS server -delegation feature, which is an optimization to reduce the number of network -round-trips needed to read or write files. To disable NFS server delegations +GitLab recommends all NFS users disable the NFS server +delegation feature. To disable NFS server delegations on an Linux NFS server, do the following: 1. On the NFS server, run: diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 058346df56d..1b01419e062 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -30,7 +30,7 @@ the `pushes_since_gc` value is 200 a `git gc` will be run. `git add`. - `git repack` ([man page][man-repack]) re-organize existing packs into a single, more efficient pack. -You can find this option under your **[Project] > Edit Project**. +You can find this option under your project's **Settings > General > Advanced**. --- diff --git a/doc/administration/img/custom_hooks_error_msg.png b/doc/administration/img/custom_hooks_error_msg.png Binary files differindex 845f0de19ce..4f25c471908 100644 --- a/doc/administration/img/custom_hooks_error_msg.png +++ b/doc/administration/img/custom_hooks_error_msg.png diff --git a/doc/administration/img/housekeeping_settings.png b/doc/administration/img/housekeeping_settings.png Binary files differindex acc4506993a..356de51f0cc 100644 --- a/doc/administration/img/housekeeping_settings.png +++ b/doc/administration/img/housekeeping_settings.png diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 25d85d1687b..2596e3fe68b 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -11,7 +11,7 @@ One of the things it uses these credentials for is providing access to ## How it works A detailed overview of the architecture of web terminals and how they work -can be found in [this document](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/doc/terminal.md). +can be found in [this document](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/doc/channel.md). In brief: - GitLab relies on the user to provide their own Kubernetes credentials, and to diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md index 63945506f3c..aa9d87562a3 100644 --- a/doc/administration/job_traces.md +++ b/doc/administration/job_traces.md @@ -10,13 +10,13 @@ In the following table you can see the phases a trace goes through. | Phase | State | Condition | Data flow | Stored path | | ----- | ----- | --------- | --------- | ----------- | -| 1: patching | Live trace | When a job is running | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| -| 2: overwriting | Live trace | When a job is finished | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| -| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`| +| 1: patching | Live trace | When a job is running | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| +| 2: overwriting | Live trace | When a job is finished | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| +| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`| | 4: uploading | Archived trace | After a trace is archived | Sidekiq moves archived trace to [object storage](#uploading-traces-to-object-storage) (if configured) |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`| The `ROOT_PATH` varies per your environment. For Omnibus GitLab it -would be `/var/opt/gitlab/gitlab-ci`, whereas for installations from source +would be `/var/opt/gitlab`, whereas for installations from source it would be `/home/git/gitlab`. ## Changing the job traces local location diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 36dee75bd44..3d40cda491a 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -23,16 +23,19 @@ requests from the API are logged to a separate file in `api_json.log`. Each line contains a JSON line that can be ingested by Elasticsearch, Splunk, etc. For example: ```json -{"method":"GET","path":"/gitlab/gitlab-ce/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"duration":229.03,"view":174.07,"db":13.24,"time":"2017-08-08T20:15:54.821Z","params":[{"key":"param_key","value":"param_value"}],"remote_ip":"18.245.0.1","user_id":1,"username":"admin","gitaly_calls":76,"queue_duration": 112.47} +{"method":"GET","path":"/gitlab/gitlab-ce/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"duration":229.03,"view":174.07,"db":13.24,"time":"2017-08-08T20:15:54.821Z","params":[{"key":"param_key","value":"param_value"}],"remote_ip":"18.245.0.1","user_id":1,"username":"admin","gitaly_calls":76,"gitaly_duration":7.41,"queue_duration": 112.47} ``` -In this example, you can see this was a GET request for a specific issue. Notice each line also contains performance data: +In this example, you can see this was a GET request for a specific +issue. Notice each line also contains performance data. All times are in +milliseconds: -1. `duration`: total time in milliseconds taken to retrieve the request -1. `queue_duration`: total time in milliseconds that the request was queued inside GitLab Workhorse +1. `duration`: total time taken to retrieve the request +1. `queue_duration`: total time that the request was queued inside GitLab Workhorse 1. `view`: total time taken inside the Rails views 1. `db`: total time to retrieve data from the database 1. `gitaly_calls`: total number of calls made to Gitaly +1. `gitaly_duration`: total time taken by Gitaly calls User clone/fetch activity using http transport appears in this log as `action: git_upload_pack`. @@ -85,7 +88,7 @@ Introduced in GitLab 10.0, this file lives in It helps you see requests made directly to the API. For example: ```json -{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30} +{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} ``` This entry above shows an access to an internal endpoint to check whether an diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index c34a9519ace..5e9ba4f640f 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -145,3 +145,47 @@ The connection settings match those provided by [Fog](https://github.com/fog), a ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + +### Alternative in-database storage + +Enabling external diffs may reduce the performance of merge requests, as they +must be retrieved in a separate operation to other data. A compromise may be +reached by only storing outdated diffs externally, while keeping current diffs +in the database. + +To enable this feature, perform the following steps: + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['external_diffs_when'] = 'outdated' + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + external_diffs: + enabled: true + when: outdated + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + +With this feature enabled, diffs will initially stored in the database, rather +than externally. They will be moved to external storage once any of these +conditions become true: + +- A newer version of the merge request diff exists +- The merge request was merged more than seven days ago +- The merge request was closed more than seven day ago + +These rules strike a balance between space and performance by only storing +frequently-accessed diffs in the database. Diffs that are less likely to be +accessed are moved to external storage instead. diff --git a/doc/administration/monitoring/performance/img/request_profiling_token.png b/doc/administration/monitoring/performance/img/request_profiling_token.png Binary files differindex 9f3dd7f08ca..ee819fcb437 100644 --- a/doc/administration/monitoring/performance/img/request_profiling_token.png +++ b/doc/administration/monitoring/performance/img/request_profiling_token.png diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index c293df3fc57..f8d58c6ce28 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -58,6 +58,9 @@ sudo service sshd reload Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo. +> **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in +GitLab 11.11 and later. + > **Warning:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 288ce376687..17d72b96a51 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -125,7 +125,7 @@ The Pages daemon doesn't listen to the outside world. pages_external_url 'http://example.io' ``` -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. Watch the [video tutorial][video-admin] for this configuration. @@ -157,7 +157,22 @@ outside world. where `pages-nginx.crt` and `pages-nginx.key` are the SSL cert and key, respectively. -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. + +### Additional configuration for Docker container + +The GitLab Pages daemon will not have permissions to bind mounts when it runs +in a Docker container. To overcome this issue you'll need to change the chroot +behavior: + +1. Edit `/etc/gitlab/gitlab.rb`. +1. Set the `inplace_chroot` to `true` for GitLab Pages: + + ```shell + gitlab_pages['inplace_chroot'] = true + ``` + +1. [Reconfigure GitLab][reconfigure]. ## Advanced configuration @@ -195,7 +210,7 @@ world. Custom domains are supported, but no TLS. `192.0.2.2` and `2001::2` are the secondary IPs the GitLab Pages daemon listens on. If you don't have IPv6, you can omit the IPv6 address. -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ### Custom domains with TLS support @@ -229,7 +244,7 @@ world. Custom domains and TLS are supported. `192.0.2.2` and `2001::2` are the secondary IPs where the GitLab Pages daemon listens on. If you don't have IPv6, you can omit the IPv6 address. -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ### Custom domain verification @@ -287,7 +302,7 @@ Follow the steps below to configure verbose logging of GitLab Pages daemon. gitlab_pages['log_verbose'] = true ``` -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ## Change storage path @@ -302,7 +317,7 @@ are stored. gitlab_rails['pages_path'] = "/mnt/storage/pages" ``` -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ## Configure listener for reverse proxy requests @@ -325,7 +340,7 @@ Omnibus GitLab 11.1. gitlab_pages['listen_proxy'] = "localhost:10080" ``` -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ## Set maximum pages size diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md new file mode 100644 index 00000000000..54a423b9571 --- /dev/null +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -0,0 +1,62 @@ +# Uploads Sanitize tasks + +## Requirements + +You need `exiftool` installed on your system. If you installed GitLab: + +- Using the Omnibus package, you're all set. +- From source, make sure `exiftool` is installed: + + ```sh + # Debian/Ubuntu + sudo apt-get install libimage-exiftool-perl + + # RHEL/CentOS + sudo yum install perl-Image-ExifTool + ``` + +## Remove EXIF data from existing uploads + +Since 11.9 EXIF data are automatically stripped from JPG or TIFF image uploads. +Because EXIF data may contain sensitive information (e.g. GPS location), you +can remove EXIF data also from existing images which were uploaded before +with the following command: + +```bash +sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif +``` + +This command by default runs in dry mode and it doesn't remove EXIF data. It can be used for +checking if (and how many) images should be sanitized. + +The rake task accepts following parameters. + +Parameter | Type | Description +--------- | ---- | ----------- +`start_id` | integer | Only uploads with equal or greater ID will be processed +`stop_id` | integer | Only uploads with equal or smaller ID will be processed +`dry_run` | boolean | Do not remove EXIF data, only check if EXIF data are present or not, default: true +`sleep_time` | float | Pause for number of seconds after processing each image, default: 0.3 seconds + +If you have too many uploads, you can speed up sanitization by setting +`sleep_time` to a lower value or by running multiple rake tasks in parallel, +each with a separate range of upload IDs (by setting `start_id` and `stop_id`). + +To run the command without dry mode and remove EXIF data from all uploads, you can use: + +```bash +sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[,,false,] 2>&1 | tee exif.log +``` + +To run the command without dry mode on uploads with ID between 100 and 5000 and pause for 0.1 second, you can use: + +```bash +sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[100,5000,false,0.1] 2>&1 | tee exif.log +``` + +Because the output of commands will be probably long, the output is written also into exif.log file. + +If sanitization fails for an upload, an error message should be in the output of the rake task (typical reasons may +be that the file is missing in the storage or it's not a valid image). Please +[report](https://gitlab.com/gitlab-org/gitlab-ce/issues/new) any issues at `gitlab.com` and use +prefix 'EXIF' in issue title with the error output and (if possible) the image. diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 7f25423171f..1689b0a57d6 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -62,6 +62,8 @@ files and add the full paths of the alternative repository storage paths. In the example below, we add two more mountpoints that are named `nfs` and `cephfs` respectively. +NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](./high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. + **For installations from source** 1. Edit `gitlab.yml` and add the storage paths: diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 25c3d564560..4e1e363888d 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -142,7 +142,7 @@ projects: 2. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. To schedule a complete rollback, see the -[rake task documentation for storage rollback][rake/rollback-to-legacy] for instructions. +[rake task documentation for storage rollback](raketasks/storage.md#rollback-from-hashed-storage-to-legacy-storage) for instructions. The rollback task also supports specifying a range of Project IDs. Here is an example of limiting the rollout to Project IDs 50 to 100, in an Omnibus Gitlab installation: @@ -205,6 +205,5 @@ They are also S3 compatible since **10.0** (GitLab Premium), and available in Gi [ce-2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 [ce-28283]: https://gitlab.com/gitlab-org/gitlab-ce/issues/28283 [rake/migrate-to-hashed]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage -[rake/rollback-to-legacy]: raketasks/storage.md#rollback [storage-paths]: repository_storage_types.md [gitaly]: gitaly/index.md diff --git a/doc/api/build_triggers.md b/doc/api/build_triggers.md index 20d924ab35e..bad7a655d08 100644 --- a/doc/api/build_triggers.md +++ b/doc/api/build_triggers.md @@ -1 +1,5 @@ -This document was moved to [Pipeline Triggers](pipeline_triggers.md). +--- +redirect_to: 'pipeline_triggers.md' +--- + +This document was moved to [another location](pipeline_triggers.md). diff --git a/doc/api/builds.md b/doc/api/builds.md index a6edda68bc4..0154d35cab6 100644 --- a/doc/api/builds.md +++ b/doc/api/builds.md @@ -1 +1,5 @@ +--- +redirect_to: 'jobs.md' +--- + This document was moved to [another location](jobs.md). diff --git a/doc/api/commits.md b/doc/api/commits.md index 09546fcac3f..92f53c7b5e6 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -130,6 +130,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Cont ``` Example response: + ```json { "id": "ed899a2f4b50b4370feeea94676502b42383c746", @@ -155,6 +156,32 @@ Example response: } ``` +GitLab supports [form encoding](README.md#encoding-api-parameters-of-array-and-hash-types). The following is an example using Commit API with form encoding: + +```bash +curl --request POST \ + --form "branch=master" \ + --form "commit_message=some commit message" \ + --form "start_branch=master" \ + --form "actions[][action]=create" \ + --form "actions[][file_path]=foo/bar" \ + --form "actions[][content]=</path/to/local.file" \ + --form "actions[][action]=delete" \ + --form "actions[][file_path]=foo/bar2" \ + --form "actions[][action]=move" \ + --form "actions[][file_path]=foo/bar3" \ + --form "actions[][previous_path]=foo/bar4" \ + --form "actions[][content]=</path/to/local1.file" \ + --form "actions[][action]=update" \ + --form "actions[][file_path]=foo/bar5" \ + --form "actions[][content]=</path/to/local2.file" \ + --form "actions[][action]=chmod" \ + --form "actions[][file_path]=foo/bar5" \ + --form "actions[][execute_filemode]=true" \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/1/repository/commits" +``` + ## Get a single commit Get a specific commit identified by the commit hash or name of a branch or tag. @@ -196,9 +223,9 @@ Example response: "last_pipeline" : { "id": 8, "ref": "master", - "sha": "2dc6aa325a317eda67812f05600bdf0fcdc70ab0" + "sha": "2dc6aa325a317eda67812f05600bdf0fcdc70ab0", "status": "created" - } + }, "stats": { "additions": 15, "deletions": 10, diff --git a/doc/api/environments.md b/doc/api/environments.md index 4a38dd73747..ebcdc546d08 100644 --- a/doc/api/environments.md +++ b/doc/api/environments.md @@ -29,6 +29,111 @@ Example response: ] ``` +## Get a specific environment + +``` +GET /projects/:id/environments/:environment_id +``` + +| Attribute | Type | Required | Description | +|-----------|---------|----------|---------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `environment_id` | integer | yes | The ID of the environment | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/environments/1" +``` + +Example of response + +```json +{ + "id": 1, + "name": "review/fix-foo", + "slug": "review-fix-foo-dfjre3", + "external_url": "https://review-fix-foo-dfjre3.example.gitlab.com" + "last_deployment": { + "id": 100, + "iid": 34, + "ref": "fdroid", + "sha": "416d8ea11849050d3d1f5104cf8cf51053e790ab", + "created_at": "2019-03-25T18:55:13.252Z", + "user": { + "id": 1, + "name": "Administrator", + "state": "active", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + } + "deployable": { + "id": 710, + "status": "success", + "stage": "deploy", + "name": "staging", + "ref": "fdroid", + "tag": false, + "coverage": null, + "created_at": "2019-03-25T18:55:13.215Z", + "started_at": "2019-03-25T12:54:50.082Z", + "finished_at": "2019-03-25T18:55:13.216Z", + "duration": 21623.13423, + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://gitlab.dev/root", + "created_at": "2015-12-21T13:14:24.077Z", + "bio": null, + "location": null, + "public_email": "", + "skype": "", + "linkedin": "", + "twitter": "", + "website_url": "", + "organization": null + } + "commit": { + "id": "416d8ea11849050d3d1f5104cf8cf51053e790ab", + "short_id": "416d8ea1", + "created_at": "2016-01-02T15:39:18.000Z", + "parent_ids": [ + "e9a4449c95c64358840902508fc827f1a2eab7df" + ], + "title": "Removed fabric to fix #40", + "message": "Removed fabric to fix #40\n", + "author_name": "Administrator", + "author_email": "admin@example.com", + "authored_date": "2016-01-02T15:39:18.000Z", + "committer_name": "Administrator", + "committer_email": "admin@example.com", + "committed_date": "2016-01-02T15:39:18.000Z" + }, + "pipeline": { + "id": 34, + "sha": "416d8ea11849050d3d1f5104cf8cf51053e790ab", + "ref": "fdroid", + "status": "success", + "web_url": "http://localhost:3000/Commit451/lab-coat/pipelines/34" + }, + "web_url": "http://localhost:3000/Commit451/lab-coat/-/jobs/710", + "artifacts": [ + { + "file_type": "trace", + "size": 1305, + "filename": "job.log", + "file_format": null + } + ], + "runner": null, + "artifacts_expire_at": null + } + } +} +``` + ## Create a new environment Creates a new environment with the given name and external_url. diff --git a/doc/api/group_milestones.md b/doc/api/group_milestones.md index 1c2f56581eb..260eb09cc38 100644 --- a/doc/api/group_milestones.md +++ b/doc/api/group_milestones.md @@ -12,6 +12,7 @@ GET /groups/:id/milestones?iids[]=42 GET /groups/:id/milestones?iids[]=42&iids[]=43 GET /groups/:id/milestones?state=active GET /groups/:id/milestones?state=closed +GET /groups/:id/milestones?title=1.0 GET /groups/:id/milestones?search=version ``` @@ -22,6 +23,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | Array[integer] | optional | Return only the milestones having the given `iid` | | `state` | string | optional | Return only `active` or `closed` milestones | +| `title` | string | optional | Return only the milestones having the given `title` | | `search` | string | optional | Return only milestones with a title or description matching the provided string | ```bash diff --git a/doc/api/milestones.md b/doc/api/milestones.md index 897184d51af..3b76c19dc07 100644 --- a/doc/api/milestones.md +++ b/doc/api/milestones.md @@ -10,6 +10,7 @@ GET /projects/:id/milestones?iids[]=42 GET /projects/:id/milestones?iids[]=42&iids[]=43 GET /projects/:id/milestones?state=active GET /projects/:id/milestones?state=closed +GET /projects/:id/milestones?title=1.0 GET /projects/:id/milestones?search=version ``` @@ -20,6 +21,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | Array[integer] | optional | Return only the milestones having the given `iid` | | `state` | string | optional | Return only `active` or `closed` milestones | +| `title` | string | optional | Return only the milestones having the given `title` | | `search` | string | optional | Return only milestones with a title or description matching the provided string | ```bash diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index 43bbf463c8d..1a4310ef328 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -93,6 +93,36 @@ Example of response } ``` +### Get variables of a pipeline + +``` +GET /projects/:id/pipelines/:pipeline_id/variables +``` + +| Attribute | Type | Required | Description | +|------------|---------|----------|---------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `pipeline_id` | integer | yes | The ID of a pipeline | + +``` +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipelines/46/variables" +``` + +Example of response + +```json +[ + { + "key": "RUN_NIGHTLY_BUILD", + "value": "true" + }, + { + "key": "foo", + "value": "bar" + } +] +``` + ## Create a new pipeline > [Introduced][ce-7209] in GitLab 8.14 diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index 8efb98fe1fc..f36e352da67 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -33,6 +33,7 @@ Example response: { "id":18, "name":"cluster-1", + "domain":"example.com", "created_at":"2019-01-02T20:18:12.563Z", "provider_type":"user", "platform_type":"kubernetes", @@ -90,6 +91,7 @@ Example response: { "id":18, "name":"cluster-1", + "domain":"example.com", "created_at":"2019-01-02T20:18:12.563Z", "provider_type":"user", "platform_type":"kubernetes", @@ -157,6 +159,7 @@ Parameters: | --------- | ---- | -------- | ----------- | | `id` | integer | yes | The ID of the project owned by the authenticated user | | `name` | String | yes | The name of the cluster | +| `domain` | String | no | The [base domain](../user/project/clusters/index.md#base-domain) of the cluster | | `enabled` | Boolean | no | Determines if cluster is active or not, defaults to true | | `platform_kubernetes_attributes[api_url]` | String | yes | The URL to access the Kubernetes API | | `platform_kubernetes_attributes[token]` | String | yes | The token to authenticate against Kubernetes | @@ -247,6 +250,7 @@ Parameters: | `id` | integer | yes | The ID of the project owned by the authenticated user | | `cluster_id` | integer | yes | The ID of the cluster | | `name` | String | no | The name of the cluster | +| `domain` | String | no | The [base domain](../user/project/clusters/index.md#base-domain) of the cluster | | `platform_kubernetes_attributes[api_url]` | String | no | The URL to access the Kubernetes API | | `platform_kubernetes_attributes[token]` | String | no | The token to authenticate against Kubernetes | | `platform_kubernetes_attributes[ca_cert]` | String | no | TLS certificate (needed if API is using a self-signed TLS certificate | @@ -262,7 +266,7 @@ Example request: ```bash curl --header 'Private-Token: <your_access_token>' https://gitlab.example.com/api/v4/projects/26/clusters/24 \ -H "Content-Type:application/json" \ --X PUT --data '{"name":"new-cluster-name","api_url":"https://new-api-url.com"}' +-X PUT --data '{"name":"new-cluster-name","domain":"new-domain.com","api_url":"https://new-api-url.com"}' ``` Example response: @@ -271,6 +275,7 @@ Example response: { "id":24, "name":"new-cluster-name", + "domain":"new-domain.com", "created_at":"2019-01-03T21:53:40.610Z", "provider_type":"user", "platform_type":"kubernetes", diff --git a/doc/api/runners.md b/doc/api/runners.md index 7d7215e6b80..46f7b1d2a25 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -4,6 +4,29 @@ [ce-2640]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2640 +## Registration and authentication tokens + +There are two tokens to take into account when connecting a Runner with GitLab. + +| Token | Description | +| ----- | ----------- | +| Registration token | Token used to [register the Runner](https://docs.gitlab.com/runner/register/). It can be [obtained through GitLab](../ci/runners/README.md). | +| Authentication token | Token used to authenticate the Runner with the GitLab instance. It is obtained either automatically when [registering a Runner](https://docs.gitlab.com/runner/register/), or manually when [registering the Runner via the Runners API](#register-a-new-runner). | + +Here's an example of how the two tokens are used in Runner registration: + +1. You register the Runner via the GitLab API using a registration token, and an + authentication token is returned. +1. You use that authentication token and add it to the + [Runner's configuration file](https://docs.gitlab.com/runner/commands/#configuration-file): + + ```toml + [[runners]] + token = "<authentication_token>" + ``` + +GitLab and Runner are then connected. + ## List owned runners Get a list of specific runners available to the user. @@ -256,6 +279,8 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git ## List runner's jobs +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15432) in GitLab 10.3. + List jobs that are being processed or were processed by specified Runner. ``` @@ -454,7 +479,7 @@ POST /runners | Attribute | Type | Required | Description | |-------------|---------|----------|---------------------| -| `token` | string | yes | Registration token ([Read how to obtain a token](../ci/runners/README.md)) | +| `token` | string | yes | [Registration token](#registration-and-authentication-tokens). | | `description`| string | no | Runner's description| | `info` | hash | no | Runner's metadata | | `active` | boolean| no | Whether the Runner is active | @@ -464,7 +489,7 @@ POST /runners | `maximum_timeout` | integer | no | Maximum timeout set when this Runner will handle the job | ``` -curl --request POST "https://gitlab.example.com/api/v4/runners" --form "token=ipzXrMhuyyJPifUt6ANz" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2" +curl --request POST "https://gitlab.example.com/api/v4/runners" --form "token=<registration_token>" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2" ``` Response: @@ -492,10 +517,10 @@ DELETE /runners | Attribute | Type | Required | Description | |-------------|---------|----------|---------------------| -| `token` | string | yes | Runner's authentication token | +| `token` | string | yes | Runner's [authentication token](#registration-and-authentication-tokens). | ``` -curl --request DELETE "https://gitlab.example.com/api/v4/runners" --form "token=ebb6fc00521627750c8bb750f2490e" +curl --request DELETE "https://gitlab.example.com/api/v4/runners" --form "token=<authentication_token>" ``` Response: @@ -514,10 +539,10 @@ POST /runners/verify | Attribute | Type | Required | Description | |-------------|---------|----------|---------------------| -| `token` | string | yes | Runner's authentication token | +| `token` | string | yes | Runner's [authentication token](#registration-and-authentication-tokens). | ``` -curl --request POST "https://gitlab.example.com/api/v4/runners/verify" --form "token=ebb6fc00521627750c8bb750f2490e" +curl --request POST "https://gitlab.example.com/api/v4/runners/verify" --form "token=<authentication_token>" ``` Response: diff --git a/doc/api/services.md b/doc/api/services.md index 03d0a80aa64..e8ae7ff78f4 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -449,6 +449,45 @@ Get Hangouts Chat service settings for a project. GET /projects/:id/services/hangouts-chat ``` +## HipChat + +Private group chat and IM + +### Create/Edit HipChat service + +Set HipChat service for a project. + +``` +PUT /projects/:id/services/hipchat +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | true | Room token | +| `color` | string | false | The room color | +| `notify` | boolean | false | Enable notifications | +| `room` | string | false |Room name or ID | +| `api_version` | string | false | Leave blank for default (v2) | +| `server` | string | false | Leave blank for default. For example, `https://hipchat.example.com`. | + +### Delete HipChat service + +Delete HipChat service for a project. + +``` +DELETE /projects/:id/services/hipchat +``` + +### Get HipChat service settings + +Get HipChat service settings for a project. + +``` +GET /projects/:id/services/hipchat +``` + ## Irker (IRC gateway) Send IRC messages, on update, to a list of recipients through an Irker gateway. @@ -522,7 +561,7 @@ Parameters: | `username` | string | yes | The username of the user created to be used with GitLab/JIRA. | | `password` | string | yes | The password of the user created to be used with GitLab/JIRA. | | `active` | boolean | no | Activates or deactivates the service. Defaults to false (deactivated). | -| `jira_issue_transition_id` | integer | no | The ID of a transition that moves issues to a closed state. You can find this number under the JIRA workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the **Transitions (id)** column ([see screenshot][trans]). By default, this ID is set to `2`. | +| `jira_issue_transition_id` | string | no | The ID of a transition that moves issues to a closed state. You can find this number under the JIRA workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the **Transitions (id)** column ([see screenshot][trans]). By default, this ID is set to `2`. | ### Delete JIRA service diff --git a/doc/api/suggestions.md b/doc/api/suggestions.md index e88d536282a..188989bc94e 100644 --- a/doc/api/suggestions.md +++ b/doc/api/suggestions.md @@ -24,8 +24,6 @@ Example response: ```json { "id": 36, - "from_original_line": 10, - "to_original_line": 10, "from_line": 10, "to_line": 10, "appliable": false, diff --git a/doc/articles/artifactory_and_gitlab/index.md b/doc/articles/artifactory_and_gitlab/index.md index 6a590b53727..ed9fd135e7c 100644 --- a/doc/articles/artifactory_and_gitlab/index.md +++ b/doc/articles/artifactory_and_gitlab/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../ci/examples/artifactory_and_gitlab/index.md' +--- + This document was moved to [another location](../../ci/examples/artifactory_and_gitlab/index.md) diff --git a/doc/articles/how_to_configure_ldap_gitlab_ce/index.md b/doc/articles/how_to_configure_ldap_gitlab_ce/index.md index a8320c12e6b..8e2e54711e7 100644 --- a/doc/articles/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/articles/how_to_configure_ldap_gitlab_ce/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md' +--- + This document was moved to [another location](../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md). diff --git a/doc/articles/how_to_install_git/index.md b/doc/articles/how_to_install_git/index.md index 3e6003a33b7..62598101895 100644 --- a/doc/articles/how_to_install_git/index.md +++ b/doc/articles/how_to_install_git/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../topics/git/how_to_install_git/index.md' +--- + This document was moved to [another location](../../topics/git/how_to_install_git/index.md). diff --git a/doc/articles/laravel_with_gitlab_and_envoy/index.md b/doc/articles/laravel_with_gitlab_and_envoy/index.md index b092cdb0f7a..fa4f6243410 100644 --- a/doc/articles/laravel_with_gitlab_and_envoy/index.md +++ b/doc/articles/laravel_with_gitlab_and_envoy/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../ci/examples/laravel_with_gitlab_and_envoy/index.md' +--- + This document was moved to [another location](../../ci/examples/laravel_with_gitlab_and_envoy/index.md). diff --git a/doc/articles/numerous_undo_possibilities_in_git/index.md b/doc/articles/numerous_undo_possibilities_in_git/index.md index 3f46ee9a5e6..83aac82db4e 100644 --- a/doc/articles/numerous_undo_possibilities_in_git/index.md +++ b/doc/articles/numerous_undo_possibilities_in_git/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../topics/git/numerous_undo_possibilities_in_git/index.md' +--- + This document was moved to [another location](../../topics/git/numerous_undo_possibilities_in_git/index.md). diff --git a/doc/articles/runner_autoscale_aws/index.md b/doc/articles/runner_autoscale_aws/index.md index e2667aebc5f..fb769731256 100644 --- a/doc/articles/runner_autoscale_aws/index.md +++ b/doc/articles/runner_autoscale_aws/index.md @@ -1 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/index.html' +--- + This document was moved to [another location](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/index.html). diff --git a/doc/ci/README.md b/doc/ci/README.md index 47810a8b7b6..a7ad2018b09 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -43,28 +43,28 @@ For complete control, you can manually configure GitLab CI/CD. With basic knowledge of how GitLab CI/CD works, the following documentation extends your knowledge into more features: -| Topic | Description | -|:--------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------| -| [Creating and using CI/CD pipelines](pipelines.md) | Understand, visualize, create, and use CI/CD pipelines. | -| [CI/CD Variables](variables/README.md) | How environment variables can be configured and made available in pipelines. | -| [Where variables can be used](variables/where_variables_can_be_used.md) | A deeper look into where and how CI/CD variables can be used. | -| [User](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions | Learn about the access levels a user can have for performing certain CI actions. | -| [Configuring GitLab Runners](runners/README.md) | Documentation for configuring [GitLab Runner](https://docs.gitlab.com/runner/). | -| [Introduction to environments and deployments](environments.md) | Learn how to separate your jobs into environments and use them for different purposes like testing, building and, deploying. | -| [Job artifacts](../user/project/pipelines/job_artifacts.md) | Learn about the output of jobs. | -| [Cache dependencies in GitLab CI/CD](caching/index.md) | Discover how to speed up pipelines using caching. | -| [Using Git submodules with GitLab CI](git_submodules.md) | How to run your CI jobs when using Git submodules. | -| [Pipelines for merge requests](merge_request_pipelines/index.md) | Create pipelines specifically for merge requests. | -| [Using SSH keys with GitLab CI/CD](ssh_keys/README.md) | Use SSH keys in your build environment. | -| [Triggering pipelines through the API](triggers/README.md) | Use the GitLab API to trigger a pipeline. | -| [Pipeline schedules](../user/project/pipelines/schedules.md) | Trigger pipelines on a schedule. | -| [Connecting GitLab with a Kubernetes cluster](../user/project/clusters/index.md) | Integrate one or more Kubernetes clusters to your project. | -| [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | -| [Interactive web terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | -| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes in a per-branch basis. | -| [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | -| [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | -| [Protected environments](https://docs.gitlab.com/ce/ci/environments/protected_environments.html) **[PREMIUM]** | Ensure that only people with the right privileges can deploy to an environment. | +| Topic | Description | +|:--------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------| +| [Creating and using CI/CD pipelines](pipelines.md) | Understand, visualize, create, and use CI/CD pipelines. | +| [CI/CD Variables](variables/README.md) | How environment variables can be configured and made available in pipelines. | +| [Where variables can be used](variables/where_variables_can_be_used.md) | A deeper look into where and how CI/CD variables can be used. | +| [User](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions | Learn about the access levels a user can have for performing certain CI actions. | +| [Configuring GitLab Runners](runners/README.md) | Documentation for configuring [GitLab Runner](https://docs.gitlab.com/runner/). | +| [Environments and deployments](environments.md) | Deploy the output of jobs into environments for reviewing, staging, and production. | +| [Job artifacts](../user/project/pipelines/job_artifacts.md) | Learn about the output of jobs. | +| [Cache dependencies in GitLab CI/CD](caching/index.md) | Discover how to speed up pipelines using caching. | +| [Using Git submodules with GitLab CI](git_submodules.md) | How to run your CI jobs when using Git submodules. | +| [Pipelines for merge requests](merge_request_pipelines/index.md) | Create pipelines specifically for merge requests. | +| [Using SSH keys with GitLab CI/CD](ssh_keys/README.md) | Use SSH keys in your build environment. | +| [Triggering pipelines through the API](triggers/README.md) | Use the GitLab API to trigger a pipeline. | +| [Pipeline schedules](../user/project/pipelines/schedules.md) | Trigger pipelines on a schedule. | +| [Connecting GitLab with a Kubernetes cluster](../user/project/clusters/index.md) | Integrate one or more Kubernetes clusters to your project. | +| [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | +| [Interactive web terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | +| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes in a per-branch basis. | +| [Optimizing GitLab for large repositories](large_repositories/index.md) | Useful tips on how to optimize GitLab and GitLab Runner for big repositories. | +| [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | +| [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/index.html) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | ### GitLab Pages @@ -121,7 +121,7 @@ See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJ As GitLab CI/CD has evolved, certain breaking changes have been necessary. These are: -- [CI variables renaming for GitLab 9.0](variables/README.md#gitlab-90-renaming). Read about the +- [CI variables renaming for GitLab 9.0](variables/deprecated_variables.md#gitlab-90-renamed-variables). Read about the deprecated CI variables and what you should use for GitLab 9.0+. - [New CI job permissions model](../user/project/new_ci_build_permissions_model.md). See what changed in GitLab 8.12 and how that affects your jobs. diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md index a06fe6961a7..5ecdf0f8c54 100644 --- a/doc/ci/chatops/README.md +++ b/doc/ci/chatops/README.md @@ -24,7 +24,7 @@ Since ChatOps is built upon GitLab CI/CD, the job has all the same features and * It is strongly recommended to set `only: [chat]` so the job does not run as part of the standard CI pipeline. * If the job is set to `when: manual`, the pipeline will be created however the job will wait to be started. -* It is important to keep in mind that there is very limited support for access control. If the user who triggered the slash command is a developer in the project, the job will run. The job itself can utilize existing [CI/CD variables](../variables/README.html#predefined-environment-variables) like `GITLAB_USER_ID` to perform additional rights validation, however these variables can be [overridden](https://docs.gitlab.com/ce/ci/variables/README.html#priority-of-variables). +* It is important to keep in mind that there is very limited support for access control. If the user who triggered the slash command is a developer in the project, the job will run. The job itself can utilize existing [CI/CD variables](../variables/README.html#predefined-environment-variables) like `GITLAB_USER_ID` to perform additional rights validation, however these variables can be [overridden](../variables/README.html#priority-of-environment-variables). ### Controlling the ChatOps reply diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 9266c4511be..5222cc45bc4 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -303,20 +303,19 @@ services: - docker:dind variables: - CONTAINER_IMAGE: registry.gitlab.com/$CI_PROJECT_PATH DOCKER_HOST: tcp://docker:2375 DOCKER_DRIVER: overlay2 before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build script: - - docker pull $CONTAINER_IMAGE:latest || true - - docker build --cache-from $CONTAINER_IMAGE:latest --tag $CONTAINER_IMAGE:$CI_COMMIT_SHA --tag $CONTAINER_IMAGE:latest . - - docker push $CONTAINER_IMAGE:$CI_COMMIT_SHA - - docker push $CONTAINER_IMAGE:latest + - docker pull $CI_REGISTRY_IMAGE:latest || true + - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest . + - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + - docker push $CI_REGISTRY_IMAGE:latest ``` The steps in the `script` section for the `build` stage can be summed up to: @@ -324,7 +323,7 @@ The steps in the `script` section for the `build` stage can be summed up to: 1. The first command tries to pull the image from the registry so that it can be used as a cache for the `docker build` command. 1. The second command builds a Docker image using the pulled image as a - cache (notice the `--cache-from $CONTAINER_IMAGE:latest` argument) if + cache (notice the `--cache-from $CI_REGISTRY_IMAGE:latest` argument) if available, and tags it. 1. The last two commands push the tagged Docker images to the container registry so that they may also be used as cache for subsequent builds. @@ -421,14 +420,14 @@ and depend on the visibility of your project. For all projects, mostly suitable for public ones: -- **Using the special `gitlab-ci-token` user**: This user is created for you in order to +- **Using the special `$CI_REGISTRY_USER` variable**: The user specified by this variable is created for you in order to push to the Registry connected to your project. Its password is automatically - set with the `$CI_JOB_TOKEN` variable. This allows you to automate building and deploying + set with the `$CI_REGISTRY_PASSWORD` variable. This allows you to automate building and deploying your Docker images and has read/write access to the Registry. This is ephemeral, so it's only valid for one job. You can use the following example as-is: ```sh - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY ``` For private and internal projects: @@ -436,8 +435,10 @@ For private and internal projects: - **Using a personal access token**: You can create and use a [personal access token](../../user/profile/personal_access_tokens.md) in case your project is private: - - For read (pull) access, the scope should be `read_registry`. - - For read/write (pull/push) access, use `api`. + + - For read (pull) access, the scope should be `read_registry`. + - For read/write (pull/push) access, use `api`. + Replace the `<username>` and `<access_token>` in the following example: ```sh @@ -469,9 +470,9 @@ could look like: DOCKER_DRIVER: overlay2 stage: build script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.example.com - - docker build -t registry.example.com/group/project/image:latest . - - docker push registry.example.com/group/project/image:latest + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY + - docker build -t $CI_REGISTRY/group/project/image:latest . + - docker push $CI_REGISTRY/group/project/image:latest ``` You can also make use of [other variables](../variables/README.md) to avoid hardcoding: @@ -486,7 +487,7 @@ variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build @@ -526,7 +527,7 @@ variables: CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 3314c76d234..13c26bc5f47 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -500,7 +500,7 @@ To configure access for `registry.example.com:5000`, follow these steps: bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= ``` -1. Create a [variable] `DOCKER_AUTH_CONFIG` with the content of the +1. Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: ```json @@ -642,7 +642,6 @@ creation. [postgres-hub]: https://hub.docker.com/r/_/postgres/ [mysql-hub]: https://hub.docker.com/r/_/mysql/ [runner-priv-reg]: http://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry -[variable]: ../variables/README.md#variables [entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint [cmd]: https://docs.docker.com/engine/reference/builder/#cmd [register]: https://docs.gitlab.com/runner/register/ diff --git a/doc/ci/environments.md b/doc/ci/environments.md index 05d392d54cb..3e52cc786dd 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -1,45 +1,65 @@ -# Introduction to environments and deployments +# Environments and deployments > Introduced in GitLab 8.9. -During the development of software, there can be many stages until it's ready -for public consumption. You sure want to first test your code and then deploy it -in a testing or staging environment before you release it to the public. That -way you can prevent bugs not only in your software, but in the deployment -process as well. +Environments allow control of the continuous deployment of your software, +all within GitLab. -GitLab CI is capable of not only testing or building your projects, but also +## Introduction + +There are many stages required in the software development process before the software is ready +for public consumption. + +For example: + +1. Develop your code. +1. Test your code. +1. Deploy your code into a testing or staging environment before you release it to the public. + +This helps prevent bugs not only in your software, but in the deployment process as well. + +GitLab CI/CD is capable of not only testing or building your projects, but also deploying them in your infrastructure, with the added benefit of giving you a way to track your deployments. In other words, you can always know what is currently being deployed or has been deployed on your servers. -## Overview +It's important to know that: -With environments, you can control the Continuous Deployment of your software -all within GitLab. All you need to do is define them in your project's -[`.gitlab-ci.yml`][yaml] as we will explore below. GitLab provides a full -history of your deployments per every environment. +- Environments are like tags for your CI jobs, describing where code gets deployed. +- Deployments are created when [jobs](yaml/README.md#introduction) deploy versions of code to environments, + so every environment can have one or more deployments. -Environments are like tags for your CI jobs, describing where code gets deployed. -Deployments are created when [jobs](yaml/README.md#introduction) deploy versions of code to environments, -so every environment can have one or more deployments. GitLab keeps track of -your deployments, so you always know what is currently being deployed on your -servers. If you have a deployment service such as [Kubernetes][kube] +GitLab: + +- Provides a full history of your deployments per every environment. +- Keeps track of your deployments, so you always know what is currently being deployed on your + servers. + +If you have a deployment service such as [Kubernetes](../user/project/clusters/index.md) enabled for your project, you can use it to assist with your deployments, and can even access a [web terminal](#web-terminals) for your environment from within GitLab! -To better understand how environments and deployments work, let's consider an -example. We assume that you have already created a project in GitLab and set up -a Runner. The example will cover the following: +## Configuring environments + +Configuring environments involves: + +1. Understanding how [pipelines](pipelines.md) work. +1. Defining environments in your project's [`.gitlab-ci.yml`](yaml/README.md) file. + +The rest of this section illustrates how to configure environments and deployments using an example. +It assumes you have already: -- We are developing an application -- We want to run tests and build our app on all branches -- Our default branch is `master` -- We deploy the app only when a pipeline on `master` branch is run +- Created a [project](../gitlab-basics/create-project.md) in GitLab. +- Set up [a Runner](runners/README.md). -Let's see how it all ties together. +In the scenario: -## Defining environments +- We are developing an application. +- We want to run tests and build our app on all branches. +- Our default branch is `master`. +- We deploy the app only when a pipeline on `master` branch is run. + +### Defining environments Let's consider the following `.gitlab-ci.yml` example: @@ -70,123 +90,64 @@ deploy_staging: We have defined 3 [stages](yaml/README.md#stages): -- test -- build -- deploy +- `test` +- `build` +- `deploy` The jobs assigned to these stages will run in this order. If a job fails, then the jobs that are assigned to the next stage won't run, rendering the pipeline -as failed. In our case, the `test` job will run first, then the `build` and -lastly the `deploy_staging`. With this, we ensure that first the tests pass, -then our app is able to be built successfully, and lastly we deploy to the -staging server. +as failed. + +In our case: + +- The `test` job will run first. +- Then the `build` job. +- Lastly the `deploy_staging` job. + +With this configuration, we ensure that: + +- The tests pass. +- Our app is able to be built successfully. +- Lastly we deploy to the staging server. +NOTE: **Note:** The `environment` keyword is just a hint for GitLab that this job actually -deploys to this environment's `name`. It can also have a `url` which, as we -will later see, is exposed in various places within GitLab. Each time a job that -has an environment specified and succeeds, a deployment is recorded, remembering +deploys to this environment's `name`. It can also have a `url` that is +exposed in various places within GitLab. Each time a job that +has an environment specified succeeds, a deployment is recorded, storing the Git SHA and environment name. -> **Note:** +In summary, with the above `.gitlab-ci.yml` we have achieved the following: + +- All branches will run the `test` and `build` jobs. +- The `deploy_staging` job will run [only](yaml/README.md#onlyexcept-basic) on the `master` + branch, which means all merge requests that are created from branches don't + get deployed to the staging server. +- When a merge request is merged, all jobs will run and the `deploy_staging` + job will deploy our code to a staging server while the deployment + will be recorded in an environment named `staging`. + > Starting with GitLab 8.15, the environment name is exposed to the Runner in > two forms: `$CI_ENVIRONMENT_NAME`, and `$CI_ENVIRONMENT_SLUG`. The first is > the name given in `.gitlab-ci.yml` (with any variables expanded), while the > second is a "cleaned-up" version of the name, suitable for use in URLs, DNS, > etc. -> -> **Note:** + > Starting with GitLab 9.3, the environment URL is exposed to the Runner via > `$CI_ENVIRONMENT_URL`. The URL would be expanded from `.gitlab-ci.yml`, or if > the URL was not defined there, the external URL from the environment would be > used. -To sum up, with the above `.gitlab-ci.yml` we have achieved that: - -- All branches will run the `test` and `build` jobs. -- The `deploy_staging` job will run [only](yaml/README.md#onlyexcept-basic) on the `master` - branch which means all merge requests that are created from branches don't - get to deploy to the staging server -- When a merge request is merged, all jobs will run and the `deploy_staging` - in particular will deploy our code to a staging server while the deployment - will be recorded in an environment named `staging`. - -Let's now see how that information is exposed within GitLab. - -## Viewing the current status of an environment - -The environment list under your project's **Operations > Environments**, is -where you can find information of the last deployment status of an environment. - -Here's how the Environments page looks so far. - - - -There's a bunch of information there, specifically you can see: - -- The environment's name with a link to its deployments -- The last deployment ID number and who performed it -- The job ID of the last deployment with its respective job name -- The commit information of the last deployment such as who committed, to what - branch and the Git SHA of the commit -- The exact time the last deployment was performed -- A button that takes you to the URL that you have defined under the - `environment` keyword in `.gitlab-ci.yml` -- A button that re-deploys the latest deployment, meaning it runs the job - defined by the environment name for that specific commit - -> **Notes:** -> -> - While you can create environments manually in the web interface, we recommend -> that you define your environments in `.gitlab-ci.yml` first. They will -> be automatically created for you after the first deploy. -> - The environments page can only be viewed by Reporters and above. For more -> information on the permissions, see the [permissions documentation][permissions]. -> - Only deploys that happen after your `.gitlab-ci.yml` is properly configured -> will show up in the "Environment" and "Last deployment" lists. - -The information shown in the Environments page is limited to the latest -deployments, but as you may have guessed an environment can have multiple -deployments. - -## Viewing the deployment history of an environment - -GitLab keeps track of your deployments, so you always know what is currently -being deployed on your servers. That way you can have the full history of your -deployments per every environment right in your browser. Clicking on an -environment will show the history of its deployments. Assuming you have deployed -multiple times already, here's how a specific environment's page looks like. - - - -We can see the same information as when in the Environments page, but this time -all deployments are shown. As you may have noticed, apart from the **Re-deploy** -button there are now **Rollback** buttons for each deployment. Let's see how -that works. - -## Rolling back changes - -You can't control everything, so sometimes things go wrong. When that unfortunate -time comes GitLab has you covered. Simply by clicking the **Rollback** button -that can be found in the deployments page -(**Operations > Environments > `environment name`**) you can relaunch the -job with the commit associated with it. - ->**Note:** -Bear in mind that your mileage will vary and it's entirely up to how you define -the deployment process in the job's `script` whether the rollback succeeds or not. -GitLab CI is just following orders. +### Configuring manual deployments -Thankfully that was the staging server that we had to rollback, and since we -learn from our mistakes, we decided to not make the same again when we deploy -to the production server. Enter manual actions for deployments. +Converting automatically executed job into jobs requiring to a manual action involves +adding `when: manual` to the job's configuration. -## Manually deploying to environments +To expand on the [previous example](#defining-environments), the following includes +another job that deploys our app to a production server and is +tracked by a `production` environment. -Turning a job from running automatically to a manual action is as simple as -adding `when: manual` to it. To expand on our previous example, let's add -another job that this time deploys our app to a production server and is -tracked by a `production` environment. The `.gitlab-ci.yml` looks like this -so far: +The `.gitlab-ci.yml` file for this is as follows: ```yaml stages: @@ -224,41 +185,62 @@ deploy_prod: - master ``` -The `when: manual` action exposes a play button in GitLab's UI and the -`deploy_prod` job will only be triggered if and when we click that play button. -You can find it in the pipeline, job, environment, and deployment views. +The `when: manual` action: -| Pipelines | Single pipeline | Environments | Deployments | jobs | -| --------- | ----------------| ------------ | ----------- | -------| -|  |  |  |  |  | +- Exposes a "play" button in GitLab's UI. +- Means the `deploy_prod` job will only be triggered when the "play" button is clicked. -Clicking on the play button in either of these places will trigger the -`deploy_prod` job, and the deployment will be recorded under a new +You can find the "play" button in the pipelines, environments, deployments, and jobs views. + +| View | Screenshot | +|:----------------|:-------------------------------------------------------------------------------| +| Pipelines |  | +| Single pipeline |  | +| Environments |  | +| Deployments |  | +| Jobs |  | + +Clicking on the play button in any view will trigger the `deploy_prod` job, and the deployment will be recorded under a new environment named `production`. ->**Note:** -Remember that if your environment's name is `production` (all lowercase), then +NOTE: **Note:** +If your environment's name is `production` (all lowercase), it will get recorded in [Cycle Analytics](../user/project/cycle_analytics.md). -Double the benefit! -## Dynamic environments +### Configuring dynamic environments -As the name suggests, it is possible to create environments on the fly by just -declaring their names dynamically in `.gitlab-ci.yml`. Dynamic environments is -the basis of [Review apps](review_apps/index.md). +Other environments are good for deploying to stable environments like staging or production. -NOTE: **Note:** -The `name` and `url` parameters can use most of the CI/CD variables, -including [predefined](variables/README.md#predefined-environment-variables), -[project/group ones](variables/README.md#variables) and -[`.gitlab-ci.yml` variables](yaml/README.md#variables). You however cannot use variables -defined under `script` or on the Runner's side. There are also other variables that -are unsupported in the context of `environment:name`. You can read more about -[where variables can be used](variables/where_variables_can_be_used.md). - -GitLab Runner exposes various [environment variables][variables] when a job runs, -and as such, you can use them as environment names. Let's add another job in -our example which will deploy to all branches except `master`: +However, what about environments for branches other than `master`? Dynamic environments can be used to achieve these. + +Dynamic environments make it possible to create environments on the fly by +declaring their names dynamically in `.gitlab-ci.yml`. + +Dynamic environments form the basis of [Review apps](review_apps/index.md). + +#### Allowed variables + +The `name` and `url` parameters for dynamic environments can use most available CI/CD variables, +including: + +- [Predefined environment variables](variables/README.md#predefined-environment-variables) +- [Project and group variables](variables/README.md#gitlab-cicd-environment-variables) +- [`.gitlab-ci.yml` variables](yaml/README.md#variables) + +However, you cannot use variables defined: + +- Under `script`. +- On the Runner's side. + +There are also other variables that are unsupported in the context of `environment:name`. +For more information, see [Where variables can be used](variables/where_variables_can_be_used.md). + +#### Example configuration + +GitLab Runner exposes various [environment variables](variables/README.md) when a job runs and so +you can use them as environment names. + +In the following example, a job will deploy to all branches except `master`: ```yaml deploy_review: @@ -274,39 +256,49 @@ deploy_review: - master ``` -Let's break it down in pieces. The job's name is `deploy_review` and it runs -on the `deploy` stage. The `script` at this point is fictional, you'd have to -use your own based on your deployment. Then, we set the `environment` with the -`environment:name` being `review/$CI_COMMIT_REF_NAME`. Now that's an interesting -one. Since the [environment name][env-name] can contain slashes (`/`), we can -use this pattern to distinguish between dynamic environments and the regular -ones. - -So, the first part is `review`, followed by a `/` and then `$CI_COMMIT_REF_NAME` -which takes the value of the branch name. Since `$CI_COMMIT_REF_NAME` itself may -also contain `/`, or other characters that would be invalid in a domain name or -URL, we use `$CI_ENVIRONMENT_SLUG` in the `environment:url` so that the -environment can get a specific and distinct URL for each branch. In this case, -given a `$CI_COMMIT_REF_NAME` of `100-Do-The-Thing`, the URL will be something -like `https://100-do-the-4f99a2.example.com`. Again, the way you set up -the web server to serve these requests is based on your setup. - -You could also use `$CI_COMMIT_REF_SLUG` in `environment:url`, e.g.: -`https://$CI_COMMIT_REF_SLUG.example.com`. We use `$CI_ENVIRONMENT_SLUG` -here because it is guaranteed to be unique, but if you're using a workflow like -[GitLab Flow][gitlab-flow], collisions are very unlikely, and you may prefer -environment names to be more closely based on the branch name - the example -above would give you an URL like `https://100-do-the-thing.example.com` - -Last but not least, we tell the job to run [`only`](yaml/README.md#onlyexcept-basic) on branches -[`except`](yaml/README.md#onlyexcept-basic) master. - ->**Note:** +In this example: + +- The job's name is `deploy_review` and it runs on the `deploy` stage. +- We set the `environment` with the `environment:name` as `review/$CI_COMMIT_REF_NAME`. + Since the [environment name](yaml/README.md#environmentname) can contain slashes (`/`), we can + use this pattern to distinguish between dynamic environments and the regular ones. +- We tell the job to run [`only`](yaml/README.md#onlyexcept-basic) on branches [`except`](yaml/README.md#onlyexcept-basic) `master`. + +For the value of: + +- `environment:name`, the first part is `review`, followed by a `/` and then `$CI_COMMIT_REF_NAME`, + which takes the value of the branch name. +- `environment:url`, since `$CI_COMMIT_REF_NAME` itself may also contain `/`, or other characters that + would be invalid in a domain name or URL, we use `$CI_ENVIRONMENT_SLUG` so that the environment can get a specific and distinct URL for each branch. + + For example, given a `$CI_COMMIT_REF_NAME` of `100-Do-The-Thing`, the URL will be something + like `https://100-do-the-4f99a2.example.com`. Again, the way you set up + the web server to serve these requests is based on your setup. + + You could also use `$CI_COMMIT_REF_SLUG` in `environment:url`. For example, `https://$CI_COMMIT_REF_SLUG.example.com`. + We have used `$CI_ENVIRONMENT_SLUG` here because it is guaranteed to be unique. If you're using a workflow like + [GitLab Flow](../workflow/gitlab_flow.md), collisions are unlikely and you may prefer environment names to be more closely based on the branch name. The example + above would give you an URL like `https://100-do-the-thing.example.com`. + +NOTE: **Note:** You are not bound to use the same prefix or only slashes in the dynamic -environments' names (`/`), but as we will see later, this will enable the -[grouping similar environments](#grouping-similar-environments) feature. +environments' names (`/`). However, this will enable the [grouping similar environments](#grouping-similar-environments) feature. + +### Complete example + +The configuration in this section provides a full development workflow where your app is: -The whole `.gitlab-ci.yml` looks like this so far: +- Tested. +- Built. +- Deployed as a Review App. +- Deployed to a staging server once the merge request is merged. +- Finally, manually deployed to the production server. + +The following combines the previous configuration examples, including: + +- Defining [simple environments](#defining-environments) for testing, building, and deployment to staging. +- Adding [manual actions](#configuring-manual-deployments) for deployment to production. +- Creating [dynamic environments](#configuring-dynamic-environments) for deployments for reviewing. ```yaml stages: @@ -357,8 +349,9 @@ deploy_prod: ``` A more realistic example would include copying files to a location where a -webserver (NGINX) could then read and serve. The example below will copy the -`public` directory to `/srv/nginx/$CI_COMMIT_REF_SLUG/public`: +webserver (for example, NGINX) could then read and serve. + +The example below will copy the `public` directory to `/srv/nginx/$CI_COMMIT_REF_SLUG/public`: ```yaml review_app: @@ -370,76 +363,155 @@ review_app: url: https://$CI_COMMIT_REF_SLUG.example.com ``` -It is assumed that the user has already set up NGINX and GitLab Runner in the -server this job will run on. +This example requires that NGINX and GitLab Runner are set up on the server this job will run on. + +NOTE: **Note:** +See the [limitations](#limitations) section for some edge cases regarding naming of your branches and Review Apps. + +The complete example provides the following workflow for developers: + +- Create a branch locally. +- Make changes and commit them +- Push the branch to GitLab. +- Create a merge request. + +Behind the scenes, GitLab runner will: + +- Pick up the changes and start running the jobs. +- Run the jobs sequentially as defined in `stages`: + - First, run the tests. + - If the tests succeed, build the app. + - If the build succeeds, the app will be is deployed to an environment with a name specific to the + branch. + +So now, every branch: + +- Gets its own environment. +- Is deployed to its own location, with the added benefit of: + - Having a [history of deployments](#viewing-deployment-history). + - Being able to [rollback changes](#retrying-and-rolling-back) if needed. + +For more information on using the URL, see [Using the environment URL](#using-the-environment-url). + +### Protected environments + +Environments can be "protected", restricting access to them. + +For more information, see [Protected environments](environments/protected_environments.md). + +## Working with environments + +Having configured environments, GitLab provides many features to work with them. These are documented below. + +### Viewing environments and deployments + +A list of environments and deployment statuses is available on project's **Operations > Environments** page. + +For example: + + + +This example shows: + +- The environment's name with a link to its deployments. +- The last deployment ID number and who performed it. +- The job ID of the last deployment with its respective job name. +- The commit information of the last deployment such as who committed, to what + branch, and the Git SHA of the commit. +- The exact time the last deployment was performed. +- A button that takes you to the URL that you have defined under the + `environment` keyword in `.gitlab-ci.yml`. +- A button that re-deploys the latest deployment, meaning it runs the job + defined by the environment name for that specific commit. + +The information shown in the **Environments** page is limited to the latest +deployments, but an environment can have multiple deployments. + +> **Notes:** +> +> - While you can create environments manually in the web interface, we recommend +> that you define your environments in `.gitlab-ci.yml` first. They will +> be automatically created for you after the first deploy. +> - The environments page can only be viewed by Reporters and above. For more +> information on the permissions, see the [permissions documentation](../user/permissions.md). +> - Only deploys that happen after your `.gitlab-ci.yml` is properly configured +> will show up in the **Environment** and **Last deployment** lists. + +### Viewing deployment history + +GitLab keeps track of your deployments, so you: ->**Note:** -Be sure to check out the [limitations](#limitations) section for some edge -cases regarding naming of your branches and Review Apps. +- Always know what is currently being deployed on your servers. +- Can have the full history of your deployments per every environment. ---- +Clicking on an environment shows the history of its deployments. Here's an example **Environments** page +with multiple deployments: -The development workflow would now be: + + +This view is similar to the **Environments** page, but all deployments are shown. Also in this view +is a **Rollback** button. For more information, see [Retrying and rolling back](#retrying-and-rolling-back). -- Developer creates a branch locally -- Developer makes changes, commits and pushes the branch to GitLab -- Developer creates a merge request +### Retrying and rolling back -Behind the scenes: +If there is a problem with a deployment, you can retry it or roll it back. -- GitLab Runner picks up the changes and starts running the jobs -- The jobs run sequentially as defined in `stages` - - First, the tests pass - - Then, the job begins and successfully also passes - - Lastly, the app is deployed to an environment with a name specific to the - branch +To retry or rollback a deployment: -So now, every branch gets its own environment and is deployed to its own place -with the added benefit of having a [history of deployments](#viewing-the-deployment-history-of-an-environment) -and also being able to [rollback changes](#rolling-back-changes) if needed. -Let's briefly see where URL that's defined in the environments is exposed. +1. Navigate to **Operations > Environments**. +1. Click on the environment. +1. On the page that lists the deployment history for the environment, click the: + - **Rollback** button against a previously successful deployment, to roll back to that deployment. + - **Retry** button against the last deployment, to retry that deployment. -## Making use of the environment URL +NOTE: **Note:** +The defined deployment process in the job's `script` determines whether the rollback succeeds or not. + +### Using the environment URL The [environment URL](yaml/README.md#environmenturl) is exposed in a few places within GitLab. -| In a merge request widget as a link | In the Environments view as a button | In the Deployments view as a button | -| -------------------- | ------------ | ----------- | -|  |  |  | +These are: + +- In a merge request widget as a link: +  +- In the Environments view as a button: +  +- In the Deployments view as a button: +  -If a merge request is eventually merged to the default branch (in our case -`master`) and that branch also deploys to an environment (in our case `staging` -and/or `production`) you can see this information in the merge request itself. +You can see this information in a merge request itself if: + +- The merge request is eventually merged to the default branch (usually `master`). +- That branch also deploys to an environment (for example, `staging` or `production`). + +For example:  -### Go directly from source files to public pages on the environment +#### Going from source files to public pages With GitLab's [Route Maps](review_apps/index.md#route-maps) you can go directly from source files to public pages on the environment set for Review Apps. -From then on, you have a full development cycle, where your app is tested, built, deployed -as a Review App, deployed to a staging server once the merge request is merged, -and finally manually deployed to the production server. This is a simple workflow, -but when you have multiple developers working on a project -at the same time, each of them pushing to their own branches, dynamic environments are -created all the time. In which case, you probably want to do some clean up. Read -next how environments can be stopped. +### Stopping an environment -## Stopping an environment +Stopping an environment: -By stopping an environment, you are effectively terminating its recording of the -deployments that happen in it. +- Moves it from the list of **Available** environments to the list of **Stopped** environments on the [**Environments** page](#viewing-environments-and-deployments). +- Executes an [`on_stop` action](yaml/README.md#environmenton_stop), if defined. -A branch is associated with an environment when the CI pipeline that is created -for this branch, was recently deployed to this environment. You can think of -the CI pipeline as the glue between the branch and the environment: -`branch ➔ CI pipeline ➔ environment`. +This is often used when multiple developers are working on a project at the same time, +each of them pushing to their own branches, causing many dynamic environments to be created. -There is a special case where environments can be manually stopped. That can -happen if you provide another job for that matter. The syntax is a little -tricky since a job calls another job to do the job. +NOTE: **Note:** +Starting with GitLab 8.14, dynamic environments will be stopped automatically +when their associated branch is deleted. + +#### Automatically stopping an environment + +Environments can be stopped automatically using special configuration. Consider the following example where the `deploy_review` calls the `stop_review` to clean up and stop the environment: @@ -470,35 +542,31 @@ stop_review: action: stop ``` -Setting the [`GIT_STRATEGY`][git-strategy] to `none` is necessary on the -`stop_review` job so that the [GitLab Runner] won't try to checkout the code +Setting the [`GIT_STRATEGY`](yaml/README.md#git-strategy) to `none` is necessary on the +`stop_review` job so that the [GitLab Runner](https://docs.gitlab.com/runner/) won't try to check out the code after the branch is deleted. ->**Note:** -Starting with GitLab 8.14, dynamic environments will be stopped automatically -when their associated branch is deleted. - When you have an environment that has a stop action defined (typically when -the environment describes a review app), GitLab will automatically trigger a +the environment describes a Review App), GitLab will automatically trigger a stop action when the associated branch is deleted. The `stop_review` job must be in the same `stage` as the `deploy_review` one in order for the environment to automatically stop. -You can read more in the [`.gitlab-ci.yml` reference][onstop]. +You can read more in the [`.gitlab-ci.yml` reference](yaml/README.md#environmenton_stop). -## Grouping similar environments +### Grouping similar environments -> [Introduced][ce-7015] in GitLab 8.14. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7015) in GitLab 8.14. -As we've seen in the [dynamic environments](#dynamic-environments), you can -prepend their name with a word, then followed by a `/` and finally the branch -name which is automatically defined by the `CI_COMMIT_REF_NAME` variable. +As documented in [Configuring dynamic environments](#configuring-dynamic-environments), you can +prepend environment name with a word, followed by a `/`, and finally the branch +name, which is automatically defined by the `CI_COMMIT_REF_NAME` variable. In short, environments that are named like `type/foo` are presented under a group named `type`. -In our minimal example, we name the environments `review/$CI_COMMIT_REF_NAME` -where `$CI_COMMIT_REF_NAME` is the branch name: +In our [minimal example](#example-configuration), we named the environments `review/$CI_COMMIT_REF_NAME` +where `$CI_COMMIT_REF_NAME` is the branch name. Here is a snippet of the example: ```yaml deploy_review: @@ -509,49 +577,47 @@ deploy_review: name: review/$CI_COMMIT_REF_NAME ``` -In that case, if you visit the Environments page, and provided the branches +In this case, if you visit the **Environments** page and the branches exist, you should see something like:  -## Monitoring environments +### Monitoring environments > **Notes:** > > - For the monitoring dashboard to appear, you need to: -> - Have enabled the [Prometheus integration][prom] -> - Configured Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/index.md) -> - With GitLab 9.2, all deployments to an environment are shown directly on the -> monitoring dashboard +> - Enable the [Prometheus integration](../user/project/integrations/prometheus.md). +> - Configure Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/index.md) +> - With GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. -If you have enabled [Prometheus for monitoring system and response metrics](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html), you can monitor the performance behavior of your app running in each environment. +If you have enabled [Prometheus for monitoring system and response metrics](../user/project/integrations/prometheus.md), you can monitor the performance behavior of your app running in each environment. -Once configured, GitLab will attempt to retrieve [supported performance metrics](https://docs.gitlab.com/ee/user/project/integrations/prometheus_library/index.html) for any -environment which has had a successful deployment. If monitoring data was -successfully retrieved, a Monitoring button will appear for each environment. +Once configured, GitLab will attempt to retrieve [supported performance metrics](../user/project/integrations/prometheus_library/index.md) for any +environment that has had a successful deployment. If monitoring data was +successfully retrieved, a **Monitoring** button will appear for each environment.  -Clicking on the Monitoring button will display a new page, showing up to the last +Clicking on the **Monitoring** button will display a new page showing up to the last 8 hours of performance data. It may take a minute or two for data to appear after initial deployment. -All deployments to an environment are shown directly on the monitoring dashboard +All deployments to an environment are shown directly on the monitoring dashboard, which allows easy correlation between any changes in performance and a new version of the app, all without leaving GitLab.  -## Web terminals +### Web terminals + +> Web terminals were added in GitLab 8.15 and are only available to project Maintainers and Owners. ->**Note:** -Web terminals were added in GitLab 8.15 and are only available to project -maintainers and owners. +If you deploy to your environments with the help of a deployment service (for example, +the [Kubernetes integration](../user/project/clusters/index.md)), GitLab can open +a terminal session to your environment. -If you deploy to your environments with the help of a deployment service (e.g., -the [Kubernetes integration][kube]), GitLab can open -a terminal session to your environment! This is a very powerful feature that -allows you to debug issues without leaving the comfort of your web browser. To +This is a powerful feature that allows you to debug issues without leaving the comfort of your web browser. To enable it, just follow the instructions given in the service integration documentation. @@ -568,41 +634,71 @@ establish the terminal session:  -This works just like any other terminal - you'll be in the container created -by your deployment, so you can run shell commands and get responses in real -time, check the logs, try out configuration or code tweaks, etc. You can open -multiple terminals to the same environment - they each get their own shell -session - and even a multiplexer like `screen` or `tmux`! +This works just like any other terminal. You'll be in the container created +by your deployment so you can: ->**Note:** -Container-based deployments often lack basic tools (like an editor), and may -be stopped or restarted at any time. If this happens, you will lose all your -changes! Treat this as a debugging tool, not a comprehensive online IDE. +- Run shell commands and get responses in real time. +- Check the logs. +- Try out configuration or code tweaks etc. ---- +You can open multiple terminals to the same environment, they each get their own shell +session and even a multiplexer like `screen` or `tmux`. -While this is fine for deploying to some stable environments like staging or -production, what happens for branches? So far we haven't defined anything -regarding deployments for branches other than `master`. Dynamic environments -will help us achieve that. +NOTE: **Note:** +Container-based deployments often lack basic tools (like an editor), and may +be stopped or restarted at any time. If this happens, you will lose all your +changes. Treat this as a debugging tool, not a comprehensive online IDE. -## Checkout deployments locally +### Check out deployments locally -Since 8.13, a reference in the git repository is saved for each deployment, so +Since GitLab 8.13, a reference in the Git repository is saved for each deployment, so knowing the state of your current environments is only a `git fetch` away. -In your git config, append the `[remote "<your-remote>"]` block with an extra +In your Git configuration, append the `[remote "<your-remote>"]` block with an extra fetch line: -``` +```text fetch = +refs/environments/*:refs/remotes/origin/environments/* ``` +### Scoping environments with specs **[PREMIUM]** + +Some GitLab [Enterprise Edition](https://about.gitlab.com/pricing/) features can behave differently for each +environment. For example, you can [create a secret variable to be injected only into a production environment](https://docs.gitlab.com/ee/ci/variables/#limiting-environment-scopes-of-environment-variables-premium). + +In most cases, these features use the _environment specs_ mechanism, which offers +an efficient way to implement scoping within each environment group. + +Let's say there are four environments: + +- `production` +- `staging` +- `review/feature-1` +- `review/feature-2` + +Each environment can be matched with the following environment spec: + +| Environment Spec | `production` | `staging` | `review/feature-1` | `review/feature-2` | +|:-----------------|:-------------|:----------|:-------------------|:-------------------| +| * | Matched | Matched | Matched | Matched | +| production | Matched | | | | +| staging | | Matched | | | +| review/* | | | Matched | Matched | +| review/feature-1 | | | Matched | | + +As you can see, you can use specific matching for selecting a particular environment, +and also use wildcard matching (`*`) for selecting a particular environment group, +such as [Review apps](review_apps/index.md) (`review/*`). + +NOTE: **Note:** +The most _specific_ spec takes precedence over the other wildcard matching. +In this case, `review/feature-1` spec takes precedence over `review/*` and `*` specs. + ## Limitations -1. You are limited to use only the [CI predefined variables][variables] in the - `environment: name`. If you try to re-use variables defined inside `script` - as part of the environment name, it will not work. +You are limited to use only the [CI predefined variables](variables/README.md) in the +`environment: name`. If you try to re-use variables defined inside `script` +as part of the environment name, it will not work. ## Further reading @@ -611,18 +707,3 @@ Below are some links you may find interesting: - [The `.gitlab-ci.yml` definition of environments](yaml/README.md#environment) - [A blog post on Deployments & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) - [Review Apps - Use dynamic environments to deploy your code for every branch](review_apps/index.md) - -[Pipelines]: pipelines.md -[yaml]: yaml/README.md -[environments]: #environments -[deployments]: #deployments -[permissions]: ../user/permissions.md -[variables]: variables/README.md -[env-name]: yaml/README.md#environmentname -[onstop]: yaml/README.md#environmenton_stop -[ce-7015]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7015 -[gitlab-flow]: ../workflow/gitlab_flow.md -[gitlab runner]: https://docs.gitlab.com/runner/ -[git-strategy]: yaml/README.md#git-strategy -[kube]: ../user/project/clusters/index.md -[prom]: ../user/project/integrations/prometheus.md diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md new file mode 100644 index 00000000000..219af4ced9d --- /dev/null +++ b/doc/ci/environments/protected_environments.md @@ -0,0 +1,48 @@ +# Protected Environments **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6303) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. + +## Overview + +[Environments](../environments.md) can be used for different reasons: + +- Some of them are just for testing. +- Others are for production. + +Because deploy jobs can be raised by different users with different roles, it is important that +specific environments are "protected" to avoid unauthorized people affecting them. + +By default, a protected environment does one thing: it ensures that only people +with the right privileges can deploy to it, thus keeping it safe. + +NOTE: **Note**: +A GitLab admin is always allowed to use environments, even if they are protected. + +To protect, update, or unprotect an environment, you need to have at least +[Maintainer permissions](../../user/permissions.md). + +## Protecting environments + +To protect an environment: + +1. Navigate to your project's **Settings > CI/CD**. +1. Expand the **Protected Environments** section. +1. From the **Environment** dropdown menu, select the environment you want to protect. +1. In the **Allowed to Deploy** dropdown menu, select the role, users, or groups you want to have deploy access. + There are some considerations to have in mind: + - There are two roles to choose from: + - **Maintainers**: will allow access to all maintainers in the project. + - **Developers**: will allow access to all maintainers and all developers in the project. + - You can only select groups that are associated with the project. + - Only users that have at least Developer permission level will appear on + the **Allowed to Deploy** dropdown menu. +1. Click the **Protect** button. + +The protected environment will now appear in the list of protected environments. + +## Modifying and unprotecting environments + +Maintainers can: + +- Update existing protected environments at any time by changing the access on **Allowed to deploy** dropdown menu. +- Unprotect a protected environment by clicking the **Unprotect** button of the environment to unprotect. diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md index 908cf85980e..d6ad00a77da 100644 --- a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -511,7 +511,7 @@ Errors can be easily debugged through GitLab's build logs, and within minutes of you can see the changes live on your game. Setting up Continuous Integration and Continuous Deployment from the start with Dark Nova enables -rapid but stable development. We can easily test changes in a separate [environment](../../../ci/environments.md#introduction-to-environments-and-deployments), +rapid but stable development. We can easily test changes in a separate [environment](../../environments.md), or multiple environments if needed. Balancing and updating a multiplayer game can be ongoing and tedious, but having faith in a stable deployment with GitLab CI/CD allows a lot of breathing room in quickly getting changes to players. diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png b/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png Binary files differindex 4675e20ef79..edeaa011ada 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 0f809ed05ca..f56d5429fb7 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -116,7 +116,7 @@ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys cat ~/.ssh/id_rsa ``` -Now, let's add it to your GitLab project as a [variable](../../variables/README.md#variables). +Now, let's add it to your GitLab project as a [variable](../../variables/README.md#gitlab-cicd-environment-variables). Variables are user-defined variables and are stored out of `.gitlab-ci.yml`, for security purposes. They can be added per project by navigating to the project's **Settings** > **CI/CD**. diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md index fa18cb22aed..e1164b8d03a 100644 --- a/doc/ci/examples/test-scala-application.md +++ b/doc/ci/examples/test-scala-application.md @@ -69,5 +69,5 @@ in the `.gitlab-ci.yml` file with your application's name. ## Heroku API key You can look up your Heroku API key in your -[account](https://dashboard.heroku.com/account). Add a [protected variable](../variables/README.md#protected-variables) with +[account](https://dashboard.heroku.com/account). Add a [protected variable](../variables/README.md#protected-environment-variables) with this value in **Project ➔ Variables** with key `HEROKU_API_KEY`. diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index 1a909e8892a..4a5fda661df 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -11,7 +11,7 @@ last_updated: 2019-03-06 [Phoenix][phoenix-site] is a web development framework written in [Elixir][elixir-site], which is a functional language designed for productivity and maintainability that runs on the -[Erlang VM][erlang-site]. Erlang VM is really really fast and can handle very large numbers of +[Erlang VM](https://www.erlang.org). Erlang VM is really really fast and can handle very large numbers of simultaneous users. That's why we're hearing so much about Phoenix today. @@ -27,7 +27,7 @@ and GitLab UI._ ### What is Phoenix? [Phoenix][phoenix-site] is a web development framework written in [Elixir][elixir-site] very useful - to build fast, reliable, and high-performance applications, as it uses [Erlang VM][erlang-site]. + to build fast, reliable, and high-performance applications, as it uses [Erlang VM](https://www.erlang.org). Many components and concepts are similar to Ruby on Rails or Python's Django. High developer productivity and high application performance are only a few advantages on learning how to use it. @@ -406,7 +406,6 @@ other reasons][ci-reasons] to keep using GitLab CI/CD. The benefits to our teams [elixir-site]: http://elixir-lang.org/ "Elixir" [elixir-mix]: http://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html "Introduction to mix" [elixir-docs]: http://elixir-lang.org/getting-started/introduction.html "Elixir Documentation" -[erlang-site]: http://erlang.org "Erlang" [elixir-install]: https://elixir-lang.org/install.html "Elixir Installation" [ecto]: http://hexdocs.pm/ecto "Ecto" [ecto-repo]: https://hexdocs.pm/ecto/Ecto.html#module-repositories "Ecto Repositories" diff --git a/doc/ci/introduction/img/gitlab_workflow_example.png b/doc/ci/introduction/img/gitlab_workflow_example.png Binary files differdeleted file mode 100644 index 94e7753c3b2..00000000000 --- a/doc/ci/introduction/img/gitlab_workflow_example.png +++ /dev/null diff --git a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png Binary files differnew file mode 100644 index 00000000000..204e9c462e5 --- /dev/null +++ b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png diff --git a/doc/ci/introduction/img/job_running.png b/doc/ci/introduction/img/job_running.png Binary files differindex d5f922ceb8c..d5f922ceb8c 100755..100644 --- a/doc/ci/introduction/img/job_running.png +++ b/doc/ci/introduction/img/job_running.png diff --git a/doc/ci/introduction/img/pipeline_status.png b/doc/ci/introduction/img/pipeline_status.png Binary files differindex 96881f072e1..96881f072e1 100755..100644 --- a/doc/ci/introduction/img/pipeline_status.png +++ b/doc/ci/introduction/img/pipeline_status.png diff --git a/doc/ci/introduction/img/rollback.png b/doc/ci/introduction/img/rollback.png Binary files differindex 38e0552f4f1..38e0552f4f1 100755..100644 --- a/doc/ci/introduction/img/rollback.png +++ b/doc/ci/introduction/img/rollback.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index 1b423a4696e..6055d8c282a 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -127,7 +127,7 @@ displayed by GitLab:  At the end, if anything goes wrong, you can easily -[roll back](../environments.md#rolling-back-changes) all the changes: +[roll back](../environments.md#retrying-and-rolling-back) all the changes:  @@ -154,7 +154,7 @@ Once you're happy with your implementation: - GitLab CI/CD deploys your changes automatically to a production environment. - And finally, you and your team can easily roll it back if something goes wrong. -<img src="img/gitlab_workflow_example.png" alt="GitLab workflow example" class="image-noshadow"> +<img src="img/gitlab_workflow_example_11_9.png" alt="GitLab workflow example" class="image-noshadow"> GitLab CI/CD is capable of a doing a lot more, but this workflow exemplifies GitLab's ability to track the entire process, diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md new file mode 100644 index 00000000000..cfe638c0a22 --- /dev/null +++ b/doc/ci/large_repositories/index.md @@ -0,0 +1,235 @@ +# Optimizing GitLab for large repositories + +Large repositories consisting of more than 50k files in a worktree +often require special consideration because of +the time required to clone and check out. + +GitLab and GitLab Runner handle this scenario well +but require optimized configuration to efficiently perform its +set of operations. + +The general guidelines for handling big repositories are simple. +Each guideline is described in more detail in the sections below: + +- Always fetch incrementally. Do not clone in a way that results in recreating all of the worktree. +- Always use shallow clone to reduce data transfer. Be aware that this puts more burden + on GitLab instance due to higher CPU impact. +- Control the clone directory if you heavily use a fork-based workflow. +- Optimize `git clean` flags to ensure that you remove or keep data that might affect or speed-up your build. + +## Shallow cloning + +> Introduced in GitLab Runner 8.9. + +GitLab and GitLab Runner always perform a full clone by default. +While it means that all changes from GitLab are received, +it often results in receiving extra commit logs. + +Ideally, you should always use `GIT_DEPTH` with a small number +like 10. This will instruct GitLab Runner to perform shallow clones. +Shallow clones makes Git request only the latest set of changes for a given branch, +up to desired number of commits as defined by the `GIT_DEPTH` variable. + +This significantly speeds up fetching of changes from Git repositories, +especially if the repository has a very long backlog consisting of number +of big files as we effectively reduce amount of data transfer. + +The following example makes GitLab Runner shallow clone to fetch only a given branch, +it does not fetch any other branches nor tags. + +```yaml +variables: + GIT_DEPTH: 10 + +test: + script: + - ls -al +``` + +## Git strategy + +> Introduced in GitLab Runner 8.9. + +By default, GitLab is configured to always prefer the `GIT_STRATEGY: fetch` strategy. +The `GIT_STRATEGY: fetch` strategy will re-use existing worktrees if found +on disk. This is different to the `GIT_STRATEGY: clone` strategy +as in case of clones, if a worktree is found, it is removed before clone. + +Usage of `fetch` is preferred because it reduces the amount of data to transfer and +does not really impact the operations that you might do on a repository from CI. + +However, `fetch` does require access to the previous worktree. This works +well when using the `shell` or `docker` executor because these +try to preserve worktrees and try to re-use them by default. + +This does not work today for `kubernetes` executor and has limitations when using +`docker+machine`. `kubernetes` executor today always clones into ephemeral directory. + +GitLab also offers the `GIT_STRATEGY: none` strategy. This disables any `fetch` and `checkout` commands +done by GitLab, requiring you to do them. + +## Git clone path + +> Introduced in GitLab Runner 11.10. + +`GIT_CLONE_PATH` allows you to control where you clone your sources. +This can have implications if you heavily use big repositories with fork workflow. + +Fork workflow from GitLab Runner's perspective is stored as a separate repository +with separate worktree. That means that GitLab Runner cannot optimize the usage +of worktrees and you might have to instruct GitLab Runner to use that. + +In such cases, ideally you want to make the GitLab Runner executor be used only used only +for the given project and not shared across different projects to make this +process more efficient. + +The `GIT_CLONE_PATH` has to be within the `$CI_BUILDS_DIR`. Currently, +it is impossible to pick any path from disk. + +## Git clean flags + +> Introduced in GitLab Runner 11.10. + +`GIT_CLEAN_FLAGS` allows you to control whether or not you require +the `git clean` command to be executed for each CI job. +By default, GitLab ensures that you have your worktree on the given SHA, +and that your repository is clean. + +`GIT_CLEAN_FLAGS` is disabled when set to `none`. On very big repositories, this +might be desired because `git clean` is disk I/O intensive. Controlling that +with `GIT_CLEAN_FLAGS: -ffdx -e .build/`, for example, allows you to control and +disable removal of some directories within the worktree between subsequent runs, +which can speed-up the incremental builds. This has the biggest effect +if you re-use existing machines, and have an existing worktree that you can re-use +for builds. + +For exact parameters accepted by `GIT_CLEAN_FLAGS`, see the documentation +for [git clean](https://git-scm.com/docs/git-clean). The +available parameters are dependent on Git version. + +## Fork-based workflow + +> Introduced in GitLab Runner 11.10. + +Following the guidelines above, lets imagine that we want to: + +- Optimize for a big project (more than 50k files in directory). +- Use forks-based workflow for contributing. +- Reuse existing worktrees. Have preconfigured runners that are pre-cloned with repositories. +- Runner assigned only to project and all forks. + +Lets consider the following two examples, one using `shell` executor and +other using `docker` executor. + +### `shell` executor example + +Lets assume that you have the following [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "shell" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.custom_build_dir] + enabled = true +``` + +This `config.toml`: + +- Uses the `shell` executor, +- Specifies a custom `/builds` directory where all clones will be stored. +- Enables the ability to specify `GIT_CLONE_PATH`, +- Runs at most 4 jobs at once. + +### `docker` executor example + +Lets assume that you have the following [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` + +This `config.toml`: + +- Uses the `docker` executor, +- Specifies a custom `/builds` directory on disk where all clones will be stored. + We host mount the `/builds` directory to make it reusable between subsequent runs + and be allowed to override the cloning strategy. +- Doesn't enable the ability to specify `GIT_CLONE_PATH` as it is enabled by default. +- Runs at most 4 jobs at once. + +### Our `.gitlab-ci.yml` + +Once we have the executor configured, we need to fine tune our `.gitlab-ci.yml`. + +Our pipeline will be most performant if we use the following `.gitlab-ci.yml`: + +```yaml +variables: + GIT_DEPTH: 10 + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME + +build: + script: ls -al +``` + +The above configures a: + +- Shallow clone of 10, to speed up subsequent `git fetch` commands. +- Custom clone path to make it possible to re-use worktrees between parent project and all forks + because we use the same clone path for all forks. + +Why use `$CI_CONCURRENT_ID`? The main reason is to ensure that worktrees used are not conflicting +between projects. The `$CI_CONCURRENT_ID` represents a unique identifier within the given executor, +so as long as we use it to construct the path, it is guaranteed that this directory will not conflict +with other concurrent jobs running. + +### Store custom clone options in `config.toml` + +Ideally, all job-related configuration should be stored in `.gitlab-ci.yml`. +However, sometimes it is desirable to make these schemes part of Runner configuration. + +In the above example of Forks, making this configuration discoverable for users may be preferred, +but this brings administrative overhead as the `.gitlab-ci.yml` needs to be updated for each branch. +In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it +a configuration of Runner. + +We can extend our [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) +with the following specification that will be used by Runner if `.gitlab-ci.yml` will not override it: + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" + + environment = [ + "GIT_DEPTH=10", + "GIT_CLONE_PATH=$CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME" + ] + + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` + +This makes the cloning configuration to be part of given Runner, +and does not require us to update each `.gitlab-ci.yml`. diff --git a/doc/ci/merge_request_pipelines/img/merge_request.png b/doc/ci/merge_request_pipelines/img/merge_request.png Binary files differindex cf9c628e9a0..d03fdc6a885 100644 --- a/doc/ci/merge_request_pipelines/img/merge_request.png +++ b/doc/ci/merge_request_pipelines/img/merge_request.png diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png b/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png Binary files differnew file mode 100644 index 00000000000..58d5581f628 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png b/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png Binary files differnew file mode 100644 index 00000000000..0a84e61d284 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png diff --git a/doc/ci/merge_request_pipelines/img/pipeline_detail.png b/doc/ci/merge_request_pipelines/img/pipeline_detail.png Binary files differindex 6094a0975fb..90e7c449a66 100644 --- a/doc/ci/merge_request_pipelines/img/pipeline_detail.png +++ b/doc/ci/merge_request_pipelines/img/pipeline_detail.png diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index e8953d235a7..4f61e97bd8a 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -2,14 +2,16 @@ > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/15310) in GitLab 11.6. -Usually, when you create a new merge request, a pipeline runs on the +Usually, when you create a new merge request, a pipeline runs with the new change and checks if it's qualified to be merged into a target branch. This -pipeline should contain only necessary jobs for checking the new changes. +pipeline should contain only necessary jobs for validating the new changes. For example, unit tests, lint checks, and [Review Apps](../review_apps/index.md) are often used in this cycle. With pipelines for merge requests, you can design a specific pipeline structure -for merge requests. +for when you are running a pipeline in a merge request. This +could be either adding or removing steps in the pipeline, to make sure that +your pipelines are as efficient as possible. ## Configuring pipelines for merge requests @@ -30,9 +32,7 @@ build: stage: build script: ./build only: - - branches - - tags - - merge_requests + - master test: stage: test @@ -43,6 +43,8 @@ test: deploy: stage: deploy script: ./deploy + only: + - master ``` After the merge request is updated with new commits: @@ -50,18 +52,58 @@ After the merge request is updated with new commits: - GitLab detects that changes have occurred and creates a new pipeline for the merge request. - The pipeline fetches the latest code from the source branch and run tests against it. -In the above example, the pipeline contains only `build` and `test` jobs. -Since the `deploy` job doesn't have the `only: merge_requests` parameter, -deployment jobs will not happen in the merge request. +In the above example, the pipeline contains only a `test` job. +Since the `build` and `deploy` jobs don't have the `only: merge_requests` parameter, +they will not run in the merge request. -Pipelines tagged with the **merge request** badge indicate that they were triggered +Pipelines tagged with the **detached** badge indicate that they were triggered when a merge request was created or updated. For example:  -The same tag is shown on the pipeline's details: +## Combined ref pipelines **[PREMIUM]** + +> [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. + +It's possible for your source and target branches to diverge, which can result +in the scenario that source branch's pipeline was green, the target's pipeline was green, +but the combined output fails. By having your merge request pipeline automatically +create a new ref that contains the merge result of the source and target branch +(then running a pipeline on that ref), we can better test that the combined result +is also valid. + +From GitLab 11.10, pipelines for merge requests run by default +on this merged result. That is, where the source and target branches are combined into a +new ref and a pipeline for this ref validates the result prior to merging. + + + +There are some cases where creating a combined ref is not possible or not wanted. +For example, a source branch that has conflicts with the target branch +or a merge request that is still in WIP status. In this case, the merge request pipeline falls back to a "detached" state +and runs on the source branch ref as if it was a regular pipeline. + +The detached state serves to warn you that you are working in a situation +subjected to merge problems, and helps to highlight that you should +get out of WIP status or resolve merge conflicts as soon as possible. - +### Enabling combined ref pipelines + +This feature disabled by default until we resolve issues with [contention handling](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186). It can be enabled at the project level: + +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. Check **Merge pipelines will try to validate the post-merge result prior to merging**. +1. Click **Save changes** button. + + + +### Combined ref pipeline's limitations + +- This feature requires [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. +- This feature requires [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. +- After the merge request pipeline succeeds, if the target branch has moved forward, the result of the pipeline is stale and must be retried. In busy repos, this can become a problem as it is highly probable that the target branch will have moved ahead. Improvements are [planned](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) for future versions of GitLab. +- Forking/cross-repo workflows are not currently supported. To follow progress, see [#9713](https://gitlab.com/gitlab-org/gitlab-ee/issues/9713). +- This feature is not available for [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). ## Excluding certain jobs @@ -138,3 +180,12 @@ External users could steal secret variables from the parent project by modifying We're discussing a secure solution of running pipelines for merge requests that submitted from forked projects, see [the issue about the permission extension](https://gitlab.com/gitlab-org/gitlab-ce/issues/23902). + +## Additional predefined variables + +By using pipelines for merge requests, GitLab exposes additional predefined variables to the pipeline jobs. +Those variables contain information of the associated merge request, so that it's useful +to integrate your job with [GitLab Merge Request API](../../api/merge_requests.md). + +You can find the list of avilable variables in [the reference sheet](../variables/predefined_variables.md). +The variable names begin with the `CI_MERGE_REQUEST_` prefix. diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index c509c341d1e..2ffa3d4edc7 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -45,12 +45,12 @@ Pipeline graphs can be displayed in two different ways, depending on the page yo access the graph from. NOTE: **Note:** -GitLab capitalizes the stages' names when shown in the [pipeline graphs](#pipeline-graphs). +GitLab capitalizes the stages' names when shown in the pipeline graphs (below). ### Regular pipeline graphs Regular pipeline graphs show the names of the jobs of each stage. Regular pipeline graphs can -be found when you are on a [single pipeline page](#seeing-pipeline-status). For example: +be found when you are on a [single pipeline page](#accessing-pipelines). For example:  @@ -295,7 +295,7 @@ You can do this straight from the pipeline graph. Just click on the play button to execute that particular job. For example, your pipeline start automatically, but require manual action to -[deploy to production](environments.md#manually-deploying-to-environments). In the example below, the `production` +[deploy to production](environments.md#configuring-manual-deployments). In the example below, the `production` stage has a job with a manual action.  @@ -313,7 +313,7 @@ For example, if you start rolling out new code and: - Users do not experience trouble, GitLab can automatically complete the deployment from 0% to 100%. - Users experience trouble with the new code, you can stop the timed incremental rollout by canceling the pipeline - and [rolling](environments.md#rolling-back-changes) back to the last stable version. + and [rolling](environments.md#retrying-and-rolling-back) back to the last stable version.  diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 9684cb6ed98..65886400c64 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -105,7 +105,7 @@ Jobs are used to create jobs, which are then picked by What is important is that each job is run independently from each other. If you want to check whether the `.gitlab-ci.yml` of your project is valid, there is a -Lint tool under the page `/ci/lint` of your project namespace. You can also find +Lint tool under the page `/-/ci/lint` of your project namespace. You can also find a "CI Lint" button to go to this page under **CI/CD ➔ Pipelines** and **Pipelines ➔ Jobs** in your project. diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 9dbcf9d1155..2fc1e14f02e 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -33,7 +33,7 @@ In this example, you can see a branch was: ## How do Review Apps work? -The basis of Review Apps in GitLab is [dynamic environments](../environments.md#dynamic-environments), which allow you to dynamically create a new environment for each branch. +The basis of Review Apps in GitLab is [dynamic environments](../environments.md#configuring-dynamic-environments), which allow you to dynamically create a new environment for each branch. Access to the Review App is made available as a link on the [merge request](../../user/project/merge_requests.md) relevant to the branch. Review Apps enable you to review all changes proposed by the merge request in live environment. @@ -60,14 +60,14 @@ To get a better understanding of Review Apps, review documentation on how enviro 1. Learn about [environments](../environments.md) and their role in the development workflow. 1. Learn about [CI variables](../variables/README.md) and how they can be used in your CI jobs. 1. Explore the [`environment` syntax](../yaml/README.md#environment) as defined in `.gitlab-ci.yml`. This will become a primary reference. -1. Additionally, find out about [manual actions](../environments.md#manually-deploying-to-environments) and how you can use them to deploy to critical environments like production with the push of a button. +1. Additionally, find out about [manual actions](../environments.md#configuring-manual-deployments) and how you can use them to deploy to critical environments like production with the push of a button. 1. Follow the [example tutorials](#examples). These will guide you through setting up infrastructure and using Review Apps. ### Configuring dynamic environments Configuring Review Apps dynamic environments depends on your technology stack and infrastructure. -For more information, see [dynamic environments](../environments.md#dynamic-environments) documentation to understand how to define and create them. +For more information, see [dynamic environments](../environments.md#configuring-dynamic-environments) documentation to understand how to define and create them. ### Creating and destroying Review Apps @@ -81,7 +81,7 @@ The process of adding Review Apps in your workflow is as follows: 1. Set up the infrastructure to host and deploy the Review Apps. 1. [Install](https://docs.gitlab.com/runner/install/) and [configure](https://docs.gitlab.com/runner/commands/) a Runner to do deployment. -1. Set up a job in `.gitlab-ci.yml` that uses the predefined [predefined CI environment variable](../variables/README.md) `${CI_COMMIT_REF_NAME}` to create dynamic environments and restrict it to run only on branches. +1. Set up a job in `.gitlab-ci.yml` that uses the [predefined CI environment variable](../variables/README.md) `${CI_COMMIT_REF_NAME}` to create dynamic environments and restrict it to run only on branches. 1. Optionally, set a job that [manually stops](../environments.md#stopping-an-environment) the Review Apps. After adding Review Apps to your workflow, you follow the branched Git flow. That is: diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index ff63d0bd69f..9ed1ec5aa5c 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -49,7 +49,7 @@ to access it. This is where an SSH key pair comes in handy. **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. -1. Create a new [variable](../variables/README.md#variables). +1. Create a new [variable](../variables/README.md#gitlab-cicd-environment-variables). As **Key** enter the name `SSH_PRIVATE_KEY` and in the **Value** field paste the content of your _private_ key that you created earlier. @@ -157,7 +157,7 @@ ssh-keyscan example.com ssh-keyscan 1.2.3.4 ``` -Create a new [variable](../variables/README.md#variables) with +Create a new [variable](../variables/README.md#gitlab-cicd-environment-variables) with `SSH_KNOWN_HOSTS` as "Key", and as a "Value" add the output of `ssh-keyscan`. NOTE: **Note:** diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 398b017277f..0e72bdddee7 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -20,7 +20,7 @@ A unique trigger token can be obtained when [adding a new trigger](#adding-a-new DANGER: **Danger:** Passing plain text tokens in public projects is a security issue. Potential attackers can impersonate the user that exposed their trigger token publicly in -their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#variables) +their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#gitlab-cicd-environment-variables) to protect trigger tokens. ## Adding a new trigger diff --git a/doc/ci/triggers/img/trigger_variables.png b/doc/ci/triggers/img/trigger_variables.png Binary files differindex d273b1fe3a2..910622d74cc 100644 --- a/doc/ci/triggers/img/trigger_variables.png +++ b/doc/ci/triggers/img/trigger_variables.png diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 4d15d58cb40..61d1a904f76 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -2,174 +2,242 @@ table_display_block: true --- -# GitLab CI/CD Variables +# GitLab CI/CD environment variables -When receiving a job from GitLab CI, the [Runner](https://docs.gitlab.com/runner/) prepares the build environment. -It starts by setting a list of: +After a brief overview over the use of environment +variables, this document teaches you how to use GitLab CI/CD's +variables, presents the full reference for predefined variables, +and dives into more advanced applications. -- [Predefined environment variables](#predefined-environment-variables). -- Other variables. +## Overview -## Priority of variables +An environment variable is a dynamic-named value that can +affect the way running processes will behave on an operating +system. -Variables of different types can take precedence over other variables, depending on where they are defined. +They are part of the environment in which a process runs. +For example, a running process can query the value of the +`TEMP` environment variable to discover a suitable location +to store temporary files, or to define a `URL` for a database +that can be reused in different scripts. -The order of precedence for variables is (from highest to lowest): +Variables are useful for customizing your jobs in GitLab +CI/CD's pipelines. Using variables means no hardcoded values. -1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables) or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#making-use-of-scheduled-pipeline-variables). -1. Project-level [variables](#variables) or [protected variables](#protected-variables). -1. Group-level [variables](#variables) or [protected variables](#protected-variables). -1. YAML-defined [job-level variables](../yaml/README.md#variables). -1. YAML-defined [global variables](../yaml/README.md#variables). -1. [Deployment variables](#deployment-variables). -1. [Predefined environment variables](#predefined-environment-variables). +### Predefined environment variables -For example, you define: +GitLab CI/CD has a default set of +[predefined variables](predefined_variables.md) +which can be used without any specification needed. +You can call issues numbers, user names, branch names, +pipeline and commit IDs, and much more. -- `API_TOKEN=secure` as a project variable. -- `API_TOKEN=yaml` in your `.gitlab-ci.yml`. +Predefined environment variables are the ones that GitLab +provides out of the box for the local environment of the Runner. -`API_TOKEN` will take the value `secure` as the project variables take precedence over those defined -in `.gitlab-ci.yml`. +GitLab reads the .gitlab-ci.yml file, sends the information +to the Runner (which runs the script commands), under which +the variables are exposed. -## Unsupported variables +For example, two jobs under the same pipeline can share the same +`CI_PIPELINE_ID` variable, but each one has its own `CI_JOB_ID` +variable. -There are cases where some variables cannot be used in the context of a -`.gitlab-ci.yml` definition (for example under `script`). Read more -about which variables are [not supported](where_variables_can_be_used.md). +### Custom environment variables -## Predefined environment variables +When your use case requires a specific variable, you can +[set them up easily from the UI](#creating-a-custom-environment-variable) +or directly in the `.gitlab-ci.yml` file and reuse them as you wish. -Some of the predefined environment variables are available only if a minimum -version of [GitLab Runner](https://docs.gitlab.com/runner/) is used. Consult the table below to find the -version of Runner required. +That can be very powerful as it can be used for scripting without +the need to specify the value itself. -NOTE: **Note:** -Starting with GitLab 9.0, we have deprecated some variables. Read the -[9.0 Renaming](#gitlab-90-renaming) section to find out their replacements. **You are -strongly advised to use the new variables as we will remove the old ones in -future GitLab releases.** - -| Variable | GitLab | Runner | Description | -|-------------------------------------------|--------|--------|-------------| -| **ARTIFACT_DOWNLOAD_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to download artifacts running a job | -| **CHAT_INPUT** | 10.6 | all | Additional arguments passed in the [ChatOps](../chatops/README.md) command | -| **CHAT_CHANNEL** | 10.6 | all | Source chat channel which triggered the [ChatOps](../chatops/README.md) command | -| **CI** | all | 0.4 | Mark that job is executed in CI environment | -| **CI_COMMIT_BEFORE_SHA** | 11.2 | all | The previous latest commit present on a branch before a push request. | -| **CI_COMMIT_DESCRIPTION** | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | -| **CI_COMMIT_MESSAGE** | 10.8 | all | The full commit message. | -| **CI_COMMIT_REF_NAME** | 9.0 | all | The branch or tag name for which project is built | -| **CI_COMMIT_REF_SLUG** | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | -| **CI_COMMIT_SHA** | 9.0 | all | The commit revision for which project is built | -| **CI_COMMIT_SHORT_SHA** | 11.7 | all | The first eight characters of `CI_COMMIT_SHA` | -| **CI_COMMIT_TAG** | 9.0 | 0.5 | The commit tag name. Present only when building tags. | -| **CI_COMMIT_TITLE** | 10.8 | all | The title of the commit - the full first line of the message | -| **CI_CONFIG_PATH** | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | -| **CI_DEBUG_TRACE** | all | 1.7 | Whether [debug tracing](#debug-tracing) is enabled | -| **CI_DEPLOY_PASSWORD** | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| -| **CI_DEPLOY_USER** | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| -| **CI_DISPOSABLE_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | -| **CI_ENVIRONMENT_NAME** | 8.15 | all | The name of the environment for this job. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | -| **CI_ENVIRONMENT_SLUG** | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | -| **CI_ENVIRONMENT_URL** | 9.3 | all | The URL of the environment for this job. Only present if [`environment:url`](../yaml/README.md#environmenturl) is set. | -| **CI_JOB_ID** | 9.0 | all | The unique id of the current job that GitLab CI uses internally | -| **CI_JOB_MANUAL** | 8.12 | all | The flag to indicate that job was manually started | -| **CI_JOB_NAME** | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | -| **CI_JOB_STAGE** | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| **CI_JOB_TOKEN** | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry][registry] and downloading [dependent repositories][dependent-repositories] | -| **CI_JOB_URL** | 11.1 | 0.5 | Job details URL | -| **CI_MERGE_REQUEST_ID** | 11.6 | all | The ID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_IID** | 11.6 | all | The IID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_PROJECT_ID** | 11.6 | all | The ID of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_PROJECT_PATH** | 11.6 | all | The path of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) (e.g. `namespace/awesome-project`) | -| **CI_MERGE_REQUEST_PROJECT_URL** | 11.6 | all | The URL of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) (e.g. `http://192.168.10.15:3000/namespace/awesome-project`) | -| **CI_MERGE_REQUEST_REF_PATH** | 11.6 | all | The ref path of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). (e.g. `refs/merge-requests/1/head`) | -| **CI_MERGE_REQUEST_SOURCE_BRANCH_NAME** | 11.6 | all | The source branch name of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_SOURCE_BRANCH_SHA** | 11.9 | all | The HEAD sha of the source branch of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_SOURCE_PROJECT_ID** | 11.6 | all | The ID of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_SOURCE_PROJECT_PATH** | 11.6 | all | The path of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_SOURCE_PROJECT_URL** | 11.6 | all | The URL of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_TARGET_BRANCH_NAME** | 11.6 | all | The target branch name of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_TARGET_BRANCH_SHA** | 11.9 | all | The HEAD sha of the target branch of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_TITLE** | 11.9 | all | The title of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_ASSIGNEES** | 11.9 | all | Comma-separated list of usernames of assignees for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). [Multiple assignees for merge requests](https://gitlab.com/gitlab-org/gitlab-ee/issues/2004) is scheduled for a future release | -| **CI_MERGE_REQUEST_MILESTONE** | 11.9 | all | The milestone title of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_LABELS** | 11.9 | all | Comma-separated label names of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | -| **CI_NODE_INDEX** | 11.5 | all | Index of the job in the job set. If the job is not parallelized, this variable is not set. | -| **CI_NODE_TOTAL** | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | -| **CI_API_V4_URL** | 11.7 | all | The GitLab API v4 root URL | -| **CI_PAGES_DOMAIN** | 11.8 | all | The configured domain that hosts GitLab Pages. | -| **CI_PAGES_URL** | 11.8 | all | URL to GitLab Pages-built pages. Always belongs to a subdomain of `CI_PAGES_DOMAIN`. | -| **CI_PIPELINE_ID** | 8.10 | all | The unique id of the current pipeline that GitLab CI uses internally | -| **CI_PIPELINE_IID** | 11.0 | all | The unique id of the current pipeline scoped to project | -| **CI_PIPELINE_SOURCE** | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | -| **CI_PIPELINE_TRIGGERED** | all | all | The flag to indicate that job was [triggered] | -| **CI_PIPELINE_URL** | 11.1 | 0.5 | Pipeline details URL | -| **CI_PROJECT_DIR** | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | -| **CI_PROJECT_ID** | all | all | The unique id of the current project that GitLab CI uses internally | -| **CI_PROJECT_NAME** | 8.10 | 0.5 | The project name that is currently being built (actually it is project folder name) | -| **CI_PROJECT_NAMESPACE** | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | -| **CI_PROJECT_PATH** | 8.10 | 0.5 | The namespace with project name | -| **CI_PROJECT_PATH_SLUG** | 9.3 | all | `$CI_PROJECT_PATH` lowercased and with everything except `0-9` and `a-z` replaced with `-`. Use in URLs and domain names. | -| **CI_PROJECT_URL** | 8.10 | 0.5 | The HTTP(S) address to access project | -| **CI_PROJECT_VISIBILITY** | 10.3 | all | The project visibility (internal, private, public) | -| **CI_REGISTRY** | 8.10 | 0.5 | If the Container Registry is enabled it returns the address of GitLab's Container Registry | -| **CI_REGISTRY_IMAGE** | 8.10 | 0.5 | If the Container Registry is enabled for the project it returns the address of the registry tied to the specific project | -| **CI_REGISTRY_PASSWORD** | 9.0 | all | The password to use to push containers to the GitLab Container Registry | -| **CI_REGISTRY_USER** | 9.0 | all | The username to use to push containers to the GitLab Container Registry | -| **CI_REPOSITORY_URL** | 9.0 | all | The URL to clone the Git repository | -| **CI_RUNNER_DESCRIPTION** | 8.10 | 0.5 | The description of the runner as saved in GitLab | -| **CI_RUNNER_EXECUTABLE_ARCH** | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | -| **CI_RUNNER_ID** | 8.10 | 0.5 | The unique id of runner being used | -| **CI_RUNNER_REVISION** | all | 10.6 | GitLab Runner revision that is executing the current job | -| **CI_RUNNER_TAGS** | 8.10 | 0.5 | The defined runner tags | -| **CI_RUNNER_VERSION** | all | 10.6 | GitLab Runner version that is executing the current job | -| **CI_SERVER** | all | all | Mark that job is executed in CI environment | -| **CI_SERVER_NAME** | all | all | The name of CI server that is used to coordinate jobs | -| **CI_SERVER_REVISION** | all | all | GitLab revision that is used to schedule jobs | -| **CI_SERVER_VERSION** | all | all | GitLab version that is used to schedule jobs | -| **CI_SERVER_VERSION_MAJOR** | 11.4 | all | GitLab version major component | -| **CI_SERVER_VERSION_MINOR** | 11.4 | all | GitLab version minor component | -| **CI_SERVER_VERSION_PATCH** | 11.4 | all | GitLab version patch component | -| **CI_SHARED_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a shared environment (something that is persisted across CI invocations like `shell` or `ssh` executor). If the environment is shared, it is set to true, otherwise it is not defined at all. | -| **GET_SOURCES_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to fetch sources running a job | -| **GITLAB_CI** | all | all | Mark that job is executed in GitLab CI environment | -| **GITLAB_USER_EMAIL** | 8.12 | all | The email of the user who started the job | -| **GITLAB_USER_ID** | 8.12 | all | The id of the user who started the job | -| **GITLAB_USER_LOGIN** | 10.0 | all | The login username of the user who started the job | -| **GITLAB_USER_NAME** | 10.0 | all | The real name of the user who started the job | -| **RESTORE_CACHE_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to restore the cache running a job | - -## GitLab 9.0 renaming - -To follow conventions of naming across GitLab, and to further move away from the -`build` term and toward `job` CI variables have been renamed for the 9.0 -release. +## Getting started + +To get started with environment variables in the scope of GitLab +CI/CD, let's go over a few examples. + +### Using predefined environment variables + +To get started, choose one of the existing +[predefined variables](predefined_variables.md) +to be output by the Runner. For example, let's say that you want +a given job you're running through your script to output the +stage that job is running for. In your `.gitlab-ci.yml` file, +call the variable from your script according to the [syntaxes](#syntax-of-environment-variables-in-job-scripts) available. To +output the job stage, use the predefined variable `CI_JOB_STAGE`: + +```yaml +test_variable: + stage: test + script: + - echo $CI_JOB_STAGE +``` + +For this case, the Runner will output the `stage` for the +job `test_variable`, which is `test`: + + + +As another example, let's say you're using your own GitLab +instance you want to know what domain your GitLab Pages are +served under. You can easily call it with the predefined +variable `$CI_PAGES_DOMAIN` in your script: + +```yaml +pages: + script: + - ... + - echo $CI_PAGES_DOMAIN +``` + +For GitLab.com users, the output will be `gitlab.io`. For your +private instance, the output will be whatever your sysadmin has +defined. + +### Creating a custom environment variable + +Assume you have something you want to repeat through your scripts +in GitLab CI/CD's configuration file. To keep this example simple, +let's say you want to output `HELLO WORLD` for a `TEST` variable. + +You can either set the variable directly in the `.gitlab-ci.yml` +file or through the UI. + +#### Via [`.gitlab-ci.yml`](../yaml/README.md#variables) + +```yaml +variables: + TEST: "HELLO WORLD" +``` + +For a deeper look into them, see [`.gitlab-ci.yml` defined variables](#gitlab-ciyml-defined-variables). + +#### Via the UI + +From the UI, navigate to your project's **Settings > CI/CD** and +expand **Environment variables**. Create a new variable by naming +it in the field **Input variable key**, and define its value in the +**Input variable value** field: + + + +Once you've set the variables, call them from the `.gitlab-ci.yml` file: + +```yaml +test_variable: + stage: test + script: + - echo $CI_JOB_STAGE # calls a predefined variable + - echo $TEST # calls a custom variable +``` + +The output will be: + + + +CAUTION: **Important:** +Be aware that variables are not masked, and their values can be shown +in the job logs if explicitly asked to do so. If your project is public or +internal, you can set the pipelines private from your [project's Pipelines +settings](../../user/project/pipelines/settings.md#visibility-of-pipelines). +Follow the discussion in issue [#13784][ce-13784] for masking the variables. + +### Syntax of environment variables in job scripts + +All variables are set as environment variables in the build environment, and +they are accessible with normal methods that are used to access such variables. +In most cases `bash` or `sh` is used to execute the job script. + +To access environment variables, use the syntax for your Runner's [shell][shellexecutors]. + +| Shell | Usage | +|----------------------|-----------------| +| bash/sh | `$variable` | +| windows batch | `%variable%` | +| PowerShell | `$env:variable` | + +To access environment variables in bash, prefix the variable name with (`$`): + +```yaml +job_name: + script: + - echo $CI_JOB_ID +``` + +To access environment variables in **Windows Batch**, surround the variable +with (`%`): + +```yaml +job_name: + script: + - echo %CI_JOB_ID% +``` + +To access environment variables in a **Windows PowerShell** environment, prefix +the variable name with (`$env:`): + +```yaml +job_name: + script: + - echo $env:CI_JOB_ID +``` + +You can also list all environment variables with the `export` command, +but be aware that this will also expose the values of all the variables +you set, in the job log: + +```yaml +job_name: + script: + - export +``` + +Example values: + +```bash +export CI_JOB_ID="50" +export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" +export CI_COMMIT_SHORT_SHA="1ecfd275" +export CI_COMMIT_REF_NAME="master" +export CI_REPOSITORY_URL="https://gitlab-ci-token:abcde-1234ABCD5678ef@example.com/gitlab-org/gitlab-ce.git" +export CI_COMMIT_TAG="1.0.0" +export CI_JOB_NAME="spec:other" +export CI_JOB_STAGE="test" +export CI_JOB_MANUAL="true" +export CI_JOB_TRIGGERED="true" +export CI_JOB_TOKEN="abcde-1234ABCD5678ef" +export CI_PIPELINE_ID="1000" +export CI_PIPELINE_IID="10" +export CI_PAGES_DOMAIN="gitlab.io" +export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab-ce" +export CI_PROJECT_ID="34" +export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce" +export CI_PROJECT_NAME="gitlab-ce" +export CI_PROJECT_NAMESPACE="gitlab-org" +export CI_PROJECT_PATH="gitlab-org/gitlab-ce" +export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-ce" +export CI_REGISTRY="registry.example.com" +export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-ce" +export CI_RUNNER_ID="10" +export CI_RUNNER_DESCRIPTION="my runner" +export CI_RUNNER_TAGS="docker, linux" +export CI_SERVER="yes" +export CI_SERVER_NAME="GitLab" +export CI_SERVER_REVISION="70606bf" +export CI_SERVER_VERSION="8.9.0" +export CI_SERVER_VERSION_MAJOR="8" +export CI_SERVER_VERSION_MINOR="9" +export CI_SERVER_VERSION_PATCH="0" +export GITLAB_USER_ID="42" +export GITLAB_USER_EMAIL="user@example.com" +export CI_REGISTRY_USER="gitlab-ci-token" +export CI_REGISTRY_PASSWORD="longalfanumstring" +``` + +### `.gitlab-ci.yml` defined variables NOTE: **Note:** -Starting with GitLab 9.0, we have deprecated the `$CI_BUILD_*` variables. **You are -strongly advised to use the new variables as we will remove the old ones in -future GitLab releases.** - -| 8.x name | 9.0+ name | -| --------------------- |------------------------ | -| `CI_BUILD_ID` | `CI_JOB_ID` | -| `CI_BUILD_REF` | `CI_COMMIT_SHA` | -| `CI_BUILD_TAG` | `CI_COMMIT_TAG` | -| `CI_BUILD_BEFORE_SHA` | `CI_COMMIT_BEFORE_SHA` | -| `CI_BUILD_REF_NAME` | `CI_COMMIT_REF_NAME` | -| `CI_BUILD_REF_SLUG` | `CI_COMMIT_REF_SLUG` | -| `CI_BUILD_NAME` | `CI_JOB_NAME` | -| `CI_BUILD_STAGE` | `CI_JOB_STAGE` | -| `CI_BUILD_REPO` | `CI_REPOSITORY_URL` | -| `CI_BUILD_TRIGGERED` | `CI_PIPELINE_TRIGGERED` | -| `CI_BUILD_MANUAL` | `CI_JOB_MANUAL` | -| `CI_BUILD_TOKEN` | `CI_JOB_TOKEN` | - -## `.gitlab-ci.yml` defined variables - -NOTE **Note:** This feature requires GitLab Runner 0.5.0 or higher and GitLab 7.14 or higher. GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in the @@ -207,42 +275,60 @@ script: - 'eval $LS_CMD' # will execute 'ls -al $TMP_DIR' ``` -## Variables - -> Group-level variables were introduced in GitLab 9.4. +### Group-level environment variables -CAUTION: **Important:** -Be aware that variables are not masked, and their values can be shown -in the job logs if explicitly asked to do so. If your project is public or -internal, you can set the pipelines private from your [project's Pipelines -settings](../../user/project/pipelines/settings.md#visibility-of-pipelines). -Follow the discussion in issue [#13784][ce-13784] for masking the variables. +> Introduced in GitLab 9.4. -GitLab CI allows you to define per-project or per-group variables -that are set in the pipeline environment. The variables are stored out of +GitLab CI/CD allows you to define per-project or per-group variables +that are set in the pipeline environment. Group-level variables are stored out of the repository (not in `.gitlab-ci.yml`) and are securely passed to GitLab Runner -making them available during a pipeline run. It's the recommended method to +making them available during a pipeline run. It's the **recommended method** to use for storing things like passwords, SSH keys, and credentials. -Project-level variables can be added by: +Group-level variables can be added by: -1. Navigating to your project's **Settings > CI/CD** page. +1. Navigating to your group's **Settings > CI/CD** page. 1. Inputing variable keys and values in the **Environment variables** section. +Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. -Group-level variables can be added by: +Once you set them, they will be available for all subsequent pipelines. -1. Navigating to your group's **Settings > CI/CD** page. -1. Inputing variable keys and values in the **Environment variables** section. Any variables of - [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. +## Priority of environment variables + +Variables of different types can take precedence over other +variables, depending on where they are defined. + +The order of precedence for variables is (from highest to lowest): + +1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables) or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#using-variables). +1. Project-level [variables](#creating-a-custom-environment-variable) or [protected variables](#protected-environment-variables). +1. Group-level [variables](#group-level-environment-variables) or [protected variables](#protected-environment-variables). +1. YAML-defined [job-level variables](../yaml/README.md#variables). +1. YAML-defined [global variables](../yaml/README.md#variables). +1. [Deployment variables](#deployment-environment-variables). +1. [Predefined environment variables](predefined_variables.md). -Once you set them, they will be available for all subsequent pipelines. You can also -[protect your variables](#protected-variables). +For example, if you define: -### Protected variables +- `API_TOKEN=secure` as a project variable. +- `API_TOKEN=yaml` in your `.gitlab-ci.yml`. + +`API_TOKEN` will take the value `secure` as the project +variables take precedence over those defined in `.gitlab-ci.yml`. + +## Unsupported variables + +There are cases where some variables cannot be used in the context of a +`.gitlab-ci.yml` definition (for example under `script`). Read more +about which variables are [not supported](where_variables_can_be_used.md). + +## Advanced use + +### Protected environment variables > Introduced in GitLab 9.3. -Variables could be protected. Whenever a variable is +Variables can be protected. Whenever a variable is protected, it would only be securely passed to pipelines running on the [protected branches] or [protected tags]. The other pipelines would not get any protected variables. @@ -253,13 +339,7 @@ Protected variables can be added by going to your project's Once you set them, they will be available for all subsequent pipelines. -### Manually-specified variables - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/44059) in GitLab 10.8. - -Variables can be specified for a single pipeline run when a [manual pipeline](../pipelines.md#manually-executing-pipelines) is created. - -## Deployment variables +### Deployment environment variables > Introduced in GitLab 8.15. @@ -272,7 +352,7 @@ the project services that you are using to learn which variables they define. An example project service that defines deployment variables is the [Kubernetes integration](../../user/project/clusters/index.md#deployment-variables). -## Auto DevOps application variables +### Auto DevOps environment variables > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/49056) in GitLab 11.7. @@ -289,6 +369,112 @@ CAUTION: **Caution:** Variables with multiline values are not currently supported due to limitations with the current Auto DevOps scripting environment. +### Environment variables triggered manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/44059) in GitLab 10.8. + +[Manually triggered pipelines](../pipelines.md#manually-executing-pipelines) allow you to override the value of a current variable. + +For instance, suppose you added a +[custom variable `$TEST`](#creating-a-custom-environment-variable) +as exemplified above and you want to override it in a manual pipeline. +Navigate to your project's **CI/CD > Pipelines** and click **Run pipeline**. +Choose the branch you want to run the pipeline for, then add a new variable +pair through the UI: + + + +The Runner will override the value previously set and use the custom +value you set for this specific pipeline: + + + +## Environment variables expressions + +> Introduced in GitLab 10.7. + +It is possible to use variables expressions with only / except policies in +`.gitlab-ci.yml`. By using this approach you can limit what jobs are going to +be created within a pipeline after pushing a code to GitLab. + +This is particularly useful in combination with variables and triggered +pipeline variables. + +```yaml +deploy: + script: cap staging deploy + environment: staging + only: + variables: + - $RELEASE == "staging" + - $STAGING +``` + +Each expression provided is going to be evaluated before creating a pipeline. + +If any of the conditions in `variables` evaluates to truth when using `only`, +a new job is going to be created. If any of the expressions evaluates to truth +when `except` is being used, a job is not going to be created. + +This follows usual rules for [`only` / `except` policies](../yaml/README.md#onlyexcept-advanced). + +### Supported syntax + +Below you can find supported syntax reference: + +1. Equality matching using a string + + > Example: `$VARIABLE == "some value"` + + You can use equality operator `==` to compare a variable content to a + string. We support both, double quotes and single quotes to define a string + value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` + are supported. `"some value" == $VARIABLE` is correct too. + +1. Checking for an undefined value + + > Example: `$VARIABLE == null` + + It sometimes happens that you want to check whether a variable is defined + or not. To do that, you can compare a variable to `null` keyword, like + `$VARIABLE == null`. This expression is going to evaluate to truth if + variable is not defined. + +1. Checking for an empty variable + + > Example: `$VARIABLE == ""` + + If you want to check whether a variable is defined, but is empty, you can + simply compare it against an empty string, like `$VAR == ''`. + +1. Comparing two variables + + > Example: `$VARIABLE_1 == $VARIABLE_2` + + It is possible to compare two variables. This is going to compare values + of these variables. + +1. Variable presence check + + > Example: `$STAGING` + + If you only want to create a job when there is some variable present, + which means that it is defined and non-empty, you can simply use + variable name as an expression, like `$STAGING`. If `$STAGING` variable + is defined, and is non empty, expression will evaluate to truth. + `$STAGING` value needs to a string, with length higher than zero. + Variable that contains only whitespace characters is not an empty variable. + +1. Pattern matching _(added in 11.0)_ + + > Example: `$VARIABLE =~ /^content.*/` + + It is possible perform pattern matching against a variable and regular + expression. Expression like this evaluates to truth if matches are found. + + Pattern matching is case-sensitive by default. Use `i` flag modifier, like + `/pattern/i` to make a pattern case-insensitive. + ## Debug tracing > Introduced in GitLab Runner 1.7. @@ -311,7 +497,7 @@ variables that were set, etc. Before enabling this, you should ensure jobs are visible to [team members only](../../user/permissions.md#project-features). You should -also [erase](../pipelines.md#seeing-job-status) all generated job traces +also [erase](../pipelines.md#accessing-individual-jobs) all generated job traces before making them visible again. To enable debug traces, set the `CI_DEBUG_TRACE` variable to `true`: @@ -353,6 +539,8 @@ $'\''git'\'' "checkout" "-f" "-q" "dd648b2e48ce6518303b0bb580b2ee32fadaf045" Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-machine-1480971377-317a7d0f-digital-ocean-4gb... ++ export CI=true ++ CI=true +++ export CI_API_V4_API_URL=https://example.com:3000/api/v4 +++ CI_API_V4_API_URL=https://example.com:3000/api/v4 ++ export CI_DEBUG_TRACE=false ++ CI_DEBUG_TRACE=false ++ export CI_COMMIT_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045 @@ -391,6 +579,8 @@ Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-mach ++ GITLAB_CI=true ++ export CI=true ++ CI=true +++ export CI_API_V4_API_URL=https://example.com:3000/api/v4 +++ CI_API_V4_API_URL=https://example.com:3000/api/v4 ++ export GITLAB_CI=true ++ GITLAB_CI=true ++ export CI_JOB_ID=7046507 @@ -454,184 +644,6 @@ MIIFQzCCBCugAwIBAgIRAL/ElDjuf15xwja1ZnCocWAwDQYJKoZIhvcNAQELBQAw' ... ``` -## Using the CI variables in your job scripts - -All variables are set as environment variables in the build environment, and -they are accessible with normal methods that are used to access such variables. -In most cases `bash` or `sh` is used to execute the job script. - -To access environment variables, use the syntax for your Runner's [shell][shellexecutors]. - -| Shell | Usage | -|----------------------|-----------------| -| bash/sh | `$variable` | -| windows batch | `%variable%` | -| PowerShell | `$env:variable` | - -To access environment variables in bash, prefix the variable name with (`$`): - -```yaml -job_name: - script: - - echo $CI_JOB_ID -``` - -To access environment variables in **Windows Batch**, surround the variable -with (`%`): - -```yaml -job_name: - script: - - echo %CI_JOB_ID% -``` - -To access environment variables in a **Windows PowerShell** environment, prefix -the variable name with (`$env:`): - -```yaml -job_name: - script: - - echo $env:CI_JOB_ID -``` - -You can also list all environment variables with the `export` command, -but be aware that this will also expose the values of all the variables -you set, in the job log: - -```yaml -job_name: - script: - - export -``` - -Example values: - -```bash -export CI_JOB_ID="50" -export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" -export CI_COMMIT_SHORT_SHA="1ecfd275" -export CI_COMMIT_REF_NAME="master" -export CI_REPOSITORY_URL="https://gitlab-ci-token:abcde-1234ABCD5678ef@example.com/gitlab-org/gitlab-ce.git" -export CI_COMMIT_TAG="1.0.0" -export CI_JOB_NAME="spec:other" -export CI_JOB_STAGE="test" -export CI_JOB_MANUAL="true" -export CI_JOB_TRIGGERED="true" -export CI_JOB_TOKEN="abcde-1234ABCD5678ef" -export CI_PIPELINE_ID="1000" -export CI_PIPELINE_IID="10" -export CI_PAGES_DOMAIN="gitlab.io" -export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab-ce" -export CI_PROJECT_ID="34" -export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce" -export CI_PROJECT_NAME="gitlab-ce" -export CI_PROJECT_NAMESPACE="gitlab-org" -export CI_PROJECT_PATH="gitlab-org/gitlab-ce" -export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-ce" -export CI_REGISTRY="registry.example.com" -export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-ce" -export CI_RUNNER_ID="10" -export CI_RUNNER_DESCRIPTION="my runner" -export CI_RUNNER_TAGS="docker, linux" -export CI_SERVER="yes" -export CI_SERVER_NAME="GitLab" -export CI_SERVER_REVISION="70606bf" -export CI_SERVER_VERSION="8.9.0" -export CI_SERVER_VERSION_MAJOR="8" -export CI_SERVER_VERSION_MINOR="9" -export CI_SERVER_VERSION_PATCH="0" -export GITLAB_USER_ID="42" -export GITLAB_USER_EMAIL="user@example.com" -export CI_REGISTRY_USER="gitlab-ci-token" -export CI_REGISTRY_PASSWORD="longalfanumstring" -``` - -## Variables expressions - -> Introduced in GitLab 10.7. - -It is possible to use variables expressions with only / except policies in -`.gitlab-ci.yml`. By using this approach you can limit what jobs are going to -be created within a pipeline after pushing a code to GitLab. - -This is particularly useful in combination with variables and triggered -pipeline variables. - -```yaml -deploy: - script: cap staging deploy - environment: staging - only: - variables: - - $RELEASE == "staging" - - $STAGING -``` - -Each expression provided is going to be evaluated before creating a pipeline. - -If any of the conditions in `variables` evaluates to truth when using `only`, -a new job is going to be created. If any of the expressions evaluates to truth -when `except` is being used, a job is not going to be created. - -This follows usual rules for [`only` / `except` policies](../yaml/README.md#onlyexcept-advanced). - -### Supported syntax - -Below you can find supported syntax reference: - -1. Equality matching using a string - - > Example: `$VARIABLE == "some value"` - - You can use equality operator `==` to compare a variable content to a - string. We support both, double quotes and single quotes to define a string - value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` - are supported. `"some value" == $VARIABLE` is correct too. - -1. Checking for an undefined value - - > Example: `$VARIABLE == null` - - It sometimes happens that you want to check whether a variable is defined - or not. To do that, you can compare a variable to `null` keyword, like - `$VARIABLE == null`. This expression is going to evaluate to truth if - variable is not defined. - -1. Checking for an empty variable - - > Example: `$VARIABLE == ""` - - If you want to check whether a variable is defined, but is empty, you can - simply compare it against an empty string, like `$VAR == ''`. - -1. Comparing two variables - - > Example: `$VARIABLE_1 == $VARIABLE_2` - - It is possible to compare two variables. This is going to compare values - of these variables. - -1. Variable presence check - - > Example: `$STAGING` - - If you only want to create a job when there is some variable present, - which means that it is defined and non-empty, you can simply use - variable name as an expression, like `$STAGING`. If `$STAGING` variable - is defined, and is non empty, expression will evaluate to truth. - `$STAGING` value needs to a string, with length higher than zero. - Variable that contains only whitespace characters is not an empty variable. - -1. Pattern matching _(added in 11.0)_ - - > Example: `$VARIABLE =~ /^content.*/` - - It is possible perform pattern matching against a variable and regular - expression. Expression like this evaluates to truth if matches are found. - - Pattern matching is case-sensitive by default. Use `i` flag modifier, like - `/pattern/i` to make a pattern case-insensitive. - [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI variables" [eep]: https://about.gitlab.com/pricing/ "Available only in GitLab Premium" [envs]: ../environments.md diff --git a/doc/ci/variables/deprecated_variables.md b/doc/ci/variables/deprecated_variables.md new file mode 100644 index 00000000000..2642c9b0eb4 --- /dev/null +++ b/doc/ci/variables/deprecated_variables.md @@ -0,0 +1,27 @@ +# Deprecated GitLab CI/CD variables + +## GitLab 9.0 renamed variables + +To follow conventions of naming across GitLab, and to further move away from the +`build` term and toward `job`, some [CI/CD environment variables](README.md#predefined-environment-variables) were renamed for GitLab 9.0 +release. + +NOTE: **Note:** +Starting with GitLab 9.0, we have deprecated the `$CI_BUILD_*` variables. **You are +strongly advised to use the new variables as we will remove the old ones in +future GitLab releases.** + +| 8.x name | 9.0+ name | +| --------------------- |------------------------ | +| `CI_BUILD_ID` | `CI_JOB_ID` | +| `CI_BUILD_REF` | `CI_COMMIT_SHA` | +| `CI_BUILD_TAG` | `CI_COMMIT_TAG` | +| `CI_BUILD_BEFORE_SHA` | `CI_COMMIT_BEFORE_SHA` | +| `CI_BUILD_REF_NAME` | `CI_COMMIT_REF_NAME` | +| `CI_BUILD_REF_SLUG` | `CI_COMMIT_REF_SLUG` | +| `CI_BUILD_NAME` | `CI_JOB_NAME` | +| `CI_BUILD_STAGE` | `CI_JOB_STAGE` | +| `CI_BUILD_REPO` | `CI_REPOSITORY_URL` | +| `CI_BUILD_TRIGGERED` | `CI_PIPELINE_TRIGGERED` | +| `CI_BUILD_MANUAL` | `CI_JOB_MANUAL` | +| `CI_BUILD_TOKEN` | `CI_JOB_TOKEN` | diff --git a/doc/ci/variables/img/ci_job_stage_output_example.png b/doc/ci/variables/img/ci_job_stage_output_example.png Binary files differnew file mode 100644 index 00000000000..056238d5693 --- /dev/null +++ b/doc/ci/variables/img/ci_job_stage_output_example.png diff --git a/doc/ci/variables/img/custom_variable_output.png b/doc/ci/variables/img/custom_variable_output.png Binary files differnew file mode 100644 index 00000000000..50f3bceff9a --- /dev/null +++ b/doc/ci/variables/img/custom_variable_output.png diff --git a/doc/ci/variables/img/new_custom_variable_example.png b/doc/ci/variables/img/new_custom_variable_example.png Binary files differnew file mode 100644 index 00000000000..d169c5f1806 --- /dev/null +++ b/doc/ci/variables/img/new_custom_variable_example.png diff --git a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png Binary files differnew file mode 100644 index 00000000000..02369d57fb8 --- /dev/null +++ b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differnew file mode 100644 index 00000000000..3bcd354e096 --- /dev/null +++ b/doc/ci/variables/img/override_variable_manual_pipeline.png diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md new file mode 100644 index 00000000000..846c539daab --- /dev/null +++ b/doc/ci/variables/predefined_variables.md @@ -0,0 +1,113 @@ +# Predefined environment variables reference + +For an introduction on this subject, read through the +[getting started with environment variables](README.md) document. + +## Overview + +Some of the predefined environment variables are available only if a minimum +version of [GitLab Runner](https://docs.gitlab.com/runner/) is used. Consult the table below to find the +version of Runner required. + +NOTE: **Note:** +Starting with GitLab 9.0, we have deprecated some variables. Read the +[9.0 Renaming](deprecated_variables.md#gitlab-90-renamed-variables) section to find out their replacements. **You are +strongly advised to use the new variables as we will remove the old ones in +future GitLab releases.** + +## Variables reference + +| Variable | GitLab | Runner | Description | +|-------------------------------------------|--------|--------|-------------| +| `ARTIFACT_DOWNLOAD_ATTEMPTS` | 8.15 | 1.9 | Number of attempts to download artifacts running a job | +| `CHAT_INPUT` | 10.6 | all | Additional arguments passed in the [ChatOps](../chatops/README.md) command | +| `CHAT_CHANNEL` | 10.6 | all | Source chat channel which triggered the [ChatOps](../chatops/README.md) command | +| `CI` | all | 0.4 | Mark that job is executed in CI environment | +| `CI_BUILDS_DIR` | all | 11.10 | Top-level directory where builds are executed. | +| `CI_CONCURRENT_ID` | all | 11.10 | Unique ID of build execution within a single executor. | +| `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | Unique ID of build execution within a single executor and project. | +| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a push request. | +| `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | +| `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. | +| `CI_COMMIT_REF_NAME` | 9.0 | all | The branch or tag name for which project is built | +| `CI_COMMIT_REF_SLUG` | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | +| `CI_COMMIT_SHA` | 9.0 | all | The commit revision for which project is built | +| `CI_COMMIT_SHORT_SHA` | 11.7 | all | The first eight characters of `CI_COMMIT_SHA` | +| `CI_COMMIT_TAG` | 9.0 | 0.5 | The commit tag name. Present only when building tags. | +| `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit - the full first line of the message | +| `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | +| `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug tracing](README.md#debug-tracing) is enabled | +| `CI_DEPLOY_PASSWORD` | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| +| `CI_DEPLOY_USER` | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| +| `CI_DISPOSABLE_ENVIRONMENT` | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | +| `CI_ENVIRONMENT_NAME` | 8.15 | all | The name of the environment for this job. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | +| `CI_ENVIRONMENT_SLUG` | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | +| `CI_ENVIRONMENT_URL` | 9.3 | all | The URL of the environment for this job. Only present if [`environment:url`](../yaml/README.md#environmenturl) is set. | +| `CI_JOB_ID` | 9.0 | all | The unique id of the current job that GitLab CI uses internally | +| `CI_JOB_MANUAL` | 8.12 | all | The flag to indicate that job was manually started | +| `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | +| `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | +| `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry][registry] and downloading [dependent repositories][dependent-repositories] | +| `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | +| `CI_MERGE_REQUEST_ID` | 11.6 | all | The ID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_IID` | 11.6 | all | The IID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_PROJECT_ID` | 11.6 | all | The ID of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_PROJECT_PATH` | 11.6 | all | The path of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) (e.g. `namespace/awesome-project`) | +| `CI_MERGE_REQUEST_PROJECT_URL` | 11.6 | all | The URL of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) (e.g. `http://192.168.10.15:3000/namespace/awesome-project`) | +| `CI_MERGE_REQUEST_REF_PATH` | 11.6 | all | The ref path of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). (e.g. `refs/merge-requests/1/head`) | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | 11.6 | all | The source branch name of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD sha of the source branch of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_ID` | 11.6 | all | The ID of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` | 11.6 | all | The path of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_URL` | 11.6 | all | The URL of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | 11.6 | all | The target branch name of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD sha of the target branch of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_TITLE` | 11.9 | all | The title of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of usernames of assignees for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). [Multiple assignees for merge requests](https://gitlab.com/gitlab-org/gitlab-ee/issues/2004) is scheduled for a future release | +| `CI_MERGE_REQUEST_MILESTONE` | 11.9 | all | The milestone title of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_MERGE_REQUEST_LABELS` | 11.9 | all | Comma-separated label names of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | +| `CI_NODE_INDEX` | 11.5 | all | Index of the job in the job set. If the job is not parallelized, this variable is not set. | +| `CI_NODE_TOTAL` | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | +| `CI_API_V4_URL` | 11.7 | all | The GitLab API v4 root URL | +| `CI_PAGES_DOMAIN` | 11.8 | all | The configured domain that hosts GitLab Pages. | +| `CI_PAGES_URL` | 11.8 | all | URL to GitLab Pages-built pages. Always belongs to a subdomain of `CI_PAGES_DOMAIN`. | +| `CI_PIPELINE_ID` | 8.10 | all | The unique id of the current pipeline that GitLab CI uses internally | +| `CI_PIPELINE_IID` | 11.0 | all | The unique id of the current pipeline scoped to project | +| `CI_PIPELINE_SOURCE` | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | +| `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered] | +| `CI_PIPELINE_URL` | 11.1 | 0.5 | Pipeline details URL | +| `CI_PROJECT_DIR` | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | +| `CI_PROJECT_ID` | all | all | The unique id of the current project that GitLab CI uses internally | +| `CI_PROJECT_NAME` | 8.10 | 0.5 | The project name that is currently being built (actually it is project folder name) | +| `CI_PROJECT_NAMESPACE` | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | +| `CI_PROJECT_PATH` | 8.10 | 0.5 | The namespace with project name | +| `CI_PROJECT_PATH_SLUG` | 9.3 | all | `$CI_PROJECT_PATH` lowercased and with everything except `0-9` and `a-z` replaced with `-`. Use in URLs and domain names. | +| `CI_PROJECT_URL` | 8.10 | 0.5 | The HTTP(S) address to access project | +| `CI_PROJECT_VISIBILITY` | 10.3 | all | The project visibility (internal, private, public) | +| `CI_REGISTRY` | 8.10 | 0.5 | If the Container Registry is enabled it returns the address of GitLab's Container Registry | +| `CI_REGISTRY_IMAGE` | 8.10 | 0.5 | If the Container Registry is enabled for the project it returns the address of the registry tied to the specific project | +| `CI_REGISTRY_PASSWORD` | 9.0 | all | The password to use to push containers to the GitLab Container Registry | +| `CI_REGISTRY_USER` | 9.0 | all | The username to use to push containers to the GitLab Container Registry | +| `CI_REPOSITORY_URL` | 9.0 | all | The URL to clone the Git repository | +| `CI_RUNNER_DESCRIPTION` | 8.10 | 0.5 | The description of the runner as saved in GitLab | +| `CI_RUNNER_EXECUTABLE_ARCH` | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | +| `CI_RUNNER_ID` | 8.10 | 0.5 | The unique id of runner being used | +| `CI_RUNNER_REVISION` | all | 10.6 | GitLab Runner revision that is executing the current job | +| `CI_RUNNER_TAGS` | 8.10 | 0.5 | The defined runner tags | +| `CI_RUNNER_VERSION` | all | 10.6 | GitLab Runner version that is executing the current job | +| `CI_SERVER` | all | all | Mark that job is executed in CI environment | +| `CI_SERVER_NAME` | all | all | The name of CI server that is used to coordinate jobs | +| `CI_SERVER_REVISION` | all | all | GitLab revision that is used to schedule jobs | +| `CI_SERVER_VERSION` | all | all | GitLab version that is used to schedule jobs | +| `CI_SERVER_VERSION_MAJOR` | 11.4 | all | GitLab version major component | +| `CI_SERVER_VERSION_MINOR` | 11.4 | all | GitLab version minor component | +| `CI_SERVER_VERSION_PATCH` | 11.4 | all | GitLab version patch component | +| `CI_SHARED_ENVIRONMENT` | all | 10.1 | Marks that the job is executed in a shared environment (something that is persisted across CI invocations like `shell` or `ssh` executor). If the environment is shared, it is set to true, otherwise it is not defined at all. | +| `GET_SOURCES_ATTEMPTS` | 8.15 | 1.9 | Number of attempts to fetch sources running a job | +| `GITLAB_CI` | all | all | Mark that job is executed in GitLab CI environment | +| `GITLAB_USER_EMAIL` | 8.12 | all | The email of the user who started the job | +| `GITLAB_USER_ID` | 8.12 | all | The id of the user who started the job | +| `GITLAB_USER_LOGIN` | 10.0 | all | The login username of the user who started the job | +| `GITLAB_USER_NAME` | 10.0 | all | The real name of the user who started the job | +| `RESTORE_CACHE_ATTEMPTS` | 8.15 | 1.9 | Number of attempts to restore the cache running a job | +| `GITLAB_FEATURES` | 10.6 | all | The comma separated list of licensed features available for your instance and plan | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index ceca4af1bee..091ddcb0bae 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -15,26 +15,26 @@ There are two places defined variables can be used. On the: ### `.gitlab-ci.yml` file -| Definition | Can be expanded? | Expansion place | Description | -|--------------------------------------|-------------------|-----------------|--------------| -| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<ul><li>Supported: all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules)</li><li>Not supported: variables defined in Runner's `config.toml` and variables created in job's `script`</li></ul> | -| `environment:name` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support: <ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> | -| `variables` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `image` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `services:[]` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `services:[]:name` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `cache:key` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `artifacts:name` | yes | Runner | The variable expansion is made by GitLab Runner's shell environment | -| `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | -| `only:variables:[]`, `except:variables:[]` | no | n/a | The variable must be in the form of `$variable`.<br/>Not supported:<ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> | +| Definition | Can be expanded? | Expansion place | Description | +|:-------------------------------------------|:-----------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<br/><br/>Supported are all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules).<br/><br/>Not supported are variables defined in Runner's `config.toml` and variables created in job's `script`. | +| `environment:name` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| `variables` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `image` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `services:[]` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `services:[]:name` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `cache:key` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `artifacts:name` | yes | Runner | The variable expansion is made by GitLab Runner's shell environment | +| `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | +| `only:variables:[]`, `except:variables:[]` | no | n/a | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | ### `config.toml` file NOTE: **Note:** You can read more about `config.toml` in the [Runner's docs](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). -| Definition | Can be expanded? | Description | -|--------------------------------------|------------------|-------------| +| Definition | Can be expanded? | Description | +|:-------------------------------------|:-----------------|:---------------------------------------------------------------------------------------------------------------------------------------------| | `runners.environment` | yes | The variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `runners.kubernetes.pod_labels` | yes | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `runners.kubernetes.pod_annotations` | yes | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | @@ -111,4 +111,4 @@ They are: - Script execution shell. - Not supported: - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. - - In the `only` and `except` [variables expressions](README.md#variables-expressions). + - In the `only` and `except` [variables expressions](README.md#environment-variables-expressions). diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 9eb694a2c64..36a0bf10416 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -319,7 +319,7 @@ There are a few rules that apply to the usage of job policy: - `only` and `except` are inclusive. If both `only` and `except` are defined in a job specification, the ref is filtered by `only` and `except`. -- `only` and `except` allow the use of regular expressions (using [Ruby regexp syntax](https://ruby-doc.org/core/Regexp.html)). +- `only` and `except` allow the use of regular expressions ([supported regexp syntax](#supported-onlyexcept-regexp-syntax)). - `only` and `except` allow to specify a repository path to filter jobs for forks. @@ -351,6 +351,19 @@ job: - branches ``` +Pattern matching is case-sensitive by default. Use `i` flag modifier, like +`/pattern/i` to make a pattern case-insensitive: + +```yaml +job: + # use regexp + only: + - /^issue-.*$/i + # use special keyword + except: + - branches +``` + In this example, `job` will run only for refs that are tagged, or if a build is explicitly requested via an API trigger or a [Pipeline Schedule][schedules]: @@ -378,6 +391,11 @@ job: The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, except `master` and those with names prefixed with `release/`. +NOTE: **Note:** +Because `@` is used to denote the beginning of a ref's repository path, +matching a ref name containing the `@` character in a regular expression +requires the use of the hex character code match `\x40`. + If a job does not have an `only` rule, `only: ['branches', 'tags']` is set by default. If it doesn't have an `except` rule, it is empty. @@ -396,11 +414,28 @@ job: only: ['branches', 'tags'] ``` -### `only`/`except` (advanced) +### Supported `only`/`except` regexp syntax + +CAUTION: **Warning:** +This is a breaking change that was introduced with GitLab 11.9.4. -> - `refs` and `kubernetes` policies introduced in GitLab 10.0. -> - `variables` policy introduced in GitLab 10.7. -> - `changes` policy [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/19232) in GitLab 11.4. +In GitLab 11.9.4, GitLab begun internally converting regexp used +in `only` and `except` parameters to [RE2](https://github.com/google/re2/wiki/Syntax). + +This means that only subset of features provided by [Ruby Regexp](https://ruby-doc.org/core/Regexp.html) +is supported. [RE2](https://github.com/google/re2/wiki/Syntax) limits the set of features +provided due to computational complexity, which means some features became unavailable in GitLab 11.9.4. +For example, negative lookaheads. + +For GitLab versions from 11.9.7 and up to GitLab 12.0, GitLab provides a feature flag that can be +enabled by administrators that allows users to use unsafe regexp syntax. This brings compatibility +with previously allowed syntax version and allows users to gracefully migrate to the new syntax. + +```ruby +Feature.enable(:allow_unsafe_ruby_regexp) +``` + +### `only`/`except` (advanced) CAUTION: **Warning:** This an _alpha_ feature, and it is subject to change at any time without @@ -422,6 +457,8 @@ If you use multiple keys under `only` or `except`, they act as an AND. The logic #### `only:refs`/`except:refs` +> `refs` policy introduced in GitLab 10.0. + The `refs` strategy can take the same values as the [simplified only/except configuration](#onlyexcept-basic). @@ -438,6 +475,8 @@ deploy: #### `only:kubernetes`/`except:kubernetes` +> `kubernetes` policy introduced in GitLab 10.0. + The `kubernetes` strategy accepts only the `active` keyword. In the example below, the `deploy` job is going to be created only when the @@ -451,6 +490,8 @@ deploy: #### `only:variables`/`except:variables` +> `variables` policy introduced in GitLab 10.7. + The `variables` keyword is used to define variables expressions. In other words, you can use predefined variables / project / group or environment-scoped variables to define an expression GitLab is going to @@ -479,11 +520,13 @@ end-to-end: - $CI_COMMIT_MESSAGE =~ /skip-end-to-end-tests/ ``` -Learn more about [variables expressions](../variables/README.md#variables-expressions). +Learn more about [variables expressions](../variables/README.md#environment-variables-expressions). #### `only:changes`/`except:changes` -Using the `changes` keyword with `only` or `except`, makes it possible to define if +> `changes` policy [introduced][ce-19232] in GitLab 11.4. + +Using the `changes` keyword with `only` or `except` makes it possible to define if a job should be created based on files modified by a git push event. For example: @@ -499,36 +542,59 @@ docker build: - more_scripts/*.{rb,py,sh} ``` -In the scenario above, if you are pushing multiple commits to GitLab to an -existing branch, GitLab creates and triggers the `docker build` job, provided that -one of the commits contains changes to either: +In the scenario above, when pushing multiple commits to GitLab to an existing +branch, GitLab creates and triggers the `docker build` job, provided that one of the +commits contains changes to any of the following: - The `Dockerfile` file. - Any of the files inside `docker/scripts/` directory. - Any of the files and subdirectories inside the `dockerfiles` directory. - Any of the files with `rb`, `py`, `sh` extensions inside the `more_scripts` directory. +You can also use glob patterns to match multiple files in either the root directory of the repo, or in _any_ directory within the repo. For example: + +```yaml +test: + script: npm run test + only: + changes: + - "*.json" + - "**/*.sql" +``` + +NOTE: **Note:** +In the example above, the expressions are wrapped double quotes because they are glob patterns. GitLab will fail to parse `.gitlab-ci.yml` files with unwrapped glob patterns. + +The following example will skip the CI job if a change is detected in any file in the root directory of the repo with a `.md` extension: + +```yaml +build: + script: npm run build + except: + changes: + - "*.md" +``` + CAUTION: **Warning:** There are some caveats when using this feature with new branches and tags. See the section below. ##### Using `changes` with new branches and tags -If you are pushing a **new** branch or a **new** tag to GitLab, the policy -always evaluates to true and GitLab will create a job. This feature is not -connected with merge requests yet, and because GitLab is creating pipelines -before an user can create a merge request we don't know a target branch at -this point. +When pushing a **new** branch or a **new** tag to GitLab, the policy always +evaluates to true and GitLab will create a job. This feature is not connected +with merge requests yet and, because GitLab is creating pipelines before a user +can create a merge request, it is unknown what the target branch is at this point. ##### Using `changes` with `merge_requests` With [pipelines for merge requests](../merge_request_pipelines/index.md), -make it possible to define if a job should be created base on files modified +it is possible to define a job to be created based on files modified in a merge request. For example: -``` +```yaml docker build service one: script: docker build -t my-service-one-image:$CI_COMMIT_REF_SLUG . only: @@ -539,9 +605,9 @@ docker build service one: - service-one/**/* ``` -In the scenario above, if you create or update a merge request that changes -either files in `service-one` folder or `Dockerfile`, GitLab creates and triggers -the `docker build service one` job. +In the scenario above, if a merge request is created or updated that changes +either files in `service-one` directory or the `Dockerfile`, GitLab creates +and triggers the `docker build service one` job. ### `tags` @@ -694,7 +760,7 @@ Manual actions are a special type of job that are not executed automatically, they need to be explicitly started by a user. An example usage of manual actions would be a deployment to a production environment. Manual actions can be started from the pipeline, job, environment, and deployment views. Read more at the -[environments documentation](../environments.md#manually-deploying-to-environments). +[environments documentation](../environments.md#configuring-manual-deployments). Manual actions can be either optional or blocking. Blocking manual actions will block the execution of the pipeline at the stage this action is defined in. It's @@ -2187,7 +2253,7 @@ Runner itself](../variables/README.md#predefined-environment-variables). One example would be `CI_COMMIT_REF_NAME` which has the value of the branch or tag name for which project is built. Apart from the variables you can set in `.gitlab-ci.yml`, there are also the so called -[Variables](../variables/README.md#variables) +[Variables](../variables/README.md#gitlab-cicd-environment-variables) which can be set in GitLab's UI. Learn more about [variables and their priority][variables]. @@ -2300,8 +2366,35 @@ variables: GIT_STRATEGY: clone GIT_CHECKOUT: "false" script: - - git checkout master - - git merge $CI_BUILD_REF_NAME + - git checkout -B master origin/master + - git merge $CI_COMMIT_SHA +``` + +#### Git clean flags + +> Introduced in GitLab Runner 11.10 + +The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of +`git clean` after checking out the sources. You can set it globally or per-job in the +[`variables`](#variables) section. + +`GIT_CLEAN_FLAGS` accepts all possible options of the [git clean](https://git-scm.com/docs/git-clean) +command. + +`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. + +If `GIT_CLEAN_FLAGS` is: + +- Not specified, `git clean` flags default to `-ffdx`. +- Given the value `none`, `git clean` is not executed. + +For example: + +```yaml +variables: + GIT_CLEAN_FLAGS: -ffdx -e cache/ +script: + - ls -al cache/ ``` #### Job stages attempts @@ -2377,6 +2470,72 @@ CAUTION: **Deprecated:** `type` is deprecated, and could be removed in one of the future releases. Use [`stage`](#stage) instead. +## Custom build directories + +> [Introduced][https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1267] in Gitlab Runner 11.10 + +NOTE: **Note:** +This can only be used when `custom_build_dir` is enabled in the [Runner's +configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). +This is the default configuration for `docker` and `kubernetes` executor. + +By default, GitLab Runner clones the repository in a unique subpath of the `$CI_BUILDS_DIR` directory. +However, sometimes your project might require the code in a specific directory, +but sometimes your project might require to have the code in a specific directory, +like Go projects, for example. In that case, you can specify the `GIT_CLONE_PATH` variable +to tell the Runner in which directory to clone the repository: + +```yml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name + +test: + script: + - pwd +``` + +The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` +is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +setting. + +### Handling concurrency + +An executor using a concurrency greater than `1` might lead +to failures because multiple jobs might be working on the same directory if the `builds_dir` +is shared between jobs. +GitLab Runner does not try to prevent this situation. It is up to the administrator +and developers to comply with the requirements of Runner configuration. + +To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because Runner +exposes two additional variables that provide a unique `ID` of concurrency: + +- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. +- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. + +The most stable configuration that should work well in any scenario and on any executor +is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: + +```yml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name + +test: + script: + - pwd +``` + +The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` +as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: + +```yml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH + +test: + script: + - pwd +``` + ## Special YAML features It's possible to use special YAML features like anchors (`&`), aliases (`*`) @@ -2542,6 +2701,15 @@ Not to be confused with [`trigger`](#trigger-premium). [Read more in the triggers documentation.](../triggers/README.md) +## Processing Git pushes + +GitLab will create at most 4 branch and tags pipelines when +doing pushing multiple changes in single `git push` invocation. + +This limitation does not affect any of the updated Merge Request pipelines, +all updated Merge Requests will have a pipeline created when using +[pipelines for merge requests](../merge_request_pipelines/index.md). + ## Skipping jobs If your commit message contains `[ci skip]` or `[skip ci]`, using any @@ -2559,6 +2727,7 @@ git push -o ci.skip [ce-7983]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7983 [ce-7447]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7447 [ce-12909]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12909 +[ce-19232]: https://gitlab.com/gitlab-org/gitlab-ce/issues/19232 [environment]: ../environments.md "CI/CD environments" [schedules]: ../../user/project/pipelines/schedules.md "Pipelines schedules" [variables]: ../variables/README.md "CI/CD variables" diff --git a/doc/customization/system_header_and_footer_messages/custom_header_footer.png b/doc/customization/system_header_and_footer_messages/custom_header_footer.png Binary files differindex 691ca8a3484..15d19c87dd7 100644 --- a/doc/customization/system_header_and_footer_messages/custom_header_footer.png +++ b/doc/customization/system_header_and_footer_messages/custom_header_footer.png diff --git a/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png b/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png Binary files differindex a9c59bc9f62..51349ddf6f4 100644 --- a/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png +++ b/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png diff --git a/doc/customization/welcome_message.md b/doc/customization/welcome_message.md index 9194f847cdf..45f1fed355e 100644 --- a/doc/customization/welcome_message.md +++ b/doc/customization/welcome_message.md @@ -1 +1,5 @@ +--- +redirect_to: 'branded_login_page.md' +--- + This document was moved to [another location](branded_login_page.md). diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 501092ff2aa..8d2bfff3a5d 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -9,38 +9,6 @@ can be shared. It is also possible to add a `private_token` to the querystring, or add a `HTTP_PRIVATE_TOKEN` header. -### Authorization - -Fields can be authorized using the same abilities used in the Rails -app. This can be done by supplying the `authorize` option: - -```ruby -module Types - class QueryType < BaseObject - graphql_name 'Query' - - field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver, authorize: :read_project - end -end -``` - -Fields can be authorized against multiple abilities, in which case all -ability checks must pass. This requires explicitly passing a block to `field`: - -```ruby -field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver do - authorize [:read_project, :another_ability] -end -``` - -The object found by the resolve call is used for authorization. - -TIP: **Tip:** -When authorizing collections, try to load only what the currently -authenticated user is allowed to view with our existing finders first. -This minimizes database queries and unnecessary authorization checks of -the loaded records. - ## Types When exposing a model through the GraphQL API, we do so by creating a @@ -197,6 +165,114 @@ end policies at once. The fields for these will all have be non-nullable booleans with a default description. +## Authorization + +Authorizations can be applied to both types and fields using the same +abilities as in the Rails app. + +If the: + +- Currently authenticated user fails the authorization, the authorized +resource will be returned as `null`. +- Resource is part of a collection, the collection will be filtered to +exclude the objects that the user's authorization checks failed against. + +TIP: **Tip:** +Try to load only what the currently authenticated user is allowed to +view with our existing finders first, without relying on authorization +to filter the records. This minimizes database queries and unnecessary +authorization checks of the loaded records. + +### Type authorization + +Authorize a type by passing an ability to the `authorize` method. All +fields with the same type will be authorized by checking that the +currently authenticated user has the required ability. + +For example, the following authorization ensures that the currently +authenticated user can only see projects that they have the +`read_project` ability for (so long as the project is returned in a +field that uses `Types::ProjectType`): + +```ruby +module Types + class ProjectType < BaseObject + authorize :read_project + end +end +``` + +You can also authorize against multiple abilities, in which case all of +the ability checks must pass. + +For example, the following authorization ensures that the currently +authenticated user must have `read_project` and `another_ability` +abilities to see a project: + +```ruby +module Types + class ProjectType < BaseObject + authorize [:read_project, :another_ability] + end +end +``` + +### Field authorization + +Fields can be authorized with the `authorize` option. + +For example, the following authorization ensures that the currently +authenticated user must have the `owner_access` ability to see the +project: + +```ruby +module Types + class MyType < BaseObject + field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver, authorize: :owner_access + end +end +``` + +Fields can also be authorized against multiple abilities, in which case +all of ability checks must pass. **Note:** This requires explicitly +passing a block to `field`: + +```ruby +module Types + class MyType < BaseObject + field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver do + authorize [:owner_access, :another_ability] + end + end +end +``` + +NOTE: **Note:** If the field's type already [has a particular +authorization](#type-authorization) then there is no need to add that +same authorization to the field. + +### Type and Field authorizations together + +Authorizations are cumulative, so where authorizations are defined on +a field, and also on the field's type, then the currently authenticated +user would need to pass all ability checks. + +In the following simplified example the currently authenticated user +would need both `first_permission` and `second_permission` abilities in +order to see the author of the issue. + +```ruby +class UserType + authorize :first_permission +end +``` + +```ruby +class IssueType + field :author, UserType, authorize: :second_permission +end +``` + ## Resolvers To find objects to display in a field, we can add resolvers to diff --git a/doc/development/code_review.md b/doc/development/code_review.md index f115045dbb7..b1a32e0ed26 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -20,7 +20,7 @@ importance of involving reviewer(s) in the section on the responsibility of the If you need some guidance (e.g. it's your first merge request), feel free to ask one of the [Merge request coaches][team]. -If you need assistance with security scans or comments, feel free to include the +If you need assistance with security scans or comments, feel free to include the Security Team (`@gitlab-com/gl-security`) in the review. The `danger-review` CI job will randomly pick a reviewer and a maintainer for @@ -58,12 +58,7 @@ from teams other than your own. #### Security requirements - 1. If your merge request is processing, storing, or transferring any kind of [RED or ORANGE data](https://docs.google.com/document/d/15eNKGA3zyZazsJMldqTBFbYMnVUSQSpU14lo22JMZQY/edit) (this is a confidential document), it must be - **approved by a [Security Engineer][team]**. - 1. If your merge request involves implementing, utilizing, or is otherwise related to any type of authentication, authorization, or session handling mechanism, it must be - **approved by a [Security Engineer][team]**. - 1. If your merge request has a goal which requires a cryptographic function such as: confidentiality, integrity, authentication, or non-repudiation, it must be - **approved by a [Security Engineer][team]**. +View the updated documentation regarding [internal application security reviews](https://about.gitlab.com/handbook/engineering/security/index.html#internal-application-security-reviews) for **when** and **how** to request a security review. ### The responsibility of the merge request author @@ -138,10 +133,10 @@ as a domain expert and/or reviewer, it is recommended that they are not also pic as the maintainer to ultimately approve and merge it. Try to review in a timely manner; doing so allows everyone involved in the merge -request to iterate faster as the context is fresh in memory. Further, this +request to iterate faster as the context is fresh in memory. Further, this improves contributors' experiences significantly. Reviewers should aim to review -within two working days from the date they were assigned the merge request. If -you don't think you'll be able to review a merge request within that time, let +within two working days from the date they were assigned the merge request. If +you don't think you'll be able to review a merge request within that time, let the author know as soon as possible. When the author of the merge request has not heard anything after two days, a new reviewer should be assigned. @@ -151,7 +146,7 @@ required approvers. Maintainers must check before merging if the merge request is introducing new vulnerabilities, by inspecting the list in the Merge Request [Security Widget](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate). -When in doubt, a [Security Engineer][team] can be involved. The list of detected +When in doubt, a [Security Engineer][team] can be involved. The list of detected vulnerabilities must be either empty or containing: - dismissed vulnerabilities in case of false positives diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index bfe1ef75914..8b1d014e101 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -3,9 +3,6 @@ Thank you for your interest in contributing to GitLab. This guide details how to contribute to GitLab in a way that is easy for everyone. -We want to create a welcoming environment for everyone who is interested in contributing. -Please visit our [Code of Conduct page](https://about.gitlab.com/contributing/code-of-conduct) to learn more about our commitment to an open and welcoming environment. - For a first-time step-by-step guide to the contribution process, please see ["Contributing to GitLab"](https://about.gitlab.com/contributing/). @@ -33,8 +30,8 @@ vulnerabilities. ## Code of conduct -Our code of conduct can be found on the -["Contributing to GitLab"](https://about.gitlab.com/contributing/) page. +We want to create a welcoming environment for everyone who is interested in contributing. +Please visit our [Code of Conduct page](https://about.gitlab.com/community/contribute/code-of-conduct/) to learn more about our commitment to an open and welcoming environment. ## Closing policy for issues and merge requests @@ -52,9 +49,12 @@ for audiences of all ages. If a contributor is no longer actively working on a submitted merge request we can decide that the merge request will be finished by one of our [Merge request coaches][team] or close the merge request. We make this decision -based on how important the change is for our product vision. If a Merge request +based on how important the change is for our product vision. If a merge request coach is going to finish the merge request we assign the -~"coach will finish" label. +~"coach will finish" label. When a team member picks up a community contribution, +we credit the original author by adding a changelog entry crediting the author +and optionally include the original author on at least one of the commits +within the MR. ## Helping others diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 4c53643ed9c..827b3d7681a 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -62,10 +62,12 @@ The current team labels are: - ~Configure - ~Create +- ~Defend - ~Distribution - ~Documentation - ~Geo - ~Gitaly +- ~Growth - ~Manage - ~Monitor - ~Plan @@ -99,6 +101,8 @@ The current stage labels are: - ~"devops:configure" - ~"devops:monitor" - ~"devops:secure" +- ~"devops:defend" +- ~"devops:enablement" These labels should be mutually exclusive. If an issue belongs to multiple stages, the most relevant should be used. diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 0eedef5e14f..f319d00d7fe 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -10,7 +10,7 @@ 1. [Testing][testing] 1. [JavaScript styleguide][js-styleguide] 1. [SCSS styleguide][scss-styleguide] -1. [Shell commands](../shell_commands.md) created by GitLab +1. [Shell commands (Ruby)](../shell_commands.md) created by GitLab contributors to enhance security 1. [Database Migrations](../migration_style_guide.md) 1. [Markdown](http://www.cirosantilli.com/markdown-styleguide) @@ -22,6 +22,7 @@ text should be _sorry, we could not create your account because:_ 1. Code should be written in [US English][us-english] 1. [Go](../go_guide/index.md) +1. [Python](../python_guide/index.md) This is also the style used by linting tools such as [RuboCop](https://github.com/bbatsov/rubocop) and [Hound CI](https://houndci.com). diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index a4da34a50ce..4a2ff64807e 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -48,29 +48,21 @@ as its markdown rendering engine. See the [GitLab Markdown Guide](https://about. Adhere to the [Documentation Style Guide](styleguide.md). If a style standard is missing, you are welcome to suggest one via a merge request. -## Documentation directory structure +## Documentation types and organization The documentation is structured based on the GitLab UI structure itself, separated by [`user`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/user), [`administrator`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/administration), and [`contributor`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/development). -In order to have a [solid site structure](https://searchengineland.com/seo-benefits-developing-solid-site-structure-277456) for our documentation, -all docs should be linked. Every new document should be cross-linked to its related documentation, and linked from its topic-related index, when existent. - -The directories `/workflow/`, `/gitlab-basics/`, `/university/`, and `/articles/` have -been **deprecated** and the majority their docs have been moved to their correct location -in small iterations. Please do not create new docs in these folders. Organize docs by product area and subject, not type. +Organize docs by product area and subject, not type. For example, do not create groupings of similar media types +(e.g. indexes of all articles, videos, etc.). -### Documentation files - -- When you create a new directory, always start with an `index.md` file. - Do not use another file name and **do not** create `README.md` files. -- **Do not** use special chars and spaces, or capital letters in file names, - directory names, branch names, and anything that generates a path. -- Max screenshot size: 100KB. -- We do not support videos (yet). +Similarly, we do not use glossaries or FAQs. Such grouping of content by type makes +it difficult to browse for the information you need and difficult to maintain up-to-date content. +Instead, organize content by its subject (e.g. everything related to CI goes together) +and cross-link between any related content. -### Location and naming documents +### Location and naming of files Our goal is to have a clear hierarchical structure with meaningful URLs like `docs.gitlab.com/user/project/merge_requests/`. With this pattern, @@ -78,16 +70,8 @@ you can immediately tell that you are navigating to user-related documentation about project features; specifically about merge requests. Our site's paths match those of our repository, so the clear structure also makes documentation easier to update. -While the documentation is home to a variety of content types, we do not organize by content type. -For example, do not create groupings of similar media types (e.g. indexes of all articles, videos, etc.). -Similarly, we do not use glossaries or FAQs. Such grouping of content by type makes -it difficult to browse for the information you need and difficult to maintain up-to-date content. -Instead, organize content by its subject (e.g. everything related to CI goes together) -and cross-link between any related content. - -Do not simply link out to GitLab technical blog posts. There should be an up-to-date -single source of truth on the topic within the documentation, and the top of the -blog post should be updated to link to that doc. +In order to have a [solid site structure](https://searchengineland.com/seo-benefits-developing-solid-site-structure-277456) for our documentation, +all docs should be linked at least from its higher-level index page if not also from other relevant locations. The table below shows what kind of documentation goes where. @@ -102,13 +86,18 @@ The table below shows what kind of documentation goes where. | `doc/update/` | Same with `doc/install/`. Should be under `administration/`, but this is a well known location, better leave as-is, at least for now. | | `doc/topics/` | Indexes per Topic (`doc/topics/topic-name/index.md`): all resources for that topic (user and admin documentation, articles, and third-party docs) | -**General rules & best practices:** +**Rules and best practices:** +1. When you create a new directory, always start with an `index.md` file. + Do not use another file name and **do not** create `README.md` files. +1. **Do not** use special chars and spaces, or capital letters in file names, + directory names, branch names, and anything that generates a path. 1. When creating a new document and it has more than one word in its name, make sure to use underscores instead of spaces or dashes (`-`). For example, a proper naming would be `import_projects_from_github.md`. The same rule applies to images. -1. Start a new directory with an `index.md` file. +1. For image files, do not exceed 100KB. +1. We do not yet support embedded videos. Please link out. 1. There are four main directories, `user`, `administration`, `api` and `development`. 1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, `profile/`, `dashboard/` and `admin_area/`. @@ -129,6 +118,9 @@ The table below shows what kind of documentation goes where. 1. The `doc/topics/` directory holds topic-related technical content. Create `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. General user- and admin- related documentation, should be placed accordingly. +1. The directories `/workflow/`, `/university/`, and `/articles/` have +been **deprecated** and the majority their docs have been moved to their correct location +in small iterations. If you are unsure where a document or a content addition should live, this should not stop you from authoring and contributing. You can use your best judgment and diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index c0386290785..1cefe48b4c1 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -14,9 +14,8 @@ For programmatic help adhering to the guidelines, see [linting](index.md#linting ## Files -- [Directory structure](index.md#location-and-naming-documents): place the docs - in the correct location. -- [Documentation files](index.md#documentation-files): name the files accordingly. +See the [Documentation types and organization](index.md#documentation-types-and-organization) section +on the index page for information on how to structure and name files across the GitLab documentation. DANGER: **Attention:** **Do not** use capital letters, spaces, or special chars in file names, @@ -28,41 +27,43 @@ a test that will fail if it spots a new `README.md` file. ### Markdown -The [documentation website](https://docs.gitlab.com) had its markdown engine migrated from [Redcarpet to GitLab Kramdown](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108) -in October 2018. +The [documentation website](https://docs.gitlab.com) uses GitLab Kramdown as its Markdown rendering engine [as of October 2018](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108). For a complete Kramdown reference, see the [GitLab Markdown Kramdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/). The [`gitlab-kramdown`](https://gitlab.com/gitlab-org/gitlab_kramdown) gem will support all [GFM markup](../../user/markdown.md) in the future. For now, -use regular markdown markup, following the rules on this style guide. For a complete -Kramdown reference, check the [GitLab Markdown Kramdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/). -Use Kramdown markup wisely: do not overuse its specific markup (e.g., `{:.class}`) as it will not render properly in -[`/help`](index.md#gitlab-help). +use regular markdown markup, following the rules in the linked style guide. + +Note that Kramdown-specific markup (e.g., `{:.class}`) will not render properly in on GitLab instances in [`/help`](index.md#gitlab-help). ## Content These guidelines help toward the goal of having every user's search of documentation yield a useful result, and ensuring content is helpful and easy to consume. -- What to include: - - Any and all helpful information, processes, and tips for implementing, - using, and troubleshooting GitLab features. [The documentation is the single source of truth](https://about.gitlab.com/handbook/documentation/#documentation-as-single-source-of-truth-ssot) - for this information. - - 'Risky' or niche problem-solving steps. There is no reason to withhold these or - store them elsewhere; simply include them along with the rest of the docs including all necessary - detail, such as specific warnings and caveats about potential ramifications. - - Any content types/sources, if relevant to users or admins. You can freely - include presentations, videos, etc.; no matter who it was originally written for, - if it is helpful to any of our audiences, we can include it. If an outside source - that's under copyright, rephrase, or summarize and link out; do not copy and paste. - - All applicable subsections as described on the [structure and template](structure.md) page, - with files organized in the [correct directory](index.md#documentation-directory-structure). +### Subject matter + +[The documentation is the single source of truth (SSOT)](https://about.gitlab.com/handbook/documentation/#documentation-as-single-source-of-truth-ssot) for any and all helpful information and processes needed to learn about, implement, use, and troubleshoot GitLab features. Note that this includes problem-solving actions that may address rare cases or be considered 'risky', so long as proper context is provided. See the SSOT link for more detail. + +### Media types and sources + +Include any media types/sources, if relevant to readers. You can freely include or link presentations, diagrams, videos, etc.; no matter who it was originally composed for, if it is helpful to any of our audiences, we can include it. + + - If you use an image that has a separate source file (for example, a vector or diagram format), link the image to the source file so that it may be reused or updated by anyone. + - Do not copy and paste content from other sources unless it is a limited quotation with the source cited. Typically it is better to either rephrase relevant information in your own words or link out to the other source. + +### Structure across documents + +- Place files in the correct directory per the [Documentation directory structure](index.md#documentation-types-and-organization) guidelines. +- To avoid duplication, do not include the same information in multiple places. Instead, choose one 'single source of truth (SSOT)' location and link from other relevant locations. +- When referencing other GitLab products and features, link to their respective docs. +- When referencing third-party products or technologies, link out to their external sites, documentation, and resources. + +### Structure within documents + +- Include any and all applicable subsections as described on the [structure and template](structure.md) page, - To ensure discoverability, link to each doc from its higher-level index page and other related pages. -- When referencing other GitLab products and features, link to their - respective docs; when referencing third-party products or technologies, - link out to their external sites, documentation, and resources. -- Do not duplicate information. - Structure content in alphabetical order in tables, lists, etc., unless there is - a logical reason not to (for example, when mirroring the UI or an ordered sequence). + a logical reason not to (for example, when mirroring the UI or an otherwise ordered sequence). ## Language diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 41a64044c68..c5a1d915be6 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -19,6 +19,11 @@ CE specs should remain untouched as much as possible and extra specs should be added for EE. Licensed features can be stubbed using the spec helper `stub_licensed_features` in `EE::LicenseHelpers`. +You can force Webpack to act as CE by either deleting the `ee/` directory or by +setting the [`IS_GITLAB_EE` environment variable](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/config/helpers/is_ee_env.js) +to something that evaluates as `false`. The same works for running tests +(for example `IS_GITLAB_EE=0 yarn jest`). + [ee-as-ce]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2500 ## Separation of EE code @@ -909,14 +914,15 @@ export default { - Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](https://docs.gitlab.com/ee/development/ee_features.html#javascript-code-in-assetsjavascripts) alias we already have for webpack. - This means all the EE specific props, computed properties, methods, etc that are EE only should be in a mixin in the `ee/` folder and we need to create a CE counterpart of the mixin - ##### Example: - ```javascript - import mixin from 'ee_else_ce/path/mixin'; +##### Example: +```javascript +import mixin from 'ee_else_ce/path/mixin'; - { +{ mixins: [mixin] - } - ``` +} +``` + - Computed Properties/methods and getters only used in the child import still need a counterpart in CE - For store modules, we will need a CE counterpart too. @@ -942,7 +948,7 @@ export default { ```html - <ul v-if="renderIfEE"> + <ul v-if="ifEE"> <li>One wrapped</li> <li>element</li> <li>that is rendered</li> @@ -961,7 +967,7 @@ For regular JS files, the approach is similar. ```javascript import { ifEE } from '~/lib/utils/common_utils' -if (renderIfEE) { +if (ifEE) { $('.js-import-git-toggle-button').on('click', () => { const $projectMirror = $('#project_mirror'); @@ -975,7 +981,7 @@ if (renderIfEE) { To separate EE-specific styles in SCSS files, if a component you're adding styles for is limited to only EE, it is better to have a separate SCSS file in appropriate directory within `app/assets/stylesheets`. -See [backporting changes](#backporting-changes-from-EE-to-CE) for instructions on how to merge changes safely. +See [backporting changes](#backporting-changes-from-ee-to-ce) for instructions on how to merge changes safely. In some cases, this is not entirely possible or creating dedicated SCSS file is an overkill, e.g. a text style of some component is different for EE. In such cases, diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 3290f29530a..5f6123b5f9b 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -46,7 +46,9 @@ Read more about [Vue Apollo][vue-apollo] in the [Vue Apollo documentation][vue-a ### Local state with `apollo-link-state` It is possible to use our Apollo setup with [apollo-link-state][apollo-link-state] by passing -in the client state object when creating the default client. +in a resolvers object when creating the default client. The default state can be set by writing +to the cache after setting up the default client. + ```javascript import Vue from 'vue'; @@ -54,15 +56,23 @@ import VueApollo from 'vue-apollo'; import createDefaultClient from '~/lib/graphql'; Vue.use(VueApollo); +const defaultClient = createDefaultClient({ + Query: { + ... + }, + Mutations: { + ... + }, +}); + +defaultClient.cache.writeData({ + data: { + isLoading: true, + }, +}); + const apolloProvider = new VueApollo({ - defaultClient: createDefaultClient({ - defaults: { - testing: true, - }, - resolvers: { - ... - }, - }), + defaultClient, }); ``` diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md index b09243598d5..548d72bea93 100644 --- a/doc/development/fe_guide/style_guide_scss.md +++ b/doc/development/fe_guide/style_guide_scss.md @@ -5,6 +5,23 @@ easy to maintain, and performant for the end-user. ## Rules +### Utility Classes +As part of the effort for [cleaning up our CSS and moving our components into GitLab-UI](https://gitlab.com/groups/gitlab-org/-/epics/950) +led by the [GitLab UI WG](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20623) we prefer the use of utility classes over adding new CSS. However, complex CSS can be addressed by adding component classes. + +We have a few internal utility classes in [`common.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/framework/common.scss) +and we use [Bootstrap's Utility Classes](https://getbootstrap.com/docs/4.3/utilities/) + +New utility classes should be added to [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/utilities.scss). Existing classes include: + +**Background color**: `.bg-variant-shade` e.g. `.bg-warning-400` +**Text color**: `.text-variant-shade` e.g. `.text-success-500` +- variant is one of 'primary', 'secondary', 'success', 'warning', 'error' +- shade is on of the shades listed on [colors](https://design.gitlab.com/foundations/colors/) + +**Font size**: `.text-size` e.g. `.text-2` +- **size** is number from 1-6 from our [Type scale](https://design.gitlab.com/foundations/typography) + ### Naming Filenames should use `snake_case`. diff --git a/doc/development/fe_guide/testing.md b/doc/development/fe_guide/testing.md index 98e499b8c0f..b23e37d1eef 100644 --- a/doc/development/fe_guide/testing.md +++ b/doc/development/fe_guide/testing.md @@ -1 +1,5 @@ -This document was moved to [../testing_guide/frontend_testing.md](../testing_guide/frontend_testing.md). +--- +redirect_to: '../testing_guide/frontend_testing.md' +--- + +This document was moved to [another location](../testing_guide/frontend_testing.md). diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index a34d1fcec89..437ce9abc7d 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -38,7 +38,7 @@ _For consistency purposes, we recommend you to follow the same structure._ Let's look into each of them: -### A `index.js` file +### An `index.js` file This is the index file of your new feature. This is where the root Vue instance of the new feature should be. @@ -46,7 +46,7 @@ of the new feature should be. The Store and the Service should be imported and initialized in this file and provided as a prop to the main component. -Don't forget to follow [these steps][page_specific_javascript]. +Be sure to read about [page-specific JavaScript][page_specific_javascript]. ### Bootstrapping Gotchas #### Providing data from HAML to JavaScript @@ -240,7 +240,7 @@ One should apply to be a Vue.js expert by opening an MR when the Merge Request's [vue-docs]: http://vuejs.org/guide/index.html [issue-boards]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/app/assets/javascripts/boards [environments-table]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/app/assets/javascripts/environments -[page_specific_javascript]: https://docs.gitlab.com/ce/development/frontend.html#page-specific-javascript +[page_specific_javascript]: ./performance.md#page-specific-javascript [component-system]: https://vuejs.org/v2/guide/#Composing-with-Components [state-management]: https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch [one-way-data-flow]: https://vuejs.org/v2/guide/components.html#One-Way-Data-Flow diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 7b54fa6289c..7d52cac5f7e 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -83,7 +83,7 @@ In this file, we will write the actions that will call the respective mutations: export const requestUsers = ({ commit }) => commit(types.REQUEST_USERS); export const receiveUsersSuccess = ({ commit }, data) => commit(types.RECEIVE_USERS_SUCCESS, data); - export const receiveUsersError = ({ commit }, error) => commit(types.REQUEST_USERS_ERROR, error); + export const receiveUsersError = ({ commit }, error) => commit(types.RECEIVE_USERS_ERROR, error); export const fetchUsers = ({ state, dispatch }) => { dispatch('requestUsers'); @@ -175,7 +175,7 @@ Remember that actions only describe that something happened, they don't describe state.users = data; state.isLoading = false; }, - [types.REQUEST_USERS_ERROR](state, error) { + [types.RECEIVE_USERS_ERROR](state, error) { state.isLoading = false; }, [types.REQUEST_ADD_USER](state, user) { diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 1a4b1743a00..bfde26dbe4a 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -6,7 +6,7 @@ Workhorse and GitLab-Shell. ## Beginner's guide Start by reading the gitaly repository's -[Beginner's guide to Gitaly contributions](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/beginners_guide.md). +[Beginner's guide to Gitaly contributions](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/beginners_guide.md). It describes how to setup gitaly, the various components of gitaly and what they do, and how to run its test suites. ## Developing new Git features @@ -192,8 +192,87 @@ GITALY_REPO_URL=https://gitlab+deploy-token-1000:token-here@gitlab.com/nick.thom To use a custom Gitaly repository in CI, for instance if you want your GitLab fork to always use your own Gitaly fork, set `GITALY_REPO_URL` -as a [CI environment variable](../ci/variables/README.md#variables). +as a [CI environment variable](../ci/variables/README.md#gitlab-cicd-environment-variables). --- [Return to Development documentation](README.md) + +## Wrapping RPCs in Feature Flags + +Here are the steps to gate a new feature in Gitaly behind a feature flag. + +### Gitaly + +1. Create a package scoped flag name: + + ```go + var findAllTagsFeatureFlag = "go-find-all-tags" + ``` + +1. Create a switch in the code using the `featureflag` package: + + ```go + if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) { + // go implementation + } else { + // ruby implementation + } + ``` + +1. Create prometheus metrics: + + ```go + var findAllTagsRequests = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "gitaly_find_all_tags_requests_total", + Help: "Counter of go vs ruby implementation of FindAllTags", + }, + []string{"implementation"}, + ) + ) + + func init() { + prometheus.Register(findAllTagsRequests) + } + + if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) { + findAllTagsRequests.WithLabelValues("go").Inc() + // go implementation + } else { + findAllTagsRequests.WithLabelValues("ruby").Inc() + // ruby impelmentation + } + ``` + +1. Set headers in tests: + + ```go + import ( + "google.golang.org/grpc/metadata" + + "gitlab.com/gitlab-org/gitaly/internal/featureflag" + ) + + //... + + md := metadata.New(map[string]string{featureflag.HeaderKey(findAllTagsFeatureFlag): "true"}) + ctx = metadata.NewOutgoingContext(context.Background(), md) + + c, err = client.FindAllTags(ctx, rpcRequest) + require.NoError(t, err) + ``` + +### Gitlab-Rails + +1. Add feature flag to `lib/gitlab/gitaly_client.rb` (in gitlab-rails): + + ```ruby + SERVER_FEATURE_FLAGS = %w[go-find-all-tags].freeze + ``` + +1. Test in rails console by setting feature flag: + + ```ruby + Feature.enable('gitaly_go-find-all-tags') + ``` diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 7fd41c5e01f..6dcade3bb51 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -26,7 +26,7 @@ Reviewers and maintainers should pay attention to: - `defer` functions: ensure the presence when needed, and after `err` check. - Inject dependencies as parameters. -- Void structs when marshalling to JSON (generates `null` instead of `[]`). +- Void structs when marshaling to JSON (generates `null` instead of `[]`). ### Security @@ -185,7 +185,7 @@ There are a few guidelines one should follow when using the - When printing an error use [WithError](https://godoc.org/github.com/sirupsen/logrus#WithError). For - exmaple, `logrus.WithError(err).Error("Failed to do something")`. + example, `logrus.WithError(err).Error("Failed to do something")`. - Since we use [structured logging](#structured-json-logging) we can log fields in the context of that code path, such as the URI of the request using [`WithField`](https://godoc.org/github.com/sirupsen/logrus#WithField) or diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index dd338f6f67d..131e3edf35e 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -126,5 +126,8 @@ are very appreciative of the work done by translators and proofreaders! your previous translations by [GitLab team members](https://about.gitlab.com/team/) or [Core team members](https://about.gitlab.com/core-team/) who are fluent in the language or current proofreaders. + - When a request is made for the first proofreader for a lanuguage and there are no [GitLab team members](https://about.gitlab.com/team/) + or [Core team members](https://about.gitlab.com/core-team/) who speak the language, we will request links to previous translation work in other communities or projects. + [proofreader-src]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/i18n/proofreader.md diff --git a/doc/development/i18n_guide.md b/doc/development/i18n_guide.md index f6e949b5fd8..e588b47e203 100644 --- a/doc/development/i18n_guide.md +++ b/doc/development/i18n_guide.md @@ -1 +1,5 @@ -This document was moved to [a new location](i18n/index.md). +--- +redirect_to: 'i18n/index.md' +--- + +This document was moved to [another location](i18n/index.md). diff --git a/doc/development/python_guide/index.md b/doc/development/python_guide/index.md new file mode 100644 index 00000000000..6025dc9ebf2 --- /dev/null +++ b/doc/development/python_guide/index.md @@ -0,0 +1,79 @@ +# Python Development Guidelines + +GitLab requires Python as a dependency for [reStructuredText](http://docutils.sourceforge.net/rst.html) +markup rendering. + +As of GitLab 11.10, we require Python 3. + +## Installation + +There are several ways of installing python on your system. To be able to use the same version we use in production, +we suggest you use [pyenv](https://github.com/pyenv/pyenv). It works and behave similar to its counterpart in the +ruby world: [rbenv](https://github.com/rbenv/rbenv). + +### macOS + +To install `pyenv` on macOS, you can use [Homebrew](https://brew.sh/) with: + +```bash +brew install pyenv +``` + +### Linux + +To install `pyenv` on Linux, you can run the command below: + +```bash +curl https://pyenv.run | bash +``` + +Alternatively, you may find `pypenv` available as a system package via your distro package manager. + +You can read more about it in: <https://github.com/pyenv/pyenv-installer#prerequisites>. + +### Shell integration + +Pyenv installation will add required changes to Bash. If you use a different shell, +check for any additional steps required for it. + +For Fish, you can install a plugin for [Fisherman](https://github.com/fisherman/fisherman): + +```bash +fisher add fisherman/pyenv +``` + +Or for [Oh My Fish](https://github.com/oh-my-fish/oh-my-fish): + +```bash +omf install pyenv +``` + +## Dependency management + +While GitLab doesn't directly contain any Python scripts, because we depend on Python to render +[reStructuredText](http://docutils.sourceforge.net/rst.html) markup, we need to keep track on dependencies +on the main project level, so we can run that on our development machines. + +Recently, an equivalent to the `Gemfile` and the [Bundler](https://bundler.io/) project has been introduced to Python: +`Pipfile` and [Pipenv](https://pipenv.readthedocs.io/en/latest/). + +You will now find a `Pipfile` with the dependencies in the root folder. To install them, run: + +```bash +pipenv install +``` + +Running this command will install both the required Python version as well as required pip dependencies. + +## Use instructions + +To run any python code under the Pipenv environment, you need to first start a `virtualenv` based on the dependencies +of the application. With Pipenv, this is a simple as running: + +```bash +pipenv shell +``` + +After running that command, you can run GitLab on the same shell and it will be using the Python and dependencies +installed from the `pipenv install` command. + diff --git a/doc/development/sql.md b/doc/development/sql.md index 47519d39e74..edeca7fb298 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -155,6 +155,21 @@ The _only_ time you should use `pluck` is when you actually need to operate on the values in Ruby itself (e.g. write them to a file). In almost all other cases you should ask yourself "Can I not just use a sub-query?". +In line with our `CodeReuse/ActiveRecord` cop, you should only use forms like +`pluck(:id)` or `pluck(:user_id)` within model code. In the former case, you can +use the `ApplicationRecord`-provided `.pluck_primary_key` helper method instead. +In the latter, you should add a small helper method to the relevant model. + +## Inherit from ApplicationRecord + +Most models in the GitLab codebase should inherit from `ApplicationRecord`, +rather than from `ActiveRecord::Base`. This allows helper methods to be easily +added. + +An exception to this rule exists for models created in database migrations. As +these should be isolated from application code, they should continue to subclass +from `ActiveRecord::Base`. + ## Use UNIONs UNIONs aren't very commonly used in most Rails applications but they're very diff --git a/doc/development/testing.md b/doc/development/testing.md index 45b1519ece8..79ef8e75432 100644 --- a/doc/development/testing.md +++ b/doc/development/testing.md @@ -1 +1,5 @@ -This document was moved to [testing_guide/index.md](testing_guide/index.md). +--- +redirect_to: 'testing_guide/index.md' +--- + +This document was moved to [another location](testing_guide/index.md). diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index cfe0e6f70fc..e41148360f2 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -1,5 +1,20 @@ # Testing best practices +## Test Design + +Testing at GitLab is a first class citizen, not an afterthought. It's important we consider the design of our tests +as we do the design of our features. + +When implementing a feature, we think about developing the right capabilities the right way, which helps us +narrow our scope to a manageable level. When implementing tests for a feature, we must think about developing +the right tests, but then cover _all_ the important ways the test may fail, which can quickly widen our scope to +a level that is difficult to manage. + +Test heuristics can help solve this problem. They concisely address many of the common ways bugs +manifest themselves within our code. When designing our tests, take time to review known test heuristics to inform +our test design. We can find some helpful heuristics documented in the Handbook in the +[Test Design](https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/test-design/) section. + ## Test speed GitLab has a massive test suite that, without [parallelization], can take hours @@ -201,8 +216,10 @@ project, one project will do for the entire file. This can be achieved by using reloads or recreates the model, _only_ if needed. That is, when you changed properties or destroyed the object. -There is one gotcha; you can't reference a model defined in a `let` block in a -`set` block. +Note that you can't reference a model defined in a `let` block in a `set` block. + +Also, `set` is not supported in `:js` specs since those don't use transactions +to clean up database state after each example. ### Time-sensitive tests diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end_tests.md index 7010250b33c..51fe19c3d9e 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end_tests.md @@ -7,6 +7,24 @@ as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together. +## Branch naming + +If your contribution contains **only** changes under the +[`qa/` folder](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa), you can +speed up the CI process by following some branch naming conventions. You have +three choices: + +| Branch name | Valid example | +|:----------------------|:-----------------------------| +| Starting with `qa/` | `qa/new-oauth-login-test` | +| Starting with `qa-` | `qa-new-oauth-login-test` | +| Ending in `-qa` | `123-new-oauth-login-test-qa` | + +If your branch name matches any of the above, it will run only the QA-related +jobs. +If it does not, the whole application test suite will run (including QA-related +jobs). + ## How do we test GitLab? We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we @@ -27,10 +45,15 @@ Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests +#### Using the `package-and-qa` job + It is possible to run end-to-end tests for a merge request, eventually being run in a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project, -by triggering the `package-and-qa` manual action in the `test` stage (which should -be present in a merge request widget, unless the merge request comes from a fork). +by triggering the `package-and-qa` manual action in the `test` stage (not +available for forks). + +**This runs end-to-end tests against a custom Omnibus package built from your +merge request's changes.** Manual action that starts end-to-end tests is also available in merge requests in [Omnibus GitLab][omnibus-gitlab]. @@ -85,7 +108,25 @@ subgraph gitlab-qa pipeline 1. The result of the [GitLab QA pipeline][gitlab-qa-pipelines] is being propagated upstream, through Omnibus, back to the CE / EE merge request. -#### How do I write tests? +#### Using the `review-qa-all` jobs + +On every pipeline during the `test` stage, the `review-qa-smoke` job is +automatically started: it runs the QA smoke suite against the +[Review App][review-apps]. + +You can also manually start the `review-qa-all`: it runs the full QA suite +against the [Review App][review-apps]. + +**This runs end-to-end tests against a Review App based on [the official GitLab +Helm chart][helm-chart], itself deployed with custom +[Cloud Native components][cng] built from your merge request's changes.** + +See [Review Apps][review-apps] for more details about Review Apps. + +[helm-chart]: https://gitlab.com/charts/gitlab/ +[cng]: https://gitlab.com/gitlab-org/build/CNG + +## How do I write tests? In order to write new tests, you first need to learn more about GitLab QA architecture. See the [documentation about it][gitlab-qa-architecture]. @@ -105,9 +146,11 @@ you can find an issue you would like to work on in [omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab [gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa +[gitlab-qa-pipelines]: https://gitlab.com/gitlab-org/gitlab-qa/pipelines [gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md [quality-nightly-pipelines]: https://gitlab.com/gitlab-org/quality/nightly/pipelines [quality-staging-pipelines]: https://gitlab.com/gitlab-org/quality/staging/pipelines +[review-apps]: ./review_apps.md [gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md [gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario [gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name[]=QA&label_name[]=test diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 3d568c37fba..931cbc51cae 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -8,7 +8,7 @@ eventually. ## Quarantined tests When a test frequently fails in `master`, -[a ~"broken master" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) +[a ~"master:broken" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) should be created. If the test cannot be fixed in a timely fashion, there is an impact on the productivity of all the developers, so it should be placed in quarantine by @@ -21,7 +21,7 @@ bin/rspec --tag quarantine ``` **Before putting a test in quarantine, you should make sure that a -~"broken master" issue exists for it so it won't stay in quarantine forever.** +~"master:broken" issue exists for it so it won't stay in quarantine forever.** Once a test is in quarantine, there are 3 choices: diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 71c9637e72c..f58a8dcbcdc 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -26,6 +26,10 @@ It is not yet a requirement to use Jest. You can view the [epic](https://gitlab.com/groups/gitlab-org/-/epics/873) of issues we need to solve before being able to use Jest for all our needs. +### Debugging Jest tests + +Running `yarn jest-debug` will run Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/en/troubleshooting#tests-are-failing-and-you-don-t-know-why). + ### Timeout error The default timeout for Jest is set in diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 67e4cfeda0e..ecad9ba48a3 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -33,7 +33,7 @@ changes should be tested. ## [Testing best practices](best_practices.md) -Everything you should know about how to write good tests: RSpec, FactoryBot, +Everything you should know about how to write good tests: Test Design, RSpec, FactoryBot, system tests, parameterized tests etc. --- diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index fda3ff57316..55ca502f84a 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -90,8 +90,8 @@ subgraph GCP `gitlab-review-apps` project ## QA runs On every [pipeline][gitlab-pipeline] during the `test` stage, the -`review-qa-smoke` job is automatically started: it runs the smoke QA suite. -You can also manually start the `review-qa-all`: it runs the full QA suite. +`review-qa-smoke` job is automatically started: it runs the QA smoke suite. +You can also manually start the `review-qa-all`: it runs the QA full suite. Note that both jobs first wait for the `review-deploy` job to be finished. @@ -130,9 +130,9 @@ secure note named **gitlab-{ce,ee} Review App's root password**. 1. Find and open the `task-runner` Deployment, e.g. `review-29951-issu-id2qax-task-runner`. 1. Click on the Pod in the "Managed pods" section, e.g. `review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz`. 1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. -1. Replace `-c task-runner -- ls` with `-- /srv/gitlab/bin/rails c` from the +1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the default command or - - Run `kubectl exec --namespace review-apps-ce -it review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz -- /srv/gitlab/bin/rails c` + - Run `kubectl exec --namespace review-apps-ce review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and - Replace `review-apps-ce` with `review-apps-ee` if the Review App is running EE, and diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md index 3360031c220..30d861d7d68 100644 --- a/doc/development/testing_guide/smoke.md +++ b/doc/development/testing_guide/smoke.md @@ -7,13 +7,19 @@ functionality is working. Currently, our suite consists of this basic functionality coverage: -- User Login (Standard Auth) -- Project Creation -- Issue Creation -- Merge Request Creation +- User standard authentication +- SSH Key creation and addition to a user +- Project simple creation +- Project creation with Auto-DevOps enabled +- Issue creation +- Merge Request creation +- Snippet creation Smoke tests have the `:smoke` RSpec metadata. +See [End-to-end Testing](./end_to_end_tests.md) for more details about +end-to-end tests. + --- [Return to Testing documentation](index.md) diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 5d46833a1e2..352651fe91b 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -160,7 +160,7 @@ Every new feature should come with a [test plan]. > See [end-to-end tests](end_to_end_tests.md) for more information. Note that `qa/spec` contains unit tests of the QA framework itself, not to be -confused with the application's [unit tests](#unit-tests) or +confused with the application's [unit tests](#unit-tests) or [end-to-end tests](#black-box-tests-at-the-system-level-aka-end-to-end-tests). [multiple pieces]: ../architecture.md#components @@ -234,6 +234,8 @@ you should write an integration test using Jasmine. [big]: https://twitter.com/timbray/status/822470746773409794 [picture]: https://twitter.com/withzombies/status/829716565834752000 [tests-cost]: https://medium.com/table-xi/high-cost-tests-and-high-value-tests-a86e27a54df#.2ulyh3a4e +[RSpec]: https://github.com/rspec/rspec-rails#feature-specs +[Capybara]: https://github.com/teamcapybara/capybara --- diff --git a/doc/gitlab-basics/add-merge-request.md b/doc/gitlab-basics/add-merge-request.md index 5cc014419ad..7bb2e014738 100644 --- a/doc/gitlab-basics/add-merge-request.md +++ b/doc/gitlab-basics/add-merge-request.md @@ -8,7 +8,7 @@ request. For more information, check the --- 1. Before you start, you should have already [created a branch](create-branch.md) - and [pushed your changes](basic-git-commands.md) to GitLab. + and [pushed your changes](start-using-git.md#send-changes-to-gitlabcom) to GitLab. 1. Go to the project where you'd like to merge your changes and click on the **Merge requests** tab. 1. Click on **New merge request** on the right side of the screen. diff --git a/doc/gitlab-basics/create-group.md b/doc/gitlab-basics/create-group.md index 985a52d88f5..8c0d3561882 100644 --- a/doc/gitlab-basics/create-group.md +++ b/doc/gitlab-basics/create-group.md @@ -1,2 +1,5 @@ +--- +redirect_to: '../user/group/index.md#create-a-new-group' +--- This document was moved to [another location](../user/group/index.md#create-a-new-group). diff --git a/doc/gitlab-basics/create-issue.md b/doc/gitlab-basics/create-issue.md index abb163dbf18..6e2a09fc030 100644 --- a/doc/gitlab-basics/create-issue.md +++ b/doc/gitlab-basics/create-issue.md @@ -1,2 +1,5 @@ +--- +redirect_to: '../user/project/issues/index.md#issue-actions' +--- -This document was moved to [another location](../user/project/issues/index.md#new-issue). +This document was moved to [another location](../user/project/issues/index.md#issue-actions). diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index 3d4259ab7d3..3e99496d531 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -42,7 +42,9 @@ Project templates can pre-populate your project with necessary files to get you There are two types of project templates: -- [Built-in templates](#built-in-templates), sourced from the [`project-templates`](https://gitlab.com/gitlab-org/project-templates) group. +- [Built-in templates](#built-in-templates), sourced from the following groups: + - [`project-templates`](https://gitlab.com/gitlab-org/project-templates) + - [`pages`](https://gitlab.com/pages) - [Custom project templates](#custom-project-templates-premium-only), for custom templates configured by GitLab administrators and users. ### Built-in templates @@ -50,7 +52,7 @@ There are two types of project templates: Built-in templates are project templates that are: - Developed and maintained in the - [`project-templates`](https://gitlab.com/gitlab-org/project-templates) group. + [`project-templates`](https://gitlab.com/gitlab-org/project-templates) and [`pages`](https://gitlab.com/pages) groups. - Released with GitLab. To use a built-in template on the **New project** page: @@ -64,7 +66,7 @@ To use a built-in template on the **New project** page: TIP: **Tip:** You can improve the existing built-in templates or contribute new ones on the -[`project-templates`](https://gitlab.com/gitlab-org/project-templates) group. +[`project-templates`](https://gitlab.com/gitlab-org/project-templates) and [`pages`](https://gitlab.com/pages) groups. ### Custom project templates **[PREMIUM ONLY]** diff --git a/doc/gitlab-basics/fork-project.md b/doc/gitlab-basics/fork-project.md index 6c232fe6086..a128a7c7dd3 100644 --- a/doc/gitlab-basics/fork-project.md +++ b/doc/gitlab-basics/fork-project.md @@ -1,8 +1,8 @@ # How to fork a project -A fork is a copy of an original repository that you can put in another namespace -where you can experiment and apply changes that you can later decide if -publishing or not, without affecting your original project. +A fork is a copy of an original repository that you put in another namespace +where you can experiment and apply changes that you can later decide whether or +not to share, without affecting the original project. It takes just a few steps to fork a project in GitLab. diff --git a/doc/img/devops-stages.png b/doc/img/devops-stages.png Binary files differindex 424bce95607..2fa5357062c 100644 --- a/doc/img/devops-stages.png +++ b/doc/img/devops-stages.png diff --git a/doc/install/README.md b/doc/install/README.md index 52011526768..53778f7f0d3 100644 --- a/doc/install/README.md +++ b/doc/install/README.md @@ -55,9 +55,9 @@ need to be aware of: - It can be more expensive for smaller installations. The default installation requires more resources than a single node Omnibus deployment, as most services are deployed in a redundant fashion. -- There are some feature [limitations to be aware of](kubernetes/gitlab_chart.md#limitations). +- There are some feature [limitations to be aware of](https://docs.gitlab.com/charts/#limitations). -[**> Install GitLab on Kubernetes using the GitLab Helm charts.**](kubernetes/index.md) +[**> Install GitLab on Kubernetes using the GitLab Helm charts.**](https://docs.gitlab.com/charts/) ## Installing GitLab with Docker diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index 2fcc9b90157..0000e03f1d7 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -1,11 +1,11 @@ -# Installing GitLab on Amazon Web Services (AWS) - -To install GitLab on AWS, you can use the Amazon Machine Images (AMIs) that GitLab -provides with [each release](https://about.gitlab.com/releases/). +# Installing GitLab HA on Amazon Web Services (AWS) This page offers a walkthrough of a common HA (Highly Available) configuration for GitLab on AWS. You should customize it to accommodate your needs. +NOTE: **Note** +For organizations with 300 users or less, the recommended AWS installation method is to launch an EC2 single box [Omnibus Installation](https://about.gitlab.com/install/) and implement a snapshot strategy for backing up the data. + ## Introduction GitLab on AWS can leverage many of the services that are already @@ -55,6 +55,8 @@ Here's a list of the AWS services we will use, with links to pricing information - **ElastiCache**: An in-memory cache environment will be used to provide a High Availability Redis configuration. See the [Amazon ElastiCache pricing](https://aws.amazon.com/elasticache/pricing/). + +NOTE: **Note:** Please note that while we will be using EBS for storage, we do not recommend using EFS as it may negatively impact GitLab's performance. You can review the [relevant documentation](../../administration/high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. ## Creating an IAM EC2 instance role and profile To minimize the permissions of the user, we'll create a new [IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) diff --git a/doc/install/google_cloud_platform/index.md b/doc/install/google_cloud_platform/index.md index aa4b3dccf7d..bd60ee25bff 100644 --- a/doc/install/google_cloud_platform/index.md +++ b/doc/install/google_cloud_platform/index.md @@ -8,6 +8,12 @@ description: 'Learn how to install a GitLab instance on Google Cloud Platform.' Getting started with GitLab on a [Google Cloud Platform (GCP)][gcp] instance is quick and easy. +NOTE: **Note:** +Google provides a whitepaper for [deploying production-ready GitLab on +Google Kubernetes Engine](https://cloud.google.com/solutions/deploying-production-ready-gitlab-on-gke), +including all steps and external resource configuration. These are an alternative to using a GCP VM, and use +the [Cloud native GitLab Helm chart](https://docs.gitlab.com/charts). + ## Prerequisites There are only two prerequisites in order to install GitLab on GCP: diff --git a/doc/install/kubernetes/gitlab_chart.md b/doc/install/kubernetes/gitlab_chart.md index 9db246b3eb3..43655767002 100644 --- a/doc/install/kubernetes/gitlab_chart.md +++ b/doc/install/kubernetes/gitlab_chart.md @@ -1,156 +1,5 @@ -# GitLab Helm Chart +--- +redirect_to: https://docs.gitlab.com/charts/ +--- -This is the official way to install GitLab on a cloud native environment. - -NOTE: **Kubernetes experience required:** -Our Helm charts are recommended for those who are familiar with Kubernetes. -If you're not sure if Kubernetes is for you, our -[Omnibus GitLab packages](../README.md#installing-gitlab-using-the-omnibus-gitlab-package-recommended) -are mature, scalable, support [high availability](../../administration/high_availability/README.md) -and are used today on GitLab.com. -It is not necessary to have GitLab installed on Kubernetes in order to use [GitLab Kubernetes integration](https://docs.gitlab.com/ee/user/project/clusters/index.html). - -## Introduction - -The `gitlab` chart is the best way to operate GitLab on Kubernetes. This chart -contains all the required components to get started, and can scale to large deployments. - -The default deployment includes: - -- Core GitLab components: Unicorn, Shell, Workhorse, Registry, Sidekiq, and Gitaly -- Optional dependencies: Postgres, Redis, Minio -- An auto-scaling, unprivileged [GitLab Runner](https://docs.gitlab.com/runner/) using the Kubernetes executor -- Automatically provisioned SSL via [Let's Encrypt](https://letsencrypt.org/). - -## Limitations - -Some features of GitLab are not currently available: - -- [GitLab Pages](https://gitlab.com/charts/gitlab/issues/37) -- [GitLab Geo](https://gitlab.com/charts/gitlab/issues/8) -- [No in-cluster HA database](https://gitlab.com/charts/gitlab/issues/48) -- MySQL will not be supported, as support is [deprecated within GitLab](https://docs.gitlab.com/omnibus/settings/database.html#using-a-mysql-database-management-server-enterprise-edition-only) - -## Installing GitLab using the Helm Chart - -The `gitlab` chart includes all required dependencies, and takes a few minutes -to deploy. - -TIP: **Tip:** -For production deployments, we strongly recommend using the -[detailed installation instructions](https://gitlab.com/charts/gitlab/blob/master/doc/installation/index.md) -utilizing [external Postgres, Redis, and object storage](https://gitlab.com/charts/gitlab/tree/master/doc/advanced) services. - -### Requirements - -In order to deploy GitLab on Kubernetes, the following are required: - -1. `helm` and `kubectl` [installed on your computer](preparation/tools_installation.md). -1. A Kubernetes cluster, version 1.8 or higher. 6vCPU and 16GB of RAM is recommended. - - [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) - - [Google GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-container-cluster) - - [IBM IKS](https://console.bluemix.net/docs/tutorials/scalable-webapp-kubernetes.html#create_kube_cluster) - - [Microsoft AKS](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal) -1. A [wildcard DNS entry and external IP address](preparation/networking.md) -1. [Authenticate and connect](preparation/connect.md) to the cluster -1. Configure and initialize [Helm Tiller](preparation/tiller.md). - -### Deployment of GitLab to Kubernetes - -To deploy GitLab, the following three parameters are required: - -- `global.hosts.domain`: the [base domain](preparation/networking.md) of the - wildcard host entry. For example, `example.com` if the wild card entry is - `*.example.com`. -- `global.hosts.externalIP`: the [external IP](preparation/networking.md) which - the wildcard DNS resolves to. -- `certmanager-issuer.email`: the email address to use when requesting new SSL - certificates from Let's Encrypt. - -NOTE: **Note:** -For deployments to Amazon EKS, there are -[additional configuration requirements](preparation/eks.md). A full list of -configuration options is [also available](https://gitlab.com/charts/gitlab/blob/master/doc/installation/command-line-options.md). - -Once you have all of your configuration options collected, you can get any -dependencies and run helm. In this example, the helm release is named "gitlab": - -```sh -helm repo add gitlab https://charts.gitlab.io/ -helm repo update -helm upgrade --install gitlab gitlab/gitlab \ - --timeout 600 \ - --set global.hosts.domain=example.com \ - --set global.hosts.externalIP=10.10.10.10 \ - --set certmanager-issuer.email=email@example.com -``` - -### Monitoring the Deployment - -This will output the list of resources installed once the deployment finishes, -which may take 5-10 minutes. - -The status of the deployment can be checked by running `helm status gitlab` -which can also be done while the deployment is taking place if you run the -command in another terminal. - -### Initial login - -You can access the GitLab instance by visiting the domain name beginning with -`gitlab.` followed by the domain specified during installation. From the example -above, the URL would be `https://gitlab.example.com`. - -If you manually created the secret for initial root password, you -can use that to sign in as `root` user. If not, GitLab automatically -created a random password for `root` user. This can be extracted by the -following command (replace `<name>` by name of the release - which is `gitlab` -if you used the command above): - -```sh -kubectl get secret <name>-gitlab-initial-root-password -ojsonpath={.data.password} | base64 --decode ; echo -``` - -### Outgoing email - -By default outgoing email is disabled. To enable it, provide details for your SMTP server -using the `global.smtp` and `global.email` settings. You can find details for these settings in the -[command line options](https://gitlab.com/charts/gitlab/blob/master/doc/installation/command-line-options.md#email-configuration). - -If your SMTP server requires authentication make sure to read the section on providing -your password in the [secrets documentation](https://gitlab.com/charts/gitlab/blob/master/doc/installation/secrets.md#smtp-password). -You can disable authentication settings with `--set global.smtp.authentication=""`. - -If your Kubernetes cluster is on GKE, be aware that SMTP port [25 is blocked](https://cloud.google.com/compute/docs/tutorials/sending-mail/#using_standard_email_ports). - -### Deploying the Community Edition - -To deploy the Community Edition, include these options in your `helm install` command: - -```sh ---set gitlab.migrations.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-rails-ce ---set gitlab.sidekiq.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce ---set gitlab.unicorn.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce ---set gitlab.unicorn.workhorse.image=registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce ---set gitlab.task-runner.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ce -``` - -## Updating GitLab using the Helm Chart - -Once your GitLab Chart is installed, configuration changes and chart updates -should be done using `helm upgrade`: - -```sh -helm repo update -helm upgrade --reuse-values gitlab gitlab/gitlab -``` - -## Uninstalling GitLab using the Helm Chart - -To uninstall the GitLab Chart, run the following: - -```sh -helm delete gitlab -``` - -[kube-srv]: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services---service-types -[storageclass]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses +This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/gitlab_omnibus.md b/doc/install/kubernetes/gitlab_omnibus.md index c0cb7694e91..43655767002 100644 --- a/doc/install/kubernetes/gitlab_omnibus.md +++ b/doc/install/kubernetes/gitlab_omnibus.md @@ -1,246 +1,5 @@ -# GitLab-Omnibus Helm Chart +--- +redirect_to: https://docs.gitlab.com/charts/ +--- -CAUTION: **Caution:** -This chart is **deprecated**. We recommend using the [`gitlab` chart](gitlab_chart.md) -instead. A comparison of the two charts is available in [this video](https://youtu.be/Z6jWR8Z8dv8). - -For more information on available GitLab Helm Charts, see [Installing GitLab on Kubernetes](index.md). - -- This GitLab-Omnibus chart has been tested on Google Kubernetes Engine and Azure Container Service. -- This work is based partially on: <https://github.com/lwolf/kubernetes-gitlab/>. GitLab would like to thank Sergey Nuzhdin for his work. - -## Introduction - -This chart provides an easy way to get started with GitLab, provisioning an -installation with nearly all functionality enabled. SSL is automatically -provisioned via [Let's Encrypt](https://letsencrypt.org/). - -This Helm chart is suited for small to medium deployments and is **deprecated** -and replaced by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). -Due to the significant architectural changes, migrating will require backing up -data out of this instance and importing it into the new deployment. - -The deployment includes: - -- A [GitLab Omnibus](https://docs.gitlab.com/omnibus/) Pod, including Mattermost, Container Registry, and Prometheus -- An auto-scaling [GitLab Runner](https://docs.gitlab.com/runner/) using the Kubernetes executor -- [Redis](https://github.com/kubernetes/charts/tree/master/stable/redis) -- [PostgreSQL](https://github.com/kubernetes/charts/tree/master/stable/postgresql) -- [NGINX Ingress](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress) -- Persistent Volume Claims for Data, Registry, Postgres, and Redis - -## Limitations - -[High Availability](../../administration/high_availability/README.md) and -[Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html) are not supported. - -## Requirements - -- _At least_ 4 GB of RAM available on your cluster. 41GB of storage and 2 CPU are also required. -- Kubernetes 1.4+ with Beta APIs enabled -- [Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) provisioner support in the underlying infrastructure -- A [wildcard DNS entry](#networking-requirements), which resolves to the external IP address -- The `kubectl` CLI installed locally and authenticated for the cluster -- The [Helm client](https://github.com/kubernetes/helm/blob/master/docs/quickstart.md) installed locally on your machine - -### Networking requirements - -This chart configures a GitLab server and Kubernetes cluster which can support -dynamic [Review Apps](../../ci/review_apps/index.md), as well as services like -the integrated [Container Registry](../../user/project/container_registry.md) -and [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/). - -To support the GitLab services and dynamic environments, a wildcard DNS entry -is required which resolves to the [load balancer](#load-balancer-ip) or -[external IP](#external-ip-recommended). Configuration of the DNS entry will depend upon -the DNS service being used. - -#### External IP (recommended) - -To provision an external IP on GCP and Azure, simply request a new address from -the Networking section. Ensure that the region matches the region your container -cluster is created in. It is important that the IP is not assigned at this point -in time. It will be automatically assigned once the Helm chart is installed, -and assigned to the Load Balancer. - -Now that an external IP address has been allocated, ensure that the wildcard -DNS entry you would like to use resolves to this IP. Please consult the -documentation for your DNS service for more information on creating DNS records. - -Finally, set the `baseIP` setting to this IP address when -[deploying GitLab](#configuring-and-installing-gitlab). - -#### Load Balancer IP - -If you do not specify a `baseIP`, an IP will be assigned to the Load Balancer or -Ingress. You can retrieve this IP by running the following command *after* deploying GitLab: - -```sh -kubectl get svc -w --namespace nginx-ingress nginx -``` - -The IP address will be displayed in the `EXTERNAL-IP` field, and should be used -to configure the Wildcard DNS entry. For more information on creating a wildcard -DNS entry, consult the documentation for the DNS server you are using. - -For production deployments of GitLab, we strongly recommend using a -[external IP](#external-ip-recommended). - -## Configuring and Installing GitLab - -For most installations, two parameters are required: - -- `baseDomain`: the [base domain](#networking-requirements) of the wildcard host entry. For example, `mycompany.io` if the wild card entry is `*.mycompany.io`. -- `legoEmail`: Email address to use when requesting new SSL certificates from Let's Encrypt. - -Other common configuration options: - -- `baseIP`: the desired [external IP address](#external-ip-recommended) -- `gitlab`: Choose the [desired edition](https://about.gitlab.com/pricing), either `ee` or `ce`. `ce` is the default. -- `gitlabEELicense`: For Enterprise Edition, the [license](https://docs.gitlab.com/ee/user/admin_area/license.html) can be installed directly via the Chart -- `provider`: Optimizes the deployment for a cloud provider. The default is `gke` for [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/), with `acs` also supported for the [Azure Container Service](https://azure.microsoft.com/en-us/services/container-service/). - -For additional configuration options, consult the -[`values.yaml`](https://gitlab.com/charts/gitlab-omnibus/blob/master/values.yaml). - -### Choosing a different GitLab release version - -The version of GitLab installed is based on the `gitlab` setting (see [section](#configuring-and-installing-gitLab) above), and -the value of the corresponding helm setting: `gitlabCEImage` or `gitabEEImage`. - -```yaml -gitlab: CE -gitlabCEImage: gitlab/gitlab-ce:9.5.2-ce.0 -gitlabEEImage: gitlab/gitlab-ee:9.5.2-ee.0 -``` - -The different images can be found in the [gitlab-ce](https://hub.docker.com/r/gitlab/gitlab-ce/tags/) and [gitlab-ee](https://hub.docker.com/r/gitlab/gitlab-ee/tags/) -repositories on Docker Hub. - -### Persistent storage - -NOTE: **Note:** -If you are using a machine type with support for less than 4 attached disks, -like an Azure trial, you should disable dedicated storage for Postgres and Redis. - -By default, persistent storage is enabled for GitLab and the charts it depends -on (Redis and PostgreSQL). Components can have their claim size set from your -`values.yaml`, along with whether to provision separate storage for Postgres and Redis. - -Basic configuration: - -```yaml -redisImage: redis:3.2.10 -redisDedicatedStorage: true -redisStorageSize: 5Gi -postgresImage: postgres:9.6.3 -# If you disable postgresDedicatedStorage, you should consider bumping up gitlabRailsStorageSize -postgresDedicatedStorage: true -postgresStorageSize: 30Gi -gitlabRailsStorageSize: 30Gi -gitlabRegistryStorageSize: 30Gi -gitlabConfigStorageSize: 1Gi -``` - -### Routing and SSL - -Ingress routing and SSL are automatically configured within this Chart. An NGINX -ingress is provisioned and configured, and will route traffic to any service. -SSL certificates are automatically created and configured by -[kube-lego](https://github.com/kubernetes/charts/tree/master/stable/kube-lego). - -NOTE: **Note:** -Let's Encrypt limits a single TLD to five certificate requests within a single -week. This means that common DNS wildcard services like [nip.io](http://nip.io) -and [xip.io](http://xip.io) are unlikely to work. - -## Installing GitLab using the Helm Chart - -NOTE: **Note:** -You may see a temporary error message `SchedulerPredicates failed due to PersistentVolumeClaim is not bound` -while storage provisions. Once the storage provisions, the pods will automatically start. -This may take a couple minutes depending on your cloud provider. If the error persists, -please review the [requirements sections](#requirements) to ensure you have enough RAM, CPU, and storage. - -Add the GitLab Helm repository and initialize Helm: - -```bash -helm init -helm repo add gitlab https://charts.gitlab.io -``` - -Once you have reviewed the [configuration settings](#configuring-and-installing-gitlab), -you can install the chart. We recommending saving your configuration options in a -`values.yaml` file for easier upgrades in the future: - -```bash -helm install --name gitlab -f values.yaml gitlab/gitlab-omnibus -``` - -Or you can pass them on the command line: - -```bash -helm install --name gitlab --set baseDomain=gitlab.io,baseIP=192.0.2.1,gitlab=ee,gitlabEELicense=$LICENSE,legoEmail=email@gitlab.com gitlab/gitlab-omnibus -``` - -## Updating GitLab using the Helm Chart - -If you are upgrading from a previous version to 0.1.35 or above, you will need to -change the access mode values for GitLab's storage. To do this, set the following -in `values.yaml` or on the CLI: - -```sh -gitlabDataAccessMode=ReadWriteMany -gitlabRegistryAccessMode=ReadWriteMany -gitlabConfigAccessMode=ReadWriteMany -``` - -Once your GitLab Chart is installed, configuration changes and chart updates -should be done using `helm upgrade`: - -```sh -helm upgrade -f values.yaml gitlab gitlab/gitlab-omnibus -``` - -## Upgrading from CE to EE using the Helm Chart - -If you have installed the Community Edition using this chart, upgrading to -Enterprise Edition is easy. - -If you are using a `values.yaml` file to specify the configuration options, edit -the file and set `gitlab=ee`. If you would like to run a specific version of -GitLab EE, set `gitlabEEImage` to be the desired GitLab -[docker image](https://hub.docker.com/r/gitlab/gitlab-ee/tags/). Then you can -use `helm upgrade` to update your GitLab instance to EE: - -```bash -helm upgrade -f values.yaml gitlab gitlab/gitlab-omnibus -``` - -You can also upgrade and specify these options via the command line: - -```bash -helm upgrade gitlab --set gitlab=ee,gitlabEEImage=gitlab/gitlab-ee:9.5.5-ee.0 gitlab/gitlab-omnibus -``` - -## Uninstalling GitLab using the Helm Chart - -To uninstall the GitLab Chart, run the following: - -```bash -helm delete --purge gitlab -``` - -## Troubleshooting - -### Storage errors when updating `gitlab-omnibus` versions prior to 0.1.35 - -Users upgrading `gitlab-omnibus` from a version prior to 0.1.35, may see an error -like: `Error: UPGRADE FAILED: PersistentVolumeClaim "gitlab-gitlab-config-storage" is invalid: spec: Forbidden: field is immutable after creation`. - -This is due to a change in the access mode for GitLab storage in version 0.1.35. -To successfully upgrade, the access mode flags must be set to `ReadWriteMany` -as detailed in the [update section](#updating-gitlab-using-the-helm-chart). - -[kube-srv]: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services---service-types -[storageclass]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses +This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/gitlab_runner_chart.md b/doc/install/kubernetes/gitlab_runner_chart.md index 68b2a146115..08ccf2cf9ad 100644 --- a/doc/install/kubernetes/gitlab_runner_chart.md +++ b/doc/install/kubernetes/gitlab_runner_chart.md @@ -1,269 +1,5 @@ -# GitLab Runner Helm Chart -> **Note:** -These charts have been tested on Google Kubernetes Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/gitlab-org/gitlab-runner/issues). +--- +redirect_to: https://docs.gitlab.com/runner/install/kubernetes.html +--- -The `gitlab-runner` Helm chart deploys a GitLab Runner instance into your -Kubernetes cluster. - -This chart configures the Runner to: - -- Run using the GitLab Runner [Kubernetes executor](https://docs.gitlab.com/runner/install/kubernetes.html) -- For each new job it receives from [GitLab CI](https://about.gitlab.com/features/gitlab-ci-cd/), it will provision a - new pod within the specified namespace to run it. - -For more information on available GitLab Helm Charts, please see our [overview](index.md). - -## Prerequisites - -- Your GitLab Server's API is reachable from the cluster -- Kubernetes 1.4+ with Beta APIs enabled -- The `kubectl` CLI installed locally and authenticated for the cluster -- The [Helm client](https://github.com/kubernetes/helm/blob/master/docs/quickstart.md) installed locally on your machine - -## Configuring GitLab Runner using the Helm Chart - -Create a `values.yaml` file for your GitLab Runner configuration. See [Helm docs](https://github.com/kubernetes/helm/blob/master/docs/chart_template_guide/values_files.md) -for information on how your values file will override the defaults. - -The default configuration can always be found in the [values.yaml](https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml) in the chart repository. - -### Required configuration - -In order for GitLab Runner to function, your config file **must** specify the following: - - - `gitlabUrl` - the GitLab Server URL (with protocol) to register the runner against - - `runnerRegistrationToken` - The Registration Token for adding new Runners to the GitLab Server. This must be - retrieved from your GitLab Instance. See the [GitLab Runner Documentation](../../ci/runners/README.md) for more information. - -Unless you need to specify additional configuration, you are [ready to install](#installing-gitlab-runner-using-the-helm-chart). - -### Other configuration - -The rest of the configuration is [documented in the `values.yaml`](https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml) in the chart repository. - -Here is a snippet of the important settings: - -```yaml -## The GitLab Server URL (with protocol) that want to register the runner against -## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register -## -gitlabUrl: http://gitlab.your-domain.com/ - -## The Registration Token for adding new Runners to the GitLab Server. This must -## be retrieved from your GitLab Instance. -## ref: https://docs.gitlab.com/ee/ci/runners/README.html -## -runnerRegistrationToken: "" - -## Set the certsSecretName in order to pass custom certificates for GitLab Runner to use -## Provide resource name for a Kubernetes Secret Object in the same namespace, -## this is used to populate the /etc/gitlab-runner/certs directory -## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates -## -#certsSecretName: - -## Configure the maximum number of concurrent jobs -## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section -## -concurrent: 10 - -## Defines in seconds how often to check GitLab for a new builds -## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section -## -checkInterval: 30 - -## For RBAC support: -rbac: - create: false - - ## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs - ## cluster-wide or only within namespace - clusterWideAccess: false - - ## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create) - ## - # serviceAccountName: default - -## Configuration for the Pods that the runner launches for each new job -## -runners: - ## Default container image to use for builds when none is specified - ## - image: ubuntu:16.04 - - ## Run all containers with the privileged flag enabled - ## This will allow the docker:stable-dind image to run if you need to run Docker - ## commands. Please read the docs before turning this on: - ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind - ## - privileged: false - - ## Namespace to run Kubernetes jobs in (defaults to 'default') - ## - # namespace: - - ## Build Container specific configuration - ## - builds: - # cpuLimit: 200m - # memoryLimit: 256Mi - cpuRequests: 100m - memoryRequests: 128Mi - - ## Service Container specific configuration - ## - services: - # cpuLimit: 200m - # memoryLimit: 256Mi - cpuRequests: 100m - memoryRequests: 128Mi - - ## Helper Container specific configuration - ## - helpers: - # cpuLimit: 200m - # memoryLimit: 256Mi - cpuRequests: 100m - memoryRequests: 128Mi - -``` - -### Enabling RBAC support - -If your cluster has RBAC enabled, you can choose to either have the chart create its own service account or provide one. - -To have the chart create the service account for you, set `rbac.create` to true. - -### Controlling maximum Runner concurrency - -A single GitLab Runner deployed on Kubernetes is able to execute multiple jobs in parallel by automatically starting additional Runner pods. The [`concurrent` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) controls the maximum number of pods allowed at a single time, and defaults to `10`. - -```yaml -## Configure the maximum number of concurrent jobs -## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section -## -concurrent: 10 -``` - -### Running Docker-in-Docker containers with GitLab Runners - -See [Running Privileged Containers for the Runners](#running-privileged-containers-for-the-runners) for how to enable it, -and the [GitLab CI Runner documentation](https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-in-your-builds) on running dind. - -### Running privileged containers for the Runners - -You can tell the GitLab Runner to run using privileged containers. You may need -this enabled if you need to use the Docker executable within your GitLab CI jobs. - -This comes with several risks that you can read about in the -[GitLab CI Runner documentation](https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-in-your-builds). - -If you are okay with the risks, and your GitLab CI Runner instance is registered -against a specific project in GitLab that you trust the CI jobs of, you can -enable privileged mode in `values.yaml`: - -```yaml -runners: - ## Run all containers with the privileged flag enabled - ## This will allow the docker:stable-dind image to run if you need to run Docker - ## commands. Please read the docs before turning this on: - ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind - ## - privileged: true -``` - -### Providing a custom certificate for accessing GitLab - -You can provide a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) -to the GitLab Runner Helm Chart, which will be used to populate the container's -`/etc/gitlab-runner/certs` directory. - -Each key name in the Secret will be used as a filename in the directory, with the -file content being the value associated with the key. - -More information on how GitLab Runner uses these certificates can be found in the -[Runner Documentation](https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates). - - - The key/file name used should be in the format `<gitlab-hostname>.crt`. For example: `gitlab.your-domain.com.crt`. - - Any intermediate certificates need to be concatenated to your server certificate in the same file. - - The hostname used should be the one the certificate is registered for. - -The GitLab Runner Helm Chart does not create a secret for you. In order to create -the secret, you can prepare your certificate on you local machine, and then run -the `kubectl create secret` command from the directory with the certificate - -```bash -kubectl - --namespace <NAMESPACE> - create secret generic <SECRET_NAME> - --from-file=<CERTFICATE_FILENAME> -``` - -- `<NAMESPACE>` is the Kubernetes namespace where you want to install the GitLab Runner. -- `<SECRET_NAME>` is the Kubernetes Secret resource name. For example: `gitlab-domain-cert` -- `<CERTFICATE_FILENAME>` is the filename for the certificate in your current directory that will be imported into the secret - -You then need to provide the secret's name to the GitLab Runner chart. - -Add the following to your `values.yaml` - -```yaml -## Set the certsSecretName in order to pass custom certificates for GitLab Runner to use -## Provide resource name for a Kubernetes Secret Object in the same namespace, -## this is used to populate the /etc/gitlab-runner/certs directory -## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates -## -certsSecretName: <SECRET NAME> -``` - -- `<SECRET_NAME>` is the Kubernetes Secret resource name. For example: `gitlab-domain-cert` - -## Installing GitLab Runner using the Helm Chart - -Add the GitLab Helm repository and initialize Helm: - -```bash -helm repo add gitlab https://charts.gitlab.io -helm init -``` - -Once you [have configured](#configuring-gitlab-runner-using-the-helm-chart) GitLab Runner in your `values.yml` file, -run the following: - -```bash -helm install --namespace <NAMESPACE> --name gitlab-runner -f <CONFIG_VALUES_FILE> gitlab/gitlab-runner -``` - -- `<NAMESPACE>` is the Kubernetes namespace where you want to install the GitLab Runner. -- `<CONFIG_VALUES_FILE>` is the path to values file containing your custom configuration. See the - [Configuring GitLab Runner using the Helm Chart](#configuring-gitlab-runner-using-the-helm-chart) section to create it. - -## Updating GitLab Runner using the Helm Chart - -Once your GitLab Runner Chart is installed, configuration changes and chart updates should we done using `helm upgrade` - -```bash -helm upgrade --namespace <NAMESPACE> -f <CONFIG_VALUES_FILE> <RELEASE-NAME> gitlab/gitlab-runner -``` - -Where: - -- `<NAMESPACE>` is the Kubernetes namespace where GitLab Runner is installed -- `<CONFIG_VALUES_FILE>` is the path to values file containing your custom configuration. See the - [Configuring GitLab Runner using the Helm Chart](#configuring-gitlab-runner-using-the-helm-chart) section to create it. -- `<RELEASE-NAME>` is the name you gave the chart when installing it. - In the [Installing GitLab Runner using the Helm Chart](#installing-gitlab-runner-using-the-helm-chart) section, we called it `gitlab-runner`. - -## Uninstalling GitLab Runner using the Helm Chart - -To uninstall the GitLab Runner Chart, run the following: - -```bash -helm delete --namespace <NAMESPACE> <RELEASE-NAME> -``` - -where: - -- `<NAMESPACE>` is the Kubernetes namespace where GitLab Runner is installed -- `<RELEASE-NAME>` is the name you gave the chart when installing it. - In the [Installing GitLab Runner using the Helm Chart](#installing-gitlab-runner-using-the-helm-chart) section, we called it `gitlab-runner`. +This document was moved to [another location](https://docs.gitlab.com/runner/install/kubernetes.html). diff --git a/doc/install/kubernetes/index.md b/doc/install/kubernetes/index.md index ecc956d04e9..7312bf2d4f7 100644 --- a/doc/install/kubernetes/index.md +++ b/doc/install/kubernetes/index.md @@ -21,16 +21,15 @@ of the application including how it should be deployed, upgraded, and configured ## GitLab Chart This chart contains all the required components to get started, and can scale to -large deployments. It offers a number of benefits: +large deployments. It offers a number of benefits, among others: -- Horizontal scaling of individual components -- No requirement for shared storage to scale -- Containers do not need `root` permissions -- Automatic SSL with Let's Encrypt -- An unprivileged GitLab Runner -- and plenty more. +- Horizontal scaling of individual components. +- No requirement for shared storage to scale. +- Containers do not need `root` permissions. +- Automatic SSL with Let's Encrypt. +- An unprivileged GitLab Runner. -Learn more about the [GitLab chart](gitlab_chart.md). +Learn more about the [GitLab chart](https://docs.gitlab.com/charts/). ## GitLab Runner Chart @@ -39,4 +38,4 @@ and you'd like to leverage the Runner's [Kubernetes capabilities](https://docs.gitlab.com/runner/executors/kubernetes.html), it can be deployed with the GitLab Runner chart. -Learn more about [gitlab-runner chart](gitlab_runner_chart.md). +Learn more about the [GitLab Runner chart](https://docs.gitlab.com/runner/install/kubernetes.html). diff --git a/doc/install/kubernetes/preparation/connect.md b/doc/install/kubernetes/preparation/connect.md index a3a0cba4bf2..db55e03d3d4 100644 --- a/doc/install/kubernetes/preparation/connect.md +++ b/doc/install/kubernetes/preparation/connect.md @@ -1,27 +1,5 @@ -# Connecting your computer to a cluster +--- +redirect_to: https://docs.gitlab.com/charts/installation/cloud/ +--- -In order to deploy software and settings to a cluster, you must connect and authenticate to it. - -## Connect to GKE cluster - -The command for connection to the cluster can be obtained from the -[Google Cloud Platform Console](https://console.cloud.google.com/kubernetes/list) -by the individual cluster. - -Look for the **Connect** button in the clusters list page or use the command below, -filling in your cluster's information: - -``` -gcloud container clusters get-credentials <cluster-name> --zone <zone> --project <project-id> -``` - -## Connect to EKS cluster - -For the most up to date instructions, follow the Amazon EKS documentation on -[connecting to a cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#eks-configure-kubectl). - -## Connect to local minikube cluster - -If you are doing local development, you can use `minikube` as your -local cluster. If `kubectl cluster-info` is not showing `minikube` as the current -cluster, use `kubectl config set-cluster minikube` to set the active cluster. +This document was moved to [another location](https://docs.gitlab.com/charts/installation/cloud/). diff --git a/doc/install/kubernetes/preparation/eks.md b/doc/install/kubernetes/preparation/eks.md index ea3b075dd82..975d35c11c6 100644 --- a/doc/install/kubernetes/preparation/eks.md +++ b/doc/install/kubernetes/preparation/eks.md @@ -1,45 +1,5 @@ -# Running GitLab on EKS +--- +redirect_to: https://docs.gitlab.com/charts/installation/cloud/eks.html +--- -There are a few nuances to Amazon EKS which are important to be aware of, when deploying GitLab. - -## Persistent volume management - -There are two methods to manage volume claims on Kubernetes: - -1. Manually creating each persistent volume (recommended on EKS) -1. Utilizing dynamic provisioning to automatically create the persistent volumes - -### Manual provisioning of volumes (Recommended) - -Manually creating the volumes allows you to control the zone of each volume, as well as all other details supported by the underlying storage. - -Follow our documentation on [manually creating persistent volumes](https://gitlab.com/charts/gitlab/blob/master/doc/installation/storage.md#manually-creating-static-volumes). - -### Dynamic provisioning of volumes - -Dynamic provisioning utilizes a Kubernetes provisioner, like `aws-ebs`, to automatically create persistent volumes to fulfill each claim. - -With EKS, there are a few important details to keep in mind: - -1. Clusters are required to span multiple AZ's -1. Kubernetes volume provisioners create volumes across zones without regard to which pod they belong to. This leads to scenarios where a pod with multiple volumes being unable to start due to the volumes being in different zones. -1. There is no default Storage Class. - -The easiest way to solve this and still utilize dynamic provisioning is to utilize, or create, a Storage Class that is locked to a specific zone. - -> **Note**: Restricting volumes to specific zone will cause GitLab and any other application using this Storage Class to only reside in that zone. For multiple zone support, utilize [manually provisioned volumes](#manual-provisioning-of-volumes-recommended). - -To create the storage class, download and edit Amazon EKS's [sample Storage Class](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html) and add the following parameter: - -```yaml -parameters: - zone: <desired-zone> -``` - -Then [specify the Storage Class](https://gitlab.com/charts/gitlab/blob/master/doc/installation/storage.md#using-a-custom-storage-class) name when deploying GitLab. - -## External access to GitLab - -By default, GitLab will an deploy an ingress which will create an associated Elastic Load Balancer. Since the DNS names of ELB's cannot be known ahead of time, it is difficult to utilize Let's Encrypt to automatically provision HTTPS certificates. - -We recommend [using your own certificates](https://gitlab.com/charts/gitlab/blob/master/doc/installation/tls.md#option-2-use-your-own-wildcard-certificate), and then mapping your desired DNS name to the created ELB using a CNAME record. +This document was moved to [another location](https://docs.gitlab.com/charts/installation/cloud/eks.html). diff --git a/doc/install/kubernetes/preparation/networking.md b/doc/install/kubernetes/preparation/networking.md index b9fb9a7399f..2af16a752dc 100644 --- a/doc/install/kubernetes/preparation/networking.md +++ b/doc/install/kubernetes/preparation/networking.md @@ -1,38 +1,5 @@ -# Networking Prerequisites +--- +redirect_to: https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns +--- -NOTE: **Note:** -Amazon EKS utilizes Elastic Load Balancers, which are addressed by DNS name and -cannot be known ahead of time. If you're using EKS, you can skip this section. - -The `gitlab` chart configures a GitLab server and Kubernetes cluster which can support dynamic [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/index.html), as well as services like the integrated [Container Registry](https://docs.gitlab.com/ee/user/project/container_registry.html). - -To support the GitLab services and dynamic environments, a wildcard DNS entry is required which resolves to the external IP. - -## External IP - -To provision an external IP on GCP and Azure, simply request a new address from the Networking section. Ensure that the region matches the region your container cluster is created in. Note, it is important that the IP is not assigned at this point in time. It will be automatically assigned once the Helm chart is installed, to the Load Balancer. - -Set `global.hosts.externalIP` to this IP address when [deploying GitLab](../gitlab_chart.md#installing-gitlab-using-the-helm-chart). - -Then, create a [wildcard DNS record](#wildcard-dns-entry) which resolves to this IP address. - -### Creating an external IP on GCP - -When creating the external IP, it is critical to create it in the same region as your cluster. Otherwise, the IP address will fail to bind to the Load Balancer. - -1. Open the [web console](https://console.cloud.google.com) -1. In the sidebar, browse to `VPC Network > External IP addresses` -1. Click `Reserve static address` -1. Choose `Regional` and select the region of your cluster -1. Leave `Attached to` blank, as it will be automatically assigned during deployment - -## Wildcard DNS entry - -Now that an external IP address has been allocated, ensure that the wildcard DNS entry you would like to use resolves to this IP. Typically this would be an `A record` for `*`, resolving to the external IP above. - -Please consult the documentation for your DNS service for more information on creating DNS records: - -- [Google Domains](https://support.google.com/domains/answer/3290350?hl=en) -- [GoDaddy](https://www.godaddy.com/help/add-an-a-record-19238) - -Set `global.hosts.domain` to this DNS name when [deploying GitLab](../gitlab_chart.md#installing-gitlab-using-the-helm-chart). +This document was moved to [another location](https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns). diff --git a/doc/install/kubernetes/preparation/rbac.md b/doc/install/kubernetes/preparation/rbac.md index c5f8d7a7e9e..f94e7c24cdc 100644 --- a/doc/install/kubernetes/preparation/rbac.md +++ b/doc/install/kubernetes/preparation/rbac.md @@ -1,20 +1,5 @@ -# Role Based Access Control +--- +redirect_to: https://docs.gitlab.com/charts/installation/deployment.html#rbac +--- -Until Kubernetes 1.7, there were no permissions within a cluster. With the launch -of 1.7, there is now a [role based access control system (RBAC)](https://kubernetes.io/docs/admin/authorization/rbac/) -which determines what services can perform actions within a cluster. - -RBAC affects a few different aspects of GitLab: - -- [Installation of GitLab using Helm](tiller.md#preparing-for-helm-with-rbac) -- Prometheus monitoring -- GitLab Runner - -## Checking that RBAC is enabled - -Try listing the current cluster roles, if it fails then `RBAC` is disabled. -The following command will output `false` if `RBAC` is disabled and `true` otherwise: - -```sh -kubectl get clusterroles > /dev/null 2>&1 && echo true || echo false -``` +This document was moved to [another location](https://docs.gitlab.com/charts/installation/deployment.html#rbac). diff --git a/doc/install/kubernetes/preparation/tiller.md b/doc/install/kubernetes/preparation/tiller.md index 684df14ac2c..66d6c8faece 100644 --- a/doc/install/kubernetes/preparation/tiller.md +++ b/doc/install/kubernetes/preparation/tiller.md @@ -1,109 +1,5 @@ -# Configuring and initializing Helm Tiller - -To make use of Helm, you must have a [Kubernetes][k8s-io] cluster. Ensure you can -access your cluster using `kubectl`. - -Helm consists of two parts, the `helm` client and a `tiller` server inside Kubernetes. - -NOTE: **Note:** -If you are not able to run Tiller in your cluster, for example on OpenShift, it -is possible to use [Tiller locally](https://docs.gitlab.com/charts/installation/tools.html#local-tiller) -and avoid deploying it into the cluster. This should only be used when Tiller -cannot be normally deployed. - -## Initialize Helm and Tiller - -Tiller is deployed into the cluster and interacts with the Kubernetes API to deploy your applications. If role based access control (RBAC) is enabled, Tiller will need to be [granted permissions](#preparing-for-helm-with-rbac) to allow it to talk to the Kubernetes API. - -If RBAC is not enabled, skip to [initializing Helm](#initialize-helm). - -If you are not sure whether RBAC is enabled in your cluster, or to learn more, read through our [RBAC documentation](rbac.md). - -## Preparing for Helm with RBAC - -Helm's Tiller will need to be granted permissions to perform operations. These instructions grant cluster wide permissions, however for more advanced deployments [permissions can be restricted to a single namespace](https://docs.helm.sh/using_helm/#example-deploy-tiller-in-a-namespace-restricted-to-deploying-resources-only-in-that-namespace). To grant access to the cluster, we will create a new `tiller` service account and bind it to the `cluster-admin` role. - -Create a file `rbac-config.yaml` with the following contents: - -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: tiller - namespace: kube-system --- -apiVersion: rbac.authorization.k8s.io/v1beta1 -kind: ClusterRoleBinding -metadata: - name: tiller -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin -subjects: - - kind: ServiceAccount - name: tiller - namespace: kube-system -``` - -Next we need to connect to the cluster and upload the RBAC config. - -### Upload the RBAC config - -Some clusters require authentication to use `kubectl` to create the Tiller roles. - -#### Upload the RBAC config as an admin user (GKE) - -For GKE, you need to obtain the admin credentials. This command will output the admin password: - -``` -gcloud container clusters describe <cluster-name> --zone <zone> --project <project-id> --format='value(masterAuth.password)' -``` - -Use the admin password to set the admin credentials. Replace the password value below with the output value from the above step: - -``` -kubectl config set-credentials admin --username=admin --password=xxxxxxxxxxxxxx -``` - -Once credentials have been set, create the role: - -``` -kubectl --user=admin create -f rbac-config.yaml -``` - -#### Upload the RBAC config (Non-GKE clusters) - -For other clusters like Amazon EKS, you can directly upload the RBAC configuration. - -``` -kubectl create -f rbac-config.yaml -``` - -## Initialize Helm - -Deploy Helm Tiller with a service account: - -``` -helm init --service-account tiller -``` - -If your cluster previously had Helm/Tiller installed, -run the following to ensure that the deployed version of Tiller matches the local Helm version: - -``` -helm init --upgrade --service-account tiller -``` - -### Patching Helm Tiller for Amazon EKS - -Helm Tiller requires a flag to be enabled to work properly on Amazon EKS: - -``` -kubectl -n kube-system patch deployment tiller-deploy -p '{"spec": {"template": {"spec": {"automountServiceAccountToken": true}}}}' -``` +redirect_to: https://docs.gitlab.com/charts/installation/tools.html +--- -[helm]: https://helm.sh -[helm-using]: https://docs.helm.sh/using_helm -[k8s-io]: https://kubernetes.io/ -[gcp-k8s]: https://console.cloud.google.com/kubernetes/list +This document was moved to [another location](https://docs.gitlab.com/charts/installation/tools.html). diff --git a/doc/install/kubernetes/preparation/tools_installation.md b/doc/install/kubernetes/preparation/tools_installation.md index d2f7a69a0af..66d6c8faece 100644 --- a/doc/install/kubernetes/preparation/tools_installation.md +++ b/doc/install/kubernetes/preparation/tools_installation.md @@ -1,19 +1,5 @@ -# Installing kubectl and Helm on your computer +--- +redirect_to: https://docs.gitlab.com/charts/installation/tools.html +--- -In order to work with the GitLab Helm charts, `kubectl` and `helm` must be installed and configured on your computer. - -## Installing `kubectl` - -`kubectl` is the Kubernetes command line tool, which can be used to deploy settings to the cluster. - -Follow the [official documentation](https://kubernetes.io/docs/tasks/tools/install-kubectl/) for the most up to date instructions. - -## Installing `helm` - -Helm is a package management tool for Kubernetes, and is used to deploy charts. - -You can get Helm from the project's [releases page](https://github.com/kubernetes/helm/releases), or follow other options under the official documentation of [Installing Helm](https://docs.helm.sh/using_helm/#installing-helm). - -# Next steps - -Once installed, proceed to the next [installation step](../gitlab_chart.md#installing-gitlab-using-the-helm-chart). +This document was moved to [another location](https://docs.gitlab.com/charts/installation/tools.html). diff --git a/doc/integration/README.md b/doc/integration/README.md index f5bc0693b84..a539933f223 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -29,8 +29,8 @@ See the documentation below for details on how to configure these services. ## Project services -Integration with services such as Campfire, Flowdock, Pivotal Tracker, and Slack -are available in the form of a [Project Service][]. +Integration with services such as Campfire, Flowdock, HipChat, +Pivotal Tracker, and Slack are available in the form of a [Project Service][]. [Project Service]: ../user/project/integrations/project_services.md diff --git a/doc/integration/img/github_app_entry.png b/doc/integration/img/github_app_entry.png Binary files differindex 0a1fe0ca65a..c736bed3dd6 100644 --- a/doc/integration/img/github_app_entry.png +++ b/doc/integration/img/github_app_entry.png diff --git a/doc/integration/img/github_register_app.png b/doc/integration/img/github_register_app.png Binary files differindex 5786b822f53..d03ca5cc2aa 100644 --- a/doc/integration/img/github_register_app.png +++ b/doc/integration/img/github_register_app.png diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 69bbd05c367..2932c884d04 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -256,7 +256,7 @@ gitlab_rails['omniauth_enabled'] = false You can enable profile syncing from selected OmniAuth providers and for all or for specific user information. -When authenticating using LDAP, the user's email is always synced. +When authenticating using LDAP, the user's name and email are always synced. ```ruby gitlab_rails['sync_profile_from_provider'] = ['twitter', 'google_oauth2'] diff --git a/doc/markdown/markdown.md b/doc/markdown/markdown.md index 4ac81ab3ee7..29cb6ac9164 100644 --- a/doc/markdown/markdown.md +++ b/doc/markdown/markdown.md @@ -1 +1,5 @@ -This document was moved to [user/markdown.md](../user/markdown.md). +--- +redirect_to: '../user/markdown.md' +--- + +This document was moved to [another location](../user/markdown.md). diff --git a/doc/operations/README.md b/doc/operations/README.md index d7a83948b87..319e5f48d38 100644 --- a/doc/operations/README.md +++ b/doc/operations/README.md @@ -1 +1,5 @@ +--- +redirect_to: '../administration/operations/index.md' +--- + This document was moved to [another location](../administration/operations/index.md). diff --git a/doc/operations/cleaning_up_redis_sessions.md b/doc/operations/cleaning_up_redis_sessions.md index 2a1d0a8c8eb..bde7fffd090 100644 --- a/doc/operations/cleaning_up_redis_sessions.md +++ b/doc/operations/cleaning_up_redis_sessions.md @@ -1 +1,5 @@ -This document was moved to [administration/operations/cleaning_up_redis_sessions](../administration/operations/cleaning_up_redis_sessions.md). +--- +redirect_to: '../administration/operations/cleaning_up_redis_sessions.md' +--- + +This document was moved to [another location](../administration/operations/cleaning_up_redis_sessions.md). diff --git a/doc/operations/moving_repositories.md b/doc/operations/moving_repositories.md index c54bca324a5..57d47e3e9ea 100644 --- a/doc/operations/moving_repositories.md +++ b/doc/operations/moving_repositories.md @@ -1 +1,5 @@ -This document was moved to [administration/operations/moving_repositories](../administration/operations/moving_repositories.md). +--- +redirect_to: '../administration/operations/moving_repositories.md' +--- + +This document was moved to [another location](../administration/operations/moving_repositories.md). diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md index cf7c3b2e2ed..2df4a6e5648 100644 --- a/doc/operations/sidekiq_memory_killer.md +++ b/doc/operations/sidekiq_memory_killer.md @@ -1 +1,5 @@ -This document was moved to [administration/operations/sidekiq_memory_killer](../administration/operations/sidekiq_memory_killer.md). +--- +redirect_to: '../administration/operations/sidekiq_memory_killer.md' +--- + +This document was moved to [another location](../administration/operations/sidekiq_memory_killer.md). diff --git a/doc/operations/unicorn.md b/doc/operations/unicorn.md index fbc9697b755..949f4a66c9d 100644 --- a/doc/operations/unicorn.md +++ b/doc/operations/unicorn.md @@ -1 +1,5 @@ -This document was moved to [administration/operations/unicorn](../administration/operations/unicorn.md). +--- +redirect_to: '../administration/operations/unicorn.md' +--- + +This document was moved to [another location](../administration/operations/unicorn.md). diff --git a/doc/pages/README.md b/doc/pages/README.md index 7878bce3f10..c67847f1a83 100644 --- a/doc/pages/README.md +++ b/doc/pages/README.md @@ -1 +1,5 @@ -This document was moved to [pages/index.md](../user/project/pages/index.md). +--- +redirect_to: '../user/project/pages/index.md' +--- + +This document was moved to [another location](../user/project/pages/index.md). diff --git a/doc/pages/administration.md b/doc/pages/administration.md index 4eb3bb32c77..015dd54ec7f 100644 --- a/doc/pages/administration.md +++ b/doc/pages/administration.md @@ -1 +1,5 @@ -This document was moved to [administration/pages](../administration/pages/index.md). +--- +redirect_to: '../administration/pages/index.md' +--- + +This document was moved to [another location](../administration/pages/index.md). diff --git a/doc/pages/getting_started_part_one.md b/doc/pages/getting_started_part_one.md index 1d63ccb4d2f..a0feed0b477 100644 --- a/doc/pages/getting_started_part_one.md +++ b/doc/pages/getting_started_part_one.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/pages/getting_started_part_one.md' +--- + This document was moved to [another location](../user/project/pages/getting_started_part_one.md). diff --git a/doc/pages/getting_started_part_three.md b/doc/pages/getting_started_part_three.md index 1697b5cd6b4..b65247ff7b7 100644 --- a/doc/pages/getting_started_part_three.md +++ b/doc/pages/getting_started_part_three.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/pages/getting_started_part_three.md' +--- + This document was moved to [another location](../user/project/pages/getting_started_part_three.md). diff --git a/doc/pages/getting_started_part_two.md b/doc/pages/getting_started_part_two.md index a58affec73d..05353c171fc 100644 --- a/doc/pages/getting_started_part_two.md +++ b/doc/pages/getting_started_part_two.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/pages/getting_started_part_two.md' +--- + This document was moved to [another location](../user/project/pages/getting_started_part_two.md). diff --git a/doc/project_services/hipchat.md b/doc/project_services/hipchat.md new file mode 100644 index 00000000000..4ae9f6c6b2e --- /dev/null +++ b/doc/project_services/hipchat.md @@ -0,0 +1 @@ +This document was moved to [user/project/integrations/hipchat.md](../user/project/integrations/hipchat.md). diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 90187617c41..0729875daf8 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -15,3 +15,4 @@ comments: false - [Import](import.md) of git repositories in bulk - [Rebuild authorized_keys file](http://docs.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators - [Migrate Uploads](../administration/raketasks/uploads/migrate.md) +- [Sanitize Uploads](../administration/raketasks/uploads/sanitize.md) diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 574ba961cb0..56db7b5eb3a 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -409,7 +409,7 @@ an access key from the Google console first: 1. Select "Interoperability" and create an access key 1. Make note of the "Access Key" and "Secret" and replace them in the configurations below -1. In the buckets advanced settings ensure the Access Control option "Set object-level +1. In the buckets advanced settings ensure the Access Control option "Set object-level and bucket-level permissions" is selected 1. Make sure you already have a bucket created @@ -632,7 +632,7 @@ directory (repositories, uploads). To restore a backup, you will also need to restore `/etc/gitlab/gitlab-secrets.json` (for Omnibus packages) or `/home/git/gitlab/.secret` (for installations from source). This file contains the database encryption key, -[CI/CD variables](../ci/variables/README.md#variables), and +[CI/CD variables](../ci/variables/README.md#gitlab-cicd-environment-variables), and variables used for [two-factor authentication](../user/profile/account/two_factor_authentication.md). If you fail to restore this encryption key file along with the application data backup, users with two-factor authentication enabled and GitLab Runners will @@ -848,15 +848,24 @@ including (but not restricted to): * [Project mirroring](../workflow/repository_mirroring.md) * [Web hooks](../user/project/integrations/webhooks.md) -In the case of CI/CD, variables, you might experience some weird behavior, like -stuck jobs or 500 errors. In that case, you can try removing contents of the -`ci_group_variables` and `ci_project_variables` tables from the database. +In cases like CI/CD variables and Runner authentication, you might +experience some unexpected behavior such as: + +- Stuck jobs. +- 500 errors. + +In this case, you are required to reset all the tokens for CI/CD variables +and Runner Authentication, which is described in more detail below. After +resetting the tokens, you should be able to visit your project and the jobs +will have started running again. CAUTION: **Warning:** Use the following commands at your own risk, and make sure you've taken a backup beforehand. -1. Enter the Rails console: +#### Reset CI/CD variables + +1. Enter the DB console: For Omnibus GitLab packages: @@ -889,8 +898,39 @@ backup beforehand. 1. You may need to reconfigure or restart GitLab for the changes to take effect. -You should now be able to visit your project, and the jobs will start -running again. + +#### Reset Runner registration tokens + +1. Enter the DB console: + + For Omnibus GitLab packages: + + ```sh + sudo gitlab-rails dbconsole + ``` + + For installations from source: + + ```sh + sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production + ``` + +1. Clear all the tokens for projects, groups, and the whole instance: + +CAUTION: **Caution:** +The last UPDATE operation will stop the runners being able to pick up +new jobs. You must register new runners. + + ```sql + -- Clear project tokens + UPDATE projects SET runners_token = null, runners_token_encrypted = null; + -- Clear group tokens + UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; + -- Clear instance tokens + UPDATE application_settings SET runners_registration_token_encrypted = null; + -- Clear runner tokens + UPDATE ci_runners SET token = null, token_encrypted = null; + ``` A similar strategy can be employed for the remaining features - by removing the data that cannot be decrypted, GitLab can be brought back into working order, diff --git a/doc/raketasks/check.md b/doc/raketasks/check.md index f7f6a40cd04..ceb089e80c0 100644 --- a/doc/raketasks/check.md +++ b/doc/raketasks/check.md @@ -1,3 +1,5 @@ -# Check Rake Tasks +--- +redirect_to: '../administration/raketasks/check.md' +--- -This document was moved to [administration/raketasks/check](../administration/raketasks/check.md). +This document was moved to [another location](../administration/raketasks/check.md). diff --git a/doc/raketasks/maintenance.md b/doc/raketasks/maintenance.md index 266aeb7d60e..f554a09d94d 100644 --- a/doc/raketasks/maintenance.md +++ b/doc/raketasks/maintenance.md @@ -1,3 +1,5 @@ -# Maintenance Rake Tasks +--- +redirect_to: '../administration/raketasks/maintenance.md' +--- -This document was moved to [administration/raketasks/maintenance](../administration/raketasks/maintenance.md). +This document was moved to [another location](../administration/raketasks/maintenance.md). diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index fb3f9711711..1df492ba82c 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -141,7 +141,7 @@ in any of the following places: NOTE: **Note** The Auto DevOps base domain variable (`KUBE_INGRESS_BASE_DOMAIN`) follows the same order of precedence -as other environment [variables](../../ci/variables/README.md#priority-of-variables). +as other environment [variables](../../ci/variables/README.md#priority-of-environment-variables). A wildcard DNS A record matching the base domain(s) is required, for example, given a base domain of `example.com`, you'd need a DNS entry like: @@ -444,13 +444,6 @@ This is an optional step, since many projects do not have a Kubernetes cluster available. If the [requirements](#requirements) are not met, the job will silently be skipped. -CAUTION: **Caution:** -Your apps should *not* be manipulated outside of Helm (using Kubernetes directly.) -This can cause confusion with Helm not detecting the change, and subsequent -deploys with Auto DevOps can undo your changes. Also, if you change something -and want to undo it by deploying again, Helm may not detect that anything changed -in the first place, and thus not realize that it needs to re-apply the old config. - [Review Apps][review-app] are temporary application environments based on the branch's code so developers, designers, QA, product managers, and other reviewers can actually see and interact with code changes as part of the review @@ -466,6 +459,24 @@ up in the merge request widget for easy discovery. When the branch or tag is del for example after the merge request is merged, the Review App will automatically be deleted. +Review apps are deployed using the +[auto-deploy-app](https://gitlab.com/charts/auto-deploy-app) chart with +Helm. The app will be deployed into the [Kubernetes +namespace](../../user/project/clusters/index.md#deployment-variables) +for the environment. + +Since GitLab 11.4, a [local +Tiller](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22036) is +used. Previous versions of GitLab had a Tiller installed in the project +namespace. + +CAUTION: **Caution:** +Your apps should *not* be manipulated outside of Helm (using Kubernetes directly). +This can cause confusion with Helm not detecting the change and subsequent +deploys with Auto DevOps can undo your changes. Also, if you change something +and want to undo it by deploying again, Helm may not detect that anything changed +in the first place, and thus not realize that it needs to re-apply the old config. + ### Auto DAST **[ULTIMATE]** > Introduced in [GitLab Ultimate][ee] 10.4. @@ -504,13 +515,6 @@ This is an optional step, since many projects do not have a Kubernetes cluster available. If the [requirements](#requirements) are not met, the job will silently be skipped. -CAUTION: **Caution:** -Your apps should *not* be manipulated outside of Helm (using Kubernetes directly.) -This can cause confusion with Helm not detecting the change, and subsequent -deploys with Auto DevOps can undo your changes. Also, if you change something -and want to undo it by deploying again, Helm may not detect that anything changed -in the first place, and thus not realize that it needs to re-apply the old config. - After a branch or merge request is merged into the project's default branch (usually `master`), Auto Deploy deploys the application to a `production` environment in the Kubernetes cluster, with a namespace based on the project name and unique @@ -523,6 +527,24 @@ enable them. You can make use of [environment variables](#environment-variables) to automatically scale your pod replicas. +Apps are deployed using the +[auto-deploy-app](https://gitlab.com/charts/auto-deploy-app) chart with +Helm. The app will be deployed into the [Kubernetes +namespace](../../user/project/clusters/index.md#deployment-variables) +for the environment. + +Since GitLab 11.4, a [local +Tiller](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22036) is +used. Previous versions of GitLab had a Tiller installed in the project +namespace. + +CAUTION: **Caution:** +Your apps should *not* be manipulated outside of Helm (using Kubernetes directly). +This can cause confusion with Helm not detecting the change and subsequent +deploys with Auto DevOps can undo your changes. Also, if you change something +and want to undo it by deploying again, Helm may not detect that anything changed +in the first place, and thus not realize that it needs to re-apply the old config. + > [Introduced][ce-19507] in GitLab 11.0. For internal and private projects a [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token) @@ -647,12 +669,12 @@ repo or by specifying a project variable: file in it, Auto DevOps will detect the chart and use it instead of the [default one](https://gitlab.com/charts/auto-deploy-app). This can be a great way to control exactly how your application is deployed. -- **Project variable** - Create a [project variable](../../ci/variables/README.md#variables) +- **Project variable** - Create a [project variable](../../ci/variables/README.md#gitlab-cicd-environment-variables) `AUTO_DEVOPS_CHART` with the URL of a custom chart to use or create two project variables `AUTO_DEVOPS_CHART_REPOSITORY` with the URL of a custom chart repository and `AUTO_DEVOPS_CHART` with the path to the chart. ### Custom Helm chart per environment **[PREMIUM]** -You can specify the use of a custom Helm chart per environment by scoping the environment variable +You can specify the use of a custom Helm chart per environment by scoping the environment variable to the desired environment. See [Limiting environment scopes of variables](https://docs.gitlab.com/ee/ci/variables/#limiting-environment-scopes-of-variables-premium). ### Customizing `.gitlab-ci.yml` @@ -677,6 +699,21 @@ renaming `.staging` to `staging`. Then make sure to uncomment the `when` key of the `production` job to turn it into a manual action instead of deploying automatically. +### Using components of Auto-DevOps + +If you only require a subset of the features offered by Auto-DevOps, you can include +individual Auto-DevOps jobs into your own `.gitlab-ci.yml`. + +For example, to make use of [Auto Build](#auto-build), you can add the following to +your `.gitlab-ci.yml`: + +```yaml +include: + - template: Jobs/Build.gitlab-ci.yml +``` + +Consult the [Auto DevOps template] for information on available jobs. + ### PostgreSQL database support In order to support applications that require a database, @@ -733,7 +770,7 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac TIP: **Tip:** Set up the replica variables using a -[project variable](../../ci/variables/README.md#variables) +[project variable](../../ci/variables/README.md#gitlab-cicd-environment-variables) and scale your application by just redeploying it! CAUTION: **Caution:** @@ -901,7 +938,7 @@ increasing the rollout up to 100%. If `INCREMENTAL_ROLLOUT_MODE` is set to `manual` in your project, then instead of the standard `production` job, 4 different -[manual jobs](../../ci/pipelines.md#manual-actions-from-the-pipeline-graph) +[manual jobs](../../ci/pipelines.md#manual-actions-from-pipeline-graphs) will be created: 1. `rollout 10%` @@ -918,7 +955,7 @@ required to go from `10%` to `100%`, you can jump to whatever job you want. You can also scale down by running a lower percentage job, just before hitting `100%`. Once you get to `100%`, you cannot scale down, and you'd have to roll back by redeploying the old version using the -[rollback button](../../ci/environments.md#rolling-back-changes) in the +[rollback button](../../ci/environments.md#retrying-and-rolling-back) in the environment page. Below, you can see how the pipeline will look if the rollout or staging @@ -1022,10 +1059,9 @@ planned for a subsequent release. buildpack](#custom-buildpacks). - Auto Test may fail because of a mismatch between testing frameworks. In this case, you may need to customize your `.gitlab-ci.yml` with your test commands. -- Auto Deploy may fail if it is unable to create a Kubernetes namespace and - service account for your project. See the - [troubleshooting failed deployments](../../user/project/clusters/index.md#troubleshooting-failed-deployment-jobs) - section to debug why these resources were not created. +- Auto Deploy will fail if GitLab can not create a Kubernetes namespace and + service account for your project. For help debugging this issue, see + [Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting-failed-deployment-jobs). ### Disable the banner instance wide diff --git a/doc/university/glossary/README.md b/doc/university/glossary/README.md index 254e234a22c..0af2f8d2f54 100644 --- a/doc/university/glossary/README.md +++ b/doc/university/glossary/README.md @@ -41,7 +41,7 @@ Objects (usually binary and large) created by a build process. These can include ### Atlassian -A [company](https://www.atlassian.com) that develops software products for developers and project managers including Bitbucket, Jira, Confluence, Bamboo. +A [company](https://www.atlassian.com) that develops software products for developers and project managers including Bitbucket, Jira, Hipchat, Confluence, Bamboo. ### Audit Log diff --git a/doc/university/high-availability/aws/README.md b/doc/university/high-availability/aws/README.md index 1906655dfa5..01d5073ab7e 100644 --- a/doc/university/high-availability/aws/README.md +++ b/doc/university/high-availability/aws/README.md @@ -83,7 +83,7 @@ our newly created VPC. ### Internet Gateway Now still on the same dashboard head over to Internet Gateways and -create a new one. After its created pres on the `Attach to VPC` button and +create a new one. After its created press on the `Attach to VPC` button and select our VPC.  diff --git a/doc/university/high-availability/aws/img/reference-arch2.png b/doc/university/high-availability/aws/img/reference-arch2.png Binary files differindex a9cb6663103..0f8790d0f74 100644 --- a/doc/university/high-availability/aws/img/reference-arch2.png +++ b/doc/university/high-availability/aws/img/reference-arch2.png diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index 350072186ee..b7f7d71689d 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -1,31 +1,58 @@ --- -last_updated: 2018-02-07 +last_updated: 2019-03-27 --- # Migrating from MySQL to PostgreSQL -> **Note:** This guide assumes you have a working GitLab instance with -> MySQL and want to migrate to bundled PostgreSQL database. +This guide documents how to take a working GitLab instance that uses MySQL and +migrate it to a PostgreSQL database. -## Omnibus installation +## Requirements -### Prerequisites +[pgloader](http://pgloader.io) 3.4.1+ is required. -First, we'll need to enable the bundled PostgreSQL database with up-to-date -schema. Next, we'll use [pgloader](http://pgloader.io) to migrate the data -from the old MySQL database to the new PostgreSQL one. +You can install it directly from your distribution, for example in +Debian/Ubuntu: -Here's what you'll need to have installed: +1. Search for the version: -- pgloader 3.4.1+ -- Omnibus GitLab -- MySQL + ```bash + apt-cache madison pgloader + ``` -### Enable bundled PostgreSQL database +1. If the version is 3.4.1+, install it with: + + ```bash + sudo apt-get install pgloader + ``` + + If your distribution's version is too old, use PostgreSQL's repository: + + ```bash + # Add repository + sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' + + # Add key + sudo apt-get install wget ca-certificates + wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - + + # Install package + sudo apt-get update + sudo apt-get install pgloader + ``` + +For other distributions, follow the instructions in PostrgreSQL's +[download page](https://www.postgresql.org/download/) to add their repository +and then install `pgloader`. + +## Omnibus GitLab installations + +For [Omnibus GitLab packages](https://about.gitlab.com/install/), you'll first +need to enable the bundled PostgreSQL: 1. Stop GitLab: - ``` bash + ```bash sudo gitlab-ctl stop ``` @@ -40,39 +67,34 @@ Here's what you'll need to have installed: and alike. You could just comment all of them out so that we'll just use the defaults. -1. [Reconfigure GitLab] for the changes to take effect: - - ``` bash - sudo gitlab-ctl reconfigure - ``` - +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. 1. Start Unicorn and PostgreSQL so that we can prepare the schema: - ``` bash + ```bash sudo gitlab-ctl start unicorn sudo gitlab-ctl start postgresql ``` 1. Run the following commands to prepare the schema: - ``` bash + ```bash sudo gitlab-rake db:create db:migrate ``` 1. Stop Unicorn to prevent other database access from interfering with the loading of data: - ``` bash + ```bash sudo gitlab-ctl stop unicorn ``` After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. -### Migrate data from MySQL to PostgreSQL - -Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: +Next, we'll use `pgloader` to migrate the data from the old MySQL database to the +new PostgreSQL one: 1. Save the following snippet in a `commands.load` file, and edit with your - database `username`, `password` and `host`: + MySQL database `username`, `password` and `host`: ``` LOAD DATABASE @@ -90,7 +112,7 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: 1. Start the migration: - ``` bash + ```bash sudo -u gitlab-psql pgloader commands.load ``` @@ -117,170 +139,140 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: Total import time 1894 1894 0 12.497s ``` - If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See - the [troubleshooting guide](#Troubleshooting) for more details. + If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See + the [troubleshooting guide](#troubleshooting) for more details. 1. Start GitLab: - ``` bash + ```bash sudo gitlab-ctl start ``` -Now, you can verify that everything worked by visiting GitLab. - -### Troubleshooting - -#### Permissions - -Note that the PostgreSQL user that you use for the above MUST have **superuser** privileges. Otherwise, you may see -a similar message to the following: - -``` -debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread - #<THREAD "lparallel" RUNNING {10078A3513}>: - Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger - QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL; - 2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger - QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL; -``` - -#### Experiencing 500 errors after the migration - -If you experience 500 errors after the migration, try to clear the cache: - -``` bash -sudo gitlab-rake cache:clear -``` - -[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure - -## Source installation +You can now verify that everything works as expected by visiting GitLab. -### Prerequisites +## Source installations -#### Install PostgreSQL and create database +For installations from source that use MySQL, you'll first need to +[install PostgreSQL and create a database](../install/installation.md#6-database). -See [installation guide](../install/installation.md#6-database). - -#### Install [pgloader](http://pgloader.io) 3.4.1+ - -Install directly from your distro: -``` bash -sudo apt-get install pgloader -``` - -If this version is too old, use PostgreSQL's repository: -``` bash -# add repository -sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' - -# add key -sudo apt-get install wget ca-certificates -wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - - -# install package -sudo apt-get update -sudo apt-get install pgloader -``` - -### Enable bundled PostgreSQL database +After the database is created, go on with the following steps: 1. Stop GitLab: - ``` bash - sudo service gitlab stop - ``` + ```bash + sudo service gitlab stop + ``` 1. Switch database from MySQL to PostgreSQL - ``` bash - cd /home/git/gitlab - sudo -u git mv config/database.yml config/database.yml.bak - sudo -u git cp config/database.yml.postgresql config/database.yml - sudo -u git -H chmod o-rwx config/database.yml - ``` + ```bash + cd /home/git/gitlab + sudo -u git mv config/database.yml config/database.yml.bak + sudo -u git cp config/database.yml.postgresql config/database.yml + sudo -u git -H chmod o-rwx config/database.yml + ``` + 1. Install Gems related to Postgresql - ``` bash - sudo -u git -H rm .bundle/config - sudo -u git -H bundle install --deployment --without development test mysql aws kerberos - ``` + ```bash + sudo -u git -H rm .bundle/config + sudo -u git -H bundle install --deployment --without development test mysql aws kerberos + ``` 1. Run the following commands to prepare the schema: - ``` bash - sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production - ``` + ```bash + sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production + ``` After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. -### Migrate data from MySQL to PostgreSQL - -Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: +Next, we'll use `pgloader` to migrate the data from the old MySQL database to the +new PostgreSQL one: 1. Save the following snippet in a `commands.load` file, and edit with your MySQL `username`, `password` and `host`: - ``` - LOAD DATABASE - FROM mysql://username:password@host/gitlabhq_production - INTO postgresql://postgres@unix://var/run/postgresql:/gitlabhq_production + ``` + LOAD DATABASE + FROM mysql://username:password@host/gitlabhq_production + INTO postgresql://postgres@unix://var/run/postgresql:/gitlabhq_production - WITH include no drop, truncate, disable triggers, create no tables, - create no indexes, preserve index names, no foreign keys, - data only + WITH include no drop, truncate, disable triggers, create no tables, + create no indexes, preserve index names, no foreign keys, + data only - ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public' + ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public' - ; - ``` + ; + ``` 1. Start the migration: - ``` bash - sudo -u postgres pgloader commands.load - ``` + ```bash + sudo -u postgres pgloader commands.load + ``` 1. Once the migration finishes, you should see a summary table that looks like the following: - ``` - table name read imported errors total time - ----------------------------------------------- --------- --------- --------- -------------- - fetch meta data 119 119 0 0.388s - Truncate 119 119 0 1.134s - ----------------------------------------------- --------- --------- --------- -------------- - public.abuse_reports 0 0 0 0.490s - public.appearances 0 0 0 0.488s - . - . - . - public.web_hook_logs 0 0 0 1.080s - ----------------------------------------------- --------- --------- --------- -------------- - COPY Threads Completion 4 4 0 2.008s - Reset Sequences 113 113 0 0.304s - Install Comments 0 0 0 0.000s - ----------------------------------------------- --------- --------- --------- -------------- - Total import time 1894 1894 0 12.497s - ``` - - If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See - the [troubleshooting guide](#Troubleshooting) for more details. + ``` + table name read imported errors total time + ----------------------------------------------- --------- --------- --------- -------------- + fetch meta data 119 119 0 0.388s + Truncate 119 119 0 1.134s + ----------------------------------------------- --------- --------- --------- -------------- + public.abuse_reports 0 0 0 0.490s + public.appearances 0 0 0 0.488s + . + . + . + public.web_hook_logs 0 0 0 1.080s + ----------------------------------------------- --------- --------- --------- -------------- + COPY Threads Completion 4 4 0 2.008s + Reset Sequences 113 113 0 0.304s + Install Comments 0 0 0 0.000s + ----------------------------------------------- --------- --------- --------- -------------- + Total import time 1894 1894 0 12.497s + ``` + + If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See + the [troubleshooting guide](#troubleshooting) for more details. 1. Start GitLab: - ``` bash - sudo service gitlab start - ``` + ```bash + sudo service gitlab start + ``` + +You can now verify that everything works as expected by visiting GitLab. + +## Troubleshooting + +Sometimes, you might encounter some errors during or after the migration. -Now, you can verify that everything worked by visiting GitLab. +### Database error permission denied -### Troubleshooting +The PostgreSQL user that you use for the migration MUST have **superuser** privileges. +Otherwise, you may see a similar message to the following: -#### Experiencing 500 errors after the migration +``` +debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread + #<THREAD "lparallel" RUNNING {10078A3513}>: + Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger + QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL; + 2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger + QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL; +``` + +### Experiencing 500 errors after the migration If you experience 500 errors after the migration, try to clear the cache: -``` bash +```bash +# Omnibus GitLab +sudo gitlab-rake cache:clear + +# Installations from source sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production ``` diff --git a/doc/user/discussions/img/multi-line-suggestion-preview.png b/doc/user/discussions/img/multi-line-suggestion-preview.png Binary files differnew file mode 100644 index 00000000000..4288d0ba034 --- /dev/null +++ b/doc/user/discussions/img/multi-line-suggestion-preview.png diff --git a/doc/user/discussions/img/multi-line-suggestion-syntax.png b/doc/user/discussions/img/multi-line-suggestion-syntax.png Binary files differnew file mode 100644 index 00000000000..df0c99b84ef --- /dev/null +++ b/doc/user/discussions/img/multi-line-suggestion-syntax.png diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 23b9604a456..bf41fdecd10 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -344,6 +344,24 @@ and push the suggested change directly into the codebase in the merge request's Custom commit messages will be introduced by [#54404](https://gitlab.com/gitlab-org/gitlab-ce/issues/54404). +### Multi-line suggestions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53310) in GitLab 11.10. + +Reviewers can also suggest changes to +multiple lines with a single suggestion within Merge Request diff discussions. + + + +In the example above, the suggestion covers three lines above and four lines below the commented diff line. +It'd change from 3 lines _above_ to 4 lines _below_ the commented Diff line. + + + +NOTE: **Note:** +Suggestions covering multiple lines are limited to 100 lines _above_ and 100 lines _below_ +the commented diff line, allowing up to 200 changed lines per suggestion. + ## Start a discussion by replying to a standard comment > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30299) in GitLab 11.9 diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 5dc798fd8d3..2fb6cec55fa 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -71,6 +71,14 @@ or over the size limit, you can [reduce your repository size with Git](../projec | ----------- | ----------------- | ------------- | | Repository size including LFS | 10G | Unlimited | +## IP range + +GitLab.com, CI/CD, and related services are deployed into Google Cloud Platform (GCP). Any +IP based firewall can be configured by looking up all +[IP address ranges or CIDR blocks for GCP](https://cloud.google.com/compute/docs/faq#where_can_i_find_product_name_short_ip_ranges). + +[Static endpoints](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5071) are being considered. + ## Shared Runners Shared Runners on GitLab.com run in [autoscale mode] and powered by diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index e67795b9bae..984881ef26c 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -28,6 +28,7 @@ deployments. | [Helm Tiller](https://docs.helm.sh) | 11.6+ | Helm is a package manager for Kubernetes and is required to install all the other applications. It is installed in its own pod inside the cluster which can run the `helm` CLI in a safe environment. | n/a | | [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress) | 11.6+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps](../../../topics/autodevops/index.md) or deploy your own web apps. | [stable/nginx-ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress) | | [Cert-Manager](https://docs.cert-manager.io/en/latest/) | 11.6+ | Cert-Manager is a native Kubernetes certificate management controller that helps with issuing certificates. Installing Cert-Manager on your cluster will issue a certificate by [Let's Encrypt](https://letsencrypt.org/) and ensure that certificates are valid and up-to-date. | [stable/cert-manager](https://github.com/helm/charts/tree/master/stable/cert-manager) | +| [GitLab Runner](https://docs.gitlab.com/runner/) | 11.10+ | GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with [GitLab CI/CD](../../../ci/README.md), the open-source continuous integration service included with GitLab that coordinates the jobs. When installing the GitLab Runner via the applications, it will run in **privileged mode** by default. Make sure you read the [security implications](../../project/clusters/index.md#security-implications) before doing so. | [runner/gitlab-runner](https://gitlab.com/charts/gitlab-runner) | NOTE: **Note:** Some [cluster @@ -35,8 +36,6 @@ applications](../../project/clusters/index.md#installing-applications) are installable only for a project-level cluster. Support for installing these applications in a group-level cluster is planned for future releases. For updates, see: -- Support installing [Runner in group-level - clusters](https://gitlab.com/gitlab-org/gitlab-ce/issues/51988) - Support installing [JupyterHub in group-level clusters](https://gitlab.com/gitlab-org/gitlab-ce/issues/51989) - Support installing [Prometheus in group-level diff --git a/doc/user/group/img/create_new_group_info.png b/doc/user/group/img/create_new_group_info.png Binary files differindex c2e6ed43c5b..bd724240c37 100644 --- a/doc/user/group/img/create_new_group_info.png +++ b/doc/user/group/img/create_new_group_info.png diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 3bcfd30079d..74735886350 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -58,7 +58,7 @@ By doing so: ## Issues and merge requests within a group Issues and merge requests are part of projects. For a given group, view all the -[issues](../project/issues/index.md#issues-per-group) and [merge requests](../project/merge_requests/index.md#merge-requests-per-group) across all the projects in that group, +[issues](../project/issues/index.md#issues-list) and [merge requests](../project/merge_requests/index.md#merge-requests-per-group) across all the projects in that group, together in a single list view. ## Create a new group @@ -151,6 +151,17 @@ There are two different ways to add a new project to a group:  +### Default project creation level + +Group owners or administrators can allow users with the +Developer role to create projects under groups. + +By default, [Developers and Maintainers](../permissions.md#group-members-permissions) can create projects under agroup, but this can be changed either within the group settings for a group, or +be set globally by a GitLab administrator in the Admin area +at **Settings > General > Visibility and access controls**. + +Available settings are `No one`, `Maintainers`, or `Developers + Maintainers`. + ## Transfer projects into groups Learn how to [transfer a project into a group](../project/settings/index.md#transferring-an-existing-project-into-another-namespace). diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 3cecefe11f5..4e81e28a45a 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -167,7 +167,7 @@ Here's a list of what you can't do with subgroups: - [GitLab Pages](../../project/pages/index.md) supports projects hosted under a subgroup, but not subgroup websites. That means that only the highest-level group supports - [group websites](../../project/pages/introduction.html#user-or-group-pages), + [group websites](../../project/pages/getting_started_part_one.md#gitlab-pages-domain-names), although you can have project websites under a subgroup. - It is not possible to share a project with a group that's an ancestor of the group the project is in. That means you can only share as you walk down diff --git a/doc/user/img/color_inline_colorchip_render_gfm.png b/doc/user/img/color_inline_colorchip_render_gfm.png Binary files differindex fed8ca5c34b..6f93dbeeb10 100644 --- a/doc/user/img/color_inline_colorchip_render_gfm.png +++ b/doc/user/img/color_inline_colorchip_render_gfm.png diff --git a/doc/user/img/task_list_ordered_render_gfm.png b/doc/user/img/task_list_ordered_render_gfm.png Binary files differindex 0905a8378be..98ec791e958 100644 --- a/doc/user/img/task_list_ordered_render_gfm.png +++ b/doc/user/img/task_list_ordered_render_gfm.png diff --git a/doc/user/img/unordered_check_list_render_gfm.png b/doc/user/img/unordered_check_list_render_gfm.png Binary files differindex ccdeab6e62c..2ce0fb95645 100644 --- a/doc/user/img/unordered_check_list_render_gfm.png +++ b/doc/user/img/unordered_check_list_render_gfm.png diff --git a/doc/user/index.md b/doc/user/index.md index d408504249e..8164b31c37e 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -38,8 +38,8 @@ GitLab is a Git-based platform that integrates a great number of essential tools - Hosting code in repositories with version control. - Tracking proposals for new implementations, bug reports, and feedback with a - fully featured [Issue Tracker](project/issues/index.md#issue-tracker). -- Organizing and prioritizing with [Issue Boards](project/issues/index.md#issue-board). + fully featured [Issue Tracker](project/issues/index.md#issues-list). +- Organizing and prioritizing with [Issue Boards](project/issues/index.md#issue-boards). - Reviewing code in [Merge Requests](project/merge_requests/index.md) with live-preview changes per branch with [Review Apps](../ci/review_apps/index.md). - Building, testing, and deploying with built-in [Continuous Integration](../ci/README.md). @@ -65,9 +65,7 @@ With GitLab Enterprise Edition, you can also: - View the current health and status of each CI environment running on Kubernetes with [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html). - Leverage continuous delivery method with [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html). -You can also [integrate](project/integrations/project_services.md) GitLab with -numerous third-party applications, such as Mattermost, Microsoft Teams, Trello, -Slack, Bamboo CI, JIRA, and a lot more. +You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, JIRA, and a lot more. ## Projects diff --git a/doc/user/markdown.md b/doc/user/markdown.md index d8bc3a9187e..9891a43aa61 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -267,10 +267,7 @@ However the wrapping tags cannot be mixed as such: ### Emoji -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#emoji - -``` +```md Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: :zap: You can use emoji anywhere GFM is supported. :v: @@ -288,15 +285,15 @@ On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/he Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. ``` -Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/monkey.png" width="20px" height="20px"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/star2.png" width="20px" height="20px"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/speech_balloon.png" width="20px" height="20px">. Well we have a gift for you: +Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/monkey.png" width="20px" height="20px" style="display:inline;margin:0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/star2.png" width="20px" height="20px" style="display:inline;margin:0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0">. Well we have a gift for you: -<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/zap.png" width="20px" height="20px">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/v.png" width="20px" height="20px"> +<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/zap.png" width="20px" height="20px" style="display:inline;margin:0">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/v.png" width="20px" height="20px" style="display:inline;margin:0"> -You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/bug.png" width="20px" height="20px"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/speak_no_evil.png" width="20px" height="20px"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/snail.png" width="20px" height="20px"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/birthday.png" width="20px" height="20px">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/heart.png" width="20px" height="20px"> you for that. +You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/bug.png" width="20px" height="20px" style="display:inline;margin:0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/snail.png" width="20px" height="20px" style="display:inline;margin:0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/birthday.png" width="20px" height="20px" style="display:inline;margin:0">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/heart.png" width="20px" height="20px" style="display:inline;margin:0"> you for that. -If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/fearful.png" width="20px" height="20px">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/family.png" width="20px" height="20px">. All you need to do is to look up one of the supported codes. +If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/fearful.png" width="20px" height="20px" style="display:inline;margin:0">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/family.png" width="20px" height="20px" style="display:inline;margin:0">. All you need to do is to look up one of the supported codes. -Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/thumbsup.png" width="20px" height="20px"> +Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/public/-/emojis/1/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0"> Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. @@ -581,11 +578,11 @@ Alt-H2 ------ ``` -### Header IDs and links +#### Header IDs and links -All Markdown-rendered headers automatically get IDs, except in comments. +All Markdown-rendered headers automatically get IDs, which can be linked to, except in comments. -On hover, a link to those IDs becomes visible to make it easier to copy the link to the header to give it to someone else. +On hover, a link to those IDs becomes visible to make it easier to copy the link to the header to use it somewhere else. The IDs are generated from the content of the header according to the following rules: @@ -612,8 +609,8 @@ Would generate the following link IDs: 1. `this-header-has-spaces-in-it` 1. `this-header-has-a-in-it` 1. `this-header-has-unicode-in-it-한글` -1. `this-header-has-spaces-in-it` 1. `this-header-has-spaces-in-it-1` +1. `this-header-has-spaces-in-it-2` 1. `this-header-has-3-5-in-it-and-parentheses` Note that the Emoji processing happens before the header IDs are generated, so the Emoji is converted to an image which then gets removed from the ID. @@ -718,25 +715,25 @@ Becomes: There are two ways to create links, inline-style and reference-style. - [I'm an inline-style link](https://www.google.com) - - [I'm a reference-style link][Arbitrary case-insensitive reference text] - - [I'm a relative reference to a repository file](LICENSE) - - [I am an absolute reference within the repository](/doc/user/markdown.md) - - [I link to the Milestones page](/../milestones) +```markdown +[I'm an inline-style link](https://www.google.com) +[I'm a link to a repository file in the same directory](index.md) +[I am an absolute reference within the repository](/doc/user/index.md) +[I'm a relative link to the Milestones page](../README.md) - [You can use numbers for reference-style link definitions][1] +[I link to a section on a different markdown page, using a header ID](index.md#overview) +[I link to a different section on the same page, using the header ID](#header-ids-and-links) - Or leave it empty and use the [link text itself][] +[I'm a reference-style link][Arbitrary case-insensitive reference text] +[You can use numbers for reference-style link definitions][1] +Or leave it empty and use the [link text itself][] - Some text to show that the reference links can follow later. +Some text to show that the reference links can follow later. - [arbitrary case-insensitive reference text]: https://www.mozilla.org - [1]: http://slashdot.org - [link text itself]: https://www.reddit.com +[arbitrary case-insensitive reference text]: https://www.mozilla.org +[1]: http://slashdot.org +[link text itself]: https://www.reddit.com +``` >**Note:** Relative links do not allow referencing project files in a wiki page or wiki diff --git a/doc/user/permissions.md b/doc/user/permissions.md index adc0f4d568b..a340dd063e4 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -119,7 +119,7 @@ The following table depicts the various user permission levels in a project. | Force push to protected branches [^4] | | | | | | | Remove protected branches [^4] | | | | | | | View project Audit Events | | | | ✓ | ✓ | -| View project statistics | | | | ✓ | ✓ | +| View project statistics | | ✓ | ✓ | ✓ | ✓ | ## Project features permissions diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index b74bd81d467..df413a11af0 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -18,9 +18,10 @@ the second factor of authentication. Once enabled, in addition to supplying your password to login, you'll be prompted to activate your U2F device (usually by pressing a button on it), and it will perform secure authentication on your behalf. -The U2F workflow is only [supported by](https://caniuse.com/#search=U2F) Google Chrome, Opera and Firefox at this point, so we _strongly_ recommend -that you set up both methods of two-factor authentication, so you can still access your account -from other browsers. +The U2F workflow is [supported by](https://caniuse.com/#search=U2F) Google Chrome, Opera, and Firefox. + +We recommend that you set up 2FA with both a [one-time password authenticator](#enable-2fa-via-one-time-password-authenticator) and a [U2F device](#enable-2fa-via-u2f-device), so you can still access your account +if you lose your U2F device. ## Enabling 2FA @@ -160,8 +161,8 @@ a new set of recovery codes with SSH. 1. Run `ssh git@gitlab.example.com 2fa_recovery_codes`. 1. You are prompted to confirm that you want to generate new codes. Continuing this process invalidates previously saved codes. - ``` - bash + + ```sh $ ssh git@gitlab.example.com 2fa_recovery_codes Are you sure you want to generate new two-factor recovery codes? Any existing recovery codes you saved will be invalidated. (yes/no) @@ -206,17 +207,17 @@ Sign in and re-enable two-factor authentication as soon as possible. - You need to take special care to that 2FA keeps working after [restoring a GitLab backup](../../../raketasks/backup_restore.md). - To ensure 2FA authorizes correctly with TOTP server, you may want to ensure - your GitLab server's time is synchronized via a service like NTP. Otherwise, + your GitLab server's time is synchronized via a service like NTP. Otherwise, you may have cases where authorization always fails because of time differences. - The GitLab U2F implementation does _not_ work when the GitLab instance is accessed from multiple hostnames, or FQDNs. Each U2F registration is linked to the _current hostname_ at the time of registration, and cannot be used for other hostnames/FQDNs. - For example, if a user is trying to access a GitLab instance from `first.host.xyz` and `second.host.xyz`: + For example, if a user is trying to access a GitLab instance from `first.host.xyz` and `second.host.xyz`: - - The user logs in via `first.host.xyz` and registers their U2F key. - - The user logs out and attempts to log in via `first.host.xyz` - U2F authentication succeeds. - - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because + - The user logs in via `first.host.xyz` and registers their U2F key. + - The user logs out and attempts to log in via `first.host.xyz` - U2F authentication succeeds. + - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because the U2F key has only been registered on `first.host.xyz`. [Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index b216b9f255c..61a30a775b0 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -33,6 +33,7 @@ From there, you can: - Set a [custom status](#current-status) for your profile - Manage your [commit email](#commit-email) for your profile - Manage [2FA](account/two_factor_authentication.md) +- Add details of [external accounts](#add-details-of-external-accounts). - Change your username and [delete your account](account/delete_account.md) - Manage applications that can [use GitLab as an OAuth provider](../../integration/oauth_provider.md#introduction-to-oauth) @@ -92,6 +93,16 @@ To enable private profile: NOTE: **Note:** You and GitLab admins can see your the abovementioned information on your profile even if it is private. +## Add details of external accounts + +GitLab allows you to add links to certain other external accounts you might have, like Skype and Twitter. They can help other users connect with you on other platforms. + +To add links to other accounts: + +1. Navigate to your **User Settings > Profile**. +1. In the **Main settings** section, locate and fill out fields for links to external accounts like Skype and Twitter. +1. Click the **Update profile settings** button. + ## Private contributions > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/14078) in GitLab 11.3. diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 3a4d09c35d9..52b4a72e688 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -5,12 +5,9 @@ Personal access tokens are the preferred way for third party applications and scripts to authenticate with the [GitLab API][api], if using [OAuth2](../../api/oauth2.md) is not practical. -You can also use them to authenticate against Git over HTTP. They are the only -accepted method of authentication when you have -[Two-Factor Authentication (2FA)][2fa] enabled. +You can also use personal access tokens to authenticate against Git over HTTP or SSH. They must be used when you have [Two-Factor Authentication (2FA)][2fa] enabled. Authenticate with a token in place of your password. -Once you have your token, [pass it to the API][usage] using either the -`private_token` parameter or the `Private-Token` header. +To make [authenticated requests to the API][usage], use either the `private_token` parameter or the `Private-Token` header. The expiration of personal access tokens happens on the date you define, at midnight UTC. diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 5a74ac96e83..1983513174c 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -101,14 +101,20 @@ To add an existing Kubernetes cluster to your project: It's the URL that GitLab uses to access the Kubernetes API. Kubernetes exposes several APIs, we want the "base" URL that is common to all of them, e.g., `https://kubernetes.example.com` rather than `https://kubernetes.example.com/api/v1`. + + Get the API URL by running this command: + + ```sh + kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}' + ``` - **CA certificate** (required) - A valid Kubernetes certificate is needed to authenticate to the EKS cluster. We will use the certificate created by default. - - List the secrets with `kubectl get secrets`, and one should named similar to + - List the secrets with `kubectl get secrets`, and one should named similar to `default-token-xxxxx`. Copy that token name for use below. - - Get the certificate by running this command: + - Get the certificate by running this command: - ```sh - kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode - ``` + ```sh + kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode + ``` - **Token** - GitLab authenticates against Kubernetes using service tokens, which are scoped to a particular `namespace`. @@ -124,23 +130,7 @@ To add an existing Kubernetes cluster to your project: metadata: name: gitlab-admin namespace: kube-system - ``` - - 2. Apply the service account to your cluster: - - ```bash - kubectl apply -f gitlab-admin-service-account.yaml - ``` - - Output: - - ```bash - serviceaccount "gitlab-admin" created - ``` - - 3. Create a file called `gitlab-admin-cluster-role-binding.yaml` with contents: - - ```yaml + --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: @@ -155,41 +145,42 @@ To add an existing Kubernetes cluster to your project: namespace: kube-system ``` - 4. Apply the cluster role binding to your cluster: + 1. Apply the service account and cluster role binding to your cluster: ```bash - kubectl apply -f gitlab-admin-cluster-role-binding.yaml + kubectl apply -f gitlab-admin-service-account.yaml ``` Output: ```bash + serviceaccount "gitlab-admin" created clusterrolebinding "gitlab-admin" created ``` - 5. Retrieve the token for the `gitlab-admin` service account: + 1. Retrieve the token for the `gitlab-admin` service account: ```bash kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep gitlab-admin | awk '{print $1}') ``` - Copy the `<authentication_token>` value from the output: + Copy the `<authentication_token>` value from the output: - ```yaml - Name: gitlab-admin-token-b5zv4 - Namespace: kube-system - Labels: <none> - Annotations: kubernetes.io/service-account.name=gitlab-admin - kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 + ```yaml + Name: gitlab-admin-token-b5zv4 + Namespace: kube-system + Labels: <none> + Annotations: kubernetes.io/service-account.name=gitlab-admin + kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 - Type: kubernetes.io/service-account-token + Type: kubernetes.io/service-account-token - Data - ==== - ca.crt: 1025 bytes - namespace: 11 bytes - token: <authentication_token> - ``` + Data + ==== + ca.crt: 1025 bytes + namespace: 11 bytes + token: <authentication_token> + ``` NOTE: **Note:** For GKE clusters, you will need the @@ -212,14 +203,6 @@ To add an existing Kubernetes cluster to your project: After a couple of minutes, your cluster will be ready to go. You can now proceed to install some [pre-defined applications](#installing-applications). -To determine the: - -- API URL, run `kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}'`. -- Token: - 1. List the secrets by running: `kubectl get secrets`. Note the name of the secret you need the token for. - 1. Get the token for the appropriate secret by running: `kubectl get secret <SECRET_NAME> -o jsonpath="{['data']['token']}" | base64 --decode`. -- CA certificate, run `kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode`. - ## Security implications CAUTION: **Important:** @@ -331,13 +314,7 @@ install it manually. ## Installing applications -NOTE: **Note:** -Before starting the installation of applications, make sure that time is synchronized -between your GitLab server and your Kubernetes cluster. Otherwise, installation could fail -and you may get errors like `Error: remote error: tls: bad certificate` -in the `stdout` of pods created by GitLab in your Kubernetes cluster. - -GitLab provides a one-click install for various applications which can +GitLab provides **GitLab Managed Apps**, a one-click install for various applications which can be added directly to your configured cluster. Those applications are needed for [Review Apps](../../../ci/review_apps/index.md) and [deployments](../../../ci/environments.md). You can install them after you @@ -360,8 +337,8 @@ by GitLab before installing any of the applications. | [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) | 10.2+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps] or deploy your own web apps. | [stable/nginx-ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress) | | [Cert-Manager](https://docs.cert-manager.io/en/latest/) | 11.6+ | Cert-Manager is a native Kubernetes certificate management controller that helps with issuing certificates. Installing Cert-Manager on your cluster will issue a certificate by [Let's Encrypt](https://letsencrypt.org/) and ensure that certificates are valid and up-to-date. | [stable/cert-manager](https://github.com/helm/charts/tree/master/stable/cert-manager) | | [Prometheus](https://prometheus.io/docs/introduction/overview/) | 10.4+ | Prometheus is an open-source monitoring and alerting system useful to supervise your deployed applications. | [stable/prometheus](https://github.com/helm/charts/tree/master/stable/prometheus) | -| [GitLab Runner](https://docs.gitlab.com/runner/) | 10.6+ | GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/), the open-source continuous integration service included with GitLab that coordinates the jobs. When installing the GitLab Runner via the applications, it will run in **privileged mode** by default. Make sure you read the [security implications](#security-implications) before doing so. | [runner/gitlab-runner](https://gitlab.com/charts/gitlab-runner) | -| [JupyterHub](http://jupyter.org/) | 11.0+ | [JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a multi-user service for managing notebooks across a team. [Jupyter Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) provide a web-based interactive programming environment used for data analysis, visualization, and machine learning. We use a [custom Jupyter image](https://gitlab.com/gitlab-org/jupyterhub-user-image/blob/master/Dockerfile) that installs additional useful packages on top of the base Jupyter. Authentication will be enabled only for [project members](../members/index.md) with [Developer or higher](../../permissions.md) access to the project. You will also see ready-to-use DevOps Runbooks built with Nurtch's [Rubix library](https://github.com/amit1rrr/rubix). More information on creating executable runbooks can be found in [our Nurtch documentation](runbooks/index.md#nurtch-executable-runbooks). | [jupyter/jupyterhub](https://jupyterhub.github.io/helm-chart/) | +| [GitLab Runner](https://docs.gitlab.com/runner/) | 10.6+ | GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with [GitLab CI/CD](../../../ci/README.md), the open-source continuous integration service included with GitLab that coordinates the jobs. When installing the GitLab Runner via the applications, it will run in **privileged mode** by default. Make sure you read the [security implications](#security-implications) before doing so. | [runner/gitlab-runner](https://gitlab.com/charts/gitlab-runner) | +| [JupyterHub](http://jupyter.org/) | 11.0+ | [JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a multi-user service for managing notebooks across a team. [Jupyter Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) provide a web-based interactive programming environment used for data analysis, visualization, and machine learning. We use a [custom Jupyter image](https://gitlab.com/gitlab-org/jupyterhub-user-image/blob/master/Dockerfile) that installs additional useful packages on top of the base Jupyter. Authentication will be enabled only for [project members](../members/index.md) with [Developer or higher](../../permissions.md) access to the project. You will also see ready-to-use DevOps Runbooks built with Nurtch's [Rubix library](https://github.com/amit1rrr/rubix). More information on creating executable runbooks can be found in [our Nurtch documentation](runbooks/index.md#nurtch-executable-runbooks). Note that Ingress must be installed and have an IP address assigned before JupyterHub can be installed. | [jupyter/jupyterhub](https://jupyterhub.github.io/helm-chart/) | | [Knative](https://cloud.google.com/knative) | 11.5+ | Knative provides a platform to create, deploy, and manage serverless workloads from a Kubernetes cluster. It is used in conjunction with, and includes [Istio](https://istio.io) to provide an external IP address for all programs hosted by Knative. You will be prompted to enter a wildcard domain where your applications will be exposed. Configure your DNS server to use the external IP address for that domain. For any application created and installed, they will be accessible as `<program_name>.<kubernetes_namespace>.<domain_name>`. This will require your kubernetes cluster to have [RBAC enabled](#role-based-access-control-rbac). | [knative/knative](https://storage.googleapis.com/triggermesh-charts) With the exception of Knative, the applications will be installed in a dedicated @@ -395,6 +372,29 @@ Upgrades will reset values back to the values built into the `runner` chart plus the values set by [`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/runner/values.yaml) +### Troubleshooting applications + +Applications can fail with the following error: + +```text +Error: remote error: tls: bad certificate +``` + +To avoid installation errors: + +- Before starting the installation of applications, make sure that time is synchronized + between your GitLab server and your Kubernetes cluster. +- Ensure certificates are not out of sync. When installing applications, GitLab expects a new cluster with no previous installation of Tiller. + + You can confirm that the certificates match via `kubectl`: + + ```sh + kubectl get configmaps/values-content-configuration-ingress -n gitlab-managed-apps -o \ + "jsonpath={.data['cert\.pem']}" | base64 -d > a.pem + kubectl get secrets/tiller-secret -n gitlab-managed-apps -o "jsonpath={.data['ca\.crt']}" | base64 -d > b.pem + diff a.pem b.pem + ``` + ## Getting the external endpoint NOTE: **Note:** @@ -406,14 +406,20 @@ to obtain the endpoint. You can use either In order to publish your web application, you first need to find the endpoint which will be either an IP address or a hostname associated with your load balancer. -### Let GitLab fetch the external endpoint +### Automatically determining the external endpoint > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17052) in GitLab 10.6. -If you [installed Ingress or Knative](#installing-applications), -you should see the Ingress Endpoint on this same page within a few minutes. -If you don't see this, GitLab might not be able to determine the external endpoint of -your ingress application in which case you should manually determine it. +After you install [Ingress or Knative](#installing-applications), Gitlab attempts to determine the external endpoint +and it should be available within a few minutes. If the endpoint doesn't appear +and your cluster runs on Google Kubernetes Engine: + +1. Check your [Kubernetes cluster on Google Kubernetes Engine](https://console.cloud.google.com/kubernetes) to ensure there are no errors on its nodes. +1. Ensure you have enough [Quotas](https://console.cloud.google.com/iam-admin/quotas) on Google Kubernetes Engine. For more information, see [Resource Quotas](https://cloud.google.com/compute/quotas). +1. Check [Google Cloud's Status](https://status.cloud.google.com/) to ensure they are not having any disruptions. + +If GitLab is still unable to determine the endpoint of your Ingress or Knative application, you can +manually determine it by following the steps below. ### Manually determining the external endpoint @@ -539,7 +545,7 @@ The result will then be: ## Deployment variables The Kubernetes cluster integration exposes the following -[deployment variables](../../../ci/variables/README.md#deployment-variables) in the +[deployment variables](../../../ci/variables/README.md#deployment-environment-variables) in the GitLab CI/CD build environment. | Variable | Description | @@ -559,25 +565,23 @@ service account of the cluster integration. ### Troubleshooting failed deployment jobs GitLab will create a namespace and service account specifically for your -deployment jobs. These resources are created just before the deployment -job starts. Sometimes there may be errors that cause their creation to fail. +deployment jobs, immediately before the jobs starts. -In such instances, your job will fail with the message: +However, sometimes GitLab can not create them. In such instances, your job will fail with the message: -```The job failed to complete prerequisite tasks``` +```text +This job failed because the necessary resources were not successfully created. +``` -You will need to check the [logs](../../../administration/logs.md) to debug -why the namespace and service account creation failed. +To find the cause of this error when creating a namespace and service account, check the [logs](../../../administration/logs.md#kuberneteslog). -A common reason for failure is that the token you gave GitLab did not have -[`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) -privileges as GitLab expects. +Common reasons for failure include: -Another common problem is caused by a missing `KUBECONFIG` or `KUBE_TOKEN`. -To be passed to your job, it must have a matching -[`environment:name`](../../../ci/environments.md#defining-environments). If -your job has no `environment:name` set, it will not be passed the Kubernetes -credentials. +- The token you gave GitLab did not have [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) + privileges required by GitLab. +- Missing `KUBECONFIG` or `KUBE_TOKEN` variables. To be passed to your job, they must have a matching + [`environment:name`](../../../ci/environments.md#defining-environments). If your job has no + `environment:name` set, it will not be passed the Kubernetes credentials. ## Monitoring your Kubernetes cluster **[ULTIMATE]** diff --git a/doc/user/project/clusters/runbooks/img/authorize-jupyter.png b/doc/user/project/clusters/runbooks/img/authorize-jupyter.png Binary files differindex 84cce311483..f3e868de802 100644 --- a/doc/user/project/clusters/runbooks/img/authorize-jupyter.png +++ b/doc/user/project/clusters/runbooks/img/authorize-jupyter.png diff --git a/doc/user/project/clusters/runbooks/img/gitlab-variables.png b/doc/user/project/clusters/runbooks/img/gitlab-variables.png Binary files differindex 1d338f063a9..7c9c9099851 100644 --- a/doc/user/project/clusters/runbooks/img/gitlab-variables.png +++ b/doc/user/project/clusters/runbooks/img/gitlab-variables.png diff --git a/doc/user/project/clusters/runbooks/img/helm-install.png b/doc/user/project/clusters/runbooks/img/helm-install.png Binary files differindex 003e482e756..e67cf317ec1 100644 --- a/doc/user/project/clusters/runbooks/img/helm-install.png +++ b/doc/user/project/clusters/runbooks/img/helm-install.png diff --git a/doc/user/project/clusters/runbooks/img/ingress-install.png b/doc/user/project/clusters/runbooks/img/ingress-install.png Binary files differindex 7edc11d5b45..08256a65138 100644 --- a/doc/user/project/clusters/runbooks/img/ingress-install.png +++ b/doc/user/project/clusters/runbooks/img/ingress-install.png diff --git a/doc/user/project/clusters/runbooks/img/jupyterhub-install.png b/doc/user/project/clusters/runbooks/img/jupyterhub-install.png Binary files differindex 75c6028a763..784e508ff25 100644 --- a/doc/user/project/clusters/runbooks/img/jupyterhub-install.png +++ b/doc/user/project/clusters/runbooks/img/jupyterhub-install.png diff --git a/doc/user/project/clusters/runbooks/img/postgres-query.png b/doc/user/project/clusters/runbooks/img/postgres-query.png Binary files differindex 04315d54d5e..dbb5a62ebba 100644 --- a/doc/user/project/clusters/runbooks/img/postgres-query.png +++ b/doc/user/project/clusters/runbooks/img/postgres-query.png diff --git a/doc/user/project/clusters/runbooks/img/sample-runbook.png b/doc/user/project/clusters/runbooks/img/sample-runbook.png Binary files differindex 70011202bf0..545d34fd4ad 100644 --- a/doc/user/project/clusters/runbooks/img/sample-runbook.png +++ b/doc/user/project/clusters/runbooks/img/sample-runbook.png diff --git a/doc/user/project/clusters/serverless/img/function-details-loaded.png b/doc/user/project/clusters/serverless/img/function-details-loaded.png Binary files differnew file mode 100644 index 00000000000..34465c5c087 --- /dev/null +++ b/doc/user/project/clusters/serverless/img/function-details-loaded.png diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index e6804666e22..5b7e9ef906f 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -11,13 +11,12 @@ Run serverless workloads on Kubernetes using [Knative](https://cloud.google.com/ Knative extends Kubernetes to provide a set of middleware components that are useful to build modern, source-centric, container-based applications. Knative brings some significant benefits out of the box through its main components: -- [Build](https://github.com/knative/build): Source-to-container build orchestration. -- [Eventing](https://github.com/knative/eventing): Management and delivery of events. - [Serving](https://github.com/knative/serving): Request-driven compute that can scale to zero. +- [Eventing](https://github.com/knative/eventing): Management and delivery of events. For more information on Knative, visit the [Knative docs repo](https://github.com/knative/docs). -With GitLab serverless, you can deploy both functions-as-a-service (FaaS) and serverless applications. +With GitLab Serverless, you can deploy both functions-as-a-service (FaaS) and serverless applications. ## Prerequisites @@ -41,13 +40,16 @@ To run Knative on Gitlab, you will need: wildcard domain where your applications will be served. Configure your DNS server to use the external IP address or hostname for that domain. 1. **`.gitlab-ci.yml`:** GitLab uses [Kaniko](https://github.com/GoogleContainerTools/kaniko) - to build the application and the [TriggerMesh CLI](https://github.com/triggermesh/tm) to simplify the - deployment of knative services and functions. + to build the application. We also use [gitlabktl](https://gitlab.com/gitlab-org/gitlabktl) + and [TriggerMesh CLI](https://github.com/triggermesh/tm) CLIs to simplify the + deployment of services and functions to Knative. 1. **`serverless.yml`** (for [functions only](#deploying-functions)): When using serverless to deploy functions, the `serverless.yml` file will contain the information for all the functions being hosted in the repository as well as a reference to the runtime being used. -1. **`Dockerfile`** (for [applications only](#deploying-serverless-applications): Knative requires a `Dockerfile` in order to build your application. It should be included - at the root of your project's repo and expose port `8080`. +1. **`Dockerfile`** (for [applications only](#deploying-serverless-applications): Knative requires a + `Dockerfile` in order to build your applications. It should be included at the root of your + project's repo and expose port `8080`. `Dockerfile` is not require if you plan to build serverless functions + using our [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes). 1. **Prometheus** (optional): Installing Prometheus allows you to monitor the scale and traffic of your serverless function/application. See [Installing Applications](../index.md#installing-applications) for more information. @@ -89,10 +91,11 @@ Using functions is useful for dealing with independent events without needing to maintain a complex unified infrastructure. This allows you to focus on a single task that can be executed/scaled automatically and independently. -Currently the following [runtimes](https://gitlab.com/triggermesh/runtimes) are offered: +Currently the following [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes) are offered: +- ruby - node.js -- kaniko +- Dockerfile You can find and import all the files referenced in this doc in the **[functions example project](https://gitlab.com/knative-examples/functions)**. @@ -111,13 +114,17 @@ Follow these steps to deploy a function using the Node.js runtime to your Knativ include: template: Serverless.gitlab-ci.yml - functions: + functions:build: + extends: .serverless:build:functions + environment: production + + functions:deploy: extends: .serverless:deploy:functions environment: production ``` - This `.gitlab-ci.yml` creates a `functions` job that invokes some - predefined commands to deploy your functions to your cluster. + This `.gitlab-ci.yml` creates jobs that invoke some predefined commands to + build and deploy your functions to your cluster. `Serverless.gitlab-ci.yml` is a template that allows customization. You can either import it with `include` parameter and use `extends` to @@ -135,29 +142,40 @@ Follow these steps to deploy a function using the Node.js runtime to your Knativ You can find the relevant files for this project in the [functions example project](https://gitlab.com/knative-examples/functions). ```yaml - service: my-functions - description: "Deploying functions from GitLab using Knative" + service: functions + description: "GitLab Serverless functions using Knative" provider: name: triggermesh - registry-secret: gitlab-registry environment: - FOO: BAR + FOO: value functions: - echo: - handler: echo - runtime: https://gitlab.com/triggermesh/runtimes/raw/master/nodejs.yaml - description: "echo function using node.js runtime" - buildargs: - - DIRECTORY=echo + echo-js: + handler: echo-js + source: ./echo-js + runtime: https://gitlab.com/gitlab-org/serverless/runtimes/nodejs + description: "node.js runtime function" + environment: + MY_FUNCTION: echo-js + + echo-rb: + handler: MyEcho.my_function + source: ./echo-rb + runtime: https://gitlab.com/gitlab-org/serverless/runtimes/ruby + description: "Ruby runtime function" + environment: + MY_FUNCTION: echo-rb + + echo-docker: + handler: echo-docker + source: ./echo-docker + description: "Dockerfile runtime function" environment: - FUNCTION: echo + MY_FUNCTION: echo-docker ``` -The `serverless.yml` file references both an `echo` directory (under `buildargs`) and an `echo` file (under `handler`), -which is a reference to `echo.js` in the [repository](https://gitlab.com/knative-examples/functions). Additionally, it -contains three sections with distinct parameters: +Explanation of the fields used above: ### `service` @@ -171,7 +189,6 @@ contains three sections with distinct parameters: | Parameter | Description | |-----------|-------------| | `name` | Indicates which provider is used to execute the `serverless.yml` file. In this case, the TriggerMesh `tm` CLI. | -| `registry-secret` | Indicates which registry will be used to store docker images. The sample function is using the GitLab Registry (`gitlab-registry`). A different registry host may be specified using `registry` key in the `provider` object. If changing the default, update the permission and the secret value on the `gitlab-ci.yml` file | | `environment` | Includes the environment variables to be passed as part of function execution for **all** functions in the file, where `FOO` is the variable name and `BAR` are he variable contents. You may replace this with you own variables. | ### `functions` @@ -180,10 +197,10 @@ In the `serverless.yml` example above, the function name is `echo` and the subse | Parameter | Description | |-----------|-------------| -| `handler` | The function's file name. In the example above, both the function name and the handler are the same. | +| `handler` | The function's name. | +| `source` | Directory with sources of a functions. | | `runtime` | The runtime to be used to execute the function. | | `description` | A short description of the function. | -| `buildargs` | Pointer to the function file in the repo. In the sample the function is located in the `echo` directory. | | `environment` | Sets an environment variable for the specific function only. | After the `gitlab-ci.yml` template has been added and the `serverless.yml` file has been @@ -214,7 +231,7 @@ The sample function can now be triggered from any HTTP client using a simple `PO --header "Content-Type: application/json" \ --request POST \ --data '{"GitLab":"FaaS"}' \ - <http://functions-echo.functions-1.functions.example.com/> + http://functions-echo.functions-1.functions.example.com ``` 2. Using a web-based tool (ie. postman, restlet, etc) @@ -284,3 +301,23 @@ The second to last line, labeled **Service domain** contains the URL for the dep browser to see the app live.  + +## Function details + +Go to the **Operations > Serverless** page and click on one of the function +rows to bring up the function details page. + + + +The pod count will give you the number of pods running the serverless function instances on a given cluster. + +### Prometheus support + +For the Knative function invocations to appear, +[Prometheus must be installed](../index.md#installing-applications). + +Once Prometheus is installed, a message may appear indicating that the metrics data _is +loading or is not available at this time._ It will appear upon the first access of the +page, but should go away after a few seconds. If the message does not disappear, then it +is possible that GitLab is unable to connect to the Prometheus instance running on the +cluster. diff --git a/doc/user/project/img/labels_default.png b/doc/user/project/img/labels_default.png Binary files differindex 7a7fab611a4..221c5cf55a2 100644 --- a/doc/user/project/img/labels_default.png +++ b/doc/user/project/img/labels_default.png diff --git a/doc/user/project/img/labels_list.png b/doc/user/project/img/labels_list.png Binary files differindex 6878349fc0c..6940dde68bf 100644 --- a/doc/user/project/img/labels_list.png +++ b/doc/user/project/img/labels_list.png diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 4d19464cb7a..64139f9dbe9 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -55,11 +55,11 @@ When you create a project in GitLab, you'll have access to a large number of - [Auto Deploy](../../ci/autodeploy/index.md): Configure GitLab CI/CD to automatically set up your app's deployment - [Enable and disable GitLab CI](../../ci/enable_or_disable_ci.md) - - [Pipelines](../../ci/pipelines.md#pipelines): Configure and visualize + - [Pipelines](../../ci/pipelines.md): Configure and visualize your GitLab CI/CD pipelines from the UI - [Scheduled Pipelines](pipelines/schedules.md): Schedule a pipeline to start at a chosen time - - [Pipeline Graphs](../../ci/pipelines.md#pipeline-graphs): View your + - [Pipeline Graphs](../../ci/pipelines.md#visualizing-pipelines): View your entire pipeline from the UI - [Job artifacts](pipelines/job_artifacts.md): Define, browse, and download job artifacts diff --git a/doc/user/project/integrations/hipchat.md b/doc/user/project/integrations/hipchat.md new file mode 100644 index 00000000000..0fd847d415f --- /dev/null +++ b/doc/user/project/integrations/hipchat.md @@ -0,0 +1,53 @@ +# Atlassian HipChat + +GitLab provides a way to send HipChat notifications upon a number of events, +such as when a user pushes code, creates a branch or tag, adds a comment, and +creates a merge request. + +## Setup + +GitLab requires the use of a HipChat v2 API token to work. v1 tokens are +not supported at this time. Note the differences between v1 and v2 tokens: + +HipChat v1 API (legacy) supports "API Auth Tokens" in the Group API menu. A v1 +token is allowed to send messages to *any* room. + +HipChat v2 API has tokens that are can be created using the Integrations tab +in the Group or Room admin page. By design, these are lightweight tokens that +allow GitLab to send messages only to *one* room. + +### Complete these steps in HipChat + +1. Go to: <https://admin.hipchat.com/admin> +1. Click on "Group Admin" -> "Integrations". +1. Find "Build Your Own!" and click "Create". +1. Select the desired room, name the integration "GitLab", and click "Create". +1. In the "Send messages to this room by posting this URL" column, you should +see a URL in the format: + +``` +https://api.hipchat.com/v2/room/<room>/notification?auth_token=<token> +``` + +HipChat is now ready to accept messages from GitLab. Next, set up the HipChat +service in GitLab. + +### Complete these steps in GitLab + +1. Navigate to the project you want to configure for notifications. +1. Navigate to the [Integrations page](project_services.md#accessing-the-project-services) +1. Click "HipChat". +1. Select the "Active" checkbox. +1. Insert the `token` field from the URL into the `Token` field on the Web page. +1. Insert the `room` field from the URL into the `Room` field on the Web page. +1. Save or optionally click "Test Settings". + +## Troubleshooting + +If you do not see notifications, make sure you are using a HipChat v2 API +token, not a v1 token. + +Note that the v2 token is tied to a specific room. If you want to be able to +specify arbitrary rooms, you can create an API token for a specific user in +HipChat under "Account settings" and "API access". Use the `XXX` value under +`auth_token=XXX`. diff --git a/doc/user/project/integrations/img/jira_api_token_menu.png b/doc/user/project/integrations/img/jira_api_token_menu.png Binary files differindex 55c8fb1bdb9..14037bd0b47 100644 --- a/doc/user/project/integrations/img/jira_api_token_menu.png +++ b/doc/user/project/integrations/img/jira_api_token_menu.png diff --git a/doc/user/project/integrations/img/jira_service_page.png b/doc/user/project/integrations/img/jira_service_page.png Binary files differindex 3a27b4df841..377b69d9d06 100644 --- a/doc/user/project/integrations/img/jira_service_page.png +++ b/doc/user/project/integrations/img/jira_service_page.png diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md index e2f23827360..42c7824a125 100644 --- a/doc/user/project/integrations/project_services.md +++ b/doc/user/project/integrations/project_services.md @@ -36,6 +36,7 @@ Click on the service links to see further configuration instructions and details | External Wiki | Replaces the link to the internal wiki with a link to an external wiki | | Flowdock | Flowdock is a collaboration web app for technical teams | | [Hangouts Chat](hangouts_chat.md) | Receive events notifications in Google Hangouts Chat | +| [HipChat](hipchat.md) | Private group chat and IM | | [Irker (IRC gateway)](irker.md) | Send IRC messages, on update, to a list of recipients through an Irker gateway | | [JIRA](jira.md) | JIRA issue tracker | | JetBrains TeamCity CI | A continuous integration and build server | diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index d324e0de0cc..8e1603f9ec9 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -48,7 +48,7 @@ Navigate to the webhooks page by going to your project's ## Use-cases - You can set up a webhook in GitLab to send a notification to - [Slack](https://api.slack.com/incoming-webhooks) every time a build fails, for example + [Slack](https://api.slack.com/incoming-webhooks) every time a job fails. - You can [integrate with Twilio to be notified via SMS](https://www.datadoghq.com/blog/send-alerts-sms-customizable-webhooks-twilio/) every time an issue is created for a specific project or group within GitLab - You can use them to [automatically assign labels to merge requests](https://about.gitlab.com/2016/08/19/applying-gitlab-labels-automatically/). @@ -1004,7 +1004,7 @@ X-Gitlab-Event: Pipeline Hook "email": "user@gitlab.com" } }, - "builds":[ + "jobs":[ { "id": 380, "stage": "deploy", @@ -1129,34 +1129,34 @@ X-Gitlab-Event: Pipeline Hook } ``` -### Build events +### Job events -Triggered on status change of a Build. +Triggered on status change of a job. **Request Header**: ``` -X-Gitlab-Event: Build Hook +X-Gitlab-Event: Job Hook ``` **Request Body**: ```json { - "object_kind": "build", + "object_kind": "job", "ref": "gitlab-script-trigger", "tag": false, "before_sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", "sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", - "build_id": 1977, - "build_name": "test", - "build_stage": "test", - "build_status": "created", - "build_started_at": null, - "build_finished_at": null, - "build_duration": null, - "build_allow_failure": false, - "build_failure_reason": "script_failure", + "job_id": 1977, + "job_name": "test", + "job_stage": "test", + "job_status": "created", + "job_started_at": null, + "job_finished_at": null, + "job_duration": null, + "job_allow_failure": false, + "job_failure_reason": "script_failure", "project_id": 380, "project_name": "gitlab-org/gitlab-test", "user": { diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index ca19ce4d328..ad47b848bea 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -42,7 +42,7 @@ below. ## How it works The Issue Board builds on GitLab's existing -[issue tracking functionality](issues/index.md#issue-tracker) and +[issue tracking functionality](issues/index.md#issues-list) and leverages the power of [labels](labels.md) by utilizing them as lists of the scrum board. With the Issue Board you can have a different view of your issues while diff --git a/doc/user/project/issues/create_new_issue.md b/doc/user/project/issues/create_new_issue.md index 40040e44d64..9a147deecd4 100644 --- a/doc/user/project/issues/create_new_issue.md +++ b/doc/user/project/issues/create_new_issue.md @@ -7,7 +7,7 @@ the information illustrated on the image below.  -Read through the [issues functionalities documentation](issues_functionalities.md#issues-functionalities) +Read through the [issue data and actions documentation](issue_data_and_actions.md#parts-of-an-issue) to understand these fields one by one. ## New issue from the Issue Tracker diff --git a/doc/user/project/issues/img/similar_issues.png b/doc/user/project/issues/img/similar_issues.png Binary files differindex 0dfb5b00e02..c5b7902b2ac 100644 --- a/doc/user/project/issues/img/similar_issues.png +++ b/doc/user/project/issues/img/similar_issues.png diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 675c280a12a..c82b7f100d2 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -1,170 +1,135 @@ # Issues -The GitLab Issue Tracker is an advanced and complete tool -for tracking the evolution of a new idea or the process -of solving a problem. +Issues are the fundamental medium for collaborating on ideas and planning work in GitLab. -It allows you, your team, and your collaborators to share -and discuss proposals before and while implementing them. +## Overview -GitLab Issues and the GitLab Issue Tracker are available in all -[GitLab Products](https://about.gitlab.com/pricing/) as -part of the [GitLab Workflow](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/). +The GitLab issue tracker is an advanced tool for collaboratively developing ideas, solving problems, and planning work. -## Use cases +Issues can allow you, your team, and your collaborators to share and discuss proposals before and during their implementation. +However, they can be used for a variety of other purposes, customized to your needs and workflow. -Issues can have endless applications. Just to exemplify, these are -some cases for which creating issues are most used: +Issues are always associated with a specific project, but if you have multiple projects in a group, +you can also view all the issues collectively at the group level. + +**Common use cases include:** - Discussing the implementation of a new idea -- Submitting feature proposals -- Asking questions -- Reporting bugs and malfunction -- Obtaining support +- Tracking tasks and work status +- Accepting feature proposals, questions, support requests, or bug reports - Elaborating new code implementations See also the blog post "[Always start a discussion with an issue](https://about.gitlab.com/2016/03/03/start-with-an-issue/)". -### Keep private things private - -For instance, let's assume you have a public project but want to start a discussion on something -you don't want to be public. With [Confidential Issues](#confidential-issues), -you can discuss private matters among the project members, and still keep -your project public, open to collaboration. - -### Streamline collaboration - -With [Multiple Assignees for Issues](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html), -available in [GitLab Starter](https://about.gitlab.com/pricing/) -you can streamline collaboration and allow shared responsibilities to be clearly displayed. -All assignees are shown across your workflows and receive notifications (as they -would as single assignees), simplifying communication and ownership. - -### Consistent collaboration - -Create [issue templates](#issue-templates) to make collaboration consistent and -containing all information you need. For example, you can create a template -for feature proposals and another one for bug reports. - -## Issue Tracker - -The Issue Tracker is the collection of opened and closed issues created in a project. -It is available for all projects, from the moment the project is created. - -Find the issue tracker by navigating to your **Project's homepage** > **Issues**. - -### Issues per project - -When you access your project's issues, GitLab will present them in a list, -and you can use the tabs available to quickly filter by open and closed issues. - - - -You can also [search and filter](../../search/index.md#issues-and-merge-requests-per-project) the results more deeply with GitLab's search capacities. - -### Issues per group - -View issues in all projects in the group, including all projects of all descendant subgroups of the group. Navigate to **Group > Issues** to view these issues. This view also has the open and closed issues tabs. - - - -## GitLab Issues Functionalities - -The image bellow illustrates how an issue looks like: +## Parts of an issue + +Issues contain a variety of content and metadata, enabling a large range of flexibility in how they are used. Each issue can contain the following attributes, though some items may remain unset. + +<table class="borderless-table fixed-table"> +<tr> + <td> + <ul> + <li>Content</li> + <ul> + <li>Title</li> + <li>Description and tasks</li> + <li>Comments and other activity</li> + </ul> + <li>People</li> + <ul> + <li>Author</li> + <li>Assignee(s)</li> + </ul> + <li>State</li> + <ul> + <li>Status (open/closed)</li> + <li>Confidentiality</li> + <li>Tasks (completed vs. outstanding)</li> + </ul> + </ul> + </td> + <td> + <ul> + <li>Planning and tracking</li> + <ul> + <li>Milestone</li> + <li>Due date</li> + <li>Weight</li> + <li>Time tracking</li> + <li>Labels</li> + <li>Votes</li> + <li>Reaction emoji</li> + <li>Linked issues</li> + <li>Assigned epic</li> + <li>Unique issue number and URL</li> + </ul> + </ul> + </td> +</tr> +</table> + +## Viewing and managing issues + +While you can view and manage the full detail of an issue at its URL, you can also work with multiple issues at a time using the Issues List, Issue Boards, Epics **[ULTIMATE]**, and issue references. + +### Issue page  -Learn more about it on the [GitLab Issues Functionalities documentation](issues_functionalities.md). - -## New issue +On an issue’s page, you can view all aspects of the issue, and you can also modify them if you you have the necessary [permissions](../../permissions.md). -Read through the [documentation on creating issues](create_new_issue.md). +For more information, see the [Issue Data and Actions](issue_data_and_actions.md) page. -## Closing issues +### Issues list -Learn distinct ways to [close issues](closing_issues.md) in GitLab. - -## Moving issues - -Read through the [documentation on moving issues](moving_issues.md). - -## Deleting issues + -Read through the [documentation on deleting issues](deleting_issues.md) +On the Issues List, you can view all issues in the current project, or from multiple projects when opening the Issues List from the higher-level group context. Filter the issue list by [any search query](../../search/index.md#issues-and-merge-requests-per-project) and/or specific metadata, such as label(s), assignees(s), status, and more. From this view, you can also make certain changes [in bulk](../bulk_editing.md) to the displayed issues. -## Create a merge request from an issue +For more information, see the [Issue Data and Actions](issue_data_and_actions.md) page. -Learn more about it on the [GitLab Issues Functionalities documentation](issues_functionalities.md#18-new-merge-request). +### Issue boards -## Search for an issue + -Learn how to [find an issue](../../search/index.md) by searching for and filtering them. +Issue boards are Kanban boards with columns that display issues based on their labels or their assignees**[PREMIUM]**. They offer the flexibility to manage issues using highly customizable workflows. -## Advanced features +You can reorder issues within a column, or drag an issue card to another column; its associated label or assignee will change to match that of the new column. The entire board can also be filtered to only include issues from a certain milestone or an overarching label. -### Confidential Issues +For more information, see the [Issue Boards](../issue_board.md) page. -Whenever you want to keep the discussion presented in a -issue within your team only, you can make that -[issue confidential](confidential_issues.md). Even if your project -is public, that issue will be preserved. The browser will -respond with a 404 error whenever someone who is not a project -member with at least [Reporter level](../../permissions.md#project-members-permissions) tries to -access that issue's URL. +### Epics **[ULTIMATE]** -Learn more about them on the [confidential issues documentation](confidential_issues.md). +Epics let you manage your portfolio of projects more efficiently and with less effort by tracking groups of issues that share a theme, across projects and milestones. -### Issue templates +For more information, see the [Epics](https://docs.gitlab.com/ee/user/group/epics/) page. -Create templates for every new issue. They will be available from -the dropdown menu **Choose a template** when you create a new issue: +### Related issues **[STARTER]** - +You can mark two issues as related, so that when viewing each one, the other is always listed in its Related Issues section. This can help display important context, such as past work, dependencies, or duplicates. -Learn more about them on the [issue templates documentation](../../project/description_templates.md#creating-issue-templates). +For more information, see [Related Issues](https://docs.gitlab.com/ee/user/project/issues/related_issues.html). ### Crosslinking issues -Learn more about [crosslinking](crosslinking_issues.md) issues and merge requests. - -### Issue Board - -The [GitLab Issue Board](https://about.gitlab.com/features/issueboard/) is a way to -enhance your workflow by organizing and prioritizing issues in GitLab. - - - -Find GitLab Issue Boards by navigating to your **Project's Dashboard** > **Issues** > **Board**. - -Read through the documentation for [Issue Boards](../issue_board.md) -to find out more about this feature. - -With [GitLab Starter](https://about.gitlab.com/pricing/), you can also -create various boards per project with [Multiple Issue Boards](../issue_board.html#multiple-issue-boards-starter). - -### Import Issues from CSV - -You can import a CSV file containing issue titles and descriptions to create -a batch of issues simultaneously. - -When you navigate to the Issues list page, an import button is displayed. - -For further details, see [Importing issues from CSV](csv_import.md) - -### External Issue Tracker - -Alternatively to GitLab's built-in Issue Tracker, you can also use an [external -tracker](../../../integration/external-issue-tracker.md) such as Jira, Redmine, -YouTrack, or Bugzilla. - -### Issue API +When you reference an issue from another issue or merge request by including its URL or ID, the referenced issue displays a message in the Activity stream about the reference, with a link to the other issue or MR. -See the [API documentation](../../../api/issues.md). +For more information, see [Crosslinking issues](crosslinking_issues.md). -### Bulk editing issues +## Issue actions -See the [bulk editing issues](../../project/bulk_editing.md) page. +- [Create an issue](create_new_issue.md) +- [Create an issue from a template](../../project/description_templates.md#using-the-templates) +- [Close an issue](closing_issues.md) +- [Move an issue](moving_issues.md) +- [Delete an issue](deleting_issues.md) +- [Create a merge request from an issue](issue_data_and_actions.md#18-new-merge-request) -### Similar issues +## Advanced issue management -See the [similar issues](similar_issues.md) page. +- [Bulk edit issues](../bulk_editing.md) - From the Issues List, select multiple issues in order to change their status, assignee, milestone, or labels in bulk. +- [Import issues](csv_import.md) +- [Export issues](https://docs.gitlab.com/ee/user/project/issues/csv_export.html) **[STARTER]** +- [Issues API](../../../api/issues.md) +- Configure an [external issue tracker](../../../integration/external-issue-tracker.md) such as Jira, Redmine, +or Bugzilla. diff --git a/doc/user/project/issues/issues_functionalities.md b/doc/user/project/issues/issue_data_and_actions.md index 27b9dc51760..653bd94e513 100644 --- a/doc/user/project/issues/issues_functionalities.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -1,8 +1,8 @@ -# GitLab Issues Functionalities +# Issue Data and Actions Please read through the [GitLab Issue Documentation](index.md) for an overview on GitLab Issues. -## Issues Functionalities +## Parts of an Issue The image below illustrates what an issue looks like: @@ -77,7 +77,7 @@ can be changed as many times as needed. Categorize issues by giving them [labels](../labels.md). They help to organize workflows, and they enable you to work with the -[GitLab Issue Board](index.md#issue-board). +[GitLab Issue Board](index.md#issue-boards). Group Labels, which allow you to use the same labels for a group of projects, can be also given to issues. They work exactly the same, diff --git a/doc/user/project/merge_requests/img/comment-on-any-diff-line.png b/doc/user/project/merge_requests/img/comment-on-any-diff-line.png Binary files differindex c2455c2d1e5..5b9844bf02f 100644 --- a/doc/user/project/merge_requests/img/comment-on-any-diff-line.png +++ b/doc/user/project/merge_requests/img/comment-on-any-diff-line.png diff --git a/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png b/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png Binary files differindex 2a2101719ba..ea3aff59aa1 100644 --- a/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png +++ b/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 593eb80e044..ba7d05a7ad7 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -18,7 +18,7 @@ With GitLab merge requests, you can: - Live preview the changes when [Review Apps](../../../ci/review_apps/index.md) is configured for your project - Build, test, and deploy your code in a per-branch basis with built-in [GitLab CI/CD](../../../ci/README.md) - Prevent the merge request from being merged before it's ready with [WIP MRs](#work-in-progress-merge-requests) -- View the deployment process through [Pipeline Graphs](../../../ci/pipelines.md#pipeline-graphs) +- View the deployment process through [Pipeline Graphs](../../../ci/pipelines.md#visualizing-pipelines) - [Automatically close the issue(s)](../../project/issues/closing_issues.md#via-merge-request) that originated the implementation proposed in the merge request - Assign it to any registered user, and change the assignee how many times you need - Assign a [milestone](../../project/milestones/index.md) and track the development of a broader implementation @@ -219,6 +219,64 @@ apply the patches. The target branch can be specified using the [`/target_branch` quick action](../quick_actions.md). If the source branch already exists, the patches will be applied on top of it. +## Git push options + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26752) in GitLab 11.10. + +NOTE: **Note:** +Git push options are only available with Git 2.10 or newer. + +GitLab supports using +[Git push options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt) +to perform the following actions against merge requests at the same time +as pushing changes: + +- Create a new merge request for the pushed branch. +- Set the target of the merge request to a particular branch. +- Set the merge request to merge when its pipeline succeeds. + +### Create a new merge request using git push options + +To create a new merge request for a branch, use the +`merge_request.create` push option: + +```sh +git push -o merge_request.create +``` + +### Set the target branch of a merge request using git push options + +To update an existing merge request's target branch, use the +`merge_request.target=<branch_name>` push option: + +```sh +git push -o merge_request.target=branch_name +``` + +You can also create a merge request and set its target branch at the +same time using a `-o` flag per push option: + +```sh +git push -o merge_request.create -o merge_request.target=branch_name +``` + +### Set merge when pipeline succeeds using git push options + +To set an existing merge request to +[merge when its pipeline succeeds](merge_when_pipeline_succeeds.md), use +the `merge_request.merge_when_pipeline_succeeds` push option: + +```sh +git push -o merge_request.merge_when_pipeline_succeeds +``` + +You can also create a merge request and set it to merge when its +pipeline succeeds at the same time using a `-o` flag per push option: + +```sh +git push -o merge_request.create -o merge_request.merge_when_pipeline_succeeds +``` + ## Find the merge request that introduced a change > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2383) in GitLab 10.5. @@ -254,7 +312,7 @@ To prevent merge requests from accidentally being accepted before they're completely ready, GitLab blocks the "Accept" button for merge requests that have been marked as a **Work In Progress**. -[Learn more about settings a merge request as "Work In Progress".](work_in_progress_merge_requests.md) +[Learn more about setting a merge request as "Work In Progress".](work_in_progress_merge_requests.md) ## Merge request diff file navigation diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index f8af71ab46b..1477e35dca8 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -35,11 +35,11 @@ You need to have jobs configured to enable this feature. You can prevent merge requests from being merged if their pipeline did not succeed or if there are discussions to be resolved. -Navigate to your project's settings page, select the -**Only allow merge requests to be merged if the pipeline succeeds** check box and -hit **Save** for the changes to take effect. +Navigate to your project's settings page and expand the **Merge requests** section. +In the **Merge checks** subsection, select the **Pipelines must succeed** check +box and hit **Save** for the changes to take effect. - +  From now on, every time the pipeline fails you will not be able to merge the merge request from the UI, until you make all relevant jobs pass. diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index f1e2771dcb9..7dbf58b5715 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -1,42 +1,11 @@ --- last_updated: 2018-02-16 -author: Marcia Ramos -author_gitlab: marcia -level: beginner -article_type: user guide -date: 2017-02-22 --- # Static sites and GitLab Pages domains -This document is the beginning of a comprehensive guide, made for those who want to -publish a website with GitLab Pages but aren't familiar with -the entire process involved. - -This [first document](#what-you-need-to-know-before-getting-started) of this series will present you to the concepts of -static sites, and go over how the default Pages domains work. - -The [second document](getting_started_part_two.md) covers how to get started with GitLab Pages: deploy -a website from a forked project or create a new one from scratch. - -The [third document](getting_started_part_three.md) will show you how to set up a custom domain or subdomain -to your site already deployed. - -The [fourth document](getting_started_part_four.md) will show you how to create and tweak GitLab CI for -GitLab Pages. - -To **enable** GitLab Pages for GitLab CE (Community Edition) -and GitLab EE (Enterprise Edition), please read the -[admin documentation](https://docs.gitlab.com/ce/administration/pages/index.html), -and/or watch this [video tutorial](https://youtu.be/dD8c7WNcc6s). - ->**Note:** -For this guide, we assume you already have GitLab Pages -server up and running for your GitLab instance. - -## What you need to know before getting started - -Before we begin, let's understand a few concepts first. +On this docucument, learn how to name your project for GitLab Pages +according to your intended website's URL. ## Static sites @@ -48,20 +17,10 @@ CSS, and JS, or use a [Static Site Generator (SSG)](https://www.staticgen.com/) to simplify your code and build the static site for you, which is highly recommendable and much faster than hardcoding. -### Further reading - -- Read through this technical overview on [Static versus Dynamic Websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) -- Understand [how modern Static Site Generators work](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) and what you can add to your static site -- You can use [any SSG with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) -- Fork an [example project](https://gitlab.com/pages) to build your website based upon - -## GitLab Pages domain +See the [further reading](#further-reading) section below for +references on static site concepts. -If you set up a GitLab Pages project on GitLab.com, -it will automatically be accessible under a -[subdomain of `namespace.gitlab.io`](introduction.md#gitlab-pages-on-gitlabcom). -The `namespace` is defined by your username on GitLab.com, -or the group name you created this project under. +## GitLab Pages domain names >**Note:** If you use your own GitLab instance to deploy your @@ -70,11 +29,32 @@ Pages wildcard domain. This guide is valid for any GitLab instance, you just need to replace Pages wildcard domain on GitLab.com (`*.gitlab.io`) with your own. -Learn more about [namespaces](../../group/index.md#namespaces). +If you set up a GitLab Pages project on GitLab, +it will automatically be accessible under a +subdomain of `namespace.example.io`. +The [`namespace`](../../group/index.md#namespaces) +is defined by your username on GitLab.com, +or the group name you created this project under. +For GitLab self-managed instances, replace `example.io` +with your instance's Pages domain. For GitLab.com, +Pages domains are `*.gitlab.io`. + +| Type of GitLab Pages | The name of the project created in GitLab | Website URL | +| -------------------- | ------------ | ----------- | +| User pages | `username.example.io` | `http(s)://username.example.io` | +| Group pages | `groupname.example.io` | `http(s)://groupname.example.io` | +| Project pages owned by a user | `projectname` | `http(s)://username.example.io/projectname` | +| Project pages owned by a group | `projectname` | `http(s)://groupname.example.io/projectname`| +| Project pages owned by a subgroup | `subgroup/projectname` | `http(s)://groupname.example.io/subgroup/projectname`| + +CAUTION: **Warning:** +There are some known [limitations](introduction.md#limitations) +regarding namespaces served under the general domain name and HTTPS. +Make sure to read that section. -### Practical examples +To understand Pages domains clearly, read the examples below. -#### Project Websites +### Project website examples - You created a project called `blog` under your username `john`, therefore your project URL is `https://gitlab.com/john/blog/`. @@ -92,7 +72,7 @@ Learn more about [namespaces](../../group/index.md#namespaces). GitLab Pages for this project, the site will live under `https://engineering.gitlab.io/docs/workflows`. -#### User and Group Websites +### User and Group website examples - Under your username, `john`, you created a project called `john.gitlab.io`. Your project URL will be `https://gitlab.com/john/john.gitlab.io`. @@ -103,8 +83,6 @@ Learn more about [namespaces](../../group/index.md#namespaces). Once you enable GitLab Pages for your project, your website will be published under `https://websites.gitlab.io`. -> Support for subgroup project's websites was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30548) in GitLab 11.8. - **General example:** - On GitLab.com, a project site will always be available under @@ -115,3 +93,10 @@ Learn more about [namespaces](../../group/index.md#namespaces). Pages server domain. Ask your sysadmin for this information. _Read on about [Projects for GitLab Pages and URL structure](getting_started_part_two.md)._ + +### Further reading + +- Read through this technical overview on [Static versus Dynamic Websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) +- Understand [how modern Static Site Generators work](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) and what you can add to your static site +- You can use [any SSG with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) +- Fork an [example project](https://gitlab.com/pages) to build your website based upon
\ No newline at end of file diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index 756b8b698c7..9f2bc281f85 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -115,6 +115,8 @@ If using a [DNS A record](#dns-a-record), you can place the TXT record directly under the domain. If using a [DNS CNAME record](#dns-cname-record), the two record types won't co-exist, so you need to place the TXT record in a special subdomain of its own. +If the domain cannot be verified for 7 days, it will be removed from the GitLab project. + #### TL;DR For root domains (`domain.com`), set a DNS `A` record and verify your @@ -149,7 +151,7 @@ verify your domain's ownership with a TXT record: Once you've set the DNS record, you'll need navigate to your project's **Setting > Pages** and click **+ New domain** to add your custom domain to GitLab Pages. You can choose whether to add an [SSL/TLS certificate](#ssltls-certificates) -to make your website accessible under HTTPS or leave it blank. If don't add a certificate, +to make your website accessible under HTTPS or leave it blank. If you don't add a certificate, your site will be accessible only via HTTP:  @@ -175,9 +177,6 @@ Note that [DNS propagation may take some time (up to 24h)](http://www.inmotionho although it's usually a matter of minutes to complete. Until it does, verification will fail and attempts to visit your domain will respond with a 404. -Read through the [general documentation on GitLab Pages](introduction.md#add-a-custom-domain-to-your-pages-website) to learn more about adding -custom domains to GitLab Pages sites. - ### Redirecting `www.domain.com` to `domain.com` with Cloudflare If you use Cloudflare, you can redirect `www` to `domain.com` without the need of adding both diff --git a/doc/user/project/pages/getting_started_part_two.md b/doc/user/project/pages/getting_started_part_two.md index 901fb226cda..1034ba1733d 100644 --- a/doc/user/project/pages/getting_started_part_two.md +++ b/doc/user/project/pages/getting_started_part_two.md @@ -104,8 +104,8 @@ from the Pages group into a **user/group** website, you'll need to: ### Create a project from scratch 1. From your **Project**'s **[Dashboard](https://gitlab.com/dashboard/projects)**, - click **New project**, and name it considering the - [practical examples](getting_started_part_one.md#practical-examples). + click **New project**, and name it according to the + [Pages domain names](getting_started_part_one.md#gitlab-pages-domain-names). 1. Clone it to your local computer, add your website files to your project, add, commit and push to GitLab. 1. From the your **Project**'s page, click **Set up CI/CD**: diff --git a/doc/user/project/pages/img/icons/click.png b/doc/user/project/pages/img/icons/click.png Binary files differindex a534ae29e0f..62a997a7591 100644 --- a/doc/user/project/pages/img/icons/click.png +++ b/doc/user/project/pages/img/icons/click.png diff --git a/doc/user/project/pages/img/icons/fork.png b/doc/user/project/pages/img/icons/fork.png Binary files differindex 8a3aa46eb37..1ae8fa722b7 100644 --- a/doc/user/project/pages/img/icons/fork.png +++ b/doc/user/project/pages/img/icons/fork.png diff --git a/doc/user/project/pages/img/icons/free.png b/doc/user/project/pages/img/icons/free.png Binary files differindex ae455033e94..c74cd90fa1a 100644 --- a/doc/user/project/pages/img/icons/free.png +++ b/doc/user/project/pages/img/icons/free.png diff --git a/doc/user/project/pages/img/icons/lock.png b/doc/user/project/pages/img/icons/lock.png Binary files differindex f4c35c84112..f7f32fded45 100644 --- a/doc/user/project/pages/img/icons/lock.png +++ b/doc/user/project/pages/img/icons/lock.png diff --git a/doc/user/project/pages/img/icons/monitor.png b/doc/user/project/pages/img/icons/monitor.png Binary files differindex 8bad059a74c..ce27b7e177e 100644 --- a/doc/user/project/pages/img/icons/monitor.png +++ b/doc/user/project/pages/img/icons/monitor.png diff --git a/doc/user/project/pages/img/icons/terminal.png b/doc/user/project/pages/img/icons/terminal.png Binary files differindex 377eeb4edc6..5a711f05c4e 100644 --- a/doc/user/project/pages/img/icons/terminal.png +++ b/doc/user/project/pages/img/icons/terminal.png diff --git a/doc/user/project/pages/img/pages_remove.png b/doc/user/project/pages/img/pages_remove.png Binary files differdeleted file mode 100644 index 10299880247..00000000000 --- a/doc/user/project/pages/img/pages_remove.png +++ /dev/null diff --git a/doc/user/project/pages/img/remove_pages.png b/doc/user/project/pages/img/remove_pages.png Binary files differnew file mode 100644 index 00000000000..60f76f15f93 --- /dev/null +++ b/doc/user/project/pages/img/remove_pages.png diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 885df9f0850..0cd47c65d76 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -5,6 +5,11 @@ last_updated: 2019-03-05 # GitLab Pages +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/80) in GitLab Enterprise Edition 8.3. +> - Custom CNAMEs with TLS support were [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/173) in GitLab Enterprise Edition 8.5. +> - [Ported](https://gitlab.com/gitlab-org/gitlab-ce/issues/14605) to GitLab Community Edition in GitLab 8.17. +> Support for subgroup project's websites was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30548) in GitLab 11.8. + **GitLab Pages is a feature that allows you to publish static websites directly from a repository in GitLab.** @@ -83,7 +88,7 @@ that will build your site and publish it to the GitLab Pages server. The sequenc scripts that GitLab CI/CD runs to accomplish this task is created from a file named `.gitlab-ci.yml`, which you can [create and modify](getting_started_part_four.md) at will. -You can either use GitLab's [default domain for GitLab Pages websites](getting_started_part_one.md#gitlab-pages-domain), +You can either use GitLab's [default domain for GitLab Pages websites](getting_started_part_one.md#gitlab-pages-domain-names), `*.gitlab.io`, or your own domain (`example.com`). In that case, you'll need admin access to your domain's registrar (or control panel) to set it up with Pages. @@ -128,7 +133,7 @@ To learn more about GitLab Pages, read the following tutorials: - [Projects for GitLab Pages and URL structure](getting_started_part_two.md): Forking projects and creating new ones from scratch, understanding URLs structure and baseurls - [GitLab Pages custom domains and SSL/TLS Certificates](getting_started_part_three.md): How to add custom domains and subdomains to your website, configure DNS records and SSL/TLS certificates - [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md): Understand how to create your own `.gitlab-ci.yml` for your site -- [Technical aspects, custom 404 pages, limitations](introduction.md) +- [Exploring GitLab Pages](introduction.md): Technical aspects, specific configuration options, custom 404 pages, limitations ### GitLab Pages with Static Site Generators (SSGs) diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 39f14a1126f..a14a446aead 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -1,178 +1,44 @@ # Exploring GitLab Pages -> **Notes:** -> -> - This feature was [introduced][ee-80] in GitLab EE 8.3. -> - Custom CNAMEs with TLS support were [introduced][ee-173] in GitLab EE 8.5. -> - GitLab Pages [was ported][ce-14605] to Community Edition in GitLab 8.17. -> - This document is about the user guide. To learn how to enable GitLab Pages -> across your GitLab instance, visit the [administrator documentation](../../../administration/pages/index.md). +This document is a user guide to explore the options and settings +GitLab Pages offers. -With GitLab Pages you can host for free your static websites on GitLab. -Combined with the power of [GitLab CI] and the help of [GitLab Runner] you can -deploy static pages for your individual projects, your user or your group. +To familiarize yourself with GitLab Pages first: -Read [GitLab Pages on GitLab.com](#gitlab-pages-on-gitlabcom) for specific -information, if you are using GitLab.com to host your website. +- Read an [introduction to GitLab Pages](index.md#overview). +- Learn [how to get started with Pages](index.md#getting-started). +- Learn how to enable GitLab Pages +across your GitLab instance on the [administrator documentation](../../../administration/pages/index.md). -## Getting started with GitLab Pages domains - -> **Note:** -> In the rest of this document we will assume that the general domain name that -> is used for GitLab Pages is `example.io`. - -In general there are two types of pages one might create: - -- Pages per user (`username.example.io`) or per group (`groupname.example.io`) -- Pages per project (`username.example.io/projectname` or `groupname.example.io/projectname`) - -In GitLab, usernames and groupnames are unique and we often refer to them -as [namespaces](../../group/index.md#namespaces). There can be only one namespace -in a GitLab instance. Below you -can see the connection between the type of GitLab Pages, what the project name -that is created on GitLab looks like and the website URL it will be ultimately -be served on. - -| Type of GitLab Pages | The name of the project created in GitLab | Website URL | -| -------------------- | ------------ | ----------- | -| User pages | `username.example.io` | `http(s)://username.example.io` | -| Group pages | `groupname.example.io` | `http(s)://groupname.example.io` | -| Project pages owned by a user | `projectname` | `http(s)://username.example.io/projectname` | -| Project pages owned by a group | `projectname` | `http(s)://groupname.example.io/projectname`| -| Project pages owned by a subgroup | `subgroup/projectname` | `http(s)://groupname.example.io/subgroup/projectname`| - -> **Warning:** -> There are some known [limitations](#limitations) regarding namespaces served -> under the general domain name and HTTPS. Make sure to read that section. - -### GitLab Pages requirements +## Pages requirements In brief, this is what you need to upload your website in GitLab Pages: -1. Find out the general domain name that is used for GitLab Pages - (ask your administrator). This is very important, so you should first make - sure you get that right. -1. Create a project -1. Push a [`.gitlab-ci.yml` file][yaml] in the root directory - of your repository with a specific job named [`pages`][pages] -1. Set up a GitLab Runner to build your website - -> **Note:** -If [shared runners](../../../ci/runners/README.md) are enabled by your GitLab -administrator, you should be able to use them instead of bringing your own. - -### User or group Pages - -For user and group pages, the name of the project should be specific to the -username or groupname and the general domain name that is used for GitLab Pages. -Head over your GitLab instance that supports GitLab Pages and create a -repository named `username.example.io`, where `username` is your username on -GitLab. If the first part of the project name doesn't match exactly your -username, it won’t work, so make sure to get it right. - -To create a group page, the steps are the same like when creating a website for -users. Just make sure that you are creating the project within the group's -namespace. - - - ---- - -After you push some static content to your repository and GitLab Runner uploads -the artifacts to GitLab CI, you will be able to access your website under -`http(s)://username.example.io`. Keep reading to find out how. - ->**Note:** -If your username/groupname contains a dot, for example `foo.bar`, you will not -be able to use the wildcard domain HTTPS, read more at [limitations](#limitations). +1. Domain of the instance: domain name that is used for GitLab Pages +(ask your administrator). +1. GitLab CI/CD: a `.gitlab-ci.yml` file with a specific job named [`pages`][pages] in the root directory of your repository. +1. A directory called `public` in your site's repo containing the content +to be published. +1. GitLab Runner enabled for the project. -### Project Pages - -GitLab Pages for projects can be created by both user and group accounts. -The steps to create a project page for a user or a group are identical: - -1. Create a new project -1. Push a [`.gitlab-ci.yml` file][yaml] in the root directory - of your repository with a specific job named [`pages`][pages]. -1. Set up a GitLab Runner to build your website - -A user's project will be served under `http(s)://username.example.io/projectname` -whereas a group's project under `http(s)://groupname.example.io/projectname`. - -For practical examples for group and project Pages, read through the guide -[GitLab Pages from A to Z: Part 1 - Static sites and GitLab Pages domains](getting_started_part_one.md#practical-examples). - -## Quick Start - -Read through [GitLab Pages Quick Start Guide][pages-quick] or watch the video tutorial on -[how to publish a website with GitLab Pages on GitLab.com from a forked project][video-pages-fork]. - -See also [All you Need to Know About GitLab Pages][pages-index-guide] for a list with all the resources we have for GitLab Pages. - -### Explore the contents of `.gitlab-ci.yml` - -The key thing about GitLab Pages is the `.gitlab-ci.yml` file, something that -gives you absolute control over the build process. You can actually watch your -website being built live by following the CI job traces. - -For a simplified user guide on setting up GitLab CI/CD for Pages, read through -the article [GitLab Pages from A to Z: Part 4 - Creating and Tweaking `.gitlab-ci.yml` for GitLab Pages](getting_started_part_four.md) - -> **Note:** -> Before reading this section, make sure you familiarize yourself with GitLab CI -> and the specific syntax of[`.gitlab-ci.yml`][yaml] by -> following our [quick start guide]. - -To make use of GitLab Pages, the contents of `.gitlab-ci.yml` must follow the -rules below: - -1. A special job named [`pages`][pages] must be defined -1. Any static content which will be served by GitLab Pages must be placed under - a `public/` directory -1. `artifacts` with a path to the `public/` directory must be defined +## GitLab Pages on GitLab.com -In its simplest form, `.gitlab-ci.yml` looks like: +If you are using [GitLab Pages on GitLab.com](#gitlab-pages-on-gitlabcom) to host your website, then: -```yaml -pages: - script: - - my_commands - artifacts: - paths: - - public -``` +- The domain name for GitLab Pages on GitLab.com is `gitlab.io`. +- Custom domains and TLS support are enabled. +- Shared runners are enabled by default, provided for free and can be used to + build your website. If you want you can still bring your own Runner. -When the Runner reaches to build the `pages` job, it executes whatever is -defined in the `script` parameter and if the job completes with a non-zero -exit status, it then uploads the `public/` directory to GitLab Pages. +## Example projects -The `public/` directory should contain all the static content of your website. -Depending on how you plan to publish your website, the steps defined in the -[`script` parameter](../../../ci/yaml/README.md#script) may differ. +Visit the [GitLab Pages group](https://gitlab.com/groups/pages) for a complete list of example projects. Contributions are very welcome. -Be aware that Pages are by default branch/tag agnostic and their deployment -relies solely on what you specify in `.gitlab-ci.yml`. If you don't limit the -`pages` job with the [`only` parameter](../../../ci/yaml/README.md#onlyexcept-basic), -whenever a new commit is pushed to whatever branch or tag, the Pages will be -overwritten. In the example below, we limit the Pages to be deployed whenever -a commit is pushed only on the `master` branch: +## Specific configuration options for Pages -```yaml -pages: - script: - - my_commands - artifacts: - paths: - - public - only: - - master -``` - -We then tell the Runner to treat the `public/` directory as `artifacts` and -upload it to GitLab. And since all these parameters were all under a `pages` -job, the contents of the `public` directory will be served by GitLab Pages. +Learn how to set up GitLab CI/CD for specific use cases. -#### How `.gitlab-ci.yml` looks like when the static content is in your repository +### `.gitlab-ci.yml` for plain HTML websites Supposed your repository contained the following files: @@ -201,55 +67,11 @@ pages: - master ``` -#### How `.gitlab-ci.yml` looks like when using a static generator - -In general, GitLab Pages support any kind of [static site generator][staticgen], -since `.gitlab-ci.yml` can be configured to run any possible command. - -In the root directory of your Git repository, place the source files of your -favorite static generator. Then provide a `.gitlab-ci.yml` file which is -specific to your static generator. +### `.gitlab-ci.yml` for a static site generator -The example below, uses [Jekyll] to build the static site: +See this document for a [step-by-step guide](getting_started_part_four.md). -```yaml -image: ruby:2.1 # the script will run in Ruby 2.1 using the Docker image ruby:2.1 - -pages: # the build job must be named pages - script: - - gem install jekyll # we install jekyll - - jekyll build -d public/ # we tell jekyll to build the site for us - artifacts: - paths: - - public # this is where the site will live and the Runner uploads it in GitLab - only: - - master # this script is only affecting the master branch -``` - -Here, we used the Docker executor and in the first line we specified the base -image against which our jobs will run. - -You have to make sure that the generated static files are ultimately placed -under the `public` directory, that's why in the `script` section we run the -`jekyll` command that jobs the website and puts all content in the `public/` -directory. Depending on the static generator of your choice, this command will -differ. Search in the documentation of the static generator you will use if -there is an option to explicitly set the output directory. If there is not -such an option, you can always add one more line under `script` to rename the -resulting directory in `public/`. - -We then tell the Runner to treat the `public/` directory as `artifacts` and -upload it to GitLab. - ---- - -See the [jekyll example project][pages-jekyll] to better understand how this -works. - -For a list of Pages projects, see the [example projects](#example-projects) to -get you started. - -#### How to set up GitLab Pages in a repository where there's also actual code +### `.gitlab-ci.yml` for a repository where there's also actual code Remember that GitLab Pages are by default branch/tag agnostic and their deployment relies solely on what you specify in `.gitlab-ci.yml`. You can limit @@ -294,28 +116,6 @@ also includes `.gitlab-ci.yml`. [jekyll-master]: https://gitlab.com/pages/jekyll-branched/tree/master [jekyll-pages]: https://gitlab.com/pages/jekyll-branched/tree/pages -## Next steps - -So you have successfully deployed your website, congratulations! Let's check -what more you can do with GitLab Pages. - -### Example projects - -Below is a list of example projects for GitLab Pages with a plain HTML website -or various static site generators. Contributions are very welcome. - -- [Plain HTML](https://gitlab.com/pages/plain-html) -- [Jekyll](https://gitlab.com/pages/jekyll) -- [Hugo](https://gitlab.com/pages/hugo) -- [Middleman](https://gitlab.com/pages/middleman) -- [Hexo](https://gitlab.com/pages/hexo) -- [Brunch](https://gitlab.com/pages/brunch) -- [Metalsmith](https://gitlab.com/pages/metalsmith) -- [Harp](https://gitlab.com/pages/harp) - -Visit the GitLab Pages group for a full list of example projects: -<https://gitlab.com/groups/pages>. - ### Serving compressed assets Most modern browsers support downloading files in a compressed format. This @@ -408,52 +208,6 @@ NOTE: **Note:** When `public/data/index.html` exists, it takes priority over the `public/data.html` file for both the `/data` and `/data/` URL paths. -### Add a custom domain to your Pages website - -For a complete guide on Pages domains, read through the article -[GitLab Pages from A to Z: Part 3 - GitLab Pages custom domains and SSL/TLS Certificates](getting_started_part_three.md) - -If this setting is enabled by your GitLab administrator, you should be able to -see the **New Domain** button when visiting your project's settings through the -gear icon in the top right and then navigating to **Pages**. - - - ---- - -You can add multiple domains pointing to your website hosted under GitLab. -Once the domain is added, you can see it listed under the **Domains** section. - - - ---- - -As a last step, you need to configure your DNS and add a CNAME pointing to your -user/group page. Click on the **Details** button of a domain for further -instructions. - - - ---- - ->**Note:** -Currently there is support only for custom domains on per-project basis. That -means that if you add a custom domain (`example.com`) for your user website -(`username.example.io`), a project that is served under `username.example.io/foo`, -will not be accessible under `example.com/foo`. - -### Secure your custom domain website with TLS - -When you add a new custom domain, you also have the chance to add a TLS -certificate. If this setting is enabled by your GitLab administrator, you -should be able to see the option to upload the public certificate and the -private key when adding a new domain. - - - -For a complete guide on Pages domains, read through the article -[GitLab Pages from A to Z: Part 3 - GitLab Pages custom domains and SSL/TLS Certificates](getting_started_part_three.md) - ### Custom error codes pages You can provide your own 403 and 404 error pages by creating the `403.html` and @@ -472,29 +226,17 @@ If the case of `404.html`, there are different scenarios. For example: - If you use a custom domain and try to access `/non/existing_file`, GitLab Pages will try to serve only `/404.html`. -### Remove the contents of your pages - -If you ever feel the need to purge your Pages content, you can do so by going -to your project's settings through the gear icon in the top right, and then -navigating to **Pages**. Hit the **Remove pages** button and your Pages website -will be deleted. Simple as that. - - - -## GitLab Pages on GitLab.com - -If you are using GitLab.com to host your website, then: - -- The general domain name for GitLab Pages on GitLab.com is `gitlab.io`. -- Custom domains and TLS support are enabled. -- Shared runners are enabled by default, provided for free and can be used to - build your website. If you want you can still bring your own Runner. +### Redirects in GitLab Pages -The rest of the guide still applies. +Since you cannot use any custom server configuration files, like `.htaccess` or +any `.conf` file, if you want to redirect a page to another +location, you can use the [HTTP meta refresh tag][metarefresh]. -See also: [GitLab Pages from A to Z: Part 1 - Static sites and GitLab Pages domains](getting_started_part_one.md#gitlab-pages-domain). +Some static site generators provide plugins for that functionality so that you +don't have to create and edit HTML files manually. For example, Jekyll has the +[redirect-from plugin](https://github.com/jekyll/jekyll-redirect-from). -## GitLab Pages access control **[CORE ONLY]** +### GitLab Pages Access Control **[CORE ONLY]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/33422) in GitLab 11.5. @@ -536,6 +278,15 @@ The next time someone tries to access your website and the access control is enabled, they will be presented with a page to sign into GitLab and verify they can access the website. +## Unpublishing your Pages + +If you ever feel the need to purge your Pages content, you can do so by going +to your project's settings through the gear icon in the top right, and then +navigating to **Pages**. Hit the **Remove pages** button and your Pages website +will be deleted. + + + ## Limitations When using Pages under the general domain of a GitLab instance (`*.example.io`), @@ -550,16 +301,6 @@ don't redirect HTTP to HTTPS. GitLab Pages [does **not** support group websites for subgroups](../../group/subgroups/index.md#limitations). You can only create the highest-level group website. -## Redirects in GitLab Pages - -Since you cannot use any custom server configuration files, like `.htaccess` or -any `.conf` file, if you want to redirect a page to another -location, you can use the [HTTP meta refresh tag][metarefresh]. - -Some static site generators provide plugins for that functionality so that you -don't have to create and edit HTML files manually. For example, Jekyll has the -[redirect-from plugin](https://github.com/jekyll/jekyll-redirect-from). - ## Frequently Asked Questions ### Can I download my generated pages? @@ -581,8 +322,6 @@ No, you don't. You can create your project first and it will be accessed under For a list of known issues, visit GitLab's [public issue tracker]. [jekyll]: http://jekyllrb.com/ -[ee-80]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/80 -[ee-173]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/173 [pages-daemon]: https://gitlab.com/gitlab-org/gitlab-pages [gitlab ci]: https://about.gitlab.com/gitlab-ci [gitlab runner]: https://docs.gitlab.com/runner/ @@ -592,7 +331,6 @@ For a list of known issues, visit GitLab's [public issue tracker]. [pages-jekyll]: https://gitlab.com/pages/jekyll [metarefresh]: https://en.wikipedia.org/wiki/Meta_refresh [public issue tracker]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name=pages -[ce-14605]: https://gitlab.com/gitlab-org/gitlab-ce/issues/14605 [quick start guide]: ../../../ci/quick_start/README.md [pages-index-guide]: index.md [pages-quick]: getting_started_part_one.md diff --git a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md index f639188684b..ea22f3e905b 100644 --- a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md +++ b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md @@ -134,14 +134,14 @@ Now that your certificate has been issued, let's add it to your Pages site: sudo cat /etc/letsencrypt/live/example.com/fullchain.pem | pbcopy ``` -1. Copy and paste the public key into the second field **Key (PEM)**: +1. Copy and paste the private key into the second field **Key (PEM)**: ```bash sudo cat /etc/letsencrypt/live/example.com/privkey.pem | pbcopy ``` 1. Click **Save changes** to apply them to your website. -1. Wait a few minutes for DNS propagation. +1. Wait a few minutes for the configuration changes to take effect. 1. Visit your website at `https://example.com`. To force `https` connections on your site, navigate to your diff --git a/doc/user/project/pipelines/schedules.md b/doc/user/project/pipelines/schedules.md index 2911a56cf67..89f1beb6d1f 100644 --- a/doc/user/project/pipelines/schedules.md +++ b/doc/user/project/pipelines/schedules.md @@ -2,29 +2,33 @@ > **Notes**: > -> - This feature was introduced in 9.1 as [Trigger Schedule][ce-10533]. -> - In 9.2, the feature was [renamed to Pipeline Schedule][ce-10853]. +> - Introduced in GitLab 9.1 as [Trigger Schedule](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10533). +> - [Renamed to Pipeline Schedule](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10853) in GitLab 9.2. > - Cron notation is parsed by [Fugit](https://github.com/floraison/fugit). -Pipeline schedules can be used to run a pipeline at specific intervals, for example every -month on the 22nd for a certain branch. +Pipelines are normally run based on certain conditions being met. For example, when a branch is pushed to repository. -## Using Pipeline schedules +Pipeline schedules can be used to also run [pipelines](../../../ci/pipelines.md) at specific intervals. For example: -In order to schedule a pipeline: +- Every month on the 22nd for a certain branch. +- Once every day. -1. Navigate to your project's **CI / CD ➔ Schedules** and click the - **New Schedule** button. -1. Fill in the form -1. Hit **Save pipeline schedule** for the changes to take effect. +In addition to using the GitLab UI, pipeline schedules can be maintained using the +[Pipeline schedules API](../../../api/pipeline_schedules.md). + +## Configuring pipeline schedules + +To schedule a pipeline for project: + +1. Navigate to the project's **CI / CD > Schedules** page. +1. Click the **New schedule** button. +1. Fill in the **Schedule a new pipeline** form. +1. Click the **Save pipeline schedule** button.  -> **Attention:** -The pipelines won't be executed precisely, because schedules are handled by -Sidekiq, which runs according to its interval. -See [advanced admin configuration](#advanced-admin-configuration) for more -information. +NOTE: **Note:** +Pipelines execution [timing is dependent](#advanced-configuration) on Sidekiq's own schedule. In the **Schedules** index page you can see a list of the pipelines that are scheduled to run. The next run is automatically calculated by the server GitLab @@ -32,36 +36,24 @@ is installed on.  -### Running a scheduled pipeline manually - -> [Introduced][ce-15700] in GitLab 10.4. - -To trigger a pipeline schedule manually, click the "Play" button: - - - -This will schedule a background job to run the pipeline schedule. A flash -message will provide a link to the CI/CD Pipeline index page. - -To help avoid abuse, users are rate limited to triggering a pipeline once per -minute. - -### Making use of scheduled pipeline variables +### Using variables -> [Introduced][ce-12328] in GitLab 9.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12328) in GitLab 9.4. You can pass any number of arbitrary variables and they will be available in -GitLab CI so that they can be used in your `.gitlab-ci.yml` file. +GitLab CI so that they can be used in your [`.gitlab-ci.yml` file](../../../ci/yaml/README.md).  -## Using only and except +### Using only and except To configure that a job can be executed only when the pipeline has been scheduled (or the opposite), you can use [only and except](../../../ci/yaml/README.md#onlyexcept-basic) configuration keywords. -``` +For example: + +```yaml job:on-schedule: only: - schedules @@ -75,11 +67,47 @@ job: - make build ``` -## Taking ownership +### Advanced configuration + +The pipelines won't be executed exactly on schedule because schedules are handled by +Sidekiq, which runs according to its interval. + +For example, only two pipelines will be created per day if: -Pipelines are executed as a user, who owns a schedule. This influences what -projects and other resources the pipeline has access to. If a user does not own -a pipeline, you can take ownership by clicking the **Take ownership** button. +- You set a schedule to create a pipeline every minute (`* * * * *`). +- The Sidekiq worker runs on 00:00 and 12:00 every day (`0 */12 * * *`). + +To change the Sidekiq worker's frequency: + +1. Edit the `pipeline_schedule_worker_cron` value in your instance's `gitlab.rb` file. +1. [Restart GitLab](../../../administration/restart_gitlab.md). + +For GitLab.com, refer to the [dedicated settings page](../../gitlab_com/index.md#cron-jobs). + +## Working with scheduled pipelines + +Once configured, GitLab supports many functions for working with scheduled pipelines. + +### Running manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15700) in GitLab 10.4. + +To trigger a pipeline schedule manually, click the "Play" button: + + + +This will schedule a background job to run the pipeline schedule. A flash +message will provide a link to the CI/CD Pipeline index page. + +NOTE: **Note:** +To help avoid abuse, users are rate limited to triggering a pipeline once per +minute. + +### Taking ownership + +Pipelines are executed as a user, who owns a schedule. This influences what projects and other resources the pipeline has access to. + +If a user does not own a pipeline, you can take ownership by clicking the **Take ownership** button. The next time a pipeline is scheduled, your credentials will be used.  @@ -90,20 +118,3 @@ on the target branch, the schedule will stop creating new pipelines. This can happen if, for example, the owner is blocked or removed from the project, or the target branch or tag is protected. In this case, someone with sufficient privileges must take ownership of the schedule. - -## Advanced admin configuration - -The pipelines won't be executed precisely, because schedules are handled by -Sidekiq, which runs according to its interval. For example, if you set a -schedule to create a pipeline every minute (`* * * * *`) and the Sidekiq worker -runs on 00:00 and 12:00 every day (`0 */12 * * *`), only 2 pipelines will be -created per day. To change the Sidekiq worker's frequency, you have to edit the -`pipeline_schedule_worker_cron` value in your `gitlab.rb` and restart GitLab. -For GitLab.com, you can check the [dedicated settings page][settings]. If you -don't have admin access to the server, ask your administrator. - -[ce-10533]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10533 -[ce-10853]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10853 -[ce-12328]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12328 -[ce-15700]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15700 -[settings]: https://about.gitlab.com/gitlab-com/settings/#cron-jobs diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md index 4a989afad4d..8b762307ac4 100644 --- a/doc/user/project/pipelines/settings.md +++ b/doc/user/project/pipelines/settings.md @@ -182,4 +182,4 @@ https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?st ## Environment Variables -[Environment variables](../../../ci/variables/README.html#variables) can be set in an environment to be available to a runner. +[Environment variables](../../../ci/variables/README.html#gitlab-cicd-environment-variables) can be set in an environment to be available to a runner. diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 392e72dcc5c..88f4de891a1 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -31,7 +31,7 @@ discussions, and descriptions: | `/reassign @user1 @user2` | Change assignee | ✓ | ✓ | | `/milestone %milestone` | Set milestone | ✓ | ✓ | | `/remove_milestone` | Remove milestone | ✓ | ✓ | -| `/label ~label1 ~label2` | Add label(s) | ✓ | ✓ | +| `/label ~label1 ~label2` | Add label(s). Label names can also start without ~ but mixed syntax is not supported. | ✓ | ✓ | | `/unlabel ~label1 ~label2` | Remove all or specific label(s)| ✓ | ✓ | | `/relabel ~label1 ~label2` | Replace label | ✓ | ✓ | | <code>/copy_metadata #issue | !merge_request</code> | Copy labels and milestone from other issue or merge request | ✓ | ✓ | diff --git a/doc/user/project/repository/branches/img/branch_filter_search_box.png b/doc/user/project/repository/branches/img/branch_filter_search_box.png Binary files differindex 5dc7eccf189..5dc300cf24e 100644 --- a/doc/user/project/repository/branches/img/branch_filter_search_box.png +++ b/doc/user/project/repository/branches/img/branch_filter_search_box.png diff --git a/doc/user/project/repository/img/repository_cleanup.png b/doc/user/project/repository/img/repository_cleanup.png Binary files differindex bda40d3e193..e343f23ac27 100644 --- a/doc/user/project/repository/img/repository_cleanup.png +++ b/doc/user/project/repository/img/repository_cleanup.png diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 532247a98cd..95f606fd786 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -25,7 +25,7 @@ message. ## Creating a new wiki page NOTE: **Note:** -A [permission level][permissions] of **Developer** is needed to create Wiki pages. +Requires Developer [permissions](../../permissions.md). Create a new page by clicking the **New page** button that can be found in all wiki pages. You will be asked to fill in the page name from which GitLab @@ -58,7 +58,7 @@ repository, you will have to upload them again. ## Editing a wiki page NOTE: **Note:** -A [permission level][permissions] of **Developer** is needed to edit Wiki pages. +Requires Developer [permissions](../../permissions.md). To edit a page, simply click on the **Edit** button. From there on, you can change its content. When done, click **Save changes** for the changes to take @@ -67,7 +67,7 @@ effect. ## Deleting a wiki page NOTE: **Note:** -A [permission level][permissions] of **Maintainer** is needed to delete Wiki pages. +Requires Maintainer [permissions](../../permissions.md). You can find the **Delete** button only when editing a page. Click on it and confirm you want the page to be deleted. @@ -114,8 +114,6 @@ them like you would do with every other Git repository. On the right sidebar, click on **Clone repository** and follow the on-screen instructions. -[permissions]: ../../permissions.md - ## Customizing sidebar By default, the wiki would render a sidebar which lists all the pages for the diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index 9aa81e33fc0..9e8475d8294 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -83,6 +83,7 @@ Currently the following names are reserved as top level groups: - unsubscribes - uploads - users +- v2 These group names are unavailable as subgroup names: diff --git a/doc/workflow/add-user/add-user.md b/doc/workflow/add-user/add-user.md index 35cc080d2b7..f1ec771dd9a 100644 --- a/doc/workflow/add-user/add-user.md +++ b/doc/workflow/add-user/add-user.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/members/index.md' +--- + This document was moved to [../../user/project/members/index.md](../../user/project/members/index.md) diff --git a/doc/workflow/authorization_for_merge_requests.md b/doc/workflow/authorization_for_merge_requests.md index 7bf80a3ad0d..8e43d340613 100644 --- a/doc/workflow/authorization_for_merge_requests.md +++ b/doc/workflow/authorization_for_merge_requests.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/authorization_for_merge_requests.md' +--- + This document was moved to [user/project/merge_requests/authorization_for_merge_requests](../user/project/merge_requests/authorization_for_merge_requests.md) diff --git a/doc/workflow/award_emoji.md b/doc/workflow/award_emoji.md index d74378cc564..02db97b8dd6 100644 --- a/doc/workflow/award_emoji.md +++ b/doc/workflow/award_emoji.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/award_emojis.md' +--- + This document was moved to [another location](../user/award_emojis.md). diff --git a/doc/workflow/cherry_pick_changes.md b/doc/workflow/cherry_pick_changes.md index 663ffd3f746..29c4f854416 100644 --- a/doc/workflow/cherry_pick_changes.md +++ b/doc/workflow/cherry_pick_changes.md @@ -1 +1,5 @@ -This document was moved to [user/project/merge_requests/cherry_pick_changes](../user/project/merge_requests/cherry_pick_changes.md). +--- +redirect_to: '../user/project/merge_requests/cherry_pick_changes.md' +--- + +This document was moved to [another location](../user/project/merge_requests/cherry_pick_changes.md). diff --git a/doc/workflow/importing/README.md b/doc/workflow/importing/README.md index e0a445920b4..29da321ba46 100644 --- a/doc/workflow/importing/README.md +++ b/doc/workflow/importing/README.md @@ -1 +1,5 @@ +---
+redirect_to: '../../user/project/import/index.md'
+---
+
This document was moved to [another location](../../user/project/import/index.md).
diff --git a/doc/workflow/importing/import_projects_from_bitbucket.md b/doc/workflow/importing/import_projects_from_bitbucket.md index ec9a11f390e..a42ba7d4518 100644 --- a/doc/workflow/importing/import_projects_from_bitbucket.md +++ b/doc/workflow/importing/import_projects_from_bitbucket.md @@ -1 +1,5 @@ +---
+redirect_to: '../../user/project/import/bitbucket.md'
+---
+
This document was moved to [another location](../../user/project/import/bitbucket.md).
diff --git a/doc/workflow/importing/import_projects_from_fogbugz.md b/doc/workflow/importing/import_projects_from_fogbugz.md index 876eb0434f0..f5c791dc6de 100644 --- a/doc/workflow/importing/import_projects_from_fogbugz.md +++ b/doc/workflow/importing/import_projects_from_fogbugz.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/import/fogbugz.md' +--- + This document was moved to [another location](../../user/project/import/fogbugz.md). diff --git a/doc/workflow/importing/import_projects_from_gitea.md b/doc/workflow/importing/import_projects_from_gitea.md index 8b55b6c23eb..df053835b44 100644 --- a/doc/workflow/importing/import_projects_from_gitea.md +++ b/doc/workflow/importing/import_projects_from_gitea.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/import/gitea.md' +--- + This document was moved to [another location](../../user/project/import/gitea.md). diff --git a/doc/workflow/importing/import_projects_from_github.md b/doc/workflow/importing/import_projects_from_github.md index 72dfe5403c3..6397fcc74b8 100644 --- a/doc/workflow/importing/import_projects_from_github.md +++ b/doc/workflow/importing/import_projects_from_github.md @@ -1 +1,5 @@ +---
+redirect_to: '../../user/project/import/github.md'
+---
+
This document was moved to [another location](../../user/project/import/github.md).
diff --git a/doc/workflow/importing/import_projects_from_gitlab_com.md b/doc/workflow/importing/import_projects_from_gitlab_com.md index 3256088c014..135b9704df9 100644 --- a/doc/workflow/importing/import_projects_from_gitlab_com.md +++ b/doc/workflow/importing/import_projects_from_gitlab_com.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/import/gitlab_com.md' +--- + This document was moved to [another location](../../user/project/import/gitlab_com.md). diff --git a/doc/workflow/importing/migrating_from_svn.md b/doc/workflow/importing/migrating_from_svn.md index 32a75a6c6af..99f13d6354c 100644 --- a/doc/workflow/importing/migrating_from_svn.md +++ b/doc/workflow/importing/migrating_from_svn.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/import/svn.md' +--- + This document was moved to [another location](../../user/project/import/svn.md). diff --git a/doc/workflow/labels.md b/doc/workflow/labels.md index 5c09891dfdd..3d07d411dd4 100644 --- a/doc/workflow/labels.md +++ b/doc/workflow/labels.md @@ -1,3 +1,7 @@ +--- +redirect_to: '../user/project/labels.md' +--- + # Labels This document was moved to [user/project/labels.md](../user/project/labels.md). diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md index 3fb553280d0..6d941135bf2 100644 --- a/doc/workflow/lfs/lfs_administration.md +++ b/doc/workflow/lfs/lfs_administration.md @@ -108,6 +108,18 @@ Here is a configuration example with GCS. _NOTE: The service account must have permission to access the bucket. [See more](https://cloud.google.com/storage/docs/authentication)_ +Here is a configuration example with Rackspace Cloud Files. + +| Setting | Description | example | +|---------|-------------|---------| +| `provider` | The provider name | `Rackspace` | +| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | +| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | +| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` | + +_NOTES: Regardless of whether the container has public access enabled or disabled, Fog will use the TempURL method to grant access to LFS objects. If you see errors in logs referencing instantiating storage with a temp-url-key, ensure that you have set they key properly on the Rackspace API and in gitlab.rb. You can verify the value of the key Rackspace has set by sending a GET request with token header to the service access endpoint URL and comparing the output of the returned headers._ + ### Manual uploading to an object storage There are two ways to manually do the same thing as automatic uploading (described above). diff --git a/doc/workflow/merge_requests.md b/doc/workflow/merge_requests.md index dc6da1938f3..fd9f9b81bc9 100644 --- a/doc/workflow/merge_requests.md +++ b/doc/workflow/merge_requests.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/index.md' +--- + This document was moved to [user/project/merge_requests/index.md](../user/project/merge_requests/index.md). diff --git a/doc/workflow/merge_when_build_succeeds.md b/doc/workflow/merge_when_build_succeeds.md index b4f6d6117de..41e6ff0cdd6 100644 --- a/doc/workflow/merge_when_build_succeeds.md +++ b/doc/workflow/merge_when_build_succeeds.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/merge_when_pipeline_succeeds.md' +--- + This document was moved to [merge_when_pipeline_succeeds](../user/project/merge_requests/merge_when_pipeline_succeeds.md). diff --git a/doc/workflow/milestones.md b/doc/workflow/milestones.md index 69eb6b286b0..18dc15f7327 100644 --- a/doc/workflow/milestones.md +++ b/doc/workflow/milestones.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/milestones/index.md' +--- + This document was moved to [another location](../user/project/milestones/index.md). diff --git a/doc/workflow/project_features.md b/doc/workflow/project_features.md index feb88712f5a..f54afb768a1 100644 --- a/doc/workflow/project_features.md +++ b/doc/workflow/project_features.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/index.md' +--- + This document was moved to [../user/project/index.md](../user/project/index.md) diff --git a/doc/workflow/protected_branches.md b/doc/workflow/protected_branches.md index ced7d391ace..1bcac4a2de5 100644 --- a/doc/workflow/protected_branches.md +++ b/doc/workflow/protected_branches.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/protected_branches.md' +--- + This document was moved to [another location](../user/project/protected_branches.md). diff --git a/doc/workflow/repository_mirroring.md b/doc/workflow/repository_mirroring.md index ae1624b7dc0..9fcadbf3bee 100644 --- a/doc/workflow/repository_mirroring.md +++ b/doc/workflow/repository_mirroring.md @@ -84,7 +84,7 @@ To set up a mirror from GitLab to GitHub, you need to follow these steps: 1. Fill in **Password** field with your GitHub personal access token. 1. Click the **Mirror repository** button. -The mirrored repository will be listed. For example, `https://*****:*****@github.com/<your_github_group>/<your_github_project>.git`. +The mirrored repository will be listed. For example, `https://*****:*****@github.com/<your_github_group>/<your_github_project>.git`. The repository will push soon. To force a push, click the appropriate button. @@ -138,13 +138,18 @@ upstream and GitLab will no longer automatically update this branch to prevent a ### How it works -Once you activate the pull mirroring feature, the mirror will be inserted into a queue. A scheduler -will start every minute and schedule a fixed number of mirrors for update, based on the configured maximum capacity. +Once the pull mirroring feature has been enabled for a repository, the repository is added to a queue. -If the mirror updates successfully, it will be enqueued once again with a small backoff period. +Once per minute, a Sidekiq cron job schedules repository mirrors to update, based on: -If the mirror fails (for example, a branch diverged from upstream), the project's backoff period is -increased each time it fails, up to a maximum amount of time. +- The capacity available. This is determined by Sidekiq settings. For GitLab.com, see [GitLab.com Sidekiq settings](../user/gitlab_com/index.md#sidekiq). +- The number of repository mirrors already in the queue that are due to be updated. Being due depends on when the repository mirror was last updated and how many times it's been retried. + +Repository mirrors are updated as Sidekiq becomes available to process them. If the process of updating the repository mirror: + +- Succeeds, an update will be enqueued again with at least a 30 minute wait. +- Fails (for example, a branch diverged from upstream), it will be attempted again later. Mirrors can fail + up to 14 times before they will not be enqueued for update again. ### SSH authentication diff --git a/doc/workflow/revert_changes.md b/doc/workflow/revert_changes.md index cf1292253fc..15f199af703 100644 --- a/doc/workflow/revert_changes.md +++ b/doc/workflow/revert_changes.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/revert_changes.md' +--- + This document was moved to [user/project/merge_requests/revert_changes](../user/project/merge_requests/revert_changes.md). diff --git a/doc/workflow/share_projects_with_other_groups.md b/doc/workflow/share_projects_with_other_groups.md index 2eb4d24958a..c39cd78f32d 100644 --- a/doc/workflow/share_projects_with_other_groups.md +++ b/doc/workflow/share_projects_with_other_groups.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/members/share_project_with_groups.md' +--- + This document was moved to [../user/project/members/share_project_with_groups.md](../user/project/members/share_project_with_groups.md) diff --git a/doc/workflow/share_with_group.md b/doc/workflow/share_with_group.md index 2eb4d24958a..c39cd78f32d 100644 --- a/doc/workflow/share_with_group.md +++ b/doc/workflow/share_with_group.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/members/share_project_with_groups.md' +--- + This document was moved to [../user/project/members/share_project_with_groups.md](../user/project/members/share_project_with_groups.md) diff --git a/doc/workflow/time_tracking.md b/doc/workflow/time_tracking.md index e60b6819bf1..c03dffa967d 100644 --- a/doc/workflow/time_tracking.md +++ b/doc/workflow/time_tracking.md @@ -75,6 +75,6 @@ Default conversion rates are 1mo = 4w, 1w = 5d and 1d = 8h. Other interesting links: -- [Time Tracking landing page on about.gitlab.com](https://about.gitlab.com/features/time-tracking) +- [Time Tracking landing page on about.gitlab.com](https://about.gitlab.com/solutions/time-tracking/) [quick actions]: ../user/project/quick_actions.md diff --git a/doc/workflow/web_editor.md b/doc/workflow/web_editor.md index 595c7da155b..2366372d984 100644 --- a/doc/workflow/web_editor.md +++ b/doc/workflow/web_editor.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/repository/web_editor.md' +--- + This document was moved to [user/project/repository/web_editor](../user/project/repository/web_editor.md). diff --git a/doc/workflow/wip_merge_requests.md b/doc/workflow/wip_merge_requests.md index abb8002f442..020455dcbdc 100644 --- a/doc/workflow/wip_merge_requests.md +++ b/doc/workflow/wip_merge_requests.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/work_in_progress_merge_requests.md' +--- + This document was moved to [user/project/merge_requests/work_in_progress_merge_requests](../user/project/merge_requests/work_in_progress_merge_requests.md). |
