diff options
Diffstat (limited to 'doc')
281 files changed, 10633 insertions, 2813 deletions
diff --git a/doc/README.md b/doc/README.md index c8b6b4f32b8..604f7244a34 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,5 +1,4 @@ --- -toc: false comments: false --- @@ -8,41 +7,66 @@ comments: false Welcome to [GitLab](https://about.gitlab.com/), a Git-based fully featured platform for software development! -GitLab offers the most scalable Git-based fully integrated platform for software development, with flexible products and subscription plans. +GitLab offers the most scalable Git-based fully integrated platform for +software development, with flexible products and subscriptions. +To understand what features you have access to, check the [GitLab subscriptions](#gitlab-subscriptions) below. -With GitLab self-hosted, you deploy your own GitLab instance on-premises or on a private cloud of your choice. GitLab self-hosted is available for [free and with paid subscriptions](https://about.gitlab.com/products/): Libre, Starter, Premium, and Ultimate. +**Shortcuts to GitLab's most visited docs:** -GitLab.com is our SaaS offering. It's hosted, managed, and administered by GitLab, with [free and paid plans](https://about.gitlab.com/gitlab-com/) for individuals and teams: Free, Bronze, Silver, and Gold. - -## Shortcuts to GitLab's most visited docs - -| [GitLab CI/CD](ci/README.md) | Other | +| General documentation | GitLab CI/CD docs | | :----- | :----- | -| [Quick start guide](ci/quick_start/README.md) | [API](api/README.md) | -| [Configuring `.gitlab-ci.yml`](ci/yaml/README.md) | [SSH authentication](ssh/README.md) | -| [Using Docker images](ci/docker/using_docker_images.md) | [GitLab Pages](user/project/pages/index.md) | - -- [User documentation](user/index.md) -- [Administrator documentation](administration/index.md) -- [Contributor documentation](#contributor-documentation) +| [User documentation](user/index.md) | [GitLab CI/CD](ci/README.md) | +| [Administrator documentation](administration/index.md) | [GitLab CI/CD quick start guide](ci/quick_start/README.md) | +| [Contributor documentation](#contributor-documentation) | [Configuring `.gitlab-ci.yml`](ci/yaml/README.md) | +| [Getting started with GitLab](#getting-started-with-gitlab) | [Using Docker images](ci/docker/using_docker_images.md) | +| [API](api/README.md) | [Auto DevOps](topics/autodevops/index.md) | +| [SSH authentication](ssh/README.md) | [Kubernetes integration](user/project/clusters/index.md)| +| [GitLab Pages](user/project/pages/index.md) | [GitLab Container Registry](user/project/container_registry.md) | + +## Complete DevOps with GitLab + +GitLab is the first single application for software development, security, +and operations that enables Concurrent DevOps, making the software lifecycle +three times faster and radically improving the speed of business. GitLab +provides solutions for all the stages of the DevOps lifecycle: +[plan](#plan), [create](#create), [verify](#verify), [package](#package), +[release](#release), [configure](#configure), [monitor](#monitor). + + + +### Plan + +Whether you use Waterfall, Agile, or Conversational Development, +GitLab streamlines your collaborative workflows. Visualize, prioritize, +coordinate, and track your progress your way with GitLab’s flexible project +management tools. + +- Chat operations + - [Mattermost slash commands](user/project/integrations/mattermost_slash_commands.md) + - [Slack slash commands](user/project/integrations/slack_slash_commands.md) +- [Discussions](user/discussions/index.md): Threads, comments, and resolvable discussions in issues, commits, and merge requests. +- [Issues](user/project/issues/index.md) +- [Project Issue Board](user/project/issue_board.md) +- [Issues and merge requests templates](user/project/description_templates.md): Create templates for submitting new issues and merge requests. +- [Labels](user/project/labels.md): Categorize your issues or merge requests based on descriptive titles. +- [Milestones](user/project/milestones/index.md): Organize issues and merge requests into a cohesive group, optionally setting a due date. +- [Todos](workflow/todos.md): A chronological list of to-dos that are waiting for your input, all in a simple dashboard. +- [GitLab Quick Actions](user/project/quick_actions.md): Textual shortcuts for common actions on issues or merge requests that are usually done by clicking buttons or dropdowns in GitLab's UI. -## Getting started with GitLab +#### Migrate and import your projects from other platforms -- [GitLab Basics](gitlab-basics/README.md): Start working on your command line and on GitLab. -- [GitLab Workflow](workflow/README.md): Enhance your workflow with the best of GitLab Workflow. - - See also [GitLab Workflow - an overview](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/). -- [GitLab Markdown](user/markdown.md): GitLab's advanced formatting system (GitLab Flavored Markdown). -- [GitLab Quick Actions](user/project/quick_actions.md): Textual shortcuts for common actions on issues or merge requests that are usually done by clicking buttons or dropdowns in GitLab's UI. -- [Auto DevOps](topics/autodevops/index.md) +- [Importing to GitLab](user/project/import/index.md): Import your projects from GitHub, Bitbucket, GitLab.com, FogBugz and SVN into GitLab. +- [Migrating from SVN](workflow/importing/migrating_from_svn.md): Convert a SVN repository to Git and GitLab. -### User account +### Create -- [User account](user/profile/index.md): Manage your account - - [Authentication](topics/authentication/index.md): Account security with two-factor authentication, setup your ssh keys and deploy keys for secure access to your projects. - - [Profile settings](user/profile/index.md#profile-settings): Manage your profile settings, two factor authentication and more. -- [User permissions](user/permissions.md): Learn what each role in a project (external/guest/reporter/developer/master/owner) can do. +Consolidate source code into a single [DVCS](https://en.wikipedia.org/wiki/Distributed_version_control) +that’s easily managed and controlled without disrupting your workflow. +GitLab’s git repositories come complete with branching tools and access +controls, providing a scalable, single source of truth for collaborating +on projects and code. -### Projects and groups +#### Projects and groups - [Projects](user/project/index.md): - [Project settings](user/project/settings/index.md) @@ -57,7 +81,7 @@ GitLab.com is our SaaS offering. It's hosted, managed, and administered by GitLa - [Snippets](user/snippets.md): Snippets allow you to create little bits of code. - [Wikis](user/project/wiki/index.md): Enhance your repository documentation with built-in wikis. -### Repository +#### Repositories Manage your [repositories](user/project/repository/index.md) from the UI (user interface): @@ -75,57 +99,93 @@ Manage your [repositories](user/project/repository/index.md) from the UI (user i - [Commits](user/project/repository/index.md#commits) - [Signing commits](user/project/repository/gpg_signed_commits/index.md): use GPG to sign your commits. -### Issues and Merge Requests (MRs) +#### Integrations + +- [Project Services](user/project/integrations/project_services.md): Integrate a project with external services, such as CI and chat. +- [GitLab Integration](integration/README.md): Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. +- [Trello Power-Up](integration/trello_power_up.md): Integrate with GitLab's Trello Power-Up + +#### Automation + +- [API](api/README.md): Automate GitLab via a simple and powerful API. +- [GitLab Webhooks](user/project/integrations/webhooks.md): Let GitLab notify you when new code has been pushed to your project. + +### Verify + +Spot errors sooner and shorten feedback cycles with built-in code review, code testing, +Code Quality, and Review Apps. Customize your approval workflow controls, automatically +test the quality of your code, and spin up a staging environment for every code change. +GitLab Continuous Integration is the most popular next generation testing system that +auto scales to run your tests faster. -- [Discussions](user/discussions/index.md): Threads, comments, and resolvable discussions in issues, commits, and merge requests. -- [Issues](user/project/issues/index.md) -- [Project issue Board](user/project/issue_board.md) -- [Issues and merge requests templates](user/project/description_templates.md): Create templates for submitting new issues and merge requests. -- [Labels](user/project/labels.md): Categorize your issues or merge requests based on descriptive titles. - [Merge Requests](user/project/merge_requests/index.md) - [Work In Progress Merge Requests](user/project/merge_requests/work_in_progress_merge_requests.md) - [Merge Request discussion resolution](user/discussions/index.md#moving-a-single-discussion-to-a-new-issue): Resolve discussions, move discussions in a merge request to an issue, only allow merge requests to be merged if all discussions are resolved. - [Checkout merge requests locally](user/project/merge_requests/index.md#checkout-merge-requests-locally) - [Cherry-pick](user/project/merge_requests/cherry_pick_changes.md) -- [Milestones](user/project/milestones/index.md): Organize issues and merge requests into a cohesive group, optionally setting a due date. -- [Todos](workflow/todos.md): A chronological list of to-dos that are waiting for your input, all in a simple dashboard. +- [Review Apps](ci/review_apps/index.md): Preview changes to your app right from a merge request. -### Git and GitLab +### Package -- [Git](topics/git/index.md): Getting started with Git, branching strategies, Git LFS, advanced use. -- [Git cheatsheet](https://gitlab.com/gitlab-com/marketing/raw/master/design/print/git-cheatsheet/print-pdf/git-cheatsheet.pdf): Download a PDF describing the most used Git operations. -- [GitLab Flow](workflow/gitlab_flow.md): explore the best of Git with the GitLab Flow strategy. +GitLab Container Registry gives you the enhanced security and access controls of +custom Docker images without 3rd party add-ons. Easily upload and download images +from GitLab CI/CD with full Git repository management integration. -### Migrate and import your projects from other platforms +- [GitLab CI/CD](ci/README.md): Explore the features and capabilities of Continuous Integration, Continuous Delivery, and Continuous Deployment with GitLab. +- [GitLab Container Registry](user/project/container_registry.md): Learn how to use GitLab's built-in Container Registry. -- [Importing to GitLab](user/project/import/index.md): Import your projects from GitHub, Bitbucket, GitLab.com, FogBugz and SVN into GitLab. -- [Migrating from SVN](workflow/importing/migrating_from_svn.md): Convert a SVN repository to Git and GitLab. +### Release + +Spend less time configuring your tools, and more time creating. Whether you’re +deploying to one server or thousands, build, test, and release your code +confidently and securely with GitLab’s built-in Continuous Delivery and Deployment. + +- [GitLab Pages](user/project/pages/index.md): Build, test, and deploy a static site directly from GitLab. +- [Auto Deploy](topics/autodevops/index.md#auto-deploy): Configure GitLab CI for the deployment of your application. +- [Environments and deployments](ci/environments.md): With environments, you can control the continuous deployment of your software within GitLab. + +### Configure -### Continuous Integration, Delivery, and Deployment +Automate your entire workflow from build to deploy and monitoring with GitLab +Auto Devops. Best practice templates get you started with minimal to zero +configuration. Then customize everything from buildpacks to CI/CD. + +- [Auto DevOps](topics/autodevops/index.md) + +### Monitor + +Measure how long it takes to go from planning to monitoring and ensure your +applications are always responsive and available. GitLab collects and displays +performance metrics for deployed apps using Prometheus so you can know in an +instant how code changes impact your production environment. -- [GitLab CI](ci/README.md): Explore the features and capabilities of Continuous Integration, Continuous Delivery, and Continuous Deployment with GitLab. - - [Auto Deploy](ci/autodeploy/index.md): Configure GitLab CI for the deployment of your application. - - [Review Apps](ci/review_apps/index.md): Preview changes to your app right from a merge request. - [GitLab Cycle Analytics](user/project/cycle_analytics.md): Cycle Analytics measures the time it takes to go from an [idea to production](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab) for each project you have. -- [GitLab Container Registry](user/project/container_registry.md): Learn how to use GitLab's built-in Container Registry. +- [GitLab Performance Monitoring](administration/monitoring/performance/index.md) -### Automation +## Getting started with GitLab -- [API](api/README.md): Automate GitLab via a simple and powerful API. -- [GitLab Webhooks](user/project/integrations/webhooks.md): Let GitLab notify you when new code has been pushed to your project. +- [GitLab Basics](gitlab-basics/README.md): Start working on your command line and on GitLab. +- [GitLab Workflow](workflow/README.md): Enhance your workflow with the best of GitLab Workflow. + - See also [GitLab Workflow - an overview](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/). +- [GitLab Markdown](user/markdown.md): GitLab's advanced formatting system (GitLab Flavored Markdown). -### Integrations +### User account -- [Project Services](user/project/integrations/project_services.md): Integrate a project with external services, such as CI and chat. -- [GitLab Integration](integration/README.md): Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. -- [Trello Power-Up](integration/trello_power_up.md): Integrate with GitLab's Trello Power-Up +- [User account](user/profile/index.md): Manage your account + - [Authentication](topics/authentication/index.md): Account security with two-factor authentication, setup your ssh keys and deploy keys for secure access to your projects. + - [Profile settings](user/profile/index.md#profile-settings): Manage your profile settings, two factor authentication and more. +- [User permissions](user/permissions.md): Learn what each role in a project (external/guest/reporter/developer/master/owner) can do. ----- +### Git and GitLab + +- [Git](topics/git/index.md): Getting started with Git, branching strategies, Git LFS, advanced use. +- [Git cheatsheet](https://gitlab.com/gitlab-com/marketing/raw/master/design/print/git-cheatsheet/print-pdf/git-cheatsheet.pdf): Download a PDF describing the most used Git operations. +- [GitLab Flow](workflow/gitlab_flow.md): explore the best of Git with the GitLab Flow strategy. ## Administrator documentation -[Administration documentation](administration/index.md) applies to admin users of [GitLab -self-hosted instances](#self-hosted-gitlab): Libre, Starter, Premium, Ultimate. +[Administration documentation](administration/index.md) applies to admin users of GitLab +self-hosted instances. Learn how to install, configure, update, upgrade, integrate, and maintain your own instance. Regular users don't have access to GitLab administration tools and settings. @@ -133,9 +193,48 @@ Regular users don't have access to GitLab administration tools and settings. ## Contributor documentation GitLab Community Edition is [open source](https://gitlab.com/gitlab-org/gitlab-ce/) -and Enterprise Editions are [open-core](https://gitlab.com/gitlab-org/gitlab-ee/). +and GitLab Enterprise Edition is [open-core](https://gitlab.com/gitlab-org/gitlab-ee/). Learn how to contribute to GitLab: - [Development](development/README.md): All styleguides and explanations how to contribute. - [Legal](legal/README.md): Contributor license agreements. - [Writing documentation](development/writing_documentation.md): Contributing to GitLab Docs. + +## GitLab subscriptions + +You have two options to use GitLab: + +- GitLab self-hosted: Install, administer, and maintain your own GitLab instance. +- GitLab.com: GitLab's SaaS offering. You don't need to install anything to use GitLab.com, +you only need to [sign up](https://gitlab.com/users/sign_in) and start using GitLab +straight away. + +### GitLab self-hosted + +With GitLab self-hosted, you deploy your own GitLab instance on-premises or on a private cloud of your choice. GitLab self-hosted is available for [free and with paid subscriptions](https://about.gitlab.com/products/): Core, Starter, Premium, and Ultimate. + +Every feature available in Core is also available in Starter, Premium, and Ultimate. +Starter features are also available in Premium and Ultimate, and Premium features are also +available in Ultimate. + +### GitLab.com + +GitLab.com is hosted, managed, and administered by GitLab, Inc., with +[free and paid subscriptions](https://about.gitlab.com/gitlab-com/) for individuals +and teams: Free, Bronze, Silver, and Gold. + +GitLab.com subscriptions grants access +to the same features available in GitLab self-hosted, **expect +[administration](administration/index.md) tools and settings**: + +- GitLab.com Free includes the same features available in Core +- GitLab.com Bronze includes the same features available in GitLab Starter +- GitLab.com Silver includes the same features available in GitLab Premium +- GitLab.com Gold includes the same features available in GitLab Ultimate + +For supporting the open source community and encouraging the development of +open source projects, GitLab grants access to **Gold** features +for all GitLab.com **public** projects, regardless of the subscription. + +To know more about GitLab subscriptions and licensing, please refer to the +[GitLab Product Marketing Handbook](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers). diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 9b9b8ca89e5..aa5e9513290 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -1,9 +1,12 @@ -# How to configure LDAP with GitLab CE +--- +author: Chris Wilson +author_gitlab: MrChrisW +level: intermediary +article_type: admin guide +date: 2017-05-03 +--- -> **[Article Type](../../../development/writing_documentation.html#types-of-technical-articles):** admin guide || -> **Level:** intermediary || -> **Author:** [Chris Wilson](https://gitlab.com/MrChrisW) || -> **Publication date:** 2017-05-03 +# How to configure LDAP with GitLab CE ## Introduction diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md new file mode 100644 index 00000000000..b51e705ab52 --- /dev/null +++ b/doc/administration/auth/jwt.md @@ -0,0 +1,72 @@ +# JWT OmniAuth provider + +To enable the JWT OmniAuth provider, you must register your application with JWT. +JWT will provide you with a secret key for you to use. + +1. On your GitLab server, open the configuration file. + + For Omnibus GitLab: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. +1. Add the provider configuration. + + For Omnibus GitLab: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { name: 'jwt', + app_secret: 'YOUR_APP_SECRET', + args: { + algorithm: 'HS256', + uid_claim: 'email', + required_claims: ["name", "email"], + info_maps: { name: "name", email: "email" }, + auth_url: 'https://example.com/', + valid_within: nil, + } + } + ] + ``` + + For installation from source: + + ``` + - { name: 'jwt', + app_secret: 'YOUR_APP_SECRET', + args: { + algorithm: 'HS256', + uid_claim: 'email', + required_claims: ["name", "email"], + info_map: { name: "name", email: "email" }, + auth_url: 'https://example.com/', + valid_within: nil, + } + } + ``` + + NOTE: **Note:** For more information on each configuration option refer to + the [OmniAuth JWT usage documentation](https://github.com/mbleigh/omniauth-jwt#usage). + +1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL. +1. Save the configuration file. +1. [Reconfigure GitLab][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. + +On the sign in page there should now be a JWT icon below the regular sign in form. +Click the icon to begin the authentication process. JWT will ask the user to +sign in and authorize the GitLab application. If everything goes well, the user +will be redirected to GitLab and will be signed in. + +[reconfigure GitLab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart GitLab]: ../restart_gitlab.md#installations-from-source diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 881b6a827f4..63fbb24bac1 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -181,6 +181,10 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server first_name: 'givenName' last_name: 'sn' + # If lowercase_usernames is enabled, GitLab will lower case the username. + lowercase_usernames: false + + ## EE only # Base where we can search for groups @@ -290,6 +294,41 @@ In other words, if an existing GitLab user wants to enable LDAP sign-in for themselves, they should check that their GitLab email address matches their LDAP email address, and then sign into GitLab via their LDAP credentials. +## Enabling LDAP username lowercase + +Some LDAP servers, depending on their configurations, can return uppercase usernames. This can lead to several confusing issues like, for example, creating links or namespaces with uppercase names. + +GitLab can automatically lowercase usernames provided by the LDAP server by enabling +the configuration option `lowercase_usernames`. By default, this configuration option is `false`. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + # snip... + lowercase_usernames: true + EOS + ``` + +2. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + lowercase_usernames: true + ``` +2. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + ## Encryption ### TLS Server Authentication diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 4d35b20d0c3..960970aea30 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -4,8 +4,9 @@ **Note:** Custom Git hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. Please explore [webhooks] as an option if you do not -have filesystem access. For a user configurable Git hook interface, please see -[GitLab Enterprise Edition Git Hooks](http://docs.gitlab.com/ee/git_hooks/git_hooks.html). +have filesystem access. For a user configurable Git hook interface, see +[Push Rules](https://docs.gitlab.com/ee/push_rules/push_rules.html), +available in GitLab Enterprise Edition. Git natively supports hooks that are executed on different actions. Examples of server-side git hooks include pre-receive, post-receive, and update. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 9bcd13a52f7..e6c8f59549f 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -13,7 +13,7 @@ override certain values. Variable | Type | Description -------- | ---- | ----------- -`GITLAB_CDN_HOST` | string | Sets the hostname for a CDN to serve static assets (e.g. `mycdnsubdomain.fictional-cdn.com`) +`GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (e.g. `//mycdnsubdomain.fictional-cdn.com`) `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging` or `test` diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index e3b10119090..d9a61aea6ef 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -2,7 +2,9 @@ [Gitaly](https://gitlab.com/gitlab-org/gitaly) (introduced in GitLab 9.0) is a service that provides high-level RPC access to Git -repositories. Gitaly is a mandatory component in GitLab 9.4 and newer. +repositories. Gitaly was optional when it was first introduced in +GitLab, but since GitLab 9.4 it is a mandatory component of the +application. GitLab components that access Git repositories (gitlab-rails, gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do @@ -184,14 +186,20 @@ Gitaly logs on your Gitaly server (`sudo gitlab-ctl tail gitaly` or coming in. One sure way to trigger a Gitaly request is to clone a repository from your GitLab server over HTTP. -## Disabling or enabling the Gitaly service +## Disabling or enabling the Gitaly service in a cluster environment If you are running Gitaly [as a remote service](#running-gitaly-on-its-own-server) you may want to disable the local Gitaly service that runs on your Gitlab server by default. -To disable the Gitaly service in your Omnibus installation, add the -following line to `/etc/gitlab/gitlab.rb`: +> 'Disabling Gitaly' only makes sense when you run GitLab in a custom +cluster configuration, where different services run on different +machines. Disabling Gitaly on all machines in the cluster is not a +valid configuration. + +If you are setting up a GitLab cluster where Gitaly does not need to +run on all machines, you can disable the Gitaly service in your +Omnibus installation, add the following line to `/etc/gitlab/gitlab.rb`: ```ruby gitaly['enable'] = false @@ -200,11 +208,13 @@ gitaly['enable'] = false When you run `gitlab-ctl reconfigure` the Gitaly service will be disabled. -To disable the Gitaly service in an installation from source, add the -following to `/etc/default/gitlab`: +To disable the Gitaly service in a GitLab cluster where you installed +GitLab from source, add the following to `/etc/default/gitlab` on the +machine where you want to disable Gitaly. ```shell gitaly_enabled=false ``` -When you run `service gitlab restart` Gitaly will be disabled. +When you run `service gitlab restart` Gitaly will be disabled on this +particular machine. diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index b85a166089d..e201848791c 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -25,11 +25,11 @@ for each GitLab application server in your environment. options. Here is an example snippet to add to `/etc/fstab`: ``` - 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 - 10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 ``` 1. Create the shared directories. These may be different depending on your NFS diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md new file mode 100644 index 00000000000..27a3710632d --- /dev/null +++ b/doc/administration/incoming_email.md @@ -0,0 +1,332 @@ +# Incoming email + +GitLab has several features based on receiving incoming emails: + +- [Reply by Email](reply_by_email.md): allow GitLab users to comment on issues + and merge requests by replying to notification emails. +- [New issue by email](../user/project/issues/create_new_issue.md#new-issue-via-email): + allow GitLab users to create a new issue by sending an email to a + user-specific email address. +- [New merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email): + allow GitLab users to create a new merge request by sending an email to a + user-specific email address. + +## Requirements + +Handling incoming emails requires an [IMAP]-enabled email account. GitLab +requires one of the following three strategies: + +- Email sub-addressing +- Dedicated email address +- Catch-all mailbox + +Let's walk through each of these options. + +**If your provider or server supports email sub-addressing, we recommend using it. +Most features (other than reply by email) only work with sub-addressing.** + +[IMAP]: https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol + +### Email sub-addressing + +[Sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing) is +a feature where any email to `user+some_arbitrary_tag@example.com` will end up +in the mailbox for `user@example.com`, and is supported by providers such as +Gmail, Google Apps, Yahoo! Mail, Outlook.com and iCloud, as well as the +[Postfix mail server] which you can run on-premises. + +[Postfix mail server]: reply_by_email_postfix_setup.md + +### Dedicated email address + +This solution is really simple to set up: you just have to create an email +address dedicated to receive your users' replies to GitLab notifications. + +### Catch-all mailbox + +A [catch-all mailbox](https://en.wikipedia.org/wiki/Catch-all) for a domain will +"catch all" the emails addressed to the domain that do not exist in the mail +server. + +GitLab can be set up to allow users to comment on issues and merge requests by +replying to notification emails. + +## Set it up + +If you want to use Gmail / Google Apps for incoming emails, make sure you have +[IMAP access enabled](https://support.google.com/mail/troubleshooter/1668960?hl=en#ts=1665018) +and [allowed less secure apps to access the account](https://support.google.com/accounts/answer/6010255) +or [turn-on 2-step validation](https://support.google.com/accounts/answer/185839) +and use [an application password](https://support.google.com/mail/answer/185833). + +To set up a basic Postfix mail server with IMAP access on Ubuntu, follow the +[Postfix setup documentation](reply_by_email_postfix_setup.md). + +### Security Concerns + +**WARNING:** Be careful when choosing the domain used for receiving incoming +email. + +For the sake of example, suppose your top-level company domain is `hooli.com`. +All employees in your company have an email address at that domain via Google +Apps, and your company's private Slack instance requires a valid `@hooli.com` +email address in order to sign up. + +If you also host a public-facing GitLab instance at `hooli.com` and set your +incoming email domain to `hooli.com`, an attacker could abuse the "Create new +issue by email" or +"[Create new merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email)" +features by using a project's unique address as the email when signing up for +Slack, which would send a confirmation email, which would create a new issue or +merge request on the project owned by the attacker, allowing them to click the +confirmation link and validate their account on your company's private Slack +instance. + +We recommend receiving incoming email on a subdomain, such as +`incoming.hooli.com`, and ensuring that you do not employ any services that +authenticate solely based on access to an email domain such as `*.hooli.com.` +Alternatively, use a dedicated domain for GitLab email communications such as +`hooli-gitlab.com`. + +See GitLab issue [#30366](https://gitlab.com/gitlab-org/gitlab-ce/issues/30366) +for a real-world example of this exploit. + +### Omnibus package installations + +1. Find the `incoming_email` section in `/etc/gitlab/gitlab.rb`, enable the + feature and fill in the details for your specific IMAP server and email account: + + Configuration for Postfix mail server, assumes mailbox + incoming@gitlab.example.com + + ```ruby + gitlab_rails['incoming_email_enabled'] = true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + gitlab_rails['incoming_email_address'] = "incoming+%{key}@gitlab.example.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + gitlab_rails['incoming_email_email'] = "incoming" + # Email account password + gitlab_rails['incoming_email_password'] = "[REDACTED]" + + # IMAP server host + gitlab_rails['incoming_email_host'] = "gitlab.example.com" + # IMAP server port + gitlab_rails['incoming_email_port'] = 143 + # Whether the IMAP server uses SSL + gitlab_rails['incoming_email_ssl'] = false + # Whether the IMAP server uses StartTLS + gitlab_rails['incoming_email_start_tls'] = false + + # The mailbox where incoming mail will end up. Usually "inbox". + gitlab_rails['incoming_email_mailbox_name'] = "inbox" + # The IDLE command timeout. + gitlab_rails['incoming_email_idle_timeout'] = 60 + ``` + + Configuration for Gmail / Google Apps, assumes mailbox + gitlab-incoming@gmail.com + + ```ruby + gitlab_rails['incoming_email_enabled'] = true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + gitlab_rails['incoming_email_address'] = "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + gitlab_rails['incoming_email_email'] = "gitlab-incoming@gmail.com" + # Email account password + gitlab_rails['incoming_email_password'] = "[REDACTED]" + + # IMAP server host + gitlab_rails['incoming_email_host'] = "imap.gmail.com" + # IMAP server port + gitlab_rails['incoming_email_port'] = 993 + # Whether the IMAP server uses SSL + gitlab_rails['incoming_email_ssl'] = true + # Whether the IMAP server uses StartTLS + gitlab_rails['incoming_email_start_tls'] = false + + # The mailbox where incoming mail will end up. Usually "inbox". + gitlab_rails['incoming_email_mailbox_name'] = "inbox" + # The IDLE command timeout. + gitlab_rails['incoming_email_idle_timeout'] = 60 + ``` + + Configuration for Microsoft Exchange mail server w/ IMAP enabled, assumes + mailbox incoming@exchange.example.com + + ```ruby + gitlab_rails['incoming_email_enabled'] = true + + # The email address replies are sent to - Exchange does not support sub-addressing so %{key} is not used here + gitlab_rails['incoming_email_address'] = "incoming@exchange.example.com" + + # Email account username + # Typically this is the userPrincipalName (UPN) + gitlab_rails['incoming_email_email'] = "incoming@ad-domain.example.com" + # Email account password + gitlab_rails['incoming_email_password'] = "[REDACTED]" + + # IMAP server host + gitlab_rails['incoming_email_host'] = "exchange.example.com" + # IMAP server port + gitlab_rails['incoming_email_port'] = 993 + # Whether the IMAP server uses SSL + gitlab_rails['incoming_email_ssl'] = true + ``` + +1. Reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` + +1. Verify that everything is configured correctly: + + ```sh + sudo gitlab-rake gitlab:incoming_email:check + ``` + +1. Reply by email should now be working. + +### Installations from source + +1. Go to the GitLab installation directory: + + ```sh + cd /home/git/gitlab + ``` + +1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature + and fill in the details for your specific IMAP server and email account: + + ```sh + sudo editor config/gitlab.yml + ``` + + Configuration for Postfix mail server, assumes mailbox + incoming@gitlab.example.com + + ```yaml + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "incoming+%{key}@gitlab.example.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "incoming" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "gitlab.example.com" + # IMAP server port + port: 143 + # Whether the IMAP server uses SSL + ssl: false + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + # The IDLE command timeout. + idle_timeout: 60 + ``` + + Configuration for Gmail / Google Apps, assumes mailbox + gitlab-incoming@gmail.com + + ```yaml + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "gitlab-incoming@gmail.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "imap.gmail.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + # The IDLE command timeout. + idle_timeout: 60 + ``` + + Configuration for Microsoft Exchange mail server w/ IMAP enabled, assumes + mailbox incoming@exchange.example.com + + ```yaml + incoming_email: + enabled: true + + # The email address replies are sent to - Exchange does not support sub-addressing so %{key} is not used here + address: "incoming@exchange.example.com" + + # Email account username + # Typically this is the userPrincipalName (UPN) + user: "incoming@ad-domain.example.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "exchange.example.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + # The IDLE command timeout. + idle_timeout: 60 + ``` + +1. Enable `mail_room` in the init script at `/etc/default/gitlab`: + + ```sh + sudo mkdir -p /etc/default + echo 'mail_room_enabled=true' | sudo tee -a /etc/default/gitlab + ``` + +1. Restart GitLab: + + ```sh + sudo service gitlab restart + ``` + +1. Verify that everything is configured correctly: + + ```sh + sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production + ``` + +1. Reply by email should now be working. diff --git a/doc/administration/index.md b/doc/administration/index.md index 0b199eecefd..60a45426636 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -1,9 +1,19 @@ # Administrator documentation Learn how to administer your GitLab instance (Community Edition and -[Enterprise Editions](https://about.gitlab.com/gitlab-ee/)). +Enterprise Edition). Regular users don't have access to GitLab administration tools and settings. +GitLab has two product distributions: the open source +[GitLab Community Edition (CE)](https://gitlab.com/gitlab-org/gitlab-ce), +and the open core [GitLab Enterprise Edition (EE)](https://gitlab.com/gitlab-org/gitlab-ee), +available through [different subscriptions](https://about.gitlab.com/products/). + +You can [install GitLab CE or GitLab EE](https://about.gitlab.com/installation/ce-or-ee/), +but the features you'll have access to depend on the subscription you choose +(Core, Starter, Premium, or Ultimate). GitLab Community Edition installations +only have access to Core features. + GitLab.com is administered by GitLab, Inc., therefore, only GitLab team members have access to its admin configurations. If you're a GitLab.com user, please check the [user documentation](../user/index.html). @@ -69,11 +79,19 @@ created in snippets, wikis, and repos. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication/Authorization](../topics/authentication/index.md#gitlab-administrators): Enforce 2FA, configure external authentication with LDAP, SAML, CAS and additional Omniauth providers. -- [Reply by email](reply_by_email.md): Allow users to comment on issues and merge requests by replying to notification emails. - - [Postfix for Reply by email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail +- [Incoming email](incoming_email.md): Configure incoming emails to allow + users to [reply by email], create [issues by email] and + [merge requests by email], and to enable [Service Desk]. + - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a + basic Postfix mail server with IMAP authentication on Ubuntu for incoming + emails. server with IMAP authentication on Ubuntu, to be used with Reply by email. - [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. +[reply by email]: reply_by_email.md +[issues by email]: ../user/project/issues/create_new_issue.md#new-issue-via-email +[merge requests by email]: ../user/project/merge_requests/index.md#create-new-merge-requests-by-email + ## Project settings - [Container Registry](container_registry.md): Configure Container Registry with GitLab. @@ -93,6 +111,7 @@ server with IMAP authentication on Ubuntu, to be used with Reply by email. - [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#site-wide-admin-setting): Enable or disable GitLab CI/CD for your instance. - [GitLab CI/CD admin settings](../user/admin_area/settings/continuous_integration.md): Define max artifacts size and expiration time. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). +- [Job traces](job_traces.md): Information about the job traces (logs). - [Artifacts size and expiration](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size): Define maximum artifacts limits and expiration date. - [Register Shared and specific Runners](../ci/runners/README.md#registering-a-shared-runner): Learn how to register and configure Shared and specific Runners to your own instance. - [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for Shared Runners. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 65f59b72690..d978d1dca53 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -9,7 +9,19 @@ created in snippets, wikis, and repos. ## PlantUML Server Before you can enable PlantUML in GitLab; you need to set up your own PlantUML -server that will generate the diagrams. Installing and configuring your +server that will generate the diagrams. + +### Docker + +With Docker, you can just run a container like this: + +`docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat` + +The **PlantUML URL** will be the hostname of the server running the container. + +### Debian/Ubuntu + +Installing and configuring your own PlantUML server is easy in Debian/Ubuntu distributions using Tomcat. First you need to create a `plantuml.war` file from the source code: diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 28e1fd4e12e..466bb1f851e 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -24,11 +24,11 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by **For Omnibus installations** 1. Open `/etc/gitlab/gitlab.rb` with your editor. -1. Change the value of `gitlab_rails['issue_closing_pattern']` to a regular +1. Change the value of `gitlab_rails['gitlab_issue_closing_pattern']` to a regular expression of your liking: ```ruby - gitlab_rails['issue_closing_pattern'] = "((?:[Cc]los(?:e[sd]|ing)|[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" + gitlab_rails['gitlab_issue_closing_pattern'] = "((?:[Cc]los(?:e[sd]|ing)|[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" ``` 1. [Reconfigure] GitLab for the changes to take effect. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 33f8a69c249..ac3a12930c3 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -87,10 +87,124 @@ _The artifacts are stored by default in ### Using object storage -In [GitLab Enterprise Edition Premium][eep] you can use an object storage like -AWS S3 to store the artifacts. +>**Notes:** +- [Introduced][ee-1762] in [GitLab Premium][eep] 9.4. +- Since version 9.5, artifacts are [browsable], when object storage is enabled. + 9.4 lacks this feature. +> Available in [GitLab Premium](https://about.gitlab.com/products/) and +[GitLab.com Silver](https://about.gitlab.com/gitlab-com/). +> Since version 10.6, available in [GitLab CE](https://about.gitlab.com/products/) + +If you don't want to use the local disk where GitLab is installed to store the +artifacts, you can use an object storage like AWS S3 instead. +This configuration relies on valid AWS credentials to be configured already. +Use an [Object storage option][os] like AWS S3 to store job artifacts. + +### Object Storage Settings + +For source installations the following settings are nested under `artifacts:` and then `object_store:`. On omnibus installs they are prefixed by `artifacts_object_store_`. + +| Setting | Description | Default | +|---------|-------------|---------| +| `enabled` | Enable/disable object storage | `false` | +| `remote_directory` | The bucket name where Artfacts will be stored| | +| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `connection` | Various connection options described below | | + +#### S3 compatible connection settings + +The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: + +| Setting | Description | Default | +|---------|-------------|---------| +| `provider` | Always `AWS` for compatible hosts | AWS | +| `aws_access_key_id` | AWS credentials, or compatible | | +| `aws_secret_access_key` | AWS credentials, or compatible | | +| `region` | AWS region | us-east-1 | +| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | +| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | +| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false | + +**In Omnibus installations:** + +_The artifacts are stored by default in +`/var/opt/gitlab/gitlab-rails/shared/artifacts`._ + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['artifacts_enabled'] = true + gitlab_rails['artifacts_object_store_enabled'] = true + gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts" + gitlab_rails['artifacts_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: For GitLab 9.4+, if you are using AWS IAM profiles, be sure to omit the + AWS access key and secret acces key/value pairs. For example: + + ```ruby + gitlab_rails['artifacts_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Migrate any existing local artifacts to the object storage: + + ```bash + gitlab-rake gitlab:artifacts:migrate + ``` + + Currently this has to be executed manually and it will allow you to + migrate the existing artifacts to the object storage, but all new + artifacts will still be stored on the local disk. In the future + you will be given an option to define a default storage artifacts for all + new files. + +--- + +**In installations from source:** + +_The artifacts are stored by default in +`/home/git/gitlab/shared/artifacts`._ + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + artifacts: + enabled: true + object_store: + enabled: true + remote_directory: "artifacts" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. +1. Migrate any existing local artifacts to the object storage: + + ```bash + sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production + ``` -[Learn how to use the object storage option.][ee-os] + Currently this has to be executed manually and it will allow you to + migrate the existing artifacts to the object storage, but all new + artifacts will still be stored on the local disk. In the future + you will be given an option to define a default storage artifacts for all + new files. ## Expiring artifacts @@ -194,8 +308,7 @@ When clicking on a specific file, [GitLab Workhorse] extracts it from the archive and the download begins. This implementation saves space, memory and disk I/O. -[reconfigure gitlab]: restart_gitlab.md "How to restart GitLab" -[restart gitlab]: restart_gitlab.md "How to restart GitLab" +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" [gitlab workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse "GitLab Workhorse repository" -[ee-os]: https://docs.gitlab.com/ee/administration/job_artifacts.html#using-object-storage -[eep]: https://about.gitlab.com/gitlab-ee/ "GitLab Enterprise Edition Premium" +[os]: https://docs.gitlab.com/administration/job_artifacts.html#using-object-storage diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md new file mode 100644 index 00000000000..84a1ffeec98 --- /dev/null +++ b/doc/administration/job_traces.md @@ -0,0 +1,42 @@ +# Job traces (logs) + +By default, all job traces (logs) are saved to `/var/opt/gitlab/gitlab-ci/builds` +and `/home/git/gitlab/builds` for Omnibus packages and installations from source +respectively. The job logs are organized by year and month (for example, `2017_03`), +and then by project ID. + +There isn't a way to automatically expire old job logs, but it's safe to remove +them if they're taking up too much space. If you remove the logs manually, the +job output in the UI will be empty. + +## Changing the job traces location + +To change the location where the job logs will be stored, follow the steps below. + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line: + + ``` + gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds' + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + gitlab_ci: + # The location where build traces are stored (default: builds/). + # Relative paths are relative to Rails.root. + builds_path: path/to/builds/ + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. + +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" diff --git a/doc/administration/logs.md b/doc/administration/logs.md index debaa2330d0..00a2f3d01b8 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -23,7 +23,7 @@ requests from the API are logged to a separate file in `api_json.log`. Each line contains a JSON line that can be ingested by Elasticsearch, Splunk, etc. For example: ```json -{"method":"GET","path":"/gitlab/gitlab-ce/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"duration":229.03,"view":174.07,"db":13.24,"time":"2017-08-08T20:15:54.821Z","params":{"namespace_id":"gitlab","project_id":"gitlab-ce","id":"1234"},"remote_ip":"18.245.0.1","user_id":1,"username":"admin"} +{"method":"GET","path":"/gitlab/gitlab-ce/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"duration":229.03,"view":174.07,"db":13.24,"time":"2017-08-08T20:15:54.821Z","params":[{"key":"param_key","value":"param_value"}],"remote_ip":"18.245.0.1","user_id":1,"username":"admin","gitaly_calls":76} ``` In this example, you can see this was a GET request for a specific issue. Notice each line also contains performance data: @@ -31,6 +31,9 @@ In this example, you can see this was a GET request for a specific issue. Notice 1. `duration`: the total time taken to retrieve the request 2. `view`: total time taken inside the Rails views 3. `db`: total time to retrieve data from the database +4. `gitaly_calls`: total number of calls made to Gitaly + +User clone/fetch activity using http transport appears in this log as `action: git_upload_pack`. In addition, the log contains the IP address from which the request originated (`remote_ip`) as well as the user's ID (`user_id`), and username (`username`). @@ -157,6 +160,8 @@ I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git. ``` +User clone/fetch activity using ssh transport appears in this log as `executing git command <gitaly-upload-pack...`. + ## `unicorn\_stderr.log` This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index d6333ee62b4..d18dddf09c0 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -5,5 +5,6 @@ Explore our features to monitor your GitLab instance: - [Performance monitoring](performance/index.md): GitLab Performance Monitoring makes it possible to measure a wide variety of statistics of your instance. - [Prometheus](prometheus/index.md): Prometheus is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. - [GitHub imports](github_imports.md): Monitor the health and progress of GitLab's GitHub importer with various Prometheus metrics. -- [Monitoring uptime](../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. +- [Monitoring uptime](../../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - [IP whitelists](ip_whitelist.md): Configure GitLab for monitoring endpoints that provide health check information when probed. +- [nginx_status](https://docs.gitlab.com/omnibus/settings/nginx.html#enabling-disabling-nginx_status): Monitor your Nginx server status diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differindex b3c6bc474e3..48212f6276a 100644 --- a/doc/administration/monitoring/performance/img/performance_bar.png +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png Binary files differnew file mode 100644 index 00000000000..52176df9ecd --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index b9464945cea..dc4f685d843 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -11,12 +11,18 @@ It allows you to see (from left to right): - the timing of the page (backend, frontend) - time taken and number of DB queries, click through for details of these queries  +- time taken and number of [Gitaly] calls, click through for details of these calls + +- profile of the code used to generate the page, line by line. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)). + - time taken and number of calls to Redis - time taken and number of background jobs created by Sidekiq -- profile of the code used to generate the page, line by line for either _all_, _app & lib_ , or _views_. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)). - - time taken and number of Ruby GC calls +On the far right is a request selector that allows you to view the same metrics +(excluding the page timing and line profiler) for any requests made while the +page was open. Only the first two requests per unique URL are captured. + ## Enable the Performance Bar via the Admin panel GitLab Performance Bar is disabled by default. To enable it for a given group, @@ -39,3 +45,5 @@ You can toggle the Bar using the same shortcut.  --- + +[Gitaly]: ../../gitaly/index.md diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 9d1589d84aa..bd6c7bb07b5 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -1,7 +1,7 @@ # Fast lookup of authorized SSH keys in the database > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1631) in -> [GitLab Enterprise Edition Standard](https://about.gitlab.com/gitlab-ee) 9.3. +> [GitLab Starter](https://about.gitlab.com/gitlab-ee) 9.3. > > [Available in](https://gitlab.com/gitlab-org/gitlab-ee/issues/3953) GitLab > Community Edition 10.4. @@ -56,7 +56,7 @@ new one, and attempting to pull a repo. > **Warning:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. -In the case of lookup failures (which are not uncommon), the `authorized_keys` +In the case of lookup failures (which are common), the `authorized_keys` file will still be scanned. So git SSH performance will still be slow for many users as long as a large file exists. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 7d47aaac299..00c631fdaae 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -61,6 +61,21 @@ Before proceeding with the Pages configuration, you will need to: NOTE: **Note:** If your GitLab instance and the Pages daemon are deployed in a private network or behind a firewall, your GitLab Pages websites will only be accessible to devices/users that have access to the private network. +### Add the domain to the Public Suffix List + +The [Public Suffix List](https://publicsuffix.org) is used by browsers to +decide how to treat subdomains. If your GitLab instance allows members of the +public to create GitLab Pages sites, it also allows those users to create +subdomains on the pages domain (`example.io`). Adding the domain to the Public +Suffix List prevents browsers from accepting +[supercookies](https://en.wikipedia.org/wiki/HTTP_cookie#Supercookie), +among other things. + +Follow [these instructions](https://publicsuffix.org/submit/) to submit your +GitLab Pages subdomain. For instance, if your domain is `example.io`, you should +request that `*.example.io` is added to the Public Suffix List. GitLab.com +added `*.gitlab.io` [in 2016](https://gitlab.com/gitlab-com/infrastructure/issues/230). + ### DNS configuration GitLab Pages expect to run on their own virtual host. In your DNS server/provider @@ -211,6 +226,18 @@ world. Custom domains and TLS are supported. 1. [Reconfigure GitLab][reconfigure] +### Custom domain verification + +To prevent malicious users from hijacking domains that don't belong to them, +GitLab supports [custom domain verification](../../user/project/pages/getting_started_part_three.md#dns-txt-record). +When adding a custom domain, users will be required to prove they own it by +adding a GitLab-controlled verification code to the DNS records for that domain. + +If your userbase is private or otherwise trusted, you can disable the +verification requirement. Navigate to `Admin area ➔ Settings` and uncheck +**Require users to prove ownership of custom domains** in the Pages section. +This setting is enabled by default. + ## Change storage path Follow the steps below to change the default path where GitLab Pages' contents diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md new file mode 100644 index 00000000000..c91ac3012b9 --- /dev/null +++ b/doc/administration/plugins.md @@ -0,0 +1,66 @@ +# Plugins + +**Note:** Plugins must be configured on the filesystem of the GitLab +server. Only GitLab server administrators will be able to complete these tasks. +Please explore [system hooks] or [webhooks] as an option if you do not +have filesystem access. + +Introduced in GitLab 10.6. + +A plugin will run on each event so it's up to you to filter events or projects within a plugin code. You can have as many plugins as you want. Each plugin will be triggered by GitLab asynchronously in case of an event. For a list of events please see [system hooks] documentation. + +## Setup + +Plugins must be placed directly into `plugins` directory, subdirectories will be ignored. +There is an `example` directory inside `plugins` where you can find some basic examples. + +Follow the steps below to set up a custom hook: + +1. On the GitLab server, navigate to the project's plugin directory. + For an installation from source the path is usually + `/home/git/gitlab/plugins/`. For Omnibus installs the path is + usually `/opt/gitlab/embedded/service/gitlab-rails/plugins`. +1. Inside the `plugins` directory, create a file with a name of your choice, but without spaces or special characters. +1. Make the hook file executable and make sure it's owned by the git user. +1. Write the code to make the plugin function as expected. Plugin can be + in any language. Ensure the 'shebang' at the top properly reflects the language + type. For example, if the script is in Ruby the shebang will probably be + `#!/usr/bin/env ruby`. +1. The data to the plugin will be provided as JSON on STDIN. It will be exactly same as one for [system hooks] + +That's it! Assuming the plugin code is properly implemented the hook will fire +as appropriate. Plugins file list is updated for each event. There is no need to restart GitLab to apply a new plugin. + +If a plugin executes with non-zero exit code or GitLab fails to execute it, a +message will be logged to `plugin.log`. + +## Validation + +Writing own plugin can be tricky and its easier if you can check it without altering the system. +We provided a rake task you can use with staging environment to test your plugin before using it in production. +The rake task will use a sample data and execute each of plugins. By output you should be able to determine if +system sees your plugin and if it was executed without errors. + +```bash +# Omnibus installations +sudo gitlab-rake plugins:validate + +# Installations from source +bundle exec rake plugins:validate RAILS_ENV=production +``` + +Example of output can be next: + +``` +-> bundle exec rake plugins:validate RAILS_ENV=production +Validating plugins from /plugins directory +* /home/git/gitlab/plugins/save_to_file.clj succeed (zero exit code) +* /home/git/gitlab/plugins/save_to_file.rb failure (non-zero exit code) +``` + +[hooks]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks +[system hooks]: ../system_hooks/system_hooks.md +[webhooks]: ../user/project/integrations/webhooks.md +[5073]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5073 +[93]: https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/93 + diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index d1ed152b58c..7d34d35e7d1 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -78,34 +78,45 @@ Example output: ## Uploaded Files Integrity -The uploads check Rake task will loop through all uploads in the database -and run two checks to determine the integrity of each file: +Various types of file can be uploaded to a GitLab installation by users. +Checksums are generated and stored in the database upon upload, and integrity +checks using those checksums can be run. These checks also detect missing files. -1. Check if the file exist on the file system. -1. Check if the checksum of the file on the file system matches the checksum in the database. +Currently, integrity checks are supported for the following types of file: + +* CI artifacts (Available from version 10.7.0) +* LFS objects (Available from version 10.6.0) +* User uploads (Available from version 10.6.0) **Omnibus Installation** ``` +sudo gitlab-rake gitlab:artifacts:check +sudo gitlab-rake gitlab:lfs:check sudo gitlab-rake gitlab:uploads:check ``` **Source Installation** ```bash +sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production ``` -This task also accepts some environment variables which you can use to override +These tasks also accept some environment variables which you can use to override certain values: -Variable | Type | Description --------- | ---- | ----------- -`BATCH` | integer | Specifies the size of the batch. Defaults to 200. -`ID_FROM` | integer | Specifies the ID to start from, inclusive of the value. -`ID_TO` | integer | Specifies the ID value to end at, inclusive of the value. +Variable | Type | Description +--------- | ------- | ----------- +`BATCH` | integer | Specifies the size of the batch. Defaults to 200. +`ID_FROM` | integer | Specifies the ID to start from, inclusive of the value. +`ID_TO` | integer | Specifies the ID value to end at, inclusive of the value. +`VERBOSE` | boolean | Causes failures to be listed individually, rather than being summarized. ```bash +sudo gitlab-rake gitlab:artifacts:check BATCH=100 ID_FROM=50 ID_TO=250 +sudo gitlab-rake gitlab:lfs:check BATCH=100 ID_FROM=50 ID_TO=250 sudo gitlab-rake gitlab:uploads:check BATCH=100 ID_FROM=50 ID_TO=250 ``` diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index ecf92c379fd..2b4252a7b1d 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -240,3 +240,31 @@ sudo gitlab-rake gitlab:tcp_check[example.com,80] cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:tcp_check[example.com,80] RAILS_ENV=production ``` + +## Clear exclusive lease (DANGER) + +GitLab uses a shared lock mechanism: `ExclusiveLease` to prevent simultaneous operations +in a shared resource. An example is running periodic garbage collection on repositories. + +In very specific situations, a operation locked by an Exclusive Lease can fail without +releasing the lock. If you can't wait for it to expire, you can run this task to manually +clear it. + +To clear all exclusive leases: + +DANGER: **DANGER**: +Don't run it while GitLab or Sidekiq is running + +```bash +sudo gitlab-rake gitlab:exclusive_lease:clear +``` + +To specify a lease `type` or lease `type + id`, specify a scope: + +```bash +# to clear all leases for repository garbage collection: +sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:*] + +# to clear a lease for repository garbage collection in a specific project: (id=4) +sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:4] +``` diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md new file mode 100644 index 00000000000..0cd33ffc122 --- /dev/null +++ b/doc/administration/raketasks/uploads/migrate.md @@ -0,0 +1,74 @@ +# Uploads Migrate Rake Task + +## Migrate to Object Storage + +After [configuring the object storage](../../uploads.md#using-object-storage) for GitLab's uploads, you may use this task to migrate existing uploads from the local storage to the remote storage. + +>**Note:** +All of the processing will be done in a background worker and requires **no downtime**. + +This tasks uses 3 parameters to find uploads to migrate. + +>**Note:** +These parameters are mainly internal to GitLab's structure, you may want to refer to the task list instead below. + +Parameter | Type | Description +--------- | ---- | ----------- +`uploader_class` | string | Type of the uploader to migrate from +`model_class` | string | Type of the model to migrate from +`mount_point` | string/symbol | Name of the model's column on which the uploader is mounted on. + +This task also accepts some environment variables which you can use to override +certain values: + +Variable | Type | Description +-------- | ---- | ----------- +`BATCH` | integer | Specifies the size of the batch. Defaults to 200. + +** Omnibus Installation** + +```bash +# gitlab-rake gitlab:uploads:migrate[uploader_class, model_class, mount_point] + +# Avatars +gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]" +gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]" +gitlab-rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]" + +# Attachments +gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]" +gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]" +gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]" + +# Markdown +gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]" +gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]" +gitlab-rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]" +gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" +``` + +**Source Installation** + +>**Note:** +Use `RAILS_ENV=production` for every task. + +```bash +# sudo -u git -H bundle exec rake gitlab:uploads:migrate + +# Avatars +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]" +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]" +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]" + +# Attachments +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]" +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]" +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]" + +# Markdown +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, Project]" +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]" +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]" +sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" + +``` diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index 3a2cced37bf..426245c7aca 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -5,33 +5,7 @@ replying to notification emails. ## Requirement -Reply by email requires an IMAP-enabled email account. GitLab allows you to use -three strategies for this feature: -- using email sub-addressing -- using a dedicated email address -- using a catch-all mailbox - -### Email sub-addressing - -**If your provider or server supports email sub-addressing, we recommend using it. -Some features (e.g. create new issue via email) only work with sub-addressing.** - -[Sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing) is -a feature where any email to `user+some_arbitrary_tag@example.com` will end up -in the mailbox for `user@example.com`, and is supported by providers such as -Gmail, Google Apps, Yahoo! Mail, Outlook.com and iCloud, as well as the Postfix -mail server which you can run on-premises. - -### Dedicated email address - -This solution is really simple to set up: you just have to create an email -address dedicated to receive your users' replies to GitLab notifications. - -### Catch-all mailbox - -A [catch-all mailbox](https://en.wikipedia.org/wiki/Catch-all) for a domain will -"catch all" the emails addressed to the domain that do not exist in the mail -server. +Make sure [incoming email](incoming_email.md) is setup. ## How it works? @@ -65,329 +39,3 @@ the entity the notification was about (issue, merge request, commit...). For more details about the `Message-ID`, `In-Reply-To`, and `References headers`, please consult [RFC 5322](https://tools.ietf.org/html/rfc5322#section-3.6.4). - -## Set it up - -If you want to use Gmail / Google Apps with Reply by email, make sure you have -[IMAP access enabled](https://support.google.com/mail/troubleshooter/1668960?hl=en#ts=1665018) -and [allowed less secure apps to access the account](https://support.google.com/accounts/answer/6010255) -or [turn-on 2-step validation](https://support.google.com/accounts/answer/185839) -and use [an application password](https://support.google.com/mail/answer/185833). - -To set up a basic Postfix mail server with IMAP access on Ubuntu, follow the -[Postfix setup documentation](reply_by_email_postfix_setup.md). - -### Security Concerns - -**WARNING:** Be careful when choosing the domain used for receiving incoming -email. - -For the sake of example, suppose your top-level company domain is `hooli.com`. -All employees in your company have an email address at that domain via Google -Apps, and your company's private Slack instance requires a valid `@hooli.com` -email address in order to sign up. - -If you also host a public-facing GitLab instance at `hooli.com` and set your -incoming email domain to `hooli.com`, an attacker could abuse the "Create new -issue by email" or -"[Create new merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email)" -features by using a project's unique address as the email when signing up for -Slack, which would send a confirmation email, which would create a new issue or -merge request on the project owned by the attacker, allowing them to click the -confirmation link and validate their account on your company's private Slack -instance. - -We recommend receiving incoming email on a subdomain, such as -`incoming.hooli.com`, and ensuring that you do not employ any services that -authenticate solely based on access to an email domain such as `*.hooli.com.` -Alternatively, use a dedicated domain for GitLab email communications such as -`hooli-gitlab.com`. - -See GitLab issue [#30366](https://gitlab.com/gitlab-org/gitlab-ce/issues/30366) -for a real-world example of this exploit. - -### Omnibus package installations - -1. Find the `incoming_email` section in `/etc/gitlab/gitlab.rb`, enable the - feature and fill in the details for your specific IMAP server and email account: - - ```ruby - # Configuration for Postfix mail server, assumes mailbox incoming@gitlab.example.com - gitlab_rails['incoming_email_enabled'] = true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - gitlab_rails['incoming_email_address'] = "incoming+%{key}@gitlab.example.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - gitlab_rails['incoming_email_email'] = "incoming" - # Email account password - gitlab_rails['incoming_email_password'] = "[REDACTED]" - - # IMAP server host - gitlab_rails['incoming_email_host'] = "gitlab.example.com" - # IMAP server port - gitlab_rails['incoming_email_port'] = 143 - # Whether the IMAP server uses SSL - gitlab_rails['incoming_email_ssl'] = false - # Whether the IMAP server uses StartTLS - gitlab_rails['incoming_email_start_tls'] = false - - # The mailbox where incoming mail will end up. Usually "inbox". - gitlab_rails['incoming_email_mailbox_name'] = "inbox" - # The IDLE command timeout. - gitlab_rails['incoming_email_idle_timeout'] = 60 - ``` - - ```ruby - # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com - gitlab_rails['incoming_email_enabled'] = true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - gitlab_rails['incoming_email_address'] = "gitlab-incoming+%{key}@gmail.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - gitlab_rails['incoming_email_email'] = "gitlab-incoming@gmail.com" - # Email account password - gitlab_rails['incoming_email_password'] = "[REDACTED]" - - # IMAP server host - gitlab_rails['incoming_email_host'] = "imap.gmail.com" - # IMAP server port - gitlab_rails['incoming_email_port'] = 993 - # Whether the IMAP server uses SSL - gitlab_rails['incoming_email_ssl'] = true - # Whether the IMAP server uses StartTLS - gitlab_rails['incoming_email_start_tls'] = false - - # The mailbox where incoming mail will end up. Usually "inbox". - gitlab_rails['incoming_email_mailbox_name'] = "inbox" - # The IDLE command timeout. - gitlab_rails['incoming_email_idle_timeout'] = 60 - ``` - - ```ruby - # Configuration for Microsoft Exchange mail server w/ IMAP enabled, assumes mailbox incoming@exchange.example.com - gitlab_rails['incoming_email_enabled'] = true - - # The email address replies are sent to - Exchange does not support sub-addressing so %{key} is not used here - gitlab_rails['incoming_email_address'] = "incoming@exchange.example.com" - - # Email account username - # Typically this is the userPrincipalName (UPN) - gitlab_rails['incoming_email_email'] = "incoming@ad-domain.example.com" - # Email account password - gitlab_rails['incoming_email_password'] = "[REDACTED]" - - # IMAP server host - gitlab_rails['incoming_email_host'] = "exchange.example.com" - # IMAP server port - gitlab_rails['incoming_email_port'] = 993 - # Whether the IMAP server uses SSL - gitlab_rails['incoming_email_ssl'] = true - ``` - -1. Reconfigure GitLab for the changes to take effect: - - ```sh - sudo gitlab-ctl reconfigure - ``` - -1. Verify that everything is configured correctly: - - ```sh - sudo gitlab-rake gitlab:incoming_email:check - ``` - -1. Reply by email should now be working. - -### Installations from source - -1. Go to the GitLab installation directory: - - ```sh - cd /home/git/gitlab - ``` - -1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature - and fill in the details for your specific IMAP server and email account: - - ```sh - sudo editor config/gitlab.yml - ``` - - ```yaml - # Configuration for Postfix mail server, assumes mailbox incoming@gitlab.example.com - incoming_email: - enabled: true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - address: "incoming+%{key}@gitlab.example.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - user: "incoming" - # Email account password - password: "[REDACTED]" - - # IMAP server host - host: "gitlab.example.com" - # IMAP server port - port: 143 - # Whether the IMAP server uses SSL - ssl: false - # Whether the IMAP server uses StartTLS - start_tls: false - - # The mailbox where incoming mail will end up. Usually "inbox". - mailbox: "inbox" - # The IDLE command timeout. - idle_timeout: 60 - ``` - - ```yaml - # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com - incoming_email: - enabled: true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - address: "gitlab-incoming+%{key}@gmail.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - user: "gitlab-incoming@gmail.com" - # Email account password - password: "[REDACTED]" - - # IMAP server host - host: "imap.gmail.com" - # IMAP server port - port: 993 - # Whether the IMAP server uses SSL - ssl: true - # Whether the IMAP server uses StartTLS - start_tls: false - - # The mailbox where incoming mail will end up. Usually "inbox". - mailbox: "inbox" - # The IDLE command timeout. - idle_timeout: 60 - ``` - - ```yaml - # Configuration for Microsoft Exchange mail server w/ IMAP enabled, assumes mailbox incoming@exchange.example.com - incoming_email: - enabled: true - - # The email address replies are sent to - Exchange does not support sub-addressing so %{key} is not used here - address: "incoming@exchange.example.com" - - # Email account username - # Typically this is the userPrincipalName (UPN) - user: "incoming@ad-domain.example.com" - # Email account password - password: "[REDACTED]" - - # IMAP server host - host: "exchange.example.com" - # IMAP server port - port: 993 - # Whether the IMAP server uses SSL - ssl: true - # Whether the IMAP server uses StartTLS - start_tls: false - - # The mailbox where incoming mail will end up. Usually "inbox". - mailbox: "inbox" - # The IDLE command timeout. - idle_timeout: 60 - ``` - -1. Enable `mail_room` in the init script at `/etc/default/gitlab`: - - ```sh - sudo mkdir -p /etc/default - echo 'mail_room_enabled=true' | sudo tee -a /etc/default/gitlab - ``` - -1. Restart GitLab: - - ```sh - sudo service gitlab restart - ``` - -1. Verify that everything is configured correctly: - - ```sh - sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production - ``` - -1. Reply by email should now be working. - -### Development - -1. Go to the GitLab installation directory. - -1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and fill in the details for your specific IMAP server and email account: - - ```yaml - # Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com - incoming_email: - enabled: true - - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). - address: "gitlab-incoming+%{key}@gmail.com" - - # Email account username - # With third party providers, this is usually the full email address. - # With self-hosted email servers, this is usually the user part of the email address. - user: "gitlab-incoming@gmail.com" - # Email account password - password: "[REDACTED]" - - # IMAP server host - host: "imap.gmail.com" - # IMAP server port - port: 993 - # Whether the IMAP server uses SSL - ssl: true - # Whether the IMAP server uses StartTLS - start_tls: false - - # The mailbox where incoming mail will end up. Usually "inbox". - mailbox: "inbox" - # The IDLE command timeout. - idle_timeout: 60 - ``` - - As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. - -1. Uncomment the `mail_room` line in your `Procfile`: - - ```yaml - mail_room: bundle exec mail_room -q -c config/mail_room.yml - ``` - -1. Restart GitLab: - - ```sh - bundle exec foreman start - ``` - -1. Verify that everything is configured correctly: - - ```sh - bundle exec rake gitlab:incoming_email:check RAILS_ENV=development - ``` - -1. Reply by email should now be working. diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index a1bb3851951..3e8b78e56d5 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -1,7 +1,7 @@ -# Set up Postfix for Reply by email +# Set up Postfix for incoming email This document will take you through the steps of setting up a basic Postfix mail -server with IMAP authentication on Ubuntu, to be used with [Reply by email]. +server with IMAP authentication on Ubuntu, to be used with [incoming email]. The instructions make the assumption that you will be using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. @@ -177,12 +177,12 @@ Courier, which we will install later to add IMAP authentication, requires mailbo ```sh sudo apt-get install courier-imap ``` - + And start `imapd`: ```sh imapd start ``` - + 1. The courier-authdaemon isn't started after installation. Without it, imap authentication will fail: ```sh sudo service courier-authdaemon start @@ -329,10 +329,10 @@ Courier, which we will install later to add IMAP authentication, requires mailbo ## Done! -If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [Reply by email](./reply_by_email.md) guide to configure GitLab. +If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [incoming email] guide to configure GitLab. --- _This document was adapted from https://help.ubuntu.com/community/PostfixBasicSetupHowto, by contributors to the Ubuntu documentation wiki._ -[reply by email]: reply_by_email.md +[incoming email]: incoming_email.md diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 21184fed6e9..39bd19ac851 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -4,50 +4,63 @@ ## Legacy Storage -Legacy Storage is the storage behavior prior to version 10.0. For historical reasons, GitLab replicated the same -mapping structure from the projects URLs: +Legacy Storage is the storage behavior prior to version 10.0. For historical +reasons, GitLab replicated the same mapping structure from the projects URLs: - * Project's repository: `#{namespace}/#{project_name}.git` - * Project's wiki: `#{namespace}/#{project_name}.wiki.git` +* Project's repository: `#{namespace}/#{project_name}.git` +* Project's wiki: `#{namespace}/#{project_name}.wiki.git` -This structure made simple to migrate from existing solutions to GitLab and easy for Administrators to find where the -repository is stored. +This structure made it simple to migrate from existing solutions to GitLab and +easy for Administrators to find where the repository is stored. On the other hand this has some drawbacks: -Storage location will concentrate huge amount of top-level namespaces. The impact can be reduced by the introduction of [multiple storage paths][storage-paths]. +Storage location will concentrate huge amount of top-level namespaces. The +impact can be reduced by the introduction of [multiple storage +paths][storage-paths]. -Because Backups are a snapshot of the same URL mapping, if you try to recover a very old backup, you need to verify -if any project has taken the place of an old removed project sharing the same URL. This means that `mygroup/myproject` -from your backup may not be the same original project that is today in the same URL. +Because backups are a snapshot of the same URL mapping, if you try to recover a +very old backup, you need to verify whether any project has taken the place of +an old removed or renamed project sharing the same URL. This means that +`mygroup/myproject` from your backup may not be the same original project that +is at that same URL today. -Any change in the URL will need to be reflected on disk (when groups / users or projects are renamed). This can add a lot -of load in big installations, and can be even worst if they are using any type of network based filesystem. +Any change in the URL will need to be reflected on disk (when groups / users or +projects are renamed). This can add a lot of load in big installations, +especially if using any type of network based filesystem. -Last, for GitLab Geo, this storage type means we have to synchronize the disk state, replicate renames in the correct -order or we may end-up with wrong repository or missing data temporarily. +For GitLab Geo in particular: Geo does work with legacy storage, but in some +edge cases due to race conditions it can lead to errors when a project is +renamed multiple times in short succession, or a project is deleted and +recreated under the same name very quickly. We expect these race events to be +rare, and we have not observed a race condition side-effect happening yet. -This pattern also exists in other objects stored in GitLab, like issue Attachments, GitLab Pages artifacts, -Docker Containers for the integrated Registry, etc. +This pattern also exists in other objects stored in GitLab, like issue +Attachments, GitLab Pages artifacts, Docker Containers for the integrated +Registry, etc. ## Hashed Storage -Hashed Storage is the new storage behavior we are rolling out with 10.0. It's not enabled by default yet, but we -encourage everyone to try-it and take the time to fix any script you may have that depends on the old behavior. +> **Warning:** Hashed storage is in **Beta**. For the latest updates, check the +> associated [issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821) +> and please report any problems you encounter. -Instead of coupling project URL and the folder structure where the repository will be stored on disk, we are coupling -a hash, based on the project's ID. +Hashed Storage is the new storage behavior we are rolling out with 10.0. Instead +of coupling project URL and the folder structure where the repository will be +stored on disk, we are coupling a hash, based on the project's ID. This makes +the folder structure immutable, and therefore eliminates any requirement to +synchronize state from URLs to disk structure. This means that renaming a group, +user, or project will cost only the database transaction, and will take effect +immediately. -This makes the folder structure immutable, and therefore eliminates any requirement to synchronize state from URLs to -disk structure. This means that renaming a group, user or project will cost only the database transaction, and will take -effect immediately. +The hash also helps to spread the repositories more evenly on the disk, so the +top-level directory will contain less folders than the total amount of top-level +namespaces. -The hash also helps to spread the repositories more evenly on the disk, so the top-level directory will contain less -folders than the total amount of top-level namespaces. - -Hash format is based on hexadecimal representation of SHA256: `SHA256(project.id)`. -Top-level folder uses first 2 characters, followed by another folder with the next 2 characters. They are both stored in -a special folder `@hashed`, to co-exist with existing Legacy projects: +The hash format is based on the hexadecimal representation of SHA256: +`SHA256(project.id)`. The top-level folder uses the first 2 characters, followed +by another folder with the next 2 characters. They are both stored in a special +`@hashed` folder, to be able to co-exist with existing Legacy Storage projects: ```ruby # Project's repository: @@ -57,15 +70,13 @@ a special folder `@hashed`, to co-exist with existing Legacy projects: "@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git" ``` -This new format also makes possible to restore backups with confidence, as when restoring a repository from the backup, -you will never mistakenly restore a repository in the wrong project (considering the backup is made after the migration). - ### How to migrate to Hashed Storage -In GitLab, go to **Admin > Settings**, find the **Repository Storage** section and select -"_Create new projects using hashed storage paths_". +In GitLab, go to **Admin > Settings**, find the **Repository Storage** section +and select "_Create new projects using hashed storage paths_". -To migrate your existing projects to the new storage type, check the specific [rake tasks]. +To migrate your existing projects to the new storage type, check the specific +[rake tasks]. [ce-28283]: https://gitlab.com/gitlab-org/gitlab-ce/issues/28283 [rake tasks]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage @@ -73,11 +84,13 @@ To migrate your existing projects to the new storage type, check the specific [r ### Hashed Storage coverage -We are incrementally moving every storable object in GitLab to the Hashed Storage pattern. You can check the current -coverage status below. +We are incrementally moving every storable object in GitLab to the Hashed +Storage pattern. You can check the current coverage status below (and also see +the [issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821)). -Note that things stored in an S3 compatible endpoint will not have the downsides mentioned earlier, if they are not -prefixed with `#{namespace}/#{project_name}`, which is true for CI Cache and LFS Objects. +Note that things stored in an S3 compatible endpoint will not have the downsides +mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, +which is true for CI Cache and LFS Objects. | Storable Object | Legacy Storage | Hashed Storage | S3 Compatible | GitLab Version | | --------------- | -------------- | -------------- | ------------- | -------------- | @@ -87,6 +100,6 @@ prefixed with `#{namespace}/#{project_name}`, which is true for CI Cache and LFS | Pages | Yes | No | - | - | | Docker Registry | Yes | No | - | - | | CI Build Logs | No | No | - | - | -| CI Artifacts | No | No | Yes (EEP) | - | +| CI Artifacts | No | No | Yes (Premium) | - | | CI Cache | No | No | Yes | - | -| LFS Objects | Yes | No | Yes (EEP) | - | +| LFS Objects | Yes | No | Yes (Premium) | - | diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md new file mode 100644 index 00000000000..a82735cc72c --- /dev/null +++ b/doc/administration/uploads.md @@ -0,0 +1,209 @@ +# Uploads administration + +>**Notes:** +Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled. + +### Using local storage + +>**Notes:** +This is the default configuration + +To change the location where the uploads are stored locally, follow the steps +below. + +--- + +**In Omnibus installations:** + +>**Notes:** +For historical reasons, uploads are stored into a base directory, which by default is `uploads/-/system`. It is strongly discouraged to change this configuration option on an existing GitLab installation. + +_The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/public/uploads/-/system`._ + +1. To change the storage path for example to `/mnt/storage/uploads`, edit + `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['uploads_storage_path'] = "/mnt/storage/" + gitlab_rails['uploads_base_dir'] = "uploads" + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**In installations from source:** + +_The uploads are stored by default in +`/home/git/gitlab/public/uploads/-/system`._ + +1. To change the storage path for example to `/mnt/storage/uploads`, edit + `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + uploads: + storage_path: /mnt/storage + base_dir: uploads + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. + +### Using object storage + +>**Notes:** +- [Introduced][ee-3867] in [GitLab Enterprise Edition Premium][eep] 10.5. + +If you don't want to use the local disk where GitLab is installed to store the +uploads, you can use an object storage provider like AWS S3 instead. +This configuration relies on valid AWS credentials to be configured already. + +### Object Storage Settings + +For source installations the following settings are nested under `uploads:` and then `object_store:`. On omnibus installs they are prefixed by `uploads_object_store_`. + +| Setting | Description | Default | +|---------|-------------|---------| +| `enabled` | Enable/disable object storage | `false` | +| `remote_directory` | The bucket name where Uploads will be stored| | +| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `connection` | Various connection options described below | | + +#### S3 compatible connection settings + +The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: + +| Setting | Description | Default | +|---------|-------------|---------| +| `provider` | Always `AWS` for compatible hosts | AWS | +| `aws_access_key_id` | AWS credentials, or compatible | | +| `aws_secret_access_key` | AWS credentials, or compatible | | +| `region` | AWS region | us-east-1 | +| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | +| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | +| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false | + +**In Omnibus installations:** + +_The uploads are stored by default in +`/var/opt/gitlab/gitlab-rails/public/uploads/-/system`._ + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['uploads_object_store_enabled'] = true + gitlab_rails['uploads_object_store_remote_directory'] = "uploads" + gitlab_rails['uploads_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + +>**Note:** +If you are using AWS IAM profiles, be sure to omit the AWS access key and secret acces key/value pairs. + + ```ruby + gitlab_rails['uploads_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Migrate any existing local uploads to the object storage: + +>**Notes:** +These task complies with the `BATCH` environment variable to process uploads in batch (200 by default). All of the processing will be done in a background worker and requires **no downtime**. + + ```bash + # gitlab-rake gitlab:uploads:migrate[uploader_class, model_class, mount_point] + + # Avatars + gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]" + gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]" + gitlab-rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]" + + # Attachments + gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]" + gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]" + gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]" + + # Markdown + gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]" + gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]" + gitlab-rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]" + gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" + ``` + + Currently this has to be executed manually and it will allow you to + migrate the existing uploads to the object storage, but all new + uploads will still be stored on the local disk. In the future + you will be given an option to define a default storage for all + new files. + +--- + +**In installations from source:** + +_The uploads are stored by default in +`/home/git/gitlab/public/uploads/-/system`._ + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + uploads: + object_store: + enabled: true + remote_directory: "uploads" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. +1. Migrate any existing local uploads to the object storage: + +>**Notes:** + +- These task comply with the `BATCH` environment variable to process uploads in batch (200 by default). All of the processing will be done in a background worker and requires **no downtime**. + +- To migrate in production use `RAILS_ENV=production` environment variable. + + ```bash + # sudo -u git -H bundle exec rake gitlab:uploads:migrate + + # Avatars + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]" + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]" + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]" + + # Attachments + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]" + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]" + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]" + + # Markdown + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, Project]" + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]" + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]" + sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" + + ``` + + Currently this has to be executed manually and it will allow you to + migrate the existing uploads to the object storage, but all new + uploads will still be stored on the local disk. In the future + you will be given an option to define a default storage for all + new files. + +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" +[eep]: https://about.gitlab.com/gitlab-ee/ "GitLab Enterprise Edition Premium" +[ee-3867]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3867 diff --git a/doc/api/README.md b/doc/api/README.md index f226716c3b5..ae4481b400e 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -24,9 +24,11 @@ following locations: - [GitLab CI Config templates](templates/gitlab_ci_ymls.md) - [Groups](groups.md) - [Group Access Requests](access_requests.md) +- [Group Badges](group_badges.md) - [Group Members](members.md) - [Issues](issues.md) - [Issue Boards](boards.md) +- [Group Issue Boards](group_boards.md) - [Jobs](jobs.md) - [Keys](keys.md) - [Labels](labels.md) @@ -35,6 +37,7 @@ following locations: - [Group milestones](group_milestones.md) - [Namespaces](namespaces.md) - [Notes](notes.md) (comments) +- [Discussions](discussions.md) (threaded comments) - [Notification settings](notification_settings.md) - [Open source license templates](templates/licenses.md) - [Pages Domains](pages_domains.md) @@ -43,12 +46,15 @@ following locations: - [Pipeline Schedules](pipeline_schedules.md) - [Projects](projects.md) including setting Webhooks - [Project Access Requests](access_requests.md) +- [Project Badges](project_badges.md) +- [Project import/export](project_import_export.md) - [Project Members](members.md) - [Project Snippets](project_snippets.md) - [Protected Branches](protected_branches.md) - [Repositories](repositories.md) - [Repository Files](repository_files.md) - [Runners](runners.md) +- [Search](search.md) - [Services](services.md) - [Settings](settings.md) - [Sidekiq metrics](sidekiq_metrics.md) diff --git a/doc/api/branches.md b/doc/api/branches.md index 80744258acb..01bb30c3859 100644 --- a/doc/api/branches.md +++ b/doc/api/branches.md @@ -13,6 +13,7 @@ GET /projects/:id/repository/branches | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `search` | string | no | Return list of branches matching the search criteria. | ```bash curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/repository/branches diff --git a/doc/api/commits.md b/doc/api/commits.md index 63554c63057..a6b96ba539f 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -14,6 +14,9 @@ GET /projects/:id/repository/commits | `ref_name` | string | no | The name of a repository branch or tag or if not given the default branch | | `since` | string | no | Only commits after or on this date will be returned in ISO 8601 format YYYY-MM-DDTHH:MM:SSZ | | `until` | string | no | Only commits before or on this date will be returned in ISO 8601 format YYYY-MM-DDTHH:MM:SSZ | +| `path` | string | no | The file path | +| `all` | boolean | no | Retrieve every commit from the repository | + ```bash curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/5/repository/commits" @@ -198,6 +201,41 @@ Example response: } ``` +## Get references a commit is pushed to + +> [Introduced][ce-15026] in GitLab 10.6 + +Get all references (from branches or tags) a commit is pushed to. +The pagination parameters `page` and `per_page` can be used to restrict the list of references. + +``` +GET /projects/:id/repository/commits/:sha/refs +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +| `sha` | string | yes | The commit hash | +| `type` | string | no | The scope of commits. Possible values `branch`, `tag`, `all`. Default is `all`. | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/5/repository/commits/5937ac0a7beb003549fc5fd26fc247adbce4a52e/refs?type=all" +``` + +Example response: + +```json +[ + {"type": "branch", "name": "'test'"}, + {"type": "branch", "name": "add-balsamiq-file"}, + {"type": "branch", "name": "wip"}, + {"type": "tag", "name": "v1.1.0"} + ] + +``` + ## Cherry pick a commit > [Introduced][ce-8047] in GitLab 8.15. @@ -498,5 +536,74 @@ Example response: } ``` +## List Merge Requests associated with a commit + +Get a list of Merge Requests related to the specified commit. + +``` +GET /projects/:id/repository/commits/:sha/merge_requests +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +| `sha` | string | yes | The commit SHA + + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/5/repository/commits/af5b13261899fb2c0db30abdd0af8b07cb44fdc5/merge_requests" +``` + +Example response: + +```json +[ + { + "id":45, + "iid":1, + "project_id":35, + "title":"Add new file", + "description":"", + "state":"opened", + "created_at":"2018-03-26T17:26:30.916Z", + "updated_at":"2018-03-26T17:26:30.916Z", + "target_branch":"master", + "source_branch":"test-branch", + "upvotes":0, + "downvotes":0, + "author" : { + "web_url" : "https://gitlab.example.com/thedude", + "name" : "Jeff Lebowski", + "avatar_url" : "https://gitlab.example.com/uploads/user/avatar/28/The-Big-Lebowski-400-400.png", + "username" : "thedude", + "state" : "active", + "id" : 28 + }, + "assignee":null, + "source_project_id":35, + "target_project_id":35, + "labels":[ ], + "work_in_progress":false, + "milestone":null, + "merge_when_pipeline_succeeds":false, + "merge_status":"can_be_merged", + "sha":"af5b13261899fb2c0db30abdd0af8b07cb44fdc5", + "merge_commit_sha":null, + "user_notes_count":0, + "discussion_locked":null, + "should_remove_source_branch":null, + "force_remove_source_branch":false, + "web_url":"http://https://gitlab.example.com/root/test-project/merge_requests/1", + "time_stats":{ + "time_estimate":0, + "total_time_spent":0, + "human_time_estimate":null, + "human_total_time_spent":null + } + } +] +``` + [ce-6096]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6096 "Multi-file commit" [ce-8047]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8047 +[ce-15026]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15026 diff --git a/doc/api/discussions.md b/doc/api/discussions.md new file mode 100644 index 00000000000..c341b7f2009 --- /dev/null +++ b/doc/api/discussions.md @@ -0,0 +1,411 @@ +# Discussions API + +Discussions are set of related notes on snippets or issues. + +## Issues + +### List project issue discussions + +Gets a list of all discussions for a single issue. + +``` +GET /projects/:id/issues/:issue_iid/discussions +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `issue_iid` | integer | yes | The IID of an issue | + +```json +[ + { + "id": "6a9c1750b37d513a43987b574953fceb50b03ce7", + "individual_note": false, + "notes": [ + { + "id": 1126, + "type": "DiscussionNote", + "body": "discussion text", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-03T21:54:39.668Z", + "updated_at": "2018-03-03T21:54:39.668Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Issue", + "noteable_iid": null + }, + { + "id": 1129, + "type": "DiscussionNote", + "body": "reply to the discussion", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-04T13:38:02.127Z", + "updated_at": "2018-03-04T13:38:02.127Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Issue", + "noteable_iid": null + } + ] + }, + { + "id": "87805b7c09016a7058e91bdbe7b29d1f284a39e6", + "individual_note": true, + "notes": [ + { + "id": 1128, + "type": null, + "body": "a single comment", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-04T09:17:22.520Z", + "updated_at": "2018-03-04T09:17:22.520Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Issue", + "noteable_iid": null + } + ] + } +] +``` + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions +``` + +### Get single issue discussion + +Returns a single discussion for a specific project issue + +``` +GET /projects/:id/issues/:issue_iid/discussions/:discussion_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `issue_iid` | integer | yes | The IID of an issue | +| `discussion_id` | integer | yes | The ID of a discussion | + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 +``` + +### Create new issue discussion + +Creates a new discussion to a single project issue. This is similar to creating +a note but but another comments (replies) can be added to it later. + +``` +POST /projects/:id/issues/:issue_iid/discussions +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `issue_iid` | integer | yes | The IID of an issue | +| `body` | string | yes | The content of a discussion | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions?body=comment +``` + +### Add note to existing issue discussion + +Adds a new note to the discussion. + +``` +POST /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `issue_iid` | integer | yes | The IID of an issue | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | +| `body` | string | yes | The content of a discussion | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment +``` + +### Modify existing issue discussion note + +Modify existing discussion note of an issue. + +``` +PUT /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `issue_iid` | integer | yes | The IID of an issue | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | +| `body` | string | yes | The content of a discussion | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?body=comment +``` + +### Delete an issue discussion note + +Deletes an existing discussion note of an issue. + +``` +DELETE /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `issue_iid` | integer | yes | The IID of an issue | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions/636 +``` + +## Snippets + +### List project snippet discussions + +Gets a list of all discussions for a single snippet. + +``` +GET /projects/:id/snippets/:snippet_id/discussions +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of an snippet | + +```json +[ + { + "id": "6a9c1750b37d513a43987b574953fceb50b03ce7", + "individual_note": false, + "notes": [ + { + "id": 1126, + "type": "DiscussionNote", + "body": "discussion text", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-03T21:54:39.668Z", + "updated_at": "2018-03-03T21:54:39.668Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Snippet", + "noteable_id": null + }, + { + "id": 1129, + "type": "DiscussionNote", + "body": "reply to the discussion", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-04T13:38:02.127Z", + "updated_at": "2018-03-04T13:38:02.127Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Snippet", + "noteable_id": null + } + ] + }, + { + "id": "87805b7c09016a7058e91bdbe7b29d1f284a39e6", + "individual_note": true, + "notes": [ + { + "id": 1128, + "type": null, + "body": "a single comment", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-04T09:17:22.520Z", + "updated_at": "2018-03-04T09:17:22.520Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Snippet", + "noteable_id": null + } + ] + } +] +``` + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions +``` + +### Get single snippet discussion + +Returns a single discussion for a specific project snippet + +``` +GET /projects/:id/snippets/:snippet_id/discussions/:discussion_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of an snippet | +| `discussion_id` | integer | yes | The ID of a discussion | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 +``` + +### Create new snippet discussion + +Creates a new discussion to a single project snippet. This is similar to creating +a note but but another comments (replies) can be added to it later. + +``` +POST /projects/:id/snippets/:snippet_id/discussions +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of an snippet | +| `body` | string | yes | The content of a discussion | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions?body=comment +``` + +### Add note to existing snippet discussion + +Adds a new note to the discussion. + +``` +POST /projects/:id/snippets/:snippet_id/discussions/:discussion_id/notes +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of an snippet | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | +| `body` | string | yes | The content of a discussion | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment +``` + +### Modify existing snippet discussion note + +Modify existing discussion note of an snippet. + +``` +PUT /projects/:id/snippets/:snippet_id/discussions/:discussion_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of an snippet | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | +| `body` | string | yes | The content of a discussion | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?body=comment +``` + +### Delete an snippet discussion note + +Deletes an existing discussion note of an snippet. + +``` +DELETE /projects/:id/snippets/:snippet_id/discussions/:discussion_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `snippet_id` | integer | yes | The ID of an snippet | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/636 +``` diff --git a/doc/api/events.md b/doc/api/events.md index 129af0afa35..f4d26c4de1c 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -42,6 +42,10 @@ Dates for the `before` and `after` parameters should be supplied in the followin YYYY-MM-DD ``` +### Event Time Period Limit + +GitLab removes events older than 1 year from the events table for performance reasons. The range of 1 year was chosen because user contribution calendars only show contributions of the past year. + ## List currently authenticated user's events >**Note:** This endpoint was introduced in GitLab 9.3. diff --git a/doc/api/group_badges.md b/doc/api/group_badges.md new file mode 100644 index 00000000000..3e0683f378d --- /dev/null +++ b/doc/api/group_badges.md @@ -0,0 +1,191 @@ +# Group badges API + +## Placeholder tokens + +Badges support placeholders that will be replaced in real time in both the link and image URL. The allowed placeholders are: + +- **%{project_path}**: will be replaced by the project path. +- **%{project_id}**: will be replaced by the project id. +- **%{default_branch}**: will be replaced by the project default branch. +- **%{commit_sha}**: will be replaced by the last project's commit sha. + +Because these enpoints aren't inside a project's context, the information used to replace the placeholders will be +from the first group's project by creation date. If the group hasn't got any project the original URL with the placeholders will be returned. + +## List all badges of a group + +Gets a list of a group's badges. + +``` +GET /groups/:id/badges +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/:id/badges +``` + +Example response: + +```json +[ + { + "id": 1, + "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_image_url": "https://shields.io/my/badge", + "kind": "group" + }, + { + "id": 2, + "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_image_url": "https://shields.io/my/badge", + "kind": "group" + }, +] +``` + +## Get a badge of a group + +Gets a badge of a group. + +``` +GET /groups/:id/badges/:badge_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `badge_id` | integer | yes | The badge ID | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/:id/badges/:badge_id +``` + +Example response: + +```json +{ + "id": 1, + "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_image_url": "https://shields.io/my/badge", + "kind": "group" +} +``` + +## Add a badge to a group + +Adds a badge to a group. + +``` +POST /groups/:id/badges +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `link_url` | string | yes | URL of the badge link | +| `image_url` | string | yes | URL of the badge image | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --data "link_url=https://gitlab.com/gitlab-org/gitlab-ce/commits/master&image_url=https://shields.io/my/badge1&position=0" https://gitlab.example.com/api/v4/groups/:id/badges +``` + +Example response: + +```json +{ + "id": 1, + "link_url": "https://gitlab.com/gitlab-org/gitlab-ce/commits/master", + "image_url": "https://shields.io/my/badge1", + "rendered_link_url": "https://gitlab.com/gitlab-org/gitlab-ce/commits/master", + "rendered_image_url": "https://shields.io/my/badge1", + "kind": "group" +} +``` + +## Edit a badge of a group + +Updates a badge of a group. + +``` +PUT /groups/:id/badges/:badge_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `badge_id` | integer | yes | The badge ID | +| `link_url` | string | no | URL of the badge link | +| `image_url` | string | no | URL of the badge image | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/:id/badges/:badge_id +``` + +Example response: + +```json +{ + "id": 1, + "link_url": "https://gitlab.com/gitlab-org/gitlab-ce/commits/master", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "https://gitlab.com/gitlab-org/gitlab-ce/commits/master", + "rendered_image_url": "https://shields.io/my/badge", + "kind": "group" +} +``` + +## Remove a badge from a group + +Removes a badge from a group. + +``` +DELETE /groups/:id/badges/:badge_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `badge_id` | integer | yes | The badge ID | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/:id/badges/:badge_id +``` + +## Preview a badge from a group + +Returns how the `link_url` and `image_url` final URLs would be after resolving the placeholder interpolation. + +``` +GET /groups/:id/badges/render +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `link_url` | string | yes | URL of the badge link| +| `image_url` | string | yes | URL of the badge image | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/:id/badges/render?link_url=http%3A%2F%2Fexample.com%2Fci_status.svg%3Fproject%3D%25%7Bproject_path%7D%26ref%3D%25%7Bdefault_branch%7D&image_url=https%3A%2F%2Fshields.io%2Fmy%2Fbadge +``` + +Example response: + +```json +{ + "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_image_url": "https://shields.io/my/badge", +} +``` diff --git a/doc/api/group_boards.md b/doc/api/group_boards.md new file mode 100644 index 00000000000..45a8544d6b1 --- /dev/null +++ b/doc/api/group_boards.md @@ -0,0 +1,284 @@ +# Group Issue Boards API + +Every API call to group boards must be authenticated. + +If a user is not a member of a group and the group is private, a `GET` +request will result in `404` status code. + +## Group Board + +Lists Issue Boards in the given group. + +``` +GET /groups/:id/boards +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/5/boards +``` + +Example response: + +```json +[ + { + "id": 1, + "group_id": 5, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +] +``` + +## Single board + +Gets a single board. + +``` +GET /groups/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/5/boards/1 +``` + +Example response: + +```json + { + "id": 1, + "group_id": 5, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +``` + +## List board lists + +Get a list of the board's lists. +Does not include `backlog` and `closed` lists + +``` +GET /groups/:id/boards/:board_id/lists +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/5/boards/1/lists +``` + +Example response: + +```json +[ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } +] +``` + +## Single board list + +Get a single board list. + +``` +GET /groups/:id/boards/:board_id/lists/:list_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | +| `list_id` | integer | yes | The ID of a board's list | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/5/boards/1/lists/1 +``` + +Example response: + +```json +{ + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 +} +``` + +## New board list + +Creates a new Issue Board list. + +``` +POST /groups/:id/boards/:board_id/lists +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | +| `label_id` | integer | yes | The ID of a label | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/5/boards/1/lists?label_id=5 +``` + +Example response: + +```json +{ + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 +} +``` + +## Edit board list + +Updates an existing Issue Board list. This call is used to change list position. + +``` +PUT /groups/:id/boards/:board_id/lists/:list_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | +| `list_id` | integer | yes | The ID of a board's list | +| `position` | integer | yes | The position of the list | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/group/5/boards/1/lists/1?position=2 +``` + +Example response: + +```json +{ + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 +} +``` + +## Delete a board list + +Only for admins and group owners. Soft deletes the board list in question. + +``` +DELETE /groups/:id/boards/:board_id/lists/:list_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | +| `list_id` | integer | yes | The ID of a board's list | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/5/boards/1/lists/1 +``` diff --git a/doc/api/groups.md b/doc/api/groups.md index de730cdd869..1aed8aac64e 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -15,6 +15,7 @@ Parameters: | `order_by` | string | no | Order groups by `name` or `path`. Default is `name` | | `sort` | string | no | Order groups in `asc` or `desc` order. Default is `asc` | | `statistics` | boolean | no | Include group statistics (admins only) | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | | `owned` | boolean | no | Limit to groups owned by the current user | ``` @@ -98,6 +99,7 @@ Parameters: | `order_by` | string | no | Order groups by `name` or `path`. Default is `name` | | `sort` | string | no | Order groups in `asc` or `desc` order. Default is `asc` | | `statistics` | boolean | no | Include group statistics (admins only) | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | | `owned` | boolean | no | Limit to groups owned by the current user | ``` @@ -145,6 +147,7 @@ Parameters: | `simple` | boolean | no | Return only the ID, URL, name, and path of each project | | `owned` | boolean | no | Limit by projects owned by the current user | | `starred` | boolean | no | Limit by projects starred by the current user | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | Example response: @@ -204,6 +207,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | ```bash curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/4 @@ -521,3 +525,7 @@ And to switch pages add: ``` [ce-15142]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15142 + +## Group badges + +Read more in the [Group Badges](group_badges.md) documentation. diff --git a/doc/api/issues.md b/doc/api/issues.md index da89db17cd9..7479c1d2f93 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -46,6 +46,10 @@ GET /issues?my_reaction_emoji=star | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search issues against their `title` and `description` | +| `created_after` | datetime | no | Return issues created on or after the given time | +| `created_before` | datetime | no | Return issues created on or before the given time | +| `updated_after` | datetime | no | Return issues updated on or after the given time | +| `updated_before` | datetime | no | Return issues updated on or before the given time | ```bash curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/issues @@ -96,6 +100,7 @@ Example response: }, "updated_at" : "2016-01-04T15:31:51.081Z", "closed_at" : null, + "closed_by" : null, "id" : 76, "title" : "Consequatur vero maxime deserunt laboriosam est voluptas dolorem.", "created_at" : "2016-01-04T15:31:51.081Z", @@ -118,6 +123,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## List group issues Get a list of a group's issues. @@ -152,6 +159,10 @@ GET /groups/:id/issues?my_reaction_emoji=star | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search group issues against their `title` and `description` | +| `created_after` | datetime | no | Return issues created on or after the given time | +| `created_before` | datetime | no | Return issues created on or before the given time | +| `updated_after` | datetime | no | Return issues updated on or after the given time | +| `updated_before` | datetime | no | Return issues updated on or before the given time | ```bash @@ -208,6 +219,7 @@ Example response: "updated_at" : "2016-01-04T15:31:46.176Z", "created_at" : "2016-01-04T15:31:46.176Z", "closed_at" : null, + "closed_by" : null, "user_notes_count": 1, "due_date": null, "web_url": "http://example.com/example/example/issues/1", @@ -225,6 +237,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## List project issues Get a list of a project's issues. @@ -259,8 +273,10 @@ GET /projects/:id/issues?my_reaction_emoji=star | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search project issues against their `title` and `description` | -| `created_after` | datetime | no | Return issues created after the given time (inclusive) | -| `created_before` | datetime | no | Return issues created before the given time (inclusive) | +| `created_after` | datetime | no | Return issues created on or after the given time | +| `created_before` | datetime | no | Return issues created on or before the given time | +| `updated_after` | datetime | no | Return issues updated on or after the given time | +| `updated_before` | datetime | no | Return issues updated on or before the given time | ```bash curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/4/issues @@ -316,6 +332,14 @@ Example response: "updated_at" : "2016-01-04T15:31:46.176Z", "created_at" : "2016-01-04T15:31:46.176Z", "closed_at" : "2016-01-05T15:31:46.176Z", + "closed_by" : { + "state" : "active", + "web_url" : "https://gitlab.example.com/root", + "avatar_url" : null, + "username" : "root", + "id" : 1, + "name" : "Administrator" + }, "user_notes_count": 1, "due_date": "2016-07-22", "web_url": "http://example.com/example/example/issues/1", @@ -333,6 +357,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Single issue Get a single project issue. @@ -399,6 +425,8 @@ Example response: "title" : "Ut commodi ullam eos dolores perferendis nihil sunt.", "updated_at" : "2016-01-04T15:31:46.176Z", "created_at" : "2016-01-04T15:31:46.176Z", + "closed_at" : null, + "closed_by" : null, "subscribed": false, "user_notes_count": 1, "due_date": null, @@ -422,6 +450,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## New issue Creates a new project issue. @@ -474,6 +504,7 @@ Example response: "description" : null, "updated_at" : "2016-01-07T12:44:33.959Z", "closed_at" : null, + "closed_by" : null, "milestone" : null, "subscribed" : true, "user_notes_count": 0, @@ -498,6 +529,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Edit issue Updates an existing project issue. This call is also used to mark an issue as @@ -546,6 +579,14 @@ Example response: "description" : null, "updated_at" : "2016-01-07T12:55:16.213Z", "closed_at" : "2016-01-08T12:55:16.213Z", + "closed_by" : { + "state" : "active", + "web_url" : "https://gitlab.example.com/root", + "avatar_url" : null, + "username" : "root", + "id" : 1, + "name" : "Administrator" + }, "iid" : 15, "labels" : [ "bug" @@ -577,6 +618,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Delete an issue Only for admins and project owners. Soft deletes the issue in question. @@ -630,6 +673,7 @@ Example response: "created_at": "2016-04-05T21:41:45.652Z", "updated_at": "2016-04-07T12:20:17.596Z", "closed_at": null, + "closed_by": null, "labels": [], "milestone": null, "assignees": [{ @@ -677,6 +721,8 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Subscribe to an issue Subscribes the authenticated user to an issue to receive notifications. @@ -709,6 +755,7 @@ Example response: "created_at": "2016-04-05T21:41:45.652Z", "updated_at": "2016-04-07T12:20:17.596Z", "closed_at": null, + "closed_by": null, "labels": [], "milestone": null, "assignees": [{ @@ -756,6 +803,9 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Unsubscribe from an issue Unsubscribes the authenticated user from the issue to not receive notifications @@ -797,6 +847,8 @@ Example response: "avatar_url": "http://www.gravatar.com/avatar/3e6f06a86cf27fa8b56f3f74f7615987?s=80&d=identicon", "web_url": "https://gitlab.example.com/keyon" }, + "closed_at": null, + "closed_by": null, "author": { "name": "Vivian Hermann", "username": "orville", @@ -917,6 +969,9 @@ Example response: **Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +**Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. + ## Set a time estimate for an issue Sets an estimated time of work for this issue. @@ -1102,6 +1157,8 @@ Example response: "assignee": null, "source_project_id": 1, "target_project_id": 1, + "closed_at": null, + "closed_by": null, "labels": [], "work_in_progress": false, "milestone": null, @@ -1196,3 +1253,4 @@ Example response: [ce-13004]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13004 [ce-14016]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14016 +[ce-17042]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17042 diff --git a/doc/api/jobs.md b/doc/api/jobs.md index e7060e154f4..db4fe2f6880 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -294,9 +294,10 @@ Example of response ## Get job artifacts -> [Introduced][ce-2893] in GitLab 8.5 +> **Notes**: +- [Introduced][ce-2893] in GitLab 8.5. -Get job artifacts of a project +Get job artifacts of a project. ``` GET /projects/:id/jobs/:job_id/artifacts @@ -307,8 +308,10 @@ GET /projects/:id/jobs/:job_id/artifacts | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `job_id` | integer | yes | The ID of a job | +Example requests: + ``` -curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts" +curl --location --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts" ``` Response: @@ -322,7 +325,8 @@ Response: ## Download the artifacts archive -> [Introduced][ce-5347] in GitLab 8.10. +> **Notes**: +- [Introduced][ce-5347] in GitLab 8.10. Download the artifacts archive from the given reference name and job provided the job finished successfully. @@ -339,7 +343,7 @@ Parameters | `ref_name` | string | yes | The ref from a repository (can only be branch or tag name, not HEAD or SHA) | | `job` | string | yes | The name of the job | -Example request: +Example requests: ``` curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test" diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 2957a0a5f48..b9a4f661777 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -41,12 +41,16 @@ Parameters: | `milestone` | string | no | Return merge requests for a specific milestone | | `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request | | `labels` | string | no | Return merge requests matching a comma separated list of labels | -| `created_after` | datetime | no | Return merge requests created after the given time (inclusive) | -| `created_before` | datetime | no | Return merge requests created before the given time (inclusive) | +| `created_after` | datetime | no | Return merge requests created on or after the given time | +| `created_before` | datetime | no | Return merge requests created on or before the given time | +| `updated_after` | datetime | no | Return merge requests updated on or after the given time | +| `updated_before` | datetime | no | Return merge requests updated on or before the given time | | `scope` | string | no | Return merge requests for the given scope: `created-by-me`, `assigned-to-me` or `all`. Defaults to `created-by-me` | | `author_id` | integer | no | Returns merge requests created by the given user `id`. Combine with `scope=all` or `scope=assigned-to-me` | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id` | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji` _([Introduced][ce-14016] in GitLab 10.0)_ | +| `source_branch` | string | no | Return merge requests with the given source branch | +| `target_branch` | string | no | Return merge requests with the given target branch | | `search` | string | no | Search merge requests against their `title` and `description` | ```json @@ -156,12 +160,16 @@ Parameters: | `milestone` | string | no | Return merge requests for a specific milestone | | `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request | | `labels` | string | no | Return merge requests matching a comma separated list of labels | -| `created_after` | datetime | no | Return merge requests created after the given time (inclusive) | -| `created_before` | datetime | no | Return merge requests created before the given time (inclusive) | +| `created_after` | datetime | no | Return merge requests created on or after the given time | +| `created_before` | datetime | no | Return merge requests created on or before the given time | +| `updated_after` | datetime | no | Return merge requests updated on or after the given time | +| `updated_before` | datetime | no | Return merge requests updated on or before the given time | | `scope` | string | no | Return merge requests for the given scope: `created-by-me`, `assigned-to-me` or `all` _([Introduced][ce-13060] in GitLab 9.5)_ | | `author_id` | integer | no | Returns merge requests created by the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji` _([Introduced][ce-14016] in GitLab 10.0)_ | +| `source_branch` | string | no | Return merge requests with the given source branch | +| `target_branch` | string | no | Return merge requests with the given target branch | | `search` | string | no | Search merge requests against their `title` and `description` | ```json @@ -257,20 +265,20 @@ Parameters: "upvotes": 0, "downvotes": 0, "author": { - "id": 1, - "username": "admin", - "email": "admin@example.com", - "name": "Administrator", - "state": "active", - "created_at": "2012-04-29T08:46:00Z" + "state" : "active", + "web_url" : "https://gitlab.example.com/root", + "avatar_url" : null, + "username" : "root", + "id" : 1, + "name" : "Administrator" }, "assignee": { - "id": 1, - "username": "admin", - "email": "admin@example.com", - "name": "Administrator", - "state": "active", - "created_at": "2012-04-29T08:46:00Z" + "state" : "active", + "web_url" : "https://gitlab.example.com/root", + "avatar_url" : null, + "username" : "root", + "id" : 1, + "name" : "Administrator" }, "source_project_id": 2, "target_project_id": 3, @@ -304,6 +312,26 @@ Parameters: "total_time_spent": 0, "human_time_estimate": null, "human_total_time_spent": null + }, + "closed_at": "2018-01-19T14:36:11.086Z", + "latest_build_started_at": null, + "latest_build_finished_at": null, + "first_deployed_to_production_at": null, + "pipeline": { + "id": 8, + "ref": "master", + "sha": "2dc6aa325a317eda67812f05600bdf0fcdc70ab0", + "status": "created" + }, + "merged_by": null, + "merged_at": null, + "closed_by": { + "state" : "active", + "web_url" : "https://gitlab.example.com/root", + "avatar_url" : null, + "username" : "root", + "id" : 1, + "name" : "Administrator" } } ``` @@ -470,6 +498,8 @@ Parameters: ## List MR pipelines +> [Introduced][ce-15454] in GitLab 10.5.0. + Get a list of merge request pipelines. ``` @@ -499,18 +529,19 @@ Creates a new merge request. POST /projects/:id/merge_requests ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `source_branch` | string | yes | The source branch | -| `target_branch` | string | yes | The target branch | -| `title` | string | yes | Title of MR | -| `assignee_id` | integer | no | Assignee user ID | -| `description` | string | no | Description of MR | -| `target_project_id` | integer | no | The target project (numeric id) | -| `labels` | string | no | Labels for MR as a comma-separated list | -| `milestone_id` | integer | no | The ID of a milestone | -| `remove_source_branch` | boolean | no | Flag indicating if a merge request should remove the source branch when merging | +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `source_branch` | string | yes | The source branch | +| `target_branch` | string | yes | The target branch | +| `title` | string | yes | Title of MR | +| `assignee_id` | integer | no | Assignee user ID | +| `description` | string | no | Description of MR | +| `target_project_id` | integer | no | The target project (numeric id) | +| `labels` | string | no | Labels for MR as a comma-separated list | +| `milestone_id` | integer | no | The ID of a milestone | +| `remove_source_branch` | boolean | no | Flag indicating if a merge request should remove the source branch when merging | +| `allow_maintainer_to_push` | boolean | no | Whether or not a maintainer of the target project can push to the source branch | ```json { @@ -518,7 +549,7 @@ POST /projects/:id/merge_requests "iid": 1, "target_branch": "master", "source_branch": "test1", - "project_id": 3, + "project_id": 4, "title": "test1", "state": "opened", "upvotes": 0, @@ -539,7 +570,7 @@ POST /projects/:id/merge_requests "state": "active", "created_at": "2012-04-29T08:46:00Z" }, - "source_project_id": 4, + "source_project_id": 3, "target_project_id": 4, "labels": [ ], "description": "fixed login page css paddings", @@ -566,6 +597,7 @@ POST /projects/:id/merge_requests "force_remove_source_branch": false, "web_url": "http://example.com/example/example/merge_requests/1", "discussion_locked": false, + "allow_maintainer_to_push": false, "time_stats": { "time_estimate": 0, "total_time_spent": 0, @@ -583,19 +615,20 @@ Updates an existing merge request. You can change the target branch, title, or e PUT /projects/:id/merge_requests/:merge_request_iid ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `merge_request_iid` | integer | yes | The ID of a merge request | -| `target_branch` | string | no | The target branch | -| `title` | string | no | Title of MR | -| `assignee_id` | integer | no | The ID of the user to assign the merge request to. Set to `0` or provide an empty value to unassign all assignees. | -| `milestone_id` | integer | no | The ID of a milestone to assign the merge request to. Set to `0` or provide an empty value to unassign a milestone.| -| `labels` | string | no | Comma-separated label names for a merge request. Set to an empty string to unassign all labels. | -| `description` | string | no | Description of MR | -| `state_event` | string | no | New state (close/reopen) | -| `remove_source_branch` | boolean | no | Flag indicating if a merge request should remove the source branch when merging | -| `discussion_locked` | boolean | no | Flag indicating if the merge request's discussion is locked. If the discussion is locked only project members can add, edit or resolve comments. | +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `merge_request_iid` | integer | yes | The ID of a merge request | +| `target_branch` | string | no | The target branch | +| `title` | string | no | Title of MR | +| `assignee_id` | integer | no | The ID of the user to assign the merge request to. Set to `0` or provide an empty value to unassign all assignees. | +| `milestone_id` | integer | no | The ID of a milestone to assign the merge request to. Set to `0` or provide an empty value to unassign a milestone.| +| `labels` | string | no | Comma-separated label names for a merge request. Set to an empty string to unassign all labels. | +| `description` | string | no | Description of MR | +| `state_event` | string | no | New state (close/reopen) | +| `remove_source_branch` | boolean | no | Flag indicating if a merge request should remove the source branch when merging | +| `discussion_locked` | boolean | no | Flag indicating if the merge request's discussion is locked. If the discussion is locked only project members can add, edit or resolve comments. | +| `allow_maintainer_to_push` | boolean | no | Whether or not a maintainer of the target project can push to the source branch | Must include at least one non-required attribute from above. @@ -604,7 +637,7 @@ Must include at least one non-required attribute from above. "id": 1, "iid": 1, "target_branch": "master", - "project_id": 3, + "project_id": 4, "title": "test1", "state": "opened", "upvotes": 0, @@ -625,7 +658,7 @@ Must include at least one non-required attribute from above. "state": "active", "created_at": "2012-04-29T08:46:00Z" }, - "source_project_id": 4, + "source_project_id": 3, "target_project_id": 4, "labels": [ ], "description": "description1", @@ -652,6 +685,7 @@ Must include at least one non-required attribute from above. "force_remove_source_branch": false, "web_url": "http://example.com/example/example/merge_requests/1", "discussion_locked": false, + "allow_maintainer_to_push": false, "time_stats": { "time_estimate": 0, "total_time_spent": 0, @@ -1425,3 +1459,4 @@ Example response: [ce-13060]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13060 [ce-14016]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14016 +[ce-15454]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15454 diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index 25cae5ce1f9..1f0dc700640 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -39,7 +39,7 @@ Example response: "path": "group1", "kind": "group", "full_path": "group1", - "parent_id": "null", + "parent_id": null, "members_count_with_descendants": 2 }, { @@ -48,7 +48,7 @@ Example response: "path": "bar", "kind": "group", "full_path": "foo/bar", - "parent_id": "9", + "parent_id": 9, "members_count_with_descendants": 5 } ] @@ -84,7 +84,7 @@ Example response: "path": "twitter", "kind": "group", "full_path": "twitter", - "parent_id": "null", + "parent_id": null, "members_count_with_descendants": 2 } ] @@ -117,7 +117,7 @@ Example response: "path": "group1", "kind": "group", "full_path": "group1", - "parent_id": "null", + "parent_id": null, "members_count_with_descendants": 2 } ``` @@ -137,7 +137,7 @@ Example response: "path": "group1", "kind": "group", "full_path": "group1", - "parent_id": "null", + "parent_id": null, "members_count_with_descendants": 2 } ``` diff --git a/doc/api/notes.md b/doc/api/notes.md index 1b68bd99ce2..aa38d22845c 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -15,7 +15,7 @@ GET /projects/:id/issues/:issue_iid/notes?sort=asc&order_by=updated_at | Attribute | Type | Required | Description | | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | `issue_iid` | integer | yes | The IID of an issue | `sort` | string | no | Return issue notes sorted in `asc` or `desc` order. Default is `desc` | `order_by` | string | no | Return issue notes ordered by `created_at` or `updated_at` fields. Default is `created_at` @@ -63,6 +63,10 @@ GET /projects/:id/issues/:issue_iid/notes?sort=asc&order_by=updated_at ] ``` +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/notes +``` + ### Get single issue note Returns a single note for a specific project issue @@ -73,14 +77,17 @@ GET /projects/:id/issues/:issue_iid/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `issue_iid` (required) - The IID of a project issue - `note_id` (required) - The ID of an issue note +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/notes/1 +``` + ### Create new issue note -Creates a new note to a single project issue. If you create a note where the body -only contains an Award Emoji, you'll receive this object back. +Creates a new note to a single project issue. ``` POST /projects/:id/issues/:issue_iid/notes @@ -88,11 +95,15 @@ POST /projects/:id/issues/:issue_iid/notes Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `issue_id` (required) - The IID of an issue - `body` (required) - The content of a note - `created_at` (optional) - Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/notes?body=note +``` + ### Modify existing issue note Modify existing note of an issue. @@ -103,11 +114,15 @@ PUT /projects/:id/issues/:issue_iid/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `issue_iid` (required) - The IID of an issue - `note_id` (required) - The ID of a note - `body` (required) - The content of a note +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/issues/11/notes?body=note +``` + ### Delete an issue note Deletes an existing note of an issue. @@ -120,7 +135,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `issue_iid` | integer | yes | The IID of an issue | | `note_id` | integer | yes | The ID of a note | @@ -141,11 +156,15 @@ GET /projects/:id/snippets/:snippet_id/notes?sort=asc&order_by=updated_at | Attribute | Type | Required | Description | | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | `snippet_id` | integer | yes | The ID of a project snippet | `sort` | string | no | Return snippet notes sorted in `asc` or `desc` order. Default is `desc` | `order_by` | string | no | Return snippet notes ordered by `created_at` or `updated_at` fields. Default is `created_at` +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/notes +``` + ### Get single snippet note Returns a single note for a given snippet. @@ -156,7 +175,7 @@ GET /projects/:id/snippets/:snippet_id/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `snippet_id` (required) - The ID of a project snippet - `note_id` (required) - The ID of a snippet note @@ -179,6 +198,10 @@ Parameters: } ``` +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/notes/11 +``` + ### Create new snippet note Creates a new note for a single snippet. Snippet notes are comments users can post to a snippet. @@ -190,10 +213,14 @@ POST /projects/:id/snippets/:snippet_id/notes Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `snippet_id` (required) - The ID of a snippet - `body` (required) - The content of a note +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippet/11/notes?body=note +``` + ### Modify existing snippet note Modify existing note of a snippet. @@ -204,11 +231,15 @@ PUT /projects/:id/snippets/:snippet_id/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `snippet_id` (required) - The ID of a snippet - `note_id` (required) - The ID of a note - `body` (required) - The content of a note +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/snippets/11/notes?body=note +``` + ### Delete a snippet note Deletes an existing note of a snippet. @@ -221,7 +252,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `snippet_id` | integer | yes | The ID of a snippet | | `note_id` | integer | yes | The ID of a note | @@ -242,11 +273,15 @@ GET /projects/:id/merge_requests/:merge_request_iid/notes?sort=asc&order_by=upda | Attribute | Type | Required | Description | | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | `merge_request_iid` | integer | yes | The IID of a project merge request | `sort` | string | no | Return merge request notes sorted in `asc` or `desc` order. Default is `desc` | `order_by` | string | no | Return merge request notes ordered by `created_at` or `updated_at` fields. Default is `created_at` +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/notes +``` + ### Get single merge request note Returns a single note for a given merge request. @@ -257,7 +292,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `merge_request_iid` (required) - The IID of a project merge request - `note_id` (required) - The ID of a merge request note @@ -283,6 +318,10 @@ Parameters: } ``` +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/notes/1 +``` + ### Create new merge request note Creates a new note for a single merge request. @@ -295,7 +334,7 @@ POST /projects/:id/merge_requests/:merge_request_iid/notes Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `merge_request_iid` (required) - The IID of a merge request - `body` (required) - The content of a note @@ -309,11 +348,15 @@ PUT /projects/:id/merge_requests/:merge_request_iid/notes/:note_id Parameters: -- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) - `merge_request_iid` (required) - The IID of a merge request - `note_id` (required) - The ID of a note - `body` (required) - The content of a note +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/notes?body=note +``` + ### Delete a merge request note Deletes an existing note of a merge request. @@ -326,7 +369,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of a merge request | | `note_id` | integer | yes | The ID of a note | diff --git a/doc/api/notification_settings.md b/doc/api/notification_settings.md index 3a2c398e355..f05ae647577 100644 --- a/doc/api/notification_settings.md +++ b/doc/api/notification_settings.md @@ -24,6 +24,7 @@ reopen_issue close_issue reassign_issue new_merge_request +push_to_merge_request reopen_merge_request close_merge_request reassign_merge_request @@ -75,6 +76,7 @@ curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab | `close_issue` | boolean | no | Enable/disable this notification | | `reassign_issue` | boolean | no | Enable/disable this notification | | `new_merge_request` | boolean | no | Enable/disable this notification | +| `push_to_merge_request` | boolean | no | Enable/disable this notification | | `reopen_merge_request` | boolean | no | Enable/disable this notification | | `close_merge_request` | boolean | no | Enable/disable this notification | | `reassign_merge_request` | boolean | no | Enable/disable this notification | @@ -141,6 +143,7 @@ curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab | `close_issue` | boolean | no | Enable/disable this notification | | `reassign_issue` | boolean | no | Enable/disable this notification | | `new_merge_request` | boolean | no | Enable/disable this notification | +| `push_to_merge_request` | boolean | no | Enable/disable this notification | | `reopen_merge_request` | boolean | no | Enable/disable this notification | | `close_merge_request` | boolean | no | Enable/disable this notification | | `reassign_merge_request` | boolean | no | Enable/disable this notification | @@ -164,6 +167,7 @@ Example responses: "close_issue": false, "reassign_issue": false, "new_merge_request": false, + "push_to_merge_request": false, "reopen_merge_request": false, "close_merge_request": false, "reassign_merge_request": false, diff --git a/doc/api/project_badges.md b/doc/api/project_badges.md new file mode 100644 index 00000000000..3f6e348b5b4 --- /dev/null +++ b/doc/api/project_badges.md @@ -0,0 +1,188 @@ +# Project badges API + +## Placeholder tokens + +Badges support placeholders that will be replaced in real time in both the link and image URL. The allowed placeholders are: + +- **%{project_path}**: will be replaced by the project path. +- **%{project_id}**: will be replaced by the project id. +- **%{default_branch}**: will be replaced by the project default branch. +- **%{commit_sha}**: will be replaced by the last project's commit sha. + +## List all badges of a project + +Gets a list of a project's badges and its group badges. + +``` +GET /projects/:id/badges +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/:id/badges +``` + +Example response: + +```json +[ + { + "id": 1, + "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_image_url": "https://shields.io/my/badge", + "kind": "project" + }, + { + "id": 2, + "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_image_url": "https://shields.io/my/badge", + "kind": "group" + }, +] +``` + +## Get a badge of a project + +Gets a badge of a project. + +``` +GET /projects/:id/badges/:badge_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `badge_id` | integer | yes | The badge ID | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/:id/badges/:badge_id +``` + +Example response: + +```json +{ + "id": 1, + "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_image_url": "https://shields.io/my/badge", + "kind": "project" +} +``` + +## Add a badge to a project + +Adds a badge to a project. + +``` +POST /projects/:id/badges +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project ](README.md#namespaced-path-encoding) owned by the authenticated user | +| `link_url` | string | yes | URL of the badge link | +| `image_url` | string | yes | URL of the badge image | + +```bash +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --data "link_url=https://gitlab.com/gitlab-org/gitlab-ce/commits/master&image_url=https://shields.io/my/badge1&position=0" https://gitlab.example.com/api/v4/projects/:id/badges +``` + +Example response: + +```json +{ + "id": 1, + "link_url": "https://gitlab.com/gitlab-org/gitlab-ce/commits/master", + "image_url": "https://shields.io/my/badge1", + "rendered_link_url": "https://gitlab.com/gitlab-org/gitlab-ce/commits/master", + "rendered_image_url": "https://shields.io/my/badge1", + "kind": "project" +} +``` + +## Edit a badge of a project + +Updates a badge of a project. + +``` +PUT /projects/:id/badges/:badge_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `badge_id` | integer | yes | The badge ID | +| `link_url` | string | no | URL of the badge link | +| `image_url` | string | no | URL of the badge image | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/:id/badges/:badge_id +``` + +Example response: + +```json +{ + "id": 1, + "link_url": "https://gitlab.com/gitlab-org/gitlab-ce/commits/master", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "https://gitlab.com/gitlab-org/gitlab-ce/commits/master", + "rendered_image_url": "https://shields.io/my/badge", + "kind": "project" +} +``` + +## Remove a badge from a project + +Removes a badge from a project. Only project's badges will be removed by using this endpoint. + +``` +DELETE /projects/:id/badges/:badge_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `badge_id` | integer | yes | The badge ID | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/:id/badges/:badge_id +``` + +## Preview a badge from a project + +Returns how the `link_url` and `image_url` final URLs would be after resolving the placeholder interpolation. + +``` +GET /projects/:id/badges/render +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `link_url` | string | yes | URL of the badge link| +| `image_url` | string | yes | URL of the badge image | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/:id/badges/render?link_url=http%3A%2F%2Fexample.com%2Fci_status.svg%3Fproject%3D%25%7Bproject_path%7D%26ref%3D%25%7Bdefault_branch%7D&image_url=https%3A%2F%2Fshields.io%2Fmy%2Fbadge +``` + +Example response: + +```json +{ + "link_url": "http://example.com/ci_status.svg?project=%{project_path}&ref=%{default_branch}", + "image_url": "https://shields.io/my/badge", + "rendered_link_url": "http://example.com/ci_status.svg?project=example-org/example-project&ref=master", + "rendered_image_url": "https://shields.io/my/badge", +} +``` diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md new file mode 100644 index 00000000000..5467187788a --- /dev/null +++ b/doc/api/project_import_export.md @@ -0,0 +1,170 @@ +# Project import/export API + +[Introduced][ce-41899] in GitLab 10.6 + +[See also the project import/export documentation](../user/project/settings/import_export.md) + +## Schedule an export + +Start a new export. + +The endpoint also accepts an `upload` param. This param is a hash that contains +all the necessary information to upload the exported project to a web server or +to any S3-compatible platform. At the moment we only support binary +data file uploads to the final server. + +If the `upload` params is present, `upload[url]` param is required. + (**Note:** This feature was introduced in GitLab 10.7) + +```http +POST /projects/:id/export +``` + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ---------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `description` | string | no | Overrides the project description | +| `upload` | hash | no | Hash that contains the information to upload the exported project to a web server | +| `upload[url]` | string | yes | The URL to upload the project | +| `upload[http_method]` | string | no | The HTTP method to upload the exported project. Only `PUT` and `POST` methods allowed. Default is `PUT` | + +```console +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/1/export --data "description=FooBar&upload[http_method]=PUT&upload[url]=https://example-bucket.s3.eu-west-3.amazonaws.com/backup?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIMBJHN2O62W8IELQ%2F20180312%2Feu-west-3%2Fs3%2Faws4_request&X-Amz-Date=20180312T110328Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=8413facb20ff33a49a147a0b4abcff4c8487cc33ee1f7e450c46e8f695569dbd" +``` + +```json +{ + "message": "202 Accepted" +} +``` + +## Export status + +Get the status of export. + +```http +GET /projects/:id/export +``` + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ---------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | + +```console +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/1/export +``` + +Status can be one of `none`, `started`, `after_export_action` or `finished`. The +`after_export_action` state represents that the export process has been completed successfully and +the platform is performing some actions on the resulted file. For example, sending +an email notifying the user to download the file, uploading the exported file +to a web server, etc. + +`_links` are only present when export has finished. + +```json +{ + "id": 1, + "description": "Itaque perspiciatis minima aspernatur corporis consequatur.", + "name": "Gitlab Test", + "name_with_namespace": "Gitlab Org / Gitlab Test", + "path": "gitlab-test", + "path_with_namespace": "gitlab-org/gitlab-test", + "created_at": "2017-08-29T04:36:44.383Z", + "export_status": "finished", + "_links": { + "api_url": "https://gitlab.example.com/api/v4/projects/1/export/download", + "web_url": "https://gitlab.example.com/gitlab-org/gitlab-test/download_export", + } +} +``` + +## Export download + +Download the finished export. + +```http +GET /projects/:id/export/download +``` + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ---------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | + +```console +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --remote-header-name --remote-name https://gitlab.example.com/api/v4/projects/5/export/download +``` + +```console +ls *export.tar.gz +2017-12-05_22-11-148_namespace_project_export.tar.gz +``` + +## Import a file + +```http +POST /projects/import +``` + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ---------------------------------------- | +| `namespace` | integer/string | no | The ID or path of the namespace that the project will be imported to. Defaults to the current user's namespace | +| `file` | string | yes | The file to be uploaded | +| `path` | string | yes | Name and path for new project | + +To upload a file from your filesystem, use the `--form` argument. This causes +cURL to post data using the header `Content-Type: multipart/form-data`. +The `file=` parameter must point to a file on your filesystem and be preceded +by `@`. For example: + +```console +curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --form "path=api-project" --form "file=@/path/to/file" https://gitlab.example.com/api/v4/projects/import +``` + +```json +{ + "id": 1, + "description": null, + "name": "api-project", + "name_with_namespace": "Administrator / api-project", + "path": "api-project", + "path_with_namespace": "root/api-project", + "created_at": "2018-02-13T09:05:58.023Z", + "import_status": "scheduled" +} +``` + +## Import status + +Get the status of an import. + +```http +GET /projects/:id/import +``` + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ---------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | + +```console +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/1/import +``` + +Status can be one of `none`, `scheduled`, `failed`, `started`, or `finished`. + +If the status is `failed`, it will include the import error message under `import_error`. + +```json +{ + "id": 1, + "description": "Itaque perspiciatis minima aspernatur corporis consequatur.", + "name": "Gitlab Test", + "name_with_namespace": "Gitlab Org / Gitlab Test", + "path": "gitlab-test", + "path_with_namespace": "gitlab-org/gitlab-test", + "created_at": "2017-08-29T04:36:44.383Z", + "import_status": "started" +} +``` + +[ce-41899]: https://gitlab.com/gitlab-org/gitlab-ce/issues/41899 diff --git a/doc/api/projects.md b/doc/api/projects.md index 46f5de5aa0e..271ee91dc72 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -37,6 +37,7 @@ GET /projects | `membership` | boolean | no | Limit by projects that the current user is a member of | | `starred` | boolean | no | Limit by projects starred by the current user | | `statistics` | boolean | no | Include project statistics | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | | `with_issues_enabled` | boolean | no | Limit by enabled issues feature | | `with_merge_requests_enabled` | boolean | no | Limit by enabled merge requests feature | @@ -220,6 +221,7 @@ GET /users/:user_id/projects | `membership` | boolean | no | Limit by projects that the current user is a member of | | `starred` | boolean | no | Limit by projects starred by the current user | | `statistics` | boolean | no | Include project statistics | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | | `with_issues_enabled` | boolean | no | Limit by enabled issues feature | | `with_merge_requests_enabled` | boolean | no | Limit by enabled merge requests feature | @@ -388,6 +390,7 @@ GET /projects/:id | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `statistics` | boolean | no | Include project statistics | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | ```json { @@ -664,6 +667,7 @@ GET /projects/:id/forks | `membership` | boolean | no | Limit by projects that the current user is a member of | | `starred` | boolean | no | Limit by projects starred by the current user | | `statistics` | boolean | no | Include project statistics | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only) | | `with_issues_enabled` | boolean | no | Limit by enabled issues feature | | `with_merge_requests_enabled` | boolean | no | Limit by enabled merge requests feature | @@ -1190,6 +1194,7 @@ GET /projects/:id/hooks/:hook_id "project_id": 3, "push_events": true, "issues_events": true, + "confidential_issues_events": true, "merge_requests_events": true, "tag_push_events": true, "note_events": true, @@ -1215,12 +1220,13 @@ POST /projects/:id/hooks | `url` | string | yes | The hook URL | | `push_events` | boolean | no | Trigger hook on push events | | `issues_events` | boolean | no | Trigger hook on issues events | +| `confidential_issues_events` | boolean | no | Trigger hook on confidential issues events | | `merge_requests_events` | boolean | no | Trigger hook on merge requests events | | `tag_push_events` | boolean | no | Trigger hook on tag push events | | `note_events` | boolean | no | Trigger hook on note events | | `job_events` | boolean | no | Trigger hook on job events | | `pipeline_events` | boolean | no | Trigger hook on pipeline events | -| `wiki_events` | boolean | no | Trigger hook on wiki events | +| `wiki_page_events` | boolean | no | Trigger hook on wiki events | | `enable_ssl_verification` | boolean | no | Do SSL verification when triggering the hook | | `token` | string | no | Secret token to validate received payloads; this will not be returned in the response | @@ -1239,6 +1245,7 @@ PUT /projects/:id/hooks/:hook_id | `url` | string | yes | The hook URL | | `push_events` | boolean | no | Trigger hook on push events | | `issues_events` | boolean | no | Trigger hook on issues events | +| `confidential_issues_events` | boolean | no | Trigger hook on confidential issues events | | `merge_requests_events` | boolean | no | Trigger hook on merge requests events | | `tag_push_events` | boolean | no | Trigger hook on tag push events | | `note_events` | boolean | no | Trigger hook on note events | @@ -1326,6 +1333,14 @@ POST /projects/:id/housekeeping Read more in the [Branches](branches.md) documentation. +## Project Import/Export + +Read more in the [Project import/export](project_import_export.md) documentation. + ## Project members Read more in the [Project members](members.md) documentation. + +## Project badges + +Read more in the [Project Badges](project_badges.md) documentation. diff --git a/doc/api/runners.md b/doc/api/runners.md index 7495c6cdedb..f384ac57bfe 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -153,7 +153,8 @@ Example response: "mysql" ], "version": null, - "access_level": "ref_protected" + "access_level": "ref_protected", + "maximum_timeout": 3600 } ``` @@ -174,6 +175,7 @@ PUT /runners/:id | `run_untagged` | boolean | no | Flag indicating the runner can execute untagged jobs | | `locked` | boolean | no | Flag indicating the runner is locked | | `access_level` | string | no | The access_level of the runner; `not_protected` or `ref_protected` | +| `maximum_timeout` | integer | no | Maximum timeout set when this Runner will handle the job | ``` curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/runners/6" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2" @@ -211,7 +213,8 @@ Example response: "tag2" ], "version": null, - "access_level": "ref_protected" + "access_level": "ref_protected", + "maximum_timeout": null } ``` diff --git a/doc/api/search.md b/doc/api/search.md new file mode 100644 index 00000000000..107ddaffa6a --- /dev/null +++ b/doc/api/search.md @@ -0,0 +1,800 @@ +# Search API + +[Introduced][ce-41763] in GitLab 10.5 + +Every API call to search must be authenticated. + +## Global Search API + +Search globally across the GitLab instance. + +``` +GET /search +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `scope` | string | yes | The scope to search in | +| `search` | string | yes | The search query | + +Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones, snippet_titles, snippet_blobs. + +The response depends on the requested scope. + +### Scope: projects + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/search?scope=projects&search=flight +``` + +Example response: + +```json +[ + { + "id": 6, + "description": "Nobis sed ipsam vero quod cupiditate veritatis hic.", + "name": "Flight", + "name_with_namespace": "Twitter / Flight", + "path": "flight", + "path_with_namespace": "twitter/flight", + "created_at": "2017-09-05T07:58:01.621Z", + "default_branch": "master", + "tag_list":[], + "ssh_url_to_repo": "ssh://jarka@localhost:2222/twitter/flight.git", + "http_url_to_repo": "http://localhost:3000/twitter/flight.git", + "web_url": "http://localhost:3000/twitter/flight", + "avatar_url": null, + "star_count": 0, + "forks_count": 0, + "last_activity_at": "2018-01-31T09:56:30.902Z" + } +] +``` + +### Scope: issues + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/search?scope=issues&search=file +``` + +Example response: + +```json +[ + { + "id": 83, + "iid": 1, + "project_id": 12, + "title": "Add file", + "description": "Add first file", + "state": "opened", + "created_at": "2018-01-24T06:02:15.514Z", + "updated_at": "2018-02-06T12:36:23.263Z", + "closed_at": null, + "labels":[], + "milestone": null, + "assignees": [{ + "id": 20, + "name": "Ceola Deckow", + "username": "sammy.collier", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c23d85a4f50e0ea76ab739156c639231?s=80&d=identicon", + "web_url": "http://localhost:3000/sammy.collier" + }], + "author": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "assignee": { + "id": 20, + "name": "Ceola Deckow", + "username": "sammy.collier", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c23d85a4f50e0ea76ab739156c639231?s=80&d=identicon", + "web_url": "http://localhost:3000/sammy.collier" + }, + "user_notes_count": 0, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "discussion_locked": null, + "web_url": "http://localhost:3000/h5bp/7bp/subgroup-prj/issues/1", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +] +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +### Scope: merge_requests + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/search?scope=merge_requests&search=file +``` + +Example response: + +```json +[ + { + "id": 56, + "iid": 8, + "project_id": 6, + "title": "Add first file", + "description": "This is a test MR to add file", + "state": "opened", + "created_at": "2018-01-22T14:21:50.830Z", + "updated_at": "2018-02-06T12:40:33.295Z", + "target_branch": "master", + "source_branch": "jaja-test", + "upvotes": 0, + "downvotes": 0, + "author": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "assignee": { + "id": 5, + "name": "Jacquelyn Kutch", + "username": "abigail", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/3138c66095ee4bd11a508c2f7f7772da?s=80&d=identicon", + "web_url": "http://localhost:3000/abigail" + }, + "source_project_id": 6, + "target_project_id": 6, + "labels": [ + "ruby", + "tests" + ], + "work_in_progress": false, + "milestone": { + "id": 13, + "iid": 3, + "project_id": 6, + "title": "v2.0", + "description": "Qui aut qui eos dolor beatae itaque tempore molestiae.", + "state": "active", + "created_at": "2017-09-05T07:58:29.099Z", + "updated_at": "2017-09-05T07:58:29.099Z", + "due_date": null, + "start_date": null + }, + "merge_when_pipeline_succeeds": false, + "merge_status": "can_be_merged", + "sha": "78765a2d5e0a43585945c58e61ba2f822e4d090b", + "merge_commit_sha": null, + "user_notes_count": 0, + "discussion_locked": null, + "should_remove_source_branch": null, + "force_remove_source_branch": true, + "web_url": "http://localhost:3000/twitter/flight/merge_requests/8", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +] +``` + +### Scope: milestones + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/search?scope=milestones&search=release +``` + +Example response: + +```json +[ + { + "id": 44, + "iid": 1, + "project_id": 12, + "title": "next release", + "description": "Next release milestone", + "state": "active", + "created_at": "2018-02-06T12:43:39.271Z", + "updated_at": "2018-02-06T12:44:01.298Z", + "due_date": "2018-04-18", + "start_date": "2018-02-04" + } +] +``` + +### Scope: snippet_titles + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/search?scope=snippet_titles&search=sample +``` + +Example response: + +```json +[ + { + "id": 50, + "title": "Sample file", + "file_name": "file.rb", + "description": "Simple ruby file", + "author": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "updated_at": "2018-02-06T12:49:29.104Z", + "created_at": "2017-11-28T08:20:18.071Z", + "project_id": 9, + "web_url": "http://localhost:3000/root/jira-test/snippets/50" + } +] +``` + +### Scope: snippet_blobs + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/search?scope=snippet_blos&search=test +``` + +Example response: + +```json +[ + { + "id": 50, + "title": "Sample file", + "file_name": "file.rb", + "description": "Simple ruby file", + "author": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "updated_at": "2018-02-06T12:49:29.104Z", + "created_at": "2017-11-28T08:20:18.071Z", + "project_id": 9, + "web_url": "http://localhost:3000/root/jira-test/snippets/50" + } +] +``` + + +## Group Search API + +Search within the specified group. + +If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code. + +``` +GET /groups/:id/search +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `scope` | string | yes | The scope to search in | +| `search` | string | yes | The search query | + +Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones. + +The response depends on the requested scope. + +### Scope: projects + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/search?scope=projects&search=flight +``` + +Example response: + +```json +[ + { + "id": 6, + "description": "Nobis sed ipsam vero quod cupiditate veritatis hic.", + "name": "Flight", + "name_with_namespace": "Twitter / Flight", + "path": "flight", + "path_with_namespace": "twitter/flight", + "created_at": "2017-09-05T07:58:01.621Z", + "default_branch": "master", + "tag_list":[], + "ssh_url_to_repo": "ssh://jarka@localhost:2222/twitter/flight.git", + "http_url_to_repo": "http://localhost:3000/twitter/flight.git", + "web_url": "http://localhost:3000/twitter/flight", + "avatar_url": null, + "star_count": 0, + "forks_count": 0, + "last_activity_at": "2018-01-31T09:56:30.902Z" + } +] +``` + +### Scope: issues + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/search?scope=issues&search=file +``` + +Example response: + +```json +[ + { + "id": 83, + "iid": 1, + "project_id": 12, + "title": "Add file", + "description": "Add first file", + "state": "opened", + "created_at": "2018-01-24T06:02:15.514Z", + "updated_at": "2018-02-06T12:36:23.263Z", + "closed_at": null, + "labels":[], + "milestone": null, + "assignees": [{ + "id": 20, + "name": "Ceola Deckow", + "username": "sammy.collier", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c23d85a4f50e0ea76ab739156c639231?s=80&d=identicon", + "web_url": "http://localhost:3000/sammy.collier" + }], + "author": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "assignee": { + "id": 20, + "name": "Ceola Deckow", + "username": "sammy.collier", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c23d85a4f50e0ea76ab739156c639231?s=80&d=identicon", + "web_url": "http://localhost:3000/sammy.collier" + }, + "user_notes_count": 0, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "discussion_locked": null, + "web_url": "http://localhost:3000/h5bp/7bp/subgroup-prj/issues/1", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +] +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +### Scope: merge_requests + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/search?scope=merge_requests&search=file +``` + +Example response: + +```json +[ + { + "id": 56, + "iid": 8, + "project_id": 6, + "title": "Add first file", + "description": "This is a test MR to add file", + "state": "opened", + "created_at": "2018-01-22T14:21:50.830Z", + "updated_at": "2018-02-06T12:40:33.295Z", + "target_branch": "master", + "source_branch": "jaja-test", + "upvotes": 0, + "downvotes": 0, + "author": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "assignee": { + "id": 5, + "name": "Jacquelyn Kutch", + "username": "abigail", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/3138c66095ee4bd11a508c2f7f7772da?s=80&d=identicon", + "web_url": "http://localhost:3000/abigail" + }, + "source_project_id": 6, + "target_project_id": 6, + "labels": [ + "ruby", + "tests" + ], + "work_in_progress": false, + "milestone": { + "id": 13, + "iid": 3, + "project_id": 6, + "title": "v2.0", + "description": "Qui aut qui eos dolor beatae itaque tempore molestiae.", + "state": "active", + "created_at": "2017-09-05T07:58:29.099Z", + "updated_at": "2017-09-05T07:58:29.099Z", + "due_date": null, + "start_date": null + }, + "merge_when_pipeline_succeeds": false, + "merge_status": "can_be_merged", + "sha": "78765a2d5e0a43585945c58e61ba2f822e4d090b", + "merge_commit_sha": null, + "user_notes_count": 0, + "discussion_locked": null, + "should_remove_source_branch": null, + "force_remove_source_branch": true, + "web_url": "http://localhost:3000/twitter/flight/merge_requests/8", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +] +``` + +### Scope: milestones + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/3/search?scope=milestones&search=release +``` + +Example response: + +```json +[ + { + "id": 44, + "iid": 1, + "project_id": 12, + "title": "next release", + "description": "Next release milestone", + "state": "active", + "created_at": "2018-02-06T12:43:39.271Z", + "updated_at": "2018-02-06T12:44:01.298Z", + "due_date": "2018-04-18", + "start_date": "2018-02-04" + } +] +``` + +## Project Search API + +Search within the specified project. + +If a user is not a member of a project and the project is private, a `GET` request on that project will result to a `404` status code. + +``` +GET /projects/:id/search +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `scope` | string | yes | The scope to search in | +| `search` | string | yes | The search query | + +Search the expression within the specified scope. Currently these scopes are supported: issues, merge_requests, milestones, notes, wiki_blobs, commits, blobs. + +The response depends on the requested scope. + + +### Scope: issues + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/12/search?scope=issues&search=file +``` + +Example response: + +```json +[ + { + "id": 83, + "iid": 1, + "project_id": 12, + "title": "Add file", + "description": "Add first file", + "state": "opened", + "created_at": "2018-01-24T06:02:15.514Z", + "updated_at": "2018-02-06T12:36:23.263Z", + "closed_at": null, + "labels":[], + "milestone": null, + "assignees": [{ + "id": 20, + "name": "Ceola Deckow", + "username": "sammy.collier", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c23d85a4f50e0ea76ab739156c639231?s=80&d=identicon", + "web_url": "http://localhost:3000/sammy.collier" + }], + "author": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "assignee": { + "id": 20, + "name": "Ceola Deckow", + "username": "sammy.collier", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c23d85a4f50e0ea76ab739156c639231?s=80&d=identicon", + "web_url": "http://localhost:3000/sammy.collier" + }, + "user_notes_count": 0, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "discussion_locked": null, + "web_url": "http://localhost:3000/h5bp/7bp/subgroup-prj/issues/1", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +] +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +### Scope: merge_requests + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=merge_requests&search=file +``` + +Example response: + +```json +[ + { + "id": 56, + "iid": 8, + "project_id": 6, + "title": "Add first file", + "description": "This is a test MR to add file", + "state": "opened", + "created_at": "2018-01-22T14:21:50.830Z", + "updated_at": "2018-02-06T12:40:33.295Z", + "target_branch": "master", + "source_branch": "jaja-test", + "upvotes": 0, + "downvotes": 0, + "author": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "assignee": { + "id": 5, + "name": "Jacquelyn Kutch", + "username": "abigail", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/3138c66095ee4bd11a508c2f7f7772da?s=80&d=identicon", + "web_url": "http://localhost:3000/abigail" + }, + "source_project_id": 6, + "target_project_id": 6, + "labels": [ + "ruby", + "tests" + ], + "work_in_progress": false, + "milestone": { + "id": 13, + "iid": 3, + "project_id": 6, + "title": "v2.0", + "description": "Qui aut qui eos dolor beatae itaque tempore molestiae.", + "state": "active", + "created_at": "2017-09-05T07:58:29.099Z", + "updated_at": "2017-09-05T07:58:29.099Z", + "due_date": null, + "start_date": null + }, + "merge_when_pipeline_succeeds": false, + "merge_status": "can_be_merged", + "sha": "78765a2d5e0a43585945c58e61ba2f822e4d090b", + "merge_commit_sha": null, + "user_notes_count": 0, + "discussion_locked": null, + "should_remove_source_branch": null, + "force_remove_source_branch": true, + "web_url": "http://localhost:3000/twitter/flight/merge_requests/8", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +] +``` + +### Scope: milestones + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/12/search?scope=milestones&search=release +``` + +Example response: + +```json +[ + { + "id": 44, + "iid": 1, + "project_id": 12, + "title": "next release", + "description": "Next release milestone", + "state": "active", + "created_at": "2018-02-06T12:43:39.271Z", + "updated_at": "2018-02-06T12:44:01.298Z", + "due_date": "2018-04-18", + "start_date": "2018-02-04" + } +] +``` + +### Scope: notes + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=notes&search=maxime +``` + +Example response: + +```json +[ + { + "id": 191, + "body": "Harum maxime consequuntur et et deleniti assumenda facilis.", + "attachment": null, + "author": { + "id": 23, + "name": "User 1", + "username": "user1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/111d68d06e2d317b5a59c2c6c5bad808?s=80&d=identicon", + "web_url": "http://localhost:3000/user1" + }, + "created_at": "2017-09-05T08:01:32.068Z", + "updated_at": "2017-09-05T08:01:32.068Z", + "system": false, + "noteable_id": 22, + "noteable_type": "Issue", + "noteable_iid": 2 + } +] +``` + +### Scope: wiki_blobs + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=wiki_blobs&search=bye +``` + +Example response: + +```json + +[ + { + "basename": "home", + "data": "hello\n\nand bye\n\nend", + "filename": "home.md", + "id": null, + "ref": "master", + "startline": 5, + "project_id": 6 + } +] +``` + +### Scope: commits + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=commits&search=bye +``` + +Example response: + +```json + +[ + { + "id": "4109c2d872d5fdb1ed057400d103766aaea97f98", + "short_id": "4109c2d8", + "title": "goodbye $.browser", + "created_at": "2013-02-18T22:02:54.000Z", + "parent_ids": [ + "59d05353ab575bcc2aa958fe1782e93297de64c9" + ], + "message": "goodbye $.browser\n", + "author_name": "angus croll", + "author_email": "anguscroll@gmail.com", + "authored_date": "2013-02-18T22:02:54.000Z", + "committer_name": "angus croll", + "committer_email": "anguscroll@gmail.com", + "committed_date": "2013-02-18T22:02:54.000Z", + "project_id": 6 + } +] +``` + +### Scope: blobs + +```bash +curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=blobs&search=installation +``` + +Example response: + +```json + +[ + { + "basename": "README", + "data": "```\n\n## Installation\n\nQuick start using the [pre-built", + "filename": "README.md", + "id": null, + "ref": "master", + "startline": 46, + "project_id": 6 + } +] +``` + +[ce-41763]: https://gitlab.com/gitlab-org/gitlab-ce/issues/41763 diff --git a/doc/api/services.md b/doc/api/services.md index 2928ab6cc75..92f12acbc73 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -619,6 +619,7 @@ Example response: "active": true, "push_events": true, "issues_events": true, + "confidential_issues_events": true, "merge_requests_events": true, "tag_push_events": true, "note_events": true, diff --git a/doc/api/users.md b/doc/api/users.md index 1da6fcf297d..a4447e32908 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -51,6 +51,11 @@ GET /users?blocked=true GET /users ``` +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `username`, `created_at`, or `updated_at` fields. Default is `id` | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc` | + ```json [ { @@ -160,6 +165,12 @@ You can filter by [custom attributes](custom_attributes.md) with: GET /users?custom_attributes[key]=value&custom_attributes[other_key]=other_value ``` +You can include the users' [custom attributes](custom_attributes.md) in the response with: + +``` +GET /users?with_custom_attributes=true +``` + ## Single user Get a single user. @@ -240,6 +251,12 @@ Parameters: } ``` +You can include the user's [custom attributes](custom_attributes.md) in the response with: + +``` +GET /users/:id?with_custom_attributes=true +``` + ## User creation Creates a new user. Note only administrators can create new users. Either `password` or `reset_password` should be specified (`reset_password` takes priority). diff --git a/doc/ci/README.md b/doc/ci/README.md index eabeb4510db..6aa0e5885db 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -65,11 +65,14 @@ learn how to leverage its potential even more. environments and use them for different purposes like testing, building and deploying - [Job artifacts](../user/project/pipelines/job_artifacts.md) -- [Git submodules](git_submodules.md): How to run your CI jobs when Git +- [Caching dependencies](caching/index.md) +- [Git submodules](git_submodules.md) - How to run your CI jobs when Git submodules are involved - [Use SSH keys in your build environment](ssh_keys/README.md) - [Trigger pipelines through the GitLab API](triggers/README.md) - [Trigger pipelines on a schedule](../user/project/pipelines/schedules.md) +- [Kubernetes clusters](../user/project/clusters/index.md) - Integrate one or + more Kubernetes clusters to your project ## GitLab CI/CD for Docker diff --git a/doc/ci/caching/img/clear_runners_cache.png b/doc/ci/caching/img/clear_runners_cache.png Binary files differnew file mode 100644 index 00000000000..e5db4a47b3e --- /dev/null +++ b/doc/ci/caching/img/clear_runners_cache.png diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md new file mode 100644 index 00000000000..c159198d16b --- /dev/null +++ b/doc/ci/caching/index.md @@ -0,0 +1,518 @@ +# Cache dependencies in GitLab CI/CD + +GitLab CI/CD provides a caching mechanism that can be used to save time +when your jobs are running. + +Caching is about speeding the time a job is executed by reusing the same +content of a previous job. It can be particularly useful when your are +developing software that depends on other libraries which are fetched via the +internet during build time. + +If caching is enabled, it's shared between pipelines and jobs by default, +starting from GitLab 9.0. + +Make sure you read the [`cache` reference](../yaml/README.md#cache) to learn +how it is defined in `.gitlab-ci.yml`. + +## Good caching practices + +We have the cache from the perspective of the developers (who consume a cache +within the job) and the cache from the perspective of the Runner. Depending on +which type of Runner you are using, cache can act differently. + +From the perspective of the developer, to ensure maximum availability of the +cache, when declaring `cache` in your jobs, use one or a mix of the following: + +- [Tag your Runners](../runners/README.md#using-tags) and use the tag on jobs + that share their cache. +- [Use sticky Runners](../runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) + that will be only available to a particular project. +- [Use a `key`](../yaml/README.md#cache-key) that fits your workflow (e.g., + different caches on each branch). For that, you can take advantage of the + [CI/CD predefined variables](../variables/README.md#predefined-variables-environment-variables). + +TIP: **Tip:** +Using the same Runner for your pipeline, is the most simple and efficient way to +cache files in one stage or pipeline, and pass this cache to subsequent stages +or pipelines in a guaranteed manner. + +From the perspective of the Runner, in order for cache to work effectively, one +of the following must be true: + +- Use a single Runner for all your jobs +- Use multiple Runners (in autoscale mode or not) that use + [distributed caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching), + where the cache is stored in S3 buckets (like shared Runners on GitLab.com) +- Use multiple Runners (not in autoscale mode) of the same architecture that + share a common network-mounted directory (using NFS or something similar) + where the cache will be stored + +TIP: **Tip:** +Read about the [availability of the cache](#availability-of-the-cache) +to learn more about the internals and get a better idea how cache works. + +### Sharing caches across the same branch + +Define a cache with the `key: ${CI_COMMIT_REF_SLUG}` so that jobs of each +branch always use the same cache: + +```yaml +cache: + key: ${CI_COMMIT_REF_SLUG} +``` + +While this feels like it might be safe from accidentally overwriting the cache, +it means merge requests get slow first pipelines, which might be a bad +developer experience. The next time a new commit is pushed to the branch, the +cache will be re-used. + +To enable per-job and per-branch caching: + +```yaml +cache: + key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" +``` + +To enable per-branch and per-stage caching: + +```yaml +cache: + key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" +``` + +### Sharing caches across different branches + +If the files you are caching need to be shared across all branches and all jobs, +you can use the same key for all of them: + +```yaml +cache: + key: one-key-to-rull-them-all +``` + +To share the same cache between branches, but separate them by job: + +```yaml +cache: + key: ${CI_JOB_NAME} +``` + +### Disabling cache on specific jobs + +If you have defined the cache globally, it means that each job will use the +same definition. You can override this behavior per-job, and if you want to +disable it completely, use an empty hash: + +```yaml +job: + cache: {} +``` + +For more fine tuning, read also about the +[`cache: policy`](../yaml/README.md#cache-policy). + +## Common use cases + +The most common use case of cache is to preserve contents between subsequent +runs of jobs for things like dependencies and commonly used libraries +(Nodejs packages, PHP packages, rubygems, python libraries, etc.), +so they don't have to be re-fetched from the public internet. + +NOTE: **Note:** +For more examples, check the [GitLab CI Yml](https://gitlab.com/gitlab-org/gitlab-ci-yml) +project. + +### Caching Nodejs dependencies + +Assuming your project is using [npm](https://www.npmjs.com/) or +[Yarn](https://yarnpkg.com/en/) to install the Nodejs dependencies, the +following example defines `cache` globally so that all jobs inherit it. +Nodejs modules are installed in `node_modules/` and are cached per-branch: + +```yaml +# +# https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Nodejs.gitlab-ci.yml +# +image: node:latest + +# Cache modules in between jobs +cache: + key: ${CI_COMMIT_REF_SLUG} + paths: + - node_modules/ + +before_script: + - npm install + +test_async: + script: + - node ./specs/start.js ./specs/async.spec.js +``` + +### Caching PHP dependencies + +Assuming your project is using [Composer](https://getcomposer.org/) to install +the PHP dependencies, the following example defines `cache` globally so that +all jobs inherit it. PHP libraries modules are installed in `vendor/` and +are cached per-branch: + +```yaml +# +# https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/PHP.gitlab-ci.yml +# +image: php:7.2 + +# Cache libraries in between jobs +cache: + key: ${CI_COMMIT_REF_SLUG} + paths: + - vendor/ + +before_script: +# Install and run Composer +- curl --show-error --silent https://getcomposer.org/installer | php +- php composer.phar install + +test: + script: + - vendor/bin/phpunit --configuration phpunit.xml --coverage-text --colors=never +``` + +### Caching Python dependencies + +Assuming your project is using [pip](https://pip.pypa.io/en/stable/) to install +the python dependencies, the following example defines `cache` globally so that +all jobs inherit it. Python libraries are installed in a virtualenv under `venv/`, +pip's cache is defined under `.cache/pip/` and both are cached per-branch: + +```yaml +# +# https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Python.gitlab-ci.yml +# +image: python:latest + +# Change pip's cache directory to be inside the project directory since we can +# only cache local items. +variables: + PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache" + +# Pip's cache doesn't store the python packages +# https://pip.pypa.io/en/stable/reference/pip_install/#caching +# +# If you want to also cache the installed packages, you have to install +# them in a virtualenv and cache it as well. +cache: + paths: + - .cache/ + - venv/ + +before_script: + - python -V # Print out python version for debugging + - pip install virtualenv + - virtualenv venv + - source venv/bin/activate + +test: + script: + - python setup.py test + - pip install flake8 + - flake8 . +``` + +### Caching Ruby dependencies + +Assuming your project is using [Bundler](https://bundler.io) to install the +gem dependencies, the following example defines `cache` globally so that all +jobs inherit it. Gems are installed in `vendor/ruby/` and are cached per-branch: + +```yaml +# +# https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Ruby.gitlab-ci.yml +# +image: ruby:2.5 + +# Cache gems in between builds +cache: + key: ${CI_COMMIT_REF_SLUG} + paths: + - vendor/ruby + +before_script: + - ruby -v # Print out ruby version for debugging + - gem install bundler --no-ri --no-rdoc # Bundler is not installed with the image + - bundle install -j $(nproc) --path vendor # Install dependencies into ./vendor/ruby + +rspec: + script: + - rspec spec +``` + +## Availability of the cache + +Caching is an optimization, but isn't guaranteed to always work, so you need to +be prepared to regenerate any cached files in each job that needs them. + +Assuming you have properly [defined `cache` in `.gitlab-ci.yml`](../yaml/README.md#cache) +according to your workflow, the availability of the cache ultimately depends on +how the Runner has been configured (the executor type and whether different +Runners are used for passing the cache between jobs). + +### Where the caches are stored + +Since the Runner is the one responsible for storing the cache, it's essential +to know **where** it's stored. All the cache paths defined under a job in +`.gitlab-ci.yml` are archived in a single `cache.zip` file and stored in the +Runner's configured cache location. By default, they are stored locally in the +machine where the Runner is installed and depends on the type of the executor. + +| GitLab Runner executor | Default path of the cache | +| ---------------------- | ------------------------- | +| [Shell](https://docs.gitlab.com/runner/executors/shell.html) | Locally, stored under the `gitlab-runner` user's home directory: `/home/gitlab-runner/cache/<user>/<project>/<cache-key>/cache.zip`. | +| [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, stored under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. | +| [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale Runners) | Behaves the same as the Docker executor. | + +### How archiving and extracting works + +In the most simple scenario, consider that you use only one machine where the +Runner is installed, and all jobs of your project run on the same host. + +Let's see the following example of two jobs that belong to two consecutive +stages: + +```yaml +stages: +- build +- test + +before_script: +- echo "Hello" + +job A: + stage: build + script: + - mkdir vendor/ + - echo "build" > vendor/hello.txt + cache: + key: build-cache + paths: + - vendor/ + after_script: + - echo "World" + +job B: + stage: test + script: + - cat vendor/hello.txt + cache: + key: build-cache +``` + +Here's what happens behind the scenes: + +1. Pipeline starts +1. `job A` runs +1. `before_script` is executed +1. `script` is executed +1. `after_script` is executed +1. `cache` runs and the `vendor/` directory is zipped into `cache.zip`. + This file is then saved in the directory based on the + [Runner's setting](#where-the-caches-are-stored) and the `cache: key`. +1. `job B` runs +1. The cache is extracted (if found) +1. `before_script` is executed +1. `script` is executed +1. Pipeline finishes + +By using a single Runner on a single machine, you'll not have the issue where +`job B` might execute on a Runner different from `job A`, thus guaranteeing the +cache between stages. That will only work if the build goes from stage `build` +to `test` in the same Runner/machine, otherwise, you [might not have the cache +available](#cache-mismatch). + +During the caching process, there's also a couple of things to consider: + +- If some other job, with another cache configuration had saved its + cache in the same zip file, it is overwritten. If the S3 based shared cache is + used, the file is additionally uploaded to S3 to an object based on the cache + key. So, two jobs with different paths, but the same cache key, will overwrite + their cache. +- When extracting the cache from `cache.zip`, everything in the zip file is + extracted in the job's working directory (usually the repository which is + pulled down), and the Runner doesn't mind if the archive of `job A` overwrites + things in the archive of `job B`. + +The reason why it works this way is because the cache created for one Runner +often will not be valid when used by a different one which can run on a +**different architecture** (e.g., when the cache includes binary files). And +since the different steps might be executed by Runners running on different +machines, it is a safe default. + +### Cache mismatch + +In the following table, you can see some reasons where you might hit a cache +mismatch and a few ideas how to fix it. + +| Reason of a cache mismatch | How to fix it | +| -------------------------- | ------------- | +| You use multiple standalone Runners (not in autoscale mode) attached to one project without a shared cache | Use only one Runner for your project or use multiple Runners with distributed cache enabled | +| You use Runners in autoscale mode without a distributed cache enabled | Configure the autoscale Runner to use a distributed cache | +| The machine the Runner is installed on is low on disk space or, if you've set up distributed cache, the S3 bucket where the cache is stored doesn't have enough space | Make sure you clear some space to allow new caches to be stored. Currently, there's no automatic way to do this. | +| You use the same `key` for jobs where they cache different paths. | Use different cache keys to that the cache archive is stored to a different location and doesn't overwrite wrong caches. | + +Let's explore some examples. + +--- + +Let's assume you have only one Runner assigned to your project, so the cache +will be stored in the Runner's machine by default. If two jobs, A and B, +have the same cache key, but they cache different paths, cache B would overwrite +cache A, even if their `paths` don't match: + +We want `job A` and `job B` to re-use their +cache when the pipeline is run for a second time. + +```yaml +stages: +- build +- test + +job A: + stage: build + script: make build + cache: + key: same-key + paths: + - public/ + +job B: + stage: test + script: make test + cache: + key: same-key + paths: + - vendor/ +``` + +1. `job A` runs +1. `public/` is cached as cache.zip +1. `job B` runs +1. The previous cache, if any, is unzipped +1. `vendor/` is cached as cache.zip and overwrites the previous one +1. The next time `job A` runs it will use the cache of `job B` which is different + and thus will be ineffective + +To fix that, use different `keys` for each job. + +--- + +In another case, let's assume you have more than one Runners assigned to your +project, but the distributed cache is not enabled. We want the second time the +pipeline is run, `job A` and `job B` to re-use their cache (which in this case +will be different): + +```yaml +stages: +- build +- test + +job A: + stage: build + script: build + cache: + key: keyA + paths: + - vendor/ + +job B: + stage: test + script: test + cache: + key: keyB + paths: + - vendor/ +``` + +In that case, even if the `key` is different (no fear of overwriting), you +might experience the cached files to "get cleaned" before each stage if the +jobs run on different Runners in the subsequent pipelines. + +## Clearing the cache + +GitLab Runners use [cache](../yaml/README.md#cache) to speed up the execution +of your jobs by reusing existing data. This however, can sometimes lead to an +inconsistent behavior. + +To start with a fresh copy of the cache, there are two ways to do that. + +### Clearing the cache by changing `cache:key` + +All you have to do is set a new `cache: key` in your `.gitlab-ci.yml`. In the +next run of the pipeline, the cache will be stored in a different location. + +### Clearing the cache manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41249) in GitLab 10.4. + +If you want to avoid editing `.gitlab-ci.yml`, you can easily clear the cache +via GitLab's UI: + +1. Navigate to your project's **CI/CD > Pipelines** page +1. Click on the **Clear Runner caches** button to clean up the cache + +  + +1. On the next push, your CI/CD job will use a new cache + +Behind the scenes, this works by increasing a counter in the database, and the +value of that counter is used to create the key for the cache by appending an +integer to it: `-1`, `-2`, etc. After a push, a new key is generated and the +old cache is not valid anymore. + +## Cache vs artifacts + +NOTE: **Note:** +Be careful if you use cache and artifacts to store the same path in your jobs +as **caches are restored before artifacts** and the content would be overwritten. + +Don't mix the caching with passing artifacts between stages. Caching is not +designed to pass artifacts between stages. Cache is for runtime dependencies +needed to compile the project: + +- `cache` - **Use for temporary storage for project dependencies.** Not useful + for keeping intermediate build results, like `jar` or `apk` files. + Cache was designed to be used to speed up invocations of subsequent runs of a + given job, by keeping things like dependencies (e.g., npm packages, Go vendor + packages, etc.) so they don't have to be re-fetched from the public internet. + While the cache can be abused to pass intermediate build results between stages, + there may be cases where artifacts are a better fit. +- `artifacts` - **Use for stage results that will be passed between stages.** + Artifacts were designed to upload some compiled/generated bits of the build, + and they can be fetched by any number of concurrent Runners. They are + guaranteed to be available and are there to pass data between jobs. They are + also exposed to be downloaded from the UI. + +It's sometimes confusing because the name artifact sounds like something that +is only useful outside of the job, like for downloading a final image. But +artifacts are also available in between stages within a pipeline. So if you +build your application by downloading all the required modules, you might want +to declare them as artifacts so that each subsequent stage can depend on them +being there. There are some optimizations like declaring an +[expiry time](../yaml/README.md#artifacts-expire_in) so you don't keep artifacts +around too long, and using [dependencies](../yaml/README.md#dependencies) to +control exactly where artifacts are passed around. + +So, to sum up: +- Caches are disabled if not defined globally or per job (using `cache:`) +- Caches are available for all jobs in your `.gitlab-ci.yml` if enabled globally +- Caches can be used by subsequent pipelines of that very same job (a script in + a stage) in which the cache was created (if not defined globally). +- Caches are stored where the Runner is installed **and** uploaded to S3 if + [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) +- Caches defined per job are only used either a) for the next pipeline of that job, + or b) if that same cache is also defined in a subsequent job of the same pipeline +- Artifacts are disabled if not defined per job (using `artifacts:`) +- Artifacts can only be enabled per job, not globally +- Artifacts are created during a pipeline and can be used by the subsequent + jobs of that currently active pipeline +- Artifacts are always uploaded to GitLab (known as coordinator) +- Artifacts can have an expiration value for controlling disk usage (30 days by default) diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 22afcb9199d..183808641c0 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -1,26 +1,29 @@ -# Using Docker Build +# Building Docker images with GitLab CI/CD -GitLab CI allows you to use Docker Engine to build and test docker-based projects. +GitLab CI/CD allows you to use Docker Engine to build and test docker-based projects. -**This also allows to you to use `docker-compose` and other docker-enabled tools.** +TIP: **Tip:** +This also allows to you to use `docker-compose` and other docker-enabled tools. One of the new trends in Continuous Integration/Deployment is to: -1. create an application image, -1. run tests against the created image, -1. push image to a remote registry, and -1. deploy to a server from the pushed image. +1. Create an application image +1. Run tests against the created image +1. Push image to a remote registry +1. Deploy to a server from the pushed image -It's also useful when your application already has the `Dockerfile` that can be used to create and test an image: +It's also useful when your application already has the `Dockerfile` that can be +used to create and test an image: ```bash -$ docker build -t my-image dockerfiles/ -$ docker run my-docker-image /script/to/run/tests -$ docker tag my-image my-registry:5000/my-image -$ docker push my-registry:5000/my-image +docker build -t my-image dockerfiles/ +docker run my-docker-image /script/to/run/tests +docker tag my-image my-registry:5000/my-image +docker push my-registry:5000/my-image ``` -This requires special configuration of GitLab Runner to enable `docker` support during jobs. +This requires special configuration of GitLab Runner to enable `docker` support +during jobs. ## Runner Configuration @@ -74,8 +77,8 @@ GitLab Runner then executes job scripts as the `gitlab-runner` user. 5. You can now use `docker` command and install `docker-compose` if needed. -> **Note:** -* By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. +NOTE: **Note:** +By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. For more information please read [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). ### Use docker-in-docker executor @@ -259,8 +262,66 @@ aware of the following implications: docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests ``` +## Making docker-in-docker builds faster with Docker layer caching + +When using docker-in-docker, Docker will download all layers of your image every +time you create a build. Recent versions of Docker (Docker 1.13 and above) can +use a pre-existing image as a cache during the `docker build` step, considerably +speeding up the build process. + +### How Docker caching works + +When running `docker build`, each command in `Dockerfile` results in a layer. +These layers are kept around as a cache and can be reused if there haven't been +any changes. Change in one layer causes all subsequent layers to be recreated. + +You can specify a tagged image to be used as a cache source for the `docker build` +command by using the `--cache-from` argument. Multiple images can be specified +as a cache source by using multiple `--cache-from` arguments. Keep in mind that +any image that's used with the `--cache-from` argument must first be pulled +(using `docker pull`) before it can be used as a cache source. + +### Using Docker caching + +Here's a simple `.gitlab-ci.yml` file showing how Docker caching can be utilized: + +```yaml +image: docker:latest + +services: + - docker:dind + +variables: + CONTAINER_IMAGE: registry.gitlab.com/$CI_PROJECT_PATH + DOCKER_DRIVER: overlay2 + +before_script: + - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com + +build: + stage: build + script: + - docker pull $CONTAINER_IMAGE:latest || true + - docker build --cache-from $CONTAINER_IMAGE:latest --tag $CONTAINER_IMAGE:$CI_BUILD_REF --tag $CONTAINER_IMAGE:latest . + - docker push $CONTAINER_IMAGE:$CI_BUILD_REF + - docker push $CONTAINER_IMAGE:latest +``` + +The steps in the `script` section for the `build` stage can be summed up to: + +1. The first command tries to pull the image from the registry so that it can be + used as a cache for the `docker build` command. +1. The second command builds a Docker image using the pulled image as a + cache (notice the `--cache-from $CONTAINER_IMAGE:latest` argument) if + available, and tags it. +1. The last two commands push the tagged Docker images to the container registry + so that they may also be used as cache for subsequent builds. + ## Using the OverlayFS driver +NOTE: **Note:** +The shared Runners on GitLab.com use the `overlay2` driver by default. + By default, when using `docker:dind`, Docker uses the `vfs` storage driver which copies the filesystem on every run. This is a very disk-intensive operation which can be avoided if a different driver is used, for example `overlay2`. diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index fb5bfe26bb0..bc5d3840368 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -58,7 +58,7 @@ your job and is linked to the Docker image that the `image` keyword defines. This allows you to access the service image during build time. The service image can run any application, but the most common use case is to -run a database container, eg. `mysql`. It's easier and faster to use an +run a database container, e.g., `mysql`. It's easier and faster to use an existing image and run it as an additional container than install `mysql` every time the project is built. @@ -83,6 +83,67 @@ So, in order to access your database service you have to connect to the host named `mysql` instead of a socket or `localhost`. Read more in [accessing the services](#accessing-the-services). +### How the health check of services works + +Services are designed to provide additional functionality which is **network accessible**. +It may be a database like MySQL, or Redis, and even `docker:dind` which +allows you to use Docker in Docker. It can be practically anything that is +required for the CI/CD job to proceed and is accessed by network. + +To make sure this works, the Runner: + +1. checks which ports are exposed from the container by default +1. starts a special container that waits for these ports to be accessible + +When the second stage of the check fails, either because there is no opened port in the +service, or the service was not started properly before the timeout and the port is not +responding, it prints the warning: `*** WARNING: Service XYZ probably didn't start properly`. + +In most cases it will affect the job, but there may be situations when the job +will still succeed even if that warning was printed. For example: + +- The service was started a little after the warning was raised, and the job is + not using the linked service from the very beginning. In that case, when the + job needed to access the service, it may have been already there waiting for + connections. +- The service container is not providing any networking service, but it's doing + something with the job's directory (all services have the job directory mounted + as a volume under `/builds`). In that case, the service will do its job, and + since the job is not trying to connect to it, it won't fail. + +### What services are not for + +As it was mentioned before, this feature is designed to provide **network accessible** +services. A database is the simplest example of such a service. + +NOTE: **Note:** +The services feature is not designed to, and will not add any software from the +defined `services` image(s) to the job's container. + +For example, if you have the following `services` defined in your job, the `php`, +`node` or `go` commands will **not** be available for your script, and thus +the job will fail: + +```yaml +job: + services: + - php:7 + - node:latest + - golang:1.10 + image: alpine:3.7 + script: + - php -v + - node -v + - go version +``` + +If you need to have `php`, `node` and `go` available for your script, you should +either: + +- choose an existing Docker image that contains all required tools, or +- create your own Docker image, which will have all the required tools included + and use that in your job + ### Accessing the services Let's say that you need a Wordpress instance to test some API integration with diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 0109e77935a..de60cd27cd1 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -23,7 +23,7 @@ There's also a collection of repositories with [example projects](https://gitlab - **Scala**: [Test a Scala application](test-scala-application.md) - **Clojure**: [Test a Clojure application](test-clojure-application.md) - **Elixir**: - - [Test a Phoenix application](test-phoenix-application.md) + - [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md) - [Building an Elixir Release into a Docker image using GitLab CI](https://about.gitlab.com/2016/08/11/building-an-elixir-release-into-docker-image-using-gitlab-ci-part-1/) - **iOS and macOS**: - [Setting up GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) @@ -32,34 +32,45 @@ There's also a collection of repositories with [example projects](https://gitlab - **Debian**: [Continuous Deployment with GitLab: how to build and deploy a Debian Package with GitLab CI](https://about.gitlab.com/2016/10/12/automated-debian-package-build-with-gitlab-ci/) - **Maven**: [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md) +### Game development + +- [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md) + ### Miscellaneous - [Using `dpl` as deployment tool](deployment/README.md) - [The `.gitlab-ci.yml` file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) -### Code quality analysis +## Code quality analysis [Analyze code quality with the Code Climate CLI](code_climate.md). -### Static Application Security Testing (SAST) +## Static Application Security Testing (SAST) + +**(Ultimate)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html) + +## Dependency Scanning + +**(Ultimate)** [Scan your dependencies for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) + +## Container Scanning -- **(EEU)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html) -- [Scan your Docker images for vulnerabilities](sast_docker.md) +[Scan your Docker images for vulnerabilities](container_scanning.md) -### Dynamic Application Security Testing (DAST) +## Dynamic Application Security Testing (DAST) Scan your app for vulnerabilities with GitLab [Dynamic Application Security Testing (DAST)](dast.md). -### Browser Performance Testing with Sitespeed.io +## Browser Performance Testing with Sitespeed.io Analyze your [browser performance with Sitespeed.io](browser_performance.md). -### GitLab CI/CD for Review Apps +## GitLab CI/CD for Review Apps - [Example project](https://gitlab.com/gitlab-examples/review-apps-nginx/) that shows how to use GitLab CI/CD for [Review Apps](../review_apps/index.html). - [Dockerizing GitLab Review Apps](https://about.gitlab.com/2017/07/11/dockerizing-review-apps/) -### GitLab CI/CD for GitLab Pages +## GitLab CI/CD for GitLab Pages See the documentation on [GitLab Pages](../../user/project/pages/index.md) for a complete overview. diff --git a/doc/ci/examples/artifactory_and_gitlab/index.md b/doc/ci/examples/artifactory_and_gitlab/index.md index 8e91cd05d8a..d931c9a77f4 100644 --- a/doc/ci/examples/artifactory_and_gitlab/index.md +++ b/doc/ci/examples/artifactory_and_gitlab/index.md @@ -1,14 +1,14 @@ --- redirect_from: 'https://docs.gitlab.com/ee/articles/artifactory_and_gitlab/index.html' +author: Fabio Busatto +author_gitlab: bikebilly +level: intermediary +article_type: tutorial +date: 2017-08-15 --- # How to deploy Maven projects to Artifactory with GitLab CI/CD -> **[Article Type](../../../development/writing_documentation.md#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Fabio Busatto](https://gitlab.com/bikebilly) || -> **Publication date:** 2017-08-15 - ## Introduction In this article, we will show how you can leverage the power of [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/) diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index a7945d05cd0..42dc6ef36ba 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -13,16 +13,20 @@ Once you set up the Runner, add a new job to `.gitlab-ci.yml`, called `performan services: - docker:dind script: + - mkdir gitlab-exporter + - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/10-5/index.js - mkdir sitespeed-results - - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io --outputFolder sitespeed-results https://my.website.com + - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results https://my.website.com + - mv sitespeed-results/data/performance.json performance.json artifacts: paths: + - performance.json - sitespeed-results/ ``` -This will create a `performance` job in your CI/CD pipeline and will run Sitespeed.io against the webpage you define. The full HTML Sitespeed.io report will be saved as an artifact, and if you have Pages enabled it can be viewed directly in your browser. For further customization options of Sitespeed.io, including the ability to provide a list of URLs to test, please consult their [documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/). +This will create a `performance` job in your CI/CD pipeline and will run Sitespeed.io against the webpage you define. The GitLab plugin for Sitespeed.io is downloaded in order to export key metrics to JSON. The full HTML Sitespeed.io report will also be saved as an artifact, and if you have Pages enabled it can be viewed directly in your browser. For further customization options of Sitespeed.io, including the ability to provide a list of URLs to test, please consult their [documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/). -For GitLab [Enterprise Edition Premium](https://about.gitlab.com/gitlab-ee/) users, a performance score can be automatically +For [GitLab Premium](https://about.gitlab.com/products/) users, key metrics are automatically extracted and shown right in the merge request widget. Learn more about [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). ## Performance testing on Review Apps @@ -42,9 +46,15 @@ A simple `performance` job would look like: - docker:dind script: - export CI_ENVIRONMENT_URL=$(cat environment_url.txt) + - mkdir gitlab-exporter + - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/10-5/index.js - mkdir sitespeed-results - - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io --outputFolder sitespeed-results $CI_ENVIRONMENT_URL + - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results "$CI_ENVIRONMENT_URL" + - mv sitespeed-results/data/performance.json performance.json artifacts: paths: + - performance.json - sitespeed-results/ ``` + +A complete example can be found in our [Auto DevOps CI YML](https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Auto-DevOps.gitlab-ci.yml).
\ No newline at end of file diff --git a/doc/ci/examples/code_climate.md b/doc/ci/examples/code_climate.md index f919ed3c797..64a759a9a99 100644 --- a/doc/ci/examples/code_climate.md +++ b/doc/ci/examples/code_climate.md @@ -15,20 +15,23 @@ codequality: services: - docker:dind script: - - docker pull codeclimate/codeclimate - - docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 init - - docker run --env CODECLIMATE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock --volume /tmp/cc:/tmp/cc codeclimate/codeclimate:0.69.0 analyze -f json > codeclimate.json || true + - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') + - docker run --env SOURCE_CODE="$PWD" --volume "$PWD":/code --volume /var/run/docker.sock:/var/run/docker.sock "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code artifacts: paths: [codeclimate.json] ``` -This will create a `codequality` job in your CI pipeline and will allow you to -download and analyze the report artifact in JSON format. +The above example will create a `codequality` job in your CI/CD pipeline which +will scan your source code for code quality issues. The report will be saved +as an artifact that you can later download and analyze. -For GitLab [Enterprise Edition Starter][ee] users, this information can be automatically -extracted and shown right in the merge request widget. [Learn more on code quality -diffs in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html). +TIP: **Tip:** +Starting with [GitLab Starter][ee] 9.3, this information will +be automatically extracted and shown right in the merge request widget. To do +so, the CI/CD job must be named `codequality` and the artifact path must be +`codeclimate.json`. +[Learn more on code quality diffs in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html). [cli]: https://github.com/codeclimate/codeclimate [dind]: ../docker/using_docker_build.md#use-docker-in-docker-executor -[ee]: https://about.gitlab.com/gitlab-ee/ +[ee]: https://about.gitlab.com/products/ diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md new file mode 100644 index 00000000000..3437b63748a --- /dev/null +++ b/doc/ci/examples/container_scanning.md @@ -0,0 +1,55 @@ +# Container Scanning with GitLab CI/CD + +You can check your Docker images (or more precisely the containers) for known +vulnerabilities by using [Clair](https://github.com/coreos/clair) and +[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools +for Vulnerability Static Analysis for containers. + +All you need is a GitLab Runner with the Docker executor (the shared Runners on +GitLab.com will work fine). You can then add a new job to `.gitlab-ci.yml`, +called `sast:container`: + +```yaml +sast:container: + image: docker:latest + variables: + DOCKER_DRIVER: overlay2 + ## Define two new variables based on GitLab's CI/CD predefined variables + ## https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables + CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG + CI_APPLICATION_TAG: $CI_COMMIT_SHA + allow_failure: true + services: + - docker:dind + script: + - docker run -d --name db arminc/clair-db:latest + - docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1 + - apk add -U wget ca-certificates + - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} + - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 + - mv clair-scanner_linux_amd64 clair-scanner + - chmod +x clair-scanner + - touch clair-whitelist.yml + - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-sast-container-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true + artifacts: + paths: [gl-sast-container-report.json] +``` + +The above example will create a `sast:container` job in your CI/CD pipeline, pull +the image from the [Container Registry](../../user/project/container_registry.md) +(whose name is defined from the two `CI_APPLICATION_` variables) and scan it +for possible vulnerabilities. The report will be saved as an artifact that you +can later download and analyze. + +If you want to whitelist some specific vulnerabilities, you can do so by defining +them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file), +in our case its named `clair-whitelist.yml`. + +TIP: **Tip:** +Starting with [GitLab Ultimate][ee] 10.4, this information will +be automatically extracted and shown right in the merge request widget. To do +so, the CI/CD job must be named `sast:container` and the artifact path must be +`gl-sast-container-report.json`. +[Learn more on container scanning results shown in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). + +[ee]: https://about.gitlab.com/products/ diff --git a/doc/ci/examples/dast.md b/doc/ci/examples/dast.md index 7bf647bbb8b..96de0f5ff5c 100644 --- a/doc/ci/examples/dast.md +++ b/doc/ci/examples/dast.md @@ -31,10 +31,10 @@ own) and finally write the results in the `gl-dast-report.json` file. You can then download and analyze the report artifact in JSON format. TIP: **Tip:** -Starting with [GitLab Enterprise Edition Ultimate][ee] 10.4, this information will +Starting with [GitLab Ultimate][ee] 10.4, this information will be automatically extracted and shown right in the merge request widget. To do so, the CI job must be named `dast` and the artifact path must be `gl-dast-report.json`. [Learn more about DAST results shown in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html). -[ee]: https://about.gitlab.com/gitlab-ee/ +[ee]: https://about.gitlab.com/products/ diff --git a/doc/ci/examples/deployment/README.md b/doc/ci/examples/deployment/README.md index e80e246c5dd..2dcdc2d41ec 100644 --- a/doc/ci/examples/deployment/README.md +++ b/doc/ci/examples/deployment/README.md @@ -111,7 +111,7 @@ We also use two secure variables: ## Storing API keys Secure Variables can added by going to your project's -**Settings ➔ Pipelines ➔ Secret variables**. The variables that are defined +**Settings ➔ CI / CD ➔ Secret variables**. The variables that are defined in the project settings are sent along with the build script to the Runner. The secure variables are stored out of the repository. Never store secrets in your project's `.gitlab-ci.yml`. It is also important that the secret's value diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png Binary files differnew file mode 100644 index 00000000000..76e0295722b --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/aws_config_window.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png Binary files differnew file mode 100644 index 00000000000..050a97d2726 --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/gitlab_config.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png Binary files differnew file mode 100644 index 00000000000..4ab5d5f401a --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/img/test_pipeline_pass.png diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md new file mode 100644 index 00000000000..bfc8558a580 --- /dev/null +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -0,0 +1,526 @@ +--- +author: Ryan Hall +author_gitlab: blitzgren +level: intermediary +article_type: tutorial +date: 2018-03-07 +--- + +# DevOps and Game Dev with GitLab CI/CD + +With advances in WebGL and WebSockets, browsers are extremely viable as game development +platforms without the use of plugins like Adobe Flash. Furthermore, by using GitLab and [AWS](https://aws.amazon.com/), +single game developers, as well as game dev teams, can easily host browser-based games online. + +In this tutorial, we'll focus on DevOps, as well as testing and hosting games with Continuous +Integration/Deployment methods. We assume you are familiar with GitLab, javascript, +and the basics of game development. + +## The game + +Our [demo game](http://gitlab-game-demo.s3-website-us-east-1.amazonaws.com/) consists of a simple spaceship traveling in space that shoots by clicking the mouse in a given direction. + +Creating a strong CI/CD pipeline at the beginning of developing another game, [Dark Nova](http://darknova.io/about), +was essential for the fast pace the team worked at. This tutorial will build upon my +[previous introductory article](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) and go through the following steps: + +1. Using code from the previous article to start with a barebones [Phaser](https://phaser.io) game built by a gulp file +1. Adding and running unit tests +1. Creating a `Weapon` class that can be triggered to spawn a `Bullet` in a given direction +1. Adding a `Player` class that uses this weapon and moves around the screen +1. Adding the sprites we will use for the `Player` and `Weapon` +1. Testing and deploying with Continuous Integration and Continuous Deployment methods + +By the end, we'll have the core of a [playable game](http://gitlab-game-demo.s3-website-us-east-1.amazonaws.com/) +that's tested and deployed on every push to the `master` branch of the [codebase](https://gitlab.com/blitzgren/gitlab-game-demo). +This will also provide +boilerplate code for starting a browser-based game with the following components: + +- Written in [Typescript](https://www.typescriptlang.org/) and [PhaserJs](https://phaser.io) +- Building, running, and testing with [Gulp](http://gulpjs.com/) +- Unit tests with [Chai](http://chaijs.com/) and [Mocha](https://mochajs.org/) +- CI/CD with GitLab +- Hosting the codebase on GitLab.com +- Hosting the game on AWS +- Deploying to AWS + +## Requirements and setup + +Please refer to my previous article [DevOps and Game Dev](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) to learn the foundational +development tools, running a Hello World-like game, and building this game using GitLab +CI/CD from every new push to master. The `master` branch for this game's [repository](https://gitlab.com/blitzgren/gitlab-game-demo) +contains a completed version with all configurations. If you would like to follow along +with this article, you can clone and work from the `devops-article` branch: + +```sh +git clone git@gitlab.com:blitzgren/gitlab-game-demo.git +git checkout devops-article +``` + +Next, we'll create a small subset of tests that exemplify most of the states I expect +this `Weapon` class to go through. To get started, create a folder called `lib/tests` +and add the following code to a new file `weaponTests.ts`: + +```ts +import { expect } from 'chai'; +import { Weapon, BulletFactory } from '../lib/weapon'; + +describe('Weapon', () => { + var subject: Weapon; + var shotsFired: number = 0; + // Mocked bullet factory + var bulletFactory: BulletFactory = <BulletFactory>{ + generate: function(px, py, vx, vy, rot) { + shotsFired++; + } + }; + var parent: any = { x: 0, y: 0 }; + + beforeEach(() => { + shotsFired = 0; + subject = new Weapon(bulletFactory, parent, 0.25, 1); + }); + + it('should shoot if not in cooldown', () => { + subject.trigger(true); + subject.update(0.1); + expect(shotsFired).to.equal(1); + }); + + it('should not shoot during cooldown', () => { + subject.trigger(true); + subject.update(0.1); + subject.update(0.1); + expect(shotsFired).to.equal(1); + }); + + it('should shoot after cooldown ends', () => { + subject.trigger(true); + subject.update(0.1); + subject.update(0.3); // longer than timeout + expect(shotsFired).to.equal(2); + }); + + it('should not shoot if not triggered', () => { + subject.update(0.1); + subject.update(0.1); + expect(shotsFired).to.equal(0); + }); +}); +``` + +To build and run these tests using gulp, let's also add the following gulp functions +to the existing `gulpfile.js` file: + +```ts +gulp.task('build-test', function () { + return gulp.src('src/tests/**/*.ts', { read: false }) + .pipe(tap(function (file) { + // replace file contents with browserify's bundle stream + file.contents = browserify(file.path, { debug: true }) + .plugin(tsify, { project: "./tsconfig.test.json" }) + .bundle(); + })) + .pipe(buffer()) + .pipe(sourcemaps.init({loadMaps: true}) ) + .pipe(gulp.dest('built/tests')); +}); + +gulp.task('run-test', function() { + gulp.src(['./built/tests/**/*.ts']).pipe(mocha()); +}); +``` + +We will start implementing the first part of our game and get these `Weapon` tests to pass. +The `Weapon` class will expose a method to trigger the generation of a bullet at a given +direction and speed. Later we will implement a `Player` class that ties together the user input +to trigger the weapon. In the `src/lib` folder create a `weapon.ts` file. We'll add two classes +to it: `Weapon` and `BulletFactory` which will encapsulate Phaser's **sprite** and +**group** objects, and the logic specific to our game. + +```ts +export class Weapon { + private isTriggered: boolean = false; + private currentTimer: number = 0; + + constructor(private bulletFactory: BulletFactory, private parent: Phaser.Sprite, private cooldown: number, private bulletSpeed: number) { + } + + public trigger(on: boolean): void { + this.isTriggered = on; + } + + public update(delta: number): void { + this.currentTimer -= delta; + + if (this.isTriggered && this.currentTimer <= 0) { + this.shoot(); + } + } + + private shoot(): void { + // Reset timer + this.currentTimer = this.cooldown; + + // Get velocity direction from player rotation + var parentRotation = this.parent.rotation + Math.PI / 2; + var velx = Math.cos(parentRotation); + var vely = Math.sin(parentRotation); + + // Apply a small forward offset so bullet shoots from head of ship instead of the middle + var posx = this.parent.x - velx * 10 + var posy = this.parent.y - vely * 10; + + this.bulletFactory.generate(posx, posy, -velx * this.bulletSpeed, -vely * this.bulletSpeed, this.parent.rotation); + } +} + +export class BulletFactory { + + constructor(private bullets: Phaser.Group, private poolSize: number) { + // Set all the defaults for this BulletFactory's bullet object + this.bullets.enableBody = true; + this.bullets.physicsBodyType = Phaser.Physics.ARCADE; + this.bullets.createMultiple(30, 'bullet'); + this.bullets.setAll('anchor.x', 0.5); + this.bullets.setAll('anchor.y', 0.5); + this.bullets.setAll('outOfBoundsKill', true); + this.bullets.setAll('checkWorldBounds', true); + } + + public generate(posx: number, posy: number, velx: number, vely: number, rot: number): Phaser.Sprite { + // Pull a bullet from Phaser's Group pool + var bullet = this.bullets.getFirstExists(false); + + // Set the few unique properties about this bullet: rotation, position, and velocity + if (bullet) { + bullet.reset(posx, posy); + bullet.rotation = rot; + bullet.body.velocity.x = velx; + bullet.body.velocity.y = vely; + } + + return bullet; + } +} +``` + +Lastly, we'll redo our entry point, `game.ts`, to tie together both `Player` and `Weapon` objects +as well as add them to the update loop. Here is what the updated `game.ts` file looks like: + +```ts +import { Player } from "./player"; +import { Weapon, BulletFactory } from "./weapon"; + +window.onload = function() { + var game = new Phaser.Game(800, 600, Phaser.AUTO, 'gameCanvas', { preload: preload, create: create, update: update }); + var player: Player; + var weapon: Weapon; + + // Import all assets prior to loading the game + function preload () { + game.load.image('player', 'assets/player.png'); + game.load.image('bullet', 'assets/bullet.png'); + } + + // Create all entities in the game, after Phaser loads + function create () { + // Create and position the player + var playerSprite = game.add.sprite(400, 550, 'player'); + playerSprite.anchor.setTo(0.5); + player = new Player(game.input, playerSprite, 150); + + var bulletFactory = new BulletFactory(game.add.group(), 30); + weapon = new Weapon(bulletFactory, player.sprite, 0.25, 1000); + + player.loadWeapon(weapon); + } + + // This function is called once every tick, default is 60fps + function update() { + var deltaSeconds = game.time.elapsedMS / 1000; // convert to seconds + player.update(deltaSeconds); + weapon.update(deltaSeconds); + } +} +``` + +Run `gulp serve` and you can run around and shoot. Wonderful! Let's update our CI +pipeline to include running the tests along with the existing build job. + +## Continuous Integration + +To ensure our changes don't break the build and all tests still pass, we utilize +Continuous Integration (CI) to run these checks automatically for every push. +Read through this article to understand [Continuous Integration, Continuous Delivery, and Continuous Deployment](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/), +and how these methods are leveraged by GitLab. +From the [last tutorial](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) we already have a `gitlab-ci.yml` file set up for building our app from +every push. We need to set up a new CI job for testing, which GitLab CI/CD will run after the build job using our generated artifacts from gulp. + +Please read through the [documentation on CI/CD configuration](../../../ci/yaml/README.md) file to explore its contents and adjust it to your needs. + +### Build your game with GitLab CI/CD + +We need to update our build job to ensure tests get run as well. Add `gulp build-test` +to the end of the `script` array for the existing `build` job. Once these commands run, +we know we will need to access everything in the `built` folder, given by GitLab CI/CD's `artifacts`. +We'll also cache `node_modules` to avoid having to do a full re-pull of those dependencies: +just pack them up in the cache. Here is the full `build` job: + +```yml +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules + artifacts: + paths: + - built +``` + +### Test your game with GitLab CI/CD + +For testing locally, we simply run `gulp run-tests`, which requires gulp to be installed +globally like in the `build` job. We pull `node_modules` from the cache, so the `npm i` +command won't have to do much. In preparation for deployment, we know we will still need +the `built` folder in the artifacts, which will be brought over as default behavior from +the previous job. Lastly, by convention, we let GitLab CI/CD know this needs to be run after +the `build` job by giving it a `test` [stage](../../../ci/yaml/README.md#stages). +Following the YAML structure, the `test` job should look like this: + +```yml +test: + stage: test + script: + - npm i gulp -g + - npm i + - gulp run-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ +``` + +We have added unit tests for a `Weapon` class that shoots on a specified interval. +The `Player` class implements `Weapon` along with the ability to move around and shoot. Also, +we've added test artifacts and a test stage to our GitLab CI/CD pipeline using `.gitlab-ci.yml`, +allowing us to run our tests by every push. +Our entire `.gitlab-ci.yml` file should now look like this: + +```yml +image: node:6 + +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ + +test: + stage: test + script: + - npm i gulp -g + - npm i + - gulp run-test + cache: + policy: pull + paths: + - node_modules/ + artifacts: + paths: + - built/ +``` + +### Run your CI/CD pipeline + +That's it! Add all your new files, commit, and push. For a reference of what our repo should +look like at this point, please refer to the [final commit related to this article on my sample repository](https://gitlab.com/blitzgren/gitlab-game-demo/commit/8b36ef0ecebcf569aeb251be4ee13743337fcfe2). +By applying both build and test stages, GitLab will run them sequentially at every push to +our repository. If all goes well you'll end up with a green check mark on each job for the pipeline: + + + +You can confirm that the tests passed by clicking on the `test` job to enter the full build logs. +Scroll to the bottom and observe, in all its passing glory: + +```sh +$ gulp run-test +[18:37:24] Using gulpfile /builds/blitzgren/gitlab-game-demo/gulpfile.js +[18:37:24] Starting 'run-test'... +[18:37:24] Finished 'run-test' after 21 ms + + + Weapon + ✓ should shoot if not in cooldown + ✓ should not shoot during cooldown + ✓ should shoot after cooldown ends + ✓ should not shoot if not triggered + + + 4 passing (18ms) + +Uploading artifacts... +built/: found 17 matching files +Uploading artifacts to coordinator... ok id=17095874 responseStatus=201 Created token=aaaaaaaa Job succeeded +``` + +## Continuous Deployment + +We have our codebase built and tested on every push. To complete the full pipeline with Continuous Deployment, +let's set up [free web hosting with AWS S3](https://aws.amazon.com/s/dm/optimization/server-side-test/free-tier/free_np/) and a job through which our build artifacts get +deployed. GitLab also has a free static site hosting service we could use, [GitLab Pages](https://about.gitlab.com/features/pages/), +however Dark Nova specifically uses other AWS tools that necessitates using `AWS S3`. +Read through this article that describes [deploying to both S3 and GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) +and further delves into the principles of GitLab CI/CD than discussed in this article. + +### Set up S3 Bucket + +1. Log into your AWS account and go to [S3](https://console.aws.amazon.com/s3/home) +1. Click the **Create Bucket** link at the top +1. Enter a name of your choosing and click next +1. Keep the default **Properties** and click next +1. Click the **Manage group permissions** and allow **Read** for the **Everyone** group, click next +1. Create the bucket, and select it in your S3 bucket list +1. On the right side, click **Properties** and enable the **Static website hosting** category +1. Update the radio button to the **Use this bucket to host a website** selection. Fill in `index.html` and `error.html` respectively + +### Set up AWS Secrets + +We need to be able to deploy to AWS with our AWS account credentials, but we certainly +don't want to put secrets into source code. Luckily GitLab provides a solution for this +with [Secret Variables](../../../ci/variables/README.md). This can get complicated +due to [IAM](https://aws.amazon.com/iam/) management. As a best practice, you shouldn't +use root security credentials. Proper IAM credential management is beyond the scope of this +article, but AWS will remind you that using root credentials is unadvised and against their +best practices, as they should. Feel free to follow best practices and use a custom IAM user's +credentials, which will be the same two credentials (Key ID and Secret). It's a good idea to +fully understand [IAM Best Practices in AWS](http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html). We need to add these credentials to GitLab: + +1. Log into your AWS account and go to the [Security Credentials page](https://console.aws.amazon.com/iam/home#/security_credential) +1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the id and secret around, you'll need them later +  +1. Go to your GitLab project, click **Settings > CI/CD** on the left sidebar +1. Expand the **Secret Variables** section +  +1. Add a key named `AWS_KEY_ID` and copy the key id from Step 2 into the **Value** textbox +1. Add a key named `AWS_KEY_SECRET` and copy the key secret from Step 2 into the **Value** textbox + +### Deploy your game with GitLab CI/CD + +To deploy our build artifacts, we need to install the [AWS CLI](https://aws.amazon.com/cli/) on +the Shared Runner. The Shared Runner also needs to be able to authenticate with your AWS +account to deploy the artifacts. By convention, AWS CLI will look for `AWS_ACCESS_KEY_ID` +and `AWS_SECRET_ACCESS_KEY`. GitLab's CI gives us a way to pass the secret variables we +set up in the prior section using the `variables` portion of the `deploy` job. At the end, +we add directives to ensure deployment `only` happens on pushes to `master`. This way, every +single branch still runs through CI, and only merging (or committing directly) to master will +trigger the `deploy` job of our pipeline. Put these together to get the following: + +```yml +deploy: + stage: deploy + variables: + AWS_ACCESS_KEY_ID: "$AWS_KEY_ID" + AWS_SECRET_ACCESS_KEY: "$AWS_KEY_SECRET" + script: + - apt-get update + - apt-get install -y python3-dev python3-pip + - easy_install3 -U pip + - pip3 install --upgrade awscli + - aws s3 sync ./built s3://gitlab-game-demo --region "us-east-1" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers --cache-control "no-cache, no-store, must-revalidate" --delete + only: + - master +``` + +Be sure to update the region and S3 URL in that last script command to fit your setup. +Our final configuration file `.gitlab-ci.yml` looks like: + +```yml +image: node:6 + +build: + stage: build + script: + - npm i gulp -g + - npm i + - gulp + - gulp build-test + cache: + policy: push + paths: + - node_modules/ + artifacts: + paths: + - built/ + +test: + stage: test + script: + - npm i gulp -g + - gulp run-test + cache: + policy: pull + paths: + - node_modules/ + artifacts: + paths: + - built/ + +deploy: + stage: deploy + variables: + AWS_ACCESS_KEY_ID: "$AWS_KEY_ID" + AWS_SECRET_ACCESS_KEY: "$AWS_KEY_SECRET" + script: + - apt-get update + - apt-get install -y python3-dev python3-pip + - easy_install3 -U pip + - pip3 install --upgrade awscli + - aws s3 sync ./built s3://gitlab-game-demo --region "us-east-1" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers --cache-control "no-cache, no-store, must-revalidate" --delete + only: + - master +``` + +## Conclusion + +Within the [demo repository](https://gitlab.com/blitzgren/gitlab-game-demo) you can also find a handful of boilerplate code to get +[Typescript](https://www.typescriptlang.org/), [Mocha](https://mochajs.org/), [Gulp](http://gulpjs.com/) and [Phaser](https://phaser.io) all playing +together nicely with GitLab CI/CD, which is the result of lessons learned while making [Dark Nova](http://darknova.io/). +Using a combination of free and open source software, we have a full CI/CD pipeline, a game foundation, +and unit tests, all running and deployed at every push to master - with shockingly little code. +Errors can be easily debugged through GitLab's build logs, and within minutes of a successful commit, +you can see the changes live on your game. + +Setting up Continous Integration and Continuous Deployment from the start with Dark Nova enables +rapid but stable development. We can easily test changes in a separate [environment](../../../ci/environments.md#introduction-to-environments-and-deployments), +or multiple environments if needed. Balancing and updating a multiplayer game can be ongoing +and tedious, but having faith in a stable deployment with GitLab CI/CD allows +a lot of breathing room in quickly getting changes to players. + +## Further settings + +Here are some ideas to further investigate that can speed up or improve your pipeline: + +- [Yarn](https://yarnpkg.com) instead of npm +- Setup a custom [Docker](../../../ci/docker/using_docker_images.md#define-image-and-services-from-gitlab-ci-yml) image that can preload dependencies and tools (like AWS CLI) +- Forward a [custom domain](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) to your game's S3 static website +- Combine jobs if you find it unnecessary for a small project +- Avoid the queues and set up your own [custom GitLab CI/CD runner](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/) diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index e1aff6fdf36..b62874ef029 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -1,14 +1,14 @@ --- redirect_from: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' +author: Mehran Rasulian +author_gitlab: mehranrasulian +level: intermediary +article_type: tutorial +date: 2017-08-31 --- # Test and deploy Laravel applications with GitLab CI/CD and Envoy -> **[Article Type](../../../development/writing_documentation.md#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Mehran Rasulian](https://gitlab.com/mehranrasulian) || -> **Publication date:** 2017-08-31 - ## Introduction GitLab features our applications with Continuous Integration, and it is possible to easily deploy the new code changes to the production server whenever we want. diff --git a/doc/ci/examples/sast_docker.md b/doc/ci/examples/sast_docker.md index d99cfe93afa..9f4a63e296d 100644 --- a/doc/ci/examples/sast_docker.md +++ b/doc/ci/examples/sast_docker.md @@ -1,55 +1 @@ -# Static Application Security Testing for Docker containers with GitLab CI/CD - -You can check your Docker images (or more precisely the containers) for known -vulnerabilities by using [Clair](https://github.com/coreos/clair) and -[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools -for Vulnerability Static Analysis for containers. - -All you need is a GitLab Runner with the Docker executor (the shared Runners on -GitLab.com will work fine). You can then add a new job to `.gitlab-ci.yml`, -called `sast:container`: - -```yaml -sast:container: - image: docker:latest - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - allow_failure: true - services: - - docker:dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1 - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-sast-container-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - paths: [gl-sast-container-report.json] -``` - -The above example will create a `sast:container` job in your CI/CD pipeline, pull -the image from the [Container Registry](../../user/project/container_registry.md) -(whose name is defined from the two `CI_APPLICATION_` variables) and scan it -for possible vulnerabilities. The report will be saved as an artifact that you -can later download and analyze. - -If you want to whitelist some specific vulnerabilities, you can do so by defining -them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file), -in our case its named `clair-whitelist.yml`. - -TIP: **Tip:** -Starting with [GitLab Enterprise Edition Ultimate][ee] 10.4, this information will -be automatically extracted and shown right in the merge request widget. To do -so, the CI/CD job must be named `sast:container` and the artifact path must be -`gl-sast-container-report.json`. -[Learn more on application security testing results shown in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/sast_docker.html). - -[ee]: https://about.gitlab.com/gitlab-ee/ +This document was moved to [another location](./container_scanning.md).
\ No newline at end of file diff --git a/doc/ci/examples/test-phoenix-application.md b/doc/ci/examples/test-phoenix-application.md index 7e49721daf1..52db5740c34 100644 --- a/doc/ci/examples/test-phoenix-application.md +++ b/doc/ci/examples/test-phoenix-application.md @@ -1,55 +1,5 @@ -# Test a Phoenix application with GitLab CI/CD +--- +redirect_to: '../../ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md' +--- -This example demonstrates the integration of Gitlab CI with Phoenix, Elixir and -Postgres. - -## Add `.gitlab-ci.yml` to project - -The following `.gitlab-ci.yml` should be added in the root of your -repository to trigger CI: - -```yaml -image: elixir:1.3 - -services: - - postgres:9.6 - -variables: - MIX_ENV: "test" - -before_script: - # Setup phoenix dependencies - - apt-get update - - apt-get install -y postgresql-client - - mix local.hex --force - - mix deps.get --only test - - mix ecto.reset - -test: - script: - - mix test -``` - -The variables will set the Mix environment to "test". The -`before_script` will install `psql`, some Phoenix dependencies, and will also -run your migrations. - -Finally, the test `script` will run your tests. - -## Update the Config Settings - -In `config/test.exs`, update the database hostname: - -```elixir -config :my_app, MyApp.Repo, - hostname: if(System.get_env("CI"), do: "postgres", else: "localhost"), -``` - -## Add the Migrations Folder - -If you do not have any migrations yet, you will need to create an empty -`.gitkeep` file in `priv/repo/migrations`. - -## Sources - -- https://medium.com/@nahtnam/using-phoenix-on-gitlab-ci-5a51eec81142 +The content of this page was incorporated in [this document](../../ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md). diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/job-succeeded.png b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/job-succeeded.png Binary files differnew file mode 100644 index 00000000000..0f94ac60fee --- /dev/null +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/job-succeeded.png diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/mix-phoenix-new.png b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/mix-phoenix-new.png Binary files differnew file mode 100644 index 00000000000..94828a20f51 --- /dev/null +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/mix-phoenix-new.png diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/mix-phoenix-server.png b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/mix-phoenix-server.png Binary files differnew file mode 100644 index 00000000000..68503b392ed --- /dev/null +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/mix-phoenix-server.png diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/pipelines.png b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/pipelines.png Binary files differnew file mode 100644 index 00000000000..d73140ccdd9 --- /dev/null +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/pipelines.png diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/select-template.png b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/select-template.png Binary files differnew file mode 100644 index 00000000000..38bfde0a3dd --- /dev/null +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/select-template.png diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/setup-ci.png b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/setup-ci.png Binary files differnew file mode 100644 index 00000000000..bfe85c6a10b --- /dev/null +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/setup-ci.png diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md new file mode 100644 index 00000000000..7f6519fd38e --- /dev/null +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -0,0 +1,423 @@ +--- +author: Alexandre S Hostert +author_gitlab: Hostert +level: beginner +article_type: tutorial +date: 2018-02-20 +--- + +# Testing a Phoenix application with GitLab CI/CD + +[Phoenix][phoenix-site] is a web development framework written in [Elixir][elixir-site], which is a +functional language designed for productivity and maintainability that runs on the +[Erlang VM][erlang-site]. Erlang VM is really really fast and can handle very large numbers of +simultaneous users. + +That's why we're hearing so much about Phoenix today. + +In this tutorial, we'll teach you how to set up GitLab CI/CD to build and test a Phoenix +application. + +_We assume that you know how to create a Phoenix app, run tests locally, and how to work with Git +and GitLab UI._ + +## Introduction + +### What is Phoenix? + +[Phoenix][phoenix-site] is a web development framework written in [Elixir][elixir-site] very useful + to build fast, reliable, and high-performance applications, as it uses [Erlang VM][erlang-site]. + +Many components and concepts are similar to Ruby on Rails or Python's Django. High developer +productivity and high application performance are only a few advantages on learning how to use it. +Working on the MVC pattern, it's was designed to be modular and flexible. Easy to mantain a growing +app is a plus. + +Phoenix can run in any OS where Erlang is supported: + +- Ubuntu +- CentOS +- Mac OS X +- Debian +- Windows +- Fedora +- Raspbian + +Check the [Phoenix learning guide][phoenix-learning-guide] for more information. + +### What is Elixir? + +[Elixir][elixir-site] is a dynamic, functional language created to use all the maturity of Erlang +(30 years old!) in these days, in an easy way. It has similarities with Ruby, specially on sintax, +so Ruby developers are quite excited with the rapid growing of Elixir. A full-stack Ruby developer +can learn how to use Elixir and Phoenix in just a few weeks! + +In Elixir we have a command called `mix`, which is a helper to create projects, testing, run +migrations and [much more][elixir-mix]. We'll use it later on in this tutorial. + +Check the [Elixir documentation][elixir-docs] for more information. + +## Requirements + +To follow this tutorial, you'll need to have installed: + +- Elixir [installation instructions][elixir-install] +- Phoenix Framework [installation instructions][phoenix-install] +- PostgreSQL (if you need to use MySQL server, check [Phoenix instructions][phoenix-mysql]) + +### Create a new Phoenix project + +Open your terminal and go to the directory you wish to create your project. +You don't need to create an empty directory for the project's files, because the `mix` command will +do it for us. + +When we call `mix` command, we'll pass two arguments: + +- The task we want it to run: `phoenix.new` +- And the parameter `phoenix.new` requires, which is the name of the new project. In this case, +we're calling it `hello_gitlab_ci`, but you're free to set your own name: + +```bash +mix phoenix.new hello_gitlab_ci +``` + +When asked, answer `Y` to fetch and install dependencies. + +If everything went fine, you'll get an output like this: + + + +Now, our project is located inside the directory with the same name we pass to `mix` command, for +example, `~/GitLab/hello_gitlab_ci`. +If we take a look at the directory, we'll see the Phoenix files and the dependencies needed to run. + +### Initialize the PostgreSQL database + +By default, Phoenix requires a PostgreSQL database to store whatever we need to store in our app. In +this case, we'll only create an empty database. + +First, we need to navigate to our recently created project's directory, and then execute again +`mix`. This time, `mix` will receive the parameter `ecto.create`, which is the task to create our +new database. [Ecto][ecto] is the database wrapper for Elixir. + +When we do run `mix` the first time after creating our project, it will compile our files to +bytecode, which will be interpreted by Erlang VM. In the next times, it will only compile our +changes. + +Run the commands below to create our empty database: + +```bash +cd hello_gitlab_ci +mix ecto.create +``` + +We expect to see this output at the end of the command: + +```bash +Generated hello_gitlab_ci app +The database for HelloGitlabCi.Repo has been created +``` + +> **Note:** +Phoenix assumes that our PostgreSQL database will have a `postgres` user account with the correct +permissions and a password of `postgres`. If it's not your case, check +[Ecto's instructions][ecto-repo]. + +### Start Phoenix server + +Now, it's time to see if everything we did until now went well. We'll call `mix` again, this time +with `phoenix.server` parameter, which will start Phoenix's HTTP Server. + +```bash +mix phoenix.server +``` + +This will be the output to this command: + +```bash +[info] Running HelloGitlabCi.Endpoint with Cowboy using http://localhost:4000 +23 May 11:44:35 - info: compiling +23 May 11:44:37 - info: compiled 6 files into 2 files, copied 3 in 9.8 sec +``` + +Now, we have our app running locally. We can preview it directly on our browser. Let's open +[`localhost:4000`](http://localhost:4000) to see our Phoenix Framework welcome page. If the link do +not work, open [`127.0.0.1:4000`](http://127.0.0.1:4000) instead and later, configure your OS to +point `localhost` to `127.0.0.1`. + + + +Great, now we have a local Phoenix Server running our app. + +Locally, our application is running in an `iex` session. [iex][iex] stands for Interactive Elixir. +In this interactive mode, we can type any Elixir expression and get its result. To exit `iex`, we +need to press `Ctrl+C` twice. So, when we need to stop the Phoenix server, we have to hit `Ctrl+C` +twice. + +## Introducing GitLab CI/CD + +With GitLab, we can manage our development workflow, improve our productivity, track issues, +perform code review, and much more from a single platform. With GitLab CI/CD, we can be much more +productive, because every time we, or our co-workers push any code, GitLab CI/CD will build and +test the changes, telling us in realtime if anything goes wrong. + +Certainly, when our application starts to grow, we'll need more developers working on the same +project and this process of building and testing can easely become a mess without proper management. +That's also why GitLab CI/CD is so important to our application. Every time someone pushes its code to +GitLab, we'll quickly know if their changes broke something or not. We don't need to stop everything +we're doing to test manually and locally every change our team does. + +Let's see this in practice. + +## Adjusting Phoenix configuration + +Now, we need to adjust our Phoenix configuration before configuring GitLab CI/CD. +There is a directory (`config`) in your Phoenix project that contains a configuration file for every +environment it can run. Since we will work with a single environment, we'll edit just the test +configuration file (`test.exs`). + +But, why do we need to adjust our configuration? Well, GitLab CI/CD builds and tests our code in one +isolated virtual machine, called [Runner][runner-site], using Docker technology. In this Runner, +GitLab CI/CD has access to everything our Phoenix application need to run, exactly as we have in our +`localhost`, but we have to tell GitLab CI/CD where to create and find this database using system +variables. This way, GitLab CI/CD will create our test database inside the Runner, just like we do +when running our Phoenix in our `localhost`. + +- Open `hello_gitlab_ci/config/test.exs` on your favorite code editor +- Go to **Configure your database** session and edit the block to include `System.get_env`: + + ```elixir + # Configure your database + config :hello_gitlab_ci, HelloGitlabCi.Repo, + adapter: Ecto.Adapters.Postgres, + username: System.get_env("POSTGRES_USER") || "postgres", + password: System.get_env("POSTGRES_PASSWORD") || "postgres", + database: System.get_env("POSTGRES_DB") || "hello_gitlab_ci_test", + hostname: System.get_env("POSTGRES_HOST") || "localhost", + pool: Ecto.Adapters.SQL.Sandbox + ``` + + We'll need these system variables later on. + +- Create an empty file named `.gitkeep` into `hello_gitlab_ci/priv/repo/migrations` + + As our project is still fresh, we don't have any data on our database, so, the `migrations` +directory will be empty. + Without `.gitkeep`, git will not upload this empty directory and we'll got an error when running our +test on GitLab. + + > **Note:** + If we add a folder via the GitLab UI, GitLab itself will add the `.gitkeep` to that new dir. + +Now, let's run a local test and see if everything we did didn't break anything. + +## Testing + +Earlier, when we created our project, we ran `mix phoenix.new`. +This task created everything a Phoenix application needed, including some unit tests into +`hello_gitlab_ci/test` directory. + +Let's run a new task with `mix` to run those tests for us. This time, the parameter expected is +`test`. We can add `--trace` parameter for debugging purposes. + +In your terminal, navigate to the directory `hello_gitlab_ci` and run: + +```bash +mix test +``` + +Our expected result is this: + +```bash +.... + +Finished in 0.7 seconds +4 tests, 0 failures + +Randomized with seed 610000 +``` + +Our test was successfull. It's time to push our files to GitLab. + +## Configuring CI/CD Pipeline + +The first step is to create a new file called `.gitlab-ci.yml` in `hello_gitlab_ci` directory of our +project. + +- The fastest and easiest way to do this, is to click on **Set up CI** on project's main page: + +  + +- On next screen, we can select a template ready to go. Click on **Apply a GitLab CI/CD Yaml +template** and select **Elixir**: + +  + + This template file tells GitLab CI/CD about what we wish to do every time a new commit is made. + However, we have to adapt it to run a Phoenix app. + +- The first line tells GitLab what Docker image will be used. + + Remember when we learn about Runners, the isolated virtual machine where GitLab CI/CD build and test + our application? This virtual machine must have all dependencies to run our application. This is + where a Docker image is needed. The correct image will provide the entire system for us. + + As a suggestion, you can use [trenpixster's elixir image][docker-image], which already has all + dependencies for Phoenix installed, such as Elixir, Erlang, NodeJS and PostgreSQL: + + ```yml + image: trenpixster/elixir:latest + ``` + +- At `services` session, we'll only use `postgres`, so we'll delete `mysql` and `redis` lines: + + ```yml + services: + - postgres:latest + ``` + +- Now, we'll create a new entry called `variables`, before `before_script` session: + + ```yml + variables: + POSTGRES_DB: hello_gitlab_ci_test + POSTGRES_HOST: postgres + POSTGRES_USER: postgres + POSTGRES_PASSWORD: "postgres" + MIX_ENV: "test" + ``` + + Here, we are setting up the values for GitLab CI/CD authenticate into PostgreSQL, as we did on + `config/test.exs` earlier. + +- In `before_script` session, we'll add some commands to prepare everything to the test: + + ```yml + before_script: + - apt-get update && apt-get -y install postgresql-client + - mix local.hex --force + - mix deps.get --only test + - mix ecto.create + - mix ecto.migrate + ``` + + It's important to install `postgresql-client` to let GitLab CI/CD access PostgreSQL and create our + database with the login information provided earlier. More important is to respect the identation, + to avoid syntax errors when running the build. + +- And finally, we'll let `mix` session intact. + +Let's take a look at the completed file after the editions: + +```yml +image: trenpixster/elixir:latest + +services: + - postgres:latest + +variables: + POSTGRES_DB: test_test + POSTGRES_HOST: postgres + POSTGRES_USER: postgres + POSTGRES_PASSWORD: "postgres" + MIX_ENV: "test" + +before_script: + - apt-get update && apt-get -y install postgresql-client + - mix deps.get + - mix ecto.create + - mix ecto.migrate + +mix: + script: + - mix test +``` + +For safety, we can check if we get any syntax errors before submiting this file to GitLab. Copy the +contents of `.gitlab-ci.yml` and paste it on [GitLab CI/CD Lint tool][ci-lint]. Please note that +this link will only work for logged in users. + +## Watching the build + +I don't know about you, but I love to watch that black screen being filled with compilation output. +With this, I can feel the happiness of something I made working correctly. On `localhost` it's easy +to watch our build, but on GitLab, is it possible? Yes! + +Let's go to **Pipelines** and see GitLab doing the job. Just click on **Pipelines** to find the +actual running build job. + + + +Click on build's ID to watch the entire process. If everything went as expected, we can wait for the +**Build succeeded** at the end of the process! :) + +``` +$ mix test +.... + +Finished in 0.3 seconds +4 tests, 0 failures + +Randomized with seed 206909 +Build succeeded +``` + +If we take a look at the project's main page on the GitLab UI, we can see the status of the last +build made by GitLab CI/CD. + +Time to show the world our green build badge! Navigate to your project's **Settings > CI/CD** and +expand **General pipelines settings**. Scroll down to **Pipeline status** and copy the markdown code +for your badge. Paste it on the top of your `README.md` file, to let people outside of our project +see if our latest code is running without errors. + +When we finish this edition, GitLab will start another build and show a **build running** badge. It +is expected, after all we just configured GitLab CI/CD to do this for every push! But you may think +"Why run build and tests for simple things like editing README.md?" and it is a good question. +For changes that don't affect your application, you can add the keyword [`[ci skip]`][skipping-jobs] +to commit message and the build related to that commit will be skipped. + +In the end, we finally got our pretty green build succeeded badge! By outputting the result on the +README file, it shows to whoever lands on your project's page that your code is up-to-date and +working properly. + +## Conclusion + +When we have a growing application with many developers working on it, or when we have an open +source project being watched and contributed by the community, it is really important to have our +code permanently working. GitLab CI/CD is a time saving powerfull tool to help us mantain our code +organized and working. + +As we could see in this post, GitLab CI/CD is really really easy to configure and use. We have [many +other reasons][ci-reasons] to keep using GitLab CI/CD. The benefits to our teams will be huge! + +## References + +- [GitLab CI/CD introductory guide][ci-guide] +- [GitLab CI/CD full Documentation][ci-docs] +- [GitLab Runners documentation][gitlab-runners] +- [Using Docker images documentation][using-docker] +- [Example project: Hello GitLab CI/CD on GitLab][hello-gitlab] + +[phoenix-site]: http://phoenixframework.org/ "Phoenix Framework" +[phoenix-learning-guide]: https://hexdocs.pm/phoenix/learning.html "Phoenix Learning Guide" +[phoenix-install]: http://www.phoenixframework.org/docs/installation "Phoenix Installation" +[phoenix-mysql]: http://www.phoenixframework.org/docs/using-mysql "Phoenix with MySQL" +[elixir-site]: http://elixir-lang.org/ "Elixir" +[elixir-mix]: http://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html "Introduction to mix" +[elixir-docs]: http://elixir-lang.org/getting-started/introduction.html "Elixir Documentation" +[erlang-site]: http://erlang.org "Erlang" +[elixir-install]: https://elixir-lang.org/install.html "Elixir Installation" +[ecto]: http://hexdocs.pm/ecto "Ecto" +[ecto-repo]: https://hexdocs.pm/ecto/Ecto.html#module-repositories "Ecto Repositories" +[mix-ecto]: https://hexdocs.pm/ecto/Mix.Tasks.Ecto.Create.html "mix and Ecto" +[iex]: http://elixir-lang.org/getting-started/introduction.html#interactive-mode "Interactive Mode" +[ci-lint]: https://gitlab.com/ci/lint "CI Lint Tool" +[ci-reasons]: https://about.gitlab.com/2015/02/03/7-reasons-why-you-should-be-using-ci/ "7 Reasons Why You Should Be Using CI" +[ci-guide]: https://about.gitlab.com/2015/12/14/getting-started-with-gitlab-and-gitlab-ci/ "Getting Started With GitLab And GitLab CI/CD" +[ci-docs]: ../../README.md "GitLab CI/CD Documentation" +[skipping-jobs]: ../../yaml/README.md#skipping-jobs "Skipping Jobs" +[gitlab-runners]: ../../runners/README.md "GitLab Runners Documentation" +[runner-site]: ../../runners/README.md#runners "Runners" +[docker-image]: https://hub.docker.com/r/trenpixster/elixir/ "Elixir Docker Image" +[using-docker]: ../../docker/using_docker_images.md "Using Docker Images" +[hello-gitlab]: https://gitlab.com/Hostert/hello_gitlab_ci "Hello GitLab CI/CD" diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index ac4a9b0ed27..856d7f264e4 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -121,8 +121,9 @@ The basic requirements is that there are two numbers separated with one of the following (you can even use them interchangeably): - a space -- a backslash (`/`) +- a forward slash (`/`) - a colon (`:`) +- a dot (`.`) >**Note:** More specifically, [it uses][regexp] this regular expression: `\d+[\s:\/\\]+\d+\s*`. diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index e504b81eae8..f64e868d390 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -104,8 +104,8 @@ Jobs are used to create jobs, which are then picked by What is important is that each job is run independently from each other. -If you want to check whether your `.gitlab-ci.yml` file is valid, there is a -Lint tool under the page `/ci/lint` of your GitLab instance. You can also find +If you want to check whether the `.gitlab-ci.yml` of your project is valid, there is a +Lint tool under the page `/ci/lint` of your project namespace. You can also find a "CI Lint" button to go to this page under **CI/CD ➔ Pipelines** and **Pipelines ➔ Jobs** in your project. diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 03aa6ff8e7c..60dc2ef9ac5 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -35,7 +35,7 @@ are: A Runner that is specific only runs for the specified project(s). A shared Runner can run jobs for every project that has enabled the option **Allow shared Runners** -under **Settings ➔ CI/CD**. +under **Settings > CI/CD**. Projects with high demand of CI activity can also benefit from using specific Runners. By having dedicated Runners you are guaranteed that the Runner is not @@ -76,7 +76,7 @@ Registering a specific can be done in two ways: To create a specific Runner without having admin rights to the GitLab instance, visit the project you want to make the Runner work for in GitLab: -1. Go to **Settings ➔ CI/CD** to obtain the token +1. Go to **Settings > CI/CD** to obtain the token 1. [Register the Runner][register] ### Making an existing shared Runner specific @@ -85,7 +85,7 @@ If you are an admin on your GitLab instance, you can turn any shared Runner into a specific one, but not the other way around. Keep in mind that this is a one way transition. -1. Go to the Runners in the admin area **Overview ➔ Runners** (`/admin/runners`) +1. Go to the Runners in the admin area **Overview > Runners** (`/admin/runners`) and find your Runner 1. Enable any projects under **Restrict projects for this Runner** to be used with the Runner @@ -101,7 +101,7 @@ can be changed afterwards under each Runner's settings. To lock/unlock a Runner: -1. Visit your project's **Settings ➔ CI/CD** +1. Visit your project's **Settings > CI/CD** 1. Find the Runner you wish to lock/unlock and make sure it's enabled 1. Click the pencil button 1. Check the **Lock to current projects** option @@ -115,7 +115,7 @@ you can enable the Runner also on any other project where you have Master permis To enable/disable a Runner in your project: -1. Visit your project's **Settings ➔ CI/CD** +1. Visit your project's **Settings > CI/CD** 1. Find the Runner you wish to enable/disable 1. Click **Enable for this project** or **Disable for this project** @@ -124,6 +124,13 @@ Consider that if you don't lock your specific Runner to a specific project, any user with Master role in you project can assign your runner to another arbitrary project without requiring your authorization, so use it with caution. +An admin can enable/disable a specific Runner for projects: + +1. Navigate to **Admin > Runners** +2. Find the Runner you wish to enable/disable +3. Click edit on the Runner +4. Click **Enable** or **Disable** on the project + ## Protected Runners > @@ -136,7 +143,7 @@ Whenever a Runner is protected, the Runner picks only jobs created on To protect/unprotect Runners: -1. Visit your project's **Settings ➔ CI/CD** +1. Visit your project's **Settings > CI/CD** 1. Find a Runner you want to protect/unprotect and make sure it's enabled 1. Click the pencil button besides the Runner name 1. Check the **Protected** option @@ -146,25 +153,7 @@ To protect/unprotect Runners: ## Manually clearing the Runners cache -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41249) in GitLab 10.4. - -GitLab Runners use [cache](../yaml/README.md#cache) to speed up the execution -of your jobs by reusing existing data. This however, can sometimes lead to an -inconsistent behavior. - -To start with a fresh copy of the cache, you can easily do it via GitLab's UI: - -1. Navigate to your project's **CI/CD > Pipelines** page. -1. Click on the **Clear Runner caches** to clean up the cache. -1. On the next push, your CI/CD job will use a new cache. - -That way, you don't have to change the [cache key](../yaml/README.md#cache-key) -in your `.gitlab-ci.yml`. - -Behind the scenes, this works by increasing a counter in the database, and the -value of that counter is used to create the key for the cache. After a push, a -new key is generated and the old cache is not valid anymore. Eventually, the -Runner's garbage collector will remove it form the filesystem. +Read [clearing the cache](../caching/index.md#clearing-the-cache). ## How shared Runners pick jobs @@ -228,15 +217,16 @@ that it may encounter on the projects it's shared over. This would be problematic for large amounts of projects, if it wasn't for tags. By tagging a Runner for the types of jobs it can handle, you can make sure -shared Runners will only run the jobs they are equipped to run. +shared Runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). For instance, at GitLab we have Runners tagged with "rails" if they contain the appropriate dependencies to run Rails test suites. ### Preventing Runners with tags from picking jobs without tags -You can configure a Runner to prevent it from picking jobs with tags when -the Runner does not have tags assigned. This setting can be enabled the first +You can configure a Runner to prevent it from picking +[jobs with tags](../yaml/README.md#tags) when the Runner does not have tags +assigned. This setting can be enabled the first time you [register a Runner][register] and can be changed afterwards under each Runner's settings. @@ -248,6 +238,38 @@ To make a Runner pick tagged/untagged jobs: 1. Check the **Run untagged jobs** option 1. Click **Save changes** for the changes to take effect +### Setting maximum job timeout for a Runner + +For each Runner you can specify a _maximum job timeout_. Such timeout, +if smaller than [project defined timeout], will take the precedence. This +feature can be used to prevent Shared Runner from being appropriated +by a project by setting a ridiculous big timeout (e.g. one week). + +When not configured, Runner will not override project timeout. + +How this feature will work: + +**Example 1 - Runner timeout bigger than project timeout** + +1. You set the _maximum job timeout_ for a Runner to 24 hours +1. You set the _CI/CD Timeout_ for a project to **2 hours** +1. You start a job +1. The job, if running longer, will be timeouted after **2 hours** + +**Example 2 - Runner timeout not configured** + +1. You remove the _maximum job timeout_ configuration from a Runner +1. You set the _CI/CD Timeout_ for a project to **2 hours** +1. You start a job +1. The job, if running longer, will be timeouted after **2 hours** + +**Example 3 - Runner timeout smaller than project timeout** + +1. You set the _maximum job timeout_ for a Runner to **30 minutes** +1. You set the _CI/CD Timeout_ for a project to 2 hours +1. You start a job +1. The job, if running longer, will be timeouted after **30 minutes** + ### Be careful with sensitive information With some [Runner Executors](https://docs.gitlab.com/runner/executors/README.html), @@ -276,8 +298,42 @@ Mentioned briefly earlier, but the following things of Runners can be exploited. We're always looking for contributions that can mitigate these [Security Considerations](https://docs.gitlab.com/runner/security/). +## Determining the IP address of a Runner + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17286) in GitLab 10.6. + +It may be useful to know the IP address of a Runner so you can troubleshoot +issues with that Runner. GitLab stores and displays the IP address by viewing +the source of the HTTP requests it makes to GitLab when polling for jobs. The +IP address is always kept up to date so if the Runner IP changes it will be +automatically updated in GitLab. + +The IP address for shared Runners and specific Runners can be found in +different places. + +### Shared Runners + +To view the IP address of a shared Runner you must have admin access to +the GitLab instance. To determine this: + +1. Visit **Admin area ➔ Overview ➔ Runners** +1. Look for the Runner in the table and you should see a column for "IP Address" + + + +### Specific Runners + +You can find the IP address of a Runner for a specific project by: + +1. Visit your project's **Settings ➔ CI/CD** +1. Find the Runner and click on it's ID which links you to the details page +1. On the details page you should see a row for "IP Address" + + + [install]: http://docs.gitlab.com/runner/install/ [fifo]: https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics) [register]: http://docs.gitlab.com/runner/register/ [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md +[project defined timeout]: ../../user/project/pipelines/settings.html#timeout diff --git a/doc/ci/runners/img/shared_runner_ip_address.png b/doc/ci/runners/img/shared_runner_ip_address.png Binary files differnew file mode 100644 index 00000000000..3b1542d59d3 --- /dev/null +++ b/doc/ci/runners/img/shared_runner_ip_address.png diff --git a/doc/ci/runners/img/specific_runner_ip_address.png b/doc/ci/runners/img/specific_runner_ip_address.png Binary files differnew file mode 100644 index 00000000000..3b4c3e9f2eb --- /dev/null +++ b/doc/ci/runners/img/specific_runner_ip_address.png diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 56a16f77e7f..47a576fdf5f 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -219,7 +219,7 @@ removed with one of the future versions of GitLab. You are advised to [ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017 [ci-229]: https://gitlab.com/gitlab-org/gitlab-ci/merge_requests/229 -[ee]: https://about.gitlab.com/gitlab-ee/ +[ee]: https://about.gitlab.com/products/ [variables]: ../variables/README.md [predef]: ../variables/README.md#predefined-variables-environment-variables [registry]: ../../user/project/container_registry.md diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 598a7515b01..bd4aeb006bd 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -12,7 +12,7 @@ this order: 1. [Trigger variables][triggers] or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#making-use-of-scheduled-pipeline-variables) (take precedence over all) 1. Project-level [secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) 1. Group-level [secret variables](#secret-variables) or [protected secret variables](#protected-secret-variables) -1. YAML-defined [job-level variables](../yaml/README.md#job-variables) +1. YAML-defined [job-level variables](../yaml/README.md#variables) 1. YAML-defined [global variables](../yaml/README.md#variables) 1. [Deployment variables](#deployment-variables) 1. [Predefined variables](#predefined-variables-environment-variables) (are the @@ -56,6 +56,9 @@ future GitLab releases.** | **CI_RUNNER_DESCRIPTION** | 8.10 | 0.5 | The description of the runner as saved in GitLab | | **CI_RUNNER_ID** | 8.10 | 0.5 | The unique id of runner being used | | **CI_RUNNER_TAGS** | 8.10 | 0.5 | The defined runner tags | +| **CI_RUNNER_VERSION** | all | 10.6 | GitLab Runner version that is executing the current job | +| **CI_RUNNER_REVISION** | all | 10.6 | GitLab Runner revision that is executing the current job | +| **CI_RUNNER_EXECUTABLE_ARCH** | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | | **CI_PIPELINE_ID** | 8.10 | 0.5 | The unique id of the current pipeline that GitLab CI uses internally | | **CI_PIPELINE_TRIGGERED** | all | all | The flag to indicate that job was [triggered] | | **CI_PIPELINE_SOURCE** | 10.0 | all | The source for this pipeline, one of: push, web, trigger, schedule, api, external. Pipelines created before 9.5 will have unknown as source | @@ -447,7 +450,7 @@ export CI_REGISTRY_PASSWORD="longalfanumstring" ``` [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI secret variables" -[eep]: https://about.gitlab.com/gitlab-ee/ "Available only in GitLab Enterprise Edition Premium" +[eep]: https://about.gitlab.com/products/ "Available only in GitLab Premium" [envs]: ../environments.md [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md diff --git a/doc/ci/variables/img/secret_variables.png b/doc/ci/variables/img/secret_variables.png Binary files differindex f70935069d9..3c1aa361dc2 100644 --- a/doc/ci/variables/img/secret_variables.png +++ b/doc/ci/variables/img/secret_variables.png diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 4a650303d45..c2b06e53c2f 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -3,18 +3,19 @@ This document describes the usage of `.gitlab-ci.yml`, the file that is used by GitLab Runner to manage your project's jobs. -If you want a quick introduction to GitLab CI, follow our -[quick start guide](../quick_start/README.md). - -## .gitlab-ci.yml - From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (`.gitlab-ci.yml`) for the project configuration. It is placed in the root of your repository and contains definitions of how your project should be built. +If you want a quick introduction to GitLab CI, follow our +[quick start guide](../quick_start/README.md). + +## Jobs + The YAML file defines a set of jobs with constraints stating when they should -be run. The jobs are defined as top-level elements with a name and always have -to contain at least the `script` clause: +be run. You can specify an unlimited number of jobs which are defined as +top-level elements with an arbitrary name and always have to contain at least +the `script` clause. ```yaml job1: @@ -24,9 +25,8 @@ job2: script: "execute-script-for-job2" ``` -The above example is the simplest possible CI configuration with two separate +The above example is the simplest possible CI/CD configuration with two separate jobs, where each of the jobs executes a different command. - Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. @@ -34,78 +34,115 @@ Jobs are picked up by [Runners](../runners/README.md) and executed within the environment of the Runner. What is important, is that each job is run independently from each other. -The YAML syntax allows for using more complex job specifications than in the -above example: +Each job must have a unique name, but there are a few **reserved `keywords` that +cannot be used as job names**: -```yaml -image: ruby:2.1 -services: - - postgres +- `image` +- `services` +- `stages` +- `types` +- `before_script` +- `after_script` +- `variables` +- `cache` -before_script: - - bundle install +A job is defined by a list of parameters that define the job behavior. -after_script: - - rm secrets +| Keyword | Required | Description | +|---------------|----------|-------------| +| script | yes | Defines a shell script which is executed by Runner | +| image | no | Use docker image, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | +| services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | +| stage | no | Defines a job stage (default: `test`) | +| type | no | Alias for `stage` | +| variables | no | Define job variables on a job level | +| only | no | Defines a list of git refs for which job is created | +| except | no | Defines a list of git refs for which job is not created | +| tags | no | Defines a list of tags which are used to select Runner | +| allow_failure | no | Allow job to fail. Failed job doesn't contribute to commit status | +| when | no | Define when to run job. Can be `on_success`, `on_failure`, `always` or `manual` | +| dependencies | no | Define other jobs that a job depends on so that you can pass artifacts between them| +| artifacts | no | Define list of [job artifacts](#artifacts) | +| cache | no | Define list of files that should be cached between subsequent runs | +| before_script | no | Override a set of commands that are executed before job | +| after_script | no | Override a set of commands that are executed after job | +| environment | no | Defines a name of environment to which deployment is done by this job | +| coverage | no | Define code coverage settings for a given job | +| retry | no | Define how many times a job can be auto-retried in case of a failure | -stages: - - build - - test - - deploy +### `pages` -job1: - stage: build +`pages` is a special job that is used to upload static content to GitLab that +can be used to serve your website. It has a special syntax, so the two +requirements below must be met: + +1. Any static content must be placed under a `public/` directory +1. `artifacts` with a path to the `public/` directory must be defined + +The example below simply moves all files from the root of the project to the +`public/` directory. The `.public` workaround is so `cp` doesn't also copy +`public/` to itself in an infinite loop: + +``` +pages: + stage: deploy script: - - execute-script-for-job1 + - mkdir .public + - cp -r * .public + - mv .public public + artifacts: + paths: + - public only: - - master - tags: - - docker + - master ``` -There are a few reserved `keywords` that **cannot** be used as job names: - -| Keyword | Required | Description | -|---------------|----------|-------------| -| image | no | Use docker image, covered in [Use Docker](../docker/README.md) | -| services | no | Use docker services, covered in [Use Docker](../docker/README.md) | -| stages | no | Define build stages | -| types | no | Alias for `stages` (deprecated) | -| before_script | no | Define commands that run before each job's script | -| after_script | no | Define commands that run after each job's script | -| variables | no | Define build variables | -| cache | no | Define list of files that should be cached between subsequent runs | +Read more on [GitLab Pages user documentation](../../user/project/pages/index.md). -### image and services +## `image` and `services` This allows to specify a custom Docker image and a list of services that can be used for time of the job. The configuration of this feature is covered in [a separate document](../docker/README.md). -### before_script - -`before_script` is used to define the command that should be run before all -jobs, including deploy jobs, but after the restoration of artifacts. This can -be an array or a multi-line string. - -### after_script +## `before_script` and `after_script` > Introduced in GitLab 8.7 and requires Gitlab Runner v1.2 +`before_script` is used to define the command that should be run before all +jobs, including deploy jobs, but after the restoration of [artifacts](#artifacts). +This can be an array or a multi-line string. + `after_script` is used to define the command that will be run after for all jobs, including failed ones. This has to be an array or a multi-line string. -> **Note:** The `before_script` and the main `script` are concatenated and run in a single context/container. The `after_script` is run separately, so depending on the executor, changes done outside of the working tree might not be visible, e.g. software installed in the `before_script`. -### stages +It's possible to overwrite the globally defined `before_script` and `after_script` +if you set it per-job: -`stages` is used to define stages that can be used by jobs. -The specification of `stages` allows for having flexible multi stage pipelines. +```yaml +before_script: +- global before script + +job: + before_script: + - execute this instead of global before script + script: + - my command + after_script: + - execute this after my script +``` + +## `stages` +`stages` is used to define stages that can be used by jobs and is defined +globally. + +The specification of `stages` allows for having flexible multi stage pipelines. The ordering of elements in `stages` defines the ordering of jobs' execution: 1. Jobs of the same stage are run in parallel. @@ -134,280 +171,45 @@ There are also two edge cases worth mentioning: `test` and `deploy` are allowed to be used as job's stage by default. 2. If a job doesn't specify a `stage`, the job is assigned the `test` stage. -### types - -> Deprecated, and could be removed in one of the future releases. Use [stages](#stages) instead. - -Alias for [stages](#stages). - -### variables - -> Introduced in GitLab Runner v0.5.0. - -GitLab CI allows you to add variables to `.gitlab-ci.yml` that are set in the -job environment. The variables are stored in the Git repository and are meant -to store non-sensitive project configuration, for example: - -```yaml -variables: - DATABASE_URL: "postgres://postgres@postgres/my_database" -``` - ->**Note:** -Integers (as well as strings) are legal both for variable's name and value. -Floats are not legal and cannot be used. - -These variables can be later used in all executed commands and scripts. -The YAML-defined variables are also set to all created service containers, -thus allowing to fine tune them. Variables can be also defined on a -[job level](#job-variables). - -Except for the user defined variables, there are also the ones set up by the -Runner itself. One example would be `CI_COMMIT_REF_NAME` which has the value of -the branch or tag name for which project is built. Apart from the variables -you can set in `.gitlab-ci.yml`, there are also the so called secret variables -which can be set in GitLab's UI. - -[Learn more about variables.][variables] - -### cache - -> -**Notes:** -- Introduced in GitLab Runner v0.7.0. -- Prior to GitLab 9.2, caches were restored after artifacts. -- From GitLab 9.2, caches are restored before artifacts. - -`cache` is used to specify a list of files and directories which should be -cached between jobs. You can only use paths that are within the project -workspace. - -**By default caching is enabled and shared between pipelines and jobs, -starting from GitLab 9.0** - -If `cache` is defined outside the scope of jobs, it means it is set -globally and all jobs will use that definition. - -Cache all files in `binaries` and `.config`: - -```yaml -rspec: - script: test - cache: - paths: - - binaries/ - - .config -``` - -Cache all Git untracked files: - -```yaml -rspec: - script: test - cache: - untracked: true -``` - -Cache all Git untracked files and files in `binaries`: - -```yaml -rspec: - script: test - cache: - untracked: true - paths: - - binaries/ -``` - -Locally defined cache overrides globally defined options. The following `rspec` -job will cache only `binaries/`: - -```yaml -cache: - paths: - - my/files - -rspec: - script: test - cache: - key: rspec - paths: - - binaries/ -``` - -Note that since cache is shared between jobs, if you're using different -paths for different jobs, you should also set a different **cache:key** -otherwise cache content can be overwritten. - -The cache is provided on a best-effort basis, so don't expect that the cache -will be always present. For implementation details, please check GitLab Runner. - -#### cache:key - -> Introduced in GitLab Runner v1.0.0. - -The `key` directive allows you to define the affinity of caching -between jobs, allowing to have a single cache for all jobs, -cache per-job, cache per-branch or any other way you deem proper. - -This allows you to fine tune caching, allowing you to cache data between -different jobs or even different branches. - -The `cache:key` variable can use any of the [predefined variables](../variables/README.md). - -The default key is **default** across the project, therefore everything is -shared between each pipelines and jobs by default, starting from GitLab 9.0. - ->**Note:** The `cache:key` variable cannot contain the `/` character, or the equivalent URI encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden. - ---- - -**Example configurations** - -To enable per-job caching: - -```yaml -cache: - key: "$CI_JOB_NAME" - untracked: true -``` - -To enable per-branch caching: - -```yaml -cache: - key: "$CI_COMMIT_REF_NAME" - untracked: true -``` - -To enable per-job and per-branch caching: - -```yaml -cache: - key: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" - untracked: true -``` - -To enable per-branch and per-stage caching: - -```yaml -cache: - key: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" - untracked: true -``` - -If you use **Windows Batch** to run your shell scripts you need to replace -`$` with `%`: - -```yaml -cache: - key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" - untracked: true -``` - -If you use **Windows PowerShell** to run your shell scripts you need to replace -`$` with `$env:`: - -```yaml -cache: - key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" - untracked: true -``` - -### cache:policy - -> Introduced in GitLab 9.4. - -The default behaviour of a caching job is to download the files at the start of -execution, and to re-upload them at the end. This allows any changes made by the -job to be persisted for future runs, and is known as the `pull-push` cache -policy. +## `stage` -If you know the job doesn't alter the cached files, you can skip the upload step -by setting `policy: pull` in the job specification. Typically, this would be -twinned with an ordinary cache job at an earlier stage to ensure the cache -is updated from time to time: +`stage` is defined per-job and relies on [`stages`](#stages) which is defined +globally. It allows to group jobs into different stages, and jobs of the same +`stage` are executed in `parallel`. For example: ```yaml stages: - - setup + - build - test + - deploy -prepare: - stage: setup - cache: - key: gems - paths: - - vendor/bundle - script: - - bundle install --deployment - -rspec: - stage: test - cache: - key: gems - paths: - - vendor/bundle - policy: pull - script: - - bundle exec rspec ... -``` - -This helps to speed up job execution and reduce load on the cache server, -especially when you have a large number of cache-using jobs executing in -parallel. - -Additionally, if you have a job that unconditionally recreates the cache without -reference to its previous contents, you can use `policy: push` in that job to -skip the download step. - -## Jobs +job 1: + stage: build + script: make build dependencies -`.gitlab-ci.yml` allows you to specify an unlimited number of jobs. Each job -must have a unique name, which is not one of the keywords mentioned above. -A job is defined by a list of parameters that define the job behavior. +job 2: + stage: build + script: make build artifacts -```yaml -job_name: - script: - - rake spec - - coverage +job 3: stage: test - only: - - master - except: - - develop - tags: - - ruby - - postgres - allow_failure: true + script: make test + +job 4: + stage: deploy + script: make deploy ``` -| Keyword | Required | Description | -|---------------|----------|-------------| -| script | yes | Defines a shell script which is executed by Runner | -| image | no | Use docker image, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | -| services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | -| stage | no | Defines a job stage (default: `test`) | -| type | no | Alias for `stage` | -| variables | no | Define job variables on a job level | -| only | no | Defines a list of git refs for which job is created | -| except | no | Defines a list of git refs for which job is not created | -| tags | no | Defines a list of tags which are used to select Runner | -| allow_failure | no | Allow job to fail. Failed job doesn't contribute to commit status | -| when | no | Define when to run job. Can be `on_success`, `on_failure`, `always` or `manual` | -| dependencies | no | Define other jobs that a job depends on so that you can pass artifacts between them| -| artifacts | no | Define list of [job artifacts](../../user/project/pipelines/job_artifacts.md) | -| cache | no | Define list of files that should be cached between subsequent runs | -| before_script | no | Override a set of commands that are executed before job | -| after_script | no | Override a set of commands that are executed after job | -| environment | no | Defines a name of environment to which deployment is done by this job | -| coverage | no | Define code coverage settings for a given job | -| retry | no | Define how many times a job can be auto-retried in case of a failure | +## `types` + +CAUTION: **Deprecated:** +`types` is deprecated, and could be removed in one of the future releases. +Use [stages](#stages) instead. -### script +## `script` -`script` is a shell script which is executed by the Runner. For example: +`script` is the only required keyword that a job needs. It's a shell script +which is executed by the Runner. For example: ```yaml job: @@ -429,13 +231,7 @@ that the YAML parser knows to interpret the whole thing as a string rather than a "key: value" pair. Be careful when using special characters: `:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. -### stage - -`stage` allows to group jobs into different stages. Jobs of the same `stage` -are executed in `parallel`. For more info about the use of `stage` please check -[stages](#stages). - -### only and except (simplified) +## `only` and `except` (simplified) `only` and `except` are two parameters that set a job policy to limit when jobs are created: @@ -505,12 +301,13 @@ job: The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, except master. -### only and except (complex) +## `only` and `except` (complex) > Introduced in GitLab 10.0 -> This an _alpha_ feature, and it it subject to change at any time without - prior notice! +CAUTION: **Warning:** +This an _alpha_ feature, and it it subject to change at any time without +prior notice! Since GitLab 10.0 it is possible to define a more elaborate only/except job policy configuration. @@ -535,24 +332,7 @@ job: kubernetes: active ``` -### Job variables - -It is possible to define job variables using a `variables` keyword on a job -level. It works basically the same way as its [global-level equivalent](#variables), -but allows you to define job-specific variables. - -When the `variables` keyword is used on a job level, it overrides the global YAML -job variables and predefined ones. To turn off global defined variables -in your job, define an empty hash: - -```yaml -job_name: - variables: {} -``` - -Job variables priority is defined in the [variables documentation][variables]. - -### tags +## `tags` `tags` is used to select specific Runners from the list of all Runners that are allowed to run this project. @@ -573,7 +353,7 @@ job: The specification above, will make sure that `job` is built by a Runner that has both `ruby` AND `postgres` tags defined. -### allow_failure +## `allow_failure` `allow_failure` is used when you want to allow a job to fail without impacting the rest of the CI suite. Failed jobs don't contribute to the commit status. @@ -606,7 +386,7 @@ job3: - deploy_to_staging ``` -### when +## `when` `when` is used to implement jobs that are run in case of failure or despite the failure. @@ -619,7 +399,7 @@ failure. fails. 1. `always` - execute job regardless of the status of jobs from prior stages. 1. `manual` - execute job manually (added in GitLab 8.10). Read about - [manual actions](#manual-actions) below. + [manual actions](#when-manual) below. For example: @@ -667,42 +447,41 @@ The above script will: success or failure. 3. Allow you to manually execute `deploy_job` from GitLab's UI. -#### Manual actions - -> Introduced in GitLab 8.10. -> Blocking manual actions were introduced in GitLab 9.0 -> Protected actions were introduced in GitLab 9.2 +### `when:manual` -Manual actions are a special type of job that are not executed automatically; -they need to be explicitly started by a user. Manual actions can be started -from pipeline, build, environment, and deployment views. +> **Notes:** +- Introduced in GitLab 8.10. +- Blocking manual actions were introduced in GitLab 9.0. +- Protected actions were introduced in GitLab 9.2. -An example usage of manual actions is deployment to production. +Manual actions are a special type of job that are not executed automatically, +they need to be explicitly started by a user. An example usage of manual actions +would be a deployment to a production environment. Manual actions can be started +from the pipeline, job, environment, and deployment views. Read more at the +[environments documentation][env-manual]. -Read more at the [environments documentation][env-manual]. - -Manual actions can be either optional or blocking. Blocking manual action will -block execution of the pipeline at stage this action is defined in. It is +Manual actions can be either optional or blocking. Blocking manual actions will +block the execution of the pipeline at the stage this action is defined in. It's possible to resume execution of the pipeline when someone executes a blocking -manual actions by clicking a _play_ button. +manual action by clicking a _play_ button. -When pipeline is blocked it will not be merged if Merge When Pipeline Succeeds +When a pipeline is blocked, it will not be merged if Merge When Pipeline Succeeds is set. Blocked pipelines also do have a special status, called _manual_. - Manual actions are non-blocking by default. If you want to make manual action blocking, it is necessary to add `allow_failure: false` to the job's definition in `.gitlab-ci.yml`. -Optional manual actions have `allow_failure: true` set by default. +Optional manual actions have `allow_failure: true` set by default and their +Statuses do not contribute to the overall pipeline status. So, if a manual +action fails, the pipeline will eventually succeed. -**Statuses of optional actions do not contribute to overall pipeline status.** +Manual actions are considered to be write actions, so permissions for +[protected branches](../../user/project/protected_branches.md) are used when +user wants to trigger an action. In other words, in order to trigger a manual +action assigned to a branch that the pipeline is running for, user needs to +have ability to merge to this branch. -**Manual actions are considered to be write actions, so permissions for -protected branches are used when user wants to trigger an action. In other -words, in order to trigger a manual action assigned to a branch that the -pipeline is running for, user needs to have ability to merge to this branch.** - -### environment +## `environment` > **Notes:** @@ -727,7 +506,7 @@ deploy to production: In the above example, the `deploy to production` job will be marked as doing a deployment to the `production` environment. -#### environment:name +### `environment:name` > **Notes:** @@ -766,7 +545,7 @@ deploy to production: name: production ``` -#### environment:url +### `environment:url` > **Notes:** @@ -793,7 +572,7 @@ deploy to production: url: https://prod.example.com ``` -#### environment:on_stop +### `environment:on_stop` > **Notes:** @@ -808,7 +587,7 @@ the environment. Read the `environment:action` section for an example. -#### environment:action +### `environment:action` > [Introduced][ce-6669] in GitLab 8.13. @@ -849,7 +628,7 @@ The `stop_review_app` job is **required** to have the following keywords defined - `stage` should be the same as the `review_app` in order for the environment to stop automatically when the branch is deleted -#### dynamic environments +### Dynamic environments > **Notes:** @@ -885,13 +664,186 @@ The common use case is to create dynamic environments for branches and use them as Review Apps. You can see a simple example using Review Apps at <https://gitlab.com/gitlab-examples/review-apps-nginx/>. -### artifacts +## `cache` + +> +**Notes:** +- Introduced in GitLab Runner v0.7.0. +- `cache` can be set globally and per-job. +- From GitLab 9.0, caching is enabled and shared between pipelines and jobs + by default. +- From GitLab 9.2, caches are restored before [artifacts](#artifacts). + +TIP: **Learn more:** +Read how caching works and find out some good practices in the +[caching dependencies documentation](../caching/index.md). + +`cache` is used to specify a list of files and directories which should be +cached between jobs. You can only use paths that are within the project +workspace. + +If `cache` is defined outside the scope of jobs, it means it is set +globally and all jobs will use that definition. + +### `cache:paths` + +Use the `paths` directive to choose which files or directories will be cached. +Wildcards can be used as well. + +Cache all files in `binaries` that end in `.apk` and the `.config` file: + +```yaml +rspec: + script: test + cache: + paths: + - binaries/*.apk + - .config +``` + +Locally defined cache overrides globally defined options. The following `rspec` +job will cache only `binaries/`: + +```yaml +cache: + paths: + - my/files + +rspec: + script: test + cache: + paths: + - binaries/ +``` + +### `cache:key` + +> Introduced in GitLab Runner v1.0.0. + +Since the cache is shared between jobs, if you're using different +paths for different jobs, you should also set a different `cache:key` +otherwise cache content can be overwritten. + +The `key` directive allows you to define the affinity of caching between jobs, +allowing to have a single cache for all jobs, cache per-job, cache per-branch +or any other way that fits your workflow. This way, you can fine tune caching, +allowing you to cache data between different jobs or even different branches. + +The `cache:key` variable can use any of the +[predefined variables](../variables/README.md), and the default key, if not set, +is `$CI_JOB_NAME-$CI_COMMIT_REF_NAME` which translates as "per-job and +per-branch". It is the default across the project, therefore everything is +shared between pipelines and jobs running on the same branch by default. + +NOTE: **Note:** +The `cache:key` variable cannot contain the `/` character, or the equivalent +URI-encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden. + +For example, to enable per-branch caching: + +```yaml +cache: + key: "$CI_COMMIT_REF_SLUG" + paths: + - binaries/ +``` + +If you use **Windows Batch** to run your shell scripts you need to replace +`$` with `%`: + +```yaml +cache: + key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%" + paths: + - binaries/ +``` + +If you use **Windows PowerShell** to run your shell scripts you need to replace +`$` with `$env:`: + +```yaml +cache: + key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG" + paths: + - binaries/ +``` + +### `cache:untracked` + +Set `untracked: true` to cache all files that are untracked in your Git +repository: + +```yaml +rspec: + script: test + cache: + untracked: true +``` + +Cache all Git untracked files and files in `binaries`: + +```yaml +rspec: + script: test + cache: + untracked: true + paths: + - binaries/ +``` + +### `cache:policy` + +> Introduced in GitLab 9.4. + +The default behaviour of a caching job is to download the files at the start of +execution, and to re-upload them at the end. This allows any changes made by the +job to be persisted for future runs, and is known as the `pull-push` cache +policy. + +If you know the job doesn't alter the cached files, you can skip the upload step +by setting `policy: pull` in the job specification. Typically, this would be +twinned with an ordinary cache job at an earlier stage to ensure the cache +is updated from time to time: + +```yaml +stages: + - setup + - test + +prepare: + stage: setup + cache: + key: gems + paths: + - vendor/bundle + script: + - bundle install --deployment + +rspec: + stage: test + cache: + key: gems + paths: + - vendor/bundle + policy: pull + script: + - bundle exec rspec ... +``` + +This helps to speed up job execution and reduce load on the cache server, +especially when you have a large number of cache-using jobs executing in +parallel. + +Additionally, if you have a job that unconditionally recreates the cache without +reference to its previous contents, you can use `policy: push` in that job to +skip the download step. + +## `artifacts` > **Notes:** - Introduced in GitLab Runner v0.7.0 for non-Windows platforms. - Windows support was added in GitLab Runner v.1.0.0. -- Prior to GitLab 9.2, caches were restored after artifacts. - From GitLab 9.2, caches are restored before artifacts. - Currently not all executors are supported. - Job artifacts are only collected for successful jobs by default. @@ -960,7 +912,9 @@ release-job: The artifacts will be sent to GitLab after the job finishes successfully and will be available for download in the GitLab UI. -#### artifacts:name +[Read more about artifacts.](../../user/project/pipelines/job_artifacts.md) + +### `artifacts:name` > Introduced in GitLab 8.6 and GitLab Runner v1.1.0. @@ -970,10 +924,6 @@ useful when you'd like to download the archive from GitLab. The `artifacts:name` variable can make use of any of the [predefined variables](../variables/README.md). The default name is `artifacts`, which becomes `artifacts.zip` when downloaded. ---- - -**Example configurations** - To create an archive with a name of the current job: ```yaml @@ -998,7 +948,7 @@ tag including only the files that are untracked by Git: ```yaml job: artifacts: - name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}" + name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME" untracked: true ``` @@ -1007,7 +957,7 @@ To create an archive with a name of the current [stage](#stages) and branch name ```yaml job: artifacts: - name: "${CI_JOB_STAGE}_${CI_COMMIT_REF_NAME}" + name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME" untracked: true ``` @@ -1019,7 +969,7 @@ If you use **Windows Batch** to run your shell scripts you need to replace ```yaml job: artifacts: - name: "%CI_JOB_STAGE%_%CI_COMMIT_REF_NAME%" + name: "%CI_JOB_STAGE%-%CI_COMMIT_REF_NAME%" untracked: true ``` @@ -1029,11 +979,11 @@ If you use **Windows PowerShell** to run your shell scripts you need to replace ```yaml job: artifacts: - name: "$env:CI_JOB_STAGE_$env:CI_COMMIT_REF_NAME" + name: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_NAME" untracked: true ``` -#### artifacts:when +### `artifacts:when` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. @@ -1046,11 +996,7 @@ failure. 1. `on_failure` - upload artifacts only when the job fails. 1. `always` - upload artifacts regardless of the job status. ---- - -**Example configurations** - -To upload artifacts only when job fails. +To upload artifacts only when job fails: ```yaml job: @@ -1058,22 +1004,23 @@ job: when: on_failure ``` -#### artifacts:expire_in +### `artifacts:expire_in` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. -`artifacts:expire_in` is used to delete uploaded artifacts after the specified -time. By default, artifacts are stored on GitLab forever. `expire_in` allows you -to specify how long artifacts should live before they expire, counting from the -time they are uploaded and stored on GitLab. +`expire_in` allows you to specify how long artifacts should live before they +expire and therefore deleted, counting from the time they are uploaded and +stored on GitLab. If the expiry time is not defined, it defaults to the +[instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration) +(30 days by default, forever on GitLab.com). You can use the **Keep** button on the job page to override expiration and keep artifacts forever. -After expiry, artifacts are actually deleted hourly by default (via a cron job), -but they are not accessible after expiry. +After their expiry, artifacts are deleted hourly by default (via a cron job), +and are not accessible anymore. -The value of `expire_in` is an elapsed time. Examples of parseable values: +The value of `expire_in` is an elapsed time. Examples of parsable values: - '3 mins 4 sec' - '2 hrs 20 min' @@ -1082,10 +1029,6 @@ The value of `expire_in` is an elapsed time. Examples of parseable values: - '47 yrs 6 mos and 4d' - '3 weeks and 2 days' ---- - -**Example configurations** - To expire artifacts 1 week after being uploaded: ```yaml @@ -1094,7 +1037,7 @@ job: expire_in: 1 week ``` -### dependencies +## `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1153,7 +1096,7 @@ deploy: script: make deploy ``` -#### When a dependent job will fail +### When a dependent job will fail > Introduced in GitLab 10.3. @@ -1167,27 +1110,9 @@ You can ask your administrator to [flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) and bring back the old behavior. -### before_script and after_script - -It's possible to overwrite the globally defined `before_script` and `after_script`: - -```yaml -before_script: -- global before script - -job: - before_script: - - execute this instead of global before script - script: - - my command - after_script: - - execute this after my script -``` - -### coverage +## `coverage` -**Notes:** -- [Introduced][ce-7447] in GitLab 8.17. +> [Introduced][ce-7447] in GitLab 8.17. `coverage` allows you to configure how code coverage will be extracted from the job output. @@ -1205,10 +1130,9 @@ job1: coverage: '/Code coverage: \d+\.\d+/' ``` -### retry +## `retry` -**Notes:** -- [Introduced][ce-3442] in GitLab 9.5. +> [Introduced][ce-12909] in GitLab 9.5. `retry` allows you to configure how many times a job is going to be retried in case of a failure. @@ -1228,16 +1152,57 @@ test: retry: 2 ``` -## Git Strategy +## `variables` + +> Introduced in GitLab Runner v0.5.0. + +NOTE: **Note:** +Integers (as well as strings) are legal both for variable's name and value. +Floats are not legal and cannot be used. + +GitLab CI/CD allows you to define variables inside `.gitlab-ci.yml` that are +then passed in the job environment. They can be set globally and per-job. +When the `variables` keyword is used on a job level, it overrides the global +YAML variables and predefined ones. + +They are stored in the Git repository and are meant to store non-sensitive +project configuration, for example: + +```yaml +variables: + DATABASE_URL: "postgres://postgres@postgres/my_database" +``` + +These variables can be later used in all executed commands and scripts. +The YAML-defined variables are also set to all created service containers, +thus allowing to fine tune them. + +To turn off global defined variables in a specific job, define an empty hash: + +```yaml +job_name: + variables: {} +``` + +Except for the user defined variables, there are also the ones [set up by the +Runner itself](../variables/README.md#predefined-variables-environment-variables). +One example would be `CI_COMMIT_REF_NAME` which has the value of +the branch or tag name for which project is built. Apart from the variables +you can set in `.gitlab-ci.yml`, there are also the so called +[secret variables](../variables/README.md#secret-variables) +which can be set in GitLab's UI. + +[Learn more about variables and their priority.][variables] + +### Git strategy > Introduced in GitLab 8.9 as an experimental feature. May change or be removed completely in future releases. `GIT_STRATEGY=none` requires GitLab Runner v1.7+. You can set the `GIT_STRATEGY` used for getting recent application code, either -in the global [`variables`](#variables) section or the [`variables`](#job-variables) -section for individual jobs. If left unspecified, the default from project -settings will be used. +globally or per-job in the [`variables`](#variables) section. If left +unspecified, the default from project settings will be used. There are three possible values: `clone`, `fetch`, and `none`. @@ -1269,44 +1234,13 @@ variables: GIT_STRATEGY: none ``` -## Git Checkout - -> Introduced in GitLab Runner 9.3 - -The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either -`clone` or `fetch` to specify whether a `git checkout` should be run. If not -specified, it defaults to true. Like `GIT_STRATEGY`, it can be set in either the -global [`variables`](#variables) section or the [`variables`](#job-variables) -section for individual jobs. - -If set to `false`, the Runner will: - -- when doing `fetch` - update the repository and leave working copy on - the current revision, -- when doing `clone` - clone the repository and leave working copy on the - default branch. - -Having this setting set to `true` will mean that for both `clone` and `fetch` -strategies the Runner will checkout the working copy to a revision related -to the CI pipeline: - -```yaml -variables: - GIT_STRATEGY: clone - GIT_CHECKOUT: "false" -script: - - git checkout master - - git merge $CI_BUILD_REF_NAME -``` - -## Git Submodule Strategy +### Git submodule strategy > Requires GitLab Runner v1.10+. The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git -submodules are included when fetching the code before a build. Like -`GIT_STRATEGY`, it can be set in either the global [`variables`](#variables) -section or the [`variables`](#job-variables) section for individual jobs. +submodules are included when fetching the code before a build. You can set them +globally or per-job in the [`variables`](#variables) section. There are three possible values: `none`, `normal`, and `recursive`: @@ -1336,8 +1270,36 @@ Note that for this feature to work correctly, the submodules must be configured - a relative path to another repository on the same GitLab server. See the [Git submodules](../git_submodules.md) documentation. +### Git checkout + +> Introduced in GitLab Runner 9.3 + +The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either +`clone` or `fetch` to specify whether a `git checkout` should be run. If not +specified, it defaults to true. You can set them globally or per-job in the +[`variables`](#variables) section. + +If set to `false`, the Runner will: + +- when doing `fetch` - update the repository and leave working copy on + the current revision, +- when doing `clone` - clone the repository and leave working copy on the + default branch. + +Having this setting set to `true` will mean that for both `clone` and `fetch` +strategies the Runner will checkout the working copy to a revision related +to the CI pipeline: + +```yaml +variables: + GIT_STRATEGY: clone + GIT_CHECKOUT: "false" +script: + - git checkout master + - git merge $CI_BUILD_REF_NAME +``` -## Job stages attempts +### Job stages attempts > Introduced in GitLab, it requires GitLab Runner v1.9+. @@ -1359,10 +1321,9 @@ variables: GET_SOURCES_ATTEMPTS: 3 ``` -You can set them in the global [`variables`](#variables) section or the -[`variables`](#job-variables) section for individual jobs. +You can set them globally or per-job in the [`variables`](#variables) section. -## Shallow cloning +### Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. May change in future releases or be removed completely. @@ -1393,7 +1354,17 @@ variables: GIT_DEPTH: "3" ``` -## Hidden keys (jobs) +You can set it globally or per-job in the [`variables`](#variables) section. + +## Special YAML features + +It's possible to use special YAML features like anchors (`&`), aliases (`*`) +and map merging (`<<`), which will allow you to greatly reduce the complexity +of `.gitlab-ci.yml`. + +Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). + +### Hidden keys (jobs) > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1419,14 +1390,6 @@ Use this feature to ignore jobs, or use the [special YAML features](#special-yaml-features) and transform the hidden keys into templates. -## Special YAML features - -It's possible to use special YAML features like anchors (`&`), aliases (`*`) -and map merging (`<<`), which will allow you to greatly reduce the complexity -of `.gitlab-ci.yml`. - -Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). - ### Anchors > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1556,50 +1519,22 @@ with an API call. [Read more in the triggers documentation.](../triggers/README.md) -### pages - -`pages` is a special job that is used to upload static content to GitLab that -can be used to serve your website. It has a special syntax, so the two -requirements below must be met: - -1. Any static content must be placed under a `public/` directory -1. `artifacts` with a path to the `public/` directory must be defined - -The example below simply moves all files from the root of the project to the -`public/` directory. The `.public` workaround is so `cp` doesn't also copy -`public/` to itself in an infinite loop: - -``` -pages: - stage: deploy - script: - - mkdir .public - - cp -r * .public - - mv .public public - artifacts: - paths: - - public - only: - - master -``` +## Skipping jobs -Read more on [GitLab Pages user documentation](../../user/project/pages/index.md). +If your commit message contains `[ci skip]` or `[skip ci]`, using any +capitalization, the commit will be created but the pipeline will be skipped. ## Validate the .gitlab-ci.yml -Each instance of GitLab CI has an embedded debug tool called Lint. -You can find the link under `/ci/lint` of your gitlab instance. +Each instance of GitLab CI has an embedded debug tool called Lint, which validates the +content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your +project namespace (e.g, `http://gitlab-example.com/gitlab-org/project-123/ci/lint`) ## Using reserved keywords If you get validation error when using specific values (e.g., `true` or `false`), try to quote them, or change them to a different form (e.g., `/bin/true`). -## Skipping jobs - -If your commit message contains `[ci skip]` or `[skip ci]`, using any -capitalization, the commit will be created but the jobs will be skipped. - ## Examples Visit the [examples README][examples] to see a list of examples using GitLab @@ -1613,5 +1548,5 @@ CI with various languages. [variables]: ../variables/README.md [ce-7983]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7983 [ce-7447]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7447 -[ce-3442]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3442 +[ce-12909]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12909 [schedules]: ../../user/project/pipelines/schedules.md diff --git a/doc/customization/branded_login_page.md b/doc/customization/branded_login_page.md index d4d9f5f7b5e..b892f59d777 100644 --- a/doc/customization/branded_login_page.md +++ b/doc/customization/branded_login_page.md @@ -1,6 +1,6 @@ # Changing the appearance of the login page -GitLab Community Edition offers a way to put your company's identity on the login page of your GitLab server and make it a branded login page. +GitLab offers a way to put your company's identity on the login page of your GitLab server and make it a branded login page. By default, the page shows the GitLab logo and description. diff --git a/doc/development/README.md b/doc/development/README.md index 12cca9f84b7..45e9565f9a7 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -75,6 +75,7 @@ comments: false - [Ordering table columns](ordering_table_columns.md) - [Verifying database capabilities](verifying_database_capabilities.md) - [Database Debugging and Troubleshooting](database_debugging.md) +- [Query Count Limits](query_count_limits.md) ## Testing guides diff --git a/doc/development/architecture.md b/doc/development/architecture.md index d1ba7d3dfc3..31117b5e723 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -2,9 +2,11 @@ ## Software delivery -There are two editions of GitLab: [Enterprise Edition](https://about.gitlab.com/gitlab-ee/) (EE) and [Community Edition](https://about.gitlab.com/gitlab-ce/) (CE). GitLab CE is delivered via git from the [gitlabhq repository](https://gitlab.com/gitlab-org/gitlab-ce/tree/master). New versions of GitLab are released in stable branches and the master branch is for bleeding edge development. +There are two software distributions of GitLab: the open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-ce/) (CE), and the open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab-ee/) (EE). GitLab is available under [different subscriptions](https://about.gitlab.com/products/). -EE releases are available not long after CE releases. To obtain the GitLab EE there is a [repository at gitlab.com](https://gitlab.com/gitlab-org/gitlab-ee). For more information about the release process see the section 'New versions and upgrading' in the readme. +New versions of GitLab are released in stable branches and the master branch is for bleeding edge development. + +For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/release/docs/tree/master#gitlab-release-process). Both EE and CE require some add-on components called gitlab-shell and Gitaly. These components are available from the [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. diff --git a/doc/development/automatic_ce_ee_merge.md b/doc/development/automatic_ce_ee_merge.md index cf6314f9521..a85e5b1b1cc 100644 --- a/doc/development/automatic_ce_ee_merge.md +++ b/doc/development/automatic_ce_ee_merge.md @@ -103,6 +103,99 @@ Notes: - You can use [`git rerere`](https://git-scm.com/blog/2010/03/08/rerere.html) to avoid resolving the same conflicts multiple times. +### Cherry-picking from CE to EE + +For avoiding merge conflicts, we use a method of creating equivalent branches +for CE and EE. If the `ee-compat-check` job fails, this process is required. + +This method only requires that you have cloned both CE and EE into your computer. +If you don't have them yet, please go ahead and clone them: + +- Clone CE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ce.git` +- Clone EE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ee.git` + +And the only additional setup we need is to add CE as remote of EE and vice-versa: + +- Open two terminal windows, one in CE, and another one in EE: + - In EE: `git remote add ce git@gitlab.com:gitlab-org/gitlab-ce.git` + - In CE: `git remote add ee git@gitlab.com:gitlab-org/gitlab-ee.git` + +That's all setup we need, so that we can cherry-pick a commit from CE to EE, and +from EE to CE. + +Now, every time you create an MR for CE and EE: + +1. Open two terminal windows, one in CE, and another one in EE +1. In the CE terminal: + 1. Create the CE branch, e.g., `branch-example` + 1. Make your changes and push a commit (commit A) + 1. Create the CE merge request in GitLab +1. In the EE terminal: + 1. Create the EE-equivalent branch ending with `-ee`, e.g., + `git checkout -b branch-example-ee` + 1. Fetch the CE branch: `git fetch ce branch-example` + 1. Cherry-pick the commit A: `git cherry-pick commit-A-SHA` + 1. If Git prompts you to fix the conflicts, do a `git status` + to check which files contain conflicts, fix them, save the files + 1. Add the changes with `git add .` but **DO NOT commit** them + 1. Continue cherry-picking: `git cherry-pick --continue` + 1. Push to EE: `git push origin branch-example-ee` +1. Create the EE-equivalent MR and link to the CE MR from the +description "Ports [CE-MR-LINK] to EE" +1. Once all the jobs are passing in both CE and EE, you've addressed the +feedback from your own team, and got them approved, the merge requests can be merged. +1. When both MRs are ready, the EE merge request will be merged first, and the +CE-equivalent will be merged next. + +**Important notes:** + +- The commit SHA can be easily found from the GitLab UI. From a merge request, +open the tab **Commits** and click the copy icon to copy the commit SHA. +- To cherry-pick a **commit range**, such as [A > B > C > D] use: + + ```shell + git cherry-pick "oldest-commit-SHA^..newest-commit-SHA" + ``` + + For example, suppose the commit A is the oldest, and its SHA is `4f5e4018c09ed797fdf446b3752f82e46f5af502`, + and the commit D is the newest, and its SHA is `80e1c9e56783bd57bd7129828ec20b252ebc0538`. + The cherry-pick command will be: + + ```shell + git cherry-pick "4f5e4018c09ed797fdf446b3752f82e46f5af502^..80e1c9e56783bd57bd7129828ec20b252ebc0538" + ``` + +- To cherry-pick a **merge commit**, use the flag `-m 1`. For example, suppose that the +merge commit SHA is `138f5e2f20289bb376caffa0303adb0cac859ce1`: + + ```shell + git cherry-pick -m 1 138f5e2f20289bb376caffa0303adb0cac859ce1 + ``` +- To cherry-pick multiple commits, such as B and D in a range [A > B > C > D], use: + + ```shell + git cherry-pick commmit-B-SHA commit-D-SHA + ``` + + For example, suppose commit B SHA = `4f5e4018c09ed797fdf446b3752f82e46f5af502`, + and the commit D SHA = `80e1c9e56783bd57bd7129828ec20b252ebc0538`. + The cherry-pick command will be: + + ```shell + git cherry-pick 4f5e4018c09ed797fdf446b3752f82e46f5af502 80e1c9e56783bd57bd7129828ec20b252ebc0538 + ``` + + This case is particularly useful when you have a merge commit in a sequence of + commits and you want to cherry-pick all but the merge commit. + +- If you push more commits to the CE branch, you can safely repeat the procedure +to cherry-pick them to the EE-equivalent branch. You can do that as many times as +necessary, using the same CE and EE branches. +- If you submitted the merge request to the CE repo and the `ee-compat-check` job passed, +you are not required to submit the EE-equivalent MR, but it's still recommended. If the +job failed, you are required to submit the EE MR so that you can fix the conflicts in EE +before merging your changes into CE. + --- [Return to Development documentation](README.md) diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index af2026c483e..fc1b202b5eb 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -94,6 +94,18 @@ jobs = [['BackgroundMigrationClassName', [1]], BackgroundMigrationWorker.bulk_perform_in(5.minutes, jobs) ``` +### Rescheduling background migrations + +If one of the background migrations contains a bug that is fixed in a patch +release, the background migration needs to be rescheduled so the migration would +be repeated on systems that already performed the initial migration. + +When you reschedule the background migration, make sure to turn the original +scheduling into a no-op by clearing up the `#up` and `#down` methods of the +migration performing the scheduling. Otherwise the background migration would be +scheduled multiple times on systems that are upgrading multiple patch releases at +once. + ## Cleaning Up >**Note:** diff --git a/doc/development/changelog.md b/doc/development/changelog.md index c1f783ce877..1962392a9eb 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -125,7 +125,7 @@ author: type: ``` If you're working on the GitLab EE repository, the entry will be added to -`changelogs/unreleased-ee/` instead. +`ee/changelogs/unreleased/` instead. ### Arguments @@ -279,8 +279,8 @@ After much discussion we settled on the current solution of one file per entry, and then compiling the entries into the overall `CHANGELOG.md` file during the [release process]. -[boring solution]: https://about.gitlab.com/handbook/#boring-solutions -[release managers]: https://gitlab.com/gitlab-org/release-tools/blob/master/doc/release-manager.md +[boring solution]: https://about.gitlab.com/handbook/values/#boring-solutions +[release managers]: https://gitlab.com/gitlab-org/release/docs/blob/master/quickstart/release-manager.md [started brainstorming]: https://gitlab.com/gitlab-org/gitlab-ce/issues/17826 [release process]: https://gitlab.com/gitlab-org/release-tools diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 50eb8005b44..32f392f1303 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -53,3 +53,38 @@ bundle exec rails db RAILS_ENV=development - `CREATE TABLE board_labels();`: Create a table called `board_labels` - `SELECT * FROM schema_migrations WHERE version = '20170926203418';`: Check if a migration was run - `DELETE FROM schema_migrations WHERE version = '20170926203418';`: Manually remove a migration + + +## FAQ + +### `ActiveRecord::PendingMigrationError` with Spring + +When running specs with the [Spring preloader](./rake_tasks.md#speed-up-tests-rake-tasks-and-migrations), +the test database can get into a corrupted state. Trying to run the migration or +dropping/resetting the test database has no effect. + +```sh +$ bundle exec spring rspec some_spec.rb +... +Failure/Error: ActiveRecord::Migration.maintain_test_schema! + +ActiveRecord::PendingMigrationError: + + + Migrations are pending. To resolve this issue, run: + + bin/rake db:migrate RAILS_ENV=test +# ~/.rvm/gems/ruby-2.3.3/gems/activerecord-4.2.10/lib/active_record/migration.rb:392:in `check_pending!' +... +0 examples, 0 failures, 1 error occurred outside of examples +``` + +To resolve, you can kill the spring server and app that lives between spec runs. + +```sh +$ ps aux | grep spring +eric 87304 1.3 2.9 3080836 482596 ?? Ss 10:12AM 4:08.36 spring app | gitlab | started 6 hours ago | test mode +eric 37709 0.0 0.0 2518640 7524 s006 S Wed11AM 0:00.79 spring server | gitlab | started 29 hours ago +$ kill 87304 +$ kill 37709 +``` diff --git a/doc/development/doc_styleguide.md b/doc/development/doc_styleguide.md index cfeeed2506d..41e3412c7ff 100644 --- a/doc/development/doc_styleguide.md +++ b/doc/development/doc_styleguide.md @@ -1,89 +1,28 @@ -# Documentation styleguide +# Documentation style guidelines -This styleguide recommends best practices to improve documentation and to keep -it organized and easy to find. +The documentation style guide defines the markup structure used in +GitLab documentation. Check the +[documentation guidelines](writing_documentation.md) for general development instructions. -See also [writing documentation](writing_documentation.md). - -## Location and naming of documents - ->**Note:** -These guidelines derive from the discussion taken place in issue [#3349][ce-3349]. - -The documentation hierarchy can be vastly improved by providing a better layout -and organization of directories. - -Having a structured document layout, we will be able to have meaningful URLs -like `docs.gitlab.com/user/project/merge_requests.html`. With this pattern, -you can immediately tell that you are navigating a user related documentation -and is about the project and its merge requests. - -Do not create summaries of similar types of content (e.g. an index of all articles, videos, etc.), -rather organise content by its subject (e.g. everything related to CI goes together) -and cross-link between any related content. - -The table below shows what kind of documentation goes where. - -| Directory | What belongs here | -| --------- | -------------- | -| `doc/user/` | User related documentation. Anything that can be done within the GitLab UI goes here including `/admin`. | -| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed via GitLab's interface go under `doc/user/admin_area/`. | -| `doc/api/` | API related documentation. | -| `doc/development/` | Documentation related to the development of GitLab. Any styleguides should go here. | -| `doc/legal/` | Legal documents about contributing to GitLab. | -| `doc/install/`| Probably the most visited directory, since `installation.md` is there. Ideally this should go under `doc/administration/`, but it's best to leave it as-is in order to avoid confusion (still debated though). | -| `doc/update/` | Same with `doc/install/`. Should be under `administration/`, but this is a well known location, better leave as-is, at least for now. | -| `doc/topics/` | Indexes per Topic (`doc/topics/topic-name/index.md`): all resources for that topic (user and admin documentation, articles, and third-party docs) | - ---- - -**General rules:** - -1. The correct naming and location of a new document, is a combination - of the relative URL of the document in question and the GitLab Map design - that is used for UX purposes ([source][graffle], [image][gitlab-map]). -1. When creating a new document and it has more than one word in its name, - make sure to use underscores instead of spaces or dashes (`-`). For example, - a proper naming would be `import_projects_from_github.md`. The same rule - applies to images. -1. There are four main directories, `user`, `administration`, `api` and `development`. -1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, - `profile/`, `dashboard/` and `admin_area/`. - 1. `doc/user/project/` should contain all project related documentation. - 1. `doc/user/group/` should contain all group related documentation. - 1. `doc/user/profile/` should contain all profile related documentation. - Every page you would navigate under `/profile` should have its own document, - i.e. `account.md`, `applications.md`, `emails.md`, etc. - 1. `doc/user/dashboard/` should contain all dashboard related documentation. - 1. `doc/user/admin_area/` should contain all admin related documentation - describing what can be achieved by accessing GitLab's admin interface - (_not to be confused with `doc/administration` where server access is - required_). - 1. Every category under `/admin/application_settings` should have its - own document located at `doc/user/admin_area/settings/`. For example, - the **Visibility and Access Controls** category should have a document - located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. -1. The `doc/topics/` directory holds topic-related technical content. Create - `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. - General user- and admin- related documentation, should be placed accordingly. - ---- - -If you are unsure where a document should live, you can ping `@axil` or `@marcia` in your -merge request. +Check the GitLab hanbook for the [writing styles guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines). ## Text -- Split up long lines, this makes it much easier to review and edit. Only +- Split up long lines (wrap text), this makes it much easier to review and edit. Only double line breaks are shown as a full line break in [GitLab markdown][gfm]. 80-100 characters is a good line length -- Make sure that the documentation is added in the correct directory and that +- Make sure that the documentation is added in the correct + [directory](writing_documentation.md#documentation-directory-structure) and that there's a link to it somewhere useful - Do not duplicate information - Be brief and clear - Unless there's a logical reason not to, add documents in alphabetical order - Write in US English - Use [single spaces][] instead of double spaces +- Jump a line between different markups (e.g., after every paragraph, hearder, list, etc) +- Capitalize "G" and "L" in GitLab +- Capitalize feature, products, and methods names. E.g.: GitLab Runner, Geo, +Issue Boards, Git, Prometheus, Continuous Integration. ## Formatting @@ -103,6 +42,8 @@ merge request. links shift too, which eventually leads to dead links. If you think it is compelling to add numbers in headings, make sure to at least discuss it with someone in the Merge Request +- [Avoid using symbols and special chars](https://gitlab.com/gitlab-com/gitlab-docs/issues/84) + in headers. Whenever possible, they should be plain and short text. - Avoid adding things that show ephemeral statuses. For example, if a feature is considered beta or experimental, put this info in a note, not in the heading. - When introducing a new document, be careful for the headings to be @@ -121,71 +62,18 @@ merge request. you can use `[Text][identifier]` and at the bottom of the section or the document add: `[identifier]: https://example.com`, in which case, we do encourage you to also add an alternative text: `[identifier]: https://example.com "Alternative text"` that appears when hovering your mouse on a link. - -### Linking to inline docs - -Sometimes it's needed to link to the built-in documentation that GitLab provides -under `/help`. This is normally done in files inside the `app/views/` directory -with the help of the `help_page_path` helper method. - -In its simplest form, the HAML code to generate a link to the `/help` page is: - -```haml -= link_to 'Help page', help_page_path('user/permissions') -``` - -The `help_page_path` contains the path to the document you want to link to with -the following conventions: - -- it is relative to the `doc/` directory in the GitLab repository -- the `.md` extension must be omitted -- it must not end with a slash (`/`) - -Below are some special cases where should be used depending on the context. -You can combine one or more of the following: - -1. **Linking to an anchor link.** Use `anchor` as part of the `help_page_path` - method: - - ```haml - = link_to 'Help page', help_page_path('user/permissions', anchor: 'anchor-link') - ``` - -1. **Opening links in a new tab.** This should be the default behavior: - - ```haml - = link_to 'Help page', help_page_path('user/permissions'), target: '_blank' - ``` - -1. **Linking to a circle icon.** Usually used in settings where a long - description cannot be used, like near checkboxes. You can basically use - any font awesome icon, but prefer the `question-circle`: - - ```haml - = link_to icon('question-circle'), help_page_path('user/permissions') - ``` - -1. **Using a button link.** Useful in places where text would be out of context - with the rest of the page layout: - - ```haml - = link_to 'Help page', help_page_path('user/permissions'), class: 'btn btn-info' - ``` - -1. **Using links inline of some text.** - - ```haml - Description to #{link_to 'Help page', help_page_path('user/permissions')}. - ``` - -1. **Adding a period at the end of the sentence.** Useful when you don't want - the period to be part of the link: - - ```haml - = succeed '.' do - Learn more in the - = link_to 'Help page', help_page_path('user/permissions') - ``` +- To link to internal documentation, use relative links, not full URLs. Use `../` to + navigate tp high-level directories, and always add the file name `file.md` at the + end of the link with the `.md` extension, not `.html`. + Example: instead of `[text](../../merge_requests/)`, use + `[text](../../merge_requests/index.md)` or, `[text](../../ci/README.md)`, or, + for anchor links, `[text](../../ci/README.md#examples)`. + Using the markdown extension is necessary for the [`/help`](writing_documentation.md#gitlab-help) + section of GitLab. +- To link from CE to EE-only documentation, use the EE-only doc full URL. +- Use [meaningful anchor texts](https://www.futurehosting.com/blog/links-should-have-meaningful-anchor-text-heres-why/). + E.g., instead of writing something like `Read more about GitLab Issue Boards [here](LINK)`, + write `Read more about [GitLab Issue Boards](LINK)`. ## Images @@ -222,7 +110,7 @@ Inside the document: - Notes should be quoted with the word `Note:` being bold. Use this form: - ``` + ```md >**Note:** This is something to note. ``` @@ -234,144 +122,70 @@ Inside the document: If the note spans across multiple lines it's OK to split the line. -## New features +## Specific sections and terms + +To mention and/or reference specific terms in GitLab, please follow the styles +below. + +### GitLab versions and tiers - Every piece of documentation that comes with a new feature should declare the GitLab version that feature got introduced. Right below the heading add a note: - ``` + ```md > Introduced in GitLab 8.3. ``` -- If possible every feature should have a link to the MR that introduced it. +- If possible every feature should have a link to the MR, issue, or epic that introduced it. The above note would be then transformed to: - ``` + ```md > [Introduced][ce-1242] in GitLab 8.3. ``` , where the [link identifier](#links) is named after the repository (CE) and the MR number. -- If the feature is only in GitLab Enterprise Edition, don't forget to mention - it, like: +- If the feature is only available in GitLab Enterprise Edition, don't forget to mention + the [paid tier](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers) + the feature is available in: - ``` - > Introduced in GitLab Enterprise Edition 8.3. + ```md + > [Introduced][ee-1234] in [GitLab Starter](https://about.gitlab.com/products/) 8.3. ``` Otherwise, leave this mention out. -## References - -- **GitLab Restart:** - There are many cases that a restart/reconfigure of GitLab is required. To - avoid duplication, link to the special document that can be found in - [`doc/administration/restart_gitlab.md`][doc-restart]. Usually the text will - read like: - - ``` - Save the file and [reconfigure GitLab](../administration/restart_gitlab.md) - for the changes to take effect. - ``` - If the document you are editing resides in a place other than the GitLab CE/EE - `doc/` directory, instead of the relative link, use the full path: - `http://docs.gitlab.com/ce/administration/restart_gitlab.html`. - Replace `reconfigure` with `restart` where appropriate. - -## Installation guide - -- **Ruby:** - In [step 2 of the installation guide](../install/installation.md#2-ruby), - we install Ruby from source. Whenever there is a new version that needs to - be updated, remember to change it throughout the codeblock and also replace - the sha256sum (it can be found in the [downloads page][ruby-dl] of the Ruby - website). - -[ruby-dl]: https://www.ruby-lang.org/en/downloads/ "Ruby download website" - -## Changing document location - -Changing a document's location is not to be taken lightly. Remember that the -documentation is available to all installations under `help/` and not only to -GitLab.com or http://docs.gitlab.com. Make sure this is discussed with the -Documentation team beforehand. - -If you indeed need to change a document's location, do NOT remove the old -document, but rather replace all of its contents with a new line: - -``` -This document was moved to [another location](path/to/new_doc.md). -``` - -where `path/to/new_doc.md` is the relative path to the root directory `doc/`. +### GitLab Restart ---- - -For example, if you were to move `doc/workflow/lfs/lfs_administration.md` to -`doc/administration/lfs.md`, then the steps would be: - -1. Copy `doc/workflow/lfs/lfs_administration.md` to `doc/administration/lfs.md` -1. Replace the contents of `doc/workflow/lfs/lfs_administration.md` with: - - ``` - This document was moved to [another location](../../administration/lfs.md). - ``` - -1. Find and replace any occurrences of the old location with the new one. - A quick way to find them is to use `git grep`. First go to the root directory - where you cloned the `gitlab-ce` repository and then do: - - ``` - git grep -n "workflow/lfs/lfs_administration" - git grep -n "lfs/lfs_administration" - ``` +There are many cases that a restart/reconfigure of GitLab is required. To +avoid duplication, link to the special document that can be found in +[`doc/administration/restart_gitlab.md`][doc-restart]. Usually the text will +read like: -NOTE: **Note:** -If the document being moved has any Disqus comments on it, there are extra steps -to follow documented just [below](#redirections-for-pages-with-disqus-comments). + ``` + Save the file and [reconfigure GitLab](../administration/restart_gitlab.md) + for the changes to take effect. + ``` -Things to note: +If the document you are editing resides in a place other than the GitLab CE/EE +`doc/` directory, instead of the relative link, use the full path: +`http://docs.gitlab.com/ce/administration/restart_gitlab.html`. +Replace `reconfigure` with `restart` where appropriate. -- Since we also use inline documentation, except for the documentation itself, - the document might also be referenced in the views of GitLab (`app/`) which will - render when visiting `/help`, and sometimes in the testing suite (`spec/`). -- The above `git grep` command will search recursively in the directory you run - it in for `workflow/lfs/lfs_administration` and `lfs/lfs_administration` - and will print the file and the line where this file is mentioned. - You may ask why the two greps. Since we use relative paths to link to - documentation, sometimes it might be useful to search a path deeper. -- The `*.md` extension is not used when a document is linked to GitLab's - built-in help page, that's why we omit it in `git grep`. -- Use the checklist on the documentation MR description template. +### Installation guide -### Redirections for pages with Disqus comments +**Ruby:** +In [step 2 of the installation guide](../install/installation.md#2-ruby), +we install Ruby from source. Whenever there is a new version that needs to +be updated, remember to change it throughout the codeblock and also replace +the sha256sum (it can be found in the [downloads page][ruby-dl] of the Ruby +website). -If the documentation page being relocated already has any Disqus comments, -we need to preserve the Disqus thread. - -Disqus uses an identifier per page, and for docs.gitlab.com, the page identifier -is configured to be the page URL. Therefore, when we change the document location, -we need to preserve the old URL as the same Disqus identifier. - -To do that, add to the frontmatter the variable `redirect_from`, -using the old URL as value. For example, let's say I moved the document -available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, -`https://docs.gitlab.com/my-new-location/index.html`. - -Into the **new document** frontmatter add the following: - -```yaml ---- -redirect_from: 'https://docs.gitlab.com/my-old-location/README.html' ---- -``` - -Note: it is necessary to include the file name in the `redirect_from` URL, -even if it's `index.html` or `README.html`. +[ruby-dl]: https://www.ruby-lang.org/en/downloads/ "Ruby download website" -## Configuration documentation for source and Omnibus installations +### Configuration documentation for source and Omnibus installations GitLab currently officially supports two installation methods: installations from source and Omnibus packages installations. @@ -388,7 +202,7 @@ When there is a list of steps to perform, usually that entails editing the configuration file and reconfiguring/restarting GitLab. In such case, follow the style below as a guide: -```` +```md **For Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb`: @@ -415,7 +229,7 @@ the style below as a guide: [reconfigure]: path/to/administration/restart_gitlab.md#omnibus-gitlab-reconfigure [restart]: path/to/administration/restart_gitlab.md#installations-from-source -```` +``` In this case: @@ -427,7 +241,7 @@ In this case: - different highlighting languages are used for each config in the code block - the [references](#references) guide is used for reconfigure/restart -## Fake tokens +### Fake tokens There may be times where a token is needed to demonstrate an API call using cURL or a secret variable used in CI. It is strongly advised not to use real @@ -450,7 +264,7 @@ You can use the following fake tokens as examples. | Health check token | `Tu7BgjR9qeZTEyRzGG2P` | | Request profile token | `7VgpS4Ax5utVD2esNstz` | -## API +### API Here is a list of must-have items. Use them in the exact order that appears on this document. Further explanation is given below. @@ -466,10 +280,10 @@ on this document. Further explanation is given below. - Every method must have a cURL example. - Every method must have a response body (in JSON format). -### Method description +#### Method description Use the following table headers to describe the methods. Attributes should -always be in code blocks using backticks (`). +always be in code blocks using backticks (``` ` ```). ``` | Attribute | Type | Required | Description | @@ -482,7 +296,7 @@ Rendered example: | --------- | ---- | -------- | ----------- | | `user` | string | yes | The GitLab username | -### cURL commands +#### cURL commands - Use `https://gitlab.example.com/api/v4/` as an endpoint. - Wherever needed use this personal access token: `9koXpg98eAheJpvBs5tK`. @@ -499,11 +313,11 @@ Rendered example: | `-X PUT` | Use this method when updating existing objects | | `-X DELETE` | Use this method when removing existing objects | -### cURL Examples +#### cURL Examples Below is a set of [cURL][] examples that you can use in the API documentation. -#### Simple cURL command +##### Simple cURL command Get the details of a group: @@ -511,7 +325,7 @@ Get the details of a group: curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/groups/gitlab-org ``` -#### cURL example with parameters passed in the URL +##### cURL example with parameters passed in the URL Create a new project under the authenticated user's namespace: @@ -519,7 +333,7 @@ Create a new project under the authenticated user's namespace: curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects?name=foo" ``` -#### Post data using cURL's --data +##### Post data using cURL's --data Instead of using `-X POST` and appending the parameters to the URI, you can use cURL's `--data` option. The example below will create a new project `foo` under @@ -529,7 +343,7 @@ the authenticated user's namespace. curl --data "name=foo" --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects" ``` -#### Post data using JSON content +##### Post data using JSON content > **Note:** In this example we create a new group. Watch carefully the single and double quotes. @@ -538,7 +352,7 @@ and double quotes. curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' https://gitlab.example.com/api/v4/groups ``` -#### Post data using form-data +##### Post data using form-data Instead of using JSON or urlencode you can use multipart/form-data which properly handles data encoding: @@ -550,7 +364,7 @@ curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --form "title The above example is run by and administrator and will add an SSH public key titled ssh-key to user's account which has an id of 25. -#### Escape special characters +##### Escape special characters Spaces or slashes (`/`) may sometimes result to errors, thus it is recommended to escape them when possible. In the example below we create a new issue which @@ -563,7 +377,7 @@ curl --request POST --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitl Use `%2F` for slashes (`/`). -#### Pass arrays to API calls +##### Pass arrays to API calls The GitLab API sometimes accepts arrays of strings or integers. For example, to restrict the sign-up e-mail domains of a GitLab instance to `*.example.com` and @@ -578,6 +392,3 @@ curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --data "domain [gfm]: http://docs.gitlab.com/ce/user/markdown.html#newlines "GitLab flavored markdown documentation" [ce-1242]: https://gitlab.com/gitlab-org/gitlab-ce/issues/1242 [doc-restart]: ../administration/restart_gitlab.md "GitLab restart documentation" -[ce-3349]: https://gitlab.com/gitlab-org/gitlab-ce/issues/3349 "Documentation restructure" -[graffle]: https://gitlab.com/gitlab-org/gitlab-design/blob/d8d39f4a87b90fb9ae89ca12dc565347b4900d5e/production/resources/gitlab-map.graffle -[gitlab-map]: https://gitlab.com/gitlab-org/gitlab-design/raw/master/production/resources/gitlab-map.png diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index f8cee89e650..3ba03d2d591 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -28,12 +28,65 @@ we still need to merge changes from GitLab CE to EE. To help us get there, we should make sure that we no longer edit CE files in place in order to implement EE features. -Instead, all EE codes should be put inside the `ee/` top-level directory, and -tests should be put inside `spec/ee/`. We don't use `ee/spec` for now due to -technical limitation. The rest of codes should be as close as to the CE files. +Instead, all EE code should be put inside the `ee/` top-level directory. The +rest of the code should be as close to the CE files as possible. [single code base]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2952#note_41016454 +### EE-specific comments + +When complete separation can't be achieved with the `ee/` directory, you can wrap +code in EE specific comments to designate the difference from CE/EE and add +some context for someone resolving a conflict. + +```rb +# EE-specific start +stub_licensed_features(variable_environment_scope: true) +# EE specific end +``` + +```haml +-# EE-specific start += render 'ci/variables/environment_scope', form_field: form_field, variable: variable +-# EE-specific end +``` + +EE-specific comments should not be backported to CE. + +### Detection of EE-only files + +For each commit (except on `master`), the `ee-files-location-check` CI job tries +to detect if there are any new files that are EE-only. If any file is detected, +the job fails with an explanation of why and what to do to make it pass. + +Basically, the fix is simple: `git mv <file> ee/<file>`. + +#### How to name your branches? + +For any EE branch, the job will try to detect its CE counterpart by removing any +`ee-` prefix or `-ee` suffix from the EE branch name, and matching the last +branch that contains it. + +For instance, from the EE branch `new-shiny-feature-ee` (or +`ee-new-shiny-feature`), the job would find the corresponding CE branches: + +- `new-shiny-feature` +- `ce-new-shiny-feature` +- `new-shiny-feature-ce` +- `my-super-new-shiny-feature-in-ce` + +#### Whitelist some EE-only files that cannot be moved to `ee/` + +The `ee-files-location-check` CI job provides a whitelist of files or folders +that cannot or should not be moved to `ee/`. Feel free to open an issue to +discuss adding a new file/folder to this whitelist. + +For instance, it was decided that moving EE-only files from `qa/` to `ee/qa/` +would make it difficult to build the `gitLab-{ce,ee}-qa` Docker images and it +was [not worth the complexity]. + +[not worth the complexity]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4997#note_59764702 + ### EE-only features If the feature being developed is not present in any form in CE, we don't @@ -53,6 +106,11 @@ is applied not only to models. Here's a list of other examples: - `ee/app/validators/foo_attr_validator.rb` - `ee/app/workers/foo_worker.rb` +This works because for every path that are present in CE's eager-load/auto-load +paths, we add the same `ee/`-prepended path in [`config/application.rb`]. + +[`config/application.rb`]: https://gitlab.com/gitlab-org/gitlab-ee/blob/d278b76d6600a0e27d8019a0be27971ba23ab640/config/application.rb#L41-51 + ### EE features based on CE features For features that build on existing CE features, write a module in the @@ -312,37 +370,274 @@ class beneath the `EE` module just as you would normally. For example, if CE has LDAP classes in `lib/gitlab/ldap/` then you would place EE-specific LDAP classes in `ee/lib/ee/gitlab/ldap`. +### Code in `lib/api/` + +It can be very tricky to extend EE features by a single line of `prepend`, +and for each different [Grape](https://github.com/ruby-grape/grape) feature, +we might need different strategies to extend it. To apply different strategies +easily, we would use `extend ActiveSupport::Concern` in the EE module. + +Put the EE module files following +[EE features based on CE features](#ee-features-based-on-ce-features). + +#### EE API routes + +For EE API routes, we put them in a `prepended` block: + +``` ruby +module EE + module API + module MergeRequests + extend ActiveSupport::Concern + + prepended do + params do + requires :id, type: String, desc: 'The ID of a project' + end + resource :projects, requirements: ::API::API::PROJECT_ENDPOINT_REQUIREMENTS do + # ... + end + end + end + end +end +``` + +Note that due to namespace differences, we need to use the full qualifier for some +constants. + +#### EE params + +We can define `params` and utilize `use` in another `params` definition to +include params defined in EE. However, we need to define the "interface" first +in CE in order for EE to override it. We don't have to do this in other places +due to `prepend`, but Grape is complex internally and we couldn't easily do +that, so we'll follow regular object-oriented practices that we define the +interface first here. + +For example, suppose we have a few more optional params for EE, given this CE +API code: + +``` ruby +module API + class MergeRequests < Grape::API + # EE::API::MergeRequests would override the following helpers + helpers do + params :optional_params_ee do + end + end + + prepend EE::API::MergeRequests + + params :optional_params do + # CE specific params go here... + + use :optional_params_ee + end + end +end +``` + +And then we could override it in EE module: + +``` ruby +module EE + module API + module MergeRequests + extend ActiveSupport::Concern + + prepended do + helpers do + params :optional_params_ee do + # EE specific params go here... + end + end + end + end + end +end +``` + +This way, the only difference between CE and EE for that API file would be +`prepend EE::API::MergeRequests`. + +#### EE helpers + +To make it easy for an EE module to override the CE helpers, we need to define +those helpers we want to extend first. Try to do that immediately after the +class definition to make it easy and clear: + +``` ruby +module API + class JobArtifacts < Grape::API + # EE::API::JobArtifacts would override the following helpers + helpers do + def authorize_download_artifacts! + authorize_read_builds! + end + end + + prepend EE::API::JobArtifacts + end +end +``` + +And then we can follow regular object-oriented practices to override it: + +``` ruby +module EE + module API + module JobArtifacts + extend ActiveSupport::Concern + + prepended do + helpers do + def authorize_download_artifacts! + super + check_cross_project_pipelines_feature! + end + end + end + end + end +end +``` + +#### EE-specific behaviour + +Sometimes we need EE-specific behaviour in some of the APIs. Normally we could +use EE methods to override CE methods, however API routes are not methods and +therefore can't be simply overridden. We need to extract them into a standalone +method, or introduce some "hooks" where we could inject behavior in the CE +route. Something like this: + +``` ruby +module API + class MergeRequests < Grape::API + helpers do + # EE::API::MergeRequests would override the following helpers + def update_merge_request_ee(merge_request) + end + end + + prepend EE::API::MergeRequests + + put ':id/merge_requests/:merge_request_iid/merge' do + merge_request = find_project_merge_request(params[:merge_request_iid]) + + # ... + + update_merge_request_ee(merge_request) + + # ... + end + end +end +``` + +Note that `update_merge_request_ee` doesn't do anything in CE, but +then we could override it in EE: + +``` ruby +module EE + module API + module MergeRequests + extend ActiveSupport::Concern + + prepended do + helpers do + def update_merge_request_ee(merge_request) + # ... + end + end + end + end + end +end +``` + +#### EE `route_setting` + +It's very hard to extend this in an EE module, and this is simply storing +some meta-data for a particular route. Given that, we could simply leave the +EE `route_setting` in CE as it won't hurt and we are just not going to use +those meta-data in CE. + +We could revisit this policy when we're using `route_setting` more and whether +or not we really need to extend it from EE. For now we're not using it much. + +#### Utilizing class methods for setting up EE-specific data + +Sometimes we need to use different arguments for a particular API route, and we +can't easily extend it with an EE module because Grape has different context in +different blocks. In order to overcome this, we could use class methods from the +API class. + +For example, in one place we need to pass an extra argument to +`at_least_one_of` so that the API could consider an EE-only argument as the +least argument. This is not quite beautiful but it's working: + +``` ruby +module API + class MergeRequests < Grape::API + def self.update_params_at_least_one_of + %i[ + assignee_id + description + ] + end + + prepend EE::API::MergeRequests + + params do + at_least_one_of(*::API::MergeRequests.update_params_at_least_one_of) + end + end +end +``` + +And then we could easily extend that argument in the EE class method: + +``` ruby +module EE + module API + module MergeRequests + extend ActiveSupport::Concern + + class_methods do + def update_params_at_least_one_of + super.push(*%i[ + squash + ]) + end + end + end + end +end +``` + +It could be annoying if we need this for a lot of routes, but it might be the +simplest solution right now. + ### Code in `spec/` When you're testing EE-only features, avoid adding examples to the existing CE specs. Also do no change existing CE examples, since they should remain working as-is when EE is running without a license. -Instead place EE specs in the `spec/ee/spec` folder. +Instead place EE specs in the `ee/spec` folder. ## JavaScript code in `assets/javascripts/` -To separate EE-specific JS-files we can also move the files into an `ee` folder. +To separate EE-specific JS-files we should also move the files into an `ee` folder. For example there can be an `app/assets/javascripts/protected_branches/protected_branches_bundle.js` and an EE counterpart `ee/app/assets/javascripts/protected_branches/protected_branches_bundle.js`. -That way we can create a separate webpack bundle in `webpack.config.js`: - -```javascript - protected_branches: '~/protected_branches', - ee_protected_branches: 'ee/protected_branches/protected_branches_bundle.js', -``` - -With the separate bundle in place, we can decide which bundle to load inside the -view, using the `page_specific_javascript_bundle_tag` helper. - -```haml -- content_for :page_specific_javascripts do - = page_specific_javascript_bundle_tag('protected_branches') -``` +See the frontend guide [performance section](./fe_guide/performance.md) for +information on managing page-specific javascript within EE. ## SCSS code in `assets/stylesheets` @@ -379,12 +674,13 @@ to avoid conflicts during CE to EE merge. } } -/* EE-specific styles */ +// EE-specific start .section-body.ee-section-body { .section-title { background: $gl-header-color-cyan; } } +// EE-specific end ``` ## gitlab-svgs diff --git a/doc/development/emails.md b/doc/development/emails.md index 18f47f44cb5..4dbf064fd75 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -18,6 +18,62 @@ See the [Rails guides] for more info. [previews]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/spec/mailers/previews [Rails guides]: http://guides.rubyonrails.org/action_mailer_basics.html#previewing-emails +## Incoming email + +1. Go to the GitLab installation directory. + +1. Find the `incoming_email` section in `config/gitlab.yml`, enable the + feature and fill in the details for your specific IMAP server and email + account: + + Configuration for Gmail / Google Apps, assumes mailbox gitlab-incoming@gmail.com + + ```yaml + incoming_email: + enabled: true + + # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. + # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + address: "gitlab-incoming+%{key}@gmail.com" + + # Email account username + # With third party providers, this is usually the full email address. + # With self-hosted email servers, this is usually the user part of the email address. + user: "gitlab-incoming@gmail.com" + # Email account password + password: "[REDACTED]" + + # IMAP server host + host: "imap.gmail.com" + # IMAP server port + port: 993 + # Whether the IMAP server uses SSL + ssl: true + # Whether the IMAP server uses StartTLS + start_tls: false + + # The mailbox where incoming mail will end up. Usually "inbox". + mailbox: "inbox" + # The IDLE command timeout. + idle_timeout: 60 + ``` + + As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. + +1. Run this command in the GitLab root directory to launch `mail_room`: + + ```sh + bundle exec mail_room -q -c config/mail_room.yml + ``` + +1. Verify that everything is configured correctly: + + ```sh + bundle exec rake gitlab:incoming_email:check RAILS_ENV=development + ``` + +1. Reply by email should now be working. + --- [Return to Development documentation](README.md) diff --git a/doc/development/fe_guide/components.md b/doc/development/fe_guide/components.md new file mode 100644 index 00000000000..66a8abe42f7 --- /dev/null +++ b/doc/development/fe_guide/components.md @@ -0,0 +1,61 @@ +# Components + +## Contents +* [Dropdowns](#dropdowns) +* [Modals](#modals) + +## Dropdowns + +See also the [corresponding UX guide](../ux_guide/components.md#dropdowns). + +### How to style a bootstrap dropdown +1. Use the HTML structure provided by the [docs][bootstrap-dropdowns] +1. Add a specific class to the top level `.dropdown` element + + + ```Haml + .dropdown.my-dropdown + %button{ type: 'button', data: { toggle: 'dropdown' }, 'aria-haspopup': true, 'aria-expanded': false } + %span.dropdown-toggle-text + Toggle Dropdown + = icon('chevron-down') + + %ul.dropdown-menu + %li + %a + item! + ``` + + Or use the helpers + ```Haml + .dropdown.my-dropdown + = dropdown_toggle('Toogle!', { toggle: 'dropdown' }) + = dropdown_content + %li + %a + item! + ``` + +[bootstrap-dropdowns]: https://getbootstrap.com/docs/3.3/javascript/#dropdowns + +## Modals + +See also the [corresponding UX guide](../ux_guide/components.md#modals). + +We have a reusable Vue component for modals: [vue_shared/components/gl-modal.vue](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/vue_shared/components/gl-modal.vue) + +Here is an example of how to use it: + +```html + <gl-modal + id="dogs-out-modal" + :header-title-text="s__('ModalExample|Let the dogs out?')" + footer-primary-button-variant="danger" + :footer-primary-button-text="s__('ModalExample|Let them out')" + @submit="letOut(theDogs)" + > + {{ s__('ModalExample|You’re about to let the dogs out.') }} + </gl-modal> +``` + + diff --git a/doc/development/fe_guide/dropdowns.md b/doc/development/fe_guide/dropdowns.md index 6314f8f38d2..e9d6244355c 100644 --- a/doc/development/fe_guide/dropdowns.md +++ b/doc/development/fe_guide/dropdowns.md @@ -1,32 +1 @@ -# Dropdowns - - -## How to style a bootstrap dropdown -1. Use the HTML structure provided by the [docs][bootstrap-dropdowns] -1. Add a specific class to the top level `.dropdown` element - - - ```Haml - .dropdown.my-dropdown - %button{ type: 'button', data: { toggle: 'dropdown' }, 'aria-haspopup': true, 'aria-expanded': false } - %span.dropdown-toggle-text - Toggle Dropdown - = icon('chevron-down') - - %ul.dropdown-menu - %li - %a - item! - ``` - - Or use the helpers - ```Haml - .dropdown.my-dropdown - = dropdown_toggle('Toogle!', { toggle: 'dropdown' }) - = dropdown_content - %li - %a - item! - ``` - -[bootstrap-dropdowns]: https://getbootstrap.com/docs/3.3/javascript/#dropdowns +This page has moved [here](components.md#dropdowns). diff --git a/doc/development/fe_guide/img/gl-modal.png b/doc/development/fe_guide/img/gl-modal.png Binary files differnew file mode 100644 index 00000000000..47302e857bc --- /dev/null +++ b/doc/development/fe_guide/img/gl-modal.png diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 72cb557d054..2280cf79f86 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -14,13 +14,15 @@ support through [webpack][webpack]. We also utilize [webpack][webpack] to handle the bundling, minification, and compression of our assets. -Working with our frontend assets requires Node (v4.3 or greater) and Yarn -(v0.17 or greater). You can find information on how to install these on our +Working with our frontend assets requires Node (v6.0 or greater) and Yarn +(v1.2 or greater). You can find information on how to install these on our [installation guide][install]. [jQuery][jquery] is used throughout the application's JavaScript, with [Vue.js][vue] for particularly advanced, dynamic elements. +We also use [Axios][axios] to handle all of our network requests. + ### Browser Support For our currently-supported browsers, see our [requirements][requirements]. @@ -77,8 +79,10 @@ Axios specific practices and gotchas. ## [Icons](icons.md) How we use SVG for our Icons. -## [Dropdowns](dropdowns.md) -How we use dropdowns. +## [Components](components.md) + +How we use UI components. + --- ## Style Guides @@ -122,6 +126,7 @@ The [externalization part of the guide](../i18n/externalization.md) explains the [webpack]: https://webpack.js.org/ [jquery]: https://jquery.com/ [vue]: http://vuejs.org/ +[axios]: https://github.com/axios/axios [airbnb-js-style-guide]: https://github.com/airbnb/javascript [scss-lint]: https://github.com/brigade/scss-lint [install]: ../../install/installation.md#4-node diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 14ac1133cc0..1320efaf767 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -23,7 +23,7 @@ controlled by the server. 1. The backend code will most likely be using etags. You do not and should not check for status `304 Not Modified`. The browser will transform it for you. -### Lazy Loading +### Lazy Loading Images To improve the time to first render we are using lazy loading for images. This works by setting the actual image source on the `data-src` attribute. After the HTML is rendered and JavaScript is loaded, @@ -36,43 +36,114 @@ If you are asynchronously adding content which contains lazy images then you nee `gl.lazyLoader.searchLazyImages()` which will search for lazy images and load them if needed. But in general it should be handled automatically through a `MutationObserver` in the lazy loading function. +### Animations + +Only animate `opacity` & `transform` properties. Other properties (such as `top`, `left`, `margin`, and `padding`) all cause +Layout to be recalculated, which is much more expensive. For details on this, see "Styles that Affect Layout" in +[High Performance Animations][high-perf-animations]. + +If you _do_ need to change layout (e.g. a sidebar that pushes main content over), prefer [FLIP][flip] to change expensive +properties once, and handle the actual animation with transforms. + ## Reducing Asset Footprint -### Page-specific JavaScript +### Universal code -Certain pages may require the use of a third party library, such as [d3][d3] for -the User Activity Calendar and [Chart.js][chartjs] for the Graphs pages. These -libraries increase the page size significantly, and impact load times due to -bandwidth bottlenecks and the browser needing to parse more JavaScript. - -In cases where libraries are only used on a few specific pages, we use -"page-specific JavaScript" to prevent the main `main.js` file from -becoming unnecessarily large. - -Steps to split page-specific JavaScript from the main `main.js`: - -1. Create a directory for the specific page(s), e.g. `graphs/`. -1. In that directory, create a `namespace_bundle.js` file, e.g. `graphs_bundle.js`. -1. Add the new "bundle" file to the list of entry files in `config/webpack.config.js`. - - For example: `graphs: './graphs/graphs_bundle.js',`. -1. Move code reliant on these libraries into the `graphs` directory. -1. In `graphs_bundle.js` add CommonJS `require('./path_to_some_component.js');` statements to load any other files in this directory. Make sure to use relative urls. -1. In the relevant views, add the scripts to the page with the following: - -```haml -- content_for :page_specific_javascripts do - = webpack_bundle_tag 'lib_chart' - = webpack_bundle_tag 'graphs' -``` +Code that is contained within `main.js` and `commons/index.js` are loaded and +run on _all_ pages. **DO NOT ADD** anything to these files unless it is truly +needed _everywhere_. These bundles include ubiquitous libraries like `vue`, +`axios`, and `jQuery`, as well as code for the main navigation and sidebar. +Where possible we should aim to remove modules from these bundles to reduce our +code footprint. -The above loads `chart.js` and `graphs_bundle.js` for this page only. `chart.js` -is separated from the bundle file so it can be cached separately from the bundle -and reused for other pages that also rely on the library. For an example, see -[this Haml file][page-specific-js-example]. +### Page-specific JavaScript + +Webpack has been configured to automatically generate entry point bundles based +on the file structure within `app/assets/javascripts/pages/*`. The directories +within the `pages` directory correspond to Rails controllers and actions. These +auto-generated bundles will be automatically included on the corresponding +pages. + +For example, if you were to visit [gitlab.com/gitlab-org/gitlab-ce/issues](https://gitlab.com/gitlab-org/gitlab-ce/issues), +you would be accessing the `app/controllers/projects/issues_controller.rb` +controller with the `index` action. If a corresponding file exists at +`pages/projects/issues/index/index.js`, it will be compiled into a webpack +bundle and included on the page. + +> **Note:** Previously we had encouraged the use of +> `content_for :page_specific_javascripts` within haml files, along with +> manually generated webpack bundles. However under this new system you should +> not ever need to manually add an entry point to the `webpack.config.js` file. + +> **Tip:** +> If you are unsure what controller and action corresponds to a given page, you +> can find this out by inspecting `document.body.dataset.page` within your +> browser's developer console while on any page within gitlab. + +#### Important Considerations: + +- **Keep Entry Points Lite:** + Page-specific javascript entry points should be as lite as possible. These + files are exempt from unit tests, and should be used primarily for + instantiation and dependency injection of classes and methods that live in + modules outside of the entry point script. Just import, read the DOM, + instantiate, and nothing else. + +- **Entry Points May Be Asynchronous:** + _DO NOT ASSUME_ that the DOM has been fully loaded and available when an + entry point script is run. If you require that some code be run after the + DOM has loaded, you should attach an event handler to the `DOMContentLoaded` + event with: + + ```javascript + import initMyWidget from './my_widget'; + + document.addEventListener('DOMContentLoaded', () => { + initMyWidget(); + }); + ``` + +- **Supporting Module Placement:** + - If a class or a module is _specific to a particular route_, try to locate + it close to the entry point it will be used. For instance, if + `my_widget.js` is only imported within `pages/widget/show/index.js`, you + should place the module at `pages/widget/show/my_widget.js` and import it + with a relative path (e.g. `import initMyWidget from './my_widget';`). + + - If a class or module is _used by multiple routes_, place it within a + shared directory at the closest common parent directory for the entry + points that import it. For example, if `my_widget.js` is imported within + both `pages/widget/show/index.js` and `pages/widget/run/index.js`, then + place the module at `pages/widget/shared/my_widget.js` and import it with + a relative path if possible (e.g. `../shared/my_widget`). + +- **Enterprise Edition Caveats:** + For GitLab Enterprise Edition, page-specific entry points will override their + Community Edition counterparts with the same name, so if + `ee/app/assets/javascripts/pages/foo/bar/index.js` exists, it will take + precedence over `app/assets/javascripts/pages/foo/bar/index.js`. If you want + to minimize duplicate code, you can import one entry point from the other. + This is not done automatically to allow for flexibility in overriding + functionality. ### Code Splitting -> *TODO* flesh out this section once webpack is ready for code-splitting +For any code that does not need to be run immediately upon page load, (e.g. +modals, dropdowns, and other behaviors that can be lazy-loaded), you can split +your module into asynchronous chunks with dynamic import statements. These +imports return a Promise which will be resolved once the script has loaded: + +```javascript +import(/* webpackChunkName: 'emoji' */ '~/emoji') + .then(/* do something */) + .catch(/* report error */) +``` + +Please try to use `webpackChunkName` when generating these dynamic imports as +it will provide a deterministic filename for the chunk which can then be cached +the browser across GitLab versions. + +More information is available in [webpack's code splitting documentation](https://webpack.js.org/guides/code-splitting/#dynamic-imports). ### Minimizing page size @@ -86,7 +157,9 @@ General tips: - Prefer font formats with better compression, e.g. WOFF2 is better than WOFF, which is better than TTF. - Compress and minify assets wherever possible (For CSS/JS, Sprockets and webpack do this for us). - If some functionality can reasonably be achieved without adding extra libraries, avoid them. -- Use page-specific JavaScript as described above to dynamically load libraries that are only needed on certain pages. +- Use page-specific JavaScript as described above to load libraries that are only needed on certain pages. +- Use code-splitting dynamic imports wherever possible to lazy-load code that is not needed initially. +- [High Performance Animations][high-perf-animations] ------- @@ -102,6 +175,5 @@ General tips: [pagespeed-insights]: https://developers.google.com/speed/pagespeed/insights/ [google-devtools-profiling]: https://developers.google.com/web/tools/chrome-devtools/profile/?hl=en [browser-diet]: https://browserdiet.com/ -[d3]: https://d3js.org/ -[chartjs]: http://www.chartjs.org/ -[page-specific-js-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/13bb9ed77f405c5f6ee4fdbc964ecf635c9a223f/app/views/projects/graphs/_head.html.haml#L6-8 +[high-perf-animations]: https://www.html5rocks.com/en/tutorials/speed/high-performance-animations/ +[flip]: https://aerotwist.com/blog/flip-your-animations/ diff --git a/doc/development/fe_guide/style_guide_js.md b/doc/development/fe_guide/style_guide_js.md index 02773162801..7b5fa6ca42f 100644 --- a/doc/development/fe_guide/style_guide_js.md +++ b/doc/development/fe_guide/style_guide_js.md @@ -207,10 +207,39 @@ Do not use them anymore and feel free to remove them when refactoring legacy cod var c = pureFunction(values.foo); ``` -1. Avoid constructors with side-effects +1. Avoid constructors with side-effects. +Although we aim for code without side-effects we need some side-effects for our code to run. + +If the class won't do anything if we only instantiate it, it's ok to add side effects into the constructor (_Note:_ The following is just an example. If the only purpose of the class is to add an event listener and handle the callback a function will be more suitable.) + +```javascript +// Bad +export class Foo { + constructor() { + this.init(); + } + init() { + document.addEventListener('click', this.handleCallback) + }, + handleCallback() { + + } +} + +// Good +export class Foo { + constructor() { + document.addEventListener() + } + handleCallback() { + } +} +``` + +On the other hand, if a class only needs to extend a third party/add event listeners in some specific cases, they should be initialized oustside of the constructor. 1. Prefer `.map`, `.reduce` or `.filter` over `.forEach` -A forEach will cause side effects, it will be mutating the array being iterated. Prefer using `.map`, +A forEach will most likely cause side effects, it will be mutating the array being iterated. Prefer using `.map`, `.reduce` or `.filter` ```javascript const users = [ { name: 'Foo' }, { name: 'Bar' } ]; @@ -302,20 +331,20 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. #### Naming 1. **Extensions**: Use `.vue` extension for Vue components. -1. **Reference Naming**: Use camelCase for their instances: +1. **Reference Naming**: Use PascalCase for their instances: ```javascript // bad - import CardBoard from 'cardBoard' + import cardBoard from 'cardBoard.vue' components: { - CardBoard: + cardBoard, }; // good - import cardBoard from 'cardBoard' + import CardBoard from 'cardBoard.vue' components: { - cardBoard: + CardBoard, }; ``` @@ -519,6 +548,57 @@ On those a default key should not be provided. 1. Properties in a Vue Component: Check [order of properties in components rule][vue-order]. +#### `:key` +When using `v-for` you need to provide a *unique* `:key` attribute for each item. + +1. If the elements of the array being iterated have an unique `id` it is advised to use it: + ```html + <div + v-for="item in items" + :key="item.id" + > + <!-- content --> + </div> + ``` + +1. When the elements being iterated don't have a unique id, you can use the array index as the `:key` attribute + ```html + <div + v-for="(item, index) in items" + :key="index" + > + <!-- content --> + </div> + ``` + + +1. When using `v-for` with `template` and there is more than one child element, the `:key` values must be unique. It's advised to use `kebab-case` namespaces. + ```html + <template v-for="(item, index) in items"> + <span :key="`span-${index}`"></span> + <button :key="`button-${index}`"></button> + </template> + ``` + +1. When dealing with nested `v-for` use the same guidelines as above. + ```html + <div + v-for="item in items" + :key="item.id" + > + <span + v-for="element in array" + :key="element.id" + > + <!-- content --> + </span> + </div> + ``` + + +Useful links: +1. [`key`](https://vuejs.org/v2/guide/list.html#key) +1. [Vue Style Guide: Keyed v-for](https://vuejs.org/v2/style-guide/#Keyed-v-for-essential ) #### Vue and Bootstrap 1. Tooltips: Do not rely on `has-tooltip` class name for Vue components diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md index 86a8b4135af..655d94793dd 100644 --- a/doc/development/fe_guide/style_guide_scss.md +++ b/doc/development/fe_guide/style_guide_scss.md @@ -7,6 +7,8 @@ easy to maintain, and performant for the end-user. ### Naming +Filenames should use `snake_case`. + CSS classes should use the `lowercase-hyphenated` format rather than `snake_case` or `camelCase`. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 6c93c29124d..c1170fa3b13 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -53,13 +53,13 @@ you can find a clear separation of concerns: ``` new_feature ├── components -│ └── component.js.es6 +│ └── component.vue │ └── ... -├── store -│ └── new_feature_store.js.es6 -├── service -│ └── new_feature_service.js.es6 -├── new_feature_bundle.js.es6 +├── stores +│ └── new_feature_store.js +├── services +│ └── new_feature_service.js +├── new_feature_bundle.js ``` _For consistency purposes, we recommend you to follow the same structure._ @@ -507,13 +507,15 @@ This is the entry point for our store. You can use the following as a guide: import Vue from 'vue'; import Vuex from 'vuex'; import * as actions from './actions'; -import * as mutations from './mutations'; +import * as getters from './getters'; +import mutations from './mutations'; Vue.use(Vuex); export default new Vuex.Store({ actions, getters, + mutations, state: { users: [], }, @@ -525,7 +527,7 @@ _Note:_ If the state of the application is too complex, an individual file for t An action commits a mutatation. In this file, we will write the actions that will call the respective mutation: ```javascript - import * as types from './mutation-types' + import * as types from './mutation_types'; export const addUser = ({ commit }, user) => { commit(types.ADD_USER, user); @@ -575,7 +577,8 @@ import { mapGetters } from 'vuex'; The only way to actually change state in a Vuex store is by committing a mutation. ```javascript - import * as types from './mutation-types' + import * as types from './mutation_types'; + export default { [types.ADD_USER](state, user) { state.users.push(user); @@ -684,4 +687,3 @@ describe('component', () => { [vuex-testing]: https://vuex.vuejs.org/en/testing.html [axios]: https://github.com/axios/axios [axios-interceptors]: https://github.com/axios/axios#interceptors - diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 59e8a087e02..5d1f657015c 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -1,6 +1,6 @@ # Manage feature flags -Starting from GitLab 9.3 we support feature flags via +Starting from GitLab 9.3 we support feature flags for features in GitLab via [Flipper](https://github.com/jnunemaker/flipper/). You should use the `Feature` class (defined in `lib/feature.rb`) in your code to get, set and list feature flags. @@ -19,3 +19,8 @@ dynamic (querying the DB etc.). Once defined in `lib/feature.rb`, you will be able to activate a feature for a given feature group via the [`feature_group` param of the features API](../api/features.md#set-or-create-a-feature) + +## Feature flags for user applications + +GitLab does not yet support the use of feature flags in deployed user applications. +You can follow the progress on that [in the issue on our issue tracker](https://gitlab.com/gitlab-org/gitlab-ee/issues/779).
\ No newline at end of file diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index cf00e24e11a..34a02bd2c3c 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -14,9 +14,9 @@ There are many places where file uploading is used, according to contexts: - User snippet attachments * Project - Project avatars - - Issues/MR Markdown attachments - - Issues/MR Legacy Markdown attachments - - CI Build Artifacts + - Issues/MR/Notes Markdown attachments + - Issues/MR/Notes Legacy Markdown attachments + - CI Artifacts (archive, metadata, trace) - LFS Objects @@ -25,7 +25,7 @@ There are many places where file uploading is used, according to contexts: GitLab started saving everything on local disk. While directory location changed from previous versions, they are still not 100% standardized. You can see them below: -| Description | In DB? | Relative path | Uploader class | model_type | +| Description | In DB? | Relative path (from CarrierWave.root) | Uploader class | model_type | | ------------------------------------- | ------ | ----------------------------------------------------------- | ---------------------- | ---------- | | Instance logo | yes | uploads/-/system/appearance/logo/:id/:filename | `AttachmentUploader` | Appearance | | Header logo | yes | uploads/-/system/appearance/header_logo/:id/:filename | `AttachmentUploader` | Appearance | @@ -33,17 +33,107 @@ they are still not 100% standardized. You can see them below: | User avatars | yes | uploads/-/system/user/avatar/:id/:filename | `AvatarUploader` | User | | User snippet attachments | yes | uploads/-/system/personal_snippet/:id/:random_hex/:filename | `PersonalFileUploader` | Snippet | | Project avatars | yes | uploads/-/system/project/avatar/:id/:filename | `AvatarUploader` | Project | -| Issues/MR Markdown attachments | yes | uploads/:project_path_with_namespace/:random_hex/:filename | `FileUploader` | Project | -| Issues/MR Legacy Markdown attachments | no | uploads/-/system/note/attachment/:id/:filename | `AttachmentUploader` | Note | -| CI Artifacts (CE) | yes | shared/artifacts/:year_:month/:project_id/:id | `ArtifactUploader` | Ci::Build | +| Issues/MR/Notes Markdown attachments | yes | uploads/:project_path_with_namespace/:random_hex/:filename | `FileUploader` | Project | +| Issues/MR/Notes Legacy Markdown attachments | no | uploads/-/system/note/attachment/:id/:filename | `AttachmentUploader` | Note | +| CI Artifacts (CE) | yes | shared/artifacts/:disk_hash[0..1]/:disk_hash[2..3]/:disk_hash/:year_:month_:date/:job_id/:job_artifact_id (:disk_hash is SHA256 digest of project_id) | `JobArtifactUploader` | Ci::JobArtifact | | LFS Objects (CE) | yes | shared/lfs-objects/:hex/:hex/:object_hash | `LfsObjectUploader` | LfsObject | CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader` -while in EE they inherit the `ObjectStoreUploader` and store files in and S3 API compatible object store. +while in EE they inherit the `ObjectStorage` and store files in and S3 API compatible object store. -In the case of Issues/MR Markdown attachments, there is a different approach using the [Hashed Storage] layout, +In the case of Issues/MR/Notes Markdown attachments, there is a different approach using the [Hashed Storage] layout, instead of basing the path into a mutable variable `:project_path_with_namespace`, it's possible to use the hash of the project ID instead, if project migrates to the new approach (introduced in 10.2). +### Path segments + +Files are stored at multiple locations and use different path schemes. +All the `GitlabUploader` derived classes should comply with this path segment schema: + +``` +| GitlabUploader +| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- | +| `<gitlab_root>/public/` | `uploads/-/system/` | `user/avatar/:id/` | `:filename` | +| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- | +| `CarrierWave.root` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` | +| | `CarrierWave::Uploader#store_dir` | | + +| FileUploader +| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- | +| `<gitlab_root>/shared/` | `artifacts/` | `:year_:month/:id` | `:filename` | +| `<gitlab_root>/shared/` | `snippets/` | `:secret/` | `:filename` | +| ----------------------- + ------------------------- + --------------------------------- + -------------------------------- | +| `CarrierWave.root` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` | +| | `CarrierWave::Uploader#store_dir` | | +| | | `FileUploader#upload_path | + +| ObjectStore::Concern (store = remote) +| ----------------------- + ------------------------- + ----------------------------------- + -------------------------------- | +| `<bucket_name>` | <ignored> | `user/avatar/:id/` | `:filename` | +| ----------------------- + ------------------------- + ----------------------------------- + -------------------------------- | +| `#fog_dir` | `GitlabUploader.base_dir` | `GitlabUploader#dynamic_segment` | `CarrierWave::Uploader#filename` | +| | | `ObjectStorage::Concern#store_dir` | | +| | | `ObjectStorage::Concern#upload_path | +``` + +The `RecordsUploads::Concern` concern will create an `Upload` entry for every file stored by a `GitlabUploader` persisting the dynamic parts of the path using +`GitlabUploader#dynamic_path`. You may then use the `Upload#build_uploader` method to manipulate the file. + +## Object Storage + +By including the `ObjectStorage::Concern` in the `GitlabUploader` derived class, you may enable the object storage for this uploader. To enable the object storage +in your uploader, you need to either 1) include `RecordsUpload::Concern` and prepend `ObjectStorage::Extension::RecordsUploads` or 2) mount the uploader and create a new field named `<mount>_store`. + +The `CarrierWave::Uploader#store_dir` is overriden to + + - `GitlabUploader.base_dir` + `GitlabUploader.dynamic_segment` when the store is LOCAL + - `GitlabUploader.dynamic_segment` when the store is REMOTE (the bucket name is used to namespace) + +### Using `ObjectStorage::Extension::RecordsUploads` + +> Note: this concern will automatically include `RecordsUploads::Concern` if not already included. + +The `ObjectStorage::Concern` uploader will search for the matching `Upload` to select the correct object store. The `Upload` is mapped using `#store_dirs + identifier` for each store (LOCAL/REMOTE). + +```ruby +class SongUploader < GitlabUploader + include RecordsUploads::Concern + include ObjectStorage::Concern + prepend ObjectStorage::Extension::RecordsUploads + + ... +end + +class Thing < ActiveRecord::Base + mount :theme, SongUploader # we have a great theme song! + + ... +end +``` + +### Using a mounted uploader + +The `ObjectStorage::Concern` will query the `model.<mount>_store` attribute to select the correct object store. +This column must be present in the model schema. + +```ruby +class SongUploader < GitlabUploader + include ObjectStorage::Concern + + ... +end + +class Thing < ActiveRecord::Base + attr_reader :theme_store # this is an ActiveRecord attribute + mount :theme, SongUploader # we have a great theme song! + + def theme_store + super || ObjectStorage::Store::LOCAL + end + + ... +end +``` + [CarrierWave]: https://github.com/carrierwaveuploader/carrierwave [Hashed Storage]: ../administration/repository_storage_types.md diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index f493ad4ae66..856ef882453 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -45,7 +45,7 @@ We basically have 4 types of files: 1. Ruby files: basically Models and Controllers. 1. HAML files: these are the view files. 1. ERB files: used for email templates. -1. JavaScript files: we mostly need to work with VUE JS templates. +1. JavaScript files: we mostly need to work with Vue templates. ### Ruby files @@ -107,99 +107,28 @@ You can mark that content for translation with: ### JavaScript files -In JavaScript we added the `__()` (double underscore parenthesis) function -for translations. +In JavaScript we added the `__()` (double underscore parenthesis) function that +you can import from the `~/locale` file. For instance: -## Updating the PO files with the new content - -Now that the new content is marked for translation, we need to update the PO -files with the following command: - -```sh -bundle exec rake gettext:find -``` - -This command will update the `locale/gitlab.pot` file with the newly externalized -strings and remove any strings that aren't used anymore. You should check this -file in. Once the changes are on master, they will be picked up by -[Crowdin](http://translate.gitlab.com) and be presented for translation. - -The command also updates the translation files for each language: `locale/*/gitlab.po` -These changes can be discarded, the languange files will be updated by Crowdin -automatically. - -Discard all of them at once like this: - -```sh -git checkout locale/*/gitlab.po -``` - -### Validating PO files - -To make sure we keep our translation files up to date, there's a linter that is -running on CI as part of the `static-analysis` job. - -To lint the adjustments in PO files locally you can run `rake gettext:lint`. - -The linter will take the following into account: - -- Valid PO-file syntax -- Variable usage - - Only one unnamed (`%d`) variable, since the order of variables might change - in different languages - - All variables used in the message-id are used in the translation - - There should be no variables used in a translation that aren't in the - message-id -- Errors during translation. - -The errors are grouped per file, and per message ID: - -``` -Errors in `locale/zh_HK/gitlab.po`: - PO-syntax errors - SimplePoParser::ParserErrorSyntax error in lines - Syntax error in msgctxt - Syntax error in msgid - Syntax error in msgstr - Syntax error in message_line - There should be only whitespace until the end of line after the double quote character of a message text. - Parseing result before error: '{:msgid=>["", "You are going to remove %{project_name_with_namespace}.\\n", "Removed project CANNOT be restored!\\n", "Are you ABSOLUTELY sure?"]}' - SimplePoParser filtered backtrace: SimplePoParser::ParserError -Errors in `locale/zh_TW/gitlab.po`: - 1 pipeline - <%d 條流水線> is using unknown variables: [%d] - Failure translating to zh_TW with []: too few arguments +```js +import { __ } from '~/locale'; +const label = __('Subscribe'); ``` -In this output the `locale/zh_HK/gitlab.po` has syntax errors. -The `locale/zh_TW/gitlab.po` has variables that are used in the translation that -aren't in the message with id `1 pipeline`. - -## Working with special content - - -### Just marking content for parsing - -- In Ruby/HAML: - - ```ruby - _('Subscribe') - ``` - -- In JavaScript: - - ```js - import { __ } from '../../../locale'; - const label = __('Subscribe'); - ``` +In order to test JavaScript translations you have to change the GitLab +localization to other language than English and you have to generate JSON files +using `bin/rake gettext:po_to_json` or `bin/rake gettext:compile`. +### Dynamic translations Sometimes there are some dynamic translations that can't be found by the -parser when running `bundle exec rake gettext:find`. For these scenarios you can -use the [`_N` method](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#unfound-translations-with-rake-gettextfind). +parser when running `bin/rake gettext:find`. For these scenarios you can +use the [`N_` method](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#unfound-translations-with-rake-gettextfind). There is also and alternative method to [translate messages from validation errors](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#option-a). +## Working with special content + ### Interpolation - In Ruby/HAML: @@ -211,7 +140,7 @@ There is also and alternative method to [translate messages from validation erro - In JavaScript: ```js - import { __, sprintf } from '../../../locale'; + import { __, sprintf } from '~/locale'; sprintf(__('Hello %{username}'), { username: 'Joe' }) => 'Hello Joe' ``` @@ -223,24 +152,30 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. - In Ruby/HAML: ```ruby - n_('Apple', 'Apples', 3) => 'Apples' + n_('Apple', 'Apples', 3) + # => 'Apples' ``` Using interpolation: ```ruby n_("There is a mouse.", "There are %d mice.", size) % size + # => When size == 1: 'There is a mouse.' + # => When size == 2: 'There are 2 mice.' ``` - In JavaScript: ```js - n__('Apple', 'Apples', 3) => 'Apples' + n__('Apple', 'Apples', 3) + // => 'Apples' ``` Using interpolation: ```js - n__('Last day', 'Last %d days', 30) => 'Last 30 days' + n__('Last day', 'Last %d days', x) + // => When x == 1: 'Last day' + // => When x == 2: 'Last 2 days' ``` ### Namespaces @@ -262,12 +197,15 @@ Sometimes you need to add some context to the text that you want to translate s__('OpenedNDaysAgo|Opened') ``` +Note: The namespace should be removed from the translation. See the [translation +guidelines for more details](./translation.md#namespaced-strings). + ### Dates / times - In JavaScript: ```js -import { createDateTimeFormat } from '.../locale'; +import { createDateTimeFormat } from '~/locale'; const dateFormat = createDateTimeFormat({ year: 'numeric', month: 'long', day: 'numeric' }); console.log(dateFormat.format(new Date('2063-04-05'))) // April 5, 2063 @@ -277,6 +215,100 @@ This makes use of [`Intl.DateTimeFormat`]. [`Intl.DateTimeFormat`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat +## Best practices + +### Splitting sentences + +Please never split a sentence as that would assume the sentence grammar and +structure is the same in all languages. + +For instance, the following + +```js +{{ s__("mrWidget|Set by") }} +{{ author.name }} +{{ s__("mrWidget|to be merged automatically when the pipeline succeeds") }} +``` + +should be externalized as follows: + +```js +{{ sprintf(s__("mrWidget|Set by %{author} to be merged automatically when the pipeline succeeds"), { author: author.name }) }} +``` + +When in doubt, try to follow the best practices described in this [Mozilla +Developer documentation][mdn]. + +[mdn]: https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_content_best_practices#Splitting + +## Updating the PO files with the new content + +Now that the new content is marked for translation, we need to update the PO +files with the following command: + +```sh +bin/rake gettext:find +``` + +This command will update the `locale/gitlab.pot` file with the newly externalized +strings and remove any strings that aren't used anymore. You should check this +file in. Once the changes are on master, they will be picked up by +[Crowdin](http://translate.gitlab.com) and be presented for translation. + +If there are merge conflicts in the `gitlab.pot` file, you can delete the file +and regenerate it using the same command. Confirm that you are not deleting any strings accidentally by looking over the diff. + +The command also updates the translation files for each language: `locale/*/gitlab.po` +These changes can be discarded, the languange files will be updated by Crowdin +automatically. + +Discard all of them at once like this: + +```sh +git checkout locale/*/gitlab.po +``` + +### Validating PO files + +To make sure we keep our translation files up to date, there's a linter that is +running on CI as part of the `static-analysis` job. + +To lint the adjustments in PO files locally you can run `rake gettext:lint`. + +The linter will take the following into account: + +- Valid PO-file syntax +- Variable usage + - Only one unnamed (`%d`) variable, since the order of variables might change + in different languages + - All variables used in the message-id are used in the translation + - There should be no variables used in a translation that aren't in the + message-id +- Errors during translation. + +The errors are grouped per file, and per message ID: + +``` +Errors in `locale/zh_HK/gitlab.po`: + PO-syntax errors + SimplePoParser::ParserErrorSyntax error in lines + Syntax error in msgctxt + Syntax error in msgid + Syntax error in msgstr + Syntax error in message_line + There should be only whitespace until the end of line after the double quote character of a message text. + Parseing result before error: '{:msgid=>["", "You are going to remove %{project_name_with_namespace}.\\n", "Removed project CANNOT be restored!\\n", "Are you ABSOLUTELY sure?"]}' + SimplePoParser filtered backtrace: SimplePoParser::ParserError +Errors in `locale/zh_TW/gitlab.po`: + 1 pipeline + <%d 條流水線> is using unknown variables: [%d] + Failure translating to zh_TW with []: too few arguments +``` + +In this output the `locale/zh_HK/gitlab.po` has syntax errors. +The `locale/zh_TW/gitlab.po` has variables that are used in the translation that +aren't in the message with id `1 pipeline`. + ## Adding a new language Let's suppose you want to add translations for a new language, let's say French. @@ -295,14 +327,14 @@ Let's suppose you want to add translations for a new language, let's say French. 1. Next, you need to add the language: ```sh - bundle exec rake gettext:add_language[fr] + bin/rake gettext:add_language[fr] ``` If you want to add a new language for a specific region, the command is similar, you just need to separate the region with an underscore (`_`). For example: ```sh - bundle exec rake gettext:add_language[en_GB] + bin/rake gettext:add_language[en_GB] ``` Please note that you need to specify the region part in capitals. @@ -316,7 +348,7 @@ Let's suppose you want to add translations for a new language, let's say French. containing the translations: ```sh - bundle exec rake gettext:compile + bin/rake gettext:compile ``` 1. In order to see the translated content we need to change our preferred language diff --git a/doc/development/i18n/index.md b/doc/development/i18n/index.md index 8aa0462d213..7290a175501 100644 --- a/doc/development/i18n/index.md +++ b/doc/development/i18n/index.md @@ -40,37 +40,12 @@ See [Translation guidelines](translation.md). ### Proof reading -Proof reading helps ensure the accuracy and consistency of translations. -All translations are proof read before being accepted. -If a translations requires changes, you will be notified with a comment explaining why. - -Community assistance proof reading translations is encouraged and appreciated. -Requests to become a proof reader will be considered on the merits of previous translations. - -- Bulgarian -- Chinese Simplified - - [Huang Tao](https://crowdin.com/profile/htve) -- Chinese Traditional - - [Huang Tao](https://crowdin.com/profile/htve) -- Chinese Traditional, Hong Kong - - [Huang Tao](https://crowdin.com/profile/htve) -- Dutch -- Esperanto -- French -- German -- Italian - - [Paolo Falomo](https://crowdin.com/profile/paolo.falomo) -- Japanese -- Korean - - [Huang Tao](https://crowdin.com/profile/htve) -- Portuguese, Brazilian -- Russian - - [Alexy Lustin](https://crowdin.com/profile/lustin) - - [Nikita Grylov](https://crowdin.com/profile/nixel2007) -- Spanish -- Ukrainian - -If you would like to be added as a proof reader, please [open an issue](https://gitlab.com/gitlab-org/gitlab-ce/issues). +Proof reading helps ensure the accuracy and consistency of translations. All +translations are proof read before being accepted. If a translations requires +changes, you will be notified with a comment explaining why. + +See [Proofreading Translations](proofreader.md) for more information on who's +able to proofread and instructions on becoming a proofreader yourself. ## Release diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md new file mode 100644 index 00000000000..cf62314bc29 --- /dev/null +++ b/doc/development/i18n/proofreader.md @@ -0,0 +1,74 @@ +# Proofread Translations + +Most translations are contributed, reviewed, and accepted by the community. We +are very appreciative of the work done by translators and proofreaders! + +## Proofreaders + +- Bulgarian +- Chinese Simplified + - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) +- Chinese Traditional + - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) + - Weizhe Ding - [GitLab](https://gitlab.com/d.weizhe), [Crowdin](https://crowdin.com/profile/d.weizhe) +- Chinese Traditional, Hong Kong + - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) +- Dutch +- Esperanto +- French + - Rémy Coutable - [GitLab](https://gitlab.com/rymai), [Crowdin](https://crowdin.com/profile/rymai) +- German +- Indonesian + - Ahmad Naufal Mukhtar - [GitLab](https://gitlab.com/anaufalm), [Crowdin](https://crowdin.com/profile/anaufalm) +- Italian + - Paolo Falomo - [GitLab](https://gitlab.com/paolofalomo), [Crowdin](https://crowdin.com/profile/paolo.falomo) +- Japanese +- Korean + - Chang-Ho Cha - [GitLab](https://gitlab.com/changho-cha), [Crowdin](https://crowdin.com/profile/zzazang) + - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) +- Polish + - Filip Mech - [GitLab](https://gitlab.com/mehenz), [Crowdin](https://crowdin.com/profile/mehenz) +- Portuguese, Brazilian + - Paulo George Gomes Bezerra - [GitLab](https://gitlab.com/paulobezerra), [Crowdin](https://crowdin.com/profile/paulogomes.rep) +- Russian + - Nikita Grylov - [GitLab](https://gitlab.com/nixel2007), [Crowdin](https://crowdin.com/profile/nixel2007) + - Alexy Lustin - [GitLab](https://gitlab.com/allustin), [Crowdin](https://crowdin.com/profile/lustin) +- Spanish +- Ukrainian + - Volodymyr Sobotovych - [GitLab](https://gitlab.com/wheleph), [Crowdin](https://crowdin.com/profile/wheleph) + - Andrew Vityuk - [GitLab](https://gitlab.com/3_1_3_u), [Crowdin](https://crowdin.com/profile/andruwa13) + +## Become a proofreader + +> **Note:** Before requesting Proofreader permissions in Crowdin please make +> sure that you have a history of contributing translations to the GitLab +> project. + +1. Contribute translations to GitLab. See instructions for + [translating GitLab](translation.md). + + Translating GitLab is a community effort that requires team work and + attention to detail. Proofreaders play an important role helping new + contributors, and ensuring the consistency and quality of translations. + Your conduct and contributions as a translator should reflect this before + requesting to be a proofreader. + +1. Request proofreader permissions by opening a merge request to add yourself + to the list of proofreaders. + + Open the [proofreader.md source file][proofreader-src] and click **Edit**. + + Add your language in alphabetical order, and add yourself to the list + including: + + - name + - link to your GitLab profile + - link to your CrowdIn profile + + In the merge request description, please include links to any projects you + have previously translated. + +1. Your request to become a proofreader will be considered on the merits of + your previous translations. + +[proofreader-src]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/i18n/proofreader.md diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index b34ec754742..99c0fe6db1d 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -37,33 +37,43 @@ Comments can be added to discuss a translation with the community. Remember to **Save** each translation. -## Translation Guidelines +## General Translation Guidelines Be sure to check the following guidelines before you translate any strings. +### Namespaced strings + +When an externalized string is prepended with a namespace, e.g. +`s_('OpenedNDaysAgo|Opened')`, the namespace should be removed from the final +translation. +For example in French `OpenedNDaysAgo|Opened` would be translated to +`Ouvert•e`, not `OpenedNDaysAgo|Ouvert•e`. + ### Technical terms -Technical terms should be treated like proper nouns and not be translated. -This helps maintain a logical connection and consistency between tools (e.g. `git` client) and -GitLab. +Some technical terms should be treated like proper nouns and not be translated. -Technical terms that should always be in English are noted in the glossary when using -[translate.gitlab.com](https://translate.gitlab.com). +Technical terms that should always be in English are noted in the glossary when +using [translate.gitlab.com](https://translate.gitlab.com). + +This helps maintain a logical connection and consistency between tools (e.g. +`git` client) and GitLab. ### Formality The level of formality used in software varies by language. -For example, in French we translate `you` as the informal `tu`. +For example, in French we translate `you` as the formal `vous`. -You can refer to other translated strings and notes in the glossary to assist determining a -suitable level of formality. +You can refer to other translated strings and notes in the glossary to assist +determining a suitable level of formality. ### Inclusive language [Diversity] is one of GitLab's values. -We ask you to avoid translations which exclude people based on their gender or ethnicity. -In languages which distinguish between a male and female form, -use both or choose a neutral formulation. +We ask you to avoid translations which exclude people based on their gender or +ethnicity. +In languages which distinguish between a male and female form, use both or +choose a neutral formulation. For example in German, the word "user" can be translated into "Benutzer" (male) or "Benutzerin" (female). Therefore "create a new user" would translate into "Benutzer(in) anlegen". @@ -74,3 +84,14 @@ Therefore "create a new user" would translate into "Benutzer(in) anlegen". To propose additions to the glossary please [open an issue](https://gitlab.com/gitlab-org/gitlab-ce/issues). + +## French Translation Guidelines + +### Inclusive language in French + +In French, we should follow the guidelines from [ecriture-inclusive.fr]. For +instance: + +- Utilisateur•rice•s + +[ecriture-inclusive.fr]: http://www.ecriture-inclusive.fr/ diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 274923c2d43..c06bc0d4731 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -1,6 +1,6 @@ # GitLab Licensing and Compatibility -GitLab CE is licensed under the terms of the MIT License. GitLab EE is licensed under "The GitLab Enterprise Edition (EE) license" wherein there are more restrictions. See their respective LICENSE files ([CE][CE], [EE][EE]) for more information. +[GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-ce/) (CE) is licensed [under the terms of the MIT License][CE]. [GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab-ee/) (EE) is licensed under "[The GitLab Enterprise Edition (EE) license][EE]" wherein there are more restrictions. ## Automated Testing diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 243ac7f0c98..a211effdfa7 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -23,10 +23,6 @@ When downtime is necessary the migration has to be approved by: An up-to-date list of people holding these titles can be found at <https://about.gitlab.com/team/>. -The document ["What Requires Downtime?"](what_requires_downtime.md) specifies -various database operations, whether they require downtime and how to -work around that whenever possible. - When writing your migrations, also consider that databases might have stale data or inconsistencies and guard for that. Try to make as few assumptions as possible about the state of the database. @@ -41,6 +37,18 @@ Migrations that make changes to the database schema (e.g. adding a column) can only be added in the monthly release, patch releases may only contain data migrations _unless_ schema changes are absolutely required to solve a problem. +## What Requires Downtime? + +The document ["What Requires Downtime?"](what_requires_downtime.md) specifies +various database operations, such as + +- [adding, dropping, and renaming columns](what_requires_downtime.md#adding-columns) +- [changing column constraints and types](what_requires_downtime.md#changing-column-constraints) +- [adding and dropping indexes, tables, and foreign keys](what_requires_downtime.md#adding-indexes) + +and whether they require downtime and how to work around that whenever possible. + + ## Downtime Tagging Every migration must specify if it requires downtime or not, and if it should @@ -136,11 +144,14 @@ class MyMigration < ActiveRecord::Migration disable_ddl_transaction! def up - remove_concurrent_index :table_name, :column_name if index_exists?(:table_name, :column_name) + remove_concurrent_index :table_name, :column_name end end ``` +Note that it is not necessary to check if the index exists prior to +removing it. + ## Adding indexes If you need to add a unique index please keep in mind there is the possibility diff --git a/doc/development/new_fe_guide/dependencies.md b/doc/development/new_fe_guide/dependencies.md new file mode 100644 index 00000000000..3417d77a06d --- /dev/null +++ b/doc/development/new_fe_guide/dependencies.md @@ -0,0 +1,3 @@ +# Dependencies + +> TODO: Add Dependencies
\ No newline at end of file diff --git a/doc/development/new_fe_guide/development/accessibility.md b/doc/development/new_fe_guide/development/accessibility.md new file mode 100644 index 00000000000..ed35f08432f --- /dev/null +++ b/doc/development/new_fe_guide/development/accessibility.md @@ -0,0 +1,3 @@ +# Accessibility + +> TODO: Add content diff --git a/doc/development/new_fe_guide/development/components.md b/doc/development/new_fe_guide/development/components.md new file mode 100644 index 00000000000..637099d1e83 --- /dev/null +++ b/doc/development/new_fe_guide/development/components.md @@ -0,0 +1,3 @@ +# Components + +> TODO: Add content diff --git a/doc/development/new_fe_guide/development/design_patterns.md b/doc/development/new_fe_guide/development/design_patterns.md new file mode 100644 index 00000000000..ee06566ed30 --- /dev/null +++ b/doc/development/new_fe_guide/development/design_patterns.md @@ -0,0 +1,3 @@ +# Design patterns + +> TODO: Add content diff --git a/doc/development/new_fe_guide/development/index.md b/doc/development/new_fe_guide/development/index.md new file mode 100644 index 00000000000..cee8e43ebad --- /dev/null +++ b/doc/development/new_fe_guide/development/index.md @@ -0,0 +1,29 @@ +# Development + +## [Design patterns](design_patterns.md) + +Examples of proven design patterns used in our codebase. + +## [Components](components.md) + +Documentation on existing components and how to best create a new component. + +## [Accessibility](accessibility.md) + +Learn how to implement an accessible frontend. + +## [Network requests](network_requests.md) + +Learn how to handle network requests in our codebase. + +## [Security](security.md) + +Learn how to ensure that our frontend is secure. + +## [Performance](performance.md) + +Learn how to keep our frontend performant. + +## [Testing](testing.md) + +Learn how to keep our frontend tested. diff --git a/doc/development/new_fe_guide/development/network_requests.md b/doc/development/new_fe_guide/development/network_requests.md new file mode 100644 index 00000000000..047c00313bc --- /dev/null +++ b/doc/development/new_fe_guide/development/network_requests.md @@ -0,0 +1,3 @@ +# Network requests + +> TODO: Add content diff --git a/doc/development/new_fe_guide/development/performance.md b/doc/development/new_fe_guide/development/performance.md new file mode 100644 index 00000000000..244dfb3756f --- /dev/null +++ b/doc/development/new_fe_guide/development/performance.md @@ -0,0 +1,16 @@ +# Performance + +## Monitoring + +We have a performance dashboard available in one of our [grafana instances](https://performance.gprd.gitlab.com/dashboard/db/sitespeed-page-summary?orgId=1). This dashboard automatically aggregates metric data from [sitespeed.io](https://sitespeed.io) every 6 hours. These changes are displayed after a set number of pages are aggregated. + +These pages can be found inside a text file in the gitlab-build-images [repository](https://gitlab.com/gitlab-org/gitlab-build-images) called [gitlab.txt](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/scripts/gitlab.txt) +Any frontend engineer can contribute to this dashboard. They can contribute by adding or removing urls of pages from this text file. Please have a [frontend monitoring expert](https://about.gitlab.com/team) review your changes before assigning to a maintainer of the `gitlab-build-images` project. The changes will go live on the next scheduled run after the changes are merged into `master`. + +There are 3 recommended high impact metrics to review on each page + +* [First visual change](https://developers.google.com/web/tools/lighthouse/audits/first-meaningful-paint) +* [Speed Index](https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/speed-index) +* [Visual Complete 95%](https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics/speed-index) + +For these metrics, lower numbers are better as it means that the website is more performant. diff --git a/doc/development/new_fe_guide/development/security.md b/doc/development/new_fe_guide/development/security.md new file mode 100644 index 00000000000..5bb38f17988 --- /dev/null +++ b/doc/development/new_fe_guide/development/security.md @@ -0,0 +1,14 @@ +# Security + +## Avoid inline scripts and styles + +Inline scripts and styles should be avoided in almost all cases. In an effort to protect users from [XSS vulnerabilities](https://en.wikipedia.org/wiki/Cross-site_scripting), we will be disabling inline scripts using Content Security Policy. + +## Including external resources + +External fonts, CSS, and JavaScript should never be used with the exception of Google Analytics and Piwik - and only when the instance has enabled it. Assets should always be hosted and served locally from the GitLab instance. Embedded resources via `iframes` should never be used except in certain circumstances such as with ReCaptcha, which cannot be used without an `iframe`. + +## Resources for security testing + +- [Mozilla's HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) +- [Qualys SSL Labs Server Test](https://www.ssllabs.com/ssltest/analyze.html) diff --git a/doc/development/new_fe_guide/development/testing.md b/doc/development/new_fe_guide/development/testing.md new file mode 100644 index 00000000000..c359bd83ed1 --- /dev/null +++ b/doc/development/new_fe_guide/development/testing.md @@ -0,0 +1,3 @@ +# Testing + +> TODO: Add content diff --git a/doc/development/new_fe_guide/index.md b/doc/development/new_fe_guide/index.md new file mode 100644 index 00000000000..78931defa24 --- /dev/null +++ b/doc/development/new_fe_guide/index.md @@ -0,0 +1,28 @@ +# Frontend Development Guidelines + +This guide contains all the information to successfully contribute to GitLab's frontend. +This is a living document, and we welcome contributions, feedback and suggestions. + +## [Principles](principles.md) + +Ensure that your frontend contribution starts off in the right direction. + +## [Initiatives](initiatives.md) + +High level overview of where we are going from a frontend perspective. + +## [Development](development/index.md) + +Guidance on topics related to development. + +## [Dependencies](dependencies.md) + +Learn about all the dependencies that make up our frontend, including some of our own custom built libraries. + +## [Style guides](style/index.md) + +Style guides to keep our code consistent. + +## [Tips](tips.md) + +Tips from our frontend team to develop more efficiently and effectively. diff --git a/doc/development/new_fe_guide/initiatives.md b/doc/development/new_fe_guide/initiatives.md new file mode 100644 index 00000000000..c81ed3579f0 --- /dev/null +++ b/doc/development/new_fe_guide/initiatives.md @@ -0,0 +1,3 @@ +# Initiatives + +> TODO: Add Initiatives diff --git a/doc/development/new_fe_guide/principles.md b/doc/development/new_fe_guide/principles.md new file mode 100644 index 00000000000..0af5f506a91 --- /dev/null +++ b/doc/development/new_fe_guide/principles.md @@ -0,0 +1,35 @@ +# Principles + +These principles will ensure that your frontend contribution starts off in the right direction. + +## Discuss architecture before implementation + +Discuss your architecture design in an issue before writing code. This helps decrease the review time and also provides good practice for writing and thinking about system design. + +## Be consistent + +There are multiple ways of writing code to accomplish the same results. We should be as consistent as possible in how we write code across our codebases. This will make it more easier us to maintain our code across GitLab. + +## Enhance progressively + +Whenever you see with existing code that does not follow our current style guide, update it proactively. Refrain from changing everything but each merge request should progressively enhance our codebase and reduce technical debt. + +## When to use Vue + +- Use Vue for feature that make use of heavy DOM manipulation +- Use Vue for reusable components + +## When to use jQuery + +- Use jQuery to interact with Bootstrap JavaScript components +- Avoid jQuery when a better alternative exists. We are slowly moving away from it [#43559][jquery-future] + +## Mixing Vue and jQuery + +- Mixing Vue and jQuery is not recommended. +- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it][select2]. +- It is acceptable for Vue to listen to existing jQuery events using jQuery event listeners. +- It is not recommended to add new jQuery events for Vue to interact with jQuery. + +[jquery-future]: https://gitlab.com/gitlab-org/gitlab-ce/issues/43559 +[select2]: https://vuejs.org/v2/examples/select2.html diff --git a/doc/development/new_fe_guide/style/html.md b/doc/development/new_fe_guide/style/html.md new file mode 100644 index 00000000000..2d5b7d048ab --- /dev/null +++ b/doc/development/new_fe_guide/style/html.md @@ -0,0 +1,53 @@ +# HTML style guide + +## Buttons + +<a name="button-type"></a><a name="1.1"></a> +- [1.1](#button-type) **Use button type** Button tags requires a `type` attribute according to the [W3C HTML specification][button-type-spec]. + +``` +// bad +<button></button> + +// good +<button type="button"></button> +``` + +<a name="button-role"></a><a name="1.2"></a> +- [1.2](#button-role) **Use button role for non buttons** If an HTML element has an onClick handler but is not a button, it should have `role="button"`. This is more [accessible][button-role-accessible]. + +``` +// bad +<div onClick="doSomething"></div> + +// good +<div role="button" onClick="doSomething"></div> +``` + +## Links + +<a name="blank-links"></a><a name="2.1"></a> +- [2.1](#blank-links) **Use rel for target blank** Use `rel="noopener noreferrer"` whenever your links open in a new window i.e. `target="_blank"`. This prevents [the following][jitbit-target-blank] security vulnerability documented by JitBit + +``` +// bad +<a href="url" target="_blank"></a> + +// good +<a href="url" target="_blank" rel="noopener noreferrer"></a> +``` + +<a name="fake-links"></a><a name="2.2"></a> +- [2.2](#fake-links) **Do not use fake links** Use a button tag if a link only invokes JavaScript click event handlers. This is more semantic. + +``` +// bad +<a class="js-do-something" href="#"></a> + +// good +<button class="js-do-something" type="button"></button> +``` + +[button-type-spec]: https://www.w3.org/TR/2011/WD-html5-20110525/the-button-element.html#dom-button-type +[button-role-accessible]: https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_button_role +[jitbit-target-blank]: https://www.jitbit.com/alexblog/256-targetblank---the-most-underestimated-vulnerability-ever/ diff --git a/doc/development/new_fe_guide/style/index.md b/doc/development/new_fe_guide/style/index.md new file mode 100644 index 00000000000..335d9e66240 --- /dev/null +++ b/doc/development/new_fe_guide/style/index.md @@ -0,0 +1,15 @@ +# Style guides + +## [HTML style guide](html.md) + +## [SCSS style guide](scss.md) + +## [JavaScript style guide](javascript.md) + +## [Vue style guide](vue.md) + +# Tooling + +## [Prettier](prettier.md) + +Our code is automatically formatted with [Prettier](https://prettier.io) to follow our guidelines. diff --git a/doc/development/new_fe_guide/style/javascript.md b/doc/development/new_fe_guide/style/javascript.md new file mode 100644 index 00000000000..57efd9353bc --- /dev/null +++ b/doc/development/new_fe_guide/style/javascript.md @@ -0,0 +1,195 @@ +# JavaScript style guide + +We use [Airbnb's JavaScript Style Guide][airbnb-style-guide] and it's accompanying linter to manage most of our JavaScript style guidelines. + +In addition to the style guidelines set by Airbnb, we also have a few specific rules listed below. + +> **Tip:** +You can run eslint locally by running `yarn eslint` + +## Arrays + +<a name="avoid-foreach"></a><a name="1.1"></a> +- [1.1](#avoid-foreach) **Avoid ForEach when mutating data** Use `map`, `reduce` or `filter` instead of `forEach` when mutating data. This will minimize mutations in functions ([which is aligned with Airbnb's style guide][airbnb-minimize-mutations]) + +``` +// bad +users.forEach((user, index) => { + user.id = index; +}); + +// good +const usersWithId = users.map((user, index) => { + return Object.assign({}, user, { id: index }); +}); +``` + +## Functions + +<a name="limit-params"></a><a name="2.1"></a> +- [2.1](#limit-params) **Limit number of parameters** If your function or method has more than 3 parameters, use an object as a parameter instead. + +``` +// bad +function a(p1, p2, p3) { + // ... +}; + +// good +function a(p) { + // ... +}; +``` + +## Classes & constructors + +<a name="avoid-constructor-side-effects"></a><a name="3.1"></a> +- [3.1](#avoid-constructor-side-effects) **Avoid side effects in constructors** Avoid making some operations in the `constructor`, such as asynchronous calls, API requests and DOM manipulations. Prefer moving them into separate functions. This will make tests easier to write and code easier to maintain. + + ```javascript + // bad + class myClass { + constructor(config) { + this.config = config; + axios.get(this.config.endpoint) + } + } + + // good + class myClass { + constructor(config) { + this.config = config; + } + + makeRequest() { + axios.get(this.config.endpoint) + } + } + const instance = new myClass(); + instance.makeRequest(); + + ``` + +<a name="avoid-classes-to-handle-dom-events"></a><a name="3.2"></a> +- [3.2](#avoid-classes-to-handle-dom-events) **Avoid classes to handle DOM events** If the only purpose of the class is to bind a DOM event and handle the callback, prefer using a function. + +``` +// bad +class myClass { + constructor(config) { + this.config = config; + } + + init() { + document.addEventListener('click', () => {}); + } +} + +// good + +const myFunction = () => { + document.addEventListener('click', () => { + // handle callback here + }); +} +``` + +<a name="element-container"></a><a name="3.3"></a> +- [3.3](#element-container) **Pass element container to constructor** When your class manipulates the DOM, receive the element container as a parameter. +This is more maintainable and performant. + +``` +// bad +class a { + constructor() { + document.querySelector('.b'); + } +} + +// good +class a { + constructor(options) { + options.container.querySelector('.b'); + } +} +``` + +## Type Casting & Coercion + +<a name="use-parseint"></a><a name="4.1"></a> +- [4.1](#use-parseint) **Use ParseInt** Use `ParseInt` when converting a numeric string into a number. + +``` +// bad +Number('10') + + +// good +parseInt('10', 10); +``` + +## CSS Selectors + +<a name="use-js-prefix"></a><a name="5.1"></a> +- [5.1](#use-js-prefix) **Use js prefix** If a CSS class is only being used in JavaScript as a reference to the element, prefix the class name with `js-` + +``` +// bad +<button class="add-user"></button> + +// good +<button class="js-add-user"></button> +``` + +## Modules + +<a name="use-absolute-paths"></a><a name="6.1"></a> +- [6.1](#use-absolute-paths) **Use absolute paths for nearby modules** Use absolute paths if the module you are importing is less than two levels up. + +``` +// bad +import GitLabStyleGuide from '~/guides/GitLabStyleGuide'; + +// good +import GitLabStyleGuide from '../GitLabStyleGuide'; +``` + +<a name="use-relative-paths"></a><a name="6.2"></a> +- [6.2](#use-relative-paths) **Use relative paths for distant modules** If the module you are importing is two or more levels up, use a relative path instead of an absolute path. + +``` +// bad +import GitLabStyleGuide from '../../../guides/GitLabStyleGuide'; + +// good +import GitLabStyleGuide from '~/GitLabStyleGuide'; +``` + +<a name="global-namespace"></a><a name="6.3"></a> +- [6.3](#global-namespace) **Do not add to global namespace** + +<a name="domcontentloaded"></a><a name="6.4"></a> +- [6.4](#domcontentloaded) **Do not use DOMContentLoaded in non-page modules** Imported modules should act the same each time they are loaded. `DOMContentLoaded` events are only allowed on modules loaded in the `/pages/*` directory because those are loaded dynamically with webpack. + +## Security + +<a name="avoid-xss"></a><a name="7.1"></a> +- [7.1](#avoid-xss) **Avoid XSS** Do not use `innerHTML`, `append()` or `html()` to set content. It opens up too many vulnerabilities. + +## ESLint + +<a name="disable-eslint-file"></a><a name="8.1"></a> +- [8.1](#disable-eslint-file) **Disabling ESLint in new files** Do not disable ESLint when creating new files. Existing files may have existing rules disabled due to legacy compatibility reasons but they are in the process of being refactored. + +<a name="disable-eslint-rule"></a><a name="8.2"></a> +- [8.2](#disable-eslint-rule) **Disabling ESLint rule** Do not disable specific ESLint rules. Due to technical debt, you may disable the following rules only if you are invoking/instantiating existing code modules + + - [no-new][no-new] + - [class-method-use-this][class-method-use-this] + +> Note: Disable these rules on a per line basis. This makes it easier to refactor in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line` + +[airbnb-style-guide]: https://github.com/airbnb/javascript +[airbnb-minimize-mutations]: https://github.com/airbnb/javascript#testing--for-real +[no-new]: http://eslint.org/docs/rules/no-new +[class-method-use-this]: http://eslint.org/docs/rules/class-methods-use-this diff --git a/doc/development/new_fe_guide/style/prettier.md b/doc/development/new_fe_guide/style/prettier.md new file mode 100644 index 00000000000..eb18189282b --- /dev/null +++ b/doc/development/new_fe_guide/style/prettier.md @@ -0,0 +1,45 @@ +# Formatting with Prettier + +Our code is automatically formatted with [Prettier](https://prettier.io) to follow our style guides. Prettier is taking care of formatting .js, .vue, and .scss files based on the standard prettier rules. You can find all settings for Prettier in `.prettierrc`. + +## Editor + +The easiest way to include prettier in your workflow is by setting up your preferred editor (all major editors are supported) accordingly. We suggest setting up prettier to run automatically when each file is saved. Find [here](https://prettier.io/docs/en/editors.html) the best way to set it up in your preferred editor. + +Please take care that you only let Prettier format the same file types as the global Yarn script does (.js, .vue, and .scss). In VSCode by example you can easily exclude file formats in your settings file: + +``` + "prettier.disableLanguages": [ + "json", + "markdown" + ], +``` + +## Yarn Script + +The following yarn scripts are available to do global formatting: + +``` +yarn prettier-staged-save +``` + +Updates all currently staged files (based on `git diff`) with Prettier and saves the needed changes. + +``` +yarn prettier-staged +``` +Checks all currently staged files (based on `git diff`) with Prettier and log which files would need manual updating to the console. + +``` +yarn prettier-all +``` + +Checks all files with Prettier and logs which files need manual updating to the console. + +``` +yarn prettier-all-save +``` + +Formats all files in the repository with Prettier. (This should only be used to test global rule updates otherwise you would end up with huge MR's). + +The source of these Yarn scripts can be found in `/scripts/frontend/prettier.js`. diff --git a/doc/development/new_fe_guide/style/scss.md b/doc/development/new_fe_guide/style/scss.md new file mode 100644 index 00000000000..6f5e818d7db --- /dev/null +++ b/doc/development/new_fe_guide/style/scss.md @@ -0,0 +1,3 @@ +# SCSS style guide + +> TODO: Add content diff --git a/doc/development/new_fe_guide/style/vue.md b/doc/development/new_fe_guide/style/vue.md new file mode 100644 index 00000000000..fd9353e0d3f --- /dev/null +++ b/doc/development/new_fe_guide/style/vue.md @@ -0,0 +1,3 @@ +# Vue style guide + +> TODO: Add content diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md new file mode 100644 index 00000000000..f0cdf52d618 --- /dev/null +++ b/doc/development/new_fe_guide/tips.md @@ -0,0 +1,3 @@ +# Tips + +> TODO: Add tips diff --git a/doc/development/profiling.md b/doc/development/profiling.md index 97c997e0568..11878b4009b 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -27,6 +27,17 @@ Gitlab::Profiler.profile('/my-user') # Returns a RubyProf::Profile where 100 seconds is spent in UsersController#show ``` +For routes that require authorization you will need to provide a user to +`Gitlab::Profiler`. You can do this like so: + +```ruby +Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first) +``` + +The user you provide will need to have a [personal access +token](https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html) in +the GitLab instance. + Passing a `logger:` keyword argument to `Gitlab::Profiler.profile` will send ActiveRecord and ActionController log output to that logger. Further options are documented with the method source. diff --git a/doc/development/query_count_limits.md b/doc/development/query_count_limits.md new file mode 100644 index 00000000000..310e3faf61b --- /dev/null +++ b/doc/development/query_count_limits.md @@ -0,0 +1,64 @@ +# Query Count Limits + +Each controller or API endpoint is allowed to execute up to 100 SQL queries and +in test environments we'll raise an error when this threshold is exceeded. + +## Solving Failing Tests + +When a test fails because it executes more than 100 SQL queries there are two +solutions to this problem: + +1. Reduce the number of SQL queries that are executed. +2. Whitelist the controller or API endpoint. + +You should only resort to whitelisting when an existing controller or endpoint +is to blame as in this case reducing the number of SQL queries can take a lot of +effort. Newly added controllers and endpoints are not allowed to execute more +than 100 SQL queries and no exceptions will be made for this rule. _If_ a large +number of SQL queries is necessary to perform certain work it's best to have +this work performed by Sidekiq instead of doing this directly in a web request. + +## Whitelisting + +In the event that you _have_ to whitelist a controller you'll first need to +create an issue. This issue should (preferably in the title) mention the +controller or endpoint and include the appropriate labels (`database`, +`performance`, and at least a team specific label such as `Discussion`). + +Once the issue has been created you can whitelist the code in question. For +Rails controllers it's best to create a `before_action` hook that runs as early +as possible. The called method in turn should call +`Gitlab::QueryLimiting.whitelist('issue URL here')`. For example: + +```ruby +class MyController < ApplicationController + before_action :whitelist_query_limiting, only: [:show] + + def index + # ... + end + + def show + # ... + end + + def whitelist_query_limiting + Gitlab::QueryLimiting.whitelist('https://gitlab.com/gitlab-org/...') + end +end +``` + +By using a `before_action` you don't have to modify the controller method in +question, reducing the likelihood of merge conflicts. + +For Grape API endpoints there unfortunately is not a reliable way of running a +hook before a specific endpoint. This means that you have to add the whitelist +call directly into the endpoint like so: + +```ruby +get '/projects/:id/foo' do + Gitlab::QueryLimiting.whitelist('...') + + # ... +end +``` diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index dc88ce1522c..fdfa1f10402 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -102,6 +102,12 @@ variable to `1`: export ENABLE_SPRING=1 ``` +Alternatively you can use the following on each spec run, + +``` +bundle exec spring rspec some_spec.rb +``` + ## Compile Frontend Assets You shouldn't ever need to compile frontend assets manually in development, but diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 59ebf41e09f..76ff51446ba 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -17,6 +17,9 @@ would be `process_something`. If you're not sure what queue a worker uses, you can find it using `SomeWorker.queue`. There is almost never a reason to manually override the queue name using `sidekiq_options queue: :some_queue`. +You must always add any new queues to `app/workers/all_queues.yml` otherwise +your worker will not run. + ## Queue Namespaces While different workers cannot share a queue, they can share a queue namespace. diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end_tests.md index 5b4f6511f04..d10a797a142 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end_tests.md @@ -22,7 +22,7 @@ You can find these nightly pipelines at [GitLab QA pipelines page][gitlab-qa-pip It is possible to run end-to-end tests (eventually being run within a [GitLab QA pipeline][gitlab-qa-pipelines]) for a merge request by triggering -the `package-qa` manual action, that should be present in a merge request +the `package-and-qa` manual action, that should be present in a merge request widget. Manual action that starts end-to-end tests is also available in merge requests diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 4adf0dc7c7a..e86c1f5232a 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -134,6 +134,10 @@ learn more. [GitLab QA]: https://gitlab.com/gitlab-org/gitlab-qa [part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa +## EE-specific tests + +EE-specific tests follows the same organization, but under the `ee/spec` folder. + ## How to test at the correct level? As many things in life, deciding what to test at each level of testing is a diff --git a/doc/development/writing_documentation.md b/doc/development/writing_documentation.md index 2a1d744668b..d6a13e7483a 100644 --- a/doc/development/writing_documentation.md +++ b/doc/development/writing_documentation.md @@ -1,13 +1,9 @@ -# Writing documentation +# GitLab Documentation guidelines - **General Documentation**: written by the [developers responsible by creating features](#contributing-to-docs). Should be submitted in the same merge request containing code. Feature proposals (by GitLab contributors) should also be accompanied by its respective documentation. They can be later improved by PMs and Technical Writers. - **[Technical Articles](#technical-articles)**: written by any [GitLab Team](https://about.gitlab.com/team/) member, GitLab contributors, or [Community Writers](https://about.gitlab.com/handbook/product/technical-writing/community-writers/). - **Indexes per topic**: initially prepared by the Technical Writing Team, and kept up-to-date by developers and PMs in the same merge request containing code. They gather all resources for that topic in a single page (user and admin documentation, articles, and third-party docs). -## Documentation style guidelines - -All the docs follow the same [styleguide](doc_styleguide.md). - ## Contributing to docs Whenever a feature is changed, updated, introduced, or deprecated, the merge @@ -19,7 +15,7 @@ The one responsible for writing the first piece of documentation is the develope wrote the code. It's the job of the Product Manager to ensure all features are shipped with its docs, whether is a small or big change. At the pace GitLab evolves, this is the only way to keep the docs up-to-date. If you have any questions about it, -please ask a Technical Writer. Otherwise, when your content is ready, assign one of +ask a Technical Writer. Otherwise, when your content is ready, assign one of them to review it for you. We use the [monthly release blog post](https://about.gitlab.com/handbook/marketing/blog/release-posts/#monthly-releases) as a changelog checklist to ensure everything @@ -27,23 +23,18 @@ is documented. Whenever you submit a merge request for the documentation, use the documentation MR description template. -### Documentation directory structure +Please check the [documentation workflow](https://about.gitlab.com/handbook/product/technical-writing/workflow/) before getting started. -The documentation is structured based on the GitLab UI structure itself, -separated by [`user`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/user), -[`administrator`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/administration), and [`contributor`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/development). +## Documentation structure -To learn where to place a new document, check the [documentation style guide](doc_styleguide.md#location-and-naming-of-documents). +- Overview and use cases: what it is, why it is necessary, why one would use it +- Requirements: what do we need to get started +- Tutorial: how to set it up, how to use it -In order to have a [solid site structure](https://searchengineland.com/seo-benefits-developing-solid-site-structure-277456) for our documentation, -all docs should be linked. Every new document should be cross-linked to its related documentation, and linked from its topic-related index, when existent. +Always link a new document from its topic-related index, otherwise, it will +not be included it in the documentation site search. -The directories `/workflow/`, `/gitlab-basics/`, `/university/`, and `/articles/` have -been deprecated and the majority their docs have been moved to their correct location -in small iterations. Please don't create new docs in these folders. - -To move a document from its location to another directory, read the section -[changing document location](doc_styleguide.md#changing-document-location) of the doc style guide. +_Note: to be extended._ ### Feature overview and use cases @@ -73,16 +64,169 @@ overview there. > **Overview** and **use cases** are required to **every** Enterprise Edition feature, and for every **major** feature present in Community Edition. -### Markdown +## Markdown and styles Currently GitLab docs use Redcarpet as [markdown](../user/markdown.md) engine, but there's an [open discussion](https://gitlab.com/gitlab-com/gitlab-docs/issues/50) for implementing Kramdown in the near future. -### Previewing locally +All the docs follow the [documentation style guidelines](doc_styleguide.md). -To preview your changes to documentation locally, please follow -this [development guide](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md#development). +## Documentation directory structure + +The documentation is structured based on the GitLab UI structure itself, +separated by [`user`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/user), +[`administrator`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/administration), and [`contributor`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/development). + +In order to have a [solid site structure](https://searchengineland.com/seo-benefits-developing-solid-site-structure-277456) for our documentation, +all docs should be linked. Every new document should be cross-linked to its related documentation, and linked from its topic-related index, when existent. + +The directories `/workflow/`, `/gitlab-basics/`, `/university/`, and `/articles/` have +been deprecated and the majority their docs have been moved to their correct location +in small iterations. Please don't create new docs in these folders. + +### Location and naming documents + +The documentation hierarchy can be vastly improved by providing a better layout +and organization of directories. + +Having a structured document layout, we will be able to have meaningful URLs +like `docs.gitlab.com/user/project/merge_requests/index.html`. With this pattern, +you can immediately tell that you are navigating a user related documentation +and is about the project and its merge requests. + +Do not create summaries of similar types of content (e.g. an index of all articles, videos, etc.), +rather organize content by its subject (e.g. everything related to CI goes together) +and cross-link between any related content. + +The table below shows what kind of documentation goes where. + +| Directory | What belongs here | +| --------- | -------------- | +| `doc/user/` | User related documentation. Anything that can be done within the GitLab UI goes here including `/admin`. | +| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed via GitLab's interface go under `doc/user/admin_area/`. | +| `doc/api/` | API related documentation. | +| `doc/development/` | Documentation related to the development of GitLab. Any styleguides should go here. | +| `doc/legal/` | Legal documents about contributing to GitLab. | +| `doc/install/`| Probably the most visited directory, since `installation.md` is there. Ideally this should go under `doc/administration/`, but it's best to leave it as-is in order to avoid confusion (still debated though). | +| `doc/update/` | Same with `doc/install/`. Should be under `administration/`, but this is a well known location, better leave as-is, at least for now. | +| `doc/topics/` | Indexes per Topic (`doc/topics/topic-name/index.md`): all resources for that topic (user and admin documentation, articles, and third-party docs) | + +--- + +**General rules:** + +1. The correct naming and location of a new document, is a combination + of the relative URL of the document in question and the GitLab Map design + that is used for UX purposes ([source][graffle], [image][gitlab-map]). +1. When creating a new document and it has more than one word in its name, + make sure to use underscores instead of spaces or dashes (`-`). For example, + a proper naming would be `import_projects_from_github.md`. The same rule + applies to images. +1. Start a new directory with an `index.md` file. +1. There are four main directories, `user`, `administration`, `api` and `development`. +1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, + `profile/`, `dashboard/` and `admin_area/`. + 1. `doc/user/project/` should contain all project related documentation. + 1. `doc/user/group/` should contain all group related documentation. + 1. `doc/user/profile/` should contain all profile related documentation. + Every page you would navigate under `/profile` should have its own document, + i.e. `account.md`, `applications.md`, `emails.md`, etc. + 1. `doc/user/dashboard/` should contain all dashboard related documentation. + 1. `doc/user/admin_area/` should contain all admin related documentation + describing what can be achieved by accessing GitLab's admin interface + (_not to be confused with `doc/administration` where server access is + required_). + 1. Every category under `/admin/application_settings` should have its + own document located at `doc/user/admin_area/settings/`. For example, + the **Visibility and Access Controls** category should have a document + located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. +1. The `doc/topics/` directory holds topic-related technical content. Create + `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. + General user- and admin- related documentation, should be placed accordingly. + +If you are unsure where a document should live, you can ping `@axil` or `@marcia` in your +merge request. + +### Changing document location + +Changing a document's location is not to be taken lightly. Remember that the +documentation is available to all installations under `help/` and not only to +GitLab.com or http://docs.gitlab.com. Make sure this is discussed with the +Documentation team beforehand. + +If you indeed need to change a document's location, do NOT remove the old +document, but rather replace all of its contents with a new line: + +``` +This document was moved to [another location](path/to/new_doc.md). +``` + +where `path/to/new_doc.md` is the relative path to the root directory `doc/`. + +--- + +For example, if you were to move `doc/workflow/lfs/lfs_administration.md` to +`doc/administration/lfs.md`, then the steps would be: + +1. Copy `doc/workflow/lfs/lfs_administration.md` to `doc/administration/lfs.md` +1. Replace the contents of `doc/workflow/lfs/lfs_administration.md` with: + + ``` + This document was moved to [another location](../../administration/lfs.md). + ``` + +1. Find and replace any occurrences of the old location with the new one. + A quick way to find them is to use `git grep`. First go to the root directory + where you cloned the `gitlab-ce` repository and then do: + + ``` + git grep -n "workflow/lfs/lfs_administration" + git grep -n "lfs/lfs_administration" + ``` + +NOTE: **Note:** +If the document being moved has any Disqus comments on it, there are extra steps +to follow documented just [below](#redirections-for-pages-with-disqus-comments). + +Things to note: + +- Since we also use inline documentation, except for the documentation itself, + the document might also be referenced in the views of GitLab (`app/`) which will + render when visiting `/help`, and sometimes in the testing suite (`spec/`). +- The above `git grep` command will search recursively in the directory you run + it in for `workflow/lfs/lfs_administration` and `lfs/lfs_administration` + and will print the file and the line where this file is mentioned. + You may ask why the two greps. Since we use relative paths to link to + documentation, sometimes it might be useful to search a path deeper. +- The `*.md` extension is not used when a document is linked to GitLab's + built-in help page, that's why we omit it in `git grep`. +- Use the checklist on the documentation MR description template. + +### Redirections for pages with Disqus comments + +If the documentation page being relocated already has any Disqus comments, +we need to preserve the Disqus thread. + +Disqus uses an identifier per page, and for docs.gitlab.com, the page identifier +is configured to be the page URL. Therefore, when we change the document location, +we need to preserve the old URL as the same Disqus identifier. + +To do that, add to the frontmatter the variable `redirect_from`, +using the old URL as value. For example, let's say I moved the document +available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, +`https://docs.gitlab.com/my-new-location/index.html`. + +Into the **new document** frontmatter add the following: + +```yaml +--- +redirect_from: 'https://docs.gitlab.com/my-old-location/README.html' +--- +``` + +Note: it is necessary to include the file name in the `redirect_from` URL, +even if it's `index.html` or `README.html`. -### Testing +## Testing We treat documentation as code, thus have implemented some testing. Currently, the following tests are in place: @@ -101,7 +245,7 @@ Currently, the following tests are in place: Submitting an EE-equivalent merge request cherry-picking all commits from CE to EE is essential to avoid them. -### Branch naming +## Branch naming If your contribution contains **only** documentation changes, you can speed up the CI process by following some branch naming conventions. You have three @@ -116,7 +260,53 @@ choices: If your branch name matches any of the above, it will run only the docs tests. If it doesn't, the whole test suite will run (including docs). -### Previewing the changes live +## Merge requests for GitLab documentation + +Before getting started, make sure you read the introductory section +"[contributing to docs](#contributing-to-docs)" above and the +[tech writing workflow](https://about.gitlab.com/handbook/product/technical-writing/workflow/) +for GitLab Team members. + +- Use the current [merge request description template](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab/merge_request_templates/Documentation.md) +- Use the correct [branch name](#branch-naming) +- Label the MR `Documentation` +- Assign the correct milestone (see note below) + + +NOTE: **Note:** +If the release version you want to add the documentation to has already been +frozen or released, use the label `Pick into X.Y` to get it merged into +the correct release. Avoid picking into a past release as much as you can, as +it increases the work of the release managers. + +### Cherry-picking from CE to EE + +As we have the `master` branch of CE merged into EE once a day, it's common to +run into merge conflicts. To avoid them, we [test for merge conflicts against EE](#testing) +with the `ee-compat-check` job, and use the following method of creating equivalent +branches for CE and EE. + +Follow this [method for cherry-picking from CE to EE](automatic_ce_ee_merge.md#cherry-picking-from-ce-to-ee), with a few adjustments: + +- Create the [CE branch](#branch-naming) starting with `docs-`, + e.g.: `git checkout -b docs-example` +- Create the EE-equivalent branch ending with `-ee`, e.g., + `git checkout -b docs-example-ee` +- Once all the jobs are passing in CE and EE, and you've addressed the +feedback from your own team, assign the CE MR to a technical writer for review +- When both MRs are ready, the EE merge request will be merged first, and the +CE-equivalent will be merged next. +- Note that the review will occur only in the CE MR, as the EE MR +contains the same commits as the CE MR. +- If you have a few more changes that apply to the EE-version only, you can submit +a couple more commits to the EE branch, but ask the reviewer to review the EE merge request +additionally to the CE MR. If there are many EE-only changes though, start a new MR +to EE only. + +## Previewing the changes live + +To preview your changes to documentation locally, please follow +this [development guide](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md#development). If you want to preview the doc changes of your merge request live, you can use the manual `review-docs-deploy` job in your merge request. You will need at @@ -176,7 +366,7 @@ working on. If you don't, the remote docs branch won't be removed either, and the server where the Review Apps are hosted will eventually be out of disk space. -#### Technical aspects +### Technical aspects If you want to know the hot details, here's what's really happening: @@ -211,6 +401,74 @@ The following GitLab features are used among others: - [Artifacts](../ci/yaml/README.md#artifacts) - [Specific Runner](../ci/runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) +## GitLab `/help` + +Every GitLab instance includes the documentation, which is available from `/help` +(`http://my-instance.com/help`), e.g., <https://gitlab.com/help>. + +When you're building a new feature, you may need to link the documentation +from GitLab, the application. This is normally done in files inside the +`app/views/` directory with the help of the `help_page_path` helper method. + +In its simplest form, the HAML code to generate a link to the `/help` page is: + +```haml += link_to 'Help page', help_page_path('user/permissions') +``` + +The `help_page_path` contains the path to the document you want to link to with +the following conventions: + +- it is relative to the `doc/` directory in the GitLab repository +- the `.md` extension must be omitted +- it must not end with a slash (`/`) + +Below are some special cases where should be used depending on the context. +You can combine one or more of the following: + +1. **Linking to an anchor link.** Use `anchor` as part of the `help_page_path` + method: + + ```haml + = link_to 'Help page', help_page_path('user/permissions', anchor: 'anchor-link') + ``` + +1. **Opening links in a new tab.** This should be the default behavior: + + ```haml + = link_to 'Help page', help_page_path('user/permissions'), target: '_blank' + ``` + +1. **Linking to a circle icon.** Usually used in settings where a long + description cannot be used, like near checkboxes. You can basically use + any font awesome icon, but prefer the `question-circle`: + + ```haml + = link_to icon('question-circle'), help_page_path('user/permissions') + ``` + +1. **Using a button link.** Useful in places where text would be out of context + with the rest of the page layout: + + ```haml + = link_to 'Help page', help_page_path('user/permissions'), class: 'btn btn-info' + ``` + +1. **Using links inline of some text.** + + ```haml + Description to #{link_to 'Help page', help_page_path('user/permissions')}. + ``` + +1. **Adding a period at the end of the sentence.** Useful when you don't want + the period to be part of the link: + + ```haml + = succeed '.' do + Learn more in the + = link_to 'Help page', help_page_path('user/permissions') + ``` + ## General Documentation vs Technical Articles ### General documentation @@ -225,7 +483,7 @@ They are topic-related documentation, written with an user-friendly approach and A technical article guides users and/or admins to achieve certain objectives (within guides and tutorials), or provide an overview of that particular topic or feature (within technical overviews). It can also describe the use, implementation, or integration of third-party tools with GitLab. -They should be placed in a new directory named `/article-title/index.md` under a topic-related folder, and their images should be placed in `/article-title/img/`. For example, a new article on GitLab Pages should be placed in `doc/user/project/pages/article-title/` and a new article on GitLab CI/CD should be placed in `doc/ci/article-title/`. +They should be placed in a new directory named `/article-title/index.md` under a topic-related folder, and their images should be placed in `/article-title/img/`. For example, a new article on GitLab Pages should be placed in `doc/user/project/pages/article-title/` and a new article on GitLab CI/CD should be placed in `doc/ci/examples/article-title/`. #### Types of Technical Articles @@ -240,7 +498,7 @@ Suppose there's a process to go from point A to point B in 5 steps: `(A) 1 > 2 > A **guide** can be understood as a description of certain processes to achieve a particular objective. A guide brings you from A to B describing the characteristics of that process, but not necessarily going over each step. It can mention, for example, steps 2 and 3, but does not necessarily explain how to accomplish them. -- Live example: "GitLab Pages from A to Z - [Part 1](../user/project/pages/getting_started_part_one.md) to [Part 4](../user/project/pages/getting_started_part_four.md)" +- Live example: "[Static sites and GitLab Pages domains (Part 1)](../user/project/pages/getting_started_part_one.md) to [Creating and Tweaking GitLab CI/CD for GitLab Pages (Part 4)](../user/project/pages/getting_started_part_four.md)" A **tutorial** requires a clear **step-by-step** guidance to achieve a singular objective. It brings you from A to B, describing precisely all the necessary steps involved in that process, showing each of the 5 steps to go from A to B. It does not only describes steps 2 and 3, but also shows you how to accomplish them. @@ -254,21 +512,30 @@ through the process of how to use it systematically. #### Special format -Every **Technical Article** contains, in the very beginning, a blockquote with the following information: +Every **Technical Article** contains a frontmatter at the beginning of the doc +with the following information: + +- **Type of article** (user guide, admin guide, technical overview, tutorial) +- **Knowledge level** expected from the reader to be able to follow through (beginner, intermediate, advanced) +- **Author's name** and **GitLab.com handle** +- **Publication date** (ISO format YYYY-MM-DD) + +For example: -- A reference to the **type of article** (user guide, admin guide, tech overview, tutorial) -- A reference to the **knowledge level** expected from the reader to be able to follow through (beginner, intermediate, advanced) -- A reference to the **author's name** and **GitLab.com handle** -- A reference of the **publication date** -```md -> **[Article Type](../../development/writing_documentation.html#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Name Surname](https://gitlab.com/username) || -> **Publication date:** AAAA-MM-DD +```yaml +--- +author: John Doe +author_gitlab: johnDoe +level: beginner +article_type: user guide +date: 2017-02-01 +--- ``` #### Technical Articles - Writing Method Use the [writing method](https://about.gitlab.com/handbook/product/technical-writing/#writing-method) defined by the Technical Writing team. +[gitlab-map]: https://gitlab.com/gitlab-org/gitlab-design/raw/master/production/resources/gitlab-map.png +[graffle]: https://gitlab.com/gitlab-org/gitlab-design/blob/d8d39f4a87b90fb9ae89ca12dc565347b4900d5e/production/resources/gitlab-map.graffle diff --git a/doc/downgrade_ee_to_ce/README.md b/doc/downgrade_ee_to_ce/README.md index 75bae324585..f656057e3da 100644 --- a/doc/downgrade_ee_to_ce/README.md +++ b/doc/downgrade_ee_to_ce/README.md @@ -70,7 +70,7 @@ To downgrade an Omnibus installation, it is sufficient to install the Community Edition package on top of the currently installed one. You can do this manually, by directly [downloading the package](https://packages.gitlab.com/gitlab/gitlab-ce) you need, or by adding our CE package repository and following the -[CE installation instructions](https://about.gitlab.com/downloads/). +[CE installation instructions](https://about.gitlab.com/installation/?version=ce). **Source Installation** diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index e18711f3392..10e8059756d 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -33,5 +33,40 @@ 1. Click **Create project**. +## Push to create a new project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/26388) in GitLab 10.5. + +When you create a new repo locally, instead of going to GitLab to manually +create a new project and then push the repo, you can directly push it to +GitLab to create the new project, all without leaving your terminal. If you have access to that +namespace, we will automatically create a new project under that GitLab namespace with its +visibility set to Private by default (you can later change it in the [project's settings](../public_access/public_access.md#how-to-change-project-visibility)). + +This can be done by using either SSH or HTTP: + +``` +## Git push using SSH +git push --set-upstream git@gitlab.example.com:namespace/nonexistent-project.git master + +## Git push using HTTP +git push --set-upstream https://gitlab.example.com/namespace/nonexistent-project.git master +``` + +Once the push finishes successfully, a remote message will indicate +the command to set the remote and the URL to the new project: + +``` +remote: +remote: The private project namespace/nonexistent-project was created. +remote: +remote: To configure the remote, run: +remote: git remote add origin https://gitlab.example.com/namespace/nonexistent-project.git +remote: +remote: To view the project, visit: +remote: https://gitlab.example.com/namespace/nonexistent-project +remote: +``` + [import it]: ../workflow/importing/README.md [reserved]: ../user/reserved_names.md diff --git a/doc/img/devops_lifecycle.png b/doc/img/devops_lifecycle.png Binary files differnew file mode 100644 index 00000000000..0616be46df8 --- /dev/null +++ b/doc/img/devops_lifecycle.png diff --git a/doc/install/README.md b/doc/install/README.md index 87f6969b415..9724b56910d 100644 --- a/doc/install/README.md +++ b/doc/install/README.md @@ -36,7 +36,7 @@ the full process of installing GitLab on Google Container Engine (GKE), pushing - [Install on AWS](https://about.gitlab.com/aws/) - _Testing only!_ [DigitalOcean and Docker Machine](digitaloceandocker.md) - Quickly test any version of GitLab on DigitalOcean using Docker Machine. -- [Getting started with GitLab and DigitalOcean](ttps://about.gitlab.com/2016/04/27/getting-started-with-gitlab-and-digitalocean/): requirements, installation process, updates. +- [Getting started with GitLab and DigitalOcean](https://about.gitlab.com/2016/04/27/getting-started-with-gitlab-and-digitalocean/): requirements, installation process, updates. - [Demo: Cloud Native Development with GitLab](https://about.gitlab.com/2017/04/18/cloud-native-demo/): video demonstration on how to install GitLab on Kubernetes, build a project, create Review Apps, store Docker images in Container Registry, deploy to production on Kubernetes, and monitor with Prometheus. ## Database diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index 7afe338ae8b..b0c3ad960bb 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -38,9 +38,10 @@ create SQL Databases, author websites, and perform lots of other cloud tasks. ## Create New VM The [Azure Marketplace][Azure-Marketplace] is an online store for pre-configured applications and -services which have been optimized for the cloud by software vendors like GitLab, and both -the [Community Edition ("CE")][CE] and the [Enterprise Edition ("EE")][EE] versions of GitLab are -available on the Azure Marketplace as pre-configured solutions. +services which have been optimized for the cloud by software vendors like GitLab, +available on the Azure Marketplace as pre-configured solutions. In this tutorial +we will install GitLab Community Edition, but for GitLab Enterprise Edition you +can follow the same process. To begin creating a new GitLab VM, click on the **+ New** icon, type "GitLab" into the search box, and then click the **"GitLab Community Edition"** search result: diff --git a/doc/install/database_mysql.md b/doc/install/database_mysql.md index f9ba1508705..5c7557ed2b3 100644 --- a/doc/install/database_mysql.md +++ b/doc/install/database_mysql.md @@ -6,7 +6,6 @@ and [problems](https://bugs.mysql.com/bug.php?id=65830) that [suggested](https://bugs.mysql.com/bug.php?id=50909) [fixes](https://bugs.mysql.com/bug.php?id=65830) [have](https://bugs.mysql.com/bug.php?id=63164). -- We recommend using MySQL version 5.6 or later. Please see the following [issue][ce-38152]. ## Initial database setup diff --git a/doc/install/google_cloud_platform/img/boot_disk.png b/doc/install/google_cloud_platform/img/boot_disk.png Binary files differnew file mode 100644 index 00000000000..37b2d9eaae7 --- /dev/null +++ b/doc/install/google_cloud_platform/img/boot_disk.png diff --git a/doc/install/google_cloud_platform/img/change_admin_passwd_email.png b/doc/install/google_cloud_platform/img/change_admin_passwd_email.png Binary files differdeleted file mode 100644 index 1ffe14f60ff..00000000000 --- a/doc/install/google_cloud_platform/img/change_admin_passwd_email.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/chrome_not_secure_page.png b/doc/install/google_cloud_platform/img/chrome_not_secure_page.png Binary files differdeleted file mode 100644 index e732066908f..00000000000 --- a/doc/install/google_cloud_platform/img/chrome_not_secure_page.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/first_signin.png b/doc/install/google_cloud_platform/img/first_signin.png Binary files differnew file mode 100644 index 00000000000..6eb3392d674 --- /dev/null +++ b/doc/install/google_cloud_platform/img/first_signin.png diff --git a/doc/install/google_cloud_platform/img/gcp_gitlab_being_deployed.png b/doc/install/google_cloud_platform/img/gcp_gitlab_being_deployed.png Binary files differdeleted file mode 100644 index 2a1859da6e3..00000000000 --- a/doc/install/google_cloud_platform/img/gcp_gitlab_being_deployed.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/gcp_gitlab_overview.png b/doc/install/google_cloud_platform/img/gcp_gitlab_overview.png Binary files differdeleted file mode 100644 index 1c4c870dbc9..00000000000 --- a/doc/install/google_cloud_platform/img/gcp_gitlab_overview.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/gcp_landing.png b/doc/install/google_cloud_platform/img/gcp_landing.png Binary files differindex 6398d247ba0..d6390c4dd4f 100644 --- a/doc/install/google_cloud_platform/img/gcp_landing.png +++ b/doc/install/google_cloud_platform/img/gcp_landing.png diff --git a/doc/install/google_cloud_platform/img/gcp_launcher_console_home_page.png b/doc/install/google_cloud_platform/img/gcp_launcher_console_home_page.png Binary files differdeleted file mode 100644 index f492888ea4a..00000000000 --- a/doc/install/google_cloud_platform/img/gcp_launcher_console_home_page.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/gcp_search_for_gitlab.png b/doc/install/google_cloud_platform/img/gcp_search_for_gitlab.png Binary files differdeleted file mode 100644 index b38af3966e2..00000000000 --- a/doc/install/google_cloud_platform/img/gcp_search_for_gitlab.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/gitlab_deployed_page.png b/doc/install/google_cloud_platform/img/gitlab_deployed_page.png Binary files differdeleted file mode 100644 index fef9ae45f32..00000000000 --- a/doc/install/google_cloud_platform/img/gitlab_deployed_page.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/gitlab_first_sign_in.png b/doc/install/google_cloud_platform/img/gitlab_first_sign_in.png Binary files differdeleted file mode 100644 index 381c0fe48a5..00000000000 --- a/doc/install/google_cloud_platform/img/gitlab_first_sign_in.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/gitlab_launch_button.png b/doc/install/google_cloud_platform/img/gitlab_launch_button.png Binary files differdeleted file mode 100644 index 50f66f66118..00000000000 --- a/doc/install/google_cloud_platform/img/gitlab_launch_button.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/launch_vm.png b/doc/install/google_cloud_platform/img/launch_vm.png Binary files differnew file mode 100644 index 00000000000..3fd13f232bb --- /dev/null +++ b/doc/install/google_cloud_platform/img/launch_vm.png diff --git a/doc/install/google_cloud_platform/img/new_gitlab_deployment_settings.png b/doc/install/google_cloud_platform/img/new_gitlab_deployment_settings.png Binary files differdeleted file mode 100644 index 00060841619..00000000000 --- a/doc/install/google_cloud_platform/img/new_gitlab_deployment_settings.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/ssh_terminal.png b/doc/install/google_cloud_platform/img/ssh_terminal.png Binary files differnew file mode 100644 index 00000000000..6a1a418d8e9 --- /dev/null +++ b/doc/install/google_cloud_platform/img/ssh_terminal.png diff --git a/doc/install/google_cloud_platform/img/ssh_via_button.png b/doc/install/google_cloud_platform/img/ssh_via_button.png Binary files differdeleted file mode 100644 index 26106f159ad..00000000000 --- a/doc/install/google_cloud_platform/img/ssh_via_button.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/img/vm_created.png b/doc/install/google_cloud_platform/img/vm_created.png Binary files differnew file mode 100644 index 00000000000..fb467f40838 --- /dev/null +++ b/doc/install/google_cloud_platform/img/vm_created.png diff --git a/doc/install/google_cloud_platform/img/vm_details.png b/doc/install/google_cloud_platform/img/vm_details.png Binary files differnew file mode 100644 index 00000000000..2d230416a4b --- /dev/null +++ b/doc/install/google_cloud_platform/img/vm_details.png diff --git a/doc/install/google_cloud_platform/index.md b/doc/install/google_cloud_platform/index.md index c6b767fff02..3389f0260f9 100644 --- a/doc/install/google_cloud_platform/index.md +++ b/doc/install/google_cloud_platform/index.md @@ -2,12 +2,7 @@  -The fastest way to get started on [Google Cloud Platform (GCP)][gcp] is through -the [Google Cloud Launcher][launcher] program. - -GitLab's official Google Launcher apps: -1. [GitLab Community Edition](https://console.cloud.google.com/launcher/details/gitlab-public/gitlab-community-edition?project=gitlab-public) -2. [GitLab Enterprise Edition](https://console.cloud.google.com/launcher/details/gitlab-public/gitlab-enterprise-edition?project=gitlab-public) +Gettung started with GitLab on a [Google Cloud Platform (GCP)][gcp] instance is quick and easy. ## Prerequisites @@ -17,84 +12,52 @@ There are only two prerequisites in order to install GitLab on GCP: 1. You need to sign up for the GCP program. If this is your first time, Google gives you [$300 credit for free][freetrial] to consume over a 60-day period. -Once you have performed those two steps, you can visit the -[GCP launcher console][console] which has a list of all the things you can -deploy on GCP. - - - -The next step is to find and install GitLab. +Once you have performed those two steps, you can [create a VM](#creating-the-vm). -## Configuring and deploying the VM +## Creating the VM To deploy GitLab on GCP you need to follow five simple steps: -1. Go to https://cloud.google.com/launcher and login with your Google credentials -1. Search for GitLab from GitLab Inc. (not the same as Bitnami) and click on - the tile. +1. Go to https://console.cloud.google.com/compute/instances and login with your Google credentials. -  +1. Click on **Create** -1. In the next page, you can see an overview of the GitLab VM as well as some - estimated costs. Click the **Launch on Compute Engine** button to choose the - hardware and network settings. +  -  +1. On the next page, you can select the type of VM as well as the + estimated costs. Provide the name of the instance, desired datacenter, and machine type. Note that GitLab recommends at least 2 vCPU's and 4GB of RAM. -1. In the settings page you can choose things like the datacenter where your GitLab - server will be hosted, the number of CPUs and amount of RAM, the disk size - and type, etc. Read GitLab's [requirements documentation][req] for more - details on what to choose depending on your needs. +  -  +1. Click **Change** under Boot disk to select the size, type, and desired operating system. GitLab supports a [variety of linux operating systems][req], including Ubuntu and Debian. Click **Select** when finished. -1. As a last step, hit **Deploy** when ready. The process will finish in a few - seconds. +  -  +1. As a last step allow HTTP and HTTPS traffic, then click **Create**. The process will finish in a few seconds. +## Installing GitLab -## Visiting GitLab for the first time +After a few seconds, the instance will be created and available to log in. The next step is to install GitLab onto the instance. -After a few seconds, GitLab will be successfully deployed and you should be -able to see the IP address that Google assigned to the VM, as well as the -credentials to the GitLab admin account. + - +1. Make a note of the IP address of the instance, as you will need that in a later step. +1. Click on the SSH button to connect to the instance. +1. A new window will appear, with you logged into the instance. -1. Click on the IP under **Site address** to visit GitLab. -1. Accept the self-signed certificate that Google automatically deployed in - order to securely reach GitLab's login page. -1. Use the username and password that are present in the Google console page - to login into GitLab and click **Sign in**. +  -  +1. Next, follow the instructions for installing GitLab for the operating system you choose, at https://about.gitlab.com/installation/. You can use the IP address from the step above, as the hostname. -Congratulations! GitLab is now installed and you can access it via your browser, -but we're not done yet. There are some steps you need to take in order to have -a fully functional GitLab installation. +1. Congratulations! GitLab is now installed and you can access it via your browser. To finish installation, open the URL in your browser and provide the initial administrator password. The username for this account is `root`. + +  ## Next steps These are the most important next steps to take after you installed GitLab for the first time. -### Changing the admin password and email - -Google assigned a random password for the GitLab admin account and you should -change it ASAP: - -1. Visit the GitLab admin page through the link in the Google console under - **Admin URL**. -1. Find the Administrator user under the **Users** page and hit **Edit**. -1. Change the email address to a real one and enter a new password. - -  - -1. Hit **Save changes** for the changes to take effect. -1. After changing the password, you will be signed out from GitLab. Use the - new credentials to login again. - ### Assigning a static IP By default, Google assigns an ephemeral IP to your instance. It is strongly @@ -112,7 +75,7 @@ here's how you configure GitLab to be aware of the change: 1. SSH into the VM. You can easily use the **SSH** button in the Google console and a new window will pop up. -  +  In the future you might want to set up [connecting with an SSH key][ssh] instead. @@ -161,7 +124,6 @@ Kerberos, etc. Here are some documents you might be interested in reading: - [GitLab Pages configuration](https://docs.gitlab.com/ce/administration/pages/index.html) - [GitLab Container Registry configuration](https://docs.gitlab.com/ce/administration/container_registry.html) -[console]: https://console.cloud.google.com/launcher "GCP launcher console" [freetrial]: https://console.cloud.google.com/freetrial "GCP free trial" [ip]: https://cloud.google.com/compute/docs/configure-instance-ip-addresses#promote_ephemeral_ip "Configuring an Instance's IP Addresses" [gcp]: https://cloud.google.com/ "Google Cloud Platform" diff --git a/doc/install/installation.md b/doc/install/installation.md index 6eb8890cc4f..1abbfd78738 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -93,9 +93,9 @@ Is the system packaged Git too old? Remove it and compile from source. # Download and compile from source cd /tmp - curl --remote-name --progress https://www.kernel.org/pub/software/scm/git/git-2.14.3.tar.gz - echo '023ffff6d3ba8a1bea779dfecc0ed0bb4ad68ab8601d14435dd8c08416f78d7f git-2.14.3.tar.gz' | shasum -a256 -c - && tar -xzf git-2.14.3.tar.gz - cd git-2.14.3/ + curl --remote-name --progress https://www.kernel.org/pub/software/scm/git/git-2.16.2.tar.gz + echo '9acc4339b7a2ab484eea69d705923271682b7058015219cf5a7e6ed8dee5b5fb git-2.16.2.tar.gz' | shasum -a256 -c - && tar -xzf git-2.16.2.tar.gz + cd git-2.16.2/ ./configure make prefix=/usr/local all @@ -162,13 +162,14 @@ page](https://golang.org/dl). ## 4. Node -Since GitLab 8.17, GitLab requires the use of node >= v4.3.0 to compile -javascript assets, and yarn >= v0.17.0 to manage javascript dependencies. -In many distros the versions provided by the official package repositories -are out of date, so we'll need to install through the following commands: +Since GitLab 8.17, GitLab requires the use of Node to compile javascript +assets, and Yarn to manage javascript dependencies. The current minimum +requirements for these are node >= v6.0.0 and yarn >= v1.2.0. In many distros +the versions provided by the official package repositories are out of date, so +we'll need to install through the following commands: - # install node v7.x - curl --location https://deb.nodesource.com/setup_7.x | sudo bash - + # install node v8.x + curl --location https://deb.nodesource.com/setup_8.x | sudo bash - sudo apt-get install -y nodejs curl --silent --show-error https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - @@ -299,9 +300,9 @@ sudo usermod -aG redis git ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 10-4-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 10-6-stable gitlab -**Note:** You can change `10-4-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `10-6-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It diff --git a/doc/install/kubernetes/gitlab_chart.md b/doc/install/kubernetes/gitlab_chart.md index 96968c1e3ab..84eeacac3fd 100644 --- a/doc/install/kubernetes/gitlab_chart.md +++ b/doc/install/kubernetes/gitlab_chart.md @@ -1,17 +1,17 @@ # GitLab Helm Chart -> **Note**: -* This chart is deprecated, and is being replaced by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). For more information on available charts, please see our [overview](index.md#chart-overview). -* These charts have been tested on Google Kubernetes Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). +> **Note:** +* This chart has been tested on Google Kubernetes Engine and Azure Container Service. + +**This chart is deprecated.** For small installations on Kubernetes today, we recommend the beta [`gitlab-omnibus` Helm chart](gitlab_omnibus.md). +A new [cloud native GitLab chart](index.md#cloud-native-gitlab-chart) is in development with increased scalability and resilience, among other benefits. The cloud native chart will replace both the `gitlab` and `gitlab-omnibus` charts when available later this year. -For more information on available GitLab Helm Charts, please see our [overview](index.md#chart-overview). +Due to the significant architectural changes, migrating will require backing up data out of this instance and restoring it into the new deployment. For more information on available GitLab Helm Charts, please see our [overview](index.md#chart-overview). ## Introduction The `gitlab` Helm chart deploys just GitLab into your Kubernetes cluster, and offers extensive configuration options. This chart requires advanced knowledge of Kubernetes to successfully use. We **strongly recommend** the [gitlab-omnibus](gitlab_omnibus.md) chart. -This chart is deprecated, and will be replaced by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). Due to the difficulty in supporting upgrades, migrating will require exporting data out of this instance and importing it into the new deployment. - This chart includes the following: - Deployment using the [gitlab-ce](https://hub.docker.com/r/gitlab/gitlab-ce) or [gitlab-ee](https://hub.docker.com/r/gitlab/gitlab-ee) container image diff --git a/doc/install/kubernetes/gitlab_omnibus.md b/doc/install/kubernetes/gitlab_omnibus.md index 5a5f8d67ff5..9c5258c2cdf 100644 --- a/doc/install/kubernetes/gitlab_omnibus.md +++ b/doc/install/kubernetes/gitlab_omnibus.md @@ -1,17 +1,18 @@ # GitLab-Omnibus Helm Chart -> **Note:** -* This Helm chart is in beta, and will be deprecated by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). -* These charts have been tested on Google Kubernetes Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/charts/charts.gitlab.io/issues). +> **Note:**. +* This chart has been tested on Google Kubernetes Engine and Azure Container Service. -This work is based partially on: https://github.com/lwolf/kubernetes-gitlab/. GitLab would like to thank Sergey Nuzhdin for his work. +**[This chart is beta](#limitations), and is the best way to install GitLab on Kubernetes today.** A new [cloud native GitLab chart](index.md#cloud-native-gitlab-chart) is in development with increased scalability and resilience, among other benefits. Once available, the cloud native chart will be the recommended installation method for Kubernetes, and this chart will be deprecated. For more information on available GitLab Helm Charts, please see our [overview](index.md#chart-overview). +This work is based partially on: https://github.com/lwolf/kubernetes-gitlab/. GitLab would like to thank Sergey Nuzhdin for his work. + ## Introduction This chart provides an easy way to get started with GitLab, provisioning an installation with nearly all functionality enabled. SSL is automatically provisioned via [Let's Encrypt](https://letsencrypt.org/). -This Helm chart is in beta, and will be deprecated by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md) once available. Due to the difficulty in supporting upgrades, migrating will require exporting data out of this instance and importing it into the new deployment. +This Helm chart is in beta, and is suited for small to medium deployments. It will be deprecated by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md) once available. Due to the significant architectural changes, migrating will require backing up data out of this instance and importing it into the new deployment. The deployment includes: diff --git a/doc/install/kubernetes/gitlab_runner_chart.md b/doc/install/kubernetes/gitlab_runner_chart.md index ca9c95aeced..1f53e12d5f8 100644 --- a/doc/install/kubernetes/gitlab_runner_chart.md +++ b/doc/install/kubernetes/gitlab_runner_chart.md @@ -31,7 +31,7 @@ The default configuration can always be found in the [values.yaml](https://gitla In order for GitLab Runner to function, your config file **must** specify the following: - - `gitlabURL` - the GitLab Server URL (with protocol) to register the runner against + - `gitlabUrl` - the GitLab Server URL (with protocol) to register the runner against - `runnerRegistrationToken` - The Registration Token for adding new Runners to the GitLab Server. This must be retrieved from your GitLab Instance. See the [GitLab Runner Documentation](../../ci/runners/README.md#creating-and-registering-a-runner) for more information. @@ -47,7 +47,7 @@ Here is a snippet of the important settings: ## The GitLab Server URL (with protocol) that want to register the runner against ## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register ## -gitlabURL: http://gitlab.your-domain.com/ +gitlabUrl: http://gitlab.your-domain.com/ ## The Registration Token for adding new Runners to the GitLab Server. This must ## be retreived from your GitLab Instance. diff --git a/doc/install/kubernetes/index.md b/doc/install/kubernetes/index.md index cd889e74487..aa9b8777359 100644 --- a/doc/install/kubernetes/index.md +++ b/doc/install/kubernetes/index.md @@ -10,7 +10,7 @@ should be deployed, upgraded, and configured. ## Chart Overview * **[GitLab-Omnibus](gitlab_omnibus.md)**: The best way to run GitLab on Kubernetes today, suited for small deployments. The chart is in beta and will be deprecated by the [cloud native GitLab chart](#cloud-native-gitlab-chart). -* **[Cloud Native GitLab Chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md)**: The next generation GitLab chart, currently in development. Will support large deployments with horizontal scaling of individual GitLab components. +* **[Cloud Native GitLab Chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md)**: The next generation GitLab chart, currently in alpha. Will support large deployments with horizontal scaling of individual GitLab components. * Other Charts * [GitLab Runner Chart](gitlab_runner_chart.md): For deploying just the GitLab Runner. * [Advanced GitLab Installation](gitlab_chart.md): Deprecated, being replaced by the [cloud native GitLab chart](#cloud-native-gitlab-chart). Provides additional deployment options, but provides less functionality out-of-the-box. @@ -35,9 +35,9 @@ By offering individual containers and charts, we will be able to provide a numbe * Potential for rolling updates and canaries within a service, * and plenty more. -This is a large project and will be worked on over the span of multiple releases. For the most up-to-date status and release information, please see our [tracking issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2420). We are planning to launch this chart in beta by the end of 2017. +Presently this chart is available in alpha for testing, and not recommended for production use. -Learn more about the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). +Learn more about the [cloud native GitLab chart here ](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md) and [here [Video]](https://youtu.be/Z6jWR8Z8dv8). ## Other Charts diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index 448cbe1077d..e6ccfccd33f 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -1,9 +1,12 @@ -# Getting started with OpenShift Origin 3 and GitLab +--- +author: Achilleas Pipinellis +author_gitlab: axil +level: intermediary +article_type: tutorial +date: 2016-06-28 +--- -> **[Article Type](../../development/writing_documentation.html#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Achilleas Pipinellis](https://gitlab.com/axil) || -> **Publication date:** 2016-06-28 +# Getting started with OpenShift Origin 3 and GitLab ## Introduction @@ -461,7 +464,9 @@ bother us. In any case, it is something to keep in mind when deploying GitLab on a production cluster. In order to deploy GitLab on a production cluster, you will need to assign the -GitLab service account to the `anyuid` Security Context. +GitLab service account to the `anyuid` [Security Context Constraints][scc]. + +For OpenShift v3.0, you will need to do this manually: 1. Edit the Security Context: ```sh @@ -474,6 +479,12 @@ GitLab service account to the `anyuid` Security Context. 1. Save and exit the editor +For OpenShift v3.1 and above, you can do: + +```sh +oc adm policy add-scc-to-user anyuid system:serviceaccount:gitlab:gitlab-ce-user +``` + ## Conclusion By now, you should have an understanding of the basic OpenShift Origin concepts @@ -510,3 +521,4 @@ PaaS and managing your applications with the ease of containers. [autoscaling]: https://docs.openshift.org/latest/dev_guide/pod_autoscaling.html "Documentation - Autoscale" [basic-cli]: https://docs.openshift.org/latest/cli_reference/basic_cli_operations.html "Documentation - Basic CLI operations" [openshift-docs]: https://docs.openshift.org "OpenShift documentation" +[scc]: https://docs.openshift.org/latest/admin_guide/manage_scc.html "Documentation - Managing Security Context Constraints"
\ No newline at end of file diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 4324b4ca0b8..1f2b4d9d3d9 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -137,6 +137,14 @@ CREATE EXTENSION pg_trgm; On some systems you may need to install an additional package (e.g. `postgresql-contrib`) for this extension to become available. +#### Additional requirements for GitLab Geo + +If you are using [GitLab Geo](https://docs.gitlab.com/ee/development/geo.html), the [tracking database](https://docs.gitlab.com/ee/development/geo.html#geo-tracking-database) also requires the `postgres_fdw` extension. + +``` +CREATE EXTENSION postgres_fdw; +``` + ## Unicorn Workers It's possible to increase the amount of unicorn workers and this will usually help to reduce the response time of the applications and increase the ability to handle parallel requests. @@ -192,3 +200,6 @@ use the CI features. We support the current and the previous major release of Firefox, Chrome/Chromium, Safari and Microsoft browsers (Microsoft Edge and Internet Explorer 11). Each time a new browser version is released, we begin supporting that version and stop supporting the third most recent version. + +Note: We do not support running GitLab with JavaScript disabled in the browser and have no plans of supporting that +in the future because we have features such as Issue Boards which require JavaScript extensively. diff --git a/doc/integration/auth0.md b/doc/integration/auth0.md index c39d7ab57c6..a75836a915a 100644 --- a/doc/integration/auth0.md +++ b/doc/integration/auth0.md @@ -56,7 +56,8 @@ for initial settings. "name" => "auth0", "args" => { client_id: 'YOUR_AUTH0_CLIENT_ID', client_secret: 'YOUR_AUTH0_CLIENT_SECRET', - namespace: 'YOUR_AUTH0_DOMAIN' + domain: 'YOUR_AUTH0_DOMAIN', + scope: 'openid profile email' } } ] @@ -69,8 +70,8 @@ for initial settings. args: { client_id: 'YOUR_AUTH0_CLIENT_ID', client_secret: 'YOUR_AUTH0_CLIENT_SECRET', - namespace: 'YOUR_AUTH0_DOMAIN' - } + domain: 'YOUR_AUTH0_DOMAIN', + scope: 'openid profile email' } } ``` diff --git a/doc/integration/google.md b/doc/integration/google.md index 07a700f7b64..ae1d848f439 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -35,7 +35,7 @@ In Google's side: 1. You should now be able to see a Client ID and Client secret. Note them down or keep this page open as you will need them later. -1. From the **Dashboard** select **ENABLE APIS AND SERVICES > Compute > Google Container Engine API > Enable** +1. From the **Dashboard** select **ENABLE APIS AND SERVICES > Compute > Google Kubernetes Engine API > Enable** On your GitLab server: diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 20087a981f9..3edde3de83d 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -32,6 +32,7 @@ contains some settings that are common for all providers. - [Auth0](auth0.md) - [Authentiq](../administration/auth/authentiq.md) - [OAuth2Generic](oauth2_generic.md) +- [JWT](../administration/auth/jwt.md) ## Initial OmniAuth Configuration diff --git a/doc/integration/openid_connect_provider.md b/doc/integration/openid_connect_provider.md index 56f367d841e..ad41be52045 100644 --- a/doc/integration/openid_connect_provider.md +++ b/doc/integration/openid_connect_provider.md @@ -39,6 +39,7 @@ Currently the following user information is shared with clients: | `website` | `string` | URL for the user's website | `profile` | `string` | URL for the user's GitLab profile | `picture` | `string` | URL for the user's GitLab avatar +| `groups` | `array` | Names of the groups the user is a member of [OpenID Connect]: http://openid.net/connect/ "OpenID Connect website" [doorkeeper-openid_connect]: https://github.com/doorkeeper-gem/doorkeeper-openid_connect "Doorkeeper::OpenidConnect website" diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 3ae98adc465..3f49432ce93 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -102,15 +102,15 @@ in your SAML IdP: installation to generate the correct value). 1. Change the values of `idp_cert_fingerprint`, `idp_sso_target_url`, - `name_identifier_format` to match your IdP. Check + `name_identifier_format` to match your IdP. If a fingerprint is used it must + be a SHA1 fingerprint; check [the omniauth-saml documentation](https://github.com/omniauth/omniauth-saml) - for details on these options. + for more details on these options. 1. Change the value of `issuer` to a unique name, which will identify the application to the IdP. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. For the changes to take effect, you must [reconfigure][] GitLab if you installed via Omnibus or [restart GitLab][] if you installed from source. 1. Register the GitLab SP in your SAML 2.0 IdP, using the application name specified in `issuer`. @@ -312,6 +312,7 @@ need to be validated using a fingerprint, a certificate or a validator. For this you need take the following into account: +- If a fingerprint is used, it must be the SHA1 fingerprint - If no certificate is provided in the settings, a fingerprint or fingerprint validator needs to be provided and the response from the server must contain a certificate (`<ds:KeyInfo><ds:X509Data><ds:X509Certificate>`) diff --git a/doc/integration/slash_commands.md b/doc/integration/slash_commands.md index 36a8844e953..7d73026a6c6 100644 --- a/doc/integration/slash_commands.md +++ b/doc/integration/slash_commands.md @@ -15,9 +15,10 @@ Taking the trigger term as `project-name`, the commands are: | `/project-name issue new <title> <shift+return> <description>` | Creates a new issue with title `<title>` and description `<description>` | | `/project-name issue show <id>` | Shows the issue with id `<id>` | | `/project-name issue search <query>` | Shows up to 5 issues matching `<query>` | +| `/project-name issue move <id> to <project>` | Moves issue ID `<id>` to `<project>` | | `/project-name deploy <from> to <to>` | Deploy from the `<from>` environment to the `<to>` environment | -Note that if you are using the [GitLab Slack application](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html) for +Note that if you are using the [GitLab Slack application](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html) for your GitLab.com projects, you need to [add the `gitlab` keyword at the beginning of the command](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html#usage). ## Issue commands diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 8d0afa9e692..7f028565412 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -44,7 +44,7 @@ This decision is made on a case-by-case basis. ## Upgrade recommendations -We encourage everyone to run the latest stable release to ensure that you can +We encourage everyone to run the [latest stable release](https://about.gitlab.com/blog/categories/release/) to ensure that you can easily upgrade to the most secure and feature-rich GitLab experience. In order to make sure you can easily run the most recent stable release, we are working hard to keep the update process simple and reliable. diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 2f916f5dea7..90187617c41 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -14,3 +14,4 @@ comments: false - [Webhooks](web_hooks.md) - [Import](import.md) of git repositories in bulk - [Rebuild authorized_keys file](http://docs.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators +- [Migrate Uploads](../administration/raketasks/uploads/migrate.md) diff --git a/doc/ssh/README.md b/doc/ssh/README.md index 33a2d7a88a7..aa14a39e4c9 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -35,8 +35,8 @@ to clipboard step. If you don't see the string or would like to generate a SSH key pair with a custom name continue onto the next step. -> -**Note:** Public SSH key may also be named as follows: +Note that Public SSH key may also be named as follows: + - `id_dsa.pub` - `id_ecdsa.pub` - `id_ed25519.pub` @@ -73,7 +73,7 @@ custom name continue onto the next step. key pair, but it is not required and you can skip creating a password by pressing enter. - >**Note:** + NOTE: **Note:** If you want to change the password of your SSH key pair, you can use `ssh-keygen -p <keyname>`. @@ -162,11 +162,13 @@ That's why it needs to uniquely map to a single user. ## Deploy keys +### Per-repository deploy keys + Deploy keys allow read-only or read-write (if enabled) access to one or multiple projects with a single SSH key pair. This is really useful for cloning repositories to your Continuous -Integration (CI) server. By using deploy keys, you don't have to setup a +Integration (CI) server. By using deploy keys, you don't have to set up a dummy user account. If you are a project master or owner, you can add a deploy key in the @@ -185,6 +187,47 @@ a group. Deploy keys can be shared between projects, you just need to add them to each project. +### Global shared deploy keys + +Global Shared Deploy keys allow read-only or read-write (if enabled) access to +be configured on any repository in the entire GitLab installation. + +This is really useful for integrating repositories to secured, shared Continuous +Integration (CI) services or other shared services. +GitLab administrators can set up the Global Shared Deploy key in GitLab and +add the private key to any shared systems. Individual repositories opt into +exposing their repsitory using these keys when a project masters (or higher) +authorizes a Global Shared Deploy key to be used with their project. + +Global Shared Keys can provide greater security compared to Per-Project Deploy +Keys since an administrator of the target integrated system is the only one +who needs to know and configure the private key. + +GitLab administrators set up Global Deploy keys in the Admin area under the +section **Deploy Keys**. Ensure keys have a meaningful title as that will be +the primary way for project masters and owners to identify the correct Global +Deploy key to add. For instance, if the key gives access to a SaaS CI instance, +use the name of that service in the key name if that is all it is used for. +When creating Global Shared Deploy keys, give some thought to the granularity +of keys - they could be of very narrow usage such as just a specific service or +of broader usage for something like "Anywhere you need to give read access to +your repository". + +Once a GitLab administrator adds the Global Deployment key, project masters +and owners can add it in project's **Settings > Repository** section by expanding the +**Deploy Key** section and clicking **Enable** next to the appropriate key listed +under **Public deploy keys available to any project**. + +NOTE: **Note:** +The heading **Public deploy keys available to any project** only appears +if there is at least one Global Deploy Key configured. + +CAUTION: **Warning:** +Defining Global Deploy Keys does not expose any given repository via +the key until that respository adds the Global Deploy Key to their project. +In this way the Global Deploy Keys enable access by other systems, but do +not implicitly give any access just by setting them up. + ## Applications ### Eclipse diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 144cd4c26b0..e88b787187c 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -20,7 +20,8 @@ project in an easy and automatic way: 1. [Auto Test](#auto-test) 1. [Auto Code Quality](#auto-code-quality) 1. [Auto SAST (Static Application Security Testing)](#auto-sast) -1. [Auto SAST for Docker images](#auto-sast-for-docker-images) +1. [Auto Dependency Scanning](#auto-dependency-scanning) +1. [Auto Container Scanning](#auto-container-scanning) 1. [Auto Review Apps](#auto-review-apps) 1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast) 1. [Auto Deploy](#auto-deploy) @@ -95,7 +96,9 @@ Auto Deploy, and Auto Monitoring will be silently skipped. The Auto DevOps base domain is required if you want to make use of [Auto Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It is defined -under the project's CI/CD settings while [enabling Auto DevOps](#enabling-auto-devops). +either under the project's CI/CD settings while +[enabling Auto DevOps](#enabling-auto-devops) or in instance-wide settings in +the CI/CD section. It can also be set at the project or group level as a variable, `AUTO_DEVOPS_DOMAIN`. A wildcard DNS A record matching the base domain is required, for example, @@ -198,24 +201,37 @@ static analysis and other code checks on the current code. The report is created, and is uploaded as an artifact which you can later download and check out. -In GitLab Enterprise Edition Starter, differences between the source and +In GitLab Starter, differences between the source and target branches are also [shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html). ### Auto SAST -> Introduced in [GitLab Enterprise Edition Ultimate][ee] 10.3. +> Introduced in [GitLab Ultimate][ee] 10.3. Static Application Security Testing (SAST) uses the -[gl-sast Docker image](https://gitlab.com/gitlab-org/gl-sast) to run static +[SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) to run static analysis on the current code and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -In GitLab Enterprise Edition Ultimate, any security warnings are also +In GitLab Ultimate, any security warnings are also [shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html). -### Auto SAST for Docker images +### Auto Dependency Scanning + +> Introduced in [GitLab Ultimate][ee] 10.7. + +Dependency Scanning uses the +[Dependency Scanning Docker image](https://gitlab.com/gitlab-org/security-products/dependency-scanning) +to run analysis on the project dependencies and checks for potential security issues. Once the +report is created, it's uploaded as an artifact which you can later download and +check out. + +In GitLab Ultimate, any security warnings are also +[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html). + +### Auto Container Scanning > Introduced in GitLab 10.4. @@ -225,8 +241,8 @@ Docker image and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -In GitLab Enterprise Edition Ultimate, any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/sast_docker.html). +In GitLab Ultimate, any security warnings are also +[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). ### Auto Review Apps @@ -256,7 +272,7 @@ be deleted. ### Auto DAST -> Introduced in [GitLab Enterprise Edition Ultimate][ee] 10.4. +> Introduced in [GitLab Ultimate][ee] 10.4. Dynamic Application Security Testing (DAST) uses the popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy) @@ -264,12 +280,12 @@ to perform an analysis on the current code and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -In GitLab Enterprise Edition Ultimate, any security warnings are also +In GitLab Ultimate, any security warnings are also [shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html). ### Auto Browser Performance Testing -> Introduced in [GitLab Enterprise Edition Premium][ee] 10.4. +> Introduced in [GitLab Premium][ee] 10.4. Auto Browser Performance Testing utilizes the [Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) to measure the performance of a web page. A JSON report is created and uploaded as an artifact, which includes the overall performance score for each page. By default, the root page of Review and Production environments will be tested. If you would like to add additional URL's to test, simply add the paths to a file named `.gitlab-urls.txt` in the root directory, one per line. For example: @@ -279,7 +295,7 @@ Auto Browser Performance Testing utilizes the [Sitespeed.io container](https://h /direction ``` -In GitLab Enterprise Edition Premium, performance differences between the source and target branches are [shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). +In GitLab Premium, performance differences between the source and target branches are [shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). ### Auto Deploy @@ -307,6 +323,18 @@ enable them. You can make use of [environment variables](#helm-chart-variables) to automatically scale your pod replicas. +It's important to note that when a project is deployed to a Kubernetes cluster, +it relies on a Docker image that has been pushed to the +[GitLab Container Registry](../../user/project/container_registry.md). Kubernetes +fetches this image and uses it to run the application. If the project is public, +the image can be accessed by Kubernetes without any authentication, allowing us +to have deployments more usable. If the project is private/internal, the +Registry requires credentials to pull the image. Currently, this is addressed +by providing `CI_JOB_TOKEN` as the password that can be used, but this token will +no longer be valid as soon as the deployment job finishes. This means that +Kubernetes can run the application, but in case it should be restarted or +executed somewhere else, it cannot be accessed again. + ### Auto Monitoring NOTE: **Note:** @@ -395,7 +423,7 @@ If you want to modify the CI/CD pipeline used by Auto DevOps, you can copy the Assuming that your project is new or it doesn't have a `.gitlab-ci.yml` file present: -1. From your project home page, either click on the "Set up CI" button, or click +1. From your project home page, either click on the "Set up CI/CD" button, or click on the plus button and (`+`), then "New file" 1. Pick `.gitlab-ci.yml` as the template type 1. Select "Auto-DevOps" from the template dropdown @@ -593,4 +621,4 @@ curl --data "value=true" --header "PRIVATE-TOKEN: personal_access_token" https:/ [postgresql]: https://www.postgresql.org/ [Auto DevOps template]: https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Auto-DevOps.gitlab-ci.yml [GitLab Omnibus Helm Chart]: ../../install/kubernetes/gitlab_omnibus.md -[ee]: https://about.gitlab.com/gitlab-ee/ +[ee]: https://about.gitlab.com/products/ diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index 4858735ee86..15567715c98 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -102,6 +102,11 @@ running: kubectl get svc ruby-app-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` +NOTE: **Note:** +If your ingress controller has been installed in a different way, you can find +how to get the external IP address in the +[Cluster documentation](../../user/project/clusters/index.md#getting-the-external-ip-address). + Use this IP address to configure your DNS. This part heavily depends on your preferences and domain provider. But in case you are not sure, just create an A record with a wildcard host like `*.<your-domain>`. diff --git a/doc/topics/git/how_to_install_git/index.md b/doc/topics/git/how_to_install_git/index.md index 7fb578e9ea8..6c909a1ba86 100644 --- a/doc/topics/git/how_to_install_git/index.md +++ b/doc/topics/git/how_to_install_git/index.md @@ -1,9 +1,12 @@ -# Installing Git +--- +author: Sean Packham +author_gitlab: SeanPackham +level: beginner +article_type: user guide +date: 2017-05-15 +--- -> **[Article Type](../../../development/writing_documentation.html#types-of-technical-articles):** user guide || -> **Level:** beginner || -> **Author:** [Sean Packham](https://gitlab.com/SeanPackham) || -> **Publication date:** 2017-05-15 +# Installing Git To begin contributing to GitLab projects you will need to install the Git client on your computer. diff --git a/doc/topics/git/numerous_undo_possibilities_in_git/index.md b/doc/topics/git/numerous_undo_possibilities_in_git/index.md index 6a2f7b30dd3..4cb8f083fb5 100644 --- a/doc/topics/git/numerous_undo_possibilities_in_git/index.md +++ b/doc/topics/git/numerous_undo_possibilities_in_git/index.md @@ -1,9 +1,12 @@ -# Numerous undo possibilities in Git +--- +author: Crt Mori +author_gitlab: Letme +level: intermediary +article_type: tutorial +date: 2017-05-15 +--- -> **[Article Type](../../../development/writing_documentation.md#types-of-technical-articles):** tutorial || -> **Level:** intermediary || -> **Author:** [Crt Mori](https://gitlab.com/Letme) || -> **Publication date:** 2017-08-17 +# Numerous undo possibilities in Git ## Introduction diff --git a/doc/update/10.4-to-10.5.md b/doc/update/10.4-to-10.5.md new file mode 100644 index 00000000000..313419ed13d --- /dev/null +++ b/doc/update/10.4-to-10.5.md @@ -0,0 +1,361 @@ +--- +comments: false +--- + +# From 10.4 to 10.5 + +Make sure you view this update guide from the tag (version) of GitLab you would +like to install. In most cases this should be the highest numbered production +tag (without rc in it). You can select the tag in the version dropdown at the +top left corner of GitLab (below the menu bar). + +If the highest number stable branch is unclear please check the +[GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation +guide links by version. + +### 1. Stop server + +```bash +sudo service gitlab stop +``` + +### 2. Backup + +NOTE: If you installed GitLab from source, make sure `rsync` is installed. + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 3. Update Ruby + +NOTE: GitLab 9.0 and higher only support Ruby 2.3.x and dropped support for Ruby 2.1.x. Be +sure to upgrade your interpreter if necessary. + +You can check which version you are running with `ruby -v`. + +Download and compile Ruby: + +```bash +mkdir /tmp/ruby && cd /tmp/ruby +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.6.tar.gz +echo '4e6a0f828819e15d274ae58485585fc8b7caace0 ruby-2.3.6.tar.gz' | shasum -c - && tar xzf ruby-2.3.6.tar.gz +cd ruby-2.3.6 +./configure --disable-install-rdoc +make +sudo make install +``` + +Install Bundler: + +```bash +sudo gem install bundler --no-ri --no-rdoc +``` + +### 4. Update Node + +GitLab now runs [webpack](http://webpack.js.org) to compile frontend assets. +We require a minimum version of node v6.0.0. + +You can check which version you are running with `node -v`. If you are running +a version older than `v6.0.0` you will need to update to a newer version. You +can find instructions to install from community maintained packages or compile +from source at the nodejs.org website. + +<https://nodejs.org/en/download/> + +Since 8.17, GitLab requires the use of yarn `>= v0.17.0` to manage +JavaScript dependencies. + +```bash +curl --silent --show-error https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - +echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list +sudo apt-get update +sudo apt-get install yarn +``` + +More information can be found on the [yarn website](https://yarnpkg.com/en/docs/install). + +### 5. Update Go + +NOTE: GitLab 9.2 and higher only supports Go 1.8.3 and dropped support for Go +1.5.x through 1.7.x. Be sure to upgrade your installation if necessary. + +You can check which version you are running with `go version`. + +Download and install Go: + +```bash +# Remove former Go installation folder +sudo rm -rf /usr/local/go + +curl --remote-name --progress https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz +echo '1862f4c3d3907e59b04a757cfda0ea7aa9ef39274af99a784f5be843c80c6772 go1.8.3.linux-amd64.tar.gz' | shasum -a256 -c - && \ + sudo tar -C /usr/local -xzf go1.8.3.linux-amd64.tar.gz +sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ +rm go1.8.3.linux-amd64.tar.gz +``` + +### 6. Get latest code + +```bash +cd /home/git/gitlab + +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +sudo -u git -H git checkout -- locale +``` + +For GitLab Community Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 10-5-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 10-5-stable-ee +``` + +### 7. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell + +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION) +sudo -u git -H bin/compile +``` + +### 8. Update gitlab-workhorse + +Install and compile gitlab-workhorse. GitLab-Workhorse uses +[GNU Make](https://www.gnu.org/software/make/). +If you are not using Linux you may have to run `gmake` instead of +`make` below. + +```bash +cd /home/git/gitlab-workhorse + +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION) +sudo -u git -H make +``` + +### 9. Update Gitaly + +#### New Gitaly configuration options required + +In order to function Gitaly needs some additional configuration information. Below we assume you installed Gitaly in `/home/git/gitaly` and GitLab Shell in `/home/git/gitlab-shell`. + +```shell +echo ' +[gitaly-ruby] +dir = "/home/git/gitaly/ruby" + +[gitlab-shell] +dir = "/home/git/gitlab-shell" +' | sudo -u git tee -a /home/git/gitaly/config.toml +``` + +#### Check Gitaly configuration + +Due to a bug in the `rake gitlab:gitaly:install` script your Gitaly +configuration file may contain syntax errors. The block name +`[[storages]]`, which may occur more than once in your `config.toml` +file, should be `[[storage]]` instead. + +```shell +sudo -u git -H sed -i.pre-10.1 's/\[\[storages\]\]/[[storage]]/' /home/git/gitaly/config.toml +``` + +#### Compile Gitaly + +```shell +cd /home/git/gitaly +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) +sudo -u git -H make +``` + +### 10. Update MySQL permissions + +If you are using MySQL you need to grant the GitLab user the necessary +permissions on the database: + +```bash +mysql -u root -p -e "GRANT TRIGGER ON \`gitlabhq_production\`.* TO 'git'@'localhost';" +``` + +If you use MySQL with replication, or just have MySQL configured with binary logging, +you will need to also run the following on all of your MySQL servers: + +```bash +mysql -u root -p -e "SET GLOBAL log_bin_trust_function_creators = 1;" +``` + +You can make this setting permanent by adding it to your `my.cnf`: + +``` +log_bin_trust_function_creators=1 +``` + +### 11. Update configuration files + +#### New configuration options for `gitlab.yml` + +There might be configuration options available for [`gitlab.yml`][yaml]. View them with the command below and apply them manually to your current `gitlab.yml`: + +```sh +cd /home/git/gitlab + +git diff origin/10-4-stable:config/gitlab.yml.example origin/10-5-stable:config/gitlab.yml.example +``` + +#### Nginx configuration + +Ensure you're still up-to-date with the latest NGINX configuration changes: + +```sh +cd /home/git/gitlab + +# For HTTPS configurations +git diff origin/10-4-stable:lib/support/nginx/gitlab-ssl origin/10-5-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/10-4-stable:lib/support/nginx/gitlab origin/10-5-stable:lib/support/nginx/gitlab +``` + +If you are using Strict-Transport-Security in your installation to continue using it you must enable it in your Nginx +configuration as GitLab application no longer handles setting it. + +If you are using Apache instead of NGINX please see the updated [Apache templates]. +Also note that because Apache does not support upstreams behind Unix sockets you +will need to let gitlab-workhorse listen on a TCP port. You can do this +via [/etc/default/gitlab]. + +[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache +[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-5-stable/lib/support/init.d/gitlab.default.example#L38 + +#### SMTP configuration + +If you're installing from source and use SMTP to deliver mail, you will need to add the following line +to config/initializers/smtp_settings.rb: + +```ruby +ActionMailer::Base.delivery_method = :smtp +``` + +See [smtp_settings.rb.sample] as an example. + +[smtp_settings.rb.sample]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-5-stable/config/initializers/smtp_settings.rb.sample#L13 + +#### Init script + +There might be new configuration options available for [`gitlab.default.example`][gl-example]. View them with the command below and apply them manually to your current `/etc/default/gitlab`: + +```sh +cd /home/git/gitlab + +git diff origin/10-4-stable:lib/support/init.d/gitlab.default.example origin/10-5-stable:lib/support/init.d/gitlab.default.example +``` + +Ensure you're still up-to-date with the latest init script changes: + +```bash +cd /home/git/gitlab + +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +For Ubuntu 16.04.1 LTS: + +```bash +sudo systemctl daemon-reload +``` + +### 12. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment + +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment + +# Optional: clean up old gems +sudo -u git -H bundle clean + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Compile GetText PO files + +sudo -u git -H bundle exec rake gettext:compile RAILS_ENV=production + +# Update node dependencies and recompile assets +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production + +# Clean up cache +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + +**MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). + +### 13. Start application + +```bash +sudo service gitlab start +sudo service nginx restart +``` + +### 14. Check application status + +Check if GitLab and its environment are configured correctly: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` + +To make sure you didn't miss anything run a more thorough check: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production +``` + +If all items are green, then congratulations, the upgrade is complete! + +## Things went south? Revert to previous version (10.4) + +### 1. Revert the code to the previous version + +Follow the [upgrade guide from 10.3 to 10.4](10.3-to-10.4.md), except for the +database migration (the backup is already migrated to the previous version). + +### 2. Restore from the backup + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. + +[yaml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-5-stable/config/gitlab.yml.example +[gl-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-5-stable/lib/support/init.d/gitlab.default.example diff --git a/doc/update/10.5-to-10.6.md b/doc/update/10.5-to-10.6.md new file mode 100644 index 00000000000..2f90fb62c4a --- /dev/null +++ b/doc/update/10.5-to-10.6.md @@ -0,0 +1,361 @@ +--- +comments: false +--- + +# From 10.5 to 10.6 + +Make sure you view this update guide from the tag (version) of GitLab you would +like to install. In most cases this should be the highest numbered production +tag (without rc in it). You can select the tag in the version dropdown at the +top left corner of GitLab (below the menu bar). + +If the highest number stable branch is unclear please check the +[GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation +guide links by version. + +### 1. Stop server + +```bash +sudo service gitlab stop +``` + +### 2. Backup + +NOTE: If you installed GitLab from source, make sure `rsync` is installed. + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 3. Update Ruby + +NOTE: GitLab 9.0 and higher only support Ruby 2.3.x and dropped support for Ruby 2.1.x. Be +sure to upgrade your interpreter if necessary. + +You can check which version you are running with `ruby -v`. + +Download and compile Ruby: + +```bash +mkdir /tmp/ruby && cd /tmp/ruby +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.6.tar.gz +echo '4e6a0f828819e15d274ae58485585fc8b7caace0 ruby-2.3.6.tar.gz' | shasum -c - && tar xzf ruby-2.3.6.tar.gz +cd ruby-2.3.6 +./configure --disable-install-rdoc +make +sudo make install +``` + +Install Bundler: + +```bash +sudo gem install bundler --no-ri --no-rdoc +``` + +### 4. Update Node + +GitLab utilizes [webpack](http://webpack.js.org) to compile frontend assets. +This requires a minimum version of node v6.0.0. + +You can check which version you are running with `node -v`. If you are running +a version older than `v6.0.0` you will need to update to a newer version. You +can find instructions to install from community maintained packages or compile +from source at the nodejs.org website. + +<https://nodejs.org/en/download/> + +GitLab also requires the use of yarn `>= v1.2.0` to manage JavaScript +dependencies. + +```bash +curl --silent --show-error https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - +echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list +sudo apt-get update +sudo apt-get install yarn +``` + +More information can be found on the [yarn website](https://yarnpkg.com/en/docs/install). + +### 5. Update Go + +NOTE: GitLab 9.2 and higher only supports Go 1.8.3 and dropped support for Go +1.5.x through 1.7.x. Be sure to upgrade your installation if necessary. + +You can check which version you are running with `go version`. + +Download and install Go: + +```bash +# Remove former Go installation folder +sudo rm -rf /usr/local/go + +curl --remote-name --progress https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz +echo '1862f4c3d3907e59b04a757cfda0ea7aa9ef39274af99a784f5be843c80c6772 go1.8.3.linux-amd64.tar.gz' | shasum -a256 -c - && \ + sudo tar -C /usr/local -xzf go1.8.3.linux-amd64.tar.gz +sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ +rm go1.8.3.linux-amd64.tar.gz +``` + +### 6. Get latest code + +```bash +cd /home/git/gitlab + +sudo -u git -H git fetch --all --prune +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +sudo -u git -H git checkout -- locale +``` + +For GitLab Community Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 10-6-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 10-6-stable-ee +``` + +### 7. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell + +sudo -u git -H git fetch --all --tags --prune +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION) +sudo -u git -H bin/compile +``` + +### 8. Update gitlab-workhorse + +Install and compile gitlab-workhorse. GitLab-Workhorse uses +[GNU Make](https://www.gnu.org/software/make/). +If you are not using Linux you may have to run `gmake` instead of +`make` below. + +```bash +cd /home/git/gitlab-workhorse + +sudo -u git -H git fetch --all --tags --prune +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION) +sudo -u git -H make +``` + +### 9. Update Gitaly + +#### New Gitaly configuration options required + +In order to function Gitaly needs some additional configuration information. Below we assume you installed Gitaly in `/home/git/gitaly` and GitLab Shell in `/home/git/gitlab-shell`. + +```shell +echo ' +[gitaly-ruby] +dir = "/home/git/gitaly/ruby" + +[gitlab-shell] +dir = "/home/git/gitlab-shell" +' | sudo -u git tee -a /home/git/gitaly/config.toml +``` + +#### Check Gitaly configuration + +Due to a bug in the `rake gitlab:gitaly:install` script your Gitaly +configuration file may contain syntax errors. The block name +`[[storages]]`, which may occur more than once in your `config.toml` +file, should be `[[storage]]` instead. + +```shell +sudo -u git -H sed -i.pre-10.1 's/\[\[storages\]\]/[[storage]]/' /home/git/gitaly/config.toml +``` + +#### Compile Gitaly + +```shell +cd /home/git/gitaly +sudo -u git -H git fetch --all --tags --prune +sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) +sudo -u git -H make +``` + +### 10. Update MySQL permissions + +If you are using MySQL you need to grant the GitLab user the necessary +permissions on the database: + +```bash +mysql -u root -p -e "GRANT TRIGGER ON \`gitlabhq_production\`.* TO 'git'@'localhost';" +``` + +If you use MySQL with replication, or just have MySQL configured with binary logging, +you will need to also run the following on all of your MySQL servers: + +```bash +mysql -u root -p -e "SET GLOBAL log_bin_trust_function_creators = 1;" +``` + +You can make this setting permanent by adding it to your `my.cnf`: + +``` +log_bin_trust_function_creators=1 +``` + +### 11. Update configuration files + +#### New configuration options for `gitlab.yml` + +There might be configuration options available for [`gitlab.yml`][yaml]. View them with the command below and apply them manually to your current `gitlab.yml`: + +```sh +cd /home/git/gitlab + +git diff origin/10-5-stable:config/gitlab.yml.example origin/10-6-stable:config/gitlab.yml.example +``` + +#### Nginx configuration + +Ensure you're still up-to-date with the latest NGINX configuration changes: + +```sh +cd /home/git/gitlab + +# For HTTPS configurations +git diff origin/10-5-stable:lib/support/nginx/gitlab-ssl origin/10-6-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/10-5-stable:lib/support/nginx/gitlab origin/10-6-stable:lib/support/nginx/gitlab +``` + +If you are using Strict-Transport-Security in your installation to continue using it you must enable it in your Nginx +configuration as GitLab application no longer handles setting it. + +If you are using Apache instead of NGINX please see the updated [Apache templates]. +Also note that because Apache does not support upstreams behind Unix sockets you +will need to let gitlab-workhorse listen on a TCP port. You can do this +via [/etc/default/gitlab]. + +[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache +[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-6-stable/lib/support/init.d/gitlab.default.example#L38 + +#### SMTP configuration + +If you're installing from source and use SMTP to deliver mail, you will need to add the following line +to config/initializers/smtp_settings.rb: + +```ruby +ActionMailer::Base.delivery_method = :smtp +``` + +See [smtp_settings.rb.sample] as an example. + +[smtp_settings.rb.sample]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-6-stable/config/initializers/smtp_settings.rb.sample#L13 + +#### Init script + +There might be new configuration options available for [`gitlab.default.example`][gl-example]. View them with the command below and apply them manually to your current `/etc/default/gitlab`: + +```sh +cd /home/git/gitlab + +git diff origin/10-5-stable:lib/support/init.d/gitlab.default.example origin/10-6-stable:lib/support/init.d/gitlab.default.example +``` + +Ensure you're still up-to-date with the latest init script changes: + +```bash +cd /home/git/gitlab + +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +For Ubuntu 16.04.1 LTS: + +```bash +sudo systemctl daemon-reload +``` + +### 12. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment + +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment + +# Optional: clean up old gems +sudo -u git -H bundle clean + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Compile GetText PO files + +sudo -u git -H bundle exec rake gettext:compile RAILS_ENV=production + +# Update node dependencies and recompile assets +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production + +# Clean up cache +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + +**MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). + +### 13. Start application + +```bash +sudo service gitlab start +sudo service nginx restart +``` + +### 14. Check application status + +Check if GitLab and its environment are configured correctly: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` + +To make sure you didn't miss anything run a more thorough check: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production +``` + +If all items are green, then congratulations, the upgrade is complete! + +## Things went south? Revert to previous version (10.5) + +### 1. Revert the code to the previous version + +Follow the [upgrade guide from 10.4 to 10.5](10.4-to-10.5.md), except for the +database migration (the backup is already migrated to the previous version). + +### 2. Restore from the backup + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. + +[yaml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-6-stable/config/gitlab.yml.example +[gl-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/10-6-stable/lib/support/init.d/gitlab.default.example diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index fff47180099..44e9f6c5516 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -1,13 +1,15 @@ --- -last_updated: 2017-10-05 +last_updated: 2018-02-07 --- # Migrating from MySQL to PostgreSQL -> **Note:** This guide assumes you have a working Omnibus GitLab instance with +> **Note:** This guide assumes you have a working GitLab instance with > MySQL and want to migrate to bundled PostgreSQL database. -## Prerequisites +## Omnibus installation + +### Prerequisites First, we'll need to enable the bundled PostgreSQL database with up-to-date schema. Next, we'll use [pgloader](http://pgloader.io) to migrate the data @@ -19,7 +21,7 @@ Here's what you'll need to have installed: - Omnibus GitLab - MySQL -## Enable bundled PostgreSQL database +### Enable bundled PostgreSQL database 1. Stop GitLab: @@ -65,7 +67,7 @@ Here's what you'll need to have installed: After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. -## Migrate data from MySQL to PostgreSQL +### Migrate data from MySQL to PostgreSQL Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: @@ -104,122 +106,9 @@ the following: ----------------------------------------------- --------- --------- --------- -------------- public.abuse_reports 0 0 0 0.490s public.appearances 0 0 0 0.488s - public.approvals 0 0 0 0.273s - public.application_settings 1 1 0 0.266s - public.approvers 0 0 0 0.339s - public.approver_groups 0 0 0 0.357s - public.audit_events 1 1 0 0.410s - public.award_emoji 0 0 0 0.441s - public.boards 0 0 0 0.505s - public.broadcast_messages 0 0 0 0.498s - public.chat_names 0 0 0 0.576s - public.chat_teams 0 0 0 0.617s - public.ci_builds 0 0 0 0.611s - public.ci_group_variables 0 0 0 0.620s - public.ci_pipelines 0 0 0 0.599s - public.ci_pipeline_schedules 0 0 0 0.622s - public.ci_pipeline_schedule_variables 0 0 0 0.573s - public.ci_pipeline_variables 0 0 0 0.594s - public.ci_runners 0 0 0 0.533s - public.ci_runner_projects 0 0 0 0.584s - public.ci_sources_pipelines 0 0 0 0.564s - public.ci_stages 0 0 0 0.595s - public.ci_triggers 0 0 0 0.569s - public.ci_trigger_requests 0 0 0 0.596s - public.ci_variables 0 0 0 0.565s - public.container_repositories 0 0 0 0.605s - public.conversational_development_index_metrics 0 0 0 0.571s - public.deployments 0 0 0 0.607s - public.emails 0 0 0 0.602s - public.deploy_keys_projects 0 0 0 0.557s - public.events 160 160 0 0.677s - public.environments 0 0 0 0.567s - public.features 0 0 0 0.639s - public.events_for_migration 160 160 0 0.582s - public.feature_gates 0 0 0 0.579s - public.forked_project_links 0 0 0 0.660s - public.geo_nodes 0 0 0 0.686s - public.geo_event_log 0 0 0 0.626s - public.geo_repositories_changed_events 0 0 0 0.677s - public.geo_node_namespace_links 0 0 0 0.618s - public.geo_repository_renamed_events 0 0 0 0.696s - public.gpg_keys 0 0 0 0.704s - public.geo_repository_deleted_events 0 0 0 0.638s - public.historical_data 0 0 0 0.729s - public.geo_repository_updated_events 0 0 0 0.634s - public.index_statuses 0 0 0 0.746s - public.gpg_signatures 0 0 0 0.667s - public.issue_assignees 80 80 0 0.769s - public.identities 0 0 0 0.655s - public.issue_metrics 80 80 0 0.781s - public.issues 80 80 0 0.720s - public.labels 0 0 0 0.795s - public.issue_links 0 0 0 0.707s - public.label_priorities 0 0 0 0.793s - public.keys 0 0 0 0.734s - public.lfs_objects 0 0 0 0.812s - public.label_links 0 0 0 0.725s - public.licenses 0 0 0 0.813s - public.ldap_group_links 0 0 0 0.751s - public.members 52 52 0 0.830s - public.lfs_objects_projects 0 0 0 0.738s - public.merge_requests_closing_issues 0 0 0 0.825s - public.lists 0 0 0 0.769s - public.merge_request_diff_commits 0 0 0 0.840s - public.merge_request_metrics 0 0 0 0.837s - public.merge_requests 0 0 0 0.753s - public.merge_request_diffs 0 0 0 0.771s - public.namespaces 30 30 0 0.874s - public.merge_request_diff_files 0 0 0 0.775s - public.notes 0 0 0 0.849s - public.milestones 40 40 0 0.799s - public.oauth_access_grants 0 0 0 0.979s - public.namespace_statistics 0 0 0 0.797s - public.oauth_applications 0 0 0 0.899s - public.notification_settings 72 72 0 0.818s - public.oauth_access_tokens 0 0 0 0.807s - public.pages_domains 0 0 0 0.958s - public.oauth_openid_requests 0 0 0 0.832s - public.personal_access_tokens 0 0 0 0.965s - public.projects 8 8 0 0.987s - public.path_locks 0 0 0 0.925s - public.plans 0 0 0 0.923s - public.project_features 8 8 0 0.985s - public.project_authorizations 66 66 0 0.969s - public.project_import_data 8 8 0 1.002s - public.project_statistics 8 8 0 1.001s - public.project_group_links 0 0 0 0.949s - public.project_mirror_data 0 0 0 0.972s - public.protected_branch_merge_access_levels 0 0 0 1.017s - public.protected_branches 0 0 0 0.969s - public.protected_branch_push_access_levels 0 0 0 0.991s - public.protected_tags 0 0 0 1.009s - public.protected_tag_create_access_levels 0 0 0 0.985s - public.push_event_payloads 0 0 0 1.041s - public.push_rules 0 0 0 0.999s - public.redirect_routes 0 0 0 1.020s - public.remote_mirrors 0 0 0 1.034s - public.releases 0 0 0 0.993s - public.schema_migrations 896 896 0 1.057s - public.routes 38 38 0 1.021s - public.services 0 0 0 1.055s - public.sent_notifications 0 0 0 1.003s - public.slack_integrations 0 0 0 1.022s - public.spam_logs 0 0 0 1.024s - public.snippets 0 0 0 1.058s - public.subscriptions 0 0 0 1.069s - public.taggings 0 0 0 1.099s - public.timelogs 0 0 0 1.104s - public.system_note_metadata 0 0 0 1.038s - public.tags 0 0 0 1.034s - public.trending_projects 0 0 0 1.140s - public.uploads 0 0 0 1.129s - public.todos 80 80 0 1.085s - public.users_star_projects 0 0 0 1.153s - public.u2f_registrations 0 0 0 1.061s - public.web_hooks 0 0 0 1.179s - public.users 26 26 0 1.163s - public.user_agent_details 0 0 0 1.068s + . + . + . public.web_hook_logs 0 0 0 1.080s ----------------------------------------------- --------- --------- --------- -------------- COPY Threads Completion 4 4 0 2.008s @@ -240,9 +129,9 @@ the following: Now, you can verify that everything worked by visiting GitLab. -## Troubleshooting +### Troubleshooting -### Permissions +#### Permissions Note that the PostgreSQL user that you use for the above MUST have **superuser** privileges. Otherwise, you may see a similar message to the following: @@ -256,7 +145,7 @@ debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL; ``` -### Experiencing 500 errors after the migration +#### Experiencing 500 errors after the migration If you experience 500 errors after the migration, try to clear the cache: @@ -265,3 +154,130 @@ sudo gitlab-rake cache:clear ``` [reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure + +## Source installation + +### Prerequisites + +#### Install PostgreSQL and create database + +See [installation guide](../install/installation.md#6-database). + +#### Install [pgloader](http://pgloader.io) 3.4.1+ + +Install directly from your distro: +``` bash +sudo apt-get install pgloader +``` + +If this version is too old, use PostgreSQL's repository: +``` bash +# add repository +sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' + +# add key +sudo apt-get install wget ca-certificates +wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - + +# install package +sudo apt-get update +sudo apt-get install pgloader +``` + +### Enable bundled PostgreSQL database + +1. Stop GitLab: + + ``` bash + sudo service gitlab stop + ``` + +1. Switch database from MySQL to PostgreSQL + + ``` bash + cd /home/git/gitlab + sudo -u git mv config/database.yml config/database.yml.bak + sudo -u git cp config/database.yml.postgresql config/database.yml + sudo -u git -H chmod o-rwx config/database.yml + ``` + +1. Run the following commands to prepare the schema: + + ``` bash + sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production + ``` + +After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. + +### Migrate data from MySQL to PostgreSQL + +Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: + +1. Save the following snippet in a `commands.load` file, and edit with your + MySQL `username`, `password` and `host`: + + ``` + LOAD DATABASE + FROM mysql://username:password@host/gitlabhq_production + INTO postgresql://postgres@unix://var/run/postgresql:/gitlabhq_production + + WITH include no drop, truncate, disable triggers, create no tables, + create no indexes, preserve index names, no foreign keys, + data only + + ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public' + + ; + ``` + +1. Start the migration: + + ``` bash + sudo -u postgres pgloader commands.load + ``` + +1. Once the migration finishes, you should see a summary table that looks like +the following: + + + ``` + table name read imported errors total time + ----------------------------------------------- --------- --------- --------- -------------- + fetch meta data 119 119 0 0.388s + Truncate 119 119 0 1.134s + ----------------------------------------------- --------- --------- --------- -------------- + public.abuse_reports 0 0 0 0.490s + public.appearances 0 0 0 0.488s + . + . + . + public.web_hook_logs 0 0 0 1.080s + ----------------------------------------------- --------- --------- --------- -------------- + COPY Threads Completion 4 4 0 2.008s + Reset Sequences 113 113 0 0.304s + Install Comments 0 0 0 0.000s + ----------------------------------------------- --------- --------- --------- -------------- + Total import time 1894 1894 0 12.497s + ``` + + If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See + the [troubleshooting guide](#Troubleshooting) for more details. + +1. Start GitLab: + + ``` bash + sudo service gitlab start + ``` + +Now, you can verify that everything worked by visiting GitLab. + +### Troubleshooting + +#### Experiencing 500 errors after the migration + +If you experience 500 errors after the migration, try to clear the cache: + +``` bash +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + diff --git a/doc/user/admin_area/settings/img/update-available.png b/doc/user/admin_area/settings/img/update-available.png Binary files differnew file mode 100644 index 00000000000..0dafdad618e --- /dev/null +++ b/doc/user/admin_area/settings/img/update-available.png diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index d874688cc29..381efdf5d67 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -8,20 +8,26 @@ under **Admin area > Settings > Usage statistics**. ## Version check -GitLab can inform you when an update is available and the importance of it. +If enabled, version check will inform you if a new version is available and the +importance of it through a status. This is shown on the help page (i.e. `/help`) +for all signed in users, and on the admin pages. The statuses are: -No information other than the GitLab version and the instance's hostname (through the HTTP referer) -are collected. +* Green: You are running the latest version of GitLab. +* Orange: An updated version of GitLab is available. +* Red: The version of GitLab you are running is vulnerable. You should install + the latest version with security fixes as soon as possible. -In the **Overview** tab you can see if your GitLab version is up to date. There -are three cases: 1) you are up to date (green), 2) there is an update available -(yellow) and 3) your version is vulnerable and a security fix is released (red). + -In any case, you will see a message informing you of the state and the -importance of the update. +GitLab Inc. collects your instance's version and hostname (through the HTTP +referer) as part of the version check. No other information is collected. -If enabled, the version status will also be shown in the help page (`/help`) -for all signed in users. +This information is used, among other things, to identify to which versions +patches will need to be back ported, making sure active GitLab instances remain +secure. + +If you disable version check, this information will not be collected. Enable or +disable the version check at **Admin area > Settings > Usage statistics**. ## Usage ping diff --git a/doc/user/feature_highlight.md b/doc/user/feature_highlight.md new file mode 100644 index 00000000000..bd98ea00757 --- /dev/null +++ b/doc/user/feature_highlight.md @@ -0,0 +1,15 @@ +# Feature highlight + +> [Introduced][ce-16379] in GitLab 10.5 + +Feature highlights are represented by a pulsing blue dot. Hovering over the dot +will open up callout with more information. +They are used to emphasize a certain feature and make something more visible to the user. + +You can dismiss any feature highlight permanently by clicking the "Got it" link +at the bottom of the callout. There isn't a way to restore the feature highlight +after it has been dismissed. + + + +[ce-16379]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16379 diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md new file mode 100644 index 00000000000..d5f77191938 --- /dev/null +++ b/doc/user/gitlab_com/index.md @@ -0,0 +1,270 @@ +# GitLab.com settings + +In this page you will find information about the settings that are used on +[GitLab.com](https://about.gitlab.com/pricing). + +## SSH host keys fingerprints + +Below are the fingerprints for GitLab.com's SSH host keys. + +| Algorithm | MD5 | SHA256 | +| --------- | --- | ------- | +| DSA | `7a:47:81:3a:ee:89:89:64:33:ca:44:52:3d:30:d4:87` | `p8vZBUOR0XQz6sYiaWSMLmh0t9i8srqYKool/Xfdfqw` | +| ECDSA | `f1:d0:fb:46:73:7a:70:92:5a:ab:5d:ef:43:e2:1c:35` | `HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw` | +| ED25519 | `2e:65:6a:c8:cf:bf:b2:8b:9a:bd:6d:9f:11:5c:12:16` | `eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8` | +| RSA | `b6:03:0e:39:97:9e:d0:e7:24:ce:a3:77:3e:01:42:09` | `ROQFvPThGrW4RuWLoL9tq9I9zJ42fK4XywyRtbOz/EQ` | + +## Mail configuration + +GitLab.com sends emails from the `mg.gitlab.com` domain via [Mailgun] and has +its own dedicated IP address (`198.61.254.240`). + +## Alternative SSH port + +GitLab.com can be reached via a [different SSH port][altssh] for `git+ssh`. + +| Setting | Value | +| --------- | ------------------- | +| `Hostname` | `altssh.gitlab.com` | +| `Port` | `443` | + +An example `~/.ssh/config` is the following: + +``` +Host gitlab.com + Hostname altssh.gitlab.com + User git + Port 443 + PreferredAuthentications publickey + IdentityFile ~/.ssh/gitlab +``` + +## GitLab Pages + +Below are the settings for [GitLab Pages]. + +| Setting | GitLab.com | Default | +| ----------------------- | ---------------- | ------------- | +| Domain name | `gitlab.io` | - | +| IP address | `52.167.214.135` | - | +| Custom domains support | yes | no | +| TLS certificates support| yes | no | + +The maximum size of your Pages site is regulated by the artifacts maximum size +which is part of [GitLab CI/CD](#gitlab-ci-cd). + +## GitLab CI/CD + +Below are the current settings regarding [GitLab CI/CD](../../ci/README.md). + +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ------------- | +| Artifacts maximum size | 1G | 100M | + +## Repository size limit + +The maximum size your Git repository is allowed to be including LFS. + +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ------------- | +| Repository size including LFS | 10G | Unlimited | + +## Shared Runners + +Shared Runners on GitLab.com run in [autoscale mode] and powered by +DigitalOcean. Autoscaling means reduced waiting times to spin up builds, +and isolated VMs for each project, thus maximizing security. + +They're free to use for public open source projects and limited to 2000 CI +minutes per month per group for private projects. Read about all +[GitLab.com plans](https://about.gitlab.com/pricing/). + +All your builds run on 2GB (RAM) ephemeral instances, with CoreOS and the latest +Docker Engine installed. The default region of the VMs is NYC. + +Below are the shared Runners settings. + +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ---------- | +| [GitLab Runner] | [Runner versions dashboard][ci_version_dashboard] | - | +| Executor | `docker+machine` | - | +| Default Docker image | `ruby:2.1` | - | +| `privileged` (run [Docker in Docker]) | `true` | `false` | + +[ci_version_dashboard]: https://monitor.gitlab.net/dashboard/db/ci?refresh=5m&orgId=1&panelId=12&fullscreen&from=now-1h&to=now&var-runner_type=All&var-cache_server=All&var-gl_monitor_fqdn=postgres-01.db.prd.gitlab.com&var-has_minutes=yes&var-hanging_droplets_cleaner=All&var-droplet_zero_machines_cleaner=All&var-runner_job_failure_reason=All&theme=light + +### `config.toml` + +The full contents of our `config.toml` are: + +```toml +[[runners]] + name = "docker-auto-scale" + limit = X + request_concurrency = X + url = "https://gitlab.com/ci" + token = "SHARED_RUNNER_TOKEN" + executor = "docker+machine" + environment = [ + "DOCKER_DRIVER=overlay2" + ] + [runners.docker] + image = "ruby:2.1" + privileged = true + [runners.machine] + IdleCount = 40 + IdleTime = 1800 + MaxBuilds = 1 + MachineDriver = "digitalocean" + MachineName = "machine-%s-digital-ocean-2gb" + MachineOptions = [ + "digitalocean-image=coreos-stable", + "digitalocean-ssh-user=core", + "digitalocean-access-token=DIGITAL_OCEAN_ACCESS_TOKEN", + "digitalocean-region=nyc1", + "digitalocean-size=2gb", + "digitalocean-private-networking", + "digitalocean-userdata=/etc/gitlab-runner/cloudinit.sh", + "engine-registry-mirror=http://IP_TO_OUR_REGISTRY_MIRROR" + ] + [runners.cache] + Type = "s3" + ServerAddress = "IP_TO_OUR_CACHE_SERVER" + AccessKey = "ACCESS_KEY" + SecretKey = "ACCESS_SECRET_KEY" + BucketName = "runner" + Shared = true +``` + +## Sidekiq + +GitLab.com runs [Sidekiq][sidekiq] with arguments `--timeout=4 --concurrency=4` +and the following environment variables: + +| Setting | GitLab.com | Default | +|-------- |----------- |-------- | +| `SIDEKIQ_MEMORY_KILLER_MAX_RSS` | `1000000` | `1000000` | +| `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL` | `SIGKILL` | - | +| `SIDEKIQ_LOG_ARGUMENTS` | `1` | - | + +## Cron jobs + +Periodically executed jobs by Sidekiq, to self-heal Gitlab, do external +synchronizations, run scheduled pipelines, etc.: + +| Setting | GitLab.com | Default | +|-------- |------------- |------------- | +| `pipeline_schedule_worker` | `19 * * * *` | `19 * * * *` | + +## PostgreSQL + +GitLab.com being a fairly large installation of GitLab means we have changed +various PostgreSQL settings to better suit our needs. For example, we use +streaming replication and servers in hot-standby mode to balance queries across +different database servers. + +The list of GitLab.com specific settings (and their defaults) is as follows: + +| Setting | GitLab.com | Default | +|:------------------------------------|:--------------------------------------------------------------------|:--------------------------------------| +| archive_command | `/usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-push %p` | empty | +| archive_mode | on | off | +| autovacuum_analyze_scale_factor | 0.01 | 0.01 | +| autovacuum_max_workers | 6 | 3 | +| autovacuum_vacuum_cost_limit | 1000 | -1 | +| autovacuum_vacuum_scale_factor | 0.01 | 0.02 | +| checkpoint_completion_target | 0.7 | 0.9 | +| checkpoint_segments | 32 | 10 | +| effective_cache_size | 338688MB | Based on how much memory is available | +| hot_standby | on | off | +| hot_standby_feedback | on | off | +| log_autovacuum_min_duration | 0 | -1 | +| log_checkpoints | on | off | +| log_line_prefix | `%t [%p]: [%l-1] ` | empty | +| log_min_duration_statement | 1000 | -1 | +| log_temp_files | 0 | -1 | +| maintenance_work_mem | 2048MB | 16 MB | +| max_replication_slots | 5 | 0 | +| max_wal_senders | 32 | 0 | +| max_wal_size | 5GB | 1GB | +| shared_buffers | 112896MB | Based on how much memory is available | +| shared_preload_libraries | pg_stat_statements | empty | +| shmall | 30146560 | Based on the server's capabilities | +| shmmax | 123480309760 | Based on the server's capabilities | +| wal_buffers | 16MB | -1 | +| wal_keep_segments | 512 | 10 | +| wal_level | replica | minimal | +| statement_timeout | 15s | 60s | +| idle_in_transaction_session_timeout | 60s | 60s | + +Some of these settings are in the process being adjusted. For example, the value +for `shared_buffers` is quite high and as such we are looking into adjusting it. +More information on this particular change can be found at +<https://gitlab.com/gitlab-com/infrastructure/issues/1555>. An up to date list +of proposed changes can be found at +<https://gitlab.com/gitlab-com/infrastructure/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=database&label_name[]=change>. + +## Unicorn + +GitLab.com adjusts the memory limits for the [unicorn-worker-killer][unicorn-worker-killer] gem. + +Base default: +* `memory_limit_min` = 750MiB +* `memory_limit_max` = 1024MiB + +Web front-ends: +* `memory_limit_min` = 1024MiB +* `memory_limit_max` = 1280MiB + +## GitLab.com at scale + +In addition to the GitLab Enterprise Edition Omnibus install, GitLab.com uses +the following applications and settings to achieve scale. All settings are +located publicly available [chef cookbooks](https://gitlab.com/gitlab-cookbooks). + +### ELK + +We use Elasticsearch, logstash, and Kibana for part of our monitoring solution: + +- [gitlab-cookbooks / gitlab-elk · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-elk) +- [gitlab-cookbooks / gitlab_elasticsearch · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab_elasticsearch) + +### Prometheus + +Prometheus complete our monitoring stack: + +- [gitlab-cookbooks / gitlab-prometheus · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-prometheus) + +### Grafana + +For the visualization of monitoring data: + +- [gitlab-cookbooks / gitlab-grafana · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-grafana) + +### Sentry + +Open source error tracking: + +- [gitlab-cookbooks / gitlab-sentry · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-sentry) + +### Consul + +Service discovery: + +- [gitlab-cookbooks / gitlab_consul · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab_consul) + +### Haproxy + +High Performance TCP/HTTP Load Balancer: + +- [gitlab-cookbooks / gitlab-haproxy · GitLab](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy) + +[autoscale mode]: https://docs.gitlab.com/runner/configuration/autoscale.html "How Autoscale works" +[runners-post]: https://about.gitlab.com/2016/04/05/shared-runners/ "Shared Runners on GitLab.com" +[GitLab Runner]: https://gitlab.com/gitlab-org/gitlab-runner +[altssh]: https://about.gitlab.com/2016/02/18/gitlab-dot-com-now-supports-an-alternate-git-plus-ssh-port/ "GitLab.com now supports an alternate git+ssh port" +[GitLab Pages]: https://about.gitlab.com/features/pages "GitLab Pages" +[docker in docker]: https://hub.docker.com/_/docker/ "Docker in Docker at DockerHub" +[mailgun]: https://www.mailgun.com/ "Mailgun website" +[sidekiq]: http://sidekiq.org/ "Sidekiq website" +[unicorn-worker-killer]: https://rubygems.org/gems/unicorn-worker-killer "unicorn-worker-killer" diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 7f77a33aadc..88efddbfba8 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -168,6 +168,20 @@ Alternatively, you can [lock the sharing with group feature](#share-with-group-l In GitLab Enterprise Edition it is possible to manage GitLab group memberships using LDAP groups. See [the GitLab Enterprise Edition documentation](../../integration/ldap.md) for more information. +## Transfer groups to another group + +From 10.5 there are two different ways to transfer a group: + +- Either by transferring a group into another group (making it a subgroup of that group). +- Or by converting a subgroup into a root group (a group with no parent). + +Please make sure to understand that: + +- Changing a group's parent can have unintended side effects. See [Redirects when changing repository paths](https://docs.gitlab.com/ce/user/project/index.html#redirects-when-changing-repository-paths) +- You can only transfer the group to a group you manage. +- You will need to update your local repositories to point to the new location. +- If the parent group's visibility is lower than the group current visibility, visibility levels for subgroups and projects will be changed to match the new parent group's visibility. + ## Group settings Once you have created a group, you can manage its settings by navigating to @@ -231,20 +245,22 @@ To enable this feature, navigate to the group settings page. Select  -#### Member Lock (EES/EEP) +#### Member Lock + +> Available in [GitLab Starter](https://about.gitlab.com/products/) and +[GitLab.com Bronze](https://about.gitlab.com/gitlab-com/). -Available in [GitLab Enterprise Edition Starter](https://about.gitlab.com/gitlab-ee/), -with **Member Lock** it is possible to lock membership in project to the +With **Member Lock** it is possible to lock membership in project to the level of members in group. -Learn more about [Member Lock](https://docs.gitlab.com/ee/user/group/index.html#member-lock-ees-eep). +Learn more about [Member Lock](https://docs.gitlab.com/ee/user/group/index.html#member-lock). ### Advanced settings - **Projects**: view all projects within that group, add members to each project, access each project's settings, and remove any project from the same screen. - **Webhooks**: configure [webhooks](../project/integrations/webhooks.md) -and [push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html#push-rules) to your group (Push Rules is available in [GitLab Enteprise Edition Starter](https://about.gitlab.com/products/).) +and [push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html#push-rules) to your group (Push Rules is available in [GitLab Starter](https://about.gitlab.com/products/).) - **Audit Events**: view [Audit Events](https://docs.gitlab.com/ee/administration/audit_events.html#audit-events) -for the group (GitLab admins only, available in [GitLab Enterprise Edition Starter][ee]). +for the group (GitLab admins only, available in [GitLab Starter][ee]). - **Pipelines quota**: keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group diff --git a/doc/user/img/feature_highlight_example.png b/doc/user/img/feature_highlight_example.png Binary files differnew file mode 100644 index 00000000000..32ca05a6087 --- /dev/null +++ b/doc/user/img/feature_highlight_example.png diff --git a/doc/user/index.md b/doc/user/index.md index 01db8becc43..43b6fd53b91 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -107,6 +107,8 @@ personal access tokens, authorized applications, etc. methods available in GitLab. - [Permissions](permissions.md): Learn the different set of permissions levels for each user type (guest, reporter, developer, master, owner). +- [Feature highlight](feature_highlight.md): Learn more about the little blue dots +around the app that explain certain features ## Groups diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 552abac747b..650d60f1585 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -36,12 +36,16 @@ GFM honors the markdown specification in how [paragraphs and line breaks are han A paragraph is simply one or more consecutive lines of text, separated by one or more blank lines. Line-breaks, or softreturns, are rendered if you end a line with two or more spaces: - Roses are red [followed by two or more spaces] +[//]: # (Do *NOT* remove the two ending whitespaces in the following line.) +[//]: # (They are needed for the Markdown text to render correctly.) + Roses are red [followed by two or more spaces] Violets are blue Sugar is sweet -Roses are red +[//]: # (Do *NOT* remove the two ending whitespaces in the following line.) +[//]: # (They are needed for the Markdown text to render correctly.) +Roses are red Violets are blue Sugar is sweet @@ -226,7 +230,7 @@ https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#emoji If you are new to this, don't be :fearful:. You can easily join the emoji :family:. All you need to do is to look up on the supported codes. - Consult the [Emoji Cheat Sheet](http://emoji.codes) for a list of all supported emoji codes. :thumbsup: + Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: @@ -236,7 +240,7 @@ You can use it to point out a :bug: or warn about :speak_no_evil: patches. And i If you are new to this, don't be :fearful:. You can easily join the emoji :family:. All you need to do is to look up on the supported codes. -Consult the [Emoji Cheat Sheet](http://emoji.codes) for a list of all supported emoji codes. :thumbsup: +Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: ### Special GitLab References @@ -253,7 +257,7 @@ GFM will recognize the following: | `@user_name` | specific user | | `@group_name` | specific group | | `@all` | entire team | -| `#123` | issue | +| `#12345` | issue | | `!123` | merge request | | `$123` | snippet | | `~123` | label by ID | @@ -379,6 +383,45 @@ _Be advised that KaTeX only supports a [subset][katex-subset] of LaTeX._ >**Note:** This also works for the asciidoctor `:stem: latexmath`. For details see the [asciidoctor user manual][asciidoctor-manual]. +### Colors + +> If this is not rendered correctly, see +https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#colors + +It is possible to have color written in HEX, RGB or HSL format rendered with a color indicator. + +Color written inside backticks will be followed by a color "chip". + +Examples: + + `#F00` + `#F00A` + `#FF0000` + `#FF0000AA` + `RGB(0,255,0)` + `RGB(0%,100%,0%)` + `RGBA(0,255,0,0.7)` + `HSL(540,70%,50%)` + `HSLA(540,70%,50%,0.7)` + +Becomes: + +`#F00` +`#F00A` +`#FF0000` +`#FF0000AA` +`RGB(0,255,0)` +`RGB(0%,100%,0%)` +`RGBA(0,255,0,0.7)` +`HSL(540,70%,50%)` +`HSLA(540,70%,50%,0.7)` + +#### Supported formats: + +* HEX: `` `#RGB[A]` `` or `` `#RRGGBB[AA]` `` +* RGB: `` `RGB[A](R, G, B[, A])` `` +* HSL: `` `HSL[A](H, S, L[, A])` `` + ### Mermaid > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15107) in @@ -711,7 +754,7 @@ This line is separated from the one above by two newlines, so it will be a *sepa This line is also a separate paragraph, but... This line is only separated by a single newline, so it *does not break* and just follows the previous line in the *same paragraph*. -This line is also a separate paragraph, and... +This line is also a separate paragraph, and... This line is *on its own line*, because the previous line ends with two spaces. (but still in the *same paragraph*) spaces. @@ -724,7 +767,7 @@ This line is separated from the one above by two newlines, so it will be a *sepa This line is also a separate paragraph, but... This line is only separated by a single newline, so it *does not break* and just follows the previous line in the *same paragraph*. -This line is also a separate paragraph, and... +This line is also a separate paragraph, and... This line is *on its own line*, because the previous line ends with two spaces. (but still in the *same paragraph*) spaces. diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 708d07fcec9..a520279c29e 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -25,7 +25,8 @@ The following table depicts the various user permission levels in a project. | Create confidential issue | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | View confidential issues | (✓) [^2] | ✓ | ✓ | ✓ | ✓ | | Leave comments | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| Lock discussions (issues and merge requests) | | | | ✓ | ✓ | +| Lock issue discussions | | ✓ | ✓ | ✓ | ✓ | +| Lock merge request discussions | | | ✓ | ✓ | ✓ | | See a list of jobs | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | See a job log | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | Download and browse job artifacts | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | @@ -117,14 +118,16 @@ and drag issues around. Read though the [documentation on Issue Boards permissions](project/issue_board.md#permissions) to learn more. -### File Locking permissions (EEP) +### File Locking permissions + +> Available in [GitLab Premium](https://about.gitlab.com/products/). The user that locks a file or directory is the only one that can edit and push their changes back to the repository where the locked objects are located. Read through the documentation on [permissions for File Locking](https://docs.gitlab.com/ee/user/project/file_lock.html#permissions-on-file-locking) to learn more. File Locking is available in -[GitLab Enterprise Edition Premium](https://about.gitlab.com/gitlab-ee/) only. +[GitLab Premium](https://about.gitlab.com/products/) only. ### Confidential Issues permissions @@ -251,12 +254,14 @@ for details about the pipelines security model. Since GitLab 8.15, LDAP user permissions can now be manually overridden by an admin user. Read through the documentation on [LDAP users permissions](https://docs.gitlab.com/ee/articles/how_to_configure_ldap_gitlab_ee/index.html#updating-user-permissions-new-feature) to learn more. -## Auditor users permissions (EEP) +## Auditor users permissions + +> Available in [GitLab Premium](https://about.gitlab.com/products/). An Auditor user should be able to access all projects and groups of a GitLab instance with the permissions described on the documentation on [auditor users permissions](https://docs.gitlab.com/ee/administration/auditor_users.html#permissions-and-restrictions-of-an-auditor-user). -Auditor users are available in [GitLab Enterprise Edition Premium](https://about.gitlab.com/gitlab-ee/) +Auditor users are available in [GitLab Premium](https://about.gitlab.com/products/) only. [^1]: On public and internal projects, all users are able to perform this action diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index e7596f5c577..910bd20f882 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -1,7 +1,7 @@ # Deleting a User Account - As a user, you can delete your own account by navigating to **Settings** > **Account** and selecting **Delete account** -- As an admin, you can delete a user account by navigating to the **Admin Area**, selecting the **Users** tab, selecting a user, and clicking on **Remove user** +- As an admin, you can delete a user account by navigating to the **Admin Area**, selecting the **Users** tab, selecting a user, and clicking on **Delete user** ## Associated Records diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 022d6317555..930e506802a 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -41,7 +41,7 @@ select few, the amount of activity on the default Dashboard page can be overwhelming. Changing this setting allows you to redefine what your default dashboard will be. -You have 6 options here that you can use for your default dashboard view: +You have 8 options here that you can use for your default dashboard view: - Your projects (default) - Starred projects @@ -49,6 +49,8 @@ You have 6 options here that you can use for your default dashboard view: - Starred projects' activity - Your groups - Your [Todos] +- Assigned Issues +- Assigned Merge Requests ### Project home page content diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index e87b4403854..716787532fc 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -5,20 +5,23 @@ Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster in a few steps. -With a cluster associated to your project, you can use Review Apps, deploy your -applications, run your pipelines, and much more, in an easy way. +## Overview + +With a Kubernetes cluster associated to your project, you can use +[Review Apps](../../../ci/review_apps/index.md), deploy your applications, run +your pipelines, and much more, in an easy way. There are two options when adding a new cluster to your project; either associate your account with Google Kubernetes Engine (GKE) so that you can [create new clusters](#adding-and-creating-a-new-gke-cluster-via-gitlab) from within GitLab, or provide the credentials to an [existing Kubernetes cluster](#adding-an-existing-kubernetes-cluster). -## Prerequisites +## Adding and creating a new GKE cluster via GitLab -In order to be able to manage your Kubernetes cluster through GitLab, the -following prerequisites must be met. +NOTE: **Note:** +You need Master [permissions] and above to access the Kubernetes page. -**For a cluster hosted on GKE:** +Before proceeding, make sure the following requirements are met: - The [Google authentication integration](../../../integration/google.md) must be enabled in GitLab at the instance level. If that's not the case, ask your @@ -28,30 +31,16 @@ following prerequisites must be met. account](https://cloud.google.com/billing/docs/how-to/manage-billing-account) must be set up and that you have to have permissions to access it. - You must have Master [permissions] in order to be able to access the - **Cluster** page. + **Kubernetes** page. - You must have [Cloud Billing API](https://cloud.google.com/billing/) enabled - You must have [Resource Manager API](https://cloud.google.com/resource-manager/) -**For an existing Kubernetes cluster:** - -- Since the cluster is already created, there are no prerequisites. - ---- - -If all of the above requirements are met, you can proceed to add a new Kubernetes -cluster. - -## Adding and creating a new GKE cluster via GitLab - -NOTE: **Note:** -You need Master [permissions] and above to access the Clusters page. - -Before proceeding, make sure all [prerequisites](#prerequisites) are met. -To add a new cluster hosted on GKE to your project: +If all of the above requirements are met, you can proceed to create and add a +new Kubernetes cluster that will be hosted on GKE to your project: -1. Navigate to your project's **CI/CD > Clusters** page. -1. Click on **Add cluster**. +1. Navigate to your project's **CI/CD > Kubernetes** page. +1. Click on **Add Kubernetes cluster**. 1. Click on **Create with GKE**. 1. Connect your Google account if you haven't done already by clicking the **Sign in with Google** button. @@ -66,7 +55,7 @@ To add a new cluster hosted on GKE to your project: - **Machine type** - The [machine type](https://cloud.google.com/compute/docs/machine-types) of the Virtual Machine instance that the cluster will be based on. - **Environment scope** - The [associated environment](#setting-the-environment-scope) to this cluster. -1. Finally, click the **Create cluster** button. +1. Finally, click the **Create Kubernetes cluster** button. After a few moments, your cluster should be created. If something goes wrong, you will be notified. @@ -77,14 +66,14 @@ enable the Cluster integration. ## Adding an existing Kubernetes cluster NOTE: **Note:** -You need Master [permissions] and above to access the Clusters page. +You need Master [permissions] and above to access the Kubernetes page. To add an existing Kubernetes cluster to your project: -1. Navigate to your project's **CI/CD > Clusters** page. -1. Click on **Add cluster**. -1. Click on **Add an existing cluster** and fill in the details: - - **Cluster name** (required) - The name you wish to give the cluster. +1. Navigate to your project's **CI/CD > Kubernetes** page. +1. Click on **Add Kubernetes cluster**. +1. Click on **Add an existing Kubernetes cluster** and fill in the details: + - **Kubernetes cluster name** (required) - The name you wish to give the cluster. - **Environment scope** (required)- The [associated environment](#setting-the-environment-scope) to this cluster. - **API URL** (required) - @@ -112,15 +101,48 @@ To add an existing Kubernetes cluster to your project: - If you or someone created a secret specifically for the project, usually with limited permissions, the secret's namespace and project namespace may be the same. -1. Finally, click the **Create cluster** button. - -The Kubernetes service takes the following parameters: +1. Finally, click the **Create Kubernetes cluster** button. After a few moments, your cluster should be created. If something goes wrong, you will be notified. You can now proceed to install some pre-defined applications and then -enable the Cluster integration. +enable the Kubernetes cluster integration. + +## Security implications + +CAUTION: **Important:** +The whole cluster security is based on a model where [developers](../../permissions.md) +are trusted, so **only trusted users should be allowed to control your clusters**. + +The default cluster configuration grants access to a wide set of +functionalities needed to successfully build and deploy a containerized +application. Bare in mind that the same credentials are used for all the +applications running on the cluster. + +When GitLab creates the cluster, it enables and uses the legacy +[Attribute-based access control (ABAC)](https://kubernetes.io/docs/admin/authorization/abac/). +The newer [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/) +authorization will be supported in a +[future release](https://gitlab.com/gitlab-org/gitlab-ce/issues/29398). + +### Security of GitLab Runners + +GitLab Runners have the [privileged mode](https://docs.gitlab.com/runner/executors/docker.html#the-privileged-mode) +enabled by default, which allows them to execute special commands and running +Docker in Docker. This functionality is needed to run some of the [Auto DevOps] +jobs. This implies the containers are running in privileged mode and you should, +therefore, be aware of some important details. + +The privileged flag gives all capabilities to the running container, which in +turn can do almost everything that the host can do. Be aware of the +inherent security risk associated with performing `docker run` operations on +arbitrary images as they effectively have root access. + +If you don't want to use GitLab Runner in privileged mode, first make sure that +you don't have it installed via the applications, and then use the +[Runner's Helm chart](../../../install/kubernetes/gitlab_runner_chart.md) to +install it manually. ## Installing applications @@ -131,14 +153,85 @@ added directly to your configured cluster. Those applications are needed for | Application | GitLab version | Description | | ----------- | :------------: | ----------- | | [Helm Tiller](https://docs.helm.sh/) | 10.2+ | Helm is a package manager for Kubernetes and is required to install all the other applications. It will be automatically installed as a dependency when you try to install a different app. It is installed in its own pod inside the cluster which can run the `helm` CLI in a safe environment. | -| [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) | 10.2+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps](../../../topics/autodevops/index.md) or deploy your own web apps. | +| [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) | 10.2+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps] or deploy your own web apps. | | [Prometheus](https://prometheus.io/docs/introduction/overview/) | 10.4+ | Prometheus is an open-source monitoring and alerting system useful to supervise your deployed applications | +| [GitLab Runner](https://docs.gitlab.com/runner/) | 10.6+ | GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/), the open-source continuous integration service included with GitLab that coordinates the jobs. When installing the GitLab Runner via the applications, it will run in **privileged mode** by default. Make sure you read the [security implications](#security-implications) before doing so. | + +## Getting the external IP address + +NOTE: **Note:** +You need a load balancer installed in your cluster in order to obtain the +external IP address with the following procedure. It can be deployed using the +[**Ingress** application](#installing-applications). + +In order to publish your web application, you first need to find the external IP +address associated to your load balancer. + +### Let GitLab fetch the IP address + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17052) in GitLab 10.6. + +If you installed the Ingress [via the **Applications**](#installing-applications), +you should see the Ingress IP address on this same page within a few minutes. +If you don't see this, GitLab might not be able to determine the IP address of +your ingress application in which case you should manually determine it. + +### Manually determining the IP address + +If the cluster is on GKE, click on the **Google Kubernetes Engine** link in the +**Advanced settings**, or go directly to the +[Google Kubernetes Engine dashboard](https://console.cloud.google.com/kubernetes/) +and select the proper project and cluster. Then click on **Connect** and execute +the `gcloud` command in a local terminal or using the **Cloud Shell**. + +If the cluster is not on GKE, follow the specific instructions for your +Kubernetes provider to configure `kubectl` with the right credentials. + +If you installed the Ingress [via the **Applications**](#installing-applications), +run the following command: + +```bash +kubectl get svc --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' +``` + +Otherwise, you can list the IP addresses of all load balancers: + +```bash +kubectl get svc --all-namespaces -o jsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} ' +``` + +The output is the external IP address of your cluster. This information can then +be used to set up DNS entries and forwarding rules that allow external access to +your deployed applications. + +### Using a static IP + +By default, an ephemeral external IP address is associated to the cluster's load +balancer. If you associate the ephemeral IP with your DNS and the IP changes, +your apps will not be able to be reached, and you'd have to change the DNS +record again. In order to avoid that, you should change it into a static +reserved IP. + +[Read how to promote an ephemeral external IP address in GKE.](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip) + +### Pointing your DNS at the cluster IP + +Once you've set up the static IP, you should associate it to a [wildcard DNS +record](https://en.wikipedia.org/wiki/Wildcard_DNS_record), in order to be able +to reach your apps. This heavily depends on your domain provider, but in case +you aren't sure, just create an A record with a wildcard host like +`*.example.com.`. ## Setting the environment scope -When adding more than one clusters, you need to differentiate them with an -environment scope. The environment scope associates clusters and -[environments](../../../ci/environments.md) in an 1:1 relationship similar to how the +NOTE: **Note:** +This is only available for [GitLab Premium][ee] where you can add more than +one Kubernetes cluster. + +When adding more than one Kubernetes clusters to your project, you need to +differentiate them with an environment scope. The environment scope associates +clusters and [environments](../../../ci/environments.md) in an 1:1 relationship +similar to how the [environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-secret-variables) work. @@ -148,7 +241,7 @@ cluster in a project, and a validation error will occur if otherwise. --- -For example, let's say the following clusters exist in a project: +For example, let's say the following Kubernetes clusters exist in a project: | Cluster | Environment scope | | ---------- | ------------------- | @@ -190,14 +283,13 @@ The result will then be: ## Multiple Kubernetes clusters -> Introduced in [GitLab Enterprise Edition Premium][ee] 10.3. +> Introduced in [GitLab Premium][ee] 10.3. -With GitLab EEP, you can associate more than one Kubernetes clusters to your +With GitLab Premium, you can associate more than one Kubernetes clusters to your project. That way you can have different clusters for different environments, like dev, staging, production, etc. -To add another cluster, follow the same steps as described in [adding a -Kubernetes cluster](#adding-a-kubernetes-cluster) and make sure to +Simply add another cluster, like you did the first time, and make sure to [set an environment scope](#setting-the-environment-scope) that will differentiate the new cluster with the rest. @@ -205,53 +297,50 @@ differentiate the new cluster with the rest. The Kubernetes cluster integration exposes the following [deployment variables](../../../ci/variables/README.md#deployment-variables) in the -GitLab CI/CD build environment: - -- `KUBE_URL` - Equal to the API URL. -- `KUBE_TOKEN` - The Kubernetes token. -- `KUBE_NAMESPACE` - The Kubernetes namespace is auto-generated if not specified. - The default value is `<project_name>-<project_id>`. You can overwrite it to - use different one if needed, otherwise the `KUBE_NAMESPACE` variable will - receive the default value. -- `KUBE_CA_PEM_FILE` - Only present if a custom CA bundle was specified. Path - to a file containing PEM data. -- `KUBE_CA_PEM` (deprecated) - Only if a custom CA bundle was specified. Raw PEM data. -- `KUBECONFIG` - Path to a file containing `kubeconfig` for this deployment. - CA bundle would be embedded if specified. - -## Enabling or disabling the Cluster integration +GitLab CI/CD build environment. + +| Variable | Description | +| -------- | ----------- | +| `KUBE_URL` | Equal to the API URL. | +| `KUBE_TOKEN` | The Kubernetes token. | +| `KUBE_NAMESPACE` | The Kubernetes namespace is auto-generated if not specified. The default value is `<project_name>-<project_id>`. You can overwrite it to use different one if needed, otherwise the `KUBE_NAMESPACE` variable will receive the default value. | +| `KUBE_CA_PEM_FILE` | Only present if a custom CA bundle was specified. Path to a file containing PEM data. | +| `KUBE_CA_PEM` | (**deprecated**) Only if a custom CA bundle was specified. Raw PEM data. | +| `KUBECONFIG` | Path to a file containing `kubeconfig` for this deployment. CA bundle would be embedded if specified. | + +## Enabling or disabling the Kubernetes cluster integration After you have successfully added your cluster information, you can enable the -Cluster integration: +Kubernetes cluster integration: 1. Click the "Enabled/Disabled" switch 1. Hit **Save** for the changes to take effect You can now start using your Kubernetes cluster for your deployments. -To disable the Cluster integration, follow the same procedure. +To disable the Kubernetes cluster integration, follow the same procedure. -## Removing the Cluster integration +## Removing the Kubernetes cluster integration NOTE: **Note:** -You need Master [permissions] and above to remove a cluster integration. +You need Master [permissions] and above to remove a Kubernetes cluster integration. NOTE: **Note:** When you remove a cluster, you only remove its relation to GitLab, not the cluster itself. To remove the cluster, you can do so by visiting the GKE dashboard or using `kubectl`. -To remove the Cluster integration from your project, simply click on the +To remove the Kubernetes cluster integration from your project, simply click on the **Remove integration** button. You will then be able to follow the procedure -and [add a cluster](#adding-a-cluster) again. +and add a Kubernetes cluster again. ## What you can get with the Kubernetes integration Here's what you can do with GitLab if you enable the Kubernetes integration. -### Deploy Boards (EEP) +### Deploy Boards -> Available in [GitLab Enterprise Edition Premium][ee]. +> Available in [GitLab Premium][ee]. GitLab's Deploy Boards offer a consolidated view of the current health and status of each CI [environment](../../../ci/environments.md) running on Kubernetes, @@ -261,9 +350,9 @@ workflow they already use without any need to access Kubernetes. [> Read more about Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) -### Canary Deployments (EEP) +### Canary Deployments -> Available in [GitLab Enterprise Edition Premium][ee]. +> Available in [GitLab Premium][ee]. Leverage [Kubernetes' Canary deployments](https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments) and visualize your canary deployments right inside the Deploy Board, without @@ -303,4 +392,5 @@ the deployment variables above, ensuring any pods you create are labelled with `app=$CI_ENVIRONMENT_SLUG`. GitLab will do the rest! [permissions]: ../../permissions.md -[ee]: https://about.gitlab.com/gitlab-ee/ +[ee]: https://about.gitlab.com/products/ +[Auto DevOps]: ../../../topics/autodevops/index.md diff --git a/doc/user/project/img/label_priority_sort_order.png b/doc/user/project/img/label_priority_sort_order.png Binary files differdeleted file mode 100644 index 21c7a76a322..00000000000 --- a/doc/user/project/img/label_priority_sort_order.png +++ /dev/null diff --git a/doc/user/project/img/labels_assign_label_sidebar.png b/doc/user/project/img/labels_assign_label_sidebar.png Binary files differdeleted file mode 100644 index d74796fdb4d..00000000000 --- a/doc/user/project/img/labels_assign_label_sidebar.png +++ /dev/null diff --git a/doc/user/project/img/labels_assign_label_sidebar_saved.png b/doc/user/project/img/labels_assign_label_sidebar_saved.png Binary files differdeleted file mode 100644 index dabffe956dc..00000000000 --- a/doc/user/project/img/labels_assign_label_sidebar_saved.png +++ /dev/null diff --git a/doc/user/project/img/labels_default.png b/doc/user/project/img/labels_default.png Binary files differindex 7934e3bfb5e..7a7fab611a4 100644 --- a/doc/user/project/img/labels_default.png +++ b/doc/user/project/img/labels_default.png diff --git a/doc/user/project/img/labels_description_tooltip.png b/doc/user/project/img/labels_description_tooltip.png Binary files differdeleted file mode 100644 index eea4f8cf0f4..00000000000 --- a/doc/user/project/img/labels_description_tooltip.png +++ /dev/null diff --git a/doc/user/project/img/labels_filter.png b/doc/user/project/img/labels_filter.png Binary files differdeleted file mode 100644 index 6a1ebfc2ecb..00000000000 --- a/doc/user/project/img/labels_filter.png +++ /dev/null diff --git a/doc/user/project/img/labels_generate.png b/doc/user/project/img/labels_generate.png Binary files differdeleted file mode 100644 index 987f4b5be71..00000000000 --- a/doc/user/project/img/labels_generate.png +++ /dev/null diff --git a/doc/user/project/img/labels_generate_default.png b/doc/user/project/img/labels_generate_default.png Binary files differnew file mode 100644 index 00000000000..fca2a06e04f --- /dev/null +++ b/doc/user/project/img/labels_generate_default.png diff --git a/doc/user/project/img/labels_group_issues.png b/doc/user/project/img/labels_group_issues.png Binary files differnew file mode 100644 index 00000000000..29dcf7ff45e --- /dev/null +++ b/doc/user/project/img/labels_group_issues.png diff --git a/doc/user/project/img/labels_list.png b/doc/user/project/img/labels_list.png Binary files differnew file mode 100644 index 00000000000..12c47ea9766 --- /dev/null +++ b/doc/user/project/img/labels_list.png diff --git a/doc/user/project/img/labels_new_label.png b/doc/user/project/img/labels_new_label.png Binary files differdeleted file mode 100644 index e26425d0188..00000000000 --- a/doc/user/project/img/labels_new_label.png +++ /dev/null diff --git a/doc/user/project/img/labels_new_label_on_the_fly.png b/doc/user/project/img/labels_new_label_on_the_fly.png Binary files differdeleted file mode 100644 index 2ac9805b1ab..00000000000 --- a/doc/user/project/img/labels_new_label_on_the_fly.png +++ /dev/null diff --git a/doc/user/project/img/labels_new_label_on_the_fly_create.png b/doc/user/project/img/labels_new_label_on_the_fly_create.png Binary files differdeleted file mode 100644 index 02ccf68553b..00000000000 --- a/doc/user/project/img/labels_new_label_on_the_fly_create.png +++ /dev/null diff --git a/doc/user/project/img/labels_prioritize.png b/doc/user/project/img/labels_prioritize.png Binary files differdeleted file mode 100644 index d602a3c90ec..00000000000 --- a/doc/user/project/img/labels_prioritize.png +++ /dev/null diff --git a/doc/user/project/img/labels_prioritized.png b/doc/user/project/img/labels_prioritized.png Binary files differnew file mode 100644 index 00000000000..57dcfe89b3d --- /dev/null +++ b/doc/user/project/img/labels_prioritized.png diff --git a/doc/user/project/img/labels_promotion.png b/doc/user/project/img/labels_promotion.png Binary files differnew file mode 100644 index 00000000000..8a5efd210a2 --- /dev/null +++ b/doc/user/project/img/labels_promotion.png diff --git a/doc/user/project/img/labels_sidebar.png b/doc/user/project/img/labels_sidebar.png Binary files differnew file mode 100644 index 00000000000..7349c6d4f0c --- /dev/null +++ b/doc/user/project/img/labels_sidebar.png diff --git a/doc/user/project/img/labels_sidebar_assign.png b/doc/user/project/img/labels_sidebar_assign.png Binary files differnew file mode 100644 index 00000000000..61e8d04fc85 --- /dev/null +++ b/doc/user/project/img/labels_sidebar_assign.png diff --git a/doc/user/project/img/labels_sidebar_inline.png b/doc/user/project/img/labels_sidebar_inline.png Binary files differnew file mode 100644 index 00000000000..31fa397761d --- /dev/null +++ b/doc/user/project/img/labels_sidebar_inline.png diff --git a/doc/user/project/img/labels_sort_label_priority.png b/doc/user/project/img/labels_sort_label_priority.png Binary files differnew file mode 100644 index 00000000000..c8b97639121 --- /dev/null +++ b/doc/user/project/img/labels_sort_label_priority.png diff --git a/doc/user/project/img/labels_sort_priority.png b/doc/user/project/img/labels_sort_priority.png Binary files differnew file mode 100644 index 00000000000..a95198e7f72 --- /dev/null +++ b/doc/user/project/img/labels_sort_priority.png diff --git a/doc/user/project/img/labels_subscribe.png b/doc/user/project/img/labels_subscribe.png Binary files differdeleted file mode 100644 index 56f24ae7bc8..00000000000 --- a/doc/user/project/img/labels_subscribe.png +++ /dev/null diff --git a/doc/user/project/img/labels_subscriptions.png b/doc/user/project/img/labels_subscriptions.png Binary files differnew file mode 100644 index 00000000000..8bcb3b57f6c --- /dev/null +++ b/doc/user/project/img/labels_subscriptions.png diff --git a/doc/user/project/img/new_label_from_sidebar.gif b/doc/user/project/img/new_label_from_sidebar.gif Binary files differnew file mode 100644 index 00000000000..572b29a86e1 --- /dev/null +++ b/doc/user/project/img/new_label_from_sidebar.gif diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 72def9d1d1d..8c639bd5343 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -46,7 +46,7 @@ namespace that started the import process. The importer will also import branches on forks of projects related to open pull requests. These branches will be imported with a naming scheme similar to -GH-SHA-Username/Pull-Request-number/fork-name/branch. This may lead to a discrepency +GH-SHA-Username/Pull-Request-number/fork-name/branch. This may lead to a discrepancy in branches compared to the GitHub Repository. For a more technical description and an overview of the architecture you can diff --git a/doc/user/project/import/img/import_projects_from_repo_url.png b/doc/user/project/import/img/import_projects_from_repo_url.png Binary files differnew file mode 100644 index 00000000000..ec867da1087 --- /dev/null +++ b/doc/user/project/import/img/import_projects_from_repo_url.png diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index e2b285678c3..72cc58546b7 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -10,6 +10,7 @@ 1. [From Perforce](perforce.md) 1. [From SVN](svn.md) 1. [From TFS](tfs.md) +1. [From repo by URL](repo_by_url.md) In addition to the specific migration documentation above, you can import any Git repository via HTTP from the New Project page. Be aware that if the diff --git a/doc/user/project/import/perforce.md b/doc/user/project/import/perforce.md index aa7508e1e8e..a1ea716b606 100644 --- a/doc/user/project/import/perforce.md +++ b/doc/user/project/import/perforce.md @@ -48,3 +48,9 @@ Here's a few links to get you started: - [git-p4 manual page](https://www.kernel.org/pub/software/scm/git/docs/git-p4.html) - [git-p4 example usage](https://git.wiki.kernel.org/index.php/Git-p4_Usage) - [Git book migration guide](https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git#_perforce_import) + +Note that `git p4` and `git filter-branch` are not very good at +creating small and efficient Git pack files. So it might be a good +idea to spend time and CPU to properly repack your repository before +sending it for the first time to your GitLab server. See +[this StackOverflow question](https://stackoverflow.com/questions/28720151/git-gc-aggressive-vs-git-repack/). diff --git a/doc/user/project/import/repo_by_url.md b/doc/user/project/import/repo_by_url.md new file mode 100644 index 00000000000..f43e384de88 --- /dev/null +++ b/doc/user/project/import/repo_by_url.md @@ -0,0 +1,12 @@ +# Import project from repo by URL + +You can import your existing repositories by providing the Git URL: + +1. From your GitLab dashboard click **New project** +1. Switch to the **Import project** tab +1. Click on the **Repo by URL** button +1. Fill in the "Git repository URL" and the remaining project fields +1. Click **Create project** to being the import process +1. Once complete, you will be redirected to your newly created project + + diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 4c772c62f8d..f94e93dd7d8 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -17,7 +17,7 @@ When you create a project in GitLab, you'll have access to a large number of - [Issue tracker](issues/index.md): Discuss implementations with your team within issues - [Issue Boards](issue_board.md): Organize and prioritize your workflow - - [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards) (**EES/EEP**): Allow your teams to create their own workflows (Issue Boards) for the same project + - [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards) (**Starter/Premium**): Allow your teams to create their own workflows (Issue Boards) for the same project - [Repositories](repository/index.md): Host your code in a fully integrated platform - [Branches](repository/branches/index.md): use Git branching strategies to @@ -29,7 +29,7 @@ integrated platform - [Signing commits](gpg_signed_commits/index.md): use GPG to sign your commits - [Merge Requests](merge_requests/index.md): Apply your branching strategy and get reviewed by your team - - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) (**EES/EEP**): Ask for approval before + - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) (**Starter/Premium**): Ask for approval before implementing a change - [Fix merge conflicts from the UI](merge_requests/resolve_conflicts.md): Your Git diff tool right from GitLab's UI @@ -128,12 +128,9 @@ and Git push/pull redirects. Depending on the situation, different things apply. -When [transferring a project](settings/index.md#transferring-an-existing-project-into-another-namespace), -or [renaming a user](../profile/index.md#changing-your-username) or -[changing a group path](../group/index.md#changing-a-group-s-path): +When [renaming a user](../profile/index.md#changing-your-username), +[changing a group path](../group/index.md#changing-a-group-s-path) or [renaming a repository](settings/index.md#renaming-a-repository): -- **The redirect to the new URL is permanent**, which means that the original - namespace can't be claimed again by any group or user. - Existing web URLs for the namespace and anything under it (e.g., projects) will redirect to the new URLs. - Starting with GitLab 10.3, existing Git remote URLs for projects under the @@ -142,9 +139,5 @@ or [renaming a user](../profile/index.md#changing-your-username) or your remote will be displayed instead of rejecting your action. This means that any automation scripts, or Git clients will continue to work after a rename, making any transition a lot smoother. - To avoid pulling from or pushing to an entirely incorrect repository, the old - path will be reserved. - -When [renaming-a-repository](settings/index.md#renaming-a-repository), the same -things apply, except for the Git push/pull actions which will be rejected with a -warning message to change to the new remote URL. +- The redirects will be available as long as the original path is not claimed by + another group, user or project. diff --git a/doc/user/project/integrations/bugzilla.md b/doc/user/project/integrations/bugzilla.md index ba2adc1afda..671804035cc 100644 --- a/doc/user/project/integrations/bugzilla.md +++ b/doc/user/project/integrations/bugzilla.md @@ -11,11 +11,7 @@ in the table below. | `issues_url` | The URL to the issue in Bugzilla project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. | | `new_issue_url` | This is the URL to create a new issue in Bugzilla for the project linked to this GitLab project. Note that the `new_issue_url` requires PRODUCT_NAME to be updated with the product/project name in Bugzilla. | -Once you have configured and enabled Bugzilla: - -- the **Issues** link on the GitLab project pages takes you to the appropriate - Bugzilla product page -- clicking **New issue** on the project dashboard takes you to Bugzilla for entering a new issue +Once you have configured and enabled Bugzilla you'll see the Bugzilla link on the GitLab project pages that takes you to the appropriate Bugzilla project. ## Referencing issues in Bugzilla diff --git a/doc/user/project/integrations/custom_issue_tracker.md b/doc/user/project/integrations/custom_issue_tracker.md index 757522c2ae3..731291ebe84 100644 --- a/doc/user/project/integrations/custom_issue_tracker.md +++ b/doc/user/project/integrations/custom_issue_tracker.md @@ -13,6 +13,8 @@ in the table below. | `issues_url` | The URL to the issue in the issue tracker project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. For example, `https://customissuetracker.com/project-name/:id`. | | `new_issue_url` | Currently unused. Will be changed in a future release. | +Once you have configured and enabled Custom Issue Tracker Service you'll see a link on the GitLab project pages that takes you to that custom issue tracker. + ## Referencing issues diff --git a/doc/user/project/integrations/img/prometheus_dashboard.png b/doc/user/project/integrations/img/prometheus_dashboard.png Binary files differnew file mode 100644 index 00000000000..bd19f1b44cc --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_dashboard.png diff --git a/doc/user/project/integrations/img/prometheus_deploy.png b/doc/user/project/integrations/img/prometheus_deploy.png Binary files differnew file mode 100644 index 00000000000..d39081bcc7b --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_deploy.png diff --git a/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png b/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png Binary files differdeleted file mode 100644 index e30cba211e6..00000000000 --- a/doc/user/project/integrations/img/prometheus_gcp_firewall_rule.png +++ /dev/null diff --git a/doc/user/project/integrations/img/prometheus_gcp_node_name.png b/doc/user/project/integrations/img/prometheus_gcp_node_name.png Binary files differdeleted file mode 100644 index ea289431454..00000000000 --- a/doc/user/project/integrations/img/prometheus_gcp_node_name.png +++ /dev/null diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index 2f6e415c5cf..5933bcedc8b 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -129,7 +129,7 @@ Note that the transition ID may vary between workflows (e.g., bug vs. story), even if the status you are changing to is the same. After saving the configuration, your GitLab project will be able to interact -with all JIRA projects in your JIRA instance. +with all JIRA projects in your JIRA instance and you'll see the JIRA link on the GitLab project pages that takes you to the appropriate JIRA project.  diff --git a/doc/user/project/integrations/kubernetes.md b/doc/user/project/integrations/kubernetes.md index 543baaa81e1..f502d1c9821 100644 --- a/doc/user/project/integrations/kubernetes.md +++ b/doc/user/project/integrations/kubernetes.md @@ -81,9 +81,9 @@ GitLab CI/CD build environment: Here's what you can do with GitLab if you enable the Kubernetes integration. -### Deploy Boards (EEP) +### Deploy Boards -> Available in [GitLab Enterprise Edition Premium][ee]. +> Available in [GitLab Premium][ee]. GitLab's Deploy Boards offer a consolidated view of the current health and status of each CI [environment](../../../ci/environments.md) running on Kubernetes, @@ -93,9 +93,9 @@ workflow they already use without any need to access Kubernetes. [> Read more about Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) -### Canary Deployments (EEP) +### Canary Deployments -> Available in [GitLab Enterprise Edition Premium][ee]. +> Available in [GitLab Premium][ee]. Leverage [Kubernetes' Canary deployments](https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments) and visualize your canary deployments right inside the Deploy Board, without @@ -134,4 +134,4 @@ containers. To use this integration, you should deploy to Kubernetes using the deployment variables above, ensuring any pods you create are labelled with `app=$CI_ENVIRONMENT_SLUG`. GitLab will do the rest! -[ee]: https://about.gitlab.com/gitlab-ee/ +[ee]: https://about.gitlab.com/products/ diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 5fefb3b69c4..fa7e504c4aa 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -2,119 +2,73 @@ > [Introduced][ce-8935] in GitLab 9.0. -GitLab offers powerful integration with [Prometheus] for monitoring your apps. -Metrics are retrieved from the configured Prometheus server, and then displayed +GitLab offers powerful integration with [Prometheus] for monitoring key metrics of your apps, directly within GitLab. +Metrics for each environment are retrieved from Prometheus, and then displayed within the GitLab interface. -Each project can be configured with its own specific Prometheus server, see the -[configuration](#configuration) section for more details. If you have a single -Prometheus server which monitors all of your infrastructure, you can pre-fill -the settings page with a default template. To configure the template, see the -[Services templates](services_templates.md) document. + -## Requirements +There are two ways to setup Prometheus integration, depending on where your apps are running: +* For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes) +* For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). -Integration with Prometheus requires the following: - -1. GitLab 9.0 or higher -1. Prometheus must be configured to collect one of the [supported metrics](prometheus_library/metrics.md) -1. Each metric must be have a label to indicate the environment -1. GitLab must have network connectivity to the Prometheus server - -## Getting started with Prometheus monitoring +Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-ci-cd-environments). -Depending on your deployment and where you have located your GitLab server, there are a few options to get started with Prometheus monitoring. +## Enabling Prometheus Integration -* If both GitLab and your applications are installed in the same Kubernetes cluster, you can leverage the [bundled Prometheus server within GitLab](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes). -* If your applications are deployed on Kubernetes, but GitLab is not in the same cluster, then you can [configure a Prometheus server in your Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes). -* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-prometheus-outside-of-kubernetes). +### Managed Prometheus on Kubernetes +> **Note**: [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28916) in GitLab 10.5 -### Getting started with Prometheus outside of Kubernetes - -Installing and configuring Prometheus to monitor applications is fairly straight forward. - -1. [Install Prometheus](https://prometheus.io/docs/introduction/install/) -1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md) -1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config) +GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.md), making monitoring of your apps easy. -### Configuring Omnibus GitLab Prometheus to monitor Kubernetes deployments +#### Requirements -With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled -version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>). +* A [connected Kubernetes cluster](../clusters/index.md) +* Helm Tiller [installed by GitLab](../clusters/index.md#installing-applications) -1. Read how to configure the bundled Prometheus server in the - [Administration guide][gitlab-prometheus-k8s-monitor]. -1. Now that Prometheus is configured, proceed on - [configuring the Prometheus project service in GitLab](#configuration-in-gitlab). +#### Getting started -### Configuring your own Prometheus server within Kubernetes +Once you have a connected Kubernetes cluster with Helm installed, deploying a managed Prometheus is as easy as a single click. -Setting up and configuring Prometheus within Kubernetes is quick and painless. -The Prometheus project provides an [official Docker image][prometheus-docker-image] -which we can use as a starting point. +1. Go to the `CI/CD > Kubernetes` page, to view your connected clusters +1. Select the cluster you would like to deploy Prometheus to +1. Click the **Install** button to deploy Prometheus to the cluster -To get started quickly, we have provided a [sample YML file][prometheus-yml] -that can be used as a template. This file will create a `prometheus` **Namespace**, -**Service**, **Deployment**, and **ConfigMap** in Kubernetes. You can upload -this file to the Kubernetes dashboard using **+ Create** at the top right. + - +#### About managed Prometheus deployments -Or use `kubectl`: +Prometheus is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/prometheus). Prometheus is only accessible within the cluster, with GitLab communicating through the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/). -```bash -kubectl apply -f path/to/prometheus.yml -``` +The Prometheus server will [automatically detect and monitor](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E) nodes, pods, and endpoints. To configure a resource to be monitored by Prometheus, simply set the following [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/): +* `prometheus.io/scrape` to `true` to enable monitoring of the resource. +* `prometheus.io/port` to define the port of the metrics endpoint. +* `prometheus.io/path` to define the path of the metrics endpoint. Defaults to `/metrics`. -Once deployed, you should see the Prometheus service, deployment, and -pod start within the `prometheus` namespace. The server will begin to collect -metrics from each Kubernetes Node in the cluster, based on the configuration -provided in the template. It will also attempt to collect metrics from any Kubernetes Pods that have been [annotated for Prometheus](https://prometheus.io/docs/operating/configuration/#pod). +CPU and Memory consumption is monitored, but requires [naming conventions](prometheus_library/kubernetes.html#specifying-the-environment) in order to determine the environment. If you are using [Auto DevOps](../../../topics/autodevops/), this is handled automatically. -Since GitLab is not running within Kubernetes, the template provides external -network access via a `NodePort` running on `30090`. This method allows access -to be controlled using provider firewall rules, like within Google Compute Engine. +The [NGINX Ingress](../clusters/index.md#installing-applications) that is deployed by GitLab to clusters, is automatically annotated for monitoring providing key response metrics: latency, throughput, and error rates. -Since a `NodePort` does not automatically have firewall rules created for it, -one will need to be created manually to allow access. In GCP/GKE, you will want -to confirm the Node that the Prometheus pod is running on. This can be done -either by looking at the Pod in the Kubernetes dashboard, or by running: +### Manual configuration of Prometheus -```bash -kubectl describe pods -n prometheus -``` +#### Requirements -Next on GKE, we need to get the `tag` of the Node or VM Instance, so we can -create an accurate firewall rule. The easiest way to do this is to go into the -Google Cloud Platform Compute console and select the VM instance that matches -the name of the Node gathered from the step above. In this case, the node tag -needed is `gke-prometheus-demo-5d5ada10-node`. Also make a note of the -**External IP**, which will be the IP address the Prometheus server is reachable -on. - - - -Armed with the proper Node tag, the firewall rule can now be created -specifically for this node. To create the firewall rule, open the Google Cloud -Platform Networking console, and select **Firewall Rules**. - -Create a new rule: +Integration with Prometheus requires the following: -- Specify the source IP range to match your desired access list, which should - include your GitLab server. A sample of GitLab.com's IP address range is - available [in this issue][gitlab.com-ip-range], but note that GitLab.com's IPs - are subject to change without prior notification. -- Allowed protocol and port should be `tcp:30090`. -- The target tags should match the Node tag identified earlier in this step. +1. GitLab 9.0 or higher +1. Prometheus must be configured to collect one of the [supported metrics](prometheus_library/metrics.md) +1. Each metric must be have a label to indicate the environment +1. GitLab must have network connectivity to the Prometheus server - +#### Getting started ---- +Installing and configuring Prometheus to monitor applications is fairly straight forward. -Now that Prometheus is configured, proceed to -[configure the Prometheus project service in GitLab](##configuration-in-gitlab). +1. [Install Prometheus](https://prometheus.io/docs/introduction/install/) +1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md) +1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config) -## Configuration in GitLab +#### Configuration in GitLab The actual configuration of Prometheus integration within GitLab is very simple. All you will need is the DNS or IP address of the Prometheus server you'd like @@ -133,9 +87,9 @@ to integrate with. Once configured, GitLab will attempt to retrieve performance metrics for any environment which has had a successful deployment. -GitLab will automatically scan the Prometheus server for known metrics and attempt to identify the metrics for a particular environment. The supported metrics and scan process is detailed in our [Prometheus Metric Library documentation](prometheus_library/metrics.html). +GitLab will automatically scan the Prometheus server for metrics from known serves like Kubernetes and NGINX, and attempt to identify individual environment. The supported metrics and scan process is detailed in our [Prometheus Metric Library documentation](prometheus_library/metrics.html). -[Learn more about monitoring environments.](../../../ci/environments.md#monitoring-environments) +You can view the performance dashboard for an environment by [clicking on the monitoring button](../../../ci/environments.md#monitoring-environments). ## Determining the performance impact of a merge @@ -143,7 +97,7 @@ GitLab will automatically scan the Prometheus server for known metrics and attem > GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-ce/issues/27439) of the 30 minute averages. > Requires [Kubernetes](prometheus_library/kubernetes.md) metrics -Developers can view theperformance impact of their changes within the merge +Developers can view the performance impact of their changes within the merge request workflow. When a source branch has been deployed to an environment, a sparkline and numeric comparison of the average memory consumption will appear. On the sparkline, a dot indicates when the current changes were deployed, with up to 30 minutes of performance data displayed before and after. The comparison shows the difference between the 30 minute average before and after the deployment. This information is updated after @@ -159,7 +113,7 @@ Prometheus server. ## Troubleshooting -If the "Attempting to load performance data" screen continues to appear, it could be due to: +If the "No data found" screen continues to appear, it could be due to: - No successful deployments have occurred to this environment. - Prometheus does not have performance data for this environment, or the metrics diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index a6673fa2a00..6b190deaa6c 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -11,22 +11,27 @@ integration services must be enabled. ## Metrics supported -| Name | Query | -| ---- | ----- | -| Average Memory Usage (MB) | (sum(avg(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) without (job))) / count(avg(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) without (job)) /1024/1024 | -| Average CPU Utilization (%) | sum(avg(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="%{ci_environment_slug}"}[2m])) without (job)) * 100 | +- Average Memory Usage (MB): -## Configuring Prometheus to monitor for Kubernetes node metrics + ``` + avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 + ``` -In order for Prometheus to collect Kubernetes metrics, you first must have a -Prometheus server up and running. You have two options here: +- Average CPU Utilization (%): -- If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes). -- To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) or [our guide](../../../../administration/monitoring/prometheus/index.md#configuring-your-own-prometheus-server-within-kubernetes). + ``` + avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) + ``` -## Specifying the Environment label +## Configuring Prometheus to monitor for Kubernetes metrics -In order to isolate and only display relevant metrics for a given environment -however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments). +Prometheus needs to be deployed into the cluster and configured properly in order to gather Kubernetes metrics. GitLab supports two methods for doing so: -If you are using [GitLab Auto-Deploy](../../../../ci/autodeploy/index.md) and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the `environment` label will be automatically added. +- GitLab [integrates with Kubernetes](../../clusters/index.md), and can [deploy Prometheus into a connected cluster](../prometheus.html#managed-prometheus-on-kubernetes). It is automatically configured to collect Kubernetes metrics. +- To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/). + +## Specifying the Environment + +In order to isolate and only display relevant CPU and Memory metrics for a given environment, GitLab needs a method to detect which containers it is running. Because these metrics are tracked at the container level, traditional Kubernetes labels are not available. + +Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with [CI_ENVIRONMENT_SLUG](../../../../ci/variables/README.md#predefined-variables-environment-variables). It can be followed by a `-` and additional content if desired. For example, a deployment name of `review-homepage-5620p5` would match the `review/homepage` environment. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index e6f13d0630b..49b34c82ae6 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -2,11 +2,11 @@ > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13438) in GitLab 9.5 -GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included in [version 0.9.0](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/Changelog.md#09-beta1) of the ingress. +GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included in [version 0.9.0](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/Changelog.md#09-beta1) and above of the ingress. ## Requirements -The [Prometheus service](../prometheus/index.md) must be enabled. +[Prometheus integration](../prometheus/index.md) must be active. ## Metrics supported @@ -18,24 +18,34 @@ The [Prometheus service](../prometheus/index.md) must be enabled. ## Configuring NGINX ingress monitoring -If you have deployed with the [gitlab-omnibus](https://docs.gitlab.com/ee/install/kubernetes/gitlab_omnibus.md) Helm chart, and your application is running in the same cluster, no further action is required. The ingress metrics will be automatically enabled and annotated for Prometheus monitoring. Simply ensure Prometheus monitoring is [enabled for your project](../prometheus.md), which is on by default. +If you have deployed NGINX Ingress using GitLab's [Kubernetes cluster integration](../../clusters/index.md#installing-applications), it will [automatically be monitored](#about-managed-nginx-ingress-deployments) by Prometheus. -For other deployments, there is some configuration required depending on your installation: -* NGINX Ingress should be version 0.9.0 or above +For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation: +* NGINX Ingress should be version 0.9.0 or above, with metrics enabled * NGINX Ingress should be annotated for Prometheus monitoring * Prometheus should be configured to monitor annotated pods -### Setting up NGINX Ingress for Prometheus monitoring +### About managed NGINX Ingress deployments + +NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). NGINX Ingress will be [externally reachable via the Load Balancer's IP](https://docs.gitlab.com/ce/user/project/clusters/index.html#getting-the-external-ip-address). + +NGINX is configured for Prometheus monitoring, by setting: +* `enable-vts-status: "true"`, to export Prometheus metrics +* `prometheus.io/scrape: "true"`, to enable automatic discovery +* `prometheus.io/port: "10254"`, to specify the metrics port + +When used in conjunction with the GitLab deployed Prometheus service, response metrics will be automatically collected. + +### Manually setting up NGINX Ingress for Prometheus monitoring Version 0.9.0 and above of [NGINX ingress](https://github.com/kubernetes/ingress/tree/master/controllers/nginx) have built-in support for exporting Prometheus metrics. To enable, a ConfigMap setting must be passed: `enable-vts-status: "true"`. Once enabled, a Prometheus metrics endpoint will start running on port 10254. -With metric data now available, Prometheus needs to be configured to collect it. The easiest way to do this is to leverage Prometheus' [built-in Kubernetes service discovery](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config), which automatically detects a variety of Kubernetes components and makes them available for monitoring. Since NGINX ingress metrics are exposed per pod, a scrape job for Kubernetes pods is required. A sample pod scraping configuration [is available](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L248). This configuration will detect pods and enable collection of metrics **only if** they have been specifically annotated for monitoring. +Next, the ingress needs to be annotated for Prometheus monitoring. Two new annotations need to be added: -Depending on how NGINX ingress was deployed, typically a DaemonSet or Deployment, edit the corresponding YML spec. Two new annotations need to be added: * `prometheus.io/scrape: "true"` * `prometheus.io/port: "10254"` -Prometheus should now be collecting NGINX ingress metrics. To validate view the Prometheus Targets, available under `Status > Targets` on the Prometheus dashboard. New entries for NGINX should be listed in the kubernetes pod monitoring job, `kubernetes-pods`. +Managing these settings depends on how NGINX ingress has been deployed. If you have deployed via the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress), metrics can be enabled with `controller.stats.enabled` along with the required annotations. Alternatively it is possible edit the NGINX ingress YML directly in the [Kubernetes dashboard](https://github.com/kubernetes/dashboard). ## Specifying the Environment label diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index cc3218fbfd1..de2cf6d4647 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -12,6 +12,8 @@ in the table below. | `issues_url` | The URL to the issue in Redmine project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. | | `new_issue_url` | This is the URL to create a new issue in Redmine for the project linked to this GitLab project. **This is currently not being used and will be removed in a future release.** | + Once you have configured and enabled Redmine you'll see the Redmine link on the GitLab project pages that takes you to the appropriate Redmine project. + As an example, below is a configuration for a project named gitlab-ci.  diff --git a/doc/user/project/integrations/samples/prometheus.yml b/doc/user/project/integrations/samples/prometheus.yml deleted file mode 100644 index 3a4735d282f..00000000000 --- a/doc/user/project/integrations/samples/prometheus.yml +++ /dev/null @@ -1,107 +0,0 @@ -apiVersion: v1 -kind: Namespace -metadata: - name: prometheus ---- -apiVersion: v1 -kind: ConfigMap -metadata: - name: prometheus - namespace: prometheus -data: - prometheus.yml: |- - scrape_configs: - - job_name: 'kubernetes-nodes' - scheme: https - tls_config: - ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt - insecure_skip_verify: true - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token - kubernetes_sd_configs: - - role: node - metric_relabel_configs: - - source_labels: [pod_name] - target_label: environment - regex: (.+)-.+-.+ - replacement: $1 - - job_name: kubernetes-pods - tls_config: - ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" - insecure_skip_verify: true - bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token" - kubernetes_sd_configs: - - role: pod - api_server: https://kubernetes.default.svc:443 - tls_config: - ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" - bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token" - relabel_configs: - - source_labels: - - __meta_kubernetes_pod_annotation_prometheus_io_scrape - action: keep - regex: 'true' - - source_labels: - - __meta_kubernetes_pod_annotation_prometheus_io_path - action: replace - target_label: __metrics_path__ - regex: "(.+)" - - source_labels: - - __address__ - - __meta_kubernetes_pod_annotation_prometheus_io_port - action: replace - regex: "([^:]+)(?::[0-9]+)?;([0-9]+)" - replacement: "$1:$2" - target_label: __address__ - - action: labelmap - regex: __meta_kubernetes_pod_label_(.+) - - source_labels: - - __meta_kubernetes_namespace - action: replace - target_label: kubernetes_namespace - - source_labels: - - __meta_kubernetes_pod_name - action: replace - target_label: kubernetes_pod_name ---- -apiVersion: v1 -kind: Service -metadata: - name: prometheus - namespace: prometheus -spec: - selector: - app: prometheus - ports: - - name: prometheus - protocol: TCP - port: 9090 - nodePort: 30090 - type: NodePort ---- -apiVersion: extensions/v1beta1 -kind: Deployment -metadata: - name: prometheus - namespace: prometheus -spec: - replicas: 1 - template: - metadata: - labels: - app: prometheus - spec: - containers: - - name: prometheus - image: prom/prometheus:latest - args: - - '--config.file=/prometheus-data/prometheus.yml' - ports: - - name: prometheus - containerPort: 9090 - volumeMounts: - - name: data-volume - mountPath: /prometheus-data - volumes: - - name: data-volume - configMap: - name: prometheus diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 8c2690ec3b2..b4a842f33d6 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -34,7 +34,7 @@ and deploy from one single platform. Issue Boards help you to visualize and manage the entire process _in_ GitLab. With [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards), available -only in [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/), +only in [GitLab Ultimate](https://about.gitlab.com/products/), you go even further, as you can not only keep yourself and your project organized from a broader perspective with one Issue Board per project, but also allow your team members to organize their own workflow by creating @@ -235,6 +235,27 @@ to another list the label changes and a system not is recorded. [Developers and up](../permissions.md) can use all the functionality of the Issue Board, that is create/delete lists and drag issues around. +## Group Issue Board + +>Introduced in GitLab 10.6 + +Group issue board is analogous to project-level issue board and it is accessible at the group +navigation level. A group-level issue board allows you to view all issues from all projects in that group +(currently, it does not see issues from projects in subgroups). Similarly, you can only filter by group labels for these +boards. When updating milestones and labels for an issue through the sidebar update mechanism, again only +group-level objects are available. + +## Features per tier + +Different issue board features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: + +| Tier | Number of project issue boards | Board with configuration in project issue boards | Number of group issue boards | Board with configuration in group issue boards | +| --- | --- | --- | --- | --- | +| Core | 1 | No | 1 | No | +| Starter | Multiple | Yes | 1 | No | +| Premium | Multiple | Yes | Multiple | Yes | +| Ultimate | Multiple | Yes | Multiple | Yes | + ## Tips A few things to remember: diff --git a/doc/user/project/issues/create_new_issue.md b/doc/user/project/issues/create_new_issue.md index 9af088374a1..1688edc1ee2 100644 --- a/doc/user/project/issues/create_new_issue.md +++ b/doc/user/project/issues/create_new_issue.md @@ -36,3 +36,25 @@ From an Issue Board, create a new issue by clicking on the plus sign (**+**) on It opens a new issue for that project labeled after its respective list.  + +## New issue via email + +*This feature needs [incoming email](../../../administration/incoming_email.md) +to be configured by a GitLab administrator to be available for CE/EE users, and +it's available on GitLab.com.* + +At the bottom of a project's issue page, click +**Email a new issue to this project**, and you will find an email address +which belongs to you. You could add this address to your contact. + +This is a private email address, generated just for you. +**Keep it to yourself** as anyone who gets ahold of it can create issues or +merge requests as if they were you. You can add this address to your contact +list for easy access. + +Sending an email to this address will create a new issue on your behalf for +this project, where the email subject becomes the issue title, and the email +body becomes the issue description. [Markdown] and [quick actions] are +supported. + + diff --git a/doc/user/project/issues/img/new_issue_from_email.png b/doc/user/project/issues/img/new_issue_from_email.png Binary files differnew file mode 100644 index 00000000000..775ea0cdffb --- /dev/null +++ b/doc/user/project/issues/img/new_issue_from_email.png diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 3e81dcb78c6..be4436749f9 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -35,7 +35,7 @@ your project public, open to collaboration. ### Streamline collaboration With [Multiple Assignees for Issues](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html), -available in [GitLab Enterprise Edition Starter](https://about.gitlab.com/gitlab-ee/) +available in [GitLab Starter](https://about.gitlab.com/products/) you can streamline collaboration and allow shared responsibilities to be clearly displayed. All assignees are shown across your workflows and receive notifications (as they would as single assignees), simplifying communication and ownership. @@ -64,9 +64,7 @@ You can also [search and filter](../../search/index.md#issues-and-merge-requests ### Issues per group -View all the issues in a group (that is, all the issues across all projects in that -group) by navigating to **Group > Issues**. This view also has the open and closed -issue tabs. +View issues in all projects in the group, including all projects of all descendant subgroups of the group. Navigate to **Group > Issues** to view these issues. This view also has the open and closed issues tabs.  @@ -141,7 +139,7 @@ Find GitLab Issue Boards by navigating to your **Project's Dashboard** > **Issue Read through the documentation for [Issue Boards](../issue_board.md) to find out more about this feature. -With [GitLab Enterprise Edition Starter](https://about.gitlab.com/gitlab-ee/), you can also +With [GitLab Starter](https://about.gitlab.com/products/), you can also create various boards per project with [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards). ### External Issue Tracker diff --git a/doc/user/project/issues/issues_functionalities.md b/doc/user/project/issues/issues_functionalities.md index 66140f389af..f2ca6a6822e 100644 --- a/doc/user/project/issues/issues_functionalities.md +++ b/doc/user/project/issues/issues_functionalities.md @@ -41,16 +41,17 @@ it's reassigned to someone else to take it from there. if a user is not member of that project, it can only be assigned to them if they created the issue themselves. -##### 3.1. Multiple Assignees (EES/EEP) +##### 3.1. Multiple Assignees -Multiple Assignees are only available in [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/). +> Available in [GitLab Starter](https://about.gitlab.com/products/) and +[GitLab.com Bronze](https://about.gitlab.com/gitlab-com/). Often multiple people likely work on the same issue together, which can especially be difficult to track in large teams where there is shared ownership of an issue. -In GitLab Enterprise Edition, you can also select multiple assignees -to an issue. +In [GitLab Starter](https://about.gitlab.com/products/), you can also +select multiple assignees to an issue. Learn more on the [Multiple Assignees documentation](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html). @@ -88,9 +89,10 @@ but they are immediately available to all projects in the group. > **Tip:** if the label doesn't exist yet, when you click **Edit**, it opens a dropdown menu from which you can select **Create new label**. -#### 8. Weight (EES/EEP) +#### 8. Weight -Issue Weights are only available in [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/). +> Available in [GitLab Starter](https://about.gitlab.com/products/) and +[GitLab.com Bronze](https://about.gitlab.com/gitlab-com/). - Attribute a weight (in a 0 to 9 range) to that issue. Easy to complete should weight 1 and very hard to complete should weight 9. diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index d7eb4bca89c..dabffaec5fa 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -1,174 +1,125 @@ # Labels -Labels provide an easy way to categorize the issues or merge requests based on -descriptive titles like `bug`, `documentation` or any other text you feel like. -They can have different colors, a description, and are visible throughout -the issue tracker or inside each issue individually. +## Overview -With labels, you can navigate the issue tracker and filter any bloated -information to visualize only the issues you are interested in. Let's see how -that works. +Labels allow you to categorize issues or merge requests using descriptive titles like `bug`, `feature request`, or `docs`. Each label also has a customizable color. They allow you to quickly and dynamically filter and manage issues or merge requests you care about, and are visible throughout GitLab in most places where issues and merge requests are located. -## Create new labels +## Project labels and group labels + +In GitLab, you can create project and group labels: + +- **Project labels** can be assigned to issues or merge requests in that project only. +- **Group labels** can be assigned to any issue or merge request of any project in that group. +- In the [future](https://gitlab.com/gitlab-org/gitlab-ce/issues/40915), you will be able to assign group labels to issues and merge reqeusts of projects in [subgroups](../group/subgroups/index.md). + +## Creating labels >**Note:** -A permission level of `Developer` or higher is required in order to manage -labels. +A permission level of `Developer` or higher is required to create labels. -Head over a single project and navigate to **Issues > Labels**. +### New project label -The first time you visit this page, you'll notice that there are no labels -created yet. +To create a **project label**, navigate to **Issues > Labels** in the project. -Creating a new label from scratch is as easy as pressing the **New label** -button. From there on you can choose the name, give it an optional description, -a color and you are set. +Click the **New label** button. Enter the title, an optional description, and the background color. Click **Create label** to create the label. -When you are ready press the **Create label** button to create the new label. +If a project has no labels, you can generate a default set of project labels from its empty label list page: - + ---- +GitLab will add the following default labels to the project: -## Default labels + -The very first time you visit the labels area, it's gonna be empty. In that -case, it's possible to populate the labels for your project from a set of -predefined labels. +### New group label -Click the link to 'Generate a default set of labels' and GitLab will -generate them for you. There are 8 default generated labels in total: +To create a **group label**, follow similar steps from above to project labels. Navigate to **Issues > Labels** in the group and create it from there. -- bug -- confirmed -- critical -- discussion -- documentation -- enhancement -- suggestion -- support +Group labels appear in every label list page of the group's child projects. -## Labels Overview + - +### New project label from sidebar -You can see that from the labels page you can have an overview of the number of -issues and merge requests assigned to each label. +From the sidebar of an issue or a merge request, you can create a create a new **project label** inline immediately, instead of navigating to the project label list page. -## Prioritize labels + ->**Notes:** -> -> - Introduced in GitLab 8.9. -> - Priority sorting is based on the highest priority label only. This might -> change in the future, follow the discussion in -> https://gitlab.com/gitlab-org/gitlab-ce/issues/18554. +## Editing labels -Prioritized labels are like any other label, but sorted by priority. This allows -you to sort issues and merge requests by label priority. +NOTE: **Note:** +A permission level of `Developer` or higher is required to edit labels. -To prioritize labels, navigate to your project's **Issues > Labels** and click -on the star icon next to them to put them in the priority list. Click on the -star icon again to remove them from the list. +You can update a label by navigating to **Issues > Labels** in the project or group and clicking the pencil icon. -From there, you can drag them around to set the desired priority. Priority is -set from high to low with an ascending order. Labels with no priority, count as -having their priority set to null. +You can delete a label by clicking the trash icon. - +### Promoting project labels to group labels -Now that you have labels prioritized, you can use the 'Label priority' and 'Priority' -sort orders in the issues or merge requests tracker. +If you are expanding from a few projects to a larger number of projects within the same group, you may want to share the same label among multiple projects in the same group. If you previously created a project label and now want to make it available for other projects, you can promote it to a group label. -In the following, everything applies to both issues and merge requests, but we'll -refer to just issues for brevity. +From the project label list page, you can promote a project label to a group label. This will merge all project labels across all projects in this group with the same name into a single group label. All issues and merge requests that previously were assigned one of these project labels will now be assigned the new group label. This action cannot be reversed and the changes are permanent. -The 'Label priority' sort order positions issues with higher priority labels -toward the top, and issues with lower priority labels toward the bottom. A non-prioritized -label is considered to have the lowest priority. For a given issue, we _only_ consider the -highest priority label assigned to it in the comparison. ([We are discussing](https://gitlab.com/gitlab-org/gitlab-ce/issues/18554) -including all the labels in a given issue for this comparison.) Given two issues -are equal according to this sort comparison, their relative order is equal, and -therefore it's not guaranteed that one will be always above the other. + - +## Assigning labels from the sidebar -The 'Priority' sort order comparison first considers an issue's milestone's due date, -(if the issue is assigned a milestone and the milestone's due date exists), and then -secondarily considers the label priority comparison above. Sooner due dates results -a higher sort order. If an issue doesn't have a milestone due date, it is equivalent to -being assigned to a milestone that has a due date in the infinite future. Given two issues -are equal according to this two-stage sort comparison, their relative order is equal, and -therefore it's not guaranteed that one will be always above the other. +Every issue and merge request can be assigned any number of labels. The labels are visible on every issue and merge request page, in the sidebar. They are also visible in the issue board. From the sidebar, you can assign or unassign a label to the object (i.e. label or unlabel it). You can also perform this as a [quick action](quick_actions.md) in a comment. - +| View labels in sidebar | Assign labels from sidebar | +|:---:|:---:| +|  |  | +## Filtering issues and merge requests by label -## Subscribe to labels +### Filtering in list pages -If you don’t want to miss issues or merge requests that are important to you, -simply subscribe to a label. You’ll get notified whenever the label gets added -to an issue or merge request, making sure you don’t miss a thing. +From the project issue list page and the project merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels and project labels. -Go to your project's **Issues > Labels** area, find the label(s) you want to -subscribe to and click on the eye icon. Click again to unsubscribe. +From the group issue list page and the group merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels and project labels. - + -If you work on a large or popular project, try subscribing only to the labels -that are relevant to you. You’ll notice it’ll be much easier to focus on what’s -important. +### Filtering in issue boards -## Create a new label when inside an issue +- From [project boards](issue_board.md), you can filter by both group labels and project labels in the [search and filter bar](../search/index.md#issue-boards). -There are times when you are already inside an issue searching to assign a -label, only to realize it doesn't exist. Instead of going to the **Labels** -page and being distracted from your original purpose, you can create new -labels on the fly. +## Subscribing to labels -Expand the issue sidebar and select **Create new label** from the labels dropdown -list. Provide a name, pick a color and hit **Create**. The new label will be -ready to used right away! +From the project label list page and the group label list page, you can subscribe to [notifications](../../workflow/notifications.md) of a given label, to alert you that that label has been assigned to an issue or merge request. - + -## Assigning labels to issues and merge requests +## Label priority -There are generally two ways to assign a label to an issue or merge request. +>**Notes:** +> +> - Introduced in GitLab 8.9. +> - Priority sorting is based on the highest priority label only. [This discussion](https://gitlab.com/gitlab-org/gitlab-ce/issues/18554) considers changing this. -The first one is to assign a label when you first create or edit an issue or -merge request. +Labels can have relative priorities, which are used in the "Label priority" and "Priority" sort orders of the issue and merge request list pages. -The second way is by using the right sidebar when inside an issue or merge -request. Expand it and hit **Edit** in the labels area. Start typing the name -of the label you are looking for to narrow down the list, and select it. You -can add more than one labels at once. When done, click outside the sidebar area -for the changes to take effect. +From the project label list page, star a label to indicate that it has a priority. Drag starred labels up and down to change their priority. Higher means higher priority. Prioritization happens at the project level, only on the project label list page, and not on the group label list page. However, both project and group labels can be prioritized on the project label list page since both types are displayed on the project label list page. - - + ---- +On the project and group issue and merge request list pages, you can sort by `Label priority` and `Priority`, which account for objects (issues and merge requests) that have prioritized labels assigned to them. -To remove labels, expand the left sidebar and unmark them from the labels list. -Simple as that. +If you sort by `Label priority`, GitLab considers this sort comparison order: -## Use labels to filter issues +- Object with a higher priority prioritized label. +- Object without a prioritized label. -Once you start adding labels to your issues, you'll see the benefit of it. -Labels can have several uses, one of them being the quick filtering of issues -or merge requests. +Ties are broken arbitrarily. (Note that we _only_ consider the highest prioritized label in an object, and not any of the lower prioritized labels. [This discussion](https://gitlab.com/gitlab-org/gitlab-ce/issues/18554) considers changing this.) -Pick an existing label from the dropdown _Label_ menu or click on an existing -label from the issue tracker. In the latter case, you also get to see the -label description like shown below. + - +If you sort by `Priority`, GitLab considers this sort comparison order: ---- +- Object's assigned [milestone](milestones/index.md)'s due date is sooner, provided the object has a milestone and the milestone has a due date. If this isn't the case, consider the object having a due date in the infinite future. +- Object with a higher priority prioritized label. +- Object without a prioritized label. -And if you added a description to your label, you can see it by hovering your -mouse over the label in the issue tracker or wherever else the label is -rendered. +Ties are broken arbitrarily. - + diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index f5c748a03b3..5d819998dd9 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -16,19 +16,29 @@ say 'Project Acme', in GitLab is to make the 'Engineering' group the owner of 'P Acme'. But what if 'Project Acme' already belongs to another group, say 'Open Source'? This is where the group sharing feature can be of use. -To share 'Project Acme' with the 'Engineering' group, go to the project settings page for 'Project Acme' and use the left navigation menu to go to the **Settings > Members** section. +To share 'Project Acme' with the 'Engineering' group: - +1. For 'Project Acme' use the left navigation menu to go to **Settings > Members** -Then select the 'Share with group' tab by clicking it. +  -Now you can add the 'Engineering' group with the maximum access level of your choice. Click 'Share' to share it. +1. Select the 'Share with group' tab +1. Add the 'Engineering' group with the maximum access level of your choice +1. Click **Share** to share it - +  -After sharing 'Project Acme' with 'Engineering', the project will be listed on the group dashboard. +1. After sharing 'Project Acme' with 'Engineering', the project will be listed + on the group dashboard - +  + +Note that you can only share a project with: + +- groups for which you have an explicitly defined membership +- groups that contain a nested subgroup or project for which you have an explicitly defined role + +Admins are able to share projects with any group in the system. ## Maximum access level diff --git a/doc/user/project/merge_requests/img/allow_maintainer_push.png b/doc/user/project/merge_requests/img/allow_maintainer_push.png Binary files differnew file mode 100644 index 00000000000..91cc399f4ff --- /dev/null +++ b/doc/user/project/merge_requests/img/allow_maintainer_push.png diff --git a/doc/user/project/merge_requests/img/remove_source_branch_status.png b/doc/user/project/merge_requests/img/remove_source_branch_status.png Binary files differnew file mode 100644 index 00000000000..1377fab54ec --- /dev/null +++ b/doc/user/project/merge_requests/img/remove_source_branch_status.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 7037d7f5989..3640d236db4 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -28,13 +28,14 @@ With GitLab merge requests, you can: - Enable [fast-forward merge requests](#fast-forward-merge-requests) - Enable [semi-linear history merge requests](#semi-linear-history-merge-requests) as another security layer to guarantee the pipeline is passing in the target branch - [Create new merge requests by email](#create-new-merge-requests-by-email) +- Allow maintainers of the target project to push directly to the fork by [allowing edits from maintainers](maintainer_access.md) With **[GitLab Enterprise Edition][ee]**, you can also: -- View the deployment process across projects with [Multi-Project Pipeline Graphs](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html#multi-project-pipeline-graphs) (available only in GitLab Enterprise Edition Premium) -- Request [approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your managers (available in GitLab Enterprise Edition Starter) -- [Squash and merge](https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html) for a cleaner commit history (available in GitLab Enterprise Edition Starter) -- Analise the impact of your changes with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html) (available in GitLab Enterprise Edition Starter) +- View the deployment process across projects with [Multi-Project Pipeline Graphs](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html#multi-project-pipeline-graphs) (available only in GitLab Premium) +- Request [approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your managers (available in GitLab Starter) +- [Squash and merge](https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html) for a cleaner commit history (available in GitLab Starter) +- Analyze the impact of your changes with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html) (available in GitLab Starter) ## Use cases @@ -42,10 +43,10 @@ A. Consider you are a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request 1. You gather feedback from your team -1. You work on the implementation optimizing code with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html) (available in GitLab Enterprise Edition Starter) +1. You work on the implementation optimizing code with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html) (available in GitLab Starter) 1. You build and test your changes with GitLab CI/CD 1. You request the approval from your manager -1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Enterprise Edition Starter) +1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#manual-actions) for GitLab CI/CD 1. Your implementations were successfully shipped to your customer @@ -55,8 +56,8 @@ B. Consider you're a web developer writing a webpage for your company's: 1. You gather feedback from your reviewers 1. Your changes are previewed with [Review Apps](../../../ci/review_apps/index.md) 1. You request your web designers for their implementation -1. You request the [approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your manager (available in GitLab Enterprise Edition Starter) -1. Once approved, your merge request is [squashed and merged](https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html), and [deployed to staging with GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) (Squash and Merge is available in GitLab Enterprise Edition Starter) +1. You request the [approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your manager (available in GitLab Starter) +1. Once approved, your merge request is [squashed and merged](https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html), and [deployed to staging with GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) (Squash and Merge is available in GitLab Starter) 1. Your production team [cherry picks](#cherry-pick-changes) the merge commit into production ## Merge requests per project @@ -70,12 +71,28 @@ and you can use the tabs available to quickly filter by open and closed. You can ## Merge requests per group -View all the merge requests in a group (that is, all the merge requests across all projects in that -group) by navigating to **Group > Merge Requests**. This view also has the open, merged, and closed -merge request tabs, from which you can [search and filter the results](../../search/index.md#issues-and-merge-requests-per-group). +View merge requests in all projects in the group, including all projects of all descendant subgroups of the group. Navigate to **Group > Merge Requests** to view these merge requests. This view also has the open and closed merge requests tabs. + +You can [search and filter the results](../../search/index.md#issues-and-merge-requests-per-group) from here.  +## Removing the source branch + +When creating a merge request, select the "Remove source branch when merge +request accepted" option and the source branch will be removed when the merge +request is merged. + +This option is also visible in an existing merge request next to the merge +request button and can be selected/deselected before merging. It's only visible +to users with [Master permissions](../../permissions.md) in the source project. + +If the user viewing the merge request does not have the correct permissions to +remove the source branch and the source branch is set for removal, the merge +request widget will show the "Removes source branch" text. + + + ## Authorization for merge requests There are two main ways to have a merge request flow with GitLab: @@ -134,6 +151,10 @@ those conflicts in the GitLab UI. ## Create new merge requests by email +*This feature needs [incoming email](../../../administration/incoming_email.md) +to be configured by a GitLab administrator to be available for CE/EE users, and +it's available on GitLab.com.* + You can create a new merge request by sending an email to a user-specific email address. The address can be obtained on the merge requests page by clicking on a **Email a new merge request to this project** button. The subject will be @@ -146,6 +167,19 @@ administrator to do so.  +## Find the merge request that introduced a change + +> **Note**: this feature was [implemented in GitLab 10.5](https://gitlab.com/gitlab-org/gitlab-ce/issues/2383). + +When viewing the commit details page, GitLab will link to the merge request (or +merge requests, if it's in more than one) containing that commit. + +This only applies to commits that are in the most recent version of a merge +request - if a commit was in a merge request, then rebased out of that merge +request, they will not be linked. + +[Read more about merge request versions](versions.md) + ## Revert changes GitLab implements Git's powerful feature to revert any commit with introducing @@ -160,7 +194,7 @@ of merge request diff is created. When you visit a merge request that contains more than one pushes, you can select and compare the versions of those merge request diffs. -[Read more about the merge requests versions.](versions.md) +[Read more about merge request versions](versions.md) ## Work In Progress merge requests @@ -287,4 +321,4 @@ git checkout origin/merge-requests/1 ``` [protected branches]: ../protected_branches.md -[ee]: https://about.gitlab.com/gitlab-ee/ "GitLab Enterprise Edition" +[ee]: https://about.gitlab.com/products/ "GitLab Enterprise Edition" diff --git a/doc/user/project/merge_requests/maintainer_access.md b/doc/user/project/merge_requests/maintainer_access.md new file mode 100644 index 00000000000..c9763a3fe02 --- /dev/null +++ b/doc/user/project/merge_requests/maintainer_access.md @@ -0,0 +1,18 @@ +# Allow maintainer pushes for merge requests across forks + +> [Introduced][ce-17395] in GitLab 10.6. + +This feature is available for merge requests across forked projects that are +publicly accessible. It makes it easier for maintainers of projects to +collaborate on merge requests across forks. + +When enabled for a merge request, members with merge access to the target +branch of the project will be granted write permissions to the source branch +of the merge request. + +The feature can only be enabled by users who already have push access to the +source project, and only lasts while the merge request is open. + +Enable this functionality while creating a merge request: + + diff --git a/doc/user/project/merge_requests/work_in_progress_merge_requests.md b/doc/user/project/merge_requests/work_in_progress_merge_requests.md index 546c8bdc5e5..f01da06fa6e 100644 --- a/doc/user/project/merge_requests/work_in_progress_merge_requests.md +++ b/doc/user/project/merge_requests/work_in_progress_merge_requests.md @@ -7,7 +7,8 @@ have been marked a **Work In Progress**.  To mark a merge request a Work In Progress, simply start its title with `[WIP]` -or `WIP:`. +or `WIP:`. As an alternative, you're also able to do it by sending a commit +with its title starting with `wip` or `WIP` to the merge request's source branch.  diff --git a/doc/user/project/milestones/img/milestone_create.png b/doc/user/project/milestones/img/milestone_create.png Binary files differdeleted file mode 100644 index beb2caa897f..00000000000 --- a/doc/user/project/milestones/img/milestone_create.png +++ /dev/null diff --git a/doc/user/project/milestones/img/milestone_group_create.png b/doc/user/project/milestones/img/milestone_group_create.png Binary files differdeleted file mode 100644 index 7aaa7c56c15..00000000000 --- a/doc/user/project/milestones/img/milestone_group_create.png +++ /dev/null diff --git a/doc/user/project/milestones/img/milestones_new_group_milestone.png b/doc/user/project/milestones/img/milestones_new_group_milestone.png Binary files differnew file mode 100644 index 00000000000..8780394d72e --- /dev/null +++ b/doc/user/project/milestones/img/milestones_new_group_milestone.png diff --git a/doc/user/project/milestones/img/milestones_new_project_milestone.png b/doc/user/project/milestones/img/milestones_new_project_milestone.png Binary files differnew file mode 100644 index 00000000000..ba058428dfa --- /dev/null +++ b/doc/user/project/milestones/img/milestones_new_project_milestone.png diff --git a/doc/user/project/milestones/img/milestones_project_milestone_page.png b/doc/user/project/milestones/img/milestones_project_milestone_page.png Binary files differnew file mode 100644 index 00000000000..9717075b8d0 --- /dev/null +++ b/doc/user/project/milestones/img/milestones_project_milestone_page.png diff --git a/doc/user/project/milestones/img/milestones_promote_milestone.png b/doc/user/project/milestones/img/milestones_promote_milestone.png Binary files differnew file mode 100644 index 00000000000..99bee1240d4 --- /dev/null +++ b/doc/user/project/milestones/img/milestones_promote_milestone.png diff --git a/doc/user/project/milestones/img/sidebar.png b/doc/user/project/milestones/img/sidebar.png Binary files differdeleted file mode 100644 index 274962a936c..00000000000 --- a/doc/user/project/milestones/img/sidebar.png +++ /dev/null diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 27832b0fa2b..10e6321eb82 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -1,63 +1,111 @@ # Milestones -Milestones allow you to organize issues and merge requests into a cohesive group, -optionally setting a due date. A common use is keeping track of an upcoming -software version. Milestones can be created per-project or per-group. +## Overview -## Creating a project milestone +Milestones in GitLab are a way to track issues and merge requests created to achieve a broader goal in a certain period of time. + +Milestones allow you to organize issues and merge requests into a cohesive group, with an optional start date and an optional due date. + +## Project milestones and group milestones + +- **Project milestones** can be assigned to issues or merge requests in that project only. +- **Group milestones** can be assigned to any issue or merge request of any project in that group. +- In the [future](https://gitlab.com/gitlab-org/gitlab-ce/issues/36862), you will be able to assign group milestones to issues and merge reqeusts of projects in [subgroups](../../group/subgroups/index.md). + +## Creating milestones + +>**Note:** +A permission level of `Developer` or higher is required to create milestones. + +### New project milestone + +To create a **project milestone**, navigate to **Issues > Milestones** in the project. + +Click the **New milestone** button. Enter the title, an optional description, an optional start date, and an optional due date. Click **Create milestone** to create the milestone. + + + +### New group milestone + +To create a **group milestone**, follow similar steps from above to project milestones. Navigate to **Issues > Milestones** in the group and create it from there. + + + +## Editing milestones >**Note:** -You need [Master permissions](../../permissions.md) in order to create a milestone. +A permission level of `Developer` or higher is required to edit milestones. + +You can update a milestone by navigating to **Issues > Milestones** in the project or group and clicking the **Edit** button. -You can find the milestones page under your project's **Issues ➔ Milestones**. -To create a new milestone, simply click the **New milestone** button when in the -milestones page. A milestone can have a title, a description and start/due dates. -Once you fill in all the details, hit the **Create milestone** button. +You can delete a milestone by clicking the **Delete** button. - +### Promoting project milestones to group milestones -## Creating a group milestone +If you are expanding from a few projects to a larger number of projects within the same group, you may want to share the same milestone among multiple projects in the same group. If you previously created a project milestone and now want to make it available for other milestones, you can promote it to a group milestone. + +From the project milestone list page, you can promote a project milestone to a group milestone. This will merge all project milestones across all projects in this group with the same name into a single group milestones. All issues and merge requests that previously were assigned one of these project milestones will now be assigned the new group milestones. This action cannot be reversed and the changes are permanent. >**Note:** -You need [Master permissions](../../permissions.md) in order to create a milestone. +Not all features on the project milestone view are available on the group milestone view. If you promote a project milestone to a group milestone, you will lose these features. See [Milestone view](#milestone-view) to see which features are missing from the group milestone view. + + + +## Assigning milestones from the sidebar + +Every issue and merge request can be assigned a milestone. The milestones are visible on every issue and merge request page, in the sidebar. They are also visible in the issue board. From the sidebar, you can assign or unassign a milestones to the object. You can also perform this as a [quick action](../quick_actions.md) in a comment. [As mentioned](#project-milestones-and-group-milestones), for a given issue or merge request, both project milestones and group milestones can be selected and assigned to the object. + +## Filtering issues and merge requests by milestone + +### Filtering in list pages + +From the project issue/merge request list pages and the group issue/merge request list pages, you can [filter](../../search/index.md#issues-and-merge-requests) by both group milestones and project milestones. + +### Filtering in issue boards + +From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [search and filter bar](../../search/index.md#issue-boards). + +### Special milestone filters + +When filtering by milestone, in addition to choosing a specific project milestone or group milestone, you can choose a special milestone filter. -You can create a milestone for a group that will be shared across group projects. -On the group's **Issues ➔ Milestones** page, you will be able to see the state -of that milestone and the issues/merge requests count that it shares across the group projects. To create a new milestone click the **New milestone** button. The form is the same as when creating a milestone for a specific project which you can find in the previous item. +- **No Milestone**: Show issues or merge requests with no assigned milestone. +- **Upcoming**: Show issues or merge requests that have been assigned the open milestone that has the next upcoming due date (i.e. nearest due date in the future). +- **Started**: Show issues or merge requests that have an assigned milestone with a start date that is before today. -In addition to that you will be able to filter issues or merge requests by group milestones in all projects that belongs to the milestone group. +## Milestone view -## Milestone promotion +Not all features in the project milestone view are available in the group milestone view. This table summarizes the differences: -Project milestones can be promoted to group milestones if its project belongs to a group. When a milestone is promoted all other milestones across the group projects with the same title will be merged into it, which means all milestone's children like issues, merge requests and boards will be moved into the new promoted milestone. -The promote button can be found in the milestone view or milestones list. +| Feature | Project milestone view | Group milestone view | +|---|:---:|:---:| +| Title an description | ✓ | ✓ | +| Issues assigned to milestone | ✓ | | +| Merge requests assigned to milestone | ✓ | | +| Participants and labels used | ✓ | | +| Percentage complete | ✓ | ✓ | +| Start date and due date | ✓ | ✓ | +| Total issue time spent | ✓ | ✓ | +| Total issue weight | ✓ | | -## Special milestone filters +The milestone view shows the title and description. -In addition to the milestones that exist in the project or group, there are some -special options available when filtering by milestone: +### Project milestone features -* **No Milestone** - only show issues or merge requests without a milestone. -* **Upcoming** - show issues or merge request that belong to the next open - milestone with a due date, by project. (For example: if project A has - milestone v1 due in three days, and project B has milestone v2 due in a week, - then this will show issues or merge requests from milestone v1 in project A - and milestone v2 in project B.) -* **Started** - show issues or merge requests from any milestone with a start - date less than today. Note that this can return results from several - milestones in the same project. +These features are only available for project milestones and not group milestones. -## Milestone sidebar +- Issues assigned to the milestone are displayed in three columns: Unstarted issues, ongoing issues, and completed issues. +- Merge requests assigned to the milestone are displayed in four columns: Work in progress merge requests, waiting for merge, rejected, and closed. +- Participants and labels that are used in issues and merge requests that have the milestone assigned are displayed. -The milestone sidebar shows percentage complete, start date and due date, -issues, total issue weight, total issue time spent, and merge requests. +### Milestone sidebar -The percentage complete is calculated as: Closed and merged merge requests plus all closed issues divided by -total merge requests and issues. +The milestone sidebar on the milestone view shows the following: - +- Percentage complete, which is calculated as number of closed issues plus number of closed/merged merge requests divided by total number issues and merge requests. +- The start date and due date. +- The total time spent on all issues that have the milestone assigned. -## Quick actions +For project milestones only, the milestone sidebar shows the total issue weight of all issues that have the milestone assigned. -[Quick actions](../quick_actions.md) are available for assigning and removing -project and group milestones. + diff --git a/doc/user/project/pages/getting_started_part_four.md b/doc/user/project/pages/getting_started_part_four.md index bd0cb437924..e4ee2f7cdfa 100644 --- a/doc/user/project/pages/getting_started_part_four.md +++ b/doc/user/project/pages/getting_started_part_four.md @@ -1,16 +1,13 @@ -# GitLab Pages from A to Z: Part 4 +--- +last_updated: 2018-02-16 +author: Marcia Ramos +author_gitlab: marcia +level: intermediate +article_type: user guide +date: 2017-02-22 +--- -> **Article [Type](../../../development/writing_documentation.html#types-of-technical-articles)**: user guide || -> **Level**: intermediate || -> **Author**: [Marcia Ramos](https://gitlab.com/marcia) || -> **Publication date:** 2017/02/22 - -- [Part 1: Static sites and GitLab Pages domains](getting_started_part_one.md) -- [Part 2: Quick start guide - Setting up GitLab Pages](getting_started_part_two.md) -- [Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](getting_started_part_three.md) -- **Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages** - -## Creating and Tweaking `.gitlab-ci.yml` for GitLab Pages +# Creating and Tweaking GitLab CI/CD for GitLab Pages [GitLab CI](https://about.gitlab.com/gitlab-ci/) serves numerous purposes, to build, test, and deploy your app @@ -19,10 +16,13 @@ from GitLab through methods. You will need it to build your website with GitLab Pages, and deploy it to the Pages server. +To implement GitLab CI/CD, the first thing we need is a configuration +file called `.gitlab-ci.yml` placed at your website's root directory. + What this file actually does is telling the [GitLab Runner](https://docs.gitlab.com/runner/) to run scripts as you would do from the command line. The Runner acts as your -terminal. GitLab CI tells the Runner which commands to run. +terminal. GitLab CI/CD tells the Runner which commands to run. Both are built-in in GitLab, and you don't need to set up anything for them to work. @@ -34,7 +34,7 @@ need to understand just a few things to be able to write our own with its own syntax. You can always check your CI syntax with the [GitLab CI Lint Tool](https://gitlab.com/ci/lint). -**Practical Example:** +## Practical example Let's consider you have a [Jekyll](https://jekyllrb.com/) site. To build it locally, you would open your terminal, and run `jekyll build`. @@ -384,7 +384,3 @@ in parallel, or build a custom pipeline](https://about.gitlab.com/2016/07/29/the [pulling specific directories from different projects](https://about.gitlab.com/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/) to deploy this website you're looking at, docs.gitlab.com. - On this blog post, we teach you [how to use GitLab Pages to produce a code coverage report](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/). - -||| -|:--|--:| -|[**← Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates**](getting_started_part_three.md)|| diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index 1e19f422d94..290dfa5af84 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -1,31 +1,28 @@ -# GitLab Pages from A to Z: Part 1 - -> **Article [Type](../../../development/writing_documentation.html#types-of-technical-articles)**: user guide || -> **Level**: beginner || -> **Author**: [Marcia Ramos](https://gitlab.com/marcia) || -> **Publication date:** 2017/02/22 - -- **Part 1: Static sites and GitLab Pages domains** -- [Part 2: Quick start guide - Setting up GitLab Pages](getting_started_part_two.md) -- [Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](getting_started_part_three.md) -- [Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages](getting_started_part_four.md) - -## GitLab Pages from A to Z - -This is a comprehensive guide, made for those who want to +--- +last_updated: 2018-02-16 +author: Marcia Ramos +author_gitlab: marcia +level: beginner +article_type: user guide +date: 2017-02-22 +--- + +# Static sites and GitLab Pages domains + +This document is the beginning of a comprehensive guide, made for those who want to publish a website with GitLab Pages but aren't familiar with the entire process involved. -This [first part](#what-you-need-to-know-before-getting-started) of this series will present you to the concepts of +This [first document](#what-you-need-to-know-before-getting-started) of this series will present you to the concepts of static sites, and go over how the default Pages domains work. -The [second part](getting_started_part_two.md) covers how to get started with GitLab Pages: deploy +The [second document](getting_started_part_two.md) covers how to get started with GitLab Pages: deploy a website from a forked project or create a new one from scratch. -The [third part](getting_started_part_three.md) will show you how to set up a custom domain or subdomain +The [third document](getting_started_part_three.md) will show you how to set up a custom domain or subdomain to your site already deployed. -The [fourth part](getting_started_part_four.md) will show you how to create and tweak GitLab CI for +The [fourth document](getting_started_part_four.md) will show you how to create and tweak GitLab CI for GitLab Pages. To **enable** GitLab Pages for GitLab CE (Community Edition) @@ -113,6 +110,4 @@ You can only create the highest level group website. - On your GitLab instance, replace `gitlab.io` above with your Pages server domain. Ask your sysadmin for this information. -||| -|:--|--:| -||[**Part 2: Quick start guide - Setting up GitLab Pages →**](getting_started_part_two.md)| +_Read on about [Projects for GitLab Pages and URL structure](getting_started_part_two.md)._ diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index 0096f8507d2..430fe3af1f8 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -1,27 +1,19 @@ --- -last_updated: 2017-09-28 +last_updated: 2018-02-16 +author: Marcia Ramos +author_gitlab: marcia +level: beginner +article_type: user guide +date: 2017-02-22 --- -# GitLab Pages from A to Z: Part 3 +# GitLab Pages custom domains and SSL/TLS Certificates -> **[Article Type](../../../development/writing_documentation.md#types-of-technical-articles)**: user guide || -> **Level**: beginner || -> **Author**: [Marcia Ramos](https://gitlab.com/marcia) || -> **Publication date:** 2017-02-22 || -> **Last updated**: 2017-09-28 - -- [Part 1: Static sites and GitLab Pages domains](getting_started_part_one.md) -- [Part 2: Quick start guide - Setting up GitLab Pages](getting_started_part_two.md) -- **Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates** -- [Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages](getting_started_part_four.md) - -## Setting Up Custom Domains - DNS Records and SSL/TLS Certificates - -As described in the previous part of this series, setting up GitLab Pages with custom domains, and adding SSL/TLS certificates to them, are optional features of GitLab Pages. +Setting up GitLab Pages with custom domains, and adding SSL/TLS certificates to them, are optional features of GitLab Pages. These steps assume you've already [set your site up](getting_started_part_two.md) and and it's served under the default Pages domain `namespace.gitlab.io`, or `namespace.gitlab.io/project-name`. -### Adding your custom domain to GitLab Pages +## Adding your custom domain to GitLab Pages To use one or more custom domain with your Pages site, there are two things you should consider first, which we'll cover in this guide: @@ -36,7 +28,7 @@ Let's start from the beginning with [DNS records](#dns-records). If you already know how they work and want to skip the introduction to DNS, you may be interested in skipping it until the [TL;DR](#tl-dr) section below. -### DNS Records +## DNS Records A Domain Name System (DNS) web service routes visitors to websites by translating domain names (such as `www.example.com`) into the @@ -70,9 +62,9 @@ for the most popular hosting services: - [Microsoft](https://msdn.microsoft.com/en-us/library/bb727018.aspx) If your hosting service is not listed above, you can just try to -search the web for "how to add dns record on <my hosting service>". +search the web for `how to add dns record on <my hosting service>`. -#### DNS A record +### DNS A record In case you want to point a root domain (`example.com`) to your GitLab Pages site, deployed to `namespace.gitlab.io`, you need to @@ -87,7 +79,7 @@ running on your instance).  -#### DNS CNAME record +### DNS CNAME record In case you want to point a subdomain (`hello-world.example.com`) to your GitLab Pages site initially deployed to `namespace.gitlab.io`, @@ -103,12 +95,32 @@ without any `/project-name`.  +#### DNS TXT record + +Unless your GitLab administrator has [disabled custom domain verification](../../../administration/pages/index.md#custom-domain-verification), +you'll have to prove that you own the domain by creating a `TXT` record +containing a verification code. The code will be displayed after you +[add your custom domain to GitLab Pages settings](#add-your-custom-domain-to-gitlab-pages-settings). + +If using a [DNS A record](#dns-a-record), you can place the TXT record directly +under the domain. If using a [DNS CNAME record](#dns-cname-record), the two record types won't +co-exist, so you need to place the TXT record in a special subdomain of its own. + #### TL;DR +If the domain has multiple uses (e.g., you host email on it as well): + | From | DNS Record | To | | ---- | ---------- | -- | | domain.com | A | 52.167.214.135 | -| subdomain.domain.com | CNAME | namespace.gitlab.io | +| domain.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | + +If the domain is dedicated to GitLab Pages use and no other services run on it: + +| From | DNS Record | To | +| ---- | ---------- | -- | +| subdomain.domain.com | CNAME | gitlab.io | +| _gitlab-pages-verification-code.subdomain.domain.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | > **Notes**: > @@ -119,7 +131,7 @@ domain. E.g., **do not** point your `subdomain.domain.com` to `namespace.gitlab.io.` or `namespace.gitlab.io/`. > - GitLab Pages IP on GitLab.com [has been changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) from `104.208.235.32` to `52.167.214.135`. -### Add your custom domain to GitLab Pages settings +## Add your custom domain to GitLab Pages settings Once you've set the DNS record, you'll need navigate to your project's **Setting > Pages** and click **+ New domain** to add your custom domain to @@ -129,6 +141,17 @@ your site will be accessible only via HTTP:  +Once you have added a new domain, you will need to **verify your ownership** +(unless the GitLab administrator has disabled this feature). A verification code +will be shown to you; add it as a [DNS TXT record](#dns-txt-record), then press +the "Verify ownership" button to activate your new domain: + + + +Once your domain has been verified, leave the verification record in place - +your domain will be periodically reverified, and may be disabled if the record +is removed. + You can add more than one alias (custom domains and subdomains) to the same project. An alias can be understood as having many doors leading to the same room. @@ -136,13 +159,13 @@ All the aliases you've set to your site will be listed on **Setting > Pages**. From that page, you can view, add, and remove them. Note that [DNS propagation may take some time (up to 24h)](http://www.inmotionhosting.com/support/domain-names/dns-nameserver-changes/domain-names-dns-changes), -although it's usually a matter of minutes to complete. Until it does, visit attempts -to your domain will respond with a 404. +although it's usually a matter of minutes to complete. Until it does, verification +will fail and attempts to visit your domain will respond with a 404. Read through the [general documentation on GitLab Pages](introduction.md#add-a-custom-domain-to-your-pages-website) to learn more about adding custom domains to GitLab Pages sites. -### SSL/TLS Certificates +## SSL/TLS Certificates Every GitLab Pages project on GitLab.com will be available under HTTPS for the default Pages domain (`*.gitlab.io`). Once you set @@ -155,16 +178,41 @@ Certificates are NOT required to add to your custom (sub)domain on your GitLab Pages project, though they are highly recommendable. -The importance of having any website securely served under HTTPS -is explained on the introductory section of the blog post -[Secure GitLab Pages with StartSSL](https://about.gitlab.com/2016/06/24/secure-gitlab-pages-with-startssl/#https-a-quick-overview). +Let's start with an introduction to the importance of HTTPS. +Alternatively, jump ahead to [adding certificates to your project](#adding-certificates-to-your-project). + +### Why should I care about HTTPS? + +This might be your first question. If our sites are hosted by GitLab Pages, +they are static, hence we are not dealing with server-side scripts +nor credit card transactions, then why do we need secure connections? -The reason why certificates are so important is that they encrypt +Back in the 1990s, where HTTPS came out, [SSL](https://en.wikipedia.org/wiki/Transport_Layer_Security#SSL_1.0.2C_2.0_and_3.0) was considered a "special" +security measure, necessary just for big companies, like banks and shoppings sites +with financial transactions. +Now we have a different picture. [According to Josh Aas](https://letsencrypt.org/2015/10/29/phishing-and-malware.html), Executive Director at [ISRG](https://en.wikipedia.org/wiki/Internet_Security_Research_Group): + +> _We’ve since come to realize that HTTPS is important for almost all websites. It’s important for any website that allows people to log in with a password, any website that [tracks its users](https://www.washingtonpost.com/news/the-switch/wp/2013/12/10/nsa-uses-google-cookies-to-pinpoint-targets-for-hacking/) in any way, any website that [doesn’t want its content altered](http://arstechnica.com/tech-policy/2014/09/why-comcasts-javascript-ad-injections-threaten-security-net-neutrality/), and for any site that offers content people might not want others to know they are consuming. We’ve also learned that any site not secured by HTTPS [can be used to attack other sites](http://krebsonsecurity.com/2015/04/dont-be-fodder-for-chinas-great-cannon/)._ + +Therefore, the reason why certificates are so important is that they encrypt the connection between the **client** (you, me, your visitors) and the **server** (where you site lives), through a keychain of authentications and validations. -### Issuing Certificates +How about taking Josh's advice and protecting our sites too? We will be +well supported, and we'll contribute to a safer internet. + +### Organizations supporting HTTPS + +There is a huge movement in favor of securing all the web. W3C fully +[supports the cause](https://w3ctag.github.io/web-https/) and explains very well +the reasons for that. Richard Barnes, a writer for Mozilla Security Blog, +suggested that [Firefox would deprecate HTTP](https://blog.mozilla.org/security/2015/04/30/deprecating-non-secure-http/), +and would no longer accept unsecured connections. Recently, Mozilla published a +[communication](https://blog.mozilla.org/security/2016/03/29/march-2016-ca-communication/) +reiterating the importance of HTTPS. + +## Issuing Certificates GitLab Pages accepts [PEM](https://support.quovadisglobal.com/kb/a37/what-is-pem-format.aspx) certificates issued by [Certificate Authorities (CA)](https://en.wikipedia.org/wiki/Certificate_authority) @@ -193,7 +241,7 @@ Their certs are valid up to 15 years. Read through the tutorial on Regardless the CA you choose, the steps to add your certificate to your Pages project are the same. -#### What do you need +### What do you need 1. A PEM certificate 1. An intermediate certificate @@ -203,7 +251,7 @@ your Pages project are the same. These fields are found under your **Project**'s **Settings** > **Pages** > **New Domain**. -#### What's what? +### What's what? - A PEM certificate is the certificate generated by the CA, which needs to be added to the field **Certificate (PEM)**. @@ -216,7 +264,7 @@ are one of these cases. - A public key is an encrypted key which validates your PEM against your domain. -#### Now what? +### Now what? Now that you hopefully understand why you need all of this, it's simple: @@ -233,6 +281,4 @@ just jumping a line between them. regular text editors. Always use code editors (such as Sublime Text, Atom, Dreamweaver, Brackets, etc). -||| -|:--|--:| -|[**← Part 2: Quick start guide - Setting up GitLab Pages**](getting_started_part_two.md)|[**Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages →**](getting_started_part_four.md)| +_Read on about [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md)_ diff --git a/doc/user/project/pages/getting_started_part_two.md b/doc/user/project/pages/getting_started_part_two.md index 64de0463dad..2274cac8ace 100644 --- a/doc/user/project/pages/getting_started_part_two.md +++ b/doc/user/project/pages/getting_started_part_two.md @@ -1,27 +1,23 @@ -# GitLab Pages from A to Z: Part 2 +--- +last_updated: 2018-02-16 +author: Marcia Ramos +author_gitlab: marcia +level: beginner +article_type: user guide +date: 2017-02-22 +--- -> **Article [Type](../../../development/writing_documentation.html#types-of-technical-articles)**: user guide || -> **Level**: beginner || -> **Author**: [Marcia Ramos](https://gitlab.com/marcia) || -> **Publication date:** 2017/02/22 - -- [Part 1: Static sites and GitLab Pages domains](getting_started_part_one.md) -- **Part 2: Quick start guide - Setting up GitLab Pages** -- [Part 3: Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](getting_started_part_three.md) -- [Part 4: Creating and tweaking `.gitlab-ci.yml` for GitLab Pages](getting_started_part_four.md) - -## Setting up GitLab Pages - -For a complete step-by-step tutorial, please read the -blog post [Hosting on GitLab.com with GitLab Pages](https://about.gitlab.com/2016/04/07/gitlab-pages-setup/). The following sections will explain -what do you need and why do you need them. +# Projects for GitLab Pages and URL structure ## What you need to get started +To get started with GitLab Pages, you need: + 1. A project 1. A configuration file (`.gitlab-ci.yml`) to deploy your site 1. A specific `job` called `pages` in the configuration file that will make GitLab aware that you are deploying a GitLab Pages website +1. A `public` directory with the content of the website Optional Features: @@ -51,35 +47,26 @@ containing the most popular SSGs templates. Watch the [video tutorial](https://youtu.be/TWqh9MtT4Bg) we've created for the steps below. -1. Choose your SSG template -1. Fork a project from the [Pages group](https://gitlab.com/pages) -1. Remove the fork relationship by navigating to your **Project**'s **Settings** > **Edit Project** +1. [Fork a sample project](../../../gitlab-basics/fork-project.md) from the [Pages group](https://gitlab.com/pages) +1. Trigger a build (push a change to any file) +1. As soon as the build passes, your website will have been deployed with GitLab Pages. Your website URL will be available under your project's **Settings** > **Pages** +1. Optionally, remove the fork relationship by navigating to your project's **Settings** > expanding **Advanced settings** and scrolling down to **Remove fork relashionship**:  -1. Enable Shared Runners for your fork: navigate to your **Project**'s **Settings** > **Pipelines** -1. Trigger a build (push a change to any file) -1. As soon as the build passes, your website will have been deployed with GitLab Pages. Your website URL will be available under your **Project**'s **Settings** > **Pages** - To turn a **project website** forked from the Pages group into a **user/group** website, you'll need to: -- Rename it to `namespace.gitlab.io`: navigate to **Project**'s **Settings** > **Edit Project** > **Rename repository** +- Rename it to `namespace.gitlab.io`: navigate to project's **Settings** > expand **Advanced settings** > and scroll down to **Rename repository** - Adjust your SSG's [base URL](#urls-and-baseurls) to from `"project-name"` to `""`. This setting will be at a different place for each SSG, as each of them have their own structure and file tree. Most likelly, it will be in the SSG's config file. > **Notes:** > ->1. Why do I need to remove the fork relationship? +> Why do I need to remove the fork relationship? > -> Unless you want to contribute to the original project, +> Unless you want to contribute to the original project, you won't need it connected to the upstream. A [fork](https://about.gitlab.com/2016/12/01/how-to-keep-your-fork-up-to-date-with-its-origin/#fork) is useful for submitting merge requests to the upstream. -> -> 2. Why do I need to enable Shared Runners? -> -> Shared Runners will run the script set by your GitLab CI -configuration file. They're enabled by default to new projects, -but not to forks. ### Create a project from scratch @@ -88,9 +75,9 @@ click **New project**, and name it considering the [practical examples](getting_started_part_one.md#practical-examples). 1. Clone it to your local computer, add your website files to your project, add, commit and push to GitLab. -1. From the your **Project**'s page, click **Set up CI**: +1. From the your **Project**'s page, click **Set up CI/CD**: -  +  1. Choose one of the templates from the dropbox menu. Pick up the template corresponding to the SSG you're using (or plain HTML). @@ -98,7 +85,7 @@ Pick up the template corresponding to the SSG you're using (or plain HTML).  Once you have both site files and `.gitlab-ci.yml` in your project's -root, GitLab CI will build your site and deploy it with Pages. +root, GitLab CI/CD will build your site and deploy it with Pages. Once the first build passes, you see your site is live by navigating to your **Project**'s **Settings** > **Pages**, where you'll find its default URL. @@ -108,7 +95,7 @@ where you'll find its default URL. > - GitLab Pages [supports any SSG](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/), but, if you don't find yours among the templates, you'll need to configure your own `.gitlab-ci.yml`. Do do that, please -read through the article [Creating and Tweaking `.gitlab-ci.yml` for GitLab Pages](getting_started_part_four.md). New SSGs are very welcome among +read through the article [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md). New SSGs are very welcome among the [example projects](https://gitlab.com/pages). If you set up a new one, please [contribute](https://gitlab.com/pages/pages.gitlab.io/blob/master/CONTRIBUTING.md) @@ -121,7 +108,7 @@ you can run `git init` in your local website directory, add the remote URL: `git remote add origin git@gitlab.com:namespace/project-name.git`, then add, commit, and push. -### URLs and Baseurls +## URLs and Baseurls Every Static Site Generator (SSG) default configuration expects to find your website under a (sub)domain (`example.com`), not @@ -149,11 +136,7 @@ example we've just mentioned, you'd have to change Jekyll's `_config.yml` to: baseurl: "" ``` -### Custom Domains +## Custom Domains -GitLab Pages supports custom domains and subdomains, served under HTTPS or HTTPS. +GitLab Pages supports custom domains and subdomains, served under HTTP or HTTPS. Please check the [next part](getting_started_part_three.md) of this series for an overview. - -||| -|:--|--:| -|[**← Part 1: Static sites, domains, DNS records, and SSL/TLS certificates**](getting_started_part_one.md)|[**Setting Up Custom Domains - DNS Records and SSL/TLS Certificates →**](getting_started_part_three.md)| diff --git a/doc/user/project/pages/img/verify_your_domain.png b/doc/user/project/pages/img/verify_your_domain.png Binary files differnew file mode 100644 index 00000000000..89c69cac9a5 --- /dev/null +++ b/doc/user/project/pages/img/verify_your_domain.png diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 8404d789de6..a65aa758198 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -37,10 +37,10 @@ to secure them. Read the following tutorials to know more about: -- [Static websites and GitLab Pages domains](getting_started_part_one.md) -- [Forking projects and creating new ones from scratch, URLs and baseurls](getting_started_part_two.md) -- [Custom domains and subdomains, DNS records, SSL/TLS certificates](getting_started_part_three.md) -- [How to create your own `.gitlab-ci.yml` for your site](getting_started_part_four.md) +- [Static websites and GitLab Pages domains](getting_started_part_one.md): Understand what is a static website, and how GitLab Pages default domains work +- [Projects for GitLab Pages and URL structure](getting_started_part_two.md): Forking projects and creating new ones from scratch, understanding URLs structure and baseurls +- [GitLab Pages custom domains and SSL/TLS Certificates](getting_started_part_three.md): How to add custom domains and subdomains to your website, configure DNS records, and SSL/TLS certificates +- [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md): Understand how to create your own `.gitlab-ci.yml` for your site - [Technical aspects, custom 404 pages, limitations](introduction.md) - [Hosting on GitLab.com with GitLab Pages](https://about.gitlab.com/2016/04/07/gitlab-pages-setup/) (outdated) @@ -54,7 +54,6 @@ _Blog posts for securing GitLab Pages custom domains with SSL/TLS certificates:_ - [CloudFlare](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) - [Let's Encrypt](https://about.gitlab.com/2016/04/11/tutorial-securing-your-gitlab-pages-with-tls-and-letsencrypt/) (outdated) -- [StartSSL](https://about.gitlab.com/2016/06/24/secure-gitlab-pages-with-startssl/) (deprecated) ## Advanced use diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index f52f66f518a..0b5076b8c5d 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -316,6 +316,47 @@ or various static site generators. Contributions are very welcome. Visit the GitLab Pages group for a full list of example projects: <https://gitlab.com/groups/pages>. +### Serving compressed assets + +Most modern browsers support downloading files in a compressed format. This +speeds up downloads by reducing the size of files. + +Before serving an uncompressed file, Pages will check whether the same file +exists with a `.gz` extension. If it does, and the browser supports receiving +compressed files, it will serve that version instead of the uncompressed one. + +To take advantage of this feature, the artifact you upload to the Pages should +have this structure: + +``` +public/ +├─┬ index.html +│ └ index.html.gz +│ +├── css/ +│ └─┬ main.css +│ └ main.css.gz +│ +└── js/ + └─┬ main.js + └ main.js.gz +``` + +This can be achieved by including a `script:` command like this in your +`.gitlab-ci.yml` pages job: + +```yaml +pages: + # Other directives + script: + - # build the public/ directory first + - find public -type f -iregex '.*\.\(htm\|html\|txt\|text\|js\|css\)$' -execdir gzip -f --keep {} \; +``` + +By pre-compressing the files and including both versions in the artifact, Pages +can serve requests for both compressed and uncompressed content without +needing to compress files on-demand. + ### Add a custom domain to your Pages website For a complete guide on Pages domains, read through the article diff --git a/doc/user/project/pipelines/schedules.md b/doc/user/project/pipelines/schedules.md index 34809a2826f..a13b1b4561c 100644 --- a/doc/user/project/pipelines/schedules.md +++ b/doc/user/project/pipelines/schedules.md @@ -12,7 +12,7 @@ month on the 22nd for a certain branch. In order to schedule a pipeline: -1. Navigate to your project's **Pipelines ➔ Schedules** and click the +1. Navigate to your project's **CI / CD ➔ Schedules** and click the **New Schedule** button. 1. Fill in the form 1. Hit **Save pipeline schedule** for the changes to take effect. diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md index 43451844f2d..6cead7b9961 100644 --- a/doc/user/project/pipelines/settings.md +++ b/doc/user/project/pipelines/settings.md @@ -27,6 +27,13 @@ The default value is 60 minutes. Decrease the time limit if you want to impose a hard limit on your jobs' running time or increase it otherwise. In any case, if the job surpasses the threshold, it is marked as failed. +### Timeout overriding on Runner level + +> - [Introduced][ce-17221] in GitLab 10.7. + +Project defined timeout (either specific timeout set by user or the default +60 minutes timeout) may be [overridden on Runner level][timeout overriding]. + ## Custom CI config path > - [Introduced][ce-12509] in GitLab 9.4. @@ -152,5 +159,7 @@ into your `README.md`: [var]: ../../../ci/yaml/README.md#git-strategy [coverage report]: #test-coverage-parsing +[timeout overriding]: ../../../ci/runners/README.html#setting-maximum-job-timeout-for-a-runner [ce-9362]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9362 [ce-12509]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12509 +[ce-17221]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17221 diff --git a/doc/user/project/repository/img/jupyter_notebook.png b/doc/user/project/repository/img/jupyter_notebook.png Binary files differnew file mode 100644 index 00000000000..52c5c5aea32 --- /dev/null +++ b/doc/user/project/repository/img/jupyter_notebook.png diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 9501db88f57..376f4e3cbe4 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -18,7 +18,7 @@ documentation. > **Important:** For security reasons, when using the command line, we strongly recommend -you to [connect with GitLab via SSH](../../../ssh/README.md). +that you [connect with GitLab via SSH](../../../ssh/README.md). ## Files @@ -53,6 +53,22 @@ To get started with the command line, please read through the Use GitLab's [file finder](../../../workflow/file_finder.md) to search for files in a repository. +### Jupyter Notebook files + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2508) in GitLab 9.1 + +[Jupyter][jupyter] Notebook (previously IPython Notebook) files are used for +interactive computing in many fields and contain a complete record of the +user's sessions and include code, narrative text, equations and rich output. + +When added to a repository, Jupyter Notebooks with a `.ipynb` extension will be +rendered to HTML when viewed. + + + +Interactive features, including JavaScript plots, will not work when viewed in +GitLab. + ## Branches When you submit changes in a new [branch](branches/index.md), you create a new version @@ -66,9 +82,8 @@ your implementation with your team. You can live preview changes submitted to a new branch with [Review Apps](../../../ci/review_apps/index.md). -With [GitLab Enterprise Edition](https://about.gitlab.com/gitlab-ee/) -subscriptions, you can also request -[approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html#merge-request-approvals) from your managers. +With [GitLab Starter](https://about.gitlab.com/products/), you can also request +[approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your managers. To create, delete, and [branches](branches/index.md) via GitLab's UI: @@ -117,8 +132,9 @@ Use GPG to [sign your commits](gpg_signed_commits/index.md). ## Repository size -In GitLab.com, your repository size limit it 10GB. For other instances, -the repository size is limited by your system administrators. +On GitLab.com, your [repository size limit is 10GB](../../gitlab_com/index.md#repository-size-limit) +(including LFS). For other instances, the repository size is limited by your +system administrators. You can also [reduce a repository size using Git](reducing_the_repo_size_using_git.md). @@ -147,13 +163,17 @@ Select branches to compare and view the changes inline: Find it under your project's **Repository > Compare**. -## Locked files (EEP) +## Locked files + +> Available in [GitLab Premium](https://about.gitlab.com/products/). Lock your files to prevent any conflicting changes. [File Locking](https://docs.gitlab.com/ee/user/project/file_lock.html) is available only in -[GitLab Enterprise Edition Premium](https://about.gitlab.com/gitlab-ee/). +[GitLab Premium](https://about.gitlab.com/products/). ## Repository's API You can access your repos via [repository API](../../../api/repositories.md). + +[jupyter]: https://jupyter.org diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index db0c3ed9d59..33c9a1a4d6b 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -45,7 +45,7 @@ has already been created, which creates a link to the license itself.  >**Note:** -The **Set up CI** button will not appear on an empty repository. You have to at +The **Set up CI/CD** button will not appear on an empty repository. You have to at least add a file in order for the button to show up. ## Upload a file diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index b8f865679a2..dedf102fc37 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -22,6 +22,7 @@ > in the import side is required to map the users, based on email or username. > Otherwise, a supplementary comment is left to mention the original author and > the MRs, notes or issues will be owned by the importer. +> - Control project Import/Export with the [API](../../../api/project_import_export.md). Existing projects running on any GitLab instance or GitLab.com can be exported with all their related data and be moved into a new GitLab instance. diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index f01fa5b1860..888dd0e143a 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -34,7 +34,7 @@ Set up your project's merge request settings: - Set up the merge request method (merge commit, [fast-forward merge](../merge_requests/fast_forward_merge.html)). - Merge request [description templates](../description_templates.md#description-templates). -- Enable [merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html#merge-request-approvals), _available in [GitLab Enterprise Edition Starter](https://about.gitlab.com/gitlab-ee/)_. +- Enable [merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html#merge-request-approvals), _available in [GitLab Starter](https://about.gitlab.com/products/)_. - Enable [merge only of pipeline succeeds](../merge_requests/merge_when_pipeline_succeeds.md). - Enable [merge only when all discussions are resolved](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-discussions-are-resolved). @@ -42,7 +42,7 @@ Set up your project's merge request settings: ### Service Desk -Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) for your project to offer customer support. Service Desk is available in [GitLab Enterprise Edition Premium](https://about.gitlab.com/gitlab-ee/). +Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) for your project to offer customer support. Service Desk is available in [GitLab Premium](https://about.gitlab.com/products/). ### Export project diff --git a/doc/user/project/wiki/img/wiki_move_page_1.png b/doc/user/project/wiki/img/wiki_move_page_1.png Binary files differnew file mode 100644 index 00000000000..0331c9d3a5c --- /dev/null +++ b/doc/user/project/wiki/img/wiki_move_page_1.png diff --git a/doc/user/project/wiki/img/wiki_move_page_2.png b/doc/user/project/wiki/img/wiki_move_page_2.png Binary files differnew file mode 100644 index 00000000000..a8e0c055051 --- /dev/null +++ b/doc/user/project/wiki/img/wiki_move_page_2.png diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index c0b8a87f038..d084ee41d8a 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -64,6 +64,18 @@ effect. You can find the **Delete** button only when editing a page. Click on it and confirm you want the page to be deleted. +## Moving a wiki page + +You can move a wiki page from one directory to another by specifying the full +path in the wiki page title in the [edit](#editing-a-wiki-page) form. + + + + + +In order to move a wiki page to the root directory, the wiki page title must +be preceded by the slash (`/`) character. + ## Viewing a list of all created wiki pages Every wiki has a sidebar from which a short list of the created pages can be diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md index d768b73286d..cac3cb599dd 100644 --- a/doc/workflow/lfs/lfs_administration.md +++ b/doc/workflow/lfs/lfs_administration.md @@ -5,6 +5,7 @@ Documentation on how to use Git LFS are under [Managing large binary files with ## Requirements * Git LFS is supported in GitLab starting with version 8.2. +* Support for object storage, such as AWS S3, was introduced in 10.0. * Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up. ## Configuration @@ -12,16 +13,18 @@ Documentation on how to use Git LFS are under [Managing large binary files with Git LFS objects can be large in size. By default, they are stored on the server GitLab is installed on. -There are two configuration options to help GitLab server administrators: +There are various configuration options to help GitLab server administrators: * Enabling/disabling Git LFS support * Changing the location of LFS object storage +* Setting up AWS S3 compatible object storage ### Omnibus packages In `/etc/gitlab/gitlab.rb`: ```ruby +# Change to true to enable lfs gitlab_rails['lfs_enabled'] = false # Optionally, change the storage path location. Defaults to @@ -35,11 +38,115 @@ gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" In `config/gitlab.yml`: ```yaml +# Change to true to enable lfs lfs: enabled: false storage_path: /mnt/storage/lfs-objects ``` +## Setting up S3 compatible object storage + +> **Note:** [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. +> Available in [GitLab CE][ce] 10.7 + +It is possible to store LFS objects on remote object storage instead of on a local disk. + +This allows you to offload storage to an external AWS S3 compatible service, freeing up disk space locally. You can also host your own S3 compatible storage decoupled from GitLab, with with a service such as [Minio](https://www.minio.io/). + +Object storage currently transfers files first to GitLab, and then on the object storage in a second stage. This can be done either by using a rake task to transfer existing objects, or in a background job after each file is received. + +### Object Storage Settings + +For source installations the following settings are nested under `lfs:` and then `object_store:`. On omnibus installs they are prefixed by `lfs_object_store_`. + +| Setting | Description | Default | +|---------|-------------|---------| +| `enabled` | Enable/disable object storage | `false` | +| `remote_directory` | The bucket name where LFS objects will be stored| | +| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | +| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `connection` | Various connection options described below | | + +#### S3 compatible connection settings + +The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: + +| Setting | Description | Default | +|---------|-------------|---------| +| `provider` | Always `AWS` for compatible hosts | AWS | +| `aws_access_key_id` | AWS credentials, or compatible | | +| `aws_secret_access_key` | AWS credentials, or compatible | | +| `region` | AWS region | us-east-1 | +| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | +| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | +| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false | + + +### From source + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + lfs: + enabled: true + object_store: + enabled: false + remote_directory: lfs-objects # Bucket name + connection: + provider: AWS + aws_access_key_id: 1ABCD2EFGHI34JKLM567N + aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE + region: eu-central-1 + # Use the following options to configure an AWS compatible host such as Minio + host: 'localhost' + endpoint: 'http://127.0.0.1:9000' + path_style: true + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. +1. Migrate any existing local LFS objects to the object storage: + + ```bash + sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production + ``` + + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless + `gitlab_rails['lfs_object_store_background_upload']` is set to false. + +### In Omnibus + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['lfs_object_store_enabled'] = true + gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects" + gitlab_rails['lfs_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N', + 'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE', + # The below options configure an S3 compatible host instead of AWS + 'host' => 'localhost', + 'endpoint' => 'http://127.0.0.1:9000', + 'path_style' => true + } + ``` + +1. Save the file and [reconfigure GitLab]s for the changes to take effect. +1. Migrate any existing local LFS objects to the object storage: + + ```bash + gitlab-rake gitlab:lfs:migrate + ``` + + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless + `gitlab_rails['lfs_object_store_background_upload']` is set to false. + ## Storage statistics You can see the total storage used for LFS objects on groups and projects @@ -48,10 +155,13 @@ and [projects APIs](../../api/projects.md). ## Known limitations -* Currently, storing GitLab Git LFS objects on a non-local storage (like S3 buckets) - is not supported * Support for removing unreferenced LFS objects was added in 8.14 onwards. * LFS authentications via SSH was added with GitLab 8.12 * Only compatible with the GitLFS client versions 1.1.0 and up, or 1.0.2. * The storage statistics currently count each LFS object multiple times for every project linking to it + +[reconfigure gitlab]: ../../administration/restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: ../../administration/restart_gitlab.md#installations-from-source "How to restart GitLab" +[eep]: https://about.gitlab.com/products/ "GitLab Premium" +[ee-2760]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2760 diff --git a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md index ce7895780c3..377eee69c11 100644 --- a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -34,7 +34,7 @@ Documentation for GitLab instance administrators is under [LFS administration do credentials store is recommended * Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have to add the URL to Git config manually (see [troubleshooting](#troubleshooting)) - + >**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication still goes over HTTP, but now the SSH client passes the correct credentials to the Git LFS client, so no action is required by the user. @@ -83,6 +83,74 @@ that are on the remote repository, eg. from branch `master`: git lfs fetch master ``` +## File Locking + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/35856) in GitLab 10.5. + +The first thing to do before using File Locking is to tell Git LFS which +kind of files are lockable. The following command will store PNG files +in LFS and flag them as lockable: + +```bash +git lfs track "*.png" --lockable +``` + +After executing the above command a file named `.gitattributes` will be +created or updated with the following content: + +```bash +*.png filter=lfs diff=lfs merge=lfs -text lockable +``` + +You can also register a file type as lockable without using LFS +(In order to be able to lock/unlock a file you need a remote server that implements the LFS File Locking API), +in order to do that you can edit the `.gitattributes` file manually: + +```bash +*.pdf lockable +``` + +After a file type has been registered as lockable, Git LFS will make +them readonly on the file system automatically. This means you will +need to lock the file before editing it. + +### Managing Locked Files + +Once you're ready to edit your file you need to lock it first: + +```bash +git lfs lock images/banner.png +Locked images/banner.png +``` + +This will register the file as locked in your name on the server: + +```bash +git lfs locks +images/banner.png joe ID:123 +``` + +Once you have pushed your changes, you can unlock the file so others can +also edit it: + +```bash +git lfs unlock images/banner.png +``` + +You can also unlock by id: + +```bash +git lfs unlock --id=123 +``` + +If for some reason you need to unlock a file that was not locked by you, +you can use the `--force` flag as long as you have a `master` access on +the project: + +```bash +git lfs unlock --id=123 --force +``` + ## Troubleshooting ### error: Repository or object not found diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index 37265a5b771..c4095ee0f69 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -67,7 +67,7 @@ Below is the table of events users can be notified of: ### Issue / Merge request events -In all of the below cases, the notification will be sent to: +In most of the below cases, the notification will be sent to: - Participants: - the author and assignee of the issue/merge request - authors of comments on the issue/merge request @@ -87,6 +87,7 @@ In all of the below cases, the notification will be sent to: | Reassign issue | The above, plus the old assignee | | Reopen issue | | | New merge request | | +| Push to merge request | Participants and Custom notification level with this event selected | | Reassign merge request | The above, plus the old assignee | | Close merge request | | | Reopen merge request | | diff --git a/doc/workflow/todos.md b/doc/workflow/todos.md index 3d8d3ce8f13..e612646cfbc 100644 --- a/doc/workflow/todos.md +++ b/doc/workflow/todos.md @@ -28,11 +28,10 @@ A Todo appears in your Todos dashboard when: - an issue or merge request is assigned to you, - you are `@mentioned` in an issue or merge request, be it the description of the issue/merge request or in a comment, +- you are `@mentioned` in a comment on a commit, - a job in the CI pipeline running for your merge request failed, but this job is not allowed to fail. ->**Note:** Commenting on a commit will _not_ trigger a Todo. - ### Directly addressed Todos > [Introduced][ce-7926] in GitLab 9.0. |
