diff options
Diffstat (limited to 'doc')
19 files changed, 531 insertions, 130 deletions
diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 550b3b07a95..39174780e24 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -10,10 +10,23 @@ all you need to do is update GitLab itself: 1. Log into each node (**primary** and **secondary** nodes). 1. [Update GitLab][update]. -1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when - the tracking database is enabled. 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. +### Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo raketask on all nodes, everything should be green: + + ```sh + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** node's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** node and see if it + is received by **secondary** nodes. + ## Upgrading to GitLab 12.1 By default, GitLab 12.1 will attempt to automatically upgrade the embedded PostgreSQL server to 10.7 from 9.6. Please see [the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) for the recommended procedure. @@ -419,22 +432,7 @@ is prepended with the relevant node for better clarity: sudo gitlab-ctl start ``` -## Check status after updating - -Now that the update process is complete, you may want to check whether -everything is working correctly: - -1. Run the Geo raketask on all nodes, everything should be green: - - ```sh - sudo gitlab-rake gitlab:geo:check - ``` - -1. Check the **primary** node's Geo dashboard for any errors. -1. Test the data replication by pushing code to the **primary** node and see if it - is received by **secondary** nodes. - -## Update tracking database on **secondary** node +### Update tracking database on **secondary** node After updating a **secondary** node, you might need to run migrations on the tracking database. The tracking database was added in GitLab 9.1, diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index bb824c9259b..56665ba8b9a 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -172,6 +172,8 @@ environment that supports about 10,000 users. The specifications below are a representation of the work so far. The specifications may be adjusted in the future based on additional testing and iteration. +NOTE: **Note:** The specifications here were performance tested against a specific coded workload. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + - 3 PostgreSQL - 4 CPU, 16GiB memory per node - 1 PgBouncer - 2 CPU, 4GiB memory - 2 Redis - 2 CPU, 8GiB memory per node diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index ea69378b249..e787af798bc 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -71,10 +71,10 @@ sudo service sshd reload Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo. -> **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in +NOTE: **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in GitLab 11.11 and later. -> **Warning:** Do not disable writes until SSH is confirmed to be working +CAUTION: **Caution:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. In the case of lookup failures (which are common), the `authorized_keys` diff --git a/doc/api/services.md b/doc/api/services.md index df15e6892b0..45b49d7eb92 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -972,22 +972,28 @@ Parameters: | `channel` | string | false | Default channel to use if others are not configured | | `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | | `notify_only_default_branch` | boolean | false | Send notifications only for the default branch | -| `push_events` | boolean | false | Enable notifications for push events | -| `issues_events` | boolean | false | Enable notifications for issue events | +| `commit_events` | boolean | false | Enable notifications for commit events | +| `confidential_issue_channel` | string | false | The name of the channel to receive confidential issues events notifications | | `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `confidential_note_channel` | string | false | The name of the channel to receive confidential note events notifications | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | +| `deployment_channel` | string | false | The name of the channel to receive deployment events notifications | +| `deployment_events` | boolean | false | Enable notifications for deployment events | +| `issue_channel` | string | false | The name of the channel to receive issues events notifications | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `job_events` | boolean | false | Enable notifications for job events | +| `merge_request_channel` | string | false | The name of the channel to receive merge request events notifications | | `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_channel` | string | false | The name of the channel to receive note events notifications | | `note_events` | boolean | false | Enable notifications for note events | +| `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | | `pipeline_events` | boolean | false | Enable notifications for pipeline events | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | | `push_channel` | string | false | The name of the channel to receive push events notifications | -| `issue_channel` | string | false | The name of the channel to receive issues events notifications | -| `confidential_issue_channel` | string | false | The name of the channel to receive confidential issues events notifications | -| `merge_request_channel` | string | false | The name of the channel to receive merge request events notifications | -| `note_channel` | string | false | The name of the channel to receive note events notifications | +| `push_events` | boolean | false | Enable notifications for push events | | `tag_push_channel` | string | false | The name of the channel to receive tag push events notifications | -| `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | | `wiki_page_channel` | string | false | The name of the channel to receive wiki page events notifications | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | ### Delete Slack service diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index c48817a5e30..c63b1e104ed 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -371,8 +371,8 @@ variables take precedence over those defined in `.gitlab-ci.yml`. There are cases where some variables cannot be used in the context of a `.gitlab-ci.yml` definition (for example under `script`). Read more about which variables are [not supported](where_variables_can_be_used.md). - -## Where variables can be used + +## Where variables can be used Click [here](where_variables_can_be_used.md) for a section that describes where and how the different types of variables can be used. @@ -484,81 +484,86 @@ Below you can find supported syntax reference: 1. Equality matching using a string - > Example: `$VARIABLE == "some value"` + Examples: - > Example: `$VARIABLE != "some value"` (introduced in GitLab 11.11) + - `$VARIABLE == "some value"` + - `$VARIABLE != "some value"` (introduced in GitLab 11.11) - You can use equality operator `==` or `!=` to compare a variable content to a - string. We support both, double quotes and single quotes to define a string - value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` - are supported. `"some value" == $VARIABLE` is correct too. + You can use equality operator `==` or `!=` to compare a variable content to a + string. We support both, double quotes and single quotes to define a string + value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` + are supported. `"some value" == $VARIABLE` is correct too. 1. Checking for an undefined value - > Example: `$VARIABLE == null` + Examples: - > Example: `$VARIABLE != null` (introduced in GitLab 11.11) + - `$VARIABLE == null` + - `$VARIABLE != null` (introduced in GitLab 11.11) - It sometimes happens that you want to check whether a variable is defined - or not. To do that, you can compare a variable to `null` keyword, like - `$VARIABLE == null`. This expression is going to evaluate to truth if - variable is not defined when `==` is used, or to falsey if `!=` is used. + It sometimes happens that you want to check whether a variable is defined + or not. To do that, you can compare a variable to `null` keyword, like + `$VARIABLE == null`. This expression is going to evaluate to truth if + variable is not defined when `==` is used, or to falsey if `!=` is used. 1. Checking for an empty variable - > Example: `$VARIABLE == ""` - - > Example: `$VARIABLE != ""` (introduced in GitLab 11.11) + Examples: + + - `$VARIABLE == ""` + - `$VARIABLE != ""` (introduced in GitLab 11.11) - If you want to check whether a variable is defined, but is empty, you can - simply compare it against an empty string, like `$VAR == ''` or non-empty - string `$VARIABLE != ""`. + If you want to check whether a variable is defined, but is empty, you can + simply compare it against an empty string, like `$VAR == ''` or non-empty + string `$VARIABLE != ""`. 1. Comparing two variables - > Example: `$VARIABLE_1 == $VARIABLE_2` + Examples: - > Example: `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) + - `$VARIABLE_1 == $VARIABLE_2` + - `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) - It is possible to compare two variables. This is going to compare values - of these variables. + It is possible to compare two variables. This is going to compare values + of these variables. 1. Variable presence check - > Example: `$STAGING` + Example: `$STAGING` - If you only want to create a job when there is some variable present, - which means that it is defined and non-empty, you can simply use - variable name as an expression, like `$STAGING`. If `$STAGING` variable - is defined, and is non empty, expression will evaluate to truth. - `$STAGING` value needs to a string, with length higher than zero. - Variable that contains only whitespace characters is not an empty variable. + If you only want to create a job when there is some variable present, + which means that it is defined and non-empty, you can simply use + variable name as an expression, like `$STAGING`. If `$STAGING` variable + is defined, and is non empty, expression will evaluate to truth. + `$STAGING` value needs to a string, with length higher than zero. + Variable that contains only whitespace characters is not an empty variable. 1. Pattern matching (introduced in GitLab 11.0) - > Example: `$VARIABLE =~ /^content.*/` + Examples: - > Example: `$VARIABLE_1 !~ /^content.*/` (introduced in GitLab 11.11) + - `$VARIABLE =~ /^content.*/` + - `$VARIABLE_1 !~ /^content.*/` (introduced in GitLab 11.11) - It is possible perform pattern matching against a variable and regular - expression. Expression like this evaluates to truth if matches are found - when using `=~`. It evaluates to truth if matches are not found when `!~` is used. + It is possible perform pattern matching against a variable and regular + expression. Expression like this evaluates to truth if matches are found + when using `=~`. It evaluates to truth if matches are not found when `!~` is used. - Pattern matching is case-sensitive by default. Use `i` flag modifier, like - `/pattern/i` to make a pattern case-insensitive. + Pattern matching is case-sensitive by default. Use `i` flag modifier, like + `/pattern/i` to make a pattern case-insensitive. 1. Conjunction / Disjunction ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27925) in GitLab 12.0) - > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` - - > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` + Examples: - > Example: `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` + - `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` + - `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` + - `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` - It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise - supported syntax may be used in a conjunctive or disjunctive statement. - Precedence of operators follows standard Ruby 2.5 operation - [precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html). + It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise + supported syntax may be used in a conjunctive or disjunctive statement. + Precedence of operators follows standard Ruby 2.5 operation + [precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html). ## Debug tracing diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 79c701d7abf..39f12e6886e 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -92,9 +92,6 @@ The following team labels are **true** teams per our [organization structure](ht The descriptions on the [labels page](https://gitlab.com/gitlab-org/gitlab-ce/-/labels) explain what falls under the responsibility of each team. -Within those team labels, we also have the ~backend and ~frontend labels to -indicate if an issue needs backend work, frontend work, or both. - Team labels are always capitalized so that they show up as the first label for any issue. @@ -107,15 +104,6 @@ The current stage labels can be found by [searching the labels list for `devops: These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) and thus are mutually exclusive. -They differ from the [Team labels](#team-labels) because teams may work on -issues outside their stage. - -Normally there is a 1:1 relationship between Stage labels and Team labels, but -any issue can be picked up by any team, depending on current priorities. -So, an issue labeled ~"devops:create" may be scheduled by the ~Plan team, for -example. In such cases, it's usual to include both team labels so each team can -be aware of the progress. - The Stage labels are used to generate the [direction pages][direction-pages] automatically. [devops-stages]: https://about.gitlab.com/direction/#devops-stages @@ -130,9 +118,16 @@ The current group labels can be found by [searching the labels list for `group:: These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) and thus are mutually exclusive. -Groups are nested beneath a particular stage, so only one stage label and one group label -can be applied to a single issue. You can find the groups listed in the -[Product Categories pages][product-categories]. +You can find the groups listed in the [Product Stages, Groups, and Categories][product-categories] page. + +We use the term group to map down product requirements from our product stages. +As a team needs some way to collect the work their members are planning to be assigned to, we use the `~group::` labels to do so. + +Normally there is a 1:1 relationship between Stage labels and Group labels. In the spirit of "Everyone can contribute", +any issue can be picked up by any group, depending on current priorities. For example, an issue labeled ~"devops::create" may be picked up by the ~"group::access" group. + +We also use stage and group labels to help quantify our [throughput](https://about.gitlab.com/handbook/engineering/management/throughput). +Please read [Stage and Group labels in Throughtput](https://about.gitlab.com/handbook/engineering/management/throughput/#stage-and-group-labels-in-throughput) for more information on how the labels are used in this context. [structure-groups]: https://about.gitlab.com/company/team/structure/#groups [product-categories]: https://about.gitlab.com/handbook/product/categories/ diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md index 14a169dcc1d..e1df8be8b6f 100644 --- a/doc/development/testing_guide/end_to_end/quick_start_guide.md +++ b/doc/development/testing_guide/end_to_end/quick_start_guide.md @@ -110,7 +110,7 @@ end ``` > Notice that the test itself is simple. The most challenging part is the creation of the application state, which will be covered later. - +> > The exemplified test case's MVC is not enough for the change to be merged, but it helps to build up the test logic. The reason is that we do not want to use locators directly in the tests, and tests **must** use [Page Objects] before they can be merged. This way we better separate the responsibilities, where the Page Objects encapsulate elements and methods that allow us to interact with pages, while the spec files describe the test cases in more business-related language. Below are the steps that the test covers: @@ -211,7 +211,7 @@ A pre-condition for the entire test suite is defined in the `before :context` bl > For our test suite, due to the need of the tests being completely independent of each other, we won't use the `before :context` block. The `before :context` block would make the tests dependent on each other because the first test changes the label of the issue, and the second one depends on the `'animal::fox'` label being set. -> **Tip:** In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. +TIP: **Tip:** In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. #### `before` @@ -274,11 +274,11 @@ end In the `before` block we create all the application state needed for the tests to run. We do that by using the `Runtime::Browser.visit` method to go to the login page, by performing a `sign_in_using_credentials` from the `Login` Page Object, by fabricating resources via APIs (`issue`, and `Resource::Label`), and by using the `issue.visit!` to visit the issue page. > A project is created in the background by creating the `issue` resource. - +> > When creating the [Resources], notice that when calling the `fabricate_via_api` method, we pass some attribute:values, like `title`, and `labels` for the `issue` resource; and `project` and `title` for the `label` resource. - +> > What's important to understand here is that by creating the application state mostly using the public APIs we save a lot of time in the test suite setup stage. - +> > Soon we will cover the use of the already existing resources' methods and the creation of your own `fabricate_via_api` methods for resources where this is still not available, but first, let's optimize our implementation. ### 6. Optimization @@ -362,7 +362,7 @@ First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d358 Add the following `attribute :id` and `attribute :labels` right above the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). > This line is needed to allow for the issue fabrication, and for labels to be automatically added to the issue when fabricating it via API. - +> > We add the attributes above the existing attribute to keep them alphabetically organized. Then, let's initialize an instance variable for labels to allow an empty array as default value when such information is not passed during the resource fabrication, since this optional. [Between the attributes and the `fabricate!` method](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a1f1408728f19b2aa15887cd20bddab7e70c8bd/qa/qa/resource/issue.rb#L18), add the following: @@ -437,7 +437,7 @@ By defining the `resource_web_url(resource)` method, we override the one from th By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the publich API, we raise a `NotImplementedError` instead. -By defining the `api_post_path` method, we allow for the [`ApiFabricator `](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. +By defining the `api_post_path` method, we allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. @@ -580,7 +580,7 @@ filter_output = search_field_tag search_id, nil, class: "dropdown-input-field", > `data-qa-*` data attributes and CSS classes starting with `qa-` are used solely for the purpose of QA and testing. > By defining these, we add **testability** to the application. - +> > When defining a data attribute like: `qa_selector: 'labels_block'`, it should match the element definition: `element :labels_block`. We use a [sanity test](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective selectors in the specified views. #### Updates in the `QA::Page::Base` class @@ -599,8 +599,6 @@ This method receives an element (`name`) and the `keys` that it will send to tha As you might remember, in the Issue Page Object we call this method like this: `send_keys_to_element(:dropdown_input_field, [label, :enter])`. -___ - With that, you should be able to start writing end-to-end tests yourself. *Congratulations!* [Page Objects]: page_objects.md diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index 24edd05da2f..f0da1cc2ddc 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -140,7 +140,7 @@ done without requiring downtime. However, this does require that any application changes are deployed _first_. Thus, changing the constraints of a column should happen in a post-deployment migration. NOTE: Avoid using `change_column` as it produces inefficient query because it re-defines -the whole column type. For example, to add a NOT NULL constraint, prefer `change_column_null ` +the whole column type. For example, to add a NOT NULL constraint, prefer `change_column_null` ## Changing Column Types diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 365609ef757..ed5b23a122f 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -62,6 +62,8 @@ NOTE: **Note:** Since file system performance may affect GitLab's overall perfor ### CPU +This is the recommended minimum hardware for a handful of example GitLab user base sizes. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + - 1 core supports up to 100 users but the application can be a bit slower due to having all workers and background jobs running on the same core - **2 cores** is the **recommended** minimum number of cores and supports up to 100 users - 4 cores supports up to 500 users @@ -71,6 +73,8 @@ NOTE: **Note:** Since file system performance may affect GitLab's overall perfor ### Memory +This is the recommended minimum hardware for a handful of example GitLab user base sizes. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + You need at least 8GB of addressable memory (RAM + swap) to install and use GitLab! The operating system and any other running applications will also be using memory so keep in mind that you need at least 4GB available before running GitLab. With diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index cdcd8215b23..5b227ebebe0 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -48,6 +48,7 @@ The following are resources about version control concepts: The following resources may help you become more efficient at using Git: +- [Useful Git commands](useful_git_commands.md) collected by the GitLab support team. - [Git Tips & Tricks](https://about.gitlab.com/2016/12/08/git-tips-and-tricks/) - [Eight Tips to help you work better with Git](https://about.gitlab.com/2015/02/19/8-tips-to-help-you-work-better-with-git/) @@ -82,6 +83,8 @@ Git-related queries from GitLab. The following relate to Git Large File Storage: - [Getting Started with Git LFS](https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/) -- [GitLab Git LFS documentation](../../workflow/lfs/manage_large_binaries_with_git_lfs.md) +- [Migrate an existing Git repo with Git LFS](migrate_to_git_lfs/index.md) +- [GitLab Git LFS user documentation](../../workflow/lfs/manage_large_binaries_with_git_lfs.md) +- [GitLab Git LFS admin documentation](../../workflow/lfs/lfs_administration.md) - [Git-Annex to Git-LFS migration guide](../../workflow/lfs/migrate_from_git_annex_to_git_lfs.md) - [Towards a production quality open source Git LFS server](https://about.gitlab.com/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) diff --git a/doc/topics/git/migrate_to_git_lfs/index.md b/doc/topics/git/migrate_to_git_lfs/index.md new file mode 100644 index 00000000000..c879e404997 --- /dev/null +++ b/doc/topics/git/migrate_to_git_lfs/index.md @@ -0,0 +1,174 @@ +--- +type: tutorial, concepts +description: "How to migrate an existing Git repository to Git LFS with BFG." +last_updated: 2019-07-11 +--- + +# Migrate a Git repo into Git LFS with BFG + +Using Git LFS can help you to reduce the size of your Git +repository and improve its performance. + +However, simply adding the +large files that are already in your repository to Git LFS, +will not actually reduce the size of your repository because +the files are still referenced by previous commits. + +Through the method described on this document, first migrate +to Git LFS with [BFG](https://rtyley.github.io/bfg-repo-cleaner/) +through a mirror repo, then clean up the repository's history, +and lastly create LFS tracking rules to prevent new binary files +from being added. + +This tutorial was inspired by the guide +[Use BFG to migrate a repo to Git LFS](https://confluence.atlassian.com/bitbucket/use-bfg-to-migrate-a-repo-to-git-lfs-834233484.html). +For more information on Git LFS, see the [references](#references) +below. + +CAUTION: **Warning:** +The method described on this guide rewrites Git history. Make +sure to back up your repo before beginning and use it at your +own risk. + +## Requirements + +Before beginning, make sure: + +- You have enough LFS storage for the files you want to convert. + Storage is required for the entire history of all files. +- All the team members you share the repository with have pushed all changes. + Branches based on the repository before applying this method cannot be merged. + Branches based on the repo before applying this method cannot be merged. + +To follow this tutorial, you'll need: + +- Maintainer permissions to the existing Git repository + you'd like to migrate to LFS with access through the command line. +- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) + and [Java Runtime Environment](https://www.java.com/en/download/manual.jsp) + (Java 7 or above) installed locally. +- BFG installed locally: + + ```bash + brew install bfg + ``` + +- Git LFS installed locally: + + ```bash + brew install git-lfs + ``` + +NOTE: **Note:** +This guide was tested on macOS Mojave. + +## Steps + +Consider an example upstream project, `git@gitlab.com:gitlab-tests/test-git-lfs-repo-migration.git`. + +1. Back up your repository: + + Create a copy of your repository so that you can + recover it in case something goes wrong. + +1. Clone `--mirror` the repo: + + Cloning with the mirror flag will create a bare repository. + This ensures you get all the branches within the repo. + + It creates a directory called `<repo-name>.git` + (in our example, `test-git-lfs-repo-migration.git`), + mirroring the upstream project: + + ```bash + git clone --mirror git@gitlab.com:gitlab-tests/test-git-lfs-repo-migration.git + ``` + +1. Convert the Git history with BFG: + + ```bash + bfg --convert-to-git-lfs "*.{png,mp4,jpg,gif}" --no-blob-protection test-git-lfs-repo-migration.git + ``` + + It is scanning all the history, and looking for any files with + that extension, and then converting them to an LFS pointer. + +1. Clean up the repository: + + ```bash + # cd path/to/mirror/repo: + cd test-git-lfs-repo-migration.git + # clean up the repo: + git reflog expire --expire=now --all && git gc --prune=now --aggressive + ``` + + You can also take a look on how to further [clean the repo](../../../user/project/repository/reducing_the_repo_size_using_git.md), + but it's not necessary for the purposes of this guide. + +1. Install Git LFS in the mirror repository: + + ```bash + git lfs install + ``` + +1. [Unprotect the default branch](../../../user/project/protected_branches.md), + so that we can force-push the rewritten repository: + + 1. Navigate to your project's **Settings > Repository** and + expand **Protected Branches**. + 1. Scroll down to locate the protected branches and click + **Unprotect** the default branch. + +1. Force-push to GitLab: + + ```bash + git push --force + ``` + +1. Track the files you want with LFS: + + ```bash + # cd path/to/upstream/repo: + cd test-git-lfs-repo-migration + # You may need to reset your local copy with upstream's `master` after force-pushing from the mirror: + git reset --hard origin/master + # Track the files with LFS: + git lfs track "*.gif" "*.png" "*.jpg" "*.psd" "*.mp4" ".gitattributes" "img/" + ``` + + Now all existing the files you converted, as well as the new + ones you add, will be properly tracked with LFS. + +1. [Re-protect the default branch](../../../user/project/protected_branches.md): + + 1. Navigate to your project's **Settings > Repository** and + expand **Protected Branches**. + 1. Select the default branch from the **Branch** dropdown menu, + and set up the + **Allowed to push** and **Allowed to merge** rules. + 1. Click **Protect**. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> + +## References + +- [Getting Started with Git LFS](https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/) +- [Migrate from Git Annex to Git LFS](../../../workflow/lfs/migrate_from_git_annex_to_git_lfs.md) +- [GitLab's Git LFS user documentation](../../../workflow/lfs/manage_large_binaries_with_git_lfs.md) +- [GitLab's Git LFS administrator documentation](../../../workflow/lfs/lfs_administration.md) +- Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs) + +<!-- +Test project: +https://gitlab.com/gitlab-tests/test-git-lfs-repo-migration +--> diff --git a/doc/topics/git/useful_git_commands.md b/doc/topics/git/useful_git_commands.md new file mode 100644 index 00000000000..84406805350 --- /dev/null +++ b/doc/topics/git/useful_git_commands.md @@ -0,0 +1,210 @@ +--- +type: reference +--- + +# Useful Git commands + +Here are some useful Git commands collected by the GitLab support team. You may not +need to use often, but they can can come in handy when needed. + +## Remotes + +### Add another URL to a remote, so both remotes get updated on each push + +```sh +git remote set-url --add <remote_name> <remote_url> +``` + +## Staging and reverting changes + +### Remove last commit and leave the changes in unstaged + +```sh +git reset --soft HEAD^ +``` + +### Unstage a certain number of commits from HEAD + +To unstage 3 commits, for example, run: + +```sh +git reset HEAD^3 +``` + +### Unstage changes to a certain file from HEAD + +```sh +git reset <filename> +``` + +### Revert a file to HEAD state and remove changes + +There are two options to revert changes to a file: + +- `git checkout <filename>` +- `git reset --hard <filename>` + +### Undo a previous commit by creating a new replacement commit + +```sh +git revert <commit-sha> +``` + +### Create a new message for last commit + +```sh +git commit --amend +``` + +### Add a file to the last commit + +```sh +git add <filename> +git commit --amend +``` + +Append `--no-edit` to the `commit` command if you do not want to edit the commit +message. + +## Stashing + +### Stash changes + +```sh +git stash save +``` + +The default behavor of `stash` is to save, so you can also use just: + +```sh +git stash +``` + +### Unstash your changes + +```sh +git stash apply +``` + +### Discard your stashed changes + +```sh +git stash drop +``` + +### Apply and drop your stashed changes + +```sh +git stash pop +``` + +## Refs and Log + +### Use reflog to show the log of reference changes to HEAD + +```sh +git reflog +``` + +### Check the Git history of a file + +The basic command to check the git history of a file: + +```sh +git log <file> +``` + +If you get this error message: + +```text +fatal: ambiguous argument <file_name>: unknown revision or path not in the working tree. +Use '--' to separate paths from revisions, like this: +``` + +Use this to check the Git history of the file: + +```sh +git log -- <file> +``` + +### Find the tags that contain a particular SHA + +```sh +git tag --contains <sha> +``` + +### Check the content of each change to a file + +```sh +gitk <file> +``` + +### Check the content of each change to a file, follows it past file renames + +```sh +gitk --follow <file> +``` + +## Debugging + +### Use a custom SSH key for a git command + +```text +GIT_SSH_COMMAND="ssh -i ~/.ssh/gitlabadmin" git <command> +``` + +### Debug cloning + +With SSH: + +```text +GIT_SSH_COMMAND="ssh -vvv" git clone <git@url> +``` + +With HTTPS: + +```text +GIT_TRACE_PACKET=1 GIT_TRACE=2 GIT_CURL_VERBOSE=1 git clone <url> +``` + +## Rebasing + +### Rebase your branch onto master + +The -i flag stands for 'interactive': + +```sh +git rebase -i master +``` + +### Continue the rebase if paused + +```sh +git rebase --continue +``` + +### Use git rerere + +To _reuse_ recorded solutions to the same problems when repeated: + +```sh +git rerere +``` + +To enable `rerere` functionality: + +```sh +git config --global rerere.enabled true +``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index 59835aeba01..cb533538047 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -29,6 +29,7 @@ SAST supports the following official analyzers: - [Security Code Scan (.NET)](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) - [TSLint (Typescript)](https://gitlab.com/gitlab-org/security-products/analyzers/tslint) - [Sobelow (Elixir Phoenix)](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) +- [PMD (Apex only)](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex) The analyzers are published as Docker images that SAST will use to launch dedicated containers for each analysis. @@ -116,24 +117,24 @@ custom analyzer can scan the source code. ## Analyzers Data -| Property \ Tool | Bandit | Brakeman | ESLint security | Find Sec Bugs | Flawfinder | Go AST Scanner | NodeJsScan | Php CS Security Audit | Security code Scan (.NET) | TSLint Security | Sobelow | -| --------------------------------------- | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :---------------------: | :-------------------------: | :-------------: | :----------------: | -| Severity | ✓ | 𐄂 | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | -| Title | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | -| Description | 𐄂 | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | -| File | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | -| Start line | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | -| End line | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | 𐄂 | -| Start column | 𐄂 | 𐄂 | ✓ | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | -| End column | 𐄂 | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | 𐄂 | -| External id (e.g. CVE) | 𐄂 | ⚠ | 𐄂 | ⚠ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | -| URLs | 𐄂 | ✓ | 𐄂 | ⚠ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | -| Internal doc/explanation | ⚠ | ✓ | 𐄂 | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | -| Solution | 𐄂 | 𐄂 | 𐄂 | ⚠ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | -| Confidence | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | -| Affected item (e.g. class or package) | 𐄂 | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | -| Source code extract | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | -| Internal ID | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | ✓ | +| Property \ Tool | Apex | Bandit | Brakeman | ESLint security | Find Sec Bugs | Flawfinder | Go AST Scanner | NodeJsScan | Php CS Security Audit | Security code Scan (.NET) | TSLint Security | Sobelow | +| --------------------------------------- | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :---------------------: | :-------------------------: | :-------------: | :----------------: | +| Severity | ✓ | ✓ | 𐄂 | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | +| Title | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Description | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | +| File | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Start line | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| End line | ✓ | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | 𐄂 | +| Start column | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | +| End column | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | 𐄂 | +| External id (e.g. CVE) | 𐄂 | 𐄂 | ⚠ | 𐄂 | ⚠ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| URLs | ✓ | 𐄂 | ✓ | 𐄂 | ⚠ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| Internal doc/explanation | ✓ | ⚠ | ✓ | 𐄂 | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | +| Solution | ✓ | 𐄂 | 𐄂 | 𐄂 | ⚠ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| Confidence | 𐄂 | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | +| Affected item (e.g. class or package) | ✓ | 𐄂 | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| Source code extract | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| Internal ID | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | ✓ | - ✓ => we have that data - ⚠ => we have that data but it's partially reliable, or we need to extract it from unstructured content diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index aac881112ff..5149f628345 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -59,6 +59,7 @@ The following table shows which languages, package managers and frameworks are s |-----------------------------------------------------------------------------|----------------------------------------------------------------------------------------|------------------------------| | .NET | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | | Any | [Gitleaks](https://github.com/zricethezav/gitleaks) and [TruffleHog](https://github.com/dxa4481/truffleHog) | 11.9 | +| Apex (Salesforce) | [pmd](https://pmd.github.io/pmd/index.html) | 12.1 | | C/C++ | [Flawfinder](https://www.dwheeler.com/flawfinder/) | 10.7 | | Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.10 | | Go | [Gosec](https://github.com/securego/gosec) | 10.7 | diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index 7c24f6db86f..a29df76f4b7 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -253,6 +253,7 @@ The applications below can be uninstalled. | Application | GitLab version | Notes | | ----------- | -------------- | ----- | | GitLab Runner | 12.2+ | Any running pipelines will be canceled. | +| Helm | 12.2+ | The associated Tiller pod will be deleted and cannot be restored. | | Ingress | 12.1+ | The associated load balancer and IP will be deleted and cannot be restored. Furthermore, it can only be uninstalled if JupyterHub is not installed. | | JupyterHub | 12.1+ | All data not committed to GitLab will be deleted and cannot be restored. | | Knative | 12.1+ | The associated IP will be deleted and cannot be restored. | diff --git a/doc/user/group/bulk_editing/index.md b/doc/user/group/bulk_editing/index.md index 5b5f75c2dd9..c8715577eb2 100644 --- a/doc/user/group/bulk_editing/index.md +++ b/doc/user/group/bulk_editing/index.md @@ -5,22 +5,21 @@ > - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12719) for merge requests in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. -> NOTE: **Note:** -> -> - A permission level of `Reporter` or higher is required in order to manage issues. -> - A permission level of `Developer` or higher is required in order to manage merge requests. - Milestones can be updated simultaneously across multiple issues or merge requests by using the bulk editing feature.  +NOTE: **Note:** +A permission level of `Reporter` or higher is required in order to manage issues, and +a permission level of `Developer` or higher is required in order to manage merge requests. + To bulk update group issue or merge request milestones: 1. Navigate to the issues or merge requests list. 1. Click the **Edit issues** or **Edit merge requests** button. - - This will open a sidebar on the right-hand side of your screen where an editable field - for milestones will be displayed. - - Checkboxes will also appear beside each issue or merge request. + - This will open a sidebar on the right-hand side of your screen where an editable field + for milestones will be displayed. + - Checkboxes will also appear beside each issue or merge request. 1. Check the checkbox beside each issue to be edited. 1. Select the desired milestone from the sidebar. 1. Click **Update all**. diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md index 9c72fe33d0d..d7178506b64 100644 --- a/doc/user/project/issues/related_issues.md +++ b/doc/user/project/issues/related_issues.md @@ -19,7 +19,7 @@ Issues from a different project require additional information like the group and the project name. For example: - same project: `#44` -- same group: `project#44 ` +- same group: `project#44` - different group: `group/project#44` Valid references will be added to a temporary list that you can review. diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 54ecc42d2b9..6a9900d48f9 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -191,7 +191,7 @@ can use the following setup: 1. In Cloudflare, create a DNS `TXT` record to verify your domain. 1. In GitLab, verify your domain. 1. In Cloudflare, create a DNS `CNAME` record pointing `www` to `domain.com`. -1. In Cloudflare, add a Page Rule pointing `www.domain,com` to `domain.com`: +1. In Cloudflare, add a Page Rule pointing `www.domain.com` to `domain.com`: - Navigate to your domain's dashboard and click **Page Rules** on the top nav. - Click **Create Page Rule**. diff --git a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md index b6bba57049d..264372a512d 100644 --- a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -84,6 +84,10 @@ that are on the remote repository, eg. for a branch from origin: git lfs fetch origin master ``` +### Migrate an existing repo to Git LFS + +Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md). + ## File Locking > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/35856) in GitLab 10.5. |