Add latest changes from gitlab-org/gitlab@master

8 files changed, 169 insertions, 16 deletions

diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index 3b32baf28b9..d0c78031055 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -564,6 +564,109 @@ concurrency limiter, not a rate limiter. If a client makes 1000 requests
 in a row in a very short timespan, the concurrency will not exceed 1,
 and this mechanism (the concurrency limiter) will do nothing.
 
+## Rotating a Gitaly authentication token
+
+Rotating credentials in a production environment often either requires
+downtime, or causes outages, or both. If you are careful, though, you
+*can* rotate Gitaly credentials without a service interruption.
+
+This procedure also works if you are running GitLab on a single server.
+In that case, "Gitaly servers" and "Gitaly clients" refers to the same
+machine.
+
+### 1. Monitor current authentication behavior
+
+Use prometheus to see what the current authentication behavior of your
+GitLab installation is.
+
+```
+sum(rate(gitaly_authentications_total[5m])) by (enforced, status)
+```
+
+In a system where authentication is configured correctly, and where you
+have live traffic, you will see something like this:
+
+```
+{enforced="true",status="ok"}  4424.985419441742
+```
+
+There may also be other numbers with rate 0. We only care about the
+non-zero numbers.
+
+The only non-zero number should have `enforced="true",status="ok"`. If
+you have other non-zero numbers, something is wrong in your
+configuration.
+
+The 'status="ok"' number reflects your current request rate. In the example
+above, Gitaly is handling about 4000 requests per second.
+
+Now you have established that you can monitor the Gitaly authentication
+behavior of your GitLab installation.
+
+### 2. Reconfigure all Gitaly servers to be in "auth transitioning" mode
+
+The second step is to temporarily disable authentication on the Gitaly servers.
+
+```ruby
+# in /etc/gitlab/gitlab.rb
+gitaly['auth_transitioning'] = true
+```
+
+After you have applied this, your prometheus query should return
+something like this:
+
+```
+{enforced="false",status="would be ok"}  4424.985419441742
+```
+
+Because `enforced="false"`, it will be safe to start rolling out the new
+token.
+
+### 3. Update Gitaly token on all clients and servers
+
+```ruby
+# in /etc/gitlab/gitlab.rb
+
+gitaly['auth_token'] = 'my new secret token'
+```
+
+Remember to apply this on both your Gitaly clients *and* servers. If you
+check your prometheus query while this change is being rolled out, you
+will see non-zero values for the `enforced="false",status="denied"` counter.
+
+### 4. Use prometheus to ensure there are no authentication failures
+
+After you applied the Gitaly token change everywhere, and all services
+involved have been restarted, you should will temporarily see a mix of
+`status="would be ok"` and `status="denied"`.
+
+After the new token has been picked up by all Gitaly clients and
+servers, the **only non-zero rate** should be
+`enforced="false",status="would be ok"`.
+
+### 5. Disable "auth transitioning" Mode
+
+Now we turn off the 'auth transitioning' mode. These final steps are
+important: without them, you have **no authentication**.
+
+Update the configuration on your Gitaly servers:
+
+```ruby
+# in /etc/gitlab/gitlab.rb
+gitaly['auth_transitioning'] = false
+```
+
+### 6. Verify that authentication is enforced again
+
+Refresh your prometheus query. You should now see the same kind of
+result as you did in the beginning:
+
+```
+{enforced="true",status="ok"}  4424.985419441742
+```
+
+Note that `enforced="true"`, meaning that authentication is being enforced.
+
 ## Troubleshooting Gitaly
 
 ### `gitaly-debug`
diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
index a064dfbfbe2..5a414cc7e32 100644
--- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
+++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md
@@ -715,13 +715,15 @@ Namespace.find_by_full_path("user/proj").namespace_statistics.update(shared_runn
 project = Project.find_by_full_path('')
 builds_with_artifacts =  project.builds.with_artifacts_archive
 
-# Prior to 10.6 the above line would be:
-# builds_with_artifacts = project.builds.with_artifacts
-
 # Instance-wide:
-builds_with_artifacts = Ci::Build.with_artifacts
+builds_with_artifacts = Ci::Build.with_artifacts_archive
+
+# Prior to 10.6 the above lines would be:
+# builds_with_artifacts =  project.builds.with_artifacts
+# builds_with_artifacts = Ci::Build.with_artifacts
 
 ### CLEAR THEM OUT
+# Note that this will also erase artifacts that developers marked to "Keep"
 builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago)
 builds_to_clear.each do |build|
   build.artifacts_expire_at = Time.now
diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md
index 8ae38db5c96..4ca6547747b 100644
--- a/doc/ci/multi_project_pipelines.md
+++ b/doc/ci/multi_project_pipelines.md
@@ -116,7 +116,8 @@ staging:
 ```
 
 Use a `project` keyword to specify full path to a downstream project. Use
-a `branch` keyword to specify a branch name.
+a `branch` keyword to specify a branch name. Variable expansion is supported in
+the `branch` property.
 
 GitLab will use a commit that is currently on the HEAD of the branch when
 creating a downstream pipeline.
diff --git a/doc/development/img/memory_ruby_heap_fragmentation.png b/doc/development/img/memory_ruby_heap_fragmentation.png
new file mode 100644
index 00000000000..6567abe58bb
--- /dev/null
+++ b/doc/development/img/memory_ruby_heap_fragmentation.png
diff --git a/doc/development/performance.md b/doc/development/performance.md
index 39fcc8ff806..786b590ec70 100644
--- a/doc/development/performance.md
+++ b/doc/development/performance.md
@@ -251,6 +251,36 @@ These results can also be placed into a PostgreSQL database by setting the
 `RSPEC_PROFILING_POSTGRES_URL` variable. This is used to profile the test suite
 when running in the CI environment.
 
+## Memory profiling
+
+One of the reasons of the increased memory footprint could be Ruby memory fragmentation.
+
+To diagnose it, you can visualize Ruby heap as described in [this post by Aaron Patterson](https://tenderlovemaking.com/2017/09/27/visualizing-your-ruby-heap.html).
+
+To start, you want to dump the heap of the process you are investigating to a JSON file.  
+
+You need to run the command inside the process you are exploring, you may do that with `rbtrace`.  
+`rbtrace` is already present in GitLab `Gemfile`, you just need to require it.  
+It could be achieved running webserver or Sidekiq with the environment variable set to `ENABLE_RBTRACE=1`.
+
+To get the heap dump:
+
+```ruby
+bundle exec rbtrace -p <PID> -e 'File.open("heap.json", "wb") { |t| ObjectSpace.dump_all(output: t) }'
+```
+
+Having the JSON, you finally could render a picture using the script [provided by Aaron](https://gist.github.com/tenderlove/f28373d56fdd03d8b514af7191611b88) or similar:
+
+```sh
+ruby heapviz.rb heap.json
+```
+  
+Fragmented Ruby heap snapshot could look like this:
+
+![Ruby heap fragmentation](img/memory_ruby_heap_fragmentation.png)
+
+Memory fragmentation could be reduced by tuning GC parameters as described in [this post by Nate Berkopec](https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html), which should be considered as a tradeoff, as it may affect overall performance of memory allocation and GC cycles.
+
 ## Importance of Changes
 
 When working on performance improvements, it's important to always ask yourself
diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md
index a144bf70a86..20604cce9c6 100644
--- a/doc/development/rake_tasks.md
+++ b/doc/development/rake_tasks.md
@@ -143,12 +143,6 @@ This will compile and minify all JavaScript and CSS assets and copy them along
 with all other frontend assets (images, fonts, etc) into `/public/assets` where
 they can be easily inspected.
 
-## Generate API documentation for project services (e.g. Slack)
-
-```
-bundle exec rake services:doc
-```
-
 ## Updating Emoji Aliases
 
 To update the Emoji aliases file (used for Emoji autocomplete) you must run the
diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md
index 0618c14a3d1..2972421501d 100644
--- a/doc/user/application_security/sast/index.md
+++ b/doc/user/application_security/sast/index.md
@@ -45,12 +45,15 @@ The results are sorted by the priority of the vulnerability:
 
 ## Requirements
 
-To run a SAST job, you need GitLab Runner with the
+To run a SAST job, by default, you need GitLab Runner with the
 [`docker`](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode) or
 [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners)
 executor running in privileged mode. If you're using the shared Runners on GitLab.com,
 this is enabled by default.
 
+Privileged mode is not necessary if you've [disabled Docker in Docker
+for SAST](#disabling-docker-in-docker-for-sast)
+
 CAUTION: **Caution:**
 If you use your own Runners, make sure that the Docker version you have installed
 is **not** `19.03.00`. See [troubleshooting information](#error-response-from-daemon-error-processing-tar-file-docker-tar-relocation-error) for details.
@@ -144,6 +147,21 @@ under your project's settings:
 | ---- | --- | ----- |
 | Variable | `MAVEN_CLI_OPTS` | `-Drepository.password=verysecret -Drepository.user=myuser` |
 
+### Disabling Docker in Docker for SAST
+
+You can avoid the need for Docker in Docker by running the individual analyzers.
+This does not require running the executor in privileged mode. For example:
+
+```yaml
+include:
+  template: SAST.gitlab-ci.yml
+
+variables:
+  SAST_DISABLE_DIND: "true"
+```
+
+This will create individual `<analyzer-name>-sast` jobs for each analyzer that runs in your CI/CD pipeline.
+
 ### Overriding the SAST template
 
 If you want to override the job definition (for example, change properties like
@@ -173,6 +191,7 @@ The following are Docker image-related variables.
 | `SAST_ANALYZER_IMAGE_PREFIX`  | Override the name of the Docker registry providing the default images (proxy). Read more about [customizing analyzers](analyzers.md). |
 | `SAST_ANALYZER_IMAGE_TAG`     | Override the Docker tag of the default images. Read more about [customizing analyzers](analyzers.md).                                 |
 | `SAST_DEFAULT_ANALYZERS`      | Override the names of default images. Read more about [customizing analyzers](analyzers.md).                                          |
+| `SAST_DISABLE_DIND`           | Disable Docker in Docker and run analyzers [individually](#disabling-docker-in-docker-for-sast).                                      |
 | `SAST_PULL_ANALYZER_IMAGES`   | Pull the images from the Docker registry (set to 0 to disable). Read more about [customizing analyzers](analyzers.md).                 |
 
 ### Vulnerability filters
diff --git a/doc/user/project/integrations/generic_alerts.md b/doc/user/project/integrations/generic_alerts.md
index 46f05f235df..ec43696fdee 100644
--- a/doc/user/project/integrations/generic_alerts.md
+++ b/doc/user/project/integrations/generic_alerts.md
@@ -13,11 +13,11 @@ authored by the GitLab Alert Bot.
 
 ## Setting up generic alerts
 
-To set up the generic alerts integration:  
+To set up the generic alerts integration:
 
-1. Navigate to **Settings > Integrations** in a project.  
+1. Navigate to **Settings > Integrations** in a project.
 1. Click on **Alert endpoint**.
-1. Toggle the **Active**  alert setting. The `URL` and `Authorization Key` for the webhook configuration can be found there.  
+1. Toggle the **Active**  alert setting. The `URL` and `Authorization Key` for the webhook configuration can be found there.
 
 ## Customizing the payload
 
@@ -35,7 +35,11 @@ You can customize the payload by sending the following parameters. All fields ar
 Example request:
 
 ```sh
-curl --request POST --data '{"title": "Incident title"}' --header "Authorization: Bearer <autorization_key>" <url>
+curl --request POST \
+  --data '{"title": "Incident title"}' \
+  --header "Authorization: Bearer <autorization_key>" \
+  --header "Content-Type: application/json" \
+  <url>
 ```
 
 The `<autorization_key>` and `<url>` values can be found when [setting up generic alerts](#setting-up-generic-alerts).