Add latest changes from gitlab-org/gitlab@13-1-stable-ee

2 files changed, 295 insertions, 0 deletions

diff --git a/doc/topics/offline/index.md b/doc/topics/offline/index.md
new file mode 100644
index 00000000000..6d4c486d350
--- /dev/null
+++ b/doc/topics/offline/index.md
@@ -0,0 +1,138 @@
+# Offline GitLab
+
+Computers in an offline environment are isolated from the public internet as a security measure. This
+page lists all the information available for running GitLab in an offline environment.
+
+## Quick start
+
+If you plan to deploy a GitLab instance on a physically-isolated and offline network, see the
+[quick start guide](quick_start_guide.md) for configuration steps.
+
+## Features
+
+Follow these best practices to use GitLab's features in an offline environment:
+
+- [Operating the GitLab Secure scanners in an offline environment](../../user/application_security/offline_deployments/index.md).
+
+## Loading Docker images onto your offline host
+
+To use many GitLab features, including
+[security scans](../../user/application_security/index.md#working-in-an-offline-environment)
+and [Auto DevOps](../autodevops/), the GitLab Runner must be able to fetch the
+relevant Docker images.
+
+The process for making these images available without direct access to the public internet
+involves downloading the images then packaging and transferring them to the offline host. Here's an
+example of such a transfer:
+
+1. Download Docker images from public internet.
+1. Package Docker images as tar archives.
+1. Transfer images to offline environment.
+1. Load transferred images into offline Docker registry.
+
+### Using the official GitLab template
+
+GitLab provides a [vendored template](../../ci/yaml/README.md#includetemplate)
+to ease this process.
+
+This template should be used in a new, empty project, with a `gitlab-ci.yml` file containing:
+
+```yaml
+include:
+  - template: Secure-Binaries.gitlab-ci.yml
+```
+
+The pipeline downloads the Docker images needed for the Security Scanners and saves them as
+[job artifacts](../../ci/pipelines/job_artifacts.md) or pushes them to the [Container Registry](../../user/packages/container_registry/index.md)
+of the project where the pipeline is executed. These archives can be transferred to another location
+and [loaded](https://docs.docker.com/engine/reference/commandline/load/) in a Docker daemon.
+This method requires a GitLab Runner with access to both `gitlab.com` (including
+`registry.gitlab.com`) and the local offline instance. This runner must run in
+[privileged mode](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode)
+to be able to use the `docker` command inside the jobs. This runner can be installed in a DMZ or on
+a bastion, and used only for this specific project.
+
+#### Scheduling the updates
+
+By default, this project's pipeline will run only once, when the `.gitlab-ci.yml` is added to the
+repo. To update the GitLab security scanners and signatures, it's necessary to run this pipeline
+regularly. GitLab provides a way to [schedule pipelines](../../ci/pipelines/schedules.md). For
+example, you can set this up to download and store the Docker images every week.
+
+Some images can be updated more frequently than others. For example, the [vulnerability database](https://hub.docker.com/r/arminc/clair-db/tags)
+for Container Scanning is updated daily. To update this single image, create a new Scheduled
+Pipeline that runs daily and set `SECURE_BINARIES_ANALYZERS` to `clair-vulnerabilities-db`. Only
+this job will be triggered, and the image will be updated daily and made available in the project
+registry.
+
+#### Using the secure bundle created
+
+The project using the `Secure-Binaries.gitlab-ci.yml` template should now host all the required
+images and resources needed to run GitLab Security features.
+
+Next, you must tell the offline instance to use these resources instead of the default ones on
+GitLab.com. To do so, set the environment variable `SECURE_ANALYZERS_PREFIX` with the URL of the
+project [container registry](../../user/packages/container_registry/index.md).
+
+You can set this variable in the projects' `.gitlab-ci.yml`, or
+in the GitLab UI at the project or group level. See the [GitLab CI/CD environment variables page](../../ci/variables/README.md#custom-environment-variables)
+for more information.
+
+#### Variables
+
+The following table shows which variables you can use with the `Secure-Binaries.gitlab-ci.yml`
+template:
+
+| VARIABLE                                  | Description                                   | Default value                     |
+|-------------------------------------------|-----------------------------------------------|-----------------------------------|
+| `SECURE_BINARIES_ANALYZERS`               | Comma-separated list of analyzers to download | `"bandit, brakeman, gosec, and so on..."` |
+| `SECURE_BINARIES_DOWNLOAD_IMAGES`         | Used to disable jobs                          | `"true"`                          |
+| `SECURE_BINARIES_PUSH_IMAGES`             | Push files to the project registry            | `"true"`                          |
+| `SECURE_BINARIES_SAVE_ARTIFACTS`          | Also save image archives as artifacts         | `"false"`                         |
+| `SECURE_BINARIES_ANALYZER_VERSION`        | Default analyzer version (Docker tag)         | `"2"`                             |
+
+### Alternate way without the official template
+
+If it's not possible to follow the above method, the images can be transferred manually instead:
+
+#### Example image packager script
+
+```shell
+#!/bin/bash
+set -ux
+
+# Specify needed analyzer images
+analyzers=${SAST_ANALYZERS:-"bandit eslint gosec"}
+gitlab=registry.gitlab.com/gitlab-org/security-products/analyzers/
+
+for i in "${analyzers[@]}"
+do
+  tarname="${i}_2.tar"
+  docker pull $gitlab$i:2
+  docker save $gitlab$i:2 -o ./analyzers/${tarname}
+  chmod +r ./analyzers/${tarname}
+done
+```
+
+#### Example image loader script
+
+This example loads the images from a bastion host to an offline host. In certain configurations,
+physical media may be needed for such a transfer:
+
+```shell
+#!/bin/bash
+set -ux
+
+# Specify needed analyzer images
+analyzers=${SAST_ANALYZERS:-"bandit eslint gosec"}
+registry=$GITLAB_HOST:4567
+
+for i in "${analyzers[@]}"
+do
+  tarname="${i}_2.tar"
+  scp ./analyzers/${tarname} ${GITLAB_HOST}:~/${tarname}
+  ssh $GITLAB_HOST "sudo docker load -i ${tarname}"
+  ssh $GITLAB_HOST "sudo docker tag $(sudo docker images | grep $i | awk '{print $3}') ${registry}/analyzers/${i}:2"
+  ssh $GITLAB_HOST "sudo docker push ${registry}/analyzers/${i}:2"
+done
+```
diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md
new file mode 100644
index 00000000000..0abdd08ffcf
--- /dev/null
+++ b/doc/topics/offline/quick_start_guide.md
@@ -0,0 +1,157 @@
+# Getting started with an offline GitLab Installation
+
+This is a step-by-step guide that helps you install, configure, and use a self-managed GitLab
+instance entirely offline.
+
+## Installation
+
+NOTE: **Note:**
+This guide assumes the server is Ubuntu 18.04. Instructions for other servers may vary.
+
+NOTE: **Note:**
+This guide assumes the server host resolves as `my-host`, which you should replace with your
+server's name.
+
+Follow the installation instructions [as outlined in the omnibus install
+guide](https://about.gitlab.com/install/#ubuntu), but make sure to specify an `http`
+URL for the `EXTERNAL_URL` installation step. Once installed, we will manually
+configure the SSL ourselves.
+
+It is strongly recommended to setup a domain for IP resolution rather than bind
+to the server's IP address. This better ensures a stable target for our certs' CN
+and will make long-term resolution simpler.
+
+```shell
+sudo EXTERNAL_URL="http://my-host.internal" install gitlab-ee
+```
+
+## Enabling SSL
+
+Follow these steps to enable SSL for your fresh instance. Note that these steps reflect those for
+[manually configuring SSL in Omnibus's NGINX configuration](https://docs.gitlab.com/omnibus/settings/nginx.html#manually-configuring-https):
+
+1. Make the following changes to `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   # Update external_url from "http" to "https"
+   external_url "https://example.gitlab.com"
+
+   # Set Let's Encrypt to false
+   letsencrypt['enable'] = false
+   ```
+
+1. Create the following directories with the appropriate permissions for generating self-signed
+   certificates:
+
+   ```shell
+   sudo mkdir -p /etc/gitlab/ssl
+   sudo chmod 755 /etc/gitlab/ssl
+   sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/gitlab/ssl/my-host.internal.key -out /etc/gitlab/ssl/my-host.internal.crt
+   ```
+
+1. Reconfigure your instance to apply the changes:
+
+   ```shell
+   sudo gitlab-ctl reconfigure
+   ```
+
+## Enabling the GitLab Container Registry
+
+Follow these steps to enable the container registry. Note that these steps reflect those for
+[configuring the container registry under an existing domain](../../administration/packages/container_registry.md#configure-container-registry-under-an-existing-gitlab-domain):
+
+1. Make the following changes to `/etc/gitlab/gitlab.rb`:
+
+   ```ruby
+   # Change external_registry_url to match external_url, but append the port 4567
+   external_url "https://example.gitlab.com"
+   registry_external_url "https://example.gitlab.com:4567"
+   ```
+
+1. Reconfigure your instance to apply the changes:
+
+   ```shell
+   sudo gitlab-ctl reconfigure
+   ```
+
+## Allow the Docker daemon to trust the registry and GitLab Runner
+
+Provide your Docker daemon with your certs by
+[following the steps for using trusted certificates with your registry](../../administration/packages/container_registry.md#using-self-signed-certificates-with-container-registry):
+
+```shell
+sudo mkdir -p /etc/docker/certs.d/my-host.internal:5000
+
+sudo cp /etc/gitlab/ssl/my-host.internal.crt /etc/docker/certs.d/my-host.internal:5000/ca.crt
+```
+
+Provide your GitLab Runner (to be installed next) with your certs by
+[following the steps for using trusted certificates with your Runner](https://docs.gitlab.com/runner/install/docker.html#installing-trusted-ssl-server-certificates):
+
+```shell
+sudo mkdir -p /etc/gitlab-runner/certs
+
+sudo cp /etc/gitlab/ssl/my-host.internal.crt /etc/gitlab-runner/certs/ca.crt
+```
+
+## Enabling GitLab Runner
+
+[Following a similar process to the steps for installing our GitLab Runner as a
+Docker service](https://docs.gitlab.com/runner/install/docker.html#docker-image-installation), we must first register our Runner:
+
+```shell
+$ sudo docker run --rm -it -v /etc/gitlab-runner:/etc/gitlab-runner gitlab/gitlab-runner register
+Updating CA certificates...
+Runtime platform                                    arch=amd64 os=linux pid=7 revision=1b659122 version=12.8.0
+Running in system-mode.
+
+Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
+https://my-host.internal
+Please enter the gitlab-ci token for this runner:
+XXXXXXXXXXX
+Please enter the gitlab-ci description for this runner:
+[eb18856e13c0]:
+Please enter the gitlab-ci tags for this runner (comma separated):
+
+Registering runner... succeeded                     runner=FSMwkvLZ
+Please enter the executor: custom, docker, virtualbox, kubernetes, docker+machine, docker-ssh+machine, docker-ssh, parallels, shell, ssh:
+docker
+Please enter the default Docker image (e.g. ruby:2.6):
+ruby:2.6
+Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
+```
+
+Now we must add some additional configuration to our runner:
+
+Make the following changes to `/etc/gitlab-runner/config.toml`:
+
+- Add Docker socket to volumes `volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]`
+- Add `pull_policy = "if-not-present"` to the executor configuration
+
+Now we can start our Runner:
+
+```shell
+sudo docker run -d --restart always --name gitlab-runner -v /etc/gitlab-runner:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest
+90646b6587127906a4ee3f2e51454c6e1f10f26fc7a0b03d9928d8d0d5897b64
+```
+
+### Authenticating the registry against the host OS
+
+As noted in [Docker's registry authentication documentation](https://docs.docker.com/registry/insecure/#docker-still-complains-about-the-certificate-when-using-authentication),
+certain versions of Docker require trusting the certificate chain at the OS level.
+
+In the case of Ubuntu, this involves using `update-ca-certificates`:
+
+```shell
+sudo cp /etc/docker/certs.d/my-host.internal\:5000/ca.crt /usr/local/share/ca-certificates/my-host.internal.crt
+
+sudo update-ca-certificates
+```
+
+If all goes well, this is what you should see:
+
+```plaintext
+1 added, 0 removed; done.
+Running hooks in /etc/ca-certificates/update.d...
+done.
+```