diff options
Diffstat (limited to 'doc/topics/offline/quick_start_guide.md')
-rw-r--r-- | doc/topics/offline/quick_start_guide.md | 92 |
1 files changed, 89 insertions, 3 deletions
diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index 80ce703f7db..57c7a80fa3e 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -23,7 +23,7 @@ For a video walkthrough of this process, see [Offline GitLab Installation: Downl You should [manually download the GitLab package](../../update/package/index.md#upgrade-using-a-manually-downloaded-package) and relevant dependencies using a server of the same operating system type that has access to the Internet. -If your offline environment has no local network access, you must manually transport across the relevant package files through physical media, such as a USB drive, or writable DVD. +If your offline environment has no local network access, you must manually transport the relevant package through physical media, such as a USB drive. In Ubuntu, this can be performed on a server with Internet access using the following commands: @@ -71,7 +71,7 @@ sudo EXTERNAL_URL="http://my-host.internal" dpkg -i <gitlab_package_name>.deb ## Enabling SSL Follow these steps to enable SSL for your fresh instance. These steps reflect those for -[manually configuring SSL in Omnibus's NGINX configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-https-manually): +[manually configuring SSL in Omnibus's NGINX configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-https-manually): 1. Make the following changes to `/etc/gitlab/gitlab.rb`: @@ -200,7 +200,7 @@ done. ### Disable Version Check and Service Ping -The Version Check and Service Ping services improve the GitLab user experience and ensure that +Version Check and Service Ping improve the GitLab user experience and ensure that users are on the most up-to-date instances of GitLab. These two services can be turned off for offline environments so that they do not attempt and fail to reach out to GitLab services. @@ -214,3 +214,89 @@ and Praefect servers so they can use an accessible NTP server. On offline instances, the [GitLab Geo check Rake task](../../administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-site-correctly) always fails because it uses `pool.ntp.org`. This error can be ignored but you can [read more about how to work around it](../../administration/geo/replication/troubleshooting.md#message-machine-clock-is-synchronized--exception). + +## Enabling the package metadata database + +Enabling the package metadata database is required to enable [license scanning of CycloneDX files](../../user/compliance/license_scanning_of_cyclonedx_files). +This process requires usage of the GitLab License Database, which is licensed under the [EE License](https://storage.googleapis.com/prod-export-license-bucket-1a6c642fc4de57d4/v1/LICENSE). +Note the following in relation to use of the License Database: + +- We may change or discontinue all or any part of the License Database, at any time and without notice, at our sole discretion. +- The License Database may contain links to third-party websites or resources. We provide these links only as a convenience and are not responsible for any third-party data, content, products, or services from those websites or resources or links displayed on such websites. +- The License Database is based in part on information made available by third parties, and GitLab is not responsible for the accuracy or completeness of content made available. + +### Using the gsutil tool to download the package metadata exports + +1. Install the [`gsutil`](https://cloud.google.com/storage/docs/gsutil_install) tool. +1. Find the root of the GitLab Rails directory. + + ```shell + export GITLAB_RAILS_ROOT_DIR="$(gitlab-rails runner 'puts Rails.root.to_s')" + echo $GITLAB_RAILS_ROOT_DIR + ``` + +1. Download the package metadata exports. + + ```shell + # To download the package metadata exports, an outbound connection to Google Cloud Storage bucket must be allowed. + mkdir $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + gsutil -m rsync -r -d gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + + # Alternatively, if the GitLab instance is not allowed to connect to the Google Cloud Storage bucket, the package metadata + # exports can be downloaded using a machine with the allowed access, and then copied to the root of the GitLab Rails directory. + rsync rsync://example_username@gitlab.example.com/package_metadata_db $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + ``` + +### Using the Google Cloud Storage REST API to download the package metadata exports + +The package metadata exports can also be downloaded using the Google Cloud Storage API. The contents are available at [https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o](https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o). The following is an example of how this can be downloaded using [cURL](https://curl.se/) and [jq](https://stedolan.github.io/jq/). + +```shell +#!/bin/bash + +set -euo pipefail + +GITLAB_RAILS_ROOT_DIR="$(gitlab-rails runner 'puts Rails.root.to_s')" +PKG_METADATA_DIR="$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db" +PKG_METADATA_MANIFEST_OUTPUT_FILE="/tmp/license_db_export_manifest.json" +PKG_METADATA_DOWNLOADS_OUTPUT_FILE="/tmp/license_db_object_links.tsv" + +# Download the contents of the bucket +curl --silent --show-error --request GET "https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o" > "$PKG_METADATA_MANIFEST_OUTPUT_FILE" + +# Parse the links and names for the bucket objects and output them into a tsv file +jq -r '.items[] | [.name, .mediaLink] | @tsv' "$PKG_METADATA_MANIFEST_OUTPUT_FILE" > "$PKG_METADATA_DOWNLOADS_OUTPUT_FILE" + +echo -e "Saving package metadata exports to $PKG_METADATA_DIR\n" + +# Track how many objects will be downloaded +INDEX=1 +TOTAL_OBJECT_COUNT="$(wc -l $PKG_METADATA_DOWNLOADS_OUTPUT_FILE | awk '{print $1}')" + +# Download the objects +while IFS= read -r line; do + FILE="$(echo -n $line | awk '{print $1}')" + URL="$(echo -n $line | awk '{print $2}')" + OUTPUT_DIR="$(dirname $PKG_METADATA_DIR/$FILE)" + OUTPUT_PATH="$PKG_METADATA_DIR/$FILE" + + echo "Downloading $FILE" + + curl --progress-bar --create-dirs --output "$OUTPUT_PATH" --request "GET" "$URL" + + echo -e "$INDEX of $TOTAL_OBJECT_COUNT objects downloaded\n" + + let INDEX=(INDEX+1) +done < "$PKG_METADATA_DOWNLOADS_OUTPUT_FILE" + +echo "All objects saved to $PKG_METADATA_DIR" +``` + +### Automatic synchronization + +Your GitLab instance is synchronized [every hour](https://gitlab.com/gitlab-org/gitlab/-/blob/d4331343d26d6e2a81fadd8f7ecd72f7cb74d04d/config/initializers/1_settings.rb#L831-832) with the contents of the `package_metadata_db` directory. +To automatically update your local copy with the upstream changes, a cron job can be added to periodically download new exports. For example, the following crontabs can be added to setup a cron job that runs every 30 minutes. + +```plaintext +*/30 * * * * gsutil -m rsync -r -d gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ +``` |