diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-06-19 00:08:50 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-06-19 00:08:50 +0300 |
commit | 047eb7596938953e0e927dfae9096a9326c98a53 (patch) | |
tree | db08cc80ba802c085b6f5c37e5102d0debe1b5db /doc | |
parent | 40c103c53fcc97e0a94ed7744f247318c14492c0 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/administration/object_storage.md | 4 | ||||
-rw-r--r-- | doc/administration/raketasks/check.md | 38 | ||||
-rw-r--r-- | doc/administration/secure_files.md | 201 |
3 files changed, 242 insertions, 1 deletions
diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index a6f3a96cb0f..f2b966bd180 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -114,6 +114,7 @@ The following table lists the valid `objects` that can be used: | `dependency_proxy` | [Dependency Proxy](packages/dependency_proxy.md) | | `terraform_state` | [Terraform state files](terraform_state.md) | | `pages` | [Pages](pages/index.md) | +| `ci_secure_files` | [Project-level Secure Files](secure_files.md) | Within each object type, three parameters can be defined: @@ -161,6 +162,7 @@ supported by consolidated form, refer to the following guides: | Object storage type | Supported by consolidated form? | |---------------------|------------------------------------------| +| [Project-level Secure Files](secure_files.md#using-object-storage) | **{dotted-circle}** No | | [Backups](../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | | [Mattermost](https://docs.mattermost.com/configure/file-storage-configuration-settings.html)| **{dotted-circle}** No | @@ -463,7 +465,6 @@ The following example uses AWS S3 to enable object storage for all supported ser gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages' gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy' gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state' - gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files' gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages' ``` @@ -726,6 +727,7 @@ To migrate existing local data to object storage see the following guides: - [Dependency Proxy](packages/dependency_proxy.md#migrate-local-dependency-proxy-blobs-and-manifests-to-object-storage) - [Terraform state files](terraform_state.md#migrate-to-object-storage) - [Pages content](pages/index.md#migrate-pages-deployments-to-object-storage) +- [Project-level Secure Files](secure_files.md#migrate-to-object-storage) ## Transition to consolidated form diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 2cb664b0859..e55a0f1c8a7 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -121,12 +121,14 @@ Integrity checks are supported for the following types of file: - CI artifacts (introduced in GitLab 10.7.0) - LFS objects (introduced in GitLab 10.6.0) +- Project-level Secure Files (introduced in GitLab 16.1.0) - User uploads (introduced in GitLab 10.6.0) **Omnibus Installation** ```shell sudo gitlab-rake gitlab:artifacts:check +sudo gitlab-rake gitlab:ci_secure_files:check sudo gitlab-rake gitlab:lfs:check sudo gitlab-rake gitlab:uploads:check ``` @@ -135,6 +137,7 @@ sudo gitlab-rake gitlab:uploads:check ```shell sudo -u git -H bundle exec rake gitlab:artifacts:check RAILS_ENV=production +sudo -u git -H bundle exec rake gitlab:ci_secure_files:check RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:lfs:check RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:uploads:check RAILS_ENV=production ``` @@ -151,6 +154,7 @@ Variable | Type | Description ```shell sudo gitlab-rake gitlab:artifacts:check BATCH=100 ID_FROM=50 ID_TO=250 +sudo gitlab-rake gitlab:ci_secure_files:check BATCH=100 ID_FROM=50 ID_TO=250 sudo gitlab-rake gitlab:lfs:check BATCH=100 ID_FROM=50 ID_TO=250 sudo gitlab-rake gitlab:uploads:check BATCH=100 ID_FROM=50 ID_TO=250 ``` @@ -415,3 +419,37 @@ To update these references to point to local storage: ``` The script to [delete references to missing artifacts](check.md#delete-references-to-missing-artifacts) now functions correctly and cleans up the database. + +### Delete references to missing secure files + +`VERBOSE=1 gitlab-rake gitlab:ci_secure_files:check` detects when secure files: + +- Are deleted outside of GitLab. +- Have references still in the GitLab database. + +When this scenario is detected, the Rake task displays an error message. For example: + +```shell +Checking integrity of CI Secure Files +- 1..15: Failures: 2 + - Job SecureFile: 9: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/ci_secure_files/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/2022_06_30/8/9/distribution.cer> + - Job SecureFile: 15: Remote object does not exist +Done! + +``` + +To delete these references to missing local or remote secure files: + +1. Open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session). +1. Run the following Ruby code: + + ```ruby + secure_files_deleted = 0 + ::Ci::SecureFile.find_each do |secure_file| ### Iterate secure files + next if secure_file.file.file.exists? ### Skip if the file reference is valid + secure_files_deleted += 1 + puts "#{secure_file.id} #{secure_file.file.path} is missing." ### Allow verification before destroy + # secure_file.destroy! ### Uncomment to actually destroy + end + puts "Count of identified/destroyed invalid references: #{secure_files_deleted}" + ``` diff --git a/doc/administration/secure_files.md b/doc/administration/secure_files.md new file mode 100644 index 00000000000..3c9d40c3e73 --- /dev/null +++ b/doc/administration/secure_files.md @@ -0,0 +1,201 @@ +--- +stage: Mobile +group: Mobile DevOps +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Secure Files administration **(FREE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78227) in GitLab 14.8 [with a flag](feature_flags.md) named `ci_secure_files`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/350748) in GitLab 15.7. Feature flag `ci_secure_files` removed. + +You can securely store up to 100 files for use in CI/CD pipelines as secure files. +These files are stored securely outside of your project's repository and are not version controlled. +It is safe to store sensitive information in these files. Secure files support both plain text +and binary file types, and must be 5 MB or less. + +The storage location of these files can be configured using the options described below, +but the default locations are: + +- `/var/opt/gitlab/gitlab-rails/shared/ci_secure_files` for installations using the Linux package. +- `/home/git/gitlab/shared/ci_secure_files` for self-compiled installations. + +Use [external object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-terraform-state-dependency-proxy) +configuration for [GitLab Helm chart](https://docs.gitlab.com/charts/) installations. + +## Disabling Secure Files + +You can disable Secure Files across the entire GitLab instance. You might want to disable +Secure Files to reduce disk space, or to remove access to the feature. + +To disable Secure Files, follow the steps below according to your installation. + +Prerequisite: + +- You must be an administrator. + +**For Linux package installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['ci_secure_files_enabled'] = false + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +**For self-compiled installations** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + ci_secure_files: + enabled: false + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + +## Using local storage + +The default configuration uses local storage. To change the location where Secure Files +are stored locally, follow the steps below. + +**For Linux package installations** + +1. To change the storage path for example to `/mnt/storage/ci_secure_files`, edit + `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['ci_secure_files_storage_path'] = "/mnt/storage/ci_secure_files" + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +**For self-compiled installations** + +1. To change the storage path for example to `/mnt/storage/ci_secure_files`, edit + `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + ci_secure_files: + enabled: true + storage_path: /mnt/storage/ci_secure_files + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) + for the changes to take effect. + +## Using object storage **(FREE SELF)** + +Instead of storing Secure Files on disk, you should use [one of the supported object storage options](object_storage.md#supported-object-storage-providers). +This configuration relies on valid credentials to be configured already. + +[Read more about using object storage with GitLab](object_storage.md). + +NOTE: +This feature is not supported by consolidated object storage configuration. +Adding support is proposed in [issue 414673](https://gitlab.com/gitlab-org/gitlab/-/issues/414673). + +### Object storage settings + +The following settings are: + +- Nested under `ci_secure_files:` and then `object_store:` on source installations. +- Prefixed by `ci_secure_files_object_store_` on Linux package installations. + +| Setting | Description | Default | +|---------|-------------|---------| +| `enabled` | Enable/disable object storage | `false` | +| `remote_directory` | The bucket name where Secure Files are stored | | +| `connection` | Various connection options described below | | + +### S3-compatible connection settings + +See [the available connection settings for different providers](object_storage.md#configure-the-connection-settings). + +**For Linux package installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, but using + the values you want: + + ```ruby + gitlab_rails['ci_secure_files_object_store_enabled'] = true + gitlab_rails['ci_secure_files_object_store_remote_directory'] = "terraform" + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: + If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs: + + ```ruby + gitlab_rails['ci_secure_files_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#reconfigure-a-linux-package-installation) +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. [Migrate any existing local states to the object storage](#migrate-to-object-storage) + +**For self-compiled installations** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + ci_secure_files: + enabled: true + object_store: + enabled: true + remote_directory: "ci_secure_files" # The bucket name + connection: + provider: AWS # Only AWS supported at the moment + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +1. [Migrate any existing local states to the object storage](#migrate-to-object-storage) + +### Migrate to object storage + +> [Introduced](https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/readme/-/issues/125) in GitLab 16.1. + +WARNING: +It's not possible to migrate Secure Files from object storage back to local storage, +so proceed with caution. + +To migrate Secure Files to object storage, follow the instructions below. + +- For Linux package installations: + + ```shell + sudo gitlab-rake gitlab:ci_secure_files:migrate + ``` + +- For self-compiled installations: + + ```shell + sudo -u git -H bundle exec rake gitlab:ci_secure_files:migrate RAILS_ENV=production + ``` |