diff options
Diffstat (limited to 'doc/administration/object_storage.md')
| -rw-r--r-- | doc/administration/object_storage.md | 186 |
1 files changed, 93 insertions, 93 deletions
diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index d133e8c3721..f1025bd1846 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Object Storage +# Object storage **(FREE SELF)** GitLab supports using an object storage service for holding numerous types of data. It's recommended over NFS and @@ -29,7 +29,7 @@ GitLab has been tested on a number of object storage providers: - Dell EMC ECS: Prior to GitLab 13.3, there is a [known bug in GitLab Workhorse that prevents HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806). - Be sure to upgrade to GitLab v13.3.0 or above if you use S3 storage with this hardware. + Be sure to upgrade to GitLab 13.3.0 or above if you use S3 storage with this hardware. - Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying). @@ -47,7 +47,7 @@ For more information on the differences and to transition from one form to anoth ### Consolidated object storage configuration -> Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368). +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368) in GitLab 13.2. Using the consolidated object storage configuration has a number of advantages: @@ -209,13 +209,13 @@ gitlab_rails['object_store']['connection'] = { } ``` -| Setting | Description | -|---------|-------------| -| `enabled` | Enable/disable object storage | -| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | -| `connection` | Various [connection options](#connection-settings) described below | -| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3 | -| `objects` | [Object-specific configuration](#object-specific-configuration) +| Setting | Description | +|-------------------|-----------------------------------| +| `enabled` | Enable or disable object storage. | +| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | +| `connection` | Various [connection options](#connection-settings) described below. | +| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3. | +| `objects` | [Object-specific configuration](#object-specific-configuration). | ### Connection settings @@ -226,27 +226,27 @@ in the `connection` setting. The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws): -| Setting | Description | Default | -|---------|-------------|---------| -| `provider` | Always `AWS` for compatible hosts | `AWS` | -| `aws_access_key_id` | AWS credentials, or compatible | | -| `aws_secret_access_key` | AWS credentials, or compatible | | -| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | -| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | -| `region` | AWS region. | | -| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | -| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | -| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false` | -| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys | `false` +| Setting | Description | Default | +|---------------------------------|------------------------------------|---------| +| `provider` | Always `AWS` for compatible hosts. | `AWS` | +| `aws_access_key_id` | AWS credentials, or compatible. | | +| `aws_secret_access_key` | AWS credentials, or compatible. | | +| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | +| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | +| `region` | AWS region. | | +| `host` | S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | +| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | +| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false`. | +| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` | #### Oracle Cloud S3 connection settings Note that Oracle Cloud S3 must be sure to use the following settings: -| Setting | Value | -|---------|-------| +| Setting | Value | +|---------------------------------|---------| | `enable_signature_v4_streaming` | `false` | -| `path_style` | `true` | +| `path_style` | `true` | If `enable_signature_v4_streaming` is set to `true`, you may see the following error in `production.log`: @@ -259,13 +259,13 @@ STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported Here are the valid connection parameters for GCS: -| Setting | Description | example | -|---------|-------------|---------| -| `provider` | The provider name | `Google` | -| `google_project` | GCP project name | `gcp-project-12345` | -| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | -| `google_json_key_location` | The JSON key path | `/path/to/gcp-project-12345-abcde.json` | -| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | +| Setting | Description | Example | +|------------------------------|-------------------|---------| +| `provider` | Provider name. | `Google` | +| `google_project` | GCP project name. | `gcp-project-12345` | +| `google_client_email` | Email address of the service account. | `foo@gcp-project-12345.iam.gserviceaccount.com` | +| `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` | +| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | | The service account must have permission to access the bucket. Learn more in Google's @@ -328,12 +328,12 @@ The following are the valid connection parameters for Azure. Read the [Azure Blob storage documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) to learn more. -| Setting | Description | Example | -|---------|-------------|---------| -| `provider` | Provider name | `AzureRM` | -| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage | `azuretest` | -| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | -| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | +| Setting | Description | Example | +|------------------------------|----------------|-----------| +| `provider` | Provider name. | `AzureRM` | +| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage. | `azuretest` | +| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | +| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | ##### Azure example (consolidated form) @@ -382,15 +382,15 @@ consolidated form, see the [S3 settings](#s3-compatible-connection-settings). Here are the valid connection settings for the Swift API, provided by [fog-openstack](https://github.com/fog/fog-openstack): -| Setting | Description | Default | -|---------|-------------|---------| -| `provider` | Always `OpenStack` for compatible hosts | `OpenStack` | -| `openstack_username` | OpenStack username | | -| `openstack_api_key` | OpenStack API key | | +| Setting | Description | Default | +|--------------------------|----------------------|---------| +| `provider` | Always `OpenStack` for compatible hosts. | `OpenStack` | +| `openstack_username` | OpenStack username. | | +| `openstack_api_key` | OpenStack API key. | | | `openstack_temp_url_key` | OpenStack key for generating temporary URLs | | -| `openstack_auth_url` | OpenStack authentication endpoint | | -| `openstack_region` | OpenStack region | | -| `openstack_tenant` | OpenStack tenant ID | +| `openstack_auth_url` | OpenStack authentication endpoint | | +| `openstack_region` | OpenStack region. | | +| `openstack_tenant` | OpenStack tenant ID. | | #### Rackspace Cloud Files @@ -400,13 +400,13 @@ Rackspace Cloud, provided by [fog-rackspace](https://github.com/fog/fog-rackspac This isn't compatible with the consolidated object storage form. Rackspace Cloud is supported only with the storage-specific form. -| Setting | Description | example | -|---------|-------------|---------| -| `provider` | The provider name | `Rackspace` | -| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | -| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | -| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | -| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | +| Setting | Description | Example | +|--------------------------|----------------|-------------| +| `provider` | Provider name. | `Rackspace` | +| `rackspace_username` | Username of the Rackspace account with access to the container. | `joe.smith` | +| `rackspace_api_key` | API key of the Rackspace account with access to the container. | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_region` | Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/). | `iad` | +| `rackspace_temp_url_key` | Private key you set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | Regardless of whether the container has public access enabled or disabled, Fog uses the TempURL method to grant access to LFS objects. If you see error @@ -465,24 +465,24 @@ gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages' This is the list of valid `objects` that can be used: -| Type | Description | -|--------------------|---------------| -| `artifacts` | [CI artifacts](job_artifacts.md) | -| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | -| `uploads` | [User uploads](uploads.md) | -| `lfs` | [Git Large File Storage objects](lfs/index.md) | -| `packages` | [Project packages (e.g. PyPI, Maven, NuGet, etc.)](packages/index.md) | -| `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | -| `terraform_state` | [Terraform state files](terraform_state.md) | -| `pages` | [GitLab Pages](pages/index.md) | +| Type | Description | +|--------------------|----------------------------------------------------------------------------| +| `artifacts` | [CI artifacts](job_artifacts.md) | +| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | +| `uploads` | [User uploads](uploads.md) | +| `lfs` | [Git Large File Storage objects](lfs/index.md) | +| `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) | +| `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | +| `terraform_state` | [Terraform state files](terraform_state.md) | +| `pages` | [GitLab Pages](pages/index.md) | Within each object type, three parameters can be defined: -| Setting | Required? | Description | -|------------------|-----------|-------------| -| `bucket` | Yes | The bucket name for the object storage. | -| `enabled` | No | Overrides the common parameter | -| `proxy_download` | No | Overrides the common parameter | +| Setting | Required? | Description | +|------------------|------------------------|-------------------------------------| +| `bucket` | **{check-circle}** Yes | Bucket name for the object storage. | +| `enabled` | **{dotted-circle}** No | Overrides the common parameter. | +| `proxy_download` | **{dotted-circle}** No | Overrides the common parameter. | #### Selectively disabling object storage @@ -542,21 +542,21 @@ original configuration (for example, `artifacts_object_store_enabled`, or For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides: -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](uploads.md#using-object-storage) | Yes | -| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | Yes | -| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](terraform_state.md#using-object-storage) | Yes | -| [GitLab Pages content](pages/index.md#using-object-storage) | Yes | +| Object storage type | Supported by consolidated configuration? | +|---------------------|------------------------------------------| +| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | +| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes | +| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes | +| [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes | +| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | +| [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| **{dotted-circle}** No | +| [Packages](packages/index.md#using-object-storage) (optional feature) | **{check-circle}** Yes | +| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | **{check-circle}** Yes | +| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | **{dotted-circle}** No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | +| [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | +| [GitLab Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | ### Other alternatives to file system storage @@ -618,20 +618,20 @@ This can result in some of the following problems: - If GitLab is using non-secure HTTP to access the object storage, clients may generate `https->http` downgrade errors and refuse to process the redirect. The solution to this -is for GitLab to use HTTPS. LFS, for example, will generate this error: +is for GitLab to use HTTPS. LFS, for example, generates this error: ```plaintext LFS: lfsapi/client: refusing insecure redirect, https->http ``` -- Clients will need to trust the certificate authority that issued the object storage +- Clients need to trust the certificate authority that issued the object storage certificate, or may return common TLS errors such as: ```plaintext x509: certificate signed by unknown authority ``` -- Clients will need network access to the object storage. +- Clients need network access to the object storage. Network firewalls could block access. Errors that might result if this access is not in place include: @@ -667,7 +667,7 @@ The first option is recommended for MinIO. Otherwise, the is to use the `--compat` parameter on the server. Without consolidated object store configuration or instance profiles enabled, -GitLab Workhorse will upload files to S3 using pre-signed URLs that do +GitLab Workhorse uploads files to S3 using pre-signed URLs that do not have a `Content-MD5` HTTP header computed for them. To ensure data is not corrupted, Workhorse checks that the MD5 hash of the data sent equals the ETag header returned from the S3 server. When encryption is @@ -683,7 +683,7 @@ eliminates the need to compare ETag headers returned from the S3 server. Instead of supplying AWS access and secret keys in object storage configuration, GitLab can be configured to use IAM roles to set up an [Amazon instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). -When this is used, GitLab will fetch temporary credentials each time an +When this is used, GitLab fetches temporary credentials each time an S3 bucket is accessed, so no hard-coded values are needed in the configuration. @@ -709,10 +709,10 @@ only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowl To do this, you must configure GitLab to send the proper encryption headers in the `storage_options` configuration section: -| Setting | Description | -|-------------------------------------|-------------| -| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`) | -| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) | +| Setting | Description | +|-------------------------------------|------------------------------------------| +| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`). | +| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). | As with the case for default encryption, these options only work when the Workhorse S3 client is enabled. One of the following two conditions @@ -721,7 +721,7 @@ must be fulfilled: - `use_iam_profile` is `true` in the connection settings. - Consolidated object storage settings are in use. -[ETag mismatch errors](#etag-mismatch) will occur if server side +[ETag mismatch errors](#etag-mismatch) occur if server side encryption headers are used without enabling the Workhorse S3 client. ##### Disabling the feature |