diff options
Diffstat (limited to 'doc/user/infrastructure/index.md')
| -rw-r--r-- | doc/user/infrastructure/index.md | 211 |
1 files changed, 150 insertions, 61 deletions
diff --git a/doc/user/infrastructure/index.md b/doc/user/infrastructure/index.md index c17d1831b50..e24e669d994 100644 --- a/doc/user/infrastructure/index.md +++ b/doc/user/infrastructure/index.md @@ -6,8 +6,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Infrastructure as code with Terraform and GitLab +## Motivation + +The Terraform integration features within GitLab enable your GitOps / Infrastructure-as-Code (IaC) +workflows to tie into GitLab's authentication and authorization. These features focus on +lowering the barrier to entry for teams to adopt Terraform, collaborate effectively within +GitLab, and support Terraform best practices. + ## GitLab managed Terraform State +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2673) in GitLab 13.0. + [Terraform remote backends](https://www.terraform.io/docs/backends/index.html) enable you to store the state file in a remote, shared store. GitLab uses the [Terraform HTTP backend](https://www.terraform.io/docs/backends/types/http.html) @@ -27,6 +36,14 @@ To get started with a GitLab-managed Terraform State, there are two different op - [Use a local machine](#get-started-using-local-development). - [Use GitLab CI](#get-started-using-gitlab-ci). +## Permissions for using Terraform + +In GitLab version 13.1, [Maintainer access](../permissions.md) was required to use a +GitLab managed Terraform state backend. In GitLab versions 13.2 and greater, +[Maintainer access](../permissions.md) is required to lock, unlock and write to the state +(using `terraform apply`), while [Developer access](../permissions.md) is required to read +the state (using `terraform plan -lock=false`). + ## Get started using local development If you plan to only run `terraform plan` and `terraform apply` commands from your @@ -45,8 +62,7 @@ local machine, this is a simple way to get started: ``` 1. Create a [Personal Access Token](../profile/personal_access_tokens.md) with - the `api` scope. The Terraform backend is restricted to users with - [Maintainer access](../permissions.md) to the repository. + the `api` scope. 1. On your local machine, run `terraform init`, passing in the following options, replacing `<YOUR-PROJECT-NAME>`, `<YOUR-PROJECT-ID>`, `<YOUR-USERNAME>` and @@ -80,10 +96,6 @@ Next, [configure the backend](#configure-the-backend). After executing the `terraform init` command, you must configure the Terraform backend and the CI YAML file: -CAUTION: **Important:** -The Terraform backend is restricted to users with [Maintainer access](../permissions.md) -to the repository. - 1. In your Terraform project, define the [HTTP backend](https://www.terraform.io/docs/backends/types/http.html) by adding the following code block in a `.tf` file (such as `backend.tf`) to define the remote backend: @@ -95,64 +107,75 @@ to the repository. } ``` -1. In the root directory of your project repository, configure a `.gitlab-ci.yaml` file. - This example uses a pre-built image: +1. In the root directory of your project repository, configure a + `.gitlab-ci.yaml` file. This example uses a pre-built image which includes a + `gitlab-terraform` helper. For supported Terraform versions, see the [GitLab + Terraform Images project](https://gitlab.com/gitlab-org/terraform-images). ```yaml - image: - name: hashicorp/terraform:light - entrypoint: - - '/usr/bin/env' - - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest ``` -1. In the `.gitlab-ci.yaml` file, define some environment variables to ease development. In this - example, `GITLAB_TF_ADDRESS` is the URL of the GitLab instance where this pipeline - runs, and `TF_ROOT` is the directory where the Terraform commands must be executed: +1. In the `.gitlab-ci.yaml` file, define some environment variables to ease + development. In this example, `TF_ROOT` is the directory where the Terraform + commands must be executed, `TF_ADDRESS` is the URL to the state on the GitLab + instance where this pipeline runs, and the final path segment in `TF_ADDRESS` + is the name of the Terraform state. Projects may have multiple states, and + this name is arbitrary, so in this example we will set it to the name of the + project, and we will ensure that the `.terraform` directory is cached between + jobs in the pipeline using a cache key based on the state name: ```yaml variables: - GITLAB_TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME} TF_ROOT: ${CI_PROJECT_DIR}/environments/cloudflare/production + TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME} cache: + key: ${CI_PROJECT_NAME} paths: - - .terraform + - ${TF_ROOT}/.terraform ``` -1. In a `before_script`, pass a `terraform init` call containing configuration parameters - corresponding to variables required by the - [HTTP backend](https://www.terraform.io/docs/backends/types/http.html): +1. In a `before_script`, change to your `TF_ROOT`: ```yaml before_script: - cd ${TF_ROOT} - - terraform --version - - terraform init -backend-config="address=${GITLAB_TF_ADDRESS}" -backend-config="lock_address=${GITLAB_TF_ADDRESS}/lock" -backend-config="unlock_address=${GITLAB_TF_ADDRESS}/lock" -backend-config="username=gitlab-ci-token" -backend-config="password=${CI_JOB_TOKEN}" -backend-config="lock_method=POST" -backend-config="unlock_method=DELETE" -backend-config="retry_wait_min=5" stages: + - prepare - validate - build - - test - deploy + init: + stage: prepare + script: + - gitlab-terraform init + validate: stage: validate script: - - terraform validate + - gitlab-terraform validate plan: stage: build script: - - terraform plan - - terraform show + - gitlab-terraform plan + - gitlab-terraform plan-json + artifacts: + name: plan + paths: + - ${TF_ROOT}/plan.cache + reports: + terraform: ${TF_ROOT}/plan.json apply: stage: deploy environment: name: production script: - - terraform apply + - gitlab-terraform apply dependencies: - plan when: manual @@ -160,8 +183,9 @@ to the repository. - master ``` -1. Push your project to GitLab, which triggers a CI job pipeline. This pipeline runs - the `terraform init`, `terraform validate`, and `terraform plan` commands. +1. Push your project to GitLab, which triggers a CI job pipeline. This pipeline + runs the `gitlab-terraform init`, `gitlab-terraform validate`, and + `gitlab-terraform plan` commands. The output from the above `terraform` commands should be viewable in the job logs. @@ -176,15 +200,18 @@ you can expose details from `terraform plan` runs directly into a merge request enabling you to see statistics about the resources that Terraform will create, modify, or destroy. -Let's explore how to configure a GitLab Terraform Report artifact: +Let's explore how to configure a GitLab Terraform Report artifact. You can +either use a pre-built image which includes a `gitlab-terraform` helper as +above, where `gitlab-terraform plan-json` outputs the required artifact, or you +can configure this manually as follows: 1. For simplicity, let's define a few reusable variables to allow us to refer to these files multiple times: ```yaml variables: - PLAN: plan.tfplan - PLAN_JSON: tfplan.json + PLAN: plan.cache + PLAN_JSON: plan.json ``` 1. Install `jq`, a @@ -198,6 +225,18 @@ Let's explore how to configure a GitLab Terraform Report artifact: - alias convert_report="jq -r '([.resource_changes[]?.change.actions?]|flatten)|{\"create\":(map(select(.==\"create\"))|length),\"update\":(map(select(.==\"update\"))|length),\"delete\":(map(select(.==\"delete\"))|length)}'" ``` + NOTE: **Note:** + In distributions that use Bash (for example, Ubuntu), `alias` statements are not + expanded in non-interactive mode. If your pipelines fail with the error + `convert_report: command not found`, alias expansion can be activated explicitly + by adding a `shopt` command to your script: + + ```yaml + before_script: + - shopt -s expand_aliases + - alias convert_report="jq -r '([.resource_changes[]?.change.actions?]|flatten)|{\"create\":(map(select(.==\"create\"))|length),\"update\":(map(select(.==\"update\"))|length),\"delete\":(map(select(.==\"delete\"))|length)}'" + ``` + 1. Define a `script` that runs `terraform plan` and `terraform show`. These commands pipe the output and convert the relevant bits into a store variable `PLAN_JSON`. This JSON is used to create a @@ -212,18 +251,18 @@ Let's explore how to configure a GitLab Terraform Report artifact: - terraform plan -out=$PLAN - terraform show --json $PLAN | convert_report > $PLAN_JSON artifacts: - name: plan - paths: - - $PLAN reports: terraform: $PLAN_JSON ``` - For a full example, see [Example `.gitlab-ci.yaml` file](#example-gitlab-ciyaml-file). + For a full example using the pre-built image, see [Example `.gitlab-ci.yaml` + file](#example-gitlab-ciyaml-file). + + For an example displaying multiple reports, see [`.gitlab-ci.yaml` multiple reports file](#multiple-terraform-plan-reports). 1. Running the pipeline displays the widget in the merge request, like this: -  +  1. Clicking the **View Full Log** button in the widget takes you directly to the plan output present in the pipeline logs: @@ -233,64 +272,114 @@ Let's explore how to configure a GitLab Terraform Report artifact: ### Example `.gitlab-ci.yaml` file ```yaml -image: - name: hashicorp/terraform:light - entrypoint: - - '/usr/bin/env' - - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' +image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest -# Default output file for Terraform plan variables: - GITLAB_TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME} - PLAN: plan.tfplan - PLAN_JSON: tfplan.json - TF_ROOT: ${CI_PROJECT_DIR} + TF_ROOT: ${CI_PROJECT_DIR}/environments/cloudflare/production + TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME} cache: + key: ${CI_PROJECT_NAME} paths: - - .terraform + - ${TF_ROOT}/.terraform before_script: - - apk --no-cache add jq - - alias convert_report="jq -r '([.resource_changes[]?.change.actions?]|flatten)|{\"create\":(map(select(.==\"create\"))|length),\"update\":(map(select(.==\"update\"))|length),\"delete\":(map(select(.==\"delete\"))|length)}'" - cd ${TF_ROOT} - - terraform --version - - terraform init -backend-config="address=${GITLAB_TF_ADDRESS}" -backend-config="lock_address=${GITLAB_TF_ADDRESS}/lock" -backend-config="unlock_address=${GITLAB_TF_ADDRESS}/lock" -backend-config="username=${GITLAB_USER_LOGIN}" -backend-config="password=${GITLAB_TF_PASSWORD}" -backend-config="lock_method=POST" -backend-config="unlock_method=DELETE" -backend-config="retry_wait_min=5" stages: + - prepare - validate - build - deploy +init: + stage: prepare + script: + - gitlab-terraform init + validate: stage: validate script: - - terraform validate + - gitlab-terraform validate plan: stage: build script: - - terraform plan -out=$PLAN - - terraform show --json $PLAN | convert_report > $PLAN_JSON + - gitlab-terraform plan + - gitlab-terraform plan-json artifacts: name: plan paths: - - ${TF_ROOT}/plan.tfplan + - ${TF_ROOT}/plan.cache reports: - terraform: ${TF_ROOT}/tfplan.json + terraform: ${TF_ROOT}/plan.json -# Separate apply job for manual launching Terraform as it can be destructive -# action. apply: stage: deploy environment: name: production script: - - terraform apply -input=false $PLAN + - gitlab-terraform apply dependencies: - plan when: manual only: - master +``` + +### Multiple Terraform Plan reports + +Starting with 13.2, you can display mutiple reports on the Merge Request page. The reports will also display the `artifact: name:`. See example below for a suggested setup. + +```yaml +image: + name: registry.gitlab.com/gitlab-org/gitlab-build-images:terraform + entrypoint: + - '/usr/bin/env' + - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + +cache: + paths: + - .terraform +stages: + - build + +.terraform-plan-generation: + stage: build + variables: + PLAN: plan.tfplan + JSON_PLAN_FILE: tfplan.json + before_script: + - cd ${TERRAFORM_DIRECTORY} + - terraform --version + - terraform init + - apk --no-cache add jq + script: + - terraform validate + - terraform plan -out=${PLAN} + - terraform show --json ${PLAN} | jq -r '([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}' > ${JSON_PLAN_FILE} + artifacts: + reports: + terraform: ${TERRAFORM_DIRECTORY}/${JSON_PLAN_FILE} + +review_plan: + extends: .terraform-plan-generation + variables: + TERRAFORM_DIRECTORY: "review/" + # Review will not include an artifact name + +staging_plan: + extends: .terraform-plan-generation + variables: + TERRAFORM_DIRECTORY: "staging/" + artifacts: + name: Staging + +production_plan: + extends: .terraform-plan-generation + variables: + TERRAFORM_DIRECTORY: "production/" + artifacts: + name: Production ``` |