diff options
Diffstat (limited to 'doc/topics/autodevops')
| -rw-r--r-- | doc/topics/autodevops/index.md | 271 | ||||
| -rw-r--r-- | doc/topics/autodevops/quick_start_guide.md | 6 |
2 files changed, 161 insertions, 116 deletions
diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index a1373639a87..93549ac4de5 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -95,58 +95,85 @@ Auto DevOps. To make full use of Auto DevOps, you will need: +- **Kubernetes** (for Auto Review Apps, Auto Deploy, and Auto Monitoring) + + To enable deployments, you will need: + + 1. A [Kubernetes 1.12+ cluster](../../user/project/clusters/index.md) for the project. The easiest + way is to add a [new cluster using the GitLab UI](../../user/project/clusters/add_remove_clusters.md#add-new-cluster). + 1. NGINX Ingress. You can deploy it to your Kubernetes cluster by installing + the [GitLab-managed app for Ingress](../../user/clusters/applications.md#ingress), + once you have configured GitLab's Kubernetes integration in the previous step. + + Alternatively, you can use the + [`nginx-ingress`](https://github.com/helm/charts/tree/master/stable/nginx-ingress) + Helm chart to install Ingress manually. + + NOTE: **Note:** + If you are using your own Ingress instead of the one provided by GitLab's managed + apps, ensure you are running at least version 0.9.0 of NGINX Ingress and + [enable Prometheus metrics](https://github.com/helm/charts/tree/master/stable/nginx-ingress#prometheus-metrics) + in order for the response metrics to appear. You will also have to + [annotate](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) + the NGINX Ingress deployment to be scraped by Prometheus using + `prometheus.io/scrape: "true"` and `prometheus.io/port: "10254"`. + +- **Base domain** (for Auto Review Apps, Auto Deploy, and Auto Monitoring) + + You will need a domain configured with wildcard DNS which is going to be used + by all of your Auto DevOps applications. If you're using the + [GitLab-managed app for Ingress](../../user/clusters/applications.md#ingress), + the URL endpoint will be automatically configured for you. + + You will then need to [specify the Auto DevOps base domain](#auto-devops-base-domain). + - **GitLab Runner** (for all stages) Your Runner needs to be configured to be able to run Docker. Generally this means using either the [Docker](https://docs.gitlab.com/runner/executors/docker.html) or [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html) executors, with [privileged mode enabled](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode). - The Runners do not need to be installed in the Kubernetes cluster, but the Kubernetes executor is easy to use and is automatically autoscaling. Docker-based Runners can be configured to autoscale as well, using [Docker Machine](https://docs.gitlab.com/runner/install/autoscaling.html). + If you have configured GitLab's Kubernetes integration in the first step, you + can deploy it to your cluster by installing the + [GitLab-managed app for GitLab Runner](../../user/clusters/applications.md#gitlab-runner). + Runners should be registered as [shared Runners](../../ci/runners/README.md#registering-a-shared-runner) for the entire GitLab instance, or [specific Runners](../../ci/runners/README.md#registering-a-specific-runner) - that are assigned to specific projects. -- **Base domain** (for Auto Review Apps and Auto Deploy) - - You will need a domain configured with wildcard DNS which is going to be used - by all of your Auto DevOps applications. - - Read the [specifics](#auto-devops-base-domain). -- **Kubernetes** (for Auto Review Apps, Auto Deploy, and Auto Monitoring) - - To enable deployments, you will need: + that are assigned to specific projects (the default if you have installed the + GitLab Runner managed application). - - Kubernetes 1.5+. - - A [Kubernetes cluster][kubernetes-clusters] for the project. - - A load balancer. You can use NGINX Ingress by deploying it to your - Kubernetes cluster by either: - - Using the [`nginx-ingress`](https://github.com/helm/charts/tree/master/stable/nginx-ingress) Helm chart. - - Installing the Ingress [GitLab Managed App](../../user/clusters/applications.md#ingress). - **Prometheus** (for Auto Monitoring) - To enable Auto Monitoring, you - will need Prometheus installed somewhere (inside or outside your cluster) and - configured to scrape your Kubernetes cluster. To get response metrics - (in addition to system metrics), you need to - [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring). + To enable Auto Monitoring, you will need Prometheus installed somewhere + (inside or outside your cluster) and configured to scrape your Kubernetes cluster. + If you have configured GitLab's Kubernetes integration, you can deploy it to + your cluster by installing the + [GitLab-managed app for Prometheus](../../user/clusters/applications.md#prometheus). The [Prometheus service](../../user/project/integrations/prometheus.md) - integration needs to be enabled for the project, or enabled as a + integration needs to be enabled for the project (or enabled as a [default service template](../../user/project/integrations/services_templates.md) - for the entire GitLab installation. + for the entire GitLab installation). + + To get response metrics (in addition to system metrics), you need to + [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring). If you do not have Kubernetes or Prometheus installed, then Auto Review Apps, Auto Deploy, and Auto Monitoring will be silently skipped. +One all requirements are met, you can go ahead and [enable Auto DevOps](#enablingdisabling-auto-devops). + ## Auto DevOps base domain -The Auto DevOps base domain is required if you want to make use of [Auto -Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It can be defined -in any of the following places: +The Auto DevOps base domain is required if you want to make use of +[Auto Review Apps](#auto-review-apps), [Auto Deploy](#auto-deploy), and +[Auto Monitoring](#auto-monitoring). It can be defined in any of the following +places: - either under the cluster's settings, whether for [projects](../../user/project/clusters/index.md#base-domain) or [groups](../../user/group/clusters/index.md#base-domain) - or in instance-wide settings in the **admin area > Settings** under the "Continuous Integration and Delivery" section @@ -156,9 +183,15 @@ in any of the following places: The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of precedence as other environment [variables](../../ci/variables/README.md#priority-of-environment-variables). -NOTE: **Note** -`AUTO_DEVOPS_DOMAIN` environment variable is deprecated and -[is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-foss/issues/56959). +TIP: **Tip:** +If you're using the [GitLab managed app for Ingress](../../user/clusters/applications.md#ingress), +the URL endpoint should be automatically configured for you. All you have to do +is use its value for the `KUBE_INGRESS_BASE_DOMAIN` variable. + +NOTE: **Note:** +`AUTO_DEVOPS_DOMAIN` was [deprecated in GitLab 11.8](https://gitlab.com/gitlab-org/gitlab-foss/issues/52363) +and replaced with `KUBE_INGRESS_BASE_DOMAIN`. It was removed in +[GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-foss/issues/56959). A wildcard DNS A record matching the base domain(s) is required, for example, given a base domain of `example.com`, you'd need a DNS entry like: @@ -179,77 +212,28 @@ Auto DevOps base domain to `1.2.3.4.nip.io`. Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s). -NOTE: **Note:** -From GitLab 11.8, `KUBE_INGRESS_BASE_DOMAIN` replaces `AUTO_DEVOPS_DOMAIN`. -Support for `AUTO_DEVOPS_DOMAIN` was [removed in GitLab -12.0](https://gitlab.com/gitlab-org/gitlab-foss/issues/56959). - -## Using multiple Kubernetes clusters **(PREMIUM)** - -When using Auto DevOps, you may want to deploy different environments to -different Kubernetes clusters. This is possible due to the 1:1 connection that -[exists between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters-premium). - -In the [Auto DevOps template] (used behind the scenes by Auto DevOps), there -are currently 3 defined environment names that you need to know: - -- `review/` (every environment starting with `review/`) -- `staging` -- `production` - -Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so -except for the environment scope, they would also need to have a different -domain they would be deployed to. This is why you need to define a separate -`KUBE_INGRESS_BASE_DOMAIN` variable for all the above -[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables). - -The following table is an example of how the three different clusters would -be configured. - -| Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes | -|--------------|---------------------------|-------------------------------------------|----------------------------|---| -| review | `review/*` | `review.example.com` | `review/*` | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. | -| staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). | -| production | `production` | `example.com` | `production` | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](#incremental-rollout-to-production-premium). | - -To add a different cluster for each environment: - -1. Navigate to your project's **Operations > Kubernetes** and create the Kubernetes clusters - with their respective environment scope as described from the table above. - -  - -1. After the clusters are created, navigate to each one and install Helm Tiller - and Ingress. Wait for the Ingress IP address to be assigned. -1. Make sure you have [configured your DNS](#auto-devops-base-domain) with the - specified Auto DevOps domains. -1. Navigate to each cluster's page, through **Operations > Kubernetes**, - and add the domain based on its Ingress IP address. - -Now that all is configured, you can test your setup by creating a merge request -and verifying that your app is deployed as a review app in the Kubernetes -cluster with the `review/*` environment scope. Similarly, you can check the -other environments. - ## Enabling/Disabling Auto DevOps -When first using Auto Devops, review the [requirements](#requirements) to ensure all necessary components to make +When first using Auto DevOps, review the [requirements](#requirements) to ensure all necessary components to make full use of Auto DevOps are available. If this is your fist time, we recommend you follow the [quick start guide](quick_start_guide.md). GitLab.com users can enable/disable Auto DevOps at the project-level only. Self-managed users can enable/disable Auto DevOps at the project-level, group-level or instance-level. -### At the instance level (Administrators only) +### At the project level -Even when disabled at the instance level, group owners and project maintainers can still enable -Auto DevOps at the group and project level, respectively. +If enabling, check that your project doesn't have a `.gitlab-ci.yml`, or if one exists, remove it. -1. Go to **Admin area > Settings > Continuous Integration and Deployment**. -1. Toggle the checkbox labeled **Default to Auto DevOps pipeline for all projects**. -1. If enabling, optionally set up the Auto DevOps [base domain](#auto-devops-base-domain) which will be used for Auto Deploy and Auto Review Apps. +1. Go to your project's **Settings > CI/CD > Auto DevOps**. +1. Toggle the **Default to Auto DevOps pipeline** checkbox (checked to enable, unchecked to disable) +1. When enabling, it's optional but recommended to add in the [base domain](#auto-devops-base-domain) + that will be used by Auto DevOps to [deploy your application](#auto-deploy) + and choose the [deployment strategy](#deployment-strategy). 1. Click **Save changes** for the changes to take effect. +When the feature has been enabled, an Auto DevOps pipeline is triggered on the default branch. + ### At the group level > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/52447) in GitLab 11.10. @@ -266,19 +250,16 @@ When enabling or disabling Auto DevOps at group-level, group configuration will the subgroups and projects inside that group, unless Auto DevOps is specifically enabled or disabled on the subgroup or project. -### At the project level +### At the instance level (Administrators only) -If enabling, check that your project doesn't have a `.gitlab-ci.yml`, or if one exists, remove it. +Even when disabled at the instance level, group owners and project maintainers can still enable +Auto DevOps at the group and project level, respectively. -1. Go to your project's **Settings > CI/CD > Auto DevOps**. -1. Toggle the **Default to Auto DevOps pipeline** checkbox (checked to enable, unchecked to disable) -1. When enabling, it's optional but recommended to add in the [base domain](#auto-devops-base-domain) - that will be used by Auto DevOps to [deploy your application](#auto-deploy) - and choose the [deployment strategy](#deployment-strategy). +1. Go to **Admin area > Settings > Continuous Integration and Deployment**. +1. Toggle the checkbox labeled **Default to Auto DevOps pipeline for all projects**. +1. If enabling, optionally set up the Auto DevOps [base domain](#auto-devops-base-domain) which will be used for Auto Deploy and Auto Review Apps. 1. Click **Save changes** for the changes to take effect. -When the feature has been enabled, an Auto DevOps pipeline is triggered on the default branch. - ### Enable for a percentage of projects There is also a feature flag to enable Auto DevOps by default to your chosen percentage of projects. @@ -310,6 +291,53 @@ The available options are: - `master` branch is directly deployed to staging. - Manual actions are provided for incremental rollout to production. +## Using multiple Kubernetes clusters **(PREMIUM)** + +When using Auto DevOps, you may want to deploy different environments to +different Kubernetes clusters. This is possible due to the 1:1 connection that +[exists between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters-premium). + +In the [Auto DevOps template] (used behind the scenes by Auto DevOps), there +are currently 3 defined environment names that you need to know: + +- `review/` (every environment starting with `review/`) +- `staging` +- `production` + +Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so +except for the environment scope, they would also need to have a different +domain they would be deployed to. This is why you need to define a separate +`KUBE_INGRESS_BASE_DOMAIN` variable for all the above +[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables). + +The following table is an example of how the three different clusters would +be configured. + +| Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes | +|--------------|---------------------------|-------------------------------------------|----------------------------|---| +| review | `review/*` | `review.example.com` | `review/*` | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. | +| staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). | +| production | `production` | `example.com` | `production` | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](#incremental-rollout-to-production-premium). | + +To add a different cluster for each environment: + +1. Navigate to your project's **Operations > Kubernetes** and create the Kubernetes clusters + with their respective environment scope as described from the table above. + +  + +1. After the clusters are created, navigate to each one and install Helm Tiller + and Ingress. Wait for the Ingress IP address to be assigned. +1. Make sure you have [configured your DNS](#auto-devops-base-domain) with the + specified Auto DevOps domains. +1. Navigate to each cluster's page, through **Operations > Kubernetes**, + and add the domain based on its Ingress IP address. + +Now that all is configured, you can test your setup by creating a merge request +and verifying that your app is deployed as a review app in the Kubernetes +cluster with the `review/*` environment scope. Similarly, you can check the +other environments. + ## Stages of Auto DevOps The following sections describe the stages of Auto DevOps. Read them carefully @@ -670,9 +698,28 @@ workers: terminationGracePeriodSeconds: 60 ``` -### Auto Monitoring +#### Running commands in the container + +Applications built with [Auto Build](#auto-build) using Herokuish, the default +unless you have [a custom Dockerfile](#auto-build-using-a-dockerfile), may require +commands to be wrapped as follows: + +```shell +/bin/herokuish procfile exec $COMMAND +``` + +This might be neccessary, for example, when: + +- Attaching using `kubectl exec`. +- Using GitLab's [Web Terminal](../../ci/environments.md#web-terminals). + +For example, to start a Rails console from the application root directory, run: + +```sh +/bin/herokuish procfile exec bin/rails c +``` -See the [requirements](#requirements) for Auto Monitoring to enable this stage. +### Auto Monitoring Once your application is deployed, Auto Monitoring makes it possible to monitor your application's server and response metrics right out of the box. Auto @@ -687,18 +734,15 @@ The metrics include: - **Response Metrics:** latency, throughput, error rate - **System Metrics:** CPU utilization, memory utilization -In order to make use of monitoring you need to: +To make use of Auto Monitoring: -1. [Deploy Prometheus](../../user/project/integrations/prometheus.md) into your Kubernetes cluster -1. If you would like response metrics, ensure you are running at least version - 0.9.0 of NGINX Ingress and - [enable Prometheus metrics](https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/customization/custom-vts-metrics-prometheus/nginx-vts-metrics-conf.yaml). -1. Finally, [annotate](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) - the NGINX Ingress deployment to be scraped by Prometheus using - `prometheus.io/scrape: "true"` and `prometheus.io/port: "10254"`. - -To view the metrics, open the -[Monitoring dashboard for a deployed environment](../../ci/environments.md#monitoring-environments). +1. [Install and configure the requirements](#requirements). +1. [Enable Auto DevOps](#enablingdisabling-auto-devops) if you haven't done already. +1. Finally, go to your project's **CI/CD > Pipelines** and run a pipeline. +1. Once the pipeline finishes successfully, open the + [monitoring dashboard for a deployed environment](../../ci/environments.md#monitoring-environments) + to view the metrics of your deployed application. To view the metrics of the + whole Kubernetes cluster, navigate to **Operations > Metrics**.  @@ -725,6 +769,8 @@ or a `.buildpacks` file in your project: CAUTION: **Caution:** Using multiple buildpacks isn't yet supported by Auto DevOps. +CAUTION: **Caution:** When using the `.buildpacks` file, Auto Test will not work. The buildpack [heroku-buildpack-multi](https://github.com/heroku/heroku-buildpack-multi/) (which is used under the hood to parse the `.buildpacks` file) doesn't provide the necessary commands `bin/test-compile` and `bin/test`. Make sure to provide the project variable `BUILDPACK_URL` instead. + ### Custom `Dockerfile` If your project has a `Dockerfile` in the root of the project repo, Auto DevOps @@ -924,6 +970,7 @@ applications. | `AUTO_DEVOPS_CHART_REPOSITORY_NAME` | From GitLab 11.11, used to set the name of the Helm repository. Defaults to `gitlab`. | | `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From GitLab 11.11, used to set a username to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD`. | | `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From GitLab 11.11, used to set a password to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME`. | +| `AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE` | From GitLab 12.5, used in combination with [Modsecurity feature flag](../../user/clusters/applications.md#web-application-firewall-modsecurity) to toggle [Modsecurity's `SecRuleEngine`](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecRuleEngine) behavior. Defaults to `DetectionOnly`. | | `BUILDPACK_URL` | Buildpack's full URL. Can point to either Git repositories or a tarball URL. For Git repositories, it is possible to point to a specific `ref`. For example `https://github.com/heroku/heroku-buildpack-ruby.git#v142`. | | `CANARY_ENABLED` | From GitLab 11.0, used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments-premium). | | `CANARY_PRODUCTION_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. Takes precedence over `CANARY_REPLICAS`. Defaults to 1. | @@ -968,7 +1015,6 @@ The following table lists variables related to security tools. | **Variable** | **Description** | | `SAST_CONFIDENCE_LEVEL` | Minimum confidence level of security issues you want to be reported; `1` for Low, `2` for Medium, `3` for High. Defaults to `3`. | -| `DS_DISABLE_REMOTE_CHECKS` | Whether remote Dependency Scanning checks are disabled. Defaults to `"false"`. Set to `"true"` to disable checks that send data to GitLab central servers. [Read more about remote checks](../../user/application_security/dependency_scanning/index.md#remote-checks). | #### Disable jobs @@ -1301,7 +1347,6 @@ curl --data "value=true" --header "PRIVATE-TOKEN: personal_access_token" https:/ ``` [ce-37115]: https://gitlab.com/gitlab-org/gitlab-foss/issues/37115 -[kubernetes-clusters]: ../../user/project/clusters/index.md [docker-in-docker]: ../../docker/using_docker_build.md#use-docker-in-docker-executor [review-app]: ../../ci/review_apps/index.md [container-registry]: ../../user/packages/container_registry/index.md diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index d9bdd73221f..ce3a3dd5ca6 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -25,7 +25,7 @@ Google account (for example, one that you use to access Gmail, Drive, etc.) or c TIP: **Tip:** Every new Google Cloud Platform (GCP) account receives [$300 in credit](https://console.cloud.google.com/freetrial), and in partnership with Google, GitLab is able to offer an additional $200 for new GCP accounts to get started with GitLab's -Google Kubernetes Engine Integration. All you have to do is [follow this link](https://cloud.google.com/partners/partnercredit/?PCN=a0n60000006Vpz4AAC) and apply for credit. +Google Kubernetes Engine Integration. All you have to do is [follow this link](https://cloud.google.com/partners/partnercredit/?pcn_code=0014M00001h35gDQAQ#contact-form) and apply for credit. ## Creating a new project from a template @@ -212,7 +212,7 @@ under **Settings > CI/CD > Environment variables**. ### Working with branches -Following the [GitLab flow](../../workflow/gitlab_flow.md#working-with-feature-branches), +Following the [GitLab flow](../gitlab_flow.md#working-with-feature-branches), let's create a feature branch that will add some content to the application. Under your repository, navigate to the following file: `app/views/welcome/index.html.erb`. @@ -280,6 +280,6 @@ and customized to fit your workflow. Here are some helpful resources for further 1. [Multiple Kubernetes clusters](index.md#using-multiple-kubernetes-clusters-premium) **(PREMIUM)** 1. [Incremental rollout to production](index.md#incremental-rollout-to-production-premium) **(PREMIUM)** 1. [Disable jobs you don't need with environment variables](index.md#environment-variables) -1. [Use a static IP for your cluster](../../user/project/clusters/index.md#using-a-static-ip) +1. [Use a static IP for your cluster](../../user/clusters/applications.md#using-a-static-ip) 1. [Use your own buildpacks to build your application](index.md#custom-buildpacks) 1. [Prometheus monitoring](../../user/project/integrations/prometheus.md) |