diff options
Diffstat (limited to 'doc/topics')
43 files changed, 1143 insertions, 648 deletions
diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index c1d0a69e1f4..a7611784284 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -6,18 +6,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Authentication **(FREE)** -This page gathers all the resources for the topic **Authentication** within GitLab. +This page gathers all the resources for the topic **Authentication** in GitLab. ## GitLab users - [SSH](../../user/ssh.md) - [Two-factor authentication](../../user/profile/account/two_factor_authentication.md) -- [Why do you keep getting signed out?](../../user/profile/index.md#why-do-you-keep-getting-signed-out) +- [Stay signed in indefinitely](../../user/profile/index.md#stay-signed-in-indefinitely) - **Articles:** - [Support for Universal 2nd Factor Authentication - YubiKeys](https://about.gitlab.com/blog/2016/06/22/gitlab-adds-support-for-u2f/) - [Security Webcast with Yubico](https://about.gitlab.com/blog/2016/08/31/gitlab-and-yubico-security-webcast/) - **Integrations:** - - [GitLab as OAuth2 authentication service provider](../../integration/oauth_provider.md) + - [GitLab as OAuth 2.0 authentication service provider](../../integration/oauth_provider.md) - [GitLab as OpenID Connect identity provider](../../integration/openid_connect_provider.md) ## GitLab administrators @@ -38,7 +38,7 @@ This page gathers all the resources for the topic **Authentication** within GitL ## API -- [OAuth 2 Tokens](../../api/rest/index.md#oauth2-tokens) +- [OAuth 2.0 tokens](../../api/rest/index.md#oauth-20-tokens) - [Personal access tokens](../../api/rest/index.md#personalprojectgroup-access-tokens) - [Project access tokens](../../api/rest/index.md#personalprojectgroup-access-tokens) - [Group access tokens](../../api/rest/index.md#personalprojectgroup-access-tokens) diff --git a/doc/topics/autodevops/cicd_variables.md b/doc/topics/autodevops/cicd_variables.md index b22b4677f24..529f449e452 100644 --- a/doc/topics/autodevops/cicd_variables.md +++ b/doc/topics/autodevops/cicd_variables.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -62,8 +62,7 @@ Use these variables to customize and deploy your build. ## Database variables WARNING: -The default value of `true` for `POSTGRES_ENABLED` was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/387766) -in GitLab 15.8. In [GitLab 16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/343988), the default value will change to `false`. +From [GitLab 16.0](https://gitlab.com/gitlab-org/gitlab/-/issues/343988), `POSTGRES_ENABLED` is no longer set by default. Use these variables to integrate CI/CD with PostgreSQL databases. @@ -71,7 +70,7 @@ Use these variables to integrate CI/CD with PostgreSQL databases. |-----------------------------------------|------------------------------------| | `DB_INITIALIZE` | Used to specify the command to run to initialize the application's PostgreSQL database. Runs inside the application pod. | | `DB_MIGRATE` | Used to specify the command to run to migrate the application's PostgreSQL database. Runs inside the application pod. | -| `POSTGRES_ENABLED` | Whether PostgreSQL is enabled. Defaults to `true` until GitLab 16.0, when the default will change to `false`. Set to `false` to disable the automatic deployment of PostgreSQL. | +| `POSTGRES_ENABLED` | Whether PostgreSQL is enabled. Set to `true` to enable the automatic deployment of PostgreSQL. | | `POSTGRES_USER` | The PostgreSQL user. Defaults to `user`. Set it to use a custom username. | | `POSTGRES_PASSWORD` | The PostgreSQL password. Defaults to `testing-password`. Set it to use a custom password. | | `POSTGRES_DB` | The PostgreSQL database name. Defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/index.md#predefined-cicd-variables). Set it to use a custom database name. | @@ -87,45 +86,39 @@ Use these variables to disable CI/CD jobs. | **Job name** | **CI/CD variable** | **GitLab version** | **Description** | |----------------------------------------|---------------------------------|-----------------------|-----------------| -| `.fuzz_base` | `COVFUZZ_DISABLED` | [From GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34984) | [Read more](../../user/application_security/coverage_fuzzing/index.md) about how `.fuzz_base` provide capability for your own jobs. If the variable is present, your jobs aren't created. | -| `apifuzzer_fuzz` | `API_FUZZING_DISABLED` | [From GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39135) | If the variable is present, the job isn't created. | +| `.fuzz_base` | `COVFUZZ_DISABLED` | [From GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34984) | [Read more](../../user/application_security/coverage_fuzzing/index.md) about how `.fuzz_base` provide capability for your own jobs. The job isn't created if the value is `"true"`. | +| `apifuzzer_fuzz` | `API_FUZZING_DISABLED` | [From GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39135) | The job isn't created if the value is `"true"`. | | `build` | `BUILD_DISABLED` | | If the variable is present, the job isn't created. | | `build_artifact` | `BUILD_DISABLED` | | If the variable is present, the job isn't created. | -| `bandit-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `brakeman-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | +| `brakeman-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | | `canary` | `CANARY_ENABLED` | | This manual job is created if the variable is present. | | `code_intelligence` | `CODE_INTELLIGENCE_DISABLED` | From GitLab 13.6 | If the variable is present, the job isn't created. | -| `code_quality` | `CODE_QUALITY_DISABLED` | | If the variable is present, the job isn't created. | -| `container_scanning` | `CONTAINER_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | -| `dast` | `DAST_DISABLED` | | If the variable is present, the job isn't created. | -| `dast_environment_deploy` | `DAST_DISABLED_FOR_DEFAULT_BRANCH` or `DAST_DISABLED` | [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17789) | If either variable is present, the job isn't created. | -| `dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | -| `eslint-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `flawfinder-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `gemnasium-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | -| `gemnasium-maven-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | -| `gemnasium-python-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | -| `gosec-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `kubesec-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | +| `code_quality` | `CODE_QUALITY_DISABLED` | | The job isn't created if the value is `"true"`. | +| `container_scanning` | `CONTAINER_SCANNING_DISABLED` | | The job isn't created if the value is `"true"`. | +| `dast` | `DAST_DISABLED` | | The job isn't created if the value is `"true"`. | +| `dast_environment_deploy` | `DAST_DISABLED_FOR_DEFAULT_BRANCH` or `DAST_DISABLED` | [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17789) | The job isn't created if the value is `"true"`. | +| `dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | The job isn't created if the value is `"true"`. | +| `flawfinder-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | +| `gemnasium-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | The job isn't created if the value is `"true"`. | +| `gemnasium-maven-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | The job isn't created if the value is `"true"`. | +| `gemnasium-python-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | The job isn't created if the value is `"true"`. | +| `kubesec-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | | `license_management` | `LICENSE_MANAGEMENT_DISABLED` | GitLab 12.7 and earlier | If the variable is present, the job isn't created. Job deprecated [from GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | -| `license_scanning` | `LICENSE_MANAGEMENT_DISABLED` | [From GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | If the variable is present, the job isn't created. | +| `license_scanning` | `LICENSE_MANAGEMENT_DISABLED` | [From GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | The job isn't created if the value is `"true"`.| | `load_performance` | `LOAD_PERFORMANCE_DISABLED` | From GitLab 13.2 | If the variable is present, the job isn't created. | -| `nodejs-scan-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | +| `nodejs-scan-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | | `performance` | `PERFORMANCE_DISABLED` | GitLab 13.12 and earlier | Browser performance. If the variable is present, the job isn't created. Replaced by `browser_performance`. | | `browser_performance` | `BROWSER_PERFORMANCE_DISABLED` | From GitLab 14.0 | Browser performance. If the variable is present, the job isn't created. Replaces `performance`. | -| `phpcs-security-audit-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `pmd-apex-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | +| `phpcs-security-audit-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | +| `pmd-apex-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | | `review` | `REVIEW_DISABLED` | | If the variable is present, the job isn't created. | | `review:stop` | `REVIEW_DISABLED` | | Manual job. If the variable is present, the job isn't created. | -| `sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `sast:container` | `CONTAINER_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | -| `secret_detection` | `SECRET_DETECTION_DISABLED` | From GitLab 13.1 | If the variable is present, the job isn't created. | -| `secret_detection_default_branch` | `SECRET_DETECTION_DISABLED` | [From GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | If the variable is present, the job isn't created. | -| `security-code-scan-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `secrets-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `sobelaw-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `stop_dast_environment` | `DAST_DISABLED_FOR_DEFAULT_BRANCH` or `DAST_DISABLED` | [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17789) | If either variable is present, the job isn't created. | -| `spotbugs-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | +| `secret_detection` | `SECRET_DETECTION_DISABLED` | From GitLab 13.1 | The job isn't created if the value is `"true"`. | +| `secret_detection_default_branch` | `SECRET_DETECTION_DISABLED` | [From GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | The job isn't created if the value is `"true"`. | +| `semgrep-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | +| `sobelow-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | +| `stop_dast_environment` | `DAST_DISABLED_FOR_DEFAULT_BRANCH` or `DAST_DISABLED` | [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17789) | The job isn't created if the value is `"true"`. | +| `spotbugs-sast` | `SAST_DISABLED` | | The job isn't created if the value is `"true"`. | | `test` | `TEST_DISABLED` | | If the variable is present, the job isn't created. | | `staging` | `STAGING_ENABLED` | | The job is created if the variable is present. | | `stop_review` | `REVIEW_DISABLED` | | If the variable is present, the job isn't created. | diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_ec2.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_ec2.md index 6bebe9eb9e3..381326d68ea 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_ec2.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_ec2.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md index 76fd6ad82d8..0d865b5cf0d 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md new file mode 100644 index 00000000000..32a47bb790b --- /dev/null +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md @@ -0,0 +1,301 @@ +--- +stage: Deploy +group: Environments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Use Auto DevOps to deploy an application to Amazon Elastic Kubernetes Service (EKS) + +In this tutorial, we'll help you to get started with [Auto DevOps](../index.md) +through an example of how to deploy an application to Amazon Elastic Kubernetes Service (EKS). + +The tutorial uses the GitLab native Kubernetes integration, so you don't need +to create a Kubernetes cluster manually using the AWS console. + +You can also follow this tutorial on a self-managed instance. +Ensure your own [runners are configured](../../../ci/runners/index.md). + +To deploy a project to EKS: + +1. [Configure your Amazon account](#configure-your-amazon-account) +1. [Create a Kubernetes cluster and deploy the agent](#create-a-kubernetes-cluster) +1. [Create a new project from a template](#create-an-application-project-from-a-template) +1. [Configure the agent](#configure-the-agent) +1. [Install Ingress](#install-ingress) +1. [Configure Auto DevOps](#configure-auto-devops) +1. [Enable Auto DevOps and run the pipeline](#enable-auto-devops-and-run-the-pipeline) +1. [Deploy the application](#deploy-the-application) + +## Configure your Amazon account + +Before you create and connect your Kubernetes cluster to your GitLab project, +you need an [Amazon Web Services account](https://aws.amazon.com/). +Sign in with an existing Amazon account or create a new one. + +## Create a Kubernetes cluster + +To create an new cluster on Amazon EKS: + +- Follow the steps in [Create an Amazon EKS cluster](../../../user/infrastructure/clusters/connect/new_eks_cluster.md). + +If you prefer, you can also create a cluster manually using `eksctl`. + +## Create an application project from a template + +Use a GitLab project template to get started. As the name suggests, +those projects provide a bare-bones application built on some well-known frameworks. + +WARNING: +Create the application project in the group hierarchy at the same level or below the project for cluster management. Otherwise, it fails to [authorize the agent](../../../user/clusters/agent/ci_cd_workflow.md#authorize-the-agent). + +1. On the top bar in GitLab, select the plus icon (**{plus-square}**), and select + **New project/repository**. +1. Select **Create from template**. +1. Select the **Ruby on Rails** template. +1. Give your project a name, optionally a description, and make it public so that + you can take advantage of the features available in the + [GitLab Ultimate plan](https://about.gitlab.com/pricing/). +1. Select **Create project**. + +Now you have an application project you are going to deploy to the EKS cluster. + +## Configure the agent + +Next, we'll configure the GitLab agent for Kubernetes so we can use it to deploy the application project. + +1. Go to the project [we created to manage the cluster](#create-a-kubernetes-cluster). +1. Navigate to the [agent configuration file](../../../user/clusters/agent/install/index.md#create-an-agent-configuration-file) (`.gitlab/agents/eks-agent/config.yaml`) and edit it. +1. Configure `ci_access:projects` attribute. Use the application project path as `id`: + +```yaml +ci_access: + projects: + - id: path/to/application-project +``` + +## Install Ingress + +After your cluster is running, you must install NGINX Ingress Controller as a +load balancer to route traffic from the internet to your application. +Install the NGINX Ingress Controller +through the GitLab [Cluster management project template](../../../user/clusters/management_project_template.md), +or manually via the command line: + +1. Ensure you have `kubectl` and Helm installed on your machine. +1. Create an IAM role to access the cluster. +1. Create an access token to access the cluster. +1. Use `kubectl` to connect to your cluster: + + ```shell + helm upgrade --install ingress-nginx ingress-nginx \ + --repo https://kubernetes.github.io/ingress-nginx \ + --namespace gitlab-managed-apps --create-namespace + + # Check that the ingress controller is installed successfully + kubectl get service ingress-nginx-controller -n gitlab-managed-apps + ``` + +## Configure Auto DevOps + +Follow these steps to configure the base domain and other settings required for Auto DevOps. + +1. A few minutes after you install NGINX, the load balancer obtains an IP address, and you can + get the external IP address with the following command: + + ```shell + kubectl get all -n gitlab-managed-apps --selector app.kubernetes.io/instance=ingress-nginx + ``` + + Replace `gitlab-managed-apps` if you have overwritten your namespace. + + Next, find the actual external IP address for your cluster with the following command: + + ```shell + nslookup [External IP] + ``` + + Where the `[External IP]` is the hostname found with the previous command. + + The IP address might be listed in the `Non-authoritative answer:` section of the response. + + Copy this IP address, as you need it in the next step. + +1. Go back to the application project. +1. On the left sidebar, select **Settings > CI/CD** and expand **Variables**. + - Add a key called `KUBE_INGRESS_BASE_DOMAIN` with the application deployment domain as the value. For this example, use the domain `<IP address>.nip.io`. + - Add a key called `KUBE_NAMESPACE` with a value of the Kubernetes namespace for your deployments to target. You can use different namespaces per environment. Configure the environment, use the environment scope. + - Add a key called `KUBE_CONTEXT` with a value like `path/to/agent/project:eks-agent`. Select the environment scope of your choice. + - Select **Save changes**. + +## Enable Auto DevOps and run the pipeline + +While Auto DevOps is enabled by default, Auto DevOps can be disabled at both +the instance level (for self-managed instances) and the group level. Complete +these steps to enable Auto DevOps if it's disabled: + +1. On the top bar, select **Main menu > Projects** and find the application project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Auto DevOps**. +1. Select **Default to Auto DevOps pipeline** to display more options. +1. In **Deployment strategy**, select your desired [continuous deployment strategy](../requirements.md#auto-devops-deployment-strategy) + to deploy the application to production after the pipeline successfully runs on the default branch. +1. Select **Save changes**. +1. Edit `.gitlab-ci.yml` file to include the Auto DevOps template and commit the change to the default branch: + + ```yaml + include: + - template: Auto-DevOps.gitlab-ci.yml + ``` + +The commit should trigger a pipeline. In the next section, we explain what each job does in the pipeline. + +## Deploy the application + +When your pipeline runs, what is it doing? + +To view the jobs in the pipeline, select the pipeline's status badge. The +**{status_running}** icon displays when pipeline jobs are running, and updates +without refreshing the page to **{status_success}** (for success) or +**{status_failed}** (for failure) when the jobs complete. + +The jobs are separated into stages: + + + +- **Build** - The application builds a Docker image and uploads it to your project's + [Container Registry](../../../user/packages/container_registry/index.md) ([Auto Build](../stages.md#auto-build)). +- **Test** - GitLab runs various checks on the application, but all jobs except `test` + are allowed to fail in the test stage: + + - The `test` job runs unit and integration tests by detecting the language and + framework ([Auto Test](../stages.md#auto-test-deprecated)) + - The `code_quality` job checks the code quality and is allowed to fail + ([Auto Code Quality](../stages.md#auto-code-quality)) + - The `container_scanning` job checks the Docker container if it has any + vulnerabilities and is allowed to fail ([Auto Container Scanning](../stages.md#auto-container-scanning)) + - The `dependency_scanning` job checks if the application has any dependencies + susceptible to vulnerabilities and is allowed to fail + ([Auto Dependency Scanning](../stages.md#auto-dependency-scanning)) + - Jobs suffixed with `-sast` run static analysis on the current code to check for potential + security issues, and are allowed to fail ([Auto SAST](../stages.md#auto-sast)) + - The `secret-detection` job checks for leaked secrets and is allowed to fail ([Auto Secret Detection](../stages.md#auto-secret-detection)) + - The `license_scanning` job searches the application's dependencies to determine each of their + licenses and is allowed to fail + ([Auto License Compliance](../stages.md#auto-license-compliance)) + +- **Review** - Pipelines on the default branch include this stage with a `dast_environment_deploy` job. + To learn more, see [Dynamic Application Security Testing (DAST)](../../../user/application_security/dast/index.md). + +- **Production** - After the tests and checks finish, the application deploys in + Kubernetes ([Auto Deploy](../stages.md#auto-deploy)). + +- **Performance** - Performance tests are run on the deployed application + ([Auto Browser Performance Testing](../stages.md#auto-browser-performance-testing)). + +- **Cleanup** - Pipelines on the default branch include this stage with a `stop_dast_environment` job. + +After running a pipeline, you should view your deployed website and learn how +to monitor it. + +### Monitor your project + +After successfully deploying your application, you can view its website and check +on its health on the **Environments** page by navigating to +**Deployments > Environments**. This page displays details about +the deployed applications, and the right-hand column displays icons that link +you to common environment tasks: + + + +- **Open live environment** (**{external-link}**) - Opens the URL of the application deployed in production +- **Monitoring** (**{chart}**) - Opens the metrics page where Prometheus collects data + about the Kubernetes cluster and how the application + affects it in terms of memory usage, CPU usage, and latency +- **Deploy to** (**{play}** **{chevron-lg-down}**) - Displays a list of environments you can deploy to +- **Terminal** (**{terminal}**) - Opens a [web terminal](../../../ci/environments/index.md#web-terminals-deprecated) + session inside the container where the application is running +- **Re-deploy to environment** (**{repeat}**) - For more information, see + [Retrying and rolling back](../../../ci/environments/index.md#retry-or-roll-back-a-deployment) +- **Stop environment** (**{stop}**) - For more information, see + [Stopping an environment](../../../ci/environments/index.md#stopping-an-environment) + +GitLab displays the [deploy board](../../../user/project/deploy_boards.md) below the +environment's information, with squares representing pods in your +Kubernetes cluster, color-coded to show their status. Hovering over a square on +the deploy board displays the state of the deployment, and selecting the square +takes you to the pod's logs page. + +Although the example shows only one pod hosting the application at the moment, you can add +more pods by defining the [`REPLICAS` CI/CD variable](../cicd_variables.md) +in **Settings > CI/CD > Variables**. + +### Work with branches + +Following the [GitLab flow](../../gitlab_flow.md#working-with-feature-branches), +you should next create a feature branch to add content to your application: + +1. In your project's repository, go to the following file: `app/views/welcome/index.html.erb`. + This file should only contain a paragraph: `<p>You're on Rails!</p>`. +1. Open the GitLab [Web IDE](../../../user/project/web_ide/index.md) to make the change. +1. Edit the file so it contains: + + ```html + <p>You're on Rails! Powered by GitLab Auto DevOps.</p> + ``` + +1. Stage the file. Add a commit message, then create a new branch and a merge request + by selecting **Commit**. + +  + +After submitting the merge request, GitLab runs your pipeline, and all the jobs +in it, as [described previously](#deploy-the-application), in addition to +a few more that run only on branches other than the default branch. + +After a few minutes a test fails, which means a test was +'broken' by your change. Select the failed `test` job to see more information +about it: + +```plaintext +Failure: +WelcomeControllerTest#test_should_get_index [/app/test/controllers/welcome_controller_test.rb:7]: +<You're on Rails!> expected but was +<You're on Rails! Powered by GitLab Auto DevOps.>.. +Expected 0 to be >= 1. + +bin/rails test test/controllers/welcome_controller_test.rb:4 +``` + +To fix the broken test: + +1. Return to your merge request. +1. In the upper right corner, select **Code**, then select **Open in Gitpod**. +1. In the left-hand directory of files, find the `test/controllers/welcome_controller_test.rb` + file, and select it to open it. +1. Change line 7 to say `You're on Rails! Powered by GitLab Auto DevOps.` +1. Select **Commit**. +1. In the left-hand column, under **Unstaged changes**, select the checkmark icon + (**{stage-all}**) to stage the changes. +1. Write a commit message, and select **Commit**. + +Return to the **Overview** page of your merge request, and you should not only +see the test passing, but also the application deployed as a +[review application](../stages.md#auto-review-apps). You can visit it by selecting +the **View app** **{external-link}** button to see your changes deployed. + +After merging the merge request, GitLab runs the pipeline on the default branch, +and then deploys the application to production. + +## Conclusion + +After implementing this project, you should have a solid understanding of the basics of Auto DevOps. +You started from building and testing, to deploying and monitoring an application +all in GitLab. Despite its automatic nature, Auto DevOps can also be configured +and customized to fit your workflow. Here are some helpful resources for further reading: + +1. [Auto DevOps](../index.md) +1. [Multiple Kubernetes clusters](../multiple_clusters_auto_devops.md) +1. [Incremental rollout to production](../cicd_variables.md#incremental-rollout-to-production) +1. [Disable jobs you don't need with CI/CD variables](../cicd_variables.md) +1. [Use your own buildpacks to build your application](../customize.md#custom-buildpacks) +1. [Prometheus monitoring](../../../user/project/integrations/prometheus.md) diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md index 9bd1d30e1b1..1a03a66d17a 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -221,7 +221,7 @@ you to common environment tasks: - **Re-deploy to environment** (**{repeat}**) - For more information, see [Retrying and rolling back](../../../ci/environments/index.md#retry-or-roll-back-a-deployment) - **Stop environment** (**{stop}**) - For more information, see - [Stopping an environment](../../../ci/environments/index.md#stop-an-environment) + [Stopping an environment](../../../ci/environments/index.md#stopping-an-environment) GitLab displays the [deploy board](../../../user/project/deploy_boards.md) below the environment's information, with squares representing pods in your @@ -274,7 +274,7 @@ bin/rails test test/controllers/welcome_controller_test.rb:4 To fix the broken test: 1. Return to your merge request. -1. In the upper right corner, select **Code**, then select **Open in Gitpod**. +1. In the upper-right corner, select **Code**, then select **Open in Gitpod**. 1. In the left-hand directory of files, find the `test/controllers/welcome_controller_test.rb` file, and select it to open it. 1. Change line 7 to say `You're on Rails! Powered by GitLab Auto DevOps.` diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index b8ea242df1a..e7fe4ce6e0b 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 588be855659..87810193a8d 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -185,6 +185,7 @@ and clear the **Default to Auto DevOps pipeline** checkbox. ### Deploy your app to a cloud provider - [Use Auto DevOps to deploy to a Kubernetes cluster on Google Kubernetes Engine (GKE)](cloud_deployments/auto_devops_with_gke.md) +- [Use Auto DevOps to deploy to a Kubernetes cluster on Amazon Elastic Kubernetes Service (EKS)](cloud_deployments/auto_devops_with_eks.md) - [Use Auto DevOps to deploy to EC2](cloud_deployments/auto_devops_with_ec2.md) - [Use Auto DevOps to deploy to ECS](cloud_deployments/auto_devops_with_ecs.md) diff --git a/doc/topics/autodevops/multiple_clusters_auto_devops.md b/doc/topics/autodevops/multiple_clusters_auto_devops.md index 3411beedefb..0b1699d8230 100644 --- a/doc/topics/autodevops/multiple_clusters_auto_devops.md +++ b/doc/topics/autodevops/multiple_clusters_auto_devops.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/autodevops/prepare_deployment.md b/doc/topics/autodevops/prepare_deployment.md index ebba20ca821..6f434e8fb84 100644 --- a/doc/topics/autodevops/prepare_deployment.md +++ b/doc/topics/autodevops/prepare_deployment.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/autodevops/requirements.md b/doc/topics/autodevops/requirements.md index a409c6f1520..dc126edf1aa 100644 --- a/doc/topics/autodevops/requirements.md +++ b/doc/topics/autodevops/requirements.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -63,8 +63,8 @@ To define the base domain, either: - In the project or group level: add it as an environment variable: `KUBE_INGRESS_BASE_DOMAIN`. - In the instance level: go to **Main menu > Admin > Settings > CI/CD > Continuous Integration and Delivery** and add it there. -The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of precedence -as other environment [variables](../../ci/variables/index.md#cicd-variable-precedence). +The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of +[precedence as other environment variables](../../ci/variables/index.md#cicd-variable-precedence). If you don't specify the base domain in your projects and groups, Auto DevOps uses the instance-wide **Auto DevOps domain**. @@ -108,10 +108,7 @@ To make full use of Auto DevOps with Kubernetes, you need: or manually by using the [`ingress-nginx`](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx) Helm chart. - NOTE: - For metrics to appear when using the [Prometheus cluster integration](../../user/clusters/integrations.md#prometheus-cluster-integration), you must [enable Prometheus metrics](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx#prometheus-metrics). - - When deploying [using custom charts](customize.md#custom-helm-chart), you must also + When deploying [using custom charts](customize.md#custom-helm-chart), you must [annotate](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) the Ingress manifest to be scraped by Prometheus using `prometheus.io/scrape: "true"` and `prometheus.io/port: "10254"`. @@ -142,21 +139,6 @@ To make full use of Auto DevOps with Kubernetes, you need: for the entire GitLab instance, or [project runners](../../ci/runners/runners_scope.md#project-runners) that are assigned to specific projects. -- **Prometheus** (for [Auto Monitoring](stages.md#auto-monitoring)) - - To enable Auto Monitoring, you need Prometheus installed either inside or - outside your cluster, and configured to scrape your Kubernetes cluster. - If you've configured the GitLab integration with Kubernetes, you can - instruct GitLab to query an in-cluster Prometheus by enabling - the [Prometheus cluster integration](../../user/clusters/integrations.md#prometheus-cluster-integration). - - The [Prometheus integration](../../user/project/integrations/prometheus.md) - integration must be activated for the project, or activated at the group or instance level. - For more information, see [Project integration management](../../user/admin_area/settings/project_integration_management.md). - - To get response metrics (in addition to system metrics), you must - [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring). - - **cert-manager** (optional, for TLS/HTTPS) To enable HTTPS endpoints for your application, you can [install cert-manager](https://cert-manager.io/docs/installation/supported-releases/), diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index 0c5c87cdf0f..c95426b03e6 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -491,7 +491,7 @@ deletion). You can configure database initialization and migrations for PostgreSQL to run within the application pod by setting the project CI/CD variables `DB_INITIALIZE` and -`DB_MIGRATE` respectively. +`DB_MIGRATE`. If present, `DB_INITIALIZE` is run as a shell command within an application pod as a Helm post-install hook. As some applications can't run without a successful @@ -614,8 +614,7 @@ To use Auto Monitoring: 1. Select **Run pipeline**. 1. After the pipeline finishes successfully, open the [monitoring dashboard for a deployed environment](../../ci/environments/index.md#monitor-environments) - to view the metrics of your deployed application. To view the metrics of the - whole Kubernetes cluster, on the left sidebar, select **Monitor > Metrics**. + to view the metrics of your deployed application.  diff --git a/doc/topics/autodevops/troubleshooting.md b/doc/topics/autodevops/troubleshooting.md index dd715e95512..21a74cca07a 100644 --- a/doc/topics/autodevops/troubleshooting.md +++ b/doc/topics/autodevops/troubleshooting.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md index 858562eef48..0c31ccbfe32 100644 --- a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md +++ b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md @@ -1,6 +1,6 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- @@ -52,7 +52,7 @@ The following table explains the version compatibility between GitLab and Auto D | GitLab version | `auto-deploy-image` version | Notes | |------------------|-----------------------------|-------| | v10.0 to v14.0 | v0.1.0 to v2.0.0 | v0 and v1 auto-deploy-image are backwards compatible. | -| v13.4 and higher | v2.0.0 and higher | v2 auto-deploy-image contains breaking changes, as explained in the [upgrade guide](#upgrade-deployments-to-the-v2-auto-deploy-image). | +| v13.4 and later | v2.0.0 and later | v2 auto-deploy-image contains breaking changes, as explained in the [upgrade guide](#upgrade-deployments-to-the-v2-auto-deploy-image). | You can find the current stable version of auto-deploy-image in the [Auto Deploy stable template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). @@ -179,7 +179,7 @@ used the `v0.17.0` chart, add `AUTO_DEVOPS_FORCE_DEPLOY_V2`. ## Early adopters -If you want to use the latest [Beta](../../policy/alpha-beta-support.md#beta-features) or unstable version of `auto-deploy-image`, include +If you want to use the latest [Beta](../../policy/alpha-beta-support.md#beta) or unstable version of `auto-deploy-image`, include the latest Auto Deploy template into your `.gitlab-ci.yml`: ```yaml @@ -189,7 +189,7 @@ include: ``` WARNING: -Using a [Beta](../../policy/alpha-beta-support.md#beta-features) or unstable `auto-deploy-image` could cause unrecoverable damage to +Using a [Beta](../../policy/alpha-beta-support.md#beta) or unstable `auto-deploy-image` could cause unrecoverable damage to your environments. Do not test it with important projects or environments. The next stable template update is [planned for GitLab v14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/232788). diff --git a/doc/topics/autodevops/upgrading_postgresql.md b/doc/topics/autodevops/upgrading_postgresql.md index f18d5c49be5..ffb72b802f7 100644 --- a/doc/topics/autodevops/upgrading_postgresql.md +++ b/doc/topics/autodevops/upgrading_postgresql.md @@ -1,13 +1,13 @@ --- -stage: Configure -group: Configure +stage: Deploy +group: Environments info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Upgrading PostgreSQL for Auto DevOps **(FREE)** -Auto DevOps provides an [in-cluster PostgreSQL database](customize.md#postgresql-database-support) -for your application. +When `POSTGRES_ENABLED` is `true`, Auto DevOps provides an +[in-cluster PostgreSQL database](customize.md#postgresql-database-support) for your application. The version of the chart used to provision PostgreSQL: @@ -49,12 +49,12 @@ being modified after the database dump is created. 1. Get the Kubernetes namespace for the environment. It typically looks like `<project-name>-<project-id>-<environment>`. In our example, the namespace is called `minimal-ruby-app-4349298-production`. - ```shell - $ kubectl get ns + ```shell + $ kubectl get ns - NAME STATUS AGE - minimal-ruby-app-4349298-production Active 7d14h - ``` + NAME STATUS AGE + minimal-ruby-app-4349298-production Active 7d14h + ``` 1. For ease of use, export the namespace name: @@ -64,20 +64,20 @@ being modified after the database dump is created. 1. Get the deployment name for your application with the following command. In our example, the deployment name is `production`. - ```shell - $ kubectl get deployment --namespace "$APP_NAMESPACE" - NAME READY UP-TO-DATE AVAILABLE AGE - production 2/2 2 2 7d21h - production-postgres 1/1 1 1 7d21h - ``` + ```shell + $ kubectl get deployment --namespace "$APP_NAMESPACE" + NAME READY UP-TO-DATE AVAILABLE AGE + production 2/2 2 2 7d21h + production-postgres 1/1 1 1 7d21h + ``` 1. To prevent the database from being modified, set replicas to 0 for the deployment with the following command. We use the deployment name from the previous step (`deployments/<DEPLOYMENT_NAME>`). - ```shell - $ kubectl scale --replicas=0 deployments/production --namespace "$APP_NAMESPACE" - deployment.extensions/production scaled - ``` + ```shell + $ kubectl scale --replicas=0 deployments/production --namespace "$APP_NAMESPACE" + deployment.extensions/production scaled + ``` 1. You must also set replicas to zero for workers if you have any. @@ -85,26 +85,26 @@ being modified after the database dump is created. 1. Get the service name for PostgreSQL. The name of the service should end with `-postgres`. In our example the service name is `production-postgres`. - ```shell - $ kubectl get svc --namespace "$APP_NAMESPACE" - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - production-auto-deploy ClusterIP 10.30.13.90 <none> 5000/TCP 7d14h - production-postgres ClusterIP 10.30.4.57 <none> 5432/TCP 7d14h - ``` + ```shell + $ kubectl get svc --namespace "$APP_NAMESPACE" + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + production-auto-deploy ClusterIP 10.30.13.90 <none> 5000/TCP 7d14h + production-postgres ClusterIP 10.30.4.57 <none> 5432/TCP 7d14h + ``` 1. Get the pod name for PostgreSQL with the following command. In our example, the pod name is `production-postgres-5db86568d7-qxlxv`. - ```shell - $ kubectl get pod --namespace "$APP_NAMESPACE" -l app=production-postgres - NAME READY STATUS RESTARTS AGE - production-postgres-5db86568d7-qxlxv 1/1 Running 0 7d14h - ``` + ```shell + $ kubectl get pod --namespace "$APP_NAMESPACE" -l app=production-postgres + NAME READY STATUS RESTARTS AGE + production-postgres-5db86568d7-qxlxv 1/1 Running 0 7d14h + ``` 1. Connect to the pod with: - ```shell - kubectl exec -it production-postgres-5db86568d7-qxlxv --namespace "$APP_NAMESPACE" -- bash - ``` + ```shell + kubectl exec -it production-postgres-5db86568d7-qxlxv --namespace "$APP_NAMESPACE" -- bash + ``` 1. Once, connected, create a dump file with the following command. @@ -114,20 +114,20 @@ being modified after the database dump is created. - When prompted for the database password, the default is `testing-password`. - ```shell - ## Format is: - # pg_dump -h SERVICE_NAME -U USERNAME DATABASE_NAME > /tmp/backup.sql + ```shell + ## Format is: + # pg_dump -h SERVICE_NAME -U USERNAME DATABASE_NAME > /tmp/backup.sql - pg_dump -h production-postgres -U user production > /tmp/backup.sql - ``` + pg_dump -h production-postgres -U user production > /tmp/backup.sql + ``` 1. Once the backup dump is complete, exit the Kubernetes exec process with `Control-D` or `exit`. 1. Download the dump file with the following command: - ```shell - kubectl cp --namespace "$APP_NAMESPACE" production-postgres-5db86568d7-qxlxv:/tmp/backup.sql backup.sql - ``` + ```shell + kubectl cp --namespace "$APP_NAMESPACE" production-postgres-5db86568d7-qxlxv:/tmp/backup.sql backup.sql + ``` ## Retain persistent volumes @@ -184,8 +184,7 @@ You can also 1. Set `AUTO_DEVOPS_POSTGRES_DELETE_V1` to a non-empty value. This flag is a safeguard to prevent accidental deletion of databases. <!-- DO NOT REPLACE when upgrading GitLab's supported version. This is NOT related to GitLab's PostgreSQL version support, but the one deployed by Auto DevOps. --> -1. If you have a `POSTGRES_VERSION` set, make sure it is set to `9.6.16` *or -higher*. This is the +1. If you have a `POSTGRES_VERSION` set, make sure it is set to `9.6.16` *or later*. This is the minimum PostgreSQL version supported by Auto DevOps. See also the list of [tags available](https://hub.docker.com/r/bitnami/postgresql/tags). 1. Set `PRODUCTION_REPLICAS` to `0`. For other environments, use @@ -205,11 +204,11 @@ higher*. This is the 1. Get the pod name for the new PostgreSQL, in our example, the pod name is `production-postgresql-0`: - ```shell - $ kubectl get pod --namespace "$APP_NAMESPACE" -l app=postgresql - NAME READY STATUS RESTARTS AGE - production-postgresql-0 1/1 Running 0 19m - ```` + ```shell + $ kubectl get pod --namespace "$APP_NAMESPACE" -l app=postgresql + NAME READY STATUS RESTARTS AGE + production-postgresql-0 1/1 Running 0 19m + ```` 1. Copy the dump file from the backup steps to the pod: diff --git a/doc/topics/awesome_co.md b/doc/topics/awesome_co.md deleted file mode 100644 index ff3c81a1b90..00000000000 --- a/doc/topics/awesome_co.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -stage: Manage -group: Foundations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -description: AwesomeCo test data harness created by the Test Data Working Group https://about.gitlab.com/company/team/structure/working-groups/demo-test-data/ -comments: false ---- - -# AwesomeCo - -AwesomeCo is a test data seeding harness, that can seed test data into a user or group namespace. - -AwesomeCo uses FactoryBot in the backend which makes maintenance extremely easy. When a Model is changed, -FactoryBot will already be reflected to account for the change. - -## Docker Setup - -See [AwesomeCo Docker Demo](https://gitlab.com/-/snippets/2390362) - -## GDK Setup - -```shell -$ gdk start db -ok: run: services/postgresql: (pid n) 0s, normally down -ok: run: services/redis: (pid n) 74s, normally down -$ bundle install -Bundle complete! -$ bundle exec rake db:migrate -main: migrated -ci: migrated -``` - -### Run - -The `ee:gitlab:seed:awesome_co` Rake task takes two arguments. `:name` and `:namespace_id`. - -```shell -$ bundle exec rake "ee:gitlab:seed:awesome_co[awesome_co,1]" -Seeding AwesomeCo for Administrator -``` - -#### `:name` - -Where `:name` is the name of the AwesomeCo. (This will reflect .rb files located in db/seeds/awesome_co/*.rb) - -#### `:namespace_id` - -Where `:namespace_id` is the ID of the User or Group Namespace - -## List of Awesome Companies - -Each company (i.e. test data template) is represented as a Ruby file (.rb) in `db/seeds/awesome_co`. - -### AwesomeCo (db/seeds/awesome_co/awesome_co.rb) - -```shell -$ bundle exec rake "ee:gitlab:seed:awesome_co[awesome_co,:namespace_id]" -Seeding AwesomeCo for :namespace_id -``` - -AwesomeCo is an automated seeding of [this demo repository](https://gitlab.com/tech-marketing/demos/gitlab-agile-demo/awesome-co). - -## Develop - -AwesomeCo seeding uses FactoryBot definitions from `spec/factories` which ... - -1. Saves time on development -1. Are easy-to-read -1. Are easy to maintain -1. Do not rely on an API that may change in the future -1. Are always up-to-date -1. Execute on the lowest-level (`ActiveRecord`) possible to create data as quickly as possible - -> From the [FactoryBot README](https://github.com/thoughtbot/factory_bot#readme_) : `factory_bot` is a fixtures replacement with a straightforward definition syntax, support for multiple build -> strategies (saved instances, unsaved instances, attribute hashes, and stubbed objects), and support for multiple factories for the same class, including factory -> inheritance - -Factories reside in `spec/factories/*` and are fixtures for Rails models found in `app/models/*`. For example, For a model named `app/models/issue.rb`, the factory will -be named `spec/factories/issues.rb`. For a model named `app/models/project.rb`, the factory will be named `app/models/projects.rb`. - -### Taxonomy of a Factory - -Factories consist of three main parts - the **Name** of the factory, the **Traits** and the **Attributes**. - -Given: `create(:iteration, :with_title, :current, title: 'My Iteration')` - -||| -|:-|:-| -| **:iteration** | This is the **Name** of the factory. The file name will be the plural form of this **Name** and reside under either `spec/factories/iterations.rb` or `ee/spec/factories/iterations.rb`. | -| **:with_title** | This is a **Trait** of the factory. [See how it's defined](https://gitlab.com/gitlab-org/gitlab/-/blob/9c2a1f98483921dd006d70fdaed316e21fc5652f/ee/spec/factories/iterations.rb#L21-23). | -| **:current** | This is a **Trait** of the factory. [See how it's defined](https://gitlab.com/gitlab-org/gitlab/-/blob/9c2a1f98483921dd006d70fdaed316e21fc5652f/ee/spec/factories/iterations.rb#L29-31). | -| **title: 'My Iteration'** | This is an **Attribute** of the factory that will be passed to the Model for creation. | - -### Examples - -In these examples, you will see an instance variable `@owner`. This is the `root` user (`User.first`). - -#### Create a Group - -```ruby -my_group = create(:group, name: 'My Group', path: 'my-group-path') -``` - -#### Create a Project - -```ruby -# create a Project belonging to a Group -my_project = create(:project, :public, name: 'My Project', namespace: my_group, creator: @owner) -``` - -#### Create an Issue - -```ruby -# create an Issue belonging to a Project -my_issue = create(:issue, title: 'My Issue', project: my_project, weight: 2) -``` - -#### Create an Iteration - -```ruby -# create an Iteration under a Group -my_iteration = create(:iteration, :with_title, :current, title: 'My Iteration', group: my_group) -``` - -### Frequently encountered issues - -#### ActiveRecord::RecordInvalid: Validation failed: Email has already been taken, Username has already been taken - -This is because, by default, our factories are written to backfill any data that is missing. For instance, when a project -is created, the project must have somebody that created it. If the owner is not specified, the factory attempts to create it. - -**How to fix** - -Check the respective Factory to find out what key is required. Usually `:author` or `:owner`. - -```ruby -# This throws ActiveRecord::RecordInvalid -create(:project, name: 'Throws Error', namespace: create(:group, name: 'Some Group')) - -# Specify the user where @owner is a [User] record -create(:project, name: 'No longer throws error', owner: @owner, namespace: create(:group, name: 'Some Group')) -create(:epic, group: create(:group), author: @owner) -``` diff --git a/doc/topics/data_seeder.md b/doc/topics/data_seeder.md new file mode 100644 index 00000000000..19c0e05d8ed --- /dev/null +++ b/doc/topics/data_seeder.md @@ -0,0 +1,331 @@ +--- +stage: Manage +group: Foundations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +description: Data Seeder test data harness created by the Test Data Working Group https://about.gitlab.com/company/team/structure/working-groups/demo-test-data/ +--- + +# GitLab Data Seeder + +GitLab Data Seeder (GDS) is a test data seeding harness, that can seed test data into a user or group namespace. + +The Data Seeder uses FactoryBot in the backend which makes maintenance extremely easy. When a Model is changed, +FactoryBot will already be reflected to account for the change. + +## Docker Setup + +See [Data Seeder Docker Demo](https://gitlab.com/-/snippets/2390362) + +## GDK Setup + +```shell +$ gdk start db +ok: run: services/postgresql: (pid n) 0s, normally down +ok: run: services/redis: (pid n) 74s, normally down +$ bundle install +Bundle complete! +$ bundle exec rake db:migrate +main: migrated +ci: migrated +``` + +### Run + +The `ee:gitlab:seed:data_seeder` Rake task takes two arguments. `:name` and `:namespace_id`. + +```shell +$ bundle exec rake "ee:gitlab:seed:data_seeder[data_seeder,1]" +Seeding Data for Administrator +``` + +#### `:name` + +Where `:name` is the file name. (This will reflect relative `.rb`, `.yml`, or `.json` files located in `ee/db/seeds/data_seeder`, or absolute paths to seed files) + +#### `:namespace_id` + +Where `:namespace_id` is the ID of the User or Group Namespace + +## Develop + +The Data Seeder uses FactoryBot definitions from `spec/factories` which ... + +1. Saves time on development +1. Are easy-to-read +1. Are easy to maintain +1. Do not rely on an API that may change in the future +1. Are always up-to-date +1. Execute on the lowest-level (`ActiveRecord`) possible to create data as quickly as possible + +> From the [FactoryBot README](https://github.com/thoughtbot/factory_bot#readme_) : `factory_bot` is a fixtures replacement with a straightforward definition syntax, support for multiple build +> strategies (saved instances, unsaved instances, attribute hashes, and stubbed objects), and support for multiple factories for the same class, including factory +> inheritance + +Factories reside in `spec/factories/*` and are fixtures for Rails models found in `app/models/*`. For example, For a model named `app/models/issue.rb`, the factory will +be named `spec/factories/issues.rb`. For a model named `app/models/project.rb`, the factory will be named `app/models/projects.rb`. + +There are currently three parsers that the GitLab Data Seeder supports. Ruby, YAML, and JSON. + +### Ruby + +All Ruby Seeds must define a `DataSeeder` class with a `#seed` instance method. You may structure your Ruby class as you wish. All FactoryBot [methods](https://www.rubydoc.info/gems/factory_bot/FactoryBot/Syntax/Methods) (`create`, `build`, `create_list`) will be included in the class automatically and may be called. + +The `DataSeeder` class will have the following instance variables defined upon seeding: + +- `@seed_file` - The `File` object. +- `@owner` - The owner of the seed data. +- `@name` - The name of the seed. This will be the seed file name without the extension. +- `@group` - The root group that all seeded data will be created under. + +```ruby +# frozen_string_literal: true + +class DataSeeder + def seed + my_group = create(:group, name: 'My Group', path: 'my-group-path', parent: @group) + my_project = create(:project, :public, name: 'My Project', namespace: my_group, creator: @owner) + end +end +``` + +### YAML + +The YAML Parser is a DSL that supports Factory definitions and allows you to seed data using a human-readable format. + +```yaml +name: My Seeder +groups: + - _id: my_group + name: My Group + path: my-group-path + +projects: + - _id: my_project + name: My Project + namespace_id: <%= groups.my_group.id %> + creator_id: <%= @owner.id %> + traits: + - public +``` + +### JSON + +The JSON Parser allows you to house seed files in JSON format. + +```json +{ + "name": "My Seeder", + "groups": [ + { "_id": "my_group", "name": "My Group", "path": "my-group-path" } + ], + "projects": [ + { + "_id": "my_project", + "name": "My Project", + "namespace_id": "<%= groups.my_group.id %>", + "creator_id": "<%= @owner.id %>", + "traits": ["public"] + } + ] +} +``` + +### Taxonomy of a Factory + +Factories consist of three main parts - the **Name** of the factory, the **Traits** and the **Attributes**. + +Given: `create(:iteration, :with_title, :current, title: 'My Iteration')` + +||| +|:-|:-| +| **:iteration** | This is the **Name** of the factory. The file name will be the plural form of this **Name** and reside under either `spec/factories/iterations.rb` or `ee/spec/factories/iterations.rb`. | +| **:with_title** | This is a **Trait** of the factory. [See how it's defined](https://gitlab.com/gitlab-org/gitlab/-/blob/9c2a1f98483921dd006d70fdaed316e21fc5652f/ee/spec/factories/iterations.rb#L21-23). | +| **:current** | This is a **Trait** of the factory. [See how it's defined](https://gitlab.com/gitlab-org/gitlab/-/blob/9c2a1f98483921dd006d70fdaed316e21fc5652f/ee/spec/factories/iterations.rb#L29-31). | +| **title: 'My Iteration'** | This is an **Attribute** of the factory that will be passed to the Model for creation. | + +### Examples + +In these examples, you will see an instance variable `@owner`. This is the `root` user (`User.first`). + +#### Create a Group + +```ruby +my_group = create(:group, name: 'My Group', path: 'my-group-path') +``` + +#### Create a Project + +```ruby +# create a Project belonging to a Group +my_project = create(:project, :public, name: 'My Project', namespace: my_group, creator: @owner) +``` + +#### Create an Issue + +```ruby +# create an Issue belonging to a Project +my_issue = create(:issue, title: 'My Issue', project: my_project, weight: 2) +``` + +#### Create an Iteration + +```ruby +# create an Iteration under a Group +my_iteration = create(:iteration, :with_title, :current, title: 'My Iteration', group: my_group) +``` + +### Frequently encountered issues + +#### ActiveRecord::RecordInvalid: Validation failed: Email has already been taken, Username has already been taken + +This is because, by default, our factories are written to backfill any data that is missing. For instance, when a project +is created, the project must have somebody that created it. If the owner is not specified, the factory attempts to create it. + +**How to fix** + +Check the respective Factory to find out what key is required. Usually `:author` or `:owner`. + +```ruby +# This throws ActiveRecord::RecordInvalid +create(:project, name: 'Throws Error', namespace: create(:group, name: 'Some Group')) + +# Specify the user where @owner is a [User] record +create(:project, name: 'No longer throws error', owner: @owner, namespace: create(:group, name: 'Some Group')) +create(:epic, group: create(:group), author: @owner) +``` + +#### `parsing id "my id" as "my_id"` + +See [specifying variables](#specify-a-variable) + +#### `id is invalid` + +Given that non-Ruby parsers parse IDs as Ruby Objects, the [naming conventions](https://docs.ruby-lang.org/en/2.0.0/syntax/methods_rdoc.html#label-Method+Names) of Ruby must be followed when specifying an ID. + +Examples of invalid IDs: + +- IDs that start with a number +- IDs that have special characters (-, !, $, @, `, =, <, >, ;, :) + +#### ActiveRecord::AssociationTypeMismatch: Model expected, got ... which is an instance of String + +This is currently a limitation for the seeder. + +See the issue for [allowing parsing of raw Ruby objects](https://gitlab.com/gitlab-org/gitlab/-/issues/403079). + +## YAML Factories + +### Generator to generate _n_ amount of records + +### [Group Labels](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/factories/labels.rb) + +```yaml +group_labels: + # Group Label with Name and a Color + - name: Group Label 1 + group_id: <%= @group.id %> + color: "#FF0000" +``` + +### [Group Milestones](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/factories/milestones.rb) + +```yaml +group_milestones: + # Past Milestone + - name: Past Milestone + group_id: <%= @group.id %> + group: + start_date: <%= 1.month.ago %> + due_date: <%= 1.day.ago %> + + # Ongoing Milestone + - name: Ongoing Milestone + group_id: <%= @group.id %> + group: + start_date: <%= 1.day.ago %> + due_date: <%= 1.month.from_now %> + + # Future Milestone + - name: Ongoing Milestone + group_id: <%= @group.id %> + group: + start_date: <%= 1.month.from_now %> + due_date: <%= 2.months.from_now %> +``` + +#### Quirks + +- You _must_ specify `group:` and have it be empty. This is because the Milestones factory will manipulate the factory in an `after(:build)`. If this is not present, the Milestone will not be associated properly with the Group. + +### [Epics](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/factories/epics.rb) + +```yaml +epics: + # Simple Epic + - title: Simple Epic + group_id: <%= @group.id %> + author_id: <%= @owner.id %> + + # Epic with detailed Markdown description + - title: Detailed Epic + group_id: <%= @group.id %> + author_id: <%= @owner.id %> + description: | + # Markdown + + **Description** + + # Epic with dates + - title: Epic with dates + group_id: <%= @group.id %> + author_id: <%= @owner.id %> + start_date: <%= 1.day.ago %> + due_date: <%= 1.month.from_now %> +``` + +## Variables + +Each created factory can be assigned an identifier to be used in future seeding. + +You can specify an ID for any created factory that you may use later in the seed file. + +### Specify a variable + +You may pass an `_id` attribute on any factory to refer back to it later in non-Ruby parsers. + +Variables are under the factory definitions that they reside in. + +```yaml +--- +group_labels: + - _id: my_label #=> group_labels.my_label + +projects: + - _id: my_project #=> projects.my_project +``` + +Variables: + +NOTE: +It is not advised, but you may specify variables with spaces. These variables may be referred back to with underscores. + +### Referencing a variable + +Given a YAML seed file: + +```yaml +--- +group_labels: + - _id: my_group_label #=> group_labels.my_group_label + name: My Group Label + color: "#FF0000" + - _id: my_other_group_label #=> group_labels.my_other_group_label + color: <%= group_labels.my_group_label.color %> + +projects: + - _id: my_project #=> projects.my_project + name: My Project +``` + +When referring to a variable, the variable refers to the _already seeded_ models. In other words, the model's `id` attribute will +be populated. diff --git a/doc/topics/git/bisect.md b/doc/topics/git/bisect.md index a7f8b2a846c..eaf619ce36f 100644 --- a/doc/topics/git/bisect.md +++ b/doc/topics/git/bisect.md @@ -1,76 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false +redirect_to: 'index.md' +remove_date: '2023-06-09' --- -# Bisect **(FREE)** +This document was moved to [another location](index.md). -- Find a commit that introduced a bug -- Works through a process of elimination -- Specify a known good and bad revision to begin - -## Bisect sample workflow - -1. Start the bisect process -1. Enter the bad revision (usually latest commit) -1. Enter a known good revision (commit/branch) -1. Run code to see if bug still exists -1. Tell bisect the result -1. Repeat the previous 2 items until you find the offending commit - -## Setup - -```shell - mkdir bisect-ex - cd bisect-ex - touch index.html - git add -A - git commit -m "starting out" - vi index.html - # Add all good - git add -A - git commit -m "second commit" - vi index.html - # Add all good 2 - git add -A - git commit -m "third commit" - vi index.html -``` - -```shell - # Add all good 3 - git add -A - git commit -m "fourth commit" - vi index.html - # This looks bad - git add -A - git commit -m "fifth commit" - vi index.html - # Really bad - git add -A - git commit -m "sixth commit" - vi index.html - # again just bad - git add -A - git commit -m "seventh commit" -``` - -## Commands - -```shell - git bisect start - # Test your code - git bisect bad - git bisect next - # Say yes to the warning - # Test - git bisect good - # Test - git bisect bad - # Test - git bisect good - # done - git bisect reset -``` +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/cherry_picking.md b/doc/topics/git/cherry_picking.md index ce2a1d4b954..bd589562ed1 100644 --- a/doc/topics/git/cherry_picking.md +++ b/doc/topics/git/cherry_picking.md @@ -2,7 +2,6 @@ stage: Create group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false --- # Cherry-pick a Git commit **(FREE)** diff --git a/doc/topics/git/feature_branch_development.md b/doc/topics/git/feature_branch_development.md deleted file mode 100644 index 4125d8e8fdb..00000000000 --- a/doc/topics/git/feature_branch_development.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2023-03-31' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after <2023-03-31>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/feature_branching.md b/doc/topics/git/feature_branching.md index 73de351fde2..eaf619ce36f 100644 --- a/doc/topics/git/feature_branching.md +++ b/doc/topics/git/feature_branching.md @@ -1,31 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false +redirect_to: 'index.md' +remove_date: '2023-06-09' --- -# Feature branching **(FREE)** +This document was moved to [another location](index.md). -- Efficient parallel workflow for teams -- Develop each feature in a branch -- Keeps changes isolated -- Consider a 1-to-1 link to issues -- Push branches to the server frequently - - Hint: Pushing branches is a cheap backup for your work-in-progress code. - -## Feature branching sample workflow - -1. Create a new feature branch called `squash_some_bugs` -1. Edit '`bugs.rb`' and remove all the bugs. -1. Commit -1. Push - -```shell -git checkout -b squash_some_bugs -# Edit `bugs.rb` -git status -git add bugs.rb -git commit -m 'Fix some buggy code' -git push origin squash_some_bugs -``` +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/getting_started.md b/doc/topics/git/getting_started.md deleted file mode 100644 index 790fd3aa6c0..00000000000 --- a/doc/topics/git/getting_started.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../tutorials/make_your_first_git_commit.md' -remove_date: '2023-04-23' ---- - -This document was moved to [another location](../../tutorials/make_your_first_git_commit.md). - -<!-- This redirect file can be deleted after <2023-04-23>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/git_add.md b/doc/topics/git/git_add.md index 722701dbb49..1089202bbd3 100644 --- a/doc/topics/git/git_add.md +++ b/doc/topics/git/git_add.md @@ -2,7 +2,6 @@ stage: Create group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false --- # Git add **(FREE)** diff --git a/doc/topics/git/git_log.md b/doc/topics/git/git_log.md index e2e9c0e8804..eaf619ce36f 100644 --- a/doc/topics/git/git_log.md +++ b/doc/topics/git/git_log.md @@ -1,60 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false +redirect_to: 'index.md' +remove_date: '2023-06-09' --- -# Git Log **(FREE)** +This document was moved to [another location](index.md). -Git log lists commit history. It allows searching and filtering. - -- Initiate log: - - ```shell - git log - ``` - -- Retrieve set number of records: - - ```shell - git log -n 2 - ``` - -- Search commits by author. Allows user name or a regular expression. - - ```shell - git log --author="user_name" - ``` - -- Search by comment message: - - ```shell - git log --grep="<pattern>" - ``` - -- Search by date: - - ```shell - git log --since=1.month.ago --until=3.weeks.ago - ``` - -## Git Log Workflow - -1. Change to workspace directory -1. Clone the multi runner projects -1. Change to project dir -1. Search by author -1. Search by date -1. Combine - -## Commands - -```shell -cd ~/workspace -git clone git@gitlab.com:gitlab-org/gitlab-runner.git -cd gitlab-runner -git log --author="Travis" -git log --since=1.month.ago --until=3.weeks.ago -git log --since=1.month.ago --until=1.day.ago --author="Travis" -``` +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/git_rebase.md b/doc/topics/git/git_rebase.md index ec28d0b6431..bc9337481d4 100644 --- a/doc/topics/git/git_rebase.md +++ b/doc/topics/git/git_rebase.md @@ -226,5 +226,4 @@ git push --force origin my-feature ## Related topics - [Numerous undo possibilities in Git](numerous_undo_possibilities_in_git/index.md#undo-staged-local-changes-without-modifying-history) -for a deeper look into interactive rebases. -- [Git documentation for branches and rebases](https://git-scm.com/book/en/v2/Git-Branching-Rebasing). +- [Git documentation for branches and rebases](https://git-scm.com/book/en/v2/Git-Branching-Rebasing) diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index b47a34fa7b2..d5aa74935f2 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -26,13 +26,12 @@ The following resources can help you get started with Git: - [Git-ing started with Git](https://www.youtube.com/watch?v=Ce5nz5n41z4), a video introduction to Git. -- [Make your first Git commit](../../tutorials/make_your_first_git_commit.md) +- [Make your first Git commit](../../tutorials/make_first_git_commit/index.md) - [Git Basics](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics) - [Git on the Server - GitLab](https://git-scm.com/book/en/v2/Git-on-the-Server-GitLab) - [How to install Git](how_to_install_git/index.md) -- [Git terminology](terminology.md) +- [Git concepts](terminology.md) - [Start using Git on the command line](../../gitlab-basics/start-using-git.md) -- [Edit files through the command line](../../gitlab-basics/command-line-commands.md) - [GitLab Git Cheat Sheet (download)](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) - Commits: - [Revert a commit](../../user/project/merge_requests/revert_changes.md#revert-a-commit) @@ -43,7 +42,7 @@ The following resources can help you get started with Git: - [Git stash](stash.md) - [Git file blame](../../user/project/repository/git_blame.md) - [Git file history](../../user/project/repository/git_history.md) -- [Git tags](tags.md) +- [Git tags](../../user/project/repository/tags/index.md) ### Concepts @@ -58,16 +57,11 @@ The following are resources on version control concepts: You can do many Git tasks from the command line: -- [Bisect](bisect.md). - [Cherry-pick](cherry_picking.md). -- [Feature branching](feature_branching.md). -- [Getting started with Git](getting_started.md). +- [Getting started with Git](../../tutorials/make_first_git_commit/index.md). - [Git add](git_add.md). -- [Git log](git_log.md). - [Git stash](stash.md). -- [Merge conflicts](merge_conflicts.md). - [Rollback commits](rollback_commits.md). -- [Subtree](subtree.md). - [Unstage](unstage.md). ## Git tips @@ -88,7 +82,6 @@ If you have problems with Git, the following may help: ## Branching strategies - [Feature branch workflow](../../gitlab-basics/feature_branch_workflow.md) -- [Develop on a feature branch](feature_branch_development.md) - [GitLab Flow](../gitlab_flow.md) - [Git Branching - Branches in a Nutshell](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell) - [Git Branching - Branching Workflows](https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows) diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index cac203ffac0..4f0d1bfc5e6 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -3,7 +3,6 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" type: reference, howto -disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs/index.html' --- # Git Large File Storage (LFS) **(FREE)** @@ -27,10 +26,10 @@ instructions from where to fetch or where to push the large file. Documentation for GitLab instance administrators is under [LFS administration doc](../../../administration/lfs/index.md). -## Requirements +## Prerequisites - Git LFS must be [enabled in project settings](../../../user/project/settings/index.md#configure-project-visibility-features-and-permissions). -- [Git LFS client](https://git-lfs.com/) version 1.0.1 or higher must be installed. +- [Git LFS client](https://git-lfs.com/) version 1.0.1 or later must be installed. ## Known limitations diff --git a/doc/topics/git/lfs/migrate_to_git_lfs.md b/doc/topics/git/lfs/migrate_to_git_lfs.md index 07c89e52653..d14d0db6087 100644 --- a/doc/topics/git/lfs/migrate_to_git_lfs.md +++ b/doc/topics/git/lfs/migrate_to_git_lfs.md @@ -35,7 +35,7 @@ The method described on this guide rewrites Git history. Make sure to back up your repository before beginning and use it at your own risk. -## Requirements +## Prerequisites Before beginning, make sure: @@ -161,7 +161,7 @@ Consider an example upstream project, `git@gitlab.com:gitlab-tests/test-git-lfs- expand **Protected branches**. 1. Select the default branch from the **Branch** dropdown list, and set up the - **Allowed to push** and **Allowed to merge** rules. + **Allowed to push and merge** and **Allowed to merge** rules. 1. Select **Protect**. <!-- ## Troubleshooting diff --git a/doc/topics/git/merge_conflicts.md b/doc/topics/git/merge_conflicts.md deleted file mode 100644 index ea37afe1b31..00000000000 --- a/doc/topics/git/merge_conflicts.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../user/project/merge_requests/conflicts.md#resolve-conflicts-from-the-command-line' -remove_date: '2023-02-01' ---- - -This document was moved to [another location](../../user/project/merge_requests/conflicts.md#resolve-conflicts-from-the-command-line). - -<!-- This redirect file can be deleted after <2023-02-01>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/partial_clone.md b/doc/topics/git/partial_clone.md index f5f11b17675..de0547ae49d 100644 --- a/doc/topics/git/partial_clone.md +++ b/doc/topics/git/partial_clone.md @@ -94,9 +94,7 @@ Updating files: 100% (28/28), done. $ cd www-gitlab-com -$ git sparse-checkout init --cone - -$ git sparse-checkout add data +$ git sparse-checkout set data --cone remote: Enumerating objects: 301, done. remote: Counting objects: 100% (301/301), done. remote: Compressing objects: 100% (292/292), done. diff --git a/doc/topics/git/rollback_commits.md b/doc/topics/git/rollback_commits.md index 56d740f3d1b..d1f34c7cf9c 100644 --- a/doc/topics/git/rollback_commits.md +++ b/doc/topics/git/rollback_commits.md @@ -2,7 +2,6 @@ stage: Create group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false --- # Roll back commits **(FREE)** diff --git a/doc/topics/git/stash.md b/doc/topics/git/stash.md index ee931bbbb7c..9f621308a1d 100644 --- a/doc/topics/git/stash.md +++ b/doc/topics/git/stash.md @@ -2,7 +2,6 @@ stage: Create group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false --- # Git stash **(FREE)** diff --git a/doc/topics/git/subtree.md b/doc/topics/git/subtree.md index a8a665d4e13..eaf619ce36f 100644 --- a/doc/topics/git/subtree.md +++ b/doc/topics/git/subtree.md @@ -1,50 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false +redirect_to: 'index.md' +remove_date: '2023-06-09' --- -# Subtree **(FREE)** +This document was moved to [another location](index.md). -- Used when there are nested repositories. -- Not recommended when the amount of dependencies is too large. -- For these cases we need a dependency control system. -- Command are painfully long so aliases are necessary. - -## Subtree Aliases - -- Add: `git subtree add --prefix <target-folder> <url> <branch> --squash` -- Pull: `git subtree pull --prefix <target-folder> <url> <branch> --squash` -- Push: `git subtree push --prefix <target-folder> <url> <branch>` - -```shell - # Add an alias - # Add - git config alias.sba 'subtree add --prefix st / - git@gitlab.com:balameb/subtree-nested-example.git master --squash' - # Pull - git config alias.sbpl 'subtree pull --prefix st / - git@gitlab.com:balameb/subtree-nested-example.git master --squash' - # Push - git config alias.sbph 'subtree push --prefix st / - git@gitlab.com:balameb/subtree-nested-example.git master' - - # Adding this subtree adds a st dir with a readme - git sba - vi st/README.md - # Edit file - git status shows differences - -``` - -```shell - # Adding, or committing won't change the sub repo at remote - # even if we push - git add -A - git commit -m "Adding to subtree readme" - - # Push to subtree repo - git sbph - # now we can check our remote sub repo -``` +<!-- This redirect file can be deleted after <2023-06-09>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/tags.md b/doc/topics/git/tags.md index ab196f409f7..c66c97717f2 100644 --- a/doc/topics/git/tags.md +++ b/doc/topics/git/tags.md @@ -1,41 +1,11 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../../user/project/repository/tags/index.md' +remove_date: '2023-05-27' --- -# Tags **(FREE)** +This document was moved to [another location](../../user/project/repository/tags/index.md). -Tags help you mark certain deployments and releases for later -reference. Git supports two types of tags: - -- Annotated tags: An unchangeable part of Git history. -- Lightweight (soft) tags: Tags that can be set and removed as needed. - -Many projects combine an annotated release tag with a stable branch. Consider -setting deployment or release tags automatically. - -## Tags sample workflow - -1. Create a lightweight tag. -1. Create an annotated tag. -1. Push the tags to the remote repository. - -```shell -git checkout master - -# Lightweight tag -git tag my_lightweight_tag - -# Annotated tag -git tag -a v1.0 -m 'Version 1.0' - -# Show list of the existing tags -git tag - -git push origin --tags -``` - -## Related topics - -- [Tagging](https://git-scm.com/book/en/v2/Git-Basics-Tagging) Git reference page +<!-- This redirect file can be deleted after <2023-05-27>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/git/terminology.md b/doc/topics/git/terminology.md index ac097396bef..cef9b7cc35b 100644 --- a/doc/topics/git/terminology.md +++ b/doc/topics/git/terminology.md @@ -4,9 +4,9 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Git terminology +# Git concepts -The following are commonly-used Git terms. +The following are commonly-used Git concepts. ## Repository @@ -27,7 +27,7 @@ In GitLab, a repository is contained in a **project**. ## Fork When you want to contribute to someone else's repository, you make a copy of it. -This copy is called a [**fork**](../../user/project/repository/forking_workflow.md#creating-a-fork). +This copy is called a [**fork**](../../user/project/repository/forking_workflow.md#create-a-fork). The process is called "creating a fork." When you fork a repository, you create a copy of the project in your own diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index a116f6cc978..502acf5f7b4 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -75,7 +75,7 @@ Configuring both the client and the server is unnecessary. ``` - On Windows, if you are using PuTTY, go to your session properties, then - navigate to "Connection" and under "Sending of null packets to keep + go to "Connection" and under "Sending of null packets to keep session active", set `Seconds between keepalives (0 to turn off)` to `60`. **To configure SSH on the server side**, edit `/etc/ssh/sshd_config` and add: @@ -186,7 +186,7 @@ fatal: early EOF fatal: index-pack failed ``` -This is a common problem with Git itself, due to its inability to handle large files or large quantities of files. +This problem is common in Git itself, due to its inability to handle large files or large quantities of files. [Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) was created to work around this problem; however, even it has limitations. It's usually due to one of these reasons: - The number of files in the repository. @@ -221,11 +221,17 @@ apply more than one: 1. Modify the GitLab instance's [`gitlab.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/13.5.1+ee.0/files/gitlab-config-template/gitlab.rb.template#L1435-1455) file: - ```shell - gitaly['gitconfig'] = [ - # Set the http.postBuffer size, in bytes - {key: "http.postBuffer", value: "524288000"}, - ] + ```ruby + gitaly['configuration'] = { + # ... + git: { + # ... + config: [ + # Set the http.postBuffer size, in bytes + {key: "http.postBuffer", value: "524288000"}, + ], + }, + } ``` 1. After applying this change, apply the configuration change: @@ -282,3 +288,105 @@ The bug was reported [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issu If you receive an `HTTP Basic: Access denied` error when using Git over HTTP(S), refer to the [two-factor authentication troubleshooting guide](../../user/profile/account/two_factor_authentication.md#troubleshooting). + +## `401` errors logged during successful `git clone` + +When cloning a repository via HTTP, the +[`production_json.log`](../../administration/logs/index.md#production_jsonlog) file +may show an initial status of `401` (unauthorized), quickly followed by a `200`. + +```json +{ + "method":"GET", + "path":"/group/project.git/info/refs", + "format":"*/*", + "controller":"Repositories::GitHttpController", + "action":"info_refs", + "status":401, + "time":"2023-04-18T22:55:15.371Z", + "remote_ip":"x.x.x.x", + "ua":"git/2.39.2", + "correlation_id":"01GYB98MBM28T981DJDGAD98WZ", + "duration_s":0.03585 +} +{ + "method":"GET", + "path":"/group/project.git/info/refs", + "format":"*/*", + "controller":"Repositories::GitHttpController", + "action":"info_refs", + "status":200, + "time":"2023-04-18T22:55:15.714Z", + "remote_ip":"x.x.x.x", + "user_id":1, + "username":"root", + "ua":"git/2.39.2", + "correlation_id":"01GYB98MJ0CA3G9K8WDH7HWMQX", + "duration_s":0.17111 +} +``` + +You should expect this initial `401` log entry for each Git operation performed over HTTP, +due to how [HTTP Basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) works. + +When the Git client initiates a clone, the initial request sent to GitLab does not provide +any authentication details. GitLab returns a `401 Unauthorized` result for that request. +A few milliseconds later, the Git client sends a follow-up request containing authentication +details. This second request should succeed, and result in a `200 OK` log entry. + +If a `401` log entry lacks a corresponding `200` log entry, the Git client is likely using either: + +- An incorrect password. +- An expired or revoked token.an incorrect + +If not rectified, you could encounter +[`403` (Forbidden) errors](#403-error-when-performing-git-operations-over-http) +instead. + +## `403` error when performing Git operations over HTTP + +When performing Git operations over HTTP, a `403` (Forbidden) error indicates that +your IP address has been blocked by the failed-authentication ban: + +```plaintext +fatal: unable to access 'https://gitlab.com/group/project.git/': The requested URL returned error: 403 +``` + +The `403` can be seen in the [`production_json.log`](../../administration/logs/index.md#production_jsonlog): + +```json +{ + "method":"GET", + "path":"/group/project.git/info/refs", + "format":"*/*", + "controller":"Repositories::GitHttpController", + "action":"info_refs", + "status":403, + "time":"2023-04-19T22:14:25.894Z", + "remote_ip":"x.x.x.x", + "user_id":1, + "username":"root", + "ua":"git/2.39.2", + "correlation_id":"01GYDSAKAN2SPZPAMJNRWW5H8S", + "duration_s":0.00875 +} +``` + +If your IP address has been blocked, a corresponding log entry exists in the +[`auth_json.log`](../../administration/logs/index.md#auth_jsonlog): + +```json +{ + "severity":"ERROR", + "time":"2023-04-19T22:14:25.893Z", + "correlation_id":"01GYDSAKAN2SPZPAMJNRWW5H8S", + "message":"Rack_Attack", + "env":"blocklist", + "remote_ip":"x.x.x.x", + "request_method":"GET", + "path":"/group/project.git/info/refs?service=git-upload-pack"} +``` + +The failed authentication ban limits differ depending if you are using a +[self-managed instance](../../security/rate_limits.md#failed-authentication-ban-for-git-and-container-registry) +or [GitLab.com](../../user/gitlab_com/index.md#ip-blocks). diff --git a/doc/topics/git/unstage.md b/doc/topics/git/unstage.md index 142a80a9ad9..3f509cdacef 100644 --- a/doc/topics/git/unstage.md +++ b/doc/topics/git/unstage.md @@ -2,18 +2,22 @@ stage: Create group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -comments: false --- -# Unstage **(FREE)** +# Unstage a file in Git **(FREE)** -- To remove files from stage use reset HEAD where HEAD is the last commit of the current branch. This unstages the file but maintain the modifications. +When you _stage_ a file in Git, you instruct Git to track changes to the file in +preparation for a commit. To instruct Git to disregard changes to a file, and not +include it in your next commit, _unstage_ the file. + +- To remove files from stage use `reset HEAD`, where HEAD is the last commit of + the current branch. This unstages the file but maintains the modifications. ```shell git reset HEAD <file> ``` -- To revert the file back to the state it was in before the changes we can use: +- To revert the file back to the state it was in before the changes: ```shell git checkout -- <file> @@ -26,7 +30,8 @@ comments: false git rm -r <dirname> ``` -- If we want to remove a file from the repository but keep it on disk, say we forgot to add it to our `.gitignore` file then use `--cache`: +- To keep a file on disk but remove it from the repository (such as a file you want + to add to `.gitignore`), use the `rm` command with the `--cache` flag: ```shell git rm <filename> --cache diff --git a/doc/topics/git/useful_git_commands.md b/doc/topics/git/useful_git_commands.md index 235412f511a..22548be2e8b 100644 --- a/doc/topics/git/useful_git_commands.md +++ b/doc/topics/git/useful_git_commands.md @@ -130,12 +130,6 @@ Use this to check the Git history of the file: git log -- <file> ``` -### Find the tags that contain a particular SHA - -```shell -git tag --contains <sha> -``` - ### Check the content of each change to a file ```shell diff --git a/doc/topics/gitlab_flow.md b/doc/topics/gitlab_flow.md index 0bf2e1fcc20..eb298841247 100644 --- a/doc/topics/gitlab_flow.md +++ b/doc/topics/gitlab_flow.md @@ -2,25 +2,94 @@ stage: Create group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -disqus_identifier: 'https://docs.gitlab.com/ee/workflow/gitlab_flow.html' --- -# Introduction to GitLab Flow **(FREE)** +# Introduction to Git workflows **(FREE)** With Git, you can use a variety of branching strategies and workflows. +Having a structured workflow for collaboration in complex projects is +crucial for several reasons: -Because the default workflow is not specifically defined, many organizations -end up with workflows that are too complicated, not clearly defined, or -not integrated with their issue tracking systems. +- **Code organization**: Keep the codebase organized, prevent + overlapping work, and ensure focused efforts towards a common goal. + +- **Version control**: Allow simultaneous work on different features + without conflicts, maintaining code stability. + +- **Code quality**: A code review and approval process helps maintain high + code quality and adherence to coding standards. + +- **Traceability and accountability**: Enable tracking of changes and their authors, + simplifying issue identification and responsibility assignment. + +- **Easier onboarding**: Help new team members quickly grasp the + development process, and start contributing effectively. + +- **Time and resource management**: Enable better planning, resource + allocation, and meeting deadlines, ensuring an efficient development + process. + +- **CI/CD**: Incorporate automated testing and deployment + processes, streamlining the release cycle and delivering high-quality + software consistently. + +A structured workflow promotes organization, efficiency, and code +quality, leading to a more successful and streamlined development process. + +If the default workflow is not specifically defined, many organizations +end up with workflows that are: + +- Too complicated. +- Not clearly defined. +- Not integrated with their issue tracking systems. Your organization can use GitLab with any workflow you choose. -However, if you are looking for guidance on best practices, you can use -the GitLab Flow. This workflow combines [feature-driven development](https://en.wikipedia.org/wiki/Feature-driven_development) -and [feature branches](https://martinfowler.com/bliki/FeatureBranch.html) with issue tracking. +## Workflow types -While this workflow used at GitLab, you can choose whichever workflow -suits your organization best. +Here are some of the most common Git workflows. + +### Centralized workflow + +Best suited for small teams transitioning from a centralized version +control system like SVN. All team members work on a single branch, +usually `main`, and push their changes directly to the central +repository. + +### Feature branch workflow + +Developers create separate branches for each feature or bugfix, +keeping the 'main' branch stable. When a feature is complete, the +developer submits a merge request to integrate the +changes back into `main` after a code review. + +### Forking workflow + +Commonly used in open-source projects, this workflow allows external +contributors to work without direct access to the main repository. +Developers create a fork (personal copy) of the main repository and +make changes in it. They then submit a merge request to have those changes +integrated into the main repository. + +### Git flow workflow + +This workflow is best for projects with a structured release cycle. +It introduces two long-lived branches: `main` for production-ready +code and `develop` for integrating features. Additional branches like +`feature`, `release`, and `hotfix` are used for specific purposes, +ensuring a strict and organized development process. + +### GitLab/GitHub flow + +A simplified workflow primarily used for web development and +continuous deployment. It combines aspects of the Feature branch +workflow and the Git flow workflow. Developers create feature branches +from `main`, and after the changes are complete, they are merged back +into the `main` branch, which is then immediately deployed. + +Each of these Git workflows has its advantages and is suited to +different project types and team structures. Below the most popular +workflows are reviewed in more details. ## Git workflow @@ -97,12 +166,22 @@ graph TD ``` This flow is clean and straightforward, and many organizations have adopted it with great success. -Atlassian recommends [a similar strategy](https://www.atlassian.com/blog/git/simple-git-workflow-is-simple), although they rebase feature branches. +Another way to integrate change from one branch to another is [rebasing](https://git-scm.com/book/en/v2/Git-Branching-Rebasing). Merging everything into the `main` branch and frequently deploying means you minimize the amount of unreleased code. This approach is in line with lean and continuous delivery best practices. However, this flow still leaves a lot of questions unanswered regarding deployments, environments, releases, and integrations with issues. -With GitLab flow, we offer additional guidance for these questions. -## Production branch with GitLab flow +## Introduction to GitLab Flow **(FREE)** + +However, if you are looking for guidance on best practices, you can use +the GitLab Flow. This workflow combines [feature-driven development](https://en.wikipedia.org/wiki/Feature-driven_development) +and [feature branches](https://martinfowler.com/bliki/FeatureBranch.html) with issue tracking. + +While this workflow used at GitLab, you can choose whichever workflow +suits your organization best. + +With GitLab Flow, we offer additional guidance for these questions. + +## Production branch with GitLab Flow GitHub flow assumes you can deploy to production every time you merge a feature branch. While this is possible in some cases, such as SaaS applications, there are some cases where this is not possible, such as: @@ -114,7 +193,8 @@ While this is possible in some cases, such as SaaS applications, there are some In these cases, you can create a production branch that reflects the deployed code. You can deploy a new version by merging `main` into the `production` branch. -While not shown in the graph below, the work on the `main` branch works just like in GitHub flow, i.e. with feature-branches being merged into `main`. +While not shown in the graph below, the work on the `main` branch works just like in GitHub flow: +with feature branches being merged into `main`. ```mermaid graph TD @@ -136,7 +216,7 @@ This time is pretty accurate if you automatically deploy your production branch. If you need a more exact time, you can have your deployment script create a tag on each deployment. This flow prevents the overhead of releasing, tagging, and merging that happens with Git flow. -## Environment branches with GitLab flow +## Environment branches with GitLab Flow It might be a good idea to have an environment that is automatically updated to the `staging` branch. Only, in this case, the name of this environment might differ from the branch name. @@ -170,12 +250,12 @@ In this case, deploy the `staging` branch to your staging environment. To deploy to pre-production, create a merge request from the `staging` branch to the `pre-prod` branch. Go live by merging the `pre-prod` branch into the `production` branch. This workflow, where commits only flow downstream, ensures that everything is tested in all environments. -If you need to cherry-pick a commit with a hotfix, it is common to develop it on a feature branch and merge it into `production` with a merge request. +To cherry-pick a commit with a hotfix, develop it on a feature branch and merge it into `production` with a merge request. In this case, do not delete the feature branch yet. If `production` passes automatic testing, you then merge the feature branch into the other branches. If this is not possible because more manual testing is required, you can send merge requests from the feature branch to the downstream branches. -## Release branches with GitLab flow +## Release branches with GitLab Flow You should work with release branches only if you need to release software to the outside world. In this case, each branch contains a minor version, such as @@ -202,13 +282,13 @@ Create stable branches using `main` as a starting point, and branch as late as p By doing this, you minimize the length of time during which you have to apply bug fixes to multiple branches. After announcing a release branch, only add serious bug fixes to the branch. If possible, first merge these bug fixes into `main`, and then cherry-pick them into the release branch. -If you start by merging into the release branch, you might forget to cherry-pick them into `main`, and then you'd encounter the same bug in subsequent releases. +If you initially merged into the release branch and then forgot to cherry-pick to `main`, you'd encounter the same bug in subsequent releases. Merging into `main` and then cherry-picking into release is called an "upstream first" policy, which is also practiced by [Google](https://www.chromium.org/chromium-os/chromiumos-design-docs/upstream-first/) and [Red Hat](https://www.redhat.com/en/blog/a-community-for-using-openstack-with-red-hat-rdo). Every time you include a bug fix in a release branch, increase the patch version (to comply with [Semantic Versioning](https://semver.org/)) by setting a new tag. Some projects also have a stable branch that points to the same commit as the latest released branch. In this flow, it is not common to have a production branch (or Git flow `main` branch). -## Merge/pull requests with GitLab flow +## Merge/pull requests with GitLab Flow  @@ -226,12 +306,17 @@ The merge request serves as a code review tool, and no separate code review tool If the review reveals shortcomings, anyone can commit and push a fix. Usually, the person to do this is the creator of the merge request. The diff in the merge request automatically updates when new commits are pushed to the branch. +In GitLab Flow, you can configure your pipeline to run every time you commit changes to a branch. This type of pipeline is called a branch pipeline. Alternatively, you can configure your pipeline to run every time you make changes to the source branch for a merge request. This type of pipeline is called a [merge request pipeline](../ci/pipelines/merge_request_pipelines.md). In GitLab Flow, you can also take advantage of our [Review Apps](../ci/review_apps/index.md) capability, which are a collaboration tool that provide an environment to showcase product changes. Review Apps provide an automatic live preview of changes made in a feature branch by spinning up a dynamic environment for your merge requests allowing stakeholders to see your changes without needing to check out your branch and run your changes in a sandbox environment. When your changes are merged, Review Apps clean up the dynamic environment and related resources preventing environment sprawl. -When you are ready for your feature branch to be merged, assign the merge request to the person who knows most about the codebase you are changing. +When you are ready to merge your feature branch, assign the merge request to a maintainer for the project. Also, mention any other people from whom you would like feedback. After the assigned person feels comfortable with the result, they can merge the branch. +In GitLab Flow, a [merged results pipeline](../ci/pipelines/merged_results_pipelines.md) runs against the results of the source and target branches merged together. If the assigned person does not feel comfortable, they can request more changes or close the merge request without merging. +NOTE: +In your pipelines, you can include any automated CI tests for unit, security vulnerabilities, code quality, performance, dependency, etc. Information related to the pipeline runs as well as results of tests are all displayed as widgets in the merge request window, so that you can access and visualize all these from a central location. + In GitLab, it is common to protect the long-lived branches, such as the `main` branch, so [most developers can't modify them](../user/permissions.md). So, if you want to merge into a protected branch, assign your merge request to someone with the Maintainer role. @@ -246,9 +331,9 @@ When you reopen an issue you need to create a new merge request.  -## Issue tracking with GitLab flow +## Issue tracking with GitLab Flow -GitLab flow is a way to make the relation between the code and the issue tracker more transparent. +GitLab Flow is a way to make the relation between the code and the issue tracker more transparent. Any significant change to the code should start with an issue that describes the goal. Having a reason for every code change helps to inform the rest of the team and to keep the scope of a feature branch small. @@ -256,7 +341,8 @@ In GitLab, each change to the codebase starts with an issue in the issue trackin If there is no issue yet, create the issue if the change requires more than an hour's work. In many organizations, raising an issue is part of the development process because they are used in sprint planning. The issue title should describe the desired state of the system. -For example, the issue title "As an administrator, I want to remove users without receiving an error" is better than "Administrators can't remove users." +For example, the issue title `As an administrator, I want to remove users without receiving an error` +is better than "Administrators can't remove users." When you are ready to code, create a branch for the issue from the `main` branch. This branch is the place for any work related to this change. @@ -268,7 +354,7 @@ When you are done or want to discuss the code, open a merge request. A merge request is an online place to discuss the change and review the code. If you open the merge request but do not assign it to anyone, it is a [draft merge request](../user/project/merge_requests/drafts.md). -These are used to discuss the proposed implementation but are not ready for inclusion in the `main` branch yet. +Drafts are used to discuss the proposed implementation but are not ready for inclusion in the `main` branch yet. Start the title of the merge request with `[Draft]`, `Draft:` or `(Draft)` to prevent it from being merged before it's ready. When you think the code is ready, assign the merge request to a reviewer. @@ -284,6 +370,8 @@ In this case, it is no problem to reuse the same branch name, because the first At any time, there is at most one branch for every issue. It is possible that one feature branch solves more than one issue. +In GitLab Flow, you can create a merge request from the issue itself. When you do it this way, a feature branch and its related merge request are automatically created and associated to each other and the merge request is automatically related to the issue. In addition, when the merge request is merged the issue is automatically closed for you. + ## Linking and closing issues from merge requests Link to issues by mentioning them in commit messages or the description of a merge request, for example, "Fixes #16" or "Duck typing is preferred. See #12." @@ -291,6 +379,8 @@ GitLab then creates links to the mentioned issues and creates comments in the is To automatically close linked issues, mention them with the words "fixes" or "closes," for example, "fixes #14" or "closes #67." GitLab closes these issues when the code is merged into the default branch. +Like mentioned in the previous section, in GitLab Flow, you can create a merge request from the issue itself. When you do it this way, a feature branch and its related merge request are automatically created and associated to each other and the merge request is automatically related to the issue. In addition, when the merge request is merged the issue is automatically closed for you. + If you have an issue that spans across multiple repositories, create an issue for each repository and link all issues to a parent issue. ## Squashing commits with rebase @@ -354,12 +444,16 @@ However, as discussed in [the section about rebasing](#squashing-commits-with-re Rebasing could create more work, as every time you rebase, you may need to resolve the same conflicts. Sometimes you can reuse recorded resolutions (`rerere`), but merging is better, because you only have to resolve conflicts once. -Atlassian has [a more thorough explanation of the tradeoffs between merging and rebasing](https://www.atlassian.com/blog/git/git-team-workflows-merge-or-rebase) on their blog. +You can read a more thorough explanation of the tradeoffs between merging and rebasing [here](https://git-scm.com/book/en/v2/Git-Branching-Rebasing#:~:text=Final%20commit%20history-,The,-Perils%20of%20Rebasing). A good way to prevent creating many merge commits is to not frequently merge `main` into the feature branch. -There are three reasons to merge in `main`: utilizing new code, resolving merge conflicts, and updating long-running branches. +Three reasons to merge in `main`: -If you need to use some code that was introduced in `main` after you created the feature branch, you can often solve this by just cherry-picking a commit. +1. Utilizing new code. +1. Resolving merge conflicts. +1. Updating long-running branches. + +To use some code that was introduced in `main` after you created the feature branch, cherry-pick a commit. If your feature branch has a merge conflict, creating a merge commit is a standard way of solving this. @@ -375,7 +469,7 @@ If your feature branches often take more than a day of work, try to split your f If you need to keep a feature branch open for more than a day, there are a few strategies to keep it up-to-date. One option is to use continuous integration (CI) to merge in `main` at the start of the day. Another option is to only merge in from well-defined points in time, for example, a tagged release. -You could also use [feature toggles](https://martinfowler.com/bliki/FeatureToggle.html) to hide incomplete features so you can still merge back into `main` every day. +You could also use [feature toggles](https://martinfowler.com/bliki/FeatureToggle.html) or [feature flags](../operations/feature_flags.md) to hide incomplete features so you can still merge back into `main` every day. NOTE: Don't confuse automatic branch testing with continuous integration. @@ -387,7 +481,7 @@ In conclusion, you should try to prevent merge commits, but not eliminate them. Your codebase should be clean, but your history should represent what actually happened. Developing software happens in small, messy steps, and it is OK to have your history reflect this. You can use tools to view the network graphs of commits and understand the messy history that created your code. -If you rebase code, the history is incorrect, and there is no way for tools to remedy this because they can't deal with changing commit identifiers. +If you rebase code, the commit history changes. Because of changed commit identifiers, tools can't restore the commit history. ## Commit often and push frequently @@ -419,8 +513,8 @@ The words "change," "improve," "fix," and "refactor" don't add much information For more information, see Tim Pope's excellent [note about formatting commit messages](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). To add more context to a commit message, consider adding information regarding the -origin of the change. For example, the URL of a GitLab issue, or a Jira issue number, -containing more information for users who need in-depth context about the change. +origin of the change, such the GitLab issue URL or Jira issue number. That way, you provide +more information for users who need in-depth context about the change. For example: @@ -434,9 +528,9 @@ Issue: gitlab.com/gitlab-org/gitlab/-/issues/1 In old workflows, the continuous integration (CI) server commonly ran tests on the `main` branch only. Developers had to ensure their code did not break the `main` branch. -When using GitLab flow, developers create their branches from this `main` branch, so it is essential that it never breaks. +When using GitLab Flow, developers create their branches from this `main` branch, so it is essential that it never breaks. Therefore, each merge request must be tested before it is accepted. -CI software like Travis CI and GitLab CI/CD show the build results right in the merge request itself to simplify the process. +CI software like GitLab CI/CD shows the build results right in the merge request itself to simplify the process. There is one drawback to testing merge requests: the CI server only tests the feature branch itself, not the merged result. Ideally, the server could also test the `main` branch after each change. @@ -445,6 +539,8 @@ Because feature branches should be short-lived, testing just the branch is an ac If new commits in `main` cause merge conflicts with the feature branch, merge `main` back into the branch to make the CI server re-run the tests. As said before, if you often have feature branches that last for more than a few days, you should make your issues smaller. +In GitLab Flow, your can include automated CI tests in your branch or merge request pipelines, which can run when you commit changes to a branch. + ## Working with feature branches When creating a feature branch, always branch from an up-to-date `main`. diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index 80ce703f7db..57c7a80fa3e 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -23,7 +23,7 @@ For a video walkthrough of this process, see [Offline GitLab Installation: Downl You should [manually download the GitLab package](../../update/package/index.md#upgrade-using-a-manually-downloaded-package) and relevant dependencies using a server of the same operating system type that has access to the Internet. -If your offline environment has no local network access, you must manually transport across the relevant package files through physical media, such as a USB drive, or writable DVD. +If your offline environment has no local network access, you must manually transport the relevant package through physical media, such as a USB drive. In Ubuntu, this can be performed on a server with Internet access using the following commands: @@ -71,7 +71,7 @@ sudo EXTERNAL_URL="http://my-host.internal" dpkg -i <gitlab_package_name>.deb ## Enabling SSL Follow these steps to enable SSL for your fresh instance. These steps reflect those for -[manually configuring SSL in Omnibus's NGINX configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-https-manually): +[manually configuring SSL in Omnibus's NGINX configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#configure-https-manually): 1. Make the following changes to `/etc/gitlab/gitlab.rb`: @@ -200,7 +200,7 @@ done. ### Disable Version Check and Service Ping -The Version Check and Service Ping services improve the GitLab user experience and ensure that +Version Check and Service Ping improve the GitLab user experience and ensure that users are on the most up-to-date instances of GitLab. These two services can be turned off for offline environments so that they do not attempt and fail to reach out to GitLab services. @@ -214,3 +214,89 @@ and Praefect servers so they can use an accessible NTP server. On offline instances, the [GitLab Geo check Rake task](../../administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-site-correctly) always fails because it uses `pool.ntp.org`. This error can be ignored but you can [read more about how to work around it](../../administration/geo/replication/troubleshooting.md#message-machine-clock-is-synchronized--exception). + +## Enabling the package metadata database + +Enabling the package metadata database is required to enable [license scanning of CycloneDX files](../../user/compliance/license_scanning_of_cyclonedx_files). +This process requires usage of the GitLab License Database, which is licensed under the [EE License](https://storage.googleapis.com/prod-export-license-bucket-1a6c642fc4de57d4/v1/LICENSE). +Note the following in relation to use of the License Database: + +- We may change or discontinue all or any part of the License Database, at any time and without notice, at our sole discretion. +- The License Database may contain links to third-party websites or resources. We provide these links only as a convenience and are not responsible for any third-party data, content, products, or services from those websites or resources or links displayed on such websites. +- The License Database is based in part on information made available by third parties, and GitLab is not responsible for the accuracy or completeness of content made available. + +### Using the gsutil tool to download the package metadata exports + +1. Install the [`gsutil`](https://cloud.google.com/storage/docs/gsutil_install) tool. +1. Find the root of the GitLab Rails directory. + + ```shell + export GITLAB_RAILS_ROOT_DIR="$(gitlab-rails runner 'puts Rails.root.to_s')" + echo $GITLAB_RAILS_ROOT_DIR + ``` + +1. Download the package metadata exports. + + ```shell + # To download the package metadata exports, an outbound connection to Google Cloud Storage bucket must be allowed. + mkdir $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + gsutil -m rsync -r -d gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + + # Alternatively, if the GitLab instance is not allowed to connect to the Google Cloud Storage bucket, the package metadata + # exports can be downloaded using a machine with the allowed access, and then copied to the root of the GitLab Rails directory. + rsync rsync://example_username@gitlab.example.com/package_metadata_db $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ + ``` + +### Using the Google Cloud Storage REST API to download the package metadata exports + +The package metadata exports can also be downloaded using the Google Cloud Storage API. The contents are available at [https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o](https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o). The following is an example of how this can be downloaded using [cURL](https://curl.se/) and [jq](https://stedolan.github.io/jq/). + +```shell +#!/bin/bash + +set -euo pipefail + +GITLAB_RAILS_ROOT_DIR="$(gitlab-rails runner 'puts Rails.root.to_s')" +PKG_METADATA_DIR="$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db" +PKG_METADATA_MANIFEST_OUTPUT_FILE="/tmp/license_db_export_manifest.json" +PKG_METADATA_DOWNLOADS_OUTPUT_FILE="/tmp/license_db_object_links.tsv" + +# Download the contents of the bucket +curl --silent --show-error --request GET "https://storage.googleapis.com/storage/v1/b/prod-export-license-bucket-1a6c642fc4de57d4/o" > "$PKG_METADATA_MANIFEST_OUTPUT_FILE" + +# Parse the links and names for the bucket objects and output them into a tsv file +jq -r '.items[] | [.name, .mediaLink] | @tsv' "$PKG_METADATA_MANIFEST_OUTPUT_FILE" > "$PKG_METADATA_DOWNLOADS_OUTPUT_FILE" + +echo -e "Saving package metadata exports to $PKG_METADATA_DIR\n" + +# Track how many objects will be downloaded +INDEX=1 +TOTAL_OBJECT_COUNT="$(wc -l $PKG_METADATA_DOWNLOADS_OUTPUT_FILE | awk '{print $1}')" + +# Download the objects +while IFS= read -r line; do + FILE="$(echo -n $line | awk '{print $1}')" + URL="$(echo -n $line | awk '{print $2}')" + OUTPUT_DIR="$(dirname $PKG_METADATA_DIR/$FILE)" + OUTPUT_PATH="$PKG_METADATA_DIR/$FILE" + + echo "Downloading $FILE" + + curl --progress-bar --create-dirs --output "$OUTPUT_PATH" --request "GET" "$URL" + + echo -e "$INDEX of $TOTAL_OBJECT_COUNT objects downloaded\n" + + let INDEX=(INDEX+1) +done < "$PKG_METADATA_DOWNLOADS_OUTPUT_FILE" + +echo "All objects saved to $PKG_METADATA_DIR" +``` + +### Automatic synchronization + +Your GitLab instance is synchronized [every hour](https://gitlab.com/gitlab-org/gitlab/-/blob/d4331343d26d6e2a81fadd8f7ecd72f7cb74d04d/config/initializers/1_settings.rb#L831-832) with the contents of the `package_metadata_db` directory. +To automatically update your local copy with the upstream changes, a cron job can be added to periodically download new exports. For example, the following crontabs can be added to setup a cron job that runs every 30 minutes. + +```plaintext +*/30 * * * * gsutil -m rsync -r -d gs://prod-export-license-bucket-1a6c642fc4de57d4 $GITLAB_RAILS_ROOT_DIR/vendor/package_metadata_db/ +``` diff --git a/doc/topics/release_your_application.md b/doc/topics/release_your_application.md index 819ed0a1224..e8dba553cab 100644 --- a/doc/topics/release_your_application.md +++ b/doc/topics/release_your_application.md @@ -26,6 +26,7 @@ release features incrementally. - [Auto Deploy](autodevops/stages.md#auto-deploy) is the DevOps stage dedicated to software deployment using GitLab CI/CD. Auto Deploy has built-in support for EC2 and ECS deployments. - Deploy to Kubernetes clusters by using the [GitLab agent](../user/clusters/agent/install/index.md). +- View an example of [how to structure a GitOps deployment repository](../user/clusters/agent/gitops/example_repository_structure.md). - Use Docker images to run AWS commands from GitLab CI/CD, and a template to facilitate [deployment to AWS](../ci/cloud_deployment). - Use GitLab CI/CD to target any type of infrastructure accessible by GitLab Runner. diff --git a/doc/topics/set_up_organization.md b/doc/topics/set_up_organization.md index 855b4962960..cf0a0ae4ab7 100644 --- a/doc/topics/set_up_organization.md +++ b/doc/topics/set_up_organization.md @@ -11,7 +11,7 @@ and give everyone access to the projects they need. - [Namespaces](../user/namespace/index.md) - [Members](../user/project/members/index.md) -- [Organization](../user/workspace/index.md) _(In development)_ +- [Organization](../user/organization/index.md) _(In development)_ - [Groups](../user/group/index.md) - [User account options](../user/profile/index.md) - [SSH keys](../user/ssh.md) diff --git a/doc/topics/your_work.md b/doc/topics/your_work.md index 862f9ae8430..c57c9b52604 100644 --- a/doc/topics/your_work.md +++ b/doc/topics/your_work.md @@ -1,18 +1,11 @@ --- -stage: Manage -group: Foundations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: '../tutorials/left_sidebar/index.md' +remove_date: '2023-08-03' --- -# Your work sidebar +This document was moved to [another location](../tutorials/left_sidebar/index.md). -- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/384342) in GitLab 15.9. - -The **Your work** left sidebar provides access to your: - -- [Projects](../user/project/working_with_projects.md#view-projects) -- [Groups](../user/group/index.md) -- [Issues](../user/project/issues/index.md) -- [Merge requests](../user/project/merge_requests/index.md) -- [To-do List](../user/todos.md) -- [Milestones](../user/project/milestones/index.md) +<!-- This redirect file can be deleted after <2023-08-03>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> |