diff options
Diffstat (limited to 'doc/topics')
-rw-r--r-- | doc/topics/authentication/index.md | 3 | ||||
-rw-r--r-- | doc/topics/autodevops/cicd_variables.md | 2 | ||||
-rw-r--r-- | doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md | 2 | ||||
-rw-r--r-- | doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md | 2 | ||||
-rw-r--r-- | doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md | 2 | ||||
-rw-r--r-- | doc/topics/autodevops/index.md | 8 | ||||
-rw-r--r-- | doc/topics/autodevops/prepare_deployment.md | 2 | ||||
-rw-r--r-- | doc/topics/autodevops/requirements.md | 4 | ||||
-rw-r--r-- | doc/topics/git/how_to_install_git/index.md | 6 | ||||
-rw-r--r-- | doc/topics/git/index.md | 188 | ||||
-rw-r--r-- | doc/topics/git/lfs/index.md | 22 | ||||
-rw-r--r-- | doc/topics/git/troubleshooting_git.md | 2 | ||||
-rw-r--r-- | doc/topics/offline/quick_start_guide.md | 32 |
13 files changed, 166 insertions, 109 deletions
diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 6137ade4559..8efa5afcb80 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -1,5 +1,5 @@ --- -stage: Manage +stage: Govern group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -30,7 +30,6 @@ This page gathers all the resources for the topic **Authentication** in GitLab. - **Integrations:** - [OmniAuth](../../integration/omniauth.md) - [Atlassian Crowd OmniAuth Provider](../../administration/auth/crowd.md) - - [CAS OmniAuth Provider](../../integration/cas.md) - [SAML OmniAuth Provider](../../integration/saml.md) - [SAML for GitLab.com Groups](../../user/group/saml_sso/index.md) - [SCIM user provisioning for GitLab.com Groups](../../user/group/saml_sso/scim_setup.md) diff --git a/doc/topics/autodevops/cicd_variables.md b/doc/topics/autodevops/cicd_variables.md index a5897e0b233..21d9dd0b3d3 100644 --- a/doc/topics/autodevops/cicd_variables.md +++ b/doc/topics/autodevops/cicd_variables.md @@ -136,7 +136,7 @@ Prerequisite: To configure secret variables: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Variables**. 1. Create a CI/CD variable with the prefix `K8S_SECRET_`. For example, you diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md index af396e159da..6ac8f87940e 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_ecs.md @@ -13,7 +13,7 @@ You can choose to target AWS ECS as a deployment platform instead of using Kuber To get started on Auto DevOps to AWS ECS, you must add a specific CI/CD variable. To do so, follow these steps: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Specify which AWS platform to target during the Auto DevOps deployment diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md index 53b20f880a9..d0a7814348c 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_eks.md @@ -132,7 +132,7 @@ While Auto DevOps is enabled by default, Auto DevOps can be disabled at both the instance level (for self-managed instances) and the group level. Complete these steps to enable Auto DevOps if it's disabled: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the application project. +1. On the left sidebar, select **Search or go to** and find the application project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Select **Default to Auto DevOps pipeline** to display more options. diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md index a939fe108a1..d50bdcfa056 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md @@ -136,7 +136,7 @@ While Auto DevOps is enabled by default, Auto DevOps can be disabled at both the instance level (for self-managed instances) and the group level. Complete these steps to enable Auto DevOps if it's disabled: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find the application project. +1. On the left sidebar, select **Search or go to** and find the application project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Select **Default to Auto DevOps pipeline** to display more options. diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 0668ea0df14..4191ba257ca 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -101,7 +101,7 @@ Prerequisites: To enable Auto DevOps for a project: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Select the **Default to Auto DevOps pipeline** checkbox. @@ -132,7 +132,7 @@ Prerequisites: To enable Auto DevOps for a group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your group. +1. On the left sidebar, select **Search or go to** and find your group. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Select the **Default to Auto DevOps pipeline** checkbox. @@ -144,7 +144,7 @@ clear the **Default to Auto DevOps pipeline** checkbox. After enabling Auto DevOps at the group level, you can trigger the Auto DevOps pipeline for any project that belongs to that group: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Make sure the project doesn't contain a `.gitlab-ci.yml` file. 1. Select **Build > Pipelines**. 1. To trigger the Auto DevOps pipeline, select **Run pipeline**. @@ -164,7 +164,7 @@ Prerequisites: To enable Auto DevOps for your instance: -1. On the left sidebar, expand the top-most chevron (**{chevron-down}**). +1. On the left sidebar, select **Search or go to**. 1. Select **Admin Area**. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. diff --git a/doc/topics/autodevops/prepare_deployment.md b/doc/topics/autodevops/prepare_deployment.md index c5a758e5d70..1dced373461 100644 --- a/doc/topics/autodevops/prepare_deployment.md +++ b/doc/topics/autodevops/prepare_deployment.md @@ -43,7 +43,7 @@ To define the base domain, either: - In the project, group, or instance level: go to your cluster settings and add it there. - In the project or group level: add it as an environment variable: `KUBE_INGRESS_BASE_DOMAIN`. -- In the instance level: go to **Main menu > Admin > Settings > CI/CD > Continuous Integration and Delivery** and add it there. +- In the instance level: go to the Admin Area, then **Settings > CI/CD > Continuous Integration and Delivery** and add it there. The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of precedence as other environment [variables](../../ci/variables/index.md#cicd-variable-precedence). diff --git a/doc/topics/autodevops/requirements.md b/doc/topics/autodevops/requirements.md index 47d79fcfc6b..a0aaba99a59 100644 --- a/doc/topics/autodevops/requirements.md +++ b/doc/topics/autodevops/requirements.md @@ -41,7 +41,7 @@ that works best for your needs: You can choose the deployment method when enabling Auto DevOps or later: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > CI/CD**. 1. Expand **Auto DevOps**. 1. Choose the deployment strategy. @@ -60,7 +60,7 @@ To define the base domain, either: - In the project, group, or instance level: go to your cluster settings and add it there. - In the project or group level: add it as an environment variable: `KUBE_INGRESS_BASE_DOMAIN`. -- In the instance level: go to **Main menu > Admin > Settings > CI/CD > Continuous Integration and Delivery** and add it there. +- In the instance level: go to the Admin Area, then **Settings > CI/CD > Continuous Integration and Delivery** and add it there. The base domain variable `KUBE_INGRESS_BASE_DOMAIN` follows the same order of [precedence as other environment variables](../../ci/variables/index.md#cicd-variable-precedence). diff --git a/doc/topics/git/how_to_install_git/index.md b/doc/topics/git/how_to_install_git/index.md index 40c5147e20b..52e8e2e1259 100644 --- a/doc/topics/git/how_to_install_git/index.md +++ b/doc/topics/git/how_to_install_git/index.md @@ -34,16 +34,16 @@ Prerequisites: To install Git on macOS: -1. Open a terminal and install the XCode Command Line Tools: +1. Open a terminal and install Xcode Command Line Tools: ```shell xcode-select --install ``` - Alternatively, you can install the entire [XCode](https://developer.apple.com/xcode/) + Alternatively, you can install the entire [Xcode](https://developer.apple.com/xcode/) package through the macOS App Store. -1. Select **Install** to download and install XCode Command Line Tools. +1. Select **Install** to download and install Xcode Command Line Tools. 1. Install Homebrew according to the [official Homebrew installation instructions](https://brew.sh/index.html). 1. Install Git by running `brew install git` from your terminal. 1. In a terminal, verify that Git works on your computer: diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index 05b14b21f20..e349cf0bb92 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -8,104 +8,116 @@ type: index # Git **(FREE ALL)** Git is a [free and open source](https://git-scm.com/about/free-and-open-source) -distributed version control system designed to handle everything from small to -large projects with speed and efficiency. +distributed version control system. It handles projects of all sizes quickly and +efficiently, while providing support for rolling back changes when needed. -[GitLab](https://about.gitlab.com) is a Git-based fully integrated platform for -software development. Besides Git functionalities, GitLab has a lot of -powerful [features](https://about.gitlab.com/features/) to enhance your -[workflow](https://about.gitlab.com/topics/version-control/what-is-gitlab-flow/). +GitLab is built on top of (and with) Git, and provides you a Git-based, fully-integrated +platform for software development. GitLab adds many powerful +[features](https://about.gitlab.com/features/) on top of Git to enhance your workflow. -We've gathered some resources to help you to get the best from Git with GitLab. +These resources can help you to get the best from using Git with GitLab. -More information is also available on the [Git website](https://git-scm.com). +## Learn about Git -## Getting started +New to Git? These resources can help you understand basic Git concepts before +you dive in: -The following resources can help you get started with Git: - -- [Git-ing started with Git](https://www.youtube.com/watch?v=Ce5nz5n41z4), - a video introduction to Git. -- [Make your first Git commit](../../tutorials/make_first_git_commit/index.md) -- [Git Basics](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics) -- [Git on the Server - GitLab](https://git-scm.com/book/en/v2/Git-on-the-Server-GitLab) -- [How to install Git](how_to_install_git/index.md) - [Git concepts](terminology.md) -- [Start using Git on the command line](../../gitlab-basics/start-using-git.md) -- [GitLab Git Cheat Sheet (download)](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) -- Commits: - - [Revert a commit](../../user/project/merge_requests/revert_changes.md#revert-a-commit) - - [Cherry-picking a commit](../../user/project/merge_requests/cherry_pick_changes.md) - - [Squash-and-merge](../../user/project/merge_requests/squash_and_merge.md) - - [Signing commits](../../user/project/repository/gpg_signed_commits/index.md) -- [Git stash](stash.md) -- [Git file blame](../../user/project/repository/git_blame.md) -- [Git file history](../../user/project/repository/git_history.md) -- [Git tags](../../user/project/repository/tags/index.md) - -### Concepts - -The following are resources on version control concepts: - -- [Why Git is Worth the Learning Curve](https://about.gitlab.com/blog/2017/05/17/learning-curve-is-the-biggest-challenge-developers-face-with-git/) -- [The future of SaaS hosted Git repository pricing](https://about.gitlab.com/blog/2016/05/11/git-repository-pricing/) -- [Git website on version control](https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control) -- [GitLab University presentation about Version Control](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit?usp=sharing) - -### Work with Git on the command line - -You can do many Git tasks from the command line: - -- [Cherry-pick](cherry_picking.md). -- [Getting started with Git](../../tutorials/make_first_git_commit/index.md). -- [Git add](git_add.md). -- [Git stash](stash.md). -- [Rollback commits](rollback_commits.md). -- [Unstage](unstage.md). - -## Git tips - -The following resources may help you become more efficient at using Git: +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> + Video tutorial: [Git-ing started with Git](https://www.youtube.com/watch?v=Ce5nz5n41z4) +- PDF download: [GitLab Git Cheat Sheet](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) -- [Useful Git commands](useful_git_commands.md) collected by the GitLab support team. -- [Git Tips & Tricks](https://about.gitlab.com/blog/2016/12/08/git-tips-and-tricks/) -- [Eight Tips to help you work better with Git](https://about.gitlab.com/blog/2015/02/19/8-tips-to-help-you-work-better-with-git/) +The official Git documentation also offers information on +[Git basics](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics). -## Troubleshooting Git +## Begin using Git -If you have problems with Git, the following may help: +After you learn how Git works, you're ready to try it out. These resources are +appropriate for when you're ready to start learning Git by doing: -- [Numerous _undo_ possibilities in Git](numerous_undo_possibilities_in_git/index.md) -- Learn a few [Git troubleshooting](troubleshooting_git.md) techniques - -## Branching strategies +- [How to install Git](how_to_install_git/index.md) +- [Start using Git on the command line](../../gitlab-basics/start-using-git.md) +- Tutorial: [Make your first Git commit](../../tutorials/make_first_git_commit/index.md) +- Tutorial: [How to update Git commit messages](../../tutorials/update_commit_messages/index.md) +- The [GitLab CLI](https://gitlab.com/gitlab-org/cli/) + +A typical Git user encounters these concepts soon after starting to use Git: + +- [`git add`](git_add.md) to start tracking files with Git. +- [Tags](../../user/project/repository/tags/index.md) and + [branches](../../user/project/repository/branches/index.md). +- [How to undo mistakes](numerous_undo_possibilities_in_git/index.md), + including [`git reset`](rollback_commits.md). +- View a chronological list of changes to a file with + [Git history](../../user/project/repository/git_history.md). +- View a line-by-line editing history of a file with + [`git blame`](../../user/project/repository/git_blame.md). +- [Sign commits](../../user/project/repository/signed_commits/gpg.md) + for increased accountability and trust. + +## Learn more complex commands + +When you're comfortable with basic Git commands, you're ready to dive into the +more complex features of Git. These commands aren't required when creating +straightforward changes. When you begin managing multiple branches or need more complex +change management, you're ready for these features: + +- To stop tracking changes to a file, because you don't want to commit them, + [unstage the changes](unstage.md). +- [Stash your changes](stash.md) when your current work isn't ready to create a commit locally, + but you need to switch branches to work on something else. +- If you create many small commits locally, you can use + [squash and merge](../../user/project/merge_requests/squash_and_merge.md) + to combine them into fewer commits before pushing them. +- [Cherry-pick](../../user/project/merge_requests/cherry_pick_changes.md) the contents + of a commit from one branch to another. +- [Revert an existing commit](../../user/project/merge_requests/revert_changes.md#revert-a-commit) + if it contains changes you no longer want. + +## Learn branching and workflow strategies + +When you're comfortable with the creation and handling of individual branches, +you're ready to learn about Git workflows and branching strategies: - [Feature branch workflow](../../gitlab-basics/feature_branch_workflow.md) -- [Git Branching - Branches in a Nutshell](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell) -- [Git Branching - Branching Workflows](https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows) - -## Advanced use - -The following are advanced topics for those who want to get the most out of Git: - - [Introduction to Git rebase, force-push, and merge conflicts](git_rebase.md) -- [Server Hooks](../../administration/server_hooks.md) -- [Git Attributes](../../user/project/git_attributes.md) -- Git Submodules: [Using Git submodules with GitLab CI](../../ci/git_submodules.md) -- [Partial Clone](partial_clone.md) - -## API - -[Gitignore templates](../../api/templates/gitignores.md) API allow for -Git-related queries from GitLab. - -## Git Large File Storage (LFS) - -The following relate to Git Large File Storage: - -- [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) -- [Migrate an existing Git repository with Git LFS](lfs/migrate_to_git_lfs.md) -- [Removing objects from LFS](lfs/index.md#removing-objects-from-lfs) -- [GitLab Git LFS user documentation](lfs/index.md) -- [GitLab Git LFS administrator documentation](../../administration/lfs/index.md) -- [Towards a production quality open source Git LFS server](https://about.gitlab.com/blog/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) +- From the official Git documentation: + - [Git Branching - Branches in a Nutshell](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell) + - [Git Branching - Branching Workflows](https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows) + +## Learn advanced topics in Git management + +Git and GitLab, combined together, provide advanced features for repository management: + +- Enforce commit policies and run tasks with [Git server hooks](../../administration/server_hooks.md). +- Define which file types to treat as binary, and set the languages to use for + syntax highlighting with [the `.gitattributes` file](../../user/project/git_attributes.md). +- To keep a Git repository as a subdirectory in another repository, + [use Git submodules with GitLab CI](../../ci/git_submodules.md). +- When working with extremely large repositories, you can use a [partial clone](partial_clone.md) + of a repository instead of a complete clone. +- GitLab APIs for [`.gitignore` files](../../api/templates/gitignores.md), + [commits](../../api/commits.md), [tags](../../api/tags.md), + and [repositories](../../api/repositories.md). + +### Git Large File Storage (LFS) + +Many Git projects must manage large binary assets, such as videos and images. +Implementing Git Large File Storage can help manage these assets while keeping +your repository small: + +- [User documentation](lfs/index.md) for Git LFS at GitLab +- [Administrator documentation](../../administration/lfs/index.md) for Git LFS at GitLab +- Blog post: [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) +- [Migrate an existing Git repository](lfs/migrate_to_git_lfs.md) with Git LFS +- [Remove objects](lfs/index.md#removing-objects-from-lfs) from Git LFS +- Blog post: [Towards a production-quality open source Git LFS server](https://about.gitlab.com/blog/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) + +## Related topics + +- Official [Git documentation](https://git-scm.com), including + [Git on the Server - GitLab](https://git-scm.com/book/en/v2/Git-on-the-Server-GitLab) +- [Git troubleshooting](troubleshooting_git.md) techniques +- [Git commands](useful_git_commands.md) collected by the GitLab support team +- Blog post: [Git Tips & Tricks](https://about.gitlab.com/blog/2016/12/08/git-tips-and-tricks/) +- Blog post: [Eight Tips to help you work better with Git](https://about.gitlab.com/blog/2015/02/19/8-tips-to-help-you-work-better-with-git/) diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index d9c906ccd81..23d972e9aa7 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -32,7 +32,7 @@ Prerequisites: To do this: -1. On the left sidebar, at the top, select **Search GitLab** (**{search}**) to find your project. +1. On the left sidebar, select **Search or go to** and find your project. 1. Select **Settings > General**. 1. Expand the **Visibility, project features, permissions** section. 1. Turn on the **Git Large File Storage (LFS)** toggle. @@ -299,3 +299,23 @@ You might choose to do this if you are using an appliance like a Nexus Repositor GitLab can't verify LFS objects. Pushes then fail if you have GitLab LFS support enabled. To stop push failure, LFS support can be disabled in the [Project settings](../../../user/project/settings/index.md), which also disables GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS). + +### I/O timeout when pushing LFS objects + +You might get an error that states: + +```shell +LFS: Put "http://your-instance.com/root/project.git/gitlab-lfs/objects/cc29e205d04a4062d0fb131700e8bfc8e54c44d0176a8dca22f40b24ef26d325/15": read tcp your-instance-ip:54544->your-instance-ip:443: i/o timeout +error: failed to push some refs to 'ssh://your-instance.com:2222/root/project.git' +``` + +When network conditions are unstable, the Git LFS client might time out when trying to upload files +if network conditions are unstable. + +The workaround is to set the client activity timeout a higher value. + +For example, to set the timeout to 60 seconds: + +```shell +git config lfs.activitytimeout 60 +``` diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index 43d5d746539..0cc7259a104 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -197,7 +197,7 @@ The root causes vary, so multiple potential solutions exist, and you may need to apply more than one: - If this error occurs when cloning a large repository, you can - [decrease the cloning depth](../../ci/large_repositories/index.md#shallow-cloning) + [decrease the cloning depth](../../user/project/repository/managing_large_repositories.md#shallow-cloning) to a value of `1`. For example: ```shell diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index 82a88b53dcf..301f73a268d 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -217,7 +217,9 @@ always fails because it uses `pool.ntp.org`. This error can be ignored but you c ## Enabling the Package Metadata Database -Enabling the Package Metadata Database is required to enable [license scanning of CycloneDX files](../../user/compliance/license_scanning_of_cyclonedx_files). +Enabling the Package Metadata Database is required to enable +[Continuous Vulnerability Scanning](../../user/application_security/continuous_vulnerability_scanning/index.md) +and [license scanning of CycloneDX files](../../user/compliance/license_scanning_of_cyclonedx_files/index.md). This process requires the use of License and/or Advisory Data under what is collectively called the Package Metadata Database, which is licensed under the [EE License](https://storage.googleapis.com/prod-export-license-bucket-1a6c642fc4de57d4/LICENSE). Note the following in relation to use of the Package Metadata Database: @@ -225,8 +227,6 @@ Note the following in relation to use of the Package Metadata Database: - The Package Metadata Database may contain links to third-party websites or resources. We provide these links only as a convenience and are not responsible for any third-party data, content, products, or services from those websites or resources or links displayed on such websites. - The Package Metadata Database is based in part on information made available by third parties, and GitLab is not responsible for the accuracy or completeness of content made available. -Enabling the Package Metadata Database is also required to enable Continuous Vulnerability Scans for Dependency Scanning (see [epic 9534](https://gitlab.com/groups/gitlab-org/-/epics/9534) tracking this work for more info). - Package metadata is stored in the following Google Cloud Provider (GCP) buckets: - License Scanning - prod-export-license-bucket-1a6c642fc4de57d4 @@ -351,3 +351,29 @@ The directory for package metadata changed with the release of 16.2 from `vendor ```shell sed -i '.bckup' -e 's#vendor/package_metadata_db#vendor/package_metadata/licenses#g' [FILE ...] ``` + +### Troubleshooting + +#### Missing database data + +If license or advisory data is missing from the dependency list or MR pages, one possible cause of this is that the database has not been synchronized with the export data. + +`package_metadata` synchronization is triggered by using cron jobs ([advisory sync](https://gitlab.com/gitlab-org/gitlab/-/blob/16-3-stable-ee/config/initializers/1_settings.rb#L864-866) and [license sync](https://gitlab.com/gitlab-org/gitlab/-/blob/16-3-stable-ee/config/initializers/1_settings.rb#L855-857)) and imports only the package registry types enabled in [admin settings](../../administration/settings/security_and_compliance.md#choose-package-registry-metadata-to-sync). + +The file structure in `vendor/package_metadata` must coincide with the package registry type enabled above. For example, to sync `maven` license or advisory data, the package metadata directory under the Rails directory must have the following structure: + +- For licenses:`$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/licenses/v2/maven/**/*.ndjson`. +- For advisories:`$GITLAB_RAILS_ROOT_DIR/vendor/package_metadata/advisories/v2/maven/**/*.ndjson`. + +After a successful run, data under the `pm_` tables in the database should be populated (check using [Rails console](../../administration/operations/rails_console.md)): + +- For licenses: `sudo gitlab-rails runner "puts \"Package model has #{PackageMetadata::Package.where(purl_type: 'maven').size} packages\""` +- For advisories: `sudo gitlab-rails runner "puts \"Advisory model has #{PackageMetadata::AffectedPackage.where(purl_type: 'maven').size} packages\""` + +Additionally, checkpoint data should exist for the particular package registry being synchronized. For Maven, for example, there should be a checkpoint created after a successful sync run: + +- For licenses: `sudo gitlab-rails runner "puts \"maven data has been synced up to #{PackageMetadata::Checkpoint.where(data_type: 'licenses', purl_type: 'maven')}\""` +- For advisories: `sudo gitlab-rails runner "puts \"maven data has been synced up to #{PackageMetadata::Checkpoint.where(data_type: 'advisories', purl_type: 'maven')}\""` + +Finally, you can check the [`application_json.log`](../../administration/logs/index.md#application_jsonlog) logs to verify that the +sync job has run and is without error by searching for `DEBUG` messages where the class is `PackageMetadata::SyncService`. Example: `{"severity":"DEBUG","time":"2023-06-22T16:41:00.825Z","correlation_id":"a6e80150836b4bb317313a3fe6d0bbd6","class":"PackageMetadata::SyncService","message":"Evaluating data for licenses:gcp/prod-export-license-bucket-1a6c642fc4de57d4/v2/pypi/1694703741/0.ndjson"}`. |