diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-08 06:15:41 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-08 06:15:41 +0300 |
commit | 777ef5e9b2cb049db6c25867d8321a191cec8f72 (patch) | |
tree | 8e2458bdde3047bb367e99e9d7de9862107e6212 /doc | |
parent | 6352d2f7c2a9e62e07f854861f9f57dacbd6f4a4 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/api/graphql/reference/index.md | 68 | ||||
-rw-r--r-- | doc/development/bitbucket_cloud_importer.md | 5 | ||||
-rw-r--r-- | doc/user/application_security/policies/scan-result-policies.md | 2 | ||||
-rw-r--r-- | doc/user/project/import/bitbucket.md | 7 | ||||
-rw-r--r-- | doc/user/project/repository/monorepos/troubleshooting.md | 98 |
5 files changed, 167 insertions, 13 deletions
diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 71c72ccdfe4..6b281e1ade8 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -11806,6 +11806,43 @@ The edge type for [`Milestone`](#milestone). | <a id="milestoneedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="milestoneedgenode"></a>`node` | [`Milestone`](#milestone) | The item at the end of the edge. | +#### `MlCandidateConnection` + +The connection type for [`MlCandidate`](#mlcandidate). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlcandidateconnectionedges"></a>`edges` | [`[MlCandidateEdge]`](#mlcandidateedge) | A list of edges. | +| <a id="mlcandidateconnectionnodes"></a>`nodes` | [`[MlCandidate]`](#mlcandidate) | A list of nodes. | +| <a id="mlcandidateconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +##### Fields with arguments + +###### `MlCandidateConnection.count` + +Limited count of collection. Returns limit + 1 for counts greater than the limit. + +Returns [`Int!`](#int). + +####### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlcandidateconnectioncountlimit"></a>`limit` | [`Int`](#int) | Limit value to be applied to the count query. Default is 1000. | + +#### `MlCandidateEdge` + +The edge type for [`MlCandidate`](#mlcandidate). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlcandidateedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="mlcandidateedgenode"></a>`node` | [`MlCandidate`](#mlcandidate) | The item at the end of the edge. | + #### `MlModelVersionConnection` The connection type for [`MlModelVersion`](#mlmodelversion). @@ -21235,6 +21272,17 @@ Represents an entry from the Cloud License history. | <a id="locationblobpath"></a>`blobPath` | [`String`](#string) | HTTP URI path to view the input file in GitLab. | | <a id="locationpath"></a>`path` | [`String`](#string) | Path, relative to the root of the repository, of the filewhich was analyzed to detect the dependency. | +### `MLCandidateLinks` + +Represents links to perform actions on the candidate. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlcandidatelinksartifactpath"></a>`artifactPath` | [`String`](#string) | Path to the artifact. | +| <a id="mlcandidatelinksshowpath"></a>`showPath` | [`String`](#string) | Path to the details page of the candidate. | + ### `MLModelVersionLinks` Represents links to perform actions on the model version. @@ -22849,6 +22897,19 @@ Contains statistics about a milestone. | <a id="milestonestatsclosedissuescount"></a>`closedIssuesCount` | [`Int`](#int) | Number of closed issues associated with the milestone. | | <a id="milestonestatstotalissuescount"></a>`totalIssuesCount` | [`Int`](#int) | Total number of issues associated with the milestone. | +### `MlCandidate` + +Candidate for a model version in the model registry. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mlcandidate_links"></a>`_links` | [`MLCandidateLinks!`](#mlcandidatelinks) | Map of links to perform actions on the candidate. | +| <a id="mlcandidatecreatedat"></a>`createdAt` | [`Time!`](#time) | Date of creation. | +| <a id="mlcandidateid"></a>`id` | [`MlCandidateID!`](#mlcandidateid) | ID of the candidate. | +| <a id="mlcandidatename"></a>`name` | [`String!`](#string) | Name of the candidate. | + ### `MlModel` Machine learning model in the model registry. @@ -22857,6 +22918,7 @@ Machine learning model in the model registry. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="mlmodelcandidates"></a>`candidates` | [`MlCandidateConnection`](#mlcandidateconnection) | Version candidates of the model. (see [Connections](#connections)) | | <a id="mlmodelid"></a>`id` | [`MlModelID!`](#mlmodelid) | ID of the model. | | <a id="mlmodelname"></a>`name` | [`String!`](#string) | Name of the model. | | <a id="mlmodelversions"></a>`versions` | [`MlModelVersionConnection`](#mlmodelversionconnection) | Versions of the model. (see [Connections](#connections)) | @@ -32380,6 +32442,12 @@ A `MilestoneID` is a global ID. It is encoded as a string. An example `MilestoneID` is: `"gid://gitlab/Milestone/1"`. +### `MlCandidateID` + +A `MlCandidateID` is a global ID. It is encoded as a string. + +An example `MlCandidateID` is: `"gid://gitlab/Ml::Candidate/1"`. + ### `MlModelID` A `MlModelID` is a global ID. It is encoded as a string. diff --git a/doc/development/bitbucket_cloud_importer.md b/doc/development/bitbucket_cloud_importer.md index 7d8874417a9..98620ca39eb 100644 --- a/doc/development/bitbucket_cloud_importer.md +++ b/doc/development/bitbucket_cloud_importer.md @@ -6,11 +6,6 @@ info: Any user with at least the Maintainer role can merge updates to this conte # Bitbucket Cloud importer developer documentation -The Bitbucket Cloud importer can be configured with the `bitbucket_parallel_importer` feature flag. When the feature flag is: - -- Enabled, the importer uses Sidekiq to schedule work asynchronously. -- Disabled, the importer does all the work in a single thread. - ## Prerequisites You must be authenticated with Bitbucket: diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index 704b1455a08..3db9c3db749 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can use scan result policies to take action based on scan results. For example, one type of scan result policy is a security approval policy that allows approval to be required based on the -findings of one or more security scan jobs. Scan result policies are evaluated after a CI scanning job is fully executed. +findings of one or more security scan jobs. Scan result policies are evaluated after a CI scanning job is fully executed and both vulnerability and license type policies are evaluated based on the job artifact reports that are published in the completed pipeline. NOTE: Scan result policies are applicable only to [protected](../../project/protected_branches.md) target branches. diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index 8b2c4199e91..ebd868256a2 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -6,15 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Import your project from Bitbucket Cloud **(FREE ALL)** -> Parallel imports from Bitbucket Cloud [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/412614) in GitLab 16.6 [with a flag](../../../administration/feature_flags.md) named `bitbucket_parallel_importer`. Disabled by default. - Import your projects from Bitbucket Cloud to GitLab. -FLAG: -On self-managed GitLab, parallel imports are not available. Parallel imports can help when importing large projects. -To make parallel imports available, an administrator can [enable the feature flag](../../../administration/feature_flags.md) -named `bitbucket_parallel_importer`. On GitLab.com, this feature is available. - The Bitbucket importer can import: - Repository description diff --git a/doc/user/project/repository/monorepos/troubleshooting.md b/doc/user/project/repository/monorepos/troubleshooting.md new file mode 100644 index 00000000000..208464c3e0f --- /dev/null +++ b/doc/user/project/repository/monorepos/troubleshooting.md @@ -0,0 +1,98 @@ +--- +stage: Systems +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting monorepo performance + +Review these suggestions for performance problems with monorepos. + +## Slowness during `git clone` or `git fetch` + +There are a few key causes of slowness with clones and fetches. + +### High CPU utilization + +If the CPU utilization on your Gitaly nodes is high, you can also check +how much CPU is taken up from clones by [filtering on certain values](../../../../user/project/repository/monorepos/observability.md#cpu-and-memory). + +In particular, the `command.cpu_time_ms` field can indicate how +much CPU is being taken up by clones and fetches. + +In most cases, the bulk of server load is generated from `git-pack-objects` +processes, which is initiated during clones and fetches. Monorepos are often very busy +and CI/CD systems send a lot of clone and fetch commands to the server. + +High CPU utilization is a common cause of slow performance. +The following non-mutually exclusive causes are possible: + +- [Too many clones for Gitaly to handle](#cause-too-many-large-clones). +- [Poor read distribution on Gitaly Cluster](#cause-poor-read-distribution). + +#### Cause: too many large clones + +You might have too many large clones for Gitaly to handle. Gitaly can struggle to keep up +because of a number of factors: + +- The size of a repository. +- The volume of clones and fetches. +- Lack of CPU capacity. + +To help Gitaly process many large clones, you might need to reduce the burden on Gitaly servers through some optimization strategies +such as: + +- Turn on [pack-objects-cache](../../../../administration/gitaly/configure_gitaly.md#pack-objects-cache) + to reduce the work that `git-pack-objects` has to do. +- Change [Git strategy](../../../../user/project/repository/monorepos/index.md#git-strategy) + in CI/CD settings from `clone` to `fetch` or `none`. +- [Stop fetching tags](../../../../user/project/repository/monorepos/index.md#git-fetch-extra-flags), + unless your tests require them. +- [Use shallow clones](../../../../user/project/repository/monorepos/index.md#shallow-cloning) + whenever possible. + +The other option is to increase CPU capacity on Gitaly servers. + +#### Cause: poor read distribution + +You might have poor read distribution on Gitaly Cluster. + +To observe if most read traffic is going to the primary Gitaly node instead of +getting distributed across the cluster, use the +[read distribution Prometheus metric](observability.md#read-distribution). + +If the secondary Gitaly nodes aren't receiving much traffic, it might be that +the secondary nodes are perpetually out of sync. This problem is exacerbated in +a monorepo. + +Monorepos are often both large and busy. This leads to two effects. Firstly, +monorepos are pushed to often and have lots of CI jobs running. There can be +times when write operations such as deleting a branch fails a proxy call to the +secondary nodes. This triggers a replication job in Gitaly Cluster so that +the secondary node will catch up eventually. + +The replication job is essentially a `git fetch` from the secondary node to the +primary node, and because monorepos are often very large, this fetch can take a +long time. + +If the next call fails before the previous replication job completes, and this +keeps happening, you can end up in a state where your monorepo is constantly +behind in its secondaries. This leads to all traffic going to the primary node. + +One reason for these failed proxied writes is a known issue with the Git +`$GIT_DIR/packed-refs` file. The file must be locked to +remove an entry in the file, which can lead to a race condition that causes a +delete to fail when concurrent deletes happen. + +Engineers at GitLab have developed mitigations to try to batch reference deletions. + +Turn on the following [feature flags](../../../../administration/feature_flags.md) to allow GitLab to batch ref deletions. +These feature flags do not need downtime to enable. + +- `merge_request_cleanup_ref_worker_async` +- `pipeline_cleanup_ref_worker_async` +- `pipeline_delete_gitaly_refs_in_batches` +- `merge_request_delete_gitaly_refs_in_batches` + +[Epic 4220](https://gitlab.com/groups/gitlab-org/-/epics/4220) proposes to add RefTable support in GitLab, +which is considered a long-term solution. |