diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-02-08 18:08:59 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-02-08 18:08:59 +0300 |
commit | c6c5dd8848b78528d7ad7f044a0c95be629d372e (patch) | |
tree | 261577e229ade85472353eb5b380c1e4fed9bc60 /doc | |
parent | d0aeb5df3d6b06165355b023a25b79c7bd74a27d (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
-rw-r--r-- | doc/api/graphql/reference/index.md | 40 | ||||
-rw-r--r-- | doc/api/projects.md | 15 | ||||
-rw-r--r-- | doc/api/runners.md | 7 | ||||
-rw-r--r-- | doc/architecture/blueprints/search/code_search_with_zoekt.md | 305 | ||||
-rw-r--r-- | doc/development/fe_guide/customizable_dashboards.md | 30 | ||||
-rw-r--r-- | doc/user/application_security/vulnerability_report/index.md | 2 |
6 files changed, 360 insertions, 39 deletions
diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 83626d921f8..9bf303234f4 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -9111,28 +9111,28 @@ The edge type for [`ProductAnalyticsDashboard`](#productanalyticsdashboard). | <a id="productanalyticsdashboardedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="productanalyticsdashboardedgenode"></a>`node` | [`ProductAnalyticsDashboard`](#productanalyticsdashboard) | The item at the end of the edge. | -#### `ProductAnalyticsDashboardWidgetConnection` +#### `ProductAnalyticsDashboardPanelConnection` -The connection type for [`ProductAnalyticsDashboardWidget`](#productanalyticsdashboardwidget). +The connection type for [`ProductAnalyticsDashboardPanel`](#productanalyticsdashboardpanel). ##### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="productanalyticsdashboardwidgetconnectionedges"></a>`edges` | [`[ProductAnalyticsDashboardWidgetEdge]`](#productanalyticsdashboardwidgetedge) | A list of edges. | -| <a id="productanalyticsdashboardwidgetconnectionnodes"></a>`nodes` | [`[ProductAnalyticsDashboardWidget]`](#productanalyticsdashboardwidget) | A list of nodes. | -| <a id="productanalyticsdashboardwidgetconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | +| <a id="productanalyticsdashboardpanelconnectionedges"></a>`edges` | [`[ProductAnalyticsDashboardPanelEdge]`](#productanalyticsdashboardpaneledge) | A list of edges. | +| <a id="productanalyticsdashboardpanelconnectionnodes"></a>`nodes` | [`[ProductAnalyticsDashboardPanel]`](#productanalyticsdashboardpanel) | A list of nodes. | +| <a id="productanalyticsdashboardpanelconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | -#### `ProductAnalyticsDashboardWidgetEdge` +#### `ProductAnalyticsDashboardPanelEdge` -The edge type for [`ProductAnalyticsDashboardWidget`](#productanalyticsdashboardwidget). +The edge type for [`ProductAnalyticsDashboardPanel`](#productanalyticsdashboardpanel). ##### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="productanalyticsdashboardwidgetedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | -| <a id="productanalyticsdashboardwidgetedgenode"></a>`node` | [`ProductAnalyticsDashboardWidget`](#productanalyticsdashboardwidget) | The item at the end of the edge. | +| <a id="productanalyticsdashboardpaneledgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="productanalyticsdashboardpaneledgenode"></a>`node` | [`ProductAnalyticsDashboardPanel`](#productanalyticsdashboardpanel) | The item at the end of the edge. | #### `ProjectConnection` @@ -17552,32 +17552,32 @@ Represents a product analytics dashboard. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="productanalyticsdashboarddescription"></a>`description` | [`String`](#string) | Description of the dashboard. | +| <a id="productanalyticsdashboardpanels"></a>`panels` | [`ProductAnalyticsDashboardPanelConnection!`](#productanalyticsdashboardpanelconnection) | Panels shown on the dashboard. (see [Connections](#connections)) | | <a id="productanalyticsdashboardtitle"></a>`title` | [`String!`](#string) | Title of the dashboard. | -| <a id="productanalyticsdashboardwidgets"></a>`widgets` | [`ProductAnalyticsDashboardWidgetConnection!`](#productanalyticsdashboardwidgetconnection) | Widgets shown on the dashboard. (see [Connections](#connections)) | -### `ProductAnalyticsDashboardVisualization` +### `ProductAnalyticsDashboardPanel` -Represents a product analytics dashboard visualization. +Represents a product analytics dashboard panel. #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="productanalyticsdashboardvisualizationdata"></a>`data` | [`JSON!`](#json) | Data of the visualization. | -| <a id="productanalyticsdashboardvisualizationoptions"></a>`options` | [`JSON!`](#json) | Options of the visualization. | -| <a id="productanalyticsdashboardvisualizationtype"></a>`type` | [`String!`](#string) | Type of the visualization. | +| <a id="productanalyticsdashboardpanelgridattributes"></a>`gridAttributes` | [`JSON`](#json) | Description of the position and size of the panel. | +| <a id="productanalyticsdashboardpaneltitle"></a>`title` | [`String!`](#string) | Title of the panel. | +| <a id="productanalyticsdashboardpanelvisualization"></a>`visualization` | [`ProductAnalyticsDashboardVisualization!`](#productanalyticsdashboardvisualization) | Visualization of the panel. | -### `ProductAnalyticsDashboardWidget` +### `ProductAnalyticsDashboardVisualization` -Represents a product analytics dashboard widget. +Represents a product analytics dashboard visualization. #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="productanalyticsdashboardwidgetgridattributes"></a>`gridAttributes` | [`JSON`](#json) | Description of the position and size of the widget. | -| <a id="productanalyticsdashboardwidgettitle"></a>`title` | [`String!`](#string) | Title of the widget. | -| <a id="productanalyticsdashboardwidgetvisualization"></a>`visualization` | [`ProductAnalyticsDashboardVisualization!`](#productanalyticsdashboardvisualization) | Visualization of the widget. | +| <a id="productanalyticsdashboardvisualizationdata"></a>`data` | [`JSON!`](#json) | Data of the visualization. | +| <a id="productanalyticsdashboardvisualizationoptions"></a>`options` | [`JSON!`](#json) | Options of the visualization. | +| <a id="productanalyticsdashboardvisualizationtype"></a>`type` | [`String!`](#string) | Type of the visualization. | ### `Project` diff --git a/doc/api/projects.md b/doc/api/projects.md index d53c515bfbe..529e86d2c94 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -207,6 +207,7 @@ When the user is authenticated and `simple` is not set this returns something li "security_and_compliance_access_level": "private", "emails_disabled": null, "shared_runners_enabled": true, + "group_runners_enabled": true, "lfs_enabled": true, "creator_id": 1, "import_status": "none", @@ -391,6 +392,7 @@ GET /users/:user_id/projects "archived": false, "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png", "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 0, "runners_token": "b8547b1dc37721d05889db52fa2f02", @@ -502,6 +504,7 @@ GET /users/:user_id/projects "archived": false, "avatar_url": null, "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 0, "runners_token": "b8547b1dc37721d05889db52fa2f02", @@ -653,6 +656,7 @@ Example response: "archived": false, "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png", "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 0, "runners_token": "b8547b1dc37721d05889db52fa2f02", @@ -757,6 +761,7 @@ Example response: "archived": false, "avatar_url": null, "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 0, "runners_token": "b8547b1dc37721d05889db52fa2f02", @@ -916,6 +921,7 @@ GET /projects/:id "source_url": "http://www.gnu.org/licenses/lgpl-3.0.txt" }, "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 0, "runners_token": "b8bc4a7a29eb76ea83cf79e4908c2b", @@ -1287,6 +1293,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your-token>" \ | `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | | `security_and_compliance_access_level` | string | **{dotted-circle}** No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | | `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | +| `group_runners_enabled` | boolean | **{dotted-circle}** No | Enable group runners for this project. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | | `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | @@ -1371,6 +1378,7 @@ POST /projects/user/:user_id | `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | | `security_and_compliance_access_level` | string | **{dotted-circle}** No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | | `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | +| `group_runners_enabled` | boolean | **{dotted-circle}** No | Enable group runners for this project. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | | `issue_branch_template` | string | **{dotted-circle}** No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | @@ -1481,6 +1489,7 @@ Supported attributes: | `security_and_compliance_access_level` | string | **{dotted-circle}** No | (GitLab 14.9 and later) Security and compliance access level. One of `disabled`, `private`, or `enabled`. | | `service_desk_enabled` | boolean | **{dotted-circle}** No | Enable or disable Service Desk feature. | | `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | +| `group_runners_enabled` | boolean | **{dotted-circle}** No | Enable group runners for this project. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | | `issue_branch_template` | string | **{dotted-circle}** No | Template used to suggest names for [branches created from issues](../user/project/merge_requests/creating_merge_requests.md#from-an-issue). _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21243) in GitLab 15.6.)_ | @@ -1600,6 +1609,7 @@ Example responses: "archived": true, "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png", "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 1, "public_jobs": true, @@ -1707,6 +1717,7 @@ Example response: "source_url": "http://www.gnu.org/licenses/lgpl-3.0.txt" }, "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 1, "public_jobs": true, @@ -1812,6 +1823,7 @@ Example response: "source_url": "http://www.gnu.org/licenses/lgpl-3.0.txt" }, "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 0, "public_jobs": true, @@ -2008,6 +2020,7 @@ Example response: "source_url": "http://www.gnu.org/licenses/lgpl-3.0.txt" }, "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 0, "runners_token": "b8bc4a7a29eb76ea83cf79e4908c2b", @@ -2136,6 +2149,7 @@ Example response: "source_url": "http://www.gnu.org/licenses/lgpl-3.0.txt" }, "shared_runners_enabled": true, + "group_runners_enabled": true, "forks_count": 0, "star_count": 0, "runners_token": "b8bc4a7a29eb76ea83cf79e4908c2b", @@ -2819,6 +2833,7 @@ Example response: "security_and_compliance_access_level": "enabled", "emails_disabled": null, "shared_runners_enabled": true, + "group_runners_enabled": true, "lfs_enabled": true, "creator_id": 2, "import_status": "none", diff --git a/doc/api/runners.md b/doc/api/runners.md index 4f39eac0c7d..f791d727a51 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -739,9 +739,10 @@ Validates authentication credentials for a registered runner. POST /runners/verify ``` -| Attribute | Type | Required | Description | -|-------------|---------|----------|-------------------------------------------------------------------------------| -| `token` | string | yes | The runner's [authentication token](#registration-and-authentication-tokens). | +| Attribute | Type | Required | Description | +|-------------|---------|----------|------------------------------------------------------------------------------------------------| +| `token` | string | yes | The runner's [authentication token](#registration-and-authentication-tokens). | +| `system_id` | string | no | The runner's system identifier. This attribute is required if the `token` starts with `glrt-`. | ```shell curl --request POST "https://gitlab.example.com/api/v4/runners/verify" \ diff --git a/doc/architecture/blueprints/search/code_search_with_zoekt.md b/doc/architecture/blueprints/search/code_search_with_zoekt.md new file mode 100644 index 00000000000..d0d347f1ff4 --- /dev/null +++ b/doc/architecture/blueprints/search/code_search_with_zoekt.md @@ -0,0 +1,305 @@ +--- +status: ongoing +creation-date: "2022-12-28" +authors: [ "@dgruzd", "@DylanGriffith" ] +coach: "@DylanGriffith" +approvers: [ "@joshlambert", "@changzhengliu" ] +owning-stage: "~devops::enablement" +participating-stages: [] +--- + +# Use Zoekt For code search + +## Summary + +We will be implementing an additional code search functionality in GitLab that +is backed by [Zoekt](https://github.com/sourcegraph/zoekt), an open source +search engine that is specifically designed for code search. Zoekt will be used as +an API by GitLab and remain an implementation detail while the user interface +in GitLab will not change much except for some new features made available by +Zoekt. + +This will be rolled out in phases to ensure that the system will actually meet +our scaling and cost expectations and will run alongside code search backed by +Elasticsearch until we can be sure it is a viable replacement. The first step +will be making it available for `gitlab-org` for internal and expanding +customer by customer based on customer interest. + +## Motivation + +GitLab code search functionality today is backed by Elasticsearch. +Elasticsearch has proven useful for other types of search (issues, merge +requests, comments and so-on) but is by design not a good choice for code +search where users expect matches to be precise (ie. no false positives) and +flexible (e.g. support +[substring matching](https://gitlab.com/gitlab-org/gitlab/-/issues/325234) +and +[regexes](https://gitlab.com/gitlab-org/gitlab/-/issues/4175)). We have +[investigated our options](https://gitlab.com/groups/gitlab-org/-/epics/7404) +and [Zoekt](https://github.com/sourcegraph/zoekt) is pretty much the only well +maintained open source technology that is suited to code search. Based on our +research we believe it will be better to adopt a well maintained open source +database than attempt to build our own. This is mostly due to the fact that our +research indicates that the fundamental architecture of Zoekt is what we would +implement again if we tried to implement something ourselves. + +Our +[early benchmarking](https://gitlab.com/gitlab-org/gitlab/-/issues/370832#note_1183611955) +suggests that Zoekt will be viable at our scale, but we feel strongly +that investing in building a beta integration with Zoekt and rolling it out +group by group on GitLab.com will provide better insights into scalability and +cost than more accurate benchmarking efforts. It will also be relatively low +risk as it will be rolled out internally first and later rolled out to +customers that wish to participate in the trial. + +### Goals + +The main goals of this integration will be to implement the following highly +requested improvements to code search: + +1. [Exact match (substring match) code searches in Advanced Search](https://gitlab.com/gitlab-org/gitlab/-/issues/325234) +1. [Support regular expressions with Advanced Global Search](https://gitlab.com/gitlab-org/gitlab/-/issues/4175) +1. [Support multiple line matches in the same file](https://gitlab.com/gitlab-org/gitlab/-/issues/668) + +The initial phases of the rollout will be designed to catch and resolve scaling +or infrastructure cost issues as early as possible so that we can pivot early +before investing too much in this technology if it is not suitable. + +### Non-Goals + +The following are not goals initially but could theoretically be built upon +this solution: + +1. Improving security scanning features by having access to quickly perform + regex scans across many repositories +1. Saving money on our search infrastructure - this may be possible with + further optimizations, but initial estimates suggest the cost is similar +1. AI/ML features of search used to predict what users might be interested in + finding +1. Code Intelligence and Navigation - likely code intelligence and navigation + features should be built on structured data rather than a trigram index but + regex based searches (using Zoekt) may be a suitable fallback for code which + does not have structured metadata enabled or dynamic languages where static + analysis is not very accurate. Zoekt in particular may not be well suited + initially, despite existing symbol extraction using ctags, because ctags + symbols may not contain enough data for accurate navigation and Zoekt + doesn't undersand dependencies which would be necessary for cross-project + navigation. + +## Proposal + +An +[initial implementation of a Zoekt integration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/105049) +was created to demonstrate the feasibility of using Zoekt as a drop-in +replacement for Elasticsearch code searches. This blueprint will extend on all +the details needed to provide a minimum viable change as well steps needed to +scale this to a larger customer rollout on GitLab.com. + +## Design and implementation details + +### User Experience + +When a user performs an advanced search on a group or project that is part +of the Zoekt rollout we will present a toggle somewhere in the UI to change +to "precise search" (or some other UX TBD) which switches them from +Elasticsearch to Zoekt. Early user feedback will help us assess the best way +to present these choices to users and ultimately we will want to remove the +Elasticsearch option if we find Zoekt is a suitable long term option. + +### Indexing + +Similar to our Elasticsearch integration, GitLab will notify Zoekt every time +there are updates to a repository. Zoekt, unlike Elasticsearch, is designed to +clone and index Git repositories so we will simply notify Zoekt of the URL of +the repository that has changed and it will update its local copy of the Git +repo and then update its local index files. The Zoekt side of this logic will +be implemented in a new server-side indexing endpoint we add to Zoekt which is +currently in +[an open Pull request](https://github.com/sourcegraph/zoekt/pull/496). +While the details of +this pull request are still being debated, we may choose to deploy a fork with +the functionality we need, but our strongest intention is not to maintain a +fork of Zoekt and the maintainers have already expressed they are open to this +new functionality. + +The rails side of the integration will be a Sidekiq worker that is scheduled +every time there is an update to a repository and it will simply call this +`/index` endpoint in Zoekt. This will also need to generate a one-time token +that can allow Zoekt to clone a private repository. + +```mermaid +sequenceDiagram + participant user as User + participant gitlab_git as GitLab Git + participant gitlab_sidekiq as GitLab Sidekiq + participant zoekt as Zoekt + user->>gitlab_git: git push git@gitlab.com:gitlab-org/gitlab.git + gitlab_git->>gitlab_sidekiq: ZoektIndexerWorker.perform_async(278964) + gitlab_sidekiq->>zoekt: POST /index {"RepoUrl":"https://zoekt:SECRET_TOKEN@gitlab.com/gitlab-org/gitlab.git","RepoId":278964}' + zoekt->>gitlab_git: git clone https://zoekt:SECRET_TOKEN@gitlab.com/gitlab-org/gitlab.git +``` + +The Sidekiq worker can leverage de-duplication based on the `project_id`. + +Zoekt supports indexing multiple projects we'll likely need to, eventually, +allow a way for users to configure additional branches (beyond the default +branch) and this will need to be sent to Zoekt. We will need to decide if these +branch lists are sent every time we index the project or only when they change +configuration. + +There may be race conditions with multiple Zoekt processes indexing the same +repo at the same time. For this reason we should implement a locking mechanism +somewhere to ensure we are only indexing 1 project in 1 place at a time. We +could make use of the same Redis locking we use for indexing projects in +Elasticsearch. + +### Searching + +Searching will be implemented using the `/api/search` functionality in +Zoekt. There is also +[an open PR to fix this endpoint in Zoekt](https://github.com/sourcegraph/zoekt/pull/506), +and again we may consider working from a fork until this is fixed. GitLab will +prepend all searches with the appropriate filter for repositories based on the +user's search context (group or project) in the same way we do for +Elasticsearch. For Zoekt this will be implemented as a query string regex that +matches all the searched repositories. + +### Zoekt infrastructure + +Each Zoekt node will need to run a +[zoekt-dynamic-indexserver](https://github.com/sourcegraph/zoekt/pull/496) and +a +[zoekt-webserver](https://github.com/sourcegraph/zoekt/blob/main/cmd/zoekt-webserver/main.go). +These are both webservers with different responsibilities. Considering that the +Zoekt indexing process needs to keep a full clone of the bare repo +([unless we come up with a better option](https://gitlab.com/gitlab-org/gitlab/-/issues/384722)) +these bare repos will be stored on spinning disks to save space. These are only +used as an intermediate step to generate the actual `.zoekt` index files which +will be stored on an SSD for fast searches. These web servers need to run on +the same node as they access the same files. The `zoekt-dynamic-indexserver` is +responsible for writing the `.zoekt` index files. The `zoekt-webserver` is +responsible for responding to searches that it performs by reading these +`.zoekt` index files. + +### Rollout strategy + +Initially Zoekt code search will only be available to `gitlab-org`. After that +we'll start rolling it out to specific customers that have requested better +code search experience. As we learn about scaling and make improvements we will +gradually roll it out to all licensed groups on GitLab.com. We will use a +similar approach to Elasticsearch for keeping track of which groups are indexed +and which are not. This will be based on a new table `zoekt_indexed_namespaces` +with a `namespace_id` reference. We will only allow rolling out to top level +namespaces to simplify the logic of checking for all layers of group +inheritance. Once we've rolled out to all licensed groups we'll enable logic to +automatically enroll newly licensed groups. This table also may be a place to +store per-namespace sharding and replication data as described below. + +### Sharding and replication strategy + +Zoekt does not have any inbuilt sharding, and we expect that we'll need +multiple Zoekt servers to reach the scale to provide search functionality to +all of GitLab licensed customers. + +There are 2 clear ways to implement sharding: + +1. Build it on top of, or in front of Zoekt, as an independent component. Building + all the complexities of a distributed database into Zoekt is not likely to + be a good direction for the project so most likely this would be an + independent piece of infrastructure that proxied requests to the correct + shard. +1. Manage the shards inside GitLab. This would be an application layer in + GitLab which chooses the correct shard to send indexing and search requests + to. + +Likewise, there are a few ways to implement replication: + +1. Server-side where Zoekt replicas are aware of other Zoekt replicas and they + stream updates from some primary to remain in sync +1. Client-side replication where clients send indexing requests to all replicas + and search requests to any replica + +We plan to implement sharding inside GitLab application but replication may be +best served at the level of the filesystem of Zoekt servers rather than sending +duplicated updates from GitLab to all replicas. This could be some process on +Zoekt servers that monitors for changes to the `.zoekt` files in a specific +directory and syncs those updates to the replicas. This will need to be +slightly more sophisticated than `rsync` because the files are constantly +changing and files may be getting deleted while the sync is happening so we +would want to be syncing the updates in batches somehow without slowing down +indexing. + +Implementing sharding in GitLab simplifies the additional infrastructure +components that need to be deployed and allows more flexibility to control our +rollout to many customers alongside our rollout of multiple shards. + +Implementing syncing from primary -> replica on Zoekt nodes at the filesystem +level optimizes that overall resource usage. We only need to sync the index +files to replicas as the bare repo is just a cache. This saves on: + +1. Disk space on replicas +1. CPU usage on replicas as it does not need to rebuild the index +1. Load on Gitaly to clone the repos + +We plan to defer the implementation of these high availability aspects until +later, but a preliminary plan would be: + +1. GitLab is configured with a pool of Zoekt servers +1. GitLab assigns groups randomly a Zoekt primary server +1. There will also be Zoekt replica servers +1. Periodically Zoekt primary servers will sync their `.zoekt` index files to + their respective replicas +1. There will need to be some process by which to promote a replica to a + primary if the primary is having issues. We will be using Consul for + keeping track of which is the primary and which are the replicas. +1. When indexing a project GitLab will queue a Sidekiq job to update the index + on the primary +1. When searching we will randomly select one of the Zoekt primaries or replica + servers for the group being searched. We don't care which is "more up to + date" as code search will be "eventually consistent" and all reads may read + slightly out of date indexes. We will have a target of maximum latency of + index updates and may consider removing nodes from rotation if they are too + far out of date. +1. We will shard everything by top level group as this ensures group search can + always search a single Zoekt server. Aggregation may be possible for global + searches at some point in future if this turns out to be important. Smaller + self-managed instances may use a single Zoekt server allowing global + searches to work without any aggregation being implemented. Depending on our + largest group sizes and scaling limitations of a single node Zoekt server we + may consider implementing an approach where a group can be assigned multiple + shards. + +The downside of the chosen path will be added complexity of managing all these +Zoekt servers from GitLab when compared with a "proxy" layer outside of GitLab +that is managing all of these shards. We will consider this decision a work in +progress and reassess if it turns out to add too much complexity to GitLab. + +#### Sharding proposal using GitLab `::Zoekt::Shard` model + +This is already implemented as the `::Zoekt::IndexedNamespace` +implements a many-to-many relationship between namespaces and shards. + +#### Replication and service discovery using Consul + +If we plan to replicate at the Zoekt node level as described above we need to +change our data model to use a one-to-many relationship from `zoekt_shards -> +namespaces`. This means making the `namespace_id` column unique in +`zoekt_indexed_namespaces`. Then we need to implement a service discovery +approach where the `index_url` always points at a primary Zoekt node and the +`search_url` is a DNS record with N replicas and the primary. We then choose +randomly from `search_url` records when searching. + +### Iterations + +1. Make available for `gitlab-org` +1. Improve monitoring +1. Improve performance +1. Make available for select customers +1. Implement sharding +1. Implement replication +1. Make available to many more licensed groups +1. Implement automatic (re)balancing of shards +1. Estimate costs for rolling out to all licensed groups and decide if it's worth it or if we need to optimize further or adjust our plan +1. Rollout to all licensed groups +1. Improve performance +1. Assess costs and decide whether we should roll out to all free customers diff --git a/doc/development/fe_guide/customizable_dashboards.md b/doc/development/fe_guide/customizable_dashboards.md index 62dd974cccb..26c73b26126 100644 --- a/doc/development/fe_guide/customizable_dashboards.md +++ b/doc/development/fe_guide/customizable_dashboards.md @@ -19,7 +19,7 @@ To use customizable dashboards: 1. Create your dashboard component. 1. Render an instance of `CustomizableDashboard`. -1. Pass a list of widgets to render. +1. Pass a list of panels to render. For example, a customizable dashboard for users over time: @@ -35,10 +35,10 @@ export default { }, data() { return { - widgets: [ + panels: [ { - component: 'CubeLineChart', // The name of the widget component. - title: s__('ProductAnalytics|Users / Time'), // The title shown on the widget component. + component: 'CubeLineChart', // The name of the panel component. + title: s__('ProductAnalytics|Users / Time'), // The title shown on the panel component. // Gridstack settings based upon https://github.com/gridstack/gridstack.js/tree/master/doc#item-options. // All values are grid row/column numbers up to 12. // We use the default 12 column grid https://github.com/gridstack/gridstack.js#change-grid-columns. @@ -50,17 +50,17 @@ export default { xPos: 0, yPos: 0, }, - // Options that are used to set bespoke values for each widget. - // Available customizations are determined by the widget itself. + // Options that are used to set bespoke values for each panel. + // Available customizations are determined by the panel itself. customizations: {}, - // Chart options defined by the charting library being used by the widget. + // Chart options defined by the charting library being used by the panel. chartOptions: { xAxis: { name: __('Time'), type: 'time' }, yAxis: { name: __('Counts') }, }, - // The data for the widget. - // This could be imported or in this case, a query passed to be used by the widgets API. - // Each widget type determines how it handles this property. + // The data for the panel. + // This could be imported or in this case, a query passed to be used by the panels API. + // Each panel type determines how it handles this property. data: { query: { users: { @@ -78,20 +78,20 @@ export default { <template> <h1>{{ s__('ProductAnalytics|Analytics dashboard') }}</h1> - <customizable-dashboard :widgets="widgets" /> + <customizable-dashboard :panels="panels" /> </template> ``` -The widgets data can be retrieved from a file or API request, or imported through HTML data attributes. +The panels data can be retrieved from a file or API request, or imported through HTML data attributes. -For each widget, a `component` is defined. Each `component` is a component declaration and should be included in -[`vue_shared/components/customizable_dashboard/widgets_base.vue`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/assets/javascripts/vue_shared/components/customizable_dashboard/widgets_base.vue) +For each panel, a `component` is defined. Each `component` is a component declaration and should be included in +[`vue_shared/components/customizable_dashboard/panels_base.vue`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/assets/javascripts/vue_shared/components/customizable_dashboard/panels_base.vue) as a dynamic import, to keep the memory usage down until it is used. For example: ```javascript components: { - CubeLineChart: () => import('ee/product_analytics/dashboards/components/widgets/cube_line_chart.vue') + CubeLineChart: () => import('ee/product_analytics/dashboards/components/panels/cube_line_chart.vue') } ``` diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 58607ac4727..e6353264f39 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -202,7 +202,7 @@ Fields included are: - Other identifiers - Detected At - Location -- Activity +- Activity: Returns `true` if the vulnerability is resolved on the default branch, and `false` if not. - Comments NOTE: |