diff options
30 files changed, 581 insertions, 80 deletions
diff --git a/GITLAB_PAGES_VERSION b/GITLAB_PAGES_VERSION index 76d05362056..af92bdd9f58 100644 --- a/GITLAB_PAGES_VERSION +++ b/GITLAB_PAGES_VERSION @@ -1 +1 @@ -1.62.0 +1.63.0 diff --git a/app/graphql/types/project_type.rb b/app/graphql/types/project_type.rb index 07ca17bca79..900f37a4b72 100644 --- a/app/graphql/types/project_type.rb +++ b/app/graphql/types/project_type.rb @@ -37,6 +37,10 @@ module Types null: false, description: 'Path of the project.' + field :incident_management_timeline_event_tags, [Types::IncidentManagement::TimelineEventTagType], + null: true, + description: 'Timeline event tags for the project.' + field :sast_ci_configuration, Types::CiConfiguration::Sast::Type, null: true, calls_gitaly: true, diff --git a/config/metrics/counts_28d/20221108101211_merge_request_authors_monthly.yml b/config/metrics/counts_28d/20221108101211_merge_request_authors_monthly.yml new file mode 100644 index 00000000000..5987bde2d14 --- /dev/null +++ b/config/metrics/counts_28d/20221108101211_merge_request_authors_monthly.yml @@ -0,0 +1,22 @@ +--- +key_path: usage_activity_by_stage_monthly.create.merge_request_authors_monthly +description: Number of unique merge request authors +product_section: dev +product_stage: create +product_group: code_review +product_category: code_review +value_type: number +status: active +milestone: "15.6" +introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103334 +time_frame: 28d +data_source: database +data_category: optional +instrumentation_class: CountMergeRequestAuthorsMetric +distribution: +- ce +- ee +tier: +- free +- premium +- ultimate diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 834e044110c..94b9d63bdbb 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -16590,6 +16590,7 @@ Represents a product analytics dashboard widget. | <a id="projecthttpurltorepo"></a>`httpUrlToRepo` | [`String`](#string) | URL to connect to the project via HTTPS. | | <a id="projectid"></a>`id` | [`ID!`](#id) | ID of the project. | | <a id="projectimportstatus"></a>`importStatus` | [`String`](#string) | Status of import background job of the project. | +| <a id="projectincidentmanagementtimelineeventtags"></a>`incidentManagementTimelineEventTags` | [`[TimelineEventTagType!]`](#timelineeventtagtype) | Timeline event tags for the project. | | <a id="projectissuesenabled"></a>`issuesEnabled` | [`Boolean`](#boolean) | Indicates if Issues are enabled for the current user. | | <a id="projectjiraimportstatus"></a>`jiraImportStatus` | [`String`](#string) | Status of Jira import background job of the project. | | <a id="projectjiraimports"></a>`jiraImports` | [`JiraImportConnection`](#jiraimportconnection) | Jira imports into the project. (see [Connections](#connections)) | diff --git a/doc/user/project/integrations/mlflow_client.md b/doc/user/project/integrations/mlflow_client.md new file mode 100644 index 00000000000..82bfd08e926 --- /dev/null +++ b/doc/user/project/integrations/mlflow_client.md @@ -0,0 +1,81 @@ +--- +stage: Create +group: Incubation +info: Machine Learning Experiment Tracking is a GitLab Incubation Engineering program. No technical writer assigned to this group. +--- + +# MLFlow Client Integration **(FREE)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8560) in GitLab 15.6 as an [Alpha](../../../policy/alpha-beta-support.md#alpha-features) release [with a flag](../../../administration/feature_flags.md) named `ml_experiment_tracking`. Disabled by default. + +DISCLAIMER: +MLFlow Client Integration is an experimental feature being developed by the Incubation Engineering Department, +and will receive significant changes over time. + +[MLFlow](https://mlflow.org/) is one of the most popular open source tools for Machine Learning Experiment Tracking. +GitLabs works as a backend to the MLFlow Client, [logging experiments](../ml/experiment_tracking/index.md). +Setting up your integrations requires minimal changes to existing code. + +GitLab plays the role of proxy server, both for artifact storage and tracking data. It reflects the +MLFlow [Scenario 5](https://www.mlflow.org/docs/latest/tracking.html#scenario-5-mlflow-tracking-server-enabled-with-proxied-artifact-storage-access). + +## Enable MFlow Client Integration + +Complete this task to enable MFlow Client Integration. + +Prerequisites: + +- A [personal access token](../../../user/profile/personal_access_tokens.md) for the project, with minimum access level of `api`. +- The project ID. To find the project ID, on the top bar, select **Main menu > Projects** and find your project. On the left sidebar, select **Settings > General**. + +1. Set the tracking URI and token environment variables on the host that runs the code (your local environment, CI pipeline, or remote host). + + For example: + + ```shell + export MLFLOW_TRACKING_URI="http://<your gitlab endpoint>/api/v4/projects/<your project id>/ml/mlflow" + export MLFLOW_TRACKING_TOKEN="<your_access_token>" + ``` + +1. If your training code contains the call to `mlflow.set_tracking_uri()`, remove it. + +When running the training code, MLFlow will create experiments, runs, log parameters, metrics, +and artifacts on GitLab. + +After experiments are logged, they are listed under `/<your project>/-/ml/experiments`. Runs are registered as Model Candidates, +that can be explored by selecting an experiment. + +## Limitations + +- The API GitLab supports is the one defined at MLFlow version 1.28.0. +- API endpoints not listed above are not supported. +- During creation of experiments and runs, tags are ExperimentTags and RunTags are ignored. +- MLFLow Model Registry is not supported. + +## Supported methods and caveats + +This is a list of methods we support from the MLFlow client. Other methods might be supported but were not +tested. More information can be found in the [MLFlow Documentation](https://www.mlflow.org/docs/1.28.0/python_api/mlflow.html). + +### `set_experiment` + +Accepts both experiment_name and experiment_id + +### `start_run()` + +- Nested runs have not been tested. +- `run_name` is not supported + +### `log_param()`, `log_params()`, `log_metric()`, `log_metrics()` + +Work as defined by the documentation + +### `log_artifact()`, `log_artifacts()` + +`artifact_path` must be empty string. + +### `log_model()` + +This is an experimental method in MLFlow, and partial support is offered. It stores the model artifacts, but does +not log the model information. The `artifact_path` parameter must be set to `''`, because Generic Packages do not support folder +structure. diff --git a/doc/user/project/ml/experiment_tracking/img/candidates.png b/doc/user/project/ml/experiment_tracking/img/candidates.png Binary files differnew file mode 100644 index 00000000000..df70a01a2bd --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/img/candidates.png diff --git a/doc/user/project/ml/experiment_tracking/img/experiments.png b/doc/user/project/ml/experiment_tracking/img/experiments.png Binary files differnew file mode 100644 index 00000000000..a6472406b90 --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/img/experiments.png diff --git a/doc/user/project/ml/experiment_tracking/index.md b/doc/user/project/ml/experiment_tracking/index.md new file mode 100644 index 00000000000..e274bd7f38e --- /dev/null +++ b/doc/user/project/ml/experiment_tracking/index.md @@ -0,0 +1,76 @@ +--- +stage: Create +group: Incubation +info: Machine Learning Experiment Tracking is a GitLab Incubation Engineering program. No technical writer assigned to this group. +--- + +# Machine Learning Experiment Tracking **(FREE)** + +DISCLAIMER: +Machine Learning Experiment Tracking is an experimental feature being developed by the Incubation Engineering Department, +and will receive significant changes over time. This feature is being release with the aim of getting user feedback, but +is not stable and can lead to performance degradation. See below on how to disable this feature. + +When creating machine learning models, data scientists often experiment with different parameters, configurations, feature +engineering, and so on, to improve the performance of the model. Keeping track of all this metadata and the associated +artifacts so that the data scientist can later replicate the experiment is not trivial. Machine learning experiment +tracking enables them to log parameters, metrics, and artifacts directly into GitLab, giving easy access later on. + +![List of Experiments](img/experiments.png) + +![Experiment Candidates](img/candidates.png) + +## What is an experiment? + +An experiment is a collection of comparable model candidates. Experiments can be long lived (for example, when they represent +a use case), or short lived (results from hyperparameter tuning triggered by a merge request), but usually hold model candidates +that have a similar set of parameters and metrics. + +## Model candidate + +A model candidate is a variation of the training of a machine learning model, that can be eventually promoted to a version +of the model. The goal of a data scientist is to find the model candidate whose parameter values lead to the best model +performance, as indicated by the given metrics. + +Example parameters: + +- Algorithm (linear regression, decision tree, and so on). +- Hyperparameters for the algorithm (learning rate, tree depth, number of epochs). +- Features included. + +## Usage + +### User access management + +An experiment is always associated to a project. Only users with access to the project an experiment is associated with +can view that experiment data. + +### Tracking new experiments and trials + +Experiment and trials can only be tracked through the [MLFlow](https://www.mlflow.org/docs/latest/tracking.html) client +integration. More information on how to use GitLab as a backend for MLFlow Client can be found [at the documentation page](../../integrations/mlflow_client.md). + +### Exploring model candidates + +To list the current active experiments, navigate to `https/-/ml/experiments`. To display all trials +that have been logged, along with their metrics and parameters, selecting an experiment. + +### Logging artifacts + +Trial artifacts are saved as [generic packages](../../../packages/generic_packages/index.md), and follow all their +conventions. After an artifact is logged for a candidate, all artifacts logged for the candidate are listed in the +package registry. The package name for a candidate is `ml_candidate_<candidate_id>`, with version `-`. + +### Limitations and future + +- Searching experiments, searching trials, visual comparison of trials, and creating, deleting and updating experiments and trials through GitLab UI is under development. +- No support for experiment and trial metadata that do not classify as parameters or metrics. + +## Disabling or enabling the Feature + +On self-managed GitLab, ML Experiment Tracking is disabled by default. To enable the feature, ask an administrator to [disable the feature flag](../../../../administration/feature_flags.md) named `ml_experiment_tracking`. +On GitLab.com, this feature is currently on private testing. + +## Feedback, roadmap and reports + +For updates on the development, feedback and bug reports, refer to the [development epic](https://gitlab.com/groups/gitlab-org/-/epics/8560). diff --git a/lib/api/deploy_keys.rb b/lib/api/deploy_keys.rb index c8ddf081ffc..ffe0b6589bc 100644 --- a/lib/api/deploy_keys.rb +++ b/lib/api/deploy_keys.rb @@ -4,6 +4,8 @@ module API class DeployKeys < ::API::Base include PaginationParams + deploy_keys_tags = %w[deploy_keys] + before { authenticate! } feature_category :continuous_delivery @@ -24,8 +26,12 @@ module API desc 'List all deploy keys' do detail 'Get a list of all deploy keys across all projects of the GitLab instance. This endpoint requires administrator access and is not available on GitLab.com.' success Entities::DeployKey + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' } + ] is_array true - tags %w[deploy_keys] + tags deploy_keys_tags end params do use :pagination @@ -48,8 +54,12 @@ module API desc 'List deploy keys for project' do detail "Get a list of a project's deploy keys." success Entities::DeployKeysProject + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] is_array true - tags %w[deploy_keys] + tags deploy_keys_tags end params do use :pagination @@ -65,7 +75,11 @@ module API desc 'Get a single deploy key' do detail 'Get a single key.' success Entities::DeployKeysProject - tags %w[deploy_keys] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_keys_tags end params do requires :key_id, type: Integer, desc: 'The ID of the deploy key' @@ -79,7 +93,12 @@ module API desc 'Add deploy key' do detail "Creates a new deploy key for a project. If the deploy key already exists in another project, it's joined to the current project only if the original one is accessible by the same user." success Entities::DeployKeysProject - tags %w[deploy_keys] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_keys_tags end params do requires :key, type: String, desc: 'New deploy key' @@ -124,7 +143,13 @@ module API desc 'Update deploy key' do detail 'Updates a deploy key for a project.' success Entities::DeployKey - tags %w[deploy_keys] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' }, + { code: 404, message: 'Not found' } + ] + tags deploy_keys_tags end params do requires :key_id, type: Integer, desc: 'The ID of the deploy key' @@ -160,7 +185,11 @@ module API desc 'Enable a deploy key' do detail 'Enables a deploy key for a project so this can be used. Returns the enabled key, with a status code 201 when successful. This feature was added in GitLab 8.11.' success Entities::DeployKey - tags %w[deploy_keys] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_keys_tags end params do requires :key_id, type: Integer, desc: 'The ID of the deploy key' @@ -178,7 +207,11 @@ module API desc 'Delete deploy key' do detail "Removes a deploy key from the project. If the deploy key is used only for this project, it's deleted from the system." - tags %w[deploy_keys] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_keys_tags end params do requires :key_id, type: Integer, desc: 'The ID of the deploy key' diff --git a/lib/api/deploy_tokens.rb b/lib/api/deploy_tokens.rb index 37824ec7856..975a65af285 100644 --- a/lib/api/deploy_tokens.rb +++ b/lib/api/deploy_tokens.rb @@ -4,6 +4,8 @@ module API class DeployTokens < ::API::Base include PaginationParams + deploy_tokens_tags = %w[deploy_tokens] + feature_category :continuous_delivery urgency :low @@ -28,8 +30,12 @@ module API desc 'List all deploy tokens' do detail 'Get a list of all deploy tokens across the GitLab instance. This endpoint requires administrator access. This feature was introduced in GitLab 12.9.' success Entities::DeployToken + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' } + ] is_array true - tags %w[deploy_tokens] + tags deploy_tokens_tags end params do use :pagination @@ -58,8 +64,13 @@ module API desc 'List project deploy tokens' do detail "Get a list of a project's deploy tokens. This feature was introduced in GitLab 12.9." success Entities::DeployToken + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' }, + { code: 404, message: 'Not found' } + ] is_array true - tags %w[deploy_tokens] + tags deploy_tokens_tags end get ':id/deploy_tokens' do authorize!(:read_deploy_token, user_project) @@ -86,7 +97,12 @@ module API desc 'Create a project deploy token' do detail 'Creates a new deploy token for a project. This feature was introduced in GitLab 12.9.' success Entities::DeployTokenWithToken - tags %w[deploy_tokens] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_tokens_tags end post ':id/deploy_tokens' do authorize!(:create_deploy_token, user_project) @@ -105,7 +121,11 @@ module API desc 'Get a project deploy token' do detail "Get a single project's deploy token by ID. This feature was introduced in GitLab 14.9." success Entities::DeployToken - tags %w[deploy_tokens] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_tokens_tags end params do requires :token_id, type: Integer, desc: 'The ID of the deploy token' @@ -120,7 +140,11 @@ module API desc 'Delete a project deploy token' do detail 'This feature was introduced in GitLab 12.9.' - tags %w[deploy_tokens] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_tokens_tags end params do requires :token_id, type: Integer, desc: 'The ID of the deploy token' @@ -147,8 +171,13 @@ module API desc 'List group deploy tokens' do detail "Get a list of a group's deploy tokens. This feature was introduced in GitLab 12.9." success Entities::DeployToken + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' }, + { code: 404, message: 'Not found' } + ] is_array true - tags %w[deploy_tokens] + tags deploy_tokens_tags end get ':id/deploy_tokens' do authorize!(:read_deploy_token, user_group) @@ -175,7 +204,12 @@ module API desc 'Create a group deploy token' do detail 'Creates a new deploy token for a group. This feature was introduced in GitLab 12.9.' success Entities::DeployTokenWithToken - tags %w[deploy_tokens] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_tokens_tags end post ':id/deploy_tokens' do authorize!(:create_deploy_token, user_group) @@ -194,7 +228,11 @@ module API desc 'Get a group deploy token' do detail "Get a single group's deploy token by ID. This feature was introduced in GitLab 14.9. " success Entities::DeployToken - tags %w[deploy_tokens] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_tokens_tags end params do requires :token_id, type: Integer, desc: 'The ID of the deploy token' @@ -209,7 +247,11 @@ module API desc 'Delete a group deploy token' do detail 'Removes a deploy token from the group. This feature was introduced in GitLab 12.9.' - tags %w[deploy_tokens] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deploy_tokens_tags end params do requires :token_id, type: Integer, desc: 'The ID of the deploy token' diff --git a/lib/api/deployments.rb b/lib/api/deployments.rb index 084236807c9..1b00ec64d73 100644 --- a/lib/api/deployments.rb +++ b/lib/api/deployments.rb @@ -5,6 +5,8 @@ module API class Deployments < ::API::Base include PaginationParams + deployments_tags = %w[deployments] + before { authenticate! } feature_category :continuous_delivery @@ -17,8 +19,13 @@ module API desc 'List project deployments' do detail 'Get a list of deployments in a project. This feature was introduced in GitLab 8.11.' success Entities::Deployment + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] is_array true - tags %w[deployments] + tags deployments_tags end params do use :pagination @@ -68,7 +75,11 @@ module API desc 'Get a specific deployment' do detail 'This feature was introduced in GitLab 8.11.' success Entities::DeploymentExtended - tags %w[deployments] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deployments_tags end params do requires :deployment_id, type: Integer, desc: 'The ID of the deployment' @@ -84,7 +95,12 @@ module API desc 'Create a deployment' do detail 'This feature was introduced in GitLab 12.4.' success Entities::DeploymentExtended - tags %w[deployments] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags deployments_tags end params do requires :environment, @@ -137,7 +153,13 @@ module API desc 'Update a deployment' do detail 'This feature was introduced in GitLab 12.4.' success Entities::DeploymentExtended - tags %w[deployments] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' }, + { code: 404, message: 'Not found' } + ] + tags deployments_tags end params do requires :status, @@ -172,7 +194,7 @@ module API [403, 'Forbidden'], [400, '"Cannot destroy running deployment" or "Deployment currently deployed to environment"'] ] - tags %w[deployments] + tags deployments_tags end params do requires :deployment_id, type: Integer, desc: 'The ID of the deployment' @@ -196,7 +218,12 @@ module API desc 'List of merge requests associated with a deployment' do detail 'Retrieves the list of merge requests shipped with a given deployment. This feature was introduced in GitLab 12.7.' success Entities::MergeRequestBasic - tags %w[deployments] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + is_array true + tags deployments_tags end params do use :pagination diff --git a/lib/api/entities/deploy_key.rb b/lib/api/entities/deploy_key.rb index 2c9c33549a1..1bcd06f2c88 100644 --- a/lib/api/entities/deploy_key.rb +++ b/lib/api/entities/deploy_key.rb @@ -3,9 +3,15 @@ module API module Entities class DeployKey < Entities::SSHKey - expose :key - expose :fingerprint, if: ->(key, _) { key.fingerprint.present? } - expose :fingerprint_sha256 + expose :key, + documentation: { type: 'string', example: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDNJAkI3Wdf0r13c8a5pEExB2YowPWCSVzfZV22pNBc1CuEbyYLHpUyaD0GwpGvFdx2aP7lMEk35k6Rz3ccBF6jRaVJyhsn5VNnW92PMpBJ/P1UebhXwsFHdQf5rTt082cSxWuk61kGWRQtk4ozt/J2DF/dIUVaLvc+z4HomT41fQ==' } + + expose :fingerprint, + documentation: { type: 'string', example: '4a:9d:64:15:ed:3a:e6:07:6e:89:36:b3:3b:03:05:d9' }, + if: ->(key, _) { key.fingerprint.present? } + + expose :fingerprint_sha256, + documentation: { type: 'string', example: 'SHA256:Jrs3LD1Ji30xNLtTVf9NDCj7kkBgPBb2pjvTZ3HfIgU' } expose :projects_with_write_access, using: Entities::ProjectIdentity, if: -> (_, options) { options[:include_projects_with_write_access] } end diff --git a/lib/api/entities/deploy_keys_project.rb b/lib/api/entities/deploy_keys_project.rb index 12a86fbdf8e..4501af88067 100644 --- a/lib/api/entities/deploy_keys_project.rb +++ b/lib/api/entities/deploy_keys_project.rb @@ -4,7 +4,7 @@ module API module Entities class DeployKeysProject < Grape::Entity expose :deploy_key, merge: true, using: Entities::DeployKey - expose :can_push + expose :can_push, documentation: { type: 'boolean' } end end end diff --git a/lib/api/entities/deploy_token.rb b/lib/api/entities/deploy_token.rb index daee104ba6b..9861467e35d 100644 --- a/lib/api/entities/deploy_token.rb +++ b/lib/api/entities/deploy_token.rb @@ -4,8 +4,13 @@ module API module Entities class DeployToken < Grape::Entity # exposing :token is a security risk and should be avoided - expose :id, :name, :username, :expires_at, :scopes, :revoked - expose :expired?, as: :expired + expose :id, documentation: { type: 'integer', example: 1 } + expose :name, documentation: { type: 'string', example: 'MyToken' } + expose :username, documentation: { type: 'string', example: 'gitlab+deploy-token-1' } + expose :expires_at, documentation: { type: 'dateTime', example: '2020-02-14T00:00:00.000Z' } + expose :scopes, documentation: { type: 'array', example: ['read_repository'] } + expose :revoked, documentation: { type: 'boolean' } + expose :expired?, documentation: { type: 'boolean' }, as: :expired end end end diff --git a/lib/api/entities/deploy_token_with_token.rb b/lib/api/entities/deploy_token_with_token.rb index 11efe3720fa..a96051e1403 100644 --- a/lib/api/entities/deploy_token_with_token.rb +++ b/lib/api/entities/deploy_token_with_token.rb @@ -3,7 +3,7 @@ module API module Entities class DeployTokenWithToken < Entities::DeployToken - expose :token + expose :token, documentation: { type: 'string', example: 'jMRvtPNxrn3crTAGukpZ' } end end end diff --git a/lib/api/entities/deployment.rb b/lib/api/entities/deployment.rb index 4e3a4c289d9..426e92e7723 100644 --- a/lib/api/entities/deployment.rb +++ b/lib/api/entities/deployment.rb @@ -3,11 +3,16 @@ module API module Entities class Deployment < Grape::Entity - expose :id, :iid, :ref, :sha, :created_at, :updated_at + expose :id, documentation: { type: 'integer', example: 41 } + expose :iid, documentation: { type: 'integer', example: 1 } + expose :ref, documentation: { type: 'string', example: 'main' } + expose :sha, documentation: { type: 'string', example: '99d03678b90d914dbb1b109132516d71a4a03ea8' } + expose :created_at, documentation: { type: 'dateTime', example: '2016-08-11T11:32:35.444Z' } + expose :updated_at, documentation: { type: 'dateTime', example: '2016-08-11T11:32:35.444Z' } expose :user, using: Entities::UserBasic expose :environment, using: Entities::EnvironmentBasic expose :deployable, using: Entities::Ci::Job - expose :status + expose :status, documentation: { type: 'string', example: 'created' } end end end diff --git a/lib/api/entities/feature.rb b/lib/api/entities/feature.rb index d1151849cd7..48dd5a22a7e 100644 --- a/lib/api/entities/feature.rb +++ b/lib/api/entities/feature.rb @@ -3,8 +3,8 @@ module API module Entities class Feature < Grape::Entity - expose :name - expose :state + expose :name, documentation: { type: 'string', example: 'experimental_feature' } + expose :state, documentation: { type: 'string', example: 'off' } expose :gates, using: Entities::FeatureGate do |model| model.gates.map do |gate| value = model.gate_values[gate.key] diff --git a/lib/api/entities/feature_gate.rb b/lib/api/entities/feature_gate.rb index bea9c9474b3..ad4702bf210 100644 --- a/lib/api/entities/feature_gate.rb +++ b/lib/api/entities/feature_gate.rb @@ -3,8 +3,8 @@ module API module Entities class FeatureGate < Grape::Entity - expose :key - expose :value + expose :key, documentation: { type: 'string', example: 'percentage_of_actors' } + expose :value, documentation: { type: 'integer', example: 34 } end end end diff --git a/lib/api/environments.rb b/lib/api/environments.rb index 33f27f4dace..01d46ee7bfb 100644 --- a/lib/api/environments.rb +++ b/lib/api/environments.rb @@ -5,6 +5,8 @@ module API class Environments < ::API::Base include PaginationParams + environments_tags = %w[environments] + before { authenticate! } feature_category :continuous_delivery @@ -19,10 +21,10 @@ module API success Entities::Environment is_array true failure [ - { code: 403, message: 'Unauthenticated' }, + { code: 401, message: 'Unauthorized' }, { code: 404, message: 'Not found' } ] - tags %w[environments] + tags environments_tags end params do use :pagination @@ -46,11 +48,11 @@ module API detail 'Creates a new environment with the given name and `external_url`. It returns `201` if the environment was successfully created, `400` for wrong parameters. This feature was introduced in GitLab 8.11.' success Entities::Environment failure [ - { code: 403, message: 'Unauthenticated' }, - { code: 404, message: 'Not found' }, - { code: 422, message: 'Unprocessable entity' } + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } ] - tags %w[environments] + tags environments_tags end params do requires :name, type: String, desc: 'The name of the environment' @@ -74,11 +76,11 @@ module API detail 'Updates an existing environment name and/or `external_url`. It returns `200` if the environment was successfully updated. In case of an error, a status code `400` is returned. This feature was introduced in GitLab 8.11.' success Entities::Environment failure [ - { code: 403, message: 'Unauthenticated' }, - { code: 404, message: 'Not found' }, - { code: 422, message: 'Unprocessable entity' } + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } ] - tags %w[environments] + tags environments_tags end params do requires :environment_id, type: Integer, desc: 'The ID of the environment' @@ -106,11 +108,11 @@ module API success Entities::EnvironmentBasic failure [ { code: 400, message: 'Bad request' }, - { code: 403, message: 'Unauthenticated' }, + { code: 401, message: 'Unauthorized' }, { code: 404, message: 'Not found' }, { code: 409, message: 'Conflict' } ] - tags %w[environments] + tags environments_tags end params do optional :before, type: Time, desc: "The date before which environments can be deleted. Defaults to 30 days ago. Expected in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`)", default: -> { 30.days.ago } @@ -139,7 +141,7 @@ module API detail 'It returns 204 if the environment was successfully deleted, and 404 if the environment does not exist. This feature was introduced in GitLab 8.11.' success Entities::Environment failure [ - { code: 403, message: 'Unauthenticated' }, + { code: 401, message: 'Unauthorized' }, { code: 404, message: 'Not found' } ] tags %w[environments] @@ -160,7 +162,7 @@ module API detail 'It returns 200 if the environment was successfully stopped, and 404 if the environment does not exist.' success Entities::Environment failure [ - { code: 403, message: 'Unauthenticated' }, + { code: 401, message: 'Unauthorized' }, { code: 404, message: 'Not found' } ] tags %w[environments] @@ -183,7 +185,7 @@ module API desc 'Get a specific environment' do success Entities::Environment failure [ - { code: 403, message: 'Unauthenticated' }, + { code: 401, message: 'Unauthorized' }, { code: 404, message: 'Not found' } ] tags %w[environments] diff --git a/lib/api/feature_flags.rb b/lib/api/feature_flags.rb index b1599ab9b47..1846ddf6833 100644 --- a/lib/api/feature_flags.rb +++ b/lib/api/feature_flags.rb @@ -4,6 +4,8 @@ module API class FeatureFlags < ::API::Base include PaginationParams + feature_flags_tags = %w[feature_flags] + FEATURE_FLAG_ENDPOINT_REQUIREMENTS = API::NAMESPACE_OR_PROJECT_REQUIREMENTS .merge(name: API::NO_SLASH_URL_PART_REGEX) @@ -22,8 +24,12 @@ module API desc 'List feature flags for a project' do detail 'Gets all feature flags of the requested project. This feature was introduced in GitLab 12.5.' success ::API::Entities::FeatureFlag + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] is_array true - tags %w[feature_flags] + tags feature_flags_tags end params do optional :scope, @@ -43,7 +49,12 @@ module API desc 'Create a new feature flag' do detail 'Creates a new feature flag. This feature was introduced in GitLab 12.5.' success ::API::Entities::FeatureFlag - tags %w[feature_flags] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' } + ] + tags feature_flags_tags end params do requires :name, type: String, desc: 'The name of the feature flag' @@ -88,7 +99,11 @@ module API desc 'Get a single feature flag' do detail 'Gets a single feature flag. This feature was introduced in GitLab 12.5.' success ::API::Entities::FeatureFlag - tags %w[feature_flags] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags feature_flags_tags end get do authorize_read_feature_flag! @@ -100,7 +115,13 @@ module API desc 'Update a feature flag' do detail 'Updates a feature flag. This feature was introduced in GitLab 13.2.' success ::API::Entities::FeatureFlag - tags %w[feature_flags] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' }, + { code: 404, message: 'Not found' }, + { code: 422, message: 'Unprocessable entity' } + ] + tags feature_flags_tags end params do optional :name, type: String, desc: 'The new name of the feature flag. Supported in GitLab 13.3 and later' @@ -144,7 +165,12 @@ module API desc 'Delete a feature flag' do detail 'Deletes a feature flag. This feature was introduced in GitLab 12.5.' success ::API::Entities::FeatureFlag - tags %w[feature_flags] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 403, message: 'Forbidden' }, + { code: 404, message: 'Not found' } + ] + tags feature_flags_tags end delete do authorize_destroy_feature_flag! diff --git a/lib/api/feature_flags_user_lists.rb b/lib/api/feature_flags_user_lists.rb index c5060eae763..aed277d28a2 100644 --- a/lib/api/feature_flags_user_lists.rb +++ b/lib/api/feature_flags_user_lists.rb @@ -4,6 +4,8 @@ module API class FeatureFlagsUserLists < ::API::Base include PaginationParams + feature_flags_user_lists_tags = %w[feature_flags_user_lists] + error_formatter :json, -> (message, _backtrace, _options, _env, _original_exception) { message.is_a?(String) ? { message: message }.to_json : message.to_json } @@ -24,8 +26,12 @@ module API detail 'Gets all feature flag user lists for the requested project. ' \ 'This feature was introduced in GitLab 12.10.' success ::API::Entities::FeatureFlag::UserList + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] is_array true - tags %w[feature_flags_user_lists] + tags feature_flags_user_lists_tags end params do optional :search, type: String, desc: 'Return user lists matching the search criteria' @@ -41,7 +47,12 @@ module API desc 'Create a feature flag user list' do detail 'Creates a feature flag user list. This feature was introduced in GitLab 12.10.' success ::API::Entities::FeatureFlag::UserList - tags %w[feature_flags_user_lists] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags feature_flags_user_lists_tags end params do requires :name, type: String, desc: 'The name of the list' @@ -69,7 +80,11 @@ module API desc 'Get a feature flag user list' do detail 'Gets a feature flag user list. This feature was introduced in GitLab 12.10.' success ::API::Entities::FeatureFlag::UserList - tags %w[feature_flags_user_lists] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags feature_flags_user_lists_tags end get do present user_project.operations_feature_flags_user_lists.find_by_iid!(params[:iid]), @@ -79,7 +94,12 @@ module API desc 'Update a feature flag user list' do detail 'Updates a feature flag user list. This feature was introduced in GitLab 12.10.' success ::API::Entities::FeatureFlag::UserList - tags %w[feature_flags_user_lists] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags feature_flags_user_lists_tags end params do optional :name, type: String, desc: 'The name of the list' @@ -101,7 +121,12 @@ module API desc 'Delete feature flag user list' do detail 'Deletes a feature flag user list. This feature was introduced in GitLab 12.10.' - tags %w[feature_flags_user_lists] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' }, + { code: 409, message: 'Conflict' } + ] + tags feature_flags_user_lists_tags end delete do # TODO: Move the business logic to a service class in app/services/feature_flags. diff --git a/lib/api/features.rb b/lib/api/features.rb index d3c38b3894f..c5255f4e5ef 100644 --- a/lib/api/features.rb +++ b/lib/api/features.rb @@ -4,6 +4,8 @@ module API class Features < ::API::Base before { authenticated_as_admin! } + features_tags = %w[features] + feature_category :feature_flags urgency :low @@ -48,7 +50,7 @@ module API detail 'Get a list of all persisted features, with its gate values.' success Entities::Feature is_array true - tags %w[features] + tags features_tags end get do features = Feature.all @@ -60,7 +62,7 @@ module API detail 'Get a list of all feature definitions.' success Entities::Feature::Definition is_array true - tags %w[features] + tags features_tags end get :definitions do definitions = ::Feature::Definition.definitions.values.map(&:to_h) @@ -72,7 +74,10 @@ module API detail "Set a feature's gate value. If a feature with the given name doesn't exist yet, it's created. " \ "The value can be a boolean, or an integer to indicate percentage of time." success Entities::Feature - tags %w[features] + failure [ + { code: 400, message: 'Bad request' } + ] + tags features_tags end params do requires :value, @@ -149,7 +154,7 @@ module API desc 'Delete a feature' do detail "Removes a feature gate. Response is equal when the gate exists, or doesn't." - tags %w[features] + tags features_tags end delete ':name' do Feature.remove(params[:name]) diff --git a/lib/api/freeze_periods.rb b/lib/api/freeze_periods.rb index 0e991b92901..40f1be83028 100644 --- a/lib/api/freeze_periods.rb +++ b/lib/api/freeze_periods.rb @@ -20,6 +20,10 @@ module API detail 'Paginated list of Freeze Periods, sorted by created_at in ascending order. ' \ 'This feature was introduced in GitLab 13.0.' success Entities::FreezePeriod + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] is_array true tags freeze_periods_tags end @@ -38,6 +42,10 @@ module API desc 'Get a freeze period' do detail 'Get a freeze period for the given `freeze_period_id`. This feature was introduced in GitLab 13.0.' success Entities::FreezePeriod + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] tags freeze_periods_tags end params do @@ -52,6 +60,10 @@ module API desc 'Create a freeze period' do detail 'Creates a freeze period. This feature was introduced in GitLab 13.0.' success Entities::FreezePeriod + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' } + ] tags freeze_periods_tags end params do @@ -78,6 +90,10 @@ module API desc 'Update a freeze period' do detail 'Updates a freeze period for the given `freeze_period_id`. This feature was introduced in GitLab 13.0.' success Entities::FreezePeriod + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' } + ] tags freeze_periods_tags end params do @@ -100,6 +116,9 @@ module API desc 'Delete a freeze period' do detail 'Deletes a freeze period for the given `freeze_period_id`. This feature was introduced in GitLab 13.0.' success Entities::FreezePeriod + failure [ + { code: 401, message: 'Unauthorized' } + ] tags freeze_periods_tags end params do diff --git a/lib/api/release/links.rb b/lib/api/release/links.rb index 3338ea56b8a..c72f90dfdf3 100644 --- a/lib/api/release/links.rb +++ b/lib/api/release/links.rb @@ -5,6 +5,8 @@ module API class Links < ::API::Base include PaginationParams + release_links_tags = %w[release_links] + RELEASE_ENDPOINT_REQUIREMENTS = API::NAMESPACE_OR_PROJECT_REQUIREMENTS .merge(tag_name: API::NO_SLASH_URL_PART_REGEX) @@ -25,8 +27,12 @@ module API desc 'List links of a release' do detail 'Get assets as links from a release. This feature was introduced in GitLab 11.7.' success Entities::Releases::Link + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] is_array true - tags %w[release_links] + tags release_links_tags end params do use :pagination @@ -41,7 +47,11 @@ module API desc 'Create a release link' do detail 'Create an asset as a link from a release. This feature was introduced in GitLab 11.7.' success Entities::Releases::Link - tags %w[release_links] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' } + ] + tags release_links_tags end params do requires :name, type: String, desc: 'The name of the link. Link names must be unique in the release' @@ -73,7 +83,11 @@ module API desc 'Get a release link' do detail 'Get an asset as a link from a release. This feature was introduced in GitLab 11.7.' success Entities::Releases::Link - tags %w[release_links] + failure [ + { code: 401, message: 'Unauthorized' }, + { code: 404, message: 'Not found' } + ] + tags release_links_tags end route_setting :authentication, job_token_allowed: true get do @@ -85,7 +99,11 @@ module API desc 'Update a release link' do detail 'Update an asset as a link from a release. This feature was introduced in GitLab 11.7.' success Entities::Releases::Link - tags %w[release_links] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' } + ] + tags release_links_tags end params do optional :name, type: String, desc: 'The name of the link' @@ -113,7 +131,11 @@ module API desc 'Delete a release link' do detail 'Deletes an asset as a link from a release. This feature was introduced in GitLab 11.7.' success Entities::Releases::Link - tags %w[release_links] + failure [ + { code: 400, message: 'Bad request' }, + { code: 401, message: 'Unauthorized' } + ] + tags release_links_tags end route_setting :authentication, job_token_allowed: true delete do diff --git a/lib/gitlab/usage/metrics/instrumentations/count_merge_request_authors_metric.rb b/lib/gitlab/usage/metrics/instrumentations/count_merge_request_authors_metric.rb new file mode 100644 index 00000000000..a7f8bca8e08 --- /dev/null +++ b/lib/gitlab/usage/metrics/instrumentations/count_merge_request_authors_metric.rb @@ -0,0 +1,15 @@ +# frozen_string_literal: true + +module Gitlab + module Usage + module Metrics + module Instrumentations + class CountMergeRequestAuthorsMetric < DatabaseMetric + operation :distinct_count, column: :author_id + + relation { MergeRequest } + end + end + end + end +end diff --git a/spec/graphql/types/project_type_spec.rb b/spec/graphql/types/project_type_spec.rb index 25f178ca954..30fabb8e9e2 100644 --- a/spec/graphql/types/project_type_spec.rb +++ b/spec/graphql/types/project_type_spec.rb @@ -37,6 +37,7 @@ RSpec.describe GitlabSchema.types['Project'] do ci_template timelogs merge_commit_template squash_commit_template work_item_types recent_issue_boards ci_config_path_or_default packages_cleanup_policy ci_variables timelog_categories fork_targets branch_rules ci_config_variables pipeline_schedules languages + incident_management_timeline_event_tags ] expect(described_class).to include_graphql_fields(*expected_fields) @@ -508,6 +509,12 @@ RSpec.describe GitlabSchema.types['Project'] do it { is_expected.to have_graphql_resolver(Resolvers::Ci::JobTokenScopeResolver) } end + describe 'incident_management_timeline_event_tags field' do + subject { described_class.fields['incidentManagementTimelineEventTags'] } + + it { is_expected.to have_graphql_type(Types::IncidentManagement::TimelineEventTagType) } + end + describe 'agent_configurations' do let_it_be(:project) { create(:project) } let_it_be(:user) { create(:user) } @@ -732,6 +739,60 @@ RSpec.describe GitlabSchema.types['Project'] do end end + describe 'timeline_event_tags' do + let_it_be(:user) { create(:user) } + let_it_be(:project) do + create(:project, + :private, + :repository, + creator_id: user.id, + namespace: user.namespace) + end + + let_it_be(:tag1) do + create(:incident_management_timeline_event_tag, + project: project, + name: 'Tag 1') + end + + let_it_be(:tag2) do + create(:incident_management_timeline_event_tag, + project: project, + name: 'Tag 2') + end + + let(:query) do + %( + query { + project(fullPath: "#{project.full_path}") { + incidentManagementTimelineEventTags { + name + id + } + } + } + ) + end + + let(:tags) do + subject.dig('data', 'project', 'incidentManagementTimelineEventTags') + end + + subject { GitlabSchema.execute(query, context: { current_user: user }).as_json } + + context 'when user has permissions to read project' do + before do + project.add_developer(user) + end + + it 'contains timeline event tags' do + expect(tags.count).to eq(2) + expect(tags.first['name']).to eq(tag1.name) + expect(tags.last['name']).to eq(tag2.name) + end + end + end + describe 'languages' do let_it_be(:user) { create(:user) } let_it_be(:project) do diff --git a/spec/lib/gitlab/request_forgery_protection_spec.rb b/spec/lib/gitlab/request_forgery_protection_spec.rb index fe93fe9c828..10842173365 100644 --- a/spec/lib/gitlab/request_forgery_protection_spec.rb +++ b/spec/lib/gitlab/request_forgery_protection_spec.rb @@ -14,15 +14,9 @@ RSpec.describe Gitlab::RequestForgeryProtection, :allow_forgery_protection do end it 'logs to /dev/null' do - logger = described_class::Controller.new.logger + expect(ActiveSupport::Logger).to receive(:new).with(File::NULL) - # Taken from ActiveSupport.logger_outputs_to? - # There is no equivalent /dev/null stream like STDOUT, so - # we need to extract the path. - logdev = logger.instance_variable_get(:@logdev) - logger_source = logdev.dev - - expect(logger_source.path).to eq(File::NULL) + described_class::Controller.new.logger end describe '.call' do diff --git a/spec/lib/gitlab/usage/metrics/instrumentations/count_merge_request_authors_metric_spec.rb b/spec/lib/gitlab/usage/metrics/instrumentations/count_merge_request_authors_metric_spec.rb new file mode 100644 index 00000000000..92459e92eac --- /dev/null +++ b/spec/lib/gitlab/usage/metrics/instrumentations/count_merge_request_authors_metric_spec.rb @@ -0,0 +1,25 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe Gitlab::Usage::Metrics::Instrumentations::CountMergeRequestAuthorsMetric do + let(:expected_value) { 1 } + let(:start) { 30.days.ago.to_s(:db) } + let(:finish) { 2.days.ago.to_s(:db) } + + let(:expected_query) do + "SELECT COUNT(DISTINCT \"merge_requests\".\"author_id\") FROM \"merge_requests\"" \ + " WHERE \"merge_requests\".\"created_at\" BETWEEN '#{start}' AND '#{finish}'" + end + + before do + user = create(:user) + user2 = create(:user) + + create(:merge_request, created_at: 1.year.ago, author: user) + create(:merge_request, created_at: 1.week.ago, author: user2) + create(:merge_request, created_at: 1.week.ago, author: user2) + end + + it_behaves_like 'a correct instrumented metric value and query', { time_frame: '28d' } +end diff --git a/spec/support/matchers/graphql_matchers.rb b/spec/support/matchers/graphql_matchers.rb index db7d4269945..155a6dba52c 100644 --- a/spec/support/matchers/graphql_matchers.rb +++ b/spec/support/matchers/graphql_matchers.rb @@ -169,7 +169,11 @@ RSpec::Matchers.define :have_graphql_type do |expected, opts = {}| include GraphQLTypeHelpers match do |object| - expect(object.type).to eq(nullified(expected, opts[:null])) + if object.type.list? + expect(object.type.unwrap).to eq(nullified(expected, opts[:null])) + else + expect(object.type).to eq(nullified(expected, opts[:null])) + end end failure_message do |object| diff --git a/vendor/gems/bundler-checksum/lib/bundler/checksum.rb b/vendor/gems/bundler-checksum/lib/bundler/checksum.rb index c8d78eba111..40c42644964 100644 --- a/vendor/gems/bundler-checksum/lib/bundler/checksum.rb +++ b/vendor/gems/bundler-checksum/lib/bundler/checksum.rb @@ -19,7 +19,8 @@ module Bundler cached_checksum = fetch_checksum_from_file(spec) if cached_checksum.nil? - raise SecurityError, "Cached checksum for #{spec.full_name} not found. Please (re-)generate Gemfile.checksum" + raise SecurityError, "Cached checksum for #{spec.full_name} not found. Please (re-)generate Gemfile.checksum with " \ + "`bundle exec bundler-checksum init`. See https://docs.gitlab.com/ee/development/gemfile.html#updating-the-checksum-file." end validate_file_checksum(cached_checksum) |