diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-12 03:22:18 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-12-12 03:22:18 +0300 |
commit | 101d32c3b8d57c73088f1103f87899b0b20531eb (patch) | |
tree | 743e32b7b63be5ce60b28ff0caccedaff65e9fdc /doc/user | |
parent | 6d08f67e10d01b6198daaad31b402e54bfe068e8 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/user')
-rw-r--r-- | doc/user/profile/index.md | 4 | ||||
-rw-r--r-- | doc/user/project/deploy_keys/index.md | 2 | ||||
-rw-r--r-- | doc/user/project/ml/experiment_tracking/mlflow_client.md | 221 | ||||
-rw-r--r-- | doc/user/project/ml/model_registry/index.md | 79 |
4 files changed, 284 insertions, 22 deletions
diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index e3f7513befd..81e2f3d7a55 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -134,7 +134,7 @@ to match your username. > - Mastodon user account [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132892) in GitLab 16.6 [with a flag](../feature_flags.md) named `mastodon_social_ui`. Disabled by default. > - Mastodon user account [generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/428163) in GitLab 16.7. Feature flag `mastodon_social_ui` removed. -You can add links to certain other external accounts you might have, like Skype and Twitter. +You can add links to certain other external accounts you might have, like Skype and X (formerly Twitter). They can help other users connect with you on other platforms. To add links to other accounts: @@ -146,7 +146,7 @@ To add links to other accounts: - LinkedIn profile name. - Mastodon username. - Skype username. - - Twitter @username. + - X (formerly Twitter) @username. Your user ID or username must be 500 characters or less. 1. Select **Update profile settings**. diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index d7b6f1ffc6c..6c2d4763526 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -65,7 +65,7 @@ As with all sensitive information, you should ensure only those who need access For human interactions, use credentials tied to users such as Personal Access Tokens. To help detect a potential secret leak, you can use the -[Audit Event](../../../administration/audit_event_streaming/examples.md#example-payloads-for-ssh-events-with-deploy-key) feature. +[Audit Event](../../../administration/audit_event_streaming/examples.md#example-payloads-for-git-over-ssh-events-with-deploy-key) feature. ## View deploy keys diff --git a/doc/user/project/ml/experiment_tracking/mlflow_client.md b/doc/user/project/ml/experiment_tracking/mlflow_client.md index 9cedb5780ed..35972f0ad7f 100644 --- a/doc/user/project/ml/experiment_tracking/mlflow_client.md +++ b/doc/user/project/ml/experiment_tracking/mlflow_client.md @@ -1,20 +1,20 @@ --- -stage: Create -group: Incubation -info: Machine Learning Experiment Tracking is a GitLab Incubation Engineering program. No technical writer assigned to this group. +stage: ModelOps +group: MLOps +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# MLflow client compatibility **(FREE ALL)** +# MLflow client compatibility **(FREE ALL EXPERIMENT)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8560) in GitLab 15.11 as an [Experiment](../../../../policy/experiment-beta-support.md#experiment) release [with a flag](../../../../administration/feature_flags.md) named `ml_experiment_tracking`. Disabled by default. NOTE: -Model experiment tracking is an [experimental feature](../../../../policy/experiment-beta-support.md). -Refer to <https://gitlab.com/gitlab-org/gitlab/-/issues/381660> for feedback and feature requests. +Model registry and model experiment tracking are [Experiments](../../../../policy/experiment-beta-support.md). +Provide feedback [for model experiment tracking](https://gitlab.com/gitlab-org/gitlab/-/issues/381660). Provide feedback for [model registry](https://gitlab.com/gitlab-org/gitlab/-/epics/9423). [MLflow](https://mlflow.org/) is a popular open source tool for Machine Learning Experiment Tracking. -GitLab [Model experiment tracking](index.md) is compatible with MLflow Client, -[logging experiments](index.md). The setup requires minimal changes to existing code. +GitLab [Model experiment tracking](index.md) and GitLab +[Model registry](../model_registry/index.md) are compatible with the MLflow client. The setup requires minimal changes to existing code. GitLab plays the role of a MLflow server. Running `mlflow server` is not necessary. @@ -37,18 +37,18 @@ To use MLflow client compatibility from a local environment: export MLFLOW_TRACKING_TOKEN="<your_access_token>" ``` -1. If your training code contains the call to `mlflow.set_tracking_uri()`, remove it. +1. If the training code contains the call to `mlflow.set_tracking_uri()`, remove it. -When running the training code, MLflow creates experiments, runs, log parameters, metrics, metadata -and artifacts on GitLab. +## Model experiments + +When running the training code, MLflow client can be used to create experiments, runs, +models, model versions, log parameters, metrics, metadata and artifacts on GitLab. After experiments are logged, they are listed under `/<your project>/-/ml/experiments`. -Runs are registered as: -- Model Candidates, which can be explored by selecting an experiment. -- Tags, which are registered as metadata. +Runs are registered as candidates, which can be explored by selecting an experiment, model, or model version. -## Associating a candidate to a CI/CD job +### Associating a candidate to a CI/CD job > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/119454) in GitLab 16.1. @@ -71,10 +71,179 @@ candidate metadata. To associate a candidate to a CI/CD job: # End of snippet to be included ``` +## Model registry + +You can also manage models and model versions by using the MLflow +client. Models are registered under `/<your project>/-/ml/models`. + +### Models + +#### Creating a model + +```python +client = MlflowClient() +model_name = '<your_model_name>' +description = 'Model description' +model = client.create_registered_model(model_name, description=description) +``` + +**Notes** + +- `create_registered_model` argument `tags` is ignored. +- `name` must be unique within the project. +- `name` cannot be the name of an existing experiment. + +#### Fetching a model + +```python +client = MlflowClient() +model_name = '<your_model_name>' +model = client.get_registered_model(model_name) +``` + +#### Updating a model + +```python +client = MlflowClient() +model_name = '<your_model_name>' +description = 'New description' +client.update_registered_model(model_name, description=description) +``` + +#### Deleting a model + +```python +client = MlflowClient() +model_name = '<your_model_name>' +client.delete_registered_model(model_name) +``` + +### Logging candidates to a model + +Every model has an associated experiment with the same name. To log a candidate/run to the model, +use the experiment with the name of the model: + +```python +client = MlflowClient() +model_name = '<your_model_name>' +exp = client.get_experiment_by_name(model_name) +run = client.create_run(exp.experiment_id) +``` + +### Model version + +#### Creating a model version + +```python +client = MlflowClient() +model_name = '<your_model_name>' +description = 'Model version description' +model_version = client.create_model_version(model_name, source="", description=description) +``` + +**Notes** + +- Argument `run_id` is ignored. Every model version behaves as a Candidate/Run. Creating a mode version from a run is not yet supported. +- Argument `source` is ignored. GitLab will create a package location for the model version files. +- Argument `tags` is ignored. +- Argument `run_link` is ignored. +- Argument `await_creation_for` is ignored. + +#### Updating a model + +```python +client = MlflowClient() +model_name = '<your_model_name>' +version = '<your_version>' +description = 'New description' +client.update_model_version(model_name, version, description=description) +``` + +#### Fetching a model version + +```python +client = MlflowClient() +model_name = '<your_model_name>' +version = '<your_version>' +client.get_model_version(model_name, version) +``` + +#### Getting latest versions of a model + +```python +client = MlflowClient() +model_name = '<your_model_name>' +client.get_latest_versions(model_name) +``` + +**Notes** + +- Argument `stages` is ignored. +- Versions are ordered by last created. + +#### Logging metrics and parameters to a model version + +Every model version is also a candidate/run, allowing users to log parameters +and metrics. The run ID can either be found at the Model version page in GitLab, +or by using the MLflow client: + +```python +client = MlflowClient() +model_name = '<your_model_name>' +version = '<your_version>' +model_version = client.get_model_version(model_name, version) +run_id = model_version.run_id + +# Your training code + +client.log_metric(run_id, '<metric_name>', '<metric_value>') +client.log_param(run_id, '<param_name>', '<param_value>') +client.log_batch(run_id, metric_list, param_list, tag_list) +``` + +#### Logging artifacts to a model version + +GitLab creates a package that can be used by the MLflow client to upload files. + +```python +client = MlflowClient() +model_name = '<your_model_name>' +version = '<your_version>' +model_version = client.get_model_version(model_name, version) +run_id = model_version.run_id + +# Your training code + +client.log_artifact(run_id, '<local/path/to/file.txt>', artifact_path="") +client.log_figure(run_id, figure, artifact_file="my_plot.png") +client.log_dict(run_id, my_dict, artifact_file="my_dict.json") +client.log_image(run_id, image, artifact_file="image.png") +``` + +Artifacts will then be available under `https/<your project>/-/ml/models/<model_id>/versions/<version_id>`. + +#### Linking a model version to a CI/CD job + +Similar to candidates, it is also possible to link a model version to a CI/CD job: + +```python +client = MlflowClient() +model_name = '<your_model_name>' +version = '<your_version>' +model_version = client.get_model_version(model_name, version) +run_id = model_version.run_id + +# Your training code + +if os.getenv('GITLAB_CI'): + client.set_tag(model_version.run_id, 'gitlab.CI_JOB_ID', os.getenv('CI_JOB_ID')) +``` + ## Supported MLflow client methods and caveats GitLab supports these methods from the MLflow client. Other methods might be supported but were not -tested. More information can be found in the [MLflow Documentation](https://www.mlflow.org/docs/1.28.0/python_api/mlflow.html). +tested. More information can be found in the [MLflow Documentation](https://www.mlflow.org/docs/1.28.0/python_api/mlflow.html). The MlflowClient counterparts +of the methods below are also supported with the same caveats. | Method | Supported | Version Added | Comments | |--------------------------|------------------|----------------|-------------------------------------------------------------------------------------| @@ -102,9 +271,23 @@ tested. More information can be found in the [MLflow Documentation](https://www. | `update_run` | Yes | 15.11 | | | `log_model` | Partial | 15.11 | (15.11) Saves the artifacts, but not the model data. `artifact_path` must be empty. | +Other MLflowClient methods: + +| Method | Supported | Version added | Comments | +|---------------------------|------------------|---------------|--------------------------------------------------| +| `create_registered_model` | Yes with caveats | 16.8 | [See notes](#creating-a-model) | +| `get_registered_model` | Yes | 16.8 | | +| `delete_registered_model` | Yes | 16.8 | | +| `update_registered_model` | Yes | 16.8 | | +| `create_model_version` | Yes with caveats | 16.8 | [See notes](#creating-a-model-version) | +| `get_model_version` | Yes | 16.8 | | +| `get_latest_versions` | Yes with caveats | 16.8 | [See notes](#getting-latest-versions-of-a-model) | +| `update_model_version` | Yes | 16.8 | | +| `create_registered_model` | Yes | 16.8 | | +| `create_registered_model` | Yes | 16.8 | | + ## Limitations -- The API GitLab supports is the one defined at MLflow version 1.28.0. -- API endpoints not listed above are not supported. +- The API GitLab supports is the one defined at MLflow version 2.7.1. +- MLflow client methods not listed above are not supported. - During creation of experiments and runs, ExperimentTags are stored, even though they are not displayed. -- MLflow Model Registry is not supported. diff --git a/doc/user/project/ml/model_registry/index.md b/doc/user/project/ml/model_registry/index.md new file mode 100644 index 00000000000..492ec9940ab --- /dev/null +++ b/doc/user/project/ml/model_registry/index.md @@ -0,0 +1,79 @@ +--- +stage: ModelOps +group: MLOps +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Model registry **(FREE ALL EXPERIMENT)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/9423) in GitLab 16.8 as an [Experiment](../../../../policy/experiment-beta-support.md#experiment) release [with a flag](../../../../administration/feature_flags.md) named `ml_experiment_tracking`. Disabled by default. To enable the feature, an administrator can [enable the feature flag](../../../../administration/feature_flags.md) named `model_registry`. + +NOTE: +Model registry is an [Experiment](../../../../policy/experiment-beta-support.md). [Provide feedback](https://gitlab.com/groups/gitlab-org/-/epics/9423). + +Model registry allows data scientists and developers to manage their machine learning +models, along with all metadata associated with their creation: parameters, performance +metrics, artifacts, logs and more. For the full list of currently supported features, +see [epic 9423](https://gitlab.com/groups/gitlab-org/-/epics/9423). + +## Access the model registry + +To set the model registry [visibility level](../../../public_access.md) to public, private or disabled: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Under **Model registry**, ensure the toggle is on and select who you want to have access. + Users must have +at least the [Reporter role](../../../permissions.md#roles) to modify or delete models and model versions. + +## Exploring models, model versions and model candidates + +Model registry can be accessed on `https/<your-project>-/ml/models`. + +## Creating machine learning models and model versions + +Models and model versions can be created using the [MLflow](https://www.mlflow.org/docs/latest/tracking.html) client compatibility. +See [MLflow client compatibility](../experiment_tracking/mlflow_client.md#model-registry) on how to +create and manage models and model versions. + +## Upload files, log metrics, log parameters to a model version + +Files can either be uploaded to a model version using: + +- The package registry, where a model version is associated to a package of name `<model_name>/<model_version>`. +- The MLflow client compatibility. [View details](../experiment_tracking/mlflow_client.md#logging-artifacts-to-a-model-version). + +Users can log metrics and a parameters of a model version through the MLflow client compatibility, [see details](../experiment_tracking/mlflow_client.md#logging-metrics-and-parameters-to-a-model-version) + +## Link a model version to a CI/CD job + +When creating a model version through a GitLab CI/CD job, you can link the model +version to the job, giving easy access to the job's logs, merge request, and pipeline. +This can be done through the MLflow client compatibility. [View details](../experiment_tracking/mlflow_client.md#linking-a-model-version-to-a-cicd-job). + +## Model versions and semantic versioning + +The version of a model version in GitLab must follow [Semantic Version specification](https://semver.org/). +Using semantic versioning facilitates model deployment, by communicating which +if a new version can be deployed without changes to the application: + +- A change in the major component signifies a breaking change in the model, and that the application + that consumes the model must be updated to properly use this new version. + A new algorithm or the addition of a mandatory feature column are examples of breaking + changes that would require a bump at the major component. + +- A change in the minor component signifies a non-breaking change, and that the + consumer can safely use the new version without breaking, although it might + need to be updated to use its new functionality. For example, adding a non-mandatory + feature column to the model is a minor bump, because when that feature is not passed, + it will still work. + +- A change in the patch component means that a new version is out that does not + require any action by the application. For example, a daily retrain of the + model does not change the feature set or how the application consumes the + model version. Auto updating to a new patch is a safe update. + +## Related topics + +- Development details, feedback, and feature requests in [epic 9423](https://gitlab.com/groups/gitlab-org/-/epics/9423). |