diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2023-05-19 06:08:19 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2023-05-19 06:08:19 +0300 |
commit | ec1f0dd7f46a6d25e4a47c8254ac67b720f81e89 (patch) | |
tree | cea26c43c6a48b785eba0e97a20592e4545028e1 /doc/development | |
parent | 1f9bddaf87f0b6c93a9617bad0ae731baf16d268 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development')
-rw-r--r-- | doc/development/ai_features.md | 6 | ||||
-rw-r--r-- | doc/development/feature_flags/index.md | 40 | ||||
-rw-r--r-- | doc/development/sec/analyzer_development_guide.md | 6 |
3 files changed, 49 insertions, 3 deletions
diff --git a/doc/development/ai_features.md b/doc/development/ai_features.md index 25da99addc1..f8d19dc26bf 100644 --- a/doc/development/ai_features.md +++ b/doc/development/ai_features.md @@ -66,7 +66,9 @@ All AI features are experimental. 1. Enable **Experiment features** 1. Enable **Third-party AI services** 1. Enable the specific feature flag for the feature you want to test -1. Set either the required access token `OpenAi` or `Vertex`. Ask in [`#ai_enablement_team`](https://gitlab.slack.com/archives/C051K31F30R) to receive an access token. +1. Set the required access token. To receive an access token: + 1. For Vertex, follow the [instructions below](#configure-gcp-vertex-access). + 1. For all other providers, create an access request where `@m_gill`, `@wayne`, and `@timzallmann` are the tech stack owners. ### Set up the embedding database @@ -102,7 +104,7 @@ tail -f log/llm.log In order to obtain a GCP service key for local development, please follow the steps below: -- Create a sandbox GCP environment by visiting [this page](https://about.gitlab.com/handbook/infrastructure-standards/#individual-environment) and following the instructions +- Create a sandbox GCP environment by visiting [this page](https://about.gitlab.com/handbook/infrastructure-standards/#individual-environment) and following the instructions, or by requesting access to our existing group environment by using [this template](https://gitlab.com/gitlab-com/it/infra/issue-tracker/-/issues/new?issuable_template=gcp_group_account_iam_update_request). At this time, access to any endpoints outside of `text-bison` or `chat-bison` must be made through the group environment. - In the GCP console, go to `IAM & Admin` > `Service Accounts` and click on the "Create new service account" button - Name the service account something specific to what you're using it for. Select Create and Continue. Under `Grant this service account access to project`, select the role `Vertex AI User`. Select `Continue` then `Done` - Select your new service account and `Manage keys` > `Add Key` > `Create new key`. This will download the **private** JSON credentials for your service account. Your full settings should then be: diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index c2026ab8966..4b0ad30e3ad 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -149,7 +149,8 @@ created using the [Experiment Tracking template](https://gitlab.com/gitlab-org/g `worker` feature flags are used for controlling Sidekiq workers behavior, such as deferring Sidekiq jobs. `worker` feature flags likely do not have any YAML definition as the name could be dynamically generated using -the worker name itself, e.g. `defer_sidekiq_jobs:AuthorizedProjectsWorker`. +the worker name itself, e.g. `defer_sidekiq_jobs:AuthorizedProjectsWorker`. Some examples for using `worker` type feature +flags can be found in [deferring Sidekiq jobs](#deferring-sidekiq-jobs). ## Feature flag definition and validation @@ -705,3 +706,40 @@ test is written to enable/disable a feature flag explicitly. When a feature flag is changed on Staging or on GitLab.com, a Slack message will be posted to the `#qa-staging` or `#qa-production` channels to inform the pipeline triage DRI so that they can more easily determine if any failures are related to a feature flag change. However, if you are working on a change you can help to avoid unexpected failures by [confirming that the end-to-end tests pass with a feature flag enabled.](../testing_guide/end_to_end/feature_flags.md#confirming-that-end-to-end-tests-pass-with-a-feature-flag-enabled) + +## Controlling Sidekiq worker behavior with feature flags + +Feature flags with [`worker` type](#worker-type) can be used to control the behavior of a Sidekiq worker. + +### Deferring Sidekiq jobs + +Feature flags with the format of `defer_sidekiq_jobs:{WorkerName}` delay the execution of the worker +by scheduling the job at a later time. +Deferring jobs can be useful during an incident where contentious behavior from +worker instances are saturating infrastructure resources (such as database and database connection pool). +The implementation can be found at [DeferJobs Sidekiq server middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/sidekiq_middleware/defer_jobs.rb). + +NOTE: +Jobs are deferred indefinitely as long as the feature flag is enabled. It is important to disable the +feature flag after the worker is deemed safe to continue processing. + +When set to true, 100% of the jobs are deferred. When you want processing to resume, you can +use a **percentage of time** rollout. For example: + +```shell +# defer 100% of the jobs +/chatops run feature set defer_sidekiq_jobs:SlowRunningWorker true + +# defer 99% of the jobs, only letting 1% processed +/chatops run feature set defer_sidekiq_jobs:SlowRunningWorker 99 + +# defer 50% of the jobs +/chatops run feature set defer_sidekiq_jobs:SlowRunningWorker 50 + +# stop deferring the jobs, jobs are being processed normally +/chatops run feature set defer_sidekiq_jobs:SlowRunningWorker false +``` + +NOTE: +The percentage of time value denotes the percentage of time the jobs are being deferred (instead of being processed). +For example, setting to `99` means only 1% of the jobs are being processed at random. diff --git a/doc/development/sec/analyzer_development_guide.md b/doc/development/sec/analyzer_development_guide.md index 31058427b55..3f1420b8d3d 100644 --- a/doc/development/sec/analyzer_development_guide.md +++ b/doc/development/sec/analyzer_development_guide.md @@ -361,3 +361,9 @@ This issue will guide you through the whole release process. In general, you hav - Max role: `Developer` - Scope of the associated `GITLAB_TOKEN`: - Expiry Date of the associated `GITLAB_TOKEN`: + +#### Dependency updates + +All dependencies and upstream scanners (if any) used in the analyzer source are updated on a monthly cadence which primarily includes security fixes and non-breaking changes. + +- Static Analysis team uses a custom internal tool ([SastBot](https://gitlab.com/gitlab-org/security-products/analyzers/sast-analyzer-deps-bot#dependency-update-automation)) to automate dependency management of all the SAST analyzers. SastBot generates MRs on the **8th of each month** and distributes their assignment among Static Analysis team members to take them forward for review. For details on the process, see [Dependency Update Automation](https://gitlab.com/gitlab-org/security-products/analyzers/sast-analyzer-deps-bot#dependency-update-automation). |