diff options
Diffstat (limited to 'doc/administration/operations')
10 files changed, 94 insertions, 80 deletions
diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index 194dd8f39e2..ed5014b65e1 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -1,64 +1,9 @@ --- -stage: Enablement -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2021-12-10' --- -# Cleaning up stale Redis sessions +This document was moved to [another location](index.md). -Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis. -Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If -you have been running a large GitLab server (thousands of users) since before -GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis -database after you upgrade to GitLab 7.3. You can also perform a cleanup while -still running GitLab 7.2 or older, but in that case new stale sessions will -start building up again after you clean up. - -In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte -hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with -GitLab 7.3.0, the keys are -prefixed with `session:gitlab:`, so they would look like -`session:gitlab:976aa289e2189b17d7ef525a6702ace9`. Below we describe how to -remove the keys in the old format. - -NOTE: -The instructions below must be modified in accordance with your -configuration settings if you have used the advanced Redis -settings outlined in -[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/README.md). - -First we define a shell function with the proper Redis connection details. - -```shell -rcli() { - # This example works for Omnibus installations of GitLab 7.3 or newer. For an - # installation from source you will have to change the socket path and the - # path to redis-cli. - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" -} - -# test the new shell function; the response should be PONG -rcli ping -``` - -Now we do a search to see if there are any session keys in the old format for -us to clean up. - -```shell -# returns the number of old-format session keys in Redis -rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l -``` - -If the number is larger than zero, you can proceed to expire the keys from -Redis. If the number is zero there is nothing to clean up. - -```shell -# Tell Redis to expire each matched key after 600 seconds. -rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli -# This will print '(integer) 1' for each key that gets expired. -``` - -Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis -background save interval) your Redis database will be compacted. If you are -still using GitLab 7.2, users who are not clicking around in GitLab during the -10 minute expiry window will be signed out of GitLab. +<!-- This redirect file can be deleted after 2021-12-10. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 2cc4e3a4551..31cbd6c457b 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -90,7 +90,7 @@ To start multiple processes: To view the Sidekiq processes in GitLab: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. ## Negate settings diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index 6938f8a7012..bb8eb184302 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -40,6 +40,8 @@ In `/etc/gitlab/gitlab.rb`: ```ruby sidekiq['routing_rules'] = [ + # Do not re-route workers that require their own queue + ['tags=needs_own_queue', nil], # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue ['resource_boundary!=cpu&urgency=high', 'high-urgency'], # Route all database, gitaly and global search workers that are throttled to `throttled` queue @@ -164,3 +166,32 @@ with the migration to avoid losing jobs entirely, especially in a system with long queues of jobs. The migration can be done by following the migration steps mentioned in [Sidekiq job migration](../../raketasks/sidekiq_job_migration.md) + +### Workers that cannot be migrated + +Some workers cannot share a queue with other workers - typically because +they check the size of their own queue - and so must be excluded from +this process. We recommend excluding these from any further worker +routing by adding a rule to keep them in their own queue, for example: + +```ruby +sidekiq['routing_rules'] = [ + ['tags=needs_own_queue', nil], + # ... +] +``` + +These queues will also need to be included in at least one [Sidekiq +queue group](extra_sidekiq_processes.md#start-multiple-processes). + +The following table shows the workers that should have their own queue: + +| Worker name | Queue name | GitLab issue | +| --- | --- | --- | +| `EmailReceiverWorker` | `email_receiver` | [`gitlab-com/gl-infra/scalability#1263`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1263) | +| `ServiceDeskEmailReceiverWorker` | `service_desk_email_receiver` | [`gitlab-com/gl-infra/scalability#1263`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1263) | +| `ProjectImportScheduleWorker` | `project_import_schedule` | [`gitlab-org/gitlab#340630`](https://gitlab.com/gitlab-org/gitlab/-/issues/340630) | +| `HashedStorage::MigratorWorker` | `hashed_storage:hashed_storage_migrator` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | +| `HashedStorage::ProjectMigrateWorker` | `hashed_storage:hashed_storage_project_migrate` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | +| `HashedStorage::ProjectRollbackWorker` | `hashed_storage:hashed_storage_project_rollback` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | +| `HashedStorage::RollbackerWorker` | `hashed_storage:hashed_storage_rollbacker` | [`gitlab-org/gitlab#340629`](https://gitlab.com/gitlab-org/gitlab/-/issues/340629) | diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 5c1271486c0..e30ad4d8248 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -106,7 +106,7 @@ users as long as a large file exists. To disable any more writes to the `authorized_keys` file: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. 1. Clear the **Write to "authorized_keys" file** checkbox. diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 4b16c3b3a7e..7ccfa2739bb 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -8,11 +8,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w Keep your GitLab instance up and running smoothly. -- [Clean up Redis sessions](cleaning_up_redis_sessions.md): Prior to GitLab 7.3, - user sessions did not automatically expire from Redis. If - you have been running a large GitLab server (thousands of users) since before - GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis - database after you upgrade to GitLab 7.3. - [Rake tasks](../../raketasks/index.md): Tasks for common administration and operational processes such as [cleaning up unneeded items from GitLab instance](../../raketasks/cleanup.md), integrity checks, and more. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 765cf64e735..61a9ec8e7d4 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -39,6 +39,11 @@ WARNING: To move repositories into a [Gitaly Cluster](../gitaly/index.md#gitaly-cluster) in GitLab versions 13.12 to 14.1, you must [enable the `gitaly_replicate_repository_direct_fetch` feature flag](../feature_flags.md). +WARNING: +Repositories can be **permanently deleted** by a call to `/projects/:project_id/repository_storage_moves` +that attempts to move a project already stored in a Gitaly Cluster back into that cluster. +See [this issue for more details](https://gitlab.com/gitlab-org/gitaly/-/issues/3752). + Each repository is made read-only for the duration of the move. The repository is not writable until the move has completed. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index e8477eaf686..775761d655d 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -36,6 +36,14 @@ For more details about the Puma configuration, see the ## Puma Worker Killer +Puma forks worker processes as part of a strategy to reduce memory use. + +Each time a worker is created, it shares memory with the primary process and +only uses additional memory when it makes changes or additions to its memory pages. + +Memory use by workers therefore increases over time, and Puma Worker Killer is the +mechanism that recovers this memory. + By default: - The [Puma Worker Killer](https://github.com/schneems/puma_worker_killer) restarts a worker if it @@ -56,6 +64,47 @@ To change the memory limit setting: sudo gitlab-ctl reconfigure ``` +There are costs associated with killing and replacing workers including +reduced capacity to run GitLab, and CPU that is consumed +restarting the workers. `per_worker_max_memory_mb` should be set to a +higher value if the worker killer is replacing workers too often. + +Worker count is calculated based on CPU cores, so a small GitLab deployment +with 4-8 workers may experience performance issues if workers are being restarted +frequently, once or more per minute. This is too often. + +A higher value of `1200` or more would be beneficial if the server has free memory. + +The worker killer checks every 20 seconds, and can be monitored using +[the Puma log](../logs.md#puma_stdoutlog) `/var/log/gitlab/puma/puma_stdout.log`. +For example, for GitLab 13.5: + +```plaintext +PumaWorkerKiller: Out of memory. 4 workers consuming total: 4871.23828125 MB +out of max: 4798.08 MB. Sending TERM to pid 26668 consuming 1001.00390625 MB. +``` + +From this output: + +- The formula that calculates the maximum memory value results in workers + being killed before they reach the `per_worker_max_memory_mb` value. +- The default values for the formula before GitLab 13.5 were 550MB for the primary + and `per_worker_max_memory_mb` specified 850MB for each worker. +- As of GitLab 13.5 the values are primary: 800MB, worker: 1024MB. +- The threshold for workers to be killed is set at 98% of the limit: + + ```plaintext + 0.98 * ( 800 + ( worker_processes * 1024MB ) ) + ``` + +- In the log output above, `0.98 * ( 800 + ( 4 * 1024 ) )` returns the + `max: 4798.08 MB` value. + +Increasing the maximum to `1200`, for example, would set a `max: 5488 MB` value. + +Workers use additional memory on top of the shared memory, how much +depends on a site's use of GitLab. + ## Worker timeout A [timeout of 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/rack_timeout.rb) @@ -95,7 +144,6 @@ considered as a fair tradeoff in a memory-constraint environment. When running Puma in Single mode, some features are not supported: -- Phased restart do not work: [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) - [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) - [Puma Worker Killer](https://gitlab.com/gitlab-org/gitlab/-/issues/300664) diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index 598baa4fcc7..e48ac6e65eb 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -4,7 +4,7 @@ group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Sidekiq MemoryKiller +# Sidekiq MemoryKiller **(FREE SELF)** The GitLab Rails application code suffers from memory leaks. For web requests this problem is made manageable using diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 374eebeb773..814e742b026 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # User lookup via OpenSSH's AuthorizedPrincipalsCommand **(FREE SELF)** -> [Available in](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19911) GitLab -> Community Edition 11.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19911) in GitLab 11.2. The default SSH authentication for GitLab requires users to upload their SSH public keys before they can use the SSH transport. diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md deleted file mode 100644 index 6cee19186f9..00000000000 --- a/doc/administration/operations/unicorn.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'puma.md' -remove_date: '2021-08-26' ---- - -This file was moved to [another location](puma.md). - -<!-- This redirect file can be deleted after <2021-08-26>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> |