diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-03-19 12:09:27 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-03-19 12:09:27 +0300 |
commit | 2af90cef2e2e9c776eae4394a43dba3be7f33d1e (patch) | |
tree | bb4bc691caa6cc74b45720ecd779517f9c8c2cd3 /doc/administration/high_availability | |
parent | cf58004721ee715dd3884476f6fa0c62a7e7f247 (diff) |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/administration/high_availability')
-rw-r--r-- | doc/administration/high_availability/README.md | 2 | ||||
-rw-r--r-- | doc/administration/high_availability/sidekiq.md | 186 |
2 files changed, 187 insertions, 1 deletions
diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index c74beb11241..88f1079dec5 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -110,7 +110,7 @@ them. | [PgBouncer](../../development/architecture.md#pgbouncer) | Database Pool Manager | [PgBouncer HA configuration](pgbouncer.md) **(PREMIUM ONLY)** | | [Redis](../../development/architecture.md#redis)[^3] with Redis Sentinel | Key/Value store for shared data with HA watcher service | [Redis HA configuration](redis.md) | | [Gitaly](../../development/architecture.md#gitaly)[^2] [^5] [^7] | Recommended high-level storage for Git repository data | [Gitaly HA configuration](gitaly.md) | -| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/Background jobs | | +| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/Background jobs | [Sidekiq configuration](sidekiq.md) | | [GitLab application nodes](../../development/architecture.md#unicorn)[^1] | (Unicorn / Puma, Workhorse) - Web-requests (UI, API, Git over HTTP) | [GitLab app HA/scaling configuration](gitlab.md) | | [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling/HA](monitoring_node.md) | diff --git a/doc/administration/high_availability/sidekiq.md b/doc/administration/high_availability/sidekiq.md new file mode 100644 index 00000000000..3d120ea8f09 --- /dev/null +++ b/doc/administration/high_availability/sidekiq.md @@ -0,0 +1,186 @@ +--- +type: reference +--- + +# Configuring Sidekiq + +This section discusses how to configure an external Sidekiq instance. + +Sidekiq requires connection to the Redis, PostgreSQL and Gitaly instance. +To configure the Sidekiq node: + +1. SSH into the Sidekiq server. + +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package +you want using steps 1 and 2 from the GitLab downloads page. +**Do not complete any other steps on the download page.** + +1. Open `/etc/gitlab/gitlab.rb` with your editor. + +1. Generate the Sidekiq configuration: + + ```ruby + sidekiq['listen_address'] = "10.10.1.48" + + ## Optional: Enable extra Sidekiq processes + sidekiq_cluster['enable'] = true + sidekiq_cluster['enable'] = true + "elastic_indexer" + ] + ``` + +1. Setup Sidekiq's connection to Redis: + + ```ruby + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' + + ## The same password for Redis authentication you set up for the master node. + redis['master_password'] = 'YOUR_PASSOWORD' + + ## A list of sentinels with `host` and `port` + gitlab_rails['redis_sentinels'] = [ + {'host' => '10.10.1.34', 'port' => 26379}, + {'host' => '10.10.1.35', 'port' => 26379}, + {'host' => '10.10.1.36', 'port' => 26379}, + ] + ``` + +1. Setup Sidekiq's connection to Gitaly: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, + }) + gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' + ``` + +1. Setup Sidekiq's connection to Postgres: + + ```ruby + gitlab_rails['db_host'] = '10.10.1.30' + gitlab_rails['db_password'] = 'YOUR_PASSOWORD' + gitlab_rails['db_port'] = '5432' + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['auto_migrate'] = false + ``` + + Remember to add the Sidekiq nodes to the Postgres whitelist: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32 10.10.1.31/32 10.10.1.32/32 10.10.1.33/32 10.10.1.38/32) + ``` + +1. Disable other services: + + ```ruby + nginx['enable'] = false + grafana['enable'] = false + prometheus['enable'] = false + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + unicorn['enable'] = false + gitlab_exporter['enable'] = false + ``` + +1. Run `gitlab-ctl reconfigure`. + +## Example configuration + +Here's what the ending `/etc/gitlab/gitlab.rb` would look like: + +```ruby +######################################## +##### Services Disabled ### +######################################## + +nginx['enable'] = false +grafana['enable'] = false +prometheus['enable'] = false +gitlab_rails['auto_migrate'] = false +alertmanager['enable'] = false +gitaly['enable'] = false +gitlab_monitor['enable'] = false +gitlab_workhorse['enable'] = false +nginx['enable'] = false +postgres_exporter['enable'] = false +postgresql['enable'] = false +redis['enable'] = false +redis_exporter['enable'] = false +unicorn['enable'] = false +gitlab_exporter['enable'] = false + +######################################## +#### Redis ### +######################################## + +## Must be the same in every sentinel node +redis['master_name'] = 'gitlab-redis' + +## The same password for Redis authentication you set up for the master node. +redis['master_password'] = 'YOUR_PASSOWORD' + +## A list of sentinels with `host` and `port` +gitlab_rails['redis_sentinels'] = [ + {'host' => '10.10.1.34', 'port' => 26379}, + {'host' => '10.10.1.35', 'port' => 26379}, + {'host' => '10.10.1.36', 'port' => 26379}, + ] + +####################################### +### Gitaly ### +####################################### + +git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, +}) +gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' + +####################################### +### Postgres ### +####################################### +gitlab_rails['db_host'] = '10.10.1.30' +gitlab_rails['db_password'] = 'YOUR_PASSOWORD' +gitlab_rails['db_port'] = '5432' +gitlab_rails['db_adapter'] = 'postgresql' +gitlab_rails['db_encoding'] = 'unicode' +gitlab_rails['auto_migrate'] = false + +####################################### +### Sidekiq configuration ### +####################################### +sidekiq['listen_address'] = "10.10.1.48" + +####################################### +### Monitoring configuration ### +####################################### +consul['enable'] = true +consul['monitoring_service_discovery'] = true + +consul['configuration'] = { + bind_addr: '10.10.1.48', + retry_join: %w(10.10.1.34 10.10.1.35 10.10.1.36) +} + +# Set the network addresses that the exporters will listen on +node_exporter['listen_address'] = '10.10.1.48:9100' + +# Rails Status for prometheus +gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] +``` + +## Further reading + +Related Sidekiq configuration: + +1. [Extra Sidekiq processes](../operations/extra_sidekiq_processes.md) +1. [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) |