terraform: Create Ansible playbook to roll out configuration

Right now, our Terraform scripts only handle creation of machines in
GCP, but not their respective configuration. As it's a task we're doing
rather frequently which takes some time, this commit creates an Ansible
playbook which automates this task.

On creation of the cluster, our scripts now automatically generate a
`hosts.ini` file containing all necessary connection information. With
this file, the admin may now run `./configure-demo-cluster`, which will
invoke Ansible and automatically generate and apply configuration for
each of the nodes, restarting services as required. The task can be run
repeatedly in order to update configuration on the target nodes.

15 files changed, 263 insertions, 5 deletions

diff --git a/_support/terraform/.gitignore b/_support/terraform/.gitignore
index 5a6a4426c..5102a820f 100644
--- a/_support/terraform/.gitignore
+++ b/_support/terraform/.gitignore
@@ -1,3 +1,4 @@
-/.terraform*
 /*.tfstate*
+/.terraform*
+/hosts.ini
 /terraform.tfvars
diff --git a/_support/terraform/README.md b/_support/terraform/README.md
index cbd1f1ea0..50b00455b 100644
--- a/_support/terraform/README.md
+++ b/_support/terraform/README.md
@@ -31,10 +31,21 @@ running in the background on your new VM's.
 One of the provisioned resources is the database, which can take up to 10
 minutes to be created.
 
-### 4. Use SSH to manually configure the hosts
+### 4. Configure demo cluster
 
-Updating the config for all the demo cluster hosts is not yet
-automated. Please follow the documentation at
+```
+./configure-demo-cluster
+```
+
+Configuration of the cluster has been automated via Ansible. The cluster
+creation script has automatically created a `hosts.ini` file for use by
+Ansible containing all necessary information to configure the cluster.
+
+If you do not have Ansible installed, please refer to [Ansible's
+documentation](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html)
+to install it on your system.
+
+If you wish to manually configure the cluster, please consult
 https://docs.gitlab.com/ee/administration/gitaly/praefect.html.
 
 To see the list of IP's for your machines, run:
diff --git a/_support/terraform/configure-demo-cluster b/_support/terraform/configure-demo-cluster
new file mode 100755
index 000000000..4471cb7f1
--- /dev/null
+++ b/_support/terraform/configure-demo-cluster
@@ -0,0 +1,2 @@
+#!/usr/bin/env sh
+exec ansible-playbook -i hosts.ini playbook.yml "$@"
diff --git a/_support/terraform/create-demo-cluster b/_support/terraform/create-demo-cluster
index da352cb72..1e8a57e3c 100755
--- a/_support/terraform/create-demo-cluster
+++ b/_support/terraform/create-demo-cluster
@@ -4,10 +4,12 @@
 require 'erb'
 require 'etc'
 require 'io/console'
+require 'json'
 
 require_relative 'helper.rb'
 
 TFVARS = 'terraform.tfvars'
+HOSTS = 'hosts.ini'
 
 def main
   unless gcloud_appliction_default_logged_in?
@@ -25,6 +27,12 @@ def main
   end
 
   run!(%w[terraform apply])
+
+  @tfstate = JSON.parse(File.read("terraform.tfstate"))
+
+  unless File.exist?(HOSTS)
+    render!(HOSTS, 'hosts.ini.erb')
+  end
 end
 
 def praefect_demo_cluster_name
@@ -33,7 +41,7 @@ def praefect_demo_cluster_name
 end
 
 def praefect_sql_password
-  get_input(
+  @praefect_sql_password ||= get_input(
     'Enter a password for the praefect PostgreSQL user',
     'PRAEFECT_SQL_PASSWORD',
     echo: false
diff --git a/_support/terraform/hosts.ini.erb b/_support/terraform/hosts.ini.erb
new file mode 100644
index 000000000..74307ca63
--- /dev/null
+++ b/_support/terraform/hosts.ini.erb
@@ -0,0 +1,22 @@
+[gitalies]<% @tfstate["outputs"]["gitaly_ssh_ip"]["value"].each do |gitaly, address| %>
+<%= address %> internal=<%= @tfstate["outputs"]["gitaly_internal_ip"]["value"][gitaly] %><% end %>
+
+[praefects]<% @tfstate["outputs"]["praefect_ssh_ip"]["value"].each do |praefect, address| %>
+<%= address %> internal=<%= @tfstate["outputs"]["praefect_internal_ip"]["value"][praefect] %><% end %>
+
+[databases]
+<%= @tfstate["outputs"]["praefect_pgbouncer_ip"]["value"] %>
+
+[gitlabs]
+<%= @tfstate["outputs"]["gitlab_external_ip"]["value"] %> internal=<%= @tfstate["outputs"]["gitlab_internal_ip"]["value"] %>
+
+[loadbalancers]
+<%= @tfstate["outputs"]["praefect_loadbalancer_ip"]["value"] %>
+
+[all:vars]
+ansible_become=yes
+praefect_database_password=<%= praefect_sql_password %>
+praefect_external_token=PRAEFECT_EXTERNAL_TOKEN
+praefect_internal_token=PRAEFECT_INTERNAL_TOKEN
+gitlab_shell_secret_token=GITLAB_SHELL_SECRET_TOKEN
+grafana_password=GRAFANA_PASSWORD
diff --git a/_support/terraform/playbook.yml b/_support/terraform/playbook.yml
new file mode 100644
index 000000000..0dceb96a2
--- /dev/null
+++ b/_support/terraform/playbook.yml
@@ -0,0 +1,20 @@
+- name: Gitaly setup
+  hosts: gitalies
+  roles:
+    - common
+    - gitaly
+  gather_facts: no
+
+- name: Praefect setup
+  hosts: praefects
+  roles:
+    - common
+    - praefect
+  gather_facts: no
+
+- name: GitLab Application server setup
+  hosts: gitlabs
+  roles:
+    - common
+    - gitlab
+  gather_facts: no
diff --git a/_support/terraform/roles/common/handlers/main.yml b/_support/terraform/roles/common/handlers/main.yml
new file mode 100644
index 000000000..3907d4fca
--- /dev/null
+++ b/_support/terraform/roles/common/handlers/main.yml
@@ -0,0 +1,7 @@
+- name: gitlab-ctl reconfigure
+  command:
+    cmd: gitlab-ctl reconfigure
+
+- name: gitlab-ctl restart
+  command:
+    cmd: gitlab-ctl restart
diff --git a/_support/terraform/roles/gitaly/tasks/main.yml b/_support/terraform/roles/gitaly/tasks/main.yml
new file mode 100644
index 000000000..d5c47eea1
--- /dev/null
+++ b/_support/terraform/roles/gitaly/tasks/main.yml
@@ -0,0 +1,7 @@
+- name: /etc/gitlab/gitlab.rb
+  template:
+    src: gitaly-gitlab.rb.j2
+    dest: /etc/gitlab/gitlab.rb
+  notify:
+    - gitlab-ctl reconfigure
+    - gitlab-ctl restart
diff --git a/_support/terraform/roles/gitaly/templates/gitaly-gitlab.rb.j2 b/_support/terraform/roles/gitaly/templates/gitaly-gitlab.rb.j2
new file mode 100644
index 000000000..2f8da8f4e
--- /dev/null
+++ b/_support/terraform/roles/gitaly/templates/gitaly-gitlab.rb.j2
@@ -0,0 +1,47 @@
+# Disable all other services on the Praefect node
+postgresql['enable'] = false
+redis['enable'] = false
+nginx['enable'] = false
+grafana['enable'] = false
+puma['enable'] = false
+sidekiq['enable'] = false
+gitlab_workhorse['enable'] = false
+prometheus_monitoring['enable'] = false
+
+# Enable only the Gitaly service
+gitaly['enable'] = true
+
+# Enable Prometheus if needed
+prometheus['enable'] = true
+
+# Prevent database connections during 'gitlab-ctl reconfigure'
+gitlab_rails['rake_cache_clear'] = false
+gitlab_rails['auto_migrate'] = false
+
+# Make Gitaly accept connections on all network interfaces.
+# Use firewalls to restrict access to this address/port.
+gitaly['listen_addr'] = '0.0.0.0:8075'
+
+# Enable Prometheus metrics access to Gitaly. You must use firewalls
+# to restrict access to this address/port.
+gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
+
+gitaly['auth_token'] = '{{ praefect_internal_token }}'
+
+gitlab_shell['secret_token'] = '{{ gitlab_shell_secret_token }}'
+
+# Configure the gitlab-shell API callback URL. Without this, `git push` will
+# fail. This can be your front door GitLab URL or an internal load balancer.
+# Examples: 'https://example.gitlab.com', 'http://1.2.3.4'
+gitlab_rails['internal_api_url'] = 'http://{{ hostvars[groups['gitlabs'][0]]['internal'] }}'
+
+# You can include the data dirs for all nodes in the same config, because
+# Praefect will only route requests according to the addresses provided in the
+# prior step.
+git_data_dirs({
+{% for host in groups['gitalies'] %}
+  'gitaly-{{ loop.index }}' => {
+    "path" => "/var/opt/gitlab/git-data"
+  },
+{% endfor %}
+})
diff --git a/_support/terraform/roles/gitlab/handlers/main.yml b/_support/terraform/roles/gitlab/handlers/main.yml
new file mode 100644
index 000000000..0c2b34ec2
--- /dev/null
+++ b/_support/terraform/roles/gitlab/handlers/main.yml
@@ -0,0 +1,18 @@
+- name: wait gitlab
+  uri:
+    url: "http://{{ groups.gitlabs[0] }}"
+  register: result
+  until: result.status == 200
+  retries: 60
+  delay: 5
+  changed_when: false
+
+- name: verify gitaly configuration
+  command:
+    cmd: gitlab-rake gitlab:gitaly:check
+
+- name: verify gitlab-shell
+  command:
+    cmd: /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml
+  delegate_to: '{{ item }}'
+  with_items: '{{ groups.gitalies }}'
diff --git a/_support/terraform/roles/gitlab/tasks/main.yml b/_support/terraform/roles/gitlab/tasks/main.yml
new file mode 100644
index 000000000..6dce64975
--- /dev/null
+++ b/_support/terraform/roles/gitlab/tasks/main.yml
@@ -0,0 +1,15 @@
+- name: /etc/gitlab/gitlab.rb
+  template:
+    src: gitlab-gitlab.rb.j2
+    dest: /etc/gitlab/gitlab.rb
+  notify:
+    - gitlab-ctl reconfigure
+    - gitlab-ctl restart
+    - wait gitlab
+    - verify gitaly configuration
+    - verify gitlab-shell
+
+- name: Set Grafana password
+  command:
+    cmd: gitlab-ctl set-grafana-password
+    stdin: "{{ grafana_password }}\n{{ grafana_password }}\n"
diff --git a/_support/terraform/roles/gitlab/templates/gitlab-gitlab.rb.j2 b/_support/terraform/roles/gitlab/templates/gitlab-gitlab.rb.j2
new file mode 100644
index 000000000..814e81e7d
--- /dev/null
+++ b/_support/terraform/roles/gitlab/templates/gitlab-gitlab.rb.j2
@@ -0,0 +1,37 @@
+external_url 'http://{{ groups['gitlabs'][0] }}'
+
+gitaly['enable'] = false
+
+git_data_dirs({
+  "default" => {
+    "gitaly_address" => "tcp://{{ groups['loadbalancers'][0] }}:2305",
+    "gitaly_token" => '{{ praefect_external_token }}'
+  }
+})
+
+gitlab_shell['secret_token'] = '{{ gitlab_shell_secret_token }}'
+
+prometheus['scrape_configs'] = [
+  {
+    'job_name' => 'praefect',
+    'static_configs' => [
+      'targets' => [
+{% for host in groups['praefects'] %}
+        '{{ hostvars[host]['internal'] }}:9652', # praefect-{{ loop.index }}
+{% endfor %}
+      ]
+    ]
+  },
+  {
+    'job_name' => 'praefect-gitaly',
+    'static_configs' => [
+      'targets' => [
+{% for host in groups['gitalies'] %}
+        '{{ hostvars[host]['internal'] }}:9236', # gitaly-{{ loop.index }}
+{% endfor %}
+      ]
+    ]
+  }
+]
+
+grafana['disable_login_form'] = false
diff --git a/_support/terraform/roles/praefect/handlers/main.yml b/_support/terraform/roles/praefect/handlers/main.yml
new file mode 100644
index 000000000..d7865437b
--- /dev/null
+++ b/_support/terraform/roles/praefect/handlers/main.yml
@@ -0,0 +1,7 @@
+- name: verify database configuration
+  command:
+    cmd: /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping
+
+- name: verify node configuration
+  command:
+    cmd: /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes
diff --git a/_support/terraform/roles/praefect/tasks/main.yml b/_support/terraform/roles/praefect/tasks/main.yml
new file mode 100644
index 000000000..595178158
--- /dev/null
+++ b/_support/terraform/roles/praefect/tasks/main.yml
@@ -0,0 +1,9 @@
+- name: /etc/gitlab/gitlab.rb
+  template:
+    src: praefect-gitlab.rb.j2
+    dest: /etc/gitlab/gitlab.rb
+  notify:
+    - gitlab-ctl reconfigure
+    - gitlab-ctl restart
+    - verify database configuration
+    - verify node configuration
diff --git a/_support/terraform/roles/praefect/templates/praefect-gitlab.rb.j2 b/_support/terraform/roles/praefect/templates/praefect-gitlab.rb.j2
new file mode 100644
index 000000000..9eb2a11a3
--- /dev/null
+++ b/_support/terraform/roles/praefect/templates/praefect-gitlab.rb.j2
@@ -0,0 +1,47 @@
+# Disable all other services on the Praefect node
+postgresql['enable'] = false
+redis['enable'] = false
+nginx['enable'] = false
+alertmanager['enable'] = false
+prometheus['enable'] = false
+grafana['enable'] = false
+puma['enable'] = false
+sidekiq['enable'] = false
+gitlab_workhorse['enable'] = false
+gitaly['enable'] = false
+
+# Enable only the Praefect service
+praefect['enable'] = true
+
+# Prevent database connections during 'gitlab-ctl reconfigure'
+gitlab_rails['rake_cache_clear'] = false
+gitlab_rails['auto_migrate'] = false
+
+praefect['listen_addr'] = '0.0.0.0:2305'
+
+# Enable Prometheus metrics access to Praefect. You must use firewalls
+# to restrict access to this address/port.
+praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
+
+praefect['auth_token'] = '{{ praefect_external_token }}'
+
+praefect['database_host'] = '{{ groups['databases'][0] }}'
+praefect['database_port'] = 5432
+praefect['database_user'] = 'praefect'
+praefect['database_password'] = '{{ praefect_database_password }}'
+praefect['database_dbname'] = 'praefect_production'
+praefect['database_sslmode'] = 'disable'
+
+# Name of storage hash must match storage name in git_data_dirs on GitLab
+# server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1')
+praefect['virtual_storages'] = {
+  'default' => {
+{% for host in groups['gitalies'] %}
+    'gitaly-{{ loop.index }}' => {
+      'address' => 'tcp://{{ hostvars[host]['internal'] }}:8075',
+      'token'   => '{{ praefect_internal_token }}',
+      'primary' => {{ (loop.index == 0)|string|lower }},
+    },
+{% endfor %}
+  },
+}