issue template: Update with two scripts

Prior to this change, the demo template didn't have any scripts. This
change adds two:

1. Test the ansible setup, to reduce the demo setup time
2. Verify the automatic repair of repositories

This allows a consistent test script throughout demos.

1 files changed, 36 insertions, 11 deletions

diff --git a/.gitlab/issue_templates/Demo.md b/.gitlab/issue_templates/Demo.md
index 4a0356ce9..bebb3a147 100644
--- a/.gitlab/issue_templates/Demo.md
+++ b/.gitlab/issue_templates/Demo.md
@@ -4,8 +4,6 @@
 
 <!-- Replace due date below with the date of the demo -->
 
-/due YYYY-MM-DD
-
 <!--
 ## Contributing
 
@@ -63,29 +61,56 @@ This issue is used to conduct a demo for exhibiting and verifying new behavior
 for Gitaly and Praefect. Before the demo, run all `Prep:` steps. During the
 demo, run through all remaining `Demo:` and `Verify` steps. Check each
 step as completed or verified. Do not check a `Verify:` step if it does not
-succeed. 
+succeed.
 
 ## General Setup
 
-1. [ ] Prep: Check the
-   [latest version of this issue template](https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab/issue_templates/Demo.md)
+1. [ ] Prep:
+  - [ ] Check the [latest version of this issue template](https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab/issue_templates/Demo.md)
    for any new steps and update this issue accordingly.
-1. [ ] Prep: [Follow instructions to setup and configure a GitLab instance with
-   a Praefect
-   cluster](https://gitlab.com/gitlab-org/gitaly/-/blob/master/_support/terraform/README.md).
+1. [ ] Setup
+   - [ ] Checkout the latest changes as part of: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2461
+   - [ ] `cd _support/terraform`
+   - [ ] `./create-demo-cluster`
+   - [ ] `./configure-demo-cluster`
+1. [ ] Verify:
+   - [ ] Sign in as admin during the demo
+   - [ ] Create a new repository on the GitLab instance
 1. [ ] Prep: Log into the GitLab web interface and upload license
 
 ## Features
-<!-- Keep older features near the top of this section, add new features at
-the bottom -->
+
+### Automatic repository repair
+1. Prep:
+   - [ ] Have a repository in the demo cluster
+   - [ ] SSH to any Praefect node:
+      - [ ] Enable the auto reconciliation scheduler in the toml of at least one Praefect node: `[reconciliation]\n
+scheduling_interval = "5s"`
+      - [ ] Reboot the Praefect node with `sudo gitlab-ctl restart`
+      - [ ] `gitlab-ctl tail` should include `automatic reconciler started`
+1. [ ] Demo:
+   - [ ] Stop one of the Gitaly nodes
+   - [ ] Verify the Gitaly node is down on the Grafana dashboards 'Virtual storage primary flapping'
+   - [ ] Write new data to the repository
+   - [ ] Turn off the remaining Gitaly nodes
+   - [ ] Bring back first Gitaly node, which is missing the new Git data
+1. [ ] Verify:
+   - [ ] Check the `Read only repositories` [dashboard exists](https://gitlab.com/gitlab-org/gitaly/-/issues/3126) and is at least 1
+   - [ ] Check that the web interface is missing the new data
+   - [ ] Try to write to the repository, it should fail as it's in read only
+   - [ ] Run the dataloss command on any Praefect node, `sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss`
+   - [ ] Bring a second Gitaly node back online
+   - [ ] Check the logs for `"msg":"reconciliation jobs scheduled"`
+   - [ ] Check there's no dataloss anymore with `sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss`
+   - [ ] Check the `Read only repositories` [dashboard exists](https://gitlab.com/gitlab-org/gitaly/-/issues/3126) and is 0
 
 ## After Demo
 
-1. [ ] Open a new demo issue and assign to the next demo runner.
 1. [ ] Create any follow up issues discovered during the demo and assign label
    ~demo.
 1. [ ] [Follow teardown instructions to remove demo
    resources](https://gitlab.com/gitlab-org/gitaly/-/blob/master/_support/terraform/README.md#destroying-a-demo-cluster)
+    - Link the issues as related to this issue
 1. [ ] Close this issue
 
 /label ~demo ~"group::gitaly" ~"devops::create"