benchmarking: Update README.md with new options

Add description of benchmarking output to the README.

1 files changed, 47 insertions, 3 deletions

diff --git a/_support/benchmarking/README.md b/_support/benchmarking/README.md
index f5deb804e..76367e6c5 100644
--- a/_support/benchmarking/README.md
+++ b/_support/benchmarking/README.md
@@ -26,7 +26,7 @@ This will create a Gitaly node and a small client node to send requests to
 Gitaly over gRPC. This will prompt for the Gitaly revision to be built,
 instance name, and public SSH key to use for connections.
 
-Use the `gitaly_bench` user to SSH into the instance:
+Use the `gitaly_bench` user to SSH into the instance if desired:
 
 ```shell
 ssh gitaly_bench@<INSTANCE_ADDRESS>
@@ -42,7 +42,51 @@ Build and install Gitaly from source with from desired reference and install
 profiling tools like `perf` and `libbpf-tools`. A disk image containing the
 test repositories will be mounted to `/mnt/git-repositories` on the Gitaly node.
 
-### 3. Destroy instance
+### 3. Run benchmarks
+
+```shell
+./run-benchmarks
+```
+
+Run the benchmarks specified in `group_vars/all.yml`. By default Gitaly is
+profiled with `perf` and `libbpf-tools` for flamegraphs and other metrics, which
+may add ~10% overhead. Set the `profile` variable to `false` to disable profiling:
+
+```shell
+./run-benchmarks --extra-vars "profile=false"
+```
+
+On completion a tarball of the benchmark output will be written to
+`results/benchmark-<GITALY_REVISION>-<BENCH_TIMESTAMP>.tar.gz`. This will
+have a directory for each repository tested against each RPC containing:
+
+- `ghz.json` - Output in JSON format from [ghz](https://ghz.sh) for the run.
+- `gitaly.log` - The main Gitaly log file. Gitaly-Ruby logs are not included.
+
+To retrieve the 99th percentile duration in milliseconds from `ghz.json` use:
+
+```shell
+jq '.latencyDistribution[] | select(.percentage==99) | .latency / 1000000' ghz.json
+```
+
+When profiling is enabled, the following are also present:
+
+- `all-perf.svg` - Flamegraph built from a system-wide `perf` capture. This uses
+  `--call-graph=dwarf` and will provide accurate stack traces for Git but
+  Gitaly's will be invalid.
+- `biolatency.txt` - Histogram of block I/O latency, separated by disk.
+  `/mnt/git-repositories` will be disk `/dev/sdb`.
+- `biotop.txt` - List of the processes performing the most block I/O.
+- `cpu-dist-off.txt` - Histogram of duration programs spent unscheduled by the
+  kernel.
+- `cpu-dist-on.txt` - Histogram of duration programs spent running.
+- `gitaly-execs.txt` - List of all processes forked by Gitaly and their command
+  line arguments.
+- `gitaly-perf.svg` - Flamegraph built from running `perf` against Gitaly only.
+  This uses `--call-graph=fp` for accurate stack traces for Golang.
+- `page-cachestat.txt` - Kernel page cache hit rate.
+
+### 4. Destroy instance
 
 ```shell
 ./destroy-benchmark-instance
@@ -50,4 +94,4 @@ test repositories will be mounted to `/mnt/git-repositories` on the Gitaly node.
 
 All nodes will be destroyed. As GCP will frequently reuse public IP addresses,
 the addresses of the now destroyed instances are automatically removed from
-your ~/.ssh/known_hosts file to prevent connection failures on future runs.
+your `~/.ssh/known_hosts` file to prevent connection failures on future runs.