From 12e7eae8cb3ea2e5f497ad652f8dddb9548e716b Mon Sep 17 00:00:00 2001 From: Will Chandler Date: Thu, 5 Jan 2023 14:37:58 -0500 Subject: benchmarking: Update README.md with new options Add description of benchmarking output to the README. --- _support/benchmarking/README.md | 50 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-) (limited to '_support') diff --git a/_support/benchmarking/README.md b/_support/benchmarking/README.md index f5deb804e..76367e6c5 100644 --- a/_support/benchmarking/README.md +++ b/_support/benchmarking/README.md @@ -26,7 +26,7 @@ This will create a Gitaly node and a small client node to send requests to Gitaly over gRPC. This will prompt for the Gitaly revision to be built, instance name, and public SSH key to use for connections. -Use the `gitaly_bench` user to SSH into the instance: +Use the `gitaly_bench` user to SSH into the instance if desired: ```shell ssh gitaly_bench@ @@ -42,7 +42,51 @@ Build and install Gitaly from source with from desired reference and install profiling tools like `perf` and `libbpf-tools`. A disk image containing the test repositories will be mounted to `/mnt/git-repositories` on the Gitaly node. -### 3. Destroy instance +### 3. Run benchmarks + +```shell +./run-benchmarks +``` + +Run the benchmarks specified in `group_vars/all.yml`. By default Gitaly is +profiled with `perf` and `libbpf-tools` for flamegraphs and other metrics, which +may add ~10% overhead. Set the `profile` variable to `false` to disable profiling: + +```shell +./run-benchmarks --extra-vars "profile=false" +``` + +On completion a tarball of the benchmark output will be written to +`results/benchmark--.tar.gz`. This will +have a directory for each repository tested against each RPC containing: + +- `ghz.json` - Output in JSON format from [ghz](https://ghz.sh) for the run. +- `gitaly.log` - The main Gitaly log file. Gitaly-Ruby logs are not included. + +To retrieve the 99th percentile duration in milliseconds from `ghz.json` use: + +```shell +jq '.latencyDistribution[] | select(.percentage==99) | .latency / 1000000' ghz.json +``` + +When profiling is enabled, the following are also present: + +- `all-perf.svg` - Flamegraph built from a system-wide `perf` capture. This uses + `--call-graph=dwarf` and will provide accurate stack traces for Git but + Gitaly's will be invalid. +- `biolatency.txt` - Histogram of block I/O latency, separated by disk. + `/mnt/git-repositories` will be disk `/dev/sdb`. +- `biotop.txt` - List of the processes performing the most block I/O. +- `cpu-dist-off.txt` - Histogram of duration programs spent unscheduled by the + kernel. +- `cpu-dist-on.txt` - Histogram of duration programs spent running. +- `gitaly-execs.txt` - List of all processes forked by Gitaly and their command + line arguments. +- `gitaly-perf.svg` - Flamegraph built from running `perf` against Gitaly only. + This uses `--call-graph=fp` for accurate stack traces for Golang. +- `page-cachestat.txt` - Kernel page cache hit rate. + +### 4. Destroy instance ```shell ./destroy-benchmark-instance @@ -50,4 +94,4 @@ test repositories will be mounted to `/mnt/git-repositories` on the Gitaly node. All nodes will be destroyed. As GCP will frequently reuse public IP addresses, the addresses of the now destroyed instances are automatically removed from -your ~/.ssh/known_hosts file to prevent connection failures on future runs. +your `~/.ssh/known_hosts` file to prevent connection failures on future runs. -- cgit v1.2.3