doc: Add a doc about using Jaeger for local development

Distributed tracing and Jaeger are powerful tool. They are integrated
deep into Gitaly and other services at GitLab. Unfortunately, the
adoption at GitLab is not high. This commit adds a development doc to
get started with using Jaeger for Gitaly local development.

3 files changed, 67 insertions, 0 deletions

diff --git a/doc/img/jaeger-1.png b/doc/img/jaeger-1.png
new file mode 100644
index 000000000..ac91b66a5
--- /dev/null
+++ b/doc/img/jaeger-1.png
diff --git a/doc/img/jaeger-2.png b/doc/img/jaeger-2.png
new file mode 100644
index 000000000..b167eb2e0
--- /dev/null
+++ b/doc/img/jaeger-2.png
diff --git a/doc/jaeger_for_local_development.md b/doc/jaeger_for_local_development.md
new file mode 100644
index 000000000..9fd0e9f0c
--- /dev/null
+++ b/doc/jaeger_for_local_development.md
@@ -0,0 +1,67 @@
+# Using Jaeger for Gitaly local development
+
+Distributed tracing is an essential tool for understanding how services interact
+with each other, especially during local development. This tool is very helpful
+for tracking a flow of a request from GitLab services (Workhorse, GitLab Shell,
+GitLab Rails) to Gitaly and identifying any issues that might be affecting the
+performance of your system.
+
+If you're working on local development for Gitaly, you can use Jaeger, a popular
+open-source distributed tracing system. Jaeger can:
+
+- Visualize the flow of requests between services and identify any bottlenecks.
+- Comprehend data propagation and correlation between them.
+
+When combined with [Trace2](https://git-scm.com/docs/api-trace2), Jaeger can
+provide a much deeper look into how Gitaly spawns and manages Git processes.
+
+## Configure Jaeger in local environment
+
+Fortunately, Trace2 and distributed tracing are already enabled in Gitaly by
+default. To start tracing, all you need to do is set up Jaeger:
+
+- If you are using the GDK for local development, getting Jaeger setup involves
+  some additional configuration to the `gdk.yml` and restarting. You can follow
+  [this tutorial](https://docs.gitlab.com/ee/development/distributed_tracing.html#using-jaeger-in-the-gitlab-development-kit)
+  for more information.
+- Otherwise, you'll need to set up Jaeger manually. You can follow
+  [this tutorial](https://docs.gitlab.com/ee/development/distributed_tracing.html#using-jaeger-without-the-gitlab-developer-kit).
+
+## How to use Jaeger for local development
+
+For a video introduction to using Jaeger, see
+[Jaeger for Gitaly local development](https://www.youtube.com/watch?v=jJDH4hxmHv4).
+The video walks through some common scenarios you might encounter during
+development.
+
+The follow section describes a simple scenario. The video has more information
+on other view models, as well as more complicated scenarios.
+
+1. When Jaeger starts, go to `http://localhost:16686` to access the Jaeger dashboard.
+1. All resulting traces from any actions you perform are captured by Jaeger through
+   another endpoint using the `GITLAB_TRACING` environment variable, which is
+   set up automatically by GDK. As the traces are collected asynchronously, it
+   may take a short moment until the collection is complete.
+1. Perform a Git-related action, such as
+   `git clone http://localhost:3000/gitlab-org/gitaly.git`.
+1. Access the UI dashboard again. The left side includes some filters while the
+   right side shows the timeline and corresponding events. Gitaly is not exposed
+   directly, so any Remote Procedure Calls (RPCs) originate through entry-point
+   components such as Workhorse, GitLab-Shell, and GitLab Rails. You’ll see the
+   traces of those services there.
+1. Select a trace to enter the trace detail view. The default view displays
+   the timeline of spans (events) in chronological order, through multiple
+   services. In this example, it shows the details of each `git` command spawned
+   by Gitaly.
+
+   ![Jaeger detail page](./img/jaeger-1.png)
+
+1. Each span consists of detailed data that's very helpful for development
+   purposes. Of course, we can add custom instrumentation.
+
+   ![Jaeger trace2 details](./img/jaeger-2.png)
+
+One great value distributed tracing offers is the ability to
+visualize asynchronous flow in a comprehensive manner. This ability is
+particularly helpful for debugging a complicated action flowing through multiple
+services in different orders. Unfortunately, this use case is very common.