Merge branch 'zj-update-observibility-docs' into 'master'

Expand on Gitaly observibility

See merge request gitlab-org/gitaly!1252

3 files changed, 59 insertions, 6 deletions

diff --git a/README.md b/README.md
index 0fe2d2225..d1f10d7c8 100644
--- a/README.md
+++ b/README.md
@@ -5,7 +5,6 @@
   [Want to Contribute?](https://gitlab.com/gitlab-org/gitaly/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Accepting%20merge%20requests) |
   [GitLab Gitaly Issues](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&state=opened&utf8=%E2%9C%93&label_name%5B%5D=Gitaly) |
   [GitLab Gitaly Merge Requests](https://gitlab.com/groups/gitlab-org/-/merge_requests?label_name%5B%5D=Gitaly) |
-  [GitLab.com Monitoring Dashboard][dashboards]
 
 --------------------------------------------
 
@@ -40,7 +39,7 @@ configuration](https://gitlab.com/gitlab-org/gitaly/issues/1282).
 In the meantime we are building features according to our [roadmap][roadmap].
 
 If you're interested in seeing how well Gitaly is performing on
-GitLab.com, we have [dashboards][dashboards]!
+GitLab.com, read about our [observibiltiy story](doc/observibility.md)!
 
 ##### Overall
 
@@ -185,5 +184,4 @@ GITLAB_TRACING=opentracing://jaeger ./gitaly config.toml
 - [Gitaly Basics, 2017-05-01](https://docs.google.com/presentation/d/1cLslUbXVkniOaeJ-r3s5AYF0kQep8VeNfvs0XSGrpA0/edit#slide=id.g1c73db867d_0_0)
 - [Git Paris meetup, 2017-02-22](https://docs.google.com/presentation/d/19OZUalFMIDM8WujXrrIyCuVb_oVeaUzpb-UdGThOvAo/edit?usp=sharing) a high-level overview of what our plans are and where we are.
 
-[dashboards]: https://dashboards.gitlab.com/d/000000176/gitaly
 [roadmap]: https://gitlab.com/groups/gitlab-org/-/roadmap?label_name%5B%5D=Gitaly&scope=all&sort=start_date_asc&state=opened
diff --git a/doc/dashboards.md b/doc/dashboards.md
index cb395917b..7e602f08e 100644
--- a/doc/dashboards.md
+++ b/doc/dashboards.md
@@ -1,11 +1,34 @@
-# Editing Gitaly dashboards
+# Observibility
+
+## Logging
+
+Gitaly leverages the [logrus](https://github.com/sirupsen/logrus) for exposing
+log messages. Most request messages are created in the [logrus middleware][logrus-middleware].
+
+Messages can be queried leveraging our [Kibana instance](https://log.gitlab.net).
+
+[logrus-middleware]: https://github.com/grpc-ecosystem/go-grpc-middleware/tree/master/logging/logrus
+
+## Prometheus
+
+Gitaly emits low cardinality metrics through Prometheus. Most of these are added
+by [go-grpc-prometheus](https://github.com/grpc-ecosystem/go-grpc-prometheus).
+Many custom metrics were also added.
+
+### Grafana
+
+To display the prometheus metrics, GitLab leverages Grafana. Two instances are
+available to view the dashboards. The dashboards can be found at:
+[dashboards.gitlab.com](https://dashboards.gitlab.com).
+
+#### Editing Gitaly dashboards
 
 Use the Grafana web interface to make changes to dashboards if
 necessary.
 
 **Remember to hit the 'Save' button at the top of the Grafana screen when making changes.**
 
-## Tiled (repeated) dashboards
+##### Tiled (repeated) dashboards
 
 If you want to make a change across a tiled Grafana dashboard such as
 the [feature request rate
@@ -14,7 +37,7 @@ then edit the first tile (top left). Its settings get applied to the
 other tiles as well. If you edit any tile other than the first your
 changes will be lost.
 
-## Drop-down values
+##### Drop-down values
 
 At the top of most of our Grafana dashboards you find drop-down menus
 for GRPC method names, Prometheus jobs etc. The possible values in these
@@ -25,3 +48,8 @@ entries.
 
 Note that Grafana 'templates' use a combination of PromQL and
 Grafana-specific modifiers.
+
+## Sentry
+
+Errors are tracked in our sentry instance, and due to their sensitive nature only viewable
+by developers at GitLab at [the error tracking page](https://gitlab.com/gitlab-org/gitaly/error_tracking).
diff --git a/doc/observibility.md b/doc/observibility.md
new file mode 100644
index 000000000..869e4460a
--- /dev/null
+++ b/doc/observibility.md
@@ -0,0 +1,27 @@
+# Editing Gitaly dashboards
+
+Use the Grafana web interface to make changes to dashboards if
+necessary.
+
+**Remember to hit the 'Save' button at the top of the Grafana screen when making changes.**
+
+## Tiled (repeated) dashboards
+
+If you want to make a change across a tiled Grafana dashboard such as
+the [feature request rate
+overview](https://performance.gitlab.net/dashboard/db/gitaly-features-overview),
+then edit the first tile (top left). Its settings get applied to the
+other tiles as well. If you edit any tile other than the first your
+changes will be lost.
+
+## Drop-down values
+
+At the top of most of our Grafana dashboards you will find drop-down menus
+for GRPC method names, Prometheus jobs etc. The possible values in these
+drop-downs are defined with Prometheus queries. To see or change these
+queries, go into the dashboard's global settings (the gear icon at the
+top of the page) and look in the 'Templating' section. You can then edit
+entries.
+
+Note that Grafana 'templates' use a combination of PromQL and
+Grafana-specific modifiers.