Expose command stats (rusage) metrics via prometheuscommand-stats-metrics

We currently track rusage metrics in logs on a per-RPC basis. This
allows us to get a very fine-grained view into resource attribution.

However, logs often do not lend themselves to corse-grained and long-
term analysis. For this reason it is useful to expose metrics via
prometheus.

By aggregating that data as metrics, we aim to partially close an
observability gap that exists for short-lived processes. The existing
`process-exporter` metrics are severely under-reporting the utilization
of short-lived processes, which gitaly spawns many of.

See also:

- https://gitlab.com/gitlab-com/gl-infra/scalability#1655

This patch introduces a set of `gitaly_command_*` metrics which
provide aggregated resource attribution along the following
dimensions:

- `cmd` - the basename of the command being executed.
- `subcmd` - an optional subcommand, e.g. `archive` for `git archive`
- `grpc_service` - the grpc service caller
- `grpc_method` - the grpc method caller

The newly introduced metrics are:

- `gitaly_command_cpu_seconds_total` Sum of CPU time spent
- `gitaly_command_real_seconds_total` Sum of real time spent
- `gitaly_command_minor_page_faults_total` Sum of minor page faults
- `gitaly_command_major_page_faults_total` Sum of major page faults
- `gitaly_command_signals_received_total` Sum of signals received
- `gitaly_command_context_switches_total` Sum of context switches

This feature is being introduced behind a feature flag. However,
since metrics are sticky, once the metric has been defined, it
will be returned by the process until the next restart.

The cardinality of the metrics should be relatively well-bounded
in any case.

1 files changed, 2 insertions, 0 deletions

diff --git a/internal/gitaly/linguist/linguist.go b/internal/gitaly/linguist/linguist.go
index fdd7ee6e7..9c498e0cf 100644
--- a/internal/gitaly/linguist/linguist.go
+++ b/internal/gitaly/linguist/linguist.go
@@ -128,6 +128,8 @@ func (inst *Instance) startGitLinguist(ctx context.Context, repoPath string, com
 		return nil, fmt.Errorf("creating command: %w", err)
 	}
 
+	internalCmd.SetMetricsSubCmd(linguistCommand)
+
 	return internalCmd, nil
 }