diff options
| author | Patrick Steinhardt <psteinhardt@gitlab.com> | 2021-09-07 15:49:32 +0300 |
|---|---|---|
| committer | Patrick Steinhardt <psteinhardt@gitlab.com> | 2021-09-09 09:08:36 +0300 |
| commit | 4dae068e3b29b2faf3edaa61a6461ed1df52a70e (patch) | |
| tree | b9d9d5ce2328699ca2db548d4dffd9be10587d6b | |
| parent | d63338d5a5e13cd36de6621ecbf2eff62ebefc7a (diff) | |
command: Create separate tracing span when acquiring spawn tokens
When trying to dig into performance-related issues, one frequently sees
trace spans where commands are seemingly hanging for a long time while
doing nothing. More often than not, the root cause of this is that too
many requests come in and as a result have emptied available spawn
tokens. While this becomes visible as soon as one takes a look at logs,
it is highly misleading when only taking a look at trace spans.
Create a separate span when retrieving spawn tokens to better highlight
what's happening and avoid confusion.
| -rw-r--r-- | internal/command/spawntoken.go | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/internal/command/spawntoken.go b/internal/command/spawntoken.go index 95874d88c..b28206917 100644 --- a/internal/command/spawntoken.go +++ b/internal/command/spawntoken.go @@ -7,6 +7,7 @@ import ( "github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus/ctxlogrus" "github.com/kelseyhightower/envconfig" + "github.com/opentracing/opentracing-go" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promauto" ) @@ -57,6 +58,9 @@ func getSpawnToken(ctx context.Context) (putToken func(), err error) { // https://gitlab.com/gitlab-org/gitaly/issues/823. start := time.Now() + span, ctx := opentracing.StartSpanFromContext(ctx, "command.getSpawnToken") + defer span.Finish() + select { case spawnTokens <- struct{}{}: logTime(ctx, start, "spawn token acquired") |