praefect: Fix stale client connections due to keepalive policy

While the Gitaly server sets up a keepalive enforcement policy, the
Praefect server doesn't. As a result, the default policy is active which
says that a keepalive ping is only allowed ever 5 minutes and only if
there is an active stream.

This may in fact explain some latency issues we've been seeing with the
transaction service. Given the current infrequent accesses to it, there
are extended periods of time where no calls are being made from Gitaly
to Praefect. But while Gitaly clients have keepalive set up to ping
Praefect every 20 seconds even if there are no streams, Praefect will
just discard these pings. As a result we may see stale connections which
will time out only to reconnect afterwards.

Let's specify the same keepalive enforcement policy for Praefect as we
do for Gitaly to fix this issue.

1 files changed, 7 insertions, 0 deletions

diff --git a/internal/praefect/server.go b/internal/praefect/server.go
index 7965fbb92..757a46784 100644
--- a/internal/praefect/server.go
+++ b/internal/praefect/server.go
@@ -3,6 +3,8 @@ calls to a set of Gitaly services.*/
 package praefect
 
 import (
+	"time"
+
 	grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
 	grpc_logrus "github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus"
 	grpc_ctxtags "github.com/grpc-ecosystem/go-grpc-middleware/tags"
@@ -30,6 +32,7 @@ import (
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/health"
 	healthpb "google.golang.org/grpc/health/grpc_health_v1"
+	"google.golang.org/grpc/keepalive"
 )
 
 // NewGRPCServer returns gRPC server with registered proxy-handler and actual services praefect serves on its own.
@@ -81,6 +84,10 @@ func NewGRPCServer(
 			// converted to errors and logged
 			panichandler.UnaryPanicHandler,
 		)),
+		grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
+			MinTime:             20 * time.Second,
+			PermitWithoutStream: true,
+		}),
 	}...)
 
 	warnDupeAddrs(logger, conf)