housekeeping: Enable writing of multi-pack-indices

When writing packfiles, Git also writes an index for every packfile that
gives it the ability to quickly look up objects in that packfile. While
this works great when you have repositories with only a very limited
number of packfiles, these indices start to become insufficient once
repositories have larger numbers of packfiles because Git may have to
scan through every index in order to find a specific object. This is
also a problem for monorepostiories in Gitaly, as we allow for more
packfiles to exist the bigger the repository is.

With the multi-pack index Git has thus introduced a new data structure
to tackle this problem. The MIDX doesn't only span over a single pack,
but instead it typically spans over all packfiles that exist in the
repository. As such, Git does not have to scan through each index
anymore in order to find an object, but only has to consult the MIDX to
find an object.

Furthermore, the MIDX fixes another long-standing issue in Git with
bitmaps. Given an object, bitmaps allow you to find out in constant time
which other objects are transitively reachable from it and are thus an
important optimization especially when serving fetches. But previous to
the introduction of the MIDX, bitmaps could only be written when doing a
full repack of the repository. Doing full repacks is inefficient though,
so consequentially these bitmaps will become stale and thus less able to
speed up specific actions. With the MIDX though it now becomes possible
to also update bitmaps whenever we're doing an incremental repack of
loose objects as the MIDX itself can also have a bitmap.

And last but not least, it is now also possible to have multiple MIDX
bitmaps in a repository and the repositories it links to via the
alternates. This lifts another restriction we had where we couldn't
write bitmaps in repositories connected to an object pool.

So let's enable writing multi-pack-indices when optimizing repositories,
and furthermore let's enable writing bitmaps for pooled repositories
and when doing incremental repacks.

Changelog: performance

8 files changed, 442 insertions, 105 deletions

diff --git a/internal/git/housekeeping/optimization_strategy.go b/internal/git/housekeeping/optimization_strategy.go
index e1c7df86b..4fe51a863 100644
--- a/internal/git/housekeeping/optimization_strategy.go
+++ b/internal/git/housekeeping/optimization_strategy.go
@@ -5,6 +5,7 @@ import (
 	"math"
 
 	"gitlab.com/gitlab-org/gitaly/v15/internal/git/stats"
+	"gitlab.com/gitlab-org/gitaly/v15/internal/metadata/featureflag"
 )
 
 // OptimizationStrategy is an interface to determine which parts of a repository should be
@@ -42,7 +43,7 @@ func NewHeuristicalOptimizationStrategy(info stats.RepositoryInfo) HeuristicalOp
 // ShouldRepackObjects checks whether the repository's objects need to be repacked. This uses a
 // set of heuristics that scales with the size of the object database: the larger the repository,
 // the less frequent does it get a full repack.
-func (s HeuristicalOptimizationStrategy) ShouldRepackObjects(context.Context) (bool, RepackObjectsConfig) {
+func (s HeuristicalOptimizationStrategy) ShouldRepackObjects(ctx context.Context) (bool, RepackObjectsConfig) {
 	// If there are neither packfiles nor loose objects in this repository then there is no need
 	// to repack anything.
 	if s.info.Packfiles.Count == 0 && s.info.LooseObjects.Count == 0 {
@@ -59,7 +60,12 @@ func (s HeuristicalOptimizationStrategy) ShouldRepackObjects(context.Context) (b
 	// a bitmap on their own. We do not yet use multi-pack indices, and in that case Git can
 	// only use one bitmap. We already generate this bitmap in the pool, so member of it
 	// shouldn't have another bitmap on their own.
-	if !s.info.Packfiles.Bitmap.Exists && len(s.info.Alternates) == 0 {
+	//
+	// Addendum: this isn't true anymore as we're rolling out multi-pack-indices. For one, they
+	// allow us to have bitmaps even in repositories that are linked via alternates. And second,
+	// they don't require a full repack in order to write the bitmap. So once we have fully
+	// rolled out multi-pack-indices this whole condition can go away.
+	if featureflag.WriteMultiPackIndex.IsDisabled(ctx) && !s.info.Packfiles.Bitmap.Exists && len(s.info.Alternates) == 0 {
 		return true, RepackObjectsConfig{
 			FullRepack:  true,
 			WriteBitmap: true,
@@ -107,8 +113,9 @@ func (s HeuristicalOptimizationStrategy) ShouldRepackObjects(context.Context) (b
 	if uint64(math.Max(lowerLimit,
 		math.Log(float64(s.info.Packfiles.Size/1024/1024))/math.Log(log))) <= s.info.Packfiles.Count {
 		return true, RepackObjectsConfig{
-			FullRepack:  true,
-			WriteBitmap: len(s.info.Alternates) == 0,
+			FullRepack:          true,
+			WriteBitmap:         len(s.info.Alternates) == 0 || featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+			WriteMultiPackIndex: featureflag.WriteMultiPackIndex.IsEnabled(ctx),
 		}
 	}
 
@@ -127,8 +134,19 @@ func (s HeuristicalOptimizationStrategy) ShouldRepackObjects(context.Context) (b
 	// it is necessary on the client side. We thus take a much stricter limit of 1024 objects.
 	if s.info.LooseObjects.Count > looseObjectLimit {
 		return true, RepackObjectsConfig{
-			FullRepack:  false,
-			WriteBitmap: false,
+			FullRepack:          false,
+			WriteBitmap:         featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+			WriteMultiPackIndex: featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+		}
+	}
+
+	// In case both packfiles and loose objects are in a good state, but we don't yet have a
+	// multi-pack-index we perform an incremental repack to generate one.
+	if featureflag.WriteMultiPackIndex.IsEnabled(ctx) && !s.info.Packfiles.HasMultiPackIndex {
+		return true, RepackObjectsConfig{
+			FullRepack:          false,
+			WriteBitmap:         true,
+			WriteMultiPackIndex: true,
 		}
 	}
 
@@ -246,10 +264,11 @@ func NewEagerOptimizationStrategy(info stats.RepositoryInfo) EagerOptimizationSt
 // ShouldRepackObjects always instructs the caller to repack objects. The strategy will always be to
 // repack all objects into a single packfile. The bitmap will be written in case the repository does
 // not have any alterantes.
-func (s EagerOptimizationStrategy) ShouldRepackObjects(context.Context) (bool, RepackObjectsConfig) {
+func (s EagerOptimizationStrategy) ShouldRepackObjects(ctx context.Context) (bool, RepackObjectsConfig) {
 	return true, RepackObjectsConfig{
-		FullRepack:  true,
-		WriteBitmap: len(s.info.Alternates) == 0,
+		FullRepack:          true,
+		WriteBitmap:         len(s.info.Alternates) == 0 || featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+		WriteMultiPackIndex: featureflag.WriteMultiPackIndex.IsEnabled(ctx),
 	}
 }
 
diff --git a/internal/git/housekeeping/optimization_strategy_test.go b/internal/git/housekeeping/optimization_strategy_test.go
index 405b3de84..b1ba6cc54 100644
--- a/internal/git/housekeeping/optimization_strategy_test.go
+++ b/internal/git/housekeeping/optimization_strategy_test.go
@@ -7,13 +7,17 @@ import (
 
 	"github.com/stretchr/testify/require"
 	"gitlab.com/gitlab-org/gitaly/v15/internal/git/stats"
+	"gitlab.com/gitlab-org/gitaly/v15/internal/metadata/featureflag"
 	"gitlab.com/gitlab-org/gitaly/v15/internal/testhelper"
 )
 
 func TestHeuristicalOptimizationStrategy_ShouldRepackObjects(t *testing.T) {
 	t.Parallel()
+	testhelper.NewFeatureSets(featureflag.WriteMultiPackIndex).Run(t, testHeuristicalOptimizationStrategyShouldRepackObjects)
+}
 
-	ctx := testhelper.Context(t)
+func testHeuristicalOptimizationStrategyShouldRepackObjects(t *testing.T, ctx context.Context) {
+	t.Parallel()
 
 	for _, tc := range []struct {
 		desc           string
@@ -40,8 +44,9 @@ func TestHeuristicalOptimizationStrategy_ShouldRepackObjects(t *testing.T) {
 			},
 			expectedNeeded: true,
 			expectedConfig: RepackObjectsConfig{
-				FullRepack:  true,
-				WriteBitmap: true,
+				FullRepack:          featureflag.WriteMultiPackIndex.IsDisabled(ctx),
+				WriteBitmap:         true,
+				WriteMultiPackIndex: featureflag.WriteMultiPackIndex.IsEnabled(ctx),
 			},
 		},
 		{
@@ -60,10 +65,23 @@ func TestHeuristicalOptimizationStrategy_ShouldRepackObjects(t *testing.T) {
 			// If we have no bitmap in the repository we'd normally want to fully repack
 			// the repository. But because we have an alternates file we know that the
 			// repository must not have a bitmap anyway, so we can skip the repack here.
-			expectedNeeded: false,
+			//
+			// This changes though with multi-pack-indices, which allow for bitmaps to
+			// exist in pooled repositories.
+			expectedNeeded: featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+			expectedConfig: func() RepackObjectsConfig {
+				if featureflag.WriteMultiPackIndex.IsEnabled(ctx) {
+					return RepackObjectsConfig{
+						WriteBitmap:         true,
+						WriteMultiPackIndex: true,
+					}
+				}
+
+				return RepackObjectsConfig{}
+			}(),
 		},
 		{
-			desc: "no repack needed",
+			desc: "no repack needed without multi-pack-index",
 			strategy: HeuristicalOptimizationStrategy{
 				info: stats.RepositoryInfo{
 					Packfiles: stats.PackfilesInfo{
@@ -74,7 +92,41 @@ func TestHeuristicalOptimizationStrategy_ShouldRepackObjects(t *testing.T) {
 					},
 				},
 			},
-			expectedNeeded: false,
+			expectedNeeded: featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+			expectedConfig: func() RepackObjectsConfig {
+				if featureflag.WriteMultiPackIndex.IsEnabled(ctx) {
+					return RepackObjectsConfig{
+						FullRepack:          false,
+						WriteBitmap:         true,
+						WriteMultiPackIndex: true,
+					}
+				}
+
+				return RepackObjectsConfig{}
+			}(),
+		},
+		{
+			desc: "no repack needed with multi-pack-index",
+			strategy: HeuristicalOptimizationStrategy{
+				info: stats.RepositoryInfo{
+					Packfiles: stats.PackfilesInfo{
+						Count:             1,
+						HasMultiPackIndex: true,
+					},
+				},
+			},
+			expectedNeeded: featureflag.WriteMultiPackIndex.IsDisabled(ctx),
+			expectedConfig: func() RepackObjectsConfig {
+				if featureflag.WriteMultiPackIndex.IsDisabled(ctx) {
+					return RepackObjectsConfig{
+						FullRepack:          true,
+						WriteBitmap:         true,
+						WriteMultiPackIndex: false,
+					}
+				}
+
+				return RepackObjectsConfig{}
+			}(),
 		},
 	} {
 		t.Run(tc.desc, func(t *testing.T) {
@@ -159,6 +211,7 @@ func TestHeuristicalOptimizationStrategy_ShouldRepackObjects(t *testing.T) {
 								Bitmap: stats.BitmapInfo{
 									Exists: true,
 								},
+								HasMultiPackIndex: true,
 							},
 							Alternates: tc.alternates,
 						},
@@ -174,8 +227,9 @@ func TestHeuristicalOptimizationStrategy_ShouldRepackObjects(t *testing.T) {
 					repackNeeded, repackCfg := strategy.ShouldRepackObjects(ctx)
 					require.True(t, repackNeeded)
 					require.Equal(t, RepackObjectsConfig{
-						FullRepack:  true,
-						WriteBitmap: len(tc.alternates) == 0,
+						FullRepack:          true,
+						WriteBitmap:         len(tc.alternates) == 0 || featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+						WriteMultiPackIndex: featureflag.WriteMultiPackIndex.IsEnabled(ctx),
 					}, repackCfg)
 				})
 			}
@@ -234,6 +288,7 @@ func TestHeuristicalOptimizationStrategy_ShouldRepackObjects(t *testing.T) {
 							Bitmap: stats.BitmapInfo{
 								Exists: true,
 							},
+							HasMultiPackIndex: true,
 						},
 					},
 				}
@@ -241,8 +296,9 @@ func TestHeuristicalOptimizationStrategy_ShouldRepackObjects(t *testing.T) {
 				repackNeeded, repackCfg := strategy.ShouldRepackObjects(ctx)
 				require.Equal(t, outerTC.expectedRepack, repackNeeded)
 				require.Equal(t, RepackObjectsConfig{
-					FullRepack:  false,
-					WriteBitmap: false,
+					FullRepack:          false,
+					WriteBitmap:         repackNeeded && featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+					WriteMultiPackIndex: repackNeeded && featureflag.WriteMultiPackIndex.IsEnabled(ctx),
 				}, repackCfg)
 			})
 		}
@@ -527,8 +583,11 @@ func TestHeuristicalOptimizationStrategy_NeedsWriteCommitGraph(t *testing.T) {
 
 func TestEagerOptimizationStrategy(t *testing.T) {
 	t.Parallel()
+	testhelper.NewFeatureSets(featureflag.WriteMultiPackIndex).Run(t, testEagerOptimizationStrategy)
+}
 
-	ctx := testhelper.Context(t)
+func testEagerOptimizationStrategy(t *testing.T, ctx context.Context) {
+	t.Parallel()
 
 	for _, tc := range []struct {
 		desc                     string
@@ -548,6 +607,7 @@ func TestEagerOptimizationStrategy(t *testing.T) {
 					Alternates: []string{"path/to/alternate"},
 				},
 			},
+			expectWriteBitmap:        featureflag.WriteMultiPackIndex.IsEnabled(ctx),
 			expectShouldPruneObjects: true,
 		},
 		{
@@ -557,7 +617,8 @@ func TestEagerOptimizationStrategy(t *testing.T) {
 					IsObjectPool: true,
 				},
 			},
-			expectWriteBitmap: true,
+			expectWriteBitmap:        true,
+			expectShouldPruneObjects: false,
 		},
 		{
 			desc: "object pool with alternate",
@@ -567,14 +628,17 @@ func TestEagerOptimizationStrategy(t *testing.T) {
 					Alternates:   []string{"path/to/alternate"},
 				},
 			},
+			expectWriteBitmap:        featureflag.WriteMultiPackIndex.IsEnabled(ctx),
+			expectShouldPruneObjects: false,
 		},
 	} {
 		t.Run(tc.desc, func(t *testing.T) {
 			shouldRepackObjects, repackObjectsCfg := tc.strategy.ShouldRepackObjects(ctx)
 			require.True(t, shouldRepackObjects)
 			require.Equal(t, RepackObjectsConfig{
-				FullRepack:  true,
-				WriteBitmap: tc.expectWriteBitmap,
+				FullRepack:          true,
+				WriteBitmap:         tc.expectWriteBitmap,
+				WriteMultiPackIndex: featureflag.WriteMultiPackIndex.IsEnabled(ctx),
 			}, repackObjectsCfg)
 
 			shouldWriteCommitGraph, writeCommitGraphCfg := tc.strategy.ShouldWriteCommitGraph(ctx)
diff --git a/internal/git/housekeeping/optimize_repository_ext_test.go b/internal/git/housekeeping/optimize_repository_ext_test.go
index a956afb9d..087cf2fc1 100644
--- a/internal/git/housekeeping/optimize_repository_ext_test.go
+++ b/internal/git/housekeeping/optimize_repository_ext_test.go
@@ -25,7 +25,10 @@ import (
 
 func TestPruneIfNeeded(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testPruneIfNeeded)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testPruneIfNeeded)
 }
 
 func testPruneIfNeeded(t *testing.T, ctx context.Context) {
@@ -34,6 +37,13 @@ func testPruneIfNeeded(t *testing.T, ctx context.Context) {
 	cfg := testcfg.Build(t)
 	cfg.SocketPath = testserver.RunGitalyServer(t, cfg, nil, setup.RegisterAll)
 
+	midxEnabledOrDisabled := func(enabled, disabled map[string]string) map[string]string {
+		if featureflag.WriteMultiPackIndex.IsEnabled(ctx) {
+			return enabled
+		}
+		return disabled
+	}
+
 	for _, tc := range []struct {
 		desc               string
 		isPool             bool
@@ -51,20 +61,34 @@ func testPruneIfNeeded(t *testing.T, ctx context.Context) {
 			looseObjects: []string{
 				filepath.Join("ab/12345"),
 			},
-			expectedLogEntries: map[string]string{
-				"packed_objects_full": "success",
-				"written_bitmap":      "success",
-			},
+			expectedLogEntries: midxEnabledOrDisabled(
+				map[string]string{
+					"packed_objects_incremental": "success",
+					"written_bitmap":             "success",
+					"written_multi_pack_index":   "success",
+				},
+				map[string]string{
+					"packed_objects_full": "success",
+					"written_bitmap":      "success",
+				},
+			),
 		},
 		{
 			desc: "object in 17 shard",
 			looseObjects: []string{
 				filepath.Join("17/12345"),
 			},
-			expectedLogEntries: map[string]string{
-				"packed_objects_full": "success",
-				"written_bitmap":      "success",
-			},
+			expectedLogEntries: midxEnabledOrDisabled(
+				map[string]string{
+					"packed_objects_incremental": "success",
+					"written_bitmap":             "success",
+					"written_multi_pack_index":   "success",
+				},
+				map[string]string{
+					"packed_objects_full": "success",
+					"written_bitmap":      "success",
+				},
+			),
 		},
 		{
 			desc: "objects in different shards",
@@ -74,10 +98,17 @@ func testPruneIfNeeded(t *testing.T, ctx context.Context) {
 				filepath.Join("12/12345"),
 				filepath.Join("17/12345"),
 			},
-			expectedLogEntries: map[string]string{
-				"packed_objects_full": "success",
-				"written_bitmap":      "success",
-			},
+			expectedLogEntries: midxEnabledOrDisabled(
+				map[string]string{
+					"packed_objects_incremental": "success",
+					"written_bitmap":             "success",
+					"written_multi_pack_index":   "success",
+				},
+				map[string]string{
+					"packed_objects_full": "success",
+					"written_bitmap":      "success",
+				},
+			),
 		},
 		{
 			desc:   "exceeding boundary on pool",
@@ -93,10 +124,17 @@ func testPruneIfNeeded(t *testing.T, ctx context.Context) {
 
 				return looseObjects
 			}(),
-			expectedLogEntries: map[string]string{
-				"packed_objects_full": "success",
-				"written_bitmap":      "success",
-			},
+			expectedLogEntries: midxEnabledOrDisabled(
+				map[string]string{
+					"packed_objects_incremental": "success",
+					"written_bitmap":             "success",
+					"written_multi_pack_index":   "success",
+				},
+				map[string]string{
+					"packed_objects_full": "success",
+					"written_bitmap":      "success",
+				},
+			),
 		},
 		{
 			desc: "on boundary shouldn't prune",
@@ -115,10 +153,17 @@ func testPruneIfNeeded(t *testing.T, ctx context.Context) {
 				t := time.Now().Add(stats.StaleObjectsGracePeriod).Add(-1 * time.Minute)
 				return &t
 			}(),
-			expectedLogEntries: map[string]string{
-				"packed_objects_full": "success",
-				"written_bitmap":      "success",
-			},
+			expectedLogEntries: midxEnabledOrDisabled(
+				map[string]string{
+					"packed_objects_incremental": "success",
+					"written_bitmap":             "success",
+					"written_multi_pack_index":   "success",
+				},
+				map[string]string{
+					"packed_objects_full": "success",
+					"written_bitmap":      "success",
+				},
+			),
 		},
 		{
 			desc: "exceeding boundary should prune",
@@ -137,11 +182,19 @@ func testPruneIfNeeded(t *testing.T, ctx context.Context) {
 				t := time.Now().Add(stats.StaleObjectsGracePeriod).Add(-1 * time.Minute)
 				return &t
 			}(),
-			expectedLogEntries: map[string]string{
-				"packed_objects_full": "success",
-				"pruned_objects":      "success",
-				"written_bitmap":      "success",
-			},
+			expectedLogEntries: midxEnabledOrDisabled(
+				map[string]string{
+					"packed_objects_incremental": "success",
+					"pruned_objects":             "success",
+					"written_bitmap":             "success",
+					"written_multi_pack_index":   "success",
+				},
+				map[string]string{
+					"packed_objects_full": "success",
+					"pruned_objects":      "success",
+					"written_bitmap":      "success",
+				},
+			),
 		},
 	} {
 		tc := tc
diff --git a/internal/git/housekeeping/optimize_repository_test.go b/internal/git/housekeeping/optimize_repository_test.go
index 31b840f2b..c13f08ead 100644
--- a/internal/git/housekeeping/optimize_repository_test.go
+++ b/internal/git/housekeeping/optimize_repository_test.go
@@ -28,7 +28,10 @@ import (
 
 func TestRepackIfNeeded(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testRepackIfNeeded)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testRepackIfNeeded)
 }
 
 func testRepackIfNeeded(t *testing.T, ctx context.Context) {
@@ -154,7 +157,10 @@ func TestPackRefsIfNeeded(t *testing.T) {
 
 func TestOptimizeRepository(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testOptimizeRepository)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testOptimizeRepository)
 }
 
 func testOptimizeRepository(t *testing.T, ctx context.Context) {
@@ -168,6 +174,13 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 		count        int
 	}
 
+	midxEnabledOrDisabled := func(enabled, disabled []metric) []metric {
+		if featureflag.WriteMultiPackIndex.IsEnabled(ctx) {
+			return enabled
+		}
+		return disabled
+	}
+
 	for _, tc := range []struct {
 		desc                   string
 		setup                  func(t *testing.T, relativePath string) *gitalypb.Repository
@@ -198,12 +211,21 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 				gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("main"))
 				return repo
 			},
-			expectedMetrics: []metric{
-				{name: "packed_objects_full", status: "success", count: 1},
-				{name: "written_commit_graph_full", status: "success", count: 1},
-				{name: "written_bitmap", status: "success", count: 1},
-				{name: "total", status: "success", count: 1},
-			},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "packed_objects_full", status: "success", count: 1},
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 		},
 		{
 			desc: "repository without commit-graph writes commit-graph",
@@ -216,10 +238,19 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 				gittest.Exec(t, cfg, "-C", repoPath, "repack", "-A", "-d", "--write-bitmap-index")
 				return repo
 			},
-			expectedMetrics: []metric{
-				{name: "written_commit_graph_full", status: "success", count: 1},
-				{name: "total", status: "success", count: 1},
-			},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 		},
 		{
 			desc: "repository with commit-graph without generation data writes commit-graph",
@@ -233,10 +264,44 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 				gittest.Exec(t, cfg, "-c", "commitGraph.generationVersion=1", "-C", repoPath, "commit-graph", "write", "--split", "--changed-paths")
 				return repo
 			},
-			expectedMetrics: []metric{
-				{name: "written_commit_graph_full", status: "success", count: 1},
-				{name: "total", status: "success", count: 1},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
+		},
+		{
+			desc: "repository without multi-pack-index performs incremental repack",
+			setup: func(t *testing.T, relativePath string) *gitalypb.Repository {
+				repo, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{
+					SkipCreationViaService: true,
+					RelativePath:           relativePath,
+				})
+				gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("main"))
+				gittest.Exec(t, cfg, "-C", repoPath, "repack", "-A", "-d", "-b")
+				return repo
 			},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 		},
 		{
 			desc: "repository with multiple packfiles packs only for object pool",
@@ -258,15 +323,33 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 
 				return repo
 			},
-			expectedMetrics: []metric{
-				{name: "total", status: "success", count: 1},
-			},
-			expectedMetricsForPool: []metric{
-				{name: "packed_objects_full", status: "success", count: 1},
-				{name: "written_bitmap", status: "success", count: 1},
-				{name: "written_commit_graph_incremental", status: "success", count: 1},
-				{name: "total", status: "success", count: 1},
-			},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "total", status: "success", count: 1},
+				},
+			),
+			expectedMetricsForPool: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_full", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "packed_objects_full", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 		},
 		{
 			desc: "well-packed repository does not optimize",
@@ -280,9 +363,42 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 				gittest.Exec(t, cfg, "-c", "commitGraph.generationVersion=2", "-C", repoPath, "commit-graph", "write", "--split", "--changed-paths")
 				return repo
 			},
-			expectedMetrics: []metric{
-				{name: "total", status: "success", count: 1},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "total", status: "success", count: 1},
+				},
+			),
+		},
+		{
+			desc: "well-packed repository with multi-pack-index does not optimize",
+			setup: func(t *testing.T, relativePath string) *gitalypb.Repository {
+				repo, repoPath := gittest.CreateRepository(t, ctx, cfg, gittest.CreateRepositoryConfig{
+					SkipCreationViaService: true,
+					RelativePath:           relativePath,
+				})
+				gittest.WriteCommit(t, cfg, repoPath, gittest.WithBranch("main"))
+				gittest.Exec(t, cfg, "-C", repoPath, "repack", "-A", "-d", "--write-bitmap-index", "--write-midx")
+				gittest.Exec(t, cfg, "-c", "commitGraph.generationVersion=2", "-C", repoPath, "commit-graph", "write", "--split", "--changed-paths")
+				return repo
 			},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "packed_objects_full", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 		},
 		{
 			desc: "recent loose objects don't get pruned",
@@ -312,11 +428,20 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 
 				return repo
 			},
-			expectedMetrics: []metric{
-				{name: "packed_objects_incremental", status: "success", count: 1},
-				{name: "written_commit_graph_incremental", status: "success", count: 1},
-				{name: "total", status: "success", count: 1},
-			},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 		},
 		{
 			desc: "old loose objects get pruned",
@@ -343,18 +468,37 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 
 				return repo
 			},
-			expectedMetrics: []metric{
-				{name: "packed_objects_incremental", status: "success", count: 1},
-				{name: "written_commit_graph_full", status: "success", count: 1},
-				{name: "pruned_objects", status: "success", count: 1},
-				{name: "total", status: "success", count: 1},
-			},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "pruned_objects", status: "success", count: 1},
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_commit_graph_full", status: "success", count: 1},
+					{name: "pruned_objects", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 			// Object pools never prune objects.
-			expectedMetricsForPool: []metric{
-				{name: "packed_objects_incremental", status: "success", count: 1},
-				{name: "written_commit_graph_incremental", status: "success", count: 1},
-				{name: "total", status: "success", count: 1},
-			},
+			expectedMetricsForPool: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 		},
 		{
 			desc: "loose refs get packed",
@@ -373,10 +517,20 @@ func testOptimizeRepository(t *testing.T, ctx context.Context) {
 
 				return repo
 			},
-			expectedMetrics: []metric{
-				{name: "packed_refs", status: "success", count: 1},
-				{name: "total", status: "success", count: 1},
-			},
+			expectedMetrics: midxEnabledOrDisabled(
+				[]metric{
+					{name: "packed_objects_incremental", status: "success", count: 1},
+					{name: "packed_refs", status: "success", count: 1},
+					{name: "written_commit_graph_incremental", status: "success", count: 1},
+					{name: "written_bitmap", status: "success", count: 1},
+					{name: "written_multi_pack_index", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+				[]metric{
+					{name: "packed_refs", status: "success", count: 1},
+					{name: "total", status: "success", count: 1},
+				},
+			),
 		},
 	} {
 		tc := tc
diff --git a/internal/git/objectpool/fetch_test.go b/internal/git/objectpool/fetch_test.go
index 06e097a36..8ca529aa3 100644
--- a/internal/git/objectpool/fetch_test.go
+++ b/internal/git/objectpool/fetch_test.go
@@ -24,7 +24,10 @@ import (
 
 func TestFetchFromOrigin_dangling(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchFromOriginDangling)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchFromOriginDangling)
 }
 
 func testFetchFromOriginDangling(t *testing.T, ctx context.Context) {
@@ -113,7 +116,10 @@ func TestFetchFromOrigin_fsck(t *testing.T) {
 
 func TestFetchFromOrigin_deltaIslands(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchFromOriginDeltaIslands)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchFromOriginDeltaIslands)
 }
 
 func testFetchFromOriginDeltaIslands(t *testing.T, ctx context.Context) {
@@ -126,6 +132,13 @@ func testFetchFromOriginDeltaIslands(t *testing.T, ctx context.Context) {
 	require.NoError(t, pool.FetchFromOrigin(ctx, repo), "seed pool")
 	require.NoError(t, pool.Link(ctx, repo))
 
+	// With multi-pack-indices we don't do full repacks of repositories that
+	// aggressively anymore, but in order to test delta islands we need to trigger one.
+	// We thus write a second packfile so that `OptimizeRepository()` decides to
+	// rewrite packfiles.
+	gittest.WriteCommit(t, cfg, poolPath, gittest.WithBranch("irrelevant"))
+	gittest.Exec(t, cfg, "-C", poolPath, "repack")
+
 	// The setup of delta islands is done in the normal repository, and thus we pass `false`
 	// for `isPoolRepo`. Verification whether we correctly handle repacking though happens in
 	// the pool repository.
@@ -136,7 +149,10 @@ func testFetchFromOriginDeltaIslands(t *testing.T, ctx context.Context) {
 
 func TestFetchFromOrigin_bitmapHashCache(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchFromOriginBitmapHashCache)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchFromOriginBitmapHashCache)
 }
 
 func testFetchFromOriginBitmapHashCache(t *testing.T, ctx context.Context) {
@@ -166,7 +182,10 @@ func testFetchFromOriginBitmapHashCache(t *testing.T, ctx context.Context) {
 
 func TestFetchFromOrigin_refUpdates(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchFromOriginRefUpdates)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchFromOriginRefUpdates)
 }
 
 func testFetchFromOriginRefUpdates(t *testing.T, ctx context.Context) {
@@ -217,7 +236,10 @@ func testFetchFromOriginRefUpdates(t *testing.T, ctx context.Context) {
 
 func TestFetchFromOrigin_refs(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchFromOriginRefs)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchFromOriginRefs)
 }
 
 func testFetchFromOriginRefs(t *testing.T, ctx context.Context) {
diff --git a/internal/gitaly/service/objectpool/fetch_into_object_pool_test.go b/internal/gitaly/service/objectpool/fetch_into_object_pool_test.go
index a50f49e00..2d9bec21c 100644
--- a/internal/gitaly/service/objectpool/fetch_into_object_pool_test.go
+++ b/internal/gitaly/service/objectpool/fetch_into_object_pool_test.go
@@ -38,7 +38,10 @@ import (
 
 func TestFetchIntoObjectPool_Success(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchIntoObjectPoolSuccess)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchIntoObjectPoolSuccess)
 }
 
 func testFetchIntoObjectPoolSuccess(t *testing.T, ctx context.Context) {
@@ -91,7 +94,10 @@ func testFetchIntoObjectPoolSuccess(t *testing.T, ctx context.Context) {
 
 func TestFetchIntoObjectPool_transactional(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchIntoObjectPoolTransactional)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchIntoObjectPoolTransactional)
 }
 
 func testFetchIntoObjectPoolTransactional(t *testing.T, ctx context.Context) {
@@ -207,7 +213,10 @@ func testFetchIntoObjectPoolTransactional(t *testing.T, ctx context.Context) {
 
 func TestFetchIntoObjectPool_CollectLogStatistics(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchIntoObjectPoolCollectLogStatistics)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchIntoObjectPoolCollectLogStatistics)
 }
 
 func testFetchIntoObjectPoolCollectLogStatistics(t *testing.T, ctx context.Context) {
@@ -303,7 +312,10 @@ func TestFetchIntoObjectPool_Failure(t *testing.T) {
 
 func TestFetchIntoObjectPool_dfConflict(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testFetchIntoObjectPoolDfConflict)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testFetchIntoObjectPoolDfConflict)
 }
 
 func testFetchIntoObjectPoolDfConflict(t *testing.T, ctx context.Context) {
diff --git a/internal/gitaly/service/repository/optimize_test.go b/internal/gitaly/service/repository/optimize_test.go
index f713de3c2..618a6f2b5 100644
--- a/internal/gitaly/service/repository/optimize_test.go
+++ b/internal/gitaly/service/repository/optimize_test.go
@@ -28,7 +28,10 @@ import (
 
 func TestOptimizeRepository(t *testing.T) {
 	t.Parallel()
-	testhelper.NewFeatureSets(featureflag.WriteBitmapLookupTable).Run(t, testOptimizeRepository)
+	testhelper.NewFeatureSets(
+		featureflag.WriteBitmapLookupTable,
+		featureflag.WriteMultiPackIndex,
+	).Run(t, testOptimizeRepository)
 }
 
 func testOptimizeRepository(t *testing.T, ctx context.Context) {
diff --git a/internal/metadata/featureflag/ff_write_multi_pack_index.go b/internal/metadata/featureflag/ff_write_multi_pack_index.go
new file mode 100644
index 000000000..9f2565788
--- /dev/null
+++ b/internal/metadata/featureflag/ff_write_multi_pack_index.go
@@ -0,0 +1,10 @@
+package featureflag
+
+// WriteMultiPackIndex determines whether Gitaly should write multi-pack-indices in
+// OptimizeRepository.
+var WriteMultiPackIndex = NewFeatureFlag(
+	"write_multi_pack_index",
+	"v15.9.0",
+	"https://gitlab.com/gitlab-org/gitaly/-/issues/4741",
+	false,
+)