git: Fix commit-graph corruption caused by corrected committer datespks-git-disable-commit-graph-generation-v2

In staging systems we have observed corruption in commit-graphs with the
following error message:

    fatal: commit-graph requires overflow generation data but has none

This bug is caused by the rollout of Git v2.36.0, which has fixed a set
of bugs with reading corrected committer dates in commit-graphs [1].
Unfortunately, these fixes surface a corruption in commit-graphs that
seemingly happens when running with a mixture of commit-graphs generated
by Git v2.35.0 or earlier and Git v2.36.0 and later. Multiple attempts
to find the root cause of this corruption have failed until now and
nobody has yet been able to find a reproducer for this issue [2].

Disable use of corrected committer dates for now. Due to the bug that
existed in Git v2.35.0 and earlier we haven't ever read them anyway, so
this is not a performance regression for us. Instead, we'll continue to
use topological generation numbers to still speed up certain queries.

Given that the root cause seems to be bad interaction between these two
Git versions, we can reintroduce usage of corrected committer dates when
we have bumped the minimum required Git version to v2.36.0 or later.
Note that we must make sure that there are no commit-graphs left though
that still have the old GCOV or GDAT chunks.

[1]: http://public-inbox.org/git/pull.1163.git.1645735117.gitgitgadget@gmail.com/
[2]: http://public-inbox.org/git/YqD5dgalb9EPnz85@coredump.intra.peff.net/

Changelog: fixed

2 files changed, 39 insertions, 0 deletions

diff --git a/internal/git/command_factory.go b/internal/git/command_factory.go
index 86918985f..4ca0e347f 100644
--- a/internal/git/command_factory.go
+++ b/internal/git/command_factory.go
@@ -514,6 +514,43 @@ func (cf *ExecCommandFactory) globalConfiguration(ctx context.Context) ([]Global
 		// mechanism to replace malicious commits with seemingly benign ones. We thus globally
 		// disable this mechanism.
 		ConfigPair{Key: "core.useReplaceRefs", Value: "false"},
+
+		// Commit-graphs are used as an optimization to speed up reading commits and to be
+		// able to perform certain commit-related queries faster. One property that these
+		// graphs are storing is the generation number of a commit, where there are two
+		// different types of generation numbers:
+		//
+		//     - Topological level: a commit with no parents has a level of 1. A commit with
+		//       at least one parent has a level one more than the largest topological level
+		//       of its parents.
+		//
+		//     - Corrected committer date: a commit with no parents has a corrected commit
+		//       date equal to its committer date. A commit with at least one parent has a
+		//       corrected committer date equal to the maximum between either its own
+		//       committer date or the largest corrected committer date across its parents
+		//       plus 1.
+		//
+		// By default, newer Git versions store both generation numbers for commits, where
+		// the corrected committer date allows for some more optimizations. But due to a bug
+		// in Git v2.35.0 and earlier, the corrected committer date wasn't ever read.
+		//
+		// This bug was fixed in Git v2.36.0, together with a few other bugs in this area.
+		// But unfortunately, we have observed cases in production where we now end up with
+		// commit-graphs which are corrupt when the corrected committer date is enabled: the
+		// commit-graphs have the generation data storing the corrected committer date, but
+		// may be missing generation data overflow chunks which are required in case the
+		// corrected commit date requires more storage than 31 bits.
+		//
+		// We haven't been able to find the root cause until now, and chances are high that
+		// this is a bad interaction when commit-graphs are written with a mixture of Git
+		// v2.35.0 or earlier and v2.36.0 or later.
+		//
+		// Let's disable reading and writing corrected committer dates for now until we have
+		// figured out what's happening here. It is likely that we can eventually reenable
+		// this when we have fully phased out support for Git v2.35.0 and earlier _and_ when
+		// we have made sure that no commit-graph contains either GDAT (Generation Data) or
+		// GDOV (Generation Data Overflow) chunks anymore.
+		ConfigPair{Key: "commitGraph.generationVersion", Value: "1"},
 	}
 
 	// Git v2.36.0 introduced new fine-grained configuration for what data should be fsynced and
diff --git a/internal/git/command_factory_test.go b/internal/git/command_factory_test.go
index d709dd130..91672db47 100644
--- a/internal/git/command_factory_test.go
+++ b/internal/git/command_factory_test.go
@@ -483,6 +483,7 @@ func TestExecCommandFactory_config(t *testing.T) {
 		"gc.auto=0",
 		"core.autocrlf=input",
 		"core.usereplacerefs=false",
+		"commitgraph.generationversion=1",
 	}
 
 	for _, tc := range []struct {
@@ -543,6 +544,7 @@ func TestExecCommandFactory_SidecarGitConfiguration(t *testing.T) {
 		{Key: "gc.auto", Value: "0"},
 		{Key: "core.autocrlf", Value: "input"},
 		{Key: "core.useReplaceRefs", Value: "false"},
+		{Key: "commitGraph.generationVersion", Value: "1"},
 	}
 
 	commonTail := []git.ConfigPair{