blob: Implement ListAllLFSPointers() RPC

In order to verify whether a push is allowed or not, we do a call to
Rails' `/allowed` endpoint. This endpoint does multiple checks: next to
determining whether the reference updates are allowed in the first
place, there's also several checks which inspect all new objects which
are part of the push.

One of these checks is the check for LFS pointers. For each push, we get
a call to `GetNewLFSPointers` which computes the set of new objects and
then extracts all new LFS pointers from this set. If any of the new LFS
pointers does not have a corresponding LFS object in the repository,
then we refuse the push.

Computation of new objects can be heavily expensive though, depending on
the repository's size: we need to do a complete graph walk to correctly
determine preexisting objects and new objects. For big repositories with
lots of references and commits, this can take several seconds and in the
most extreme cases lead to context cancellations as the walks exceed the
30 seconds allowed for those checks. The user cannot do anything about
this, except restricting repository size (which we definitely don't want
to recommend) or disabling LFS pointer checks altogether (potentially
compromising repository consistency).

There is one realization to be had though: when doing pushes into git,
git will first accept all objects into a quarantine environment. As
such, there is a single place which contains all new objects which have
been part of the push. So if we'd be able to just single out pushed
objects and check these instead of doing a graph walk, then we'd start
to scale with push size, not with repository size.

There is an easy way to do this via `git cat-file --batch-all-objects`,
which prints out all of the ODB's objects no matter whether reachable or
not. Given that git spawns processes with the main object directory set
to the quarantine environment and the normal object directory part of
the alternative object directories, the only thing we need to do to
single out only pushed objects is to unset the alternative object
directories: `env --unset=GIT_ALTERNATIVE_OBJECT_DIRECTORIES git
cat-file --batch-all-objects`.

A quick benchmark with gitlab-org/gitlab.git shows that this is much
faster. The following tests have been done by pushing into the target
repository which had the LFS pointer checks as pre-receive hook. Output
has been formatted such that it becomes more readable.

    # 1000 commits with one change each
    $ git push origin master
    Benchmark #1: LFS pointers via rev-list
      Time (mean ± σ):     554.3 ms ±  20.6 ms    [User: 527.5 ms, System: 27.0 ms]
      Range (min … max):   521.9 ms … 590.5 ms    10 runs

    Benchmark #2: LFS pointers via --batch--all-objects
      Time (mean ± σ):       3.8 ms ±   1.6 ms    [User: 5.8 ms, System: 2.5 ms]
      Range (min … max):     2.4 ms …  23.0 ms    555 runs

    Summary
      'LFS pointers via --batch--all-objects' ran
      145.14 ± 59.30 times faster than 'LFS pointers via rev-list'

    # push 100 branches, where each has the same 1000 commits plus one that is different per branch
    $ git push origin $(seq -f 'branch-%g'  100)
    Benchmark #1: LFS pointers via rev-list
      Time (mean ± σ):     620.9 ms ±   7.0 ms    [User: 584.8 ms, System: 36.0 ms]
      Range (min … max):   613.3 ms … 633.1 ms    10 runs

    Benchmark #2: LFS pointers via --batch--all-objects
      Time (mean ± σ):       4.4 ms ±   1.6 ms    [User: 6.3 ms, System: 3.1 ms]
      Range (min … max):     0.2 ms …  26.5 ms    636 runs

    Summary
      'LFS pointers via --batch--all-objects' ran
      140.34 ± 49.49 times faster than 'LFS pointers via rev-list'

    # push of unrelated history to emulate lots of objects (pushing Gitaly into the GitLab repo)
    $ git push origin gitaly/master:refs/heads/gitaly
    Benchmark #1: LFS pointers via rev-list
      Time (mean ± σ):     625.5 ms ±  10.1 ms    [User: 590.0 ms, System: 35.5 ms]
      Range (min … max):   615.3 ms … 651.2 ms    10 runs

    Benchmark #2: LFS pointers via --batch--all-objects
      Time (mean ± σ):       6.4 ms ±   1.5 ms    [User: 7.9 ms, System: 3.8 ms]
      Range (min … max):     2.2 ms …  14.8 ms    467 runs

    Summary
      'LFS pointers via --batch--all-objects' ran
       98.11 ± 23.32 times faster than 'LFS pointers via rev-list'

So even for biggish pushes, `--batch-all-objects` is about 50x faster
than doing the graph walk.

In order to allow Rails to make use of this new way of doing things,
this commit implements a new interface `ListAllLFSPointers()`. In
contrast to the existing-but-deprecated `GetAllLFSPointers()` RPC, it
will return all LFS pointers regardless of their reachability. In order
to only make use of quarantined objects, the caller will then have to
modify the `Repository` message to unset alternative oject directories.

6 files changed, 405 insertions, 45 deletions

diff --git a/internal/gitaly/service/blob/lfs_pointers.go b/internal/gitaly/service/blob/lfs_pointers.go
index 329fff832..6e6330f1a 100644
--- a/internal/gitaly/service/blob/lfs_pointers.go
+++ b/internal/gitaly/service/blob/lfs_pointers.go
@@ -16,6 +16,7 @@ import (
 	"gitlab.com/gitlab-org/gitaly/internal/git/localrepo"
 	"gitlab.com/gitlab-org/gitaly/internal/metadata/featureflag"
 	"gitlab.com/gitlab-org/gitaly/proto/go/gitalypb"
+	"golang.org/x/text/transform"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
 )
@@ -71,6 +72,46 @@ func (s *server) ListLFSPointers(in *gitalypb.ListLFSPointersRequest, stream git
 	return nil
 }
 
+// ListAllLFSPointers finds all LFS pointers which exist in the repository, including those which
+// are not reachable via graph walks.
+func (s *server) ListAllLFSPointers(in *gitalypb.ListAllLFSPointersRequest, stream gitalypb.BlobService_ListAllLFSPointersServer) error {
+	ctx := stream.Context()
+
+	if in.GetRepository() == nil {
+		return status.Error(codes.InvalidArgument, "empty repository")
+	}
+
+	repo := localrepo.New(s.gitCmdFactory, in.Repository, s.cfg)
+	cmd, err := repo.Exec(ctx, git.SubCmd{
+		Name: "cat-file",
+		Flags: []git.Option{
+			git.Flag{Name: "--batch-all-objects"},
+			git.Flag{Name: "--batch-check=%(objecttype) %(objectsize) %(objectname)"},
+			git.Flag{Name: "--buffer"},
+			git.Flag{Name: "--unordered"},
+		},
+	})
+	if err != nil {
+		return status.Errorf(codes.Internal, "could not run batch-check: %v", err)
+	}
+
+	filteredReader := transform.NewReader(cmd, lfsPointerFilter{})
+	lfsPointers, err := readLFSPointers(ctx, repo, filteredReader, int(in.Limit))
+	if err != nil {
+		return status.Errorf(codes.Internal, "could not read LFS pointers: %v", err)
+	}
+
+	if err := sliceLFSPointers(lfsPointers, func(slice []*gitalypb.LFSPointer) error {
+		return stream.Send(&gitalypb.ListAllLFSPointersResponse{
+			LfsPointers: slice,
+		})
+	}); err != nil {
+		return err
+	}
+
+	return nil
+}
+
 // GetLFSPointers takes the list of requested blob IDs and filters them down to blobs which are
 // valid LFS pointers. It is fine to pass blob IDs which do not point to a valid LFS pointer, but
 // passing blob IDs which do not exist results in an error.
diff --git a/internal/gitaly/service/blob/lfs_pointers_test.go b/internal/gitaly/service/blob/lfs_pointers_test.go
index 2e0cab071..18d22b4db 100644
--- a/internal/gitaly/service/blob/lfs_pointers_test.go
+++ b/internal/gitaly/service/blob/lfs_pointers_test.go
@@ -6,7 +6,9 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"os"
 	"os/exec"
+	"path/filepath"
 	"strings"
 	"testing"
 
@@ -14,6 +16,7 @@ import (
 	"gitlab.com/gitlab-org/gitaly/internal/git"
 	"gitlab.com/gitlab-org/gitaly/internal/git/gittest"
 	"gitlab.com/gitlab-org/gitaly/internal/git/localrepo"
+	"gitlab.com/gitlab-org/gitaly/internal/helper/text"
 	"gitlab.com/gitlab-org/gitaly/internal/metadata/featureflag"
 	"gitlab.com/gitlab-org/gitaly/internal/testhelper"
 	"gitlab.com/gitlab-org/gitaly/internal/testhelper/testcfg"
@@ -175,6 +178,120 @@ func TestListLFSPointers(t *testing.T) {
 	}
 }
 
+func TestListAllLFSPointers(t *testing.T) {
+	receivePointers := func(t *testing.T, stream gitalypb.BlobService_ListAllLFSPointersClient) []*gitalypb.LFSPointer {
+		t.Helper()
+
+		var pointers []*gitalypb.LFSPointer
+		for {
+			resp, err := stream.Recv()
+			if err == io.EOF {
+				break
+			}
+			require.Nil(t, err)
+			pointers = append(pointers, resp.GetLfsPointers()...)
+		}
+		return pointers
+	}
+
+	ctx, cancel := testhelper.Context()
+	defer cancel()
+
+	lfsPointerContents := `version https://git-lfs.github.com/spec/v1
+oid sha256:1111111111111111111111111111111111111111111111111111111111111111
+size 12345`
+
+	t.Run("normal repository", func(t *testing.T) {
+		_, repo, _, client := setup(t)
+		stream, err := client.ListAllLFSPointers(ctx, &gitalypb.ListAllLFSPointersRequest{
+			Repository: repo,
+		})
+		require.NoError(t, err)
+		require.ElementsMatch(t, []*gitalypb.LFSPointer{
+			lfsPointers[lfsPointer1],
+			lfsPointers[lfsPointer2],
+			lfsPointers[lfsPointer3],
+			lfsPointers[lfsPointer4],
+			lfsPointers[lfsPointer5],
+			lfsPointers[lfsPointer6],
+		}, receivePointers(t, stream))
+	})
+
+	t.Run("dangling LFS pointer", func(t *testing.T) {
+		_, repo, repoPath, client := setup(t)
+
+		lfsPointerOID := text.ChompBytes(testhelper.MustRunCommand(t, strings.NewReader(lfsPointerContents),
+			"git", "-C", repoPath, "hash-object", "-w", "--stdin"))
+
+		stream, err := client.ListAllLFSPointers(ctx, &gitalypb.ListAllLFSPointersRequest{
+			Repository: repo,
+		})
+		require.NoError(t, err)
+		require.ElementsMatch(t, []*gitalypb.LFSPointer{
+			&gitalypb.LFSPointer{
+				Oid:  lfsPointerOID,
+				Data: []byte(lfsPointerContents),
+				Size: int64(len(lfsPointerContents)),
+			},
+			lfsPointers[lfsPointer1],
+			lfsPointers[lfsPointer2],
+			lfsPointers[lfsPointer3],
+			lfsPointers[lfsPointer4],
+			lfsPointers[lfsPointer5],
+			lfsPointers[lfsPointer6],
+		}, receivePointers(t, stream))
+	})
+
+	t.Run("quarantine", func(t *testing.T) {
+		cfg, repoProto, repoPath, client := setup(t)
+
+		// We're emulating the case where git is receiving data via a push, where objects
+		// are stored in a separate quarantine environment. In this case, LFS pointer checks
+		// may want to inspect all newly pushed objects, denoted by a repository proto
+		// message which only has its object directory set to the quarantine directory.
+		quarantineDir := "objects/incoming-123456"
+		require.NoError(t, os.Mkdir(filepath.Join(repoPath, quarantineDir), 0777))
+		repoProto.GitObjectDirectory = quarantineDir
+		repoProto.GitAlternateObjectDirectories = nil
+
+		// There are no quarantined objects yet, so none should be returned here.
+		stream, err := client.ListAllLFSPointers(ctx, &gitalypb.ListAllLFSPointersRequest{
+			Repository: repoProto,
+		})
+		require.NoError(t, err)
+		require.Empty(t, receivePointers(t, stream))
+
+		// Write a new object into the repository. Because we set GIT_OBJECT_DIRECTORY to
+		// the quarantine directory, objects will be written in there instead of into the
+		// repository's normal object directory.
+		repo := localrepo.New(git.NewExecCommandFactory(cfg), repoProto, cfg)
+		var buffer, stderr bytes.Buffer
+		err = repo.ExecAndWait(ctx, git.SubCmd{
+			Name: "hash-object",
+			Flags: []git.Option{
+				git.Flag{Name: "-w"},
+				git.Flag{Name: "--stdin"},
+			},
+		}, git.WithStdin(strings.NewReader(lfsPointerContents)), git.WithStdout(&buffer), git.WithStderr(&stderr))
+		require.NoError(t, err)
+
+		stream, err = client.ListAllLFSPointers(ctx, &gitalypb.ListAllLFSPointersRequest{
+			Repository: repoProto,
+		})
+		require.NoError(t, err)
+
+		// We only expect to find a single LFS pointer, which is the one we've just written
+		// into the quarantine directory.
+		require.ElementsMatch(t, []*gitalypb.LFSPointer{
+			&gitalypb.LFSPointer{
+				Oid:  text.ChompBytes(buffer.Bytes()),
+				Data: []byte(lfsPointerContents),
+				Size: int64(len(lfsPointerContents)),
+			},
+		}, receivePointers(t, stream))
+	})
+}
+
 func TestSuccessfulGetLFSPointersRequest(t *testing.T) {
 	testhelper.NewFeatureSets([]featureflag.FeatureFlag{
 		featureflag.LFSPointersUseBitmapIndex,
diff --git a/proto/blob.proto b/proto/blob.proto
index db7eda834..65cbfe6b3 100644
--- a/proto/blob.proto
+++ b/proto/blob.proto
@@ -67,6 +67,15 @@ service BlobService {
     };
   }
 
+  // ListAllLFSPointers retrieves all LFS pointers in the repository. In
+  // contrast to `GetAllLFSPointers`, this RPC also includes LFS pointers which
+  // are not reachable by any reference.
+  rpc ListAllLFSPointers(ListAllLFSPointersRequest) returns (stream ListAllLFSPointersResponse) {
+    option (op_type) = {
+      op: ACCESSOR
+    };
+  }
+
 }
 
 message GetBlobRequest {
@@ -205,3 +214,18 @@ message ListLFSPointersResponse {
   // LfsPointers is the list of LFS pointers which were requested.
   repeated LFSPointer lfs_pointers = 1;
 }
+
+// ListAllLFSPointersRequest is a request for the ListAllLFSPointers RPC.
+message ListAllLFSPointersRequest {
+  // Repository is the repository for which LFS pointers should be retrieved
+  // from.
+  Repository repository = 1[(target_repository)=true];
+  // Limit limits the number of LFS pointers returned.
+  int32 limit = 3;
+}
+
+// ListAllLFSPointersResponse is a response for the ListAllLFSPointers RPC.
+message ListAllLFSPointersResponse {
+  // LfsPointers is the list of LFS pointers which were requested.
+  repeated LFSPointer lfs_pointers = 1;
+}
diff --git a/proto/go/gitalypb/blob.pb.go b/proto/go/gitalypb/blob.pb.go
index cd5c554a7..3a76fb489 100644
--- a/proto/go/gitalypb/blob.pb.go
+++ b/proto/go/gitalypb/blob.pb.go
@@ -858,6 +858,98 @@ func (m *ListLFSPointersResponse) GetLfsPointers() []*LFSPointer {
 	return nil
 }
 
+// ListAllLFSPointersRequest is a request for the ListAllLFSPointers RPC.
+type ListAllLFSPointersRequest struct {
+	// Repository is the repository for which LFS pointers should be retrieved
+	// from.
+	Repository *Repository `protobuf:"bytes,1,opt,name=repository,proto3" json:"repository,omitempty"`
+	// Limit limits the number of LFS pointers returned.
+	Limit                int32    `protobuf:"varint,3,opt,name=limit,proto3" json:"limit,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *ListAllLFSPointersRequest) Reset()         { *m = ListAllLFSPointersRequest{} }
+func (m *ListAllLFSPointersRequest) String() string { return proto.CompactTextString(m) }
+func (*ListAllLFSPointersRequest) ProtoMessage()    {}
+func (*ListAllLFSPointersRequest) Descriptor() ([]byte, []int) {
+	return fileDescriptor_6903d1e8a20272e8, []int{14}
+}
+
+func (m *ListAllLFSPointersRequest) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_ListAllLFSPointersRequest.Unmarshal(m, b)
+}
+func (m *ListAllLFSPointersRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_ListAllLFSPointersRequest.Marshal(b, m, deterministic)
+}
+func (m *ListAllLFSPointersRequest) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_ListAllLFSPointersRequest.Merge(m, src)
+}
+func (m *ListAllLFSPointersRequest) XXX_Size() int {
+	return xxx_messageInfo_ListAllLFSPointersRequest.Size(m)
+}
+func (m *ListAllLFSPointersRequest) XXX_DiscardUnknown() {
+	xxx_messageInfo_ListAllLFSPointersRequest.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ListAllLFSPointersRequest proto.InternalMessageInfo
+
+func (m *ListAllLFSPointersRequest) GetRepository() *Repository {
+	if m != nil {
+		return m.Repository
+	}
+	return nil
+}
+
+func (m *ListAllLFSPointersRequest) GetLimit() int32 {
+	if m != nil {
+		return m.Limit
+	}
+	return 0
+}
+
+// ListAllLFSPointersResponse is a response for the ListAllLFSPointers RPC.
+type ListAllLFSPointersResponse struct {
+	// LfsPointers is the list of LFS pointers which were requested.
+	LfsPointers          []*LFSPointer `protobuf:"bytes,1,rep,name=lfs_pointers,json=lfsPointers,proto3" json:"lfs_pointers,omitempty"`
+	XXX_NoUnkeyedLiteral struct{}      `json:"-"`
+	XXX_unrecognized     []byte        `json:"-"`
+	XXX_sizecache        int32         `json:"-"`
+}
+
+func (m *ListAllLFSPointersResponse) Reset()         { *m = ListAllLFSPointersResponse{} }
+func (m *ListAllLFSPointersResponse) String() string { return proto.CompactTextString(m) }
+func (*ListAllLFSPointersResponse) ProtoMessage()    {}
+func (*ListAllLFSPointersResponse) Descriptor() ([]byte, []int) {
+	return fileDescriptor_6903d1e8a20272e8, []int{15}
+}
+
+func (m *ListAllLFSPointersResponse) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_ListAllLFSPointersResponse.Unmarshal(m, b)
+}
+func (m *ListAllLFSPointersResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_ListAllLFSPointersResponse.Marshal(b, m, deterministic)
+}
+func (m *ListAllLFSPointersResponse) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_ListAllLFSPointersResponse.Merge(m, src)
+}
+func (m *ListAllLFSPointersResponse) XXX_Size() int {
+	return xxx_messageInfo_ListAllLFSPointersResponse.Size(m)
+}
+func (m *ListAllLFSPointersResponse) XXX_DiscardUnknown() {
+	xxx_messageInfo_ListAllLFSPointersResponse.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ListAllLFSPointersResponse proto.InternalMessageInfo
+
+func (m *ListAllLFSPointersResponse) GetLfsPointers() []*LFSPointer {
+	if m != nil {
+		return m.LfsPointers
+	}
+	return nil
+}
+
 func init() {
 	proto.RegisterType((*GetBlobRequest)(nil), "gitaly.GetBlobRequest")
 	proto.RegisterType((*GetBlobResponse)(nil), "gitaly.GetBlobResponse")
@@ -874,58 +966,62 @@ func init() {
 	proto.RegisterType((*GetAllLFSPointersResponse)(nil), "gitaly.GetAllLFSPointersResponse")
 	proto.RegisterType((*ListLFSPointersRequest)(nil), "gitaly.ListLFSPointersRequest")
 	proto.RegisterType((*ListLFSPointersResponse)(nil), "gitaly.ListLFSPointersResponse")
+	proto.RegisterType((*ListAllLFSPointersRequest)(nil), "gitaly.ListAllLFSPointersRequest")
+	proto.RegisterType((*ListAllLFSPointersResponse)(nil), "gitaly.ListAllLFSPointersResponse")
 }
 
 func init() { proto.RegisterFile("blob.proto", fileDescriptor_6903d1e8a20272e8) }
 
 var fileDescriptor_6903d1e8a20272e8 = []byte{
-	// 724 bytes of a gzipped FileDescriptorProto
+	// 756 bytes of a gzipped FileDescriptorProto
 	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xac, 0x56, 0xcd, 0x6e, 0xd3, 0x40,
 	0x10, 0x96, 0xe3, 0x24, 0x75, 0x26, 0xe9, 0x0f, 0x2b, 0x68, 0x5d, 0xab, 0x14, 0xd7, 0x42, 0xc8,
-	0x07, 0x48, 0xaa, 0x22, 0x24, 0x4e, 0x48, 0xad, 0x50, 0xa3, 0xd2, 0xaa, 0xad, 0xb6, 0x5c, 0xa8,
-	0x90, 0x22, 0xbb, 0xd9, 0xb4, 0x8b, 0x36, 0x5e, 0xe3, 0xdd, 0xb6, 0x0a, 0x0f, 0xc0, 0x2b, 0xc0,
-	0x85, 0xb7, 0x41, 0xbc, 0x04, 0xef, 0xc0, 0x03, 0x70, 0x42, 0xfe, 0x8b, 0x9d, 0xd8, 0xe1, 0x92,
-	0xdc, 0x76, 0x66, 0x3c, 0xf3, 0x7d, 0x33, 0xf3, 0xed, 0x26, 0x00, 0x2e, 0xe3, 0x6e, 0xdb, 0x0f,
-	0xb8, 0xe4, 0xa8, 0x7e, 0x4d, 0xa5, 0xc3, 0x46, 0x06, 0x30, 0xea, 0xc9, 0xd8, 0x67, 0xb4, 0xc4,
-	0x8d, 0x13, 0x90, 0x7e, 0x6c, 0x59, 0x01, 0xac, 0x74, 0x89, 0x3c, 0x60, 0xdc, 0xc5, 0xe4, 0xf3,
-	0x2d, 0x11, 0x12, 0xbd, 0x06, 0x08, 0x88, 0xcf, 0x05, 0x95, 0x3c, 0x18, 0xe9, 0x8a, 0xa9, 0xd8,
-	0xcd, 0x3d, 0xd4, 0x8e, 0x0b, 0xb5, 0xf1, 0x38, 0x72, 0x50, 0xfd, 0xfe, 0xeb, 0xb9, 0x82, 0x73,
-	0xdf, 0xa2, 0x35, 0x50, 0x39, 0xed, 0xeb, 0x15, 0x53, 0xb1, 0x1b, 0x38, 0x3c, 0xa2, 0x87, 0x50,
-	0x63, 0x74, 0x48, 0xa5, 0xae, 0x9a, 0x8a, 0xad, 0xe2, 0xd8, 0xb0, 0x8e, 0x61, 0x75, 0x8c, 0x29,
-	0x7c, 0xee, 0x09, 0x82, 0x10, 0x54, 0x05, 0xfd, 0x42, 0x22, 0x38, 0x15, 0x47, 0xe7, 0xd0, 0xd7,
-	0x77, 0xa4, 0x13, 0xd5, 0x6b, 0xe1, 0xe8, 0x9c, 0x42, 0xa8, 0x63, 0x08, 0xeb, 0x8f, 0x32, 0xae,
-	0x26, 0xe6, 0x6f, 0xe1, 0x18, 0x56, 0x02, 0x72, 0x47, 0x05, 0xe5, 0x5e, 0xcf, 0x77, 0xe4, 0x8d,
-	0xd0, 0x2b, 0xa6, 0x6a, 0x37, 0xf7, 0x9e, 0xa6, 0xd9, 0x53, 0x50, 0x6d, 0x9c, 0x7c, 0x7d, 0xee,
-	0xc8, 0x1b, 0xbc, 0x1c, 0xe4, 0x2c, 0x51, 0xde, 0xbd, 0xf1, 0x06, 0x5a, 0xf9, 0x24, 0x64, 0x80,
-	0x96, 0xa6, 0x45, 0x54, 0x1b, 0x78, 0x6c, 0x87, 0x23, 0x08, 0x59, 0xa4, 0x23, 0x08, 0xcf, 0xd6,
-	0x6f, 0x05, 0xd6, 0x32, 0x16, 0xf3, 0xce, 0x0f, 0xed, 0x40, 0x8b, 0x8a, 0x9e, 0xb8, 0x75, 0x87,
-	0xbc, 0x7f, 0xcb, 0x88, 0x5e, 0x35, 0x15, 0x5b, 0xc3, 0x4d, 0x2a, 0x2e, 0x52, 0x57, 0x58, 0x68,
-	0xc8, 0xfb, 0x44, 0xaf, 0x99, 0x8a, 0x5d, 0xc3, 0xd1, 0x79, 0x82, 0x75, 0x7d, 0x06, 0xeb, 0xa5,
-	0x8c, 0x35, 0x7a, 0x06, 0x55, 0x39, 0xf2, 0x89, 0xae, 0x99, 0x8a, 0xbd, 0x92, 0x2d, 0xe3, 0xcc,
-	0xfd, 0x44, 0xae, 0xe4, 0xfb, 0x91, 0x4f, 0x70, 0x14, 0xb7, 0x0e, 0x01, 0x4e, 0x0e, 0x2f, 0xce,
-	0x39, 0xf5, 0x24, 0x09, 0xe6, 0x90, 0xc5, 0x11, 0x2c, 0x9f, 0x92, 0xfb, 0x70, 0x48, 0x31, 0x44,
-	0x69, 0xa9, 0xa2, 0x60, 0x53, 0xea, 0x6a, 0x6e, 0xe0, 0x0c, 0x1e, 0x75, 0x89, 0xcc, 0x58, 0x2d,
-	0x40, 0x66, 0x9b, 0xa0, 0x85, 0xb7, 0xb4, 0x47, 0xfb, 0xb1, 0xc0, 0x1a, 0x78, 0x29, 0xb4, 0x8f,
-	0xfa, 0xc2, 0x3a, 0x83, 0xf5, 0x69, 0xb4, 0x64, 0xc7, 0xaf, 0xa0, 0xc5, 0x06, 0xa2, 0xe7, 0x27,
-	0x7e, 0x5d, 0x89, 0x94, 0x39, 0x06, 0xcc, 0x52, 0x70, 0x93, 0x0d, 0x44, 0x9a, 0x6e, 0xfd, 0x54,
-	0x40, 0xef, 0x12, 0x79, 0x4a, 0xee, 0x17, 0xda, 0x42, 0x5e, 0x00, 0xf1, 0x2a, 0x32, 0x01, 0x4c,
-	0x08, 0xbf, 0x96, 0x08, 0x1f, 0x6d, 0x01, 0x78, 0x5c, 0xf6, 0xa8, 0xd7, 0x73, 0x18, 0x4b, 0x74,
-	0xa6, 0x79, 0x5c, 0x1e, 0x79, 0xfb, 0x8c, 0xa1, 0x6d, 0x68, 0x26, 0xd1, 0x80, 0x0c, 0x84, 0x5e,
-	0x33, 0x55, 0xbb, 0x85, 0x1b, 0x51, 0x18, 0x93, 0x81, 0xb0, 0x30, 0x6c, 0x96, 0x74, 0x31, 0xdf,
-	0x68, 0x2e, 0xa3, 0xc9, 0xec, 0x33, 0xb6, 0xc8, 0xc9, 0xbc, 0xab, 0x6a, 0x95, 0x35, 0x35, 0xe1,
-	0x3b, 0x5d, 0x7b, 0x3e, 0xbe, 0x5f, 0x15, 0x58, 0x3f, 0xa1, 0x62, 0xb1, 0x5a, 0xdc, 0x82, 0x46,
-	0xba, 0xb8, 0x54, 0x8c, 0x99, 0xa3, 0x7c, 0x95, 0xd6, 0x39, 0x6c, 0x14, 0x78, 0xcc, 0xd5, 0xda,
-	0xde, 0x8f, 0x2a, 0x34, 0xc3, 0xdb, 0x7a, 0x41, 0x82, 0x3b, 0x7a, 0x45, 0xd0, 0x5b, 0x58, 0x4a,
-	0x1e, 0x39, 0xb4, 0x3e, 0xf5, 0xf6, 0x26, 0x2d, 0x1b, 0x1b, 0x05, 0x7f, 0x4c, 0xc1, 0xaa, 0xff,
-	0xfd, 0x66, 0x57, 0xb4, 0xca, 0xae, 0x82, 0xba, 0xa0, 0xa5, 0x4f, 0x25, 0xda, 0x98, 0xf1, 0x84,
-	0x1b, 0x7a, 0x31, 0x50, 0x28, 0xf4, 0x21, 0xfa, 0x99, 0xcc, 0xf5, 0x8b, 0x1e, 0xe7, 0xb2, 0x8a,
-	0xfb, 0x30, 0xb6, 0x67, 0x85, 0x0b, 0xa5, 0x5d, 0x78, 0x50, 0x10, 0x36, 0x32, 0x73, 0xe9, 0xa5,
-	0x37, 0xd7, 0xd8, 0xf9, 0xcf, 0x17, 0x33, 0x30, 0x26, 0xc5, 0x38, 0x81, 0x51, 0x7a, 0x07, 0x26,
-	0x30, 0xca, 0x95, 0x9c, 0xc3, 0xf8, 0x08, 0xab, 0x53, 0x9a, 0x40, 0xe3, 0x21, 0x94, 0x8b, 0xd6,
-	0x78, 0x32, 0x33, 0x3e, 0x5d, 0xfd, 0x60, 0xf7, 0x32, 0xfc, 0x96, 0x39, 0x6e, 0xfb, 0x8a, 0x0f,
-	0x3b, 0xf1, 0xf1, 0x05, 0x0f, 0xae, 0x3b, 0x71, 0x85, 0x4e, 0xf4, 0x6f, 0xa6, 0x73, 0xcd, 0x13,
-	0xdb, 0x77, 0xdd, 0x7a, 0xe4, 0x7a, 0xf9, 0x2f, 0x00, 0x00, 0xff, 0xff, 0xab, 0x7a, 0x42, 0x65,
-	0x10, 0x09, 0x00, 0x00,
+	0x07, 0x48, 0xaa, 0x22, 0x24, 0x4e, 0x48, 0xad, 0x50, 0xa3, 0xd2, 0xaa, 0xad, 0x36, 0x5c, 0xa8,
+	0x90, 0x22, 0xbb, 0xd9, 0xb4, 0x0b, 0x1b, 0xaf, 0xf1, 0x6e, 0x5b, 0x85, 0x33, 0xe2, 0x15, 0xe0,
+	0x81, 0x10, 0x2f, 0xc1, 0x3b, 0xf0, 0x00, 0x9c, 0x90, 0xff, 0x12, 0x27, 0x76, 0x7a, 0x49, 0x6e,
+	0xbb, 0x33, 0x3b, 0xf3, 0x7d, 0x33, 0xf3, 0x79, 0xd7, 0x00, 0x2e, 0xe3, 0x6e, 0xd3, 0x0f, 0xb8,
+	0xe4, 0xa8, 0x7a, 0x45, 0xa5, 0xc3, 0x86, 0x06, 0x30, 0xea, 0xc9, 0xd8, 0x66, 0x34, 0xc4, 0xb5,
+	0x13, 0x90, 0x5e, 0xbc, 0xb3, 0x02, 0x58, 0x69, 0x13, 0x79, 0xc0, 0xb8, 0x8b, 0xc9, 0x97, 0x1b,
+	0x22, 0x24, 0x7a, 0x0d, 0x10, 0x10, 0x9f, 0x0b, 0x2a, 0x79, 0x30, 0xd4, 0x15, 0x53, 0xb1, 0xeb,
+	0x7b, 0xa8, 0x19, 0x27, 0x6a, 0xe2, 0x91, 0xe7, 0xa0, 0xfc, 0xf3, 0xf7, 0x73, 0x05, 0x67, 0xce,
+	0xa2, 0x35, 0x50, 0x39, 0xed, 0xe9, 0x25, 0x53, 0xb1, 0x6b, 0x38, 0x5c, 0xa2, 0x87, 0x50, 0x61,
+	0x74, 0x40, 0xa5, 0xae, 0x9a, 0x8a, 0xad, 0xe2, 0x78, 0x63, 0x1d, 0xc3, 0xea, 0x08, 0x53, 0xf8,
+	0xdc, 0x13, 0x04, 0x21, 0x28, 0x0b, 0xfa, 0x95, 0x44, 0x70, 0x2a, 0x8e, 0xd6, 0xa1, 0xad, 0xe7,
+	0x48, 0x27, 0xca, 0xd7, 0xc0, 0xd1, 0x3a, 0x85, 0x50, 0x47, 0x10, 0xd6, 0x5f, 0x65, 0x94, 0x4d,
+	0xcc, 0x5f, 0xc2, 0x31, 0xac, 0x04, 0xe4, 0x96, 0x0a, 0xca, 0xbd, 0xae, 0xef, 0xc8, 0x6b, 0xa1,
+	0x97, 0x4c, 0xd5, 0xae, 0xef, 0x3d, 0x4d, 0xa3, 0xa7, 0xa0, 0x9a, 0x38, 0x39, 0x7d, 0xee, 0xc8,
+	0x6b, 0xbc, 0x1c, 0x64, 0x76, 0xa2, 0xb8, 0x7a, 0xe3, 0x0d, 0x34, 0xb2, 0x41, 0xc8, 0x00, 0x2d,
+	0x0d, 0x8b, 0xa8, 0xd6, 0xf0, 0x68, 0x1f, 0xb6, 0x20, 0x64, 0x91, 0xb6, 0x20, 0x5c, 0x5b, 0x7f,
+	0x14, 0x58, 0x1b, 0xb3, 0x98, 0xb7, 0x7f, 0x68, 0x07, 0x1a, 0x54, 0x74, 0xc5, 0x8d, 0x3b, 0xe0,
+	0xbd, 0x1b, 0x46, 0xf4, 0xb2, 0xa9, 0xd8, 0x1a, 0xae, 0x53, 0xd1, 0x49, 0x4d, 0x61, 0xa2, 0x01,
+	0xef, 0x11, 0xbd, 0x62, 0x2a, 0x76, 0x05, 0x47, 0xeb, 0x09, 0xd6, 0xd5, 0x19, 0xac, 0x97, 0xc6,
+	0xac, 0xd1, 0x33, 0x28, 0xcb, 0xa1, 0x4f, 0x74, 0xcd, 0x54, 0xec, 0x95, 0xf1, 0x30, 0xce, 0xdc,
+	0x4f, 0xe4, 0x52, 0xbe, 0x1f, 0xfa, 0x04, 0x47, 0x7e, 0xeb, 0x10, 0xe0, 0xe4, 0xb0, 0x73, 0xce,
+	0xa9, 0x27, 0x49, 0x30, 0x87, 0x2c, 0x8e, 0x60, 0xf9, 0x94, 0xdc, 0x85, 0x4d, 0x8a, 0x21, 0x0a,
+	0x53, 0xe5, 0x05, 0x9b, 0x52, 0x57, 0x33, 0x0d, 0x67, 0xf0, 0xa8, 0x4d, 0xe4, 0x98, 0xd5, 0x02,
+	0x64, 0xb6, 0x09, 0x5a, 0xf8, 0x95, 0x76, 0x69, 0x2f, 0x16, 0x58, 0x0d, 0x2f, 0x85, 0xfb, 0xa3,
+	0x9e, 0xb0, 0xce, 0x60, 0x7d, 0x1a, 0x2d, 0x99, 0xf1, 0x2b, 0x68, 0xb0, 0xbe, 0xe8, 0xfa, 0x89,
+	0x5d, 0x57, 0x22, 0x65, 0x8e, 0x00, 0xc7, 0x21, 0xb8, 0xce, 0xfa, 0x22, 0x0d, 0xb7, 0x7e, 0x29,
+	0xa0, 0xb7, 0x89, 0x3c, 0x25, 0x77, 0x0b, 0x2d, 0x21, 0x2b, 0x80, 0x78, 0x14, 0x63, 0x01, 0x4c,
+	0x08, 0xbf, 0x92, 0x08, 0x1f, 0x6d, 0x01, 0x78, 0x5c, 0x76, 0xa9, 0xd7, 0x75, 0x18, 0x4b, 0x74,
+	0xa6, 0x79, 0x5c, 0x1e, 0x79, 0xfb, 0x8c, 0xa1, 0x6d, 0xa8, 0x27, 0xde, 0x80, 0xf4, 0x85, 0x5e,
+	0x31, 0x55, 0xbb, 0x81, 0x6b, 0x91, 0x1b, 0x93, 0xbe, 0xb0, 0x30, 0x6c, 0x16, 0x54, 0x31, 0x5f,
+	0x6b, 0x2e, 0xa2, 0xce, 0xec, 0x33, 0xb6, 0xc8, 0xce, 0xbc, 0x2b, 0x6b, 0xa5, 0x35, 0x35, 0xe1,
+	0x3b, 0x9d, 0x7b, 0x3e, 0xbe, 0xdf, 0x15, 0x58, 0x3f, 0xa1, 0x62, 0xb1, 0x5a, 0xdc, 0x82, 0x5a,
+	0x3a, 0xb8, 0x54, 0x8c, 0x63, 0x43, 0xf1, 0x28, 0xad, 0x73, 0xd8, 0xc8, 0xf1, 0x98, 0xaf, 0xb4,
+	0xcf, 0xb0, 0x19, 0x66, 0x5c, 0xf0, 0x2c, 0x66, 0xd0, 0xef, 0x80, 0x51, 0x04, 0x36, 0x57, 0x05,
+	0x7b, 0xdf, 0x2a, 0x50, 0x0f, 0xef, 0x9b, 0x0e, 0x09, 0x6e, 0xe9, 0x25, 0x41, 0x6f, 0x61, 0x29,
+	0xb9, 0xa6, 0xd1, 0xfa, 0xd4, 0xeb, 0x91, 0xd4, 0x65, 0x6c, 0xe4, 0xec, 0x31, 0x05, 0xab, 0xfa,
+	0xef, 0x87, 0x5d, 0xd2, 0x4a, 0xbb, 0x0a, 0x6a, 0x83, 0x96, 0x5e, 0xf6, 0x68, 0x63, 0xc6, 0x23,
+	0x64, 0xe8, 0x79, 0x47, 0x2e, 0xd1, 0x87, 0xe8, 0xa1, 0xcf, 0xd4, 0x8b, 0x1e, 0x67, 0xa2, 0xf2,
+	0x4d, 0x37, 0xb6, 0x67, 0xb9, 0x73, 0xa9, 0x5d, 0x78, 0x90, 0xfb, 0x34, 0x91, 0x99, 0x09, 0x2f,
+	0xbc, 0x7b, 0x8c, 0x9d, 0x7b, 0x4e, 0xcc, 0xc0, 0x98, 0x9c, 0xd8, 0x04, 0x46, 0xa1, 0x72, 0x26,
+	0x30, 0x8a, 0xc7, 0x9d, 0xc1, 0xf8, 0x08, 0xab, 0x53, 0xaa, 0x46, 0xa3, 0x26, 0x14, 0x7f, 0x76,
+	0xc6, 0x93, 0x99, 0xfe, 0x5c, 0x76, 0x02, 0x28, 0x2f, 0x3a, 0xb4, 0x93, 0x4d, 0x50, 0x5c, 0x83,
+	0x75, 0xdf, 0x91, 0x69, 0x98, 0x83, 0xdd, 0x8b, 0xf0, 0x38, 0x73, 0xdc, 0xe6, 0x25, 0x1f, 0xb4,
+	0xe2, 0xe5, 0x0b, 0x1e, 0x5c, 0xb5, 0xe2, 0x24, 0xad, 0xe8, 0xb7, 0xaf, 0x75, 0xc5, 0x93, 0xbd,
+	0xef, 0xba, 0xd5, 0xc8, 0xf4, 0xf2, 0x7f, 0x00, 0x00, 0x00, 0xff, 0xff, 0xbd, 0x71, 0xf0, 0xaa,
+	0x39, 0x0a, 0x00, 0x00,
 }
 
 // Reference imports to suppress errors if they are not otherwise used.
@@ -970,6 +1066,10 @@ type BlobServiceClient interface {
 	// transitively reference any LFS pointers are ignored. It is not valid to
 	// pass revisions which do not resolve to an existing object.
 	ListLFSPointers(ctx context.Context, in *ListLFSPointersRequest, opts ...grpc.CallOption) (BlobService_ListLFSPointersClient, error)
+	// ListAllLFSPointers retrieves all LFS pointers in the repository. In
+	// contrast to `GetAllLFSPointers`, this RPC also includes LFS pointers which
+	// are not reachable by any reference.
+	ListAllLFSPointers(ctx context.Context, in *ListAllLFSPointersRequest, opts ...grpc.CallOption) (BlobService_ListAllLFSPointersClient, error)
 }
 
 type blobServiceClient struct {
@@ -1172,6 +1272,38 @@ func (x *blobServiceListLFSPointersClient) Recv() (*ListLFSPointersResponse, err
 	return m, nil
 }
 
+func (c *blobServiceClient) ListAllLFSPointers(ctx context.Context, in *ListAllLFSPointersRequest, opts ...grpc.CallOption) (BlobService_ListAllLFSPointersClient, error) {
+	stream, err := c.cc.NewStream(ctx, &_BlobService_serviceDesc.Streams[6], "/gitaly.BlobService/ListAllLFSPointers", opts...)
+	if err != nil {
+		return nil, err
+	}
+	x := &blobServiceListAllLFSPointersClient{stream}
+	if err := x.ClientStream.SendMsg(in); err != nil {
+		return nil, err
+	}
+	if err := x.ClientStream.CloseSend(); err != nil {
+		return nil, err
+	}
+	return x, nil
+}
+
+type BlobService_ListAllLFSPointersClient interface {
+	Recv() (*ListAllLFSPointersResponse, error)
+	grpc.ClientStream
+}
+
+type blobServiceListAllLFSPointersClient struct {
+	grpc.ClientStream
+}
+
+func (x *blobServiceListAllLFSPointersClient) Recv() (*ListAllLFSPointersResponse, error) {
+	m := new(ListAllLFSPointersResponse)
+	if err := x.ClientStream.RecvMsg(m); err != nil {
+		return nil, err
+	}
+	return m, nil
+}
+
 // BlobServiceServer is the server API for BlobService service.
 type BlobServiceServer interface {
 	// GetBlob returns the contents of a blob object referenced by its object
@@ -1204,6 +1336,10 @@ type BlobServiceServer interface {
 	// transitively reference any LFS pointers are ignored. It is not valid to
 	// pass revisions which do not resolve to an existing object.
 	ListLFSPointers(*ListLFSPointersRequest, BlobService_ListLFSPointersServer) error
+	// ListAllLFSPointers retrieves all LFS pointers in the repository. In
+	// contrast to `GetAllLFSPointers`, this RPC also includes LFS pointers which
+	// are not reachable by any reference.
+	ListAllLFSPointers(*ListAllLFSPointersRequest, BlobService_ListAllLFSPointersServer) error
 }
 
 // UnimplementedBlobServiceServer can be embedded to have forward compatible implementations.
@@ -1228,6 +1364,9 @@ func (*UnimplementedBlobServiceServer) GetAllLFSPointers(req *GetAllLFSPointersR
 func (*UnimplementedBlobServiceServer) ListLFSPointers(req *ListLFSPointersRequest, srv BlobService_ListLFSPointersServer) error {
 	return status.Errorf(codes.Unimplemented, "method ListLFSPointers not implemented")
 }
+func (*UnimplementedBlobServiceServer) ListAllLFSPointers(req *ListAllLFSPointersRequest, srv BlobService_ListAllLFSPointersServer) error {
+	return status.Errorf(codes.Unimplemented, "method ListAllLFSPointers not implemented")
+}
 
 func RegisterBlobServiceServer(s *grpc.Server, srv BlobServiceServer) {
 	s.RegisterService(&_BlobService_serviceDesc, srv)
@@ -1359,6 +1498,27 @@ func (x *blobServiceListLFSPointersServer) Send(m *ListLFSPointersResponse) erro
 	return x.ServerStream.SendMsg(m)
 }
 
+func _BlobService_ListAllLFSPointers_Handler(srv interface{}, stream grpc.ServerStream) error {
+	m := new(ListAllLFSPointersRequest)
+	if err := stream.RecvMsg(m); err != nil {
+		return err
+	}
+	return srv.(BlobServiceServer).ListAllLFSPointers(m, &blobServiceListAllLFSPointersServer{stream})
+}
+
+type BlobService_ListAllLFSPointersServer interface {
+	Send(*ListAllLFSPointersResponse) error
+	grpc.ServerStream
+}
+
+type blobServiceListAllLFSPointersServer struct {
+	grpc.ServerStream
+}
+
+func (x *blobServiceListAllLFSPointersServer) Send(m *ListAllLFSPointersResponse) error {
+	return x.ServerStream.SendMsg(m)
+}
+
 var _BlobService_serviceDesc = grpc.ServiceDesc{
 	ServiceName: "gitaly.BlobService",
 	HandlerType: (*BlobServiceServer)(nil),
@@ -1394,6 +1554,11 @@ var _BlobService_serviceDesc = grpc.ServiceDesc{
 			Handler:       _BlobService_ListLFSPointers_Handler,
 			ServerStreams: true,
 		},
+		{
+			StreamName:    "ListAllLFSPointers",
+			Handler:       _BlobService_ListAllLFSPointers_Handler,
+			ServerStreams: true,
+		},
 	},
 	Metadata: "blob.proto",
 }
diff --git a/ruby/proto/gitaly/blob_pb.rb b/ruby/proto/gitaly/blob_pb.rb
index d950765a4..e6e6b1ed6 100644
--- a/ruby/proto/gitaly/blob_pb.rb
+++ b/ruby/proto/gitaly/blob_pb.rb
@@ -77,6 +77,13 @@ Google::Protobuf::DescriptorPool.generated_pool.build do
     add_message "gitaly.ListLFSPointersResponse" do
       repeated :lfs_pointers, :message, 1, "gitaly.LFSPointer"
     end
+    add_message "gitaly.ListAllLFSPointersRequest" do
+      optional :repository, :message, 1, "gitaly.Repository"
+      optional :limit, :int32, 3
+    end
+    add_message "gitaly.ListAllLFSPointersResponse" do
+      repeated :lfs_pointers, :message, 1, "gitaly.LFSPointer"
+    end
   end
 end
 
@@ -96,4 +103,6 @@ module Gitaly
   GetAllLFSPointersResponse = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitaly.GetAllLFSPointersResponse").msgclass
   ListLFSPointersRequest = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitaly.ListLFSPointersRequest").msgclass
   ListLFSPointersResponse = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitaly.ListLFSPointersResponse").msgclass
+  ListAllLFSPointersRequest = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitaly.ListAllLFSPointersRequest").msgclass
+  ListAllLFSPointersResponse = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("gitaly.ListAllLFSPointersResponse").msgclass
 end
diff --git a/ruby/proto/gitaly/blob_services_pb.rb b/ruby/proto/gitaly/blob_services_pb.rb
index 72ecd6a2e..21a3bc6e5 100644
--- a/ruby/proto/gitaly/blob_services_pb.rb
+++ b/ruby/proto/gitaly/blob_services_pb.rb
@@ -44,6 +44,10 @@ module Gitaly
       # transitively reference any LFS pointers are ignored. It is not valid to
       # pass revisions which do not resolve to an existing object.
       rpc :ListLFSPointers, Gitaly::ListLFSPointersRequest, stream(Gitaly::ListLFSPointersResponse)
+      # ListAllLFSPointers retrieves all LFS pointers in the repository. In
+      # contrast to `GetAllLFSPointers`, this RPC also includes LFS pointers which
+      # are not reachable by any reference.
+      rpc :ListAllLFSPointers, Gitaly::ListAllLFSPointersRequest, stream(Gitaly::ListAllLFSPointersResponse)
     end
 
     Stub = Service.rpc_stub_class