diff options
author | Sami Hiltunen <shiltunen@gitlab.com> | 2021-12-17 12:28:58 +0300 |
---|---|---|
committer | Sami Hiltunen <shiltunen@gitlab.com> | 2021-12-17 12:43:27 +0300 |
commit | 8f4dbe26d8a9fda9e95686be917c67464c12381c (patch) | |
tree | 979212e70dd84bf5d9a45c1349c7db00d0641819 | |
parent | 61471fff6a262bc2c9c5270016f8a677e48638bb (diff) |
Add versioning to Praefect disk storage path schemassmh-version-replica-paths
Praefect is about to begin generating unique relative paths for
repositories to avoid on disk conflicts. The current format of
`@repositories/xx/yy/<repo-id>` is sufficient for now and guarantees
unique storage directories for repositories. The uniqueness of the paths
is guaranteed by the repository ID sequence stored in Postgres. While
this is sufficient for now, it doesn't leave room for evolving the schema of
paths in the future. For example, if we wanted to ensure each replica lands
in a unique directory to avoid the same races we had with repositories
landing in the same directories, we'd have to define a new top level directory
in the storage to avoid conflicts with the existing paths. Having to define
new top level directories in the storage might be more confusing than having
a single Praefect owned directory that can contain the disk path changes.
This commit changes the top level directory in the storage from @repositories
to @praefect. This clearly marks the repository as Praefect owned. A version
number is added in to the path to allow for changing the expectations of the
uniquness of the directories. For example, when we in future make replicas land
in unique directories as well, we can do so by incrementing the path format
version and giving the replicas paths like `@praefect/v2/xx/yy/<replica_id>`.
If we didn't increment the version, the replica ids could conflict with the
repository ids in the current path schema.
Praefect is not yet generating unique paths, so this is a good time to change
the format before creating the @repositories directory in the storages. Nothing
would break even if we did but we can directly go to a format that can handle
evolving the schema in the future.
-rw-r--r-- | internal/praefect/praefectutil/replica_path.go | 7 | ||||
-rw-r--r-- | internal/praefect/praefectutil/replica_path_test.go | 4 |
2 files changed, 7 insertions, 4 deletions
diff --git a/internal/praefect/praefectutil/replica_path.go b/internal/praefect/praefectutil/replica_path.go index 5f35e1e81..2b93f65c4 100644 --- a/internal/praefect/praefectutil/replica_path.go +++ b/internal/praefect/praefectutil/replica_path.go @@ -8,7 +8,10 @@ import ( // DeriveReplicaPath derives a repository's disk storage path from its repository ID. The repository ID // is hashed with SHA256 and the first four hex digits of the hash are used as the two subdirectories to -// ensure even distribution into subdirectories. The format is @repositories/ab/cd/<repository-id>. +// ensure even distribution into subdirectories. The format is @praefect/v<version number>/ab/cd/<repository-id>. +// The prefix '@praefect' marks the directory clearly as Praefect owned. The <version> component leaves room for +// evolving our disk storage schema in future. If we make changes that may conflict with existing storage paths, +// we can simply increment the version. func DeriveReplicaPath(repositoryID int64) string { hasher := sha256.New() // String representation of the ID is used to make it easier to derive the replica paths with @@ -16,5 +19,5 @@ func DeriveReplicaPath(repositoryID int64) string { // an error. hasher.Write([]byte(strconv.FormatInt(repositoryID, 10))) hash := hasher.Sum(nil) - return fmt.Sprintf("@repositories/%x/%x/%d", hash[0:1], hash[1:2], repositoryID) + return fmt.Sprintf("@praefect/v1/%x/%x/%d", hash[0:1], hash[1:2], repositoryID) } diff --git a/internal/praefect/praefectutil/replica_path_test.go b/internal/praefect/praefectutil/replica_path_test.go index 6084c13ae..4761d8440 100644 --- a/internal/praefect/praefectutil/replica_path_test.go +++ b/internal/praefect/praefectutil/replica_path_test.go @@ -7,6 +7,6 @@ import ( ) func TestDeriveReplicaPath(t *testing.T) { - require.Equal(t, "@repositories/6b/86/1", DeriveReplicaPath(1)) - require.Equal(t, "@repositories/d4/73/2", DeriveReplicaPath(2)) + require.Equal(t, "@praefect/v1/6b/86/1", DeriveReplicaPath(1)) + require.Equal(t, "@praefect/v1/d4/73/2", DeriveReplicaPath(2)) } |